Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose a method to infer the arithmetic expressions performed by a language model while solving a mathematical reasoning question with a chain-of-thought style output. The inferred expressions are then calculated outside of the LM. This aims to improve the performance of LLMs on mathematical reasoning tasks by increasing the accuracy on arithmetic calculations. 
strength_and_weaknesses: Strength:

- The method itself appears to be novel. It involves extracting the relevant numbers and substituting large numbers in the chain-of-thought answer with random small numbers where the LM can reliably perform calculations on. By sampling a set of these input and output correspondences, the authors specify and solve a linear system of equations to elicit the arithmetic relationships. 

- The method leads to performance gains in the settings evaluated in this paper.

Weakness:

- The main issue with this work is that this method is a very complicated way to do something that is much simpler in comparison (simple arithmetic calculations). Despite its complexity, the method does not guarantee that the correct calculations are extracted and does not scale well to more complex expressions. It seems that similar performance gains can be achieved by prompting the LLMs to output its reasoning steps in the format of mathematical expressions, and sending the evaluation to an external calculator. This approach was already explored in [1] with some success. One can also imagine finetuning the model to output well-formatted arithmetic expressions in a task-agnostic way. A version of this was explored in [2] on the GSM8k dataset. These alternatives are much simpler and more general than the proposed approach, and the paper has not benchmarked against these simpler approaches to show that it offers a meaningful improvement. 

- The idea of formalizing the reasoning steps of a LM through perturbing its output and inferring the relationships may be an interesting framework, but there is no discussion of extending this particular approach to something that cannot be readily achieved by simple methods. 

[1] https://arxiv.org/pdf/2201.11903.pdf

[2] https://arxiv.org/pdf/2110.14168.pdf
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and it's nice that the paper includes a discussion on its limitations. However, there are important benchmarks missing and thus the evaluation is incomplete. 
summary_of_the_review: The paper presents an interesting idea for eliciting the mathematical expressions that the LM is implicitly computing, and offloading such computations to an external calculator. However, the proposed approach is extremely complicated compared to the problem that it's trying to solve, and does not generalize well to other situations nor scale to more complex arithmetic expressions. There are simpler and more general approaches that are likely to perform well and these alternatives are not benchmarked. Therefore I think the current paper does not convincingly demonstrate the value of the proposed approach.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors introduced a clever and simple method that improves the arithmetic capabilities of LMs.
strength_and_weaknesses: The method is easy to understand and implement and offers improvements on various reasoning datasets. It needs to be clarified, though, whether there exists a path toward applications beyond systems of linear equations. I also would like to know whether the method helps when combined with majority voting.

The Drop dataset used for evaluations in this paper is excellent, but its arithmetic is quite restricted. Are there any non-synthetic NLP datasets that have systems of linear equations? How much gain would we see on GSM8K and MATH? Can you perform a qualitative analysis for Drop, GSM8K, and MATH, highlighting the types of errors that were fixed?
clarity,_quality,_novelty_and_reproducibility: The substitution method proposed by the authors is interesting, novel, and easy to reproduce. I am less convinced about its usefulness, though.
summary_of_the_review: The authors introduced a clever and simple method that improves the arithmetic capabilities of LMs. Assessing the full usefulness of this method may require more work.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The author proposed a novel method combining a large language model with solving linear systems to improve the calculation over large number performance.
Contributions:
1. A novel approach that takes advantage of neural networks' ability to solve a simple math problem, and generalizes the ability through number substitution and linear system solving.
2. Natural language model is agnostic so the approach is generalizable to different models.
3. Have good performance gain even with zero shot

---------

Update about my final recommendation: 
I am convinced with the rebuttal material and happy to accept this paper. 

strength_and_weaknesses: Strengths: 
1. The paper writing is very clear and the story is convincing.
2. The study using different solving strategies demonstrates the approach can be configured toward different needs
3. The experimental studies show it is pretty generic and can improve different neural baselines

Weakness:
1. The experiment setup is a little bit vague. There are multiple setups, including zero-shot, few shots, and fine-tuning. However, it is hard to tell which setup is used to perform one experiment just from the figure and the caption in the evaluation section. 
2. Expressiveness: The system can only handle up to the linear system, and cannot generalize to aggregation, or recursion yet. 
clarity,_quality,_novelty_and_reproducibility: Clarity: 3.5/5 
Pro: The mathematics and paper writeup is clear, and motivates people well in the introduction section.
Con: The experimental presentation can still be improved. For example, label the experiments with ZS, FS, and fine-tune setups.

Quality: 4/5
The author's statements are well supported by both theory and experiments. To make the evaluation more complete, maybe an ablation study can be performed to see how much performance can be obtained under the ZS, FS, and fine-tuning setup, and compare the time consumption and accuracy using different solving strategies. 

Novelty: 4/5
The idea of converting hard numerical problems into easy ones, and using a linear system to abstract the internal reasoning structure is very interesting. 
summary_of_the_review: The author proposed a novel method combining a large language model with solving linear systems to improve the calculation over large number performance. The experiment shows this model-agnostic approach can improve the performance of different natural language models by a non-trivial amount.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes SOLIS, where during model inference, complex numbers are substituted with simpler numbers (anchors) from 1 to 20, the language model outputs the arithmetic between the anchors, and through these, the original answer of the arithmetic between the complex numbers are derived through solving linear systems. The operand (to be substituted by anchors) is chosen from the input text through the significance of their perturbations on the model performance. Finally, after having the operands and their anchors, the required output is generated through either one of the proposed 3 approaches to solving linear systems - analytical, search-based, and heuristics. 

Although missing some crucial baselines (discussed in the section below), the authors show that LMs can improve their performance for numeral prediction with the incorporation of their SOLIS framework.

## Edit After Reviewing Author Response

After reviewing the author response, I have updated my recommendation to a weak rejection of the paper.  This is primarily due to the clarification of the goals by the authors and incorporating (and demonstrating improvement on) the GenBERT baseline. The reason for non-acceptance despite the paper having some novel contributions and being well-written has to do with my initial comment on the lack of the proposed model to work with both textual and numeric data which is something (also acknowledged by authors in the response) that some previous works are able to do (see references in `summary of review` below).

strength_and_weaknesses: ## Strengths:
1. The ideation is interesting - authors leverage the fact that LMs are good at arithmetic for numerals 1-20 that are highly frequent in the training corpus and use this for eliciting arithmetic operations between more complex numbers.


2. The ideas are clearly expressed in the writing/figures/tables and thus are easy to grasp.

## Weakness:

Among the two weaknesses of the paper, the first is a deal-breaker. The authors establish the premise for the need of numeric comprehension in language models as 1) numeric reasoning forms the crux of understanding text, tables, and knowledge bases 2) extrapolation for numerals outside of the pre-training corpora. Here, the approach that the authors have taken simply appends a system on the top of untouched LMs that does the computations for the LM. This appendage does improve performance when the task is strictly to output a number. However, tracing this back to the original premise of the paper, unless the representations of numbers of these LMs change or adapt, this appendage makes no difference in the understanding of numerals by the LM. Case in point, a DROP dataset instance where the LM has to understand numerals in the paragraph but the output is textual - I believe this appendage provides no significance in the understanding of numerals. This is further corroborated by the paper itself - section 5.3 - where the authors state that for their experimentation, they have specifically chosen instances of the DROP dataset that have numeric outcomes.

Two essential works that also aim to improve numerical comprehension and also evaluate themselves on the DROP dataset (including instances where the output is textual), one of which is cited in the paper, is not used as baselines:

1. Geva, M., Gupta, A., & Berant, J. (2020). Injecting numerical reasoning skills into language models.

2. Sundararaman, Dhanasekar, et al. "Methods for Numeracy-Preserving Word Embeddings." Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020. 


clarity,_quality,_novelty_and_reproducibility: The quality and clarity of the paper is good - it is written and presented well. The proposed solutions are original, however, are not pertinent to the problem statement.
summary_of_the_review: The lack of important baselines I can overlook as the idea is interesting, however, I base my recommendation on the first part of the paper weakness - the offered solution is antithetical to the premise of the paper. If the problem statement is that language models struggle to understand numbers that appear in tables, text, and knowledge bases, the solution cannot be to approximate a calculator as an addition to the inference chain. To elaborate, take a supposed instance of any QA dataset (SQuAD or DROP) where the context is "X scored 4 goals while Y managed to only score 3", the question is "Who scored more goals?" and the answer is textual - "X". The framework suggested can compute arithmetic between 4 and 3, but does not help the LM better understand what 4 and 3 are such that it can make an informed decision. There are several lines of work that work on the premise that this paper takes, but provide interventions that can be applied to both textual and numeric data:

1. Spithourakis, Georgios, and Sebastian Riedel. "Numeracy for Language Models: Evaluating and Improving their Ability to Predict Numbers." Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2018.

2. Jiang, Chengyue, et al. "Learning Numeral Embedding." Findings of the Association for Computational Linguistics: EMNLP 2020. 2020.

3. Zhang, Xikun, et al. "Do Language Embeddings capture Scales?." Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP. 2020.

4. Geva, M., Gupta, A., & Berant, J. (2020). Injecting numerical reasoning skills into language models.

5. Sundararaman, Dhanasekar, et al. "Methods for Numeracy-Preserving Word Embeddings." Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposed a method to identify relevant numbers in text, replace them with smaller numbers that are more accurate to the LM, and use the input-output pair to search for a linear system.
strength_and_weaknesses: Strength:
1. Novelty is high in my opinion. Numerical instability is a long-standing problem in neural-based reasoning. While most research attempts to use neural-symbolic methods, this paper figured out a way to identify relevant numbers and outsource the finding solution steps to classicial methods.
2. Experiments are through with good analysis. Comprehensive ablation studies are performed to identify the contributions of variations in LMs and different linear system search methods.

Weakness:
1. Currently the method is limited to linear system, which probably is OK since most arithmetic in natural language text are not complex. But there are areas where non-linear solutions are required (imagine solving high-school math), how can the method be extended to that case?
2. It would be great if the authors can show some failure cases. I wonder which step leads to the most failures? Is it finding the right number in the text, or the search for system solution?
clarity,_quality,_novelty_and_reproducibility: The paper is well written with concise explanations and good amount of illustrations. Quality is high with good novelty. Not reproducible though as code is not provided, and it is not trivial to reproduce the paper from scratch.
summary_of_the_review: The proposed method is novel and thoroughly evaluated, and proven to be effective. It is however not clear if the method can be extended to more complex numerical reasoning tasks.

-----------After rebuttal--------------

I am happy with the rebuttal and I still recommend acceptance for this paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a novel technique for improving the performance of language models in tasks that involve arithmetic. The main idea is to do the following steps:

- Substitute the large numbers in the question with smaller numbers:
- Use the language model to generate the final answer.
- Use an algorithm to find out what was the calculation done by the language model.
- Do the same calculation on the original set of numbers.

The proposed method is novel and interesting. Furthermore, authors provide several experiments to evaluate it. The paper is well-written and very easy to follow. However, as also raised partially by some of the reviewers, there are several shortcomings:

- The proposed method is using an external tool to improve the performance of language models in reasoning. Therefore, the most important baseline is comparison with just prompting or fine-tuning language models to use calculator [1,2]. During rebuttal, authors argued that their method would work better than using a calculator when a small language model is used. However, these comparisons should be done and reported in the paper. So my concrete suggestion is to compare with prompting and finetuning language models to use calculator at different model scales. 

- GSM8K dataset [2] is a standard dataset for math-word problem which only involves the main 3 operators. It is a perfect task for this paper and I don't understand why there is not comparison on that dataset. I suggest authors to add GSM8K results.

- There is no guarantee that one can find the exact calculation just given the initial number and the final one. In fact, this probably becomes very difficult once more complex calculations are allowed. This should be clearly discussed or potentially analyzed in the paper.

- In many math word problems, it is not clear that the anchor numbers are feasible in the space of the problem. This is a big issue that cannot be fixed easy and again could get worse as the complexity of the problem/calculations grows. Therefore, it should be discussed in the paper.

[1] https://arxiv.org/pdf/2201.11903.pdf

[2] https://arxiv.org/pdf/2110.14168.pdf

Given the above issues, I recommend rejection but highly suggest authors to resubmit after addressing the above issues.
justification_for_why_not_higher_score: Lack of comparison with the calculator baseline and GSM8K dataset evaluations
Decision: Reject