Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose ELASTIC, a multi-modular network for numerical reasoning on MathQA and FinQA datasets. It consists of an encoder and a compiler with four different modules. Each module is specifically designed to solve a part of the numerical reasoning task. For example, the operator generator decides the next operation, and then the operand generator picks the operands to be used. The memory register stores some intermediate computations for future use. Overall, they achieve strong improvements over prior baselines. Additionally, this method generalizes better to higher program steps.
strengths_and_weaknesses: Strengths:
* The paper is very nicely motivated, with good supporting experiments and ablations
* The design of each component is generally well described and easy to follow through (except some parts mentioned below)
* This direction of using specific modules for a fixed set of tasks (similar to NMN's), is indeed very interesting for this area

Weakness:
* Why is NumNet+ the only baseline from the DROP dataset leaderboard used in the paper? There are several other meaningful baselines that can be tried by just keeping the arithmetic heads at classification time. Please try adding more of these baselines, as applicable, to make Table 3 stronger.
questions: Writing suggestions:
* Please rewrite Section 3 introduction using less technically repetitive terms and more general ones/synonyms. Currently, it's a bit hard to read through.
* Please add a figure demonstrating the terms in Table 2 clearly.
* Similarly, I feel it would help to have a motivating figure to explain the need and functionality of the memory register, for completeness.
limitations: Please try to add a section specific to the limitations of the method. Other than the generalization issues, I feel these modular approaches might need more training data compared to the end-to-end frameworks. You may consider discussing some points along these lines.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors present a novel model (ELASTIC) for numerical reasoning problems. The proposed model consists of an encoder and a compiler which separates the generation of operators and operands. The authors claim that the separated generators enhance the adaptability and elasticity of the model. The proposed model also incorporates a memory register to improve the model’s performance on the numerical problems with large M-MDD. The proposed model achieves state-of-the-art performance on two datasets: FinQA and MathQA.

strengths_and_weaknesses: Strengths: Numerical reasoning is an important area in machine learning and natural language processing. This work proposes novel methods to enhance the numerical reasoning ability of the model. On one hand, the generation of operators and operands are separated. On the other hand, a memory register is introduced to make it easier for the model to use the previous sub-program results. The proposed model outperforms FinQANet on both FinQA and MathQA. 

Weaknesses: 
1. The motivation and effectiveness of separating the generation of operands and operators is not clear. The authors claim that previous work that does not use separated generators suffers from cascading errors when encountering complicated tasks (line 36-37), but the results shown in figure 2 (c) and Table 4 may not be sufficient to support this claim. The proposed model does not consistently stabilize the performance on long reasoning programs.  

2. The improvement achieved by memory register seems to be marginal (Table 4). 

3. The writing should be improved. The approach section is not easy to follow. The authors use notations throughout this section and some terms and notations are not well explained at their first appearance. 

For example, DSL (line 108), NGS (line 136), W_{op} in eq 6, and the index i in eq 6 and 7 have different meanings. 

Figure 1 shows the reasoning manager generates g^{op} and g^{oe} in two different flows and the inputs are different, but the difference is not explained in line 134-143. Although the following introduction of two generators explains the difference, it might be better to move this content to line 134-143. Figure 1 and line 160-172 shows how the results from the RM updates the memory register but does not show how the vectors stored in the memory register is used during the generation.  I guess there should be another arrow from the memory register to the generator in Figure 1.  Please clarify. 

questions: Results in Figure 2 are a bit confusing. The accuracy with M-MDD=2 is much lower than that with M-MDD=3 in figure 2 (a), why? What’s the possible reason of the large fluctuations in figure 2 (b) and 2(c)? Why the performance of ELASTIC starts to soar when the program steps are bigger than 10 in figure 2 (c)?  Does the similar phenomenon exist in the other dataset?

limitations: The experiments might be designed more carefully. When analyzing the necessity of memory register, the effect of separated generators is not disentangled with the effect of the memory register. For more effective analysis, there should be a setting ELASTIC w/o separated generator, or FinQANet w memory register if possible.

To make section 3 easier to follow, it might be better to use more textual examples instead of notations when possible.

Typos: line 107-108, 144, 262
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a new model called numEricaL reASoning with adapTive symbolIc Compiler (ELASTIC) to solve numerical reasoning QA problems. ELASTIC separates the generation of operators and operands, and is adaptable to the number of operands, making it more accurate and flexible. They show promising results on two datasets including FinQA and MathQA.
strengths_and_weaknesses: Numerical reasoning is an important and challenging task of AI. Although the idea of generating operators and operands separately is already been adopted in other areas like program synthesis, it is still novel and valuable in numerical reasoning to alleviate the cascading error problem. Experimental results are solid and promising, which well supports the authors’ claims.

The writing needs to be improved, especially the method part, which is somewhat confusing. It seems that the method proposed cannot be applied to QA problems without labeled programs. Also, the influence of different programming language has not been discussed.
questions: 1. Comparison with large language models. The authors claim that pretrained language models fall short of numerical reasoning from the text. However, recent advances in program synthesis show that large language models perform well in this area: they achieve 83.8% on MathQA which outperforms the 83.00% in this paper. I agree that this paper is valuable but for the completeness could you make more comparisons with these large language models in terms of number of parameters, runtime costs, and generalization ability, etc.? 

2. Some details introduced in Section 3 are unclear. For example, what does “NGS” mean? What’s the size of the memory register (size = max program steps or size = 1)? Is it possible to train another network to predict which memory slot should be overwritten when the memory is full [2]?

3. Although the authors say ELASTIC is domain agnostic, it seems that the performance of these program-based models can be significantly influenced by the programming language. Can ELASTIC solve MathQA-python[3], which replaces the original MathQA DSL with python?

[1] Program Synthesis with Large Language Models

[2] Automatic Program Synthesis of Long Programs with a Learned Garbage Collector

[3] https://github.com/google/trax/blob/master/trax/examples/MathQA_Python_generation_notebook.ipynb
limitations: Please see questions listed above.
review_title: null

Meta-review: metareview: The paper presents a new model called Numerical Reasoning with Adaptive Symbolic Compiler that is able to perform numerical reasoning tasks. One of the key ideas in this model is to separate the generation of operators and operands and to include a memory register to remember intermediate values. The method is compared against FinQANet and shows good results on both the MathQA and FinQA datasets. In the reviews and rebuttal, it emerged that the performance of this model is marginally lower than the performance of state-of-the-art large language models on this task; however, the difference in size and training resources required by this model compared with the LLM is so vast that this still represents a significant advance over the state of the art. It would nevertheless be good for this to be mentioned in the paper itself. Overall, this is a novel approach that advances an important problem.
confidence: Certain
Decision: Accept