Weakly Supervised Neuro-Symbolic Module Networks for Numerical Reasoning
Authors: Amrita Saha
Shafiq Joty
Steven Hoi
Venue:  (2021)
Keywords: neural module networks
machine reading comprehension
numerical reasoning over text

Abstract: Neural Module Networks (NMNs) have been quite successful in incorporating explicit reasoning as learnable modules in various question answering tasks, including the most generic form of numerical reasoning over text  in Machine Reading Comprehension (MRC). However, to achieve this, contemporary NMNs need strong supervision in executing the query as a specialized program over the reasoning modules and fail to generalize to more open-ended settings where such supervision is not readily available. In this work, we propose Weakly Supervised Neuro-Symbolic Module Network (WNSMN) trained with answers as the sole supervision for numerical reasoning based MRC. It learns to execute a noisy heuristic program obtained from dependency parsing of the query as discrete actions over both neural and symbolic reasoning modules and trains it end-to-end in a reinforcement learning framework with discrete reward from answer matching. On the numerical-answer subset of the DROP dataset, WNSMN outperforms NMN by 32% and the reasoning-free language model GenBERT by 8% in exact match accuracy when trained under comparable weak supervised settings. This showcases the effectiveness and generalizability of modular networks that can handle explicit discrete reasoning over noisy programs in an end-to-end manner. 

Under review as a conference paper at ICLR 2021
WEAKLY SUPERVISED NEURO-SYMBOLIC MODULE NETWORKS FOR NUMERICAL REASONING
Anonymous authors
Paper under double-blind review
ABSTRACT
Neural Module Networks (NMNs) have been quite successful in incorporating ex- plicit reasoning as learnable modules in various question answering tasks, including the most generic form of numerical reasoning over text Machine Reading Com- prehension (MRC). However, to achieve this, contemporary NMNs need strong supervision executing the query as specialized program over reasoning modules and fail to generalize to more open-ended settings without such supervision. Hence we propose Weakly-SSperrvise Neuro-Symbolic Module Network (WNSMN) trained with answers as the sole supervision for numerical reasoning based MRC. It learns to execute a noisy heuristic program obtained from the dependency parsing of the query, as discrete actions over both neural and symbolic reasoning modules and trains it end-to-end in a reinforcement learning framework with discrete reward from answer matching. On the numerical-answed subset of DROP, WNSMN out- performs NMN by 32% and the reasoning- free language model GenBERT by 8% exact match accuracy when trained under comparable weak supervised settings. This showcases the effectiveness and generalizability of modular networks that can handle explicit discrete reasoning over noisy programs in an end-to-ent manner.
INTRODUCTION
End-to-end neural models have proven to be powerful tools for an expansive set of language and vision problems by effectively emulating the input-output behavior. However, many real problems like Question Answering (QA) or Dialog need more interpretable models that can incorporate explicit reasoning in the inference. In this work, we focus on the most generic form of numerical reasoning over text, encompassed by the reasoning based MRC framework. A particularly challenging setting for this task is where the answers are numerical in nature as in the popular MRC dataset, DROP (Dual et al. 2019). Figure I shows the intricacies involved in the task, (i) passage and query language understanding, (if) contextual understanding of the passage date and numbers, and (ii)) application of quantitative reasoning (e.g.. max, not) over dates and numbers to reach the final numerical answer.
Three broad genres of models have proven successful on the DROP numerical reasoning task. First, large-scale pretrained language models like GenBERT (Geva et al. 2020) uses monolithic Transformer architecture and decodes numerical answers digit-by-digit. Though they deliver mediocre performance when trained only the target data, their competency is derived from pretraining on massive synthetic data augmented with explicit supervision of the gold numerical reasoning. Second kind of models are the reasoning-free hybrid models like NumNet (Ran al. 2019), NAQANet (Dua et al. 2019). NABERT+ (Kinley & Lin 2019) and MTMSN (Hu et al. 2019). NeRd (Chen et al. 2020 They explicitly incorporate numerical computations in the standard extractive QA pipeline by learning a multi-type answer predictor over different reasoning types (e.g., max/min, diff/sum, count, negate) and directly predicting the corresponding numerical expression, instead of learning to reason. This is facilitated by exhaustively precomputing all possible outcomes of discrete operations and augmenting the training data with the reasoning-type supervision and numerical expressions that lead to the correct answer. Lastly, the most relevant class of models to consider for this work are the modular networks for reasoning. Neural Module Networks (NMN) GGupt et al., 2020) is the first explicit reasoning based QA model which parses the query into specialized program and executes it stse-wise over learnable reasoning modules. However, to do so, apart from the exhaustive precomputation of all discrete operations, it also needs more fine-grained supervision of the gold

---

Under review as a conference paper at ICLR 2021
Hoping
Dan
Query:: Which is the longest goul by Carpanter?
NateRuumer Ently Specific Cross Attention
Discrete Reasoning.
Carpantor made a 26-yard fold goal... In second quarter Carponter
Query Parsing Program Deperdae
pare
Wrich is the lingest
good by Corperier
BERT
Stacted Attenton Maos over Errites
Operator Sampler
Thippen making a 9-yard
followed by Tylar
42-aar fiedd geal
X2 [ [gool by Cappenter X1)
X1 - (WWichh the longest)
Passage
Passagg
Erdies
Auttor
TD pass to TE Anthony
Ana CiansteRaaaaring (Wiiii
Fiegram Eniies
Angumnent
Fasano and goal.
is Hh Junguatt X2)
Ans
mayi22 26 420 -42
Figure 1: Example (passage, query, answer) from DROP and outline of our method: executing noisy program obtained from dependency parsing of query by learning date/number entity specific cross attention, and sampling and execution of discrete operations on entity arguments to reach the answer.
program and the gold program execution, obtained heuristically, by leveraging the abundance of templatized queries DROP.
While being more pragmatic and richer at interpretabilityy, both modular and hybrid networks also tightly coupled with the additional supervision. For instance, the hybrid models cannot learn without it, and while NMN is the first to enable learning from QA pair alone, it still needs more finer-grained supervision for at least a part of the training data. With this, it manages to supercede the SoTA models NABERT and MTMSN a carefully chosen subset of DROP using the supervision. However, NMN generalizes poorly to more open- open-ended settings where such supervision is not easy to handcraft.
Need for symbolic reasoning. One striking characteristic of the modular methods is to avoid discrete reasoming by employing only learnable modules with an exhaustively precomputed space of outputs. While they perform well on DROP, their modeling complexity grows arbitrarily with more complex non-linear numerical operations (e-g., exp, log, cos). Contrarily, symbolic modular networks that execute the discrete operations are possibly more robust or pragmatic in this respect by remaining unaffected by the operation complexity. Such discrete reasoning has indeed been incorporated for simpler, well-structured tasks like math word problems KKonel-Kedziorsk et al, 2016) or KB/Table: QA (Zhong et al., 2017 Liang al., 2018 Saha et al. 2019),, with Deep Reinforcement Learning (RL) for end-to-end trainig. MRC however needs a more generalized framework of modular neural networks involving more fuzzy reasoning over noisy entities extracted from open-ended passages.
In view of this, we propose a Weakly-Supervice Neuro-Symbolic Module Network (WNSMN)
A first attempt at numerical reasoning based MRC, trained with answers as the sole supervision;
Based on a generalized framework of dependency parsing of queries into noisy heuristic programs;
End-to-end training of neuro-symbolic reasoning modules in a RL framework with discrete rewards;
To concretely compare WNSMN with contemporary NMN, consider the example in Figure I NMN parses the query into program form FAND Carpenter'), 'goal")), )) which is step-wise executed by different learnable modules with exhaustively precomputed output set. To train the network, it employs various forms of strong supervision such as gold program operations and gold query-span attention at each step of the program and gold execution i.e., supervision of the passage numbers (23, 26, 42) to execute MAX operation on.
In comparison our generalized query-parsingg
While NMN can only handle the 6 reasoning categories that the supervision was tailored to, WNSMN focuses on the full DROP with numerical answers (called DROP-num) that involves more diverse reasoning more open-ended questions. We empirically compare WNSMN on DROP-num with the SoTA NMN and GenBERT that allow learning with partial or no strong supervision. Our results showcase that the proposed WNSMN achieves 32% better accuracy than NMN in absence of least one or more types of supervision and performs 8% better than GenBERT when the latter is fine-tunee only on DROP in a comparable setup, without additional synthetic data having explicit supervision.
MODEL: WEAKLY SUPERVISED
MODULE NETWORK
We now describe our proposed WNSMN that learns to infer the answer based on weak supervision of the QA pair by generating the program form of the query and executing it through explicit reasoning. Parsing Query into Programs To keep the framework generic, we use simplified representation of the Stanford dependency parse tree (Chen & Manning 2014) of the query to get generalized program (Appendix A.5). First, a node is constructed for the subtree rooted at each child of the root by merging its descendants in the original word order. Next an edge is added from the left-most node (which we call the root clause) to every other node. Then by traversing left to right, each node is organized into a step of program having linear flow. For example, the program obtained in Figure
2

---

Under review as a conference paper at ICLR 2021
Stacked Spen Predictin Logit for
Quary
N, M.K.N.
0,0 D...
Passage
Paosage/Mubben Numbera
Passage Dates
X1, a
Dependency
Parsing
Passage Self Attonton
X2, 4
Program
BERT Contextuulizee Passage Enzoding for progre em slep 2-11
X3,
X2
A HAA
A dote
X1, A
A,
Indlcttivn Blee, Ently Specific Information Retreval
Expectod Possage Altention Map
MultScalled Stacked Attention besed Soft Spen Prediction
Span Level Altention Smoothing
Passage
X2, A
A,
Passage Program Acoonionn
Number/Date
X3. A mm
Attention
L. Zaa
L.
1-1
program
Figure 2: Modeling the interaction between the passage and (left) the program, & (right) its num- ber/date entities. For each program step k, they respectively yield (i) Stacked Span Prediction Logits and (ii) Attention over Number/Date entities for each passage token. The linear combination of these two gives the expected distribution over entities, Tham and Th for the step k
is XI ('which is the longest"); 1, X2 = the longest', X2). Each program step consists of two types of arguments (i) Query Span Argument obtained from the corresponding node, indicates the query segment referred to, in that program step e.g., 'goal by Carpenter' in Step (ii) Reference Argument(s) obtained from the incoming edges to that node, refers to the previous steps of the program that the current one depends on e.g, X/ in Step Next, final step of the program is added, which has the reference argument as the leaf node(s) obtained in the above manner and the query span argument the root-clause. This step is specifically responsible for handling the discrete operation, enabled by the root-clause which often indicative of the kind of discrete reasoning involved (e.g., max). However this being noisy heuristic, the QA model needs to be robust to such noise and additionally rely on the full query representation in order predict the discrete operation. For simplicity we limit the number of reference arguments 2.
= ('goal by Carpenter', XI); Answer = Discrete-Reasoning! Reasoning( 'which is
2.1 PROGRAM EXECUTION
Our proposed WNSMN learns to execute the program over the passage in three steps. In the preprocessing step, it identifies numbers and dates from the passage, and maintains them as separate canonicalized entity-lists along with their mention locations. Next, it learns an entity-specific cross- attention model to rank the entities w.zt. their query reley-ree (22.1.I,, and then samples relevant entities as discrete arguments (22.22) and executes appropriate discrete operations on them to reach the answer. An RL framework (...3 trains it end-to-end with the answer as the sole supervision.
2.1.1 ENTITY-SPECIHIC CROSS ATTENTION FOR INFORMATION EXTRACTION
To rank the query-relevant passage entities, we model the passage, program and entities jointly.
Modeling interaction between program and passage This module (Figure 2 left) learns to associate query span arguments of the program with the passage. For this, similar to NMN, we use a BERT-base pretrained encoder (Devlin et al. 2018) get contextualized token embeddings of the passage and query span argument of each program step, respectively denoted by Px and Qi for the k'th program step. Based on it, we learn similarity matrix S Ixaxm between the program and passage, where I, n,, and m respectively are the program length and query span argument and passage length (in tokens). Each S₁ x represents the affinity over the passage tokens for the k tht program argument and is defined as (,(i, w" [Q::: Paj; Q+i O Pkyl, where w is learnable parameter and O is element-win multiplication. From this, an attention map A computed over the exp(S, (k,)) passage tokens for the k k'th program argument as (i,j) = softmax, (i,( = Σ, exp(S, (1.9))
Similarly, for the i'th token of the k'th program argument the cumulative attention dsi w.r.. the passage is
softmax; (E, (a(i. J)). A linear combination of the attention map (i.
weighted by aμı gives the expected passage attention for the k'th step, ã₁ Σ, A4 (i.) E R"
Span-level smoothed attention. To facilitate information spotting and extraction over contiguous spans of text, regularize the passage attention that the attention on passage token is high if the attention over its neighbors is so. We achieve this by adopting a heuristic smoothing technique Huang et al. 2020). taking sliding window of different lengths w {1, 2, 100 over the passage,
3

---

Under review as a conference paper at ICLR 2021
and replacing the token-level attention with the attention averaged over the window. This results in 10 different attention maps over the passage for the k 'tth step of the program: { E (1,
Soft span prediction. This network takes a multi-scaled (Gupta et al. 2020) version of ag by multiplying the attention map with |s] different scaling factors (8 [1, 2, 5, 10)), yielding a |s|- dimensional representation for each passage token, i.e., 0% E R This is then passed through L-layered stacked self-attention transformer block (Vaswani m x d dimension, followed by a linear layer of dimension d x 1, to obtain the span prediction logits: ag Linear Linear(Trans (Trans I formerr]) [M ulllScaaling( (xt) E RT. Further the span prediction logits at each program step (say k) is additively combined with those from the previous steps referenced in the current one, through the reference argument (r))(k) at step k. i.e., O = az + Ee)ere(()) O
al. 2017). which encodes it to
Modeling interaction between program and number/date entities This module (Figure 2 right) facilitates an entity-based information spotting capability, that is, given a passage mention of number/date entity relevant to the query, the model should be able to attend to the neighborhood around it. To do this, for each program step, we first compute passage tokens to number tokens attention map A mum E R where N is the number of unique number entities. Note that this attention map different for each program step as the contextual BERT encoding of the passage tokens (Pk) is coupled with the program's 's span argument of that step. At the k-th step, the row Agu (i.-) denotes the probability distribution over the N unique number tokens w.r.f. the i-th passage token. The attention maps are obtained by a softmax normalization of each row of the corresponding passage tokens to number tokens similarity matrix, S E RR*NN N for k = {1... 7), where the elements of Sgam are computed as S (i.j) = P W W,.Kkn with Wn E Rddd being learnable projection matrix and nj being the passage location of the j-th number token. These similarity scores are additively aggregated over all mentions of the same number entity in the passage.
The relation between program and entities is then modeled T softmax E, ofg. A Ap (t, .)) RV, which gives the expected distribution over the N number tokens for the k-tth program step and using w as the smoothing window size. The final stacked attention map obtained for the different windows is T TTum (T س {1,2,... 10)). J. Similarly, for each program step k, we also compute separate stacked attention map Telate over the unique date tokens, parameterized by different W
A critical requirement for a meaningful attention over entities is to incorporate information extrac- tion capability in the number and date attention maps A num and date by enabling the model to attend over the neighborhood of the relevant entity mentions. This is achieved by minimizing the unsupervised auxiliary losses Luum and Ldate in the training objective, which impose an inductive bias over the number and date entities, similar to Gupta et al. (2020) Its purpose is to ensure that the passage attention is densely distributed inside the neighborhood of + Ω (a hyperparameter, e.g., 10) of the passage location of the entity mention, without imposing any bias on the attention distribution outside the neighborhood. Consequently, it maximises the log-form of cumulative likelihood of the attention distribution inside the window and the entropy of the attention distribution outside of it.
N
N
Σ
Σ
Log
Σ j=1
1 Enye & (k3
num
Σ 1=1
1 1nj
aum log(a nam
(I)
where 1 is indicator function and
A k
Anum
(1.
date for date entities is similarly defined. aaz
2.1.2 MODELING DISCRETE REASONING
The model next learns to execute a single step of discrete reasoning (Figure 3 based on the final program step. The final step contains (i) root-clause of the query which often indicates the type of discrete operation (e.g.. 'what is the longest' indicates max, 'how many goals' indicates count), and (ii) reference argument indicating the previous program steps the final step depends on. Each previous step (say k) is represented as stacked attention maps T and Thlate, obtained from 2.1.1
Operator Sampling Network Owing to the noisy nature of the program, the operator network takes as input: (i) BERT's [ [CLS] representation for the passage-query pair and LSTM (Hochreiter & Schmidhuber 1997) encoding (randomly initialized) of the BERT contextual representation of (ii) the root-clause from the final program step and (iii) full query (w.zt. passage), to make two predictions:
'This is a reasonable assumption for DROP with recall of 90% on the training set. However, it does not. limit the generalizability of WNSMN, with standard beam search it is possible scale to an l-step MDP.
4

---

Under review as a conference paper at ICLR 2021
EntiyTTpe, A
Sampling Prob.
Cooo
Staedwed
1-arqument
Sample
Op-
Sampled Astion
Arg-
Esscution
Cpenitio Arg.meent) Sumping Frot:
RENFORREE
Entiy Type Predeter
Operator Predictor
Mitonton
Date
Zarrument
Sample
Argurment
Op
Arg
Ari,
(ight 1 1g00
Otjective
Eniitos
Counter
Sagsiin
X
LSTM
LSTM
Network
Prob. %
Op-
Ang
Are.
Reward fusction
Banker
...
BERT ensoding Guery Root Clause, Full Query, CLB laken
Og-
Arp
And
Catsrig
Argumancs
Figure 3: Operator & &Argument Sampling Network and RL framework over sampled discrete actions
Entity- Predictor Network, an Exponential Linear Unit (Elu) activated fully-connected layer followed by softmax that outputs the probabilities of sampling either date or number types.
Operator Predictor Network, a similar Elu-activated fully connected layer followed by a softmax which Icarns probability distribution over a fixed catalog of 6 numerical and logical operations (count, ma x, min, sum, di ff, negate), each represented with learnable embeddings.
Apart from the diff operator which acts only on two arguments, all other operations can take any arbitrary number of arguments. Also some of these operations can be applied only on numbers (e.g., sum, negate) while others can be applied on both numbers or date (e.g., max, count).
Argument Sampling Network This network learns to sample date/number entities as arguments for the sampled discrete operation, given the entity-specific stacked attentions (7guum and Telate) for each previous step (say, k), that appears in the reference argument of the final program step. In order to allow sampling of fixed or arbitrary number of arguments, the argument sampler learns four types of networks, each modeled with a L-layered stacked self attention based Trans transformey former block (with output dimension d) followed by different non-linear layers embodying their functionality and a softmax normalization to get the corresponding probability of the argument sampling (Figure D
Sample n E {1, 2} Argument Module: softmax Elu( Linear (TTrans f former(T))) (T))) outputs distribution over the single entities (n 1) or a joint distribution over the entity-pairs (n 2). predicts distribution over possible count values (E [L.... 10) of number of entity arguments sample. Entity-Ranker Module: softmax(PReell) Linear ax (T (Trans/ transforment (orerr(T))) learns rerank the entities and outputs a distribution over all the entities given the stacked attention maps as input.
Counter Module: softmax [Flu(Tinear dx (C N- Encoder( ierooderrrrnnis (ormer(()))
Sample Arbitrary Argument: M Multinomial (Entany-tttttttttn Ranked Distribution, Counter Prediction).
Depending on the number of arguments needed by the discrete operation and the number of reference arguments in the final program step, the model invokes one of Sample {1, 2, Arbitrary) Argument. For instance, if the sampled operator is di ff which needs 2 arguments, and the final step has 1 or 2 reference arguments, then the model respectively invokes either Sample 2 argument or Sample I argument on the stacked attention T corresponding to each reference argument. And, for operations needing arbitrary number of arguments, the model invokes the Sampling Arbitrary Argument. For the Arbitrary Argument case, the model first predicts the number of entities c E {l..... . 10} to sample using the Counter Network, and then samples from the multinomial distribution based on the joint of c-combinations entities constructed from the output distribution of the Entity Ranker module.
2.1.3 TRAINING WITH WEAK SUPERVISION IN THE DEEP RL FRAMEWORK
We use an RL framework to train the model with only discrete binary feedback from the exact match of the gold and predicted numerical answer. particular, we use the REINFORCE (Williams 1992) policy gradient method where a stochastic policy comprising sequence of actions is learned with the goal of maximizing the expected reward. our case, the discrete operation along with argument sampling constitute the action. However, because of our assumption that single step of discrete reasoning suffices for most questions in DROP, we further simplify the RL framework to a contextual multi-arm bandit (MAB) problem with a 1-step MDP, i.e., the agent performs only one step action.
Despite the simplifying assumption of 1-step MDP, the following characteristics of the problem render it highly challenging: (i the action space A is exponential the order of number of operations and argument entities in the passage (averaging to 12K actions for DROP-num); (ii) the extreme reward sparsity owing to the binary feedback further exacerbated by the presence of spurious rewards, as the same answer can be generated by multiple diverse actions. Note that previous approaches like
5

---

Under review as a conference paper at ICLR 2021
NMN can avoid such spurious supervision because they heuristically obtain additional annotation of the question category, the gold program or gold program execution atleast for some training instances.
In our contextual MAB framework, for an input x = (passage(p), query(z)), the context or environment state sa(x)) is modeled by the entity specific cross attention 2.11 parameterized by の) between the (i) passage (7i) program-form of the query and (ii) extracted passage date/number entities. Given the state sa(a),, the layout policy (: 2.1.22 parameterized by 0) then learns the query-specifer inference layout, ie.. the discrete action sampling policy Pa(a [x)((())) for action a E A. The action sampling probability is product of the probability of sampling entities from the appropriate entity type (Paape) probability of sampling the operator (Pe"), and probabiliiy of sampling the entity argument(s) (Par))) normalized by number of arguments to sample. Therefore, with the learnable context representation sx(x) of input æ, the end-to-end objective to jointly learn {0, Φ that maximises the expected reward R(x, a) E {-1,+1) over the sampled actions (a), based on exact match with the gold answer.
To mitigate the learning instability in such sparse confounding reward settings, we intialize with a simpler iterative hard- cardttxpectttion Maximization (EM) learning objective, called Iterative Maximal Likelihood (IML) (Liang et al. (017). With the assumption that the sampled actions are extensive enough to contain the gold answer, IML greedily searches for the good actions by fixing the policy parameters, and then maximises the likelihood of the best action that led to the highest reward. We define good actions (A49ood) as those that result the gold answer itself and take a conservative approach of defining best among them as simply the most likely one according to the current policy.
JIML (0, Ø)
Σ max log (a.)(a(z)
(2)
aEAbsod
After the IML initialization, we switch to REINFORCE as the learning objective after a few epochs, where the goal is to maximise the expected reward (JRL(0, () = Σn Po R(r, a)) as
V (o ΣΣ (R) - B(x)) Po (alx()
where (xx) is simply the average (baseline) reward obtained by the policy for that instance x. Further, in order to mitigate overfitting, addition to Izgrregularizttonn and dropout, we also add entropy based regularization over the argument sampling distribution, in each of the sampling networks.
3 EXPERIMENTS
We now empirically compare the exact-match performance of WNSMN with SoTA baselines on versions of DROP dataset and also examine how it fares in comparison to strong supervised skylines. The Primary Baselines for WNSMN are the explicit reasoning based NMN (Gupta et al. 2020 which uses additional strong supervision and the BERT based language model GenBERT (Geva et al. 2020) that does not embody any reasoning and autoregressively generates numeric answer tokens. As the Primary Dataset we use DROP-num, the subset of DROP with numerical answers. This subset contains 45K and 5.8K instances respectively from the standard DROP train and development sets. Originally, NMN was showcased on a very specific subset of DROP, restricted to the 6 reasoning caasosnng-types it could handle, out of which three (count, difeedifference, extract-number) have numeric answers. This subset comprises 20K training and 1.8K development instances, out of which only 10K and 800 instances respectively have numerical answers. We further evaluate on this numerical subset, referred to as DROP- Pron--Puund-num. In both the cases, the training data was randomly split into 70%:300% for train and internal validation and the standard DROP development set was treated as the Test set.
Figure shows the t-NE plot of pretrained Sentence- BERT RRimerrs Gurevych 2019) encoding of all questions DROP- num-TTet and also the DROP-Pruned-num--
subset with different colors (red, green, yellow) representing different types. Not only the DROP-num questions more diverse than the carefully cho- sen DROP-Pruned-Pig Pruned-num subset, the latter also forms well-separated clusters corresponding the three reasoning types. Additionally, the average perplexity (using n k) of the DROP-Pruned-nune and DROP-num questions was found to be 3.9 and 10.65 respectively, further indicating the comparatively open- open-ended nature of the former.
Figure 4: t-SNE plot of DROP- num-Test questions.
For the primary baselines NMN and GenBERT, we report the perfor- mance on in-house trained models on the respective datasets, using

---

Under review as a conference paper at ICLR 2021
the code open-soureed by the authors. The remaining results, taken from Geva et al. (2020). Kinley & Lin (2119) and Ran et al. 2019); refer to models trained on the full DROP dataset. All models the same pretrained BERT-base Also note that primary requirement of all models other than GenBERT and WNSMN i.e,, for NMN, MTMSN, NABERT, NAQANET, NumNet, is the exhaustive enumeration of the output space of all possible discrete operations. This simplifies the QA task to classification setting, thus alleviating the need for discrete reasoning in the inference processs.
Table presents our primary results on DROP-num, comparing the performance of WNSMN (accuracy of the top-1 sampled action by the RL agent) with various ablations of NMN (provided in the authors' implementation) by removing atleast one of Program, Execution, and Query Attention supervision ( (Appendix AA.41 and GenBERT models with pretrained BERT that are finetuned on DROP or DROP-rum (de- noted as GenBERT and GenBERT-num). For a fair comparison with our weakly supervised model, we do not treat NMN with all forms of supervision or GenBERT model pretrained with additional synthetic numerical and textual data comparable baselines. Note that these GenBERT variants indeed enjoy strong reasoning supervision in terms of gold arithmetic expressions provided in these auxiliary datasets.
Table 1: DROP- 1100--uum--- Test Performance of Baselines and WNSMN.
Supervision Type Prog. Exec. QAtt.
Acc. (%)
NMN-num variants
x
/
11.77
17.52
/
18.27
/
X
18.54
X
/
X
12.27
X
X
V
11.80
NMN's performance is abysmally poor, indeed drastic degradation in comparison to its performance on the pruned DROP subset reported by Gupta et al. (2020) and in subsequent experiments in Table 2 This can be attributed to their limitation in handling more diverse classes of reasoning and open-ended queries in DROP-num, further exacerbated by the lack of one or more types of strong supervision i Our earlier analysis on the complexity of the questions the subset and full DROP- num further quantify the relative difficulty level of the latter. On the other hand. GenBERT delivers a mediocre performance, while GenBERT-num degrades additionally by 4%, as learning from numerical answers alone further curbs the language modeling ability. Our model performs significantly better than both these baselines, surpassing GenBERT by 8% and the NMN baseline by around 32%. This showcases the significance of incorporating explicit reasoning in neural models in comparison to the vanila large scale LMs like GenBERT. It also establishes the generalizability of such reasoning based models to more open-ended forms of QA, in comparison to contemporary modular networks like NMN, owing to its ability to handle both learnable and discrete modules in an end-to end-to-end manner.
X
11.70
GenBERT
X
42.30
GenBERT-num
X
X
38.41
WNSMN
X
X
50.97 X
Next, in Table 2. we compare the performance of the proposed WNSMN with the same NMN variants (as in Table D on PROP-Pruned-num. Some of the salient observations are: (i) WNSMN in fact reaches a performance quite close to the strongly supervised NMN variant (first row), and is able attain least an improvement margin of 4% over all other variants obtained by removing one or more types of supervi- sion. This is despite all variants of NMN additionally enjoying the exhaustive precompution of the output space of possible numerical answers; (ii) WNSMN suffers only in the case of extract-number type oper- ations (e.g., mix, that involve a more complex process of sampling arbitrary number of arguments (iii) Performance drop of NMN is not very large when all or none of the strong supervision is present, possi- bly because of the limited diversity over reasoning types and query language; and (iv) Query- supervision infact adversely affects NMN's performance, in absence of the program and execution supervision or both, possibly owing to an undesirable biasing effect. However when both supervisions are available, query-attention is able to improve the model performance by 5%. Further, we believe the test set of 800 instances too small to get an unbiased reflection of the model's performances.
Table 2:
Perfor- mance of NMN variants and WNSMN
Supervision-Type
Aec. (%)
Extract- Date-
Prog. Exec. QAtt.
Count
differ
NMN-num Variants
68.6
50,0
88.4
72.5
42.4
24.1
73.9
36.4
$4.3
47.9
80.7
40.9
/
/
X
63.3
45.5
81.1
68.7
X
48.2
38.1
72.4
41.9
61.0
44.7
81.1
63.2
62.3
43.7
84.1
67.7
62.1
46.8
83.6
66.1
WNSMN
66.5
58.8
66.8
75.2
Both the results and limitations of NMN in Table and were confirmed by the authors of NMN well.

---

Under review as a conference paper at ICLR 2021
In Table E] additionally inspect recall over the top-k actions sampled by WNSMN to estimate how it fares in comparison to the strongly supervised skylines: (i) NMN with all forms strong supervi- sion; (ii) GenBERT variants +ND, +TD and +ND+TD further pretrained on synthetic Numerical and Textual Data and both; (iii) reasoning- caasonnin--feeeee free hybrid models like MTMSN (Hu et al., 2019) and NumNet (Ran et.l. 2019), NAQANet (Dua et al. 2019) and NABERT, NABERT+ (Kinley & Lin 2019). Note
that both NumNet and NAQANet do not use pretrained BERT. MTMSN achieves SoTA performance through a supervised framework of training specialized predictors for each reasoning type predict the numerical expression directly instead of learning to reason. While top-1 performance of WNSMN D 4% worser than NABERT, Recall topp2 is equivalent to the strongly supervised NMN, top-5 and top-10 10 is comparable to NABERT+, NumNet and GenBERT models +ND, +TD and top-20 nearly achieves SoTA. Such promising recall over the top-k actions suggests that more sophisticated RL algorithms with better exploration strategies can possibly bridge this performance gap.
(in Table
4 ANALYSIS & FUTURE WORK
Table 3: Skylines & WNSMN top-
l performance DROP-num-TTe
Performance Analysis Despite the notorious instabilities of RL due high variance, the training trend, as shown in Fig- ure 5(a) is not afflicted by catastrophic forgetting. The sudden performance jump between epochs 10-15 because of switch- ing from iterative ML initialization to REINFORCE objective. Figure 5(b) shows the individual module-wise performance evaluated using the noisy pseudo-rewards, that indicate whether the action sampled by this module led to the correct answer or (details in Appendix A.6). Further, by bucketing the performance by the total number of passage entities in Figure 5(c), we observe that WNSMN remains unimpacted by the increasing number of date/numbers, despite the action space explosion. On the other hand, GenBERT's performance drops linearly beyond 25 passage entities and NMN-num degrades ex- ponentially from the beginning, owing its direct dependency on the exponentially growing exhaustively precomputed output space.
Strongly Supervised Models
Acc. (%)
NMN-nun (all supervision)
58.10
GenBERT +ND GenBERT+TD
69.20
70.50
GenBERT+ND+TD
75.20
NAQANet
44.97
NABERT
54.27
NABERT+
66.60
NumNet
69.74
MTMSN
75.00
Recall actions of WNSMN (%)
I 2 k
4
5
10 k:
20
58.6
63.0
65.4
67.4
72.3
74.2
Module
Performance
Sample Argument
54% (Acc.)
Sample Argument
52%
Counter
50%
Entity Ranker
53%
Operator Predictor
78%
Entity Type Predictor
83%
Overall Action Sampler 84% (Rec@ All)
110.
14]
115
PA (25, 29) (4)
Nambbe Phocsep Diwwirrder Frriken
Figure 5: (a) Training trend showing the Recall @top-k: and all actions, accuracy of Operator and Entity-type Predictor, estimated based on noisy psuedo rewards (Appendix A.6b, (b) Module-wise performance (using pseudo-reward) on Teos- (c) Bucketing performance by total number of passage entities for WNSMN, and the best performing NMN and GenBERT model from Table
More Stable RL Framework The training trend in Figure 5(a) shows early saturation and the module-wise performance indicates overfitting despite the regularization tricks in 22..3 and Ap- pendix[A.6 While more stable RL algorithms like Actor-Critic Trust Region Policy Optimization (Schulman et al. 2015) or Memory Augmented Policy Optimization (Liang et al. 2018) can mitigate these issues, we leave them for future exploration. Also, though this work's s objective was to train module networks with weak supervision, the sparse confounding rewards in the exponential action space indeed render the RL training quite challenging. One practical future direction to bridge the performance gap would be to pretrain with strong supervision on at least a subset of reasoning categories more constrained forms of synthetic questions, similar GenBERT. Such setting would require inspection and evaluation of generalizability of the RL model to unknown reasoning types more open-ended questions.
8

---

Under review as a conference paper at ICLR 2021
5 RELATED WORK
In this section we briefly compare our proposed WNSMN to the two closest genre of models that have proven quite successful on DR i) reasoning free hybrid models NumNet, NAQANet, NABERT, NABERT+, MTMSN, and NeRd ii) modular network for reasoning NMN. Their main distinction with WNSMN is that in order address the challenges of weak supervision, they obtain program annotation from the QA pairs through i) various heuristic parsing of the templatized queries in DROP to get supervision of the reasoning type (max/min, diff/sum, count, negate). ii) exhaustive search over all possible discrete operations to get supervision of the arguments in the reasoning.
Such heuristic supervision makes the learning problem significantly simpler in the following ways
These models enjoy supervision of specialized program that have explicit information of the type of reasoning to apply for a question e.g., SUM(10,12)
A simplistic (contextual BERT-like) reader model to read query related information from the
passage trained with direct supervision of the query span arguments at each step of the program
A programmer model that can be directly trained to decode the specialized programs
Executing numerical functions (e.g., difference, count, max, min) either by i) training purely neural modules in a strong supervised setting using the annotated programs or by ii) performing the actual discrete operation as a post processing step on the model's predicted program. For each these previous works, it is possible to directly apply the learning objective on the space of decoded program, without having to deal with the discrete answer or any non- differentiability.
However, such heuristic techniques of program annotation or exhaustive search is not practical as the language of questions or the space of discrete operations become more complex. Hence WNSMN learns in the challenging weak weak-supervised setting without any additional annotation through
A noisy symbolic query decomposition that is oblivious the reasoning type and simply based on generic text parsing techniques
An entity specific cross attention model extracting passage information relevant each step of the decomposed query and learning an attention distribution over the entities of each type Learning to apply discrete reasoning by employing neural modules that learn to sample the operation and the entity arguments
Leveraging a combination of neural and discrete modules when executing the discrete operation, instead of using only neural modules which need strong supervision of the programs for learning the functionality Fundamentally different learning strategy by incorporating inductive bias through auxiliary losses and Iterative Maximal Likelihood for more conservative initialization followed by REINFORCE
These reasoning-free free hybrid models are not comparable with WNSMN because of their inability to learn in absence of any heuristic program annotation. Instead of learning to reason based on only the final answer supervision, they reduce the task to learning to decode the program, based on heuristic program annotation. NMN is the only reasoning based model that employ various auxiliary losses to learn even in absence of any additional supervision, similar to us.
To our knowledge WNSMN is the first work on modular networks for fuzzy reasoning over text in RC framework, to handle the challenging cold start problem of the weak supervised setting without needing any additional specialized supervision of heuristic programs.
6 CONCLUSION
In this work, we presented Weakly Supervised Neuro-Symbolic Module Network for numerical reasoning based MRC based on generalized framework of query parsing to noisy heuristic programs. It trains both neural and discrete reasoning modules end-to-end in a Deep RL framework with only discrete reward based on exact answer match. Our empirical analysis on the numerical -nnserr only subset of DROP showcases significant performance improvement of the proposed model over SoTA NMNs and Transformer based language model GenBERT, when trained in comparable weakly supervised settings. While, our knowledge, this is the first effort towards training modular networks for fuzzy reasoning over RC in weakly-supervised setting, there is significant scope of improvement, such as employing more sophisticated RL framework or by leveraging the pretraining of reasoning.
A detailed related work section is presented in the Appendix A.4

---

Under review as a conference paper at ICLR 2021
REFERENCES
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Deep compositional question answering with neural module networks. CoRR, 1..0119,,,0799, 2015. URL http://arxiv
org/abs/
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In 2016 IEFE Conference on Computer Vision and Pattern Recognition, CVPR 20/6, Las Vegas, NV, USA, June 27-30, 2016, pp. 39-48. IEEE Computer Society, 2016. doi: 10. N..109/CVPR.201.... URL
https://dcis crg. 1ww.//12.../.... 2016.12
Ghulam Ahmed Ansari, Amrita Saha, Vishwajeet Kumar, Mohan Bhambhani, Karthik Sankara- narayanan, and Soumen Chakrabarti. Neural program induction for kbqa without gold pro- grams query annotations. In Proceedings of the Twenty-Fighuh International Joint Con- ference on Artificial Intelligence, IJCAI-19, pp. 4890-4896. International Joint Conferences on Artificial Intelligence Organization, 2019. doi: 10.24963/ijcai 2019/677. URL https
V/doi. org. 0. 24963/1 /ea.i/////ii...//////...iiaiicci 2019/6019 9/979
Danqi Chen and Christopher Manning. A fast and accurate dependency parser using neural networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 740-750, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.31 1//11/11111111/1 1082. URL https::/www.dat. hwtpeeeb..rrrraaaoooo/////////// 4-
Xinyun Chen, Chen Liang, Adams Wei Yu, Denny Zhou, Dawn Song, and Quoc V. Le. Neural symbolic reader: Scalable integration of distributed and symbolic representations for reading comprehension. 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL RRLL/tttss://opeeeeee
net./
forum? d ]NREFWH
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2018. URL http://arxiv.. www. abs/ 1810. 04805
cite 1eriv:1810.04005.oo
13 pages.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proc. of NAACL, 2019.
Avia Efrat, Elad Segal, and M. Shoham. Tag-based multi-span extraction in reading comprehension. ArXiv, abs/1909, 13375, 2019.
Mor Geva, Ankit Gupta, and Jonathan Berant. Injecting numerical reasoning skills into language models. In ACI, 2020.
Xiaoxiao Guo, Tim Klinger, Clemens Rosenbaum, Joseph P. Bigus, Murray Campbell, Ban Kawas, Kartik Talamadupula, Gerry Tesauro, and Satinder Singh. Learning to query, reason, and answer questions on ambiguous texts. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL htttpp:://opeeeeiiiet net
Nitish Gupta, Kevin Lin, Dan Roth, Sameer Singh, and Matt Gardner. Neural module networks for reasoning over text. In International Conference on Learning Representations, 2020. URL
https:///genreviee..nr for.:tte/ooooommmmdsss.awwwwaa
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni. and Nate Kushman. Learning to solve arithmetic word problems with verb categorization. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 523-533, Doha, Qatar, 1058.. URL
October 2014. Association for Computational Linguistics. doi:
httos: // www. awwwecllbb..... org http///ttoooooo/114-100000
Minghao Hu, Yuxing Peng, Zhen Huang, and Dongsheng Li. A multi-type multi-span network for reading comprehension that requires discrete reasoning. In Proceedings of EMNLP, 2019.
10

---

Under review as a conference paper at ICLR 2021
Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to reason: End-to-end module networks for visual question answering. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp. 804 813. IEEE Computer Society, 2017. doi: 10. 10.1109//CC 10910//222 URL WtRlhhttss::/ddoo www.oii.rgg/11..
CCVV 20177.1
Ting Huang, Zhi- Zhi-Hong Hong Deng, Gehui Shen, and Xi Chen. A window-based self-attention approach for URL
sentence encoding. Neurocomputing. 375:25-31, 2020. doi:
https://dci...
101 6/1 neucom. 2019 09.024)
Jambay Kinley and Raymond Lin. Nabert+: Improving numerical reasoning in reading comprehen- sion. 2019. URL httppp/s/gg..hub...om/aaaaaaa
com/ wwtp/////aalll//o/ooroooooo////.
Rik Kennie--Kdziorski,, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. MAWPS: A math word problem repository. In Proceedings of the 2016 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1152-1157, San Diego, California, June 2016. Association for Computational Linguistics. doi:
10.18653/1//N16-11336 URL http:::ttts//tttm //ww:://wwwww... aclweb. org/ wttp.//aaattooooo/g//1-------11/
Chen Liang, Jonathan Berant, Quoc Le, Kenneth D Forbus, and Ni Lao. Neural symbolic machines: Learning semantic parsers freebase with weak supervision. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pp. 23-33, 2017.
Chen Liang, Mohammad Norouzi, Jonathan Berant, Quoe V Le, and Ni Lao. Memory augmented policy optimization for program synthesis and semantic parsing. In Bengio, Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi. and R. Gamett (eds.), Advances in Neural Informa- tion Processing Systems 31, pp. 10015-10027. Curran Associates, Inc., 2018.
Arvind Neelakantan, Quoe V. Le, Martín Abadi, Andrew McCallum, and Dario Amodei. Learning a natural language interface with neural programmer. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https:://pen.ee////
net wwt//etttttte weeee//oommiidiemm www.e//oooooodddyyeeee.....
Qiu Ran, Yankai Lin, Peng Li, Jie Zhou, and Zhiyuan Liu, NumNet: Machine reading comprehension with numerical reasoning- In Proceedings of the 2019 Conference on Empirical Methods in Natural Language P Processing and the 9th International Joint Conference on Natural Language pp. 2474 2484, Hong Kong, China, November 2019. Association for Computational Linguisties. doi: 101.1/65///DDD11111 1251. URL https: wwww aclweb. org/
Processing
anthology/C19- 125
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing.
Association for Computational Linguistics. 11 2019. URL http::/ http://axxvv.or atx.vv.or//ab//10000
10084
Subhro Roy and Dan Roth. Solving general arithmetic word problems. In Proceedings of the
2015 Conference on Empirical Methods in Natural Language Processing, pp. 1743-1752, Lisbon, Portugal, September 2015, Association for Computational Linguistics. doi: 10. 11553/11/D15-1///
URL https://www wwwweb wttp///atttooogg///S....
Amrita Saha, Ghulam Ahmed Ansari, Abhishek Laddha, Karthik Sankaranarayanan, and Soumen
Chakrabarti. Complex program induction for querying knowledge bases in the absence of gold programs. Transactions of the Association for Computational Linguistics, 7:185-200, March 2019. doi: 10.11622tacl ..00222... URL hnLtst:: :/ wwww://wwwc..........o /ttp://ccieee..o//rttttooooo//// wttp///ntttoooyyy11111100000
Adam Santoro, David Raposo, David G. T. Barrett, Mateusz Malinowski, Razvan Pascanu, Pe- ter W. Battaglia, and Tim Lillicrap. A simple neural network module for relational reason- ing. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pp. 4967 4976, 2017. URL http: h nips cc htt.pppp.ccpppprrr//p/m paper/ 7082 simpll neural net module- for onal

---

Under review as a conference paper at ICLR 2021
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust policy optimization. volume 37 of Proceedings of Machine Learning Research, pp. 1889 1897, Lille, France, 07-09 Jul 2015. PMLR. URL
htm
Sanjay Subramanian, Ben Bogin, Nitish Gupta, Tomer Wolfson, Sameer Singh, Jonathan Berant, and Matt Gardner. Obtaining Faithful Interpretations from Compositional Neural Networks. In Association for Computational Linguistics (ACL), 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is you need. I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 5998-600000 Curran Associates, Inc., 2017. URL attpt attent on- all you- need. pdf
7/papers www.nip...cmmmmmm ccppppar//pppp11cccc.. -ttp:/77111-attttteeee
R. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning. 8:229-2555 1992.
Adams Wei Yu, David Dohan, Quoc Le, Thang Luong, Rui Zhao, and Kai Chen. Fast and accurate reading comprehension by combining self-attention and convolution. In International Conference on Learning Representations, 2018. /tRL/tttps::/opeeeeiiee
Rw
Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning, 2017.
12


Reviews:
[{"review_id": "jYCEgv7LSK4", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe paper proposes a new model for numerical reasoning in machine comprehension. Given a passage and a query, the model outputs an arithmetic expression over numbers/dates in the passage (e.g. max(23, 26, 42)). The model is trained with weak supervision in the form of numerical answers only. This weak supervision is used to define reward for reinforcement learning training. A key claimed advantage of the model compared to the prior art is that it trains end-to-end from the rewards as the only form of supervision. This is contrasted to  neural module networks, which require program supervision for good performance, as well as GenBERT, which requires additional synthetic training data for pretraining. Two key quantitative results include: \nbetter performance on the DROP-num datasets, compared to NMNs with less supervision and GenBERT without data augmentation\ncomparable to strongly-supevised NMN performance on DROP-Pruned-num.\n\nThe general approach is quite elegant and makes sense. It is encouraging that the paper reports successful training with RL. It is also important to build models that use less extra supervision. \n\nThat said, I have some concerns regarding the paper’s positioning. The introduction, as well as many other places in the text categorizes some of the prior art as “learning a multi-type answer predictor over different reasoning types (e.g., max/min, diff/sum, count, negate) and directly predicting the corresponding numerical expression, instead of learning to reason”. What is learning to reason then? For example, in NAQANet the model predicts whether the numbers in the passage should be summed or subtracted from each other, why is this not learning to reason? Second, calling this model a “module network” is misleading, in my opinion. In neural module networks modules learn to do things, and here the key modules of discrete reasoning are predefined. The model also contains “modules” that are conditioned on different question spans. But there is no experiment checking whether having multiple such modules is actually useful. There is furthermore no qualitative explanation of what these modules are supposed to do and what they are actually doing. Lastly, I am not sure the use of synthetic data for GenBERT can be called “strong supervision”. Data augmentation and strong supervision are not the same thing. \n\nThe paper is very dense and is quite hard to read. A lot of space is allocated to a very detailed technical presentation of “modelling interactions”, while the high-level picture of how the model functions is still hard to grasp. For example, Figure 2 is confusing because it has a “Stacked Span Prediction” pathway that leads to nowhere. Reading the text I find that apparently the output of this part is actually used for “modelling interaction between programs and number entities”, which is in the right part of the figure. These basic high-level architectural decisions are hard to understand as the reader is overwhelmed by technical details, such as sliding windows and scaling factors for various attentions. \n\nThe way the results are displayed in the table is somewhat confusing. The fully-supervised NMN baseline is shown in Table 2 for DROP-Pruned-num but is not shown in Table 1 for Table-num. Instead, it can be found in Table 3. I would recommend presenting all results in one table, even if it shows that the current model performs worse than others with more supervision or data augmentation. Furthermore, I think that comparing top-k accuracies for the proposed model and top-1 accuracies for other models, as it is done in Table 3, does not make sense.\n\nIn summary, while I think the paper might be proposing an interesting model with promising results, I also think that presentation needs work. It would be great to see a clearer high-level explanation of the difference between the proposed model and the prior work. To this end, the prior work should be better discussed (notably there is no Related Work section at the moment). Besides, more quantitative and/or qualitative results are needed to support the hypothesis that the model performs “noisy query execution”. \n\nOther minor comments:\n- it is the first time I see the word skyline used to mean “the baseline from above” \n- there is a lot of really long sentences in the paper, which makes the reading very hard. I’d recommend to try and break them up into shorter ones. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "VHtXLx8adyV", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "ICLR 2021 Review\n\nTitle: Elaborate modeling with cutting-edge techniques, but limited on generalizability and lacks analysis.\n\nThis paper introduces a method to train neuro-symbolic module networks with weak supervision signal, and out-performs previous models on datasets containing numerical problems. Dislike previous works, WNSMN (Weakly-Supervised Neuro-Symbolic Module Network) doesn't rely on strong supervision and annotations but incorporates many inductive biases for numerical operations. It learns to execute programs that are extracted from dependency parse with noisy heuristics and are trained end-to-end in a reinforcement learning framework. \n\nNeural Module Networks are a very promising trend for machine reading comprehension for that it explicitly involves the reasoning process, but training such a system requires expensive annotations. Weakly supervised learning was infeasible because of the exponentially large action space, but this paper manages to address this issue by some inductive biases. To make the searching efficient, WNSMN designs modules for the interaction between program and passage, as well as programs and entities. These inductive biases are implemented with an attention mechanism and make the action size tractable. Besides, the sampling process is also learnable, with two carefully designed samplers -- operator and argument samplers. \n\nThe training technique also plays an important role. Instead of training with discrete binary rewards and from scratch, WNSMN starts with hard EM objective with heuristic silver answers. The training process with the reinforcement learning objective adopts a multi-arm bandit assumption and entropy penalty. These techniques alleviate the problems of instability and overfitting.\n\nOverall this paper is brilliant, adopts proper techniques, and elaborately designs the architecture. However, it's also limited in generalizability and needs more analysis.\n\nCompared to the two baseline works, i.e. Neural Module Networks (NMNs) and GenBERT, WNSMN only works for numerical problems. The modules designed in the model, like the entity attention and operator sampler, are ad-hoc for problems involving dates and numbers. It's hard to extend WNSMN to more general cases. So it's questionable whether this framework could be applied to other problems of DROP like PERSON or VERB PHRASE.\n\nOne of the crucial pathologies of weakly supervised learning is the spurious prediction, where the model outputs the correct with wrong steps. However, this paper doesn't analyze this problem with WNSMN. So I'm wondering what percentage of the correct predictions of WNSMN is derived from the correct reasoning process? This could be studied with a small subset of human-annotated data and is helpful to rule out the possibility that the model learns some un-interpretable patterns.\n\nThe first step of WNSMN, i.e. query parser, is rule-based and makes less sense in contrast to other parts. It severely limits the ability of the model by just incorporating the first layer of the dependency parse tree. How can this model deal with complex patterns that are not step-by-step? How do you guarantee that the dependency relationship between steps is from left to right? I'm not convinced by the explanation \"the QA model needs to be robust to such noise and additionally rely on the full query representation in order to predict the discrete operation.\" It will be clear if you can post statistics on the percentage covered by this simple heuristic.\n\nQuestions:\n\n1. Is there a specific reason that you focus on numerical problems?\n2. During hard EM, you choose the best action based on your current policy. How many good actions do you have in the first step? If there are too many good actions, it's still hard for the model to learn from correct signals.\n3. Continued from question 2, why do you choose the best action, instead of smoothed over all the good actions?\n4. To train the in-house baseline models, do you train them on the whole dataset or just the subset that involves numbers and dates? Is that possible that NMNs or GenBERT could learn useful patterns from other types of questions to answer numerical questions?\n\nPresentation suggestions. The paper presentation is clear and dense overall. My suggestions would be:\n1. Limit the content of describing the interactions. Section 2.1.1 is overwhelming and intimidating to me, with a lot of definitions and equations. The idea behind the interactions, however, is not that complex. Some descriptions of the methods could either be abridged or moved to the appendices, like the multi sliding windows and multi-scaling tricks given they're not your contribution. Also, the interactions between program/passage and program/entity are very similar, and you might be able to refactor these two paragraphs.\n2. Curly brackets should be used for the set instead of ordered vectors. E.g. stacked attention windows in section 2.1.1.\n3. Words in equations should be surrounded by \\mathrm or \\mathit to make them less like the production of many variables.", "extra_scores": {}, "extra_text": {}}, {"review_id": "TMGmxPAnOYn", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a neurosymbolic module network that predicts a program structure following a dependency parse, populates that program's arguments, and executes it to answer numerical reasoning questions over text.  They claim that compared to Gupta et al. (2020), this approach doesn't require as many domain-specific heuristics to find gold programs or as much precomputation -- it is learned with weak supervision only (just the answers). The model has a number of pieces allowing the model to reference entities, numbers, and dates in a cross-attentive fashion. Results show that on numerical questions from the DROP dataset, the model outperforms that of Gupta et al. and is competitive with other approaches when appropriate assumptions are made.\n\nIn general, I like what this paper is trying to do. In a vacuum, I might recommend accepting it. However, my most pressing concern with this paper is the fairly fundamental similarity to this ICLR 2020 spotlight paper:\n\nhttps://openreview.net/pdf?id=ryxjnREFwH\nNeural Symbolic Reader: Scalable Integration of Distributed and Symbolic Representations for Reading Comprehension\nXinyun Chen, Chen Liang, Adams Wei Yu, Denny Zhou, Dawn Song, Quoc V. Le\n\nI will refer to this paper as Chen et al. and their system as NeRd.\n\nThe authors failed to cite this paper, so I presume they are not aware of it. There are many similarities, including the broad strokes behind the DSL and the learning approach with weak supervision. I will discuss this current paper in contrast to that one to highlight the differences and the contributions.\n\nModel: My favorite aspect of the current work is the fact that it leverages dependency representations. I like this idea a lot and buy that it might be a path to more general models of this form. However, it's hard for me to evaluate whether this is really more general due to the evaluation focused on this dataset, so this contribution feels somewhat theoretical. Would this dependency-based formalism work well elsewhere, or does it just happen to work well for DROP? It feels hard to make this work for larger, more compositional problems where the syntax-semantics divide is going to manifest itself more strongly (e.g., surrounding quantification).\n\nThe thing I like least about this paper is the complexity of the model.  The NeRd programmer is substantially simpler, as a basic seq2seq model. While you can argue that they have a more aggressive approach for warm-starting, like Gupta et al., there's less inductive bias built into the model directly.  By contrast, the current paper has a large number of bespoke modules for handling things from the DROP dataset.  I'm okay with things like treating entities as first-class concepts and allowing attention over them. But specialized attention maps targeting numbers and dates starts to feel very tailored to this dataset. Given the focus on numerical reasoning, perhaps some of this is to be expected, but I believe the other past approaches (NeRd, Gupta) have architectures that are a bit more general.\n\nDSL: The DSLs between these two approaches are quite similar, but this is to be expected as the task is the same.\n\nWeak supervision: Chen et al. use some heuristics as well as data augmentation to start learning off by giving the model access to a larger number of programs which find the right answer right off the bat. To handle spurious programs, they use hard EM wth an additional thresholding step to avoid training on programs that are extremely unlikely.  The present work initializes with a hard EM-style learning before switching to REINFORCE. In my view, there is not much of a conceptual advantage to this approach over NeRd. I don't think this paper has a particularly keen insight to solve the cold-start problem for RL here, and the NeRd-style data augmentation is not that big of a weakness. So I don't see a fundamental contribution here over NeRd.\n\nResults: Table 1 looks convincing but I'm having a hard time fully understanding it. It seems like the low numbers for NMN in DROP-num-Test are because this includes many question types explicitly excluded by the original Gupta et al. paper -- is this correct? This is fair but I think should be more explicit in the table.\n\nAs for Table 2, the ablations of different attention methods are interesting, but I'm not sure this is a fully fair comparison to Gupta et al. These pieces can't be mapped in an apples-to-apples fashion to aspects of the current approach. Gupta et al.  could just as easily say: well, let me ablate your specialized date handling module and do a comparison on that axis. In a new setting, we can claim such sources of supervision might be unavailable, but it's not clear to me that the system with this component deleted is the fairest point of comparison.\n\nIt also seems based on the Table 3 comparisons, we do not expect WNSMN to perform well compared to NeRd (see Table 4 in that paper).\n\nOVERALL\n\nNormally I am not a stickler for novelty over prior work. However, in this case, I feel compelled to judge this paper and directly compare it to Chen et al., as the motivation, evaluation conditions, and technical details of these papers bear striking similarities. (That paper also evaluates on MathQA, which is another point in its favor.) And in summary, this work does not offer enough new material to justify acceptance.  It is quite well-done work and has some interesting insights, though it suffers from an overly complex neural model. But the insights are not new, and the results are not enough to convince me that it should be accepted.", "extra_scores": {}, "extra_text": {}}, {"review_id": "1onfOub2uZe", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a weakly supervised module networks to solve numeric reasoning problems. The model is trained with reinforcement learning over the modules. The model solved a few challenges in training the RL model. They evaluated their model on a subset of the DROP dataset. \n\nThis paper was clearly presented and the results was promising. My only concern is that they only evaluate the model on a carefully constructed subset. I am not sure how it compare to other models in more general settings and/or on other datasets. \n\nThe authors mentioned that most questions in the dataset only requires 1-hop reasoning, so they reduced their model to 1-step MDP. Will your model work on questions that involves multi-step reasoning? Can you show that?\n\nDid you detect the query span as a preprocessing step? What if the query spans are noisy/missing/duplicated? \n\n* I did not check the correctness of the equations.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a weakly supervised model for numerical reasoning. After discussion with the reviewers it seems that it is already known that training NMNs directly on DROP is not successful and requires taking additional measures. Past work (NERD) has resorted to using data augmentation, and this work encodes it directly to the model. This paper needs to show the advantages of their approach and that it generalizes better to other scenarios. Other minor issues include (a) clarity fo writing (b) focus on a subset of questions (c) no evaluation on other numerical datasets (d) mild inaccuracies w.r.t prior work (GenBERT)
Decision: Reject