RECKONING: Reasoning through Dynamic Knowledge Encoding
Authors: Zeming Chen
Gail Weiss
Eric Mitchell
Asli Celikyilmaz
Antoine Bosselut
Venue: NeurIPS 2023 poster (2023)
Keywords: natural language processing
multi-hop reasoning
knowledge memorisation
TL;DR: RECKONING is a bi-level learning algorithm that improves language models' reasoning ability by folding contextual knowledge into parametric knowledge through back-propagation.

Abstract: Recent studies on transformer-based language models show that they can answer questions by reasoning over knowledge provided as part of the context (i.e., in-context reasoning). However, since the available knowledge is often not filtered for a particular question, in-context reasoning can be sensitive to distractor facts, additional content that is irrelevant to a question but that may be relevant for a different question (i.e., not necessarily random noise). In these situations, the model fails to
distinguish the necessary knowledge to answer the question, leading to spurious reasoning and degraded performance. This reasoning failure contrasts with the model’s apparent ability to distinguish its contextual knowledge from all the knowledge it has memorized during pre-training. Following this observation, we propose teaching the model to reason more robustly by folding the provided contextual knowledge into the model’s parameters before presenting it with a question. Our method, RECKONING, is a bi-level learning algorithm that teaches language models to reason by updating their parametric knowledge through back-propagation, allowing them to answer questions using the updated parameters. During training, the inner loop rapidly adapts a copy of the model weights to encode contextual knowledge into its parameters. In the outer loop, the model learns to use the updated weights to reproduce and answer reasoning questions about the memorized knowledge. Our experiments on three diverse multi-hop reasoning datasets show that RECKONING’s performance improves over the in-context reasoning baseline (by up to 4.5%). We also find that compared to in-context reasoning, RECKONING generalizes better to longer reasoning chains unseen during training, is more robust to distractors in the context, and is computationally more efficient when multiple questions are asked about the same knowledge.

RECKONING: Reasoning through Dynamic
Knowledge Encoding
Zeming Chen'
Gail Weiss' Eric Mitchell
Asli Celikyilmaz' Antoine Bosselut
EPFL' Stanford University? Meta Al Research"
(zeming. chen, fail.weisse antoine peeselut)@@epl..co
aric.mitcheelleee. stanford edu aslic@meta.co
Abstract
Recent studies on transformer-based language models show that they can answer questions by reasoning over knowledge provided as part of the context (i.e., in- context reasoning). However, since the available knowledge often not filtered for particular question, in-context reasoning can be sensitive to distractor facts, addi- tional content that irrelevant question but that may be relevant for different question (i.e., not necessarily random noise). In these situations, the model fails to distinguish the necessary knowledge to answer the question, leading spurious reasoning and degraded performance. This reasoning failure contrasts with the model's apparent ability distinguish contextual knowledge from all the knowl- edge it has memorized during pre-training. Following this observation, we propose teaching the model to reason more robustly by folding the provided contextual knowledge into the model's parameters before presenting it with question. Our method, RECKONING, is bi-level learning algorithm that teaches language mod- els to reason by updating their parametric knowledge through back- boockpropagation. allowing them to answer questions using the updated parameters. During training, the inner loop rapidly adapts copy of the model weights to encode contextual knowledge into its parameters. In the outer loop, the model learns to use the updated weights reproduce and answer reasoning questions about the memorized knowledge. Our experiments on three diverse multi-hop reasoning datasets show that RECKONING'S performance improves over the in-context reasoning baseline (by up to 4.5%). We also find that compared to in-context reasoning, RECKONING generalizes better to longer reasoning chains unseen during training, is more robust to distractors in the context, and is computationally more efficient when multiple questions are asked about the same knowledge.
Introduction
Consider the sentence: "John is David's dad, and Tom is John's dad". Concluding that Tom is David's grandfather involves reasoning about the information in the sentence. Specifically, requires understanding the direct information, contextual knowledge, given in the sentence: the stated relationships between John, David, and Tom; and combining it with our existing, commonsense knowledge of the world: someone's 's dad's dad is their grandfather. Achieving such logical reasoning automatically has long been a goal of AI
74 82
The example above demonstrates two necessary abilities required for successful reasoning: first, holding large amounts of commonsense or general knowledge about the world, and second, processing and combining new information with existing knowledge. Transformer-bie have shown a remarkable capacity for the first of these abilitics, repeatedly being demonstrated to memorize large amounts of data, or parametric knowledge, in their weights g
large language models
50 64
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Who is Pater to Griffin?
Knowledge Encoding
V
Relevant Facts X Peter Kyle.
Kyle is Julia's son.
L
Grandson
honer
Amy is daughter Griflin,
Amy is Kyle's sister.
0
Figure I: Our algorithm, RECKONING, solves reasoning problems by encoding external contextual knowledge into model's parameters through gradient updates. At inference time. RECKONING performs a few parameter updates using the gradients of a language modeling loss to encode the relevant facts. Then, the updated model answers the question using only its implicit knowledge.
For the second, recent work showed that transformers fine-tuned to predict answers over concate- nated context ("The cow is big; If something is big then it chases the dog; If the cow chases the dog then the cow sees the rabbit") and question ("Did the cow see the rabbit?"") achieve high performance on reasoning tasks where all necessary knowledge is given in the context 17 We refer to this general setting as in-context reasoning (ICR).
In real-world question-answan answering settings 16
large amounts of contextual knowledge may be provided at once, and the information may not be perfectly filtered for specific question. Unfortunately, in-context reasoning is highly sensitive to distractors [70] additional facts that are not relevant question (e.g.. "The cow round" for the above example). Indeed, when fine-tuning and evaluating GPT-2 from 99.4% only 70.9% accuracy for the same questions ($4..)) This sensitivity distractors in contextual knowledge contrasts with GPT-2's apparent robustness to distractors in parametric knowledge: for any specific example, most of the training data seen by GPT-2-which forms its parameters-is is likely to be completely irrelevant to that example. Naturally, we wonder whether presenting contextual knowledge in the same way as memorized knowledge, by encoding into model's 's parameters, will improve the reasoning abilities of transformer- based language models.
660 for ICR, we find that adding distractors to the context drops performance
In this work, propose novel bi-level optimization algorithm, RECKONING, that learns to memorize (and reason) over facts (i.e., knowledge) by performing inference-time parameter updates using gradients computed from a language modeling loss those facts. The updated model is then used to answer any questions about those facts. Our training framework involves two nested loops: the inner loop performs fast adaptations from set of initial weights memorize set of external knowledge through a few gradient updates, and the outer loop optimizes those same initial weights such that the updated model will solve reasoning problems associated with the memorized knowledge. In other words, the outer loop learns optimal meta- peetae-arrmssrss that can rapidly memorize and successfully reason over contextual knowledge, allowing knowledge memorization be optimized directly for downstream reasoning. At inference time, instead of including external knowledge in the input sequence as the prefix to a question prompt, the model can encode its parameters through gradient updates and then reason over its updated parametric knowledge to reach a conclusion.
We evaluate RECKONING two synthetic multi-hop reasoning datasets: ProofWriter [74] and SeerR-Syytematti--Gneeee
(CLUTRR-SG) (2)], and one real-world dataset, FOLIO 30
comparing against fine-tuned ICR (FT-ICR) baseline that uses the same underlying model. Our results show that RECKONING consistently outperforms the FI-CCR baseline on each benchmark, demonstrating that it successfully learns to answer multi-hop reasoning questions as desired. In particular, we find that RECKONING more successfully generalizes to adversarial settings, such as the presence of distractor facts and the introduction of longer reasoning chains at inference time. Finally, while the inference-time gradient updates make RECKONING slower to process new knowledge than a typical ICR forward pass, our run-time analysis shows that RECKONING is more efficient when answering multiple questions about shared knowledge set. This speedup occurs because RECKONING only needs to encode the knowledge once to answer multiple questions about it. Overall, we demonstrate that RECKONING is an effective algorithm for reasoning through dynamic and controllable knowledge encoding, overcoming an observed weakness the common reasoning setting and providing multiple additional benefits.

---

Outer-loop Learning
Vela
Reasoning Problems
For each (R,
Who is Peter Griffin?
Question (x,
Innenloop Learning
y:
V
Relevent Facts X
Roter a son to Syle.
Kyle Julies
Any is daughter ta Grifle.
L lnner
Amy kylc's
facts 3
Figure 2: The two-stage training process of RECKONING with an inner and outer loop.
Background
y to refer to parameterised functions in which X' is the set of possible inputs and 0 are their possible weights (parameters). We use fe x + f(x, 0) to easily refer to any f with given set of parameters O. We describe reasoning problems using tuples (K, æ, y" Y) such that y is the correct answer for the question æ given facts K, and use D to refer to sets of such problems. When it is clear from context, we drop Y and use only (K, v²))
Notation We use : X
Language Modeling and Memorization In the causal language modeling (CLM) objective, parameterized model fo is trained to estimate the conditional probabilities of each token in sequence given its predecessors: (x)))) Specifically, we train so to approximate p using the CLM loss:
T
(c) = Σ log
t=1
This training objective allows language models to memorize individual training examples [IO and we will exploit this ability to memorize and draw on contextual knowledge in our work.
given reasoning problems (K,, æ. y", Y) in natural language and attempt to recover the correct answer y" from the context K., questiin æ, and possible answers Y alone. In in-context reasoning, language models fo trained with CLM objective are applied to this task by selecting as the response the answer y E Y with maximum probability according to the model's next-token prediction from the concatenated context and question: supervised fine-tuning, transformer language models achieve high performance in this setting [17 29 74 though this degrades significantly in the presence of irrelevant facts (distractors) 770
Transformers as Soft Reasoners In natural language reasoning tasks,
arg m&xy'ey [a(s/ æ=)). Previous works show that, after relevant
Method
Addressing these challenges, propose RECKONING (REasoning through dynamiC KnOwledge eNcodING), which solves reasoning problems by memorizing the provided contextual knowledge, and then using this encoded knowledge when prompted with downstream questions. Specifically, RECKONING uses bi-level optimization to learn a set of meta-parameters primed to encode relevant knowledge in limited number of gradient steps. The model can then its updated weights to solve reasoning problems over this knowledge, without further presentation of the knowledge itself.
Inference Given reasoning problem (K,, Y), we initialize our model with weights copied from set of meta-parameters 0 and perform a constant number of gradient descent steps on these with the goal of minimizing the CLM objective on the knowledge set K. This allows the model to memorize K. in its updated parameters, which refer to 0% Next, pass the question x to the model, using føg to obtain a distribution over Y, and taking as output the answer y E Y with the highest probability. For this method to consistently output the ground truth y", we seek set of optimal parammaarsesers 0* that can quickly memorize (i.e., learn) the given knowledge in way that then allows accurate reasoning when queried about the knowledge downstream.
Overview:

---

Training RECKONING Similar to the meta- meaannen algorithm MAML (25], which focuses on optimizing the initial parameters of a model so that the model can be good at learning new tasks with a few examples, we also optimize the initial parameters such that the model learns to encode knowledge for more effective reasoning. Given a distribution p(D) of reasoning problems, our proposed bi-level optimization framework RECKONING (seen in Figure
2) optimizes the following objective:
0' E arg min
E
[C (3))
(K,z,))p(D)
where for all KC, E N, and 0: of O. and
@c t a (CLCLM(Oog..))
Here, (xr( y) denotes the cross-entropy (CE) loss, which we apply with the relevant parameters for each reasoning question in D. (CLM K.) सें ELexfcLM((
denotes the causal language modeling loss, and N and a are pre-defined hyperparameters of the fine-tuning. We seek our actual meta-parameters 0 through gradient descent. In particular, denoting by Oo our initial meta-parameters, and 0% the parameters 0 obtained when initializing 0%
with O,, we iteratively compute
0 = 0: v
Croaa Saas K. U).
(4)
Dr
(K,...))
where (f, (f, x,
(c)( y)
Algorithm RECKONING
and for each i, D; is randomly sampled from p(D). This continues until Crows converges.
Require: An example distribution p(D), a transformer lan- guage model f. initial meta-parameters 0, outer step size 7), inner step size a, inner loop length N. & outer loop
The training can be seen as two nested loops: at each iteration, the outer loop (Equa- tion (4) samples a random batch DD S DD of reasoning problems for evaluating (in order to update) the current meta- wetaa-parameters O:. after the inner loop (Equation (3) adapts them to encode the associated knowledge through N steps of gradient updates.
1:
while not converged
2: Sample D' ~ p(D)
3: Lo t 0
4:
for each (K,, m. E DD do
5: Initialize 6 = 0
6:
for t N do
inner loop
7:
0%+ + 0% VVVLLMMC (oor
8:
end for
9:
LT + Lr Limalfag K,, x,
Multi-Task Objective Through our exper-
iments, we find that adding knowledge- 10:
end for
recovery objective to the outer loop such 11: 0 4- 0 V fDT
that the model must also state all of K. when 12: end while
prompted with i-improves the model's reasoning performance. We evaluate knowledge recovery
with CLM loss and combine the two losses by simple addition, following prior works (2 77 78
The entire change is achieved by redefining the total loss in our outer loop (Equation as:
Toxal (J,K, x. y) = LCF( + LCLM (C)
(5)
where CCLM xx KC) the language modeling loss on K., as in Equation (3). but this time conditioned on the question x. The overall process for training RECKONING is depicted in Algorithm | and Figure [ Additionally, we dynamically learn a per-step-per-layer learning rate to replace the shared constant learning rate in the inner loop. We give more details in Appendix D
4 Experiments
Setup We conduct experiments three datasets focusing multi-hop logical reasoning over natural language knowledge: ProofWriter [74), which measures the model's ability to emulate reasoning over facts and rules expressed in natural language; CLUTRR-SG 29) which is generated from the CLUTRR 722 benchmark, a logical reasoning task that involves reasoning over family relationships between entities grounded in first-order logical proofs; and FOLIO [30], reasoning benchmark with first-order logical reasoning problems written by expert annotators based on real- world knowledge. Each problem in these datasets requires multiple reasoning hops to answer
In ProofWriter. the number of reasoning hops is called the proof depth. To unify the presentation of the results, we the term "hop" to describe the number of reasoning steps for both datasets.

---

We compare our method against the following baselines: (1) a fine-tuned model that performs forward pass on only the question without access to the knowledge (No-Facts), (2) fine-tuned model that performs forward pass on only the knowledge without access to the question (No-Question), (3) model trained using RECKONING with random knowledge that not relevant to the questions (Random-Factsy and (4) an ICR baseline that concatenates the knowledge IC with the question æ in a single context and is trained using supervised learning to predict the answer (FT-ICR). Our first three baselines sanity-check whether any surface-level patterns in the questions and facts can be exploited to make accurate predictions. The last baseline compares RECKONING to the conventional way of reasoning with language models. Unless stated otherwise, use the GPT-2-small [60] model (~ 124M parameters) as our initialization and refer by RECKONING to our method trained with the multi-task objective. We compute each score from the average across three different runs. For more details on the implementation, datasets, and examples, see Appendix and Appendix C
4.1 Multi-hop Reasoning Performance
Main Results We first evaluate whether RECKONING learns to per- form reasoning in the base setting. A model is given a set of supporting facts (without distractors) and ques- tion (or hypothesis) as input and be- gins by performing few CLM learn- ing steps on the facts. Then, the up- dated model reads only the question and generates an answer. To answer correctly, the model must reason over both facts and the question, meaning must encode the facts during the in- ner loop such that multi-hop reasoning can be performed over them later.
ProofWriter
CLUTRR-SG
Method
2-h
3-h
5-h
2-h
4-h
6-h
No-Facts
64.1
63.0
64.2
0.0
8.8
8.9
No-Question
66.2
67.0
65.2
35.7
36.4
28.7
Random-Facts
64.1
63.0
64.2
0.0
1.3
2.5
FT-ICRsT
98.4
98.8
97.8
97.4
91.3
89.1
FT-ICR
99.4
99.2
99.6
98.1
96.9
90.3
RECKONINGST RECKONING
98.3
98.3
99.1
96.0
90.2
91.2
99.5
99.7
99.8
98.3
97.6
94.8
Table 1: Label accuracy of RECKONING on ProofWriter and CLUTRR- SG,, compared to FT-ICR baselines where the supporting facts are given as part of the input. MT marks models trained with the multi-task objective, which optimizes both guestion-answering and knowledge memorization.
We train our models and the fine-tuned ICR (FT-ICR) baselines with both the single-task (Ler) and multi-task (CCE + LCLM) objectives. For multi-task (MT) training. the model learns to answer the question and generate its relevant knowledge in the outer loop. Table shows the evaluation results on question answering (or hypothesis classification). For all hop numbers in ProofWriter and CLUTRR-SG, multi-task RECKONING outperforms the best result of all baselines (consistently obtained by multi-task FT-ICR) by an average of 1%. We conclude that RECKONING can effectively solve reasoning problems through its updated parametric knowledge and do so better than existing baselines. The multi-task objective is crucial for this success: not only is RECKONING's performance consistently higher (by an average of 2.8% over the two datasets and their hop counts) when using the multi-task rather than single-task (ST) objective, but it also under- meddrr-erforms both FT- ICR baselines when trained with only the single-task objective. The multi-task objective also improves FT-CCR consistently (average 1. 1.8%), though it is not enough to beat the multi-task RECKONING. In all further experiments, we consider only RECKONING and FT-ICR with a multi-task objective.
Generalizing to Longer Reasoning Chains Our first experiments assume an alignment between the number of reasoning hops in the questions in the training and test set. However, we may not be able to train on all n-hop reasoning questions we encounter in the wild, and we rarely know the number of reasoning hops in a question a priori. Consequently, we also measure the generalization capacity of our model to questions with hop numbers unseen during training. We compile interpolation (fewer hops than the train set) and extrapolation (more hops than the train set) test sets from the CLUTRR- SG dataset. Again, we train models individually on 2-hop, 4-hop, and 6-hop examples and evaluate these three sets of models on the test sets, which contain 2- 2--0hhpp reasoning questions. Figure 3 shows that both RECKONING models and ICR basclines retain high performance on the interpolation test sets but exhibit decreasing performance as the number of hops increases. Importantly, though, RECKONING outperforms FT-ICR on all test sets regardless of the number of training hops, with the highest difference being more than 10% in every training setting (15% 30%, 10%, respectively). These performance gains when testing on extrapolation data suggest that training with RECKONING better generalizes to examples with high OOD hop counts than in nn-coneext reasoning (ICR).
5

---

Training on 2-hop Questions
Training on 4-hop Questions
Training on 6-hop Questions
RECKOONING
-B-FT-ICR RECCKONNNNG
RECKKNNING
30
Label Accuracy
60
20
20
2
10
2
3
4
5
6
10
10
Number of Test Hops
Number Hops
Number of Test Hops
Figure 3: System generalization evaluation on CLUTRR-SS -G. From left to right, the models are trained on 2-hop, 4-hop, and 6-hop CLUTRR-SS data portions. We evaluate the model on 2-10 hop test sets. The higher the hops, the more facts question has, and the more difficult that question is.
Training on hop Questions
Triining 1-hop Questions
Teaining S-hop Quedtions
BRECKONING BFTTICR
IN RECKONINNG #TTCRR
IRCCONNING BFFAC
25R
85.5
Accurtey
00.5
26.6
$1.
22.2
Distrcctioss
Number Distractors
Untractios
Figure 5: Robustness under distractors for ProofWriter. Each of the three plots corresponds to training and testing subset of questions in ProofWriter with different number of hops (2,3,5-hops). Each bar corresponds to the number of distractors in the knowledge sets for those questions.
Does RECKONING'S performance depend on the number of inner loop gradient steps? In RECK- ONING, the model performs multi-hop reasoning over facts by encoding facts using multiple gradient steps in the inner loop optimization
100
Label Accuracy (%)
80
($3) Naturally, this
process prompts the question of whether there is a
correlation between the number of reasoning hops
60
and the number of gradient steps needed to reliably encode the knowledge (i.e., problems with more rea- soning hops require more gradient steps in the inner loop encode the facts). In Figure 4 we show for CLUTRR-SG that as the number of inner loop steps increases, the label accuracy of the outer-loop task also increases. Furthermore, when considering the performance gains for reasoning with 6 inner loop steps (i.e., knowledge encoding) as opposed to one, observe that this gap is much more pronounced for 4-hop (42.3%) and 6-hop (34.7%) reasoning than it is for 2-hop reasoning (5.9%). These results show that problems requiring more hops of reasoning also greatly benefit from more steps of inner loop knowledge encoding.
2-hop
4-hop
6-hop
40
6
Inner Loop Steps
Figure 4: Multi-hop reasoning performance as a function of the number of inner loop steps (x-axis), with each line focusing (by training and testing) on CLUTRR-SG with a different number of hops.
4.2 Reasoning with Distractors
In cases where multiple questions must be answered about the same knowledge set, some knowledge that is relevant to one question will likely be irrelevant to another question. For example, in Table 7 the fact "Charlie is White." " not needed to answer the question "Harry red?". Thus, it is important to evaluate the robustness of RECKONING when there exists irrelevant information (i.e., distractors) in the knowledge set. this experiment we analyze RECKONING'S ability to focus on the correct knowledge and ignore distractors when answering questions. We use ProofWriter as the evaluation dataset since already has a setting with distractors included in the knowledge. For systematic analysis, gradually add distractors to the context (starting from 2 and finishing at all possible

---

distractors, of which there are an average of 7 per question). We train RECKONING and the baseline using the multi-task objective. where the model must (1) recall all of the facts and rules relevant to the question and (2) predict the conclusion based on the correct knowledge. In this case, we adapt training such that for each question æ, the outer-loop (Equation ( CLM loss is only computed with respect to the relevant facts from IC thereby learning to recall only relevant facts during training.
In Figure we see that RECKONING's performance is consistently more robust under distractors than the FT-ICR baseline. When we include all of the distractors in the context, RECKONING achieves significantly higher average label accuracy (82.5%) across hops than the baseline (70.9%), as computed by the average of the 3 considered hop depths. Additionally, compared to performance with no distractors, RECKONING's performance only drops 17.1% while the baseline performance drops 28.6% thereby exhibiting better ability to disentangle the correct knowledge from the distractors.
Finally, we also explore RECKONING's generalizability models with larger parameter size. We scale up the language model we used, GPT-2-small (124M), to GPT-2-XL. (1.5B) by adopting a parameter efficient finetuning method LoRA [34]. For simplicity, we only evaluate the models on the most difficult settings, i.e., Proo/Writer-5-hop with all the distractors. With GPT-2-XL-LORA in-context reasoning achieves 65% accuracy on the set, while our RECKONING model achieves 70.2% accuracy, a 5% performance gain. This result suggests that RECKONING's advantages in the presence of distractors hold even as models scale size.
4.3 Generalization to Weald knowledge
To investigate how generalizable our method is to real-world knowledge beyond the synthetic setting, we evaluate RECKONING on a more real-world multi-hop logical reasoning task, FOLIO [30], and report the result in Table 2 The dataset has rich vocabulary, diverse logic patterns, and abundant
language variations. It has been shown to challenge LLMs in both supervised fine-tuning and in-context learning settings. We fine-tune the GPT-2 model following the in-context reasoning setting as the base- line. As before, we train the GPT-2 model and RECK- ONING using the multi-task objective. We also com- pare to more advanced baselines, including GPT-3.5 (1ext-davinci-003 (56)) and ChatGPT(gpt-3.-uuubo P.. two popular large language models with around T75B parameters. For these two large models, we evaluate both in the zero-shot and few-shot settings. In the few-shot setting, we prompt the model with 8 single-task examples randomly sampled from the training set to perform in-context learning. We find that RECKONING's performance (which is initiated here from GPT-2) better than the GPT-2 in-context reasoning baseline. Compared to the two advanced large language models, RECKONING outperforms them by a significant margin (12% O-shot and 7% 8-shot). We conclude that RECKONING is effective and significantly benefits reasoning tasks using real-world knowledge.
Model
Acc.
Random
33.3
GPT-2
53.3
GPT-3.5 (text-davinci-003) O-shot
45.1
GPT-3.5 (text-davinci-003 8-shot
52.9
CharGPT (gpt-3.5-turbo) 0-shot
40.0
CharGPT (gpt-3.5-turbo) 8-shot
42.6
RECKONING
54.9
Table 2: Evaluation results on FOLIO. We compare RECKONING against the FT-ICR baseline with GPT-2 and two popular large language models.
4.4 Run-time Analysis
One of the advantages of RECKONING is the ability to memorize a large set of knowledge IC and answer multiple related questions about that knowledge at little extra cost per question. Specifically, in contrast to ICR, RECKONING can encode XC once and answer multiple questions without needing to reprocess it for each question asked. To test whether RECKONING could be a more efficient method for inference in this setting, we measure the wall-clock time (in seconds) of the complete inference pipeline of RECKONING vs. ICR. For this experiment, use a synthetic reasoning dataset in which K. is a sequence of random letters, and the question asks for the most frequent letter in the context. The total number of tokens in each exam- ple is 1024: for x. I for the answer, and the remaining 1016 for K. broken into 8 "facts".
huaps://opeeai.ccmmbooo/cgggggg
7

---

The FT-ICR baseline receives a sequence including all 8 facts and the question. In contrast, RECKONING receives the 8 facts as batch of eight segments of 127 tokens and encodes them in parallel in the inner loop. In the outer loop, the model only receives the question or batch of questions. We focus on two settings: (1) inference time for single question and (2) inference time when answering multiple questions. In the multiple-question setting, we set the number of questions 18 (the same in ProofWriter). For RECKONING, the inference process includes the inner- loop knowledge encoding and the final forward pass to encode the question. We set the number of inner loop gradient steps to 1 and 4. In Table 3 we see that when answering a single question, RECKONING does not per- form inference faster than in-context reasoning. However, RECKONING shows significant advantages under a multi- question setting. Both the 1-step inner loop and the 4-step inner loop are faster than the baseline. Since RECKONING encodes the knowledge in model parameters, it does not need to reprocess the knowledge for a related question and is more efficient. We rum this experiment on RTX 3090 GPU
Wall-clock
Model
Time (8)
Single question
FT-ICR
0.1887
RECKONING1step
0.2532
RECKONINGAster
0.9664
Multiple questions (18)
FT-ICR
2.0436
RECKONING1stn RECKONING
0,6228
1.4839
Table 3:
Wall clock run-time, in sec- onds, of the fine-tuned ICR baseline and RECKONING.
4.5 Memorizing Knowledge
In Table [ we saw that training RECKONING with a multi-task (MT) outer loop objective improved over training with the single-task (ST) objective, poten- tially because the MT objective improves the model's ability to memorize the knowledge in the inner loop. To validate our hypothesis, we analyze RECKON- ING's performance in reproducing memorized knowledge, First, we show in Table 4 the inner loop average loss (LCIM) and average change (ACCIM) (from first inner loop evaluation to last) on validation examples from the 5-hop ProofWriter data. We see that the average inner loop loss for RECKONINGșT is much higher than RECKONING and indeed starts out much higher as well. This shows that the ST outer loop objective, which optimizes the model only for question answering. does not learn to encode the knowledge in the inner loop by memorizing it. In contrast, the MT objective forces the model to learn to memorize the knowledge, too: we observe that RECKONING minimizes the inner loop loss as it processes the knowledge. This pattern is also shown in the average inner-loss difference (CLM) the inner loop loss decreases more after the gradient updates when trained with the MT objective. Next, we report in Table the model's ability to reproduce memorized facts correctly under a multi-task setting, measured by an exact match score between the reproduced facts and the We evaluate on the ProtfWriter dataset both with and without distractors in the context and compare the results to the FT-ICR baseline.
Method
LCLM
ACCLM
RECKONINGST RECKONINGMT
10.74
9.55
0.167
12.61
Table Average inner loop validation loss: final (LCLM) and difference from start to finish (ALCLM).
gold facts. [
The results show that RECKONING can successfully (average exact match score of 99.3%) recover the relevant facts from its model parameters when the context does not include any distrac- tors. Note that this is comparable to the FT-ICR baseline, for which the task is much easier as it can directly attend to and copy the facts from input, while RECKONINGMT no longer has direct access them. When the context includes distractors, both RECKONING and FT-ICR struggle to identify and reproduce only the relevant facts, However, the per-
ProofWriter
ProofWriter
Method
2-h
3-h
5-h
2-h
3-h
5-h
MT
99.8
99.0
98.7
42.3
50.3
55.6
RECKONINGMT
98.9
98.6
98.2
71.2
74.4
75.1
Table 5: Exact match score for reproducing memorized knowledge. In contrast to in-context reasoning, RECKON- ING does not have direct access to the knowledge.
*We perform this experiment in a limited setting and do not handle the case where hidden states could be cached for the forward pass of in-context reasoning, likely speeding up multi-question inference [2].
"This is done by prompting the model with the question and comparing its output (after its answer to the question) to the concatenation of all facts. The model is able to produce these facts in the expected order due to an implementation detail: they numbered and labeled when given to the inner loop.
8

---

formance for FT-ICR (average 49.4%) drops far below that of RECKONING (73.6%), demonstrating that RECKONING is much better at disentangling the relevant knowledge from the distractors.
Finally, we show that RECKONING with a multi-task ob- jective is also more robust to distractors it trains the model to reproduce only the facts that would be relevant to a particular question we ask in the outer loop. As in
single-task
multi-task
100
Label Accuracy (86)
$3.7
84.0
79.8
Section 4.2 we use the ProofWriter dataset and, for each question, add all the distractors to the context. We train
60
the model using the multi-task, objective and report the label accuracy. While in Table [ we originally saw a
2-hop
3-hop
5-hop
1% improvement from training with a multi-tast ob- jective on ProofWriter with no distractors, we see much more significant performance gap in Figure (~~ 18.2%) when distractors are available. We also note that the perfor- mance of the single-task model is essentially random (sce the Random-Facts baseline from Table D. By learning how to memorize knowledge in the inner loop to recall
Figure 6: RECKONING performance when trained with single-task and multi-task objective under distractors. When trained with the multi-task objec- tive, the model learns to memorize and reason over the relevant facts.
relevant facts in the outer loop, the model also learns how to encode facts more robustly over them.
5 Related Work
Logical Reasoning Datasets and Benchmarks As a central building block of human cognition and intelligence 228). logical reasoning has been a long-pursued topic in the field of AI 12 9 13 Logical reasoning, in general, can be categorized in a trichotomy of deductive, inductive, and abductive reasoning [26). Multiple datasets have been published that evaluate neural models' ability on these three types of logical reasoning 6 17 72). Initially, logical reasoning tasks focused on hypothesis classification, where, given a theory consisting of multiple facts and rules, a model would determine whether the hypothesis was correct. Recently, transformer- based language models have been directly used to solve this task in synthetic I7 66 real-world (30, and adversarial settings. However, simply predicting whether the hypothesis is valid does not elucidate whether the model correctly reasons over the provided knowledge. To better analyze and interpret the reasoning process of language models, new tasks focus on generating the valid proof that explains the model's decision 220 74). Our proposed method, RECKONING, is optimized for the hypothesis classification reasoning task and evaluates on many of these datasets (29 74 30
46 48 55
27
Logical Reasoning over Natural Language Historically, automatic logical reasoners used sym- bolic systems and formal languages as knowledge representation
52 55 81]. However, these systems were hard to scale up due to the knowledge-acquisition acquisition bottleneck and the brittleness of formal representation [35] 84). With recent advances in transformer-basel language modeling (76) and self-supervised pre-training [2 60 61]. novel paradigm for logical reasoning emerged, where pre-trained language models (PLMs) could be used as soft reasoners over knowledge expressed in natural language. Natural language as a knowledge representation allowed PLMs to handle raw input with diverse formats 15 32), resulting in PLMs being applied to various types of deductive
[17. abductive (6) and inductive 29 reasoning tasks. However, language models soft reasoners also showed structural weaknesses, as their performance dropped on complex logical operations 13 [0), and their reasoning process was not interpretable [55 65] Consequently, a new line of work
uses neuro- sycuooyymmolic methods to combine the best of both language models and symbolic reasoning
41 (4). Specifically, the interpretability gap motivated modular and step-wise reasoning systems that use PLMs as intermediate modules [33] 59 67] 69 75 83 to generate reasoning steps (c.g., proofs). In contrast to these works, our method RECKONING dynamically encodes natural language knowledge into the model parameters, thereby reasoning by mixing contextual knowledge with pre-encoded parametric knowledge and allowing the model to determine a conclusion based on its updated parametric knowledge.
74 36
Model Editing While our motivations are grounded in research on machine reasoning, our methods are more often used in the area of model editing. Model editing is a method to edit a model's parameters to correct its errors or update the model. Several works propose hypernetwork- methods to edit knowledge in a model by predicting updates conditioned on new factual statements 31 or transforming the gradicnts from new provided facts [54] to make local edits to model. Other

---

approaches focus on more direct edits of model behavior, such directly modifying neuron outputs [9 85], localizing distinct feed-forward layers that are responsible for factual recall, and modifying these weights [50]. and performing weight updates across multiple layers to perform simultaneous edits [5T]. Similarly, our method also rapidly edits the model parameters to add knowledge. However, our bi-level framework optimizes model edits for the reasoning task in the outer loop, allowing the model to learn to quickly memorize knowledge that can support the model's reasoning ability.
Language Models as Knowledge Bases Our work learns to reason by dynamically encoding contextual knowledge in the parameters of language models before answering questions about them. Previous studies have found that LLMs can store real-world facts learned during pre-training
10
[1 50 [4]. Learning these facts during pre-training allows language models to be prompted [88
71 86 or adapted 8 37 38 63] to produce these facts on-demand. However, LLM knowledge is latent and hard to identify control. The model generation is sensitive to specific words or phrases. L.MMs emit knowledge encoded the parameters only when prompted appropriately It is also difficult to inject update knowledge for LLMSs (50), and the memorization of knowledge in LLMs is not optimized toward their reasoning ability. our work, we seek to find way add knowledge LLMs in controllable and adaptive way that benefit downstream reasoning applications.
39
10
1g 23
57.
6 Conclusion
We present RECKONING, a bi-level learning framework for multi-hop reasoning that encodes knowl- edge verbalized using natural language into a model's parameters through gradient updates. During training, the inner loop encodes the contextual knowledge into the model parameters by backpropa- gating a language modeling loss. In the outer loop, given only the question as input, the model solves reasoning problems using the memorized knowledge. Through bi-level optimization, RECKONING finds set of meta- parameteres that allows Our experiments show that RECKONING learns to reason only by relying on its parametric knowledge after the external knowledge has been encoded. Using multi-task objective that jointly optimizes reasoning and knowledge memorization in the outer loop, RECKONING outperforms ICR baselines that are trained encode external knowledge as part of the context. Through our analysis, we show that RECKONING is more generalizable to problems with longer reasoning chains, less susceptible irrelevant distractor knowledge, and that RECKONING is more efficient than the baseline when answering multiple questions that require common knowledge.
perform quick knowledge-basee baaeed updates for reasoning.
Acknowledgements
We thank Shikhar Murty and Christopher Manning for helpful discussions in crafting ideas for this project. We also gratefully acknowledge the support of the Swiss National Science Foundation (No. 215390), Innosuisse (PFFS-21-29 the EPFL Science Seed Fund, the EPFL Center for Imaging, Sony Group Corporation, and the Allen Institute for AI.
References
[1] Lasha Abzianidze. A tableau prover for natural logic and language. Proceedings of the 2015 Conference Empirical Methods in Natural Language Processing, pages Portugal, September 2015. Association for Computational Linguistics.
Lisbon,
[2] Lasha Abzianidze. LangPro: Natural language theorem prover. In Proceedings of the 2017 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pages 115-120, Copenhagen, Denmark, September 2017. Association for Computational Linguistics.
[3] Antreas Antoniou, Harrison Edwards, and Amos Storkey. How to train your MAML. In International Conference on Learning Representations, 2019.
[4] Sungyong Baik, Myungsub Choi, Janghoo Choi, Heewon Kim, and Kyoung Mu Lee. Meta- learning with adaptive hyperparameters. ArXiv, abs/201 2020.
10

---

[5] Deniz Bayazit, Negar Foroutan, Zeming Chen, Gail Weiss, and Antoine Bosselut. Discovering knowledge-critical subnetworks pretrained language models. ArXiv, abs/2310.0308. 2023.
[6] Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Scott Wen tau Yih, and Yejin Choi. Abductive commonsense reasoning, 2020.
[7] Antoine Bosselut, Ronan Le Bras, and Yejin Choi. Dynamic neuro-symbolic knowledge graph construction for zero-shot commonsense question answering. In Proceedings of the 35th AAAI Conference Artificial Intelligence (AAAI), 2021.
[8] Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. Comet: Commonsense transformers for automatic knowledge graph construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics,
pages 4762-4779, 2019.
[9] Samuel R. Bowman, Christopher Potts, and Christopher D. Manning. Recursive neural networks can learn logical semantics. In Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality, pages 12-21, Beijing, China, July 2015, Association for Computational Linguistics.
[10] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations, 2023.
[11] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models, 2021.
[12] Carol Chen. Transformer inference arithmetic. http::/Aaipyyylloorrrmtmomeenmn infeeneee
arithmetic/, 2022.
[13] Zeming Chen and Qiyue Gao. Curriculum: A broad-coverage benchmark for linguistic phe- nomena natural language understanding. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 3204-3219, Seattle, United States, July 2022. Association for Computational Linguistics.
[14] Zeming Chen, Qiyue Gao, and Lawrence S. Moss. Neurall Natural language inference with joint neural and logical reasoning. In Proceedings of #SEM 2021: The Tenth Joint Conference on Lexical and Computational Semantics, pages 78-88, Online, August 2021. Association for Computational Linguistics.
[15] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope. James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontack Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slay Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022.
[16] Manuel Ciosici, Joe Cecil, Dong-Ho Lee, Alex Hedges, Marjorie Freedman, and Ralph Weischedel. Perhaps PTLMS should go to school task to assess open book and closed book QA. Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages (104-6111) Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[17] Peter Clark, Oyvind Tafjord, and Kyle Richardson. Transformers as soft reasoners over language. In Christian Bessiere, editor, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 3882-3890. International Joint Conferences on Artificial Intelligence Organization, 2020. Main track.

---

[18] Jeff Da, Ronan Le Bras, Ximing Lu, Yejin Choi, and Antoine Bosselut. Analyzing commonsense emergence in few-shot knowledge models. In Proceedings of the Conference on Automated Knowledge Base Construction (AKBC), 2021.
[19] Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8493-8502, Dublin, Ireland, May 2022. Association for Computational Linguistics.
[20] Bhavana Dalvi, Peter Jansen, Oyvind Tafjord, Zhengnan Xie, Hannah Smith, Leighannaa Pi- patanangkura, and Peter Clark. Explaining answers with entailment trees. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7358-7370, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[21] Jacob Devlin, Ming-Wos Wing-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Papers), pages 4171 Minneapolis. Minnesota, June 2019, Association for Computational Linguistics.
[22] Matthew Dunn, Levent Sagun, Mike Higgins, V. Ugur Guney, Volkan Cirik, and Kyunghyun Cho. Searchqa: new q&a dataset augmented with context from a search engine, 2017.
[23] Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schüizze, and Yoav Goldberg. Measuring and improving consistency in pretrained language models. Transactions of the Association for Computational Linguistics, 1:1012-103,, 2021.
[24] Christopher Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. Measuring and harnessing transference in multi-task learning, 2021.
[25] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. In Doina Precup and Yee Whye Tch, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1126-1135. PMLR, 06-11 Aug 2017.
[26] Peter A. Flach and Antonis C. Kakas. Abductive and inductive reasoning: Background and issues. In Applied Logic Series, pages 1-27. Springer Netherlands, 2000.
[27] Alexander Gaskell, Yishu Miao, Francesca Toni, and Lucia Specia. Logically consistent adver- sarial attacks for soft theorem provers. In Proceedings of the Thirty- International Joint Conference on Artificial Intelligence. International Joint Conferences on Artificial Intelligence Organization, July 2022.
[28] Vinod Goel, Gorka Navarrete, Ira A. Noveck, and Jérôme Prado. Editorial: The reasoning brain: The interplay between cognitive neuroscience and theories of reasoning. Frontiers in Human Neuroscience, 10, January 2017.
[29] Nicolas Gontier, Koustuv Sinha, Siva Reddy, and Christopher Joseph Pal. Measuring systematic generalization in neural proof generation with transformers. ArXiv, abs/2009. 144822 2020.
[30] Simeng Han, Hailey Schoelkopf. Yilun Zhao, Zhenting Qi. Martin Riddell, Luke Benson, Lucy Sun, Ekaterina Zubova, Yujie Qiao, Matthew Burtell, David Peng, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Shafiq Joty, Alexander R. Fabbri, Wojciech Kryscinski, Xi Victoria Lin, Caiming Xiong, and Dragomir Radev.
Folio: Natural language reasoning with first-order logic, 2022.
[31] Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin Stoyanov, Mohit Bansal, and Srinivasan Iyer. Do language models have beliefs? methods for detecting, updating, and visualizing model beliefs, 2021.
[32] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021.
[33] Ruixin Hong, Hongming Zhang, Xintong Yu, and Changshui Zhang. METGEN: A module-
based entailment tree generation framework for answer explanation. In Findings of the Associ- ation for Computational Linguistics: NAACI. 2022, pages 1887-1905, Seattle, United States, July 2022. Association for Computational Linguistics.
12

---

[34] Edward Hu. yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[35] Hai Hu, Qi Chen, Kyle Richardson, Atreyee Mukherjee, Lawrence S. Moss, and Sandra Kuebler. MonaLog: lightweight system for natural language inference based on monotonicity. In Proceedings of the Society for Computation in Linguistics 2020, pages 334-344, New York, New York, January 2020. Association for Computational I. Linguistics.
[36] Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard Hovy, and Eric Xing. Harnessing deep neural networks with logic rules. In Proceedings of the 54th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2410-2420, Berlin, Germany, August 2016. Association for Computational Linguistics.
[37] Jena D Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. (comet-) atomic 2020: On symbolic and neural commonsense knowledge graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 6392, 2021.
[38] Liwei Jiang, Antoine Bosselut, Chandra Bhagavatula, and Yejin Choi. "I'm m not mad": Common- sense implications of negation and contradiction. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4380-4397, Online, June 2021. Association for Computational Linguistics.
[39] Zhenghao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438, 2020.
[40] Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601-1611. 1, Vancouver, Canada, July 2017. Association for Computational Linguistics.
[41] Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi. Maieutic prompting: Logically consistent reasoning with recursive explana- tions. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1266-1279, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
[42] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466, 2019.
[43] Douglas B. Lenat, Mayank Prakash, and Mary Shepherd. Cye:
Using common sense knowledge to overcome brittleness and knowledge acquisition bottlenecks. Al Magazine, 6(4):65, Mar. 1985.
[44] Tao Li, Vivek Gupta, Maitrey Mehta, and Vivek Srikumar. A logic-driven framework for consistency of neural models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP--JJNIL) Association for Computational Linguistics.
pages 1924-3933 Hong Kong, China, November 2019.
[45] Zhengzhong Liang. Steven Bethard, and Mihai Surdeanu. Explainable multi-hop verbal reason- ing through internal monologue. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 1225-1250 Online, June 2021. Association for Computational Linguistics.
[46] Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. Evaluating the logical reasoning ability of chatgpt and gpt-4, 2023.
[47] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2017.
[48] Bill MacCartney and Christopher D. Manning. Natural logic for textual inference. In Proceed- ings of the ACI.-PASAAL Workshop on Textual Entailment and Paraphrasing, pages 193-200, Prague, June 2007. Association for Computational Linguistics.
13

---

[49] Pascual Martinez-Gómezz Koji Mineshima, Yusuke Miyao, and Daisuke Bekki. On-demand injection of lexical knowledge for recognising textual entailment. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 710-720, Valencia, Spain, April 2017. Association for Computational Linguistics.
[50] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[51] Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. Mass- editing memory in transformer. In The Eleventh International Conference Learning Representations, 2023.
[52] K. Metaxiotis, Dimitris Askounis, and John Psarras. Expert systems in production planning and scheduling: A statc-of-the-arr survey. Journal of Intelligent Manufacturing, 1344:2253-600 2002.
[53] Kostas S. Metaxiotis, Dimitris Askounis, and John E. Psarras. Expert systems in production planning and scheduling: A state-of-the-art survey, Journal of Intelligent Manufacturing, 2002.
[54] Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. Fast model editing at scale, 2021.
[55] Stephen Muggleton and Luc de Raedt. Inductive logic programming: Theory and methods. The Journal of Logic Programming, 19-20:629-679 679, May 1994.
[56] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.
[57] Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. How context affects language models' factual predictions, 2020.
[58] Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) pages 2463-2473, Hong Kong, China, November 2019. Association for Computational Linguistics.
[59] Hanhao Qu, Yu Cao, Jun Gao, Liang Ding, and Ruifeng Xu. Interpretable proof generation via iterative backward reasoning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1968829811 Seattle, United States, July 2022. Association for Computational Linguistics.
[60] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners. 2019.
[61] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2020.
[62] Kyle Richardson and Ashish Sabharwal. Pushing the limits of rule reasoning in transformers through natural language satisfiability. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10): 209-11229, June 2022.
[63] Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 14166442,, Online, November 2020. Association for Computational Linguistics.
[64] Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842-866, 2020.
[65] Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, ((5):06-215,) May 2019.
14

---

[66] Mohammed Saeed. Naser Ahmadi, Preslav Nakov, and Paolo Papotti. RuleBERT: Teaching soft rules to pre-trained language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1460-1476, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational I Linguisties.
[67] Swarnadeep Saha, Prateek Yadav, and Mohit Bansal. multiPRover: Generating multiple proofs for improved interpretability in rule reasoning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3662-3677, Online, June 2021. Association for Computational Linguisties.
[68] Soumya Sanyal, Zeyi Liao, and Xiang Ren. RobustLR: A diagnostic benchmark for evaluating logical robustness of deductive reasoners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1631-963,, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguisties.
[69] Soumya Sanyal, Harman Singh, and Xiang Ren. FaiRR: Faithful and robust deductive reasoning over natural language. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1075-1093, Dublin, Ireland, May 2022. Association for Computational Linguistics.
[70] Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. CoRR, abs/2302.00093 00099,,0000,.... 2023.
[71] Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. Auto- Prompt:
Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 4222-4235, Online, November 2020. Association for Computational Linguistics.
[72] Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William L. Hamilton. CLUTRR:
diagnostic benchmark for inductive reasoning from text. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing JEMNLP-JJCNLL pages 4515-4515, Hong Kong, China, November 2019. Association for Computational Linguistics.
[73] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexan- der W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, An- ders Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakas, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartoomiij Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekei, Bill Yuchen Lin, Blake Howald, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, César Ferri Ramírez, Chandan Singh, Charles Rathkopf Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burch, Chris Waites, Christian Voigt. Christo- pher D. Manning. Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hermandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dim- itri Coelho Mollo, Diyi Yang, Dong-Ho Ho Lee, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodola, Emma Lam, Eric Chu, Eric Tang. Erkut Erdem, Ernie Chang, Ethan A. Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martinez-Plumed, Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Wang, Gonzalo Jaimovitch-- López, Gregor Betz, Guy Gur-Ari, Hana Galijase- vic, Hannah Kim, Hannah Rashkin, Hannanch Hajishirzi, Harsh Mchta, Hayden Bogar, Henry
15

---

Shevlin, Hinrich Schütze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jachoon Lee, Jaime Fer- nández Fisac, James B. Simon, James Koppel, James Zheng, James Zou, Jan Kocoń, Jana Thompson, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang. Jason Wei, Jason Jekaterina Jekateria Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song. Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Berant, Jörg Frohberg. Jos Rozen, Jose Hernandez-Orallo, Joseph Boudeman, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gimpel, Kevin Omondi, Kory Mathewson, Kristen Chia- fullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang. Liam Dugan, Lianhui Qin, Lidia Contreras-Ochando Louis-Philippe Morency, Luca Moschella, Lucas Lam. Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros Colón, Luke Metz, Lütfi Kerem Şenel, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramírez Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L. Leavitt, Matthias Hagen, Mátýás Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael A. Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michat Swędrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang. Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr Milkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramón Risco Delgado, Raphaël Millière, Rhythm Garg. Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank. Rohan Sikand, Roman Novak, Roman Sitelew, Ronan LeBras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Ruslan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel S. Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay. Shyamolima, Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin. Stephen Prasad, Steven T. Piantadosi. Stuart M. Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra. Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsu Hashimoto, Te-Lin Wu, Théo Desbordes. Theodore Roth- schild, Thomas Phan, Tianle Wang. Tiberius Nkinyili, Timo Schick, Timofei Kornev, Timothy Telleen-Lawton, Titus Tunduny, Tobias Gerstenberg, Trenton Chang. Trishala Neeraj. Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Ramasesh, Vinay Uday Prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang. Yiding Hao. Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang. Zirui Wang, and Ziyi Wu. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, 2022.
Evans,
[74] Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. ProofWriter: Generating implications, proofs, and abductive statements over natural language. In Findings of the Association for Computa- tional Linguistics: ACL-IJCNLP 2021, pages 3621-3634, Online, August 2021. Association for Computational Linguistics.
[75] Oyvind Tafjord, Bhavana Dalvi Mishra, and Peter Clark. Entailer: Answering questions with faithful and truthful chains of reasoning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2078-2092 Abu Dhabi, United Arab Emirates,
16

---

December 2022. Association for Computational Linguistics.
[76] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[77] Xinyi Wang, Yulia Tsvetkov, and Graham Neubig. Balancing training for multilingual neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Com- putational Linguistics, pages 8526 8537, Online, July 2020. Association for Computational Linguistics.
[78] Zirui Wang, Zachary C. Lipton, and Yulia Tsvetkov. On negative interference in multilingual models: Findings and meta-learning treatment. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4438-4450, Online, November 2020. Association for Computational Linguistics.
[79] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's transformers: State-of-the-art natural language processing, 2020.
[80] Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, and Kentaro Inui. Do neural models learn systematicity of monotonicity inference natural language? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pages 6105 6017, Online, July 2020. Association for Computational Linguistics.
[81] Hitomi Yanaka, Koji Mineshima, Pascual Martinez-Gómer and Daisuke Bekki. Acquisition of phrase correspondences using natural deduction proofs. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), pages 756-766, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.
[82] Kaiyu Yang, Jia Deng, and Danqi Chen. Generating natural language proofs with verifier-guides search. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 89-105, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
[83] Kaiyu Yang. Jia Deng, and Danqi Chen. Generating natural language proofs with verifier-guided search, 2022.
[84] Zonglin Yang, Xinya Du, Rui Mao, Jinjie Ni, and Erik Cambria. Logical reasoning over natural language as knowledge representation: survey, 2023.
[85] Yunzhi Yao, Shaohan Huang. Li Dong, Furu Wei, Huajun Chen, and Ningyu Zhang. Kformer: Knowledge injection in transformer feed-forward layers, 2022.
[86] Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chen- guang Zhu Michael Zeng, and Meng Jiang. Generate rather than retrieve: Large language models are strong context generators. The Eleventh International Conference on Learning Representations, 2023.
17

---

A Dataset
The ProofWriter [74] dataset has 500k pairs of questions, answers, and proofs over natural-language rule bases. Each example in the dataset contains a set of facts, a set of rules, a hypothesis, and label indicating whether the hypothesis is true, false, unknown. The dataset comprise five datasets named DO, D1. D2, D3, D5, each with 100k examples. Each dataset's questions require reasoning up to depths D (D = 0, 1, 2, 3, 5) to determine their answers. In our experiments, we only focus on the datasets that require more reasoning depths (D2, D3, D5). We show an example from the dataset in Table 7 In these datasets, a of facts and rules are mapped to 18 questions, where the questions can be answered based on a subset of the facts and rules. Thus, some of the facts rules can be irrelevant to some questions, and we call them distractors in Section 4.22 In the experiment for knowledge encoding with distractors, we encode all the facts in the model parameters and evaluate its ability to reproduce and reason over the correct facts. We show an example of distractor and relevant knowledge of a question in Table 9 For detailed statistics on the two datasets, please see T
ProofWriter
CLUTRR-SG The CLUTRR-SG [29] is an evaluation dataset for inductive reasoning on family relations adapted from the [72] dataset for measuring systematic generalization. Each example in the dataset contains (i) set of facts representing a family graph G = (V, E¹) where nodes (V) are entities and edges (E) are the relationships. (ii) a question asking the relationship between two entities (v1, Un V). and (iii) target relationship e* E E as the answer for the question. The facts are expressed as a list of (U,. Cy, vx) tuples. The two entities in the question are separated by more than one hop in the graph. There are 272 unique entities, 20 relationship types, and nearly 1.5M possible facts in the dataset. Following the authors, we define the difficulty of examples based on the number of family graph edges (i.e., the number of reasoning hops required to determine relation), in which k edges (k-hop) correspond to & facts. We show an example from the dataset in Table X
B In-context Reasoning with Distractors
To motivate the advantage of RECKONING on mitigating interference from distractors, we an- alyze the performance change of fine-tuned in- context reasoning with and without distractors present in the context of the questions. We define distractors as additional facts or rules present in a question's ' context that are not di- rectly relevant to the questions. A model should not be able to use only these distractors to an- swer a question correctly. For an example of dis- tractors in question's 's context, please see Table 9 We evaluate the baseline on the ProofWriter dataset since it naturally contains contexts in- cluding distractors (Table 9). Recall that we have two training objectives. The single-task objective only trains the model to predict an an- swer for each question given their contexts. The multi-task objective (MT) trains the model not only to predict an answer but also to reproduce the correct facts and rules (in contrast to distractors) based on the contexts. We evaluate the baseline on 3, and 5-hop datasets with both training objectives, and we report the average label accuracy across hops in Figure 7 Compared to the baseline's performance without distractors in the context, the performance with distractors decreases significantly. For single-task, the performance drops 23.2% when adding distractors to the contexts, and the performance with the multi-task objective drops 28, 6%. The results highlight in-context reasoning's ' high sensitivity to the interference of irrelevant information in the contexts.
FT-ICR With Without Distractors
3 Without 3 With
100
98.3
99.4
Labell Accuracy
75.1
70.8
FT-ICR
ST-CCRMIT
Figure 7: Label accuracy of fine-tuned in-context reasoning on questions with and without distractors in the context. With the same questions, adding dis- tractors to contexts significantly lowers the perfor- mance of in-context reasoning, both in the single- task and multi-task settings.
C Implementation Details
We select GPT-2-base (00 as the model for our method and all the baselines. We use the version implemented by the Huggingface Transformers library [79]. All the experiments for RECKONING
18

---

Dataset
#Train
#Validation
#Test
CLUTRR-SG (2-hop)
96,012
10,972
3,102
CLUTRR-SO (4-hop)
89,972
10,086
9,946
CLUTRR-SO SG (6-hop)
90,922
10,290
8,788
riter (2-hop)
6,996
1,098
2,013
ProofWriter (3-hop)
10,854
1,641
3,057
ProofWriter (5-hop)
18,525
2,553
5,175
Table 6: Dataset splits and statistics for our experiments
Identifier
Content
fact
Harry is nice.
fact
Fiona quite Nice.
fact
Fiona is round.
fact
Fiona is white.
fact
Dave is furry.
fact
Charlie is white.
rule
Furry people are green. Round, green people are red. All red people are white. Nice, round people are furry. If someone is nice, then they are round.
rule
rule 3
rule
rule 5
nule
Charlie round and Charlie nice, then Charlie is white.
question-answer
Harry is red? True Harry is not red? False
question-answer 2
question-answer 3
Dave is not white? Unknown
Table An example from the dataset ProofWriter. There are 6 facts and rules mapped three crestion-answer pairs. Each question can be answered based on the given facts and rules.
are conducted on a cluster with NVIDIA A100 (40GB) GPUs. All the baseline experiments are conducted on local machine with NVIDIA RTX 3090 GPU (24GB).
Fine-tuned In-context Reasoning We set the train batch size to 16 and train the model for 6 epochs with early stopping based on validation label accuracy. We set the learning rate to 3e-5 and use the AdamW W optimizer with set to 1e-8. We validate the model on the development set for every epoch and select the best checkpoint using the validation accuracy as the metric.
RECKONING In the inner loop, we generally perform gradient steps for lower-hop questions (2, 3, 4-hop) and 5 gradient steps for higher-hop questions (5 and 6-hop). We select the Adam W 47 as the optimizer for the inner loop since the main task is language modeling. The inner-loop learning rate is set to 3e-5 before training. and the algorithm dynamically learns set of optimal lcarning rates when converged. In our experiments and analysis, we only report the results from RECKONING with multi-task objective since its performance is better than the single-task objective. In the outer loop, we also use the Adam with a learning rate 3e-5. For both optimizers, we set c to 1e-8. We set the train batch size 2 due memory limitations. We apply the technique of gradient accumulation and set the accumulation step to We train the model for 6 epochs with early stopping. For each epoch, validate the model twice: in the middle and once at the end. We select the best model checkpoint based on the validation label accuracy.
D Adaptive Learning Rate
Prior works 3 show that fixed learning rate shared across steps and parameters does not benefit the generalization performance of the system. Instead, [3] recommends learning a learning rate for
19

---

Identifier
Content
fact
C is H's father.
fact
is J's aunt.
fact
J S's daughter.
fact
D is C's father
fact
is B's father.
fact
H is Z's son.
question-answer
How and B related each other? Grandfather
Table 8: An example of 6-hop reasoning from the CLUTRR-SG dataset.
Identifier
Content
fact
Harry is nice.
fact 2
Fiona quite Nice.
fact 3
Fiona round.
fact 4
Fiona is white.
fact 5
Dave is furry.
fact 6
Charlie is white.
rule
Furry people are green.
rule
Round, green people are red.
rule
All red people white. Nice, round people are furry. rule If someone nice, then they are round.
rule
rule
Charlie round and Charlie nice, then Charlie is white.
question-answer
Harry red? True
Table Example of distractors (black) and relevant knowledge (red) in the ProofWriter dataset.
each network layer and each adaptation step in the inner loop. The layer parameters can learn to adjust the learning rates dynamically at each step. To control the learning rate in the inner loop adaptively, we define a as set adjustable variable: a = {(xo, a1. J. where L is the number of layers and for every 0, ..., L. a is vector with N elements given pre-defined inner loop step number N. The inner loop update equation then becomes
KC 1+1
a VCLLM((Os
(6)
where @ is an element-wise product and 0 is the parameters for layer at the inner step We learn the set of optimal inner loop learning rates ax by optimizing the parameters the outer loop:
a V
Σ (K.
Lrocal (Soxee
where 17 is the outer loop learning rate and 0 is the updated parameters from inner loop. Below, we show the final algorithm of RECKONING in Algorithm 2
20

---

Algorithm 2
Dynamic Knowledge Encoding for Reasoning
Require: An example distribution p(D), transformer language model f, initial meta-parameters 0, outer step size initial inner step size a, inner loop length N.
1: while converged do Sample D' (())
outer loop
Li <-
4: for each (K., v. y) E DD' do
5: Initialize 0%
6: for n. = 0 to N do
> inner loop
7: 0 + th a 2 VLCuM(fOR, K.)
end for
9:
LD' + Lov + Lioml (fag,
10:
end for a - a VD
11:
Update inner step size
12:
O 4 0 V
13:
end while
Static
Are dynamic learning rates necessary for RECK- ONING'S performance? Following prior works on meta-learning 3 Al. we dynamically learn set of learning rates for RECKONING. In this ablation study, we analyze whether dynamic learning rates for the inner loop effectively improve the outer loop reasoning performance. Similarly, we fix other experimental settings and set the number of inner loop steps to 4. As Figure shows, when using static learning rate (i.e., all layers and inner loop steps share constant learning rate), the performance drops by a large margin (average drop of 34.2%). The performance drop becomes more significant on ques- tions requiring more reasoning hops (45.5% drop for 4-hop and 39.5% drop for 6-hop), demonstrating the importance using a dynamic learning rate in the inner loop of our framework.
LR-ddyna
26
84.7
per-step-per-
Label Accurscy
60
50.5
2-hop
4-hop
G-hop
Figure We study how much the dynamic learning rate in the inner loop contributes to the outer loop performance. We fix all the hy- perparameters except the option of using the dynamic or fixed learning rate. We conduct the analysis using the CLUTRR-SC dataset since it is more complex and difficult (lower random performance).
E Experiments with Large Language Models
ProofWriter
ProofWriter distractor
ProofWriter.astraccorr
CLUTRR-SG
Method
2-h
3-h
5-h
2-h
3-h
5-h
2-h
4-h
6-h
GPT-3.5 0-ahot
58.4
56.4
53.7
49.1
47.1
45.3
35.6
16.0
18.5
GPT-3.5 8-shot
78.0
82.4
80.1
58.7
57.2
54.5
39.0
18.5
20.8
RECKONINGRIT
99.5
99.7
99.8
79.8
83.7
84.0
98.3
97.6
94.8
Table 10: Label accuracy of RECKONING on ProofWriter and CLUTRR-SG compared against a popular Large Language Model (LLM), GPT-3.5. We prompt GPT--355 in the zero-shot setting and also the 8-shot in-context learning setting. Models with MT are trained with the multi-task objective in the outer loop.
Recently, Large Language Models (LLMs) with large parameter sizes learned from human preferences have shown remarkable performance in language understanding and generation. These LLMs are powerful zero-shot and few-shot reasoners. Recent works find that LLMs learn perform multi-step reasoning by first generating new reasoning chains and then predicting the answers. In this experiment, we benchmark the performance of a popular new LLM, GPT-3.5, on the two multi-hop reasoning datasets we used in our paper. We first evaluate GPT-3.5's zero-shot reasoning performance in predicting the correct answers. As Table [0] shows, zero-shot prompting GPT-3.5 significantly under- performs RECKONINGE performance. GPT-3.5's performance improves on FroofWrites without distractors but still behind the performance of RECKONING. When distractors are present in the context, RECKONING performs much better than zero-shot and few-shot GPT-3.5 prompting, This
21

---

highlights RECKONING'S 's strength in disentangling irrelevant information from useful knowledge, an ability that even powerful LLMs like GPT-3.5 lack.
22


Reviews:
[{"review_id": "2ctkhBLWje", "reviewer": "Reviewer_ymTg", "summary": "The paper presents a two-phase learning algorithm that 1) encodes background knowledge in the parameters of an LM through fine-tuning (instead of providing it in context), and 2) learns how to use and reason the encoded knowledge for a given question (?).\n\nThe paper shows better performance than in-context learning, and better generalization to longer reasoning chains. ", "questions": "- Does the order in which the questions are seen effect the results? \n- Which task from the Proofwriter dataset was actually used in this paper?\n- Is the outer-loop of updates actually needed? Finetuning on knowledge and then trying to answer the question directly (without any more parameter updates) is a natural baseline to compare against. I believe this is slightly different from the FT-ICR setup. \n- In any case, the performance difference between FT-ICR and Reckoning is only somewhat apparent on the CLUTTRR dataset. Can you conjecture why that might be the case?\n- I think the bigger performance gaps are noticeable in the OOD generalization setup. Can you provide an intuition / analysis of why generalization gets better? \n- At multiple places, the paper says phase 1 is to memorize the knowledge and phase 2 is to quickly memorize the given knowledge and perform reasoning. e.g. lines 108 to 111. Can you clarify why one needs 2 steps to memorize the said knowledge. Is the knowledge being memorize different in these two phases? Moreover, it seems memorize and \"learn\" are being used interchangeably, but I believe they are different. Overall, phase 2 application and its impact are confusing. \n- If the knowledge from a question is being encoded in the model, how does the model handle contradictory information? For example: it is possible \"A is B's son\" in instance 1, but \"B is A's son\" in another instance. Is there an assumption that this is not possible?", "limitations": "-", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is easy to read for the most part.\n- The generalization experiments results support Reckoning as a useful approach.", "weaknesses": "- Most of my concerns / questions have to do with Phase 2 of the model updates. More details given below in questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Responses to Reviewer ymTg (R1)\n\nWe thank the reviewer for viewing our proposed bi-level learning algorithm as interesting and novel and for recognizing its benefits and contribution. Below, we address the questions:\n\n**Q1: The reviewer asks if the order of the questions affect the results**\n\nNo, the order of the questions in the dataset should not affect the results. We randomly shuffle the train and test data for each run. For each experiment, we conduct three runs using three different random seeds and report the average.\n\n**Q2: The reviewer wonders which task from ProofWriter was used in our paper**\n\nWe follow the deductive reasoning task originally proposed in RuleTaker, a work that ProofWriter is built on. The task is most similar to Task 1, proof generation, but we omit the proof part and only generate the answer.\n\n**Q3: The reviewer asks if the outer-loop of updates is needed**\n\nWe evaluated four additional baselines related to the outer-loop updates to confirm this:\n\n-   **FT-KG**: Here we check, without outer loop optimization, can finetuning on all facts allow the model to memorize them and perform well on the questions when no additional information is given.\n    \n-   **FT-KG-ICR**: Here we check, without outer loop optimization, can finetuning on all facts allow the model to memorize them and perform well on the questions when the background knowledge is given at test time too.\n    \n-   **RECKONING-no-outer**: We check if RECKONING with inner-loop only (i.e., a single-level optimization) can perform well when no facts are provided in the question. Note that for each question, we start from the trained model that is not yet finetuned on that question’s facts.\n    \n-   **RECKONING-no-outer (zero-shot)**: We do not train the model but directly evaluate it on the questions by dynamically doing a few gradient steps to encode the facts. We check if the model already can do inference-time knowledge encoding through gradient descent dynamically for each question, without bi-level optimization.Note that for each question, we start from the initial model that is not yet finetuned on that question’s facts.\n  \nWe report their performances below (also see Table 1, rebuttal pdf):\n\n|                                | Clutrr-2-hop | Clutrr-4-hop | Clutrr-6-hop |\n|--------------------------------|--------------|--------------|--------------|\n| FT-KG                          | 5.1          | 4.6          | 5.8          |\n| FT-KG-ICR                      | 5.1          | 4.6          | 5.8          |\n| RECKONING-no-outer (zero-shot) | 7.9          | 8.1          | 9.9          |\n| RECKONING-no-outer             | 20.7         | 12.9         | 10.3         |\n| RECKONING                      | **98.3**         | **97.6**         | **94.8**         |\n|\n\n|                                | Proof-2-hop | Proof-3-hop | Proof-5-hop |\n|--------------------------------|-------------|-------------|-------------|\n| FT-KG                          | 31.2        | 33.8        | 33.3        |\n| FT-KG-ICR                      | 32.5        | 32.4        | 33.6        |\n| RECKONING-no-outer (zero-shot) | 33.3        | 29.7        | 33.0        |\n| RECKONING-no-outer             | 17.6        | 14.2        | 6.8         |\n| RECKONING                      | **99.5**        | **99.7**        | **99.8**        |\n|\n\nAs our evaluation results show, the baselines FT-KG and FT-KG-ICR perform close or under random (33.3% for ProofWriter and 5% for CLUTRR). The baselines that remove the outer loop also perform poorly, far below RECKONING’s performance. We highlight the importance of outer-loop optimization, indicating that it's necessary for the model to learn to dynamically do few-step knowledge encoding that supports the reasoning performance.\n\n**Q4/Q5: The reviewer asks why the performance difference between FT-ICR and RECKONING is only apparent on CLUTRR, and why bigger performance gaps are noticeable in the OOD generalization setup**\n\nOur results in Table 1 show small improvement for RECKONING as the baseline (FT-ICR) is also quite strong in this idealized setting. Here, our results are partly a sanity check that RECKONING works as well under ideal conditions (even slightly better!). However, RECKONING exhibits stronger improvements when idealized conditions are removed and the model has to generalize out of distribution and handle noisy inputs.\n\n**Q6: The reviewer asks us to clarify knowledge memorization and learning and motivate the importance of outer loop optimization**\n\nIn RECKONING, we do not first fine-tune the model on all facts in the dataset and then learn to “quickly memorize the given knowledge and perform reasoning,” as the reviewer suggested. Instead, RECKONING does inference-time training on the fly. We define knowledge memorization as doing a few gradient updates on the facts. In the inner-loop, the model encodes the knowledge through these gradient updates. In the outer-loop, the model uses the encoded knowledge to perform reasoning. Please see more context on the inference process in our general response (point 1). To teach models the ability to do this kind of inference-time learning, i.e., gradient-based knowledge encoding that supports reasoning, bi-level optimization is important. As we have shown in our response to Q3, without the outer loop, models perform poorly.\n\n**Q7: The reviewer asks how does the model handle contradictory facts across examples**\n\nRECKONING would be able to handle this case. As mentioned in the general response, at inference time, RECKONING falls back to learned “meta-parameters” after every processed example, wiping the slate clean for the next example. As a result, contradictory knowledge between examples do not contaminate each other. RECKONING would likely not work if there were contradictory facts for the **same** example, but to the best of our knowledge, even the best performing models cannot reliably handle contradictory information in the context well."}, {"review_id": "6kndDTukGi", "reviewer": "Reviewer_UhBM", "summary": "Summary:\nThe paper introduces RECKONING, a bi-level learning algorithm designed to improve reasoning in transformer-based language models. RECKONING encodes contextual knowledge into the model's parameters using gradient updates, allowing the model to answer questions based on its updated parameters. The authors demonstrate, through experiments on two multi-hop reasoning datasets, that RECKONING outperforms an in-context reasoning baseline and is more robust to distractions, generalizes better to longer reasoning chains, and is more computationally efficient under certain conditions.\n\nContributions:\n  1. Propose a bi-level learning algorithm, RECKONING, to teach language models to reason by updating their parametric knowledge through back-propagation.\n  2. Conduct experiments on two multi-hop reasoning datasets, ProofWriter, and CLUTRR-SG, showing that RECKONING outperforms the in-context reasoning baseline and provides several other benefits.\n  3. Provide analyses on the ability of RECKONING to memorize knowledge, measure its performance under distractor conditions, and analyze its run-time efficiency, etc.", "questions": "1. How would RECKONING perform with more complex and diverse real-world reasoning tasks beyond the synthetic multi-hop reasoning datasets used in the experiments (ProofWriter and CLUTTR-SG)?  E.g., HotpotQA?\n2. Is this improvement agnostic to model architectures?  How does RECKONING do on seq2seq models?\n3. In the multi-task training setting, does the choice of weighting different terms affect the performance of RECKONING? (from the paper is 1:1?)", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "  1. the idea of encoding knowledge into the model's parameters through gradient updates is an interesting and novel idea in the field of natural language reasoning\n  2. RECKONING demonstrates better performance than the baseline model in several aspects, including better generalization to longer reasoning chains and its robustness to distractions\n  3. The paper is well-presented", "weaknesses": "The main weakness I see for this paper is the scope of its conducted experiments:\n  1. The experiments are conducted on two synthetic multi-hop reasoning datasets, ProofWriter and CLUTRR-SG. While these analyses provide valuable insights, further evaluation on a broader range of real-world datasets would strengthen the generalizability of RECKONING's performance.\n  2. The experiments are conducted only using the base GPT-2 model, which is really behind the state of the art.  It's hard to tell if on the best performing models we can still see this impovement.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Responses to Reviewer UhBM (R2)\n\nWe thank the reviewer for their helpful comments and are encouraged that the reviewer agrees that our algorithm is “an interesting and novel idea in the field of natural language reasoning” and recognizes its benefits. We also thank the reviewer for complimenting our paper as “well-presented.” We address the reviewer’s concerns and questions below:\n\n**W1/Q1: The reviewer asks how RECKONING would perform with more complex and diverse real-world reasoning tasks outside of the scope of synthetic data**\n\nTaking the reviewer’s suggestion, we evaluated RECKONING on FOLIO, a reasoning benchmark with first-order logical reasoning problems written by expert annotators based on real-world knowledge. We use the validation set as an in-house test set since the true test set has not been publicly released. In the training process, we randomly split the training data into train and validation sets. We show the evaluation results below:\n\n| Model                             | Acc  |\n|-----------------------------------|------|\n| FT-ICR                            | 53.3 |\n| GPT-3.5 (text-davinci-003) 0-shot | 45.1 |\n| GPT-3.5 (text-davinci-003) 8-shot | 52.9 |\n| chatGPT (gpt-3.5-turbo) 0-shot    | 40.0 |\n| chatGPT (gpt-3.5-turbo) 8-shot    | 42.6 |\n| RECKONING                         | **54.9** |\n|\n\nOur experiments show that RECKONING still outperforms the FT-ICR baseline and improves the performance compared to best-performing large language models like GPT-3.5 and ChatGPT, showing that even in less synthetic settings, RECKONING can still outperform other approaches.\n\n**Q2: The reviewer asks if the improvements are agnostic to model architectures and wonders how RECKONING would perform with seq2seq models**\n\nThis is a great suggestion! We extended RECKONING on a T5-small model and evaluated it on ProofWriter-5-hop with all the distractors. The FT-ICR baseline with T5-small achieves 69.2, and RECKONING achieves 70.8. We show that RECKONING still outperforms the baseline. We will conduct more extensive experiments on seq2seq models in our future revision.\n\n**Q3: The reviewer wonders if the choice of weighting on the two terms of the multi-task objective affect the performance of RECKONING**\n\nThis is an interesting point! In principle, the weighting would have some impact on the performance. However, we find that a 1:1 ratio works well in our study, so we stick to this setting due to the limited computation budget."}, {"review_id": "qmvJmm3tzC", "reviewer": "Reviewer_ZqBF", "summary": "This paper introduces a novel method for addressing logical reasoning in natural language question answering, specifically when the required knowledge is part of the context. One major challenge highlighted in this paper is that the included knowledge often contains irrelevant information, which can mislead the reasoning process and negatively impact question-answering performance.\nThe proposed solution involves a bi-level optimization technique during inference. The technical key is utilizing the knowledge to quickly fine-tune the model at inference time, using a causal language modeling objective. This eliminates the need for the knowledge to be explicitly included in the context, as it is encoded within the model parameters. It also helps the model to focus on the relevant parts of the knowledge. \nTo evaluate the effectiveness of the approach, multiple baselines incorporating variations of in-context learning are compared. The experimental results demonstrate improved performance using the proposed method, particularly in terms of generalization over longer reasoning chains.\n", "questions": "—While it is expected that the model learns to focus on the relevant parts of the knowledge, it is not very clear to me why the approach should make the model more generalizable to longer chains of reasoning? Any intuition? \n\n—Is there any SOTA results  better than what you report in this paper in your baseline variations?\n\n-- As it is explained in the paper, for the multiple question setting some parts of the knowledge are relevant for one question while irrelevant for others. Since the model is tuned once and the same model is used for answering all questions then the performance should be lower in this case, right?  ", "limitations": "The limitation section is not included in the paper. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Pros: \nThe proposed approach is sound and interesting.  The paper is well-written. \nThey show that their inference-time optimization makes the model more robust to the redundant information. \nIt becomes more generalizable to longer chains of reasoning. \nIt is more efficient when multiple questions are asked based on the same context. \n", "weaknesses": "The run-time analysis is to some extent misleading. For the mentioned problem setting, in general, we assume each example comes with its own context that is the context is not often shared. Therefore, I guess we expect this model to be less efficient in general for reasoning as it needs the bi-level optimization at inference time. I think highlighting the efficiency in the multi-question setting looks a bit far fetch. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Responses to Reviewer ZqBF (R3)\n\nDear reviewer, thank you for the constructive comments and we appreciate your time and effort. We thank the reviewer for recognizing our approach as interesting and sound, and our paper as well-written. We are encouraged to see the reviewer acknowledging our contributions. We address the reviewer’s concerns and questions below:\n\n**W1: The reviewer argues that the runtime analysis in the multi-question setting does not demonstrate RECKONING’s efficiency because the context is not often shared in general**\n\nWe argue that the setting where multiple questions share the same context is a common problem setting. For example, reading comprehension (e.g., SQuAD) typically involves reading a passage and answering multiple questions about it. In our own experiments, the ProofWriter typically contains fact sets about which multiple questions can be asked. Certain facts are distractors for particular questions and relevant for others. One of the benefits of RECKONING is that it can answer multiple questions based on the same set of facts, and our evaluation uses this property to improve the evaluation speed in multi-question settings.\n\n  \n\n**Q1: The reviewer asks about the intuition that our method makes the model more generalizable to longer chains of reasoning**\n\nThis is a good question. Our hypothesis is that RECKONING is more generalizable to longer reasoning chains because it encodes the multiple pieces of knowledge as separate sequences in a batch, which may lead to less of a length distribution shift, since the maximum sequence length is roughly equivalent for most facts. For in-context reasoning, reasoning is performed through the forward pass using the attention mechanism, which may be more likely to overfit to training length as the facts are concatenated into a single sequence.\n\n**Q2: The reviewer wonders if there are SOTA results on the baseline variations**\n\nWe also included an advanced large language model, GPT-3.5 (text-davinci-003) as one of the baselines. We show that RECKONING outperforms GPT-3.5 both on ProofWriter and CLUTRR:\n\n|                 |      | ProofWriter |     |   |ProofWriter (distractor)| |  |CLUTRR |      |\n|-----------------|------|:-----------:|------|------|:-----------:|------|------|:-------:|------|\n| Method          | 2-h  | 3-h  | 5-h             | 2-h  | 3-h         | 5-h   | 2-h  |   4-h  | 6-h  |\n| GPT-3.5 (0−shot)| 58.4 | 56.4  | 53.7    | 49.1 | 47.1        | 45.3  | 35.6 |   16.0 | 18.5 |\n| GPT-3.5 (8−shot)| 78.0 | 82.4  | 80.1    | 58.7 | 57.2        | 54.5  | 39.0 |   18.5 | 20.8 |\n| RECKONING       | **99.5** | **99.7** | **99.8**       | **79.8** | **83.7**        | **84.0**  | **98.3** |   **97.6** | **94.8** |\n|\n\nWe highlight the significant performance gain when there are distractors in the context of a question. While performance drops are observed for both approaches, the drop for GPT-3.5 is greater than for RECKONING.\n\n**Q3: Should the performance be lower when multiple questions share the same context and some parts of the knowledge are relevant for one question while irrelevant for others**\n\nCompared to the setting where each question is associated with a unique context with no irrelevant information, yes, the performance is lower when the setting changes to multiple questions sharing the same context. However, we show in our experiment (Figure 5, paper) that models trained with our algorithm are much more robust when there are distractors (irrelevant information) in the context, compared to the in-context reasoning baseline."}, {"review_id": "IUDAiKmHIL", "reviewer": "Reviewer_W7M8", "summary": "This work aims to solve reasoning tasks where models need to rely on knowledge provided as part of the task input. Motivated by the fact that pre-trained models encode a lot of irrelevant facts and they are good at retrieving the relevant ones for solving a downstream task, this work proposes an algorithm, RECKONING, that updates a model’s parameters to memorize the facts on the fly and then answer the question without having those facts as input explicitly. Experiments show that the proposed method can improve the model performance compared with the in-context reasoning baseline. The resulting model also generalizes better to examples requiring longer reasoning chains, is more robust to distractors and even more efficient in a certain setting.", "questions": "1.\tFor the Random-Facts baseline, do you mean the random facts are provided during inference or also during training? Do you train it with the multi-task objective?", "limitations": "No limitation is discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "-\tOriginality: Unlike prior works that propose different ways to filter irrelevant facts explicitly, this work makes use of the pre-trained LM to subsume all the facts and filter the irrelevant facts on its own. The idea is interesting and makes sense.\n-\tQuality: To validate the effectiveness, the work conducts sufficient experiments. The results demonstrate the superiority of the method in scenarios where a longer reasoning chain is needed or there exist a lot of distractors, which well motivate this study. An analysis is also provided to address the concern about the computation efficiency of the method.\n-\tClarity: The algorithm is well explained despite requiring some effort to understand it.\n", "weaknesses": "-\tIn the standard setting where there are no added distractors, the improvement brought by the method is actually minimal (Table 1). \n-\tAlso, the need to update the parameters on the fly makes the algorithm hard to be applied on large language models, which might have a greater capability in filtering irrelevant facts in-context already. I would suggest extending the idea to these large LMs by updating a subset of parameters (like adapters do).\n-\tOne potential disadvantage of the method could be that the knowledge facts contain a lot of noise or toxic content that may undermine the model’s basic ability for reasoning. In this regard, methods that first filter irrelevant facts and then provide the remaining facts in-context do not have such an issue.   \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Responses to Reviewer W7M8 (R4)\n\nWe thank the reviewer for their constructive comments and viewing our idea as interesting and sound, our experiments as sufficient and well-motivated toward the study, and our algorithm as well explained. We address the reviewer’s comments and questions below:\n\n\n**W1: The reviewer argues that in the standard setting, where there are no added distractors, the improvement brought by the method is minimal (Table 1 of our paper)**\n\nIn our paper, we mainly focus on the problem of generalization in more complex settings with distractors and longer reasoning chains. While our results in Table 1 show a small improvement for RECKONING, the baseline (FT-ICR) is also quite strong in this idealized setting, and our results are partly a sanity check that RECKONING works as well (even slightly better!). However, RECKONING exhibits even stronger improvements when idealized conditions are removed and the model has to generalize out of distribution and handle noisy inputs.\n\n  \n\n**W2: The reviewer wonders if the performance gain of RECKONING will generalize to larger language models**\n\n  \n\nThis is an interesting question. First, we want to show that more advanced large language models like GPT-3.5 (text-davinci-003) still fail to generalize when there are distractors, i.e., irrelevant information present in the context. Below our evaluation results on ProofWriter using GPT-3.5:\n\n| | | ProofWriter |      |      |ProofWriter (distractor)|  | |CLUTRR |      |\n|-----------------|------|:-----------:|------|------|:-----------:|------|------|:-------:|------|\n| Method          | 2-h  | 3-h         | 5-h  | 2-h  | 3-h         | 5-h   | 2-h  |   4-h  | 6-h  |\n| GPT-3.5 (0−shot)| 58.4 | 56.4        | 53.7 | 49.1 | 47.1        | 45.3  | 35.6 |   16.0 | 18.5 |\n| GPT-3.5 (8−shot)| 78.0 | 82.4        | 80.1 | 58.7 | 57.2        | 54.5  | 39.0 |   18.5 | 20.8 |\n| RECKONING       | **99.5** | **99.7**        | **99.8** | **79.8** | **83.7**        | **84.0**  | **98.3** |   **97.6** | **94.8** |\n|\n\nWhile the comparison is not perfect since we cannot tune GPT3.5, we note that the performance drop in the distractor setting on ProofWriter is much greater than for RECKONING.\n\n  \n\nAs a closer comparison, when we apply RECKONING to larger language models, we see similar improvements. We applied RECKONING to GPT2-XL (1.5B) with LoRA and evaluated on ProofWriter 5-hop with all distractors. Compared to FT-ICR’s performance (**65.0**), RECKONING’s performance (**70.2**) is **5.2** percentage points higher, demonstrating that RECKONING’s benefits still appear with larger language models.\n\n  \n\n**W3: The reviewer argues that noise or toxic content in knowledge may undermine the model’s basic ability for reasoning. They propose that methods that filter irrelevant facts ahead of time could be more effective than RECKONING**\n\nThis is an interesting point. However, we note that RECKONING is not in contradiction with a method that first filters irrelevant facts before providing them to the model for reasoning. Facts could be filtered before being provided to RECKONING too. We believe that combining these two approaches might work complementarily since existing approaches to filter irrelevant facts are not perfect.\n\n  \n\n**Q1: The reviewer asks if we provide random facts during inference or during training, and do we train the random facts baseline with the multi-task objective**\n\nWe provide random facts both during training and during inference. We train the random facts baseline without the multi-task objective.\n"}, {"review_id": "QgpblQcZGk", "reviewer": "Reviewer_5k3Z", "summary": "The paper introduces a two-level learning algorithm called RECKONING, which enhances the in-context reasoning performance by addressing the issue of distractor facts. The algorithm consists of two learning steps, an inner and an outer loop, where the inner loop trains the model to encode contextual knowledge into its parameters through BP, and the outer loop teaches the model to use the updated parameters to answer questions. The experimental results on two multi-hop reasoning datasets indicate that RECKONING outperforms the ICR baseline by up to 4.5%. Additionally, RECKONING generalizes better to longer reasoning chains unseen during training, is more robust to distractors in the context.", "questions": "1. In Figure 3, it would be helpful to include more experimental results with training on both 4- and 6-hop questions, or both 2- and 4-hop questions to further validate the proposed algorithm's performance.\n2. It would also be beneficial to include other subtasks like proof generation in the ProofWriter dataset to further explore the algorithm's capabilities.\n3. Conducting more experiments on public pre-trained language models like LLaMA would provide additional validation for the proposed method's superiority.", "limitations": "I can not find any potential negative societal impact.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Overall, the research topic presented in this work is of interest and importance, as it enhances transformer-based language models by incorporating in-context knowledge into parameters to improve reasoning ability. The proposed RECKONING consistently outperforms the conventional ICR-based methods. The proposed method's superiority is further confirmed through comprehensive experiments and ablation studies, which demonstrate its superiority in generalization to longer reasoning chains.", "weaknesses": "While the research topic is interesting and the proposed algorithm outperforms conventional in-context reasoning methods, there are some weaknesses in the paper. The proposed inner-loop of encoding knowledge into model parameters to improve reasoning abilities is limited in its incremental nature, which may lead to issues when facts from different contexts contradict each other. For example, the fact that \"Peter is a son to Kyle\" may be changed to \"Kyle is a son to Peter\" in another question's context. Additionally, the experiment part compares two baselines \"No-Facts\" and \"Randome-Facts,\" which are too weak, and no other publicly available baselines have been compared. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Responses to Reviewer 5k3Z (R5)\n\nWe thank the reviewer for their constructive comments, as well as describing our idea as interesting and important, and recognizing the benefits of our approach. We address the reviewer’s concerns and questions below:\n\n  \n\n**W1: The reviewer argues that if there are contradictory facts between questions, it may lead to issues with the model performance because of the incremental nature of the inner-loop**\n\n  \n\nWe first clarify the inference process of RECKONING. For each new question, we start from the trained meta parameters of the model (which does not contain any background knowledge for the test set questions) and do a few steps of gradient updates to encode the background knowledge. Next, we directly evaluate the updated parameters on the question. After that, we discard this locally updated model. For the next questions, we start from the trained meta parameters again, which do not contain any information of the background knowledge from the previous questions. Thus, the inner-loop in our proposed approach is not incremental. The current question’s inner-loop does not depend on the previous question's inner-loop. Each question’s inner-loop is unique to the associated question during inference. Suppose the previous question’s context and the current question’s context contradict each other. In that case, the model will not be affected since the trained meta parameters will not contain information on the previous question’s context when we do inference on the current question.\n\n  \n\n**W2: The reviewer argues that the experiments contain only weak baselines and no public baselines compared**\n\nWe have added more baselines; see our response to Reviewer 1’s Q3. For a stronger baseline, we have finetuned GPT-XL-Lora with in-context reasoning. We show that on ProofWriter-5-hop, RECKONING (70.2) still improves over this baseline (65.0) when there are distractors in the context of a question. For stronger public baselines, in our supplementary material, we report the performance of GPT-3.5 (text-davinci-003) on ProofWriter and CLUTRR (Table 9, supplementary material). We include the results below as a reference:\n\n\n| | | ProofWriter |      |      |ProofWriter (distractor)|  | |CLUTRR |      |\n|-----------------|------|:-----------:|------|------|:-----------:|------|------|:-------:|------|\n| Method          | 2-h  | 3-h         | 5-h  | 2-h  | 3-h         | 5-h   | 2-h  |   4-h  | 6-h  |\n| GPT-3.5 (0−shot)| 58.4 | 56.4        | 53.7 | 49.1 | 47.1        | 45.3  | 35.6 |   16.0 | 18.5 |\n| GPT-3.5 (8−shot)| 78.0 | 82.4        | 80.1 | 58.7 | 57.2        | 54.5  | 39.0 |   18.5 | 20.8 |\n| RECKONING       | **99.5** | **99.7**        | **99.8** | **79.8** | **83.7**        | **84.0**  | **98.3** |   **97.6** | **94.8** |\n|\n\nThe results show that even a best-performing model like text-davinci-003 still fails to perform well on the reasoning tasks and does not generalize well when the context includes distractors. Compared to a public best-performing model, we show that RECKONING significantly benefits reasoning tasks under a systematic complex setting. For other public baselines, we argue that our idea and the proposed algorithm are novel and have not been done before. Thus, we believe that other public baselines on the two datasets are also weak and do not fit in the scope of this study. Specifically, the public baselines do not evaluate models under a systematic generalization setting.\n\n  \n\n**Q1: The reviewer suggests experiments that mix training data from different hops of CLUTRR for a longer reasoning chain**\n\nWe trained the model using RECKONING on a mixture of 2-hop, 4-hop, and 6-hop data. We report our results in the pdf as Figure 1. We show that the performance of FT-ICR and RECKONING is roughly equivalent at low-hop reasoning. However, RECKONING shows greater improvement when generalizing to OOD hops, similar to Figure 3 in our paper.\n\n  \n\n**Q2: The reviewer suggests expanding our proposed approach to the proof generation task**\n\nWe thank the reviewer for this interesting idea! We note that our results in Table 4 show which facts the model recalls as relevant for reasoning about the question, which can be viewed as the first step of a proof, since the model identifies the relevant facts for reaching the correct answer. Extending RECKONING for the “full” proof generation task would require greater changes that we couldn’t implement in the rebuttal window. We will explore this and present the results in the camera-ready.\n\n\n**Q3: The reviewer suggests conducting more experiments on public pre-trained language models like LLaMA**\n\nThis is a great suggestion! As our responses to W2 and Reviewer 2’s W1/Q1, we have demonstrated that large pre-trained language models like GPT-3.5 and ChatGPT still struggle with complex reasoning and do not generalize well with distractors (Table 9 in the supplementary material). We will conduct more experiments on open-source language models like LLaMA in our revision.\n"}]

Author rebuttal:

## General responses to all reviewers 
We would like to thank the reviewers for providing us with thoughtful comments and constructive feedback. We appreciate that the reviewers recognize our proposed **bi-level learning algorithm** for language reasoning as novel/interesting (R2, R3, R4, R5), useful/important (R1, R5), sound (R3), and our work conducts sufficient **experiments** (R4). We are encouraged that the reviewers see our **paper** as well-written (R2, R3) and well-explained (R4). 

We address the general concerns below:

**(1) A few reviewers asked us to clarify the inference process of RECKONING and how the knowledge-encoding and question-answering were conducted.** 

Our proposed bi-level learning algorithm follows the idea of Model Agnostic Meta-Learning (MAML), where the objective is learning to do few-shot learning for downstream classification tasks. In the case of RECKONING, we are learning to do fast language modeling (e.g., knowledge encoding) for question answering. During inference time, we start from the trained meta parameters and train the model to do language modeling on the given facts using a few steps of gradient descent. Then we use the updated parameters to answer a question. After this, we discard the updated parameters and recover the trained meta parameters for the next question. This inference step is done for each new question. i.e., for each new question, we start from the same trained meta parameters. However, this ability to do inference-time gradient-based learning requires us to use a bi-level optimization algorithm during training to obtain the trained meta parameters.

**(2) Some of the reviewers ask how the performance of RECKONING would generalize with larger language models and real-world non-synthetic datasets.** 

In response to this, we applied RECKONING to GPT2-XL (1.5B, 15 times bigger than the GPT2-small model used in our paper.) with LoRA. We evaluated on ProofWriter 5-hop with all distractors. Compared to FT-ICR’s performance (**65.0**), RECKONING’s performance (**70.2**) is **5.2** percentage points higher. We demonstrate that  RECKONING still shows significant benefits when scaling up to larger models. 

To validate RECKONING’s performance on real-world reasoning tasks, we also evaluated on FOLIO, a complex logical reasoning benchmark involving real-world examples. We report the performances below:

| Model                             | Acc  |
|-----------------------------------|------|
| FT-ICR                            | 53.3 |
| GPT-3.5 (text-davinci-003) 0-shot | 45.1 |
| GPT-3.5 (text-davinci-003) 8-shot | 52.9 |
| chatGPT (gpt-3.5-turbo) 0-shot    | 40.0 |
| chatGPT (gpt-3.5-turbo) 8-shot    | 42.6 |
| RECKONING                         | **54.9** |
|

We can see that RECKONING still performs better than the in-context reasoning baseline, and it even surpasses best-performing large language models (LLMs) like GPT-3.5 and chatGPT. 

In addition, we show that although the two datasets we used are synthetic, even best-performing models like GPT-3.5 still fail under the systematic generalization tests (Table 9 in the supplementary material):
|                 |      | ProofWriter |      |      |ProofWriter (distractor)|  | |CLUTRR |      |
|-----------------|------|:-----------:|------|------|:-----------:|------|------|:-------:|------|
| Method          | 2-h  | 3-h         | 5-h  | 2-h  | 3-h         | 5-h   | 2-h  |   4-h  | 6-h  |
| GPT-3.5 (0−shot)| 58.4 | 56.4        | 53.7 | 49.1 | 47.1        | 45.3  | 35.6 |   16.0 | 18.5 |
| GPT-3.5 (8−shot)| 78.0 | 82.4        | 80.1 | 58.7 | 57.2        | 54.5  | 39.0 |   18.5 | 20.8 |
| RECKONING       | **99.5** | **99.7**        | **99.8** | **79.8** | **83.7**        | **84.0**  | **98.3** |   **97.6** | **94.8** |
|

We can see GPT-3.5 especially struggles when there are distractors in the context of the questions. In general, we use synthetic datasets to allow us to control changes to the fact base (e.g., distractors and longer reasoning chains) such that we can systematically evaluate how sensitive in-context reasoning and RECKONING are to these factors. Our proposed learning algorithm improves model performance in more complex and challenging settings where factors like distractors and longer reasoning chains can be systematically evaluated.


Meta-review: This paper proposes to encode knowledge through gradient updates instead of through concatenation of context and shows this leads to better robustness and generalization. Overall reviewers agreed the idea is novel, the paper well-written, and results show convincingly better generalization to longer reasoning chains.
Decision: Accept (poster)