Crystal: Introspective Reasoners Reinforced with Self-Feedback
Authors: Jiacheng Liu
Ramakanth Pasunuru
Hannaneh Hajishirzi
Yejin Choi
Asli Celikyilmaz
Venue: EMNLP 2023 Main (2023)
Keywords: introspective reasoning
commonsense reasoning
question answering
reinforcement learning

Abstract: Extensive work has shown that the performance and interpretability of commonsense reasoning can be improved via knowledge-augmented reasoning methods, where the knowledge that underpins the reasoning process is explicitly verbalized and utilized. However, existing implementations, including "chain-of-thought" and its variants, fall short in capturing the *introspective* nature of knowledge required in commonsense reasoning, and in accounting for the mutual adaptation between the generation and utilization of knowledge. We propose a novel method to develop an introspective commonsense reasoner, **Crystal**. To tackle commonsense problems, it first introspects for knowledge statements related to the given question, and subsequently makes an informed prediction that is grounded in the previously introspected knowledge. The knowledge introspection and knowledge-grounded reasoning modes of the model are tuned via reinforcement learning to mutually adapt, where the reward derives from the feedback given by the model itself. Experiments show that Crystal significantly outperforms both the standard supervised finetuning and chain-of-thought distilled methods, and enhances the transparency of the commonsense reasoning process. Our work ultimately validates the feasibility and potential of reinforcing a neural model with self-feedback.

CRYSTAL: Introspective Reasoners Reinforced with Self-Feedback
Jiacheng Liu
Ramakanth Pasunuru
Hannanch Hajishirzi
Yejin Choi Asli Celikyilmaz" *
Paul G. Allen School of Computer Science & Engineering, University of Washington
FAIR,, Meta
Allen Institute for Artificial Intelligence
liujcêcs. washington. edu
Abstract
Crystal
Reirforommnt with 5-leedbacck
Knowledge inteapection
Knowlodge
Knewlelge-gran grounded nassoring
Extensive work has shown that the performance and interpretability of commonsense reasoning can be improved via knowledge-augmentt soning methods, where the knowledge that derpins the reasoning process is explicitly ver- balized and utilized. However, existing imple- mentations, including "chain-of-thought" and its variants, fall short in capturing the intro- spective nature knowledge required com- monsense reasoning, and in accounting for the mutual adaptation between the generation and utilization of knowledge. We propose a novel method to develop an introspective common- sense reasoner, CRYSTAL. To tackle common- sense problems, it first introspects for knowl- edge statements related to the given question, and subsequently makes an informed prediction that is grounded in the previously introspected knowledge. The knowledge introspection and chowledge-groundee reasoning modes model tuned via reinforcement learning to mutually adapt, where the reward derives from the feedback given by the model itself. Experi- ments show that CRYSTAL significantly outper- forms both the standard supervised finetuning and chain-of of-thugght distilled methods, and hances the transparency of the commonsense reasoning process. Our work ultimately vali- dates the fcasibility and potential of reinforcing a neural model with self-feedback.
Queetion
Oowa produce milkk.
What
pork
Milkk comes from cowe.
rea-
butritious
( corn
(D)
Cosma provide nutettion.
Superdood leoming...
Chain-af- Chain-af-Thought
Question
comethinng that from chotoes, nutritouus thuld something that from oows. 8o the azsiwar (8).
Figure I: Top: CRYSTAL performing introspective rea- soning on commonsense question. The model first uses its knowledge introspection mode to generate rele- vant knowledge statements, then invokes a knowledge- grounded reasoning mode to predict an answer based on the introspected knowledge. Bottom: chain-of- thought prompting on the same question (generated by text-davinci-oe with original few-shot prompts in Wei et al. (2022)). The intermediate steps fail to provide meaningful insight into reasoning process.
math problems, logical deductions), where the reasoning process consists of explicit and closed- world deduction steps. Chain-of-thougee (CoT) (Wei et al., 2022) and its variants have been suc- cessful in multi-step logical reasoning, yet their ef- fectiveness on commonsense reasoning is marginal, largely due to the lack of the above observation when designing their few-shot prompts. Never- theless, generating the reasoning process is still instrumental for commonsense reasoning, as it im- proves both performance and interpretability of neural models (Shwartz et al., 2020; Liu al., 2022a, i.a.). For such knowledge mowledge-aaumeetee rea- soning approach, two components are indispens- able: (1) introspecting for relevant, high-quality knowledge, and (2) effectively and faithfully utiliz- ing the knowledge make informed final predic- tions.
1 Introduction
Commonsense reasoning poses unique challenges to neural reasoning models. The underlying knowl- edge that grounds the reasoning process is often obscure and inexplicable, even to humans as mainly rely on intuitive inference for such prob- lems (Mercier and Sperber, 2017). This in stark contrast with multi-step logical reasoning (e.g..
"Work done as visiting researcher FAIR, Meta. CCode: tit. com/liujche 1tujchhh98/rrsttal
Our key insight is that these two components are deeply adaptive to each other: knowledge in- trospection should aim to produce knowledge that
Model: huggingface. co/ 11wpcch11///rystal-11111
Demo: huggingface. co/ httpspss//iijcchh/////yys//////

---

would be most beneficial to grounding the subse- quent reasoning, and knowledge-groundee reason- ing should learn to best leverage the previously introspected knowledge. Existing literature does not comprehensively optimize these two compo- nents and the bi-directional interaction between them, and comes with additional complications. showledge-augmennte reasoning methods largely employ task-specifie engineering for knowledge generation and are thus difficult generalize to unseen domains. As for CoT and its variants, the reasoning chains hardly provide meaningful infor- mation due to deficiency in their prompt design (Figure 1).
forms direct QA models finetuned with standard supervised learning and the same data, improving absolute accuracy by 1.5% to 2.5% on different model sizes, and showcases good generalization to unseen benchmarks. This highlights the bene- fits of introspective reasoning over direct inference. Additionally, CRYSTAL substantially outperforms models distilled from CoT produced by large LMs, Through CRYSTAL, illustrate the potential and viability of reinforcing neural reasoning models with self-feedback. An additional benefit of our approach is the memory and time-efficient imple- mentation of PPO via model sharing, which allows this state-of -fetterrrt RL algorithm to be applied to larger models with given amount of resources.
We aim to systematically address the above con- siderations and build a strong, interpretable and generalizable model for commonsense reasoning. The introspective reasoner we develop, named CRYSTAI, tackles commonsense problems by the following (illustrated in Figure I): it first invokes knowledge introspection mode to generate knowl- edge statements related the given question, and subsequently invokes a knowledge groouddd rea- soning mode that ingests both the question and the previously introspected knowledge to predict an an- swer. CRYSTAL is trained with reinforcement learn- ing (RL) to improve the synergy between the rea- soning paths and the final predictions. The knowl- edge introspection mode of the model is trained with PPO (Schulman et al., 2017) to optimize a reward function that characterizes if the generated knowledge can fix prediction errors made by the knowledge-grounded reasoning mode of the model. In this sense, CRYSTAL is reinforced with self- generated feedback. Concurrently, the knowledge- grounded reasoning mode evolves better utilize the introspected knowledge statements for more accurate predictions. These two learning objectives are harmonized through novel interleaved opti- mization schedule, echoing the principles of the EM algorithm (Dempster et al., 1977). We employ a two-stage training process: the RL training stage is preceded by supervised training stage, where CRYSTAL acquires preliminary skills to generate and utilize knowledge by imitating larger LM (e.g., GPT-3).
2 Method
We will first introduce the concept of introspective reasoning ($2.1), followed by a description of our introspective reasoner, CRYSTAL, including its ba- sic functionality ($2.2), training objectives ($2.3), adaptation of the RL algorithm and efficiency im- provements ($2.4), and the design of model training process ($2.5, $2.6).
2.1 Introspective Reasoning
Conventionally, commonsense QA models are de- signed to directly predict answers for questions (e.g. Lourie et al., 2021). These models operate like black boxes and their predictions are difficult to interpret. We consider introspective reasoning, where system first introspects for commonsense knowledge statements that are relevant reasoning about the given question, and subsequently makes an informed prediction that grounded the in- trospected knowledge. We refer to the former step as knowledge introspection, and the latter step as knowledge-groundee reasoning.
Figure 1 exemplifies the introspective reasoning process. Given the question "What comes from cows?" with the correct answer "nutritious flu- ids" " provided among other incorrect choices, the system first generates knowledge statements like "Cows produce milk." " Taking this knowledge state- ment as additional input, the system then invokes knowledge laogge-groudeed reasoning and makes cor- rect prediction.
Experimental results on 25 commonsense QA benchmarks (10 seen, 15 unseen) show that CRYS- TAL not only enhances performance within fixed model sizes, but also amplifies the interpretabil- ity of the reasoning process. CRYSTAL outper-
Introspective reasoning has promise in enhanc- ing model performance on commonsense reasoning while making the reasoning process more inter- pretable. The introspected knowledge reveals the

---

Mode
DO Format
knowledgg-grounded reasoning, the modeling of answer prediction a is denoted as POKA q, k; 0).
Knowledge
Input: What from cows? (A) pork
introspection
organic ... (6) nutritious
fluid (H) carn An Knowledge:
2.3 Training Objectives
Output: Cows produce milk.
To yield the desired outcome of an introspective rea- soning system, we need to make knowledge intro- spection and shooweddg-groundeee adapted to each other. The knowledge introspec- tion component should aim to produce knowledge that would be most beneficial to ground the sub- sequent reasoning, and the knowledge-groundee -grondddd reasoning component should learn to best leverage the previously introspected knowledge. We design the training objectives to account for this mutual adaptation.
Knowledge-
Input: What
from cows? (A) pork
grounded
organic ... (6) nutritious
reasoning
fluid (H) corn Knowledge: Cows produce
reasoning well-
milk. 'nn Answer:
Output:
Table 1: CRYSTAL'S 1/O format for its two modes.
reasoning paths that lead to the final predictions, which human users can observe. The system can explore and ensemble multiple reasoning paths and thus make more informed final prediction.
Adapting reasoning to introspection. Suppose knowledge statement is sampled online from CRYSTAL in the knowledge introspection mode: POk Po f). We use standard supervision and minimize srowledge-groundee reasoning loss:
Knowledge introspection. The term "knowledge introspection" was introduced by Liu et al. (2022a), which also developed dedicated knowledge in- trospection model, Rainier. Our work extends this idea to unified introspective reasoning model.
LOKA (0) log POKA (a" [q, R;; 0),
2.2 CRYSTAL
where a" the correct answer for question q-
CRYSTAL, the introspective reasoner that we de- velop, is a unified model that supports the end-to- end workflow of an introspective reasoning sys- tem. CRYSTAL has two modes of operation: knowl- edge introspection and knowledge-groune soning. In knowledge introspection, the model ac- cepts question input, and outputs knowledge statement relevant to the question. In knowledge- grounded reasoning, the model ingests both the question and the previously generated knowledge statement as input, and outputs a prediction. The system produces an final prediction by consulting and aggregating predictions resulted from all the available reasoning paths, effectively harnessing the power of ensembled reasoning.
Adapting introspection to reasoning. The de- sirability of introspected knowledge is determined by its effectiveness on the subsequent knowledge- grounded reasoning. A knowledge statement is good if grounding in it can remediate other- wise incorrect prediction, and is bad if it misleads otherwise correct prediction. Formally, good knowledge statement should yield
rea-
a" + arg max POKA (alq, E0),, aEA
a" arg max PoKA
aeA
where A is the candidate set for question and e stands for no knowledge; and vice versa for bad knowledge statement.
1/O format. Since CRYSTAL has two modes of operation, it needs to discern when to conduct knowledge introspection and when to engage in kmowledge-grounderr reasoning. Drawing inspira- tion from Tafjord and Clark (2021), we structure the input/output format as demonstrated in Table 1. This format is adapted from UnifiedQA (Khashabi et al., 2020), and we detail our modifications in SA.
However, a knowledge statement consists of a quence of discrete tokens, rendering standard gra- dient methods infeasible for optimizing the intro- spected knowledge. Following Liu et al. (2022a), we formulate the problem as reinforcement learn- ing (RI., and optimize a reward function that char- acterizes the desirability of knowledge:
1 tanh (s(a")a
Notation. CRYSTAL is a sequence-to-sequencc generative model parameterized by In knowl- edge introspection, the modeling of knowledge k given question q is denoted as Pox (klq: in
max s(a'lg, (ill,,1))
2 tanh (s(a"lg,=)
a'cA((ar)
max s(a'lg. =))
d'A)(a*)

---

where s(alq, k) is the pre-softmax logit of PQKA(alq, k; 0) on the single-token answer a. The reward approaches +1 for good knowledge state- ments and -1 for bad ones.
Training Stage
Lxx gt
Sf gx
S gRa
L
L 2E1
Spgt
Sox
Syxx
training
Training Stage II
We use the PPO algorithm to optimize this reward. A knowledge introspection loss (0) can be defined as function of the reward and the model parameter 0, following Liu et al. (2022a). Since this training loss is derived from the downstream mhowledge-groundee
S
ppo
I ppo
S FOKA
QKA
Spro
Syxa
Speo
Syxa
training
Illustration of the interleaved optimization schedule for both training stages. In training stage I, during each cycle, Lox is optimized for Sox iterations. and then LOKA is optimized for SQKa iterations. Pro- gressing to training stage during each cycle, Lero is optimized for Speo iterations, and then LoKa is opti- mized for SOKA iterations.
Figure 2:
reasoning results produced by the same model, the model reinforced with feedback given by itself.
During training, the two objectives, Lpro (f) and LOKA( (0), are optimized under an interleaved sched- ule (rather than jointly), which is described in $2.6.
results in improved memory and time efficiency for PPO training, as discussed in $4.4.
Leaving out the direct QA objective. To pre- vent the model from taking reasoning shorteuts and encourage it leverage the introspected knowl- edge, we deliberately left out a potential, direct QA objective:
2.5 Two-Staged Training
PPO requires that the policy model is initialized from reasonably good state. Typically, PPO train- ing follows a supervised finetuning stage for the policy model (Stiennon al., 2020; Ouyang et al., 2022). For Rainier, an imitation learning stage, during which the model supervised on silver knowledge statements obtained from a few-shot GPT-3, precedes the RL training stage. This im- itation learning stage imparts the model with the preliminary skill of generating question-specific knowledge, and sets promising starting point for RL.
Loa = log logPo((a"lg)
(1)
As we will show in experiments, including this direct QA loss hurts performance, probably by allowing the model to take shortcuts around the knowledge.
2.4 PPO and Model Sharing
PPO, Proximal Policy Optimization (Schulman al., 2017), is RL algorithm that has been widely used in aligning LMs with human feedback (Stiennon al., 2020; Ouyang et al., 2022; OpenAT, 2022; Wu ct al., 2023). It is also adopted by Liu et al. (2022a) to train their knowledge introspection model, Rainier.
Drawing from this concept, employ two- stage training process for CRYSTAL In training stage I, the model is tuned to conduct both knowl- edge introspection (by imitating a few-shot GPT-3) and wwwwlede-groondeee reasoning. We minimize two losses: a knowledge introspection loss
Within the context of PPO terminology, CRYS- TAL's knowledge introspection mode assumes the role of the policy model, while its knowledge- grounded reasoning mode functions as the reward model. PPO further employs value model to esti- mate the value function for states containing partial knowledge statements, and we propose to reuse the parameters of CRYSTAL for the value model as well. Consequently, while in conventional PPO the policy, value and reward models are parameterized separately, when training CRYSTAL they share the same underlying parameters. CRYSTAL essen- tially a generative LM equipped with two heads: an LM head that comes into play during the policy and reward modeling, and value regression head that is activated in value modeling. This model sharing
(ox(O) = - log PQK (klq: 0),
and a knowledge-grounded reasoning loss
(oKA(O) log (aooaooaa""aaa k; 0),
where lo is silver knowledge statement generated by the few-shot GPT-3. In training stage II, we follow the procedure in $2.3 adapt the knowledge introspection and howweedge-groundee reasoning modes to each other.
2.6 Interleaved Optimization Schedule
Through empirical analysis, we have observed in- terleaving the two training losses in each stage

---

yields beneficial outcomes as opposed to jointly optimizing them. In stage we optimize Lok for a specific number of iterations, followed by opti- mizing LOKA for another set number of iterations, repeating this cycle. Similarly, In stage II, we op- timize LpPo for designated number of iterations, followed by optimizing LQKA for another set num- ber of iterations, repeating this cycle. This design bears resemblance the EM algorithm (Dempster al., 1977), wherein the hidden variable corre- sponds to the knowledge statement. Optimizing CpPo can be likened to estimating the hidden vari- ables, while optimizing LOKA is akin to updating the parameter estimation based on the current as- signment of hidden variables. The interleaved opti- mization schedule is illustrated in Figure 2.
for T5-large, 16 for T5-3b, and 64 for T5-11b). with the Huggingface Transformers and Accelerate libraries (Wolf et al., 2019; Gugger et al., 2022). See Table 10 (appendix) for the complete hyperpa- rameter settings.
Baselines. We primarily compare CRYSTAL with models trained on the same datasets using the stan- dard QA objective (i.e., Equation 1), referred as "Direct QA". These models are also based on the pretrained T5 of the three sizes above. Addition- ally, we compare our model to Rainier (Liu et al.. 2022a) and several CoT-distilled models. Among these. fine-tune-CoT (Ho et al., 2022) use zero-shot reasoning chains elicited from text-daviie rext-davinci-002 to finetune smaller variants of GPT-3; SCoTD (Li et al., 2023a) employs a similar distillation strategy, whereas the teacher model is code-davinci and the target model is OPT up to 1.3B param- eters; SCOTT (Wang et al., 2023) proposes addi- tional techniques to improve the reasoning integrity, including a contrastive decoding method to elicit more consistent reasoning chains from the teacher model and counterfactual reasoning method to train the target model, and yet does not enable full bidirectional adaptation between the reasoning pro- cess and the final prediction, as our method does. is worth noting that these CoT-distilled models are often trained on specific datasets, so we only present their reported performance on the datasets they were trained on.
3 Experimental Setup
Datasets, To promote generalization, we train CRYSTAL on 10 datasets (following Liu et al. (2022a)): OpenBookQA (Mihaylov et al., 2018), ARC (easy and hard splits) (Clark et al.. 2018). AI2Science (elementary and middle splits) (Clark al., 2018), CommonsenseQA (Talmor et al.,
2019), QASC (Khot et al., 2020), PhysicalIQA (Bisk et al., 2020), SocialIQA (Sap et al., 2019). and Winogrande (Sakaguchi et al., 2020). We use the development set of these datasets to eval- uate model performance (i.e., seen evaluation). For unseen evaluation, we use the development set of 15 additional datasets: Com2Sense (Singh et al., 1), SeiQ (Welbl al., 2017), QuaRel (Tafjord al., 2019a), QuaRTz (Tafjord et al.. 2019b), CycIC, COmVE (Wang et al., 2020), Wino- grad Schema Challenge (Levesque et al., 2011), COPA (Gordon et al., 2012), NumerSense (Lin et al., 2020), PROST (Aroca-Oucllett, et al., 2021), SWAG (Zellers et al., 2018), HellaSwag (Zellers et al., 2019), CODAH (Chen et al.. 2019), Story Cloze Test (Mostafazadch et al., 2016), and aNLI (Bhagavatula al., 2020). See Table (appendix) for details. On the training datasets, we get sil- ver knowledge from the davinci version of GPT-3 (Brown et al., 2020), with the few-shot prompts in Liu et al. (2022a).
We also report the existing SOTA performance achieved by non-retrieval methods on each seen dataset. We exclude retrieval- based methods for fair comparison, because CRYSTAL does not rely on retrieval from extra sources.
4 Results
4.1 Performance
The performance results are presented in Table 2 and Table 3. We organize the results based on the size of the models and compare them baseline models that are no smaller than our models.
On seen datasets, across all model sizes we ex- periment with, CRYSTAL consistently outperforms the direct QA baseline (by 1.5%~2.55% depending on model size). This demonstrates that our training process is superior to the standard supervised train- ing and brings substantial performance gains to the
Models. Similar to Liu et al. (2022a), initial- ize CRYSTAL with T5 (Raffel al., 2020). The value regression head is initialized from scratch at the beginning of stage II training. We exper- iment with three model sizes: T5-large, T5-3b, and T5-11b. We train models on V100 GPUs (8
2 Accesssed from the A12 leaderboards on 08/24/2023.

---

Method
Base model
Size
OBQA ARC_e ARC A A12Sci CSQA
QASC PIQA SIQA WG
SOTA (wo retrieval)
87.80
81.14
82,20
72.28 90.13 83.15 91.28
Fine-tune-CoT
GPT-3 UPT-3-babbage
1.38
43.08
SCoTD
QPT-1.3b
1.3B
67.00
67.00
Rainier (+ UnifiedQA)
T5-1arge
770M 62.58
69.60
67.72
55.18
68.29
63,20
67.24
54.97
65.67
57.01
56.91
Direct QA
T5-large
770M 65.07
63.00
64.74
48.49
72.36
65.60
67.40
54.75
75.19
69.19 69.93
CRYSTAL (ours)
T5-large
770M 66.74
64.20
65.61
52.84
71.54
68.00
70.52
56.80
75.68
69.81
72.38
Cine-tue-Cool
GPT-3-curic
6.7B
56.76
SCOTT
T5-3b
3B
75.40
65.00
Direct QA
T5-3b
72.00
77.19
83.74
75.20
67.82
83.03 76.77 82.08
CRYSTAL (ours)
T5-3b
3B
78.33
74.20
78.25
66.22
84.55
79.20
80.10
74.30
84.49
78.40 83.58
Direct QA
T5-11b
11B
82,49
80.00
84,56
72.91
87.80
84,00
81.98
78. 29 88,36
78.45 88.56
CRYSTAL (ours)
T5-11b
11B
84.58
85.40
87.54
73.24
89.43
84.80
82.31
81.97 88.08
82.24 90.77
Table 2: Results seen datasets. Accuracy the development set is reported.
Method
Size
All
C2S
SciQ QuaRel QualTz CyclC ComVE
WSC COPA NumerSense PROST SWAG HollaSwag CODAH SCT aNLI
Direct QA
770M
60.93
58.75
66.80
71.22
67.45
49,83
83.45
78.02
74.20
23.00
38.07
46.48
45.65
62.36
86.37 65.27
CRYSTAL (ours)
56.52
70.86
67.71
77.60
48.51
Direct QA
3B
67.73
58.57
75.90
81.29
70.83
55.35
82.05
36.83
57,77
49.31
75.54
95.14
CRYSTAL (ours)
3B
72,06
65.98
79.50
80,58
73.96
60.20
95.49
89.74
91.20
28.50
44.35
60.48
56.58
81.45
96.37
76.57
Direct QA
11B
76.83
75.45
83.20
86.69
77.34
69.57
96.89
94.14
94.00
30.50
44.39
65.68
69.37
84.73
97.70 82.83
CRYSTAI (ours)
76.30
Table 3: Results unseen datasets, Accuracy on the development is reported.
model. CRYSTAL also outperforms the combina- tion of Rainier and UnifiedQA, especially over the last five datasets (which UnifiedQA is not trained on). This shows the benefit of adapting knowledge- grounded reasoning to the introspected knowledge.
Support
34%
Trivial
19%
8%
CRYSTAL performs very closely to existing non- retrieval SOTA methods, setting new SOTA on two datasets (CommonsenseQAA QASC), and has less than 3% gap on other four (OpenBookQA, PIQA, SIQA, Winogrande). It is worth noting that these SOTA methods are good on different datasets, whereas CRYSTAL is single model with strong performance on all these benchmarks. CRYSTAL is also competitive when compared with CoT-distilled models with similar sizes. The large and 3b ver- sions of CRYSTAL beat Fine-tune- Mine-tune-CoT- on Com- monsenseQA by 27% and 23%, respectively, de- spite having smaller model sizes. CRysTAL-Large is comparable to SCoTD on OpenBookQA and CommonsenseQA, and CRYSTAL-3b significantly outperforms SCOTT on CommonsenseQA (by 5%) and QASC (by 9%).
Contradict
3%0
5%
31%
Unrelated
Repeat
Related
Figure 3: Expert annotation on the relationship between the introspected knowledge and the final prediction.
model, CRYSTAL-11b achieves an average accu- racy over 80% on these benchmarks.
4.2 Interpretability
Beside QA accuracy, we measure whether the in- trospective reasoning conducted by CRYSTAL pro- vides good interpretability to its reasoning process. We asked three NLP experts to annotate the rela- tionship between the introspected knowledge and the final prediction. We randomly selected 100 ex- amples (four from each of the 25 datasets, includ- ing both seen and unseen ones), and each annotator made a full pass over them. For each example, the annotator chooses one of the following labels:
Being trained multiple commonsense datasets, CRYSTAL exhibits good generalization unseen datasets. As shown in Table CRYSTAL achieves a 2.0%~4.3% average accuracy improve- ment over the direct QA baseline on the unseen evaluation benchmarks. The largest version of our

---

Question
Direct QA's pred CRYSTAL'S pred
Task
CRYSTAL's introspected knowledge
They discussed the company's s budget at the business meeting but the was boring and the A topic of the budget long. (A) budget (B) meeting
WG
The topic of the meeting was boring.
Find spices casily the kitchen. (A) Arrange spices from hot mild in the kitchen order
PIQA
to find them by taste. (B) Arrange your spices alphabetically to make finding them easy.
A spice alphabet is used to find spices.
B
What comes from cows? (A) pork (B) can be organic (C) holding nutrients (D) drinking (F) rice (F) antigens (G) nutritious fluid Cows produce milk.
A
QASC
G
Paul wants carrots docsn't need to drive anywhere. He gets them from where? (A)
CSQA
refrigerator (B) store (C) farmer's market (D) supermarket (E) dryer
Carrots are stored the refrigerator.
Frilled sharks and angler fish live far beneath the surface of the ocean, which is why they are
D
OBQA
known as (A) Deep sea animals (B) fish (C) Long Sea Fish (D) Far Sea Animals
Deep animals are found in the occan.
A
An ancmometer is a tool that measures (A) wind dircction. (B) wind speed. (C) air pressure.
B
ARC_e
(D) air temperature.
An anemometer measures wind speed and direction.
Table 4: Examples of CRYSTAL's introspected knowledge and predictions grounded in the knowledge. The first row each section the original question the prediction made by direct QA model; second row the knowledge statement generated by CRYSTAL in the knowledge introspection mode, and the prediction made by CRYSTAL under wwwwwldge-goouddee reasoning with this knowledge statement. We show correct answers in green and incorrect answers red.
The knowledge can be part of a non-trivial reasoning chain that supports the predicted answer.
Support:
the cases the knowledge trivially entails the predic- tion, and in another 31% % of the cases the knowledge is otherwise related to the question. The knowledge repeats known information in the question 5% of the time, and unrelated to the question or contra- dicts with the prediction 11 % of the time. Overall, the reasoning process has good interpretability the majority of cases.
The knowledge is trivial paraphrase of the question and the predicted answer.
Trivial:
Repeat: The knowledge is mere repetition of known information given in the question.
Related: The knowledge is topically related to the question and/or the choices, but can- not be part of a reasoning chain to support or refute any of the choices.
4.3 Qualitative Examples
We present several examples in Table 4 to illustrate the reasoning process of CRYSTAL. In most cases, the introspective reasoning carried out by CRYS- TAL leads to more accurate predictions compared the direct QA model. The knowledge introspected by CRYSTAL often proves to be beneficial in ar- riving at the correct prediction from human inter- pretation standpoint. For example, the knowledge "Cow produce milk" aids in concluding that "Nu- tritious fluid comes from cows" (with the implicit knowledge that "Milk is nutritious fluid"). This showcases how the mhowledge-grouncde reasoning of CRYS leverages introspected knowledge reach accurate predictions. However, there are ex- ceptional cases where the groouddddd rea- soning fails to incorporate the introspected knowl-
Unrelated: The knowledge is unrelated to the question.
Contradict:
The knowledge can part of reasoning chain that refutes the predicted answer, or supports a different choice.
See Table (appendix) for a detailed description of these labels and some examples.
The annotators reached a moderate level of agreement (Fleiss 1977)). As shown in Figure 3, in 34% of the cases the introspected knowledge is found to support the final prediction in a non-trivial manner. 19% of
0.53 (Landis and Koch,

---

Method
trained models
frozen models
# forwards
backwards
# optimizer steps
Rainier
2
2
+ 2s
2s
2s
CRYSTAL
4 8
8
8
Table 5: Theoretical memory and time consumption of PPO training in CRYSTAL. is the number of minor steps in each PPO iteration. (In our experiments, we use = 4.)
Model
Base model
Trainable params
# GPUs
Total GPU mem
PPO training speed
Rainier
T5-large
1.54B
8
153 GiB
10.97 s/it
CRYSTAL
T5-large
770M
8
129 GiB
6.96 s/it
CRYSTAL
T5-3b
3B
16
488 GiB
14.07 s/it
CRYSTAL
T5-11b
11B
64
2032 GiB
60.30 s/it
Table 6: Empirical memory usage and speed of training CRYSTAL (stage II). Experiments are conducted on V100 GPUs. Fully sharded data parallel (FSDP) and b6 mixed precision are enabled.
edge. For example, the correct knowledge that "An anemometer measures wind speed and direction" is introspected, but CRYSTAL still predicts "air pressure" instead of "wind speed" the thing measured by anemometers.
iteration, conducting one borward-backward pass and optimizer update the policy model and the value model, respectively. In contrast to Rainier, PPO on CRYSTAL needs to store only two models: shared pol- icy/value/reward model which is being actively up- dated, and an initial version of the policy model (to compute the KL penalty term). In each PPO iteration, CRYSTAL needs 4 + forward passes, backward passes, and optimizer updates on the model: gradient-less rollout from the policy, gradient-less forward pass the initial pol- icy model, two gradient-less forward passes on the reward model, plus for each minor step the itera- tion, one barward-backwarr pass and one optimizer update on the policy/value model.
4.4 Memory and Time Efficiency
As mentioned in $2.4, the PPO training in CRYS- TAL improves efficiency of the conventional PPO algorithm by model sharing. through theoretical and empirical analysis, we com- pare the memory and time consumption of PPO training in CRYSTAL and Rainier (Liu et al,, 2022a). which employs the conventional PPO.
this section,
PPO in Rainier requires three different mod- els: policy model, value model, and reward model. The policy model Rainier, while the re- ward model a fixed QA model. The value model is a separate model that shares the same architec- ture as Rainier, with the exception that it has a value regression head instead of language model- ing head. The policy and value models are trained simultaneously, while the reward model remains frozen. Additionally, an initial version of the pol- icy model must be retained (to calculate the KL penalty term). Therefore, total of four models are stored, with two of them being actively updated. In each PPO iteration, Rainier requires 5 + 2s for- ward passes, 2s backward passes, and 2s optimizer updates on the model. (s is the number of mi- steps in each PPO iteration.) This involves executing a gradient-less rollout from the policy, one gradient-less forward pass on the value model, another gradient-less forward pass on the initial policy model, two gradient-less forward passes on the reward model, and for each minor step in the
Table 5 summarizes the theoretical memory and time consumption of Rainier and CRYSTAL and Table 6 reports the empirical memory usage and speed in the stage II training of these models. Com- pared with Rainier, CRYSTAL has less trainable parameters, consumes less GPU memory, and has faster training speed. The superior memory and time efficiency of CRYSTAL enables training larger models, and a 1 1b model can be reinforced with 64 V100 GPUs.
4.5 Ablations
The effect of RL. We report the impact of remov- ing RL (i.e. training stage II) in Table 7. Across different model sizes, the performance of CRYSTAL on seen datasets consistently decreases by approx- imately 0.5% 0.6% when training stage Il is omitted. This highlights the significance of RI. in enchancing the knowledge introspection and knowl- edge grounded reasoning capability of CRYSTAL.

---

Method
Size
Seen
methods in Table 12 (appendix). Our method is the first account for the mutual adaptation of knowledge generation and knowledge- grouuneegruddedd reasoning in unified model setting.
CRYSTAL (ours)
770M
66.74
Stage
770M
16.16
Direct QA loss
770M
65.36
CRYSTAL (ours)
3B
78.33
Relation to shain-of-thoughh distillation. A se- ries of work endow smaller LMs with step-by-step reasoning capabilities by distilling from chain-of- thought (CoT) generated by large LMs (Li et al., 2022; Shridhar et al., 2022; Magister al., 2022; Ho et al., 2022; Fu et al., 2023; Li et al., 2023a; Wang et al., 2023). We share similarity with this line of work in that our part of training process (i.e.. training stage I) include distilling the emergent ca- pability of larger LM to smaller one. We differ in that capture the introspective nature of knowl- edge required for commonsense reasoning, and we further use reinforcement learning to improve the synergy between reasoning paths and final answer predictions.
Stage
38
77.79
CRYSTAL (ours)
11B
84.58
Stage
11B
84.08
Table 7: Ablations on the RL training stage (i.e., stage II). Average accuracy on the development set of seen datasets is reported.
Stage I
Stage II Seen
interleaved interleaved 66.74
joint
interleaved
66.66
joint
joint 66.31
Table 8: Ablations on the interleaved training objec- tives. Average accuracy on the development set of seen
datasets is reported.
Improving from self-feedback. Several papers have proposed to improve LMs using feedback from themselves. For example, Zelikman al. (2022) proposes to train a model with its self- generated reasoning steps that result in itself mak- ing the correct final predictions. Huang et al. (2022) chooses which self-generated reasoning chains to train on by selecting the high-confidence, self- consistent ones. Both papers use supervised loss to improve the model. To our best knowledge, we are the first improve models from self feedback using RL.
Impact of the direct QA loss. We experimented with training the stage model with the addition of the direct QA loss ($2.3, Equation 1). As shown in Table 7, training with this additional loss hurts performance by 0.8%. We therefore did not include this loss in the training objective of CRYSTAL
Interleaved objectives. To demonstrate the ad- vantages of intericaving the training objectives, we explore an alternative approach using joint ob- jective. In this approach, during training stage I, we optimize the joint loss, Lox + LOKA, in each iteration. Similarly, during training stage II, optimize the joint loss, LpPo + LOKA. Table presents the results of this approach, where the interleaved objectives are replaced with the joint version. As such, the performance on seen datasets decreases. This suggests that the interleaving of objectives in CRYSTAL provides benefit over the joint optimization approach.
Concurrent our work, Madaan et al. (2023) pro- poses an inference-time method to improve text generation by taking an LM's own feedback on the output, and yet it relies on the emergent behav- ior of LLMs, whereas CRYSTAL improves through RL and can be applied to smaller LMs to achieve higher performance than larger LLMs.
Conclusion
We develop a method to build introspective reason- ers that achieves superior performance and good interpretability on commonsense reasoning tasks. Compared with prior literature, our method com- prchensively accounts for the introspective nature of knowledge required in commonsense reason- ing, and the mutual adaptation of knowledge intro- reasoning. Our approach highlights the feasibility and benefit of training neural models with self-feedback.
5 Related Work
Knowledge-augmented reasoning. There has been numerous work that grounds reasoning in model-generated knowledge (Bosselut et al., 2021;
Rajani al., 2019; Latcinnik and Berant, 2020;
Shwartz al., 2020; Paranjape al., 2021; Liu et al., 2022b; Gu et al., 2022; Liu et al., 2022a; Wang et al., 2022b,a; Yu et al., 2022; Li et al., 2023b; Wei et al., 2022). We summarize these
spection and

---

Limitations
Event, February 2-9, 2021. pages 4923-4931. AAAI Press.
CRYSTAL is intended to solve commonsense QA problems, and its performance on non- commonsense applications is unknown and thus requires further investigation. There is also limit on the length of knowledge it generates in our ex- perimental setting, and it has not been tested on gen- erating long and coherent text. Extra care should be taken when applying our model in production environments, especially when making critical de- cisions or exposing its generated contents directly to human end users.
Tom B. Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah. Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, 1lya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners In Ad- vances in Neural Information Processing Syxtems 33: Anmual Conference on Neural Information Process- ing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
Acknowledgements
We thank members of the H2lab for their con- structive feedback. This work was funded in part by the DARPA MCS program through NIWC Pa- cific (N66001-19-2-4031), NSF IIS-2044660, NSF DMS-2134012, and ONR 180014-18-1-288 JL is supported in part by the Meta AT Mentorship program.
Michael Chen, Mike D' Arcy, Alisa Liu, Jared Fernan- dez, and Doug Downey. 2019. Codah: An adversar- ially authored question-anse answer dataset for common sense. ArXiv preprint, abs/1904.0436.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question an- swering? try arc, the ai2 reasoning challenge. ArXiv preprint, abs/1803.054577
References
Arthur P Dempster, Nan M Laird, and Donald Rubin. 1977. Maximum likelihood from incomplete data via the algorithm. Journal of the royal statistical sociery: series B (methodological), 39(1):1-22.
Stéphane Aroca-Ouellette, Cory Paik, Alessandro Ron- cone, and Katharina Kann. 2021. PROST: Physi- cal reasoning about objects through space and time. In Findings the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4597-4608, Online. Association for Computational Linguistics.
Yao Fu, Hao-Chun Peng. Litu Ou, Ashish Sabharwal, and Tushar Khot. 2023. Specializing smaller lan- guage models towards multi-step reasoning. ArXiv preprint, abs/2301.12726..
Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Han- nah Rashkin, Doug Downey, Wen-tau Yih, and Yejin Choi. 2020. Abductive commonsense reasoning. In 8th International Conference on Learning Represen- tations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OvenReview.net.
Andrew Gordon, Zornitsa Kozareva, and Melissa Roem- mele. 2012. SemEval-2011 task 7: Choice of plau- sible alternatives: An evaluation of commonsense causal reasoning. In #SEM 2012: The First Joint Conference Lexical and Computational Seman- tics Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop Semantic Eval- uation (SemEval 2012). pages 394-398, Montréal, Canada. Association for Computational Linguistics.
Yonatan Bisk, Rowan Zellers, Ronan LeBras, Jianfeng Gao, and Yejin Choi. 2020. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourdh AAAI Conference on Artificial Intelli- gence, AAAI 2020, The Thirty-Secont Innovative Ap- plications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432- 7439. AAAI Press.
Yuling Gu, Bhavana Dalvi, and Peter Clark. 2022. DREAM: Improving situational QA by first elab- orating the situation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1115-1127, Seattle, United States. Association for Computational Lin- guistics.
Antoine Bosselut, Ronan Le Bras, and Yejin Choi. 2021. Dynamic neuro-symbolic knowledge graph construc- tion for zero-shot commonsense question answering. In Thirty-Fifth AAAI Conference on Artificial Intel- ligence, AAAI 2021, Thirty-Third Conference on In- novative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Ad- vances in Artificial Intelligence, EAAI 2021, Virtual
Sylvain Gugger. Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, and Sourab Mangrulkar. 2022. Accelerate: Training and inference at scale made simple, efficient and adaptable. https:// // gi git..b.co

---

Namgyu Ho, Laura Schmid, and Sc- Young Yun. 2022. Large language models are reasoning teachers. ArXiv preprint, abs/2212. 10071.
Jiacheng Liu, Skyler Halliman, Ximing Lu, Pengfei He. Sean Welleck, Hannaneh Hajishirzi, and Yejin Choi. 2022a. Rainier: Reinforced knowledge introspector for commonsense question answering. In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8938-899 8958,, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang. Hongkun Yu, and Jiawei Han. 2022. Large language models can self-improve. ArXiv preprint, 0bs/2210.11610
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Han- naneh Hajishirzi. 2020. UNIFIEDQA: Crossing for- mat boundaries with single QA system. In Find- ings the Association for Computational Linguisties: EMNLP 2020, pages 1896-1907, Online. Association for Computational Linguistics.
Jiacheng Liu, Alisa Liu. Ximing Lu, Sean Welleck, Pe- ter West, Ronan Le Bras. Yejin Choi, and Hannaneh Hajishirzi. 2022b. Generated knowledge prompting for commonsense reasoning, In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 3154-3169, Dublin, Ireland. Association for Compu- tational Linguistics.
Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2020. QASC: A dataset for question answering via sentence compo- sition. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Secooy- Innovative Applications of Artificial Intelligence Con- ference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8082-8090. AAAI Press.
Nicholas Lourie, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. UNICORN on RAINBOW: A universal commonsense reasoning model on new multitask benchmark. In Thirty-Fifth AAAI Confer- ence on Artificial Intelligence, AAAI 2021, Thirty- Third Conference on Innovative Applications of Arti- ficial Intelligence, IAAI 2021, The Eleventh Sympo- sium on Educational Advances in Artificial Intelli- gence, EAAT 2021, Virtual Event, February 2-9, 2021, pages 13480-13488. AAAI Press.
J Richard Landis and Gary G Koch. 1977. The mea- surement of observer agreement for categorical data. biometrics, pages 159-174.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang. Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with self-feedback. ArXiv, abs/2303. 17651.
Veronica Latcinnkk and Jonathan Berant. 2020. Explain- ing question answering models through text genera- tion. ArXiv preprini, abs/2004.00556
Hector J. Levesque, Ernest Davis, and L. Morgenstem. 2011. The winograd schema challenge. In Interna- tional Conference on Principles of Knowledge Rep- resentation and Reasoning.
Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. 2022. Teaching small language models to reason. ArXiv preprint, abs/2212.0844
Liunian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai- wei Chang, and Yejin Choi. 2023a. Sym- bolic chain-of- th-uhht distillation: Small models can also "think" step-by-step. ArXiv.
Hugo Mercier and Dan Sperber. 2017. The enigma of reason.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct elec- tricity? new dataset for open book question an- swering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pages 2381-2391, Brussels, Belgium. Association for Computational Linguistics.
Shiyang Li, Jianshu Chen, Yelong Shen, Zhiyu Chen, Xinlu Zhang, Zekun Li, Hong Wang, Jingu Qian. Baolin Peng. Yi Mao, Wenhu Chen, and Xifeng Yan. 2022. Explanations from large language mod- els make small reasoners better. ArXiv preprint. abs/2210.06726.
Zekun Li, Baolin Peng, Pengeheng He, Michel Galley, Jianfeng Gao, and Xi Yan. 2023b. Guiding large language models via directional stimulus prompting. ArXiv preprint, 1
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages Califoria. Association for Computational Linguis- tics.
Bill Yuchen Lin, Scyeon Lee, Rahul Khanna, and Xiang Ren. 2020. Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre- Trained Language Models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6862-68666 Online. Association for Computational Linguistics.
San Diego
OP 2022. Introducing chatgpt.

---

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welin- der, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. 2022. Training language models follow instructions with human feedback. ArXiv preprint, abs/2203.02155.
commonsense question answering with self-talk. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4615-4629, Online. Association for Computa- tional Linguistics.
Shikhar Singh, Nuan Wen, Yu Hou, Pegah Alipoormo- labashi, Te-lin Wu, Xuezhe Ma, and Nanyun Peng- 2021. COM2SENSE: A commonsense reasoning benchmark with complementary sentences. In Find- ings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 883-898, Online. Associa- tion for Computational Linguistics.
Bhargavi Paranjape. Julian Michael, Marjan Ghazvininejad. Hannaneh Hajishirzi, and Luke Zeutlemoyer. 2021. Prompting contrastive explana- tions for commonsense reasoning tasks. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4192-4192, Online. Association for Computational Linguistics.
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler. Ryan J. Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2020. Learning to summarize from human feedback. ArXiv preprint, (bss/200.01325.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lec, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn Res., 21:140:11-00 1-440:67.
Oyvind Tafjord and Peter Clark. 2021. General-purpose pnvestion-answentn with macaw. ArXiv preprint, abs/2109.0259)
Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Explain your- self! leveraging language models for commonsense reasoning. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 4932-4942, Florence, Italy. Association for Computational Linguistics.
Oyvind Tafjord, Peter Clark, Matt Gardner, Wen-tau Yih, and Ashish Sabharwal. 2019a. QUAREL: A dataset and models for answering questions about qualitative relationships. The Thirty-Thire AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty- First Innovative Applications of Artificial In- telligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 -eeruary 2019, pages 7063-7071. AAAI Press.
Keisuke Sakaguchi. Ronan Le Bras, Chandra Bhagavat- ula, and Yejin Choi. 2020. Winogrande: An adver- sarial winograd schema challenge at scale. In The Thirty-Fourth AAAI Conference on Artificial Intelli- gence, AAAT 2020, The Thirty-Second Innovative Ap- plications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8732- 8740. AAAI Press.
Oyvind Tafjord, Matt Gardner, Kevin Lin, and Peter Clark. 2019b. QuaRTz: An open-domain dataset of qualitative relationship questions. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JCNLP)), pages 5941-5946, Hong Kong, China. Association for Computational Linguistics.
Maarten Sap. Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social IQa: Com- monsense reasoning about social interactions. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- pages 4463-
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A ques- tion answering challenge targeting commonsense knowledge. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies. Volume (Long and Short Papers), pages 3149-4158, Minneapolis, Minnesota. Association for Computational Linguistics.
guage Processing 4473, Hong Kong, China. Association for Computa- tional Linguistics.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. ArXiv preprint, abs/1707.06347.
Cunxiang Wang, Shuailong Liang, Yili Jin, Yilong Wang. Xiaodan Zhu, and Yue Zhang. 2020. SemEval- 2020 task 4: Commonsense validation and explana- tion. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 307-321, Barcelona (online). International Committee for Computational Linguistics.
Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan. 2022. Distilling multi-step reasoning ca- pabilities of large language models into smaller mod- els via semantic decompositions. ArXiv preprint, abs/2212.00193..
Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Unsupervised
Peifeng Wang, Aaron Chan, Filip Ilievski, Muhao Chen, and Xiang Ren. 2022a. Pinto: Faithful language

---

reasoning using prompt-generated rationales. ArXiv preprint, abs/2211.01562.
Peifeng Wang, Zhengyang Wang, Zheng Li, Yifan Gao, Bing Yin, and Xiang Ren. 2023. Scott:
Self-consistent chain-of-thouge distillation, ArXiv preprint, abs/2305.01
Wenya Wang. Vivek Srikumar, Hannaneh Hajishirzi, and Noah A. Smith. 2022b. cetboraatoongeneraane commonsense question answering at scale. ArXiv preprint, abs/2299.01232.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoe Le. and Denny Zhou. 2022. Chain thought prompting elicits reasoning in large language models. ArXiv preprint, 2t//22201.1900..
Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User- generated Text, pages 94 94-106, Copenhagen, Den- mark. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Jamie Brew. 2019. Huggingface 's transformers: State-of thee--of-hhe-----eee art natural language processing, ArXiv preprint, abs.1910.
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hanna Hajishirzi. 2023, Fine-grained human feedback gives better rewards for language model training. ArXiv, abs/2366.01693.
W. Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingx- uan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2022. Generate rather than retrieve: Large language models are strong context generators. ArXiv preprint, abs/2209. 10063.
E. 7elikman, Yuhuai Wu, and Noah D. Goodman. 2022. Star: Bootstrapping reasoning with reasoning. ArXiv preprint, abs/2203.1446.
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversarial dataset for grounded commonsense inference. Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 93-104, Brus- sels, Belgium. Association for Computational Lin- guistics.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can chine really finish your sentence? Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 4791-4800, Florence, Italy, Association for Computational Linguisties.

---

A More Method
C More on Results
I/O format of CRYSTAL. The input/output for- mat illustrated in Table is adapted from Khashabi et al. (2020), based on which we made the follow- ing changes:
Table 11 reports the detailed instructions for the human evaluation.
D More on Related Work
The input is appended with a marker that in- dicates which mode of operation is desired. In knowledge introspection this marker is "Knowledge:", and in knowledge-groundes reasoning it is "Answer:".
Table 12 compares CRYSTAL with existing meth- ods for knowledgg-augmented commonsense rea- soning.
The output is the letter for the predicted an- swer choice, not the actual content of an an- swer choice.
The input and output text are not lowercased.
B More on Experimental Setup
Table 9 shows the datasets we use for training and evaluation, along with their citations. Table 10 reports the hyperparameters.
Abbr.
Name
Citation
Link
TRAINING + EVALUATION (SEEN)
OBQA
OpenBoukQA
Mihaylov al. (2018)
ARC
ARC (casy)
Clark et al. (2018)
ARC h
ARC (hard)
Clark et al. (2018)
A12Sci e
A12 Science (elem)
Clark et al. (2018)
A12Sci m
A12 Science (middle)
Clark et al. (2018)
CSQA
CommonsenseeQA
Talmor et ul. (2019)
QASC
QASC
Khot et al. (2020)
http:://ggttuuu..cco//llalaa////
PIQA
Physical IQA
Bisk et (2020)
STQA
Social IQA
Sap et (2019)
nttas://ggttuul.cco///leeem/////
WG
Winogrande
Sakaguchi et (2020)
https///Ettuub.cco///lleee//////
EVALUATION (UNSBEN)
C2S
Com2Sense (paired)
Singh et al. (2021)
https://githubb.co///lug..d
master/data
SciQ
SciQ
Welbi al. (2017)
https://allenar
QuaRel
QuaRel
Tafjord al. (2019a)
QuaRT>
QuaRT>
Tafford al. (2019h)
https://allenai.cc wwtp://a////aaaaaaaeeeeeeeaaaaee
CycIC
CycIC (me)
https://leaderbbard, aawpaadddlllllaaaaaaaaaa allenai. o
subnissions/get-stad
ComVE
ComVE (task A)
Wang et al. (2020)
SemFFva22200-Task4- Comeoa-oooooosseeeeeeeeeee Validation-and Explanation
WSC
WSC COPA
Levesque et al. (2011)
ht http:://huuginggf http:://wuuginoaaee.ee//ttttttts co/dataseess/winggrad...
COPA
Gordon et al. (2012)
ht https://huggingfan http://aa...c.atatttteeee/eeeeee
NumerSense
NumerSense
Lin (2020)
dataa http:://wugggnnii...ccaaaaattstp
PROST SWAG
PROST SWAG
Arocu-Ouellette et (2021)
Zellers al. (2018)
data data https://ggthhcccee/aasssss/////s
HellaSwag
HellaSwag
Zellers et al. (2019)
CODAH
CODAH
Chen et al. (2019)
data
SCT oNLI
Story Cloze Test
Mostafazudeh et al. (2016)
http:://c...chaseeeeee///brrrrrs edu/nlp/rocstories/
https://cs.rrccesteee
oNLI
Bhagavatula et al. (2020)
https://leaderbont allenai. alllenaa.orr/mmml/ssuuiiissiiioo get-started
Table Dataset details. We show the link from which we retrieved each dataset.

---

Symbol
Value
Description
SHARED HYPERPARAMETERS
1.q
256
Max number of tokens question (including choices). number tokens knowledge.
LK
32
LA
2
Max number tokens answer.
GETTING SILVER KNOWLEDGE FEW-SHOT GPT-3
M
20
Number of knowledge statements sample from GPT-3, per question. Parameter for nucleus sampling from GPT-3.
0.5
STAGE I: IMITATION LEARNING
B
64
Batch size for training.
50,000
Total number of training iterations.
Sox
500
Number of iterations for knowledge introspection in each interleaving cycle.
SOKA
500
Number of iterations for knowledge-grounddd reasoning in each interleaving cycle. Learning rate of Adam optimizer.
X 10
STAGE II: REINFORCEMENT LEARNING
1.0
Weight value model
PPO.
0.2
Weight entropy bonus term reward.
1.0
Discount factor for rewards.
0.95
Parumeter advantage estimation.
0.2
Clipping range
clipped surrogate objective.
0.7
Temperature for knowledge sampling in PPO training.
2M
Total number training episodes.
B
64
Batch size for training.
S
31,250
Total number of training iterations.
4
Number of PPO update steps in each iteration for knowledge introspection.
SpPO
500
Number of iterations for knowledge introspection in each interleaving cycle.
SOKA
500
Number of iterations for knowledge-grounee grounddr reasoning in each interlcaving cycle. Learning rate of Adam optimizer (for CRYSTAL-large).
x 10
10
(for CRYSTAL--3 and -11b).
INFERENCE
M
10
Number of knowledge statements to sample from CRYSTAL, question. Parameter for nucleus sampling from CRYSTAL
0.5
Table 10: Hyperparameter settings.

---

Label
Description
Example
Support
The knowledge be part non-trivial reasoning chain that supports the pre- dicted answer.
Question: Who watches a play auditorium? (A)
building (B) crowd city (D) group (E) high school Knowledge: Audiences watch plays auditoriums. Prediction: (B)
Trivial
The knowledge is trivial paraphrase of the question and the predicted answer.
Question: An alpha particle, which emitted during alpha decay, consists of two protons and what else? 'n
(A) Two neutrons (B) nuclei (C) positrons (D)
electrons
Knowledge: Alpha particles compased of two pro-
and neutrons.
Prediction: (A)
Repeat
The knowledge is repetition of known informa- tion given in the question.
Question: The movement of crustal plates results from cireulating currents material beneath the crust of Earth. Which best describes the material which moves the crustal plates? in (A) hot water (B) molten rock (C) liquid metal (D) solid iron
Knowledge: The movement of crustal plates is caused circulating currents in material beneath the crust of Earth.
Prediction: (B)
Related
The knowledge is topi- cally related the question and/or the choices, but not be part of reasoning chain support refute any of the choices.
Question: How are the particles in block of iron af- fected when the block is melted? \n (A) The particles gain mass. (B) The particles contain less energy. (C) The particles move more rapidly. (D) The particles increase in volume.
Knowledge: Iron particles affected by heat.
Prediction: (C)
Unrelated
The knowledge is unrelated to the question.
Contradict
The knowledge can be part of a reasoning chain that refutes the predicted an- swer, or supports a different choice.
Question: I need what to calculate the length from my big toe to my little toe? In (A) Calculator (B) Tape
Measure (C) Graph (D) A Microscope
Knowledge: A calculator is used to calculate lengths. Prediction: (B)
Table 11: Instruction for the human evaluation.
Method
Citation
Unified model
KG =>
KR
KR => KG
DynaGen
Bosselut et al. (2021)
X
X
CAGE
Rajani et al. (2019)
/
X
XXX
ST-GS
Latcinnik and Berant (2020)
/
Self-talk
Shwartz al. (2020)
VIX
X
Contrastive Expl.
Paranjape et al. (2021)
/
X
GKP
Liu al. (2022b)
JI
X
X
X
DREAM
Gu al. (2022)
X
X
X
X
Rainier
Liu et al. (2022a)
X
ALEAP
Wang et al. (2022b)
X
X
PINTO
Wang et al. (2022a)
AXXA
GenRead
Yu et al. (2022)
DSP
Li et al. (2023b)
X
CoT
Wei et al. (2022)
X
X
CRYSTAL
ours
Table 12: Comparison of existing knowledge-augmeent reasoning methods. Unified model: the method employs
unified model (rather than separate) for knowledge generation and knowledge-grounded reasoning. KG => KR: if the mnowledge-grounded reasoning is trained to adapt to knowledge generation. KR => KG: if the knowledge generation trained to adapt to knowledge-groundde reasoning.


Reviews:
[{"review_id": "q9Q6BU8yWh", "reviewer": "Reviewer_PuFC", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the insightful feedback! We really appreciate your recognition of the clarity and detailedness of our presentation.\n\nBelow is our response to your concerns:\n1. **The “Direct QA” baseline:** As we describe in Sec 3 (Line 368), the “Direct QA” baseline is trained on the same datasets as Crystal, using a standard QA objective. We would like to additionally clarify that the base model is also the pretrained T5 models, of sizes ranging from T5-large to T5-11b, which is identical to the base model for Crystal. We compare the “Direct QA” baseline with our Crystal model with the same sizes (e.g., large vs large, 11b vs 11b), as shown in Table 2. We apologize for the missing details and we will add this to our next paper update.\n    * The QA objective used in “Direct QA” is formally: $\\mathcal{L}_\\text{QA} = -\\log p(a^* | q)$\n2. **SOTA performance**: Thanks for the suggestion, we surveyed the SOTA methods (w/o retrieval, for fair comparison with Crystal) and as below we report on seven datasets whose leaderboards are publicly available. On these datasets, our model Crystal is very close to the average SOTA accuracy (<1% gap, with new SOTA on 2 datasets). Note that these SOTA methods are good on different datasets, whereas Crystal is a single model with strong performance on all these benchmarks.\n| Model | Avg | OBQA | ARC_h | CSQA | QASC | PIQA | SIQA | WG |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Direct QA | 81.22 | 80.00 | 72.91 | 81.98 | 78.29 | 88.36 | 78.45 | 88.56 |\n| Crystal | 83.43 | 85.40 | 73.24 | **82.31** | **81.97** | 88.08 | 82.24 | 90.77 |\n| SOTA (w/o retrieval) | **84.00** | **87.80** | **81.14** | 82.20 | 72.28 | **90.13** | **83.15** | **91.28** |\n3. **Evaluation details:** Aside from the insufficient description of the “Direct QA” baseline, we have provided comprehensive details about our experiments and evaluation in our submission. Please find additional details in the Appendix, where Appendix B documents the datasets we use and model hyperparameters, and Appendix C offers method ablations as well as more details on human evaluation. If there are any additional details that are missing, please definitely let us know and we are delighted to add those to the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Interesting problem\n- Solid results", "reasons_to_reject": "- Missing details in the evaluation\n- Unclear baseline comparison", "questions_to_authors": "", "paper_topic_and_main_contributions": "The research paper addresses the challenging problem of commonsense reasoning by proposing a novel solution called CRYSTAL, an introspective commonsense reasoner. CRYSTAL employs a two-step approach, where it first introspects knowledge statements relevant to a given question and then utilizes this knowledge to provide an answer. The introspection and reasoning processes in CRYSTAL are refined using reinforcement learning techniques.\n\nThe evaluation of CRYSTAL's performance consists of two main components. Firstly, the authors present the main results in Tables 2 and 3, which demonstrate that CRYSTAL outperforms both the standard supervised fine-tuning method (referred to as DirectQA) and chain-of-thought distilled methods across a diverse set of commonsense QA tasks. Secondly, the authors assess the quality of the introspected knowledge by obtaining expert annotations. They show that the knowledge generated by CRYSTAL proves to be valuable for the reasoning process, leading to correct answers in a significant proportion of cases. Specifically, in 34% of instances, the generated knowledge directly supports the answer, further validating the effectiveness of CRYSTAL's knowledge introspection process.\n\nThe paper contributes to the existing body of research on commonsense reasoning, presenting clear and well-supported claims. The technical details are extensively presented, enabling a comprehensive understanding of the proposed method.\n\nHowever, one major concern with the paper lies in the evaluation section, specifically related to the DirectQA baseline model. Despite multiple readings of Section 3, the specific model used as DirectQA is not explicitly specified. It is crucial to have a meaningful baseline, ideally a state-of-the-art model from the leaderboard, to enable readers to draw informed conclusions regarding CRYSTAL's strengths and weaknesses in comparison. While it is not expected for the CRYSTAL to surpass the state-of-the-art performance on all tasks, reporting a well-defined baseline would aid in better understanding the advancements achieved by CRYSTAL in the context of the broader field of commonsense reasoning. This issue significantly affects my score for this paper, however, I am open to improving my score upon further clarification from the authors during rebuttal.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "v2oDttH5Fe", "reviewer": "Reviewer_Mz6D", "summary": "", "questions": "1. It is unclear whether there are any baselines where the knowledge introspection is trained on gold-set rationales (some datasets do have them, OpenBookQA for example). From what I could notice, silver set rationales from GPT-3 Does the quality of these reference rationales during training of the initial introspection model affect the eventual ceiling of performance that is reachable by Crystal? \n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the insightful feedback! We really appreciate your recognition of the thoroughness in our experiments and the value in our human evaluation.\n\nBelow is our response to your concerns:\n1. **Retrieval-augmented baselines:** In this paper we did not explicitly compare with knowledge-retrieval baselines because our method is most closely related to fully self-contained neural reasoners (i.e., no external sources at inference time). Prior work (Liu et al., 2022; Yu et al., 2022) have found that the performance of knowledge-retrieval methods can vary greatly with the quality of knowledge base, and model-generated knowledge is a more robust and scalable way to do knowledge-augmented reasoning.\n2. **CoT baselines:** Thanks for the suggestion, we will add the standard CoT baseline in our next paper update. For your reference, we have evaluated the largest GPT-3.5 under the zero-shot setting (no CoT), and it underperforms our Crystal model:\n| Model | Size | Avg acc on seen benchmarks |\n| --- | --- | --- |\n| Crystal-11b | 11B | 84.58 |\n| text-davinci-003 | 175B | 79.03 |\n\n    * (With CoT we expect GPT-3.5 would perform better than this baseline.)\n3. **Training on gold rationales:** In early experiments we tried replacing GPT3-generated silver knowledge with gold annotated rationales, and it did not lead to improved performance. Specifically, when replicating the supervised model in Rainier, we use the annotated rationales in ECQA (for CommonsenseQA) and QASC, and it did not improve the usefulness of model-generated knowledge over the original supervised model. In addition, gold annotated rationales do not exist for many other datasets we are interested in (e.g., Physical IQA, Social IQA), while GPT3-generated silver knowledge offers more flexibility so we can apply the knowledge introspection paradigm to more tasks.\n4. **Missing references:** Thanks for sharing the related concurrent work, we provide contrast with these paradigms as follows (and will add to our next paper update):\n    * **Self-Refine:** It shares similarities with Crystal in that both improve text generation from the model’s own feedback. The difference is that Self-Refine is a purely inference-time method and relies on the emergent behavior of LLMs, whereas Crystal improves through reinforcement learning and can be applied to smaller LMs to achieve higher performance than larger LLMs.\n    * **RL4F:** It uses reinforcement learning to optimize a critique model so that the generated critique can most effectively correct the outputs of a frozen task model. In some sense RL4F is more similar to Rainier, if we draw an analogy between RL4F’s “critique” and Rainier’s “knowledge”, and between RL4F’s “task model” and Rainier’s “QA model”. Contrast to RL4F and Rainier, in Crystal we use a single model for the task and critique, and these two aspects are both trained to co-adapt to each other.\n\nReferences\n* Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Peter West, Ronan Le Bras, Yejin Choi, and Hannaneh Hajishirzi. 2022b. Generated knowledge prompting for commonsense reasoning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3154–3169, Dublin, Ireland. Association for Computational Linguistics.\n* W. Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2022. Generate rather than retrieve: Large language models are strong context generators. ArXiv preprint, abs/2209.10063.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is extremely thorough with its experiments, including the diversity in datasets. The motivation for interdependence on the knowledge introspection with the answer generation is very relevant to current research directions on faithfulness of generated explanations with their answers. The human evaluations also further reinforce the association between the generated introspections with the label.", "reasons_to_reject": "RR1 Comparisons with some more fundamental baselines - While the paper compares Crystal to CoT distilled approaches, another line of close work in the introspect then predict paradigm are retrieval augmented generators. Furthermore, it would be good to compare Crystal with standard CoT approaches.", "questions_to_authors": "1. It is unclear whether there are any baselines where the knowledge introspection is trained on gold-set rationales (some datasets do have them, OpenBookQA for example). From what I could notice, silver set rationales from GPT-3 Does the quality of these reference rationales during training of the initial introspection model affect the eventual ceiling of performance that is reachable by Crystal? \n", "paper_topic_and_main_contributions": "This paper proposes Crystal, a method that first introspects knowledge, and then generates an answer that is conditioned on the introspected knowledge. This introspected knowledge is then treated as \"explanations\" for the given prediction. Feedback on generating good quality explanations conditions on the answer and vice version is setup using an RL-like paradigm (similar to the Rainier work).", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Some of these works maybe concurrent to this work, but it would be good to contrast crystal with Self-Refine (concurrent) and RL4F like paradigms \n1. Afra Feyza Akyurek, Ekin Akyurek, Ashwin Kalyan, Peter Clark, Derry Tanti Wijaya, and Niket Tandon. 2023. RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7716–7733, Toronto, Canada. Association for Computational Linguistics.\n2. Self-Refine: Iterative Refinement with Self-Feedback. Madaan et. al, 2023"}}, {"review_id": "6id96ZqkpM", "reviewer": "Reviewer_s1EV", "summary": "", "questions": "1.\tHow does your method improve the usefulness of the knowledge compared to Rainier? Can you also conduct the human evaluation on the generated knowledge from Rainier?\n2.\tHow does training the model with standard supervision guarantee the model would leverage the knowledge for reasoning?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the insightful feedback! We really appreciate your recognition of the importance of the research problem we study.\n\nBelow is our response to your concerns:\n1. **Leveraging knowledge:** We encourage the model to leverage the introspected knowledge through our training objective design:\n    * More specifically, in training stage I, the model is trained on two losses: a knowledge introspection loss $L_\\text{QK} = -\\log p(k | q)$ and a knowledge-grounded reasoning loss $L_\\text{QKA} = -\\log p(a^* | q, k)$ (Sec 2.5). Hypothetically we could introduce a third loss: a straightforward QA loss, $L_\\text{QA} = -\\log p(a^* | q)$, but this would allow the model to take shortcuts and ignore the knowledge. We ensure knowledge is leveraged by **not** including this QA loss. We apply the same knowledge-grounded reasoning loss $L_\\text{QKA}$ in stage II training.\n    * In early experiments, we found that including this QA loss hurts performance, probably by allowing the model to take shortcuts around the knowledge. We conducted this experiment with the Stage I trained model based on T5-large, so the baseline we compare with is Appendix Table 10 Line 2. Including this QA loss reduces accuracy from 66.16 down to 65.36, which motivated us to not include the QA loss. We will report this result in the updated paper.\n    * If we only use the QA loss, this is equivalent to our “Direct QA” baseline, and as reported in Table 2 & 3, this baseline underperforms our Crystal model. So to achieve better QA performance, it is in the best interest of the model to leverage the introspected knowledge.\n    * We do not enforce the knowledge to be leveraged when the model makes final predictions. If the model believes that the knowledge is wrong and implies a wrong answer, it is welcome to contradict the knowledge. In fact, as shown in our human evaluation (Sec 4.2), there is a small percentage of cases (8%) where the predicted answer contradicts the knowledge. However, in the majority of cases (53%) we found that the knowledge is directly indicative of the prediction.\n2. **Novelty:** While we inherit the RL formulation and algorithm from the Rainier paper, our paper carries substantial novelty. (1) We design Crystal as a self-contained introspective reasoning model that learns from its self-feedback using reinforcement learning, which is novel compared to Rainier and other existing literature. (2) We show that it is viable and more efficient to use a single model as policy and feedback provider in PPO. (3) We also propose novel training objectives and optimization schedules to effectively prevent catastrophic forgetting due to this model sharing.\n3. **Comparing usefulness of knowledge:** We did not compare our human evaluation with that reported in Rainier due to the difference in evaluation scheme. We asked if the knowledge is indicative of **the model’s final prediction**, whereas in Rainier the authors asked if the knowledge is indicative of **the correct answer**. We choose our scheme and because this research question is more helpful for understanding the interpretability of reasoning offered by our model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe mutual adaptation of knowledge generation and grounding is an important research problem as it helps reveal the reasoning process of an otherwise blackbox model. \n2.\tThe proposed training objectives and iterative training process are both novel and intuitive.\n3.\tThe work carefully examines the method on a wide range of tasks, demonstrating the generalization of the method.\n", "reasons_to_reject": "1.\tIt is unclear how the standard supervision would ensure the model indeed learns to leverage the knowledge instead of ignoring it.\n2.\tIt seems the RL part of the work is largely based on the cited work Rainier, which limits the novelty of the work.\n", "questions_to_authors": "1.\tHow does your method improve the usefulness of the knowledge compared to Rainier? Can you also conduct the human evaluation on the generated knowledge from Rainier?\n2.\tHow does training the model with standard supervision guarantee the model would leverage the knowledge for reasoning?\n", "paper_topic_and_main_contributions": "The paper aims to solve commonsense reasoning tasks, where additional knowledge is required. They adopt a unified model that operates on knowledge introspection and knowledge-grounded reasoning modes. To ensure the generated knowledge is effective, they adopt RL to encourage the model to generate knowledge statements that help the model predict the correct answers. To ensure the knowledge is leveraged for reasoning, they train the model to maximize the probabilities of the correct answers when knowledge is provided. Two training stages are conducted iteratively. Experiments demonstrate the effectiveness of the proposed model in achieving better results than the baselines in both seen and unseen tasks. The human evaluation shows the informativeness of the generated knowledge.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method Crystal that introspects knowledge statements related to a given commonsense question and uses this knowledge to provide an answer.  Crystal considers two steps: (a) introspective reasoning and (b) knowledge introspection. The idea of knowledge introspection is similar to Rainier's framework.  Crystal extends this idea to a unified introspective reasoning model. 


Pros: 
- The paper is well-written and thorough with every detail. 
- The idea of optimising the knowledge generator and reasoner in an interleaved fashion is interesting. 
- The qualitative and quantitative analysis is solid. 


Cons: 
- The main concern is that the baseline is weak. Comparison with only DirectQA does not give the full picture since it doesn’t use any knowledge. An ideal comparison should be:
 L_QA = -logp(a*|q,k) where k is the silver knowledge generated by GPT-3 (Table 3). This will show how the overall model improved knowledge quality and reasoning capability.  In Lines 288-289, it is unclear which version of the GPT-3 model is used to generate silver knowledge. Is it the same knowledge as Rainier's paper?  It will make the results in Table 2 more reliable. 

Finally, The AreaChair appreciate the authors' response to clarify the doubts of the reviewers.
Decision: Accept-Main