Tailoring Self-Rationalizers with Multi-Reward Distillation
Authors: Sahana Ramnath
Brihi Joshi
Skyler Hallinan
Ximing Lu
Liunian Harold Li
Aaron Chan
Jack Hessel
Yejin Choi
Xiang Ren
Venue: ICLR 2024 poster (2024)
Keywords: large language models
rationalization
explanation generation
explainability
rationale generation
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: Multi-reward conditioned algorithm that makes small LMs stronger rationalizers.

Abstract: Large language models (LMs) are capable of generating free-text rationales to aid question answering. However, prior work 1) suggests that useful self-rationalization is emergent only at significant scales (e.g., 175B parameter GPT-3); and 2) focuses largely on downstream performance, ignoring the semantics of the rationales themselves, e.g., are they faithful, true, and helpful for humans? In this work, we enable small-scale LMs (∼200x smaller than GPT-3) to generate rationales that not only improve downstream task performance, but are also more plausible, consistent, and diverse, assessed both by automatic and human evaluation. Our method, MaRio (Multi-rewArd RatIOnalization), is a multi-reward conditioned self-rationalization algorithm that optimizes multiple distinct properties like plausibility, diversity and consistency. Results on three difficult question-answering datasets StrategyQA, QuaRel and OpenBookQA show that not only does MaRio improve task accuracy, but it also improves the self-rationalization quality of small LMs across the aforementioned axes better than a supervised fine-tuning (SFT) baseline. Extensive human evaluations confirm that MaRio rationales are preferred vs. SFT rationales, as well as qualitative improvements in plausibility and consistency.

Published as a conference paper at ICLR 2024
TAILORING SELF-RATIONALIZERS WITH MULTI- REWARD DISTILLATION
Sahana Ramnath Brihi Joshi Skyler Hallinan Ximing Lu Liunian Harold Li
Aaron Chan Jack Hessel', Yejin Choi Xiang Ren
University of Southern California "University of Washington
University of California Los Angeles 9 "Allen Institute for Artificial Intelligence
sramnathe whh.sccc.ii
ABSTRACT
Large language models (LMs) are capable of generating free-text rationales aid question answering. However, prior work 1) suggests that useful self- rationalization is emergent only at significant scales (e.g.. 175B parameter GPT- 3); and 2) focuses largely on downstream performance, ignoring the semantics of the rationales themselves, c.g., they faithful, true, and helpful for humans? In this work, we enable small-scale LMs (~200x smaller than GPT-3) to generate rationales that not only improve downstream task performance, but also more plausible, consistent, and diverse, assessed both by automatic and human evalu- ation. Our method, MARIO Multi-rewArd RatIOnalization) is multi-reward conditioned self-rationalization algorithm that optimizes multiple distinct proper- ties like plausibility, diversity and consistency. Results five difficult question- answering datasets StrategyQA, QuaRel, OpenBookQA. NumerSense and QASC show that only does MARIO improve task accuracy, but it also improves the self-rationalizatiin quality of small LMs across the aforementioned axes better than supervised fine-tuning (SFT) baseline. Extensive human evaluations con- firm that MARIO rationales are preferred vs. SFT rationales, well as qualitative improvements in plausibility and consistency
INTRODUCTION
Quention Malletii conting bredd withi brudd Info whichhereeeee smooth cut, ahilecortin olk with brsu enore rough cu: Thiemens bas lncerrez Alhnod cake
Existing
What we do
Rationalizers
shardh
GETA Ratianda When somnethnng has les rederanss hi carke eun chnough. 1hos the bread los moranee makingit cader so can with dae bread keif.
Malto Rationale: Les cesiitance impless eate in through So, thrungh braad.
Anrns:
Famss
Rationale
Propertics
CPE3
(2) Focus only on task performance
Randslin
anplaa cotting Einsugh. the has smzaeih
(1) Large Size
Divssyy
Figure 1: Our proposed approach, MARIO. While existing self-rationalizing pipelines require exorbitantly large LMs that are used to primarily improve task performance, MARIO is small LM that is initially distilled from rationales generated by GPT-3, following by multi-reward training that improves its rationale quality w.r.f three properties: plausibility, diversity and consistency.
In recent years, there has been surge of interest in self-rationalizing LMs. LMs that generate fluent, human-like, free-text freee--tee rationales that can explain their decision (Wiegreffe et al. 2021). Early approaches in self-rate ratiinattoali involved collecting human-written gold rationales and using them as supervision for training LMS (Wiegreffe et al. 2021 Narang et al. 2020). Now, with the advent
1k ab. usc. usw.uuc.uuuuu wds..edd/aaRoo

---

Published as a conference paper at ICLR 2024
of large LMs, thain-of-thought prompting (Wei al. 2022 Marasovic et al. 2022; Kojima al. 2022) has revolutionizzd the landscape of calff-ratioaazzationn now, with just a few well-designed prompts and demonstrations, LMs can generate explanations for their predictions. The presence of rationales can make LMs both more interpretable (Lertvittayakumjom & Toni 2021) and more usable (Joshi et al. 2023) from the perspective of users.
However, prior work in self- self-rationalizzaiinn has largely overlooked the quality of generated rationales themselves; instead, their utility justified by measuring downstream task performance (Wei al.) 2022 Zelikman et al. 2022). This is particularly problematic, as downstream models and users may use these rationales as justifications for the predicted answer, which can further propagate these negative quality outcomes (Atanasova et al. [2023] Joshi et al.) [023 HHvyy & Prabhumoye| [2021). Furthermore, it observed that rationalization comparable to human quality is only observed with at significant LMM parameter scales (~100B or more) (Wei et al. (022). Despite some recent interest in using smaller LMs for rationalization (Chen et al., 2023b), is still unelear if smaller LMS can be used to generate similarly high-quality rationales.
In this work, we propose MARIO, a method that focuses on tailoring small-sized LMs (< 1B param- eters) to be strong rationalizers both in terms of improved downstream performance, and in terms of desireable properties of the rationales themselves. Instead of relying on human rationale labelling (Wiegreffe et al. 2021). MARIO considers setting where small LM only has access to rewards that measures factors underlying rationale quality, e.g. a trained LM that judges the plausibility of rationale and provides numerical score. MARIO first starts with training a small LM to self-
rationalize, with the help of GPT-3 (Brown et al. 2020p (TEXT-DAVINCI-00S as initial supervision, which are shown to be of higher quality [un et al. (2022). It then casts the problem into a multi-reward conditioned rationale generation problem, where the LM is optimized to maximize quality rewards. In order to achieve this, MARIO extends QUARK proposed by Lu itt al. (2022) to a multi-reward setup, where generations from an LM are binned according reward values; the LM learns distributions conditioned 'control-tokens' corresponding every reward and high-quality generations can be obtained via conditioning on the highest-reward token.
generated rationales
We determine that high-quality rationales should have three necessary properties: plausibility (makes logical sense), diversity (is not repetitive) and consistency (supports the correct answer for the instance). Generated rationales' rewards are assessed through automated metrics for each of the three quality properties. We then evaluate MARIO on three question- -anterrnnnn datasets, and ob- serve that small LMs like T5-LARGE can be effectively trained generate rationales that satisfy all of the quality requirements, while also leading to improvements in task performance over supervised fine-tuned self-rationalizers (SFT). Via human evaluation, we also observe that rationales generated by MARIO are more preferred over those generated by SFT, across all datasets.
We note that tailoring small LMs with multiple quality rewards is a challenging task. Some of these issues include finding high-quality, stable rewards that can be effectively incorporated in self- rationalizing pipeline. We also observed that a lot of additional desirable properties rationales (like factuality and completeness) do not have reliable automated rewards. Furthermore, improving task accuracy (which is the primary goal while generating rationales for a lot of these tasks) is challenging in a newar setup, and show that adding task accuracy as additional reward term leads to the best configuration of MARIO. By using small LMs to generate high-quality rationales that are also supported by human evaluations, we believe findings can help guide future work in efficient, real-world situated methods in rationale generation and evaluation.
2
Throughout this work, we refer to self- relf-rationalizers as LMs that are trained or prompted to specif- ically generate free-text rationales, along with their predictions. These free-text rationales are treated explanations for their predictions. For the purpose of our experiments, we explore self- rationalization on the question answering (QA) task. Specifically, given a question, the LM must first generate a free-text rationale that explains the LM's reasoning process, followed by an answer to the given question. Table shows examples of inputs and outputs by these self-rationalizznng LMs for five QA datasets: STRATEGYQA (Geva et al. 202I). QUAREL (Tafjord et al. 2019), OPEN-
note that whenever we mention GPT-3 in this work, we are referring TEXT-DAVINCI-003
2

---

Published as a conference paper at ICLR 2024
Table 1: Sample Inputs and Outputs for Rati-Rationliinng LMs. We use an I-RO setting for all our experiments, This table shows one example each from the training set of STRATEGYQA, OPENBOOKQA, QUAREL, NUMERSENSE and QASC. The rationales shown here are the ones sampled from GPT-3.
INPUT (D: Could somcone in Tokyo take taxi the The Metropolitan Muscum of Art?
StrAtegYQA
OUTPUT (RO): The Metropolitan Museum of Art is in New York City, USA. Tokyo, Japan over 6,000 miles away. So the
INPUT Our only star provides with energy that temperary inferior (c) expensive reusable OUTPUT (RO): The energy from the sun renewable and reusable, So the answer is (d).
OPENBOORQA
INPUT (I) Cutting hread with bread knife creates smooth cut, while cutting cake with hread knife creates
rough cut. This means that the _... has less resistance (A) bread (B) cake
QUAREL
OUTPUT (RO): When something bas less resistance.
casier cut through. Thus, the bread has less resis-
fance, making it easier to cut with the bread knife, So the answer is (A).
INPUT (I) Fungi reproduce in <nask> ways. (A) (B) (C) one (D) (E) three (F) four (G) five (II) six (I) seven (J) eight (K) nine (I.) ten
NumerSENSE
OUTPUT (RO): Fungi reproduce sexual
reproduction.
answer (D).
INPUT (1): Bees are necessary to (A) haploid plants (B) genetic diversity (C) spread flower seeds (D) prevent
evalution (E) Repreduction (F) eat weeds (G) chew floerss (II) important habitals
QASC
OUTPUT (RO): Rees are necessary to sprend flower seeds. Bees pollinate flowers, which helps the flowers re- produce spread their soeds, So the answer is (C).
BOOKQA (Mihaylov et al. 2018), NUMERSENSE Lin et al. (020)) and QASC Khot et al. (2020) These datasets were chosen over others which have existing human written rationales because all of them require certain level of implicit or logical reasoning in order to arrive at the answer. As we depict in the examples, we follow the I-RO format (Wiegreffe et al. 202T), wherein the input to the LM is the question, and the output is the joint generation of the rationale and the predicted answer.
In order to determine whether these generated rationales are of good quality, we focus on three properties that are necessary for any rationale to have, agnostic of the task it is meant for. First, we note that a rationale should be plausible. We define plausibility as the rationale making sense on its own whether it be common, logical factual sense depending on the dataset at hand. For example, if a rationale states 'Cows can fly', it is not plausible. Next, we identify that a rationale should be diverse, where the rationale is clean and not repetitive. Lastly, should be consistent with the gold label for the input. Consistency is important to ensure that a rationale does not spew irrelevant information, and that it supports the gold answer. Furthermore, we focus on consistency with respect to the gold label, as mislcading rationales are unhelpful as both LM justifications, and for human utility (Joshi et al. 2023). We formalise these properties as rewards in $4 and while these are necessary properties for any rationale, we also discuss other good-to-have properties in $
note that rationale
All of these properties are agnostic of the actual prediction made by the LMM Since our self- rationalization setup generates a rationale first, followed by its prediction, we aim to generate ratio- nales with good quality, which should ideally improve the answer generated by the LM. Therefore, we focus on improving self-rationalizztionn along these three properties, as well as on task accuracy. Along with the above rationale properties, we also consider task correctness as a necessary property of rationales, that they should try to improve over as a byproduct.
MARIO: OPTIMIZING FOR MULTIPLE REWARDS
To improve an LMs' rationalization across multiple properties, we leverage QUARK al. 2022), reinforcement learning-like framework effective tasks such as unlearning toxicity in genera- tions. We propose Multi-rewAAd RatIOnalization (MARIO), an extension of QUARK to multiple rewards concurrently. We further propose two variants of MARIO: CLASSIC and ADDITIVE, that explore different ways of using QUARK for in multi-reward setup. Figure 2 shows running example of MARIO. Appendix B
shows a second, more technical illustration of the same.
QUARK QUARK Lu et al. 2022) is reinforcement learning- like algorithm that trains LMs using unique control tokens prepended to the generated text. The QUARK algorithm works iteratively:
3

---

Published as a conference paper at ICLR 2024
Single Reward
Question: Malina is susting bread with bread knife which crestes smonth cut, while cunting cake wish brcad knili ewarce nough This means the has resisance (A) bread cake
[Plusihitt]]
[Pundhiin]
Reward Canonical
[Pansiiggl
[Doanty]
[Pissibikayl
Dezrasyl
[Comismncy]
[Pansbite]
[Divntyy
[Contnmog]
Roward-baaed updating
OPFF
Prefised Reward Tokens
Rationale
Ourput
Whazn womsihhng has less resieracee, easier to through. Thas, the beead resicance, making cuiet to cut with the bocad Lnik.
Kational:
Less relistance implle case curting shnough bread has smooth zut as is lese resisant.
[[]
Sathe
(A)
Divtilltion
Figure 2: MARTO pipeline. MARIO uses rationales generated by larger LM like GPT-3 as initial supervision, and uses rewards corresponding to three rationale properties: PLAUSIBIL.TYY DIVERSITY and CONSISTENCY, to improve self-rationalization of smaller LMs like T5-LARGE.
(1) sampling the a pre-trained LM to generate more training data, (2) scoring the generated data using chosen reward metric, and (3) using instance-level scores sort and bin the data into fixed number bins, each which correspond a unique control token. During training, the LM learns to associate each control token with the quality (as determined by the reward metric) of the data it assigned to. During inference, in order to obtain the best quality generations, QUARK samples the LM using the control token corresponding to the highest reward measure. We provide more detailed description of how QUARK works in Appendix B
3.1 CLASSIC MARIO
Since QUARK designed only for single reward optimizations, what if we are interested in improv- ing the rationales along multiple rewards? In order to do this, we first propose a direct extension to the QUARK algorithm: instead assigning each instance just one unique control token (which corresponds to one specific property), now assign each instance multiple unique control tokens at once; now, each control token corresponds to a different property we want to train the LM on. We call this the CLASSIC MARIO. The order in which properties represented in this method is design decision that we can choose, and we expand on this further in Appendix C
3.2 ADDITIVE MARIO
now propose a step-by-step multi-reward QUARK: instead of training the LM on all the prop- erties at the same time, we introduce them into training pipeline additively, in predefined order of properties. For example, say that we have properties P,, Pz, Ps. and that we want the LM to focus on P3 first, before moving to P1 and then later Pa. In this method, control tokens as in CLASSIC MARIO, but, we introduce each set of tokens into the training pipeline successively. For example, we train the LM with only P3 for the first steps, then train the LM on Ps and P1 from the f-th step to the 2f-th step, and then from the 2f-th step onwards, we train the LM on Ps, and Pz. We call this method the ADDITIVE MARIO. Again, the order in which add the control tokens of different rewards to the training pipeline, and whether each new control token added to the left or right of the existing control tokens are decision choices, and we expand on it further in Appendix C
multiple sets of
4 EXPERIMENTS
4.1 DATASETS
As we mention in Section We conduct experiments on 5 QA datasets: STRATEGYQA, QUAREL, OPENBOOKQA, NUMERSENSFand QASC; the task is to generate rationale followed by the pre-

---

Published as a conference paper at ICLR 2024
dicted answer. We report details of train, val and test splits in Appendix D We do not require humtn-written gold rationales; we sample GPT-3 for silver rationales for supervision. We use chain- of of-thought prompts from prior works on these datasets (refer Appendix [ for the full prompts) and sample 5 rationale generations for each training set instance with a temperature of 0.9; for super- vision only the instances where the answer predicted by GPT-3 is correct. We these silver rationales in three places: (1) to train SFT, (2) we use SFT as the reference model for the KL divergence loss in MARIO'S training process (Appendix B (3) we also add these silver rationales to the overall data pool of MARIO (Appendix B provides an expanded explanation for the same).
4.2 RATIONALE PROPERTY REWARDS AND TASK CORRECTNESS
We formalize our chosen rationale properties (from $2) with their implementations that we use as rewards within MARIO. Let Q be the input question, R represent the LM's generated rationale, and O and Ô represent the gold answer and the LM's predicted answer respectively. The formulations of each reward are follows:
PLAUSIBILITY via VERA (Liu et al., 2023a): VERA is trained commonsense statement verification T5- 1BB model, that provides numerical score between and 1, indicating the plausibility of declarative sentences.
PLAUSIBILITY R?) VERA(R)
We use the VERA release from HuggingFace
CONSISTENCY via (Wiegreffe et al. 2021): Wiegreffe et al. (2021) provide a framework evaluate the association between rationale and label with the help of two reference LMs that are trained to predict the answer with (Migz) and without (Ma) the rationale in the input. More formally,
()) Prege (0lQ, i Ps. (0IQ)
CONSISTENCY is numerical score between -1 and 1 that indicates the faithfulness of the rationale towards the gold answer, like the implementation by WWegreffe et al. (202T) Hyperparameters and training guidelines for the LMs involved in generating the CONSIS-
TENCY score is in Appendix B
DIVERSITY via n-gram uniqueness in Li et al. 2022) Li et al. 2022) calculate diversity generated text by determining the fraction of unique n- n-grams generated. DIVERSITY is a numerical score between 0 and that indicates the lexical diversity of the rationale; this metric also serves the purpose of ensuring that the rationale does not contain repeated phrases sentences.
unique n-grams( Rt
DIVERSITY(R)+
= n=2 total n-grams( Rt)
TASK-CORRECTNESS. As last reward, add task correctness of the answer that is generated following the rationale. This a binary 0/1 score, referring to the wrong and right predicted answer respectively.
TASK- (OSSECCRRECTN 1[Ō= 0]
We evaluate/report these metrics for all our baselines and experiments. To simplify comparisons, we also report an average normalized relative gain (NRG) (Chan et al. 2022) (Appendix E
4.3 HUMAN PREFERENCE EVALUATION
We first present human preference studies comparing rationales generated by MARIO and the super- vised fine-tuned baseline SFT for all five datasets. For each instance, we ask three distinct annotators from pool of qualified annotators to compare the two rationales across three settings, for given question and correct answer pair: PLAUSIBILITY and CONSISTENCY, which are defined in the
htttss:/hhugginggface.cooo
998/ver9
5

---

Published as a conference paper at ICLR 2024
Table 2: Demonstrative examples for the rationale properties
Malini is cutting bread with bread knife which creates a smooth cut, while cutting cake with a bread knife creates rough cut. This means that the has less resistance (A) bread (B) cake
QUESTION
Less resistance implies that the item would he difficult to cut through. Therefore, cake has less resistance.
PLAUSIBILITY
s Less resistance implies ease in cutting through. So the bread has smooth cut it less resistant.
Less resistance implies case in cutting through. So the cake has smooth cut as il is less resistant.
CONSISTENCY
resistance implies ease in cutting through. the bread has smooth cut less resistant.
Less resistance implies ease ease in in cutting through. Ease in cutting through, Answer bread.
DIVERSITY
1 Less resistance implies ease in cutting through. the bread has smooth cut as is less resistant.
same manner as the rewards, and an overall PREFERENCE rating. PREFERENCE is meant to indi- cate that the annotators pick the rationale that they would find acceptable (Wiegreffe et al. 2022) for the given question. In Figure 3 we plot the % of instances where majority of annotators prefer only MARIO's rationales, only SFT's rationales, both none. We note human annotators prefer MaRIo's s only rationales for 15%, 75.3%, 71.49%, 67 .44% and 66.6% of instances respectively for STRATEGYQA, QUAREI. OPENBOOKKA. SUMERSENNSS and QASC. Human annotators also find MARIO's rationales to be considerably more plausible and consistent than SF We use Ama- zon MTurk for all our human studics, and Appendix provides further details on the same.
StretegyOA
QuaRel
OpenBookQA
NumerSense
QASC
70
66
80-
Sa satorales
80-
Si
30
A R s
-0
20
20
10
Prefernnc
Reurtaiit
Consistncy
neterenca
Plasoiblity
Cenvotency
Prleeence
Plasiiully
Consssennc
Reustbilty Preferanc
Consattonyy
Prederence
Plassibllty
Comaotency
Only
Figure 3: Results of human studies comparing MA RIO with SFT. Here, plot the % of in-
stances in the test set wherein annotators prefer MARIO, SFT, both or none, with respect to PREF- ERENCE, PLAUSIBILITY and CONSISTENCY. We find that human annotators vastly prefer MARIO'S rationales, and also find them to be much more plausible and consistent.
4.4 BASELINES VS. MARIO
All baselines and MARIO are built top of T5-LARGE LMs (0.7B). We present and compare our method with four strong baseline models:
1. Supervised Fine-tuned Self-Rationalizer (SFT): A fine-tuned T5-LARGE, which serves as the supervised learning baseline (we use the training data as described in $4.1). trained to generate rationales and answers.
2. Product Rewards (ProdUcT): A multi-reward baseline where we consolidate the re- wards into single representative metric by taking their product and apply QUARK. Ag-
'We do perform human studies for DIVERSITY and TASK ACCURACY since they are auto- matic/straightforwarri metrics hhtps//www.mmurkccom

---

Published as a conference paper at ICLR 2024
Table 3: Baselines vs. MARIO Results. For each dataset, the best averaged NRG (across TASK ACCURACY, PLAUSIBILITY, DIVERSITY and CONSISTENCY) is highlighted in bold, and each best individual metric is underlined. Cells marked with a shows significant improvement for the corresponding MARIO configuration over SFT (p < 0.05).
Method
Baselines
MARIO
Dataset
Metric
SFT
PRODUCT
FILT-Ace
FILT- ALL
CLASSIC
ADDITIVE
Acc.
57.64
62.01
61.57
61.35
60.26
65.07
Plau.
0.33
0.35
0.34
0.36
0.38
0.39
STRATEGYQA
Div.
0.95
0.92
0.92
0.94
0.95
0.97
Cons.
-0.02
0.00
0.01
0.04"
Avg. NRG
58.66
59.75
59.39
60.34
60.94
63.27
Acc.
76.99
79.53
79.53
76.45
79.89
78.99
Plau.
0.71
0.72
0.71
0.73
0.77
0.75
QUAREL
Div.
0.95
0.95
0.95
0.95
0.97
0.97
Cons.
0.18
0.21
0.20
0.17
0.19
0,20
Avg. NRG
75.50
76.71
76.38
75.74
78.35
77.75
Acc.
63.65
61.65
65.86
56,63
66.06
65.55
Plau.
0.53
0.52
0.55
0.47
0.55
0.55
OPENBOOKQA
Div.
0.98
0.99
0.99
0.99
0.98
Cons. 0.05
0.07
0,08
0.01
0.09
0.09
Avg. NRG
66.79
66.54
68.47
63.28
68.64
68.29
Acc.
46.23
50.75
51.76
46.73
55.28
54.27
Plau.
0.60
0.60
0.61
0.58
0.63
0.63
NUMERSENSE
Div.
1.00
1.00
1.00
1.00
1.00
0.99
Cons. 0.17
0.20
0.21
0.16
0.23
0.23
Avg. NRG
66.18
67.69
68.32
65.68
69.95
69.44
Acc.
58.64
57.88
57.78
57.02
60.15
59.61
Plau.
0.44
0.43
0.39
0.42
0.47
0.47
QASC
Div.
0.96
0.95
0,96
0.96
0.99
0.99
Cons. 0.19
0.17
0.17
0.17
0.19
0.19
Avg. NRG
64.54
63.60
62.82
63.38
66.41
66.28
gregating several rewards into one is common in prior work and is often done through via product (Lu et al. 2023) weighted average (Wu et al. 2023).
3. Filtering rationales that lead correct answers (FILT-ACC): This a variant of STAR ZZe- likman et al., 2022). We iteratively train and sample new training data from T5-LARGE, similar to QUARK, but instead of using any control tokens, we filter out the instances which have the wrong predicted label. We train this model with only cross-entropy loss.
4. Multi-reward variant of FILT-ACC (FILT-ALL): Again, iteratively train and sample new training data from a T5-LARGE, and instead of using control tokens, we filter out the instances which have the wrong predicted label and instances that fall under a specified threshold value for PLAUSIBILITY, DIVERSITY and CONSISTENCY. The threshold value is tuned as hyperparameter. We train this model with only cross-entropy loss.
Table the comparisons between MARIO and the baselines. For all five datasets, note that MARIO the overall best setup as noted by both the individual metrics and the averaged NRG met- ric. ADDITIVE MARIO is found to be the best performing method for STR ATEGYQA, and CLASSIC MARIo is found to be the best method for the other 4 datasets (hyperparameter configurations in Appendix
important to note that not only does the rationale get better (as seen via the ratio- nale metrics), but the task accuracy also shows a marked improvement over the baselines. We show some representative examples of rationales generated by training with MARIO, in comparison with We also release the rationales generated by SET and MARIO
those generated by SFT in Table
com/ dri http:///wwwefeole///////////a/// dewd/bbkkdd wce 8US5y
360 w 11p00p880000

---

Published as a conference paper at ICLR 2024
4.5 REFERENCE LARGE LMS vs. MARIO
We now consider 3 strong reference LLMs that are used in practice for self-rationalization: GPT-3
(175B), FLAN-T5 (Chung et al., 2022) (sizes, L, XL, XXL) and LLAMA (Touvron et al.) 2023) (sizes 7B, 65B); we compare MARIO with them in terms of both average NRG (Figure and in- dividual metric scores (Table TOP. All these LMs apart from FLAN-T5-L are orders of magnitude larger than our T5-LARGE LM trained with MARIO; we include FL.AN-T5-L. in our comparison even though it's of the same size as MARIO because FLAN-T5-LE is cestructionfinettunn shot prompted to generate rationales, with the same set of demonstrations used by other large LMs (shown in Appendix D. Ideally. we want a small-sized LM (for efficiency) that achieves high perfor- mance, which corresponds to the top-left portion of the graph in Figure 4 Hence, to compare two LMs' performance, the LM which is relatively to the left and to the top is practically a better choice. We note that for QUAREL, MARIO results in an LMM that is of very small size (0.7B) but has very high performance, almost equivalent to that of GPT-3. For NUMERSENSF, MARTO beats all models except for FLAN-T5-XXC and GPT-3, and for QASC, MARIO beats all models except for -LAN-T5-XXL, LLAMA-651 and GPT-3. For OPENBO we see that MARIO beats LMs such as FLAN-T5-L, FLAN-T5-X XL and LLAMA-7B For STRATEGYQA we see that our LM beats FLAN-T5-L, while performing only a little worse than FLAN-T5-X5 XL.
and few-
ShahearO4
Cocka
SureeSeene
GACC
ANg. N5C
ANDO MOC
Nettout
ArT NSC
Aurt
AND
U
lop
Figure 4: Reference Large LMs vs. MARIO Results: Here, we show the comparison of Avg.
NRG values w.r. the LM size (in the order of billion parameters) for all the datasets.
5 DISCUSSION
5.1 PROPERTIES AND METRICS
While the properties we explored in this work are necessary for high rationale quality, the question of what are the complete set of properties remains an open problem (Joshi et al. [0233 Wiegreffe et al. 2022] Golovneva et al. 2022). Some recent works on other necessary rationale properties are REV Chen et al. (2023a) (novelty of information, faithfulness towards the predicted label), ROSCOE Golovneva et al. (2022) ReCEval Prasad et al. (2023) (score steps of reasoning), LAS Hase et al. (2020) (faithfulness towards predicted labels) etc. Further, there are also properties which do not have widespread implementations (to the best of our knowledge) such as coattaal--orreeeeess com- pleteness of rationales (existing metries require gold rationales which are not easily available, and which cannot score any alternate reasoning to the answer), etc. As future work, we hope to collect an extended of properties and corresponding metrics, and improve them with MARIO.
5.2
HACKING
As additional experimentation with alternate properties relevant to our chosen QA datasets, we worked on a set of experiments focusing on factual-correctnes, and lexical diversity; specifically for STRATEGYQA which requires historical or factual correctness of the rationale (this different from common-sene or logical correctness measured by PLAUSIBILITY, as explained in VER (Liu ct al., 2023al). We started with a fact verification metric LOREN (Chen et al. 2022)- while effec- live, we couldn't it use this metric in practice since each score prediction required a Web API call, which is inefficient given MARIO's s iterative data generation and scoring. We tried a weaker metric querying the rationale with larger LM, FLAN-T5-XXX and asking if the rationale was factually correct or not (probability of 'yes' under yes/no). We noticed that applying QUARK/MARIO with this metric led to some interesting reward hacking, we show in the first two rows of Table
8

---

Published as a conference paper at ICLR 2024
Trying to improve on just factuality caused the LM to start generating repetitive text, easy way of generating factual statements. When we tried to counter this by training MARIO on factuality and lexical diversity together, the LM started generating incomplete rationales. We further noted that this kind of repetitive generation is observed even in larger LMs which on surface seem much better (as seen in rows 3 and 4 in Table Therefore, we note that selecting strong rewards, well careful qualitative investigation is extremely important to prevent this kind of reward hacking where an increase in individual numerical reward scores do not guarantee overall qualitative improvements.
Table 4: Reward Hacking observed in training using QUARK: We show examples of how the rewards used are very weak or incompatible with each other, they can lead to strange reward hacking where the score numerically increase, but the rationales become worse qualitatively.
Question and Rationale
Excors by other LM
Question: Can leopard swim?
TS-1 ARtER Repetition
Rationale:
The snow leopard a cat. The snow loopard is a
Question: Would someone on Verus unlikell to experience hypothemmis?
Rationale Hypothermia condition which the body unable produce enough beat.
ARGE Incomplete
Question: Did Cleoputra have ethnicity cleser to Egyptians than Greeks!
Rationale: Cleopatra was member
Roman dynasty Egypt.
Roman dynast of Egypt was branch the Empine.
FLAN-TS-XL Repetition
Roman Empire was branch the Roman Empice.
Question: Is Flar Chrysler goining new ovenall corporte idencity?
Rationale: Fiat Chrysler a brand of automobiles. Fiat Chrysler is brand of cars.
FLAN-T5 XL Repetition
5.3 IS ACCURACY ENOUGH OF AN INDICATOR FOR RATIONALE QUALITY?
As we discuss in many contemporary works on self-rationalizationn ignore the independent qual- ity of the generated rationale, and focus entirely on how the rationale can contribute to the task performance. In this discussion, we analyze the reverse: if an LM trained only with respect to task performance, what does this mean for the rationale? We refer back to our main results, Ta- 3l we specifically look at the rows SFT, FILT-ACC and MARIO. We first see that both FILT-Acc all five datasets, intended. We then see
and SFT both improve upon the TASK ACCURACY that for STRATEGY AA, QUAREL, NUMERSENSE and QASC, the average quality of the rationales generated by MARIO is decidedly better than the rationales generated by FILT- ACTT-ACC, seen by the values of the individual rationale quality metrics. For OPENBOOKS QA, the analysis from just the metrics is inconclusive; hence, we perform human studies comparing FILT- ACC and MARIO, in the same manner as in $4.3 We find that human annotators prefer MARIO'S 's rationales highly over that of Acc:-ACc: for 69.65% of the questions, majority of the annotators prefer MARIO's 's rationales (as opposed to 22. 88% of preference for FILT-ACC's rationales, and 7 7.46% preference for both). We further performed human studies for PLAUSIBILITY and CONSISTENCY, and again, MARIO'S ra- tionales were found to be distinctly better (PI. IPLAAUIBILLTYY: 49.5% preference for MARTO, 32.58% for SFT, 13.43% both, 0.99% neither; CONSISTENCY: 48% preference for MARIO, 37.31% for SFT, 9.45% both, 2.48% neither). In conclusion, we find that optimizing for task performance does not naturally improve rationale performance, which further motivates the introduction of MARIO.
6 CONCLUSION AND FUTURE WORK
Existing self-rationalization LMs use rationales as a means for improving downstream task accuracy, with the help of large-scale LMs. In this work, we propose MARIO, an algorithm that performs multi-reward optimization of small self-rationalizingg LMs to jointly improve the quality of their rationales as well as their task accuracy. We present strong experimental results on a small LM, T5- LARGE, over competitive baselines, on datasets STRATEGYQA, QUAREL OPENBOOKQA, NU- MERSENSE and QASC. In addition to a strong improvement in task accuracy, we see that rationales produced by training an LM with our method are strongly preferred by human annotators. Lastly, we discuss intricacies of selecting appropriate rewards, well as shortcuts taken by QUARK to improve reward scores that do not translate well to qualitative improvement. As future work, we hope to extend our algorithm to improving rationales along more dimensions like completeness, factuality well human utility.
rationale generation for small LMs, issues faced with

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This research supported in part by the Office of the Director of National Intelligence (ODNT) In- telligence Advanced Research Projects Activity (IARPA), via the HIATUS Program contract #2022- 22072200006 and the USC + Amazon Center on Secure & Trusted Machine Learning. The views and conclusions contained herein are those of the authors and should not be interpreted as neces- sarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for govemn- mental purposes notwithstanding any copyright annotation therein. We would like to thank all of our collaborators the USC NLP Group and USC INK Research Lab for their constructive feedback on this work. We also thank the reviewers for their valuable and constructive suggestions.
ETHICAL CONSIDERATIONS
Like any natural language generation system/algorithm, MARIO can unintentionally lead toxic and harmful text: it is up to the user of the algorithm to use it responsibly, with non-harmful reward metries, to prevent the generation of biased and malicious outputs. As noted in McGuffie & New- house (2020), this is a deliberate misuse of text generation models, and we strongly denounce such practices.
Data. All the datasets that we use in our work are released publicly for usage and have been duly attributed to their original authors.
Crowdsourcing. All our crowdworkers are from countries where English is the primary language. For all our human studies, the task is setup in manner that ensure that the annotators receive com- pensation that is above minimum wage (S20/hour). Since we conduct extensive qualification tasks before amnotations, crowdworkers that participate the qualification are compensated more than the task, given the time taken to read and understand task instructions and examples. Furthermore, we ensure that we correspond with crowdworkers over email to address their queries. Crowdworkers have also been given bonuses for flagging errors in the task, or consistently providing good-quality annotations.
REPRODUCIBILITY
For all our experimental results and models, we report (1) the complete hyperparameter setting and any bounds explored (Appendix [P) as well as the sizes and vessions/petrained-modee links of all models used, (2) the time taken per experiment, and infrastructure used, (3) the mathematical equations (44. Appendix [B for all algorithms and metries used, (4) descriptions datasets, and demonstrations used to sample rationales from GPT-3. All codes and datasets are publicly
leased at
REFERENCES
Pepa Atanasova, Oana-Maria Camburu, Christina Lioma, Thomas Lukasiewicz, Jakob Grue Simon- sen, and Isabelle Augenstein. Faithfulness tests for natural language explanations. In Proceed- ings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 283-294, Toronto, Canada, July 2023, Association for Computational Lin- guistics. doi: 10.18653/w1/2023..m acl- 250 URL htt.appss//aaaaaattooooooooo///p acl-short.c 25
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, T. J. Henighan, Nicholas Joseph, Saurav Kadavath, John Kernion, Tom Conerly, Sheer EI-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Christopher Olah, Benjamin Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. ArXiv, abs/2204.05588 2022. URL https://///t....... /api. sepppiseemn wwwanticschoooooooo hwt.e.oooaooooooooooooooooooooo. CorpusTD:2481 18878
10

---

Published as a conference paper at ICLR 2024
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Hemighan, Rewon Child, Aditya Ramesh, Daniel M. Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. e- snli: Natural language inference with natural language explanations. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL
nwwries.....ee...... ccppers.cceppppprrrrr httpsccpppeeeeeee//eeeeeeeeeeeee File/4c7ale 6 7bb32 2bw.33322bc2222999eeeeee Patp:eeeee222aaaaaeaeeeeee.
Aaron Chan, Maziar Sanjabi, Lambert Mathias, Liang Tan, Shaoliang Nie, Xiaochang Peng, Xiang Ren, and Hamed Firooz. Unirex: A unified learning framework for language model rationale extraction. In International Conference on Machine Learning, pp. 2867-2889. PMLR, 2022.
Aaron Chan, Zhiyuan Zeng, Wyatt Lake, Brihi Joshi, Hanjie Chen, and Xiang Ren. Knife: Distilling reasoning knowledge from free-text rationales, 2023.
Hanjie Chen, Faeze Brahman, Xiang Ren, Yangfeng Ji, Yejin Choi, and Swabha Swayamdipta. REV; Information-thheortic evaluation of free-text rationales. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2007- 2030, Toronto, Canada, July 2023a. Association for Computational Linguistics. doi: 10.18653/ 11/203.accl-ongg.
URL pttpp::////. http://s//ccaaacoooooooorrr///// http:/22222..11111g..gg
Jiangjie Chen, Qiaoben Bao, Changzhi Sun, Xinbo Zhang, Jiaze Chen, Hao Zhou, Yanghua Xiao, and Lei Li. Loren: Logic-regularized reasoning for interpretable fact verification. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 10482-10491, 2022.
Wei-Lin Chen, An-Zi Yen, Hen-Hsen Huang, Cheng-Kuang Wu, and Hsin-Hsi Chen. Zara: Improv- ing sho-shot selfiraatinilizatioonn for small language models, 2023b.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph. Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetie language mod- els. arXiv preprint arXiv:2210. 1416, 2022.
Misha Denil, Alban Demiraj, and Nando De Freitas. Extraction of salient sentences from labelled documents. arXiv preprint arXiv: /412,684/ 2014.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and T. Zhang. Raft: Reward ranked finetuning for generative foundation model align- ment. ArXiv, abs/2304.06767,, 2023. URL
http://app...eennnccccchooo..... o
CorpusID:258170330
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. ArXiv, abs/2305. 143387, 2023. URL h
//api.semant iesaanntcccchoar... www.aaarrrroooooooo//a 258865545
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant, Did aristotle use laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346-361 -611, 2021.
Olga Golovneva, Moya Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel- Zarandi, and Asli Celikyilmaz. Roscoe: A suite of metrics for scoring step-by-step reasoning. arXiv preprint 2rXiv:22/2.079199 2022.
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Ab- hishek Sharma, Aditya Siddhant, Alexa Ahern, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, A. Doucet, Orhan Firat, and Nando de Freitas. Reinforced self-training (rest) for language modeling. ArXiv, abs//308..080 0889988 2023. URL https://///.........ttt /api. semanticscholas horgee.oooaaraa/oo CorpusTD:26103101

---

Published as a conference paper at ICLR 2024
Peter Hase, Shiyue Zhang, Harry Xie, and Mohit Bansal. Leakage Leakage-adjusted simulatability: Can models generate non-trivial explanations of their behavior in natural language? arXiv preprint arXiv:2010.04119, 2020.
Dirk Hovy and Shrimai Prabhumoye. Five sources of bias in natural language processing. Language and Linguistics Compass, 12432, 2021.
Cheng-Yu Yu Hsich, Chun-Liang Li, Chih-kuan Ych, Hootan Nakhost, Yasuhisa Fujüi, Alex Rat- ner, Ranjay Krishna, Chen- Yu Lee, and Tomas Pfister. Distilling step-by-step! outperform- ing larger language models with less training data and smaller model sizes. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 8003-8017, Toronto, Canada, July URL
2023. Association for Computational Linguistics. doi: 10. http://ww//22.....ngggggeooe https://acllanhholll www.antwwwwooooooo//o/eeoo/e///e f findings aclgs---cl... acl. 507
Sarthak Jain, Sarah Wiegreffee, Yuval Pinter, and Byron C Wallace. Learning to faithfully rationalize by construction. arXiv preprint 0001::2000.00115 2020.
Xisen Jin, Zhongyu Wei, Junyi Du, Xiangyang Xue, and Xiang Ren. Towards hierarchical impor- tance attribution: Explaining compositional semantics for neural sequence models. arXiv preprint arXiv:1911.06194 2019.
Brihi Joshi, Ziyi Liu, Sahana Ramnath, Aaron Chan, Zhewei Tong, Shaoliang Nie, Qifan Wang, Yejin Choi, and Xiang Ren. Are machine rationales (not) useful to humans? measuring and improving human utility of free-text rationales. In Proceedings of the 61st Anmual Meeting of the Association for Computational Linguisties (Volume Long Papers). pp. 7103-7128, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 392. URL /ttpt:::s//caanntolllooooo wttplooy...or//22221ll---o--000.
392
Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi. Maieutic prompting: Logically consistent reasoning with recursive explanations. arXiv preprint arXiv:2205.11882. 2022.
Akos Kádár, Grzegorz Chrupata, and Afra Alishahi. Representation of linguistic form and function in recurrent neural networks. Computational Linguistics, 43(4):761-780, 2017.
Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation, 2019.
Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. Qase: dataset for question answering via sentence composition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 8082-8090, 2020.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 0122199-222133 2022.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners, 2023.
Tao Lei, Regina Barzilay, and Tommi Jaakkola. Rationalizing neural predictions. In Jian Su, Kevin Duh, and Xavier Carreras (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 107-117, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.11633/v//D16 1011. URL
brg/D16-1011
Piyawat Lertvittayakumjorn and Francesca Toni. iittanaation-basel human debugging of NLP mod- els: A survey. Transactions of the Association for Computational Linguistics, 9: 9:1508-152 1528, 2021. doi: 10. a 004400 URL http:///aaaanalhaaao wttg//2/000
Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. Visualizing and understanding neural models in nlp. arXiv preprint arXiv: 01066, 2015.
Jiwei Li, Will Monroe, and Dan Jurafsky. Understanding neural networks through representation erasure. arXiv preprint arXiv: 16/2x/6//22082200 2016.
12

---

Published as a conference paper at ICLR 2024
Liunian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai- Kei Chang, and Yejin Choi. Symbolic chain-of- thouhu distillation: Small models can also "think" step-by-step. ArXiv, abs/2306. 14050, 2023a. httsttps:: /appp.p....///../ semant iischolarrcccara org/ Sttp://oooossSD:22222....
I Liunian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai-Wei Chang, and Yejin Choi. Sym- bolic thain-of-thoughe distillation: Small models can also "think" step-by-step. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2665-2679, Toronto, Canada, July 2023b. Association for Computational Linguis-
tics. doi: 10. 10.118653/v1/20233aallloogg 150. URL
ac 2png.150
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation optimization. arXiv preprint arXiv:2210.15000 2022.
Yifei Li, Zeqi Lin, Shizhuo Zhang. Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. Making language models better reasoners with step-aware verifier. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5315-
5333, 2023c.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step, 2023.
Bill Yuchen Lin, Seyeon Lee, Rahul Khanna, and Xiang Ren. Birds have four legs?! numersense: Probing numerical commonsense knowledge of pre-trained language models. arXiv preprint 0000:2200.000683 2020.
Jiacheng Liu, Skyler Hallinan, Ximing Lu, Pengfei He, Sean Welleck. Hannanch Hajishirzi, and Yejin Choi. Rainier: Reinforced knowledge introspector for commonsense question answering. arXiv preprint arXiv:2210.03078, 2022.
Jaachnng Liu, Wenya Wang. Dianzhuo Wang, Noah A Smith, Yejin Choi, and Hannaneh Ha- jishirzi. Vera: A general-purpose plausibility estimation model for commonsense statements. arXiv preprint arXiv:2305.0m 006950.5......... 2023a.
Wei Liu, Jun Wang. Haozhao Wang, Ruixuan Li, Yang Qiu, YuanKai Zhang, Jie Han, and Yix- iong Zou. Decoupled rationalization with asymmetric learning rates: A flexible lipschitz re- straint. In Proceedings of the 29th ACM SIGKDD Conference Knowledge Discovery and Data Mining, KDD '23, pp. 1535-1547, New York, NY, USA, 2023b. Association for Com- doi: 10.1455358800033599999,
puting Machinery. ISBN 9798400701030. 7/ //doi.org// 10. 1144//55000000 3599299
URL https:
Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Am- manabrolu, and Yejin Choi. Quark: Controllable text generation with reinforced unlearning. Advances in neural information processing systems, 35:27591-27609, 2022.
Ximing Lu, Faeze Brahman, Peter West, Jaehun Jang, Khyathi Chandu, Abhilasha Ravichander, Lianhui Qin, Prithviraj Ammanabrolu, Liwei Jiang, Sahana Ramnath, et al. Inference-time policy adapters (ipa): Tailoring extreme-scale Ims without fine-tuning. arXiv preprint arXiv:23000 15065, 2023.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Proceed- ings of the 31st international conference on neural information processing systems, pp. 4768- 4777, 2017.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Itera- tive refinement with self-feedback. ArXiv, abx/2303. 17651, 2023. URL hhttps: semant cschol ar. atr..orgg/oorruaatt55500000
13

---

Published as a conference paper at ICLR 2024
Ana Marasovic, Iz Beltagy, Doug Downey, and Matthew Peters. Few-shot self-rationalization with natural language prompts. In Findings of the Association for Computational Linguistics: NAACL2022, pp. 410 424, Seattle, United States, July 2022. Association for Computational Lin- orgg/
guistics. doi: 10. 10. (188366/120000 1/2022 http://////0...nnnnn-naaaaaaaaaa findinns--nacll.1. URL
7022.finm Findinggnnaaaee
Kris McGuffie and Alex Newhouse. The radicalization risks of gpt-3 and advanced neural language models. arXiv preprint arXiv:2009.06800 2020.
Clara Meister, Elizabeth Salesky, and Ryan Cotterell. Generalized entropy regularization or: There's ss nothing special about label smoothing. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 6870-6886, Online, July 2020. Asso- URL hetps:
ciation for Computational Linguisties. doi: 10.18653/v 12200
7/acl www.thoooo..oooooooooooo. ogg.oogg/2000000...0 attp:/22000...mmmammaa acl-ma n. 615
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering, arXiv preprint (rXii:1809.0278, 2018.
Sharan Narang, Colin Raffel, Katherine Lee, Adam Roberts, Noah Fiedel, and Karishma Malkan. Wt5?! training text-to-text models to explain their predictions. arXiv preprint arX1v:2004.15546, 2020.
Nina Poerner, Benjamin Roth, and Hinrich Schütze. Evaluating neural network explanation methods using hybrid documents and morphological agreement. arXiv preprint
2018.
Archiki Prasad, Swarnadeep Saha, Xiang Zhou, and Mohit Bansal. Receval: Evaluating reasoning chains via correctness and informativeness. arXiv preprint arXiv:2304. 10703, 2023.
Danish Pruthi, Bhuwan Dhingra, Livio Baldini Soares, Michael Collins, Zachary C Lipton, Graham Neubig, and William W Cohen. Evaluating explanations: How much do explanations from the teacher aid students? arXiv preprint arXiv:2012.00893, 2020.
Danish Pruthi, Rachit Bansal, Bhuwan Dhingra, Livio Baldini Soares, Michael Collins, Zachary C. Lipton. Graham Neubig, and William W. Cohen. Evaluating explanations: How much do ex- planations from the teacher aid students? Transactions of the Association for Computational Linguisticss 10:359 375, 2022. doi: 10. 10.1/a2/t a 00466... URL
org/2022. tac1-1.2.
Jing Qian, Li Dong, Yelong Shen, Furu Wei, and Weizhu Chen. Controllable natural language generation with contrastive prefixes, 2022.
Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. Explain yourself! leveraging language models for commonsense reasoning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pp. 4932-4942, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 11118653111119-14999 URL https: //aclanthology.ors wwwphoooooooor///11111
Swarnadeep Saha, Peter Hase, and Mohit Bansal. Can language models teach weaker agents?
teacher explanations improve students via theory of mind, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proxi- mal policy optimization algorithms. ArXiv, abs/1707..0l 707.04347, 2017. URL https://app.
semanti cscho ar. httgg//orpusatts TD 28695052
Xuelin Situ, Ingrid Zukerman, Cecile Paris, Sameen Maruf, and Gholamreza Haffari. Learning to explain: Generating stable explanations fast. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 5340-5355, 2021.
Joe Stacey, Yonatan Belinkov, and Marek Rei. Supervising model attention with human explana- tions for robust natural language inference. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 11349-11357, 2022.
14

---

Published as a conference paper at ICLR 2024
Jiao Sun, Swabha Swayamdipta, Jonathan May, and Xuezhe Ma. Investigating the benefits of free- form rationales. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 5867-5882, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguisticss. doi:
wttp://///22222.nnnnnnn---e org/2022. find ins--gggslpnpmm 4332
URL
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning, pp. 3319-3328, PMLR, 2017.
Oyvind Tafjord, Peter Clark, Matt Gardner, Wen-tau Yih, and Ashish Sabharwal. Quarel: A dataset and models for answering questions about qualitative relationships. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 7063-7071, 2019.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 1,
Peifeng Wang, Aaron Chan, Filip Ilievski, Muhao Chen, and Xiang Ren. Pinto: Faithful language reasoning using grompttgenerated rationales. ArXiv, abs/2211.011 1..5562,, 2022. URL https://
PeiFeng Wang, Aaron Chan, Filip Ilievski, Muhao Chen, and Xiang Ren. PINTO: Faithful lan- guage reasoning using prompt-generated rationales. In The Eleventh International Confer- ence on Learning Representations, 2023a. URL htttp:://ppereeeeee
net / forum? d=
WBXbRs63cVu
Peifeng Wang. Zhengyang Wang, Zheng Li, Yifan Gao, Bing Yin, and Xiang Ren. SCOTT: Self- consistent chain-of-thought distillation. In Proceedings of the 61 Annual Meeting of the Associ- ation for Computational Linguistics (Volume 1: Long Papers), pp- 5546-5558, Toronto, Canada, July 2023b. Association for Computational Linguistics. doi: URL /tttp::/aaclathhot
1/20233//1//22.aclll-l org/ 2023 acl long. 304
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022.
Sarah Wiegreffe and Yuval Pinter. Attention is not not explanation. arXiv preprint 0rXiv:1908.04620 2019.
Sarah Wiegreffe, Ana Marasović, and Noah A. Smith. Measuring association between labels and free-text rationales. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 10266-10284, Online and Punta Cana, Dominican Republic, Novem- ber 2021. Association for Computational Linguistics. doi:
1/222/..1111mmmllmmmmo URL ttttps::/aclanthoolle orgp://0oooorg//2222/ www.p-mmlle... caapppmain... 804
Sarah Wiegreffe, Jack Hessel, Swabha Swayamdipta, Mark Riedl, and Yejin Choi. Reframing human-AN AI collaboration for generating free-text explanations. In Proceedings of the 2022 Con- ference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 632-658, Seattle, United States, July 2022. Association for Compu- URL. httttss://aacaatthtoo
tational Linguistics. doi: 1
org/2022 naacl -main....... 47
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hanna Hajishirzi. Fine-grained human feedback gives better re- wards for language model training. ArXiv, abs/2306.01699, 2023. URL https::/appi.
semanti i cscho ar.
ID: 259064099
Kevin Yang and Dan Klein. FUDGE: Controlled text generation with future discriminators. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.). Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pp. 351 3511-3535, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v 1 1/202 www.aaaaannan..... main.276. 276. URL https: / h/taaaaaaaanoooooooaaaoaoaoooola .or/
2021 naac
15

---

Published as a conference paper at ICLR 2024
Kexin Yang, Dayiheng Liu, Wenqiang Lei, Baosong Yang, Mingfeng Xue, Boxing Chen, and Jun Xie. Tailor: A prompt-based approach to attribute-based controlled text generation, 2022.
Eric Zelikman, Jesse Mu, Noah D Goodman, and Yuhuai Tony Wu. Star: Self-taught reasoner bootstrapping reasoning with reasoning. 2022.
Honghua Zhang, Meihua Dang, Nanyun Peng, and Guy Van Den Broeck. Tractable control for autoregressive language generation. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Bar- bara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.). Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 40932-40945. PMLR, 23-29 Jul 2023. URL v202/ 2tta//haag23000 html
mll. press/
16

---

Published as a conference paper at ICLR 2024
A RELATED WORK
Self-rationalization and rationale-based distillation. Model decisions can be explained in two ways- by extracting rationales from the input text, or generating free-text text rationales that may not be grounded in the input. An extractive rationale explains a model's 's output on given task instance by
scoring input tokens' influence on the model's output (Denil et al. 2014 Sundararajan et al. 2017
Li al. 2016)] Din al.
2019| Lundberg & Leee 2017|JChan et al. p022). This token scoring can
be done via imput gradients (Sundararajan et al. all.|2017j1..... 2017;
& Lee 22077,Demm et al. 2014: Li
ctt al. 2015). input perturbation (Li et al. [2066; Poerner et | 2018: Kádár ct al. 2017). attention
weights (Pruthi et al., 2020, Stacey et al. 2002; Wiegreffe & Pinter
2019), or learned rationale
extraction models (Lei et al. 2016 Chan et al., [022]] Jain et al.) 2020 Situ et al. 2021] Liu et al.,
2023b). For the purpose of this work, mainly focus on free-text rationales. There are two primary methods adopted by prior works for generating free-text rationales. The first set of approaches use
gold human-writte rationales train rationale generation model (Camburu et al. 2018: Narang iet al. 2200;; Wiegreffe et al. 2021). The second set of approaches prompt large LMs with the help of curated templates with or without demonstrations containing examples of rationale generation for
the task at hand (Wei et al. 2022 Kojima et al.) 2023 Li et al. 2023c Jung et al. 2022: Lightman
jet al. 2023). Some approaches also leverage few-shot training approaches with handful of gold rationales (Marasovic et al. 2022 Chen et al. 2023b). Recent approaches also leverage rationales generated by large LMs to distill small LMs to be better at the task or better rationalizers. (Pruthi
et al. 2022 Li et al.
2023b Chan et al. 2023
Wang et al. 2023b Saha al. 2023 Hsieh et al.
2023
Evaluating free-text rationales. Existing works have conducted human and automatic evaluation of free-text rationales based on their association with predicted labels (Wiegreffe et al., 202T), ac- ceptability (Wiegreffe et al. 2022), informativeness (Chen al. (023a), benefits and human utility
(Sun et al,, (022) Joshi et al. (023), simulatability (Rajani et al. 2019 Hase et al., (020) and faith-
fulness (Atanasova et al.] 2023 Wang et al.) (023a) name few. Some recent works have also
provided frameworks to evaluate logical correctness of reasoning chains, that are similar to free-text
rationales (Golovneva et al. 2022 Prasad et 2023).
Reward-conditioned text generation. Reinforcement learning has proven to be a reliable means to optimize language models towards specific objective. One such example, proximal policy op- timization (PPO) (Schulman et al., 2017). has been commonly used for a variety of tasks, spanning
detoxification (Wu et al.) 2023] Lu et al. 2022), RLHF (Dubois et al. 2023 Bai al. 2022). im- proving commonsense reasoning capabilities (Liu et al.] 2022), and more. Adjacent to PPO, there are several other lighter-weight algorithms which condition the policy language model directly on the reward without the need for value function (Lu et al. 2022, Gulcehre et al.) 2023 Dong et al.)
2023 Lu et al. 2023 Zelikman et al., 2022). These methods rely on iterative, ofi-policy explo- rations fixed intervals to continuously aggregate new trajectories to learn from. Another line of work improves the reward directly through iterative refinement on a frozen policy model (Madaan et al. 2023). There are several algorithms and methods today to update text generation models with rewards. Lu et al. (2022) that unlearns toxicity by specifically fine-tuning the model on what not to do, Lu et al. (2023) which tailors the generation of extremely large LMs like GPT-3 using trained policy adaptor models. Zelikman et al. (202)) that leverages a small number of demonstrations to iteratively generate new data to train the model (new data such that the task prediction correct). Other recent work on controllable text generation revolves around creative text generation with sin- gle and multiple rewards (Yang & Klein 2021 Keskar al. 2019 Zhang et al. 2023 Qian et al.,
2022
Yang et al.) 2022)
B QUARK AND MARIO
Here, we describe QUARK and MARIO in more technical detail (refer the top and bottom pipelines in Figure respectivelyy
QUARK begins training with a pretrained trained language model Po(t)=); QUARK also requires reference language model Prey (t|x=) (which can be the same Po. or different), and reward function (ew(t, x) - R. Note that x [2o: Z1,... [mm stands for the input text sequence, and (o,41,-... l,,...; : 4n-1l
stands for the output sequence generation. Lastly, QUARK works with
17

---

Published as a conference paper at ICLR 2024
Sing Le reward Quark
Single control tokens: TOK1, TOK3,
the decoder side.
leopard
Can snow leopard swim?
Language Model
TOK2
Cats can swim. So the answer yes.
Mutti-roward control tokens: A.B.C.C
Control tokens an: decoder side.
TOK2 A.B.C.D
Multi-reward Quark
TOK3 A.B.C,D
CLASSIC MaRio
leopard the snow. Cats swim. So the answer is yes.
Can snow leopard swim?
Language Model
TOK1_ B
TOK2_D
TOK3
ADDITIVE MaRio
TOK1_ B
snow leopard cat that lives answer is
TOK2_D
Can a snow leopard swim?
Language
TOK1_B
TOK2_D
TOK3_A
Figure 5: Optimizing properties with QUARK (top) and MARIO (bottom)
data pool D which is constantly updated and added to the course of training (as describe below); further, D can initialized with gold-standane or silver-standard data, D gold] siluer (x, &gold/ silver; r).
As we explain in Section 3 QUARK operates in an iterative fashion:
1. sampling Po to generate more training data: D Dnew = (Ir, know)
2. scoring the generated data using Rew(x, f): Dnew (I., tnwws
3. using these instance-level scores to sort and bin the data into a fixed number of bins bg,bb,,,,b)).. each of which correspond (Ir, tnews r, b),
unique control token: D Daow
4. Adding the now control-token attached data to the (growing) training data pool: D DU D Dnew
During training, the model starts associate each control token with its corresponding quality of data (as given by Rewr(r, ()), and to obtain the best quality generations during inference, QUARK samples the trained Po using the control token corresponding to the highest reward measure. QUARK trained using the following training objectives:
Reward-based learning using implicit reward signals based on control tokens (which are obtained by sorting the reward Rew(x, t) scores), described above,
Language model objective using superviseed/coos-entropyy loss with respect to the target generation (as explained above, Ksapless training data in online manner from Po; however, gold or silver offline training data is available, that can also be injected into the training pipeline by scoring with Rew(2, (x,,t) t))
Stable text generation using the KL divergence penalty of Po generation with respect to Pref, and.
Entropy regularization of the generated text as Meister et (2020)
The objective function for QUARK is:
max (E)-u((,,)))(..)--- log (u x.rK) 3 KL (po (- yees ll (: Vet, x,, rk)) (5)
18

---

Published as a conference paper at ICLR 2024
Here, the first term stands for the supervised cross-entropy loss, and the second term stands for the KL divergence loss. Entropy regularization can also be added if/when needed. Note that x is the input text, y is the generated output sequence and Tk. E [I....,,K stands for the reward/control token.
We extend QUARK to MARIO by using multiple sets of control tokens, each corresponding dis- tinct reward/property, i.e., Rew (x, t), (aw t).... Rew,(x, (z, t); the CLASSIC and ADDTTIVE methods use these control tokens either together, in a step-by-step fashion
we explain in $3.1 3.2 Further, we want to note that step-3 of the algorithm (wherein we use instance-level scores to sort and bin the data) is done in MARIO separately for each reward: each reward/property goes through an individual scoring + binning process and gets a distinct control token. Subsequently, each reward/property also has its own set of control tokens (as depicted in Figure The rest of the training follows the same iterative process and training objectives as QUARK. The objective function for MARIO is:
max 0
log pe (y x. (yb. 3
Σ KL (po (- Det, lllo (- yee, 2., [.. Tyk -)) 1=1
(6)
Here again, the first term stands for the supervised cross-entropy loss, and the second term stands for the KL divergence loss; entropy regularization can be added if/when needed. x is the input text, y is the generated output sequence and rj E {I.....JJ JJ. E {1...., K} stands for the reward/control token corresponding to the j-th property and the k-th reward bin.
C ORDER OF TOKENS
As we explain in the above two sections, the order of the control tokens corresponding to each reward we use in training our self-rationalizing LM is design choice. Say for example, we have three properties. along with control tokens corresponding to the task accuracy (as do in this paper, refer 44.2): this means that there are potentially 24 orders of these properties that we can use in CLASSIC MARTO, and possible variations that we can use for ADDITIVE MARIO (24 orders 2 directions in which we introduce the property to the training left or right of the existing control tokens, assuming we keep the direction of addition consistent throughout training). It is impractical and inefficient to experiment with all these possible orders to pick the best possible one. Hence, we propose a simple way of picking the order, based on the relative strengths of a (supervised-trained) self rationalzii LM in each of these properties.
For example, say we have four reward metrics R3, R2, Rg, R4, and we determine through a prede- fined method which property the LM is relatively stronger in (for example, say the LM is good generating lexically diverse statements, but only moderately good at grammar, is broadly bad at generating plausible statements, and even worse at producing concise rationales). For example, we determine the relative strength of rewards based on how good the supervised finetuned baseline SFT on particular metric on the validation set, as opposed the maximum and minimum value of the metric itself.
max(R:) Ti strength(R.) max(R,) (R,) - min R&)
Here R; refers to the reward, r; refers the value the SET has the property R; the validation set, and max/min(R.) refer to the maximum and minimum value taken by the reward metric Rg-
For example, let the relative order of the four reward metrics using the above approach is R2 < R₁ R4 < Rg. Hence, we experiment with training the LM with the order R2, R, R4, R3 we want to allow the weaker rewards to improve into the mix. Additionally, we can also use the opposite order Rz, R4, R₁, R2, so that the LM can quickly optimize the stronger rewards and then try be better with the weaker rewards.
their own, before the stronger rewards are introduced
19

---

Published as a conference paper at ICLR 2024
D DATASET SPLITS
For STRATEGYQA, since labels are not available for evaluation sets, we split the train set into training, validation and test sets (taken from Joshi et al. 2023)), and report scores this test set.
the provided training dataset, tuned the val- idation set and report final performances on the test
For OPENBOOKQA and QUAREI,
For NUMERSENSE, we use the train, validation and test sets as the official GitHut release.
For QASC, we split the original train set into train and validation (900 guestions chosen randomly for validation), and use the original validation set the test set
All datasets have multi-choice questions (ycs/no for STRATEGYQA, a/b for QUAREL, a/b/cld for OPENBOOK QA, a/b/-/1 for NUMERSENSE, a/b//h for QASC), and the task is to generate rationale followed by the predicted answer.
E HYPERPARAMETERS AND EVALUATION
We use T5-LARGE (0.7B parameters) for SFT and all our MARIO experiments, and T5- BASE for our CONSISTENCY models (as used in the original work WWierreffe et al. (2021) we always start training with the pretrained model from HuggingFacq Tables and 7 show the hyperparameters used train SFT, CONSISTENCY and MARIO respectively. Note that for our MARIO training, we SFT the reference model (tre)(t(æ) from Appendix B for the KL divergence penalty. We also use the silver rationales sampled from GPT-3 as our initial data pool D (from Appendix BB. Further, during inference, always greedy decoding. We run all our experiments on NVIDIA Quadro RTX 8000 GPUs. For training SFT and CONSISTENCY models, 1 GPU per experiment; for training MARIO, use 2 GPUs per experiment the first GPU to hold Po, Pref (notation from Appendix B and the second GPU to hold the PLAUSIBILITY and CONSISTENCY reward models.
Furthermore, we aggregate metrics using Normalized Relative Gain as mentioned in Chan et al. 2022). NRG of metric value 2, (corresponding the general property 2) is formally defined as: min 2))
NRG(2;) max (2) min(2) (Z)
The average NRG of a set of metrics (such as with the four metrics in this work) is simple mathe- matical average of their individual NRG's.
Table 5: SFT training details
Hyperparameter Optimizer
Value
Adam
Adam epsilon
1e-8
Adam initial learning-rate
3e-5
Learning-rate scheduler
linear with warmup
Warmup steps
1000
Gradient clipping
0.5
Train batch-size
4/8
Training time
4 hours on GPU
Further, for our statistical significance tests, are done using one-tailed independent t-tests (using
seipy.stats..test.innd
htte:///uuuuiieeeeadeee///pblllt datasets/QuaRcl
20

---

Published as a conference paper at ICLR 2024
Table 6: Training details for the Mon and Mg models used for CONSISTENCY
Hyperparameter
Value
Optimizer
Adam
Adam epsilon
le-8
Adam initial learning-rate
3e-5
Learning-rate scheduler
linear with warmup
Warmup steps
1000
Gradient clipping
0.5
Train batch-size
4/32
Training time
4 hours on GPU
Table QUARK and MARIO training details
Hyperparameter
Value
Optimizer
Adam
Adam epsilon
1e-8
Adam initial learning-rate
3e-5
Learning-rate scheduler
linear with warmup
Warmup steps
1000
Gradient clipping
1.0
Gradient accumulation
2 steps
Kl.-divergence coef.
0.05/0.1
Entropy regularization coef.
0.05 / 0.0
1 (QUAREL, NUMERSENSE, QASC)
Sampling rate
or 2 (STRATEGYQA, OPENBOOKQA)
samples for every train sample
every 300 (STRATEGYQA, QUAREL) 4000 (OPENBOOKQA, SUMSRS QASC) steps
Frequency of exploration
Sampling strategy Temperature for sampling
Top-p (0.7) sampling 1.0
for rationale metrics,
Number of distinct reward-bing bins
for TASK ACCURACY
Train batch-size
4
Training time
day on GPUs
STRATEGY YQA: strongest to weakest,
add to right
QUAREL: strongest weakest, OPENBOOKQA: weakest to strongest, NUMERSENSE: weakest to strongest, QASC: strongest to weakest
Order of rewards
F REPRESENTATIVE EXAMPLES OF RATIONALES
Table 8 shows some examples of rationales generated by MARIO.
G SINGLE REWARD EXPERIMENTS
For completeness of analysis, we present single-reward QUARK experiments, where we focus on improving just one property. 9 shows results on the same. We first note that in most of the cases, MARIO achieves an equivalent or better improvement as compared to single-reward QUARK. Further, we note that even if individually some properties are better when trained under single reward QUARK as compared to MARIO, MARIO the only experiment where all the properties improve as compared to the SFT baseline. We also see that sometimes, single reward QUARK leads to improve- ment in other metrics as well; this could be because the metrics are positively correlated for that dataset. However, since want improve all metrics comprehensively. MARIO deterministic
21

---

Published as a conference paper at ICLR 2024
Table 8: Examples of Rationales: Here, we show rationales generated by MARIO and SFT for some examples for the three datasets. We observe that sometimes, SFT generates longer rationales which are not consistent, or logically correct, which is fixed by MARIO.
Dataset
Examples
Question: Waile on liquid diet, are there some types of soup you cansot cut! Yes or No!
MARIO Rationale: While on liquid diet, there are some types of soop canmot est, such as cream of mishrroom sougl
Which is type of savery soup,
SET Rationale: While liquid dies, there are certain types of soups that not suitable fox liquil diet such ss laquid ciett
STRATIGYQA
Question: Is beard is moss that grows on human? Yes or No?
MARIo Rationale: A beard type of hair that grows on huminn
SET Rationale: beand type of facial hair
from nool and iss typically wavy dark color. It not but
shelter
and helps grow.
Question:
bus and bicycle
going
disection.
going 40taph than the bicyele. Will the bicycle
hour
Rutionale: bus going fister
bicycle. Thus, bicycle cover less distance hour distance. the bicycle cover more distance hour
Rationale: When something going faseer, because it going fester than the bus.
QUAREL
Question: Jus and are riding bikes. Jins stops after dificos minutes and takes boook. Joe koeye on riding. Who wont forther? (A) Jim (B) Joe
MARIo Rationale: When someeone is riding for a longes period of time, they will go farther. Jim stopped after dittenn minutes and mook a beeak, while Joc kept on riding, Thus, Joe went fathar
SFT Rationale: When someone moving. they will go farther than someone who not. Thus, Joe went farther because he kept on riding
Question: order for crops to grow food safely, pessicides are used on them. When it floods, this causes he what to be paisonous?
air (b) Com Runetf (d) farmers
MaRIo Rationale: When floods, raulf from the crops is sprayal with pesticides makking them peisonous.
SET Rationale: When floods. pesticides sprayed crops, which make them poisonous.
OPEN
Quastion: Plant growth cause
uprick in the number of leaves (b) a surgo leaf disenve gradiaal decrease in leaves
(d) rapid decline the leaves
MARIO Rationale: Plant growth may uptick the mamber
SET Rationale: Plant growih is process sddding new leaves plant. process can cause gradial decresse number leaves.
Question: People
more emask> logs. (A) (B) sero (C' one (L) two (E) three (P) (Gi live (L))
seves (3) dight
Rationale: People have roo legs
SFT Rationale: People have four legs.
Question: One in <mark>> also means quarier. (A) no (D) zero (C) one (D) two (Ei three (I) four (G) live (H) six (1) sevea (3) right (K) nine (1. ten
MARIo Rutionale: One four is quarter
SFT Rationale: One i guette
Question: Whot messumes a meter? (A) meter stick (T) foothall field (C) intracemstops (D) visibility (E) wasther U.S. customury unis (G) honses (H) lampkreys
MARIO Rationale The metric system haved on the idea that meter is equal 100 centinctess
SET Rationale: meter unit length. The most
unit
ued the United States the meter.
QASC
Question:
What
used navigate? music backhones (C) astrlogy (D) cilia (E) magic (T) sponges (G) astommmm lamphreys hais
MAR1O Kationale: Cilia
he used
SE National:: Navigating requirss using
tachniuues. Some these mnols include compas., compass. compass, compues, composs, compase, comgress compasa, congres, compass, comjave, compass, comjaee, compass, compass. compass, compass, compass, compass, compass, compass, compass, compass, and com
way to achieve the same. (Note: We don't 't run the experiment on NUMERSENSE DIVERSITY, since SET already achieves the best possible value of 1.0).
H EXTENDED COMPARISON WITH FEW-SHOT LLMS
In Table T0 we present the detailed performance metrics of different reference LMs as opposed to MARIO. For QUAREL, MARIO beats all reference LLMs except for GPT-3 on all four metrics. For NUMERSENSE, MARIO beats all reference LLMS except for XLX and GPT-3 on all four metrics. The results are more varied with STRATEGYQA QA, OPEN and QASC; MARIO is better than the reference LLMs (apart from GPT-3) in the case of DIVERSITY for all three datasets, and in cases of varying comparisons with the reference LLMs (for example, MARIO is better at CONSISTENCY than FL--- and I.LAMA-7B for (OOK However, overall, we note that our model still needs to go further with respect to PLAUSIBILITY and TASK ACCURACY. We note that our method MARIO has done a significant job in bridging the gap between LMs such as the ones discussed in this section, and much smaller LMs such as T5-LARGE. We also note for TASK ACCURACY, CONSISTENCY and DIVERSITY, MARIO beats FLAN-TS-L, a model of equal size which has been trained with instruction fine-tuning for all 5 datasets (except for QASC and CONSISTENCY); and for all datasets except for STRATEGYQA. QAA MARIO also beats PLAUSIBILITY of FL.N-T5-L.
22

---

Published as a conference paper at ICLR 2024
Table 9: QUARK experiments improving single rewards. For each dataset, the best averaged NRG (across TASK ACCURACY, PLAUSIBILITY, DIVERSITY and CONSISTENCY) is highlighted bold, and each best individual metric underlined.
Method
Basclines
Single Reward QUARK
MARIO
Dataset
Metric
PRODUCT
Acc.
Plau.
Div.
Cons.
CLASSIC
ADDITIVE
Acc.
57.64
62.01
61.57
61.35
59.17
59.17
60,26
65.07
Plau.
0.33
0.35
0.36
0.36
0.36
0.36
0.38
0.39
STRATEGYQA
Div.
0.92
0.92
0.93
0.96
0.95
0.97
Cons.
-0.02
-0.01
0.01
Avg. NRG
58.66
59.75
59.77
60.21
59.79
60.17
60.94
63.27
Acc.
76.99
79.53
81.88
80.62
78.99
80.62
79.89
78.99
Plau. 0.71
0.72
0.74
0.81
0.71
0.73
0.77
0.75
QUAREL
Div.
0.95
0.95
0.93
0.97
0.97
Cons.
0.18
0.21
0.23
0.20
0.20
0.19
0.20
Avg. NRG
75.50
76.71
78.1
78.66
77.0
77.41
78.35
77.75
Acc.
61.65
64.46
61.65
64.66
66.27
Plau. 0.53
0.52
0.54
0.53
0.51
0.54
0.55
OPENBOOK
Div.
0.98
0.99
0.99
0.99
0.99
0.99
0.99
0.98
Cons.
0.05
0.07
0.09
0.07
0.07
0.11
0.09
0.09
Avg. NRG
66.79
66.54
67.99
66.79
67.04
68.69
68.64
68.29
Acc.
46.23
50.75
$1.76
50.75
S4.27
$5.28
54.27
Plau.
0.60
0.60
0.63
0.63
0.61
0.63
0.63
NUMERŠENSE
Div.
1.00
1.00
0.99
1.00
1.00
1.00
0.99
Cons.
0.17
0.20
0.21
0.21
0.22
0.23
0.23
Avg. NRG
66.18
67.69
68.57
68,56
69.07
69.95
69.44
Acc.
58.64
57.88
58.21
$7.88
58.1
58.75
60.15
59.61
Plau.
0.44
0.43
0.42
0.45
0.40
0.41
0.47
0.47
QASC
Div.
0.96
0.95
0.96
0.97
0,98
0.96
0.99
0.99
Cons. 0.19
0.17
0.17
0.17
0.17
0.20
0.19
0.19
Avg. NRG
64.54
63.60
63.68
64.6
63.65
63.94
66.41
66.28
I FEW-SHOT DEMONSTRATIONS
We include the full few-shot demonstrations used to prompt different models for three datasets in Tables 11 13 For clarity, the rationalizations are highlighted.
CROWDSOURCING FOR HUMAN EVALUATIONS
In this section, we describe the MTurk experiment setup. Each MTuur annotator paid above minimum wage. Since the dataset we used is carefully annotated by human, we can assure there is no toxic content and our experiment setup was submitted to IRB for ethical review. We limited our Turkers to English speaking nations United States, Canada, Australia, New Zealand and United Kingdom.
To ensure the quality of evaluation, conduct round of qualification tasks which include small set of evaluations. Turkers need to finish the qualification task first and get results of it, then we will show them the whole task.
J.O.1 WORKER SELECTION AND QUALITY CONTROL
Here, we describe details about how workers are selected and how annotations are ensured to be clean. First, employ multiple rounds of trials before deploying the actual task so as to get feedback from annotators whether they understand the task correctly. This includes in-house tests,
tested via Amazon Turk Sandbox []] and small batches tested on Turk. Second, create set of medium to hard qualification tasks for verifying preference, plauusiicc and consistency annotations
hhttps://requeee http:///wwsssttttm.o..occ.//eeee sandbox
23

---

Published as a conference paper at ICLR 2024
Table 10: We compare MARIO with strong few-shot reference LMs: FLAN-TS, LLAMA and GPT-3. Apart from FLAN-T5-L (which we have included to show model of equivalent size that has been instruction finetuned), all these models are much bigger than our T5-LARGE trained with MARIO.
Method
FLAN-TS
LLAMA
GPT-3
MARIO (0.7B)
Dataset
Metrie
L
XL
XXL
7B
65B
T-D-003
CLASSIC
ADDITIVE
Acc.
54.59
71.83
59.17
72.27
65.07
Plau.
0.49
0.59
0.64
0.72
0.70
0.70
0.38
0.39
STRATEGYQA
Div.
0.88
0.82
0.86
0.88
0.93
0.95
0.95
0.97
Cons.
-0.01
0.02
0.05
0.00
0.06
0.09
0.01
0.04
Avg. NRG
60.27
65.96
68.26
67.29
72.07
72.13
60.94
63.27
Acc.
77.36
76.99
77.54
56.70
76.27
83.33
79.89
Plau.
0.60
0.68
0.70
0.78
QUAREL
Div.
0.93
0.90
0.92
0.94
0.96
0.95
0.97
0.97
Cons.
0.14
0.13
0.00
0.17
0.23
0.19
0.20
Avg. NRG
71.84
72.87
73.64
66.18
75.19
79.46
78.35
77.75
Acc.
60.64
72.49
80.32
40.76
73.30
85,94
.6
Plau.
0.49
0.59
0.73
0.74
OPENBOOKOA
Div.
0.87
0.84
0.95
0.97
0.99
0,99
Cons.
0.05
0.13
0.16
0.25
0.09
Avg. NRG
62.29
68.00
75.33
63.07
75.33
80.36
68.64
68.29
Acc.
26.13
48.24
61.81
17.59
36.18
74.37
55.28
54.27
Plau.
0.51
0.65
0.62
0.68
0.76
NUMERSENSE
Div.
0.97
0.99
Cons.
0.03
0.36
Avg. NRG
56.41
66.19
74.83
59.40
67.80
80.84
69.95
69.44
Acc.
61.02
70.63
74.84
24.19
75.59
80.24
60.15
59.61
Plau.
0.44
0.55
0.59
0.71
0.75
0.47
0.47
QASC
Div.
0.78
0.63
0.89
0.74
0.98
0.97
0.99
0,99
Cons. 0.23
0.32
0.37
0.10
0.31
0.38
Avg. NRG
61.13
63.66
73.84
$3.05
77.52
80.31
66.41
66.28
that the annotators have to work on. These tasks are hand curated that cater certain parts of instruction whether the annotators are reading the rationale correctly, whether they are able to make appropriate connections between the rationale and the question. This weeds out lot of annotators who do not understand the task are cheating. We also weed out workers who too 'fast' (completing the task less than 5 seconds, which is indicative of potential slacking in the task). Third, constantly monitor task responses and feedback provided to annotators about their task. We also collect feedback from them which we adapt in new versions of the task.
The final MTurk instructions and template that land upon after the qualifications is shown in Figure 6 and 7
K LIMITATIONS
MARIO demonstrates promising improvements on the self-rationalization capability of small lan- guage models; note that using MARIO small LM like T5-LARGE leads considerable bridging of the gap between the quality of its rationales versus the quality of rationales generated by much larger language models. However, note that the results are still very much dependent on the initially available data (since we heavily depend upon silver standard rationales generated by GPT-3 to give our model a warm start). Our method is also dependent upon the mathematical rewards that we use: we discuss in Section [] this is a very new and active area of research, and we as research community are still figuring out what properties we need, and how to efficiently implement good mathematical metric for them.
24

---

Published as a conference paper at ICLR 2024
Table 11: The complete prompt of rationalization for STRATEGYQA. Demonstration examples are collected from Wei et al., 2022
Q: Do hamsters provide food for any animals?
Hamsters are prey animals. Prey animals provide food for predators.
A: So the answer is yes.
Could Brooke Shields succeed at University of Pennsylvania?
Brooke Shields graduated from Princeton University. Princeton is ranked as the number na- tional college by US news. University of Pennsylvania is ranked as number 6 national college by US news. Princeton only admits around percent of applicants as of 2018. University of Pennsylvania accepts around 9% of applicants as of 2018. A: So the answer yes.
Q: Yes or no: Hydrogen's atomic number squared exceeds number of Spice Girls? Hydrogen is the first element and has atomic number of one. To square number, you mul- tiply it by itself. The Spice Girls has five members. A: So the answer no.
Q: Yes or no: Is it common to see frost during some college commencements? College commencement ceremonies often happen during the months of December, May, and sometimes June. Frost isn't tt uncommon see during the month of December, is the win-
A: So the answer is yes.
Q: Yes or no: Could llama birth twice during War in Vietnam (1945-46)?
The War in Vietnam (1945-46) lasted around 6 months. The gestation period for llama is months.
A: So the answer is no.
Q: Yes no: Would pear sink in water?
The density of pear is about 0.59 g/em3. The density of water is about I g/cm3, Objects only sink if they are denser than the surrounding fluid.
A: So the answer is no.
25

---

Published as a conference paper at ICLR 2024
Table 12: The complete prompt of rationalization for QUAREL. Demonstration examples are col-
lected from Li al. 2023a
Q: Mike was snowboarding on the snow and hit piece of ice. He went much faster on the ice because ---- is smoother.
(A) snow (B) ice
A: When something is smoother, it is easier to slide on. Thus, he could go faster on the ice
because ice is smoother, So the answer is: (B)
Q: could hear then boy that say close to me clear as day, however could not hear the young lady sitting in the back of the room. Who am I able hear louder
(A) Boy (B) Lady
A: When someone close, is easier hear them. also could hear the young lady well.
Thus, am able hear the boy louder. So the answer is: (A)
Q: watched the snowflakes go from tiny specks in the sky nice size once they fell on my
face. When did the snowflakes seem bigger?
(A) in the sky (B) on my face
A: When something closer, it seems bigger. The snowflakes are closer when they are my face. Thus, they seem bigger when they are my face. So the answer is: (B)
Q: When Tammy tried slide the glass mixing bowl down the marble counter top to her mom,
it came to dead stop when it reached the wooden cutting board. The bowl came to a stop because the wooden cutting board has
(A) more resistance or (B) less resistance
When something has more resistance, it is harder slide. Thus, the bowl came to a stop because the wooden cutting board has more resistance. the answer is: (A)
Q: Sarah walked through the city and saw tourist attraction she wanted to visit. She had several blocks to go to get it, and the attraction looked very small. As she got close to it though, it towered over her. This is because when she was close to the attraction looked (A) much bigger (B) much smaller
A:
When something is closer, it looks bigger. Thus, the attraction looked much bigger when
she was close to it. So the answer is: (A)
26

---

Published as a conference paper at ICLR 2024
Table 13: The complete prompt of rationalization for OPENBOOKQA. Demonstration examples are collected from Wang al. 2022
Q: The sun is responsible for
(a) puppies learning new tricks (b) children growing up and getting old (c) flowers wilting in a vase (d) plants sprouting, blooming and wilting
A: plant requires sunlight for photosynthesis, which accumulates resources required for sprouting, blooming, and wilting, So the answer is: (d)
Q: When standing miles away from Mount Rushmore
(a) the mountains seem very close (b) the mountains are boring (c) the mountains look the same as from up close (d) the mountains seem smaller than in photographs
A: When an object far away, takes up less your field of view, and so seems smaller than in the photographs. So the answer (d)
Q: When food is reduced in the stomach
(a) the mind needs time to digest (b) take second digest what said (c) nutrients are being deconstructed (d) reader's digest is a body of works
A: The stomach part of the digestive system. The breaking down of food into nutrients in the digestive system. So the answer is: (c)
Poison causes harm which the following?
(a) Tree (b) a robot (c) a house (d) car
A: A tree a living thing. Poison causes harm to living things. So the answer is: (a)
Q: A magnet will stick to
belt buckle (b) wooden table (c) a plastic cup (d) paper plate
A: A belt buckle is made of metal. If a magnet is attracted to metal, then that magnet will
stick to that metal. So the answer is: (a)
Q: Deer are less safe in the woods because wolves
(a) have fur (b) howl (c) have claws (d) have tails
A: Claws used wolves to catch prey like deer. So the answer is: (c)
Q: An electric car causes
(a) more CO2 emissions (b) equal CO2 emissions (c) electric emissions (d) less CO2 emissions
A: An electric car uses less gasoline than regular car and thus causes less CO2 emissions. So
the answer is: (d)
27

---

Published as a conference paper at ICLR 2024
Table 14: The complete prompt of rationalization for NumerSense. Demonstration examples collected from Liu et al.) 2022
Q: penguins have <mask> wings.
(A) no (B) zero (C) one (D) two (E) three (F) four (G) five (H) six (I) seven (J) eight (K) nine
(L)
A: Birds have two wings. Penguin is a kind of bird. So the answer is (D).
Q: parallelogram has <mask> sides.
(A) no (B) zero (C) one (D) two (E) three (F) four (G) five (H) six (l) seven (J) eight (K) nine
(L.
A: A rectangular is a parallelogram. A square is parallelogram. So the answer is (F).
Q: there are <mask> feet in yard.
(A) no (B) zero (C) one (D) two (E) three (F) four (G) five (H) six (I) seven (J) eight (K) nine
(L) ten
A: A yard three feet. So the answer is (E).
Q: water can exist in <mask>> states.
(A) (B) zero (C) one (D) two (E) three (F) four (G) five (H) six (I) seven (J) eight (K) nine
(L) ten
A: There states for matter are solid, liquid, and gas, So the answer (E).
Q: a typical human being has <mask> limbs.
(A) (B) zero (C) one (I) two (E) three (F) four (G) five (H) six (I) seven (J) eight (K) nine
(L) ten
A: Human has arms and legs, So the answer is (F)
28

---

Published as a conference paper at ICLR 2024
Table 15: The complete prompt of rationalization for QASC. Demonstration examples are collected
from Wang et al. 2223a
How do you reduce pollution? (A) igniting fuel and oxidiser (B) transportation technology
(C) wasting (D) not recycling (E) burning fossil fuels (F) converting electricity to heat (G)
water conservation (H) using less resources
A: Conserving resources has a positive impact on the environment. Use of resources affects
the environment such as pollution. So the answer is: (H)
Q: what will move to another area if their habitat will no longer support them? (A) density (B)
Birds (C) squids (D) humans (E) clouds (F) gravity (G) cows (H) Whales A: If habitat
longer support animals then those animals will move to another area. Cows social animals. So the answer is: (G)
Q: With the exception of allergies, what may cause person seek medical attention? (A) Contact with latex (B) tree falling (C) Organs within the body. (D) Contact with baby chicks (E) prolactin release (F) Contact with peanut butter (G) hypothyroidism (H) Contact with mi- croorganisms
A: Microorganisms can cause infections. Infections usually require medical treatment, So the answer is: (H)
Q: Lavender can induce (A) healing (B) energy (C) hormones (D) mutations (E) Heart rate (F) growth (G) symptoms (H) warmth A: Healing requires rest. Lavender induces restful sleep. So the answer is: (A)
Q: what state a liquid in when frozen? (A) vapor (B) dense (C) gas (D) cooled (E) steam (F)
solid (G) boiling (H) cold
A: Freezing means changing from a liquid into a solid by reducing heat energy. Liquids freeze when they change to the solid state. So the answer is: (F)
Q: what unites to form a diploid zygote? (A) plant reproduction (B) Most plants (C) orchids
(D) sperm and ova (E) salt and pepper (F) predator and prey (G) honeybees (H) diploids and zygotes
Gametes then unite in fertilization and form diploid zygote. Collectively, the sperm and the ova are also referred as gametes. So the answer (D)
Q: What absorbs all visible light? (A) apples (B) coal (C) Green (D) coral (E) skin (F) bamboo (G) glass (H) eyes
A: If an object is black then that object absorbs all visible light. Light grains are quartz, Black grains are coal. So the answer is: (B)
29

---

Published as a conference paper at ICLR 2024
Instructions (click to expand/collapse)
Main Instructions
this HIT, you will read a Question and two Explanations that help you answer the question. These questions can be asking certain fact or about real life commonsense. Later, you will also be shown the correct answer to this question. You will be answering a question about your preference between the two explanations, along with follow-up questions.
Your Task
Given the question and two explanations, you then need to answer the following question:
Explanation Preference: Which explanation do you prefer for the above question? For explanations that are equivalent the exact same, you can prefer both as well.
You will then be shown the correct answer to the above question. You then need to answer the following followup questions:
Support: Which of the explanation(s) support the correct answer?
You need to determine if Explanation 1, 2, Both or None of them support the
correct answer. By support, we mean that you can arrive at the correct answer via the explanation. See example 1.
Repeating: Which of the explanation(s) repeats over and over?
Sometimes, explanations might repeat the same words and concepts again and again. See example 2.
Logicality: Which of the explanation(s) make logical sense?
Here, we mean that you should check if the explanations make logical sense, on their own. not try to determine if the explanation is correct or supports the correct answer. Apply commonsense or general knowledge to answer this. See example 3.
Figure 6: MTurk Instructions. We show these instructions to turkers, along with sample HIT, and more examples that contain special cases of each of the annotation questions.
30

---

Published as a conference paper at ICLR 2024
Read through the Instructions and Examples before protreding
IMPORTANT:
Question
There is most liely yoing be fog aroundl lel marsh doi landna plans resert
Explanation
Fag usualy associated with snow.
Explanation 2:
Fig a byer of wowe mhst fo stwopporeoeeeeeeoooooooooooo ween the airic end and
Explanation Preference: 2r
Which expionation degtu forthe
Expltration
Bosh
Exploration
New note that the conect arouee iha quadionis (e).
Support: 1 or 2 or Both None
Whith the expanationoo support the correct snower?
Exp pnnatorir
O
Ep anatior
Nome
Repeating: 1 or 2 Both None
Whrch the explaationni has repaatng worth owr over
Pop anstirr
O
onation
None
Logicality: Both or None
Wh the siptonationgo make
Exp anatior
Both
Expimnttor
Figure 7: MTurk Template. Given a question and two explanations, we ask annotators to choose which explanation they prefer, followed by questions about their plausibility and consistency.
31


Reviews:
[{"review_id": "d7OCsU7Rfo", "reviewer": "Reviewer_3RWN", "summary": "In this work, authors contribute to the task of rationale’s generations in question answering tasks. In previous work, rationalizers are at significant scale,  and ignored the semantics of the rationales themselves. In this work, authors invented the MARIO algorithm for much smaller scale LM's to generate higher quality rationales, with multiple rewards for different quality properties of generated text rationales. Besides, generations from the algorithms seem to be more perferred for human experts.", "questions": "In the paper, the authors mentioned that there is an initial supervision phase where GPT-3 provides the generation labels. I’m wondering what’s the relationship between this initial supervision process and the distillation of GPT-3? Is it before, after, or exactly the distillation process?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper is well-written. The novelty and contribution is clear to me. The authors try not to take advantage of the scalability of large language models and instead use a much smaller distilled version of GPT.  Furthermore, the rewards’ design is aiming at generating rationales with better semantic qualities rather than scoring better at the specific downstream task. I think this is a really good design philosophy for training algorithms.", "weaknesses": "I believe the improvement of two versions of Marios compared to baselines is not really significant, considering that all the baseline models have equal number of parameters with the Mario agent model. I’m wondering whether the extra efforts on training on multiple rewards are indeed worth it to improve the generations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lVDtQj8Dk7", "reviewer": "Reviewer_A7zX", "summary": "This paper introduces MARIO, a multi-reward approach that enhances the self-rationalization quality of small LMs and the performance of downstream tasks. The authors conducted experiments on three QA datasets and compared the results with several baselines, finding that MARIO can effectively train small LMs to generate rationales, which satisfy multiple distinct properties such as plausibility, diversity, and consistency, while also improving the performance of QA tasks. Additionally, human evaluation was carried out, confirming that the rationales generated by MARIO are more preferred by humans compared to the baselines. Furthermore, the paper discusses the importance of selecting appropriate rewards and preventing MULTI-REWARD HACKING.", "questions": "Although the model utilizes the quark framework, it should clearly present the learning objectives in a multi-reward scenario. Since quark is a single-reward algorithm, its objective function under the extension of multi-reward goals is not intuitive. Could you provide a more detailed and clear explanation of MARIO and the corresponding multi-reward loss?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The author presents an interesting and valuable research question, namely, how to enhance the self-rationalization quality of small LMs. Building upon the basis of quark, the paper effectively extends its application, utilizing multi-reward conditional generation to optimize both the rationale quality and the performance of downstream tasks. The article clearly explains the criteria for measuring three key aspects of a rationale's properties.", "weaknesses": "- The details of the MARIO algorithm are not adequately explained, such as how to determine the settings of control tokens, and the description of how to quantize samples under the quark framework is unclear (is it a comprehensive consideration of multiple attributes for ranking, or is it ranked based on a single attribute?).\n- The description of the MARIO method is overly simplistic, and it lacks the necessary explanation of the thought process behind the development of this method.\n- In relation to self-explaining rationalization, besides the generative rationales discussed in this paper, there is a series of extractive rationale works (such as Lei et al. 2016, Liu et al. 2023, and so on). Beyond the difference between generative and extractive approaches, the basic framework of these two types of work is very similar. Both require ensuring that the generated/extracted rationale is meaningful to humans while maintaining high performance in downstream tasks. Therefore, the related work section should also include this series of works.\n   - Lei et al. 2016, Rationalizing Neural Predictions, EMNLP-2016\n   - Liu et al. 2023, Decoupled Rationalization with Asymmetric Learning Rates, KDD-2023\n- Although Figure 1 and Figure 2 contain a considerable amount of text, the information conveyed is limited.\n- The experiment used multiple baselines, but in reality, it involves two baselines and their multi-reward forms of extension, lacking a comprehensive comparison with other works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2ZG1dv9fN3", "reviewer": "Reviewer_pMad", "summary": "In this paper, the authors presented MARIO, a method for training small LMs to generate rationales for question answering. The method is an extension to Quark. While Quark only allows optimizing towards one reward, MARIO allows learning from multiple-reward, allowing the LM to be trained with rewards for Plausibility, Consistency, Diversity and Task-correctness at the same time. The authors evaluated their method on 3 tasks: Strategy QA, QuaRel, and OpenBookQA, and the training rationales are sampled from InstructGPT outputs. The authors showed that their method outperformed baselines on similar-sized small LMs both on automated evaluation metrics and human preference, and can be comparable to some larger LMs on certain tasks.", "questions": "(1) Is there any way to rank/weight/balance different objectives in MARIO? (For example, if I found that the resulting model is weak in Consistency, is there a way to weigh Consistency reward a bit more in the training?)", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The authors tackle an important problem: rationale generation for question answering on small LMs. It is known that rationalization and chain-of-thought can work better on very large language models, but fine-tuning small LMs to correctly rationalize in question-answering tasks has been very challenging.\n\n2. The authors' proposed method allows learning towards multiple rewards, which can be very useful because often we want a model's generation to satisfy multiple desirable properties, and training towards a single property reward can often lead to complete loss of other desirable properties.\n\n3. The authors comprehensively evaluated their method against several baselines on similar-sized LMs, and showed that their method is superior on 3 different QA tasks, both on automated metrics and human preference.\n\n4. The presentation of the paper is generally quite clear. The figures and tables are very well made and they really help make the paper easier and better understood by the reader.", "weaknesses": "Overall this is a good paper. Below are a few weaknesses that prevented the paper from getting a \"10\":\n\n1. The paper's main contribution is extending an existing method (Quark) from single-reward to multiple rewards. So while the results are nice and the extension is valuable, the contribution is not revolutionary.\n\n2. While the description in text and Figure 2 are very helpful for readers to understand MARIO, the full picture of MARIO can still be a bit hard to grasp (especially to readers who are not already familiar with Quark). Maybe including an algorithm block for the entire training process will make the full picture a lot more clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mtZoviDhnh", "reviewer": "Reviewer_7guo", "summary": "This work studies the problem of generating plausible, consistent, and diverse rationales that also improve the downstream task performance while using small-scale LMs. The proposed method, called MARIO, is a multi-reward conditioned self-rationalization algorithm that optimizes multiple properties. Extensive evaluation show that produced rationales are plausible and improve performance.\n\nMario works a follow. First, it trains a small LM to self-rationalize, with the help of GPT-3. Then, it casts the problem into a multi-reward conditioned rationale generation problem. The key is to leverage a multi-reward setup, where the dimensions to optimize are plausibility, diversity, consistency, and task accuracy. Each reward is based on an automated metric. At the end, the propose method only extend Quark [Lu et al. 2022], which limits the novelty (the proposed method is 3 paragraphs).\n\nThe experiments focus on QA datasets. The two variants of Mario don't seem to significantly outperforms the baselines, Classical underperforms on StrategyQA and Mario on the others. However, the human evaluation show more promising results, showing that automated evaluations are not enough on their own. Other datasets than QA ones should be used to show the generalization of the proposed method. For example, beer, hotel, or amazon datasets. The analysis on reward hacking and optimizing solely on task performance is very interesting.\n\nOverall the work is clear, well-written, and well-structured. It is a bit weird that the related work is put in the appendix. I would highly encourage the authors to move it back into the main paper. My concerns remain the novelty of the method - that seems to extend QUARK for multi-reward steup - and the lack of datasets that are not QA, especially when we are talking about rationalization.\n\nPOST-REBUTTAL:\nThank you for your answers; I am willing to increase my score.", "questions": "- How would you adapt your method for other rationalization tasks that are not QA?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Good human evaluation and results\n- Interesting method to make self-rationalization work with small LMs", "weaknesses": "- Other datasets than QA ones should be used to show the generalization of the proposed method.\n- Small improvement in Table 2\n- Limited novelty", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7RvWEDKwgn", "reviewer": "Reviewer_CcB6", "summary": "The authors extend an existing LM control mechanism to work for controlling multiple attributes simultaneously, and following this mechanism they train T5-Large to do reasoning that is more plausible, non-repetitive, and consistent with the predicted answer. On some challenging QA tasks they get a small bump in accuracy, but a much bigger gain in human evaluations of the quality of the reasoning produced by the model to justify the answer.", "questions": "--Is there any particular reason to think additive or classic might be better than the other in any particular setting? Otherwise, it kind of seems like you're just giving yourself \"multiple tries\" at your benchmark, in some sense.\n\n--Is GPT3 = text-davinci-003 throughout?\n\n--does adding the finetuning for consistency fix the problems in 5.2? or is this not exactly the same?\n\n--the reward hacking discussion seems important and i'm glad you included it - there are a lot of potential hacks, e.g. a degenerate \"rationale\" could just be the answer itself or restating that in some way, right? do you get around this issue by starting with distillation so that the supervised rationales are initially reasonable, rather than doing e.g. some RL? i'm wondering how you would be able to go about this if you didn't have access to a much stronger model to distill from initially - it seems like so far you've only showed in a distillation-like setting, from GPT3 to a vastly smaller model. \n\n--nit: maybe also cite [1]?\n\n[1] Lightman, Hunter, et al. \"Let's Verify Step by Step.\" arXiv preprint arXiv:2305.20050 (2023).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "--tackles a challenging task (making LMs output reasoning for their answers that is more reliable on a couple of different axes) which is highly topical\n\n--impressive human eval results showing that their final rationales are much better than baselines'\n\n--interesting discussion of reward hacking", "weaknesses": "--Methodologically, this multi-attribute control setup seems to be a straightforward extension of Quark, and the idea of finetuning an LM to use tags to provide control i think is pretty well-explored in other work such as [1], and doing it for multi-attribute with different tags is used as a baseline in [2]. i feel that your main contribution is not so much the new algorithm in a general sense, but rather the interesting application (with convincing human eval results) to the important and currently relevant task of making LM reasoning more reliable.\n\n[1] Keskar, Nitish Shirish, et al. \"Ctrl: A conditional transformer language model for controllable generation.\" arXiv preprint arXiv:1909.05858 (2019).\n\n[2] Yang, Kevin, and Dan Klein. \"FUDGE: Controlled text generation with future discriminators.\" arXiv preprint arXiv:2104.05218 (2021).\n\n--some of the models you use to evaluate/filter the individual qualities, e.g. VERA, are way bigger than the model you're finetuning, which arguably gives you extra signal that the SFT baseline doesn't have? what was the data used for training those? on a related note, this raises some concerns that your method might not be scalable to larger LMs, unless we have e.g. an even larger version of VERA to provide supervision?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)