It Helps to Take a Second Opinion: Teaching Smaller LLMs To Deliberate Mutually via Selective Rationale Optimisation
Authors: Sohan Patnaik
Milan Aggarwal
Sumit Bhatia
Balaji Krishnamurthy
Venue: ICLR 2025 Poster (2025)
Keywords: Multi-LLM Deliberation
Smaller LLMs
Rationale Generation
Rationale Refinement
Selective Rationale Optimisation
Trainable
Task-Guided Rationale Selection
Primary area: foundation or frontier models, including LLMs
TL;DR: A trainable framework that facilitates interaction between two distinct Variants of the same LM to preferentially Generate and Refine better rationale choices guided by the end-task.

Abstract: Very large language models (LLMs) such as GPT-4 have shown the ability to handle complex tasks by generating and self-refining step-by-step rationales. Smaller language models (SLMs), typically with < 13B parameters, have been improved by using the data generated from very-large LMs through knowledge distillation. However, various practical constraints such as API costs, copyright, legal and ethical policies restrict using large (often opaque) models to train smaller models for commercial use. Limited success has been achieved at improving the ability of an SLM to explore the space of possible rationales and evaluate them by itself through self-deliberation. To address this, we propose COALITION, a trainable framework that facilitates interaction between two variants of the same SLM and trains them to generate and refine rationales optimized for the end-task. The variants exhibit different behaviors to produce a set of diverse candidate rationales during the generation and refinement steps. The model is then trained via Selective Rationale Optimization (SRO) to prefer generating rationale candidates that maximize the likelihood of producing the ground-truth answer. During inference, COALITION employs a controller to select the suitable variant for generating and refining the rationales. On five different datasets covering mathematical problems, commonsense reasoning, and natural language inference, COALITION outperforms several baselines by up to 5%. Our ablation studies reveal that cross-communication between the two variants performs better than using the single model to self-refine the rationales. We also demonstrate the applicability of COALITION for LMs of varying scales (4B to 14B parameters) and model families (Mistral, Llama, Qwen, Phi). We release the code for this work here.

Published as a conference paper at ICLR 2025
It Helps to Take a Second Opinion:
TEACHING SMALLER LLMS TO DELIBERATE MUTU- ALLY VIA SELECTIVE RATIONALE OPTIMISATION
'ohaan Patnaik, "Milan Aggarwal, Sumit Bhatia, & Balaji Krishnamurthy
Media and Data Science Research Lab, Adobe (soha, mi lagga,, sumit. baatii,, kbalaji @adobe com
ABSTRACT
Very large language models (LLMs) such as GPT-4 have shown the ability to han- dle complex tasks by generating and self-refining step-by-step rationales. Smaller language models (SLMs), typically with < 13B parameters, have been improved by using the data generated from very-large LMS through knowledge distillation. However, various practical constraints such as API costs, copyright, legal and eth- ical policies restrict using large (often opaque) models to train smaller models for commercial use. Limited success has been achieved at improving the ability of an SLM to explore the space of possible rationales and evaluate them by itself through self-deliberation. To address this, we propose COALITION, a trainable framework that facilitates interaction between two variants of the same SLM and trains them generate and refine rationales optimized for the end-task. The variants exhibit different behaviors to produce a set of diverse candidate ratio- nales during the generation and refinement steps. The model is then trained via Selective Rationale Optimization (SRO) to prefer generating rationale candidates that maximize the likelihood of producing the ground-truth answer. During infer- ence, COALITION employs a controller to select the suitable variant for gener- ating and refining the rationales. On five different datasets covering mathemati- cal problems, commonsense reasoning, and natural language inference, COALI- TION outperforms several baselines by up to 5%. Our ablation studies reveal that cossunicmmuiciioonn between the two variants performs better than using the single model to self-refine the rationales. We also demonstrate the applicability of COALITION for LMs of varying scales (4B 14B parameters) and model families (Mistral, Llama, Qwen, Phi). We release the code for this work here.
INTRODUCTION
Modern large language models (LLMs) with hundreds of billions of parameters, such as GPT- 4 (Achiam et al., 2023) and PaLM-540B (Chowdhery et al., 2022) have shown remarkable ability to solve complex tasks by generating step-by-step rationales (Wei et al., 2022a;b; Kojima et al., 2022) and refining them through self-correction (Wang al., 2023b; Welleck et al., 2023). The ability to think step-by-step becomes more prominent with scale, while smaller language models (SLMs), typically E 13B, struggle generate good quality rationales (Valmeekam et al., 2022; Weng et al., 2023). However, owing to the advantages of SLMs such as lesser costs, latency, and compute requirements, significant efforts have been made to improve their ability to handle complex tasks by using feedback obtained through interactions with LLMs (Tunstall et al., 2023; Hsich et al., 2023; Gou al., 2024; Wang al., 2024b). While such approaches are suitable for research and academic settings, lack of transparency in the training data of larger (often opaque) LMs limits their use in commercial settings owing to legal, ethical and copyright concerns. For instance, OpenAl's usage terms prohibit using GPT-generated outputs to train other models for commercial use.
Consequently, efforts have been made to improve SLM performance without reliance on an external teacher L.M. In the absence of supervision from external models, the SLM has to rely on its intrinsic
cqual contribution

---

Published as a conference paper at ICLR 2025
LM Variant
LM Verians
Select Vertent Generrter Fastonate
Selee foctart
RRetine
Final Anewer
Controller
Seloced
Generated Rationale
Conteoterr
Veriant Selecter
Relined Rationate
Ingun
Figure I: Schematic flow of inference using COALITION which leverages two variants of the same LM. The sample is fed to a controller (step 1) to select the variant (steps 2-3) that generates a rationale (step 4). The generated rationale is then fed to the controller to select the variant (steps 5-6) to refine the rationale (step 7) that can be used to obtain the final answer (step 8).
knowledge to explore and refine (Madaan et al., 2023a) the space of possible reasoning paths (Hao al., 2023). However, due limited scale and exploration capabilities, small models get trapped in redundant reasoning paths (Valmeekam et al., 2023; Qi et al., 2024). Furthermore, commonly used techniques such as prompt-based cross-communication between multiple LLMS for iteratively refining diverse reasoning paths (Mousavi et al., 2023; Yin et al., 2023) or using LM-as-a-juus paradigm to identify high-quality rationales to facilitate iterative refinement of rationales (Yuan et al., 2024), fail to generalize in case of smaller models (as we show empirically in $ 4). While training the SLMs using susk-specific ground-truth (GT) rationales has shown promise (Chen et al., 2024), the lack of availability of GT rationales for given task limits the applicability of such methods.
With this background. the key problem that we study is how to train the SLMs without relying on external LLMs and task-specific GT rationales to (i) generate (and, refine) diverse rationales, and (ii) select the high-quality rationales leading to improved performance on end-tasks. We posit that the following two abilities (A) critical to achieve this (A1) ability obtain distinct rationale didates describing varied reasoning paths and diverse opinions about how to refine them; and (A2) ability discriminate high-quality rationales from the low-quality ones enable the model to prefer generating candidates which are more useful. Driven by this intuition, we propose COALITION (TeaChing LLMs to DeliberAte MutuaLly via SelectIve RaTlonale OptimisatiON), a trainable framework that facilitates interaction between two distinct variants of the same SLM to learn to selectively Generate and Refine better rationale choices guided by the performance of the end-task.
The key intuition in COALITION is to overcome the limited ability of SLMs to generate diverse and high-quality rationales by employing different variants of the same LM that are designed exhibit distinct behavior by optimising them on separate data splits ($ 3.1). During training, different ratio- nales generated by the variants are further refined by each variant through self and cross-refinement. The generated and refined rationale candidates are assigned utility score by estimating the likeli- hood of generating the final GT answer conditioned on the rationale in input. The LM variants are then tuned via preference optimization (DPO) (Rafailov al., 2023) to prefer generating rationale candidates with higher utility scores ($ 3.2). A typical inference step in COALITION is illustrated in Figure where trained controller module ($ 3.3) employed for selecting the appropriate model variant for generating and refining the rationale before using it for answer generation.
Empirically, demonstrate the effectiveness of COALITION across five different datasets cov- ering mathematics problem-solving (GSM8K (Cobbe et al., 2021)), natural language inference (PIQA (Bisk al., 2020) and WinoGrande (Sakaguchi et al., 2020)) and commonsense reason- ing (CSQA (Talmor et al., 2019) and HellaSwag (Zellers et al., 2019)). Using Llama3-8B as the base model and without any supervision from external stronger models, COALITION leads to ab- solute gains of up to 5% over several recent baselines ($ 4.1). We also demonstrate the efficacy of COALITION for different language model families such as Phi3, Qwen 1.5, and Mistral across vary- ing parameter-scales ranging from 4B to 14B ($ 4.2). We present results that offer evidence that cross-communicationn between the two variants performs better than always using a single model to self-refine the rationales (S 4.3). Finally,
conduct extensive ablation studies to guide vari- ous design choices such as 1) the use of distinct model variants to obtain diverse rationales over sampling-based decoding through single model and 2) task-guided rationale selection ($ 4.4).
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Prompt-Driven Reasoning Generation: Very large-scale LLMss have been made to elicit reasoning chains by asking to generate step-by-step rationales via Chain-of-Thought (CoT) prompting (Wei et al., 2022a; Kojima et al., 2022; Wei et al., 2022b). Subsequently, several works have attempted to generate better reasoning through in-context learning (Li & Qiu, 2023a;b) by improving the quality of exemplar rationales in the prompt (Zhang et al., 2023; Diao et al., 2024). On the other hand, self- correction methods (Madaan et al.. 2023a) prompt the LLM to iteratively refine its rationales using its own feedback (Welleck et al., 2023; Wang et al., 2023a). However, it has been shown that LLMs are unable to revise their own outputs without external feedback (Jiang et al., 2024b; Valmeekam et al., 2023; Stechly al., 2023; Huang al., 2024) owing to the fact that using the same internal representations for refinement yields redundant or incorrect reasoning paths (Yin et al., 2023).
Performance Enhancement using External LLMs: Various methods have explored improving an LLM by facilitating interaction with other LLMs (Jiang et al., 2023b; Yu et al., 2024; Juneja et al., 2023: Ulmer et al., 2024; Lu et al.. 2024b). Exchange-of- Thought (EoT) (Yin et al., 2023) mimic the way humans conduct discussions by enabling multiple LLMs to critic (Mousavi et al.. 2023) and refine each other's 's outputs via prompting. Our experiments show that such methods work well only with larger LLMs. Other methods distil information from larger LM into a smaller one (Hsieh et al., 2023; Kang et al., 2023) or personalise the feedback of teacher LLM based on weaknesses of student LLM (Wang& & Li, 2023; Saha et al., 2023; Jiang et al., 2023c). However, it is often argued that training over GPT-generated outputs makes smaller LLM imitate just the style (Gudibande et al., 2024) but not learn the reasoning process (Mukherjee et al., 2023). Some methods train the LLM to prefer generating certain outputs (Zhang et al., 2024a) over others via preference optimisation (DPO) (Rafailov et al., 2023). Mixture-of-Agents (Wang al., 2024a) employs multiple open- source LLMs based agents and comprises of multiple layers of such L.LM agents such that responses generated by agents layer are fed to LLM agents in the subsequent layer to refine the output. COALITION creates multiple variants of same SLM without involving any external LLM.
Improving LLM Rationales through Self-Play: Some works improve an LLM by using it to ex- plore the reasoning space and discriminate between outputs (Qu et al., 2024; Tian et al., 2024) by itself. Tree-of- -heoughh (ToT) (Yao et al., 2023) organises candidates for each intermediate reasoning step in the form of tree to look-ahead to gauge the quality of initial steps and backtrack accord- ingly. Zhang et al. (2024b) tunes the I.LM on candidates obtained using ToT prompting through DPO. Other methods leverage sampling- based decoding (Wang et al., 2024d; Zhang et al., 2024a; Pang et al., 2024) to generate varied outputs. Such works rely on the scale of very-large LLMs to obtain diverse responses and fail to generalise well using smaller LLMs (as shown in experiments). Likewise, other works use very-large LLMs to rate the quality of different candidates for preference optimisation (Yuan et al., 2024; Pang et al., 2024) via -LM-as-a-judge (Zheng et al., 2023). How- ever, SCORE (Zhang et al., 2024c) shows that smaller LMs need superior LLMs to verify responses for correction (Jiang et al., 2024b). On contrary, leverage LLM's likelihood of generating final ground-truth answer conditioned on the rationale as a measure of its quality (Wang et al., 2024c). Some methods align LLM's output distribution with human-labelled data (Lai al., 2024). SPIN performs DPO by selecting the ground-truth answer over the LL --Meenerrteedd response (Chen et al., 2024). Lack of availability of GT rationales for given task limits their applicability. Differently, we employ different variants of the same LLM to generate and refine diverse
d rationales for selection.
METHODOLOGY
Overview of Approach: COALITION aims improve the ability of an SLM M to generate and refine reasoning chains for a given task without relying on any external model. To endow M with the capability to generate rationales for any input instruction, we first instruction fine-tune (IFT) M using the Chain-of Thouughht (CoT) (Kim et al., 2023) dataset. The CoT data comprises general- domain instruction-raaionale-nsss answer triples. Further, we augment the IFT data with rationale refine- ment samples where the model is tuned to generate the rationale in the CoT data sample given the model-generated rationale in the input. This augmentation teaches the model to generate and refine rationales. Recall from the earlier discussion ($ 1) that a typical SLM has a limited capability to explore and self-correct diverse reasoning paths. Hence, we create two distinct variants of the same SLM by carrying out the IFT separate data splits so that they exhibit different behaviors ($ 3.1).
3

---

triples. We format Published as a conference paper at ICLR 2025
Selective Rationale Optimisation (SRO)
LLM Veriands
Generate
Parionale
Winner fldarale
Taok Instraction
Indrccerr
Madel
Compure
Ri
Disect Peeeeence Optimiatiion
LLM Mansr122
Ga
Marr
Ciu
Ratlonale-ooddliinne GT Answer Likallood
Elininaked Hallones
X
Reñine
Inputs provided and outputs obtained time
--
Ratorsin
Teak- Intrructio
Selective Rationale Optimisation (SRO)
Varant2
Figure 2: Training procedure of COALITION through Selective Rationale Optimisation (SRO). The task-instruction fed to the two LLM variants (step 1) to generate different rationale candidates (R( The IFT model MIFT is used to score each candidate by estimating the likelihood (p)) of generating the ground-truth (GT) answer conditioned on the rationale (steps 3-4). The score used compare the rationales determine the winning and the eliminated rationale candidates (step 5) which are used to tune the LLM through DPO (step 6). During the refine stage, generated rationale candidate (RC is fed to both the variants to refine the rationale (step 7). The corre- sponding refined rationale candidates (R
Ref R42 are used to tune the model via SRO (step 8).
Subsequently, given an end-task without explicit rationale annotations, COALITION facilitates in- teraction between the two variants to generate and refine the rationales. The resulting set of diverse rationale candidates are then used for task-guided Selective Rationale Optimization (SRO) (S 3.2) to tune the variants to prefer rationales that can lead to an improved end-task performance.
Figure presents the details of various components and steps involved in COALITION illustrating rationale generation, refinement, and tuning of LM variants via SRO. Specifically, given a task- specific instruction sample, it is fed each variant of the LLM separately with the prompt to gener- ate a rationale describing the steps to derive the final answer (steps 1-2 in fig. 2). Subsequently, the rationales generated by each variant are fed again both the variants for self-refinement and cross- refinement. Owing to the distinct behavior of the variants, we obtain set of diverse rationales at both generate and refine steps. A utility score assigned to each rationale candidate by estimating the likelihood of generating the GT answer by the IFT model conditioned on the rationale in input (steps 3-4 in fig. 2). The candidates are ranked based on the utility score for tuning the variant LMs via DPO (Rafailov al., 2023) (A.1) to prefer output those generated and refined rationale choices with higher scores (steps 5-6 in fig. 2). Note that the rationale candidates for generate and refine steps are ranked and used separately for DPO training (steps 7-8 in fig. 2). We now describe the details of each component of COALITION in the subsections to follow.
3.1 MULTI-MODE INSTRUCTION FINE-TUNING (IFT) TO OBTAIN MODEL VARIANTS
Conventional IFT aims enabling an LLM follow given instruction generate an answer cordingly. However, as outlined in the approach overview, we require the IFT model MIFT for three additional purposes- 1) generate rationale describing how to derive the final answer given the struction (I - R); 2) refine a rationale to improve its quality for a given instruction ([;; R'] + R); and 3) generate/estimate the likelihood of producing an answer given the instruction and rationale as input (I: R.) + A). To enable MIrr to perform these three additional roles, we leverage a dataset D1F1 Drationale which comprises of samples containing enstruation-rationll-aaaa
the samples using different prompt templates (A.3) to indicate the model about the mode in which needs to generate the output. The model endowed with the ability to refine rationale by tuning it to generate the rationale in the dataset sample (R) given the LLM-generaie
rationale (R') in the input. Formally, given an instruction I, rationale R and final answer A in a sample, we perform IFT on four types of samples via cross-entropy loss and teacher forcing (Vaswani et al., 2017),
4

---

Published as a conference paper at ICLR 2025
LI- = log P(Ra ( (1TT)
(1)
C 11 = -log p(Ral [PI:: R2; R.cl. 0IFT)
(2)
C [IZRR|AA = -log p(Acl [P :i:: Rct 0IFT)
(3)
LT = logg p(Acl [Pr-A:ZiAA:l fIFT)
(4)
where, p represents probability, Ar and R. depict the 4 token in GT answer and rationale respec- tively, < indicates tokens before nhh index, Pm and Lm are the prompt format and loss function respectively for the m mode; [:] represents the operation to prepare LLM imput after arranging the instruction, answer and/or rationale into mode-specific prompt Pm. and 0IFT is LLM parameters. For samples in IFT data mix which do not contain the rationales, only loss LI-+A is applied. MIFT obtained by training base LLM entire IFT data. To obtain two LLM Variants (LV1, LV2), the LLM M is tuned on separate data splits by randomly dividing IFT dataset into two equal splits and assigning one split each variant randomly. For an end-task without rationale annotations, variants are used to generate and refine diverse rationales in task-guided manner as discussed subsequently.
3.2 TASK- GUIDED SELECTIVE RATIONALE OPTIMISATION (SRO)
For given end-task T, we denote the corresponding dataset as D which comprises instruction- answer pairs of the form (I" .A"), ). However, note that the dataset for the given task does not contain the rationale annotations to tune the LLM. To address this, we leverage the distinct LLM variants to construct set of diverse rationales via Generate and Refine steps. Given a task- instrinstruction, is given as input to each variant separately to generate a rationale. Each generated rationale is then fed to both the variants for self-refinement and cross-refinement. Quality of rationales obtained each generate and refine step is determined based on its usefulness to enhance end-task performance i.c. likelihood of generating the ground-truse -ruth answer A", Each variant is then optimised prefer generating better rationale candidates via Direct Preference Optimisation (Rafailov et al., 2023). The variant LLMs are tuned for numerous iterations by conducting multiple passes over DT
Generate:
We denote the distinct variants obtained through IFT LV (p E {1,2)). In particular, consider the 1th iteration E {1, 2)) such that the task iusk-inttuuction I is given as input to each variant (V((-1) (obtained till last (i 1)th iteration) to generate rationale (steps 1-2 in fig. 2):
RCm =LV(i (PT-MR; (vve-)))) (1,2)
(5)
Refine: Each generated rationale Reen (p {1,2}) is then fed as input to each variant LV(i-1) (a E (1,2}) to refine the quality of the rationale shown in following equation:
R R.Re (.CCCVI (PIZRR)-R ; RGen]
evgen)):P. [1,2}
(6)
where, Ref the refined rationale produced by feeding the rationale generated by the pt variant (during the generate step) to the 9th variant for refinement. The case when a variant refines its own self-refinmment (p- q). Likewise, when variant refines the rationale
rationale is referred generated by the other variant, it is referred to as cross-refinement (p + q). Thus, we obtain a set :P ;p, {1,2} rationale candidates. Each rationale candidate (R) is assigned a utility score i (steps 3-4 fig. 2) by estimating the likelihood of generating the ground-truth (GT) answer A by the IFT model MiFT conditioned on the rationale in input as: i = &0 (AT IPIZ. RR). Based the utility score, the rationale candidates from generate and refine steps are ranked separately and used to tune the variants via DPO training.
of generated (176,em; E {1,2}}) and refined ([TRBe
:T;
Direct Preference Optimisation (DPO): To maintain the distinctness in the behaviour of LLM variants, we tune them on separate data splits DT by dividing the samples into two equal partitions and randomly assign one partition each variant. Without loss of generality, consider (I* .A")
5

---

Published as a conference paper at ICLR 2025
DE (split assigned to hth variant LV:). Two types of samples used to tune the variant via DPO (one sample corresponding to each generate and refine steps). For the generate step, we compare the rationales R (p E (1, based on their utility score 1p such that the candidate with higher score is selected as the winner rationale (Rear) and the one with lower score is referred as the
eliminated rationale (RG The hth variant LV is tuned (in current iteration i) to prefer the &
winner rationale over the climinated one using the DPO loss C (steps 5-6 in fig. 2): LV(
T cyt (R
T Lv( (R
GGen LV(
-logo|
log " cvy 1) (RCen)
7 "cyle (R
where, 3 0.1 is a hyper-parameter to control divergence from reference model. The previous iteration (i 1) version of the variant LV(t-1) is used as the reference to obtain LV(() For the refine step, we consider the candidates obtained by refining the rationale generated by the t variant in the first turn i.e. R (q E {1,2 We compare them based on their utility score to identify the winner and eliminated rationales for the refine step as R.ee and Ref respectively. They are used Ref (steps 7-8 in fig. 2):
to train the tt variant via DPO using the following loss L 2y10
cv! (R cv (R Reff
log cv (R
T
Ref -logor[3(log cvpl
(8)
(R Re
"v
Likelihood -baed Sample Filtration: To ensure that high-quality samples are used to tune the vari- ants for DPO training, we apply filtration criteria to retain only those samples where the winning rationale (R... RGen /R enhances the likelihood of generating the GT follows:
FF (A" PPZ Rwol) TRIFT (A
Equation 9 compares the likelihood of generating the ground-truth by MIFT for a given task in- struction in the absence and presence of the winning rationale in the input. The sample is used for DPO training if the winning rationale enhances the likelihood compared to not using any rationale.
3.3 CONTROLLER-BASEE LLM VARIANT SELECTION DURING INFERENCE
Given an instruction sample during inference, COALITION employs Controller module to choose the variant LLM that should be used for the generate and refine steps. The controller is small encoder-only LM that is trained using the preference data collected during the DPO training based on which variant's 's rationale procedure of the controller. Given the instruction sample I", consider the generate step such that the controller is trained using cross-entropy loss perform two-way classification between the variants conditioned on IT as input. The output label is determined as the variant which generated the winning rationale RGen Likewise, corresponding to the refine step. controller C is conditioned on the instruction IT along with the winning rationale generated the generate step as the input. is trained to predict the variant that generates the winning refined rationale Re amongst R. (q E {1, 2)).. Once trained, is used during inference to select the variant to generate rationale followed by choosing the variant for refinement conditioned on rationale obtained at generate step.
selected. Figure 3 shows schematic diagram depicting training
EXPERIMENTS AND EVALUATION
Datasets: We evaluate COALITION using five datasets belonging to three diverse reasoning task
domains 1) Maths Problem Solving on GSM8K (Cobbe et al., 2021); 2) Natural Language
Inference (NLD) using PIQA (Bisk et al., 2020) and WinoGrande (Sakaguchi al., 2020); and

---

Published as a conference paper at ICLR 2025
X
Controller
LLM Varient
Rationale
Generated Rationale
LLM Variant
Refine
Refined Rationale
Instruction Sample
Generate
LLM Variant
Variant
Rationale
Input-Output to Train Controller
Controlier
Figure 3: Training process of Controller C. Each LLM variant generates a rationale candidate (step
1). The variant that generates the winning rationale (step 2) is selected the output label for C. (steps 3-4). For the refine step, C' is conditioned on the task instruction and rationale from the generate step, and is trained to select the LLM variant that generates the better refined rationale (steps 5-8).
3) Commonsense Reasoning through CSQA (Talmor al., 2019) and HellaSwag (Zellers al., 2019). GSMSK comprises maths word problems which require a model to understand the prob- lem text and perform sequence of calculations. PIQA requires understanding of physical relation between objects and comprises of samples with a goal text coupled with two candidate statements with the task of identifying the statement that can lead to the goal. WinoGrande lenging co-reference resolution task, comprising of a statement with two parts such that the latter half refers to some entity in the first part. CSQA tests model's ability answer MCQ questions by picking correct choice using commonsense knowledge. HellaSwag evaluates ability to predict continuation of context by choosing most plausible ending. Number of samples train/test splits of each dataset is GSM8K 7.5k/1.33, PIQA 16k/3k, WinoGrande 40k/1.7k, CSQA 9.7k/1.2k and HellaSwag- 39.9k/ 90kk. Please refer to A.2 for examples present in each dataset.
very chal-
Implementation Details: We use batch size (BS) of 16 8 80GB A000 GPUs (BS of 2/GPU), learning rate (Ir) of 1e-5, bfloat16 precision with cosine annealing (Loshchilov & Hutter, 2017) us- ing AdamW optimizer (Loshchilov & Hutter, 2019). We leverage DeepSpeed Zero 2 with sharding of optimizer states and gradients across GPUs and enable gradient check- porick-.piiing... For Multi-Mode IFT, we experiment with different base-LLM backbones .M (Phi3-3.8B, Owenl.5-4p Qwenl.5- 7B, Qwen .,514BB, Mistral-7B and LLaMA3-8B) to obtain the corresponding IFT models. We set of 140k samples selected randomly from CoT-CClllecion (Kim al., 2023) the data Drationate to enable three additional IFT modes. Additionally, we use 40k samples from the Dolly- HHRLHF (MosaicML, 2023) and the Open Assistant datasets combined create data for the con- ventional instruction-to answer IFT mode. The IFT training is performed for epochs. For task- guided SRO, we earry out iterations of task-guides DPO with 10 epochs in each iteration. Con- troller C is deberta-v3-
(He et al., 2021) model trained for 30 epochs with learning rate of le-5, BS of 128 (16 per GPU on GPUs) using adam optimizer with cosine annealing.
Evaluation Protocol:
To assess the usefulness of rationales, prompt containing the input-instructio appended (during inference) with the rationale generated by a method and fed to the instruct version of the LLM. The accuracy achieved is indicative of the usefulness of rationales.
4.1 DOES COALITION HELP IMPROVE LLM PERFORMANCE?
Table 1 compares COALITION with two categories of baselines: (I) Prompting-Techheee
to (i) generate rationales using Chain-of-Thought (Wei et al., 2022b), (ii) explore the space of rationales using Tree-of-Though (Yao et al., 2023), (iii) refine the rationales using CoT Self- Consistency (Wang et al., 2023b) & Self-Refine (Madaan et al., 2023b), or (iv) facilitate com- munication between multiple LLMs to refine the rationales through Exchange-of-Thoule et al., 2023); and (II) Rationale Enhancement via Trainable Self-Play (i) Distilling Step-by- Step (Hsich et al., 2023) where the IFT model is used to generate and refine the rationale. (ii) Self- Rewarding LMs (Yuan et al., 2024) uses sampling- baaeed based decoding to obtain diverse rationales and LLMM-as-a-judge to rate their quality for DPO, and (iii) SPIN (Chen et al., 2024) performs DPO by
(Yin

---

Published as a conference paper at ICLR 2025
Maths GSM8K
NLI
Comonsense
Method
WinoGrande
PIQA
HellaSwag
CSQA
Bara-Llama3-BB -nnsttrcctt w/ rationale (Dubey et al., 2024)
75.89
71.98
78.51
57.69
76.17
Prompt-Driven Rationale Refinement
Chain-of-Thoughhe (Wei et al., 2022h)
62.39
60.17
68.48
45.33
65.25
S Self-Consistency (Wang et al., 2023b)
64.11
62.37
71.42
46.13
68.92
(Yao al., 2023)
68.11
70.62
75.14
53.18
74.37
Exchange-of-Thoon Thought (Yin al., 2023)
69.19
66.47
73.11
52.22
75.48
Self-Refine (Madaan et al., 2023b)
77.26
72.81
79.49
60.48
78.22
Rationale Enhancement via Trainable Self-Play Distilling Step-by-Step (HKieh et al., 2023)
76.18
70.41
78.77
56.10
76.31
Self- Rewarding LMs (Yuan et al., 2024)
72.16
68.15
75.22
55.39
76.15
Self-Play Tine-Tunin (SPIN) (Chen al., 2024)
77.01
71.85
79.02
58.22
76.48
COALITION (ours)
81.06
77.13
83.26
63.23
82.06
Table 1: COALITION significantly outperforms multiple categories of baselines datasets from 3 task domains- Maths Problem Solving, Natural Language Inference (NLJ) and Commonsense Reasoning. Same LLM-backbone (Llama3-8B) used in all the methods for uniform comparison.
Rationales Generated by Best Baseline (SPIN) and LLM Variants- - LV, and ZV₃ in COALITION on GSM8K Sample Instruction: A company pays each of its employees $600 in month. The company has a policy of increasing the salaries of each of employees by 10% of the initial salary every year for those just clocked 5 years in the company last December, what's her annual salary after three more years of service? Best Baseline: Sylvic's annual salary after three more years will be $3600 because her salary increases by $600 every year for years. So after three years, will be 5600 $3600-000000000. X
company for five years. Sylvie
Generate w CV1: Sylvic's salary will increase by $600 every year for three years, her annual sulary after three more years ($600=3)=$2,00400""
of service will be
X
Generate w LV/y Sylvie receives a 10% increase her salary every year after years of service. Since her current salary is S600, will increase by each year. After three more years, her new salary will be $600+(3560)=$780. x
Cross-Retine LVV, rationale LV;: Sylvie's initial monthly salary is $600. After completing five years, her salary increases by 10% of initial salary. making it $600* 1.00%1.1-000 1-1-66 After three more yeurs of service, she will have received two additional annual 10% increases, which means monthly salary will be 6660-$60-560-88800 Therefore, annual salary after three more will be $780*12-$9360. /
Self-Refine CV// rationale w LV2: Sylvic's salary increases by 10% each year after her initial five years. Her monthly salary after five years $660, and three more years, she will have received two additional 10% raises. Therefore, her momthly salary will $660(11)) and her annual sulary will be Note: Controller selected CV, for generate step and CV, for refine step. Variant selection in this order yields correct rationale.
X
Table 2: COALITION yields better rationale using generate and refine steps via LLM variants. Wrong and right parts in a rationale are in red and green. Baseline wrongly applies increase for first five years. CV, estimates wrong annual increase while LV2 gives correct monthly increase but question asks annual salary. Cross-refining using LV, (as selected by controller) rectifies this error.
selecting general-domain GT rationales from CoT data over LLM-ggeerated rationales. We employ same L.M backbone (Llama3-8B) for COALITION and all baselines for uniform comparison.
COALITION outperforms all the baselines uniformly across the three task domains (Table 1). It per- forms better than the best baseline (SPIN) by 4% on GSMSK indicating the utility of rationales from COALITION for solving maths problems. Further, COALITION performs significantly better than SPIN for NLI with gain of 5.3% on challenging WinoGrande task and 4.2% on PIQA. On commonsense reasoming, COALITION outperforms SPIN by more than 5% on both CSQA and Hel- laSwag. Table 2 shows that COALITION generates better rationales than the best baseline (SPIN) by the virtue of employing distinct L.M variants. Please refer to A.6 for more qualitative examples.
Additionally, COALITION gives a significant performance boost compared to the case where the instruct model is evaluated without rationales, as well as using the rationales generated by the IFT model (Distilling Step-by-Step), Moreover, the baseline 'Self-Rewarding Language Models' which relies on the scale of very-large LMs to both generate and rate diverse rationales for DPO does not generalise with 8B 3a-parameter LM. COALITION performs better than this baseline by ~ 9% on GSM8K and WinoGrande, ~ 8% PIQA and HellaSwag, and 6% on CSQA. Finally, note that COALITION performs better by 3-4% than the best prompting-bart baseline i.e. Self- Refine (Madaan et al., 2023b) which uses same L.M as generator, refiner and feedback provider.
8

---

Published as a conference paper at ICLR 2025
Parameter Scale
Maths GSM8K
NLI
Commonsense
Model
WinoGrande
PIQA
HellaSwag
CSQA
Phi3 (Abdin et al., 2024)
3.8B
10.36
73.32
80.30
59.01
72.48
w/ COALITION (ours)
3.8B
14.76
76.19
84.48
63.72
75.04
Qwen1.5 5 (Bai al., 2023)
4B
3.49
67.01
75.57
52.01
74.61
w/ COALITION (ours)
5.58
69.26
76.48
53.37
78.29
Qwen1.5 (Bai et al., 2023)
57.01
69.53
79.54
61.06
81.00
w/ COALITION (ours)
7B
61.37
75.02
83.11
64.22
85.11
Qwen1.5 1.5 (Bai et al., 2023)
14B
69.37
76.01
81,45
65.57
84.19
w/ COALITION (ours)
14B
74,88
82.84
84.39
69.22
87.48
Mistral (Jiang al., 2023a)
7B
48.52
74.43
81.66
64.78
69.21
w/ COALITION (ours)
7B
54.42
78.39
85.01
68.48
74.38
LLaMA3 (Dubey et al., 2024)
75.89
71.98
78.51
$7.69
76.17
w/ COALITION (ours)
8B
81.06
77.13
83.26
63.23
82.06
Table 3: Performance evaluation of COALITION with LMs of varying scale of parameters (4B to 14B) and different model families (Phi3, Qwen Mistral, Llama3). It is observed that COALI- TION yields significant gains on all tasks for different model families and parameter scales.
4.2 How DOES COALITION WORK WITH DIFFERENT MODEL FAMILIES AND ACROSS VARYING PARAMETER SCALES?
We study if COALITION improves performance of LMs with varying scale of parameters (ranging from 4B to 14B) and belonging to different model families. We compare the accuracy achieved using the rationales generated by COALITION vs. the rationales obtained from the IFT (MIrT) version of the LLM. Specifically, we experiment with 1) Phi3-3.8B (Abdin et al., 2024), 2) Qwen1.5-(4B, 7B, 14B) (Bai et al., 2023), 3) Mistral-7B (Jiang et al., 2023a), and 4) LLaMA3-SB (Dubey et al., 2024). Table summarises the results where it can be seen that COALITION improves performance on all the tasks uniformly over different parameter-scales and LM families. In particular, consider Qwen - at 14B parameter scale, COALITION improves accuracy by 5 - 7% on GSM8K and WinoGrande, and by upto 4% PIQA, HellaSwag and CSQA. For Qwen-7B, there is similar improvement of ~ 4 - 5% on all the tasks. Likewise, performance increase is observed for Qwen- 4B model. For Phi3 model comprising of 3.8B parameters, 3 5% improvement is observed for different tasks. For Mistral-7B, there is an improvement of 5 6% on GSMSK and CSQA, and 4% the remaining tasks. Appendix A.4 shows that COALITION is effective on general benchmarks even when trained in a task-agnostic way on open-domain samples.
4.3 VARIANT SELECTION VIA CONTROLLER AND CHOSS-REANNEMET BOOSTS ACCURACY
We analyse the usefulness controller in Table 4 by evaluating the rationales obtained during inference 1) w/o any refinement (rows and 2). II) w self-refiement using a single variant (rows 3 and 4), III) cross-refinement using fixed order of variants for all samples (rows 5 and 6). Last row corresponds to dynamic selection of the LM variants by the controller for generation and refinement that yields the best performance. In particular, gen- erating and self-refining the rationale using a single LM variant usually (but not necessarily) performs better than not refining (rows 3-4 vs rows 1-2). However, generating the rationale using variant and refining it with the other variant consistently per- forms better than self-refinement (rows 5-6 vs rows 3-4). This shows that having distinct LLM variants and taking second opin- ion from the other variant through cross- cosssmmmmmuiiiaionn help- ful. Finally, selecting the LLM variant via the controller to gen- erate and refine rationales based on suitability for the given sam- ple yields significantly better results (last row).
WinoGrande
Selt-Neltioe
PIDA
Prose-Rullin
GSMBK
CBGA
Figure 4: Proportion of sam-
ples routed by Controller to 1) Generate w/o Refine (5-10%), Self-Refine (15-25%), 3) Cross- Refine (65-75%). Hence, gener- ation with one variant and refin- ing with other is preferred mode.

---

Published as a conference paper at ICLR 2025
Maths GSM8K
NLI
Commonsense
Communication Mode
WinoGrande
PIQA
HellaSwag
CSQA
Generate (w LV1) w/o Refine
77.26
75.10
80.11
59.22
78,47
Generate (w [V2) w/o Refine
77.21
74.89
79.24
59.31
78.33
Self-Refine (CV1 - 1
77.35
74.91
79.86
59.89
79.35
Self-Refine (LV2 - LV/2
77.23
74.70
79.60
59.70
79.25
Cross-Refine (CV) - LV2)
79.94
75.52
81.31
61.21
80.16
Cross Pross-Refine (CV2 + [V))
79.53
75.83
81.02
60.87
80.21
COALITION (w Controller)
81.06
77.13
83.26
63.23
82.06
Table 4: Performance analysis of rationales inferred 1) w/o refinement (rows 1-2), 2) w self- refinement (rows 3-4), 3) cross-refinement using fixed order of variants for all samples (rows 5-6), and 4) w controller (last row). Selecting LM variants using the controller for the generate and refine steps yields best results. Cross-communta cossuicmmuniaatoon between the variants is better than both self-refine (using a single variant) and not refining by directly using rationale generated by variant.
Ablation ID
Distinct LLM Variants
Likelihood-based Rationale Selection
DPO Sample
Maths GSM8K
NLI
Comonsense HellaSwag
Filtration
WinoGrande
PIQA
CSQA
1
No
Yes
Yes
71.74
69.28
76.15
54.22
75.22
Yes
No
Yes
78.24
75.69
80.
60.21
77.49
Yes
Yes
75,46
72.28
76.92
58.55
77.43
Yes
No
No 73.19
71.37
76.16
56.92
77.01
No
No
72.16
68.15
75.22
55.39
76.15
COALITION
Yes
Yes
Yos
81.06
77.13
83.26
63.23
82.06
Table 5: Ablation study to analyse the impact of different design choices - 1) In the absence of distinct LLM variants, sampling-basee based decoding is performed using single LLM; 2) LLM-as-a-judee is employed when likelihood- basedd rationale selection is omitted: 3) Entire train set is used for DPO when sample filtration is skipped. It observed that all components are critical for accuracy gains.
4.4 ABLATION STUDY IMPACT OF DIFFERENT DESIGN CHOICES
We examine the effectiveness of following components (using Llama3-8B backhone) 1) Distinet LLM Variants, 2) Task-guided Likelihood-basee Rationale Selection, and 3) Sample Filtration for DPO. Table shows the results where it can be seen that obtaining diverse refined rationales from distinct LLM variants gives significantly better performance than sampling-base -based decoding using single model (COALITION vs. row 1). To analyse the importance of using likelihood of generating the GT answer as the utility score to rate rationale, we leverage LLM-as-an juduee paradigm as an alternative where the IFT version of the LLM is prompted to rate rationales (row 2). Notable drop in performance than COALITION indicates that prompt-based rating does not work at scale of small LMs. Likewise, filtration of samples for task-guided DPO conditioned on whether the best rationale enhances the likelihood of generating GT (than not using any rationale) is critical for gains achieved by COALITION (vs. row 3). Additionally, omitting both likelihood- based rationale selection and sample filtration leads to further accuracy degradation (row 4). Finally, excluding all three components (row 5) gives significantly lower performance than COALITION.
CONCLUSION
We presented COALITION, trainable framework to improve the performance of (smaller) lan- guage models on complex tasks by employing distinct variants of the same LM and use them to generate and refine high-quality diverse rationales without any supervision from external (stronger) models and ground-truth rationale annotations. The model variants are made to exhibit distinct be- haviour by training them on separate data splits. In addition to the cost advantages of using smaller models, it has significant real-world advantages where legal and ethical constraints restrict the use of external models for supervision. Rigorous empirical evaluation over five datasets demonstrated the effectiveness of COALITION over several prompt-only and trainable self- self-play baselines. In fu- ture work, it is worthwhile to explore the impact adding more variants on the performance and incorporating domain-specific models (specialized variants) generate high-quality rationales.
10

---

Published as a conference paper at ICLR 2025
6 ETHICS AND REPRODUCIBILITY STATEMENT
We employ publicly available datasets and LLMs to conduct the study in our work which are com- monly used for ML research without any potential concerns. We do not annotate any data manually in this work. The rationales generated at different steps of the proposed method are of similar nature and domain as that of the text present in the datasets used. To encourage reproducibility, we release our code at this anonymous link and also upload it as part of the supplementary zip. We described the details of the datasets in $ 4 (under 'Datasets' in the Experiments section) and the LLMs used in S 4 (under 'Implementation Details"), ). Further,
provide the implementation details of our method in 4 (under "Imlemmentation Details") and discuss baselines used for comparison in $ 4.1. Finally, elaborate further details of our method in the Appendix 1) Example of samples for each dataset (A.2) and 2) prompt templates used to format the samples during the IFT (A.3).
Statement on Explainability: LLMs commonly used generate the final answer question/instruction for various NLP tasks. However, it was shown that eliciting the LLM to gener- ate rationale first followed by the final answer results in better accuracy. A rationale is statement in natural language that describes the steps which are required to derive the answer, or an explana- tion about how the question/instruution needs be approached to arrive at the right answer. The proposed COALITION framework improves the reasoning ability of (smaller) LLMs by improving their ability to generate better rationales. Since the rationales provide an explanation about why the LLM generated the final answer instead of just generating the final answer, the rationales can be used as a means of explainability while generating the answer to an input question/instruction.. Further, since COALITION generate and refine multiple rationales using variants of the same LLM, the generated and the refined rationales can be compared to identify differences in their explanation and quality. The identified differences provide further insights about what needs to be modified in the explanations.
input
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical re- port: A highly capable language model locally on your phone. arXiv preprint 2tti:2444.142199 2024.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0m 08774, 2023.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men. Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang. Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv:2309. 2023.
Yonatan Bisk, Rowan Zellers, Ronan Le bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7432-443, Apr. 2020. doi: 10. w3400//0.....4.......... 6239. URL https://ojs.aaai
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint urXiv:2401.01335, 2024.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsyaascchnkko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James
11

---

Published as a conference paper at ICLR 2025
Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Bren- nan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022. URL https:/aariv.or//bbb2200...0
Douglas
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2 10.14668. 14168, 2021.
Shizhe Diao, Pengcheng Wang, LIN Yong, Rui Pan, Xiang Liu, and Tong Zhang, Active prompting with chain-of-thour of-thought for large language models, 2024. URL https://ppeneevio..e//oommmm Wabp68RoSP.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:240..21783, 2024.
Zhibin Gou, Zhihong Shao, Yeyun Gong, yelong shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. ToRA: A tool-integrated reasoning agent for mathematical problem solving. The Twelfth International Conference Leurning Representations, 2024. URL https://openreview. nel/forum?id=Eip0Tiji Voap.
Arnav Gudibande, Eric Wallace, Charlic Victor Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. The false promise of imitating proprietary language mod- els. The Twelfth International Conference on Learning Representations, 2024. URL htips: wttprr/wwwei..eeffoummnnd--/-/ae
Shibo Hao, Yi Gu, Haodi Ma, Joshua Hong. Zhen Wang, Daisy Wang, and Zhiting Hu. Reasoning with language model is planning with world model. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 8154 8173, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 018663/112202..... 1/2022.emmlppppppp. www.mmmmin.nSnnt 507. URL https:/hclamttoooo.oor22222 wwtpg//22..eellpppomm main.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. (DEBERTA): {DECODING)- (enhanced) (bert) (with) (disentangled) (attention). In International Conference on Learning Representations, 2021. URL httss//ppenreviee.neeffrrr wttp://wwww...oooooooooo///aaaaa
Cheng-Yu Yu Hsich, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen- Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 8003-8017, Toronto, Canada, July 2023, Association for Computational Lin- guisties. doi: 10. 10.18653/112023, htt.:/22...nniigg--l.5.... URL https:/accanthboog.or/22223 fon..g.......2222.nnnn....g.... acl.507.
Jie Huang. Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. In The Twelfth Inter- national Conference on Learning Representations, 2024. URL http:://ppenreiie.neeffoom Pid= IkDD3KBPQ.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023a. URL https: Varniv.org//bs//2110..o
12

---

Published as a conference paper at ICLR 2025
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary. Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gi- anna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie- Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang. Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024a. URL https:/aaaai..or//ab/222/ org/abs/24/1/1 .04088.
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LLM-blender: Ensembling large language mod- els with pairwise ranking and generative fusion. Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 14165-14178, Toronto, Canada, July 2023b. Association for Computational Linguistics. doi: 10.18653/v1/2023. lo.18633///20....clloon.n... 792. URL http://ccaantooooooogg/2/222a/e
Dongwei Jiang, Jingyu Zhang, Orion Weller, Nathaniel Weir, Benjamin Van Durme, and Daniel Khashabi. Sel[-[in] correct: Lims struggle with refining self-generated responses. arXiv preprint 0429288804. 2024b.
Yuxin Jiang, Chunkit Chan, Mingyang Chen, and Wei Wang. Lion: Adversarial distillation of proprietary large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, pp. 3134-3154, Singapore, December 2023c. Association for Computational Linguistics. doi: 18653v 01116653/112222.33 http:1/22222mm2 www.0233.mmllpmmmm.. 189. URL https:/accanttboooy.o//222222222 wtw.:...rrr.200.2mmmmmmmmmmam
Gurusha Juneja, Subhabrata Dutta, Soumen Chakrabarti, Sunny Manchanda, and Tanmoy Chakraborty. Small language models fine-tuned coordinate larger language models improve complex reasoning. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.). Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 3675-3691, Singa- pore, December 2023. Association for Computational Linguistics. doi: 10. 16653/v1/0223.emm/ main.225. URL https://ecanthhooo..o////bb www.e-mminnpmnpm..
Minki Kang, Seanie Lee, Jinheon Back, Kenji Kawaguchi, and Sung Ju Hwang. Knowledge- augmented reasoning distillation for small language models in inottedge-intensive tasks. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https:// ppermeriww.n/ffouun///LLLLLLLLO
Seungone Kim, Se June Joo, Doyoung Kim, Joel Jang, Seonghyeon Ye, Jamin Shin, and Minjoon Seo. The cot collection: Improving zero-shot and few-shot learning of language models via chain-of-thought fine-tuning. arXiv preprint arXiv:2305. 14045, 2023.
Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 22199-22213. Curran Associates, Inc., 2022. URL https//ppoceedigs.eerripp 10per/2022/hie/bb000111dal9 lac44af006ff1 (tw.:/eeee110991166e 12099c16f3266 Pappe26-PPPpee--Coffeeeeeeeeppp
files/
Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. Step-dpo: Step- wise preference optimization for long-chain reasoning of llms. arXiv preprint (tth::440./8622)) 2024.
Xiaonan Li and Xipeng Qiu. Finding supporting examples for in-context learning. arXiv preprint 230.:23002.3339, 3, 2023a.
Xiaonan Li and Xipeng Qiu. Mot: Pre-thinking and recalling enable chatgpt to self-improve with ofemory-of-thoughts. arXiv preprint arXiv:2305.05781, 16, 2023b.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In Inter- national Conference on Learning Representations, 2017. URL http:///oeenevveeaaeefoommm ?id=
Skq89Sexx.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- on Learning Representations, 2019. URL
13

---

Published as a conference paper at ICLR 2025
Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, and Jingren Zhou. Routing to the expert: Efficient reward-guided ensemble of large language models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume I: Long Papers), pp. 1964-1974, Mexico City, Mexico, June 2024a. As- sociation for Computational Linguistics. doi: 10.18653/v 1120://12222..oom 1aw.://.1122.2nala1-aaa naacl-long.109 109, //aclanthloogyorrg http:////...aaaoooon....... naacl-long.109,
URL https:
Xiaoding Lu, Adian Liusie, Vyas Raina, Yuwen Zhang, and William Beauchamp. Blending is all you need: Cheaper, better alternative trillion parommeersss Ilm. arXiv preprint arX1v:2401.02994, 2024b.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazzanbakhss, and Peter Clark. Self- refine: Iterative refinement with self-feedback. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, vol- ume 36, pp. 46534-46594. Curran Associates, Inc., 2023a. URL httpss//proceedings. neurips. Ceraper-CCoffeence.pdeee
cc/paper flles//aper20023//ee/19 hollff/00232222555555500eeeeeeee
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. In Thirty-seventh Conference on Neural Information Pro- cessing Systems, 2023b. URL https://oeeeeeeeeeeee///ommmmmms
MosaicML. Dolly-hhrlhf dataset, 2023.
Sajad Mousavi, Ricardo Luna Gutiérrez, Desik Rengarajan, Vineet Gundecha, Ashwin Ramesh Babu, Avisek Naug, Antonio Guillen, and Soumyendu Sarkar. N-critics: Self-refinmment of large language models with ensemble of critics. arXiv preprint arXiv:2310.18679 2023.
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces gpt-4, 2023.
URL
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Chris- tiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with hu- man feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 27730-27744. Curran Associates, Inc., 2022. URL https://prooceeding..eem wwwp:eewwwdiinnnnnnisse..ee wwurips.cc/paperr hitppppr.iilsspppee/22//iiiii 4551::33444479944555500000711113
Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. arXiv preprint arXiv:2404. 19733, 2024.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin (eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguisties, pp- 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguis- URL https://aclanthoooo.oog/P. 1040.
tics. doi: 10.31 00.1115/1073003.1177
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon Ilm: Outperforming curated corpora with web data, and web data only, 2023. URL https://urxi..rrr/////233...11.1
Zhenting Qi, Mingyuan Ma, Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang. Mutual reason- ing makes smaller lims stronger problem-solvers. arXiv preprint arXiv:2408. 06/95, 2024.
14

---

Published as a conference paper at ICLR 2025
Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. Recursive introspection: Teaching language model agents how to self-improve. arXiv preprint arXiv:2407.18219, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances Neural Information Processing Systems, volume 36, pp. 53728-53741. Curran Associates, Inc., 2023. URI. https//proceedings.eeriip Seaper-Conference.ppde
files/paper/2223/iie
Swarnadeep Saha, Peter Hase, and Mohit Bansal. Can language models teach? teacher explana- tions improve student performance via personalization. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL
WQ.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An ad- versarial winograd schema challenge at scale. Proceedings of the AAAI Conference on Artifi- cial Intelligence, 34(05):8732-877) 8740, Apr. 2020. doi: 10.1609/aaii v34105.6399,
URL https:
//ojs.aaii.org//e. wttpa///ww.//////A/aaaaaaalaaiaa
Chenglei Si, Weijia Shi, Chen Zhao, Luke Zettlemoyer, and Jordan Boyd-Graber. Getting MoRE out of mixture of language model reasoning experts. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.). Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 8234-8249, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.18653/v/2023. findings-emnlp.552 URL https://cccaatholog.orrg/// wttgg//22...ininsseeemmeo..
Kaya Stechly, Matthew Marquez, and Subbarao Kambhampati. Gpt-4 doesn't 't know it's wrong: An analysis of iterative prompting for reasoning problems. arXiv preprint arXiv:2310. /2397, 2023.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQQ: A ques- tion answering challenge targeting commonsense knowledge. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4149-4158, Minneapolis, Minnesota, June 2019. Association for Compu- tational Linguistics. doi: 10. 11853//1//119112222 URL https://ccatthooog.oog////-
Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Haitao Mi, and Dong Yu. To- ward self-improvement of 11ms via imagination, searching, and criticizing. arXiv preprint arXiv:2404.12253, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshom, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https//arrivvorr//bs23330 099888..
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment. arXiv preprint arXiv:2310. 16944, 2023.
Dennis Ulmer, Elman Mansimov, Kaixiang Lin, Justin Sun, Xibin Gao, and Yi Zhang. Bootstrapping llmebased task-oriented dialogue agents via self-talk. arXiv preprint arXiv:2401.05033 2024.
Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Large language models still can't plan (a benchmark for LLMs on planning and reasoning about
15

---

Published as a conference paper at ICLR 2025
change). In NeurlPS 2022 Foundation Models for Decision Making Workshop, 2022. URL
Karthik Valmeekam, Matthew Marquez, and Subbarao Kambhampati. Can large language models really improve by self-critiquing their own plans? arXiv preprint arXiv:2310.08///8 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neu- ral Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: (tedddeeee1odd03eeeeeeee (ew.0o0e114884555. Paper.pdf. Papepppe
//proceedings.nemme www.ips..uripsssccc wwtpp/ppcce/////ie//ff5/e22222//
Danqing Wang and Lei Li. Learning from mistakes via cooperative study assistant for large lan- guage models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, pp. 10667-10685, Singa- pore, December 2023. Association for Computational Linguistics. doi: 10. (0.11663/11/2223emmmg main.659. URL https://ccamtbollog.or/////
wwnnp- main.659.
Jianing Wang, Qiushi Sun, Xiang Li, and Ming Gao. Boosting language models reasoning with chain-of-knowledge prompting. arXiv preprint arXiv:2306. XXiv:2306.06427, 2023a.
Junlin Wang. Jue Wang. Ben Athiwaratkun, Ce Zhang, and James Zou. Mixture-of-agents enhances large language model capabilities. arXiv preprint arXiv:2406. 200.00220 2024a.
Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li.. Mathcoder: Seamless code integration in LLMSs for enhanced mathematical reasoning. In The Twelfth International Conference on Learning Repre- sentations, 2024b. URL
http://ppnneeiiee.effffuum TTonBPP
Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Wu, and Zhifang Sui. shephere Verify and reinforce ilms step-by-step without human annotations. In Pro- ceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9426 9439, 2024c.
Tianduo Wang, Shichen Li, and Wei Lu. Self-training with direct preference optimization improves thain-of-thought reasoning, 2024d. URL https:/aaniv.orrgab//40007. 182448888.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023b. URL http://eeeeeeeeeee/ooooonmddddda
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large lan- guage models. Transactions Machine Learning Research. 2022a. ISSN 2835-8856. URL Survey Certification.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoe V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large lan- guage models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 24824-24837. Curran Associates, Inc., 2022b. URL 5af0r7b31 1abc44- wwwaaaaaaaeeer-CCCCC
files/paper/2022/file
Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Generating sequences by learning to self-correct. In The Eleventh International Conference on Learning Representations, 2023. URL
Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He,, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. Large language models are better reasoners with self-verification. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 2550-2575 Singapore, December 2023. Association for Computational Linguistics.
16

---

Published as a conference paper at ICLR 2025
doi: 10.18653/ www.diinninnnnnlp....... URL http:///ssamtboooyyyy22200/.mmmf 167.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language mod- els. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 11809-11822. Cur- ran Associates, Inc., 2023. URL. https://pooeeddingnnnnrrs..oooo hwtps//www.cccppppppcp/ppp//pp/a 27
Zhangyue Yin, Qiushi Sun, Cheng Chang, Qipeng Guo, Junqi Dai, Xuanjing Huang, and Xipeng Qiu. Sobbange-of-thughhh Enhancing large language model capabilities through cross-model communication. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.). Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 15135-155 1-153,,,, Singapore, December 2023. Association for Computational Linguisties. doi: 20.18653/v1/22222 wwm.o.eeemmlppppmmm main. 936. URL http:://cclanhhlooyyorg/// eem....mmmpppeee wwnll-mmann9955 main.936.
Yue Yu, Jiaming Shen, Tianqi Liu, Zhen Qin, Jing Nathan Yan, Jialu Liu, Chao Zhang, and Michael Bendersky, Explanation-awwe soft ensemble empowers large language model in-context learning, 2024. URL https://openeeview.eeeffomm h
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models, 2024. URL https:/aaxii..orr/bb//400...1
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Mårquez (eds.). Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 4791-4800, Florence, Italy, July 2019. Association for Computational Linguisties. doi: 10. PP939/11/19-1472. URL https:/cccaathoooo.orgg111111111
Chen Zhang, Chengguang Tang, Dading Chong, Ke Shi, Guohua Tang, Feng Jiang, and Haizhou Li. Ts-align: teacher-student collaborative framework for scalable iterative finetuning of large language models. arXiv preprint arXiv:2405.202111 2024a.
Xuan Zhang, Chao Du, Tianyu Pang, Qian Liu, Wei Gao, and Min Lin. Chain of preference op- timization: Improving chain-of-then of-thuh reasoning in llms. arXiv preprint 0rXiv:2406.091.66, 2024b.
Yunxiang Zhang, Muhammad Khalifa, Lajanugen Logeswaran, Jaekyeom Kim, Moontae Lee, Honglak Lee, and Lu Wang. Small language models need strong verifiers to self-correct rea- soning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics ACI. 2024, pp. 15637-15653, Bangkok, Thailand and virtual meet- ing, August 2024c. Association for Computational Linguistics. doi: 10.18653/v1/2022 1/202..iigg. acl.924. URL
2023. URL httpss//aclanthooog..rr////////e http::/hhaabbboogg./////////.... acc...
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. In The Eleventh International Conference Learning Representationss
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing. Hao Zhang, Joseph E Gonza- lez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In
A. Oh, T. Naumann. A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Ad- vances in Neural Information Processing Systems, volume 36, pp. 46595-46623, Cur- ran Associates, Inc., 2023. URL https///pooeediigs..uurpp.ppp wetpriisssseeiiissseppppp.... https///ppp....es//pr//////////a Datasets and Pattrprtttttteaaaaaaaattstsssss
Patprppeeeeeeeettt...eeeennnnsse Benchmarks,,dif
17

---

Published as a conference paper at ICLR 2025
Datasel
Tink--nstruch Instrucionn and Ground-Trae Truihh A Answers for Each Dalaset used to evalusae COALITION
GSM8K
Instruetion: Hetty saving mocely For waller which costs costs ST0T. Betry has only half the she nesds. Her parents decideed money Betry buy the waller?
give for that purpose, her grandparents
parents. How
Ground-Tre Truth Respouse:
WinoGrande
Instruction: Terry tned to bake the egpplani in the lowster oven but the Ground-Truth Response: A sgeplam
was too big. A. eggplant, B. louster
Instruction: How to llowrss will Find a dark, muist with circulation such allic unused closet With unilavoned denial banger complesely dry, so12: Find dark, dry area good circulation, such atti unosed closet. unflavored three weeks secure the
PIQA
the
they hang upside down
flowors
bottom of the Bowers' stems to hanger so that they hang upside down to dry, Leave flowers for two three weeks until completely dry.
Ground Truth Respouse: sal2
HellaSwag
Instruction: Then takes small stone from the Bowing diver and smadhes another stene. starss crush the small pleces.
smaller make the pieces smuller,
in half blow evcntually brings back into view and adds the smalller ones to make small trimguuar shuped piece. staris party with them and thoow the pieces by hand while they celebrase.
bigger. grind
Ground-Truth Response: B
lastruction: When learning about the world and different cultures, important you commutted eliminatinn preconceived notions. loss innocence. enlightemmet, open mind, smareess
CSQA
Cround Truth Respouse: open mind
Table 6: Examples of instructions from different datasets belonging to diverse task domains used in the experiments (i) Maths Problem Solving (GSM8K), (ii) Natural Language Inference (Wino- Grande and PIQA), and (iii) Commonsense Reasoning (HellaSwag and CSQA).
A APPENDIX
A.1 DIRECT PREFERENCE OPTIMISATION (DPO)
Direct Preference Optimisation (DPO) (Rafailov et al., 2023) was introduced as an alternative to Re- inforcement Learning using Human Feedback (RLHF) (Ouyang et al., 2022) technique to alleviate the need of training reward model. RLHF depends training reward model to assign score to the outputs generated by an LLM to fine-tune the LLM through reinforcement learning to align it with human preferences. On the other hand, DPO transforms the loss over the rewari-function to a loss over the LLM policy such that the reward is optimised implicitly by optimising the loss over the policy. It does so by leveraging human preference data which compares two possible out- puts generated by an LLM such that the better output is considered as the winner candidate while the inferior output is considered as the loser candidate- y/- Given a static dataset of the form
D = {x, Nw: } where is the input, the loss is modeled as
LR -oogoa(r(s, you) r(x, v1)) r(z,y) = :3log( To (y(xx)
(10)
(11)
Tref (y/z²)
where, 22 (y/z)) is the likelihood of generating y given x as input to the model Z {Mref, Mref is usually taken to be the instruction fine-tuned model in the case of L.M to prevent the L.LM policy from deviating too much from the initial policy, Ma represents the L.LM policy being optimised through DPO, is the sigmoid activation, and 3 is coefficient that controls the amount of deviation from the reference model. In summary, the algorithm optimises the LLM to learn to prefer generating certain outputs over other candidates without requiring an explicit reward model. Please refer to the original publication (Rafailov et al., 2023) for an elaborate discussion of the details.
A.2 DATASET SAMPLES
Details of datasets were discussed in the 'Experiments' section ($ 4) in the main paper. Table in the appendix shows samples of instructions for each dataset from all task domains (i) Maths Problem Solving (GSM8K), (ii) Natural Language Inference (WinoGrande and PIQA), and (iii) Commonsense Reasoning (HellaSwag and CSQA).
A.3 PROMPT TEMPLATES FOR MULTI- INSTRUCTION SUN-TUNING
As discussed in the Methodology section ($ 3, $ 3.1) in the main paper. the base model M is instruction fine-tuned to enable the LLM to operate in four modes in total (i) generate the rationale given the instruction as input (I R); (ii) refine rationale improve its quality for given
18

---

Published as a conference paper at ICLR 2025
instruction (I; + R); (iii) generate the answer conditioned the instruction and rationale as input (I; R] + A); and (iv) generate the final answer given the instruction as input (I + A). The inputs to the LLM are formatted using corresponding prompts (PI-R: PI:: PIz:R] - PI-+A) for each of these modes so that the L.M can generate an appropriate output accordingly.
The textual instruction for each prompt template is specified as follows:
1. PI-R- = "You are Al assistant 'M'. Provide response to the given instruction denoted by Task Description.
[TASK DESCRIPTION STARTS]
(Task Description): In this task, you will be given an 'Instruction'. Generate descriptive reasoning on how to derive the correct answer for the instruction such that the descriptive reasoning will be useful to another Al assistant generate the correct answer. 'Instruction' (instruction)
[TASK DESCRIPTION ENDS]
For the given (Task Description), give your response.
[M RESPONSE BEGINS]:
2. PIZ "You are Al assistant 'M', Provide response to the given instruction denoted by Task Description.
[TASK DESCRIPTION STARTS]
(Task Description): In this task, you will be given an 'Instruction' and rationale denoted by 'Rationale'. The 'Rationale' may or may not be correct for the given 'Instruction'. Analyse the rationale for its correctness, modify the rationale, and provide the correct elaborate descriptive reasoning or 'Rationale' which will be helpful to come up with the correct answer for the given instruction. 'Instruction' (instruction)
'Rationale' (rationale)
[TASK DESCRIPTION ENDS]
For the given (Task Description), give your response.
[M RESPONSE BEGINS]:
3. PIZ "You are an AI assistant 'M', Provide a response to the given instruction denoted by Task Description.
[TASK DESCRIPTION STARTS]
(Tak Description): In this task, you will be given an 'Instruction' and a rationale denoted by 'Rationale'. Analyse the rationale and come up with the correct answer for the given instruction.
'Instruction' (instruction)
'Rationale" (rationale)
[TASK DESCRIPTION ENDS]
For the given (Task Description), give your response. [M RESPONSE BEGINS]:
4. PZ-A= "You are an Al assistant 'M', Provide response to the given instruction denoted by Task Description.
[TASK DESCRIPTION STARTS]
(Task Description): In this task, you will be given an 'Instruction'. Generate the correct answer for the given instruction. "Instruction" (instruction)
[TASK DESCRIPTION ENDS]
For the given (Task Description), give your response. [M RESPONSE BEGINS]:
19

---

Published as a conference paper at ICLR 2025
Model
Parameter Scale
MMLU
HellaSwag
Easy
ARC
Challlnge
MCI
TruthfuIQA
MC2
WinGGrande
PIQA
GSM8K
CSQA
Phi3 (Abdin al,, 202-1)
3.8
69.94
59.01
$1.90
53.92
36.60
51.13
73.32
80.30
10.36
72.18
COALITION (ours)
3.8
72,01
60.19
82.45
58.79
37,38
56.19
74.48
82.01
12.15
73.69
Qweni. at. 2023)
52.01
34.73
29.38
3.19
COALITION (ours))
62.19
54.11
62.10
36.94
30.33
45.83
70.62
77:18
4.12
75.14
Qwen1 5 (Bai et al. 2023)
7B
69.94
61 16.06
80 35
50.94
40.
57. 35
69:51
79.54
57.01
81.00
w/ COALITION (ours)
7B
71.27
62.86
82.19
53,11
42.48
58.91
70.87
81.29
59,36
83.14
Qwenl al. 2023)
14B
78.78
65.57
05.98
60.49
51.53
68.99
76.01
81.45
69.37
COALITION (ours)
$4.26
69.91
63.14
53.15
71.28
80.31
83.48
72.08
Mistral Oling et al,, 2023g)
7B
59,60
64.78
$4.26
57.42
41.98
59.71
74.43
81.66
48.52
69.21
COALITION (ours)
7B
65.08
67.42
N7.76
59.01
44.79
62.89
76.92
85.01
53.35
74.02
LLaMA3 (Dubay et 2024)
50.17
26.81
73.24
50.11
COALITION (ours))
69.85
61.35
83.57
56.07
38.38
54.95
75.07
83.10
78.85
79,00
Table 7: Analysis of generality of COALITION trained in task-agnostic manner by performing SRO on general open-domain samples (instruction-answe pairs). The table summarises the accu- racy achieved on general benchmarks comprising of 10 tasks from the open-llm leaderboard. Ra- tionales generated by COALITION uniformly improves the performance on all the tasks for all the LMs belonging to different model families (Phi3, Qwenl.5, Mistral, Llama3) and varying parameter- scales (ranging from 4B to 14B).
In the above prompt templates, (instruction) is placeholder for the actual task instruction IT and (rationale) is placeholder for the rationale text.
A.4 EVALUATION ON GENERAL BENCHMARKS VIA TASK-AGNOSTIC SRO
We measure the generality and effectiveness of COALITION on general benchmarks (comprising of 10 tasks) in the open-llm leaderboard by performing selective rationale optimisation (SRO) of the LLM in a task-agnostic manner on a randomly selected subset of CoT data comprising of general open-domain samples (instruction-answe) the open-Ilm leaderboard, the L.M trained using COALITION is leveraged to obtain the rationales through the generate and the refine steps for evaluation. In particular, the generated rationale is ap- pended to the prompt after the sample-instruction backbones where it can be seen that rationales generated using COALITION uniformly increases the performance on all the 10 tasks for LLMs belonging to all the model families and parameter-scalee This demonstrates the generality of COALITION framework to improve the performance on tasks even when the training (SRO) is performed on general-domain samples in a task-agnostic manner.
pairs). Given a sample from the test-split of dataset from
Table 7 summarises the results for different LLM
Notably, for Mistral-7B, there is significant increase ~ 5.5% on the MMLU task which measures the ability of the model to answer questions related to the world-knowledge. Similarly, there is an improvement of ~ 3% on truthful-QA. PIQA and close to 5% improvement on GSM8K and CSQA. Likewise for Llama3-8B, there is huge increase of more than 7% on MMLU, 6% on the Challenging version of the ARC task, 11 - 12% on truthful-QA and similar improvements on the other tasks. Similar improvements observed for the Qwen model at different parameter-scales.
A.5 ADDITIONAL RELATED WORK
Mixture-of-Agents (Wang et al., 2024a) uses multiple open-source LLMs based agents to improve the output quality at inference-time by generating intermediate output simultaneously using each agent independently. Their framework comprises of multiple such layers of LLM agents such that the outputs generated by agents a layer are fed to the LLM agents in the subsequent layer which are prompted to analyze the information in the responses generated by L.LM agents in the previous layer. It observed that the accuracy on several benchmarks improve by prompting multiple L.M agents in such manner across multiple layers of agents. COALITION creates and uses multiple variants of same SLM to improve its ability to generate and refine rationales in a trainable manner without involving any external LLM.
Zephyr (Tunstall et al., 2023) rates the responses generated by Falcon (Penedo et al., 2023), Llama (Touvron et al., 2023), etc. using GPT-4 (Achiam et al., 2023) to tune Mistral-7B (Jiang et al., 2023a) via DPO. Legal liabilities and lack of transparency in the pre- pre-training data of larger models restrict their use to train smaller models for commercial use. Some methods use mixture-
20

---

Published as a conference paper at ICLR 2025
Datavi
Nationales Generated by Best Baseline (SPIN) and LLM Variants of COALITION Gemerale and Reline Sleps
Instructio:: John lakes care TO dogs. Fach takes 5 hours he spend taking John speads hours a day taking care of each dog, and he 10 dogs. So. for one day, the total time speat is .5 10w50. Since there are 7 days in a week. we multiply the duily total by the jotal time speat is 5550-250
walk and lake cure of their businous. How many hours a week does
GSMKK
dags?
Best
Generate: John spends (1.5 hours a day for each of the 10 dogs, so for one day he spends 5 1005 hours. To find the time spent in week.
muhtiply by/ days: 2*25=2 hoars a wesk x
Refine: Each dog requires 0.5 hours cure per day, and John has 10 dogs. So, the total time spent per 15*]0=5 hours. Since there days in week. the total time spent per week is 7=5 bours.
Instruction: Terry inted bake the ogpplani in the louater uven but the -...... was tou big. oggphant B, lnaster
WinoGramde
Best Baseline: The word that "was too big" is reforring to the touster hecause it is the object Tony was using to bake. Therefore, the correct answer is B, toaster, X
Generate: The sentennee says Terry tried to bake the eguplami, but something
big. The word "was big" is referring the luasler,
couldn't
egaplant inside. So the answer
loaster
"was too for the roaster owen, Since the eggpiant the item heing placed into the toaster oven, and the istne with its size, the correct anewer is A. eggplant of
Refine: sentence implies
Terry tried to bake eggplant,
Instruction: How to day flower?? sell: Find a dark, moist area with good circulation, such an attic or unused closet With unflavored dental banger that they bung upside down dry. Leave flowers for to three weeks unil completely dry, sol2: Find dark, dry area good circulation,
PIOA
Bloss, the botiom of flowers'
as attic unused closet they hang upside down to dry. Leave flowers for two (hres weeks until completely dry
unilavored dental floss. secure the
botiom of the flowers" stems to a hunger
Best Baseline: Sol I correct because flowers need a dack and moist environment to dry properly. Moisture will help retain the natural beauty of the flowers. So, Solution I is the better mothed for drying Blowrss X
Generate: Solution mentions finding dark, moiit area, which could help preserve the flowers freshness for a while before they dry. Refine: Solution
conrec appeaach Bowers without mold decay.
suggests finding
dry
good circulation, which ideal drying
Instruction: Then he takes a small stone from the Bowing river and smaches it on another stone. He starts to cris the small stane to smaller pieces. He
HellaSwag
A. cuts the center stone in half and blow it on to make it bigger. B. grind hard to make the pieces smallerr C. eventually beings it back into view and aids to the smaller to mike small triungular shaped piece. starts patty with them and throw pieces by hand while they celebrate.
Best Baseline: best answer A. cuts the center stone would magically enlage the picces. X
and blows to make bigger because cutting the stone and blowing
Generale: The correct answer is C. eventually brings it buck into view and adds it to the smaller unes to make small tringgulr-ssuped picue. This suggests he's shaping the stone into smaller pieces and eventually creating triangle, which aligns with the idea of breaking the stone down.
Refine: best answer is grinds hand to make the pieces smaller. This accurately mutches the action described the scenario, where the
small sione being smashed and crushed into smaller pieces. The gual is to break down, and "grinding it hard" reflects that process.
Instruction: When laarnin about the world and diffforont enlures, whit is imporant if you are committed to climmnting naning preconceived notions. open mind, E. smartess
CSQA
A. newness, B. loss of innocence, C. enlightenment, D. open
A. newness because when you learn about different cultures, everything you encounter will feel and unfamiliar. Newness what eliminates preconceived notions.
Best Bascline: The
Generate: The answer enlightenment because as you leam more about different cultures, you become onlightened, which helps you overcome your prezoaceived ideas.
Refine: The correct answer is D. open mind because if you be committed to eliminating preconceived notions, the most important factor being receplive and willing to accept ideas and perspectives. /
Table 8: Qualitative analysis and comparison of rationales generated by COALITION with the best baseline- SPIN on all the five datasets on the three task domains. COALITION yields better ratio- nales using generate and refine steps by leveraging the distinct LLM variants.
of-experts where the insk-instruttion is routed to suitable LLM expert either at the query-level (Lu et al., 2024a), in latent space (Jiang et al., 2024a) or at the output-layer (Si et al., 2023). On similar lines, COALITION employs a controller that is trained using the preference data collected during the DPO training select the LLM variants for generate and refine steps. COALITION creates labelled data to train the controller based on which LLM variants's 's rationale gets selected during selective rationale optimisation.
A.6 QUALITATIVE ANALYSIS
Table 8 shows qualitative comparison of rationales generated by COALITION with the best base- line SPIN on all the five datasets on the three task domains. COALITION yields better rationales using generate and refine steps by leveraging the distinct LLM variants.
A.7 HUMAN STUDY FOR RATIONALE EVALUATION
We conducted a human study to evaluate the effectiveness of rationales obtained using the proposed COALITION framework. The following steps describe creation of data for human evaluation:
Dataset Creation for Human Evaluation
1. We collected total of 75 samples by taking equal number of samples for each task i.. 15 samples randomly from the test sets of each of the 5 task datasets.
2. For each sample, we obtain the rationales Rlg. R2, from the two LLM variants at the generate step. Based on the variant selected by the controller for the generate step, the corresponding generated rationale R, is considered for refinement.
21

---

Published as a conference paper at ICLR 2025
3. The selected generated rationale Rg is used by the controller to determine the variant that should be used to refine the selected generated rationale. Once the variant selected, is used to refine the selected generated rationale to obtain the refined rationale - R.y.
Once the above rationales are obtained, employed two paid human annotators and presented them with the instruction each sample along with different rationales obtained above. The human evaluators asked to judge the quality of different rationales based the following questions and guidelines:
Questions and Guidelines
1. Question 1: Is the final rationale obtained from COALITION useful for answering the question correctly? The rationale is useful if it is correct and provides the correct explana- tion how the answer for the instruction in the sample should be derived. Provide label out of O or 1 such that 0 means that the final rationale is totally wrong; and means that the final rationale totally correct.
2. Question 2:
Compare the selected generated rationale Rg with the refined rationale R, obtained after refining Rg. Provide a label of or where means that the refinement improved the generated rationale and means there
improvement.
3. Question 3: Compare the two rationales obtained using the two variants at the generate step
- -Rlg and R2, Provide a label of 0 or where means that none of the rationales is better than the other and 1 means that one rationale is better than the other.
4. Question 4: In Question 3, in case one rationale is better than the other (between the ratio- nales obtained from two variants at generate step), select the better rationale.
Definition of Metries Estimated from Human Labels
Different rationales were presented to human evaluators in jumbled order to avoid biases while com- paring rationales. Based on the judgement labels provided by the human evaluators for questions above for the 75 samples, we estimate the following metrics:
1. Final Rationale Alignment - % proportion of samples which were assigned label i.c. totally correct.
2. Improvement using Refinement % proportion of samples where the refined rationale R, was judged to be improving the generated rationale Rg.
3. Diversity b/w two Rationales from Generate Step % proportion of samples where the two rationales RL, g and R2, obtained from two variants at generate step are different i.e. cases where one of the two rationales is better than the other (label 1). This metric is estimated to verify if the variants truly generate distinct rationales.
4. Better Rationale Alignment with Likelihood based Selection: We consider samples where label 1 is provided Question 3 i.e. one of the generated rationales is judged better than the other generated rationale (comparing RI, and (22g). We estimate the metric as % proportion cases from these samples where better rationale determined using likelihood- based utility score matches the better rationale from human judgement.
Human Study Results and Discussion
We compute the above metrics using the 75 samples used for human evaluation. We report the aver- age of metrics obtained for the two human evaluators in Table 9, We discuss following observations from results Table 9:
From Table we can observe that the final rationale alignment is 87.33% which means that final rationale obtained from COALITION is reliable and aligns with human preferences.
2. Rationale refinement helps since refinement improved the generated rationales for 36% cases. Thus, obtaining better rationales through refinement would also enable accuracy improvement on the final tasks as observed in the paper.
3. Rationales from Two Variants are diverse: It is observed that for 62.67% cases, one ratio- nale obtained at generate step judged to be better than the other generated rationale.
22

---

Published as a conference paper at ICLR 2025
Metric Name
Value (in %)
Final Rationale Alignment Improvement using Refinement
87.33
36.0
Diversity b/w two Rationales from Generate Step
62.67
Better Rationale Alignment with Likelihood-based Selection
80.85
Table 9: Human study results summarizing values of different metries evaluated using human labels. observed that for good proportion of cases, final rationale obtained from COALITION aligns with human preferences, refinement helps improving generated rationales, the rationales obtained from two variants are diverse and better rationale judged by humans matches with winner rationale selected using likelihood based utility score.
This means that employing two variants of same LLM is useful to obtain distinct and di- verse rationales which are useful improve quality of preference data for DPO.
4. Likelihood based rationale selection aligns with human preferences: For 80.85% cases, better generated rationale determined based on human preferences matches the better ratio- nale based on likelihood-based utility score. This shows that our choice of using likelihood of final GT answer for selecting winner rationale aligns with human preferences and is suitable to obtain the preference data.
Inter-Annotator Agreement: We also report the inter-annotator agreement by estimating the Co- hen's kappa coefficient which is commonly used to measure agreement between two annotators. For the human study, following is the Cohen-kappa coefficient for questions used estimate each metric:
Cohen-Kappe coefficient for Final Rationale Alignment: 0.7112
Cohen Kapea-Kappa coefficient for Improvement using Refinement: 0.4851
Cohen-Kaa Kappaa coefficient for Diversity b/w two Rationales from Generate Step: 0.7331
Cohen- Kappa coefficient for Better Rationale Alignment with Likelihood based Selection: 0.5105
Following is mapping of cohen-kappa coefficient value ranges with interpretation:
0 0.2: Slight agreement
0.21 0.4: Fair agreement
0.41 0.6: Moderate agreement
0.61 0.8: Substantial agreement
0.81 1.0: Almost Perfect agreement
Based on the coefficient obtained for different metrics and the above scale, it can seen that human labels for final rationale alignment (0.7112) and diversity b/w rationales (0.7331) have substan- tial agreement while human labels for improvement using refinement (0.4851) and better rationale alignment with likelihood based selection (0.5105) have moderate agreement.
A.8 RATIONALE EVALUATION USING LLM-AS-A-JUDGE
We perform the same evaluation as done for human study but instead of human evaluators, we use GPT-4o the judge. GPT-4o is prompted with questions as used for human study for all the samples in the test split of each task dataset. Table 10 summarizes the values of metrics obtained using GPT- 4o as judge where we report combined as well as dataset-wise metrics also since the number of samples for each dataset evaluated using GPT-4o is large.
can be seen that using GPT-40-as-a-judgc yields similar (even more profound) trends as were ob- served from human study where quality of the final rationale obtained from COALITION is judged to good for majority cases (for 82.55% samples on average) and refinement improves rationale quality (for ~60% cases average). Further, the rationales obtained from LLM variants di- verse (for 71.21% % cases on average) such that better rationale judged by GPT-4o aligns with winner rationale determined using wkelihood-basel utility score (for 88% cases average).
23

---

Published as a conference paper at ICLR 2025
Metric Name
Combined across Tasks
GSMBK
WinoGrande
PIOA
HellaSwag
CSOA
Final Rationalc Aligument
82.55
83.40
Improvement using Refinement
39.66
69.19
61.29
57.33
53.47
57.01
Diversity blw two Rationales from Generatee Step
71,21
80.18
72.24
74.27
61.21
68.
Aligament with Likelibood-besed Selection
92.71
Table 10: LLM-as-a-judge results for evaluating the rationales using GPT-4o as a judge to estimate (i) Final Rationale Alignment, (ii) Improvement using Refinement, (iii) Diversity b/w two Rationales from Generate Step, and (iv) Better Rationale Alignment with Likelihood- based Selection.
Method
GSMSK
WinoGrande
PIQA
HellaSwag 63.23
CSQA
COALITION 2 LLM Variants
81.06
77.13
83.26
82.06
COALITION W 3 LLM Variants
83,41
79.58
85.24
65.48
83.35
Table 11: Comparison of results by employing LLM variants vs. 2 LLM variants (as done in main paper) in COALITION. Employing more variants improves the accuracy further.
A.9 VARYING NUMBER OF LLM VARIANTS IN COALITION
The number of LLM variants is hyper-parameter. We experimented with 2 LLM variants in the paper. As an ablation study, we perform an experiment where we employ and train three LLM variants and compare the accuracy with 2 LLM variants in table II. is observed that the accuracy on all the tasks improve uniformly with an average increase of 2% across different tasks. Thus, accuracy improvements over different baselines also get enhanced further with using variants. We leave increasing the number of variants further to explore if it yields additional improvements as future work.
A. AUTOMATED DIVERSITY ESTIMATION BETWEEN RATIONALES FROM TWO VARIANTS
To measure diversity between the rationales obtained from the variants (for both generate as well as refine step), we estimate normalized lexical overlap between the rationales and take its com- plement as a measure of how distinct the rationales are. BLEU (Papineni al., 2002) is commonly used metric in the NLP field to estimate overlap between two text sequences. Using Bleu, we esti- mate corresponding diversity metric i.e. Bleu-Diversity b/w rationales r1, r2 generated by the two variants respectively by taking complement of Bleu as follows:
Bleu-Diversity = Average| Bleu(r1, r2)), Bleu(r2, r1)
Note: The values obtained using the overlap metric (BLEU) lie in the range of to 1.
Table 12 shows the values of diversity metrie for rationales obtained from two variants COALI- TION for generate as well as refine steps respectively on all the tasks where it is observed that the diversity metric for all the tasks (for both generate and refine step) lie the range 0.68-0.80 (which is high scale of 0-to-1) which shows that the rationales obtained using the two variants are distinct from each other.
A.11 ADDITIONAL MEASUREMENT OF RATIONALES USING PERPLEXITY
We conduct additional measurement the rationales by estimating the perplexity of generating GT answer conditioned on rationales obtained at both generate and refine steps (for Llama-3-8B back- bone). We also compare with the setting where no rationale is used. Lower perplexity means that training COALITION
winner/eliminated rationale using DPO enhances the LLM's confidence and chances of generating the correct answer. Table 13 summarizes the results where it is observed that using COALITION rationales reduces perplexity of GT answer. Also, using refined rationales results in lower perplexity compared to using rationales obtained from generate step highlighting the importance of refinement.
24

---

Published as a conference paper at ICLR 2025
Metric
GSM8K
WinoCrande
PIQA
HellaSwag
CSQA
Diversity b/w rationales obtained at Generate Step
0.7525
0.7995
0.6893
0.8018
0.6827
Diversily b/w rationales obtained at Refine Step
0.7369
0.8048
0.6974
0.8149
0.7011
Table 12: Bleu-diversity metric b/w rationales from two variants for generate and refine steps spectively. Since Bleu overlap metric lies in range [0, 1], Bleu-diversity also between 0-to-1. can be that rationales from two variants are lexically diverse due high value of the diversity metric for both generate and refine steps.
Method
GSM8K
WinoGrande
PIQA
HcllaSwag
CSQA
w/o any rationale
11.29
6.92
6.73
8.47
8.84
W COALITION rationales from Generate Step
9.61
5.24
5.19
7.28
7.38
W COALITION rationales from Refine Step
8.47
4.48
4.46
5.37
6.53
Perplexity (lower is better) generating GT answer (1) w/o any rationale, (ii) rationale from generate step in COALITION, and (iii) rationale from refine step in COALITION. It can be seen that COALITION's rationales reduces perplexity compared to not using any rationale. Further, refined rationales results in lower perplexity compared using rationales from generate step.
Table 13:
A.12 NUMBER OF SAMPLES USED TO TRAIN VARIANTS
We report the number of samples used for each the two variants to train them during the IFT stage as well as different iterations of DPO. During IFT, as discussed in implementation details section, a total of 180K samples were used. This IFT data was divided into two equal partitions such that 90K samples
used train and obtain each LLM variant. IFT is performed in task-agnostic manner. We summarize the number of training samples used for each variant during task-guided DPO in Table 14.
Training Stage
GSM8K
WinoGrande
PIQA
HellaSwag
CSQA
DPO iteration-1 Generate Step
1317
7297
2949
7649
1728
DPO iteration-1 Refine Step
1626
8934
3140
9795
1993
DPO iteration-2 Generate Step
1489
8379
3529
8029
1979
DPO iteration-2 Refine Step
1724
10093
3896
10764
2252
Table 14: Summary of number of samples used to train each variant for each DPO iteration for the generate and refine steps for different tasks.
25


Reviews:
[{"review_id": "BQNVU5styj", "reviewer": "Reviewer_6DbD", "summary": "Training on rationale or chain-of-thoughts have been known to be effective at enhancing a smaller LM's ability, yet due to transparency, copyright issues, this method has a fundamental limitations. The authors propose Coalition, a method that trains multiple LMs on different data splits & constructs a CoT preference dataset by comparing the likelihood (of producing the correct answer by conditioning on the CoT candidates) from the different models. This is effective because when training a single model, it could not generate a diverse set of response candidates. Compared to prompting-based baselines or iterative self-training baselines, the proposed methods show stronger performance on 5 benchmarks.", "questions": "Please see the questions mentioned in weaknesses. I will change my scores accordingly based on the author's response!", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper compares with multiple baselines (Table 1), show that their method works across various scales (Table 3), and is supported by ablation experiments that each component does make the method more effective (Table 4,5).", "weaknesses": "* In addition to extrinsic measurements, perhaps it would be great to include intrinsic measurements of how the preference data quality improves through (1) choosing a winner rationale and (2) refinement.\n\n* It is not clear how the number of LLM variants are decided and how the dataset is divided to train each LLM variant. Considering that this is the first step of the overall pipeline, an ablation experiment of deciding the number of variants or protocols for dividing data splits seems crucial but is missing.\n\n* Compared to using LLM-as-a-Judge or reward models to choose winner rationale/eliminated rationale, how much effective is using rationale-conditioned GT answer likelihood heuristic? There should also be an experiment for this as it is a trivial baseline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4uG18PHEBr", "reviewer": "Reviewer_FNsr", "summary": "This paper introduces a COALITION framework to improve the rationale generated from smaller language models without more powerful teacher models. LLMs typically excel at generating complex rationale but have limitations in their cost. COALITION addresses the limitations of SLMs by enabling two variants of the same model to generate and refine diverse rationales, optimizing them using a Selective Rationale Optimization (SRO) process to maximize task performance. \n\nTheir experiments verify the effectiveness of their method on three base models on five datasets. The result shows that cross-communication between model variants can boost performance compared to self-refinement and other baselines by up to 5%.", "questions": "The two variants model seems to be the most important part of the framework. How effectively do the two variants trained on separate data splits capture genuinely diverse reasoning paths? Could these variants' differences be further quantified or analysed to understand better their distinct contributions, e.g., what is the minimum data we need to train the variants?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper contributes to rationale generation by addressing a gap: improving the performance of smaller language models without relying on more powerful large language models. While existing work often emphasises knowledge distillation or prompt-based methods involving LLMs, this paper introduces COALITION, a unique framework that leverages two variants of the same model to collaboratively generate and refine rationales.\n\nTheir IFT trains two variants of an SLM on separate data splits to exhibit distinct behaviours in generating and refining rationales. During SRO, these variants independently generate and cross-refine rationales for a given task, with utility scores assigned based on the likelihood of producing correct answers. This scoring guides DPO in refining the models’ outputs. The controller, trained on preference data from DPO, dynamically selects the optimal variant for generation and refinement during inference. This coordinated approach generates a higher-quality rationale, boosting SLM performance without external supervision.\n\nThe writing is well structured, and their experiments consistently improve on different models and tasks compared with the baseline method.", "weaknesses": "The authors quantitatively analyse the quality of the rationales based on the final task accuracy, determining rationale effectiveness by how well it leads to correct answers. However, they did not focus on conducting human evaluations or employing LLMs as judges to assess rationales to provide more insight into the rationale coherence or human preferences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "luIQ2Cb5CS", "reviewer": "Reviewer_JqYh", "summary": "[Edit] I have updated my score based on the authors' response. \n\nThis paper proposes a novel framework \"COALITION\" that can be used with smaller language models (<13B) to generate rationales that lead to higher accuracy. COALITION uses two variants of a language model (variant in terms of instruction-finetune training data), and employs self/cross-refinement + DPO to train them. Finally during inference, a controller model picks a rationale to be used between the ones generated by the two variants. The authors present strong empirical results and a thorough ablation study.", "questions": "[covered in Weaknesses]", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Strong empirical results including (1) Varied, thorough and strong set of baselines show the numerical significance of COALITION, and (2) results with multiple base-LLMs such as Phi3, Qwen, Mistral, Llama3\n* Useful ablation studies that demonstrate the usefulness of their proposed components.", "weaknesses": "* Section 3 presents difficulty in understanding the training of LLM variants - I suggest the authors to clearly explain the initial training data differences between LV1, LV2, and M_IFT. It is my understanding that M_IFT was trained with the entire IFT data, whereas LV1 and LV2 (before refinement) were trained with different splits of the IFT data to ensure that they exhibited different behaviours, but I invite the authors to correct me if this is wrong, and also add a simplified, non-mathematical summary of the same in Section 3.\n* To clarify, each LLM variant was trained wth DPO with both (1) generated rationales raked by utility scores, and (2) refined rationales ranked by utility scores?\n* Section 3.3: What is the purpose of training a controller C to rank the variants, when M_IFT was already performing that role with its utility scores?\n* Since this paper depends heavily on rationales, I suggest the authors to perform a human study on the rationales (generated, refined, and final) to check their quality. I understand that this work is primarily focused on improvement of accuracy, however, a preliminary understanding of the quality of the rationales generated is essential to have trust in this system.\n* Lastly, I suggest the authors to discuss the explainability aspect of their framework in the Ethics section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nOA60t2dFt", "reviewer": "Reviewer_Xpy3", "summary": "In the paper, COALITION is introduced to improve the performance of smaller LLMs in generating rationales through cross-communication. This is done by two variants of the same model fine-tuned in different ways.", "questions": "- In lines 77–78, you mention not relying on \"external\" LLMs. However, in lines 85–86, you refer to two distinct variants of SLM, which are trained differently. From my perspective, they are essentially no longer the same model. \n- For evaluation, I think one important baseline is missing: self-refine (Madaan et al. 2023, https://openreview.net/forum?id=S37hOerQLB), which uses self-generated feedback to refine the rationales iteratively.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Presents COALITION framework, which refines smaller LLMs output by the same model\n- Conduct experiments 6 LLMs ranging from 3B to 14B on 6 datasets\n- Ablation study on components: distinct LLM variants, rationale selection and sample filtration", "weaknesses": "- One paper might be missing: Mixture-of-Agent (https://arxiv.org/abs/2406.04692) for cross-communication\n- In line 196, I believe the utility score alone may not be sufficient in this case. A multi-dimensional evaluation, such as using LLM-as-judge, might be necessary. Additionally, the correctness of the refined generations is unclear, as LLMs often struggle with self-correction (https://openreview.net/forum?id=IkmD3fKBPQ, https://aclanthology.org/2024.findings-acl.826/).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)