Training Chain-of-Thought via Latent-Variable Inference
Authors: Du Phan
Matthew Douglas Hoffman
david dohan
Sholto Douglas
Tuan Anh Le
Aaron T Parisi
Pavel Sountsov
Charles Sutton
Sharad Vikram
Rif A. Saurous
Venue: NeurIPS 2023 poster (2023)
Keywords: Large language models
latent-variable models
control variates
chain-of-thought
MCMC
TL;DR: Treating chain-of-thought-prompted question-answering LLMs as probabilistic latent-variable models lets us derive a principled, simple, effective way of tuning them to generate better rationales without training on human-generated rationales.

Abstract: Large language models (LLMs) solve problems more accurately and interpretably when instructed to work out the answer step by step using a "chain-of-thought" (CoT) prompt. One can also improve LLMs' performance on a specific task by supervised fine-tuning, i.e., by using gradient ascent on some tunable parameters to maximize the average log-likelihood of correct answers from a labeled training set. 
Naively combining CoT with supervised tuning requires supervision not just of the correct answers, but also of detailed rationales that lead to those answers; these rationales are expensive to produce by hand. Instead, we propose a fine-tuning strategy that tries to maximize the \emph{marginal} log-likelihood of generating a correct answer using CoT prompting, approximately averaging over all possible rationales. The core challenge is sampling from the posterior over rationales conditioned on the correct answer; we address it using a simple Markov-chain Monte Carlo (MCMC) expectation-maximization (EM) algorithm inspired by the self-taught reasoner (STaR), memoized wake-sleep, Markovian score climbing, and persistent contrastive divergence. This algorithm also admits a novel control-variate technique that drives the variance of our gradient estimates to zero as the model improves. Applying our technique to GSM8K and the tasks in BIG-Bench Hard, we find that this MCMC-EM fine-tuning technique typically improves the model's accuracy on held-out examples more than STaR or prompt-tuning with or without CoT.

Training Chain-of-Thought via le Inference
Du Phan"
Matthew D. Hoffman'
David Dohan' Charles Sutton Google
Sholto Douglas
Tuan Anh Le Rif A. Saurous
Aaron Parisi
Pavel Sountsov
Sharad Vikram
Abstract
Large language models (LLMs) solve problems more accurately and interpretably when instructed to work out the answer step by step using rchain-of-thougee (CoT) prompt. One can also improve LLMs' performance on a specific task by supervised fine-tuning, i.c., by using gradient ascent on some tunable parameters to maximize the average log-likelihood of correct answers from a labeled training set. Naively combining CoT with supervised tuning requires supervision not just of the correct answers, also of detailed rationales that lead those answers; these rationales are expensive to produce by hand. Instead, we propose fine-tuning strategy that tries to maximize the marginal log-likelihood of generating a correct answer using CoT prompting, approximately averaging over all possible rationales. The core challenge is sampling from the posterior over rationales conditioned the correct answer; address it using simple Markov-chain Monte Carlo (MCMC) expectation catiattionmmimmmzzatiiio (EM) algorithm inspired by the self-taught reasoner (STaR), memoized wake-sleep, Markovian score climbing, and persistent contrastive divergence. This algorithm also admits novel control-variatt technique that drives the variance of gradient estimates Applying our technique to GSM8K and the tasks in BIG-Bench Hard, we find that this MCMC-EM fine-tuning technique typically improves the model's accuracy held-out examples more than STaR or prompt-tuning with or without COT.
the model improves.
Introduction
For many mathematical, logical, and common-sense reasoning problems, large language models solve problems more accurately when instructed to work out the answer step by step in a chain of thought or scratchpad (Wei et al. 2022 Nye al. 202 Kojima et al. 2022 Rajani et al. 2019 Shwartz et al. 2020). These methods encourage the model to produce a rationale, that is, text describing a sequence of reasoning steps that leads to answer; the motivation is that it seems to be easier for the model to generate a sequence correct reasoning steps than to generate the final answer directly. Because of the striking performance of chain-of-thougl! methods, many variants have been proposed
(Wang et al. 2022h Zhou et al. 2022 Creswell et al. 2022 Ye & Durrett 2023), but there are still many
which the rationales incorrect.
One way to improve these methods is to fine-tune models to generate better rationales. If gold- standard rationales can be obtained, such as via crowdsourcing (Rajani et al. 2019) or automatically (Nye et al. 202T), then supervised methods can be applied, but obtaining this data can difficult. An appealing alternative is to start from datasets that contain questions and correct answers only, which are more readily available, and bootstrap rationales during learning. version of this strategy
*Corresponding authors: fanofffann, www.gppggooollle..ooooo The first two authors contributed equally, and their order was chosen randomly.
CCurent atfiliation: OpenAl.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

was proposed as the self-taught reasoner (STaR) Zeliikann et al., 2022), which generates proposed rationales from an LLM, and then fine-tunes on rationales that lead to the correct answer.
In this paper, we approach the problem of bootstrapping rationales from a different conceptual direction: chain--ffthought methods are probabilistic latent-variable models. The L.M defines joint probability distribution over questions, rationales, and answers; this joint distribution implies a marginal distribution of answers given questions, averaging over all possible rationales weighted by their probability given the question. The problem of self-training for reasoning then becomes one of learning with incomplete data, a core task in probabílistic machine learning (Murphy 2022 to which we can apply methods from large and sophisticated literature.
This perspective raises technical challenge, because computing the marginal distribution requires averaging over a vast set of potential rationales. To address this, we introduce a learning algorithm for rationale generation, which call TRICEFTRICE, is a simple Markov-chain Monte Carlo (MCMC) expectation-maxxt
(EM) algorithm combined with a novel control- scheme, inspired by ideas from STaR (Zelikman et al. 2022), memoized wake-sleep (Hewitt al. 20200) Markovian score climbing (Naesseth et al. 2020), and persistent contrastive divergence (Tieleman 2008).
This view unifies several threads of work in reasoning using LLMs: It provides an alternative interpretation of STaR as kind of biased stochastic expectation expectation-maxiiizzaii maximization algorithm (Nielsen 2000) that underweights difficult examples when its rationalization process fails. (Wang et al., 2022a) can be seen as a Monte Carlo algorithm for computing the most likely answer under the marginal distribution. Compared of TRICE allows us to average over rationales not only at inference time, but also at training time. Compared to STaR, TRICE is less likely to ignore difficult examples (which stabilizes convergence and improves performance), and is also able to learn from incorrect rationales as well as correct ones.
self-consistency, the probabilistic learning approach
We apply our technique to the GSM8K dataset (Cobbe al. 2021 and to the BIG-Bench Hard bench- mark (Suzgun et al.) 2022a). We find that TRICE improves the model's performance significantly, outperforming models tuned with STaR, direct tuning with or without CoT, and even supervised fine-tuning on human- uoman-generated rationales.
2
Method
Given a training set of N questions i1:N and answers y1:N. we formalize CoTT tuning as optimizing a parameter vector 0 maximize the average marginal log-likeldhood of answers given questions:
(0)
A N Σn Powx x.) = 3/2 Σ,, log E. po(2 / | x Pr(n 2z, 12,,n)
(1)
where a is an unobserved latent rationale, (o x)) is the probability of obtaining the rationale by prompting an LLM with the question x and tunable parameters 0, and po(y | 2, 2,z) is the probability of obtaining the answer y given rationale 2, question x, and parameters 0. We will be particularly interested in models where the likelihood Po x., z) E 1), that is, where the answer y is a deterministic function of z. For example, we might say that the model's answer is y "(a)" if = ends with the string "The answer is (a). For this deterministic model, we define p(y 2,x) = c(2, y) E {0, 1). Details of c(z,y) for each task can be found in Appendix FP We believe that such binary likelihood model is appropriate for question-answent a rationale- good rationale should leave no ambiguity about the correct answer. The derivations below will therefore assume binary likelihood function. It is straightforward to generalize our methods to cases where the relationship between and y is weaker and therefore p(y x 2) is more complicated; Appendix A shows how.
tasks where is
Algorithm I
outlines the method.
notebook with a reference implementation hww.oo/lee.eeeccccccssseeeeeeeee
can be found at htttps://github eexamples el wwwebooks/ttiiceeoocooooo.osceee ipynb
We start by initializing a memory containing a latent rationale nn for each example pair xn Vn. by sampling 2n from a hinted guide distribution q(2 Zn, Vn) that may condition on the correct answer Yn as well as the question Fn, For example, the guide might prompt an LLM specifically to give an
PTRICE stands for "Tuning Rationales with Independence- Chain Unless otherwise specified, we sample at temperature I throughout.

---

rationale for the answer, more details about the precise prompts used by the guide are in Appendix F In some cases sampling from the guide instead of the model po(z 2x,) increases the chances of generating correct rationale (Zelikman et al. 2022
We then proceed to the main optimization loop. Each iteration, sample minibatch of M questions and answers from the dataset, and retrieve the rationales associated with those examples from the memory. We then propose new rationales غ from the current model pe I)), and whenever the new rationale is correct (i.e., c(2, y) = 1) replace the old rationale in memory with the new one.
At this point we have all need compute gradient estimate; can just average the gradients Vo log pe (2)mm Ii,.) ) that we obtain from those rationales in the updated memory that are correct (i.e., we ignore examples where both the proposed rationale and the previous rationale were wrong). AASIC_GRADIENTTI LESTIMTTETTTTETEEEE in Algorithm shows how.
But we can also reduce the variance of our gradient estimator by incorporating a control variate, as in CONTROL VARIATE_ WARIATT_GRADIENT GRADINNT ESTIMATE in Algorithm We first compute leave-one-out estimates M of the average probability of accepting a new rationale. For each example m, we subtract off scaled control variate 8,, V log po po(mm (F...) whose expected value is zero (since is score function). If the proposed rationale 7m for example m is correct, then Pig. ēm and the mith gradient contribution becomes (1 - Bm)) ) Ve log p2(2 Zi.,) ). i.e., it is scaled down by - -mm If 2m is incorrect, then adjust the gradient estimate try to make 2m less likely under ps. As the model becomes more accurate (i.c., 3 gets closer to 1), we give more weight to incorrect rationales (when they occur) and less weight correct rationales (most of the time).
CONTROL__ SEEIF=_GRADENNTE LESTRMATEET is more expensive than BASIC_ SESIC_GRADENT_ERE EESTIAATE since we must compute gradients not only for the rationales in memory but also for any incorrect rationales we generate. This may be wasteful, especially if many of the weights on those gradients (I - 3 for correct proposals, 3 for incorrect proposals) are close to zero because 3 is close zero or BIEAANT_GGAADDDNEN BEAADDINTEESTIIMEA systematic resampling (Hol et al. 2006 to generate subsample of L question- pairs, from which we obtain an unbiased estimate of the output of CONTROL VARIATE GRADIENT ESTIMATE. We preferentially sample gradients with higher scalar weights; if 3 is small, we are less likely to sample incorrect rationales (which have weight 3), and if 8 is large, we are less likely to sample correct proposed rationales (which have weight 1 (3). This can be seen a generalization of the strategy of Burda et (2015) Section 3) for reducing the cost of computing IWAE gradients.
one. To reduce this cost, in SUBSAMPLED DUBSAMMLED_CONTROOO _ow.TONNTOAAAAA
we use
Below, we derive this variance-reduced stochastic MCMC-EM procedure in more detail.
2.1 Derivation
The true gradient. The gradient of the marginal log- lieeliidldoodd log po (y |x) with respect to 0 is
log E, x) = Σx V a log pa(swla)
= Σ. Po(z 2, y) V po(p x)) (2)
Σ. Pe(s' ull)
that is, it is the expectation with respect to the posterior po(= x. v) of the gradient of the conditional log- oog-prior log (= z). since the likelihood p(y 2, z) = c(z, y) does not depend on f. So if we can sample from the posterior over rationales conditioned on the question-answe can compute an unbiased estimate of the gradient of the marginal log- log-likelihood log Po | (x). We can interpret this as "bootstrapping" rationales that consistent with both the prior rationales (o(2) æ) and the observed answer y (cf. Zelikman et al. 2022).
pair x, y. then
Independence sampler for pe(z x, v). We cannot directly sample from pa (2 xx y). so we resort to Markov chain Monte Carlo (MCMC). We maintain memory (cf. Hewitt et al.) (2200) of single rationale z, for each guestion-answe
pair In, Yn. and each iteration we apply a random update to 2n that leaves the posterior (2, I Mn.) invariant (cf. Tieleman, 2008 Each MCMC update brings the 2, 's closer in distribution to (z En, Vr) (Cover 1999 Murray & Salakhutdinov 2008). However, updates to 0 may change the posterior po(2n En, Vn), so must keep updating the chains to control the bias of our gradient estimates.
To update the chains, we use a simple, hyperparameter-fies independence sampler (Tiemny 1994); a Sttropolis-Hastting (Hastings) 1970) update that proposes updating the current state with a draw ă from distribution Fzyy that does not depend on 2, and accepts the update with probability
3

---

Algorithm TRICE
Input: Generative model po (z, y æ), is is-correct function c(2, y). dataset 21.N, f1.N. hinted guide distribution
g(2 v. imitial parameters 0, optimizer update function h00, ( t), minibatch size M, gradient minibatch size
L. number iterations
Output: Tuned parameters 0, rationales 21:N-
I: for E I....... do (in parallel)
Initialize Markov chain states.
2: Sample in from q(z In: Un). // Sample "fallback" rationale from guide
3: end for
4: for t ...... do
// Main optimization loop.
Get next minibatch of indices i1. into the dataset. for m ....., .M do (in parallel)
6:
Take one MCMC step to update Markov chain states.
7:
Sample Em from po(z (.m)). if (eclmm Vim) then Update 2im f- Zm.
8:
I/ Accept or reject proposal.
10:
end
11:
Let Em = c( Wim) Let cm c(Zim: Fim
Whether the proposal correct.
12:
// Whether the updated rationale is correct.
13:
end for
14:
Compute 9 using either BASIC GRADIENT CELAAAINTTESTIIAAAA ESTIMATE(Z, 2,, VARIATE_GRADLELA GRADIENT ESTIMATE(2, 2, 2, c'). or SUBSAMPLED CONTROL_ VARIATE_ // Apply gradient update.
CONTROL_ CONTROL_VARIATE VARIATE,
VARIATE_GRADIENE GRADIENT BEEENNTESTTMATT..... 2,,,,, SOO.MMROLLECCOLOLLLLLCLLC Update 0 + h(0,
15:
16: end for
17:
return 0, 21
18: procedure BASIC GRADIENT ESTIMATE(Z, I, c')
19: return Em. Cm Em ém Ve log pa( 2im )-
20:
end procedure
21: procedure CONTROT VARIATE_ AARIATE_GRADIENTE BAADDINTTESSTMMATRE. 2, à, for m E E1.... M do (in parallel) Em sem &mm
22:
23: Sct 8m
Compute leave-onc-out control-variate scales.
Lm yom Smm
24: end for
25: return Em Sm
Em Cn (Vo po II Bm Ve log pa(m. Js,,)).
26:
end procedure
27: procedure SUBSAMPLED CONTROL_ VARIATE VARIATE_GRADIENT EEAODIENNTETTTMMEEC....
CONTROL_VARIATE
28: for m E 1,...,, ......,,) M do (in parallel)
Set 8... Em Ame
29:
I/ Compute leave-one-our control-variate scales.
30:
Set 0m = cm(1 (1- Om.m). WMs+m c/ (1 mm
I/ Compute unnormalized weights for subsampling.
31:
end for
32:
Choose a subset
indices jı:z using systematic resampling with probabilities
Zomm
33:
for e E 1...., T. do (in parallel)
34:
if je < M then
// Selected correct rationale.
35:
Lect je, Zin
36:
else
Selected incorrect rationale.
37:
Let vi = jr - M. = Zm. -1.
38:
end if
39:
Compute vo log po Ir,.)) // Negate gradient if 4th rationale incorrect.
40:
end for
>MM om 41: return Em cm
I 2'=100
42:
end procedure
a(² min {1,
pe(E.vla)/ro..13 p=( ) We choose (2) = po(= [
x)). which simplifies the
acceptance probability to a(3 2) = min {1 po(y(a.5) pa(a)a.a] }. This is 1 if c(2, y) 1, 0 if c(2, v) = 0 and c(2, 3/) 1, and ill-defined (implying that we have to reject) both c(2.1) = and c(2. v) So accept whenever the proposal ī is correct, and reject otherwise.

---

Remarks; Independence samplers can be understood as "Metropolized" importance samplers that spread the work of generating and evaluating proposals over time. In our setting, the update can also be interpreted as attempting to sample from the posterior by rejection sampling, then falling back on an old sample if that fails. The expected number of iterations between successful updates p(y = so mixing will be faster for easier questions z, and will accelerate as the model improves.
procedure lets us approximate the gradient of the marginal log-likelihood in Equation 2 Denoting as the state the Markov chain for an example x, y before the update, we sample proposal from po(z æ); accept the new state it is correct (i.e., if c(2, y) = 1), and compute the gradient of the log- ong-robbbilityy of the result:
Basic gradient estimator. This CCMMCrejection-sampiin
= c) + (1 c(2, 2));; 2; g = Va po (z);; E 0) R Epo(=la.y) [Vo log po(z x)]
(3)
where E,,=l- denotes an expectation with respect to both the proposal E and the previous state 2.
Remarks: The estimate will have low bias if the distribution of 2' is close to the posterior p(2 x. y). which we expect to be true if the chain mixing quickly enough relative to how fast is changing. This will happen if either the probability of getting correct answer is high, or if 0 is changing slowly due to a small learning rate and/or gradient. If the model's 's training-set accuracy improves with training and we use a decaying learning-rate schedule, then as training proceeds both of these factors should work to reduce the bias of the gradient estimate.
Adding a control variate. The mean of an estimator I is not affected by subtracting zero mean random variable from it. If b positively correlated with 9. then 0 - b can have lower variance than J. and we say that b can be used as a "control variate" (Owen & Zhou 2000). Since, by the fcnrt-function identity, pp (&Vo a log (= x²] (for any scalar /3 independent of 2), we can use the proposed samples ī to generate control variates for our gradient estimator:
E2 E 8 = Eg E Vo log
|=] [0]
(4)
E2 E Ve logpe(z' I=) - v po x)] 10].
Remarks: The value of this estimator will depend whether not accept the proposal 2:
Vo log po x) VV, log x) loggpo( x) - VV x) if 0,
[ log loogoop''' x))
if 1,
(5)
where we use the shorthand č A c(²,y).
This control variate can drive the variance of the gradient estimator to zero the model converges to perfect accuracy on the training set (cf. Roeder et al. 2017). If we set 3 = т, where π is the probability of a correct answer (i.e., that è 1), then as gets large, most of the time ē 1 and we multiply our gradient estimator by 1 - π (multiplying its variance by a factor of (1 )) If O, then we make use of both correct and incorrect rationale; the weights attached these updates will not be small, but if incorrect rationales are relatively rare then their contribution to the variance of the gradient estimator will be correspondingly small. On the other hand, the model has not yet learned to frequently generate good rationales for the training examples, then we should set 3 closer to since in this case the signal from the incorrect rationale less informative in Appendix C.I we show that the variance of gradient estimators based on incorrect rationales depends strongly on the model's 's accuracy æ. Appendix B we show that choosing B = x is in fact optimal up to ((11 2) terms, and that the variance of the resulting estimator is proportional to - n
Estimating B. For each example Im, Vm. we need to compute 8m Elén in a way that ensures that 8m is independent of V, log Pa (2m ). We assume that E(cml ~ M Σ.. E(e,] (i.e., that the per-example acceptance probability is close to the average acceptance probability across the minibatch 6 and compute the leave-one-out estimate 3m Em e' Em
where dy := c(2'n V).
E mm "
'There may be some examples (especially early in training) for which we have not yet generated any correct rationales. We omit these examples from our gradient estimate, since they have likelihood 0 and therefore cannot be representative samples from the posterior. WWe also tried keeping a running estimate of the average acceptance probability per example, but we did not find that this more complex scheme provided any empirical advantage.
5

---

We restrict the estimate to consider only examples for which we have a correct rationale (i.e., where cm 1), since these are the only examples that influence our gradient estimate. Leaving out ém and m from the estimate 9m ensures that Bm is independent of Ey-
Gradient subsampling. Finally, as described above, we can reduce the cost of our gradient estima- tor by using systematic resampling to select a subset of rationales. This does not affect the expected value of the estimator as long as the marginal probability of selecting a rationale is proportional to
2M
the corresponding weight Wm: and the averaged gradient is reweighted by
2.2 Why not variational inference, reweighted wake-sleep, or rejection sampling?
We considered three alternatives to the MCMC-EM approach that we pursue in this paper: variational EM (e.g.. Bishopp 2006). reweighted wake-sleep (RWS: Bornschein & Bengio 2015 Le et al. 2019). and rejection sampling.
Variational expectation-maximiziiinn maximization is common strategy for training latent- models, but variational inference with discrete latent variables is challenging Tucker et al. 2017).
RWS is an attractive alternative that avoids high-variance score-function gradients; it proceeds by sampling M samples 21:M from a guide model do (z x. y), assigning the samples weights Wim x poly.z(zz) vala|a,y]' and updating both the model parameters 0 and the guide parameters Φ to maximize the reweighted oogprobbaiilities Σm Wm log Po(zm Ir) and E.. Wm log (2 x. y). Unfortunately, we found that RWS training sometimes led to degenerate zero-length rationales z. Figure | suggests a partial explanation: shorter sequences get higher weights, so the model and guide learn to produce shorter and shorter sequences until they consistently produce empty rationales.
Why do longer sequences tend to get lower weights? We can write the unnormalized weights as t.
120
mean q sample lengths weighted mean sample lengths
dy. zm) ps (xml22
where Tm is the length of 2m and is added to address the case where none of the samples are correct. If there is mismatch between (Zmm,t 2 (m,,:((-11)) and polsm,als. p(zm,t I, 1m)):(t-1)); then uol3m, a.mm..l...... 1.....m..---- will usually be less than one, with rare high- ex- ceptions that ensure that Ey(p(= x)/g(z xx]] 1.
cly, 2m) II'm po(sm, sm. 11)
(a( 2.. 1 (11)))))
80-
60-
20
20
25
iteration
If these exceptions are rare enough to not typically appear in sample of M sequences 21:ar, then the normalized weights WI.M = Em tim
Figure 1: Example of rationale lengths shrink- ing during RWS training. Blue line shows the average number of tokens per rationale gener- ated by the guide, orange line shows the aver- age number of tokens per rationale weighted by the rationale's importance weight.
Muaa will tend to
assign higher mass to shorter sequences unless those shorter sequences are much less likely to be correct.
With careful initialization and learning-rate tuning, we could sometimes get RWS to avoid this problem of empty rationales. But this led to prior p(z z) until the very end of the rationale, and then simply paste in the correct answer whether or not it had anything to do with the rationale up to that point (cf. Turpin et al. 2023). Figure 5 in Appendix shows representative example in which the guide model ignores the answer it arrived at through incorrect reasoning and pastes in the correct answer.
problem: the guide go(2 xx v) learned to closely mimic the
Quantitatively, denoting by / the index of the token at which the "final answer" section of the rationale begins, in one run we found that the average KL between g(=1a. | (x,y) and p(21: a)) was about 0.61 nats, while the conditional KI. between q((((+))T)) x. y, #1:t) and p(=((+1):T x. 21:t) was about 42.5 nats, confirming that the guide was not "reasoning backwards", just copying the correct answer.

---

Finally, we considered rejection- rejection-samplingg sampling"sshhemm scheme in which we sample K proposal rationales 21:K from p(2 z), and average the gradients from those rationales that lead to correct answers. We will present the quantitative results in Section 4 our main finding is that, while this scheme can work, it requires reducing the minibatch size by factor of K to keep the per-iteration cost constant compared to TRICE, which in turn leads to slower convergence and/or worse final results.
3 Related Work
number of methods have proposed rationale generation for problem-solving tasks in neural sequence
models, including both fully supervised and few-shot approaches (Wei et al. 2022 Nye et al. 202T
Kojima et al.] 2022, Rajani al.) 2019; Shwartz et al.) 2020; Wang et al., 2022b: Zhou al. 2022
Creswell et al. [022]] Ye & Durrett (023). Particularly relevant to our approach is self-consistent
chain-of-thought (Wang et 2022b), because this can be approximately viewed as marginalizing
over rationales at test time. This technique has been successfully applied for range of quantitative
reasoning tasks (Lewkowyez et al. [022]. There is relatively much less work that does imputation or averaging over rationales at training time; perhaps the main instance is STaR Zelikman et al. 2022) which we discuss in Section 3
Dohan et al. (2022) present position paper which advocates representing composition of language
model interactions via probabilistic programming. Our treatment of rationales as latent variables is inspired by that work. Lievin (202)) offers another example of interpreting LLMs with CoT as latent-variable models.
Variational inference (e.g.. Kingma & Welling 2013) and wake-sleep methods (e.g., |Bornschein & Bengio 2015) are workhorses of the carttttvaarabl-mmdeliiii community, but we discuss in Section 2.2 we found the bias of these methods to cause serious problems. MCMC-EM is a less- common strategy these days, although a version of it based on Gibbs sampling (Geman & Geman 1984) it has been widely applied to training undirected graphical models (Tieleman 2008). TRICE can also be cast as an instance of Markovian score elimbing (Naesseth et al. 2020).
ReAct Yao et al. 2023) demonstrated that injecting reasoning into an RL-style observe-and-ad act loop significantly increases performance. This approach was extended in Reflexion (Shinn al.) (023). where an agent can conditionally reflect on an RI. trajectory, augmenting the resulting examples which can be used few-shot examples in subsequent rollouts. These approaches reported significant improvements on their respective evaluation tasks but still rely on the model being able to produce useful and actionable feedback through pure few-shot prompting, whereas our method actively tunes the model to produce thoughts amenable to the task.
Recent work on tool-use within language models also works via imputation, inferring where insert calls to tools (Parisi et al.) 2022 Schick et al. 2023). Their loss functions are similar in spirit to ours, filtering out trajectories which do not lead to valid answers. In this paper, we have treated rationales as latent variables; one could also treat tool-use as a latent variable.
3.1 Self-Taught Reasoner
The most closely related work is the self-taught reasoner (STaR: Zelikman et al. 2022). Besides the arguments in their derivations, there are three significant differences between TRICH and STaR. First, STaR uses greedy decoding, which reduces the diversity of the rationales it trains on. The authors made this choice to reduce the danger of the model getting the right answer despite having a bad rationale. While we do find that our procedure sometimes generates correct answers for the wrong reasons, this did not seem to stand in the way of the model improving on most tasks. One reason may be that our base models are more powerful than the 6B-parameter GPT-J model used in the STaR paper, so they are more likely to generate good rationales from the beginning.
second difference is that TRICE resamples rationales every iteration, so it are less likely to overfit to any particular rationale. STaR has an inner loop that runs many training iterations single set of rationales, meaning it uses stale rationales to estimate the gradient of the marginal likelihood.
We also considered optimizing an importance-weighhee bound (Burda et al. 2015 using the prior p(> | x)) as proposal distribution, but instead opted for a simple rejection sampling scheme since this is less biased and equally feasible in our setting.
7

---

In our experiments, we observed that this leads to the model effectively memorizing fixed set of rationales for the training set. Once this happens, the greedy decoding procedure will almost certainly reproduce exactly the same rationales at the beginning of the next outer loop. If these rationales all lead to the correct answer, and STaR has a rationale for each question, then this is global optimum of the marginal likelihood on the training set! But empirically, STaR often does not find a good rationale for each question, and so it ignores some fraction of the training set (see Section 4
This tendency to ignore the most difficult questions in the training set follows from STaR's s derivation as an approximate volicy-gradien algorithm trying to directly minimize the 0-1 loss Ep [p y)
1 - (y z). The derivative of this marginal likelihood is pe (v/ z) log Pe (v x), that is. it is the derivative of the marginal log- likeliihooo (which TRICE tries maximize) weighted by (o x). This weighting causes difficult examples to contribute little to the gradient used to update the model, so the model may "give up" on questions that it cannot yet solve. This one argument for trying to maximize log-likelihoods instead of likelihoods.
final, minor difference is that when STaR updates its rationales, it may replace a rationale from the model p(= x) with a rationale from surrogate ( ær, y). As the model memorizes a set of correct rationales for the training set, STaR becomes less likely to fall back on the surrogate, but this choice could affect early training dynamics.
4 Experiments
We evaluate TRICE on the GSM8K (Cobbe et al. 2021) ) dataset and the 27 BigBench-Hard (BBH)
tasks (Suzgun et al.,, 2022b) using the medium-size PaLM 2-M (Anil et al. 2023) Transformer-based
LLM (Vaswani et al. (017). For the BBH experiments, we used the Flan instruction-tuna (Chung
al.
2022) version of PaLM 2: for GSM8K, we used the base PaLM 2 model, since GSM8K is
included in the Flan training datasets, All experiments were run on TPU v4 and v5e chips (Jouppi
al. 2023). Examples of generated rationales can be found in Appendix B
Rather than fine-tune the model weights, we use prompt tuning (esterr et al. 202I); we prepend a sequence of embedding vectors 0 (a "soft prompt") to the embeddings corresponding the tokenized CoT prompt used to condition the model. Prompt tuning can achieve similar accuracy gains to full fine-tuning, but using small fraction of the parameters. We initialize the soft prompt with the embedding sequence obtained from a series of three (for BBH) or five (for GSM8K) exemplar CoT prompts, each of the form "Question: <QUESTION> AnAnserr: Let's think step by step. \m<RATIONALE>", We consider two initialization schemes: one where we use the standard few-shot CoT prompts that are provided with BBH, and one where we try bootstrap a few-shot CoT prompt by sampling random questions from the training set, generating random rationales from the base model, and picking three or five examples where the random rationales lead to correct answers. The first scheme can be seen as a way of fine-tuning a good initial few-shot prompt, but it does require a small amount of detailed CoT supervision, while the second scheme only requires label supervision.
On each BBH task, we split the examples into 60% train and 40% test sets. For all but three tasks, this is 150 training and 100 test examples. For GSM8K, we use the standard 7473-example training set and 1819-example test set. We evaluate CoT models' accuracy in two ways: first, using greedy (temperature-O) decoding, and second, using "self-consisteney" (Wang et al., 2022b). In self-consistency evaluation, we draw 40 samples and check whether the most common answer is correct; this is a plug-in estimator for the prediction arg max, p(y) .x) that minimizes 0-1 loss under the model (although this is not how Wang et al. (0022b) originally motivated the procedure).
We compare against four baseline prompt-tuning methods: direct prompt tuning, CoT prompt tuning, rejection sampling, and STaR (Zelikman et al. 2022). All methods are evaluated against the same validation sets, and use the same training labels, sew-shot prompts (except for direct tuning, where we only use question-anseee answer pairs), and initialization strategies as appropriate. Details for each method and its corresponding experimental hyperparameters can be found in Appendix [F)
Table and Table 2 summarize the results; more detailed task-by-task BBH summaries are in
Appendix D Even with no human-generated exemplar rationales, TRICE is able to learn to generate
rationales that lead to the correct answer. TRICE also outperforms a model trained directly on human-generated rationales GSM8K (cf. Uesato et al. 2022), perhaps because the cross-entropy
8

---

Prompt-Tunine Strategy
Greedy-Decoding Acc. (%)
Self Celf-Consistency Acc. (%)
% Valid Rationales
Initialization
STaR
62.0
62.1
91.6
Bootstrapped
Rejection Sampling
64.6
65.3
3-shot
TRICE without CV
67.8
68.0
98.7
Q-R-A
TRICE with CV
72.8
73.1
98.8
Direct Prompt Tuning
3-shot Q-A
70.4
TRICE without CV
73.4
75.2
98.2
3-shot Q-R-A
TRICE with CV
76.7
77.6
98.6
Table Average accuracies (columns 3 and 4) and fraction of training examples for which we can generate correct rationales (column 5) across the 27 BIG- BIG-Bench Hard (BBH) tasks. All methods but direct prompt tuning use CoT prompting. All trainable prompts are initialized with an embedding sequence obtained from few-shot prompt containing either example question-answe pairs ("Q-A") or example question-rationall-ane
triples ("Q-R-A")). For direct prompt tuning, the Q-A pairs come from the training set. For TRICE, we use either the three Q-R-A triples provided with BBH (bottom two rows) or bootstrap a set of rationales as described in the text. For STaR and rejection sampling, only evaluate on bootstrapped initializations.
Prompt-Tuning
Seeedd-Deoodinggg Acc. (%)
Self-Consistency Acc. (%)
% Valid Rationales
Strategy
STaR
53.5
60.1
80.2
CoT Prompt Tuning
58.6
73.8
Rejection Sampling
77.9
87.0
Direct Prompl Tuning
19.4
TRICE without CV
72.8
81.5
98.9
TRICE with CV
74.7
82.3
98.8
TRICE with CV (not bootstrapped)
77.7
86.6
98.4
Table 2:
Average accuracies (columns 2 and 3) and fraction of training examples for which we generate correct rationales (column 4) on GSM8K. Direct prompt tuning is initialized with embedding sequence obtained from a few-shot prompt containing example question-answe pairs ("Q-A"). All remaining prompt-tuning methods from a few-shot prompt containing example question- rationnnoatiioneansseeee triples ("Q-R-A")) obtained randomly from the GSM8K training set bootstrapped described in the text.
initialized with embedding sequence obtained
loss used in supervised fine-tuning may place weight on style than substance; takes far more bits to encode how one expresses chain of reasoning than it does to encode the reasons themselves.
Initializing the soft prompt with prompt slightly improves performance on BBH, as does evaluating with self-consistency. By the end of training, TRICE has managed to generate at least valid rationale for almost all training examples, while STaR fails to generate valid rationales for almost 10% of training examples. Unlike in the experiments done Commonsense QA (Tamor al. 2019) by Zelikman al. (2022). STaR does not outperform the direct-prompted prompt-tuned model on BBH. This may be because each BBH task includes relatively little training data (150 examples as opposed CommonsenseeA'ss 9,741), and so in its inner loop STaR overfits to its relatively small set of bootstrapped rationales. TRICE, on the other hand, can overfit the small of questions but least has chance to generate a somewhat diverse set of rationales from those questions.
gennaneeeeertedd 3-shot exemplar nuestion-rational--ansss
One piece of evidence for this overfiting--rationale hypothesis is that the final step of its final inner loop, STaR (with bootstrapped initialization) achieves a training sequence-level (not per-token) cross-entropy loss of less than 0.06 all tasks, and of less than 0.01 on 19 out of 27 tasks. This implies that has learned to exactly reproduce a single set rationales with very high probability, which makes it very likely that it will generate those same rationales in the next iteration.
Figure compares estimates GSMSK of the average training marginal likelihood (i.e., how often proposal is accepted) and the validation accuracy with greedy decoding as function of number of

---

Tain Accuracy
Test Accuracy
Method
0.6
RS
TRICE no CV
0.4-
0.5-
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
step
Figure 2: Time-varying estimates (with loess smoothers) of average training-ser accuracy p(y ( and greedy-decoding validation-set accuracy for TRICE with and without subsampled control-variate gradient estimator ("TRICE CV" and "TRICE no CV" respectively) and four- cour-particle rejection sampling ("RS") on GSMSK.
training steps for rejection sampling and for TRICE with and without the control-variate scheme. The control-variate scheme improves average convergence speed, particularly towards the end of training as the probability of generating correct answers on the training set increases. Both versions of TRICE converge to high training accuracy much faster than rejection sampling.
5 Discussion
We proposed TRICE, method for tuning LLMs to be better at solving questtoon-nssweren tasks using thain-of-thought (CoT) prompting. By framing the CoT-prompted LLM as a latent-variable model, were able derive principled and effective fine-tuning method. When applied to GSM8K and BIG-Bench Hard (BBH) tasks, TRICE outperforms three strong baselines: direct prompt-tuning, STaR, and rejection sampling. While we derived TRICE in the context of CoT question-answering its basic MCMC-EM strategy could be employed more broadly, for example to tool-use problems.
medium-size LLM; it may be that behaves differently on smaller models, larger models, or when using other tne-tuning strategies. TRICE is gradient-based tuning algorithm, but many of the most capable LLMs proprietary. and their owners often do not provide any public mechanism for gradient- based fine- ane-tuning. This makes hard to evaluate how well TRICE would work when used with, say, GPT-4 (OpenAl 2023). Finally, our quantitative evaluations focused on whether the LLM could produce the right answer; we did not formally evaluate the quality of the reasoning in the rationales themselves (cf. Uesato al. 2022)
Limitations: We only evaluated TRICE with prompt-tuning
Broader Impacts: This work aims to improve the capabilities of LLMs by making them better able to answer questions accurately and transparently. However, more-capable LLMs may be used in malicious or unsafe ways, fine-tuning on uncurated cresstion-answeriig datasets may introduce biases into the models, and more widely used LLMs will contribute larger carbon footprint.
Rationales may make it easier for motivated users to judge the trustworthiness of LLM outputs. But many users may not read and critique an LLM's rationales, taking the mere existence of rationale as evidence of truth. If chain-of-thoughh rationales promote uncritical trust, they could lead to harm.
*We the cost per iteration of rejection sampling and TRICE with and without the control-variats scheme directly comparable: rejection sampling. we reduce the minibatch size by a factor four and generate four times as many proposals per example; for TRICE with the size L equal to the number of examples minibatch M (note that this does still involve subsampling, since each example could potentially contribute both correct and incorrect rationale the gradient estimate).
scheme, we set the gradient minibatch
10

---

We appreciate Daniel Freeman and Enrique Piqueras' contributions to the infrastructure that we used our experiments. We thank Kevin Murphy, Ben Lee, Brian Patton, and Jascha Sohl-Dickstein for helpful discussions.
Acknowledgements::
References
Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey,
P., Chen, Z., al. Palm 2 technical report. arXiv preprint arXiv:2305. 10403, 2023.
Bishop, C. M. Pattern recognition and machine learning. Springer, 2006.
Bomschein, and Bengio, Reweighted wake-sleep, 2015.
Burda, Y., Grosse, R., and Salakhutdinov, R. Importance weighted autoencoders. arXiv preprint NrXiv:1509.00599, 2015.
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X.,, Dehghani, M., Brahma, S., et al. Scaling instruction finetuned language models. arXiv preprint arXiv:2210.11416. 2022.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L.,, Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168, 2021.
Cover, T. M. Elements of information theory. John Wiley Sons, 1999.
Creswell, A., Shanahan, M., and Higgins, I. Selection-Inferencc:: Exploiting large language models for interpretable logical reasoning. arXiv preprint arXiv:2205.007712 May 2022.
Dohan, D., Xu, W., Lewkowyzz, A., Austin, J., Bieber, D., Lopes, R. G., Wu, Y., Michalewski, H., Saurous, R. A., Sohl-dickstein, J., Murphy, K., and Sutton, Language model cascades, 2022.
Geman, S. and Geman, D. Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Transactions pattem analysis and machine intelligence, 6.721-741, 1984.
Hastings, W. K. Monte Carlo sampling methods using Markov chains and their applications. Biometrika, 57(1):97-109, 04 1970. ISSN 0006-3444. doi: 10.1093/biomee/57.1.99 URL https://doi wttp://...../bimmme///oooooo 1.97
Hewitt, L. B., Le, A., and Tenenbaum, J. B. Learning learn generative programs with memoised wake-sleep. In Uncertainty in Artificial Intelligence, 2020.
Hol, J. D., Schon, T. B., and Gustafsson, F. On resampling algorithms for particle filters. In 2006 IEEE Nonlinear Statistical Signal Processing Workshop, pp- 79-82, 2006. doi: 10. 109/NSSPW. 43788247
Jouppi, N., Kurian, G., Li, S., Ma, P.. Nagarajan, R., Nai, L., Patil, N., Subramanian, S., Swing, A., Towles, B., et al. Tpu v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings. In Proceedings the 50th Annual International Symposium Computer Architecture, pp. 1-14, 2023.
Kingma, D. P. and Ba, Adam: A method for stochastic optimization. Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxxy.... http/aawwww.o.aaaaaaaeo/o wwwp/bgg//bbbbbeeee/oeee abs/1412 6980
Kingma, D. and Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are Zero-Shot reasoners. arXiv preprint arXiv:2205. 1916. May 2022.
Le, A., Kosiorek, A. R., Siddharth, N., Teh, Y. W., and Wood, Revisiting reweighted wake-sleep for models with stochastic control flow. In Uncertainty in Artificial Intelligence, 2019.
11

---

Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficien prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pp. 1045-3059) 2021.
Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., Wu, Y., Neyshabur, B., Gur-Ari, G., and Misra, V. Solving quantitative reasoning problems with language models, 2022.
Lievin, V. Deep Latent Variable Models for Natural Language Processing. PhD thesis, 2022.
Loshchilov, I. and Hutter, F. SGDR: Stochastic gradient descent with warm restarts. In International
Conference on Leurning Representations, 2017. URL htttp:://operreierrs wwwp://eeeffooooe Skq899cxx
Murphy, K. P. Probabilistic Machine Learning: An introduction. MIT Press, 2022. URL probml...
Murray, I. and Salakhutdinov, R. Notes on the kl-divergence between a markov chain and its equilibrium distribution. preprint, 2008.
Naesseth, C., Lindsten, F., and Blei, D. Markovian score elimbing: Variational inference with (pll
9). Advances in Neural Information Processing Systems, 33:15499-15510, 2020.
Nielsen. F. The stochastic EM algorithm: Estimation and asymptotic results. Bernoulli, 6(3):
457-489, June 2000.
Nye, M. Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., Sutton, C., and Odena, A. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint 12.0v 14, November 2021.
OpenAl. Gpt-4 technical report, 2023.
Owen, A. and Zhou, Y. Safe and effective importance sampling. Journal of the American Statistical Association, 95(449):135-145 2000.
Parisi, A., Zhao, Y., and Fiedel, N. Talm: Tool augmented language models, 2022.
Rajani, N. F., McCann, B., Xiong, C., and Socher, R. Explain yourself! leveraging language models for commonsense reasoning. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics. pp. 1932-494,, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.18853/v 1/P19- 1487. URL https: http:://acaaathhooocooooo org/ P19- 1487
Roeder, G., Wu, Y., and Duvenaud, D. K. Sticking the landing: Simple, lower-variance gradient estimators for variational inference. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances Neural Information Prucessing Systems, neurips. www.iee..ccppppppeeeeeeeep volume 30. Curran Associates, Inc., 2017. URL https::/proceeeddiei Paper pdf
Schick, T., Dwivedi- Yu, J., Dessi, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., and Scialom, T. Toolformer: Language models can teach themselves to use tools, 2023.
Shinn, N., Labash, B., and Gopinath, A. Reflexion: an autonomous agent with dynamic memory and self-refiection, 2023.
Shwartz, V., West, P., Le Bras, R., Bhagavatula, C., and Choi, Y. Unsupervised common- sense question answering with self-talk. In Proceedings of the 2020 Conference Empiri- cal Methods in Natural Language Processing (EMNLP), pp. 4615 0615-4629,, Online, November 2020. Association for Computational Linguisties. doi: 10. 1/.11665/112220.ccmm 1:20000mm11m00.mmllmmm.. main. URL https://aclanthool www.looy..oorr/oooo ptg/2020..c....oo http://ror//0......mmamaaaaaaaa. 373
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay. Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, H., Zhou, D., and Wei, J. Challenging BIG-Bench tasks and whether Chain-of-Thouuhh solve them. "arXiv preprint 2:X0022210.092611 October 2022a.
12

---

Suzgun, M.. Scales, N., Schärli, N., Gehrmann, S., Tay. Y. Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., et al. Challenging big-bench tasks and whether chain-of- thothh can solve them. arXiv preprint arXiv:22 10.09261, 2022b.
Talmor, A., Herzig, J., Lourie, N., and Berant, J. Commonsense@Q:: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4149-4155 Minncapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 116138/11/1199/ 1421. UtL/tttts://accaattooo org/
N19-1421.
Tieleman, T. Training restricted boltzmann machines using approximations to the likelihood gradient. In Proceedings of the 25th international conference Machine learning, pp. 1064-1071, 2008
Tierney, L. Markov chains for exploring posterior distributions. Annals of Statistics, 22(4):1701-177 December 1994.
Tucker, G., Mnih, A., Maddison, C. J., Lawson, J., and Sohl-Dickstein, J. Rebar: Low-variance, unbiased gradient estimates for discrete latent variable models. Advances in Neural Information Processing Systems, 30, 2017.
Turpin, M., Michael, J., Perez, E., and Bowman, S. R. Language models don't t always say what they think: Unfaithful explanations in chain-of-thought prompting. arXiv preprint arXiv:2305.044: 04388, 2023.
Uesato, J., Kushman, N., Kumar, R., Song, F., Siegel, N., Wang. L., Creswell, A., Irving, G., and Higgins, I. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275. 2022.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, I.., Gomez, A. N., Kaiser, E.,, and Polosukhin, I. Attention is all you need, Advances in neural information processing systems, 30, 2017.
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. Self-Consisteneyy improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11177, March 2022a.
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., and Zhou, D. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203. /171, 2022b.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., and Zhou, Chain of thought prompting elicits reasoning in large language models. "arXiv preprint arXiv:2001.11903, January 2022.
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, React: Synergizing reasoning and acting in language models, 2023,
Ye. X. and Durrett, G. Explanation selection using unlabeled data for In-Context learning. "arXiv preprint arXiv:2302.048/3 February 2023.
Zelikman, E., Wu, Y., and Goodman, N. D. Star: Bootstrapping reasoning with reasoning. arXiv preprint arXiv:2203. 200.:22233...455.. 2022.
Zhou, D.. Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Bousquet, O., Le, Q.. and Chi, E. Least-to-Mer Most prompting enables complex reasoning in large language models. "arXiv preprint arXiv:2205. 10625", May 2022.
13

---

Supplemental Material for "Training Chain-of-Thought via Patent-Variable Inference"
A Generalizing TRICE to Nondeterministic Likelihood Models
To apply TRICE beyond ceasstionnanswringg problems, we might want to use a nondeterministic likelihood model. For example, our desired output y might be a summary of a text and might be a scratchpad or outline. In situations like this, there might be many y's that are appropriate for a given æ and z. So from a modeling perspective, it could make sense to make p(y | 2) have nonzero entropy. But there is also a computational reason to prefer such a model: as the number of reasonable values that y could take given increases, the probability of sampling the precise y that was observed goes down at a rate that might be exponential in the size of the output space.
Fortunately, we can easily extend TRICE to accommodate nondeterministic likelihoods. The differ- ences are:
1. As discussed in Section 2.1 the acceptance probability for an independence sampler where p(2 the proposals are generated from p(= 2)) is a( =) = min p(yla,z) So instead of deterministically accepting correct proposed latents 2, we update the memory probabilisti- cally, always accepting proposals that make the observed y more likely, but sometimes also accepting proposals that make a little less likely.
2. The control-variate weights 3m are now function of the acceptance probabilities o:: 3m = - Em #m a(zm' ).,.))
The value of the control variate in this setting may be less than it is in the deterministic-lkklliood setting. Even if we learn model that consistently produces good latents z (in the sense that they lead to valid outputs y), this does not guarantee that it will consistently generate latents that are consistent with the particular y that was observed. For example, there might be multiple reasonable ways to outline a long text, some of which lead to different summary paragraphs. In this scenario, the acceptance probability may not converge something close 1, and the ceriance-reductton effect from the control variate will be modest.
B Derivation of the Control Variate Scaling Heuristic
Our analysis will focus on the variance of the ktth element 9& of our gradient estimator. To minimize clutter, we suppress subscript indices and define g' V log logp(=z x), 9 = Vo, log p(ž x). and g Vo, logp(2 z), where z' is the updated rationale, é is the proposed rationale, and is the previous rationale in memory. Note that g' ğ if the proposal & is accepted, and g' = if is rejected. We assume that the previous rationale is a sample from p(z xx x.,). We also define the shorthands y+ Blū c(ε,v) = 11. 9- = Blg c(ž.y) = 0]. Var(ī) Bla"). e = Var(ğ v) 1), v_ Var(ğ c(ε,y) = 0). Finally, we define the probability of accepting proposal = E(((E,
Our gradient estimator
9= Bg: Eli g+-
(6)
The variance of ğ is
Bi(g' g+ - (3) El(s - 9+4 3"E 2/3 - 994
U4 + 8 2,3
(8)
In Equation g we use the fact that Elg = and g+ is nonrandom and therefore independent of
g. Breaking out the cases where c(ž, v) = 0 (so g' = g) and c(ā,v) = -1 (so g' g). the rightmost expectation is
Elg' = = =Elg2² c(ž, = 1| + (1 - Bوو c(3, v) = 0
(9)
= - + / + - x)949_.
(10)
14

---

We can simplify the term on the right. Since 0 -Eig x9+ + - (1-17)-- (1 T)94-- = -xg
Plugging this into Equation (TO). we have
Eig's g] x(u+ + + - + = TU.
(11)
So the variance of simplifies
Var(g) (1 10--2,0 223 + B....
(12)
Taking the derivative with respect to 8 shows that this is minimized when
2(3u -T--) 0;
(13)
Plugging this back into Equation
12 gives the optimal variance v*:
ex = +
2nT
=-= U사
π
(14)
We can expand by considering the case where c(2.
(which happens with probability a and
the case where c( 2 y) = 0 (which happens with probability 1 - IT):
v = Elg"| = nU + π + (1 - x/2_ + (1 g)g²
nV - (1-1111)00 - + (1 π)リー T9
mu4 + (1 - -))) 9+9-
mu4 + (1 (1-T)0) T)U_ + 1-
(15)
π
.u4 + (1- - TuV- + (1 (2 ---= + 0(1 - x)²)
mu4 + - (1- +1 T)92 + +0((1 ))
mu4 + (1- (v-
(1- + +0(0++1111 ((1)))))
where in the second line we again use the fact that πÿ+ = --1 -)g_, and in the third-to-last line approximate with the first-order Taylor approximation 1 π + O(1 7) Thus, can write
U4 π
V + +(1
TU
+ + - π)
+g)+ +
0(11 - T)3)
(16)
- π
(v_ (v_ + 9 + 0(11 - +))).
TD
So the optimal variance v² is
(1 - n F+
2² e+
0(11 (0(11m))) T)3) 1-T (v_ + +00)++(111-
(17)
04 1
04
11-mii ()+ v_ + 9²) + ((1 ())))
By contrast, plugging our heuristic value of 3 = π into Equation 12 gives the suboptimal variance
=
xx = (1 2 + 220
(1 2n 2
)u+ + #² (TV + (1 (v- + 0(11 T))))
(18)
(1-27 +  ) + >2 (1 () ((1 77
= - -11-r)(u, + + (
where we use the approximation * -- -
to simplify
the x² and 3 terms. Thus, we conclude that w² and T.Th are the same up to 0(11==)) and so as the probability и of getting the correct answer increases, the suboptimality of setting / = goes down faster than the variance docs.
15

---

C On Gradient Estimators Based Solely on Incorrect Rationales
We adopt the same shorthands in Appendix B
Our analysis centers on the identity
0 O = TH+ + (1 - r)g-
(19)
which relates the gradient want to estimate 9+ (the expected gradient given that the rationale is correct) to 9- (the expected gradient given that the rationale is incorrect).
C.1
Variance of incorrect-rationale gradient estimators
Equation
19
implies that
π
9+
9-.
(20)
Deferring for the moment the difficulty estimating (see Appendix C.2 the variance of an estimator based on the right hand side of Equation 220
below), we can consider
c(2. v)
v))
9
π
π
(U_ 9-)
9
n
(21)
π
(V_ + g2)
(1
)²2
n2
(V_ =g22)
n²
If π is small, then this becomes
π (v_ + =g²======- 8- + O(T π
(22)
so that unless the variance v_ of incorrect rationales is very low, the variance of this estimator will be O(T-²), which is very high. By contrast, the variance of a gradient estimator based purely on correct rationales is simply v4. so unless the gradient variance for incorrect rationales dramatically lower than that for correct rationales, then if π is small then incorrect rationales will lead to much noisier gradient estimates.
On the other hand, if - π is small, then we have
(v_ + =g=g) = (1 T) + ( +0(11 -))
(23)
π²
which is likely significant improvement on the variance 14 of the correct-rationale estimator; in particular, it goes to zero as и approaches 1.
C.2 TRICE control variate as a debiased estimator based incorrect rationales
Equation (20) implics that, in principle, given unbiased estimates for 1 - π and x -1 we could compute gradient updates purely based on rationales that fail to obtain the correct answer. Unbiased estimates of - π are easy to obtain; we need only compute 1 - c(², y). But unbiased estimates of - are harder to come by.
Instead, we can compute biased estimator that ignores the term and then correct for the bias:
E -1-(1 c(2. v))øl = -(1 π T)g-
(24)
= -+ (1 - x)g+.
So the expected value of the estimator -(1 c(2, y))ğ too small by factor of (1 π T)94- We can correct this bias by adding in an unbiased estimator of it that uses (the gradient for a correct rationale from memory):
Ei-(1 - (ū - 9)] = g+ - (1- =) + 7)9+ -
(25)
This is precisely the gradient estimator that TRICE uses when the sampled rationale ğ is incorrect and B = Smaller values of 3 interpolate between this estimator and an estimator based purely on correct rationales.
16

---

Boststrpped 3-shot Q-R-A
3-shor
3-shor Q-A
TRICE (no CV) GD/SC
(RICE(CV)
RS-4
TRICE CV)
TRICE(CV)
Direcr
GD) SC
GD/SC
GD/SC
GD/ SC
GD SD/SC
GD
boolean, expressions
94.0/94.0
92.0793.0
88.07 89.0
91.0/920
89.0 /2.0
%
85.0
causal, jjdgemmnt
57.3/64.0
61.3/6L3
66.7/62.7
52.0/53.3
57.37 58.7
61.3/57.3
46.7
undentanding
71.0/730
88.0
disunhiguation..g
96.0/
97.0/97.0
83.01 82.0
97.0
89.0 50.0
97.0:
81.0
jock_Janguage
90.0 89.0
85.0/87.0
64.0 65.0
89.0
51.0 60.0
68.0/
50,0
falmeefflllcces
formal fallacies
19.0/53.0
66.0 65.0
55.0/53.0
47.0/445.0
52.01 59.0
11.0 46.0
52.0
geometric slopes
99.0
92.0/91.0
hyperibtto
95.095..
WUA1 993.
94.01 94.0
95.0/950
98. 1/ 99.0
98.01
93.0
logical deduction, live objocts
58.0660.
66.11 65.0
$2.01 53.D
/9.0/599.1
57.0 61
$3.07 51.0
56.01
logical, deduuction
objects
54,0/55,0
57.0
53.0 39.0/41.0
52,07 53.0
51.0.524 52.0
44,0
deduction objects
98.0
mowie recrmmenndation
96.0/66.
85.01 87.0
95.0/950
96.0
meltissep_ariibaane wrt_jarmeeic_tvvo two
2.0
2.01200
6.01
2.0/
29.0/ 60.0
$9.0 74,0
49.0
navigate
62.0/65.5
$9.0 60.0
67.0//110
61.0/620
/3.0/
$7.0
$9.0
object counting
82,0/84.0
$2.0/82.0
52.0152.0
77.0//5.0
90.0 90.0
94.0: 95.0
90.0
penguins_ in table
64.4/61.0
59.3
72.9/
83.1
reasening_ ahouut_cllrrd_.oom obbjcct_
76.0
80.0/8100
72.07 66.0/67.0
78.0 79.0
$7.0: 86.0
70.0
tuin, namees
36.0/86.00
94,0/94.0
85.07 86.0
90.0 $9.0
R3.0/
92.0/93.0
90.0
salient wwwnstationnnoooooo detection
66.0/63,0
68.066.0
54,0/544,0
39.0/64.0
62.0 66.0
73,0 74,0
64.0
speets inderstanding
96.0
100.0
89.1 90.0
93.07
93.0/
99.03 99.0
99.0
tomporal sequences
95.0 95.0
100.0/ 100.0
85.0/ 88.0
99.0/99901
100,0/ 100.0
100.0/ 100.0
100.8
traking shurlee objects live abjects
13.0
2X/1 26.0
17.01 15.0
17.0/ 16,0
22.0/ 27.0
16.0/ 17.0
20,0
macking shurtted objects, objects
11.00 13.0
18.0/ 14.0
20.0
tacking. shurited objects three, objects
49.0
44.0/43.0
30.0/320
85.0
web_ of__af
51.0/47.0
99.0 98.0
48.0 48.0
48.0 49.0
100.0/ 100.0
100.0/ 100.0
100.0
word sorting
30.0/49.00
55.0 55.0
53.01 53.0
56.0
16.0/5500 55.0
49.0 48.0
45.0/ 48.0
52.0
Average
67.8/68.0
72.8/7311
62.0/62.1
64.6/65.3
73.4/75.
76.7
70.4
Table 3: BIG-Bench Hard per-task results. TRICE (CV) uses the control-variate gradient estimator with subsampling: TRICE (no CV) uses the basic gradient estimator. RS-4 is rejection sampling with four particles. GD stands for greedy-decoding accuracy (%) and SC stands for self-consistency accuracy (%).
D BBH Per-Task Experimental Results
Table summarizes our experimental results for each task in BBH.
E Example Rationales
Figure iilustrates examples of rationales generated by the TRICE-tuned LLM the BBH task Logical Deduction Three Objects.
Figure illustraaess some examples of rationales generated by the TRICE-tuned LLM GSM8K. Although we did find examples where the L.M got the answer right for the wrong reasons, this was much less common on GSM8K than on BBH, since the numeric output space for GSM8K is much larger than that for the typical multiple-choice BBH task.
Figure 5 shows an example where the guide model (=(2 | x, y) in reweighted wake-sleep learns to closely mimic the prior model po 2)) until the very end of the rationale, at which point pastes in the correct answer.
F Method and Template Details
In this section, we present more details on the methods and templates that we used in the experiments.
To sample from pa(z r), we prompt the LLM with the template "Question: Let's think step by step. An". We cap the length of the gen- erated rationales at 1.25 times the length of the longest of the exemplar rationales used to initialize the soft prompt. To initialize the memory (i.e., sample from q(z x, x,y) in line 2 of Algorithm D on BBH we sample from the base model with a "guide" prompt of The answer is <ANSWER>, Let's think step by step. An". We use the same guide prompt to generate rationalizations in STaR, but with temperature (see below). On GSM8K, we instead initialize the memory with samples from pa(2 z), since we found that initializing the memory using prompt that includes the answer led to
ARRESTION>AAASSwer
the form "Question:
IRRESTION>AAASweerr
17

---

Example Rationales for Logical Deduction Three Objects BBH Task
An example where TRICE gets the answer right with good rationale:
Question: On shelf, there three books: brown book, yellow book, and an orange book. The yellow book is the leftmost. The the right the brown book.
orange book
Options:
(A) The brown book is the rightmost
(B) The yellow book is the rightmost
(C) orange book is the rightmost
TRICE answer: Brown book to the right of yellow book. And orange book to the right of brown book. So. the answer is (C).
An example where TRICE gets the answer right with bad rationale:
Question: A fruit stand sells three fruits: plums, apples, and loquats. The loquats are the cheapest. The plums are less expensive than the apples.
Options:
(A) The plums the second-most expensive
(B) The apples are the second-most expensive
(C) The loquats
the second-most expensive
TRICE answer: Apples cost more than plums, but less than loquat. So. the answer is (A).
An example where TRICE gets the answer wrong:
Question: fruit stand sells three fruits: mangoes, watermelons, and kiwis. The watermelons are less expensive than the kiwis. The kiwis are the second-most expensive.
Options:
(A) The mangoes are the most expensive
(B) The watermelons are the most expensive
(C) The kiwis are the most expensive
TRICE answer: The Mangoes are more expensive than the Watermelon. And are expensive than kiwis. So. the answer is (C).
Figure 3: Examples of rationales where TRICE gets the answer right, right but for the wrong reasons, and wrong.
18

---

Example Rationales for GSM8K
An example where TRICE gets the answer right with good rationale:
Question: Gabrielle sells eggs. On Monday she sells crates of eggs. On Tuesday she sells 2 times as many crates of eggs as Monday. On Wednesday she sells 2 fewer crates than Tuesday.
Thursday sells half as many crates of eggs as she sells on Tuesday, How many total crates of eggs does she sell for the 4 days?
TRICE answer: Monday
crates
Tuesday = 2 5 10 crates
Wednesday 10 2 crates
Thursday 1/2 10 = crates
Total 5 + 10 8 5= 28 crates
<answer>
28
</answer>
An example where TRICE gets the answer right with bad rationale:
Question:
Samantha's last has three fewer letters than Bobbie's last name. If Bobbie took two letters off her last name, she would have last name twice the length of Jamie's. Jamie's full name is Jamie Grey. How many letters are in Samantha's last name?
TRICE answer: Jamie's full name consists of 5 letters. If Bobbie took letters off her last name, she would have a last name twice the length of Jamie's, which is 10 letters.
So, Bobbie's last name has 10 letters.
If Bobbie's last name has 10 letters, and Samantha's last name has three fewer letters than Bobbie's, Samantha's last name 10-3 letters.
<answer>
7
</answer>
An example where TRICE gets the answer wrong:
Question: John drives for hours at speed of 60 mph and then turns around because he realizes he forgot something very important at home. He tries to get home in 4 hours but spends the first 2 hours in standstill traffic. He spends the next half-hour driving at speed of 30mph, before being able drive the remaining time of the 4 hours going at 80 mph. How far is he from home at the end of those hours?
TRICE answer: First, he drives 3 hours 60 mph 180 miles away from home.
Then, he turns around and spends 2 hours traffic (o mph)
Then, he spends 0.5 hours at 30 mph = 15 miles
Then, he spends 1.5 hours at 80 mph 120 miles
So, at the end of the 4 hours, he is 180 15 + 120 = 285 miles away from home <answer>
285 </answer>
Figure 4: Examples of rationales where TRICE gets the answer right, right but for the wrong reasons, and wrong.
19

---

Example Rationales for GSM8K Generated by RWS Guide
Question: Josh decides to try flipping house. He buys house for $80,00 0000 and then puts in ,50,000 in repairs. This increased the value of the house by 150%. How much profit did he make?
Guide answer:
He bought the house for 80000 and did 50000 worth of repairs. So total cost = 80000 50000 = 130000. Now, the increased value of the house is 150% of 80000 120000. So his profit = 120000 130000 -10000. <answer>
70000
</answer>
Model answer:
He bought the house for 80,000 He spent 50,000 on repairs.
So total investment is 80, 000 + 50,000 = 130,000
Value of house after repairs is 150% 80,000 1.5 80, 120,000
So total profit is 120,000 - 130, 000 100,000
<answer>
10000
</answer>
Figure 5: Example where the prompt-tuned RWS guide model pastes in the correct answer the end, contradicting the rationale up to that point. The rationales generated by the guide and model are almost identical up to the final answer block.
slower convergence and worse results; it may be that including the answer in the prompt sometimes encourages the model to produce untrustworthy explanations (Turpin et al. 2023).
To evaluate the correctness c(z, y) of rationale z given the answer in BBH we search the end of the rationale for final answers in the form "the answer is <ANSWER>, In GSM8K. we initialize the soft prompt to encourage the model to wrap its answers in "<answer>" and "</answer>" tags, and then extract the answer from those tags. To encourage the bootstrapped few-shot examples in GSM8K to follow this template, we the following example to the CoT prompt: "Question: What is plus 17\nAnswer: Let's think step by step. An1 plus 1 is 2. An<answer>>nn Annnn</nnnnnnnna /nn/nn/wss/\/n/
Figure and Figure show the bootstrapped few-shot CoT examples that used to initialize the soft prompt in the experiments.
TRICE. For all BBH tasks, we run TRICE for 500 steps with batch size M = 32 and do not use subsampling (i.e., compute L = 64 gradients per batch). We use the Adam optimizer (Kingma & Ba 2015) with an initial learning rate 1.0 and a cosine decay schedule (Loshchilov Hutter 2017) that reduces the learning rate by 10x over the first 450 steps. For GSM8K, run TRICE for 5000 steps with constant learning rate of 1.0, batch size M 128, and compute 128 gradients per batch.
STaR. We use adaptation of the STaR strategy proposed by Zelikman al. (2022). where do prompt-tuning rather than fine-tuning on all weights. The method alternates between updating its memory and retuning the model from scratch on the updated memory in an inner loop. We apply this procedure for 10 outer-loop steps. Following Zelikman et al. (2022), we start with 40 inner-loop optimization steps, increasing the number of inner-loop steps by 20% each outer-loop iteration up to maximum of 200 steps. If the training loss goes below 0.01 we break out of the inner loop. For STaR's inner-loop optimization, we use the same prompt-tuning initialization, Adam hyperparameters above, but with cosine decay from 1.0 to 0.1 over the course of each inner loop. To update the
20

---

STaR memory, we first try generating rationale from the prompt-tunee tuned model by greedy decoding. then if that rationale is incorrect fall back on a rationalization generated by greedy decoding from the same guide model we use in TRICE to initialize the MCMC memory, and finally if neither procedure generates a valid rationale we omit the example from the memory.
Rejection Sampling. We reduce mini-batch size by 4 and draw 4 rationales for each example in mini-batch. We use the same mini-batch size, train steps, and optimizer as TRICE for BBH and GSM8K experiments. In BBH, we use the initial learning rate 1.0 as in TRICE. In GSM8K, we use the learning rate 0.1 because achieved better results than learning rate 0.3, and the training procedure became unstable with learning rate 1.0.
CoT Prompt Tuning. To supervised CoT tuning, we prompt-tune the model to maximize the log- likelihoods of the training rationales given questions. The BBH datasets include very few exemplar rationales, so we cannot apply this strategy to BBH. On GSM8K, we use the same hyperparameters in TRICE except that carly-stop the algorithm after only 1000 train steps, since the model overfits badly when we run longer.
Direct Prompt Tuning. In this method, the model tries guess the answer directly without generating rationale; prompt-tuning to maximize the log-likelihood of the answers in this setup is straightforward, since there is latent rationale integrate out. We initialize the soft prompt using 3 examples from the training set and truncate its length to 64. The optimization procedure carried out over 150 steps with batch size 16 and the same Adam hyperparameters above, except that the cosine decay period is set to I 150 instead of 450. We found these adjustments to the hyperparameters from different training schemes
necessary reduce overfitting.
21

---

Bootstrapped 3-shot CoT Prompt for Formal Fallacies BBH Task
Question:
not always easy grasp who consuming which products. The following argument pertains to this question: First of all, being a frequent consumer KMS shampoo is necessary for being regular consumer of Yardley London soap. Next, whoever is regular user Joico shampoo is not a frequent consumer of KMS shampoo. All this entails that no regular consumer of Yardley London soap a regular user of Joico shampoo."
Is the argument, given the explicitly stated premises, deductively valid or invalid?
Options:
valid
invalid
Answer: Let's think step by step.
Regular of product always entails being frequent consumer of that product. Although a Joico shampoo user might still buy bottles never be regular
of Yardley London soap sometimes, he/she product of both soaps. So, the answer is valid.
Question: "Here comes a perfectly valid argument: To start with, Young Boys is not an ex-fan Real Betis Balompié. Now, whoever is a friend of FC Dynamo Kyiv is not an expert of BSC Young Boys. Hence, no friend FC Dynamo Kyiv is an ex-fan of Real Betis Balompié."
whoever an expert
the argument, given the explicitly stated premises, deductively valid or invalid?
Options:
valid
invalid
Answer:
Let's think step by step. You can friend FC Dynamo Kyiv but not be an expert of BSC Young Boys. So, the answer is invalid.
who is related to whom and in which ways. The following argument pertains to this question: Some classmate of Terri is workmate of Dolores. Whoever is not workmate of Dolores is an ancestor of Cheryl. So, necessarily, some ancestor of Cheryl is not classmate of Terri." Is the argument, given the explicitly stated premises, deductively valid invalid? Options:
Question: "It is not always easy
valid
invalid
Answer: Let's think step by step.
Reasoning: Some classmate of Terri is not a workmate of Dolores. Whoever not workmate of Dolores
an ancestor of Cheryl. So, necessarily, some ancestor of Cheryl is not classmate of Terri. Remember that Terri's classmate and Dolores's workmate can be in any case: One of the three can be any person. So, the answer is invalid.
Figure 6: The bootstrapped 3-shot Q-R-A soft prompt which used the formal fallacies BBH experiments.
22

---

2. The best athlete at the rival school won tvice as many as Jessie. So he Bootstrapped 5-shot CoT prompt for GSM8K
Question: There meatballs each spaghetti plate. If Theresa's sons each eat two-thirds the meatballs on respective plates, how many meatballs are still left on their plates altogether? Answer: Let's think step by step.
There 3x3 balls each the plates.
sons each eat the meatballs on their respected plates. <answer>
</answer>
Question: Joyce, Michael, Nikki, and Ryn have a favorite movie. Joyce's favorite movie is 2 hours longer than Michael's movie. Nikki 's movie is three times long Michael's movie, and Ryn's favorite movie 4/5 as long as Nikki's favorite movie. Nikki's favorite movie 30 hours long, calculate the total number of hours of their favorite movies together. Answer: Let's think atep by step.
Nikki
30 hours
Ryn's movie Nikki's 4/5 4/5 30 24
Michael's movie Nikki's / = 10
Joyce's movie Michael's + 2 10 2 12 Total Hours Michael's Nikki's Ryn's Jayce's
76
<anawer>
76
</answer>
Susan had a bouquet of dozen roses. She gave half to her daughter, and then placed the rest wilted.
Question:
The next day, one-third of removing wilted flouers, Answer: Let's think step by step.
the
many flovers remained in the vase?
Ansver STEP 1: 3 dozen is 36
2:
18
Ansver STEP 3: 18-6 12
<answer>
12 </ </answer>
Question:
Jessie won times many athletic awards as his buddy Scott, who 4 awards. The best athlete the rival high school tuice many awards as Jessie. How many awards did the rival vin? Let's think step by step.
Answer:
1. Jessie won 3 times many avards Scott, Jessie 3 4 12 <.<sentence>
awards while Scott
won 2 x 12 = 24 awards, 12 more than Jessie, while Scott won 4. <sentence> 3. The best athlete at the rival school won 24 awards. <answer> <answer>
</answer>
Question: James buys packs sodas that 12 aodas each. had 10 sodas already. He finishes the sodas week. How many sodas does he drink a day?
Answer: Let's think step by step.
There packs with sodas each and he has 10 left over.
12 60 10 70 (number total sodas)
70 / (7 day week) = 10 drinks per day
<answer>
</answer>
Figure The bootstrapped 5-shot Q-R-A soft prompt which used in GSM8K experiments.
23


Reviews:
[{"review_id": "lROnXEydDh", "reviewer": "Reviewer_Surj", "summary": "The paper proposes a prompt-tuning strategy that tries to maximize the marginal log-likelihood of generating a correct answer using CoT prompting. ", "questions": "See weakness part. ", "limitations": "See weakness part. ", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper presents a prompt-tuning strategy that aims to optimize the marginal log-likelihood of producing accurate answers through the utilization of CoT prompting. And the paper uses  simple Markov chain Monte Carlo (MCMC) expectation-maximization (EM) algorithm, memoized wake-sleep, and persistent contrastive divergence to address the challenge of sampling from the posterior. The results look like promising over other baselines. ", "weaknesses": "The author did not provide specific information about the model details. If the model size is close to 100 billion, it would be beneficial for the paper to include a baseline comparison with text-davinci-003 (Zero-shot or few shot learning). Otherwise, I feel it unnecessary to conduct prompt tuning on a large-scale model if its performance is still inferior to that of zero/few-shot learning using the same scale model but from GPT family. \nIf the author's intention is to compare with open-source models, it is essential to include Alpaca-CoT (https://github.com/PhoebusSi/alpaca-CoT), regardless of the specific model or fine-tuning technique they employ.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your suggestion to compare against text-davinci-003. Our model size is in the 40–75 billion parameter range, making it significantly smaller than text-davinci-003’s 175B parameters. We will add published text-davinci-003 BIG-Bench Hard (BBH) 3-shot results to Table 1 for comparison (these results are taken from [2]). The average accuracy results are summarized below:\n\n| text-davinci-003 (3-shot COT) | No tuning (3-shot COT) | TRICE (3-shot COT) | TRICE (bootstrapped COT) |\n|-|-|-|-|\n| 70.7 | 44.6 | 72.2 | 70.1 |\n\nWe used the same base model to obtain results for the last 3 columns. The table above demonstrates that even though we begin with a smaller, weaker model than text-davinci-003, by doing TRICE prompt tuning, we can roughly match (if bootstrapping an initialization) or slightly improve on (if using a few-shot human-generated CoT initialization) the performance of the larger model.\n\nAlso, as we discuss in the comment at the top of the page, we reran the TRICE BBH experiments using a higher-quality pretrained model, which improved our bootstrapped-initialization results from 70.1% to 72.8%.\n\\\n\\\n[2] “Scaling Instruction-Finetuned Language Models”, Chung et al., 2022, https://arxiv.org/pdf/2210.11416.pdf, table 19"}, {"review_id": "ngc8nMUCQx", "reviewer": "Reviewer_d8Yh", "summary": "This paper proposes treating rationales as latent-variables and considers the marginal distribution over answers, averaging over possible rationales. To do so, a MCMC procedure is proposed, in which rationales are proposed via an independence sampler. The approach is compared to STaR (Zelikman et al., 2022) as a baseline on 27 BigBench-HARD tasks. Rather than fine-tune all the model weights, prompt tuning is used a parameter-efficient alternative. The proposed approach is found to outperform STaR on the subset of tasks considered.", "questions": "* What is the specific $c(z,y)$ used in the experiments? It never seems to be defined— an example is given on line 71, but it’s unclear whether that’s the specific correctness function used in the experiments.\n\n* Line 80 says that “For example, the guide might prompt an LLM specifically to give an rationale for the answer”— was this the case for all experiments?\n\n* Line 296 says that \"an adaptation of the STaR strategy\" is tested-- what aspects specifically were adapted?", "limitations": "There is a nominal discussion of limitations but I think this could be significantly expanded. For example, MCMC can be sensitive to hyperparameters and mixing is hard to diagnose.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The derivations the authors present are detailed and comprehensive. The design choices are explained well and justified in context.\n\n* Compared to alternate methods, the approach advocated in this paper seems more principled.\n\n* The authors discuss a potential alternate approach (variational Bayes) and justify why it is not appropriate.\n\n", "weaknesses": "* As the authors acknowledge, TRICE is only tested on small datasets. The behavior of the comparison model is significantly different than the expected behavior; the authors speculate this is due to overfitting of the comparison model, but do not compare TRICE’s performance on Commonsense QA or reduce the number of inner loop steps to prevent overfitting.\n\n* While the overall approach is novel, it’s not clear that the improvements and alterations proposed present significant innovations on existing methods. Furthermore, I have some concerns about the sensitivity of the approach to hyper-parameters, such as the # of MCMC steps, initialization scheme, etc.\n\n* Some implementation details are unclear. The methods as-is would not be reproducible, and it seems that the authors do not plan to release their code, raising concerns about replicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for the thoughtful questions and suggestions. We hope that our response below will address your concerns.\n\n# Larger datasets\n\nWe added an experiment on GSM8K; see the top comment for details.\n\n# Sensitivity to number of MCMC steps\n\nIn Algorithm 1 (and in all of our experiments) we just do a single MCMC update per iteration. It might be interesting to explore treating the number of MCMC steps per iteration as a hyperparameter, but we feel that 1 is a natural default that allows for the prompt to be updated as often as possible. (This is somewhat analogous to the common practice of training VAEs using single-sample estimates of the per-example loss.)\n\n# Sensitivity to initialization\n\nPrompt tuning has been known to be sensitive to initialization from its inception; for example, random initializations tend to underperform initializations based on embeddings of real text sequences (e.g., [3]). We also found that initializing the soft prompt with human-generated rationales (rather than bootstrapped rationales) does slightly improve TRICE’s performance (Table 1), but both initialization schemes worked well.\n\n# Reproducibility\n\nSince our code uses a proprietary model, we cannot release the code we used in our experiments. We tried to include enough information in the main text for a reader to reimplement these experiments, but we may have missed a few details—we will add an appendix with pseudocode describing in more detail all of the algorithms that we ran, which may be a clearer format for providing this information than the dense text blocks in Section 4. We appreciate your identifying some specific points that were not clear in the “questions” section above (which we will address below)—if you notice any other details that are missing from the main text please do let us know!\n\n# MCMC hyperparameters and mixing diagnostics\n\nWhile MCMC kernels in general are often sensitive to hyperparameters and difficult to diagnose, the particular independence-chain proposal we use in TRICE has no hyperparameters (it simply makes proposals from the prior) and, in the binary-likelihood setting we consider, generates unbiased samples as soon as it accepts a proposal (since it is effectively doing rejection sampling on the posterior). We will emphasize these points in the final version.\n\n# Answers to specific questions\n\n> What is the specific about c(z, y) used in the experiments?\n\nThank you for pointing out that this was not precise. We compare the final tokens of $z$ to the tokens of the target “the answer is {y}.”. This is to account for various formats of the answers like “So the answer is (A).” or “Thus the answer is (A).” We will clarify this in the paper.\n\n> Line 80 says that “For example, the guide might prompt an LLM specifically to give a rationale for the answer”— was this the case for all experiments?\n\nYes, we used the same prompt template across all experiments. We described the template in the Experiments section (lines 267-272).\n\n> Line 296 says that \"an adaptation of the STaR strategy\" is tested-- what aspects specifically were adapted?\n\nIn the STaR paper, the authors performed fine-tuning of all LLM parameters while in our work, we use a parameter-efficient tuning method, namely prompt tuning. We will clarify this in the paper.\n\\\n\\\n\\\n[3] Li, Xiang Lisa, and Percy Liang. \"Prefix-tuning: Optimizing continuous prompts for generation.\" _arXiv preprint arXiv:2101.00190_ (2021).\n"}, {"review_id": "gO13V8dXu7", "reviewer": "Reviewer_SKio", "summary": "This work introduces a new method for training models for chain-of-thought prompting, aimed at maximizing the marginal probability of generating a correct answer using Markov-chain Monte Carlo, expectation maximization, and a novel control variate technique, along with prompt tuning. Much of the work is spent deriving and analyzing the method (e.g. compared to related methods like self-taught reasoner STaR), with a final section dedicated to testing the method and demonstrating very significant performance gains over vanilla prompt tuning (no CoT) and STaR.", "questions": "None", "limitations": "Limitations section is quite limited. It would be useful to see e.g. a discussion of the effects of rationales on user trust (will having rationales make users trust models more even if the rationales are wrong?)", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Principled, novel method along with extensive derivation. Particularly, this well motivates the need for taking a marginalization framing rather than the greedy approach often used before\n- It is generally very useful to see more theoretical justification provided in this area. Past works (e.g. STaR) provide many novel ideas, but it is useful to take a more comprehensive (and marginalization-based) approach to the problem\n- extensive theoretical/technical rationale for method over other options (STaR, variational inference)\n- very strong experimental results, demonstrating significant gains over STaR, another recent method for training for CoT prompting", "weaknesses": "- Fairly limited analysis of experimental results (mostly aggregate results on Big Bench tasks in the main paper, without too much specific analysis)\n- It would also be useful/interesting to see what the generated rationales look like: how do they differ from e.g. STaR, and how often are they correct in a human-interpretable way rather than just leading to a correct answer? Adding some information about this in the main paper would be useful\n- Fairly limited discussion of limitations", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive feedback!\n\n# Additional experimental results\n\nWhile there is limited space in the main text, we will add an appendix with a full table of per-task BBH results.\n\nAlso, we added an experiment on GSM8K; see the top comment for details.\n\n### Qualitative analysis of rationales\n\nThank you for this suggestion—we will add some example rationales (both correct and incorrect) from TRICE and STaR to the final version.\n\n# Discussion of user trust\n\nThank you for your suggestion that we add some discussion of the impact of rationales on user trust to the limitations section. On the one hand, providing rationales may make it easier for motivated users to judge the trustworthiness of LLMs for themselves; if an LLM's reasons are bad, we should consider its conclusions suspect. On the other hand, many users may not bother to read and critique an LLM's rationales, and may take a rationale's mere existence as evidence of correctness. While the focus of this work is on improving LLM capabilities, we should at least acknowledge that much work remains to be done on alignment questions such as this."}, {"review_id": "BzFPDfeReu", "reviewer": "Reviewer_AjuJ", "summary": "This paper explores considering Chain-of-Thoughts (CoT) as a latent variable and introduces TRICE: an MCMC-EM algorithm for optimizing CoT-latent-latent variable models using a Metropolis-Hastings algorithm (rejection sampling) coupled with a control variate. \n\nThe authors tested the model by fine-tuning a proprietary LLM on 27 BIG-bench hard tasks and report improvements when compared to direct prompt tuning and STaR. The authors tested the effectiveness of the control variate but did not test the method against more conventional optimisation methods.", "questions": "1. How exactly did you compute the gradients in the RWS experiment?", "limitations": "1. I stumbled upon this PhD thesis: [Deep Latent Variable Models for Natural Language Processing, Lievin 2022](http://vlievin.github.io/deep-lvms-for-nlp.pdf), which discusses similar ideas. You might not have been the first ones to consider that \"CoT methods are latent-variable probabilistic models\"", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, This paper discusses a well-defined issue and explores innovative solutions to the optimization problem, it was a refreshing and educative read, thank you!\n\nMain strengths:\n1. Optimizing LLMs as latent variable models has great potential and the authors present the problem well\n2. The authors introduce an innovative gradient estimator (MCMC-EM + control variate) which is computationally efficient (single sample per data point) and seems to work very well\n3.  The authors demonstrate progress on a subset of the MMLU dataset\n4.  Theoretical and experimental limitations are thoroughly discussed  ", "weaknesses": "Overall, the paper lacks structure and more extensive comparison with the existing literature on Monte Carlo methods. I believe not addressing these points would greatly reduce the impact of this paper. Nevertheless, I would happily accept this paper if these limitations are resolved!\n   \n1. The paper overall lacks structure. Although I appreciated the lengthy discussions, I think the main storyline could be made more clear by separating the main ideas from the discussions. \n   \n2. The optimization method is only presented using full text and Algorithm 1. This paper would convey its idea more clearly using a set of two additional equations: A) expression of the *true* gradients and B) Monte Carlo estimate (which translates into Algorithm 1).\n   \n3. Classic gradient estimation methods are not covered sufficiently and some of the statements in section 2.2 *Why not variational inference?* are misleading. \n\t1. Posterior collapse (discussed in Lucas et al., 2019) is an issue inherent to the optimization of the proposal distribution $q(z | x, y)$ and $q$ is in this paper never optimized.  \n\t2. The author discusses the limitations of the ELBO, but the ELBO only arises when using a single-sample estimate of the log-likelihood. When working with discrete latent variables, it is more common to use the Importance Weighted Bound (IWB). I.e., use a K-samples importance-weighted estimate of the likelihood (e.g., $\\log p(y | x) \\approx \\log \\frac{1}{K} \\sum_{j=1}^K \\frac{p(y, z^{(j)} | x)}{q(z^{(j)}|x,y)})$ -- as done in [\"Revisiting Reweighted Wake-Sleep for Models with Stochastic Control Flow\", Let et al., 2019](https://arxiv.org/abs/1805.10469)) which variance scales as $\\sqrt{1/K}$. See [\"Monte Carlo Gradient Estimation in Machine Learning\", Mohamed et al., 2020] for more background on optimization methods.\n\n6. The experimental comparison with RWS is not included in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for the thoughtful questions and suggestions. We hope that our response below will address your concerns.\n\n# 1. Structural improvements\n\nThis is a great suggestion. To reduce the risk that readers will lose the main thread, in the final version we will more clearly set apart as “remarks” discussions that are meant to provide context and intuitions.\n\n# 2. Providing true gradient and Monte Carlo estimate\n\nThe true gradient of the per-example log-likelihood is given in Equation 2, and the Monte Carlo estimate is given in Equation 3 and expanded in Equation 4. Perhaps we’re misunderstanding your suggestion?\n\n# 3, 4: VI, RWS, and the IWB\n\nThank you for pointing out some ways in which we can make this section clearer. We have rewritten section 2.2 to focus more on RWS and the IWB (which we also tried before settling on MCMC-EM; see below) than on VI. Hopefully the version below is an improvement—we’d be very grateful for any further feedback!\n\n## 2.2 Why not variational inference, reweighted wake-sleep, or the importance-weighted bound?\n\nWe considered three alternatives to the MCMC-EM approach that we pursue in this paper: variational EM (e.g., Bishop, 2006), reweighted wake-sleep (RWS; Bornschein & Bengio, 2015; Le et al., 2019), and the importance-weighted bound (IWB) used in importance-weighted autoencoders (IWAE; Burda et al., 2015).\n\nVariational EM is a common strategy for training latent-variable models, but variational inference with discrete latent variables is challenging (e.g., Tucker et al., 2017). [pointer to VI discussion?] RWS is an attractive alternative that avoids high-variance score-function gradients; it proceeds by sampling $M$ samples $z_{1:M}$ from a guide model $q_\\phi(z\\mid x, y)$, assigning the samples weights $w_m\\propto \\frac{p_\\theta(y, z\\mid x)}{q_\\phi(z\\mid x)}$, and updating both the model parameters $\\theta$ and the guide parameters $\\phi$ to maximize the reweighted log-probabilities $\\sum_m w_m \\log p_\\theta(z_m\\mid x)$ and $\\sum_m w_m \\log q_\\phi (z_m\\mid x)$. Unfortunately, we found that RWS training sometimes produced models that produced degenerate zero-length rationales $z$. Digging deeper, we found that the weights $w_m$ tend to be larger for shorter sequences $z_m$, so the model and guide learn to produce shorter and shorter sequences until they consistently produce empty rationales.\n\nWhy do longer sequences tend to get lower weights? We can write the unnormalized weights as $\\tilde w_m = (c(y, z_m) + \\epsilon)\\frac{p_\\theta(z_m\\mid x)}{q_\\phi(z_m\\mid x)}\n= c(y, z_m)\\prod_{t=1}^{T_m} \\frac{p_\\theta(z_{m,t}\\mid x, z_{m,1:t-1})}{q_\\phi(z_{m,t}\\mid x, z_{m,1:t-1})}$,\nwhere $T_m$ is the length of $z_m$, and $\\epsilon$ is added to address the case where none of the $M$ samples are correct. If there is a mismatch between $q(z_{m,t}\\mid x, z_{m,1:t-1}))$ and $p(z_{m,t}\\mid x, z_{m,1:t-1}))$, then $\\frac{p_\\theta(z_{m,t}\\mid x, z_{m,1:t-1})}{q_\\phi(z_{m,t}\\mid x, z_{m,1:t-1})}$ will typically be less than one, with a few rare exceptions with extremely high weights ensuring that $\\mathbb{E}_q[p(z\\mid x)/q(z\\mid x)]=1.$\n\nIf these exceptions are rare enough that they do not typically appear in a sample of $M$ sequences $z_{1:M}$, then the normalized weights $w_{1:M} = \\frac{\\tilde w_{1:M}}{\\sum_m \\tilde w_m}$ will tend to assign higher mass to shorter sequences unless those shorter sequences are much less likely to be correct.\n\nFinally, we considered optimizing the $M$-sample IWB\n\n$\\mathcal{L}(x, y, \\theta) \\triangleq \\mathbb{E}_{p}[\\log \\frac{1}{M}\\sum_m\\frac{p(y, z\\mid x) + \\epsilon}{p(z\\mid x)}]$\n\n(where we use $p_\\theta(z\\mid x)$ as an importance-sampling proposal, $M$ is the number of draws from $p_\\theta(z\\mid x)$ and $\\epsilon$ is added to avoid taking the log of zero). This works well as long as the number of samples per example $M$ is large enough, but sampling several rationales per example carries a high per-step cost; the independence-chain MCMC strategy we use in TRICE lets us do $M$ times as many parameter updates per sample from $p_\\theta(z\\mid x)$.\n\n> How exactly did you compute the gradients in the RWS experiment?\n\nWe draw $M$ samples $z_{1:M}$ from a guide model $q_\\phi(z\\mid x, y)$, compute the unnormalized weights \n\n$\\tilde{w}_m = \\frac{p(z\\mid x)\\epsilon^{(1 - c(y,z))}}{q(z\\mid x,y)},$\n\nin the log-space, assign the weights\n\n$w_m = \\mathrm{stopgradient}(\\tilde{w}_m / \\sum_m\\tilde{w}_m),$\n\nand optimize the loss $\\sum_m w_m \\left(-\\log p_\\theta(z_m\\mid x) - \\log q_\\phi(z_m\\mid x,y)\\right)$. This objective gives us an estimation\n$\\sum_m -w_m \\nabla_\\theta \\log p_\\theta(z_m\\mid x)$ for the gradient w.r.t. $\\theta$ and  $\\sum_m -w_m \\nabla_\\phi \\log q_\\phi(z_m\\mid x,y)$ for the gradient w.r.t. $\\phi$.\n\nThe $\\epsilon^{(1 - c(y,z))}$ factor represents the likelihood $p(y\\mid x,z)$. The likelihood of arriving at the correct answer given a correct rationale is 1 and the likelihood of arriving at the correct answer given a wrong rationale is $\\epsilon$. This has the same effect as adding $\\epsilon$ to the binary likelihood that we described above: to avoid taking the log of zero when all rationales are wrong. In the experiments, we set $\\epsilon = 0.001$.\n\n# Lievin thesis\n\nThanks for the reference to Lievin thesis! We will cite it in the revision. Although that work describes a latent-variable chain-of-thought model, we note that the inference methods used for the COT model are relatively simple (in contrast to other models from the thesis), similar in spirit to self-consistent COT. Our work exploits the connection to probabilistic modeling to apply more sophisticated inference algorithms; we are not aware of previous work that takes this strategy. In contrast, previous work focuses on simple Monte Carlo (self-consistent COT, Lievin 2022), or rejection sampling (STaR).\n"}, {"review_id": "g8hLnuc9ma", "reviewer": "Reviewer_zJeM", "summary": "This paper introduces a principled and practical approach to boost the training of the \"chain-of-thought\" through the inference of latent variables. Instead of the typical variational inference, the authors opt for a MCMC-EM method to circumvent the issue of posterior collapse, a common occurrence in auto-regressive models with latent variables, which could be exacerbated when dealing with large language models. By treating rationales as latent variables, this methodology enables training without the need for detailed rationales, which can be expensive to produce.  Moreover, the authors propose a control-variate technique that reduces training variance while maintaining unbiasedness. The theoretical underpinnings of their approach are validated on the BIG-Bench Hard dataset.", "questions": "1. For accept rate in Metroplis-Hastings, why the transition kernel $p(\\tilde{z},y|x)$ is not conditional on previous state $z$?\n\n2. For sampling from intractable posterior $p(z|x,y)$, what if you directly sampling from this distribution using finite-step MCMC as $p(z|x,y)\\propto p(z|x)p(y|z,x)$? You do not need to add more experiments. Could you explain the difference?\n\n3. For control variate, in appendix you mentioned it may not be that useful in the stochastic setting. For figure 1, I want to know that suppose you train long enough, will the training estimate becomes the same for two settings (with or without control variate) based on your claim in Eq. 3? Meanwhile, what's the notation meaning for expectation over MCMC?\n\n4. How many tokens you use for latent variable $z$?\n\n5. While I commend the authors for validating their approach on an important benchmark, an aspect I deeply appreciate, I believe that testing on a more diverse dataset would bolster the paper's credibility and appeal. This is particularly true when making a comparison with [1]. \n\n6. Do you have failure case analysis and do you need human evaluation?\n\nI would be more than willing to raise my review scores if the authors could provide further clarification and deeper insights about the paper. \n\n[1] STaR: Self-Taught Reasoner Bootstrapping Reasoning With Reasoning ", "limitations": "The authors has discussed this paper's limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Although the soft-prompt approach is widely used, this proposal introduces an innovative perspective on 'prompt-finetuning' utilizing a chain-of-thought.\n\n2. The assumptions and derivations presented are straightforward and intuitive with only minor confusion.\n\n3. The authors thoroughly explore the links between their work and previous studies, while also critically examining the limitations of their own findings.\n\n4. The authors validate their theories by applying them to the significant benchmark of BBH.", "weaknesses": "See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful questions! We will try to answer them below; please let us know if anything remains unclear or if there are particular points that you would like to see emphasized in the final paper.\n\n# 1. Acceptance probability depends on previous state\n\nOur use of a binary (deterministic) likelihood and identifying the proposal $q(z\\mid x, y)$ with the prior $p(z\\mid x)$ dramatically simplifies the acceptance probability, so that we only need to consider whether or not the proposed state leads to a correct answer when deciding whether or not to accept it. If we used a “soft” likelihood function as discussed in Appendix A, we would indeed need to look more closely at the previous state to compute the acceptance probability.\n\n# 2. Why not use finite-step MCMC?\n\nThe main reason we only take a single MCMC step per iteration is to minimize the per-update cost of the algorithm; the cost of drawing a sample is comparable to the cost of a gradient update for the prompt, so it seems natural to balance these costs.\n\n# 3. Is the control variate necessary late in training?\n\nIf anything, the control variate is probably most helpful late in training, since the correlation between the control variate and the base gradient estimator increases with the accuracy of the model. There may be subtle effects on what kinds of local optima the two gradient estimators tend to yield, but these might be reduced by using a smaller learning rate and more training steps.\n\n# 3. What’s meant by $\\mathbb{E}_\\textrm{MCMC}$?\n\nThe expectation is with respect to the randomness in the MCMC proposal, which for TRICE is just the proposed sample from $p_\\theta(z\\mid x)$. We will change the notation to make this clearer.\n\n# 4. How many tokens per rationale?\n\nWe cap the number of tokens per rationale at 1.25 times the length of the longest of the three exemplar rationales used to initialize the soft prompt. The goal is to ensure that we have plenty of space to generate good rationales without unnecessarily increasing the cost of sampling and gradient evaluation.\n\nThank you for pointing out that this information is missing from the paper! We’ll include it in the next version.\n\n# 5. Other datasets.\n\nSee the comment at the top of the page for results on GSM8K.\n\n# 6. Failure cases.\n\nWe will add examples of rationales that lead to both correct and incorrect answers in an appendix.\n\n# 6. Human evaluation.\n\nFormal human evaluation is beyond the scope of this paper, but will be important in the future in order to determine to what extent models trained using TRICE get answers right for the right reasons. We will add some discussion of this point to the Limitations section.\n"}]

Author rebuttal:
We sincerely thank the reviewers for their many helpful and thoughtful comments.

Most of our responses are in the per-review comments, but we want to highlight here two sets of new experimental results that we will add to the paper.

# BIG-Bench Hard results with a stronger base model

After our initial submission, we reran the TRICE BBH experiments using bootstrapped initialization and a higher-quality pretrained model of similar size. Like our original base model, this model is in the 40–75B range*, significantly smaller than GPT-3's 175B parameters.

When prompt-tuning this improved model, TRICE gets an average greedy-decoding accuracy on BBH of 72.8%, significantly better than our previous 70.1%.

# GSM8K results

After our initial submission, we ran the algorithms on the GSM8K dataset [1], which includes 7,473 training examples and 1,319 testing examples, using the same higher-quality pretrained model discussed above. Below is a summary of the results with bootstrapped-initialization CoT (except for direct tuning):

| Direct tuning | No tune COT | STaR | SFT | SFT (with SC) | TRICE | TRICE (with SC) |
|-|-|-|-|-|-|-|
| 5.8 | 22.4 | 40.5 | 56.3 | 73.1 | 60.0 | 75.6 |

SFT: Supervised fine-tuning. The training dataset includes example rationales for each question; we prompt-tune the model using these pairs of questions and rationales.
SC: Self-consistency majority voting using 40 samples.

STaR performs better than direct tuning on this task, but still underperforms compared to TRICE and SFT. On the other hand, TRICE outperforms SFT even though TRICE does not train on human-generated rationales. We conjecture that this may be because SFT with cross-entropy loss gives more weight to the style of the rationales than to their content, whereas the marginal log-likelihood loss that TRICE minimizes is insensitive to precise wording.
\
\
\* We originally disclosed a larger range of 15–100B parameters, but proprietary models in this smaller 40–75B range have been publicly disclosed by many organizations such as Anthropic, Bloomberg, DeepMind, Google, Meta, and Stability, so we feel comfortable providing this narrower range.

[1] Cobbe, Karl, et al. "Training verifiers to solve math word problems." _arXiv preprint arXiv:2110.14168_ (2021).

Meta-review: The paper treats chain-of-thought-prompted question-answering LLMs as probabilistic latent-variable models, so we can derive a principled, simple, effective way of tuning them to generate better rationales without training on human-generated rationales. Most reviewers tend to accept the submission although one of them tends to reject it. Reasoning of LLMs is important for the current models. The paper proposes an interesting perspective to view the COT solution. Some LLM researchers would be interested in the proposed method. Moreover, the author response can be included in the revised version.
Decision: Accept (poster)