Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models
Authors: Zhanhui Zhou
Zhixuan Liu
Jie Liu
Zhichen Dong
Chao Yang
Yu Qiao
Venue: NeurIPS 2024 poster (2024)
Keywords: large language models
alignment
weak-to-strong generalization
Primary area: natural_language_processing
TL;DR: We present weak-to-strong search, framing the alignment of a large language model as a test-time greedy search to maximize the log-probability difference between smaller tuned and untuned models while sampling from the frozen large model

Abstract: Large language models are usually fine-tuned to align with human preferences. However, fine-tuning a large language model can be challenging. In this work, we introduce $\textit{weak-to-strong search}$, framing the alignment of a large language model as a test-time greedy search to maximize the log-probability difference between small tuned and untuned models while sampling from the frozen large model. This method serves both as (1) a compute-efficient model up-scaling strategy that avoids directly tuning the large model and as (2) an instance of weak-to-strong generalization that enhances a strong model with weak test-time guidance.
Empirically, we demonstrate the flexibility of weak-to-strong search across different tasks. In controlled-sentiment generation and summarization, we use tuned and untuned $\texttt{gpt2}$s to improve the alignment of large models without additional training. Crucially, in a more difficult instruction-following benchmark, AlpacaEval 2.0, we show that reusing off-the-shelf small models (e.g., $\texttt{zephyr-7b-beta}$ and its untuned version) can improve the length-controlled win rates of both white-box and black-box large models against $\texttt{gpt-4-turbo}$ (e.g., $34.4\% \rightarrow 37.9\%$ for $\texttt{Llama-3-70B-Instruct}$ and $16.0\% \rightarrow 20.1\%$ for $\texttt{gpt-3.5-turbo-instruct}$), despite the small models' low win rates $\approx 10.0\%$.

Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models
Zhanhui Zhou*i, Zhixuan Liu", Jie Liu, Zhichen Dong, Chao Yang!, Yu Qiao
Shanghai Artificial Intelligence Laboratory Core Contribution, *Corresponding Author asap www.uzzzoooulccccoooo com, yangchao@pjjab. org. Code: https://gitte httpp://ggttuubb.cm//22iiii http///wakkkooooo stto-strong-ssaaee
Abstract
Large language models are usually fine-tuned to align with human preferences. However, fine-tuning large language model can be challenging. In this work, we introduce weak-to-strong search, framing the aligmment of large language model as a test-time greedy search to maximize the log-probability difference between small tuned and untuned models while sampling from the frozen large model. This method serves both as (1) compute-efficient model up-scaling strategy that avoids directly tuning the large model and as (2) an instance weak-to-strong generalization that enhances strong model with weak test-time guidance. Empirically, we demonstrate the flexibility of weak-to-stron search across different tasks. In controlled sentimment generation and summarization, we use tuned and untuned gpt2s to improve the alignment of large models without additional training. Crucially, in more difficult crntruction-oolloiinn bench- mark, AlpacaEval 2.0, we show that reusing off-the-shelf small models (e.g., zephyr-7b-bele and its untuned version) improve the length-controlled win rates both white-box and black-box large models against torro (e.g., 34.4% + 37.9% for --maa---7------------ Ept-3.5-turie www---ttriioouuutttttttetee instruct), despite the small models' low win rates se 10.0%.
and 16.0% 20.1% for
Introduction
Learning based algorithms [1, 2, 3, 4, 5] have become the standard approach for aligning large language models (LLMs) with human preferences [3, 6, 7, 9, 10]. However, fine-tuning large language models is resource-intensive and difficult to implement [4]. These challenges have motivated recent studies on search-based algorithms that keep the large language models frozen and steer their decoding with test-time guidance [11, 12, 13, 14, 15, 16, 17]. Typical examples of search- based algorithms include rejection sampling [16, 17] and Monte Carlo Tree Search [18, 19]. These search- based algorithms are promising as they can reuse the same guiding signal to steer the decoding of any large language model without additional training. However, existing search-based methods either simplify the search over tokens bandit problem [16, 17]. which limits their steerability, require a value function learned from scratch to address preference reward sparsity and prune search space [13, 18, 14], which can be as difficult fine-tuning large language model.
To make search-based algorithms better suited for aligning large language models, we introduce weak-to-strong search, simple algorithm that frames the alignment of a large model as test-time search over the log-proo tog-probabilitees of small language models. This algorithm makes two contributions: (1) First, it builds on the theoretical foundation of the token-level MDP for alignment [20], using the
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

---.: zephyr-/t-beta (tuned)
#istral-7b-stt-bers (untuned)
Alpaca L.C. Win Ratn
Base
30-
Weak-to-strong search
Llama2-BB Llama2-70B Llama3-8B
Llama3-70B
CPT3.5
Figure 1: Weak to-strong search enhances the alignment of large models through test-time guidance from small models (dashed lines). This method is applicable to white-box models that use the same or different vocabularies as the small models, as well as to black-box models. We present the results for the instruction-tuned models from each family (e.g., Llama2-7B denotes Llama-2-7b-che
log- nee-probability difference between small tuned and untuned language models both reward and value [4, 20] to guide the decoding of large model (Section 4.1). Theoretically, this formulation is suitable for search as it converts the otherwise sequence-level sparse preference reward function to per-token dense reward function, which can be summed up as a value function [20]. Practically, this formulation allows the reuse of off-the-shelf small tuned and untuned language model pairs steering forces, avoiding the need to train a reward or value model from scratch. (2) Second, it introduces a beam search variant, Bhunk-leve) Beam Search (CBS), tailored for optimizing the proposed search objective. CBS guides the large language model towards high-reward regions by alternating between sampling from the frozen large model and expanding promising states evaluated by the small tuned and untuned models (Section 4.2). Especially, when the small models are weaker than the large model, our method can be viewed as an instance weak-to-strong generalization [21] that makes the strong model stronger with weak test-time guidance (Figure 1).
Empirically, verify weak-to-strong search's flexibility in various tasks (Section 5). First, in controlled-sentimen generation [22] and summarization [2], our method uses small language models of 124M parameters (i.e., gpt2) effectively steer much larger language models from the GPT-2 (e.g., gpt2-x1) [23], Llama-2 [7] and Llama-3 [24] families, at least as effective as existing methods. Then, in a more difficult fnstrwitionn-ollloiing benchmark, AlpacaEval 2.0 [25], show reusing off-the-shelf small models (e.g.. zephyr-7b-beta and its untuned version) as test-time guidance can significantly improve the length-controlled win rates of both white-box and black-box large models against gpt-4-turbo (e.g, 34.4% + 37.9% for Llama-3 3-00B- InBBructt and 16.0% + 20.1% for gpt-3. w-t-------iiie
www.---tooo-iiirreeeee instrrutttt despite the small models' low win rates 10.0% (Figure I).
Related Work
Large unsupervised language models trained on internet-seue scale corpus acquire broad knowledge and abilities [26, 27, 28). However, these large pre-trained language models may not always align with human values. To instill the desired behaviors into language models, most existing methods fine-tune these pre-trained language models on human comparisons of model-generated responses [1, 2, 3, 7, 24, 6, 29]. Despite these successes, fine-tuning a large language model requires substantial computational resources and engineering effort. These problems
compounded by the reality that different humans have different values [3, 30, 13, 31, 32, 33], as is nearly impossible to train a new large language model from scratch for individual preference. In light of these issues, our work takes search-based approach, folding as much of the complexity of alignment as possible into the decoding phase. This allows to keep the large pre- pre-trained language models frozen, steering their outputs test time with only small models that are easier to obtain.
Framing alignment test-time search to maximize reward function is not novel formulation. However, most existing works either simplify autoregressive decoding as a bandit problem [16, 17). which limits their stecrability, require value function learned from scratch to handle sparse

---

preference rewards and prune search space [13, 18], which can be as difficult as training a large language model from scratch. Our work avoids these issues by parametrizing the sparse preference reward function with the log-probability difference between small tuned and untuned language models [4]. This parametrization not only simplifies the search objective, allowing a simple greedy search algorithm to generate good results, but also reuses off-the-shelf models as steering forces, eliminating the need to train a reward or critic model from scratch.
Concurrently with our work, Rafailov et al. [20] proposes a token-level MDP interpretation for language model alignment, demonstrating that greedy probability search over trained language model can achieve improvements over regular decoding. Our work builds on their theoretical foundations and proposes a practical greedy search algorithm designed for weak-to- to-stonng guidance.
The idea of using small language models to align large language models has arisen in many recent works. The most related is proxy emulated fine-tuning [34, 12, 11,35], which uses the distributional difference of a small tuned and untuned model pair to modify the output distribution of a large model, approximating the output the directly tuned large model. However, these methods require that both small and large models share the same vocabulary, limiting their practical applications. In contrast, our approach does we perform a tree search that periodically prioritizes the most promising states for further expansion (as evaluated by the small models) while sampling from the frozen large model's s distribution. Thus our approach does not require shared vocabulary and is applicable to black-be box language models.
modify the sampling distribution of the large model at the token level. Instead,
3 Preliminaries
In this section, we introduce the mathematical formulation of alignment (Section 3.1) and describe the duality between language models and reward functions (Section 3.2).
3.1 Aligning Language Models with Human Preferences
The alignment of language models is typically cast as a KL-constrained optimization problem [l]: arg max xxxxp(x))y y]]
(1a)
s.t. Ex-p(xx) [KL (T(y x)) ( x))] S e,
(lb)
where p(x) is distribution of prompts, y is the complete language model response, is preference reward function that encourages human-preferred preferred responses, and DkL limits how far the optimized language model can deviate from the reference (untuned) model Tee.- There of alignment algorithms: (1) search-based algorithms that optimize Eq. 1 with graph- based search during inference [16, 13, 18, 19, 11, 12], and (2) learning- based algorithms that optimize Eq. 1 through gradient descent, aiming for a parametrized optimal language model [1, 36, 4, 37]. Our work falls in the first category, proposing search-based algorithm capable of using small language models to guide the decoding of a large language model to align with human preferences.
main categories
3.2 Duality between Language Models and Reward Functions
The analytical solution to Eq. 1 can be obtained through the following Lagrangian [38, 39]:
[(T, 3) = yx--(()) yr(x,)) + 3 (e DKL (- x)) || Tye x)))],
(2)
which has a well-known closed-form solution that expresses a duality between the reward function
y) and the optimal language model x >'(y x) [40, 41]:
y) = Blog π (y x)
+ Bog ((x).
(3)
T (y
x)
denotes the partition function. One takeaway from this duality is that we can always express a reward function using tuned and untuned language models: (1) If a reward function is given [1, 2, 31, we can first obtain the optimally tuned language model under this reward function with any learning- models (Tr", Teet) to reparametrize the reward function [42]; (2) If dataset is given from which the reward function can be derived, we can then directly parametrize the reward function with the tuned and untuned language models (πT", Tret) during reward modeling [4].
where 2(x)= Ey Frerly x) exp ()(()))) y)
algorithms, and then use the tuned and untuned
3

---

4 Weak-to-Strone Search
In this section, we introduce weak-to-strong search, search-based algorithm that aligns large language model by searching over the log-probability difference between small tuned and untuned language models. First, we discuss how using language models to parametrize the preference reward function (Eq. 1) makes the reward-maximization problem solvable by a simple greedy search algorithm (e.g., beam search) (Section 4.1). Then, we introduce a practical beam search method, Chunk-level Beam Search (CBS) (Section 4.2), that balances reward maximization and KL minimization, which is applicable to steering both white-box and black-box large language models.
4.1 Language Models as Both Reward and Value Functions
One practical challenge for search-based alignment algorithms is the sparsity of the preference reward signal. The preference reward function r(x.)) based on the Bradley-Terry model [43], only emits a terminal reward when the model response is complete. Search- algorithms often struggle without any intermediate rewards or a value function providing intermediate guidance [44, 45]. However, if we parameterize this sparse reward function with language models (Section 3.2), we can obtain both dense reward function and value function simultancouslly
Language models as a dense reward function. To obtain a dense reward function, we leverage the duality between the sparse preference reward function and the dense language model probability (Eq. 3). By explicitly factorizing the log-probability of a complete response y under the language models, we obtain sum-of-rewards style formulation for Eq. 3:
Iy]
x. xet) (y
n (y,
r y) =
log
+ Blog Z(x),
(4)
xxt)
where yee denotes the response tokens from to ( 1, and the last response token Ylyl is the EOS token. Combining Eq. and 4, we rewrite the original objective with per-token reward function:
lyl
n (y,
x. yxe)
arg max xx
log
(5a)
(y
y,t
S.t. Ixxp(x) [DLL (T(Y)
x) || Thase (y
x))] S
(5b)
where I and (xx are omitted as they do not influence the optimal solution. It is important to note that the reference model that parametrizes the reward function (Frat) (Eq. 5a) and the reference model that constrains the test-time search space (Thase) (Eq. 5b) can be different. Practically, decoupling the reference models useful as it allows using tuned and untuned language model pair namely (I²", Teet) to steer the decoding of any base language model Thuse without retraining.
Setting aside the KL constraint (Eq. 5b) for now, we can apply existing search algorithms like beam search [45, 20] to optimize Eq. 5a. Beam search is often criticized for leading to myopic solutions [45], as it tends to greedily prioritize states (x, y') (y' is incomplete)* with high cumulative reward log n (y' | x) log T x) midway through generation, which is generally viewed as poorly correlated with the overall return we care about. While this criticism is valid for most MDPs. we argue that in the token-level MDP [20] of our case, the cumulative reward mid-generation is actually a reliable indicator of the long-term value, making beam search less myopic.
Cumulative reward under language models as a value function [20]. Appendix A shows that: (-V*(x) + (x, if y' is incomplete
π (y' Tee (y')
x)
log
(6)
(x) (xx))r(x,y))
if y' is complete,
where V*(x, y") denotes the value function, predicting the expected terminal reward under the optimal
T" in the original KL--costrrined sparse reward problem. Although V* (x,y') is not necessarily
achievable by the searched policy, it approximates how good the state (x, yx) is in the long run. In other words, continuing from the state (x,y') of high cumulative reward log πr "y" x) - log (
x) is likely generate complete response y with high overall return log π. (y x) log ( x).
denotes complete response, while y' denotes response that can be either incomplete complete.

---

Prompt
Please write movie review
Hypathesis Set M4
Successor Chunk
Top-W
Hypothesis Set H4+
Yz hate the boring plot
0.8
yo My wife and Irally like the fun story
My wife and really
yz like the fun story
5.1
yr. faund it too long
0.4
y wife and indeed
y' O
My wife and lindedd
liked the nice play
yr, liked the nice play
K successors per state; Sampled Lidd from *base
Score
log
"' yr, xx) "arlyy y x)
Figure 2: Illustration of Chunk-level Beam Search with W, K 2,,2.
4.2 Chunk-level Beam Search (CBS)
After analyzing the feasibility of optimizing Eq. 5a with greedy search algorithms (e.g., beam search), we introduce practical beam search variant that optimizes the dense reward objective (Eq. 5a) while ensuring the Kl.-constraint from Tusse (Eq. 5b).
The core algorithm providing the foundation of our method, Chunk-level Beam Search (CBS), detailed Algorithm and illustrated in Figure 2. The key insight is that our beam search operates at the level of chunk. The search starts at the prompt and always maintains a hypothesis set H {xx, y' ) of W states. For each state (x, y') in H. CBS samples K continuation chunks yz of length L from Thase- This results in WK successor states. Among these successors, only the top- W successors with the highest partial return log π (y' yy x) log ( oy x) are stored in H. and expanded further. Finally, the terminal state (x,y) with the highest intermediate return
log x (y x) log (y x) is selected, from which the complete response y extracted.
Algorithm Chunk-level Beam Search (CBS)
1: Input: prompt x, beam width W, successors per state K, chunk length L, model to steer Touse. tuned model n", and untuned model Med-
2:
3: Output: optimal terminal state (x
4: Initialize H = {(x, y' = = W
5: while (x),y') E H such that y' is incomplete do
6: Initialize C = (}
7: for each (x,y') E 7 do
8:
y (yr)): iid. [: (. \x,y' C- (x, yy) yr y)
y = Ø if y' is complete
9:
10: end for
11: H + Top- W (x.y (xx. (y' yr. x) log Terly yr. |x))
12:
end while
13:
return arg (ax (log x) log Trerly xx)
CBS is unified framework that encompasses several search-based algorithms: (1) CBS with W 1. K N, oc (i.e., infinite chunk length) is equivalent to BoN sampling with log n" (y x) Ty x) as the scoring function, and (2) CBS with K = oc, L 1 (i.e., exploring possible next tokens from the vocabulary) is equivalent to vanilla token-level beam search. However, we always ensure finite chunk length and limited successor exploration via sampling to achieve the best of both worlds: (1) Using finite chunk length allows CBS to prune bad states during generation, enhancing stecrability more efficiently compared to BoN, (2) Sampling from
5

---

Thase with limited successor exploration implicitly enforces the KL-constraint from Thase (Eq. 5b); otherwise, integrating the KL-constraint into the objective (Eq. 5a) would be necessary for token-level search, but this can be challenging, especially when vocabularies of models differ or with black-box language base models Thase whose bog-robbabilities are inaccessible.
Computation costs. In practice, CBS samples W K continuation chunks in parallel from the frozen base model Thase and prune states by calling tuned and untuned model pair (T" Teet every L tokens. Larger W K and smaller L enhance steerability at the cost of increased computations. Note that high steerability, while beneficial, is not always ideal as it may lead to large KL deviation and over-optimization [16].
4.3 Application: Model Up-Scaling and Weak-to-Strong Generalization
The most practical use of CBS occurs when the tuned and untuned models, (T", Tee)) are smaller than the model to steer, Tase- (1) First, this instance serves as model up-scaling strategy, directly tuning a small model Tee - π.' by which the large model decoding can then be guided, to achieve similar outcomes as directly tuning the large model.
(2) Second, since the small models (r"", Ter) are usually weaker than the large model to steer Thase- this instance also exemplifies weak-to-strong generalization [21], enhancing the strong model with only weak test-time guidance. We refer this instance of CBS as weak-to-strong search, which is the main focus of our study.
5 Experiments
In this section, we empirically evaluate weak-to- strong search's 's ability to align large language models using only test-time guidance from small language models. First, controlled-sentiment generation [22] and summarization [2], we tune gpt2 to model the desired behaviors in each task and then use tuned and untuned gpt2 to steer larger models of various scales (Section 5.1). Next, in more difficult enstructionfoolooingg benchmark, AlpacaEval 2.0 [25]. instead of tunning small models, we reuse off-the-shelf open-source 7B models and their untuned versions to steer a series of large models, including open-source 70B models and black-box model (Section 5.2).
g
Baselines. In addition to weak-to-strone search, we evaluate several existing test- simetime approaches that steer a large language model Thase using small tuned and untuned language models (x* * mer)):
(1) Base: we explore regular decoding from the frozen large language model with n-shot prompting (see Appendix B. for prompt details).
(2) Best-of-N Sampling (BoN) [16, 17]: BoN uses log T" (y) x) log Tyer(y] x) select the highest-scoring responses among the N independent responses from the frozen large language model. Since weak-to-strong search (CBS) samples K response chunks in parallel, for fair computational comparisons, we always ensure N W K. (3) Emulated Fine- Tane-Tuning (EFT) [34, 12, 11, 35]: EFT approximates the results of directly fine- tuning the large language model by sampling from log TEFT (y; x. y<t) log Touse (y, y<y)) 3 (log π (y,, y<t)- logg Ryr x. yx)))), where 3 is the hyperparameter from Eq. 2. Note that EFT is only applicable when all models share the same vocabulary (which is necessary for composing output distributions from different models). Whenever possible, we also compare test-time methods against directly fine-tuning the large models in the same way small models are tuned.
5.1
Generation & Summarization
Setup. For these two tasks, we follow the synthetic setups from [16, 46, 4], assuming access to gold reward model Tgold- For controlled-sentimee generation, Fgold encourages positive continuations of movie reviews, while for summarization, it encourages high-quality summaries of Reddit posts (details in Appendix 1.4). We generate synthetic preference datasets D = {(x, yw. y from Tgold with p(y1 > Y2 x) o (T (x, y1) Tgotal xx, Yz)) to mimic human feedback [47].
To obtain the small language models, we optimize gpt2 (124M parameters) using the standard DPO pipeline [4]: (1) we first obtain the reference model nea through supervised fine-tuning on both chosen and rejected responses from the synthetic preference dataset, then (2) we apply DPO on the synthetic preference dataset with nee as the reference policy to obtain the optimal language model -² Note that the first stage primarily informs the language model of the desired response format, with most of the tuning occurring in the second DPO stage.

---

Controlled-Sentiment Generation
Summarization
Gold RM (1aoia)
3
2
GPT2-large
GPT2-X1
L1am2278B
Llama3-8B
GPr2-large
GPT2-X1
Llana2-7B
Llana3
GPT2-DPO (m*)
Base (Thee)
BoN (16)
Directly tuneri
GPT2-9FT (Trat)
Weak-to-strong search 4,
EFT (8*)
The gold reward achieved for different large pre-trained models under the gpt2 guidance. We show the mean reward (#: standard deviations) across three random seeds. EFT (9*) denotes the best EFT results among 3 E {1/4, 1/2, 1, 2, 4]: Weak-to-strong search (4, 4, 5) denotes CBS with W, K, = 4, 4, 5; BoN (16) denotes BoN with N 16.
Figure 3:
Given the tuned and untuned (un-DPO-tuned) gpt2 pair (I" keer), we use them to steer the large pre-trained language models without additional training. The large pre-trained language models we study fall into two categories based on whether they share the same vocabulary as the small models: (1) same vocabulary: gpt2-large (774M), gpt2-xl (1.5B) and (2) cross vocabulary: Llama-2- 2-bb, Llama-3-Ba Eventually, since we have access to the gold reward model, language model responses can be fairly evaluated on the test split of prompts using this gold reward model.
Results. Figure 3 demonstrates weak-to-soe -o--tttt-tongg search's great flexibility and steerability in both tasks. For summarization, weak-to-strone search consistently outperforms other test- cest-time methods by large margins. For cmnrrlled-sentineee generation, weak-to-strong search second only to EFT with carefully selected hyperparameter (8" = 1/4) when EFT is applicable. We hypothesize that token-level adjustments from EFT are sufficient for controlled-sentinet primarily requires minor stylistic changes at the token level (e.g., "hate" - "love"). However, in the more complex task of summarization, where broader subsequence-level manipulations are essential, weak-to-strong search excels. Please refer to Appendix D for quantitative comparisons of samples from different methods. We need to mention that we do not meaningfully tune stak-tt-strong search (CBS)'s hyperparameters to obtain the results in Figure (we use fixed set of hyperparameters of (4, 4, 5) for W. K, across all models), which may underestimate the performance of our method. In addition, method enables consistent weak-to-strong generalization in the harder task of summarization: most large pre-trained models (except for gpt2-large) are stronger than the tuned gpt2 in summarizing long text, but the weak models are still able improve the strong models through test-time guidance, nearly matching the results of direct fine-tuning. The phenomenon of weak-to-strong generalization will be further studied in Section 5.2.
generation, which
Chunk-level Beam Search ablations. We perform additional ablations understand how CBS hyperparameters (beam width W, successors per state K, and chunk length T.) influence performance. Figure displays the ablation results for W and K. With the same computation budget (i.c., W K), the optimal trade-off between W and K varies by tasks: for controlled-senting best results come from retaining the most promising state and concentrating computational efforts on expanding from it (W, K = 1, 16); in contrast, for summarization, maintaining multiple hypotheses (W, K 8, 2) yields the best results probably because it helps avoid local optima. Figure displays the ablation results for L where smaller L benefits pnatrolee--enntmmen generation, while an intermediate is optimal for summarization. These results are consistent with our findings from Figure 3, suggesting that the simple nature of cetrrolled-senttmeen generation makes token-level manipulation sufficient and cumulative reward gederatneattion a more reliable indicator of overall return. See Appendix C.I for extended ablations more models.
generation, the
7

---

8pt2-x1
Li 1ama-2-7b
8pt2-x1
Llana-2-7b
an0egeors per state (K)
a.1
16-
4.5
S000 S00c0e00 per state (IC
41.20
16
1:30
1:41
0.01
1.17
1:40
1.50
9.80
6:91
4.2
5.40
0:15
6:9
0.13
1.00
2.17
1.63
3.75
6:10
1.35
1.75
1:91
1.50
1:60
beun width (W)
bear width (W)
beua widhh (W)
beam width (W)
(a) controlled-eentmmentt sentiment generation
(b) summarization
Figure 4: W, K ablations for CBS (L = 5), We show the mean rewards across three random seeds.
With the same computation budget (i.e., same W K), the optimal perraarameee differ by tasks.
gpt2-x1 Llana 2-76
Gold RM
Gold R&
3.8
4.0
2.5
08
gpt2-x1
--7b
clunk leugth (L)
chunk length (L)
(a) controlled-sentiman generation
(b) summarization
Figure 5: L ablations for CBS (W, K = 4, 4). We show the mean rewards (#: standard deviations) across three random seeds.
5.2 Instruction Following
Setup. Next, we evaluate weak-to-strong search on a standard single-turn instruction-follooing following benchmark, AlpacaEval 2.0 [25], which consists of 805 prompts from various open-source datasets. Unlike the previous section where we steer large pre-trained language models (e.g.. A.lama-2-bb) we now steer large instruction-tunee language models (e.g., Llama-2-77 1ama-2-7b-chas chat). This is because models often require further alignment to match human preferences [48], and (2) to study weak-to-strong generalization in cestuutionnfollowingg the models must be proficient at following instructions before steering.
(1) instruction-turi
For small language models, we reuse two high-ranking 7B model pairs from the AlpacaEval 2.0 leaderboard as guidance: (1) Zephyr guidance: zephyr-7b-
and its untuned version tulu-2-dpo-77 and its untuned version tulu-2-1 2-bb. All four models use the 1lama-2 tokenizer. The large instruction -uned language models we aim fur- ther align fall into three categories: (1) same vocabulary: Llama-2 -aaaa----bb-aae chat, 11-ma-2-70b- 2-70t-chat: (2) cross vocabulary: LLama-3-8B-Ina Instructt Llama 3- Instruct; and (3) black box: gpt-3.5 3.tt-3.5-urre ceuurb-innsrrce these small models (Tee > π we do not test the baseline results of directly fine-tuning the large models as in Figure 3. Language model responses are evaluated by their length-controlled win rates (LC WR) against spt-4-turil with gpt-4-tuu turbo serving as the judge.
mistral-7b-sft -bet-----sttttet (2) Tulu guidance:
As it is nearly impossible to reproduce the exact training pipeline for
Experimental results with Zephyr and Tulu guidance are shown in Figure 6 (detailed hyperparameters in Appendix B.2.2). Weak-to-strone search consistently outperforms other test-time time baselines with great margins. There are two crucial takeaways worth mentioning: (1) Weak-to-strong search makes strong models stronger with only weak test-time guidance. Take Zephyr guidance for an example (Figure 6, left), even if most large instruction- models Thase are stronger than zephyr- 7b- beta before steering, strrongg search is still able to enhance their performances
Results.

---

Instruction Following (Zephyr)
Instruction Following (Tulu)
10
AlpocaEval 2.0 LCC Win Rate (%)
zepbyr-12 To-betalllleoeeeo (x*)
talo-2-dpo-7b (st)
33
nistral -bbaatt--baa (Rhe))
tulo-2-7h (Hoe)
30
30-
25-
25-
13
15
10
10
Llans2-70
Llara2-708
Llana3-80 80
Llara3 708
6PT3.
L1ama2-78
Llama2-700
Llana3-80 88
1lans3-700
GPT3.5
Base (ATbas)
Wenk-toosstiing search
BoN
EFT (187)
Figure 6: The length-controlled win rates against gpt-4-turbo for various instruction-tuned models under Zephyr (left) or Tulu (right) guidance. Hyperparameters are in Appendix B.2.2.
using weak models as guidance. Conversely, EFT and BoN mainly interpolate between weak and strong models, resulting in limited, if any, improvements over the strong models. We also tested beam search over the strong models without external guidance [20] but we found obvious improvements compared with regular decoding (Table 2), probably because the latent reward functions behind these language models are not well aligned with the human preference that gpt-4-turbo approximates. The same observations apply to Tulu guidance, even though the tuned tulu-2-dpo all the large instruction-tuner language models by significant margins (Figure 6, right). (2) Weak-to- strong search applies to black- language models. Our method, requiring only sampling from large language models, is also effective for black-box models like 5015-3.5-turb--111 weak-to-strong search with gpt-3. 100, as the black-box language model APIs are stateless and do retain activation caches, making repeated context embedding costly. Despite the long chunk length, our method still effectively improves the alignment of black-box models, significantly outperforming BoN, special case of weak-to-strong search (CBS) with infinite chunk length.
weaker than
For we use relatively long chunk length of
6 Discussion
We have presented weak-to-strong search, an alignment method that keeps the large language model frozen while steering its decoding through a test-time greedy search over small language models. This method builds on the insight that the log-prot peg-oobbaillty difference between small tuned and untuned language models can serve both as dense reward function and value function, and then introduces a novel beam search algorithm designed for balancing reward maximization and KL minimization. This method offers compute--fficceen model up-scaling strategy that eliminates the complexity of directly fine-tuning the large models, and exemplifies weak-to-strong generalization [21] that makes strong models stronger with only weak test-time guidance. Empirically, this approach is effective in controlled-sentinen generation, summarization, and instruction following.
Limitations & Future Work.
While our work focuses on aligning with human preferences, weak- to-strong search could also apply to tasks like reasoning [49, 50) and coding [51], where ground truth answers exist. This because any pair of tuned and untuned language models can act as test-time steering forces, without necessarily being trained on preferences. This then raises several questions beyond the scope of our current study: (1) In our study, we consistently use SFTed policy as the untuned model Tee due to the two-stage nature of preference learning; however, in single-stage fine-tuning tasks, does weak to-strong search still work with a pre-trained model serving as the untuned model ? (2) Although our method shows consistent weak- teeak-to-stroe io-strong generalization across diverse alignment tasks, it is also critical to probe potential failure modes [52]. Can weak-to-strong search enhance language models in tasks where ground truth answers exist, beyond merely tailoring their knowledge and skills human preferences? (3) Additionally, while our work mainly focuses on how dense language model reward function (Eq. 4) benefits language model decoding test time, it's

---

also worth exploring the potential benefits of this reward parametrization for RL tuning, Although Appendix C.3 presents some promising preliminary results, leave further analysis for future work.
Acknowledgements
This work supported in part by the National Key R&D Program of China (N0.2022ZD0100201 We would like to thank anonymous reviewers for their valuable feedback and helpful discussions.
Author Contributions
Zhanhui Zhou led the project, proposed the research idea, wrote the codebase, designed the experi- ments, conducted most of the initial experiments, and wrote the paper. Zhixuan Liu assisted with running many of the ablation studies throughout the experiments presented in the paper. All other authors provided feedback throughout the project.
10

---

References
[1] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford. Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: 2019.
[2] Nisan Stienmon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
[3] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[4] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
[5] Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. International Conference on Artificial Intelligence and Statistics, pages 4447-4455. PMLR, 2024.
[6] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05880 225600 2022.
[7] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint 0tt::3307.09288 2023.
[8] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b, arXiv preprint arXiv:2310.06825, 2023.
[9] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment. arXiv preprint arXiv:2310l 16944, 2023.
[10] Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richárd Nagyfi, al. Openassistant conversations-democrriiiii large language model alignment. Advances in Neural Information Processing Systems, 36, 2024.
[11] Eric Mitchell, Rafael Rafailov, Archit Sharma, Chelsea Finn, and Christopher D Manning. An emulator for fine-tuning large language models using small language models. arXiv preprint MiXiv:231011296: 2023.
[12] Alisa Liu, Xiaochuang Han, Yizhong Wang, Yulia Tsvetkov, Yejin Choi, and Noah A Smith. Tuning language models by proxy. arXiv preprint arXiv:2401.085653 2024.
[13] Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, et al. Controlled decoding from language models. arXiv preprint 200 7022,, 2023.
[14) Minbeom Kim, Hwanhee Lee, Kang Min Yoo, Joonsuk Park, Hwaran Lee, and Kyomin Jung. Critic-guided decoding for controlled text generation. Findings of the Association for Computational Linguistics: ACL 2023, pages 4598 4612, 2023.
[15] James Y Huang, Sailik Sengupta, Daniele Bonadiman, Yi-an Lai, Arshit Gupta, Nikolaos Pappas, Saab Mansour, Katrin Kirchoff, and Dan Roth. Deal: Decoding-time alignment for large language models. arXiv preprint 2rXiv:2402.061477 2024.

---

[16] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 00835-10866, PMLR, 2023.
[17] Ahmad Beirami, Alekh Agarwal, Jonathan Berant, Alexander D' Amour, Jacob Eisenstein, Chirag Nagpal, and Ananda Theertha Suresh. Theoretical guarantees on the best-of-n alignment policy. arXiv preprint urXiv:2400) 21//22411177777 .01879, 2024.
[18] Jiacheng Liu, Andrew Cohen, Ramakanth Pasumuru, Yejin Choi, Hannaneh Hajishirzi, and Asli Celikyilmaz. Making ppo even better: Value-guided monte-carlo tree search decoding, arXiv preprint arXiv:2309. 15028, 2023.
[19] Xidong Feng, Ziyu Wan, Muning Wen, Ying Wen, Weinan Zhang. and Jun Wang. Alphazero- like tree-search can guide large language model decoding and training. arXiv preprint 2rX. 2023.
[20] Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn, From 9": Your language model is secretly a q-function. arXiv preprint arXiv:2404. 12358, 2024.
[21] Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker. Leo Gao, Leopold Aschen- brenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong gener- alization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09390 2023.
[22] Andrew Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguisties: Human Language Technologies, pages 142-150, Portland, Oregon, USA, June 2. Association for Computational Linguistics.
[23] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners.
[24] AetMeta. Llama 3 model card. 2024.
[25] Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way debias automatic evaluators. arXiv preprint arXiv:2404.04475. 2024.
[26] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-
113, 2023.
[27] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
[28] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Amn Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 139771, 2023.
[29] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.080773 2022.
[30] Zhanhui Zhou, Jie Liu, Chao Yang, Jing Shao, Yu Liu, Xiangyu Yue, Wanli Ouyang, and Yu Qiao. Beyond ame-preference-for-a. Multi-objective direct preference optimization. arXiv preprint arXiv:2310.03708. 2023.
[31] Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. Advances Neural Information Processing Systems, 36, 2024.
12

---

[32] Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer. Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Per- sonalized large language model alignment via post-hoc parameter merging. arXiv preprint arXiv:2310. 1a..11..1556644 1564, 2023.
[33] Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang, Arithmetic control of Iims for diverse user preferences: Directional preference alignment with multi-objectivv rewards. arXiv preprint arXiv:2402.18577, 2024.
[34] Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A Smith, and Yejin Choi. Dexperts: Decoding-ting time controlled text generation with experts and anti-experts. arXiv preprint arXiv:2105.030033 2021.
[35] Zhanhui Zhou, Jie Liu, Zhichen Dong, Jiaheng Liu, Chao Yang, Wanli Ouyang, and Yu Qiao. Emulated disalignment: Safety alignment for large language models may backfire! arXiv preprint arXiv:2402. 123.43333, 2024.
[36] Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic- hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 10425, 2023.
[37] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024.
[38] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weightte regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:19/000011777 2019.
[39] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09955) 2020.
[40] Brian Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pages 1433-1438. Chicago. IL, USA, 2008.
[41] Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv: 00909, 2018.
[42] Nathan Lambert, Valentina Pyatkin. Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint 2rXiv:2403.13787, 2024.
[43] Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 394344) 1952.
[44] David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search. nature, 529(7587)4844488, 2016.
[45] Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34:1273- 1286, 2021.
[46] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050. 2023.
[47] Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 39(3/4)332 -45, 1952.
[48] Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuchai Pan, Juntao Dai, and Yaodong Yang. Aligner: Achieving efficient alignment through weak-to-strong correction. arXiv preprint 2XXi:2400.00211 2024.
[49] Erie Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning, Advances in Neural Information Processing Systems, 35:15476-15488, 2022.
13

---

[50] Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani. Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2024.
[51] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308. 12950, 2023.
[52] Wenkai Yang. Shiqi Shen, Guangyao Shen, Zhi Gong, and Yankai Lin. Super (ficial)-alinnment: Strong models may deceive weak models in weak-to-strong generalization. arXiv preprini arXiv:2406. / 1431, 2024.
[53] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Dekang Lin, Yuji Matsumoto, and Rada Mihalcea, editors, Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Lunguage Technologies, pages 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics.
[54] Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. Camels changing climate:
Enhancing Im adaptation with tulu 2. arXiv preprint arXiv:2311.10702 2023.
[55] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377, 2023.
[56] Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling-7b: Improving Ilm helpfulness & harmlessness with rlaif, November 2023.
[57] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: /00. 1707.00777.0347, 2017.
14

---

A Mathematical Derivations for Eq.
As introduced in Section 3, the alignment objective under the traditional contextual bandit framing is given by (Eq. 2):
arg max Ex Xx-p(xxx [r(x, y) - 3D (
x)
Frorly x))]
(7)
where p(x) is a distribution of prompts, y is the complete language model response, is a reward function that encourages purfe-preferred responses, and Dxi. limits how far the optimized language model π can deviate from the reference model Tet..
Given the autoregressive nature of language models, we can frame language model alignment as solving a token-level MDP [20]. This token-level MDP is define by the tuple (S, f. r(s,, a,)). Here, the state S; == (x,) E S consists of the prompt and all response tokens generated so far; the action a = Yt determines the next token to generate from the vocabulary A: the dynamics f is deterministic function that updates the state by concatenating the current state and action S:41 := (st, ar); r(:; at) a sparse reward that equals r(x, y) a; is EOS and otherwise. We use Pa (Se) to denote the state marginals of the trajectory distribution induced by the policy Under the token-level MDP, the objective from Eq. 7 can written as
max
(x))),
r(s,, + Blog (er 8x) (H ss))
(8)
99KL [m(m)/))) (ear(m))ss))
where T specifies the response length. The solution of Eq. 8 is given by [40] as:
T' (a: 5t) exp ( (Q" as) V (())))
(9)
where the optimal Q-function and V-function satisfies
Q" (st, az) = r(t,, ax) + B log The st)) + V (s:+1));
(10)
(s:) = Σ Bei -p (8: ( [r(s;, a:) + Boog (er |s:) + 3H((T"(a ss)))] (11)
1=t
Here, V' predicts the expected future return (the terminal reward in this sparse reward setting) penalized with the future KL constraint starting from the state st, under the optimal language model n.. Combining Eq. 9 and Eq. 10, we have
3 log π (a. 8t) (a s.)
rst,, ar) + (S++ - (st)..
(12)
Note that (1) rSt, a;) is a sparse reward that is non-zero if a, is EOS, and (2) (S++1) = 0 if a, is EOS. Then, summing Eq. 12 from timestep 1 to HI yield
Σ log n² a( 8z) Tee (a st) (vv*(s))) + r( an))
V + (*(H
if ay EOS if aa EOS,
(13)
where (* = V'(( a due to the deterministic transition. Now, returning back to the sequence-level MDP (Section 4), where we define y as a complete response, y' as response that can either complete or incomplete, we have that $1 (x, (x)) (x, Ø) = x and (SH aæ) = ((x, y<H). (x. ++)+1). Thus, can rewrite Eq. 13, with slight abuse of notations, as
V + (x, T x) (x) (xxx)rr(xx)))
x)
if Mfy' 전 EOS (y' is incomplete)
log
(14)
if I EOS (y' is complete).
15

---

B Further Details on the Experimental Setup
B.1
Controlled-Sentimeer Generation & Summarization
B.1.1
Model Specification
The following table lists the models and their corresponding links.
Models
Links
gpt2 (124M) [23]
https::/hugginngfccc.c./ httpp//a.......ooooooooooooooaaa
gpt2-large (774M) [23]
https://hugginggacee cotppppnaalccommutttt/og/ooii
gpt2-x1 (1.5B) [23]
http://ggggggaaccccoooppaaaaaaaa co
Llama-2-1 2-7b [7]
Llama-3-88 [24]
B.1.2 Hyperparameters Specification
We use fixed hyperparameters across all tested models. We use temperature T = 0.7, top-k = 50 and top-p = 1.0 when sampling from the language models. For weak-to-stronn search (CBS), use W. K, L = 4, 4, (W: beam width, K: successors per state, L: chunk length). For BoN, we use N 16 for fair computational comparison with weak-to-strong search (i.e., W K N). For EFT,
we report the best results among 3 E
B.1.3 Compute Resources Specification
Models are evaluated over 1000 test prompts, one single NVIDIA A100 GPU.
B.1.4
Gold Reward Model Details
We follow the synthetic setup in which we use the gold reward models to play the roles of humans and provide binary preference labels [16, 46, 4J.
For controlled-sentimenn the gold reward model "gold- Distilbert-imdu is a fine-tuned classifier on the imdb dataset [53] classify movie review sentiments. We define the gold reward rgold log p(positive xr, y) log p(negative xr y) to encourage positive review. Synthetic preferences are collected using the truncated movie reviews as prompts x, and pairwise completions from gpt2-imdb, ranked with p(y1 > x) goog y1) (go (x. y2)), as preferences.
generation, we reuse the publicly available distilberr imdb define
For summarization, we fit reward model the summarize. crro__feebbacck dataset [2] as the gold reward model Tgold- Specifically, this reward model is fine-tuned from -lama-2-7b with a linear projection head and binary cross entropy loss, using batch size of 32, a learning rate of 1e-5 for the projection head, and 5e-6 for other parameters, over one epoch with a cosine learning rate schedule. Synthetic preferences are generated by relabeling pairwise responses in the original dataset with p(y1 > x) oF (x, y1) (x. y2)).
Both gold reward models show high validation accuracies, 0.928 and 0.736, demonstrating strong correlation with human judgments.
B.1.5 Direct Tuning Details
Direct tuning on the synthetic preferences D [xx, un v.N involves two stages: Supervised Fine- Toni-Tunng (SFT) and Direct Preference Optimization (DPO) [4]. During SFT, models are trained on both selected and rejected responses using batch size of 64, learning rate of 2e-5, and cosine learning rate schedule over one epoch. During DPO, we use a 3 = 0.1. batch size of 256, a learning rate of 1e-6, and a cosine learning rate schedule over epoch.
B.1.6 Prompt Template for Sampling from Base Models
When sampling from large pre-trained models, crucial to provide clear task-specific instructions. For sentiment-controlle generation, we use zero-shot prompt:
16

---

Here is movie review from imdb: {prompt)
For summarization, two-shot prompt (the exemplars selected arbitrarily):
(examplar [1] -ppppptt/o;;;pppppp {examplar [1] response)
(examplar [2] DR; {examplar [2] response) {prompt\TL;DR:
B.2
Instruction Following
B.2.1
Model Specification
The following table lists the models and their corresponding links.
Models
Links
zephyr-7b-beta
mistral-76-sft-beta [9]
hatps://huggingface
tulu-2-dpo-7b [54]
co/allenaai/uuu--22dp0077
tulu-2-7b [54]
http::///uugiingffccccc
1lama-2-7b-chat [7]
chat-hf
1lama-2-70b-chat [7]
-lama-3-8B-Instruee [24]
actpe://huggingtacs
Instruct
1lam-3--0B-Instruue [24)
https: https://hugiingface.
http://awa....eeeeeae.aaaaaaaaaa Instruct
apt-3.S-turbo-instuue [3]
B.2.2 Hyperparameters Specification
When sampling from 1lamm-3-88 --B--IIssrutt and (.lam-33-70B Instructt we use the 7 0.6, top-k 1.0 and top-p 0.9 as per the official Llama-3 generation configuration. For other models, we default to temperature T = 0.7, top-k 50 and top-p 1.0. Specific hyperparameters for each method are detailed in Tables 3 and 4.
B.2.3 Compute Resources Specification
Models are evaluated on 805 test prompts. Model inference takes place on one single NVIDIA A100 GPU for 7B&8B and black-box models, and four NVIDIA A100 GPUs for 70B models.
C Extended Experimental Results
C.1 Chunk-level Beam Search Ablations
We show the extended CBS hyperparameters (W, K, L.) ablations Figures and 7.
8pt2-large
28
gpt2-xl
1.5
1lama-2-7b
Liana-3-7b
Gooel RMM
1.8
Gold
8.5
00
apt2-large
gpt2-xl
-05
Llana-2-Vb
-10
Llana-3-Vb
-15
10
20
length (l.
chunkk length (1.)
(a) ceatrolled-sentiment generation
(b) summarization
Figure 7: L ablations for CBS (W, K 4, 4). We show the mean rewards (+= standard deviations).
17

---

spt2-large
gpt2-x1
Llama-22 -lama-2-7b
Llama-3-88
StLeTSSOSS per state (K)
16
1M
16-
1-11
16
100
16
1:30
1,57
6.62
2.12
4.03
1.25
1/60
9.65
1.19
4.45
6.91
1.35
5.70
4.00
4.35
2.81
9.24
1.63
5.76
0304
3.70
3,50
2.11
2.80
3:40
5.70
2014
4.77
2.63
2.00
3.35
1.57
1.80
2.25
2.65
2.01
5.12
1:12
2.49
2,50
8.10
3.15
16
beam width (W)
boam width (W)
beam width (W)
beam width (W)
(a) controlled-sentmment generation
8pt2-x1
Llana: a-ma--7bb
Llama-3 33-BB
SHOORssOrs per state (K)
1:25
1:08
-0.15
0.22
1.02
1.35
.0.55
0.05
6.2
0.35
0.54
01
1:00
1.07
1.81
1.15
2.35
2.00
-0.31
6.15
0.02
0.61
1:31
1:50
1:43
1.00
1:73
7,000
0..02
11:90
1.35
1.45
2
16
2
10
2
2
10
beam width (W)
beam width (W)
beam width (W)
beam width (W)
(b) summarization
Figure 8: W, K ablations for CBS (L = 5). We show the mean rewards across three random seeds. With the same computation budget (i.e., same W K), the optimal hyperparameters differ by tasks.
C.2 Evaluation Results for Instruction Following
addition to gpt-4-turbo evaluations, we assess language model responses using two top-ranking reward models from RewardBench [42]: UltraRM-13b [55] and Starling ing--i-3nBMMBB [56]. Table and show that weak-to-strong search consistently outperform other methods across metries. We also test vanilla beam search without any external guidance [20], which does not consistently improve over direct sampling for instruction following (Table 2).
Models
LC WR (%)
WR (%)
URM (1)
SRM (1)
10.08
10.30
1.183
-5,849
w/ beam search (16)
10.23
10.07
1.141
55.355
16.18
14.98
1.902
-5.641
w/ beam search (4)
16.01
14.54
1.873
5.655
22.92
22.57
2.682
-5.16
w/ beam search (16)
21.93
21.66
2.432
5.655
2llama-3-70B-Inttrue w/ beam search (4)
34.42
32.18
3.833
-4.674
34.91
35.07
3.678
-4.754
Table 2: Vanilla beam search [20], without external guidance, shows limited improvements over
regular decoding. 'w/ beam search (16)' denotes beam search with beam width of 16. LC WR and WR denote length-controld and raw win rates against gpt-4-turbo; turbo; URM and SRM denote scores by UltraRM- 13b [55] and Starling -n4B [56].
RM-34B
18

---

Models
LC WR (%)
WR (%)
URM (t)
SRM (t)
weak supervision
zephyr-7b-beta (T*)
13.20
11.00
1.138
-6.143
mistral-7b-sft- settsstttttttttto (Trer)
7.54
4.77
-1.274
-7.618
same vocabulary
L1 Llama-24 -1lam-2-7b--a 2--bb-hht
Base (Tbase) )
10.08
10.30
1.183
-5.849
EFT (3" 0.25)
9.05
10.08
1.244
-5.819
BoN (16)
13.11
13.23
1.650
-5.738
Weak Weak-to-strong search (4,4, 4, 30)
13.65
14.11
2.234
-5.424
Ll Allam--i 2-70b- chat
Base (Tasse) )
16.18
14.98
1.902
-5641
EFT (3" = 0.25)
15.54
14.45
1.905
-3.81
BoN (4)
17.57
16.91
2.061
55576
Weak Weak-to-strong search (2, 2,
19.10
18.14
2.290
-5.425
cross vocabulary
Llama-3-88 3--BB-Innttucct
Base (Tasse)
22.92
22.57
2.682
5.156
EFT (8*)
NA
NA
NA
NA
BoN (16)
25.35
24.32
3.000
3.070
Weak-to-shee teak-to-strong search (4, 4,3)
27.17
27.43
3.407
-4.862
Llama-3-70B- Instruct
Base (Tbase)
31.42
32.18
3.833
4.674
EFT (B*)
NA
NA
NA
NA
BoN (4)
36.60
36.38
3.869
4.676
Weak-to-streee search (2, 2,3
37.92
38.43
4.019
-4.616
black box
gpt-3.5-turbe hew.b--turbb-instuuct
Base (Abase) EFT (3*)
16.00
10.58
0.771
-6.556
NA
NA
NA
NA
BoN (4)
19.59
12.51
1.017
6.455
Weak-to-stree to-strong search (2, 2, 100)
20.07
12.61
1.212
-6.391
Table 3: Instruction following performance under the Zephyr guidance. EFT (8*) denotes the best EFT results among 3 E (0.25. 0.5, 1, 1.5): Weak-to-strong search (2, 2, 30) denotes CBS with W = 2, K = 2, 30. LC WR and WR denote length-controlle and raw win rates against gpt-4-turbo; URM and SRM denote scores by Ul UltraRM-133 [55] and Starling-RM--3 [56].
19

---

Models
LC WR (%)
WR (%)
URM (t)
SRM (t)
weak supervision
1-lu-2-dpo-7b (T**)
9.46
8.10
0.743
-6.310
tulu-2-7b (TTeet
9.03
5.38
-1.070
-7.362
same vocabulary
2--7b-cha
Base (Tbase) )
10.08
10.30
1.183
-5.849
EFT (3"= 1)
10.07
11.63
1.924
-5.535
BoN (16)
11.60
11.67
1.536
-5.721
Weak Weak-to-strong search (4, 4, 30)
13.16
14.20
2.115
-5,451
Ll A.lam-2 1.lama22-70b 2-700-
Base (Taase) )
16.18
14.98
1.902
5.641
EFT
16.58
16.85
2.370
-5.81
BoN (4)
16.73
15.99
2.145
55515
Weak Weak-to-strong search (2, 2,330
19.04
18.15
2.300
-5.438
cross vocabulary
Llama-3-88 3--B--n-ttruct
Base (Tasse)
22.92
22.57
2.682
5.156
EFT (8*)
NA
NA
NA
NA
BoN (16)
22.42
22.54
3.039
5.020
Weak-to-shn teak-to-strong search (4, 4,3)
25.96
26.73
3.431
-4.859
Llama-3-70B- Instruct
Base (Tbase)
31.42
32.18
3.833
-4674
EFT (B*)
NA
NA
NA
NA
BoN (4)
35.96
36.43
3.876
4.668
Weak-to-stroe search (2, 2,3
39.09
39.81
4.068
-4.583
black box
gpt-3.5-turbe tww.b---urbbooistuuct
Base (Abase) EFT (3*)
16.00
10.58
0.771
-6.556
NA
NA
NA
NA
BoN (4)
18.60
13.15
1.202
6.327
tt--strong search (2, 2, 100)
19.80
13.23
1.285
-6.295
Table 4: Instruction following performance under the Tulu guidance. EFT (8*) denotes the best EFT results among 3 E (0.25, 0.5, 1, 1.5): Weak-to-strong search (2, 2, 30) denotes CBS with W = 2, K = 2, 30. LC WR and WR denote length-controlle and raw win rates against gpt-4-turbo; URM and SRM denote scores by UltraRM-13 [55] and Starling-RM-33 [56].
20

---

C.3 RL Fine-tuning with Language Model Reward
In Section we have demonstrated that by converting weak language models per-token dense reward function (Eq. 4), we can further improve strong models at test time without additional training. This section presents preliminary experiments showing that this reparametrized dense reward function can also benefit RI fine-tuning. Using the same setup from Section 5, we start with a dense reward function parametrized by tuned and unturned gpt2 pair (a" Teer and then we tune two larger models under this dense reward on sentiment-controlled generation using PPO [57]. Figure shows the fine-tuning results with different reward sparsity, where we find that dense reward function does benefit RL fine-tuning, and weak-to-strong generalization [21] also occurs consistently during training when the reward signal comes from weaker language models (note that π only achieve a 4.6 gold reward on sentiment-controlle generation from Figure 3).
imdb/gpt2-
Baalb/ggt--argg/reeunn large/return [train]
imdbbggtt2large/goo.. rewarrd [val]
L-1
L-5
L-10
L-20
L-60
L-inf
[-1
L-5 L-10
L-20
L-40
L-ind
6
2
2
epoch
epoch
10
15
20
25
10
15
20
25
(a) PPO fine-tuning results for gpt2-1arge
imdb/Llama-2-7777 777/reuum [train]
hmdb/Llama-2-7b/goll reward [val]
L=1
L-5
L-20
L-10
L:40
Liind
L=1
L-5
L-20
L-10
L:40
Loinl
3
2
2
0
epoch
epoch
0
(b) PPO fine-tuning results for Llama 2-bb w/ LoRA (r 128, a 128).
Figure 9: PPO fine-tuning with chunk-level dense rewards for controlled-sentimen generation. Using dense reward parameterized by tuned and untuned gpt2 models (see Section for details), we train larger base models with PPO. The chunk length L controls the reward sparsity. For example, means rewards are accumulated and emitted every tokens (delayed the last token of each chunk) while L = inf corresponds to vanilla PPO with sequence-level sparse rewards. Denser rewards facilitate credit assignment and accelerate training, improving both the achieved return on training prompts (cumulative dense rewards over the complete responses) (left) and the achieved gold reward on validation prompts (right). We plot mean std over three random seeds.
21

---

D
Sample Generations
D.1
Controlled-Sentiment Generation Sample Generations
Prompt
While I don't consider myself
weak supervision
GPT2-DPO
fan of the fealhorror genre, I enjoyed this movie a lot. If you have seen horror movies, I would recommend it for those interested in the genre. However, I do recommend that you take a look
[truncated for brevity]
4.54
GPT2-SFT
movie fan, did search for the trailer for this movie and couldn't find it. I was wondering if anyone had seen might have to watch the movie, but I'm not if want to. I'm [truncated for brevity] 33226
1et2-large
Base
classic horror fan, reading this book has made me feel rather old. I'm not sure if I'm a fan of Lovecraft's work or not, but this book has made me want read all his stories. would recommend this book [truncated for brevity] 1.96
EFT (3* = 1/4)
great comedy fan, I do enjoy good comedy. have many films over the years and have found that a lot of the best comedies have been comedies that enjoyed as child. think that this is one of those [truncated for brevity] 4.85
BoN (16)
a classic horror fan, reading this book has made me feel rather old. I'm not sure if I'm a fan of Lovecraft's work
but this book has made want to read all his stories. would recommend this book to [truncated for brevity] 5.03
Weak-to-strong search (4, 4,5)
a huge fan of the series, do like to see it on a regular basis. It is great example of a show that has its own unique style and style of storytelling that works. a show that very much work in [truncated for brevity] 5.19
Direct Tuning huge fan of the genre, still find this film very enjoyable movie.
5.40
gpt2-x1
Base
huge Mayweather fan, I do like the movie. I like Mayweather, like movies with Mayweather, and like boxers. It's definitely worth checking out. This movie not that good. It's not that great. It [truncated for brevity] 0.49
EFT (f* 1/4)
great comedy fan, do enjoy good comedy every now and then. I loved watching this movie because it was funny. The humor, the characters, the story, all of it was very funny. It is a very
funny film and it [truncated for brevity] 5.29
BoN (16)
fan of the original Star Trek, I do often enjoy the movies. have seen them all. have seen all the movies, and while have enjoyed them, have not been able to find single one that I would recommend to [truncated for brevity] 3.75
Weak-to-strong search (4, 4, 5)
22

---

huge fan of the Godzilla series, I do enjoy watching all of the films in the franchise, and I enjoyed watching this one with a good friend. The film has very good story, with some great action scenes. The movie was very well [truncated for brevity] 5.16
Direct Tuning
to be "old" at this point in my life, I still love this movie. It's very entertaining film. 5.52
Llama-2-1 amm---7b
Base
to be fan of the James Bond series, do like the movie series. The most recent is film with a highly complex plot, but is not the best of the series, and it's not a good idea to watch [truncated for brevity] 2.04
EFT (3*))
NA
BoN (16)
fan of the franchise, have seen most of the movies this series, I could appreciate this one to a certain extent. However, I'm not sure it is good movie. The characters are not as well [truncated for brevity] 1.65
Weak-to-strong search (4, 4,5)
fan of the original trilogy, I do think this is a fantastic remake. It's essentially the same story, yet it's different enough to be enjoyable. There's also lot of nods to the [truncated for brevity] 4.89
Direct Tuning
huge fan of fomantic comedies, Y do consider this film be delightful movie. While can how people might not like this film, I can also see why people would like it. I think it's [truncated for brevity]
4.73
Llama-3-77
Base
an intellectual, do consider myself a fan of cerebral movies, and this is easily the most intelligent movie I' I've ever seen. I'm not sure if I'd call this film uplifting, but is without doubt inspiring and thought-provolee [truncated for brevity] 5.16
EFT (B*)
NA
BoN (16)
huge fan of either comic book or of super heroes, I have been very impressed by the overall quality of the Marvel's cinematic universe. This movie a great addition to the series. The story is complex and multi-layered, with the characters [truncated for brevity] 5.33
Weak-to-strong search (4, 4,5) 5)
a horror fan, do love a good story which is why thoroughly enjoyed this movie. The story is told well, very well. The acting is great, and the special effects are great too. I thought the story was very well done. [truncated for brevity] 5.41
Direct Tuning
horrorfan, enjoy very wide variety of genres. However, is my belief that few genre films are ever truly horrorör činematicfilms; they are all variations on theme. think [truncated for brevity] 3.11
23

---

D.2 Summarization Sample Generations
Prompt
SUBREDDIT: rrelationships
TITLE: Me [23 FJ with my boyfriend [30 M] of 9 months, hurt and don't 't know what to do. POST: Sorry for the long post. I'm really hurting right now and could use some advice or wise words.
To give a brief background. my boyfriend and have been dating for about 9 months. He's physician in an intense fellowship program and generally very stressedtiredbusy, I' I'm currently in school, and stay pretty busy with an internship and working. We spend almost everyday together, support each other, have lot of fun and both refer to the other as our best friend (that we enjoy touching inappropriately.)
For the first couple months we were very happy together. The issues didn't start until brought up the dreaded question, So what are we?. Then began bit of struggle. I always wanted more from the relationship and was greeted by hesitancy, by months asked for exclusivity and he eventually agreed but only because was going leave if he didn't ' (I wanted something serious.) by 6-7 months said love you, he didn't 't say it back. This is totally fine, painful, but fine. I said I'd be patient with him until he is ready. He still hasn't said it back, but now 9 months he said that he feels like he is falling in love, but isn"t 't in love yet. This progress has given me hope, but not being on the same page has made me feel bit insecure about our relationship.
He has mentioned that he doesn't I want to fall in love unless he knows that we will have a serious future together, he doesn't get hurt like he did in his 2 prior relationships.
We have been fighting lot recently because: still haven't met his family (they live 5 minutes from him), he doesn't 't want to share our relationship on Facebook (not the biggest deal!), he isn't inn'iin in love with me, we don't 't really talk about the future, and the quality of time spent together isn't t the best due to the nature of his work and how stressed and tired he is all of the time. After expressing my concerns about these things, he said he doesn't 't want to lose me and will try to open up to me. The fighting is really getting to us both.
weak supervision
GPT2-DPO
Boyfriend of 9 months insists on exclusivity, refuses to talk about future but I don't 't know how to approach this. What are my options for dealing with the pressure he is feeling now and what should do? -105
GPT2-SFT
I don't know what should do, and I'm feeling insecure and don't know what do. -2.67
gpt2-large
Base
I love him and he doesn't it want to be a relationship unless knows we will be together. He has been fighting recently because I'm in love with him, I'm not in love with him, he wants to be in relationship with someone other than me, I'm not in love with him, I'm not in love with him, I'm not in love with him because I haven't 't met his family, he doesn't 't want to share relationship Facebook, I'm not in love with him, I I'm not in love with him because he doesn't 't want to share our relationship on Facebook, I'm not in [truncated for brevity] 33.56
EFT (B* = 2)
Girlfriend has been having fights with boyfriend over not sharing his relationship on Facebook and they are not in love. Girlfriend has expressed her fears about their relationship and is willing to commit time to them. He is trying to open up to her because he is getting his family to open up to him. Girlfriend is hurting. I want to give him the benefit of the doubt and believe that he does genuinely want to be with me and will be open to us, but I am concerned that if he doesn't 't, then he won't be able to open up me in healthy way. believe this be true because he said [truncated for brevity]
-2.02
24

---

BoN (16)
He doesn't tt want to share our relationship on Facebook, he doesn't want to get me hurt, and the quality of time we spend together isn' 't the best due our work and stress. -1.03
Weak-to-strong search (4, 4,5) 5)
We have been fighting because we have not had a serious relationship that we can feel happy and in love with yet. I want more from the relationship and very insecure about relationship. He is trying to open up to me but don't really do well with that approach. want more from our relationship and am feeling pretty insecure about it. 0.49
Direct Tuning
I feel like have been hurt by my boyfriends lack interest in my feelings for him. don't 't know how to proceed with our relationship and I'm not sure if he is ready for serious relationship.
0.33
gpt2-x1
Base
still haven't 't met his family (they live minutes from him), he doesn't want share our relationship on Facebook (not the biggest deal!). he isn 't in love with me, he doesn't 't really talk about the futurr, and the quality time spent together isn't the best due the nature of his work and how stressed and tired he is all of the time. After expressing my concerns about these things, he said he doesn't 't want lose me and will try to open up to me. The fighting is really getting to us both. 1.77
EFT (3* 1)
Boyfriend has been talking about his love life for the better half of 9 months and has mentioned wanting exclusivity within the next 6 months. We are fighting a lot because I still haven't met his family (they are 5 minutes from him), he doesn't want to list our relationship on Facebook (not the biggest deal), he isn't 't in love with me, we don't 't really talk about the future, and the quality of time spent together isn't the best due the nature of his work and how stressed and tired he is all of the time. 1.61
BoN (16)
Boyfriend has been with me for months, we have been fighting about lot of things, doesn't want to share our relationship on Facebook, he is not in love with me, we don't 't really talk about the future, and the quality of our time together isn't 't the best. He has said he doesn't want to lose
me and will try to open up to me. 2.51
fo-k-- search (4, 4, 5)
Boyfriend is a very talented (and really stressedtired) physician. He has mentioned that he doesn't want to fall in love unless he is 100% sure we can have a long term relationship. He's said he feels like he is falling in love, but isn't 't' in love yet. We are fighting lot because we don't talk about the future and he doesn't 't want to share our relationship on social media. 3.04
Direct Tuning
My boyfriend and have been together for 9 months and I've mentioned I want more from the relationship, he said he's falling in love but isn't tt in love yet, and now he won't address my concerns. 1.85
ama amaamma2777 2-7b
Base
Have been dating for 9 months, boyfriend is a physician in intense fellowship program, both in school. He's hesitant about saying I love you, and I'm getting hurt. 1.98
EFT (B*)
NA
BoN (16)
Boyfriend and I have been dating for months, we are still not sure if we are together or not. I love him but he worried about falling love and getting his heart broken. We fight a lot and am hurting :( 2.03
25

---

Weak-fo-strong search (4, 4,
Been dating 9 months, he's 's stressed, tired, and busy. I'm managing bit of an internship and working. He doesn't It want to fall in love if it's not serious. We fight a lot lately and he's having trouble opening up. Don't know if have future together, but want it, can't imagine my life without him. love him dearly. I'm really hurt and I don't know how to move forward. 2.80
Direct Tuning
My boyfriend and have been together for months and I've mentioned I want more from the relationship, he 9 month relationship with bf, he isn't completely in love with me, hasn" t introduced me to his family, don't talk about the future, and now fighting lot. 2.59
ama 3-
Base
I want serious relationship with my boyfriend. I don't know how to move forward with him. He isn't in love and doesn't know he ever will be. don't 't want to lose him, but I' I'm not sure how to make this work. 2.27
EFT (B*)
NA
BoN (16)
Boyfriend and have been dating 9 months. I want more from him than he is willing to give. We fight lot. He says he is falling in love but isn't there yet. don't 't know what to do about it. 2.67
Weak-to-strome to-ooo search (4, 4, 5)
Me. 23 F, with my boyfriend, 30 M, for 9 months. Boyfriend hasn't 't said love yet, want exclusivity and to meet his family. We have been fighting a lot since brought these issues up. just don't know what to do anymore. 2.88
Direct Tuning
month relationship, boyfriend doesn't 't say he loves me, doesn't want to post us on FB, doesn't want to meet my parents, doesn't want to plan for the future (not that have any expectations for him), and doesn't spend a ton quality time with me due work stresses. am getting insecure and we have been fighting lot. love him and don't 't want to lose him. What can I do? I'm really hoping for someone to give me some insight into the male psyche here and tell me the best things
I can do to help this relationship progress and be as happy as possible 2.61
26

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Sections 4 and Section 5,
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section 6.
27

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: See Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Appendix B. We also submit our code in a zip file along with the instruc- tions reproduce the experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
28

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: See Appendix B. We also submit our code in zip file along with the instruc- tions to reproduce the experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: /ttpsscccppbllc///eeccce htaliicc/guiessee /CodeSubmis http:////gguieeeeeeeuuuuuiiiiiia ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
29

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
See Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We run experiments across multiple random seeds for the two synthetic exper- iments and report the means and standard deviations (Figure 3). In instruction instruction-following experiments, where GPT-4 is used as the evaluator, we can not afford multiple tests (each test costs about 5 dollars). However, verified the GPT-4 evaluations using other comple- mentary metrics (Tables and 4).
Guidelines:
The answer NA
that the paper does include experiments. The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the GPU resources for running language model inferences in Appendix (Compute Resources Specifications) but we do calculate total compute.
Guidelines:
30

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips. httpp//ppw.ppppp.cccccccccccccaa
Answer: [Yes]
Justification: /
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The positive societal impact is that method is able to better align language models with human values, which include helpfulness and harmlessness. We do not expect negative societal impacts direct result of the contributions in paper.
Guidelines:
The answer NA means that there no socictal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
31

---

Answer:
[NA]
Justification:
We do not find our study present obvious risks for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We provide links to all assets used in our study B, from which the licenses can be accessed.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors are encouraged reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
32

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This study does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This study does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "xfhcryNphk", "reviewer": "Reviewer_ct59", "summary": "This paper addresses the alignment of large language models without the need for fine-tuning. It conceptualizes the alignment process as a search problem, leveraging the log-likelihood difference between small tuned and untuned language models as both a reward and a critic. By transforming a sparse preference reward into a per-token dense reward, the method achieves weak-to-strong generalization. The effectiveness of the proposed approach is empirically validated through controlled sentiment generation, summarization, and an instruction-following benchmark.", "questions": "What is the difference between the formulation in section 4.1 and the results in [3]?", "limitations": "The paper has discussed limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written, and the empirical results are promising.\n2. The method is well-motivated and theoretically sound.\n3. The proposed approach provides token-level guidance instead of sparse sequence-level rewards, enhancing precision and control.", "weaknesses": "1. The paper does not include comparisons with existing decoding-based alignment baselines [1,2].\n\n2. The token-based MDP formulation is largely derived from existing work [3]. The main innovation of this paper lies in using the log-likelihood difference between two small models for guidance based on the existing formulation.\n\n\n[1] ARGS: Alignment as Reward-Guided Search, ICLR 2024\n\n[2] Controlled Decoding from Language Models, ICML 2024\n\n[3] From r to Q*: Your Language Model is Secretly a Q-Function.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. To address your questions:\n\n---\n\n> W1: The paper does not include comparisons with existing decoding-based alignment baselines [1,2]\n\nWe would like to clarify that a key advantage of our work is its ability to **reuse off-the-shelf language models for test-time guidance**. Given the numerous open-source language models available online for different target tasks (e.g., mistral->zephyr for chatting, llama->codellama for coding), **our approach can almost always be training-free**. In contrast, ARGS [1] requires a reward model (often proprietary), and Controlled Decoding [2] requires a value function trained from scratch, **making them not truly training-free**. \n\n**However, we agree that comparing our approach with these mentioned decoding-based alignment baselines is essential for comprehensive performance benchmarking. Therefore, we have tested ARGS [1] and Controlled Decoding (CD-fudge variant) [2] on controlled-sentiment generation and summarization, and the results (averaged over three random seeds) are shown below.** Note that vanilla ARGS [1] and Controlled Decoding [2] operate on a per-token level and are not applicable to cross-vocabulary guidance (i.e., guiding Llama-2 and Llama-3). While ARGS and Controlled Decoding do enhance the performance of base models to some extent, they are less effective than our method.\n\n|                       |    imdb    |   imdb    | summarization | summarization |\n|:--------------------- |:----------:|:---------:|:-------------:|:-------------:|\n|                       | gpt2-large |  gpt2-xl  |  gpt2-large   |    gpt2-xl    |\n| Base                  |   2.127    |   1.711   |    -1.703     |    -0.871     |\n| Weak-to-strong search (ours) | **4.837**  | **4.522** |   **0.272**   |   **0.633**   |\n| ARGS [1]                 |   2.283    |   1.930   |    -1.378     |    -0.430     |\n| Controlled Decoding [2]  |   2.914    |   2.567   |    -1.777     |    -0.843     |\n\n---\n\n> W2: The token-based MDP formulation is largely derived from existing work [3]. The main innovation of this paper lies in using the log-likelihood difference between two small models for guidance based on the existing formulation.\n\nThe key contributions of this work are twofold:\n\n1. **Formulation Contribution**: We frame alignment as a KL-constrained search and, more importantly, **decouple** the reference model that parametrizes the implicit reward from the reference model that constrains the search space, **enabling training-free alignment**. While some analyses are inspired by [3] (**which is actually concurrent to our work**), this decoupling framework that allows for practical training-free alignment is novel.\n\n2. **Empirical Contribution**: Beyond demonstrating the effectiveness of training-free alignment, our framework shows consistent weak-to-strong generalization, namely enhancing strong models with weak guidance (see Figure 3, Figure 4, and our response to Reviewer [mz4P](https://openreview.net/forum?id=dOJ6CqWDf1&noteId=aC7BMDbknK)'s W4). **The empirical finding that our method achieves weak-to-strong generalization at test time is nontrivial (as other test-time baselines fail) and addresses a significant research question [4].**\n\n---\n\n> Q1: What is the difference between the formulation in section 4.1 and the results in [3]?\n\nPlease see our response to W2.\n\n---\n\nIf these answers do not fully address your concern, we are more than willing to offer additional clarifications.\n\n[1] Khanov, Maxim, Jirayu Burapacheep, and Yixuan Li. \"ARGS: Alignment as reward-guided search.\" arXiv preprint arXiv:2402.01694 (2024).\\\n[2] Mudgal, Sidharth, et al. \"Controlled decoding from language models.\" arXiv preprint arXiv:2310.17022 (2023).\\\n[3] Rafailov, Rafael, et al. \"From $r$ to $Q^*$: Your Language Model is Secretly a Q-Function.\" arXiv preprint arXiv:2404.12358 (2024).\\\n[4] Burns, Collin, et al. \"Weak-to-strong generalization: Eliciting strong capabilities with weak supervision.\" arXiv preprint arXiv:2312.09390 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aC7BMDbknK", "reviewer": "Reviewer_mz4P", "summary": "The paper proposes a search/decoding method \"weak-to-strong search\" for improving LLM's performance at test time by leveraging the log likelihood from small language models. Specifically, it utilizes the log likelihood differences between tuned and untuned small models to guide the decoding of larger models. This approach avoids the computationally intensive process of fine-tuning large models by using smaller, more manageable models for alignment at inference time, guiding the large model to generate better responses. The paper demonstrates the effectiveness of this method across various tasks, including controlled-sentiment generation, summarization, and instruction-following tasks.", "questions": "See weaknesses above.", "limitations": "Yes, the authors discussed limitations in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper introduces a novel approach to improve large model alignment at inference time, which reduces computational costs compared to fine-tuning methods. The idea of leveraging log likelihood differences between untuned and tuned small language models is quite interesting. \n* The method is tested across multiple tasks and has shown effectiveness. Results demonstrate improvements in multiple tasks. \n* Ablation studies offer further analysis and understanding of the hyperparameters and components of the method.", "weaknesses": "* Although the method avoids the computational costs associated with fine-tuning, the paper does not explain clearly the overhead introduced during inference. How does weak-to-strong search impact memory usage and inference speed?\n* How does the proposed method compare to other inference-time techniques, such as in-context learning (using few-shot examples) and prompting methods like CoT?\n* The method appears to be broadly applicable and not strictly limited to a weak-to-strong approach. For smaller models, such as Llama3-8B as used in the experiments, why would this method be preferred over techniques like LoRA fine-tuning, which can be performed at a low cost and yield better performance? Additionally, if the target model has already been fine-tuned, can the proposed method still enhance its performance? \n* (minor) It would be interesting to explore how weak-to-strong search could enhance a model's reasoning abilities, for example on math datasets like GSM8K.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. To address your questions:\n\n---\n\n> W1: The paper does not explain clearly the overhead introduced during inference. How does weak-to-strong search impact memory usage and inference speed?\n\nThroughout the paper, we select the hyperparameters for CBS (W, K, L) to ensure its inference costs are comparable to BoN sampling. Specifically, CBS samples W\\*K trajectories in parallel from the large base (target) model, while BoN samples N trajectories, so we always ensure W\\*K=N. The only overhead of CBS over BoN is running small models chunk-wise for intermediate guidance/evaluation, while BoN runs them once at the end. These overheads are negligible when the chunk length is large or the small models are much smaller than the base model, which is often the case. **Figure 1 in the supplementary PDF (global response) empirically supports this by comparing the GPU memory usage and inference speed of different test-time methods for instruction-following.** The GPU memory usage and inference speed of CBS with our chosen hyperparameters are close to BoN sampling, while CBS is substantially more effective in steering language models and achieves weak-to-strong generalization (which BoN fails to do). We will include these analyses in our camera-ready version.\n\n---\n\n> W2: How does the proposed method compare to other inference-time techniques, such as in-context learning (using few-shot examples) and prompting methods like CoT?\n\nFor summarization, the base (target) models are already few-shot prompted (see Section B.1.6). For instruction-following, the base (target) models are chat models (see L264-266) using a chat template, which does not naturally support few-shot exemplars. Therefore, we test few-shot prompting on the controlled-sentiment generation task, and the results are shown below. **These results show that our method is more effective than few-shot prompting and, more importantly, complementary to it (they can be combined to achieve the best results). For new empirical results on other test-time baselines, please refer to our response to Reviewer ct59's W1.**\n\n|                                | gpt2-large | gpt2-xl | Llama-2-7b | Llama-3-8B |\n|--------------------------------|:----------:|:-------:|:----------:|:----------:|\n| Base                           |    2.127   |  1.711  |    1.857   |    1.915   |\n| Base + 3 shot                  |    2.670   |  2.507  |    2.847   |    2.934   |\n| Weak-to-strong search  (ours)    |    *4.837*  |  *4.522*  |    *4.055*   |    *4.195*   |\n| Weak-to-strong search  (ours) + 3 shot  |    **5.028**   |  **4.855**  |    **4.448**   |    **4.647**   |\n\n---\n\n> W3.1: For smaller models, such as Llama3-8B as used in the experiments, why would this method be preferred over techniques like LoRA fine-tuning, which can be performed at a low cost and yield better performance?\n\nOur training-free framework offers three key advantages over parameter-efficient fine-tuning (e.g., LoRA):\n\n1. **No need for accessing model weights**: Our method does not need access to the weights of the base (target) models. We only require the ability to sample from potentially black-box models and periodically select promising response states (see our black-box GPT-3.5 experiments in Figure 6).\n\n2. **No need for fine-tuning data**: We sometimes lack the data necessary for fine-tuning. For instance, Meta fine-tunes and releases their models but does not release their fine-tuning datasets. Our method can reuse open-sourced models as guidance to approximate the results of fine-tuning target models on these proprietary datasets.\n\n3. **Performance**: Our method can close the performance gap with full-parameter fine-tuning in controlled sentiment generation, summarization (Figure 3), and mathematical reasoning (see our response to W4).\n\n---\n\n> W3.2: Additionally, if the target model has already been fine-tuned, can the proposed method still enhance its performance?\n\nYes, this is exactly what we do for instruction-following experiments, where we use tuned chat models as base (target) models (see L260-266). Empirical results in Figure 6 (we apologize for the caption confusion; each model ID in Figure 6 stands for its chat version, e.g., Llama3-8B refers to $\\texttt{Llama-3-8B-Instruct}$), with detailed results in Tables 3 and 4, show that our method can enhance an already tuned model with weak guidance, enabling consistent weak-to-strong generalization.\n\n---\n\n> W4: (minor) It would be interesting to explore how weak-to-strong search could enhance a model's reasoning abilities, for example on math datasets like GSM8K.\n\nExploring how our method enhances a model's reasoning abilities is indeed interesting. **We verify that our method demonstrates weak-to-strong generalizaition on mathematical reasoning when small models** $\\pi^*$ **are specifically tuned for reasoning**. We will include these ablation results in the camera-ready version. Specifically, we reuse two small model pairs tuned for mathematical reasoning:\n\n1. Qwen2-7B-Instruct-Step-DPO ($\\pi^*$) (GSM: 88.5), Qwen2-7B-Instruct ($\\pi_{\\text{ref}}$) (GSM: 82.3);\n2. DeepSeekMath-RL-7B ($\\pi^*$) (GSM: 88.2), DeepSeekMath-Instruct-7B ($\\pi_{\\text{ref}}$) (GSM: 82.9).\n\nThese models are publicly available on HuggingFace, with $\\pi^*$ tuned from $\\pi_{\\text{ref}}$. We then use these pairs to steer a larger base (target) model already strong in mathematical reasoning: Qwen2-72B-Instruct (GSM: 91.1). Weak-to-strong search with Qwen2-7B guidance enhances the performance of the 72B untuned version from 91.1 to 94.47, while weak-to-strong search with DeepSeek-7B enhances its performance from 91.1 to 94.24. **Notably, weak-to-strong search closes the performance gap and even outperforms directly fine-tuning the large model: Qwen2-72B-Instruct-Step-DPO (GSM: 94.0).**\n\n---\n\nIf these answers do not fully address your concern, we are more than willing to offer additional clarifications.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fzguH9HxKe", "reviewer": "Reviewer_2D8T", "summary": "The paper introduces the \"weak-to-strong search\" method for aligning a stronger large language models (LLMs) by leveraging two weaker LLMs during test-time without requiring fine-tuning of the large models. This approach aims to improve alignment by maximizing the log-likelihood difference between tuned and untuned small models while using the frozen large model for decoding. The method demonstrates effectiveness in various tasks, including sentiment-controlled generation, summarization, and instruction following.\n\nI generally think this is an inspiring paper towards acceptance. The reason I rate it a weak accept rather than accept is I personally think an ablation study of using the log-likelihood difference to conduct token/chunk-wise PPO is necessary. It can make the paper more complete but lacking it is kinda fine though.", "questions": "N/A no questions here; the paper is presented very clearly.", "limitations": "The paper includes a limitation section at the end.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The weak-to-strong search method is novel and provides a computationally efficient way to align LLMs without fine-tuning.\n2. The method is shown to work across different tasks and with both white-box and black-box models.\n3. The method is well-grounded in theory, with detailed mathematical formulations and explanations.\n\nI really like the overall presentation of this paper. The reading flow is good; better than many other ones in this NeruIPS review cycle. :D", "weaknesses": "1. Obtaining a tuned weaker model is still not a very trivial thing, and the performance of $\\pi_\\text{base}$ on downstream tasks could heavily depend on the tuned weaker model as well. There is not too much discussion regarding it.\n2. As we can formulate a partial reward function using a tuned/untuned model pairs, apart from using it to guide decoding, a natural next step/ablation study is using it to conduct PPO. It is worth trying this experiment and compare the downstream performance as well as the overall cost.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and we're glad you enjoyed the paper. To address your questions:\n\n---\n\n> W1: Obtaining a tuned weaker model is still not a very trivial thing, and the performance of $\\pi_{\\text{base}}$ on downstream tasks could heavily depend on the tuned weaker model as well. There is not too much discussion regarding it.\n\nThere are two practical reasons why we do not obtain a collection of weak models $\\pi^*$ to analyze how their performance influences weak-to-strong search:\n\n1. When tuning the weak model $\\pi^*$ ourselves (e.g., controlled-sentiment generation and summarization experiments): In the context of alignment, reward models are usually trained on relative preferences and it is standard practice to use the reward model with the highest validation accuracy [1,2]. Since the tuned weaker models are eventually used as rewards in the downstream search, we follow the standard practice to use the $\\pi^*$ checkpoint with the highest preference classification accuracy (we use DPO fine-tuning throughout the paper) and do not perform additional ablations.\n\n2. When reusing off-the-shelf tuned weak model $\\pi^*$ (e.g., instruction-following experiments): We have little control over these models' performance as their weights are fixed, and there are not enough open-source weak models to form a spectrum for ablation.\n\nHowever, we have demonstrated that weak-to-strong search works consistently across various weak models and outperforms other baselines.\n\n---\n\n> W2: As we can formulate a partial reward function using a tuned/untuned model pairs, apart from using it to guide decoding, a natural next step/ablation study is using it to conduct PPO. It is worth trying this experiment and compare the downstream performance as well as the overall cost.\n\nThank you for the valuable suggestion. Exploring if dense rewards can benefit PPO is indeed interesting. **Figure 2 in the supplementary PDF (global response) shows that dense rewards do accelerate PPO training. Weak-to-strong search (ours) outperforms vanilla PPO with sequence-level reward by a large margin, but it is less effective than chunk-level PPO.** \n\n**Despite these promising ablation results, we need to kindly emphasize that this work focuses more on training-free alignment and a framework for weak-to-strong generalization.** The key advantage of training-free method is its ability to guide a large model to approximate fine-tuning without significant computational resources. While training-based methods are promising, they are resource-intensive and unstable. For instance, we do not have the resources to fine-tune a 70B model, but we can approximate fine-tuning results using weak-to-strong search.\n\nWe appreciate the suggestions on experimenting with chunk-level PPO. **These ablations do make our paper more complete and we will include these results in the camera-ready version. However, a more thorough exploration and analysis of chunk-level PPO likely deserves another paper and will be addressed in future work.** For instance, given that dense rewards implicitly handle credit assignment, do we need value function modeling and GAE in PPO to stabilize training? Could vanilla REINFORCE achieve similar results as PPO under such dense rewards? We leave these questions for future investigation.\n\n---\n\nIf these answers do not fully address your concern, we are more than willing to offer additional clarifications.\n\n\n[1] Ouyang, Long, et al. \"Training language models to follow instructions with human feedback.\" Advances in neural information processing systems 35 (2022): 27730-27744.\\\n[2] Rafailov, Rafael, et al. \"Direct preference optimization: Your language model is secretly a reward model.\" Advances in Neural Information Processing Systems 36 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i86jZE2DN8", "reviewer": "Reviewer_Xaca", "summary": "In this paper, they frame alignment as a decoding time problem, allowing the large language model to be frozen. They do this by parametrizing a reward function with the difference in the log-likelihood between small untuned and tuned language models. Interestingly, their approach does not require shared vocabulary and is applicable to black-box LLMs. They perform experiments on controlled-sentiment generation, summarization, and instruction following.", "questions": "Questions: \n- Is it \\pi_base in L181?\n- What about the vocabulary of the small tuned and tuned models? Does it need to be shared?\n- How does your work compare with that of Li et al. (ACL 2023)? What about Zhao et al. (2024)?\n\nMinor comments: \n- BoN is defined in L204 but it’s used before (e.g., L166).\n\nReferences:\n\n- Contrastive Decoding: Open-ended Text Generation as Optimization (Li et al., ACL 2023)\n- Weak-to-Strong Jailbreaking on Large Language Models (Zhao et al., 2024)", "limitations": "Yes.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and organized.\n- The motivation is clear from the beginning.\n- The experimental setting is adequate and the results are good.", "weaknesses": "Although there’s a related work section, I believe the comparison between the proposed method and other methods, e.g., the ones that rely on small language models to aid the alignment of LLMs, should be described in more detail. In particular, the novelty of the proposed method should be made more evident; for non-experts, I think it’s a bit difficult to understand if some parts are novel or if they already exist in previous work. I also think that some citations might be missing (see my questions below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. To address your questions:\n\n---\n\n> Ｗ1: Although there’s a related work section, I believe the comparison between the proposed method and other methods, e.g., the ones that rely on small language models to aid the alignment of LLMs, should be described in more detail. In particular, the novelty of the proposed method should be made more evident; for non-experts, I think it’s a bit difficult to understand if some parts are novel or if they already exist in previous work. I also think that some citations might be missing (see my questions below).\n\nThank you for your suggestions on more detailed related works. Please see our response to Q3. We will incorporate these discussions into the camera-ready version.\n\n---\n\n> Q1: Is it \\pi_base in L181?\n\nThank you for pointing out this typo. We will fix it in the camera-ready version.\n\n---\n\n> Q2: What about the vocabulary of the small tuned and tuned models? Does it need to be shared?\n\nIn principle, our method does not require the vocabularies of small tuned and untuned models to be shared because we calculate the log-likelihood difference of small models over natural language chunks (which can be tokenized differently according to each model's vocabulary) rather than over particular tokens. In practice, though, when the former is fine-tuned from the latter, they almost always share the same vocabulary. Exploring cross-vocabulary guidance from different small model families (e.g., Llama2 and Llama3) is an interesting direction we plan to pursue in future work.\n\n---\n\n> Q3: How does your work compare with that of Li et al. (ACL 2023)? What about Zhao et al. (2024)?\n\nThank you for highlighting these related works. They are indeed quite relevant to ours, and we will discuss them further in our camera-ready paper. Both [1] and [2] can be interpreted as special cases of emulated fine-tuning (EFT) [3], which is a key baseline for our experiments. EFT adjusts a target model's output token distribution based on other models' distributions. Specifically, [2] uses EFT to remove a model's safety guardrails, while [1] applies EFT where $\\pi_{\\text{base}}$ and $\\pi^*$ are identical (see L210 for EFT formulations). However, these token-level manipulations have drawbacks: (1) they require all models to share the same vocabulary, (2) they do not apply to black-box models without full transparency of their output distributions, and (3) they are empirically less effective than **our proposed test-time search method, which is the only method that achieves weak-to-strong generalization, enhancing strong models with weak guidance**.\n\n---\n\n> C1: BoN is defined in L204 but it’s used before (e.g., L166).\n\nThank you for pointing out this issue. We will fix it in the camera-ready version.\n\n---\n\nIf these answers do not fully address your concern, we are more than willing to offer additional clarifications.\n\n\n[1] Li, Xiang Lisa, et al. \"Contrastive decoding: Open-ended text generation as optimization.\" arXiv preprint arXiv:2210.15097 (2022).\\\n[2] Zhao, Xuandong, et al. \"Weak-to-strong jailbreaking on large language models.\" arXiv preprint arXiv:2401.17256 (2024).\\\n[3] Mitchell, Eric, et al. \"An emulator for fine-tuning large language models using small language models.\" arXiv preprint arXiv:2310.12962 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are very thankful for the positive feedback from the reviewers. In this supplementary PDF, we include two additional empirical results: Figure 1: **inference costs analyses** [reviewer [mz4P](https://openreview.net/forum?id=dOJ6CqWDf1&noteId=aC7BMDbknK), W1] and Figure 2: **chunk-level PPO ablations** [reviewer [2D8T](https://openreview.net/forum?id=dOJ6CqWDf1&noteId=fzguH9HxKe), W2]. Please see the supplementary PDF and our individual responses to reviewer [mz4P](https://openreview.net/forum?id=dOJ6CqWDf1&noteId=aC7BMDbknK) and reviewer [2D8T](https://openreview.net/forum?id=dOJ6CqWDf1&noteId=fzguH9HxKe) for details.

Meta-review: This is a solid paper that proposes to align larger language models by using the difference in logits from smaller (tuned vs. untuned) models. The idea is interesting, and backed up by solid empirical experiments. There were some initial concerns as to the use of stronger baselines (in particular, those that incorporating search during decoding), as well as its position relative to related work. These have been largely addressed through the rebuttal. 

After reading the paper, I recommend that the authors further discussion the limitations of the proposed method (e.g., this method clearly would not work if the difference in capabilities of the two models is too large (e.g., n-gram model vs. a neural LM).
Decision: Accept (poster)