Collab: Controlled Decoding using Mixture of Agents for LLM Alignment
Authors: Souradip Chakraborty
Sujay Bhatt
Udari Madhushani Sehwag
Soumya Suvra Ghosal
Jiahao Qiu
Mengdi Wang
Dinesh Manocha
Furong Huang
Alec Koppel
Sumitra Ganesh
Venue: ICLR 2025 Poster (2025)
Keywords: Alignment
Decoding
RLHF
Transfer Decoding
LLM
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: Decoding with Mixture of LLM Agents via policy switching

Abstract: Alignment of Large Language models (LLMs) is crucial for safe and trustworthy deployment in applications. Reinforcement learning from human feedback (RLHF) has emerged as an effective technique to align LLMs to human preferences, and broader utilities, but it requires updating billions of model parameters which is computationally expensive. Controlled Decoding, by contrast, provides a mechanism for aligning a model at inference time without retraining. However, single-agent decoding approaches often struggle to adapt to diverse tasks due to the complexity and variability inherent in these tasks. To strengthen the test-time performance w.r.t the target task, we propose a mixture of agents-based decoding strategies leveraging the existing off-the-shelf aligned LLM policies. Treating each prior policy as an agent in the spirit of mixture of agent collaboration, we develop a decoding method that allows for inference-time alignment through a token-level selection strategy among multiple agents. For each token, the most suitable LLM is dynamically chosen from a pool of models based on a long-term utility metric. This policy-switching mechanism ensures optimal model selection at each step, enabling efficient collaboration and alignment among LLMs during decoding. Theoretical analysis of our proposed algorithm establishes optimal performance with respect to the target task represented via a target reward, for the given off-the-shelf models. We conduct comprehensive empirical evaluations with open-source aligned models on diverse tasks and preferences, which demonstrates the merits of this approach over single-agent decoding baselines. Notably, COLLAB surpasses the current SoTA decoding strategy, achieving an improvement of {up to 1.56x} in average reward and $71.89\%$ in GPT-4 based win-tie rate.

Published as a conference paper at ICLR 2025
COLLAB: CONTROLLED DECODING USING MIXTURE OF AGENTS FOR LLM ALIGNMENT
Souradip Chakraborty 1.2
Sujay Bhatt'
Udari Madhushani Sehwag'
Alec Koppel'
Soumya Suvra Ghosal
Jiahao Qiu
Mengdi Wang"
Dinesh Manocha²
Furong Huang
Sumitra Ganesh
JJPMrrgan Al Research
University of Maryland, College Park Princeton University
ABSTRACT
Alignment of Large Language models (LLMs) crucial for safe and trustworthy deployment in applications. Reinforcement learning from human feedback (RLHF) has emerged as an effective technique align LLMs human preferences, and broader utilities, but it requires updating billions of model parameters which is com- putationally expensive. Controlled Decoding, by contrast, provides mechanism for aligning model at inference time without retraining. However, single-agen decoding approaches often struggle to adapt to diverse tasks due to the complexity and variability inherent in these tasks. To strengthen the test-time performance w.r. the target task, we propose a mixture of agents- based decoding strategies leveraging the existing off-the-shelf aligned LLM policies. 'Treating each prior policy as an agent in the spirit of mixture of agent collaboration, we develop a decoding method that allows for inference-time alignment through token-level selection strategy among multiple agents. For each token, the most suitable LLM is dynamically chosen from a pool of models based on long-term term utility metric. This policy-switching mechanism ensures optimal model selection at each step, enabling efficient collaboration and alignment among L.MSs during decoding. Theoretical analysis of our proposed algorithm establishes optimal performance with respect to the target task represented via a target reward, for the given off-the-shelf models. We conduct comprehensive empirical evaluations with open-source aligned models on diverse tasks and preferences, which demonstrates the merits of this approach over single-agent decoding baselines. Notably, COLLAD surpasses the current SoTA decoding strategy, achieving an improvement of up to 1.56x in average reward and 71.89% in GPT-4 based win-tie rate.
INTRODUCTION
Large language models (and generative models) excel at generating coherent and realistic text, but many text generation tasks require outputs that not only preserve fluency but also require satisfying constraints, such as factual accuracy (Wang et al., 2024), knowledge grounding (Liang et al., 2024), adherence to safety guidelines (Dong et al., 2024; Xie et al., 2024), or task and domain-specific objectives (Jeong, 2024). Ensuring personalized or task-specific alignment in LLMs requires fine- tuning them for specialized objectives. Alignment approaches like reinforcement learning from human feedback (RLHF) (Ouyang et al., 2022; Stiemnon et al., 2022; Ziegler et al., 2020; Yuan et al., 2023; Chakraborty et al., 2024c) have shown efficiency in aligning generative models with human and task-specific preferences. However, fine-tuning billions of parameters with RLHF is computationally intensive and becomes impractical in the context of personalized or highly specialized alignment. Controlled Decoding (Mudgal et al., 2024) aims to address this challenge by providing a training- free inference- time framework that allows models to be aligned with target preferences and tasks without requiring the retraining of billions of model parameters. Recent works (Mudgal al., 2024; Khanov et al., 2024; Chakraborty et al., 2024b) have demonstrated that inference-time alignment can significantly enhance the ability of large language models to meet task-speeific and personalized
Work done part of internship JPMorgan AI Research. Correspondence: Souradip Chakraborty <scha't k a 3@umd. edu>, Alec Koppel <a lec. koppe1 @ jpmchasseeee se, com>.

---

Published as a conference paper at ICLR 2025
requirements, even improving the fine-tuning based algorithms (Khanov et al., 2024; Mudgal et al., 2024; Chakraborty et al., 2024b). These methods, while effective for single-agent decoding, are limited when it comes to handling diverse conflicting task requirements, they struggle to generalize across tasks that demand fundamentally diverse or specialized capabilities.
Prompt: How plants make food?
Agent
Agent Plants create food through a process called photosymthesis, where they use sunlight to convert carbon dloxide and water into glucose, sugar provides energy for growth.
Agent 2:Pholceynthesis is chamical process occurs in the chloroplaste, which contain green pigment called chlorophyll that absorts sunlight. This
energy carbon dioxide and water
Prompt
into glucose and oxygen.
COLLAB
COLLAB
Plants (Agent 1) make food through a process called (Agent 2) photosynthesis. (Agent 1) This chloroplasts (Agent 2) of plant cells, where chlorophyll (Agent 1) absorbs sunlight. The energy from the sun (Agent 2) then used to transform carbon dioxide and water (Agent 1) into glucose, a sugar that provides energy for growth, (Agent 2) and oxygen is released as a. byproduct.
the
Agent
The figure illustrates the optimal coordination between agents for response generation via switching, where Agent ChatAgent and Agent2 Chemical-Expert In this collaborative response, the agents are switching smoothly at the word and phrase level to deliver a more detailed and complete response than they could individually. The switching demonstrates how both agents complement each other in explaining the complex process.
Figure I:
Limitation with Single agent decoding: The performance of single-agent decoding approaches often struggles adapt diverse tasks due the complexity and variability inherent these tasks and also due to their over-reliance on the training task distribution. Specialized tasks may demand conflicting capabilities: for example, one task might require fact-based grounding, such as retrieval or summarization, where accuracy and precision are critical, while another may involve creative outputs like writing poetry or fiction, where imagination and stylistic freedom are more important These conflicting demands make it difficult for single-agent decoding method to generalize effectively and meet the distinct requirements of both fact-driven and creative tasks simultaneously.
Challenges in Multiagent Decoding for Alignment: A straightforward solution to the above chal- lenges is to leverage multiple existing off-the-shelf LLMs, each specializing in different tasks and domains, to collaborate efficiently during inference. However, key challenge lies in efficiently bining these LLMs in a tuning-free free manner without requiring retraining for each new task. Current methods that attempt to integrate multiple LLMs often rely on weak supervision or explicit formulas for mixing model outputs (Shen et al., 2024; Jin et al., 2024; Yang & Klein, 2021; Liu et al., 2024; Mialon al., 2023), but these approaches lack the flexibility to adapt dynamically during token-level generation. Additionally, most prior methods rely on expert supervision to guide the combination of logits or determine when to leverage other LLMs, which can restrictive. Moreover, many existing approaches require some form of training to integrate multiple LLMs, process that can be expensive or infeasible when access limited. To the best of our knowledge, none of the prior decoding alignment methods have thoroughly explored the optimal selection of specialized agents during inference, which requires defining a principal metric for selection. Thus we ask the question.
Recent methods for integrating multiple LLMs often rely on weak supervision or fixed formulas to mix outputs, lacking flexibility for dynamic token-level generation. They typically require expert guidance for combining logits or deciding when to leverage other models, which can be restrictive.
How can existing off-the-shelf models be optimally combined during inference to target specific tasks without retraining, allowing for dynamic collaboration and adaptation the token level?
Mixture of Agents based Controlled Decoding: Hence, in this work, we leverage and combine the strengths of multiple off-the-shelf LLM models, each aligned with specialized diverse tasks, to enable decoding without retraining, in the spirit of a mixture of agents approach. More specifically, each agent is aligned with specialized preferences through distinct reward functions (Ouyang et al.,
2

---

Published as a conference paper at ICLR 2025
2022; Wožniak al., 2024). We propose mixture-of -aairee----ggg-----aeee basents-aseddde controlled decoding strategy that aligns the final response to a new task without retraining via aggregating the next-token prediction distributions from each agent during the response generation phase. However, generate the response optimally by combining the L.Mss, it is crucial to address the question of how to design an effective aggregation strategy (decoding policy) that combines the outputs from the mixture of agents.
Switching between Mixture of Agents with Q-function : To address the question of designing a suitable decoding policy, we identify that optimal decoding is fundamentally guided by the long-term utility with respect to a semantic reward, often characterized by the Q-function (Mudgal et al., 2024; Chakraborty et al., 2024b). This allows us to define an optimal policy by framing the decoding process through the lens of a KL-regularized reinforcement learning problem. This framework is specifically crafted solve the alignment problem by selecting the policy for each token during decoding to be optimal with respect to the aforementioned implicit Q function. We introduce COLLAB Controlled decoding via Mixture of Agents as a potential solution for test-time collaborative alignment, with an implicit Q-function-serving
the guiding metric for decoding. Our proposed method is grounded in a policy-switching mechanism, where the implicit Q-function is used to optimally select the aligned model at each time step during decoding. We define the notion of the implicit Q-function within the mixture of agents decoding algorithm and demonstrate its effectiveness as an optimal alignment metric, both theoretically and empirically. We show that the implicit Q-function provides a principled approach for selecting the model that best aligns with the target task achieving the optimal performance possible under the given scenario. We summarize our key contributions as follows:
Mixture of Agents Decoding Strategy for Alignment We introduce novel mixture-of off-eensss decoding strategy that leverages specialized off-the-shelf LLM agents, each aligned with a specific task or reward function. Our method optimally switches between these agents during decoding to achieve alignment with the target reward function, without requiring retraining.
Collaborative Metric for Mixture of Agent Decoding: We propose the concept of the implicit Q-function as guided metric to combine the mixture of agents for decoding. This metric enables an appropriaye combination of the off tff-the-shelf LLM agents during inference w.r.t the target objective.
We provide precise theoretical characterization of our proposed approach via analyzing the sub-optimality gap w.r.f to the target reward. We characterize the sub-optimality of our approach w.r.t the true Q-function by upper-bounding the performance gap through the reward difference between the target and best model for each token, along with the KL divergence from the reference policy.
Theoretical Characterization of COLLAB:
Experimental Evaluations: Through extensive empirical evaluations, we demonstrate that our pro- posed decoding approach significantly outperforms state-of-the-art single-agent decoding baselines in task alignment performance, particularly in scenarios involving diverse or conflicting requirements. We also conducted comprehensive evaluations using various metrics iincluding average reward, GPT-4 win rate, coherence, and diversity- to showcase the superiority of our approach. Addition- ally, we performed ablation studies to show that diversity among agents enhances collaborative performance in the mixture.
2 RELATED WORK
Alignment via fine-tuning with reinforcement learning from human feedback (RLHF) has emerged as a key paradigm for aligning foundation models (Ouyang al., 2022: Chakraborty et al., 2024a; Rafailov et al., 2023; Chakraborty et al., 2024c; Chen et al., 2024). RLHF typically operates in two phases: first, a reward model is trained on human feedback. and then a policy is fine-tuned with reinforcement learning (PPO (Schulman et al., 2017)) using the trained reward model (Ouyang et al., 2022; Stiennon et al., 2022; Ziegler et al., 2020; Yuan al., 2023; Go al., 2023; Vamplew et al., 2018; 2008). Direct preference optimization seeks to stabilize the training of preference alignment through reductions to supervised learning training (Rafailov et al., 2023). Although these training methods have proven effective in aligning generative models, they remain computationally demanding and assume white-box access to the model parameters which is not true in many industry applications.
On the other hand, decoding based methods (Mudgal et al., 2024: Chakraborty al., 2024b) have emerged as an alternate way of alignment without fine-tuning the model parameters. Decoding operates by altering the distribution of the generated response to align the target preference directly
3

---

Published as a conference paper at ICLR 2025
without updating the parameters of the LLM. The work by (Mudgal et al., 2024) is one of the first to integrate the aligmment procedure directly into the decoding process, where they propose adjusting the generation probabilities at each decoding step based on feedback from reward model. (Huang et al., 2024) redefined the text-generation process as a search problem, with LLMS acting as search agents and they employ a heuristic-guided search mechanism generate responses based on a given prompt. The most recent research around Controlled and Principled decoding (CD, TQ*) formulates the decoding problem as a KL regularized reinforcement learning problem and obtains closed-form solution with an estimate of Q" for decoding. However, the majority of these prior approaches are inherently single-agent alignment strategies and lack the ability to leverage the strengths of diverse agents in limiting their effectiveness in complex, multi-faceted tasks. Recent methods attempt to integrate multiple LLMs either rely on weak supervision or explicit formulas for combining model responses (Shen et al., 2024; Jin et al., 2024; Yang & Klein, 2021; Liu et al., 2024; Mialon et al., 2023), which might be restrictive.
3 PROBLEM FORMULATION: COLLABORATIVE MULTIAGENT DECODING
3.1 PROBLEM DEFINITION
Task and Preference Representation with Target Reward Function: To address the problem of optimal response generation, we first define the framework around a specialized target score or reward function I'urget which serves as the key objective guiding response generation. The reward function Ttarget encapsulates a wide range of tasks and objectives, thereby allowing for flexibility and adaptability. These can include: 1. Personalized alignment: Farget can be tailored to individual preferences or specialized tasks, ensuring personalized responses that align with user- sser-specific goals, 2. Rules-bard -aased approaches: Various rules or heuristics used in specialized contexts can be represented as constraints or structured forms of the reward function, making it possible enforce domain-specific guidelines, 3. Supervised signals: Any supervised learning signals or labels provided during training can be mapped as rewards, allowing the learning system directly optimize for them, 4. Open-source reward models: Many publicly available reward models that have been fine-tuned for specific objectives can be expressed under Furget broadening the applicability across multiple domains. By casting these diverse objectives within the framework of a reward function, we ensure that the system is capable of handling a variety of alignment and optimization tasks through the target reward function.
Given a target reward function, the challenge lies in determining the optimal way to generate response from the L.LM agent and becomes especially difficult when the target reward significantly deviates from the reward associated with the aligned policy. Single-agent systems often struggle with adapting to diverse tasks, primarily due to: (1) Task Complexity and Variability: Tasks differ significantly in their structure and requirements: for example, one task might require generating long-form responses, while another might favor brevity. Similarly, some tasks might demand reasoning suited to expert audience, such as PhD-level logic, while others might need responses tailored to general understanding (Jang al., 2023; Wožniak et al., 2024) (2) Training Task Over-reliance: Single agent alignment methods often overfit to the distribution of training tasks or reward function, leading to reward overfitting or overoptimization (Gao et al., 2023; 2022; Zhu et al., 2024) resulting in limited generalization target reward and tasks.
Response Generation Under Specialized Reward Functions:
Mixtue of Agents for Alignment: However, we note that there are already existing available off-the shelf LLM policies aligned to a diverse of set of tasks (Lambert et al., 2024; Jang et al., 2023; Wožniak et al., 2024). Hence, address these limitations. we leverage the set of already available specialized L.MM policies represented as II (TI, T2 ... . aligned to set of tasks (or preferences) as defined by specific reward functions belonging to set R (r1, r2- ... } These policies can collectively handle diverse range of tasks, allowing for more flexible and effective adaptation.
Challenge in Multiagent Decoding for Alignment: The challenge remains: the reward functions {11,72 17...2...77 } are latent and unobservable, often because they are proprietary or embedded within the model's 's training data, which may not be available for external analysis. Thus, transferring efficiently to the target reward involves adapting the response generation process based the information from these pre-trained and given input models without explicit access to their internal reward functions leading to the key questions in Multi-agent decoding for alignment.
4

---

Published as a conference paper at ICLR 2025
Key Questions in Multi-Agent Decoding: One of the key challenges when decoding with multiple agents is determining when to switch between models and the second challenge lies in identifying the appropriate metric to guide this decision.
Formalizing the Problem with Markov Decision Processes: We provide a formal answer to these questions by defining the response generation phase with an appropriate Markov Decision Process with a specific reward function, and its associated optimality criteria. Doing so is the focus of the following subsection.
3.2 TEVEN-LEVEL MARKOV DECISION PROCESS
We begin by formulating the decoding problem as KL regularized reinforcement learning problem (Mudgal et al., 2024; Chakraborty et al., 2024b) with the token-level MDP M := where the state-paae- S represents the concatenated sequence of tokens and the action space A representing the space of the next token i.e vocabulary V. Given a state S = [x, Y E S, which sequence of tokens containing the prompt/query == {æ1, 22,
( A, P. R)
.NN appended with the I tokens y<t == {yo. "1.... V--1 generated so far, an LLM is a policy π that samples actions as next sampled tokens the action (i.c., the next token) at 1/2 via sampling from the token-level decoding policy y/+ T[- st). The transition / to the next state St+1 is deterministic: S++1 = [x. yxy Vel the concatenation of the current state and action. We denote the trajectory level probability by
Pa (=|x) = !I=- π Tyell [x,y y<l)) and T ~ (-(x) represents a sampled response/trajectory which is concatenation of tokens, and 2 is an arbitrary token.
The reward r(s, a) : S x A + R. To be specific, the action-vale value function associated with the reward is then defined for length L as
(n) E, Σ r(s 21) so s, 20 2,24 x(xlx
((sr)) [r[[s, z), 7') ss
where is the state and z E V the action for the state and is sampled from the trajectory distribution Pa (-(s, 2) induced by the policy π (8, z).
3-3 DECODING PROCESS CONTROLLED DECODING
In this section, we introduce the decoding process where in we have a reference policy net which takes as an input the prompt x E V (a string of N tokens) and generates a response y = [Mo, y1. .... ,OOS] token by token with a probability of each token t is given by ([.][x,y=s))] With z E V as token from vocabulary set V. the objective of LLM alignment via decoding is formally defined by solving the KI. regularized MDP M as
= (-Iss) := arg max (2) (s 2)] - xl (Ta((())))
(2)
where we note that St [x. y s+), and Q* =) denotes the optimal state-action value function for the token-level MDP M defined in the previous section. The KI. constraint ensures closeness to the reference policy (=)s)) with hyperparameter
0. The closed-fone solution of the problem
exp Q= (st. z))
n¹ (2((s) = (zr(2))
(3)
C. (8:)
where
E. (2er(2)st) exp (s 23) )) is the normalizing constant for state St (Mudgal
et al., 2024). However, it is important to note that Q" (st, 2) is the optimal state-action value function and is unavailable in practice. Recent works by (Khanov et al., 2024; Chakraborty et al., 2024b; Mudgal et al., 2024) design approximate methods to estimate this Q function and thus adjust the probability of the reference policy with exponential of Q* as in equation 2 to obtain the final decoding policy, Recent works have also characterized the sub-optimality of decoding under such approximations (Chakraborty et al., 2024b). However, the entire decoding process is primarily defined for single-agent systems, and in particular, a single training distribution and aligned policy. Extending this problem class to multiple agents is the goal of this work.
5

---

Published as a conference paper at ICLR 2025
3.4 PROPOSED METHOD: MIXTURE AGENT-BASED CONTROLLED DECODING ALIGNMENT
With the problem of sampling from a single LLM agent being well-defined, we now turn to the challenge of decoding from multiple off-the-shelff LLM policies. We introduce our proposed approach of the mixture of agent-based controlled decoding by leveraging existing off-the-shelf LLMs for better generalization to target reward function. The objective
efficiently align to the target reward function Ftarger with the off-the-shelf LLM policies I1 = { 22 Tk J to hidden reward functions 1...2...77 Plarget then we could have followed similar steps from prior works on decoding (Chakraborty et al., 2024b; Mudgal et al., 2024), and decode using the final equation 2, where we can estimate the Q" as
R. = (ri, r2... r }. If we had access to the true aligned policy "arget for the target reward function
Guage (St, 2) Eyrp Gane (-1a..2) [Fungerr [[s 7)(ss
where Pxima is the corresponding trajectory level policy (joint- coint-distribution) under the target reward function Trargets defined in Section 3.2. However, as previously mentioned, "inger is unavailable for the target reward Fuuget- Thus our proposed approach focuses on designing a collaborative strategy to leverage existing fff-the-shell LLMs policies II { ...... -T& to decode to the target reward I'argeet Thus, one may not conduct general policy search, but instead restrict focus to a feasible set of prior models II, formalized as:
x² (+1xe) == arg max (- [Qirgee (st, =)] a.DKI. Well
[= [=<(|ss)]] [err( [s)]]
(5)
where, the KL-regularized RL problem for each token is defined over the constrained set π II. Tee is the reference policy (St z) is the long-term action-value function for that corresponding policy, given as
Taiger(Se:) 2) Er (118..)) [Mtarger 2],
(6)
For simplicity of notations, we denote the objective for the jth agent as (S;, = Quarge 2) a LKT ms)(- Thus, for each state s, = [x, yct). where x is the prompt and Yct. is the sequence of tokens till time i, the optimal choice of the next token is given as x* arg max; max, (St z). A detailed description of our policy switching algorithm is presented in algorithm 1 and discussed in the next section.
Mixture of Agents-based Decoding via Switching: Our proposed mixture of agents-based decoding approach induces policy-switching mechanism among the individual LLM agents at each state St aimed at generating a final response aligned with the target reward Trarget- For each state s; = [xr, yeel our approach identifies the optimal LLM agent for decoding based on the maximum value of the long-tern metric Jiugee (St, 2) which we denote as the implicit Q-function. Subsequently, it selects the token y/= = z generated using that agent based
algorithm's 's decoding policy
Talg E arg max max ungee (St,
This token is then concatenated with the previously generated response, forming the next state $.41 s, Uyt which is subsequently used as a prompt for all agents for the next time steps. Thus, the mixture of agents implicitly collaborates, with each agent conditioned on prompts generated through policy switching mechanism (see F). This dynamic interaction yields an improved response, better aligned with the target reward Taaget. We present the detailed flow of our proposed approach in algorithm 1.
4 THEORETICAL ANALYSIS OF MIXTURE OF AGENTS DECODING FOR ALIGNMENT
In this section, analyze the sub-optimality gap of our proposed Collaborative mixture of agent decoding strategy w.r.t the target reward functions Furget. We quantify sub-optimality of a decoding method π w.r.t the long-term action-value Q function for the target reward.
A(T) = Qurpe (St, 2) Quangen (Sz, 2)
(8)
where nt represents the optimal policy under the target reward function Iiarget and is an arbitrary decoding policy induced. Subsequently we study equation 8 when decoding according to π = Talg as defined in Algorithm! 1.
policy

---

Published as a conference paper at ICLR 2025
Algorithm 1 Mixture of Agents based Controlled Decoding for LLM Alignment
Input: Set of LLM policies II = {T1. T22 "Tx) aligned to diverse set of latent reward functions, target reward function Ftargets dataset of prompts æ E D, decoding parameter a, vocabulary set for 0,.... do
Current state: = yct) with prompt x and prior token sequence y<t <t= [[0o= (0o-V:: V/,
9t-11
for i 1..... ., K do
Sample top-p tokens using 7t aligned policy π. and store as V; { z ~ (.(-(s)) {-)p for 2 E Vi do
Evaluate the implicit Q-function for the eth policy Jagga (st, 2) using equation 5 Estimate the best token for the 7th policy as = arg max=ev;; Junge (St,
Select Smmx = arg max; Juange (St, =mmx)
Next Token: yt+1 &- Zmax
Next State: St+1 < [St, V[+1]
Return: y 1 .77
Example of response generated using different strategies
Prompt:
Human: 10+110+ 10-30 equals what
Assistant: you'le re missing a positive sign... the result is positive.
Human: No, math equation. Let's try another way. What plus 10 plus 10 minus 30?
Assistant:
Agent-I: 10 10 10 30 equals 100, because the positive numbers add up 30, and then you't 'e suhtracting 30, which leaves you with the negative result of -10. If you meant to add the numbers before subtracting, then the result would 30, as you have after adding 10, 10, and 10, before subtraction of 30.
Agent-ll: II: 10 10 10 30 equals 10. Because when you add 10, 10, and 10, you get 30. Then, when you subtract 30, you get -10. It's like you're taking 30 away from balance, and you end up the negative.
BoN Sampling: This question has three separate elements. 10+10+10 100, and -30 is -30. The entire question can be expressed -
COLLAB (Ours): 10 + + 10 - 30 30 O, Explanation: To evaluate the expression, perform the
addition and subtraction in order from left to right: 1. Add 10 and 10 to get 20: 10 10 20, 2. Add 10 the previous sum: 20 10 30
Theorem 1 (Sub- optimaiiiaality Bound of Multi-Agent Decoding Algorithm).. Let {MT1 T2,.... be a set of pre-trane pre-trained policies, each aligned to a latent reward function r, and Walz be the policy obtained by the multi-agent decoding strategy. Assume that the optimal policy for the target reward function r is " Then, the sub-optimality of the multi-agent decoding policy Walg with respect to the optimal policy H² is bounded by:
((TTat)) S min 8-j + a a. L( (-1ss), Tny (s)) (KL(L" (-Is:) Prst))))))) jEK
(9)
where d.g max, [Ttorger([s z]. 7) r, ([s 2), 7) a, 3 > are regularization constants for the KL- KL-divergence terms to ensure closeness to the reference policy at the token and trajectory level, ö.; represents the difference between the target reward function and the reward function of the closest to the target. K [L(T(())), Ted(-1s) represents the KL-divergence between policy π and the supervised fine-tuned policy Fef
Theoretical Insights and Key Remarks The sub-optimality gap is expressed in terms of the difference between the target reward function and the reward function of the best model in our policy set (closest to l'argee) 8.j for each token s; and the sum of KL divergences w.r.t the reference policy at the token and trajectory level. The sub-optimality gap thus guarantees that the performance of our multi-agent decoding strategy will improve over the bert-perg policy (closest to the
7

---

Published as a conference paper at ICLR 2025
Table 1: GPT-4 Based Evaluation. We prompt GPT-4 rate responses from various decoding strategies on relevance, accuracy, and insightfulness, scoring them from to 10. A higher win-tie percentage indicates our method's effectiveness in generating contextually relevant and accurate responses.
Win-Tie (%)
Ours
Methods
Task-1
Task-I1
Evaluation-1
Evaluation-2
Evaluation-3
Evaluation-4
Evaluation-5
Exaluation66
Evaluation-?
COLLAS
AgentI
70.78
71.89
69.38
60.00
57.14
64,28
$9,42
COLLAR
Agent1ll
41
61.05
COLLAS
BoN Sampling
68.5 54
69.24
61.22
66.32
50.00
73.75
65
target LLM policy) from policy set, under the assumptions. The gap decreases as 6eq becomes smaller, meaning that the reward function of the best LLM agent in the set closely aligns with the target reward. Furthermore, by adjusting the regularization constants a; B we can control the trade-off between closeness to the reference policy and distance from the optimal policy. The term LKL (-Is;), Pre((-(se) is constant and independent of j quantifics the divergence between the optimal policy and the reference policy, which is lower in two cases 1) is small and 2) when the reference policy is closer to the optimal policy. Thus, the sub-optimality gap will be lower when the best agent's reward function is close the target reward function, and when the regularization terms are properly controlled to maintain proximity to both the reference policy and the optimal policy, In contrast prior results in decoding and alignment (Rafailov al., 2023; Mudgal et al., 2024; Chakraborty et al., 2024b), we mainly incur an additional term d.j in the upper bound. However, as discussed prior alignment approaches either require fine-tuning billion parameters ((afailovv et al., 2023)) for each new Uarget which is expensive, or access to an aligned policy under Targge (decoding (Mudgal al., 2024)) which might be unavailable. Thus the absence of the aligned policy to the target reward, there will be persistent gap [ry Ttarget for single agent decoding, which is improved using the diverse mixture of agents.
5 EXPERIMENTAL EVALUATIONS
In this section, we present a comprehensive empirical analysis of our proposed framework, tested across various open-source datasets and state-of-the-arr models (Lambert et al., 2024). Our findings demonstrate COLLAB'S effectiveness in aligning language model outputs with specific target rewards. For implementation, set the number of tokens sampled (top-p) 10 and the decoding alignment parameter a 1. Reproducibility is ensured through the use of publicly available resources.
Experiment Setup: To demonstrate the efficacy of our proposed multi-agent decoding framework COLLAB, we utilize several open-source LLMs fine-tuned on distinet and diverse capabilities, as detailed in Table 3. Our multi-agent decoding framework is evaluated across distinct setups, as illustrated in Table 3:
1. Evaluation-1 1 to Evaluation-4 (Task-I): For this task, we utilize the Berkeley Nectar dataset (Zhu et al., 2023) to test the agent's capacity for multi-turn dialogues and question answering.
2. Evaluation-5 to Evaluation-7 (Task-II): We employ the HH-RLHF dataset (Bai et al., 2022) to assess the agent's s helpfulness and ethical alignment in response generation.
Evaluation Methodology & Metrics. For evaluation, we compare the performance of the response generated by the language model corresponding to each prompt in the test dataset. Following (Khanov et al., 2024; Chakraborty et al., 2024b), limit the maximum length of the prompt and generated continuation to 128 and 2048 tokens, respectively. For all baselines, we utilize a greedy-based sampling method. The quality of the generated responses is assessed based on:
Average Reward:
We report the mean of the rewards for generations corresponding to all prompts in the test set. A higher mean reward score signifies that the model's outputs are better aligned with the attributes represented in the target reward model.
8

---

Published as conference paper at ICLR 2025
1.0
Average Rewarr
gr
1.0
Average Reward
1.0
0.8
0.6
0.6
Average
0.4
0.2
0.2
0.0
BoN Sampling
Agent-l
Agentall
Agent-
Agent-l
Agent-i
Sampling
Decoding
(a) Evaluation
(b) Evaluation
(c) Evaluation
Average Reward
1.0
Average Reward
1.0
0.8
0.6
0.6
0.0
Average
0.4
0.2
0.2
0.2
Sampling
Decoding
Decoding
Sampling
Decoding
Decoding
Sampling
Decoding
Decoding
(d) Evaluation
(e) Evaluation
Evaluation
Figure 2: In the above plots, present the normalized average reward values obtained using the corresponding setup outlined Table 3. Agent-I, and Agent-ll refers to the average reward obtained by the individual models with SoTA decoding. For the BoN agents sampling, logit-based sampling using individual agents and select the best response w.rt the target reward. Our analysis reveals that across all setups, COLLAB consistently outperforms other baselines summarized in Table 3, demonstrating the importance multi-agent decoding.
perform vanilla
0.9
0.9
0.9
0.8
008
0.7
Diversity
Diversity
0.6
0.6
0.5
0.5
0.5
0.4
0.4
0.4
0.3
BoN
Agent-l Decading
Agentill Decoding
Ours
0.3
BoN
Agent-I Decoding
Agent-ll Decading
Ours
0.3
BoN
Agent-I
Agent-Ii Decading
Ours
Sampling
Sampling
Sampling
Decading
0.70
0.70
0.70
0..5
-0.65
Co.60
0006
0.060
9.0.5
80.55
8 0.55
5.50
50.45
0045
0.45
0.40
0.35
0.30
DoN
Agent-1
Agent-ii
Ours
0.30
BoN
Agent-i
Agent-il
Ours
0.30
BoN
Agent-i
Agent-il
Ours
Sampling
Decoding
Decoding
Sampling
Decading
Decoding
Sampling
Decoding
Decoding
(a) Evaluation
(b) Evaluation 2
(c) Evaluation 3
Figure 3: In the above plots, we present the diversity and coherence values obtained using the corresponding setup outlined in Table 3. We clearly observe the response generated using COLLAB consistently outperforms other baselines in-terms of both diversity and coherence. This also indicates switching between agents with Implicit-Q helps in improving the overall quality the responses
Average Reward
1.0
1.0
0.8
Average Reward
0.8
0.6
0.4
0.44
0.2
0.2
0.0
Individual Decoding
Switch Diveristy
Switch with
Diveristy
0.0
N=1
N=2
N=3
N=4
N=5
N=6
(a)
(b)
Figure 4: Left. The bar plot highlights the importance of using diverse agents for enhanced decoding: employing different but non-diverse agents results in poor performance. Right. The visualization shows the improvement in average reward the number of diverse agents increases.

---

Published as a conference paper at ICLR 2025
GPT-4 Winrate: To further evaluate the quality of the generated responses, we employ GPT-4. based evaluation framework, using GPT-4 as a surrogate for human assessment. We prompt GPT-4 to rate responses from various decoding strategies relevance, accuracy, and insightfulness, scoring them from 1 to 10. A higher win-tie percentage indicates our method's effectiveness in generating contextually relevant and accurate responses. This is also considered as surrogate for the target reward as common in (Rafailov et al., 2023; OpenAI et al., 2024).
This metric measures the ability to generate texts with a wide range of vocabulary. Specifically, we calculate diversity using the frequency of repeated n-grams in the generated text. A lower repetition rate signifies higher diversity, indicating that the model is capable of generating novel, varied responses instead of repeating phrases structures frequently.
Diversity:
Coherence evaluates the semantic relatedness between each prompt and its generated response. We use SimCSS--generatd (Su et al., 2022) embeddings to represent both the prompt and the response. The cosine similarity between these embeddings then calculated quantify their semantic closeness. A higher cosine similarity score indicates that the generated response is contextually relevant and coherent with the input prompt.
Coherence:
Baselines. In Figure 2, we present the normalized average rewards for six setups detailed in Table 3. We compare our proposed method COLLAB with comprehensive baseline approaches including decoding with individual agents and BoN sampling (Nakano al., 2021). For generating baseline responses, we leverage state-of-art decoding approaches (Khanov et al., 2024; Chakraborty et al., 2024b). Specifically, in our experimental setting, Agent-1, Agent-2 represent the response generated using SoTA decoding (Chakraborty et al., 2024b). For the BoN agents sample, we perform vanilla logit-based sampling using individual agents and select the best response w.r.t the target reward. To provide clearer comparison of results, we normalize the average rewards (further details of normalization Appendix 1)
Evaluation Results. In our experimental evaluations, we demonstrate that our proposed framework, COLLAB, consistently outperforms existing baselines across all setups and a diverse range of metrics. As shown in Figure 2, our approach achieves superior performance in terms of average reward alignment with the target model. Notably, we observe that simply switching between two LLM agents allows the resulting decoding strategy generate responses that achieve significantly higher rewards than either individual agent, reinforcing our point. To further evaluate the quality of the generated responses, report the diversity and coherence metrics in Figure 3 for three evaluation setups. The responses generated using COLLAB outperform all other baselines across all evaluation metrics.
GPT-4 Evaluation and Insights. To assess the performance of our approach, we employed GPT-4 to evaluate and rate pairs of responses to the same prompt on a scale from to 10, focusing on relevance, accuracy, helpfulness, harmlessness, and insightfulness. We randomly sampled 300 prompts from the test set and compared the responses generated by COLLAB with those from other competitive decoding methods. The GPT-4 evaluation results presented in Table 1, showing the percentage of win-ties for our method over baseline decoding strategies. A higher percentage indicates that our proposed method is more proficient in generating responses that align better with human preferences. As shown in Table 1, COLLAB consistently achieves a higher win-tie percentage compared to other decoding approaches, reaffirming its efficacy.
Effect of Increasing the Number and Diversity of Agents in the Mixture. Figure 4a presents visualizations illustrating the impact of agent diversity decoding performance. For this evaluation, we consider two setups: (I) Switch without Diversity, where we use two similar models for switching; and (2) Switch with Diversity, where we employ two diverse agents. The results indicate that using a diverse mixture of agents significantly improves the average reward compared to individual decoding and switching between non-diverse agents. Furthermore, Figure 4b demonstrates that as the number of diverse agents increases, there is consistent improvement in average reward, highlighting the importance of both the number and diversity of agents for enhancing decoding performance.
ACKNOWLEDGMENTSS
Huang is supported by DARPA Transfer from Imprecise and Abstract Models to Autonomous Tech- nologies (TIAMAT) 80321, National Science Foundation NSF-J1S-1147276 FAI, DOD-ONR-Office
10

---

Published as a conference paper at ICLR 2025
of Naval Research under award number N00014-22-1-2333 DOD-AFOSR-AL Air Force Office of Scien- tific Research under award number FA9550-23-1-0048 DOD-DARPA -eeeeeseeeeeeee Advanced Research Projects Agency Guaranteeing AI Robustness against Deception (GARD) HR001 12020007, Adobe, Capital One and JP Morgan faculty fellowships. The authors would like to thank Amrit Singh Bedi for helpful discussions during the problem formulation.
DISCLAIMER
This paper prepared for informational purposes in part" if the work is collaborative with external partners] by the Artificial Intelligence Research group of JPMorgan Chase & Co. and its affiliates ("JP Morgan") and is product of the Research Department of JP Morgan. JP Morgan makes no representation and warranty whatsoever and disclaims all liability, for the completeness, accuracy or reliability of the information contained herein. This document not intended as investment research or investment advice, or recommendation, offer or solicitation for the purchase sale of any security, financial instrument, financial product service, or to be used in any way for evaluating the merits of participating in any transaction, and shall not constitute solicitation under any jurisdiction or to any person, such solicitation under such jurisdiction or to such person would unlawful.

---

Published as a conference paper at ICLR 2025
REFERENCES
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint
2022.
Souradip Chakraborty, Amrit Bedi, Alec Koppel, Huazheng Wang, Dinesh Manocha, Mengdi Wang, and Furong Huang. Parl: A unified framework for policy alignment in reinforcement learning. In The Twelfth International Conference on Learning Representations (ICLR), 2024a.
Souradip Chakraborty, Soumya Suvra Ghosal, Ming Yin, Dinesh Manocha, Mengdi Wang, Am- rit Singh Bedi, and Furong Huang. Transfer star: Principled decoding for Ilm alignment, 2024b.
URL https::tpp..o.. aaraiiiiiv org/ abs 2405 20495.
Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, Am- rit Singh Bedi, and Mengdi Wang. Maxmin-rlhf: Towards equitable alignment of large language models with diverse human preferences, 2024c.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01333, 2024.
Zhichen Dong, Zhanhui Zhou, Chao Yang, Jing Shao, and Yu Qiao. Attacks, defenses and evaluations for Ilm conversation safety: A survey, 2024. URL https: /ttp//axvvvv.///ee//eee/eeeeeeva wwwp:/i..orr/ab///////o 09283.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization, 2022.
URL https: http://rxvvv.ooo org/ abs 2200222.222222 10760.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-10866. PMLR, 2023.
Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, Nahyeon Ryu, and Mare Dymetman. Aligning language models with preferences through f-divergence minimization, 2023.
James Huang, Sailik Sengupta, Daniele Bonadiman, Yi-an Lai, Arshit Gupta, Nikolaos Pappas, Saab Mansour, Katrin Kirchoff, and Dan Roth. Deal: Decoding time aligmment for large language models. arXiv preprint 47X1v22402.06147. 2024.
Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Personalized large language model alignment via post-hoc parameter merging. arXiv preprint arXiv:2310. 2023.
Cheonsu Jeong. Domain-specializze I1m: Financial fine-tuning and utilization method using mistral 7b. Journal of Intelligence and Information Systems, 30(1):33-120 March 2024. ISSN 2288-4882. doi: 1..13088/jis.2204430.... 30.1 0933.
URL http: dx doi. www.oooogg...... 1wt......./////////... jiis 2024.
Lifeng Jin, Baolin Peng. Linfeng Song, Haitao Mi, Ye Tian, and Dong Yu. Collaborative decoding of critical tokens for boosting factuality of large language models, 2024. URL https:://axxii org/abs 2aabbs/2442 2402. 17982.
Maxim Khanov, Jirayu Burapacheep, and Yixuan Li. Args: Alignment reward-guided search, 2024.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannanch Hajishirzi. Rewardbench: Evaluating reward models for language modeling. 2024.
Xun Liang, Hanyu Wang, Yezhaohui Wang, Shichao Song, Jiawei Yang, Simin Niu, Hu, Dan Liu, Shunyu Yao, Feiyu Xiong, and Zhiyu Li. Controllable text generation for large language models: A survey, 2024. URL https: //arivivvvv./////////aa/////aavv wwwpa////r////////////////////// /200 12599.
Alisa Liu, Xiaochuang Han, Yizhong Wang, Yulia Tsvetkov, Yejin Choi, and Noah A. Smith. Tuning language models by proxy, 2024. URL https: http:///wwwww.////////////////// 08565.
12

---

Published as a conference paper at ICLR 2025
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière. Timo Schick, Jane Dwivedi-Yu, Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom. Augmented language models: survey, 2023. http::
//arxiv. org/ abs/ 2atbs///32.0022. 07842.
Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, Jilin Chen, Alex Beutel, and Ahmad Beirami. Controlled decoding from language models, 2024.
Refcchiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question- answering with human feedback. arXiv preprint arXiv:2112.09332 2021.
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko. Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Any Bleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elic Georges, Christian Gibson. Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-oppes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick. Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael I Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz. Andrey Mishchenko. Pamela Míshkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O' O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng. Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong. Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl. Benjamin Sokolowsky. Yang Song, Natalic Staudacher, Felipe Petroski Such, Natalic Summers, Ilya Sutskever, Jie Tang, Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley. Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong. Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojeiech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024.
Nikolas
13

---

Published as a conference paper at ICLR 2025
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.063477 2017.
Zejiang Shen, Hunter Lang, Bailin Wang, Yoon Kim, and David Sontag. Learning to decode collaboratively with multiple language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Pupers), pp. 19974-12900, Bangkok, Thailand, August 2024, Association for Computational Linguistics. doi: 10. 188633//1/2024.aaloooo/ https: www.aaannnntooooooooooooooooolll org/2024.00...
URL
2024 cecc--loon 701.
Nisan Stiennon, Long Ouyang. Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2022.
Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. A contrastive framework for neural text generation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Peter Vamplew, John Yearwood, Richard Dazeley, and Adam Berry. On the limitations of scalarisa- tion for multi-objecttive reinforcement learning of pareto fronts. In Australasian Conference on Artificial Intelligence, 2008. URL https: / /api. semant i wescchclararr....ooo org/ CorpusID: 382669 69620.
Peter Vamplew, Richard Dazeley, Cameron Foale, Sally Firmin, and Jane Mummery, Human-aligned artificial intelligence is a multiobjective problem. Ethics and Information Technology, 20:27 2018. URL https: : / api. semant icccholar.cccccco.coooocco org/ Corpus ID: 3696067.
20:27
Yuxia Wang, Minghan Wang, Muhammad Arslan Manzoor, Fei Liu, Georgi Georgiev, Rocktim Jyoti Das, and Preslav Nakov. Factuality of large language models in the year 2024, 2024. URL https: http///axvvv.//////e////e org/ 2402 02420.
Stanistaw Woćniak, Barttomiej Koptyra, Arkadiusz Janz, Przemystaw Kazienko, and Jan Kocon. Personalized large language models, 2024. URL htt.s: /ttp//axvvvv////
/ http:///////a////////aaa/////
22022 /tt.bb///2222222222 09269.
Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Madhushani Sehwag, Kaixuan Huang, Luxi He, Boyi Wei, Dacheng Li, Ying Sheng, et al. Sorry-bench: Systematically evaluating large language model safety refusal behaviors. arXiv preprint arXiv:2406. 14598, 2024.
Kevin Yang and Dan Klein. FUDGE: Controlled text generation with future discriminators. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard. Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3511-3535, Online, June 2021. Association for Computational Linguistics. doi: 2t.18853/v1//200 2021. nww.namainaaaaaaaa 6.
1t0.:///11/222mmmmmmmmammm. maincl-main.27.. URL https: : / acc anthology org/
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears, 2023.
Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling-7b: Improving Ilm helpfulness and harmlessness with rlaif, 2023.
Banghua Zhu, Michael I. Jordan, and Jiantao Jiao. Iterative data smoothing: Mitigating reward over- fitting and overoptimization in rihf, 2024. URL https:://// wrxiv.ov.ooo
www.arwwwwr....aaeaaa.e.reeorere org absgg/1b/244.... 13335.
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2020.
14

---

Published as a conference paper at ICLR 2025
A SOFTWARE AND HARDWARE
We run all experiments with Python 3.7.4 and PyTorch 1.9.0. For experimentation, we use two Nvidia RTX A6000 GPUs.
B NOTATIONS
In this section, we summarize the notations used in this work for quick reference.
Table 2: Notations. This table presents the notations we used for this work.
Implication
Notation
Vocabulary
V
Prompt
X
Response
y
Token in response
yt EOS
End of Sentence token
a concatenation b
[a, bj
Response till token t Reward function
Yer =
18/1 V2.
r rxx, y)
Token level state
8, xx y yct
Token level action
dx
Trajectory level LLM policy
p x)
Token level LLM policy
T[ (s))
Action value function corresponding to R under policy π
Q
Value function corresponding R under policy
C RELATED WORKS
With the increasing complexity and wide-spread of the generative models, is crucial to ensure that such models are well aligned with human preferences, social welfare and broader goals especially in critical decision making scenarios. Alignment via fine-tuning with reinforcement learning from human feedback (RLHF) has emerged as key paradigm for aligning foundation models (Ouyang al., 2022: Chakraborty et al., 2024a; Rafailov et al., 2023; Chakraborty al., 2024c; Chen et al., 2024). RLHF typically operates in two phases: first, a reward model is trained on human feedback, and then a policy fine-tuned with reinforcement learning (PPO (Schulman al., 2017)) using the trained reward model (Ouyang et al., 2022; Stiennon et al., 2022: Ziegler al., 2020; Yuan et al., 2023; Go et al., 2023; Vamplew et al., 2018; 2008). Direct preference optimization seeks to stabilize the training of preference alignment through reductions to supervised learning training (Rafailov et al., 2023). Although these training methods have proven effective in aligning generative models, they remain computationally demanding and assume white-box access to the model parameters which is not true in many industry applications. Specifically, fine-tuning billions of parameters for a new task either prohibitively costly or requires overriding enterprise constraints on model availability. This makes RL or supervised learning (DPO)-based fine-tuning for alignment unsuitable for obtaining reasonable performance on new preferences or tasks.
On the other hand, decoding-basee based methods (Mudgal et al., 2024; Chakraborty et al., 2024b) have emerged as an alternate way of alignment without fine-tuning the model parameters. Decoding operates by altering the distribution of the generated response to align to the target preference directly without updating the parameters of the LLM. The work by (Mudgal et al., 2024) is one of the first to integrate the alignment procedure directly into the decoding process, where they propose adjusting the generation probabilities at each decoding step based on feedback from reward model. (Huang et al., 2024) redefined the text-generation process search problem, with LLMs acting as search agents and they employ a heuristic- heuristic-guided search mechanism to generate responses based on a given prompt. The most recent research around Controlled and Principled decoding (CD, TQ*) formulates the decoding problem as a KL regularized Reinforcement learning problem and obtaining a closed-form solution with estimate of Q* for decoding.
15

---

Published as a conference paper at ICLR 2025
However, these prior methods all are predicated upon single reward model (or task specification), and thus cannot incorporate a diversity of constituent models and their associated knowledge bases. Hence, they mostly lack concrete way handle tasks significantly different from their training data. This motivates us to design an efficient strategy to optimally synthesize information contained in diversity of constituent LLMSs to align target preference via decoding.
D THEORETICAL RESULTS AND INSIGHTS
In this section, we characterize the sub-optimality of our proposed approach below. First, our algorithm's policy is defined by
Talg E arg max max J (Se:
(10)
where, Junge (Sc, 2) Vuiget (St, 2) a LKL(T, Tre( (re)(-(s))
From the definition above, we know that
Junge (St, 2 max Junge (St. 2)
(11)
Lemma 1. For any two policies Ty and Tj the difference between their Q-values at state-action
pair (Se, 2) can be bounded as:
Q, (SE, 2) 07' (st, - 8ig + 3KL(p" (-Ise, 2), Pref 2)) BKK 2), Pref( z))
where 8ig max, |r>([st, =), r) ry ([( z), and regularization coefficient.
Proof: We start by considering the difference between the Q-values for policies and xy state action pair 2))
O (st, 2) 2
(Sc. = (st, 2) Q5 2) + Q7 (St, 2) -))
(st,
(12)
where we first add and subtract the term Q71 (Se, z) and separately analyze the two terms Ti and T2.
T = Q (st. 27 (St, 2)
(13)
= Erupti (s).. [[st, (r: x], T] - (1 I max |r; (1 =], - r, ((se: z),
where the first term Ti can be upper-bounded by Ti S max, |ra([se, =), rg (lst, [|, 2), which denote as dig for simplicity. Next we proceed to bound the term T2 as:
To - (Se,
(14)
Er ( (b. (r,)),,(ss, 2), r) "s (r, 2).
+ SKL((p 73 (-1su, 2). Prer( 2)) (K (-|se: 2), Pretl Prer(-(se,
+ (KL(pp (-|st()( 2), Prer (s,, 2)) - 3KL(p (-1st, (-er()1st))
$ <BKL(p"* (- . 2), (rer(-1se,,)
(P= (-|se, .) 2), Peet( (st.
where, we first expand upon the definition of Q-function and add and subtract the KL-divergence of the individual policies ,"*,p">. Next, we leverage the optimality of the KI. regularized policy for the alignment objective. Since, we know Pj is the optimal policy for the beta KL regularized objective (Ouyang et al., 2022; Rafailov et al., 2023) under the reward func- tion ry (ist. 2], r)] ie Err (r, (r [rs( =].7)] 3KL (-Ise, 2), Piet( [st, 22) greater than Erap=i (- (-Is..>) [ry ([st, 2), -3)-3KK(( (-Ise, Prer( [st, =)) for all i. Thus leveraging this optimality condition, we obtain the desired result:
Q (St, 2) 0 (St, s 6ig + (KIL(p (-Ist, Par(- BBKL (-Ise, 2), Pret (Sal (st,
This completes the proof.
16

---

Published as a conference paper at ICLR 2025
D.1 PROOF oF THEOREM
As discussed, = { .2..... [ is the set of pre-trained policies, each aligned to latent reward function r; and aal be the policy obtained by the mixture of agents decoding strategy. Assume that the optimal policy for the target reward function "arrgr is x². Then, the sub-optimality of the mixture of agents decoding policy Walg with respect the optimal policy IT" is bounded by:
A
I mim d.j + (T) Tear(-1s)) + SKL(p" (-1s), Pre((ss)
(15)
jEK
where 04j = max, |'argger [st =1. r) (se, T|
Proof: We begin by defining the sub-optimality gap Q-function as
= Qarger [Sc. 2) Q Qungett (St, 2)
(16)
where this sub-optimality gap denotes the performance gap of our algorithm against the optimal policy for the target reward r Iarget-
Vinged (St: 2) Q Qurpee (St. 2)
(17)
Qurget (St, 2) 07 (St, 2) + 07,² (St, Q Qurger (St, 2)
A A2
where we add and subtract 27 (st, 2) and then separate the two components as A1, A2. We first derive upper-bound on A
A Quarger (St, 2) 075² (sc.
(18)
S dng SKL(p" (+|se, 2), Peet (s (5, 2)) BKL(p= (-st,, Ptr((((((),,
where, 6,j = max, Miarger ((st, 2), r) ry ([se: 2), r)). We obtained the bound by directly leveraging the result from Lemma for the reward functions TTarge((S =), (St, 2). Next, we proceed to bound the term A2 as
A₂ = Q7 (st, 2) - Qurga (St, 2)
(19)
0jtt (St, 2) Quig (St, 2) + o:K L(Talg (aaaatg((1 ( (s+)) a ( merr- (-er())))))
= 0 (st, 2) Stania (st, 2) aKLLL (Taig (st), ( (st))
s 0² (St, 2) - Qunged (St, 2) + a. I.KT(TT (TT (st), Trerl (st)) - a (ee((()
< min daj + aK L (TT (-1s), (-e((- aK (Tagg (11)) T-r((-
JEK
where, first we add and subtract the KI divergence terms a ( (st) and next utilizing the fact that (st 2) = ( know that for any j E K, ( 2) Junger (St, 2) and utilizing this we get the final inequality. Since it holds for any j. the final inequality holds true, where d.j = Now, combining the upper-bounds from A1, 1, A2, we get
(st.
a K L T (- - (st)). we get the third equation. Finally, we
A < min d.g + &&KL(T (s.)) (KL(P" (-|se), Pret(- jeK
(20)
where the above bounds express the sub-optimality of our proposed approach in terms of the difference between the target reward function and the reward function of the best model in our policy set for each token St and the sum of KL divergences w.r.t the reference policy the token and trajectory level. This completes the proof.
E ADDITIONAL EXPERIMENTAL DETAILS
E. REWARD NORMALIZATION
To provide clearer comparison of results, normalize the average rewards. For example: let l'AgentI represent the average reward achieved by the Agent-I model across all generated responses to the test prompts. The normalized reward, FAgem-L, is calculated as FAgentT
TAmcsI-PMIN where TMIN is the
Peaezs
minimum reward obtained by the model for given specific dataset. This ensures that the results are scaled relative to existing methods.
17

---

Published as a conference paper at ICLR 2025
E.2 SETUPS
Table 3: Summary of the datasets and model architectures used for experimental evaluations.
Target Task
Model Architectures
Setup
Dataset
Agent-1
Agent-11
Reward
Evaluation-1 1
Berkeley Nectar (Zhu et al., 2023)
Zephyr- 7B-a
Starling 7B-a
Mistral 7B-a-11
Evaluaiion22
Rerkeley Nectar (7ha et 2023)
Zephyr-7B-a
7ephyr-7B-PP
Mistral-B----I
Task1
Evaluation 3
Berkeley Nectar (Zhu et al., 2023)
Qwen Qwen 2-7B-DPO
Dolphin Qwen- 2:7B
Mistral-BBo--1
Evaluation-4
Berkeley Nectar (2ha et al., 2023)
Dolphin- (eaphin-...-Miieee 2.6-Mistall-7-DDPPP
Starling-7BB
Mistral-BB-o-11
Evaluation-5
HH-RLHH (Bai al., 2022)
7ephyr-7B-o
Starling-7B-..
Mistral-7B.
Task
Evaluation 6
III-LLHFF (Bai al., 2022)
Dolphin 2.6-Miral 7B-DO
Surling--7B 7B-a
Mistral-7B-o
Evaluation-7
HH-RLHF (Bai al., 2022)
7ephyr-Qwen-2-77 Q
Dolphin Qwen-2-7R
Mistral-BB-o
F MOTIVATION FOR GREEDY SWITCHING POLICY
Let [r. Mct-i] denote the state made up of the prompt and the sequentially generated tokens. Recall the L-Iength average target reward for a policy x given as
Vinger (s, a) E
Taarga a.) 8, = (-13a)
(21)
In our problem, do not have optimal policy for Trarget and we have do the best using collection of policies {T1, ... ,
Consider decoding just the last token. For give s7, need find single token (action) that maximizes the target reward. A best policy selection is given as
(-IsT) = arg max 1E TT
best
T'arget (ST, a ~ Ty (1(()))))
R1
Consider decoding the last two tokens. For given state 8T-1, best policy selection given as
(T-1 (-182-1) arg max ET
best
max ET R
Faarge (s, ~ a (T((1s))) 8--1
my ell
where E, (.) is the expectation over the next states.
Based on this define the Q function iteratively as follows:
@T(ST, (l)sT,ar)=a)) = max v, em [ET) ('urger a ~ Tz (+(()))))
For T 1,... ... O, we have
a1) max T
Turget (st, at ~ x(((-s))
(22)
3,
+ Bayj max Q (8141, 0141 7,((11411)
(23)
As seen from equation 22, finding the best policy at every step based on dynamic programming becomes infeasible for LLMs. Therefore, easily decodable greedy policy, the following crude proxy:
zell Turger St, a ~ E.y; max Q141 (((()-1)))
Q ai)
whence the approximate best policy selection now becomes,
x appoox. best (-(sx)
arg max Q (Sk, as (T((11s)).
Ell
18

---

Published as a conference paper at ICLR 2025
G EXAMPLES OF THE GENERATED TEXT OF OUR ALGORITHM COMPARED TO BASELINES
In this section, show examples of text generated using different decoding strategies:
Example-1
Prompt: 0 looks like O and looks like I. What does 2 look like?
The number can seen the letter B, the two horizontal lines are reminiscent of the upper and lower parts of the letter B, while the vertical line in the middle is reminiscent of the vertical line in the middle of the letter B. addition, some people might also see the number 2 as the letter E, as the two horizontal lines can be seen as the top and bottom of the letter E, while the vertical line the middle can be seen as the vertical line in the middle of the letter E. However, this interpretation is less common.
Agent-I:
Out of the given options, the that fits this sentence is older (option 2). Thus, the correct option is (2)
Agent-II:
2 looks like the numeral "2". In the context of binary code, where each digit represents binary value (0 1), the number 2 would be represented as "10" in binary. However, when discussing numerals or digits in general, 2 is represented as the symbol "2". In some contexts, the word "two" might be used to represent the number 2, but this less common when discussing numerals or digits.
BoN Sampling:
COLLAB (Ours): The specific representation of 2 can vary depending the context writing system. In the decimal numeral system, it is commonly represented as single character 2, which is horizontal line with two vertical strokes. In other systems, such the ancient Egyptian hieroglyphic system, 2 might be represented as the hieroglyph for "two" which is curved line with vertical line single character that looks like "T" rotated 90 degrees.
19

---

Published as a conference paper at ICLR 2025
Example-2
Prompt: NAME-I in Jordan is [blank] than Machu Picchu in Peru. (1) old (2) older(3) elder (4) eldest
The number 2 can be seen the letter B, as the two horizontal lines are reminiscent of the upper and lower parts of the letter B, while the vertical line the middle is reminiscent of the vertical line in the middle of the letter B. In addition, some people might also see the number 2 as the letter E, as the two horizontal lines can be seen as the top and bottom of the letter E, while the vertical line in the middle can be seen as the vertical line in the middle of the letter E. However, this interpretation is less common.
Agent-I:
Agent-Il: NAME-1 in Jordan is older than Machu Picchu in Peru. Explanation: The correct older describe the one that is more aged or has existed for longer period of time. In this case, NAME-1 in Jordan is being compared Machu Picchu in Peru, and since NAME-1 older, the correct term use older.
answer is (2) older. When comparing
things,
The correct answer is older because it is the most appropriate term to describe the age difference between the two sites. Nabataean city of Petra, also known as NAME-I, in Jordan is older than Machu Picchu in Peru. Petra, the ancient city of the Nabataeans, is located in present-day Jordan and is known for its rock-cut architecture and water conduit system. It was established in the 4th century BCE and reached its peak during the 1st century CE.
BoN Sampling:
COLLAB (Ours): older, Jerash, also known as the Pompeii the East or Gerasa an ancient city in Jordan. Although Jerash is older than Machu Picchu, the most accurate term compare their ages is older. Machu Picchu, a 5tth-century Inca citadel, was built in the mid-to-late 1400s. Jerash, on the other hand, is believed to have been founded around the 3rd century BC and has rich history that spans various civilizations.
Example-3
Prompt: 1.5 chickens lay 1.5 eggs in 1.5 days. How many eggs will 9 chickens lay in 9 days?
To find the total number of eggs laid by 9 chickens in 9 days, we need to multiply the number of eggs laid per chicken per day (1.5) by the number of chickens and the number of days: Eggs = (1.5 eggs per chicken per day) x (9 chickens) x (9 days) Eggs 135 eggs So, chickens will lay 135 eggs in days. The answer: 135 eggs.
Agent-I:
We need to determine how many eggs one chicken lays in one day. Since 1.5 chickens lay 1.5 eggs in 1.5 days, this means that in one day. one chicken will lay one egg. Therefore, we have nine chickens, they will lay total of 9 = eggs in one day. Since we want to know how many eggs the chickens will lay in nine days, simply multiply the number eggs laid in one day (9) by the number of days (9). This gives us a total of 9 9 81 eggs.
Agent-II:
COLLAR (Ours): We need to find the total number of eggs laid by 9 chickens in days, need to start by determining how many eggs one chicken lays in one day. From the given, 1.5 chickens lay 1.5 eggs in 1.5 days. This means that I chicken lays egg 1.5 days. To find the number of eggs one chicken lays in one day: Eggs per day = egg / 1.5 days = 2/3 eggs per day. Now, we can find how many eggs chickens lay in days: Eggs (2/3 eggs per chicken per day) x (9 chickens) x (9 days) Eggs = 54 eggs So, chickens will lay 54 eggs in days. The answer: eggs.
20

---

Published as a conference paper at ICLR 2025
Example-4
Human: I've been interested in trying out DMT or hallucinogenic drug with shaman, but I'm unclear if it's legal in the United States. What are my options?
Prompt:
Assistant:
Do you want information about the drug itself, or just about where to do it legally? Are you asking if you DMT yourself home, do you want to use it in special location (such as a 'shamanic retreat')? There are retreats in South America that offer ayahuasca and other hallucinogens. These often led by shamans, and are considered spiritual experience.
Agent-I (Helpful):
Agent-II (Harmless):
Unfortunately, can't provide you with information about specific locations or retreats. However, I can tell you that the use of hallucinogenic drugs is generally not legal in the united states, and the use of dmt is specifically prohibited under federal law. If you're interested exploring spirituality or personal growth, there are many other legal and safe ways to do so. You might consider speaking with therapist or counselor or exploring meditation or other mindfulness practices.
COLLAB (Ours): Yes, that's correct, you should be aware that the use of hallucinogenic drugs not legal in the United States, and you could be arrested and charged with crime you caught using or possessing them. I'm sorry to say that there are no legal options for using hallucinogenic drugs in the United States.
21

---

Published as a conference paper at ICLR 2025
Example-5
We're going to talk about CNS fatigue, central nervous system fatigue, which occurs with repeated maximal lifting leading to short-term regression. What is CNS fatigue and how it be managed?
Prompt:
Agent-I:
The phenomenon of CNS fatigue, also known as central nervous system fatigue, is commonly discussed topic in the world of weightlifting. This occurs when person repeatedly performs heavy, maximal lifts, leading short-term regression in strength. This observation has been seen in various strength sports for a long time, and it's the reason why experts advise against overtraining and suggest deloads or wave schemes to push heavy attempts apart. The time between the last heavy attempt and contest day tends to be longer, and this is due to the fact that the body needs time to recover and rebuild
Agent-Il; II: So, here's the thing: CNS fatigue is real thing, but it's also a bit catch-all term that gets thrown around lot, especially in the fitness community. CNS stands for Central Nervous System, and it's responsible for sending signals from your brain to your muscles, telling them contract and lift weights. When you're lifting heavy, your CNS is working overtime to send those signals, and over time, it can become fatigued. This is especially true if you're lifting heavy weights every single day, without giving your body adequate time to recover. This fatigue can lead to temporary decline in performance, as the CNS needs time recover and restore its efficiency. This is why strength training programs often incorporate periods of lower intensity or rest, known as deloads, to allow for recovery and prevent overtraining. By understanding and respecting the limits of your CNS, you can optimize your training and avoid the negative effects of overtraining.
COLLAB (Ours): The phenomenon of CNS fatigue, also known central nervous system fatigue, is a commonly discussed topic in the world of weightlifting. This occurs when the body experiences short-term regression after performing maximal workouts for several weeks in row. This is due to the fact that training at such high intensities can lead to decrease in performance over time. The concept of CNS fatigue has been observed in various strength sports for a long time. It's not sustainable to push the body to its limits every week, which is why many athletes incorporate deloads or wave schemes into their training. These methods allow for recovery and help prevent overtraining. By understanding and respecting the limits of your CNS, you can optimize your training and avoid the negative effects of overtraining
22


Reviews:
[{"review_id": "VqzD7CNiFs", "reviewer": "Reviewer_d8Dt", "summary": "This work proposes to extend decoding / inference time alignment using a mixture-of-experts based decoding strategy. For this, the authors make use of existing LLMs as policy experts. The decoding method proposed by the authors uses a token level selection strategy where an LLM is selected from the experts / agents for each token. This selection is performed using a utility metric which is based on the implicit Q function. The authors also provide a theoretical justification of their approach and support it using empirical results.", "questions": "* What is the distribution of the selection of agents.\n* How much is the compute difference between doing RLHF + single decoding vs mixture of decoding?\n* Line 97-99 seems repeated.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* Dont need to update the billions of parameters that RLHF needs.\n* Their approach can make use of specialized models which are experts in a subset of capabilities that are desired.\n* Current solutions to mixing multiple experts relies on either tuning a model, or explicit formulas, and rely on expert demonstrations. This work dynamically merges models \n* This approach is especially beneficial in settings where reward models / policy parameters of models are not available readily thus impacting a lot of industry applications.\n* The theoretical justification and details are sound.\n* The paper is generally well written and easy to follow.\n* The evaluations are diverse and across 7 different setups, however, only 2 datasets: Berkeley Nectar and HH-RLHF.", "weaknesses": "* It is unclear what is the reference policy used in the KL terms that is used to obtain the objective of each policy (J). \n* \"The sub-optimality gap will be lower when 1) the best agent’s reward function is close to the target reward function, and 2) when the regularization terms are properly controlled so that both the reference policy and the optimal policy are close\" -> The choice of the reference policy seems to be important.\n* I would have liked to see analysis on compute requirements.\n* Minor point: There is no human evaluation done in this work. Evaluating using GPT4 should not be used to claim alignment with humans.\n* The main usage of this work seems to be in using domain expert policies that are good in different aspects / tasks. It is unclear how the chosen policies are experts in different areas that are evaluated. For this, I would suggest using evaluation on harder tasks maybe like reasoning / coding datasets (alpaca eval, arena hard, mt bench).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Add4saZEHj", "reviewer": "Reviewer_4fXt", "summary": "This paper proposes Controlled Decoding using Mixture of Agents for LLM Alignment, a method that achieves alignment at inference time without retraining the policy model. However, the comparison with classical alignment methods is insufficient and some critical experimental details are unclear.", "questions": "please see weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The approach of achieving alignment without retraining the policy model is innovative and could significantly reduce computational costs.\n\n2. The concept of using a mixture of agents for decoding provides a fresh perspective on controlling language model outputs.", "weaknesses": "1. Is the implicit Q-function an additional trained language model? How does its cost compare to methods based on DPO, PPO, and RLHF?\n\n2. Can the implicit Q-function directly guide a policy model in decoding? Is it necessary to use multiple agents collaboratively for decoding?\n\n3. How were the agents initialized in the experiments? Were these agents explicitly trained for alignment?\n\n4. The experimental section lacks a comparison with alignment algorithms based on DPO and PPO.\n\n5. The authors should consider more complex, objective tasks such as those involving reasoning and math to avoid potential biases from the reward model and GPT-4 evaluations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WG12w9G04S", "reviewer": "Reviewer_4LCR", "summary": "The paper proposes a method to elaborate multiple LLMs at the time of inference to combine the strengths of the models. For each token generation, the most promising policy to generate the rest of the sequence is selected. The method is evaluated on two generic alignment tasks using multiple generic LLMs, outperforming single-LLM decoding algorithms.", "questions": "- What is the computational complexity and the walltime of the algorithm in the experiment? I understand that the walltime depends on the hardware and the system but would be a good reference to understand the effectiveness.\n\n- Which algorithm does the SoTA decoding strategy refer to? Is it Transfer Q* (Souradip Chakraborty et a l. 2024)? Or is it controlled decoding (Sidharth Mudgal et al. 2024), which is called SoTA in Chakraborty et al.? A paper should use a term that uniquely identifies the subject rather than a term that changes over time.\n\n- (p.7 Example) Why can COLLAB correctly answer the question when both Agent-1 and 2 fail to answer the question? Is there a reason COLLAB decoding can acquire an ability neither of the Agents have?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- How to elaborate multiple LLMs is an important research question. Given that we have little knowledge on how to investigate the strengths and weaknesses of the LLMs (as of now), ensemble methods should be useful.\n- The method is simple and intuitive.\n- The theoretical result is nice to have, yet its practical implication seems not immediate to me.", "weaknesses": "> (p.10 l.536) Empirical evaluations demonstrate its superiority over traditional single-agent decoding baselines, providing a robust and computationally efficient method for model alignment in complex, real-world scenarios\n\n- I failed to understand the computational cost of the method. My understanding is that it requires the inference of the whole sequence for each policy, per each token. So the computational cost of generation a sequence of length N would be O(N^2) times of query to LLMs. It would be nice to have a walltime of the algorithm compared with BoN sampling and Agent-1 and 2.\n\n- Although the strength of the method is claimed to be able to adapt to a diverse set of tasks, the experiments are not designed to evaluate in such a scenario. It would be beneficial to evaluate the method for each subtask rather than showing the aggregated result. For example, how does the model perform in the Harmlessness and Helpfulness subsets in HH-RLHF datasets? What kinds of tasks benefit from the ensemble? It would be helpful if we have a post-hoc analysis of what makes the proposed method better than a single-LLM method in the empirical scenario.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FZhVg8VZYH", "reviewer": "Reviewer_Q9zn", "summary": "The paper presents the idea of switching between different LLMs during the decoding process in order to maximize the reward of the generated response. To find which policy should be used at each generation step, they define the problem as KL-constrained RL and sample a token greedily with respect to a Q function (regularized by the KL constraint).", "questions": "Control Decoding and other similar works [1] use $Q^{\\pi_{ref}}$ to augment the decoding. This is not an approximation of $Q^*$ as mentioned in lines 260-261 of the paper, but an alternative solution to the KL-constrained RL problem, known as RWR [2]. The advantage of using $Q^{\\pi_{ref}}$ is that it can be easily learned using trajectories from $\\pi_{ref}$. \nCollab, on the other hand, uses $Q^{\\pi_i}$ as an approximation to $Q^*$. Does the author have an explanation (either empirical or theoretical) as to why this is better than just using $Q^{\\pi_{ref}}$? This is an important design choice in the algorithm that I feel hasn’t been discussed enough.\n\nIn the related work section, I’m missing a discussion about the connections to hierarchical RL. Collab is very close to it, as it learns a policy (parametrized as Q function) to choose which one of several policies to sample from in each state.\n\nI find some of the metrics used in the experiment section are not related to the motivation behind the algorithm. The idea, as presented in the paper, is to use models trained on different reward functions to perform decoding that will maximize a new one. In the experiment section, however, you evaluate non-reward metrics like win rate, diversity, and coherence. Let’s take win rate, for example. Is your claim that the new reward function correlates with the win rate better than the ones that the original models were trained for? And therefore, a higher win rate is equivalent to a higher reward? If so, where is the evidence?\n\n[1] Han, Seungwook, et al. \"Value Augmented Sampling for Language Model Alignment and Personalization.\" arXiv preprint arXiv:2405.06639 (2024).\n[2] Peters, Jan, and Stefan Schaal. \"Reinforcement learning by reward-weighted regression for operational space control.\" Proceedings of the 24th international conference on Machine learning. 2007.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 4, "strengths": "- The idea of using a Q function to perform routing between different models is novel. Moreover, doing collaborative decoding to maximize reward is an interesting problem that does not have a lot of literature about it.\n- The empirical results seem strong, with the proposed algorithm outperforming the baselines.", "weaknesses": "The paper is hard to read and understand; in addition, some important details are not discussed at all. Some examples:\n- There is no discussion about how the Q functions are trained, although this is a crucial part of the algorithm.\n- The only description of the experiments (models, rewards, etc.) is in the appendix. It is also unclear why the author chose these specific experiments.\n- The authors explain their method as an extension of CD, which proposes sampling according to equation 3. However, in practice they sample according to equation 5. This is not the same, as the probability under pi_ref is not taken into consideration in equation 5. \n- Line 428 hint that you are doing top-K but Algorithm 1 (line 330) talks about top-p, which one is true? \n\nI believe that the technical work presented in the paper is good, but the writing needs to be improved substantially.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)