Aligning Compound AI Systems via System-level DPO
Authors: Xiangwen Wang
Yibo Jacky Zhang
Zhoujie Ding
Katherine Tsai
Haolun Wu
Sanmi Koyejo
Venue: NeurIPS 2025 poster ()
Keywords: alignment
compound AI system
preference learning
DPO
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: Compound AI systems, comprising multiple interacting components such as LLMs, foundation models, and external tools, have demonstrated remarkable improvements compared to single models in various tasks. To ensure their effective deployment in real-world applications, aligning these systems with human preferences is crucial. However, aligning the compound system via policy optimization, unlike the alignment of a single model, is challenging for two main reasons: (i) non-differentiable interactions between components make end-to-end gradient-based optimization method inapplicable, and (ii) system-level preferences cannot be directly transformed into component-level preferences. To address these challenges, we first formulate compound AI systems as Directed Acyclic Graphs (DAGs), explicitly modeling both component interactions and the associated data flows. Building on this formulation, we introduce SysDPO, a framework that extends Direct Preference Optimization (DPO) to enable joint system-level alignment. 
We propose two variants, SysDPO-Direct and SysDPO-Sampling, tailored for scenarios depending on whether we construct a system-specific preference dataset. We empirically demonstrate the effectiveness of our approach across two applications: the joint alignment of a language model and a diffusion model, and the joint alignment of an LLM collaboration system.

Aligning Compound AI Systems via System-level DPO
Xiangwen Wang 1.2+
Yibo Jacky Zhang
Zhoujie Ding'
Katherine Tsai
Haolun Wu!!
Sanmi Koyejo'
'Stanford University University of Science and Technology of China Mila Quebec Al Institute
xiangven@stanford. edu,, yiboz&stanford edu, dingdstanford edu tsaikl@stanf ood edu, haolunwu@cs stanford edu, sanmi Acs. stanford. edu
Abstract
Compound AI systems, comprising multiple interacting components such LLMs, foundation models, and external tools, have demonstrated remarkable improve- ments compared to single models in various tasks. To ensure their effective deploy- ment real-world applications, aligning these systems with human preferences is crucial. However, aligning the compound system via policy optimization, unlike the alignment of a single model, is challenging for two main reasons: (i) non- differentiable interactions between components make end-to-end gradient-based optimization methods inapplicable, and (ii) system-level preferences cannot be di- rectly transformed into component-leve preferences. To address these challenges, we first formulate compound AI systems as Directed Acyclic Graphs (DAGs), explicitly modeling both component interactions and the associated data flows. Building on this formulation, we introduce SysDPO, a framework that extends Direct Preference Optimization (DPO) enable joint system-leve alignment. We propose two variants, SysDPO-Direct and SysDPO-Sampling, tailored for sce- narios depending on whether we construct system -pecif- preference dataset. We empirically demonstrate the effectiveness of our approach across two applica- tions: the joint alignment of a language model and a diffusion model, and the joint alignment of an LLM collaboration system.
Introduction
Compound AI systems, which consist of multiple interacting AI components, serve as promising frameworks push beyond model capabilities and achieve state-of-the-ard performance 40 13 [6]
For example, ChatGPT integrates a Large Language Model (LLM), a DALL-E image generator, a web browser plugin, and various other system components support diverse user needs M. A multi- agent system consisting of multiple LLMs working collaboratively achieves improved performance compared a single agent (36). A Retrieval-Augmented Generation (RAG) system combines large language models with information retrieval capabilities and is capable of answering time-sensitive queries. multi-LLM routing system includes router that dynamically selects among diverse set of models according to user queries to maximize model performance [1i]. These examples illustrate how compound AI systems leverage LLMs alongside complementary modules to tackle complex tasks beyond the reach of single LLM.
Ensuring effective collaboration among components is crucial for compound AI systems function reliably. It also plays a critical role in aligning the outputs of the system with human preferences and in ensuring safety and ethical standards (16). However, simply integrating multiple models does not guarantee effective coordination, as illustrated by a failure case involving an LLM (GPT-4) and
"Equal contribution
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Calm Cat
Slighdy Inritatdd Cat (a)
Very Angry Cat
Slightly Aznoyed Cat
Angry Cat (b)
Furious Cat
Figure 1: The figure illustrates the challenges in compound system composed of the GPT-4 and the image gen- erator DALL-E. Given user prompt to GPT-4, "Generate three separate images
cat being progressively angrier", , the task is to demonstrate a clear visual progression of the specified attribute, ie.. anger. (a) shows the results from one query. and (b) represents the results from another query. The captions under each image summarize the prompts generated by GPT-4 for DALL,-F (complete prompts in Appendix [A], where prompts from both queries reflect progressions in anger. Similarly, DALL-E accurately generates the images following the given prompts. However, (a) fails to demonstrate clear visual progression of anger compared to (b), highlighting GPT-4's inconsistent collaboration with DALL-E.
diffusion model (DALL-E), shown in Figure I Our experiments (Section reveal that instructed tumed Llama-3-8B combined with Stable Diffusion XL achieve a correctness rate of only 32% on similar tasks. These failure cases highlight the critical need to develop compound Al systems.
framework align
44 3 aligning compound While alignment techniques for monolithic models are well-studied 26
systems remains an open challenge. Standard methods such as Direct Preference Optimization (DPO) [6] and Reinforcement Learning from Human Feedback (RLHF) [19 3 are not directly applicable to compound systems for three primary bottlenecks: (i) Non-differentiable interactions: Components in a compound system often interact through non-differentiable channels, such natural language, preventing end-to-end optimization and making credit assignment across components difficult. (ii) Non- decomposable preferences: Aligning each component independently is inadequate, as sestem-level preferences are not simply decomposable into individual preferences. Moreover, effective coordination between components is essential but cannot be captured through isolated alignment. (iii) Lack of tite-graiied benchmarks: Most alignment benchmarks are constructed to evaluate the entire system; benchmarks for individual sub-tasks might not exist.
In light of these challenges, there is an urgent need develop methodologies for aligning com- pound AI systems. While recent studies investigate prompting techniques and instruction tuning approaches [39 [6 30), and concurrent work (37 explores optimization using local reward models, we take alternative approach. Our main contributions are summarized below:
We model compound Al systems as Directed Acyclic Graphs and propose SysDPO. Direct Preference Optimization (DPO)- based alignment framework with two variants- SysDPO-Diress and SysDPO-Sampling for settings with or without system-specitn datasets (Section 2
We provide a theoretical analysis showing that SysDPO achieves 3-perfect alignment in the population setting, generalizing standard DPO guarantees to compound systems (Section
We demonstrate SysDPO with two applications: aligning an LLM and a text-to-image diffusion model, well aligning two LLMs (Section 4. Our experimental results indicate that aligning compound AI systems increases the success rate in handling complex instructions (Section 5)
These results deepen understanding of alignment challenges in compound Al systems and provide a foundation for future research.
2 The SysDPO Framework
In this section, we introduce the framework of SysDPO through intuitive motivations, where the theoretical justification is provided later in Section 3 We start by reviewing prior work on the Bradley-Terry model and DPO 26 then move to introduce the SysDPO pipeline.
Bradley-Terry (BT) model. Given an input v, the system generates two pairs of outputs 2, 2' We represent the preference of the outputs as (z > z' x)) if preferred over z' by a preference oracle, e.g., human labelers. To model the preference, we use the Bradley-Terry model common

---

preference model used in alignment 26 31 which represents the preference distribution as
(x,(r"(z,2)
pref(z > z' x)) exp(r + exp(r"(x,=") E (0, 1),
(1)
where (xx 2) the ground truth reward model. Drawing preference from the preference oracle, the winning sample is assigned to n 2 and the losing sample is assigned to 2! <- z' with probability p= > z' r)). Using the preference oracle, one can construct a preference distribution 'D composed of preference pairs (z, 2m" 2
Direct Preference Optimization. DPO aligns the model using the preference distribution DD by minimizing the following loss:
I((0= = (a log (Blog po(="1a) B log po(2"1x=)
Po (c)) (0 [x))
(2)
where 0 denotes the reference model of 0, c(.) stands for the sigmoid function. When it comes to compound AI systems, the optimization challenge arises from the structure of 0, which represents collection of model parameters. These models may communicate in complex ways that are non- differentiable, c.g., by exchanging plain text or task-specific outputs. Moreover, for a compound system with intermediate generations y. the system's 's generation probability takes the form of an integral (o(2 z)) = po(2, y x)) dy, raising challenges for optimization. Therefore, the alignment of compound AI systems is an important yet difficult task.
2.1 SysDPO Framework
To circumvent the challenge of non-diiferentiabblity we develop SysDPO. We start by modeling the structure of compound AI systems as DAGs, which encode both the connections between models and the underlying data flow from input to final output via intermediate results. The DAG structure enables us to decompose the joint probability of generated outputs into several components. We study two decomposition methods, depending on whether the intermediate outputs are observable or not. Such decompositions lead to two variations of SysDPO: BxsDPO-Direed and Both methods address the non- diffferrentiabbiiiy and optimization issues. We then define DPO-based loss function that can be optimized from end-to-end simply via gradient descent. This ensures that the outputs of each component is aligned with human preferences.
Formulating Compound AI Systems DAGs. We model compound AI system as Directed Acyclic Graph (DAG), where nodes represent variables and edges capture the flow of information between components. Specifically, we define nodes as x, fyshe and {zy]ieJ, where a X is the input, Y/ E Y/ are intermediate outputs, and 2j E Z, are final outputs. Each non-input node is generated by a single model that consumes inputs from its parent nodes. We denote the set of all generated outputs by {ıi, ²j heh.ee.... The directed edges represent the flow of the generated data between components.
We illustrate this formulation with two examples. The first, shown in Figure 2 (a), involves an LLM generating image captions, followed by diffusion model that synthesizes coreess--rrrssondding to the motivating example in Figure [ The second example, shown in Figure (b). consists of two LLMs collaborating in multi-stage pipeline. This setup reflects recent interest LLM collaboration in improving reasoning, factuality, safety, and creativity through mechanisms such as verification,
debate, or response refinement 42 36 38).
2.2 SysDPO-Direct
The DAG structure encodes the conditional independence of the generated data 20 allowing the decomposition of the probability of generated data into multiple terms. Assume that all intermediate outputs {ye / are observed or given the preference dataset. We can factorize the probability of an associated DAG as
Pa (s|x) = II po, (yal Pa(y))) pa, (a(2)))
(3)
VETJEJ
where Pa(-) returns the parent nodes of given node in the graph, and 0 (0, TUT] denotes the parameter set of generative models in the compound Al system. This decomposition breaks down the likelihood of the system into product of multiple terms, where each term contains
3

---

0
8
21
0
02
x
0g
Y3
23
LLM Diffusion Model
(b) LLM LLM
Figure 2: Corresponding DAGs of compound AT systems. (a) The gives prompt which is processed by the LLM 0, to produce three captions y1, y2, Ma- The diffusion model 02 is queried to generate images z; given yı
1, ......,. ,,,. (b) user gives prompt x which is processed by first LLM 01 to produce an intermediate result y. Then, æ and are passed to the second model 02 to generate the final output z.
single model. Note that the labels of the models 0,, f can refer to different queries the same model. Taking the case of Figure 2 (a) as example, and denoting the set of generated contents by {y/1 V2, Vs 21, 22, 23), we have p(s I Pe, (y/(z) Po, (24 11)
Preference Dataset Construction. To learn the model parameters 0 {0, k IUIUU first construct a system-specific preference dataset. We assume that all the variables in (3) are observed. The dataset can be constructed in the following way: given generates two variants of the final outputs as well as intermediate outputs in the system. We label the preferred sample s {y."
SUUU), J}.
input x, the system
I} U {2) 1 J}, and the non -oreferrreed sample as s' = (vl | i 1 {2) j E J}. Consider the case of Figure (a) as an example, a preferred sample is in the form of 5 {yT 1/ .33 ; ; w }. Putting everything together, each preference data pair composed of (x, s*). This construction process is tightly coupled with the structure of the underlying system. Different compound system architectures may involve different sets of intermediate variables, and thus require generating distinct preference datasets
Loss Function Design. Given the dataset D composed of preference pairs (z, s" s') and com- pound AT system formulated DAG, we can apply the decomposition of to the DPO loss (
(a())())) Po (p(s5(a)))
Pa (=(
I Direet (0) [e,,," ,') log
gllg
B log
pa
where denotes the collection of reference models, a-.) stands for the sigmoid function. We can then find the optimal 0 by simply optimizing the loss function resulting in end-to-end optimization.
2.3 SysDPO-Sampling
SysDPO-Direct requires system-specific datasets with observations of the intermediate outputs, whereas most existing preference datasets are composed of only inputs and ranked outputs. One redress is to reversely sample the intermediate outputs given input and final output to construct a semi-synthetic dataset. However, such practice is costly and the samples might be of low quality. To this end, we introduce variation of SysDPO, termed SysDPO- SysDPO-Sampling The key difference between SysDPO-Sampling and SysDPO-Direct lies in how the probability pe p=(2(z) is decomposed.
The key idea of SysDPO- Sampling is as follows. Recall that s == {yr 2, hie.jeJ is the set of all the variables generated by the system. Denote the collection of intermediate samples as y == {ı Jeer, where y E y. Thus, s {z, [zy]eeJ Ug. Assuming discrete sample space (e.g., discrete tokens), by the law of total probability:
po({zzbieu x)= [e)==pon
po = Σ II
po. (ydl Pa(y:)) po, Pa(z)))
(5)
NEY
VEY (EIJEJ
However, the summation of the right-hand side of (5) is generally intractable, as it requires summing over possible sentences y,. While one method use Monte Carlo sampling to approximate expectation, it is inefficient if the sample space is large. To efficiently approximate the summation, we focus only on small number of highly probable, distinct samples yf" given that the less probable samples contribute little to the summation. Therefore, having the highly probable distinct samples

---

(v hi.. we make the following approximation.
po(1z))olz)
Σ
II po, (o po, (zyl Pa(zz))- VEIjEJJ
(6)
To generate the samples indexed by a, we employ Diverse Beam Search (DBS): an extension of standard beam search that enforces diversity across the beam groups by adding penalty to candidates similar to those already selected within group partitions [34.. During training, these intermediate samples {y, J. are regenerated after each model update step, while SysDPO-Direct always trained on a fixed dataset. Plug this into the original DPO loss ( and arrive at the loss function Sampling B Note that, in this case, end-to-end gradient- -base optimization is feasible.
as shown in Appendix
3 A Theoretical Analysis on Compound AI System Alignment
We ask a fundamental question: Does SysDPO lead to the correct alignment? To answer this question, we first define the concept of 8-perfect alignment and show that, in the population setting, both regular DPO and RLHF lead to perfect alignment. Then, we prove that SysDPO similarly achieves 9-perfect alignment on compound Al systems.
3.1 Perfect Alignment under Bradley-Terry Preference Model
Our theoretical analysis begins with defining the perfect alignment with respect the preference oracle (
Definition (3-Perfect Alignment). Define a probabilistic generative model 0* : + Z associated
with generative distribution pa-. We say 0 is 8-perfectly aligned with parameter B > 0 to the preference oracle pref(-) if a' E 2, x EE::
pref 2'
| [x)
po- p0- |x)) 3 pa- (z x²)
pref > n x))
An equivalent formulation is.
pg- (2 |x)
pref( ~ > 2 x))
pa- (2
x + pg- (z) x)
This definition has several interpretations. First, for all O, the above definition satisfies the order consistency of the generative model 6* and the preference oracle pref, i.e., pref(>" > 2 2) > pref(2' > 2" x) if and only if po- (2 xx) > pa- (2* x). Then, with varying 8, we can see that acts temperature parameter controlling how peaked the aligned policy must be match a given preference signal. Writing
22 > x) (1+
Po- (=' Pp- (5 1
one sees that smaller 33 forces larger likelihood ratios represent the same preference score from the oracle (thus a more deterministic policy). while larger 3 makes the required ratios closer to 1 (thus a near-uniform policy). When the policy is viewed as Boltzmann rational pe- (z x) exp(r* (x,(r"(x, 2)/
with Bradley-Terrs preferences (r > z! x)) = exp(r" (æ, 2²)) exp((" (x0 ())
exp(r* (z, 2)))) (e.g., see [12), then Definition holds exactly; in this view. is inverse ra- tionality parameter. Alternatively, if we view the generative model pa- as a choice model, then recovers Luce's 's choice axiom (independence from irrelevant alternatives) [17 when
Moreover, the notion of perfect alignment has been implicitly used in prior work [26]. In the following, show that given the preference oracle, we can find a 8-perfectly aligned model by optimizing the DPO objective (2), or equivalently the following RLHF objective:
max
(z, - 33 (po pε).
where D, is distribution over X; θ is reference model: and 3 is the strength of KL regularization.
5

---

Proposition 1. Suppose an optimal model 8 E ④ achieves the maximum of the RLHF objective ( or the minimum of the DPO loss function (2) where data preference distribution D given by the preference oracle (. Then, it follows that 0 is 3-perfectly aligned with the preference oracle when the output of the reference model 0 follows uniform distribution given any input E X.
The implication of the proposition is follows. In ideal case where we have infinite ground truth preference data, there is complete information about perfect alignment. In such cases, the alignment objective should not rely on reference model, which is why RLHF and DPO achieve perfect alignment when the reference model is simply a uniform distribution. However, the reference model has proven be important in practice, where we only have insufficient preference data.
In the following, we investigate the proposed heuristics, SysDPO-Direct and which also achieve perfect alignment.
3.2 SysDPO Achieves Perfect Alignment
SysDPO-Direct works by simply replacing the final output =: {zj) in the original DPO loss function by the set of all generated variables 8 == {ur U É S. Then, the corresponding preference oracle for two sets is inherited from the original preference oracle follows.
prefays(s > st x)) := pref(2" > 2 x))
In the following theorem, we show that the generative system 8% aligned by DasDPO-Direde is indeed 8-perfect given the following technical assumption. Intuitively, the assumption demands diversity in the training distribution 'D used in SysDPO.
Assumption 1. Any & E y x Z has a positive probability to be sampled from D.
Theorem 1. Under Assumption 7 suppose an optimal model fr E Q achieves the minimum of the SysDPO loss function ( where the preference for data is given by the preference oracle prefeys- Then, 0g is 3-perfectly aligned with the preference oracle pref(-) (-() when the reference model follows a uniform distribution.
Assumption Thighlights Diighhgt fundamental challenge of compound AI system alignment. In compound Al systems, intermediate outputs are typically hidden from the user, so preferences can only be given over the final outputs. Since the evaluation of a component in compound system must depend on other components, there is no preference oracle to compare two intermediate outputs directly. Thus, SysDPO-Direct may benefit from training dataset of diverse intermediate outputs.
In cases where the preference dataset lacks intermediate results, we propose an alternative variant It aims to directly optimize an approximated standard DPO loss function with respect to (=(2z) through sampling. Therefore, at the population level where an infinite number of samples can be drawn, Proposition]] implies optimality of SysDPO-Samplin We further discuss the finite-sample setting through the lens of coresets in Appendix E suggesting that SysDPO-Sampling may also benefit from diverse sampling scheme.
of SysDPO, i.e.,
4 Applications
In this section, we discuss two examples of compound systems corresponding to Figure 2 We demonstrate how SysDPO-Direct and SySDPO-Sampii can be applied to these examples.
4.1 An LLM and Diffusion Model
We apply SysDPO-Direct to the example in Figure ! which involves an LLM 1b and a diffusion model é. Given an input prompt z provided to the system, the LLM generates an intermediate output y. which can be parsed into multiple captions a prompt for the diffusion model. The diffusion model is then queried n times, generating images 21. ....... as the final outputs. Similarly to Figure [ the generated images are expected to follow a logical relationship. This demands that both the language model and the diffusion model not only recognize their roles in the overall task but also execute them accurately and coordinate effectively to ensure coherent system behavior. such, this setting serves as a strong testbed for evaluating our proposed method.
(V1, 3/2 ... Bn). Each yı. i = 1,..., n serves as

---

This multi-step process is modeled as a DAG whose special case (n = 3) is shown in Figure (a). where we can decompose the generation process as p(s[z) Pv (y(zz) II Pa (2. We start by applying the probability decomposition to the SysDPO-Direct loss function 4.. However, although
the L.LM's generation likelihood Pe (y/ .r) is accessible, the diffusion model's (=(y) not. We
address this challenge by extending [35] to accommodate our setting and obtain an upper bound of
the SysDPO loss function. We then optimize this upper bound to align the system. Detailed derivation is elaborated in the Appendix F
4.2 Compound LLM Collaboration System
We also explore systems formed purely by the collaboration of language models. In such systems, multiple LLMs cooperate to complete complex task. Specifically, we study a two-stage question- answering system, where a user poses a question as input x; the first-stage model 1/1 generates an intermediate answer y, and the second-stage model 1/2 refines to produce the final output z. This setup serves as a simple yet representative compound Al system to demonstrate how multiple L.MSs can collaborate toward shared objective. The overall generation process of the system can be formalized as:
p(s ()) = Péi (y a)) P (= I.,yy,,
where s = {y. 2} represents the intermediate and final outputs. Since both e² and 1/2 are language models with tractable likelihoods, we can apply either SysDPO-Sampling or SysDPO-Direct to align the system. SysDPO-Direct requires a preference dataset that includes both the intermediate prompt (i.e., the output of v/1) and the final output, which can be constructed using ground-truth reward model. In the absence of this constructed preference dataset, SysDPO-Sampling applies by only requiring ready-made preference dataset that contains only input and final response pairs.
Experiments
5.1
Compound AI System of a LLM and a Diffusion Model
This section evaluates the effectiveness of DusDPO-Direct for aligning compound AI systems. Motivated by the example in Figure we synthesize preference dataset and experiment on the joint alignment of an LLM and diffusion model. Examples of synthetic data are shown in the Appendix [] Our evaluation focuses on the coherence of the generated image sequences and their alignment with coherent system-level preferences.
Dataset Construction. We construct a custom dataset in three steps. First, we use the regressor from Zhuang et al. [43] to assign scores in [0, 1] to images based on 40 scene-related attributes (e.g., brightness, coldness, and boring). These attributes cover wide range of visual concepts and provide general, high-level labels that reflect diverse aspects of scene semantics. Second, for each attribute, GPT-4 used generate 250 user prompts instructing the system to produce image sequences with progressive changes. To increase prompt diversity, we adopt four prompt styles from Qin et al. (24]. Details are provided in Appendix H Third, for each prompt, four sequences are generated and ranked using the Preference Score in Eq. (8) Six comparison pairs constructed, with the higher scoring sequence in each pair labeled as preferred, resulting in 6,000 total comparisons.
Preference Score. To compare the generated image sequences, we define a preference score q that evaluates both the order consistency and the uniformity of the distribution. This metric is based on the attribute scores assigned to the images by the regressor from Zhang et al. [43 Given sequence of three images with attribute scores d1 02 and ag, the Preference Score q is computed as:
(a 43 + l22 (a) + 02)/21).
(8)
Sequences with higher values are preferred as they reflect correct ordering and smoother distribu- tions. In contrast, reversed or uneven sequences result lower q. Further details, including examples illustrating the calculation of provided in Appendix G
Models. For the construction and evaluation of the dataset, use an instruction-tur Llama-3-8B model (2). To generate image sequences for constructing chosen and rejected samples in the dataset, we employ Stable Diffusion XL (SDXL) (33. For training purposes, we use Stable Diffusion 1.5 28 which provides balance between computational efficiency and generation quality.
7

---

Evaluation. The performance of the system is evaluated using two metrics. The first metric is the Average Preference Score across all generated sequences from the test dataset. The second evaluation metric is the Order Consistency Ratio, measuring the proportion of generated sequences in the correct order, i.e., where 01 < 02 < 03.
Baselines. To evaluate the effectiveness of the proposed SysDPO-Direct joint alignment approach, we compare it against four baseline methods. (1) System Before Alignment: system prior to applying SysDPO-Direct. Notably, Llama-3-8B is instruction-tung
and serves as a baseline for separately aligned systems. (2) Best-of-N Sampling: from four generated sequences per prompt, the one with the highest Preference Score is selected.
(3) Only Train Language Model: the diffusion model is frozen, and only the language model is aligned using the dataset and the loss function of SysDPO-Direcs (4) Only Train Diffusion Model: the language model is frozen, and only the diffusion model is aligned. All baselines use the same dataset and loss.
Results We evaluate the system using the Preference Score and Order Consistency Ratio. Examples of system outputs before and after training can be found in the Appendix [
The results in Table I demonstrate the importance of alignment in compound Al systems and the effectiveness of the proposed SysDPO-Dine alignment approach. The "System Before Alignment" baseline achieves poor performance, with low Preference Score and a low Order Consistency Ratio (32%), indicating that conventionally instruction-tuned components are insufficient to ensure coherent collaboration required in compound systems. The "Only Train Language Model" baseline achieves significantly better results, with a Preference Score of 0.23 and a ratio of 65%. This highlights that the language model plays critical role in guiding the overall behavior of the system, it generates captions that directly influence the outputs of the diffusion model. In contrast, the performance gain from training only the diffusion model inherently constrained by the captions produced by the fixed language model, thus its performance gain is notably lower than that of the Only Train Language Model bascline. SysDPO-Direct achieves the best Preference Score (0.25) and the highest Order Consistency Ratio (73%). These results validate the effectiveness of our SysDPO algorithm, demonstrating its ability to optimize both components together for superior performance in generating coherent and progressive image sequences. The training dynamics of all three methods are provided
in Appendix J
Method
Pref. Score
OC Ratio
WR-
WR- Prompted
SysDPO-Direct
0.25
73%
Method
Chosen
(Proposed)
SyspPO-Sampling
19.8
66.4
System
-0,20
32%
Prompted System
12.8
Before Alignment
Separate-DPO
16.6
57.3
Best-of-Sampling
0.16
67%
Only Train
SysDPO-1/1
16.0
60.4
Language Model
0.23
65%
SysDPO-a/z
18.1
63.9
Only Train
-0.03
38%
Table 2: Overall Performance Comparison. WR- chosen denotes the win rate (%) against human- preferred responses in the dataset, and WR- prompted measures the win rate against the Prompted System baseline.
Diffusion Model
Table I: Performance comparison of the pro- posed method and baselines. Higher Preference Scores (Pref. Score) and higher Order Consis- tency Ratios (OC Ratio) are better.
5.2 Compound LLM Collaboration System
This section aims to evaluate the performance of our joint alignment methods, SysDPO, in the two-stage LLM collaboration system described in Section 4.2 In our implementation, we employ two instances of Qwen1 ww..5518--..... 8B-Chat [33] to serve as 1/ and 4/2, respectively. The two models are trained without sharing parameters. This two-L.MM configuration represents typical and illustrative setting for analyzing system-level alignment dynamics, offering a clear view of inter- model coordination and preference optimization.
We further extend our framework compound architectures. The setup and preliminary observations of this extension are presented in Appendix D In the following sections, we focus on the two-LLM system the main case study for detailed analysis and discussion.
three-LLM system, demonstrating its scalability larger
8

---

Dataset. We employ the preference dataset http//orra-dp--aaaiaiaa [2] for DPO training, con- sisting of 129000 instructions paired with corresponding preference examples (each instruction has a pair of chosen/rejected responses). We sample 193 instruction data points as the evaluation set. The remaining examples are used for the training process. We directly use the dataset without generating intermediate outputs. Since no ground truth reward model is available to give preference scores to intermediate samples, we focus on evaluating SysDPO-Sampling
Evaluation. We adopt the evaluator weighted-a allaca-paaaaaaaaaaaa -eeeea--va--gpt-tuurrre [[5] an automatic annotator based gpt-4-turbo, to assess model performance through pairwise comparisons. Given a pair of outputs-one one from the evaluated system and one from a reference- assigns preference between the pair. The Win Rate is defined the proportion of cases where the model output is judged superior to the reference output. To better understand the performance of the model from different perspectives, uses the chosen response from the preference dataset as the reference, reflecting alignment with human-labeled preferences. The second, WR-Prompted uses prompting-based baseline system as the reference, which will be introduced in detail later.
annotator
report two types of Win Rate. The first, WR-chosen,
Baselines. We compare SysDPO-Sampling with several baseline approaches to evaluate its effec- tiveness in aligning compound AI systems. As a simple baseline, we explore the prompting- based composition of the two-stage system without any alignment or additional training. Specifically, we directly connect two models and give task specific prompts to guide their collaboration. This setup referred to as the Prompted System. The second baseline, Separate-DPO, follows similar prompting scheme to coordinate the two models but introduces alignment by training each stage individually using the original DPO objective on the /orcl//orcadpoooo -ippss dataset. After alignment, the two stages are composed into a compound system. Each stage is optimized independently with its own preference signals, without joint training system-level feedback. In contrast, jointly aligns the entire two-stage system using holistic preference signals. Rather than optimiz- ing each component in isolation, it directly optimizes the composed system based end-to-end preferences, allowing both stages to adapt cooperatively to user-aligned objectives.
For the results reported below, we sampled two intermediate outputs per step during training. An analysis of the effect of sample method is provided in Section [.2.3] During both training and evaluation, we set the maximum token number at 256 in the sampling process. We trained the models using LoRA with 0.5, a learning rate of 10 7, and accumulated batch size of 128. Aligning the entire system took approximately 30 hours on a single NVIDIA H200 GPU.
Training Details
5.2.1 Is joint alignment necessary?
Results in Table Table that the non-optimized compound AI system (Prompted System) performs the worst, underscoring the limitations of relying solely prompt engincering for effective component coordination. In contrast, SosDPO-Sampling achieves substantial gains, improving the win rate against chosen responses from 12.8% 19.8% a 55% relative improvement. Its outputs are also preferred 66.4% the time over those of the prompted system.
The Separate-DPO baseline represents a conventional approach in which each component is aligned individually, without considering preferences over the behavior the composed system. Although this method yields better performance than unaligned prompting, it is still outperformed by our joint alignment method. Notably, this setting, the collaboration between the two models is relatively simple, and their roles are similar. Yet, even under these favorable conditions for apparate-POO. DAsDPO-Sampling still achieves superior results. In more general scenarios, where the interaction between components is more complex, their roles are more distinct, or there no clean training data available per component, the performance of Separate- is likely to further degrade. These results underscore the value of optimizing compound systems holistically rather than in isolation.
5.2.2 How much does each stage contribute to the final performance?
To better understand how system-level alignment affects each component, we design two variants of SysDPO-Sampling where only one stage is updated during training, while the other kept frozen. Importantly, the training is still guided by system-level preference signals, and the overall loss computed using SysDPO-Sampling In the SysDPO-1/1 setting, we train ti while freezing t/z; conversely, in SysDPO-12, we train the and freeze dh.. As shown in Table 2 SysDPO-U1

---

achieves 16.0% WR-Chosen and 60.4% WR- WR-Prompted, while SysDPO-1' - achieves 18.1% % and 63.9%, respectively. These results indicate that both components benefit from system-level alignment, but 4/2 plays a more critical role in determining the final quality of the output. This is likely because 2/2 directly generates the final response and has access to both the input x and the intermediate output Crucially, both variants outperform the Prompted System, indicating that during joint training, system-level preferences are effectively distributed across components, and each model able to learn and adapt accordingly. However, neither of them matches the full SysDPO-Sampling system, emphasizing that jointly optimizing both components leads to better performance and more effective collaboration.
We observe that L.sDPO-Sampling converges more slowly than its stage-wise variants but achieves the highest performance; detailed training dynamics are provided in Appendix
5.2.3 Effect of Finite Sample Approximation on Performance
approximate the loss function by sampling only small number of highly probable and distinct candidates yf In this section, we discuss how this sample set is generated and analyze how the sampling strategy and sample size affect training performance.
In Section 2.3
Our main experiments are based on the Diverse Beam Search (DBS) method [3], which generates k diverse candidates per input. We compare DBS with standard Monte Carlo (MC) sampling and summarize the results T
During training, we sample & intermediate candidates y'....,y"
yd ~ Prr, (- 2)) and use them to compute the loss. At evaluation time, set the temperature 0 and deterministically selecting the most probable outputs.
Method
Sample Size
WR-Prompted (%)
DBS
2
68.5
DBS
68.2
MC
66.8
MC
3
67.0
For DBS, we obtain & candidates using & beam groups and a diversity penalty of 20, reporting results for & = 2, 4. For MC sampling. we independently draw A samples with temperature 1.0 under different random seeds, evaluating
MC
4
66.7
MC
66.0
Table 3: Comparison between Diverse Beam Search (DBS) and Monte Carlo (MC) sam- pling under different sample sizes k:
{2,3, 4,5).
Table 3 summariizs the win rate (WR- (rR-Prrmpted) for each configuration. DBS consistently outperforms MC with the same number of sampled candidates.
Both methods show stable performance as k varies, suggesting that as few as two samples may suffice without significantly increasing computational costs. Compared with DBS, MC often produces near- The training dynamics are duplicate candidates, as illustrated by qualitative examples Appendix L shown in Appendix M
Conclusion and Discussion
We propose a principled framework for aligning compound AI systems by modeling them as DAGs and optimizing for system-level preferences. Our two methods, SysDPO-Direct and SysDPO- Sampling, address settings with and without intermediate output, showing both theoretical and empirical gains across two compound tasks. While demonstrated on specific systems, our approach has broader potential in domains like healthcare and education, where complex multi-component Al workflows require careful alignment to ensure safety and usability.
Despite these advances, several open challenges remain. For instance, extending SysDPO to systems with more components and complex interactions is natural next step. Another key direction improving training efficiency. SysDPO-Direct requires access to intermediate generations and may become expensive when components produce high-dimensional or latent outputs (e.g., in vision multimodal systems). Exploring approximations or sample-effir efficient estimators would be beneficial, For SysDPO SysDPO-Sampling better sampling strategies could improve gradient estimation when intermediate states are unobserved. Moreover, the broader design space compound Al systems, such as dynamic routing, feedback loops, or interactive collaboration, raises new alignment questions that extend beyond static DAGs. Future work should explore how to generalize SysDPO settings with non-static structures and latent coordination mechanisms.
10

---

Acknowledgments
SK acknowledges support by NSF 2046795 and 2205329, IES R30SC240046, ARPA-H, the MacArthur Foundation, Schmidt Sciences, HAI, OpenAI, Microsoft, and Google.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint wrXI1:2303.08774,220 2023.
[2] AI@ Meta. Llama 3 model card. 2024.
[3] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0580. 2022.
[4] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 39(3/4):3243344 1952.
[5] Trevor Campbell and Tamara Broderick. Automated scalable bayesian inference via hilbert coresets. Journal of Machine Learning Research, 20(15):1-385 2019.
[6] Lingjiao Chen, Jared Quincy Davis, Boris Hanin, Peter Bailis, lon Stoica, Matei Zaharia, and James Zou. Are more LLM calls all you need? towards the scaling properties of compound AT systems. In The Tiihth-eighth Annual Conference on Neural Information Processing Systems, 2024.
[7] Roi Cohen, May Hamri, Mor Geva, and Amir Globerson. Lm vs Im: Detecting factual errors via cross examination. In The 2023 Conference on Empirical Methods in Natural Language Processing.
[8] Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improv- ing factuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning, 2023.
[9] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835-10866. PMLR, 2023.
[10] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[11] Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, and Shriyash Kaustubh Upadhyay. Routerbench: benchmark for multi-llm routing system. arXiv preprint arXiv:2403 /2031, 2024.
[12] Hong Jun Jeon, Smitha Milli, and Anca Dragan. Reward-rational (implicit) choice: A unifying formalism for reward learning. Advances in Neural Information Processing Systems, 33:4415- 4426, 2020.
[13] Eser Kandogan, Sajjadur Rahman, Nikita Bhutani, Dan Zhang, Rafael Li Chen, Kushan Mitra, Sairam Gurajada, Pouya Pezeshkpour, Hayate Iso, Yanlin Feng, et al. A blueprint architecture of compound systems for enterprise. arXiv preprint arXiv:2406.00 20X:224066005844 2024.
[14] Tomasz Korbak, Hady Elsahar, Germán Kruszewski, and Mare Dymetman. On reinforce- ment learning and distribution matching for fine-tuning language models with no catastrophic forgetting. Advances in Neural Information Processing Systems, 35:16203-16220, 2022.
[15] Tatsu Lab. Alpacaeval: Automatic evaluation of iostruction-followee models. https://
github. wta.bb..oottttuuuucm 1ab/ hatplaaaaaaaaaeeee alpaca_evaaa 2023, Accessed on 2025-05-15.

---

[16] Matthieu Lin, Jenny Sheng, Andrew Zhao, Shenzhi Wang, Yang Yue, Yiran Wu, Huan Liu, Jun Liu, Gao Huang, and Yong-Jin Liu. LIm-based optimization of compound ai systems: A survey, arXiv preprini arXiv:2410. 16392, 2024.
[17] Duncan Luce al. Individual choice behavior, volume 4. Wiley New York, 1959.
[18] Vahab Mirrokni, Renato Paes Leme, Adrian Vladu, and Sam Chiu-wai Wong. Tight bounds for approximate carathéodory and beyond. In International Conference Machine Learning, pages 2440-2448. PMLR, 2017.
[19] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[20] Judea Pearl. Causality. Cambridge university press, 2009.
[21] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv: 2019.
[22] Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space control. Proceedings of the 24th international conference on Machine learning, pages 745-750, 2007.
[23] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arxiv preprint arXiv:2307.01995, 2023.
[24] Jie Qin, Jie Wu, Weifeng Chen, Yuxi Ren, Huixia Li, Hefeng Wu, Xuefeng Xiao, Rui Wang, and Shilei Wen. Diffusiongpt: Llm-driven text-to-image generation system. arXiv preprint arXiv:2401. 10061, 2024.
[25) Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sushil Sikchi, Joey Hejna, Brad Knox, Chelsea Finn, and Scott Niekum. Scaling laws for reward model overoptimization in direct alignment algorithms. Advances in Neural Information Processing Systems, 37:126207-126242 2024.
[26] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
[27] Yi Ren and Danica J Sutherland. Learning dynamics of I1m finetuning. CoRR, 2024.
[28) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, June 2022.
[29] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference Learning Representations, 2018.
[30] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[31] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning summarize with human feedback. Advances in neural information processing systems, 33:3008-300 3021, 2020.
[32] Intel Neural Compressor Team. Intel/orca_dpo_ppe dataset, 2023. Accessed on 2025-05-15.
[33] Qwen Team. Qwen 1.5-1..51..80 --.-cha. httts: catpr//accc.oo/eee///eeeeeeeeeee 1. 8B-Chat 2024. Accessed on 2025-05-15.
12

---

[34] Ashwin K Vijayakumar, Michael Cogswell, Ramprasath R Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. Diverse beam search: Decoding diverse solutions from neural sequence models. arXiv preprini arXiv: 1610.02424 2016.
[35] Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 8228-82288888 8238, 2024.
[36] Junlin Wang. Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. Mixture-of-agents enhances large language model capabilities. arXiv preprint S2ii:22000046692, 2024.
[37] Shirley Wu, Parth Sarthi, Shiyu Zhao, Aaron Lee, Herumb Shandilya, Adrian Mladenic Gro- belnik, Nurendra Choudhary, Eddie Huang, Karthik Subbian, Linjun Zhang, et al. Opti- mas: Optimizing compound ai systems with globally aligned local rewards. arXiv preprint
arXiv:2507.03041, 2025.
[38] Kai Xiong, Xiao Ding, Yixin Cao. Ting Liu, and Bing Qin. Examining inter-consistency of large language models collaboration: An in-depth analysis via debate. In The 2023 Conference on Empirical Methods in Natural Language Processing.
[39] Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. Textgrad: Automatic" differentiation" via text. arXiv preprint arXiv:2406.07496
2024.
[40] Matei Zaharia. Omar Khattab, Lingjiao Chen, Jared Quiney Davis, Heather Miller, Chris Potts, James Zou, Michael Carbin, Jonathan Frankle, Naveen Rao, et al. The shift from models compound ai systems. Berkeley Artificial Intelligence Research Lab. Available online at: https://bair: berkeley. 2024.
February 27, 2024),
[41] Jacky Zhang, Rajiv Khanna, Anastasios Kyrillidis, and Sanmi Koyejo. Bayesian coresets: Revisiting the nonconvex optimization perspective. In International Conference Artificial Intelligence and Statistics, pages 2782-2790. PMLR, 2021.
[42] Yusen Zhang, Ruoxi Sun, Yanfei Chen, Tomas Pfister, Rui Zhang, and Sercan Arik. Chain of agents: Large language models collaborating long-context tasks. Advances in Neural Information Processing Systems, 37: 1771132208-3222:1 2024.
[43] Peiye Zhuang, Oluwasanmi O Koyejo, and Alex Schwing. Enjoy your editing: Controllable gans for image editing via latent space navigation. In 9th International Conference on Learning Representations, ICLR 2021, 2021.
[44] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: 1909.0853, 2019.
13

---

Appendix
A
Complete Prompts in Figure !
The complete prompts generated by GPT-4 for DALL-E to generate the six images Figure are as follows.
Left picture
(a) A realistic image calm cat sitting comfortably on soft cushion. The cat has serene expression, with its eyes half-closed in contentment. It has fluffy fur, predominantly grey with white patches, and its tail is neatly curled around its body. The background a cozy room with a hint of sunlight filtering through window, casting warm glow the cat.
(b) A realistic image of slightly irritated cat sitting slight frown and its ears are tilted back slightly, indicating mild annoyance. has short fur, mix of black and white, and is staring directly the viewer with wide eyes. The back- ground is simple, with few scattered cat toys and small plant, adding to the homey atmosphere.
wooden floor. The cat has
(c) A realistic image of a very angry cat with its fur bristled and ears flattened against its head. The cat's eyes are narrowed and glowing, mouth slightly open in a hiss. It has thick, long fur that is mostly black with some orange stripes. The background shows stormy sky through a window, enhancing the dramatic mood. The is on dark rug, which contrasts with its bright, intense eyes.
Right picture
(a) A cartoon-sty style image of cat with slightly annoyed expression. The cat has fluffy white fur, bright green eyes, and its ears are slightly pulled back. It is sitting comfortably on a soft blue cushion, with its tail flicking lightly to the side. The background is simple, soft yellow keep the focus on the cat's expression.
(b) A cartoon-style image of cat with an angry expression. The cat has fluffy grey fur, narrowed yellow eyes, and its floor, with its fur bristling and its claws slightly out. The background is dimly lit room, adding the menacing atmosphere around the cat.
flat against its head. It standing with an arched back a wooden
(c) cartoon-style image of a with a furious expression. The has short black fur, glaring red eyes, and its ears are pinned back. It is hissing aggressively, showing sharp teeth, with puffed tail. The cat stands on a stormy night background, with lightning the sky, enhancing the dramatic and intense mood.
14

---

B Details of the Loss Function of SysDPO-Sampling Lampling
To approximate the system-level preference probabilities in Equation 2 we apply the sampling-basee
approximation in Equation (6) which rewrites the system likelihood product over component- level probabilities.
Plugging this into the original DPO loss, we obtain the following training objective:
Σ
II Pa. KET
(ofg) Pa(v()) II Pe, (2)) Pa(z)))
jEJ
I(0) -E [e, a
log
B log
Σ
Il Pa. (0 Pa(y.)) II Pa, (z Pa(2)))) iEl
JEJ
Σ
II iE!
pe, (b.1 Pa(y))) JEJ
II pe, (2)) Pa(z)))
- B log
(9)
Σ
Il FE7
Pa, (uff Pa(v()) JEJ
II Pa, (
This formulation allows us to perform end-to-end gradient- based optimization over all model compo- nents {0,} and {0,} in compound AI system. generate the samples indexed by a, we employ Diverse Beam Search [4] which still selects the highest-probabee sample after applying the diversity penalty. This ensures that the sampled trajectories are both diverse and high-probability, which aligns well with our approximation.
C Theoretical Analysis
Proof of Proposition 7 Given the reward function r*, it follows from prior work 26 22
21
4 that
the solution ng. to 7 satisfies
Pg- (2 x) 3 log G(æ);
(x, Blg
(10)
Pa 2 x)
with G(z) = Po(2 z) exp( d2 being the partition function. Then, we plug (IO
into with some algebraic manipulation, arrive at
Pas (2 [a) PG(2
pref(2" "F
x))
(2 Pa [2]
Pex (z'(z) Po(3(
When the reference model p a uniform distribution, we conclude
pg. (2 x
pref(=" " > [1)
Pa- Pa-(2* (2
x)
Po-
(2)
showing that 0" is perfectly aligned (Definition
Next, let examine what is solution to the DPO loss function 2 a uniform distribution, the DPO loss is simplified to
Given that the reference model
L(0)- - Bi ;DDD [log ( (olggo( (z)
3 log po (z'1z))
Po (0
- -B(sse"
log
Bog
Po (=
(2(2 2
--B log
(11)
pa(2"(x)* + po (x)
Let us review the definition of this preference data distribution D. Given any data triplet (ar, 2, z') E
X x Z x Z sampled from pre-preference data generation process D', the preference oracle would
15

---

label (2 = 2, = x²) with probability pref(z > 2' x)) and label (z" = 2', = 2) with
probability pref(# 2' æ). Therefore, denoting
Po (z|
pref (z a' x)) == p=(=)x)² p=(22x))3
the DPO loss above can be written as
-2,,,,,') [pref([= > z'1 log prefn(z > x))
+ (1 - pref(z (= x 2' x)) prefool( x 2 |x]]]
-- pprof(2 > 2' x) log proff
(pref((
.))
(1 pref(2 2 2 x)) log (1 prefo (z 2f [=)]].
Noticing that the above is precisely a cross entropy term, we conclude that the minimum is achieved when 8* satisfies
pg- (2
pref(z > 2 12) prefg. (z 12)
pe- (2|2)) Pa- (z'(z))
Thus, the solution 0 the DPO loss function agrees with 8-perfect alignment.
Proof of Theorem SysDPO-Dire works by simply replacing the final output 2 in the original DPO loss function by the set of all generated variables := Yyr U {2}. By definition, the corresponding preference oracle for two sets inherited from the original preference oracle as follows.
prefays(s" s 1 pref(2" " > 2f x),
(12)
where n E t and 2 s!.
Thus, Proposition | implies that if 0% minimizes the SysDPO loss function 44 with uniform prior, satisfies
prefays (s" " > st x)) poz. (s (x) prefays (s) x)) pe: (st
x
Applying (12 and rearranging the above equality, have
pref(2" > 2¹ x))
1/3
preffys > 5 x))
1/9
pa;, Pa; (s |x]
pref(z! > n0 x))
prefoys (st x g
x))
pa.. (s' |x)
Poz, (1v1" Fie! f2"0 x))
(13)
pe,, (Avthier {2() xx)
A key observation from the above equations is: the LHS is independent from any y/- Thus, there
should be some freedom in what y; we include on the RHS. Concretely, Assumption [ states that, for any intermediate samples v(vilse E ) and final output & E Z. we know (fy her, 2) appear with positive probability the SysDPO loss. Recall minimizing the SysDPO loss require (13) holds for all pairs (s s²) sampled from D. Thus, the above equation must hold for (vu her > (Hin)er, z z"), ).
her, Thus, it implics that the RHS is also independent from any
This result can be intuitively interpreted in the following way. Consider the assumptions that the training data D is good enough and the optimal model 8 is capable enough to achieve the minimum of the SysDPO loss. Then, the optimal model learns to extract sufficient information from all those possible intermediate outputs y; such that it achieves perfect alignment no matter what y is sampled from the data distribution.
Given this observation, we can simply replace {ul" Jirr and [v])er in 13) by any (yieer E y without changing the value of [13) Rearranging the equality and sum {ys her over its sample space
16

---

Y on both sides:
13
s. (Luihirr U {2 x)) = pas. (tuitier U {="h} 1x)
pay. ((ui)ier U (z') xx) Epor, (lustier U (2") xx)
y
pes. (z x)) = poz (2' xx)
poz (2th
x.)
poge (zl x))
Raising the above terms to the power of B, we recover the definition of perfect alignment.
D Extend to a three-LLM system
To further examine the scalability of our framework, we extend the two-LLM configuration described in Section [5.2 three-LLM system. In this setup, first-stage LLMs 1 and 1/2 independently generate responses to the same input query, and the second stage LLM t3 serves as a synthesizer that aggregates and refines these intermediate outputs into a final answer. This architecture intro- duces an additional component of coordination and preference alignment across multiple generation components, allowing us to test the framework's 's ability to align compound systems beyond pairwise interactions.
We train this system using the same SysDPO procedure employed in the main experiments, without modifying any hyper-parameters Evaluation is performed with the WR-Prompted metric, directly comparing the post-alignment system with its pre-alignment counterpart. The aligned three-LL.M system achieves 58 % win rate against the unaligned version, suggesting that the proposed method effectively generalizes larger multi-component architectures.
While large-scale experiments involving many components are currently constrained by our computa- tional budget, these preliminary results provide encouraging evidence that our framework offers solid foundation for future scaling toward more complex compound Al systems.
E Discussion on Finite-sample analysis for SysDPO-Sampling
In this section, we discuss how a diverse sampling scheme may benefit DysDPO-Sampling through a awo-generative-modde collaboration system (Figure (b)).
As shown in Section 2.3 SysDPO-Samplin aims to approximate the intractable generation prob-
ability p=(z w)) = EvEy po,(2 (z y, x p) (y w) with finite samples D = {v²) such that approximately po a)) x E. (o, yd, x))o,
This can be viewed as coreset selection problem. I.c., the goal is to find a diverse subset 5 of representative points approximate the full set y.. Denoting pyy == [ Epey Pe y, y, (y x)) (similarly for 133 ), we aim to minimize the approximation error = Jyy - Pyl-
Therefore, a strategically selected subset often outperform random Mlonte-Carle (MC) sampled subset, where the strategy typically involves selecting representative and diverse points [5] 41 22]. Diverse Beam Search (DBS) can be seen greedy coreset-selection algorithm where, every decoding step, it keeps the subset that jointly maximizes a combined objective of generation likelihood and diversity.
The core of a finite-sample analysis is the rate of approximation error w.r.t. the budget A: | Although it is unclear what the exact error e of DBS is, coresct algorithms generally achieve a worst-case guarantee of e ( + [5] For example, consider finite sample space Z, and denote
the sampled probability mass function Vy (- y. x. (y 2) E glž|, Assume all vy is bg-bounded by D. The approximate carathéodory theory [8] states that we can always find a subset
5 C y' of size l such that the approximation error O V ). To make the bound tighter, we need
fine-graine fine-grained analysis that exploits the concrete structure of the distribution, which is an exciting but
17

---

non-trivial future work. Nonetheless, in practice, coreset can be significantly better as shown in prior works 29 well as our experiments.
Deriving the Loss function of LLM + Diffusion System
In this section of the appendix, provide detailed explanation of the DDPM diffusion model, and the derivation of the loss function used for this system.
F1 Denoising Diffusion Probabilistic Model (DDPM)
DDPM [TQ] is widely used class of diffusion model. Below is highlight of the key ingredient need for DPO for DDPM [35], with our framework.
Given real image 20, consider diffusion process, which we call the forward process, gradually making the original image into Gaussian noise 2T after T steps, i...
20
21
22
2T (0.
The goal of the diffusion model is to reverse this process that recovers image from noise. The forward process and the reverse process are denoted respectively as
( y):
Po(
where y the context, i.e., the prompt to the diffusion model.
Note that both the forward and backward processes are Markovian, and particular we have the nice property that the forward process
T
(<orrly) <<>l))) II q((r12(11);
where each ((z/2((11) is Gaussian.
Similarly, the reverse process
Po = p( (r) Il Po(2(-122e.0)
where each (= v) is Gaussian.
(14)
In this formulation, the ideal goal for the diffusion model that (( (v) this is not easy to optimize directly. With some analysis, the DDPM paper [0] proposes minimize for
(s(or)) However,
Dxz(a(z-11la, 2o, () v)) for ~ U([T), 20 q(zoly),
where u((-) denotes the uniform distribution on a set, and [T] denotes set of {1, 21. T)) This done by learning denoiser es operating in the following way. For real image g(=0) v), we sample noise NO, I), and have
2z <) = Vārzo + -I- Cr,t
(15)
where o, is some parameter such that 2, ~ (2)))2)) Then, the denoiser predicts the noise e that is added to the 20. Lee.
<0(=1 4,) aims to predict
The denoiser es is essentially reparameterizztion of the mean of po(21-1l24,V) v).
The key ingredient is that, shown in 10),
w)1l DKL. (gligg(z-11 [2, z0, y) [p w)) Ee-A().)) w. ||e c0(21(za, e), 4),1 +C, (16)
where Wi is weight parameter and C constant independent of model Φb
Therefore, modeling 6 by a neural net, DDPM model trained minimize the above objective averaged over samples of y, 20. e.
18

---

F.2 Dealing with the Diffusion Model in SysDPO
Recall in the main text we obtain the System DPO loss function as:
_t) log (s (8(18g log (x() Po(y)(x))
(( ,l)
+ Σ log
(2 lv(tt)
Pe (1
log Pos (o e/ll(() pa(e!lut)
- B log
po(
The next step convert the likelihood of the diffusion model pa something optimizable. Consider the generated image as the whole process, i.e.,
21,0:T == {25,0, 21,1, ZrTh
where 2,,0 is the generated image, while the others are things in the middle. Following the same notation, we denote
2,,-1.1 := 2i,r)
The preference is considered to be given to the every processes that generates 2g as the end outcome. Following [35], we have
L(8,0) (, logo
((o8 Po (o(vro(z))
((osPolnnn Po
Pal Po (wx) P& (2 o.r (Bw)))
po (2((2)) (log
Pe (o... (y(
Σ
log
+
log
pg (yn (r)
P P 1B
Recall the decomposition of the reverse process 14
we have
L( -E -B )a) log
BE 1. a ao
Po ((oss pe Pp(uullz (2:)
Po Po 12 Vie)
log Po xx) ()(y)
Ps p 11
log
log pz p3 12,,0 y
P (2 22, Vi.)
t=1
Note that - TELuUUD for t being random variable uniformly distributed 1, Simply denoting ELxU(DD as E,, we have
... T.
L(0,0) -B
log [B neglt 1
Po (o (x) ((nos
Po (2 12 vin)
Po (y la:) pa pa(urla (x)
P polzj,-11kd,. of)
T
E, log
log
+ T Σ
E, log
Po(v
PT 2/ 1 1-2,0- vin)
p Po(2f,-11</4 wi)
logo
BE
1. w (2 si ( E.
(( log Po(uw px()w(z))
T Σ log P&l P (2 2/, v'n))
Po (31((rr
Pe Po(st,c-1124 121,,1 v/)
log
+ T Σ log
p 12,,e - Vin))
Pa (y (x)
Ps (zt,e_11 1l,., 1//
19

---

Next, we may further simplify the equation by switching E.,, " aaiaa)) aaaaa wew.aeeeeeeeee.. (11,, and Eg in the above, i.e.
logo { Ee T(0,0) - -R a
( aar
(1og Po (x) (o(
51,0 Vin) 2 P5 w.
Po(2 wiet
pa (311 (z)
Pol P f -12f,e vi)
E
log
log
+ T
log
1, Vir)
Po (311 (z)
pel pz( (2t,,-11225. vi)
log
((os
po po
Ppl= w.
8,,0 Vin)) 2 2w --1
llog pa (3r lx)) Po (31
Po Po 125 vf) pe ( p 10,00 vi)
Σ log
log
(o(
P
1,,0 Vir)
The rationale for the above can be illustrated as follows. Consider a random variables Zz,... Zr, and any function (Zt-1, 1, Zi) R for any t [T]. Then, denoting 52 the indicator function, iee, 5% = 1 only if t, and 8" = otherwise, we can derive
Eg EL-((T)) f(z- Zi) Ezur ELM((T))
Σ 8 ((
T
= E.-U(T))
Σ
8
Ez f(Z--1,
T
EI-U(TT)
Σ
s=1
= E(-U(TD)
Ez, 1. (2 21).
Next, noting that (,,,,,_,,el2,, <(220)1 ( (w) q( 120,, xx10 (similarly for q (,,(,,-1,11=,00) we can first sample t,t and then = separately, i.e.,
(), = -B(c,s" (')
log a BE, E2,, male,, .44.0)) Eyg r "i"" 1
(10g (o (x) (o
po P=(= 2,,t v'n)
pa (zul (x)
Pe (0,t),-1123 12f,,1 vi)
log
log
+ T
log
Pš p=(= ( 23.00 Var)
pa (u))z)
pø (2i,,-11=f,, -/f,e vi
Since log σ is convex, by Jensen's inequality, we have
L(0,0) S BBe, of DIE E,
log a BE,L. vellth. 1la,,0="%,)..f, 1oqltf, 11
((1ss Po (o (x)
log
Pel wn -- 23,00 Vin)
log
po (311 (31120) (x))
+ T Σ log
Pe (0(2t,--122,, vi)
Pe (oolvrl
Pzl 2 1- 27,00 Vin)
Pa (31 (x)
Pe p Po(2i,__122,, vi)
(17)
Recall that we we have been done so far are all for making the diffusion model's log probability efficiently computable. To complete the derivation, it left to convert the log- bog-prbbabilities to the
20

---

denoising loss via (6). Specifically, with C' being the constant appears in 166 we can see that
log Po Po 12 2. Hin)
E. - o
Pá (2 1 ( 22,1 Vir)
log Po ( 12, Fie)
log P5(= 12 Fir)
=BaL, melltiee 11
g k0 27,1)
a 1m, 27,1)
-DRilg(Zu,-11 - kw,0r Zi,e) (2 yur) + D (9(=,, 12 25, ll (Z 1/25ee Fww)
-DKL(g(Zw,,-1) - 27,, (=
( 1,,,, vw))
+ C Dxz(a(2ur)) - >1plppp lp (20,0-112,,, Fwe) - C
-Ee-N(o.1) We ||e c <), t, v v1,)1 + Bewiiron) we ||e E5(2 e) t, 8 b/n
To simplify the notation, we denote
(d; L, 26,11 ww)
We ||e c=(="er.. 2l)3
where the corresponds to the noise from which =,, derived (see le 1, 2h, vi) to denote the denoising loss for the losing data.
15
Similarly, we use
Thus we can write (17
L(0, ②) s -E )D) Ee Ba oo -0
logo | (((1g) log
pe (1
T (e 2/, leloit, 2i, vin))
po(Uuwlz)) (x)
(log Pe (v0l2r) pø Pa(()(z)
+ T
Σ
Le +h, vi)
lo +h, sh. vil)
Thus, we obtain tractable loss function for SysDPO-Dires Direct.
Since only approximate fog-probabilities of the diffusion model are available, SysDPO-Sampling is not applicable in this setting.
G Preference Score Calculation
Definition. The Preference Score q evaluates the quality of a sequence of three images with attribute scores 01, 02 and 03 [0, 1), and is computed as:
q -= (a- ag
+ a3
02
The Preference Score reflects two aspects: (1) Order Consistency: A correctly ordered sequence (aj < a2 < 43) yields a higher q value, while reversed sequence results in a lower value. (2) Distribution Evenness: A sequence where a2 is closer to the midpoint between aj and ag maximizes the score.
Properties.
Example Calculation. Consider four sequences of attribute scores:
Sequence [1.0.5,0 0]
Sequence b [0, 0.99
Sequence e [0, 0.,
For a:
1+01 0
1)
9a =
1
1-0+
0.5
2
21

---

For b: -
0
0.9
00- 0.9 +
0.35
2
For c:
Q6 ---00-11
0 + 1
0.5
2
Since < Tb. sequence b is preferred between sequence a and b. Sequence с is preferred between sequence and c. This illustrates how the Preference Score penalizes uneven intermediate distributions or incorrect orderings.
H Prompt Styles and Examples
To ensure diversity in user prompts, we utilize four distinct prompt styles inspired by [24). Each style varies in how frames the objective for image generation. For illustration, all the examples below are based on the attribute "bright," showcasing how this attribute can be expressed in different styles.
H.1 Prompt Styles
Prompt-Based Style. This style of prompt directly describes the objective be generated. It provides a clear and concise target for the system. For example:
A series of images showing garden with increasing brightness, from dawn midday.
Entruuction-aase Style. This style uses instructional language to explicitly direct the system on what to generate. The phrasing is structured as a command or directive. For example:
Generate series of images of morning scene, increasing the brightness and cheerfulness.
Inspiration-Batt Style. This style reflects a user's desire or inspiration for what they want to see. The prompt is expressed as a personal request imaginative wish. For example:
I want series of images of mountain as it gets progressively brighter.
Hypothesis-Bart -ased Style. This style frames the generation task as hypothetical scenario, often using conditional or reasoning-basee language. The prompt includes both the condition and the desired outcome. For example:
scene becomes brighter, the series images will show progressively more illuminated scenes.
I Examples of Input and Output of Experiments
This appendix presents illustrative examples of the system's inputs and outputs. Each example consists of user prompt input and sequence of three images the output. Examples include:
Success cases before training, highlighting potential alignment issues in untrained systems.
Failed cases before training, showcasing issues with order consistency or uneven attribute progression.
(3) Improved cases after training, demonstrating the effectiveness of the proposed SysDPO alignment method.
22

---

I.1 Success Case Before Training
Input Prompt
I want see series of images of beach, with varying intensity of sunlight from overcast bright sun.
Attribute
Sunny
Intermediate Captions
(a) Generate an image of serene beach on a cloudy day, with the sun hidden behind thick grey clouds, casting soft, diffused light on the sand and sea. The waves are calm, and few seagulls are flying overhead.
(b) Create an image of the same beach, but with the sun breaking through the clouds, casting a warm, golden light on the sand and sea. The waves are slightly choppy, and few beach umbrellas and sun loungers are scattered along the shore.
(c) Produce an image of the beach on bright, sunny day, with the sun high in the sky and shining brightly down on the sand and sea. The waves are crashing against shore, and group of people are enjoying the day, with some playing beach volleyball and others swimming in the ocean.
Final Output Images
Figure 3: An example of success output before training.
1.2
Failed Case Before Training
Input Prompt
want see a series of images of lake as the ice increases.
Attribute
Ice
23

---

Intermediate Captions
(a) Generate an image of a serene lake on a warm summer day, with calm water and lush greenery surrounding it, but with no signs of ice frost.
(b) Create image of the same lake, but with thin layer of ice starting to form the surface, with ripples and imperfections, and the surrounding landscape showing subtle signs autumnal change, such as fallen leaves and fading greenery.
(c) Produce image of the lake completely frozen, with thick layer of ice covering its surface, and the surrounding landscape transformed into winter wonderland, with snow-covered trees, frozen waterfalls, and crisp, blue-gray sky.
Final Output Images
Figure 4: An example of a failure case before training. The first and second images lack clear progression in ice intensity.
1.3
Improved Case After Training
Input Prompt
Create series of images of lake, increasing the thickness of ice.
Attribute
Ice
Intermediate Captions
(a) Generate serene summer day with a calm and clear lake shoreline, featuring lush green trees and blooming vegetation, with water as the main focus. The water should be smooth and reflective, with a few boats or sailboats in the background. Ignore the water conditions.
(b) Continue with the same shoreline and surroundings, but with the onset of autumn, add gentle winds and decreasing temperatures. The vegetation should be dormant, and while the water is still calm, start notice slight decrease in clarity. There should be ice yet, but hint the changing conditions with minimal foliage or bare trees along the shore.
(c) In the dead of winter, depict a frozen landscape with the same shoreline, but now with thick, rugged ice covering the entire lake. The trees should be bare and shown as silhouettes against the harsh winter sky. The goal is a highly realistic, textured ice surface with no signs of water, and incorporate any relevant winter elements like snow-covered ground bare branches.

---

Final Output Images
Figure 5: An example of an improved case after training. The sequence shows smooth and consistent progression in the ice intensity.
25

---

J Training Dynamics of the Compound AI System with an LLM and a Diffusion Model
We present the training dynamics of SysDPO-Dire to illustrate how the joint optimization of the language model and the diffusion model improves system performance over time. Figure O shows the evolution of the Order Consistency Ratio throughout training, which exhibits steady and consistent improvements.
Order Conásttncy Ratio Over Steps
24
25
100
Figure 6: Order Consistency Ratio of SysDPO-Direct over training steps in Application The consistent upward trend demonstrates the effectiveness of joint optimization in improving system- level coherence.
WR Prompted over Tra ning Steps
Comparizon of Samplnng Strategies over Training Steps
07.3
05.
05.
17.9
52.
38.0
583
Figure 7: Training Dynamics of Stage-wise wise and Joint Alignment. WR-Prompted scores over training steps for SysDPO-Sampling SysDPO- 4/,, and SysDPO-12- Stage-wise alignment strategies converge more quickly, but SysDPO- Sampling ultimately achieves the highest perfor- mance.
Figure 8: Comparison of Sampling Strategies During Training. WR-Prompted scores over training steps using sampling strategies: di- verse beam sampling with two candidates, and log- -iikeiihoodgguuieee contrastive selection from four candidates. Both strategies achieve com- parable final performance, suggesting that sam- pling two diverse candidates is sufficient.
K Training Dynamics of the Compound LLM Collaboration System
To better understand the learning behavior of different alignment strategies, we present the training dynamics of the two-stage LLM collaboration system in Figure 7 The curves show the WR-Prompted scores (%) evaluated periodically during training for SysDPO-SSmpling and its two stage-wise variants. SysDPO-11 and SysDPO-32-
We observe that SysDPO-1/ reaches its peak performance first, around step 10, but quickly exhibits noticeable fluctuations thereafter. This suggests that while ii rapidly adapt system-level supervision, its progress is constrained by the fixed, untrained 1/2, limiting overall stability and further improvement.
SysDPO-u22 shows a more gradual increase performance, reaching its peak around step 40 and maintaining more stable trajectory. This may be attributed to the role of 1/2 as the final decision maker, which benefits from direct access to both the input and the intermediate response y. and thus can learn more steadily from the system-level feedback.
26

---

In contrast, higher complexity jointly optimizing two interdependent models. However, continues to improve throughout training and eventually achieves the highest performance, reaching its peak at around step 53. This demonstrates that joint training, although more challenging, enables both components to co-adapt and better coordinate under shared supervision, leading to superior final results.
improves more slowly during the early training steps, reflecting the
After reaching their respective peaks, all three methods experience a slight decline in performance. possible explanation for this trend is provided by recent findings |2 9 271, which show that running DPO for long can lead to overoptimization, causing decline in evaluation performance. They further observe that smaller models tend to degrade more quickly, whereas larger models show more stable behavior. Since our system built upon 1.8B-pate -S--parameter models, some degradation is observed in later stages of training.
L Diverse Examples of Intermediate Samples
To complement Table 3 we provide qualitative examples of intermediate responses produced by different sampling strategies for the same inputs. Consistent with our quantitative results, Diverse Beam Search (DBS) yields semantically distinct candidates that support more informative preference learning, whereas Multinomial (MC) sampling often produces near-duplicates that dilute the learning signal.
Case 1: Extractive QA with abstention option.
Prompt
Extract the answer to the following question from the movie plot. If the question isn't answerable,
please output "Can't 1 answer".
Question: What does Johanna cut from Katniss's arm?
Title: The Hunger Games: Catching Fire Movie
plot: After winning the 74th Hunger Games, Katniss Everdeen (Jennifer Lawrence) and Peeta Mellark (Josh Hutcherson) return home to District 12. President Snow visits Katniss her home. The two make agreement to not lie to one another, and Snow explains that her actions in the Games have inspired rebellions across the districts. He orders her to use the upcoming victory tour convince him that her actions were out of genuine love for Peeta, not defiance against the Capitol, otherwise Katniss's 's loved ones will be killed. He shows her the clip where Gale kisses her warning that they watching her. As the tour begins, Haymitch Abernathy, Katniss and Peeta's mentor, warns them that the "show" of their relationship must continue for the rest of their lives. Katniss suggests public engagement between herself and Peeta, which is carried out and approved by Snow at his mansion in the Capitol. Returning home, Katniss warns her friend Gale Hawthorne of Snow's threat kill both their families. Peacekeepers crack down on District 12, and Gale is publicly whipped after attacking new Head Peacekeeper Romulus Thread. Snow announces that the upcoming 75th Hunger Games, the Third Quarter Quell, will feature tributes selected from previous victors. Katniss immediately devotes herself to ensuring Peeta's s survival, and recruits Haymitch's 's help to do so. At the reaping, Effie Trinket draws Katniss and Haymitch's 's names, but Peeta immediately volunteers to take Haymitch's 's place. As they prepare, Haymitch reveals that all the tributes are angry about being returned to the Games and advises a reluctant Katniss to make allies. In the pre-Games interviews, Katniss wears wedding dress, as ordered by President Snow, but her stylist Cinna rigs it to transform into representation of a mockingjay. Peeta announces that he and Katniss have secretly married and are expecting baby, which causes outrage, and the audience of Capitol citizens begins shouting for the Games to be stopped, to no avail. Just before Katniss enters the... Answer:
Intermediate candidates produced by DBS sampling
y²: Haymitch Abernathy's arm.
2²: Can't answer. The question is answerable it does not provide specific scene or moment in the movie where Johanna cuts from Katniss's arm. The plot of the movie focuses on the events leading up the 75... ....
27

---

Intermediate candidates produced by MC sampling
y':
Haymitch Abernath.
y²: Haymitch Abernathy.
Case 2: Data-to-text generation.
Prompt
Here is some data about restaurant: name = The Golden Palace, eatType coffee shop. food French, priceRange =20-25, customer rating = high, area city centre. Write sentence that includes the following data about a restaurant.
Intermediate candidates produced by DBS sampling
y²: The Golden Palace is fancy coffee shop in the city centre that serves delicious French food for price range E20-25, and it has a high customer rating, making it a great place to enjoy a meal with friends or family.
y²: There is restaurant called The Golden Palace located right in the city centre, where they serve delicious French cuisine price range of E18-22 per person, and customers have given it a high rating for its excellent service and cozy atmosphere.
Intermediate candidates produced by MC sampling
yh: The Golden Palace is a fancy coffee shop located in the heart of the city center, serving satisfied customers who love dining there.
y²: The Golden Palace is fancy coffee shop located in the heart of the city centre, where they serve delicious French cuisine at moderate price range of 620-25 per person, and customers highly rate their experience.
M Effect of Sampling process in the Compound LLM Collaboration System
To examine how the sampling process influences training behavior and final performance, we compare two strategies for generating intermediate samples used in the loss function in the two-stage LLM collaboration setting.
In the first strategy, sample two intermediate candidate per input using diverse beam search [34, We set the diversity penalty to 20 our experiments. In the second strategy, we sample four interme- diate responses for each input and then select pair based contrastive preferences. Specifically, compute the log-likelihood of the final output for each intermediate response using the second-stage model, and choose the pair for which one sample leads to the highest likelihood of heing preferred and the other the lowest. This contrastive selection aims to encourage sharper preference signals during training by focusing updates on clearly distinguishable pairs.
We further compare the training dynamics of the two sampling strategies by plotting WR-Prompted scores over training steps. As shown in Figure 8, both methods exhibit similar performance throughout training. This suggests that sampling only two candidates when combined with sufficiently high diversity penalty sufficient to yield informative preference pairs. Additionally, observe that the contrastive selection strategy exhibits slightly greater fluctuations during training, potentially due to the higher variance introduced by selecting extreme pairs based model log-likelihoods.
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We provide a detailed list of contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In the last section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]

---

Justification:
Assumptions and proofs are provided. Proofs are provided in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: All details in our main paper or appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
30

---

Answer: [No]
Justification: code will be published later.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https Cooeeeoodeesssseeeeee 7/nips.com http://ubb///uuuuuuooooo//aaaaaa wwwsinoniccccyyc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: in our experiment section.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Aligning compound Al systems is computational demanding. With resources it is intractable to obtain a meaningful statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
31

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
In Experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
This research is conducted with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
In our discussion section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
32

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: this paper focuses on fundamental ML methods.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: no such assets.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
33

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: no new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
34

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: the core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
35


Reviews:
[{"review_id": "SvNbFr4BeJ", "reviewer": "Reviewer_jnDe", "summary": "The paper studies alignment to human preferences in compound AI systems, such as those with multiple interacting components such as LLMs and tools. The authors formulate the compound AI system as a DAG to model component interactions and data flow and introduce SysDPO. They conduct empirical evaluations on two settings: joint alignment of language and diffusion models as well as LLM collaboration.", "questions": "Please see weaknesses", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The paper studies *joint* preference optimization in compound AI systems, which is an important problem (e.g., applications in multi-agent LLMs, LLMs+RAGs, multi-LLM routing systems, as the authors explain).\n- The problem considered in the paper is nontrivial as it non-differentiable interactions between components, difficultly with the assignment of credit and preferences across components, and decomposing the system-level preferences. \n- Formulating the compound AI system as a DAG and deriving a preference optimization algorithm based on the factorized probabilities is interesting.\n- Experiments show SysDPO performs better compared to prompted system, BoN sampling, and separate DPO, showing the necessity of joint alignment.\n\nWeaknesses:\n- Theoretical analysis in Section 3.1 include simple extensions of the properties of DPO and does not offer new insights. \n- Paragraph below Theorem 1 states that Assumption 1 is required; however, the theorem seems to only show that the statement holds under Assumption 1 and not the necessity of this assumption.\n- Experiments are limited to two component compound systems, and it’s unclear how the performance of joint PO compares with separate PO.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the constructive and encouraging feedback. We will incorporate clarifications and revisions, as well as new experiments in the camera-ready version. Details regarding your comments follow.\n\n- **Insights of the theoretical analysis in Section 3.1 and how it extends standard DPO**\n    - Section 3.1 indeed revisits standard DPO and RLHF to show that they achieve $\\beta$-perfect alignment. The purpose is to justify the definition of $\\beta$-perfect alignment and establish a connection with existing results. Our theoretical contribution and new insights are discussed in Section 3.2: by factoring a directed acyclic graph (DAG) of components, we prove that **SysDPO achieves the same β‑perfect alignment guarantee at the system level (under Assumption 1), even when the system has a non‑differentiable structure. To our knowledge, this is the first formal alignment guarantee in such compound AI system alignment.**\n    - Moreover, in the LLM+Diffusion model setting, deriving a tractable SysDPO loss function is non-trivial: diffusion models, unlike LLMs, do not directly output the log-likelihood of generated outputs. We address this through a tailored derivation (shown in Appendix D), extending the single-model Diffusion DPO (Wallace et al.). **This is an important and nontrivial technical contribution not emphasized in the main text due to space limitations**.\n    - We will highlight these distinctions in the camera‑ready and move the key part of Appendix D into the main text.\n- **Theorem 1 seems to only show that the statement holds under Assumption 1 and not the necessity of this assumption.**\n    - We thank the reviewer for catching this. Yes, Assumption 1 is sufficient for proving $\\beta$-perfect alignment, but it may not be necessary. In practice, diverse samples often suffice. We will clarify this point and revise the statement in the camera-ready version.\n- **Experiments beyond two components.**\n    - We acknowledge the importance of evaluation on large-scale multi-component systems. **We include a new experiment on a three‑LLM system:** LLM 1 and LLM 2 independently generate responses to the input question, and LLM 3 synthesizes these responses into a final answer. This setup extends our two-LLM system to work with more components. We train this system jointly with SysDPO and evaluate it using the WR‑Prompted metric. Without changing the hyperparameter, the aligned system achieved a 58 % win rate against its pre‑alignment version, showing promising potential for scalability. While broader experiments with many components are beyond our limited academic budget, we believe this result demonstrates that our framework provides a solid foundation for future scaling.\n- **Joint policy optimization vs separate policy optimization.**\n    - Thank you for raising this important distinction. **Even in the two-component setting, we observe that separate DPO underperforms joint SysDPO** as shown in Table 2. In this LLM+LLM setting, LLM 1 drafts an initial response, and LLM 2 refines it. For Separate-DPO, we train each model on the same task‑level dataset (Orca‑DPO‑Pairs) independently. \n    In contrast, SysDPO aligns both models jointly, allowing updates that reflect their collaboration dynamics. This leads to a more coordinated and effective system. We will highlight this difference more explicitly in the camera-ready version, as it demonstrates the necessity of joint alignment.\n\nWe would like to once again thank the reviewer for their constructive comments and for acknowledging the importance of aligning compound AI systems. The clarifications provided above will strengthen the paper both theoretically and empirically.\n\n**References.**\nWallace et al. Diffusion Model Alignment Using Direct Preference Optimization."}, {"review_id": "OgYwuYv5Dl", "reviewer": "Reviewer_7XLb", "summary": "This paper highlights the difficulty of fine-tuning models from preferences when those models are used as modules within a multi-model system. The two running examples are\n\n1. An image-generation system which takes a user prompt, uses an LLM to generate input prompts to an image diffusion model, then produces a sequence of images.\n2. A multi-step LLM system, which uses one LLM forward pass to improve a prompt, then another to respond to the user using that improved prompt.\n\nBecause these handoff points between models are in text, they are not differentiable, so we cannot simply pass gradients through the whole system. Instead, the authors propose to model the system as a Bayesian network where each node is a model (implicitly parameterizing a likelihood function) and the edges map the data flow. This admits a factorization of the overall system's likelihood function which can be used to construct a differentiable loss function. If we have a preference dataset which includes the intermediate models' outputs, we can plug that directly into the DPO loss function; the authors term this algorithm SysDPO-Direct. If we do not have the intermediate outputs, the true likelihood requires marginalizing over them, which is intractable, and the authors propose to approximate the likelihood with samples generated by Diverse Beam Search rather than Monte Carlo approximation, terming this algorithm SysDPO-Sampling.\n\nIn their theoretical analysis, the authors define $\\beta$-Perfect Alignment. This $\\beta$ is a temperature parameter, the inverse of the *inverse*-temperature parameter $\\beta$ from the Boltzmann rational model. My understanding is that if there is an underlying utility function (which I think is an implicit assumption underlying the existence of this sort of preference oracle), then a probabilistic generative model is $\\beta$-perfectly aligned if and only if its outputs match the choice probabilities of a Boltzmann rational agent with inverse temperature $\\beta$ acting with that utility function. If so, this is equivalent to the classic RLHF objective with a rationality parameter (as described in e.g. [1]).\n\nThe paper then argues that both variants of SysDPO achieve perfect alignment in the infinite-data regime (SysDPO-Sampling is only addressed in the case where infinite samples can be drawn and thus the value of the intractable integral over unobserved variables is known).\n\nThe paper concludes with experiments showing SysDPO outperforming some baselines in the two settings described in the running examples. For the image generation setting, this entails a derivation of an optimizable likelihood for the diffusion model.\n\n[1] Hong Jun Jeon, Smitha Milli, and Anca Dragan. 2020. Reward-rational (implicit) choice: A unifying formalism for reward learning. Advances in Neural Information Processing Systems, 33:4415–4426.", "questions": "1. Can you provide any more detailed theoretical analysis of SysDPO-Sampling in the finite sample case?\n2. Can you clarify the statement \"The fact that SysDPO-Direct requires Assumption 1 for perfect alignment, while standard DPO does not, highlights a fundamental challenge of compound AI system alignment.\" ? Since standard DPO doesn't have $\\mathcal Y$, the parallel assumption would be that all outputs have some positive probability to be sampled from $\\mathcal D$, which seems like a necessary condition for standard DPO to achieve perfect alignment. If I'm incorrect, please help me understand that and what fundamental challenge is highlighted here; the later sentences in that same paragraph seem unrelated to that statement.\n3. Can you provide any more firm understanding of what $\\beta$-perfect alignment actually means? The statement that jumps out at me is that it's a generalization of the IIA condition from Luce's choice axioms, to which it corresponds when $\\beta = 1$. Is there some meaningful correspondence that you can speak to there, perhaps including what varying $\\beta$ would mean under that interpretation?\n4. Some additional experiments that would go a long way in convincing me:\n  - A more general multi-image generation task (perhaps just open-ended preferences over sequences of images, or pairs of images)\n  - Aligning multiple differently-structured LLM collaboration systems (i.e. different DAGs) on the same dataset for 5.2\n  - A more significant hyperparameter study on the sampling in SysDPO-sampling\n5. Please clarify the training setup of the Separate-DPO baseline for Experiment 5.2.\n\n---\n\nA couple of significant typos I noticed:\n1. The text says the OC Ratio for SysDPO-Direct in 5.1 was 70%, whereas the table says 73%\n2. In the third line (second implication) of the math in line 575 in the appendix, $z^w$ and $z^l$ are swapped.\n\n---\n\nAll told, this is a great problem to be tackling with such a general-purpose framework, a gerat  and I look forward to further developments on  empirically validating that it can work at a practical scale.", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\nThe framework is very general, and provides an outline for how to jointly optimize any static multi-model system from preferences so long as a likelihood function (or likelihood estimator) is available for each model, whether or not the preference data contains the outputs of the intermediate models.\n\nIn particular, proposing algorithms for the case without intermediate results provides an alignment framework that could work for multiple different systems designed for the same task. One could have input-output pairs for the same task, design several different LLM collaboration pipelines to tackle the task, and use SysDPO-Sampling to align all of them from that single dataset.\n\nThe proofs are rigorous (I read and understood all but the diffusion model derivations in Appendix D) and provide firm confidence in the statements that RLHF, DPO, and SysDPO-Direct achieve $\\beta$-perfect alignment in the infinite-data regime. The related statement for SysDPO-Sampling is not as rigorously argued, but seems true in the infinite-data, infinite-samples regime.\n\nThe derivation of the tractable likelihood function for the diffusion model in Appendix D seems substantial, and I would consider promoting it more centrally as a contribution of the paper:\n1. This specific derivation, if it is novel, could be relevant for other work that needs such a likelihood estimator;\n2. This is an example demonstrating a more general point: that the Bayesian network factorization of the full system's likelihood function doesn't strictly require that we have a likelihood function for each model if we can derive a sufficiently good estimator.\n\nThe experiments seem decently well-precedented and provide some evidence that joint alignment outperforms marginally aligning individual components. They also provide some evidence that SysDPO-Sampling is effective in practice with small sample sizes.\n\n## Weaknesses\n\nI was confused at first by the example in Figure 1. After reading more of the paper I have come to understand the point being made: the second cat in (a) is not visibly more angry than the first cat in (a), whereas the cats in (b) show a more significant progression of anger; this highlights that the models' cooperation is inconsistent. My initial interpretation was that these were both clearly bad examples, because both progressions contain three clearly different cats. As a sanity check I showed the figure to a couple of family members (not in the AI field) and their interpretation was the same. This left me a little bit lost my first time reading the paper as to what practical problem we were grounding the discussion with.\n\nGenerally, the SysDPO-Direct idea feels like a very minor contribution theoretically. There is quite a bit of room to validate its effectiveness and to work through practical issues in scaling it to real practical problems (cf the original RLHF paper), but this work as written up here significantly emphasizes the theoretical aspects of this algorithm.\n\nOn the other hand, while Monte Carlo approximation of the sum when marginalizing over unobserved variables in the Bayesian network is a natural idea, the paper proposes to instead use Diverse Beam Search to generate samples, which is an interesting proposition with some intuitive motivation. On reading that I immediately wondered if that would lead to any bias in the likelihood estimation and whether that would matter in practice. For a paper that lends so much of its space in both main text and appendix to theoretical analysis and derivations, I was disappointed that for the one theoretical question that I found myself wanting an answer to was only addressed by pointing out that since the Diverse Beam Search samples are distinct, drawing infinitely many samples approaches identity and so guarantees perfect alignment. Yes, but the whole motivation is that the sum is intractable, and in the experiments we only draw a single sampled pair for SysDPO sampling. Is there anything we can say about the soundness of using DBS instead of MC in the finite data case?\n\nI struggled to garner much intuitive meaning of $\\beta$-perfect alignment. After reading the paragraph about interpreting that definition and Proposition 1, I arrived at the understanding I wrote in my summary above. But that's a bit circular with Proposition 1, and I find myself feeling that the $\\beta$-perfect alignment definition does not contribute to my understanding at all. Rather, I end up at \"an algorithm achieving perfect alignment means it gets the same result as RLHF/DPO\".\n\nBroadly speaking, I struggled to move my subjective belief significantly based on the results from the Experiments section. The first example feels a bit narrow and contrived -- a more specific task than one tends to run post-training DPO to align to. I would have liked to see something more general to motivate the LLM + image diffusion case. For the second example, I was a bit confused about the Separate-DPO baseline (the most relevant one to the core point of the paper). What data is being used to DPO-tune the two language models? Isn't an assumption here that we don't have factorized preference data that we could use to independently tune the models? I would have also liked to see much more exploration of the impact of the sample size $\\alpha$ in SysDPO-Sampling. What about, rather than sampling four responses then choosing the most contrastive pair, just using more samples?", "was_revised": true, "final_justification": "The clarifications on $\\beta$-perfect alignment are helpful.\n\nOn the finite-sample analysis, I remain hopeful for a stronger theoretical result, as much of the paper is devoted to theoretical analysis and finite-sample SysDPO-Sampling is the core setting and algorithm (since as you say the fundamental challenge is that humans only give preferences over the final outcomes, not the intermediate model outputs). Nonetheless your discussion here along with the new sample-size experiment give some foothold for future work to progress in that direction.\n\nFor practical applications, the new experiment with a multi-LLM system is a good start on proof-of-concept, and improves the paper.\n\nI have increased my score to a 3. There is great work here, and while there are remaining gaps in what I would like to see in a NeurIPS publication on this topic, I fully respect if the other reviewers and ACs find it complete in its current form. Thank you.", "rebuttal": "We sincerely appreciate the reviewer’s insightful feedback and recognition of the compound AI systems alignment problem. To our knowledge, SysDPO is the first principled framework that jointly aligns the parameters of an entire compound AI system modelled as a DAG. For better fluency, we address the reviewer's comments and questions in the order of their appearance in the manuscript. \n\n- **Detailed explanation of $\\beta$‑perfect alignment:** We thank the reviewer for suggesting the perspective of the Boltzmann rational model. **It is very interesting how $\\beta$-perfect alignment can be precisely interpreted from many different perspectives.** Details follow.\n    - $\\beta$**-perfect alignment of generative models.** Given an input $x$, consider two versions of the generated output $z^1$ and $z^2$. Consider a ground truth preference oracle $\\texttt{pref}(z^1\\succ z^2\\mid x)\\in (0, 1)$, which states how much  $z^1$ is preferred over $z^2$.  The model $\\theta^\\*$ is $\\beta$-perfectly aligned to the preference oracle $\\texttt{pref}(\\cdot)$ if for any $x\\in\\mathcal{X}$ and $z^1, z^2 \\in \\mathcal{Z}$ such that $$\n\\texttt{pref} (z^1 \\succ z^2 \\mid x) = \\frac{p _ {\\theta^\\*} (z^1\\mid x)^\\beta}{p_{\\theta^\\*}(z^1\\mid x)^\\beta+p_{\\theta^\\*}(z^2\\mid x)^\\beta} \\qquad\\qquad{(\\text{Definition 1})}\n$$\n    \n    -  **What varying $\\beta$ does**. Suppose we have $\\texttt{pref}(z^1\\succ z^2\\mid x)=0.8$ on data $(x, z^1, z^2)$. Then, given a $\\beta$-perfectly aligned model $\\theta^\\*$ that satisfies the alignment equation (Definition 1). Rearranging the alignment equation: $$\n    \\texttt{pref}(z^1\\succ z^2\\mid x) = \\frac{1}{1+\\left(\\tfrac{p_{\\theta^\\*}(z^2\\mid x)}{p_{\\theta^\\*}(z^1\\mid x)}\\right)^{\\beta}}\n    $$ If $\\beta\\ll 1$ is close to $0$, it would require the ratio $p_{\\theta^\\*}(z^2\\mid x)/p_{\\theta^\\*}(z^1\\mid x)\\ll 1$ for the equation to hold: the aligned model almost always output the most preferred result deterministically. Conversely, if $\\beta \\gg 1$, it would require the ration  $p_{\\theta^\\*}(z^2\\mid x)/p_{\\theta^\\*}(z^1\\mid x)\\approx 1$, i.e., approaching a uniform policy. **Therefore, in this view, $\\beta$ can be interpreted as temperature**. \n    \n    - **The perspective of the Boltzmann rational model**. With our notations, the generative policy $p_{\\theta^\\*}(\\cdot\\mid x)$ of $z$ is Boltzmann rational if $p_{\\theta^\\*}(z\\mid x)\\propto \\exp(r^\\*(x, z)/\\beta)$ where $r^\\*(x, z)\\in \\mathbb{R}$ is a ground-truth reward function of the input-output pair. Note that the $\\beta$ here is inverted compared to conventional Boltzmann rational literature, as the reviewer rightly point out. We can verify that this Boltzmann rational policy is indeed $\\beta$-perfectly aligned: $$\n    \\frac{p_{\\theta^\\*}(z^1\\mid x)^\\beta}{p_{\\theta^\\*}(z^1\\mid x)^\\beta+p_{\\theta^\\*}(z^2\\mid x)^\\beta}=\\frac{\\exp(r^\\*(x, z^1))}{\\exp(r^\\*(x, z^1))+\\exp(r^\\*(x, z^2))}=\\texttt{pref}(z^1\\succ z^2\\mid x)\n    $$ where the last equation is the definition of the Bradly-Terry preference model (Eq. 1 in the main text). **Therefore, $\\beta$ can be interpreted as the inverse of rationality.**    \n    \n    - **The perspective of DPO/RLHF**. As shown in Proposition 1, **the $\\beta$ is precisely the KL regularization strength parameter** appears in both the DPO and RLHF objective.\n- **Clarification on the fundamental challenge of aligning compound AI systems compared to standard single-model alignment**\n    - The fundamental challenge is due to how the preference signal is collected: the user preference is only given to the final output $z$ but not any intermediate outputs $y$. Since the evaluation of a component in a compound system must depend on other components, there is no preference oracle to compare two intermediate outputs directly, i.e., $\\texttt{pref}(y^1\\succ y^2\\mid x)$. Therefore, intermediate samples have to be drawn online constantly (SysDPO-Sampling) or have full coverage (SysDPO-Direct). The fundamental challenge is thus **credit assignment under non‑differentiable, hidden intermediate interactions**, **and the fact that humans only have access and give preferences to the final outputs.**\n- **Finite‑sample analysis for SysDPO‑Sampling with Diverse Beam Search (DBS) and MC Sampling**\n    - Consider a two-generative-model collaboration system (Figure 2 (b)). As shown in Section 2.3, SysDPO-Sampling aims to approximate the intractable generation probability $p_{\\theta}(z\\mid x)=\\sum_{y\\in \\mathcal Y} p_{\\theta_2}(z\\mid y, x)p_{\\theta_1}(y\\mid x)$ with finite samples $\\widehat{\\mathcal{Y}}:=\\\\{ y^\\alpha \\\\} _ \\alpha$ such that approximately  $p_{\\theta}(z\\mid x)\\propto\\sum_{\\alpha} p_{\\theta_2}(z\\mid y^\\alpha, x)p_{\\theta_1}(y^\\alpha\\mid x)$. **This can be viewed as a coreset selection problem.** I.e., the goal is to find a diverse subset $\\widehat{\\mathcal Y}$ of representative points to approximate the full set $\\mathcal Y$. Denoting $\\mu_{\\mathcal Y}:=\\tfrac{1}{|\\mathcal Y|}\\sum_{y\\in \\mathcal Y} p_{\\theta_2}(\\cdot\\mid y, x)p_{\\theta_1}(y\\mid x)$ (similarly for $\\mu_{\\widehat{\\mathcal Y}}$ ), we aim to minimize the approximation error $\\epsilon:=\\|\\mu_{\\mathcal Y}-\\mu_{\\widehat{\\mathcal Y}}\\|$. **Therefore, a strategically selected subset often outperform random MC sampled subset, where the strategy typically involves selecting representative and diverse points [A, B, C].** DBS can be seen as a greedy coreset‑selection algorithm where, at every decoding step, it keeps the subset that jointly maximizes a combined objective of generation likelihood and diversity.\n    - **Toward a finite-sample analysis.** The core of finite-sample analysis is the rate of approximation error $\\epsilon$ w.r.t. the budget $k:=|\\widehat{\\mathcal{Y}}|$. Although it is unclear what the exact error $\\epsilon$ of DBS is, coreset algorithms generally achieve a worst-case guarantee of $\\epsilon=\\mathcal O(\\tfrac{1}{\\sqrt{k}})$ [A]. For example, consider finite sample space $\\mathcal Z$, and denote the sampled probability mass function as $v_y:=p_{\\theta_2}(\\cdot\\mid y, x)p_{\\theta_1}(y\\mid x)\\in \\mathbb{R}^{|\\mathcal Z|}$. Assume all $v_y$ is $\\ell_2$-bounded by $D$. The approximate carathéodory theory [D] states that we can always find a subset $\\widehat{\\mathcal{Y}}\\subset \\mathcal Y$ of size $k$ such that the approximation error  $\\epsilon=\\mathcal O(\\tfrac{D}{\\sqrt{k}})$. To make the bound tighter, we need fine-grained analysis that exploits the concrete structure of the distribution, which is an exciting but non-trivial future work. Nonetheless, in practice, coreset can be significantly better as shown in prior works [A, B, C], as well as our experiments shown next.\n    - **Experimental Settings**. In this newly conducted experiment, we build on the LLM collaboration system in Section 5.2. During training, we sample $k$ intermediate candidates $y^1, \\dots y^k \\sim p_{\\psi_1}(\\cdot | x)$ and use them to calculate the loss. At evaluation time, we eliminate sampling stochasticity by setting the temperature to 0 and deterministically selecting $y^\\* = \\max_y p_{\\psi_1}(y | x)$  and $z^\\* = \\max_z p_{\\psi_2}(z | x)$. We evaluate the system performance based on the quality of $z^\\*$.\n    - **Sampling methods.** For DBS, We obtain $k$ candidates using $k$ beam groups and a diversity penalty of 20; we report results for $k=2,4$. For MC sampling, we independently sample $k$ candidates with temperature 1.0 under different random seeds, evaluating $k\\in\\{2,3,4,5\\}$.\n    - **Results.** The table below shows the win‑rate of each configuration. **DBS outperforms MC with fewer number of sampled candidates**. Both methods are consistent with varying $k$ within the test range, **suggesting that 2 samples may be sufficient without drastically increasing the budget**. Compared to DBS, MC often draws candidates that are near‑duplicates.\n        \n        \n        | Method | k | WR-Prompted (%) |\n        | --- | --- | --- |\n        | DBS | 2 | 68.5 |\n        | DBS | 4 | 68.2 |\n        | MC | 2 | 66.8 |\n        | MC | 3 | 67.0 |\n        | MC | 4 | 66.7 |\n        | MC | 5 | 66.0 |\n- **Experiments on aligning differently-structured LLM collaboration systems.**\n    - **We include a new experiment on a three‑LLM system:** LLM 1 and LLM 2 independently generate responses to the input question, and LLM 3 synthesizes these responses into a final answer. This setup extends our two-LLM system to work with more components. We train this system jointly with SysDPO and evaluate it using the WR‑Prompted metric. Without changing the hyperparameter, the aligned system achieves a 58 % win rate against its pre‑alignment version, showing promising potential for scalability. While broader experiments with many components are beyond our limited academic budget, we believe this result demonstrates a solid foundation for future scaling.\n- **Due to the space limitation, we summarize the remaining clarifications below**. Luce’s choice axioms introduce a probabilistic framework for modeling how people make choices. As we view the generative model $p_{\\theta^\\*}(z\\mid x)$ as a choice model, the axioms (Lemma 3 in Luce et al.) coincide with our Definition 1 with $\\beta=1$. In Experiment 5.2, the Separate-DPO baseline is done by training each model on the same task‑level dataset (Orca‑DPO‑Pairs) independently. In addition, clarification about Figure 1 and suggested typos will be incorporated in the camera-ready version.\n\nWe thank the reviewer again for engaging with our work. We hope these clarifications and additions will strengthen the paper and more clearly convey its contributions to the study of compound AI systems alignment.\n\n**References**\n\n[A] Campbell et al. Automated Scalable Bayesian Inference via Hilbert Coresets.\n\n[B] Zhang et al. Bayesian Coresets: Revisiting the Nonconvex Optimization Perspective.\n\n[C] Sener et al. Active Learning for Convolutional Neural Networks: A Core-Set Approach\n\n[D] Mirrokni et al. Tight Bounds for Approximate Carathéodory and Beyond\n\n Luce et al. Individual choice behavior."}, {"review_id": "dzr2SlU6uk", "reviewer": "Reviewer_sSJH", "summary": "This paper is concerned with the alignment of compound AI systems, which consist of multiple interacting components such as large language models and image generators. Existing alignment methods, such as DPO or RLHF, are not directly applicable to such systems since they involve non-differentiable operations and the preferences cannot be directly related to each of the subcomponents making up the system. The authors model these systems as Directed Acyclic Graphs (DAGs) and propose two system-level alignment techniques based on DPO, called SysDPO-Direct and SysDPO-Sample. The first method requires a preference dataset where intermediate outputs for each system component are available, while this is not required for the second, approximate, method. The proposed methods are evaluated on two tasks and compared against several well-chosen baselines.", "questions": "_Q1_:\nAre there insights into the computational cost of applying the SysDPO-Sample alignment technique?  Including a graph that shows the downstream task performance against the number of intermediate outputs sampled would be very insightful. Ideally, it would be good to report both the SysDPO-Sample and SysDPO-Direct technique for at least one of the applications, such that these methods can also be compared against each other. \n\n_Q2_:\nOne aspect of the paper that was not entirely clear to me, is the role of the parameter beta in (1) the beta-perfect alignment definition and (2) in the DPO loss function. Is this one parameter that occurs in both the theoretical alignment definition and the practical optimization objective? If this link can be made more explicit, that would increase the clarity of section 3. It would also be helpful to address the presentational issues previously addressed. \n\n_Q3_:\nIn order to make stronger claims on the effectiveness of the methods, it would be helpful to report additional, more holistic, system-specific performance metrics for both tasks. Ideally, the methods would be evaluated using a broader set of metrics commonly used in alignment research (such as MixEval scores for the language components of the systems or equivalent task-specific metrics for the other modalities).", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths:  \n\nThe proposed approach introduces a novel and well-motivated formulation for compound AI systems by modeling them as Directed Acyclic Graphs (DAGs).  This structure enables a compact factorization of the system’s output likelihood, making it possible to apply end-to-end, gradient-based alignment. By extending the established Direct Preference Optimization (DPO) framework to system-level alignment, the authors address a timely and practical problem in aligning multi-component AI systems.   \n\nAnother strength of the paper is the diversity of the evaluated modular AI systems. The first is a vision-language pipeline composed of an LLM and an image generator with a tree-like DAG structure. The second is a textual system consisting of two collaborating LLMs with a branched DAG structure. On both tasks, the reported evaluation metrics show the value of joint alignment through the SysDPO framework, compared to no or isolated / individual alignment. \n\n_The paper is well-written, the contributions are intuitive, well-defined, and well-described, supported by valuable empirical results, and I think they are an unsurprising but important evolution in the direction of AI agents of the future._ \n\n## Weaknesses:  \n\nOne key weakness of the proposed methods lies in scalability and data efficiency. The SysDPO-Direct variant requires system-specific preference data that includes not only final system outputs, but also intermediate outputs for each component. In practice, collecting such detailed annotations is likely to be difficult, especially for large or deeply modular systems with many interacting parts. \n\nThe alternative, SysDPO-Sample, avoids this requirement. However, it relies on sampling intermediate outputs to approximate the likelihood of the generated final outputs, which is computationally expensive. Accurately estimating the summation in Equation (5) requires these samples to cover a large, potentially infinite, probability space (the space of all possible intermediate outputs).  In the experiments, only two intermediate samples were drawn, which seems insufficient to reliably approximate this likelihood. This raises concerns about the quality and stability of the approximation, especially for complex systems with many intermediate outputs (potentially in different modalities).   \n\nA limitation of the proposed methods is the restricted applicability to systems that can be expressed as static DAGs. As acknowledged by the authors, many real-world modular AI systems cannot be easily or meaningfully reduced to a fixed DAG structure. For example, in the LLM-Diffusion experiment, the DAG is only valid because the number of generated images is fixed at three. If this number was input-dependent (as in many open-ended generation tasks), the structure of the DAG is no longer static and it is unclear if the alignment techniques can still be applied. More generally, many compound AI systems have dynamic workflows with cycles, feedback loops, etc. It would be meaningful to explore how SysDPO can be extended to such scenarios or discuss more extensively how the method can be adapted in future work to handle dynamic system behavior. \n\nAs indicated in the strengths section, the diversity of the evaluated modular AI systems is solid and the choice of baselines is sensible. However, the empirical evaluation can be elaborated. First, the SysDPO-Sample method is only evaluated on a single task and dataset and the evaluation set consists of only 193 data points (from the 12,900 available data points). Secondly, only two evaluation metrics are reported based on binary preference judgements. It would be meaningful to include additional metrics that are tailored to the evaluated system. \n\nThere are some presentational issues that could be improved. The formalization of compound AI systems as DAGs would benefit from more precision, for example, the meaning of edges is left vague (it is defined as the flow of information), the sets I and J (for the intermediate and final outputs) are not introduced, and it's ambiguous whether each parameter set (the thetas) is associated with a specific node (as presented in the text) or with an entire model component (as in the figure). Also, the motivating example in Figure 1 is not particularly compelling, as the difference between panels (a) and (b) is limited and subjective, making the intended contrast unclear. Additionally, as a minor remark, in Section 5.1, the same work is cited inconsistently as Zhang and Zhuang, which should be corrected.", "was_revised": true, "final_justification": "I aim to keep my score, which seems to be on the higher side among the other reviewers.  However, I think the paper brings novel and interesting contributions (albeit not unexpected in scope, given recent evolutions).  Also, the paper has sufficient technical depth and I think it has the potential to become an interesting reference work in a very popular research direction. Also, I appreciate the efforts the authors have done in order to meet the reviewers' concerns, including my own.  The rebuttal was precies and extensive, and I was happy with the additional insight.  Looking forward to the announced future work (like with more realistic dynamic graph structures, based my weakness on static DAGs), for which I agree it is out of scope for the current paper.", "rebuttal": "We sincerely thank the reviewer for the constructive and encouraging feedback. We will incorporate clarifications and revisions, as well as new experiments, in the camera-ready version. Details follow.\n\n- **Scalability, efficiency, and computational cost.**\n    - We thank the reviewer for highlighting efficiency and scalability. **This is indeed a challenge in our method and also a fundamental challenge of compound AI system alignment.** The fundamental challenge is due to how the preference signal is collected: the user preference is only given to the final output $z$ but not any intermediate outputs $y$.  In other words, it would be much easier if we had a preference oracle to compare two intermediate outputs directly, i.e., $\\texttt{pref}(y^1\\succ y^2\\mid x)$, from which running standard DPO separately on each model would suffice. Unfortunately, this is ill-defined because the evaluation of a component in a compound system must depend on other components, i.e., there is no ground-truth preference oracle of the intermediate outputs that stands independent of the generative system itself.  Therefore, as you rightly point out, intermediate samples have to be drawn online constantly (SysDPO-Sampling) or have sufficient coverage (SysDPO-Direct). **The challenge is thus fundamentally a** **credit assignment problem under non‑differentiable, hidden intermediate interactions**, **and the fact that humans only have access to and give preferences to the final outputs.**\n    - **How we tackle the efficiency challenge.** The core idea of improving the sample efficiency of the intermediate outputs is diversity. For SysDPO-Direct, we adopt a set of diverse styles of prompting to construct the preference dataset (detailed in Appendix F). For SysDPO-Sampling, we adopt diverse beam search (DBS) to improve the diversity of the intermediate outputs.\n        - **We run a new ablation study to show that DBS outperforms Monte Carlo sampling with the same sample budget, highlighting the importance of diversity for efficiency.** We begin by comparing Diverse Beam Search (DBS) with Monte‑Carlo (MC) sampling and then examine how performance scales with the number of intermediate candidates sampled during training. Our experiments reveal two clear trends: (i) within the tested range, increasing the sampling budget $k$ has little to no impact on overall performance; and (ii) DBS achieves higher win‑rates than MC with the same sample count.\n        - **Experimental Settings and sample details**. Building on the LLM collaboration system in section 5.2, each training instance starts from input $x$, model $\\psi_1$, and $\\psi_2$. During training, we sample $k$ intermediate candidates $y^1, \\dots y^k \\sim p_{\\psi_1}(\\cdot | x)$ and leverage them to calculate the loss that jointly optimize $\\psi_1, \\psi_2$. At evaluation time, we eliminate sampling stochasticity by setting the temperature to 0 and deterministically selecting $y^\\* = \\max_y p_{\\psi_1}(y | x)$  and $z^\\* = \\max_z p_{\\psi_2}(z | x)$. We evaluate the system performance based on the quality of $z^\\*$.\n        - **Sampling methods.** For DBS, We obtain $k$ candidates $y^1, \\dots, y^k$ using $k$ beam groups and a diversity penalty of 20; we report results for $k=2,4$. For MC sampling, we independently sample $y^1, \\dots, y^k$ with temperature 1.0 under different random seeds, evaluating $k=\\{2,3,4,5\\}$.\n        - **Results.** The table below summarizes the win‑rate of each configuration. **DBS outperforms MC with the same number of sampled candidates.** Both the methods are consistent with varying $k$ within the test range, **suggesting that 2 samples may be sufficient without drastically increasing the budget.** Compared to DBS, MC often draws candidates that are near‑duplicates.\n        \n        | Method | k | WR-Prompted (%) |\n        | --- | --- | --- |\n        | DBS | 2 | 68.5 |\n        | DBS | 4 | 68.2 |\n        | MC | 2 | 66.8 |\n        | MC | 3 | 67.0 |\n        | MC | 4 | 66.7 |\n        | MC | 5 | 66.0 |\n    - **Computational cost.**  Aligning the system of two LLMs jointly took approximately 30 hours on a single NVIDIA H200 GPU. We acknowledge that broader scaling is out of our limited academic budget and remains open. We therefore see this work as a principled first step that lays the groundwork for future large-scale investigations.\n- **Fixed DAG structure.**\n    - We appreciate the reviewer’s insight and agree that many real-world systems feature dynamic, input-dependent control flow.  **For settings with loops or dynamic branches, we envision two extensions**. (i) **time-unrolling for recurrent interactions.** For systems with loops, one can unroll the interaction over time, yielding a finite-horizon DAG, similar to back-propagation through time. (ii) **stochastic masking.** Some nodes may learn a policy over which downstream modules to invoke, where the random variable is a mask over the node’s out edges. Its probability naturally integrates into the SysDPO framework. Exploring these directions is an exciting future work.\n- **The role of $\\beta$ in $\\beta$-perfect alignment.**\n    - **It is very interesting how $\\beta$-perfect alignment can be precisely interpreted from many different perspectives.** However, we acknowledge that the explanation of the original manuscript can be improved for better clarity. Below, we first give the intuition of $\\beta$-perfect alignment. Then, we discuss how the $\\beta$ is the **same** as the $\\beta$ appearing in the DPO/RLHF objective as the KL divergence.\n        - **$\\beta$-perfect alignment of generative models.** Given an input $x$, consider two versions of the generated output $z^1$and $z^2$. Suppose there is a ground truth preference oracle $\\texttt{pref}(z^1\\succ z^2\\mid x)\\in (0, 1)$, which states how much  $z^1$ is preferred over $z^2$.  The model $\\theta^\\*$ is $\\beta$-perfectly aligned to the preference oracle $\\texttt{pref}(\\cdot)$ if for any $x\\in\\mathcal{X}$ and $z^1, z^2 \\in \\mathcal{Z}$ such that $$\\texttt{pref}(z^1\\succ z^2\\mid x) = \\frac{p_{\\theta^\\*}(z^1\\mid x)^\\beta}{p_{\\theta^\\*}(z^1\\mid x)^\\beta+p_{\\theta^\\*}(z^2\\mid x)^\\beta} \\qquad\\qquad{(\\text{Definition 1})}$$ \n        - **What varying $\\beta$ does**. We can understand the effect of varying $\\beta$ through a concrete example. Suppose we have a preference oracle of $\\texttt{pref}(z^1\\succ z^2\\mid x)=0.8$ on data $(x, z^1, z^2)$. Then, we obtain a $\\beta$-perfectly aligned model $\\theta^\\*$ that satisfies the alignment equation (Definition 1). We may rearrange the alignment equation (Definition 1) to be\n        $$\n    \\texttt{pref}(z^1\\succ z^2\\mid x) = \\frac{1}{1+\\left(\\tfrac{p_{\\theta^\\*}(z^2\\mid x)}{p_{\\theta^\\*}(z^1\\mid x)}\\right)^{\\beta}}\n    $$ If $\\beta\\ll 1$ is close to $0$, it would require the ratio $p_{\\theta^\\*}(z^2\\mid x)/p_{\\theta^\\*}(z^1\\mid x)\\ll 1$ for the equation to hold: the aligned model almost always output the most preferred result deterministically. Conversely, if $\\beta \\gg 1$, it would require the ration  $p_{\\theta^\\*}(z^2\\mid x)/p_{\\theta^\\*}(z^1\\mid x)\\approx 1$, i.e., approaching a uniform policy. **Therefore, in this view, $\\beta$ can be interpreted as temperature**. \n        \n        - **The perspective of DPO/RLHF**. As shown in Proposition 1, the optimal solution $\\theta^\\*$ to the DPO or the RLHF objective function, when the reference model follows a uniform distribution, achieves $\\beta$-perfect alignment. Most interestingly, the $\\beta$ is **precisely** the KL regularization strength parameter that appears in both the DPO and RLHF objective. **Thus, in this view, $\\beta$ can be interpreted as the strength of KL regularization.**\n- Suggested clarification presentation regarding formulating compound AI systems as DAGs, Figure 1 description, and typos will be incorporated in the camera-ready version.\n\nWe thank the reviewer once again for the encouraging feedback and for acknowledging the importance of compound AI systems alignment. We hope these clarifications and additions will strengthen the paper and more clearly convey its contributions to the study of alignment in compound AI systems."}, {"review_id": "eE8Q0p8KkB", "reviewer": "Reviewer_yR8i", "summary": "This paper proposes a way to align compound AI systems, where multiple generative models are used to provide an output for a given input. The authors provide two simple extensions of direct preference optimization (DPO), a standard alignment technique, for compound AI systems. The first approach, SysDPO-Direct, assumes that we have a preference dataset with full intermediate outputs of the system. Then, instead of treating the preference as the comparison between two final outputs, SysDPO-Direct treats the preference as given to the comparison between two sets of intermediate and final outputs. Then, the rest of the training process is identical to DPO. The second approach, SysDPO-Sampling, is a method for cases where we do not have intermediate outputs in the preference dataset. It relies on a sample approximation of the probability of the output conditioned on the input by using high-probability samples. The validity of SysDPO-Direct—the same optimal policy as the standard DPO—has been proved. Two empirical studies show the usefulness of the proposed approach.", "questions": "Please refer to the weakness section. In particular,\n\n- How are the samples in SysDPO-Sampling selected?\n- How does their quality influence the final performance?\n- How much better does the proposed approach perform against the trivial extension of DPO with a single Monte Carlo sample?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n\n- The problem tackled in this paper is an recently important topic.\n- The proposed approach is simple and natural. It is easy to understand.\n- The validity of SysDPO-Direct has been theoretically shown, as well as the requirement (Assumption 1), which may be nontrivial.\n\nWeakness:\n\n- Because the approach is very simple, its technical innovation is limited. The approach of SysDPO-Direct is probably one of the approaches that everyone tries. \n- It was not mentioned clearly how the intermediate samples are generated for SysDPO-Samples. It was also not clear how their quality affects the final performance.\n- Only two experiments are conducted to show the usefulness of the proposed approach. Moreover, a trivial baseline is missing: DPO with a single sample Monte-Carlo to approximate (5). \n- Related works are not mentioned. Are there any papers addressing the alignment of compounded systems?", "was_revised": true, "final_justification": "During the rebuttal phase, the authors provided a comparison with very natural baselines that I suggested in the first review, and successfully showed promising results over the baseline. The proposed approach will be a good baseline for future research in this direction. The reason for the current score (not a higher score) is that the experiments are limited in showing the goodness of the proposed approach as a general framework.", "rebuttal": "We sincerely thank the reviewer for the constructive and encouraging feedback. We will incorporate clarifications and revisions, as well as new experiments, in the camera-ready version. Details regarding your comments follow.\n\n- **Novelty and related works.** \n    - We thank the reviewer for the positive assessment and the request to clarify novelty and related work. Our contribution is **the first framework that *jointly aligns the parameters* of an entire compound AI system, modelled as a Directed Acyclic Graph (DAG).** This formulation lets us derive a DPO-style loss, and we prove the first alignment guarantee for compound AI systems. Different from our scope of parameter-level alignment, prior work primarily explores prompt-based coordination, which is an orthogonal line of research. It is an exciting future work to investigate how the two orthogonal directions interplay with each other, and how to effectively combine them.\n    - The resulting algorithm being conceptually simple is a result of its generality: it can be applied to any compound AI systems with a DAG factorization. Although the general scheme is simple, our implementations involve non-trivial procedures such as a tailored derivation of the tractable loss function for the LLM+Diffusion model system (Appendix D), and diverse sampling strategies used in both SysDPO-direct (Appendix F) and SysDPO-Sampling (Appendix B). Without such diversity-encouraged sampling procedures, MC sampling yields sub-optimal performance, as shown next.\n- **How the intermediate samples are generated and how their quality affects the final performance.** We would like to thank the reviewer for suggesting a more detailed explanation, as this is indeed a key factor for SysDPO-Sampling. We use Diverse Beam Search (DBS) for SysDPO Sampling (detailed in Appendix B), with a goal to improve diversity among the intermediate samples. We conduct a new ablation study to illustrate that the quality and diversity of those samples matter a lot. We begin by comparing Diverse Beam Search (DBS) with Monte‑Carlo (MC) sampling and then examine how performance scales with the number of intermediate candidates sampled during training. Our experiments reveal two clear trends: (i) within the tested range, increasing the sampling budget $k$ has little to no impact on overall performance; and (ii) DBS achieves higher win‑rates under the same sample count.\n    - **Experimental Settings and sample details**. Building on the LLM collaboration system in section 5.2, each training instance starts from input $x$, model $\\psi_1$, and $\\psi_2$. During training, we sample $k$ intermediate candidates $y^1, \\dots y^k \\sim p_{\\psi_1}(\\cdot | x)$ and leverage them to calculate the loss that jointly optimize $\\psi_1, \\psi_2$. At evaluation time, we eliminate sampling stochasticity by setting the temperature to 0 and deterministically selecting $y^\\* = \\max_y p_{\\psi_1}(y | x)$  and $z^\\* = \\max_z p_{\\psi_2}(z | x)$. We evaluate the system performance based on the quality of $z^\\*$.\n    - **Sampling methods.** For DBS, We obtain $k$ candidates $y^1, \\dots, y^k$ using $k$ beam groups and a diversity penalty of 20; we report results for $k=2,4$. For MC sampling, we independently sample $y^1, \\dots, y^k$ with temperature 1.0 under different random seeds, evaluating $k=2,3,4,5.$\n    - **Results.** The table below summarizes the win‑rate of each configuration. **DBS outperforms MC with fewer number of sampled candidates.** Both the methods are consistent with varying $k$ within the test range, **suggesting that 2 samples may be sufficient without drastically increasing the budget.** Compared to DBS, MC often draws candidates that are near‑duplicates.\n        \n        \n        | Method | k | WR-Prompted (%) |\n        | --- | --- | --- |\n        | DBS | 2 | 68.5 |\n        | DBS | 4 | 68.2 |\n        | MC | 2 | 66.8 |\n        | MC | 3 | 67.0 |\n        | MC | 4 | 66.7 |\n        | MC | 5 | 66.0 |\n    - **Examples of the intermediate samples.** Here are some examples of intermediate outputs generated using different sampling methods for the same questions.\n        - Case 1:\n        \n        > *Question:*  “Extract the answer to the following question from the movie plot. If the question isn't answerable, please output \"Can't answer ……”:\n\n        > *DBS Sampling:* \n        >\n        >  $y^1$: Haymitch Abernathy's arm.\n        >\n        >  $y^2$: Can't answer. The question is not answerable as it does not provide a specific scene or moment in the movie where Johanna cuts from Katniss's arm. The plot …\n        \n        > \n        > *MC  Sampling:*\n        >\n        > $y^1$: Haymitch Abernathy.\n        >\n        > $y^2$: Haymitch Abernathy.\n        > \n        - Case 2:\n        \n        > *Question:* Here is some data about a restaurant: name = The Golden Palace, eatType = coffee shop, food = French, priceRange = £20-25, customer rating = high, area = city centre. Write a sentence that includes the following data about a restaurant\n        \n        > *DBS Sampling:* \n        >\n        > $y^1$: The Golden Palace is a fancy coffee shop in the city centre that serves delicious French food for a price range of £20-25, and it has a high customer rating, making it a great place to enjoy a meal with friends or family.\n        >\n        > $y^2$: There is a restaurant called The Golden Palace located right in the city centre, where they serve delicious French cuisine at a price range of £18-22 per person, and customers have given it a high rating for its excellent service and cozy atmosphere.\n        \n        > *MC Sampling:*\n        >\n        > $y^1$: The Golden Palace is a fancy coffee shop located in the heart of the city center, serving delicious French cuisine for around £20-25 per person, with a high rating from satisfied customers who love dining there.\n        >\n        > $y^2$: The Golden Palace is a fancy coffee shop located in the heart of the city centre, where they serve delicious French cuisine at a moderate price range of £20-25 per person, and customers highly rate their experience.\n        > \n        \n        The MC samples are nearly identical, whereas DBS supplies two semantically distinct options. This illustrates how DBS enhances sampling diversity and, in turn, system performance.\n        \n- **A baseline of DPO with Monte-Carlo sampling.** We would like to thank the reviewer for suggesting this experiment. This new ablation study, as shown above, shows that MC sampling underperforms DBS even with more samples. We start the sample size with $k=2$, as a single sample $k=1$ would have the probability terms $p_{\\theta_i}(y^\\alpha_i\\mid \\texttt{Pa}(y^\\alpha_i))$ canceled out in the SysDPO-Sampling (Eq. 9 in Appendix B). \n\nWe would like to once again thank the reviewer for their constructive comments and for acknowledging the importance of aligning compound AI systems. We hope these clarifications and additions will strengthen the paper and more clearly convey its contributions to the study of alignment in compound AI systems."}]

Author rebuttal:

Meta-review: The paper studies how workflows consisting of multiple non-differentiable components can be jointly optimized using datasets of preferences over the end outputs of the workflow. The paper views the workflow as a Bayesian graphical model and derives likelihood estimators for each component, which allows them to define a differentiable loss function over the entire workflow and optimize via a variant of DPO.

All the reviewers agreed that the problem is well-motivated and significant, and the proposed approach is novel. The reviewers raised several questions and gave constructive suggestions to improve the experimental evidence; the authors incorporated the feedback, especially the additional experiments comparing to Monte Carlo sampling, evaluating on a larger (3-component) workflow, and clarifications on the derivations substantially strengthened the paper. No reviewer objects to the paper being accepted when revised with the additional experiments and discussions.
Decision: Accept (poster)