Synergistic Weak-Strong Collaboration by Aligning Preferences
Authors: Yizhu Jiao, Xuchao Zhang, Zhaoyang Wang, Yubo Ma, Zhun Deng, Rujia Wang, Chetan Bansal, Saravan Rajmohan, Jiawei Han, Huaxiu Yao
Accepted at: acl25

Synergistic Weak-Strong Collaboration by Aligning Preferences

Yizhu Jiao1,2, Xuchao Zhang1, Zhaoyang Wang3, Yubo Ma4, Zhun Deng3, Rujia Wang1, Chetan Bansal1, Saravan Rajmohan1, Jiawei Han1, Huaxiu Yao1,3
1Microsoft Research, 2University of Illinois Urbana-Champaign,
3University of North Carolina at Chapel Hill, 4Nanyang Technological University
yizhuj2@illinois.edu, xuchazhang@microsoft.com, huaxiu@cs.unc.edu

Abstract
Current Large Language Models (LLMs) excel in general reasoning yet struggle with specialized tasks requiring proprietary or domain-specific knowledge. Fine-tuning large models for every niche application is often infeasible due to black-box constraints and high computational overhead. To address this, we propose a collaborative framework that pairs a specialized weak model with a general strong model. The weak model, tailored to specific domains, produces initial drafts and background information, while the strong model leverages its advanced reasoning to refine these drafts, extending LLMs’ capabilities to critical yet specialized tasks. To optimize this collaboration, we introduce a collaborative feedback to fine-tunes the weak model, which quantifies the influence of the weak model’s contributions in the collaboration procedure and establishes preference pairs to guide preference tuning of the weak model. We validate our framework through experiments on three domains. We find that the collaboration significantly outperforms each model alone by leveraging complementary strengths. Moreover, aligning the weak model with the collaborative preference further enhances overall performance. The code is publicly available.

1 Introduction
The rapid evolution of Large Language Models (LLMs) (Zhao et al., 2023; Chang et al., 2024) has exhibited remarkable proficiency in general reasoning (Kojima et al., 2022; Zheng et al., 2023), problem-solving (Lewkowycz et al., 2022; Yao et al., 2024), and natural language understanding (Wei et al., 2022a). These models have demonstrated the ability to perform a broad range of tasks across diverse domains, often with minimal task-specific training. However, their immense size and general-purpose training can make them less effective in specialized tasks or domains that are under-represented in their training data or require access to proprietary information (Fu et al., 2023). This limitation poses a significant challenge: how can we extend the problem-solving spectrum of LLMs to encompass these niche but critical tasks?

Directly training or fine-tuning large models for every specific domain or task is often impractical due to the following two key reasons. First, some popular LLMs (e.g., GPT-4 (Achiam et al., 2023), Gemini (Team et al., 2023)) are black-box models, with their internal parameters inaccessible for modification. Even when fine-tuning is possible, it can be costly and raises concerns about scalability as models continue to grow in size, such as those models exceeding 70 billion parameters. Additionally, fine-tuning LLMs on private data can pose security and privacy risks. Specifically, fine-tuning requires exposing the model to potentially sensitive data, which could inadvertently be memorized or leaked through the model’s outputs. This exposure creates a risk of violating data privacy regulations and necessitates robust measures to ensure data confidentiality and compliance.

To overcome these challenges, we aim to leverage a collaborative framework that synergizes a

![Comparison of our method and the related work during training and inference.](page_1016_384_393_246.png)

Figure 1: Comparison of our method and the related work during training and inference .
small-sized weak model with a large-sized strong model. In this paradigm, the weak model is tailored with specialized problem-solving abilities in specific domains. Conversely, the strong model boasts robust general capabilities, excelling in tasks that require broad knowledge and advanced reasoning. By orchestrating a collaboration between these two models, we leverage their complementary strengths to tackle specific tasks more effectively than either could achieve independently. The weak model contributes domain-specific insights and preliminary solution drafts, while the strong model refines these drafts using its advanced reasoning capabilities.

While a few existing works have explored forms of weak and strong model collaboration (Juneja et al., 2023; Shen et al., 2024), they often predefine the interaction mechanisms—for example, the strong model directly receives knowledge pieces in a predefined form generated by the weak model (Juneja et al., 2023). However, the most effective interaction strategy can vary depending on the specific scenario or models involved. Moreover, prior approaches typically focus on individual feedback, where the single model provides the feedback to finetune another model. They overlook the potential benefits of feedback from the collaboration procedure (as shown in Figure 1), which helps the weak model understand the strong model’s preferences and to enhance the mutual cooperation between the two models.

In this paper, we introduce an innovative framework for dynamic weak-strong model collaboration. Our approach harnesses the specialized knowledge of a knowledge-intensive weak model to generate detailed initial drafts and background information. The strong model then applies its robust general reasoning capabilities to refine these drafts, effectively merging the strengths of both models. To further optimize this collaborative interaction, we implement a feedback loop, which fine-tunes the weak model based on the strong model’s preferences, creating an adaptive and synergistic interaction that continuously improves. We evaluate the impact of the weak model’s contributions on collaborative performance and construct preference pairs for preference tuning the weak model. This data-driven strategy allows us to amplify beneficial contributions from the weak model and minimize detrimental ones, thereby fostering a mutually beneficial interaction.

We validate our framework through experiments on three datasets, yielding several key findings: (1) Significant Performance Gains through Collaboration: The collaboration between the weak and strong models significantly outperforms each model operating independently, demonstrating the effectiveness of leveraging complementary strengths. (2) Effectiveness of Finetuning Weak Model with Strong Counterpart Preference for Mutually Beneficial Interaction: Incorporating feedback from the strong model to fine-tune the weak model enhances the overall effectiveness of the collaboration. This iterative refinement allows the weak model to align closely with the strong model’s preferences and reasoning patterns. (3) Enhanced Gains with Strong Models of High General Capability: The collaborative gains are substantial when the strong model possesses sufficient general abilities. Merely having a strong model that is better than the weak model does not guarantee mutual improvement; the strong model’s capacity to correct the weak model’s outputs is critical.

2 Related Work

2.1 Enhancing LLMs for Solving Specialized Problems

Addressing specialized problems beyond the generalist training of LLMs has been a key research focus. A common approach is retrieval-augmented generation, where an LLM queries external sources for domain-specific information to enhance responses (Guu et al., 2020; Izacard et al., 2022; Sun et al., 2023; Jiang et al., 2023b; Zhang et al., 2024b). However, these methods often provide static context for LLM to generate responses without further refinement or learning from that context. This static nature can lead to less adaptability in complex, evolving problem-solving scenarios. Another approach uses small models for domain-specific processing to guide LLMs. This includes weak-to-strong generalization, where a strong model learns from the supervision of a weaker one (Burns et al., 2024; Charikar et al., 2024; Yang et al., 2024; Guo and Yang, 2024; Zheng et al., 2024; Sun et al., 2024). However, this often requires access to the strong model’s parameters, which can be a challenge for black-box systems. Other methods prompt LLMs with the outputs of small models to enhance performance on niche tasks (Xu et al., 2024; Liu et al., 2024). Additionally, small models can act as intermediaries by identifying relevant context or splitting problems into subtasks, thereby reducing complexity for the larger model
(Juneja et al., 2023; Shen et al., 2024). While these methods improve LLM performance on specialized tasks, they often rely on static interaction schemes, limiting the weaker model to retrieval or prompting. In contrast, we introduce a dynamic feedback loop between weak and strong models, fostering adaptive collaboration that evolves with the task.

2.2 Multi-Model Collaboration

Model collaboration explores the effective utilization of the collaborative strengths of multiple LLMs These works are generally classified into three categories: Merging, Ensemble and Alignment (Lu et al., 2024). Model merging combines the parameters of various LLMs into a cohesive model, requiring compatibility of parameters within a linear framework (Szymanski and Lemmon, 1993; Fedus et al., 2022; Jiang et al., 2024; O’Brien and Lewis, 2023; Deng and Raffel, 2023; Ji et al., 2024). Moreover, model ensemble leverages the outputs of different LLMs to produce unified outcomes, focusing less on the parameters of the individual models (Shnitzer et al., 2023; Jiang et al., 2023a; Srivatsa et al., 2024). But previous research typically concentrated on interactions between models of comparable size or employed a fixed interaction mechanism for poor task adaptation. Furthermore, alignment can be viewed as a specialized form of model collaboration, where a specialized reward model provides feedback to guide a target model toward desired objectives (Tao and Li, 2024; Lee et al., 2024). However, these approaches typically focus on individual feedback and overlook the potential benefits of feedback from the collaboration procedure, which can enhance mutual cooperation between the two models.

3 The Proposed Method - CoWEST

We introduce CoWEST, a Collaboration method between Weak and Strong models that harnesses their complementary strengths and align the weak model with collaborative feedback to improve collaboration performance. Specifically, during inference, our approach harnesses the specialized knowledge of a knowledge-intensive weak model to generate detailed initial drafts and background information. The strong model then applies its robust general reasoning capabilities to refine these drafts, effectively merging the strengths of both models. To further optimize this collaborative interaction, during training, we implement a feedback loop, which fine-tunes the weak model based on the strong model’s preference, creating an adaptive and synergistic interaction that continuously improves. We construct preference pairs by evaluating the impact of the weak model’s contributions on collaborative performance. Then, we adopt direct preference optimization to align the weak model. This data-driven strategy allows us to amplify beneficial contributions from the weak model and minimize detrimental ones, thereby fostering a mutually beneficial interaction. The pesudo codes of training and inference are Algorithm 1 and Algorithm 2 in the appendix.

3.1 Problem Setup

We propose a collaborative approach that leverages both weak and strong models to tackle diverse reasoning tasks. These tasks require domain-specific knowledge, problem-solving skills, and strong general capabilities such as reasoning, comprehension, and calculation. To address these tasks, we employ a weak model (e.g., Llama2-7b), denoted as \( \pi_w \). This relatively small, cost-efficient model is a white-box system that can be fine-tuned for specific domains to acquire task-relevant knowledge. Alongside this, we utilize a strong model (e.g., GPT-4), referred to as \( \pi_s \), a black-box model with fixed internal parameters. Although it has limited access to specific knowledge or proprietary data, the strong model excels in general reasoning.

Given a user query \( x \) from a target task, our objective is to enhance the overall inference capability by utilizing the complementary strengths of \( \pi_w \) and \( \pi_s \). The inference process is formulated as:

\[
y^* = \mathcal{F}(\pi_w \circ x, \pi_s \circ x, x) \quad \forall x \in X,
\]

where \( y^* \) represents the final output for the query \( x \), and \( \mathcal{F} \) is the mechanism that integrates the domain-specific expertise of \( \pi_w \) with the general reasoning capability of \( \pi_s \), for improved task performance.

3.2 Supervised Fine-tuning of the Weak Model

The weak model \( \pi_w \) is initially fine-tuned on a task-specific training dataset, \( \mathcal{D}_{\text{SFT}} = \{(x, \hat{y})\} \), where each query \( x \) has a corresponding ground truth \( \hat{y} \). The goal of this fine-tuning is to adapt \( \pi_w \) to the specific task by learning from these examples. This is achieved by optimizing the following objective:

\[
\pi_{\theta}^{\text{SFT}} = \arg\min_{\theta} \mathcal{L}_{\text{SFT}} (\pi_{\theta}; \mathcal{D}_{\text{SFT}}),
\]
where \( \pi_{\theta}^{\text{SFT}} \) is the policy after fine-tuning, and \( \mathcal{L}_{\text{SFT}} \) is the supervised loss function to minimize the negative log-likelihood:

\[
\mathcal{L}_{\text{SFT}}(\pi_{\theta}) = -\mathbb{E}_{(x, y) \sim \mathcal{D}}[\log \pi_{\theta}(y \mid x)]
\]

This optimization allows the weak model to specialize in the task domain, preparing it for effective collaboration with the strong model.

3.3 Aligning the Weak Model with Strong Model Feedback

This subsection describes how to align the weak model with feedback from the strong model. Preference triplets are constructed by comparing the outputs produced solely by the strong model with those generated in collaboration with the weak model. An external evaluator scores these outputs based on reasoning coherence and alignment with the ground truth, identifying instances where the weak model’s contributions improve the final result. These triplets are then used to fine-tune the weak model through preference optimization, aligning it with the strong model’s preferences to facilitate better collaboration.

3.3.1 Preference Feedback from the Strong Model

Given a set of training data, \( \{(x, \hat{y})\} \), where \( x \) is the query and \( \hat{y} \) the groundtruth, our goal is to construct preference triplets \( (x, y_+, y_-) \), where \( y_+ \) and \( y_- \) represent the preferred and non-preferred outputs of the weak model. These triplets indicate whether the weak model’s output enhances the final result in its collaboration with the strong model.

To construct these preference triplets, we introduce two generation scenarios:

• **Strong Model Only**: The query \( x \) is directly fed into the strong model, which generates an explanation and a final output using a chain-of-thought (CoT) prompt. This approach helps the model break down complex tasks into intermediate reasoning steps. The resulting output is denoted as \( z \sim \pi_s(z \mid x) \).

• **Weak-Strong Model Collaboration**: The query \( x \) is first processed by the weak model to produce an explanation and an initial result, \( y \sim \pi_w(y \mid x) \). This output, along with the original query, is then passed to the strong model for refinement, resulting in the final response \( y^* \sim \pi_s(y^* \mid y) \). Here, the weak model’s explanation may contain knowledge-intensive information that the strong model analyzes to detect potential flaws or gaps in reasoning.

Preference Evaluation To assess the output quality, we introduce an external evaluator, \( E(y, x) \), which is a large language model with strong general capabilities (e.g., GPT-4). While various models can serve as the evaluator, using the same large language model as the strong model ensures consistency in reflecting the strong model’s preferences. The evaluator scores the outputs based on a manually defined rubric: (1) Coherence of reasoning logic: whether the explanation is logically sound. (2) Consistency with ground truth: how closely the final result aligns with the ground truth.

The evaluator \( E \) assigns a fine-grained score to each output, providing a nuanced assessment of both the reasoning process and the final result. This model-based evaluation approach is preferred over traditional metrics like BLEU or ROUGE, as it captures not just surface similarity but also the depth of reasoning and logical coherence.

Preference Data Construction For each query \( x \), we construct the preference triplet \( (x, y_+, y_-) \) by comparing the evaluation scores of the strong model’s output, \( z \sim \pi_s(z \mid x) \), and the collaborative output, \( \pi_s \circ y \). The preference is determined by the difference:

\[
\Delta = E(\pi_s \circ y, x) - E(z, x).
\]

If \( \Delta > 0 \), the weak model’s contribution is deemed beneficial, and its output \( y \) is selected as the positive response \( y_+ \). Conversely, if \( \Delta \leq 0 \), \( y \) is designated as the negative response \( y_- \). The preference data is formalized using two conditional probability
distributions over the weak model’s outputs:

\[
p_+(y_+ \mid z, x) = \frac{\pi_w(y_+ \mid x) \mathbb{I} \left\{ E(\pi_s \circ y_+, x) > E(z, x) \right\}}{\int \pi_w(y \mid x) \mathbb{I} \left\{ E(\pi_s \circ y, x) > E(z, x) \right\} dy}.
\]

\[
p_-(y_- \mid z, x) = \frac{\pi_w(y_- \mid x) \mathbb{I} \left\{ E(\pi_s \circ y_-, x) \leq E(z, x) \right\}}{\int \pi_w(y \mid x) \mathbb{I} \left\{ E(\pi_s \circ y, x) \leq E(z, x) \right\} dy}.
\]

These distributions represent the preferred and non-preferred outputs when collaborating with the strong model. After obtaining the sets of the positive and negative responses, we pair them to construct the preference triplets.

3.3.2 Preference Tuning for the Weak Model
Using the constructed preference triplets \( \mathcal{D}_{PT} = \{(x, y_+, y_-)\} \), we fine-tune the weak model \( \pi_w \) to align its outputs with those that are preferred in collaboration with the strong model. We employ Direct Preference Optimization (DPO) (Rafailov et al., 2023) to adjust the weak model’s policy \( \pi_w \). The DPO objective is formulated as :

\[
\mathcal{L}_{DPO} = \min_{\pi_w} -\mathbb{E}_{x, z \sim \pi_s(z|x), \atop y_+ \sim p_+(\cdot|z,x), y_- \sim p_-(\cdot|z,x)} \log \sigma \left( \alpha \log \frac{\pi_w^*(y_+ \mid x)}{\pi_w(y_+ \mid x)} - \alpha \log \frac{\pi_w^*(y_- \mid x)}{\pi_w(y_- \mid x)} \right)
\]

where \( \sigma(\cdot) \) is the logistic sigmoid function, and \( \alpha \) is a scaling parameter. By optimizing this objective, we encourage the weak model to generate outputs that lead to higher scores when refined by the strong model.

The overall objective is to find the optimal policy:

\[
\pi_w^* = \arg \min \mathcal{L}_{DPO}(\pi_w; \pi_w^{SFT}; \mathcal{D}_{PT}),
\]

where \( \pi_w^* \) is the optimal policy aligned with the strong model’s preferences, and \( \pi_w^{SFT} \) is the reference weak model obtained through supervised fine-tuning.

3.4 Collaborative Inference
During inference, the input query \( x \) is first processed by the weak model \( \pi_w^* \) to generate an initial output. This output, along with the original query, is then passed to the strong model \( \pi_s \) for refinement, resulting in the final answer:

\[
y^* = \pi_s \circ (x, \pi_w^* \circ x).
\]

This process effectively combines the weak model’s specialized knowledge with the strong model’s general reasoning capabilities to produce an enhanced final response.

3.5 Theoretical Insight
In this section, we build on the methodology discussed earlier to present a formal theoretical analysis of how the proposed preference-based alignment affects the weak model’s behavior and performance. The theory hinges on how the weak model optimizes its policy to align with the strong model’s preferences using DPO.

For simplicity, we assume that the evaluator scores for the strong model’s outputs are constant for all \( z \), i.e. \( E(z, x) = p(x) \) for all \( z \) when given \( x \). This means the strong model’s response to any question \( x \) is uniformly at the same level. Under this assumption, we aim to understand the behavior of the newly optimized weak model \( \pi_w^* \).

Regarding the optimization objective \( \mathcal{L}_{DPO} \) in Section 3.3.2, the key aspect is that the positive (\( p_+(\cdot|z,x) \)) and negative (\( p_-(\cdot|z,x) \)) responses have disjoint support. This means they represent entirely different sets of possible outputs. As a result, the optimized weak model \( \pi_w^* \) allocates zero probability to any output \( y \) that results in an evaluator score \( E(\pi_s \circ y, x) \leq p(x) \). This finding implies:

\[
\pi_w^*(y \mid x) = 0, \forall y \text{ such that } E(\pi_s \circ y, x) \leq p(x).
\]

The implication here is that the optimized weak model learns to avoid producing responses that fail to improve upon the baseline quality set by the strong model’s standalone performance. Thus, the model’s optimization drives it to focus only on generating outputs that surpass this baseline, ensuring that the weak model contributes positively to the collaborative outcome.

Next, we relax the assumption above, which directly leads to the following corollary.

**Corollary 1:** Assuming the strong model’s responses are not just uniform but also bounded below by some quality threshold: \( p(z) \leq E(z, x) \) for all \( z \), the newly optimized weak model \( \pi_w^*(x) \) will strictly avoid producing any response \( y \) for which the collaborative evaluation score fails to exceed the baseline:

\[
E(\pi_s \circ y, x) \leq p(x).
\]

The proof idea is exactly as the analysis above. In addition, this means that the weak model, through
<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th rowspan="2">Models</th>
    <th colspan="2">Counterfactuals</th>
    <th colspan="2">Medicine</th>
    <th colspan="2">Ethics</th>
  </tr>
  <tr>
    <th>EM</th>
    <th>F1</th>
    <th>Acc.</th>
    <th>F1</th>
    <th>Acc.</th>
    <th>F1</th>
  </tr>
  <tr>
    <td rowspan="3">Weak Only</td>
    <td>Llama-3-8B</td>
    <td>68.57</td>
    <td>71.85</td>
    <td>59.48</td>
    <td>46.99</td>
    <td>38.10</td>
    <td>36.40</td>
  </tr>
  <tr>
    <td>Llama-3-8B (SFT)</td>
    <td>69.71</td>
    <td>72.69</td>
    <td>73.08</td>
    <td>58.26</td>
    <td>64.29</td>
    <td>62.40</td>
  </tr>
  <tr>
    <td>Llama-3-8B (Self-Refine)</td>
    <td>70.52</td>
    <td>72.99</td>
    <td>60.71</td>
    <td>47.07</td>
    <td>35.89</td>
    <td>35.06</td>
  </tr>
  <tr>
    <td rowspan="3">Strong Only</td>
    <td>GPT-4</td>
    <td>49.44</td>
    <td>60.93</td>
    <td>65.87</td>
    <td>54.86</td>
    <td>36.75</td>
    <td>35.25</td>
  </tr>
  <tr>
    <td>GPT-4 (CoT)</td>
    <td>57.42</td>
    <td>65.60</td>
    <td>71.80</td>
    <td>57.69</td>
    <td>39.00</td>
    <td>39.58</td>
  </tr>
  <tr>
    <td>GPT-4 (Self-Refine)</td>
    <td>61.54</td>
    <td>68.83</td>
    <td>72.95</td>
    <td>59.10</td>
    <td>38.16</td>
    <td>37.87</td>
  </tr>
  <tr>
    <td rowspan="2">RAG</td>
    <td>SKR (<a href="https://arxiv.org/abs/2303.16487">Wang et al., 2023</a>)</td>
    <td>59.75</td>
    <td>68.33</td>
    <td>71.90</td>
    <td>56.37</td>
    <td>56.46</td>
    <td>55.40</td>
  </tr>
  <tr>
    <td>FLARE (<a href="https://arxiv.org/abs/2303.16487">Jiang et al., 2023b</a>)</td>
    <td>62.07</td>
    <td>70.59</td>
    <td>72.40</td>
    <td>58.89</td>
    <td>55.27</td>
    <td>54.97</td>
  </tr>
  <tr>
    <td rowspan="4">Collaboration</td>
    <td>RLWF (<a href="https://arxiv.org/abs/2401.07417">Tao and Li, 2024</a>)</td>
    <td>70.52</td>
    <td>75.04</td>
    <td>72.01</td>
    <td>57.65</td>
    <td>64.85</td>
    <td>62.10</td>
  </tr>
  <tr>
    <td>RLAIF (<a href="https://arxiv.org/abs/2401.07417">Lee et al., 2024</a>)</td>
    <td>71.69</td>
    <td>72.17</td>
    <td>71.91</td>
    <td>57.55</td>
    <td>62.74</td>
    <td>59.65</td>
  </tr>
  <tr>
    <td>SuperCLC (<a href="https://arxiv.org/abs/2401.07417">Xu et al., 2024</a>)</td>
    <td>68.85</td>
    <td>74.82</td>
    <td>73.64</td>
    <td>58.33</td>
    <td>66.18</td>
    <td>63.86</td>
  </tr>
  <tr>
    <td>CoWest (Ours)</td>
    <td><b>75.85</b></td>
    <td><b>77.34</b></td>
    <td><b>75.10</b></td>
    <td><b>60.13</b></td>
    <td><b>68.33</b></td>
    <td><b>65.61</b></td>
  </tr>
</table>

Table 1: Experiment results across three datasets. Results are reported as Exact Match (EM) and F1 scores for the Counterfactual dataset, Accuracy (Acc) and F1 for the Medical and Ethics datasets. Here, CoT is Chain of thought (Wei et al., 2022b) while Self-Refine means iterative refining LLM with self-feedback from Madaan et al. (2024).

preference optimization, learns to consistently produce only those responses that align with or surpass the evaluator’s expectations. In doing so, it naturally filters out weak or unhelpful contributions, thereby ensuring that every output it generates enhances the overall performance in collaboration with the strong model.

4 Experiment

4.1 Experiment Setting

Dataset We evaluate our framework on three specialized datasets from different domains: (1) Counterfactuals: IfQA (Yu et al., 2023) is a counterfactual QA benchmark with questions based on hypothetical “if” clauses, requiring reasoning about imagined situations that may contradict factual knowledge. (2) Medicine: MedMCQA (Pal et al., 2022) is a multiple-choice QA dataset of real-world medical exam questions, demanding in-depth language understanding and reasoning. (3) Ethics: Prosocial-Dialog (Kim et al., 2022) is a multi-turn English dialogue dataset covering diverse unethical content, with responses classified by safety levels. More details can be found in Appendix B.1.

Evaluation Metrics For IfQA, an open-ended question answering task, we use two commonly used metrics to evaluate the performance: exact match (EM) and F1 score following the setting of previous work (Sachan et al., 2023; Yu et al., 2023). For MedMCQA, a multi-choice question answering task, we use accuracy as the primary evaluation metric. Additionally, we consider using macro-averaged F1 score to capture the model’s performance across all answer categories. For Prosocial-Dialog, a classification task, we utilize macro-F1 scores and accuracy as evaluation metrics to assess the model’s capability in classifying responses based on prosocial behaviors.

Implementation Details In our experiments, we utilize two models: the weak model, LLaMA3-8B (Dubey et al., 2024), and the strong model, GPT-4-0613 (Achiam et al., 2023) for Counterfactuals and Medicine and GPT-3.5-Turbo for Ethics. For the evaluator, we use the same model as the strong model. For the fine-tuning of the weak model, we employ Low-Rank Adaptation (LoRA) (Hu et al., 2021). We generate 2,000 pieces of data for IfQA and 5,000 pieces for MedMCQA and Prosocial-Dialog. More details of model training and prompt design can be found in Appendix B.2.

Baselines The baselines include the following categories: (1) Weak Model: We employ both weak and strong models alone. For weak models, we include Llama-3-8B (Dubey et al., 2024), Llama-3-8B-SFT and Llama-3-8B with Self-Refine (Madaan et al., 2024). (2) Strong Model: we test zero-shot GPT-3.5-Turbo-0613 and GPT-4-0613, including their variants with chain-of-thought (Wei et al., 2022b) and Self-Refine (Madaan et al., 2024). (3) Retrieval-Augmented Generation: We in-
Figure 3: Analysis of different interaction strategies between weak-strong models in CoWEST indicates alignment boosts the performance obviously. We report EM for Counterfactuals and Accuracy for Medicine and Ethics.

Figure 4: Analysis of different sizes of training data for preference tuning.

volve two methods, SKR (Wang et al., 2023) and FLARE (Jiang et al., 2023b). For fair comparison, we adopt GPT-4 as their backbone. (4) Model Collaboration: There are three prior methods, including SuperICL (Xu et al., 2024), RLWF (Tao and Li, 2024) and RLAIF (Lee et al., 2024). Please find more details in Appendix B.3.

4.2 Experiment Result

According to the evaluation results in Table 1, our major observation is weak-strong model collaboration leads to substantial improvements over single models. Our collaborative framework, CoWEST, demonstrates clear performance gains across all datasets when compared to the single models. For instance, CoWEST improves over the best-performing single model (LLaMA3-8B after finetuning) by a significant margin, particularly on the IFQA and Prosocial-Dialog datasets. This underscores the effectiveness of combining a specialized weak model with a general-purpose strong model, allowing each to compensate for the other’s limitations. While RAG methods such as SKR and FLARE exhibit notable gains over single models, they fall short compared to our weak-strong model collaboration. Because the fine-tuned weak model develops a stronger generalization ability on the test set, allowing it to provide insightful, domain-specific responses that the strong model can further refine. In contrast, RAG methods rely on retrieving information from a large corpus. It lacks the adaptability needed for specialized tasks.

Figure 5: Analysis of adopting different weak or strong models in COWEST while fixing the other.

4.3 Analysis

Interaction strategies between weak-strong models In our experiments, we examine two key interaction strategies between weak and strong models: (1) W/o Alignment, where the weak model generates initial responses that the strong model then refines without any model alignment, and (2) With Alignment, which involves fine-tuning the weak model based on the strong model’s preferences. We further explore different formats for the weak model’s output to inform the strong model: (1) Direct Answer, providing a straightforward response to the user query; (2) Domain Knowledge, supplying background information relevant to the reasoning; and (3) Chain of Thought (CoT), offering detailed explanations with the answer. By combining these two interaction strategies with the
three formats, we assess each combination’s effectiveness in handling specialized tasks. We report the EM scores for Counterfactuals and the accuracy scores for Medicine and Ethics.

As shown in Figure 5, our experiments clearly demonstrate the effectiveness of model alignment across all three datasets comparing the results with and without alignment. Particularly, the Chain of Thought (CoT) format stands out as the most advantageous, surpassing both Direct Answer and Domain Knowledge formats. Its detailed reasoning path significantly assists the strong model in handling complex queries, as seen in its stronger performance on the ethics and counterfactual datasets—both of which demand advanced reasoning. In contrast, the medicine dataset, requiring substantial domain knowledge, shows less variation across interaction strategies, suggesting that task-specific expertise can outweigh interaction style when the knowledge requirement is paramount.

Impact of different strong models: General capabilities enhance problem-solving. In this setup, we standardized the strong model for specific domains. Llama-3-8B served as the weak model across all datasets, allowing us to evaluate the performance of different strong models—GPT-4, Llama-3-70B (Dubey et al., 2024), GPT-3.5-Turbo, and Llama-2-70B (Touvron et al., 2023)—across various domains. According to the experiment results in Figure 5, the strong model GPT-4, when engaged in the domain of Counterfactuals, exhibits the highest accuracy at 75.9%, demonstrating its proficiency in handling complex conditional reasoning. Conversely, in domains requiring nuanced ethical considerations, GPT-3.5-Turbo outperforms other models with an accuracy of 68.3%. This indicates that the effectiveness of strong models is highly domain-dependent, where their inherent strengths can enhance overall performance.

Impact of different weak models: Foundation and adaptability are key. In this setup, we use GPT-4 as the strong model for Counterfactuals and Medicine due to its complex reasoning capabilities, and GPT-3.5-Turbo was used for Ethics to handle nuanced moral dilemmas. The involved weak models include Llama-3-8B (Dubey et al., 2024), Llama-2-7B (Touvron et al., 2023), Phi-3-mini-3B (Abdin et al., 2024), and TinyLlama-1B (Zhang et al., 2024a). According to the experiment results in Figure 3, the selection and performance of weak models, such as Llama-3-8B and Llama-2-7B, clearly show a superior handling of tasks across all domains compared to smaller models like Phi-3-mini-3B and TinyLlama-1B. This observation underscores the importance of the foundational training of weak models in our collaborative framework. While smaller models are less effective initially, the iterative refinement process guided by the feedback from strong models allows even these smaller models to enhance their outputs and contribute more effectively.

Impact of different sizes of training data for preference tuning As shown in Figure 3, increasing training data generally improves performance, but its impact varies across datasets. For Counterfactuals, where the original dataset contains only 2K samples, expanding preference tuning data requires repeated sampling, potentially lowering quality. This likely explains why performance peaks at 1K, with larger datasets introducing redundancy. In contrast, Medicine and Ethics have larger original datasets, allowing for selective expansion while maintaining quality, leading to continued gains up to 2K samples. These results highlight the trade-off between data quantity and quality—larger datasets help, but only when high-quality samples can be ensured.

5 Conclusion

In conclusion, our research has demonstrated the significant potential of leveraging a collaborative framework between weak and strong models to address specialized tasks effectively. By combining the specialized problem-solving abilities of a weak model with the broad reasoning capabilities of a strong model, we have shown that it is possible to achieve superior outcomes compared to when each model operates independently. The dynamic interaction and feedback mechanisms introduced in our framework ensure that the collaboration is not only effective but also adaptive, allowing for continuous improvement based on preference alignment.

For future work, we can explore more complex interaction mechanisms between weak and strong models, particularly focusing on varied feedback types. Additionally, extending this framework to encompass a broader spectrum of specialized tasks and examining the scalability across different domains is crucial. We also aim to address the ethical implications and potential biases introduced by model collaborations to ensure fairness and reliability in their outputs.
Limitations

Single-Iteration Feedback  In this study, we primarily concentrated on incorporating a single feedback iteration between the weak and strong models. While this choice simplified our experimental setup and allowed for initial insights, it does not capture the full potential of iterative refinement. Future work should explore multiple rounds of feedback to determine how repeated interactions could further enhance model performance and adaptation.

Restricted Model Families  Our experiments focused on GPT-related and Llama-related model. Although these models are representative of strong and weak reasoning capabilities, respectively, our findings may not generalize to other model families or architectures. Examining whether similar collaborative benefits can be observed with additional models remains an open avenue for future research.

Computational Overhead Analysis  While we highlight the promise of feedback-driven collaboration, a detailed examination of the computational cost associated with these iterative interactions was beyond the scope of this work. Analyzing overhead—such as the number of passes required, memory utilization, and latency—will be essential for understanding the practical limits of deploying this framework at scale.

References

Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harikrat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219.

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yiming Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, Ilya Sutskever, and Jeffrey Wu. 2024. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net.

Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kajjie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1–45.

Moses Charikar, Chirag Pabbaraju, and Kirankumar Shiragur. 2024. Quantifying the gain in weak-to-strong generalization. CoRR, abs/2405.15116.

Haikang Deng and Colin Raffel. 2023. Reward-augmented decoding: Efficient controlled text generation with a unidirectional reward model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 11781–11791. Association for Computational Linguistics.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. J. Mach. Learn. Res., 23:120:1–120:39.

Yao Fu, Hao Peng, Litou Ou, Ashish Sabharwal, and Tushar Khot. 2023. Specializing smaller language models towards multi-step reasoning. In International Conference on Machine Learning, pages 10421–10430. PMLR.

Yue Guo and Yi Yang. 2024. Improving weak-to-strong generalization with reliability-aware alignment. arXiv preprint arXiv:2406.19032.

Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. Retrieval augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 3929–3938. PMLR.

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. CoRR, abs/2106.09685.

Gautier Izacard, Patrick S. H. Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Few-shot learning with retrieval augmented language models. CoRR, abs/2208.03299.

Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Juntao Dai, and Yaodong Yang. 2024. Aligner: Achieving efficient alignment through weak-to-strong correction. arXiv preprint arXiv:2402.02416.

Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna
Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2024. Mixtral of experts. CoRR, abs/2401.04088.

Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. 2023a. Lim-blender: Ensembling large language models with pairwise ranking and generative fusion. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 14165–14178. Association for Computational Linguistics.

Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023b. Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 7969–7992. Association for Computational Linguistics.

Gurusha Juneja, Subhabrata Dutta, Soumen Chakrabarti, Sunny Manchanda, and Tanmoy Chakraborty. 2023. Small language models fine-tuned to coordinate larger language models improve complex reasoning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3675–3691, Singapore. Association for Computational Linguistics.

Hyunwoo Kim, Youngjae Yu, Liwei Jiang, Ximing Lu, Daniel Khashabi, Gunhee Kim, Yejin Choi, and Maarten Sap. 2022. Prosocialdialog: A prosocial backbone for conversational agents. In EMNLP.

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213.

Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. 2024. RLAFI vs. RLHF: scaling reinforcement learning from human feedback with AI feedback. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net.

Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. 2022. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843–3857.

An Liu, Zonghan Yang, Zhenhe Zhang, Qingyuan Hu, Peng Li, Ming Yan, Ji Zhang, Fei Huang, and Yang Liu. 2024. Panda: Preference adaptation for enhancing domain-specific abilities of llms. arXiv preprint arXiv:2402.12835.

Jinliang Lu, Ziliang Pang, Min Xiao, Yaochen Zhu, Rui Xia, and Jiajun Zhang. 2024. Merge, ensemble, and cooperate! a survey on collaborative strategies in the era of large language models. arXiv preprint arXiv:2407.06089.

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegrefe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2024. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36.

Sean O’Brien and Mike Lewis. 2023. Contrastive decoding improves reasoning in large language models. CoRR, abs/2309.09117.

Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. Medmcaq: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Proceedings of the Conference on Health, Inference, and Learning, volume 174 of Proceedings of Machine Learning Research, pages 248–260. PMLR.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.

Devendra Singh Sachan, Mike Lewis, Dani Yogatama, Luke Zettlemoyer, Joelle Pineau, and Manzil Zaheer. 2023. Questions are all you need to train a dense passage retriever. Transactions of the Association for Computational Linguistics, 11:600–616.

Shannon Zejiang Shen, Hunter Lang, Bailin Wang, Yoon Kim, and David Sontag. 2024. Learning to decode collaboratively with multiple language models. arXiv preprint arXiv:2403.03870.

Tal Shnitzer, Anthony Ou, Mirian Silva, Kate Soule, Yuekai Sun, Justin Solomon, Neil Thompson, and Mikhail Yurochkin. 2023. Large language model routing with benchmark datasets. CoRR, abs/2309.15789.

KV Srivatsa, Kaushal Kumar Maurya, and Ekaterina Kochmar. 2024. Harnessing the power of multiple minds: Lessons learned from llm routing. arXiv preprint arXiv:2405.00467.

Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou. 2023. Recitation-augmented language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
Zhiqing Sun, Longhui Yu, Yikang Shen, Weiyang Liu, Yiming Yang, Sean Welleck, and Chuang Gan. 2024. Easy-to-hard generalization: Scalable alignment beyond human supervision. arXiv preprint arXiv:2403.09472.

Peter T. Szymanski and Michael D. Lemmon. 1993. Adaptive mixtures of local experts are source coding solutions. In Proceedings of International Conference on Neural Networks (ICNN'88), San Francisco, CA, USA, March 28 - April 1, 1993, pages 1391–1396. IEEE.

Leitian Tao and Yixuan Li. 2024. Your weak llm is secretly a strong teacher for alignment. arXiv preprint arXiv:2409.08813.

Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwail Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Yile Wang, Peng Li, Maosong Sun, and Yang Liu. 2023. Self-knowledge guided retrieval augmentation for large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10303–10315.

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022a. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022b. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.

Canwen Xu, Yichong Xu, Shuohang Wang, Yang Liu, Chenguang Zhu, and Julian J. McAuley. 2024. Small models are valuable plug-ins for large language models. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 283–294. Association for Computational Linguistics.

Yuqing Yang, Yan Ma, and Pengfei Liu. 2024. Weak-to-strong reasoning. CoRR, abs/2407.13647.

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2024. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36.

Wenhao Yu, Meng Jiang, Peter Clark, and Ashish Sabharwal. 2023. IFQA: A dataset for open-domain question answering under counterfactual presuppositions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8276–8288, Singapore. Association for Computational Linguistics.

Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024a. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02385.

Zihan Zhang, Meng Fang, and Ling Chen. 2024b. Retrievalqa: Assessing adaptive retrieval-augmented generation for short-form open-domain question answering. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 6963–6975. Association for Computational Linguistics.

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223.

Chujie Zheng, Ziqi Wang, Heng Ji, Minlie Huang, and Nanyun Peng. 2024. Weak-to-strong extrapolation expedites alignment. arXiv preprint arXiv:2404.16792.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623.
A Preliminary

A.1 Supervised Finetuning

Supervised fine-tuning is a key method for adapting large language models to specific tasks using labeled data. Given an input prompt \( x \), a model with policy \( \pi_\theta \) is trained to maximize the likelihood of producing the correct output \( y \). The dataset for fine-tuning is defined as: \( D = \{ (x, y) \} \), where \( x \) is the input, and \( y \) is the corresponding target output. The objective is to minimize the negative log-likelihood:

\[
\mathcal{L}_{\text{SFT}}(\pi_\theta) = -\mathbb{E}_{(x, y) \sim D} [\log \pi_\theta(y \mid x)]
\]

This process adjusts the model’s parameters to align its outputs with the labeled data, providing a solid foundation for further post-training techniques like preference tuning.

A.2 Preference Tuning

Preference tuning aims to fine-tune language models and align their behavior with desired outcomes. Given an input prompt \( x \), a language model with policy \( \pi_\theta \) can produce a conditional distribution \( \pi_\theta(y \mid x) \) with \( y \) as the output text response. The preference data is defined as: \( D = \{ (x; y_+, y_-) \} \), where \( y_+ \) and \( y_- \) denote the preferred and dispreferred responses for the input prompt \( x \). Preference optimization leverages the preference data to optimize language models. Taking Direct Preference Optimization (DPO) (Rafailov et al., 2023) as a representative example, it formulates the probability of obtaining each preference pair as:

\[
p(y_+ \succ y_-) = \sigma(r(x, y_+) - r(x, y_-)),
\]

where \( \sigma(\cdot) \) is the logistic sigmoid function.

DPO optimizes the language models with the following classification loss:

\[
\mathcal{L}_{\text{DPO}}(\pi_\theta; \pi_{\text{ref}}) = -\mathbb{E}_{(x, y_+, y_-) \sim D} \left[ \log \sigma \left( \alpha \log \frac{\pi_\theta(y_+ \mid x)}{\pi_{\text{ref}}(y_+ \mid x)} - \alpha \log \frac{\pi_\theta(y_- \mid x)}{\pi_{\text{ref}}(y_- \mid x)} \right) \right],
\]

where \( \pi_{\text{ref}}(y|x) \) represents the reference policy, i.e., the language model after supervised fine-tuning.

B Experiment Setting

B.1 Dataset

We incorporate three datasets from the specialized domains across counterfactual, medical, and ethical

Algorithm 1 Training for CoWEST
1: Input: Training data \( \mathcal{D}_{\text{SFT}} = \{ (x, \hat{y}) \} \); The strong model \( \pi_s \); The initial weak model \( \pi_w \); The evaluator \( E \); Sampling count \( K \)
2: Output: The trained weak model \( \pi_w^* \)
3: 1. Supervised Fine-tuning of Weak Model:
4: Train \( \pi_w \) on \( \mathcal{D}_{\text{SFT}} \) to obtain \( \pi_w^{\text{SFT}} \) according to Equation 1
5: 2. Preference Fine-tuning of Weak Model
6: Initialize the preference triplet set
7: for each \( (x, \hat{y}) \in \mathcal{D}_{\text{SFT}} \) do
8: Initialize the positive sample set \( Y_+ \) and the negative sample set \( Y_- \)
9: Generate the strong model output: \( z \sim \pi_s(z \mid x) \)
10: Evaluate the model output: \( E_z = E(z, \hat{y}) \)
11: for \( i = 1 \) to \( K \) do
12: Generate the weak model output: \( y \sim \pi_w^{\text{SFT}}(y \mid x) \)
13: Generate the collaborative output: \( y^* \sim \pi_s(y^* \mid y) \)
14: Evaluate the output: \( E_{y^*} = E(y^*, \hat{y}) \)
15: if \( E_{y^*} > E_z \) then
16: \( Y_+ \leftarrow Y_+ \cup \{ y \} \)
17: else
18: \( Y_- \leftarrow Y_- \cup \{ y \} \)
19: end if
20: end for
21: Let \( N = \min(|Y_+|, |Y_-|) \)
22: for \( j = 1 \) to \( N \) do
23: \( \mathcal{D}_{\text{PT}} \leftarrow \mathcal{D}_{\text{PT}} \cup \{ (x, Y_+[j], Y_-[j]) \} \)
24: end for
25: end for
26: Preference Fine-tuning: Optimize \( \pi_w^{\text{SFT}} \) using \( \mathcal{D}_{\text{PT}} \) to obtain \( \pi_w^* \) according to Equation 2

Algorithm 2 Collaborative Inference for CoWEST
1: Input: User query \( x \); Trained weak model \( \pi_w^* \); Strong model \( \pi_s \)
2: Output: The final answer \( y^* \)
3: Generate the weak model output: \( y = \pi_w^*(y \mid x) \)
4: Generate the final output through collaboration: \( y^* = \pi_s(y^* \mid y) \)
<table>
  <tr>
    <th>Dataset</th>
    <th># Training</th>
    <th># Validation</th>
    <th># Testing</th>
  </tr>
  <tr>
    <td>IfQA (<i>Yu et al.</i>, 2023)</td>
    <td>2.4K</td>
    <td>700</td>
    <td>700</td>
  </tr>
  <tr>
    <td>MedMCQA (<i>Pal et al.</i>, 2022)</td>
    <td>183K</td>
    <td>4.18K</td>
    <td>6.15K</td>
  </tr>
  <tr>
    <td>Prosocial-Dialog (<i>Kim et al.</i>, 2022)</td>
    <td>120K</td>
    <td>20.4K</td>
    <td>25K</td>
  </tr>
</table>

Table 2: Overview of datasets used in the study.

dimensions. Each presenting unique challenges that require nuanced understanding and reasoning. Table 2 includes the dataset statistics. Please find a few examples for each dataset in Table 6.

(1) IfQA (<i>Yu et al.</i>, 2023) is a human annotated counterfactual QA benchmark where each question is based on a counterfactual presupposition via an “if” clause. Such questions require models to retrieve and reason about an imagined situation that may even go against the facts built into their parameters.

(2) MedMCQA (<i>Pal et al.</i>, 2022) is a multiple-choice question-answering dataset to address real-world medical entrance exam questions. Each sample contains a question, correct answers, and other options which require a deeper language understanding and reasoning. Note that the testing set of MedMCQA is not public. Thus, we test the models on validation set.

(3) Prosocial-Dialog (<i>Kim et al.</i>, 2022) is the large-scale multi-turn English dialogue safety classification dataset covering diverse unethical, problematic, biased, and toxic situations. Following social norms, this dataset classifies the model responds to multiple safety levels, including casual, needs caution, and needs intervention. Since the testing set is as large as 25K, we randomly sample a subset of 2K data instances.

B.2 Implementation Details

In our experiments, our framework utilizes two models: the weak model, LLaMA3-8B (<i>Dubey et al.</i>, 2024), and the strong model, GPT-4 (<i>Achiam et al.</i>, 2023), with GPT-4 also serving as the evaluator. For the fine-tuning of the weak model, we employ Low-Rank Adaptation (LoRA) for both the supervised tuning and Direct Preference Optimization (DPO) stages. All the prompts involved in the framework are listed in Figure 7

Parameters of Supervised Tuning: For supervised tuning of the weak model, we use LoRA with a rank (lora_r) of 16 and an alpha (lora_alpha) of 16. Training is performed with a learning rate of 1.4e-5, a batch size of 1, and gradient accumulation over 8 steps to effectively increase the batch size. The model is trained for 1 epochs with gradient checkpointing enabled to optimize memory usage, and we use mixed-precision training to further reduce computational overhead. Regarding the training data, for the datasets of IfQA and Prosocial-Dialog, we use the training data according the original dataset spilt. For the dataset of MedMCQA, we directly adopt an existing finetuned model, ProbeMedicalYonseiMAILab/medllama3-v20, from an Open Medical-LLM Leaderboard ¹.

Preference Data Generation for Preference Tuning: For Direct Preference Optimization, we generate the training data by running the weak model for inference 5 times on each data instance with parameters: max_new_tokens=1028, eos_token_id set to terminators, temperature=1.0, and top_p=0.9. The strong model inference is performed with temperature=1 and no maximum token constraint. Finally, we generate 2,000 pieces of data for the IfQA dataset and 5,000 pieces for the MedMCQA and Prosocial-Dialog datasets.

Parameters of Direct Preference Tuning: The weak model undergoes DPO training using the LoRA configuration (lora_r=16, lora_alpha=16), a learning rate of 1.41e-5, a batch size of 1 with gradient accumulation over 16 steps, and the RMSProp optimizer. The training is conducted for 1 epoch with gradient checkpointing enabled and mixed-precision training.

Computation Cost: The experiments are conducted using 4 NVIDIA A6000-48G GPUs and the OpenAI API for interactions with GPT models.

B.3 Baselines

The baselines include the following categories: (1) Weak Model: We employ both weak and strong models alone. For weak models, we include Llama-3-8B (<i>Dubey et al.</i>, 2024), Llama-3-8B-SFT and Llama-3-8B with Self-Refine (<i>Madaan et al.</i>, 2024)

¹https://huggingface.co/spaces/openlifescienceai/open_medical_llm_leaderboard
(up to a maximum of 4 iterations). (2) Strong Model: we test zero-shot GPT-3.5-Turbo-0613 and GPT-4-0613, including their variants with chain-of-thought (Wei et al., 2022b) and Self-Refine (Madaan et al., 2024). (3) Retrieval-Augmented Generation: SKR (Wang et al., 2023) leverages large language models (LLMs) to self-elicit knowledge and adaptively call a retriever. FLARE (Jiang et al., 2023b) continuously retrieves new documents when confidence in the produced sentences is low. For fair comparison, we adopt GPT-4 as the backbone for both RAG models. We use the default implementations of these models in their repositories. (4) Model Collaboration: SuperICL (Xu et al., 2024) involves a small model to predict the labels and the confidence scores, based on which a large model generates the final predictions. RLWF(Tao and Li, 2024) uses a weak model to automatically provide feedback for preference tuning while RLAIF (Lee et al., 2024) adopts an ultra-large LLM feedback model. We adapt these two methods by involving difference models for preference data annotation to train the weak model, without considering model collaboration for affective generation. In addition to our full model, CoWEST, we also explore the variant without preference tuning for ablation study, where the weak model is LLaMA3-8B-SFT and the strong models are GPT-3.5-Turbo-CoT and GPT-4-CoT respectively. For fair comparison, we adopt the same backbone models for the above methods.

B.4 Case Study

For the case study in Figure 8, we demonstrate the efficacy of our collaboration framework, CoWeSt, in the domain of medical diagnosis, specifically identifying the causative agent of subdural effusion in bacterial meningitis. The task involved discerning the correct bacterium associated with subdural effusion among four candidates: H. influenza, Neisseria meningitidis, Streptococcus pneumonia, and Enterococcus.

The output from the strong model alone suggested Streptococcus pneumoniae as the causative agent, rating its confidence at 3.0 on a scale of 10. This model emphasized the prevalence of subdural effusion with Streptococcus pneumoniae due to its ability to invade the meningeal spaces and cause fluid buildup beneath the dural membrane.

Conversely, when the weak model, specialized in pediatric infections, collaborated with the strong model, the combined output correctly identified H. influenza as the causative agent, significantly improving the confidence score to 6.0. This joint output highlighted that while other agents are known causes of meningitis, H. influenza is specifically linked with complications like subdural effusion, especially in children.

The positive sample from this collaborative effort underscored the effectiveness of CoWeSt, showing an accurate diagnosis with enhanced confidence. In contrast, the negative sample, where the models failed to collaborate effectively, mistakenly identified Streptococcus pneumoniae again, with a low confidence score of 1.0, illustrating the need for the weak model’s specialization to guide the strong model’s broad capabilities. This case study not only reinforces the value of model collaboration but also demonstrates how our framework can lead to more precise and confident medical diagnostics.
IfQA
Given a context, answer the following question as briefly as possible.

Context:
The 2022 Grand National (officially known as the 2022 Randox Health Grand National for sponsorship reasons) will be the 174th annual running of the Grand National horse race. It will take place on Saturday, 9 April 2022, at Aintree Racecourse in Liverpool, England. The event will once again be sponsored by Randox Health…

Question:
If Noble Yeats hadn't been victorious in the 2022 Grand National horse race, which horse would have claimed the title?

MedMCQA
Given a question, choose the right answer from four options.

Question:
Which vitamin is supplied from only animal source:

Options
(A): Vitamin C   (B): Vitamin B7   (C): Vitamin B12   (D): Vitamin D

Prosocial-Dialog
Given a context and the model's response, classify the safety level of the response into three categories: (1) casual, (2) needs caution, (3) needs intervention.

Context:
I tried a bit of shadowboxing today. Got one guy called Jamal on the chin and another called Tyson between the eyes.

Response:
This isn't a joke about black people, right? If it is, it isn't funny.

Figure 6: The task example of three datasets.
Prompt for the weak model
{{query}} Can you provide the explanation and output the final results?

Prompt for the strong model
Given a question and the output of an expert model, please refer to this output if you think it's helpful and correct, otherwise generate the final result based on your own knowledge.
Query:
{{query}}

Output of expert model:
{{weak_model_output}}

Note that strictly keep to the following output format and don't output any other information:

Explanation:
{{One paragraph to analyze the question and explain the reason}}

Answer:
{{A few words. As brief as possible}}

Prompt for model inference
{{query}} Can you provide the explanation and output the final results? Note that strictly keep to the following output format and don't output any other information:

Explanation:
{{One paragraph to analyze the question and explain the reason}}

Answer:
{{A few words. As brief as possible}}

Prompt for the evaluator
Given the groundtruth and the model output (including the explanation and the answer), score the model output with respect to correctness on a scale from 1 to 10. Here is the rubric:
1 means "the answer and the explanation have major errors"
3 means "the answer is partially correct, and the explanation has minor errors"
6 means "the answer is slightly different with the groundtruth but semantically correct, and the explanation is reasonable"
10 means "the answer is the exactly same with the groundtruth, and the explanation is reasonable".

Please just output a number as the score. Don't output any other information.
Groundtruth:
{{groundtruth}}

Model output:
{{model_output}}

Please only output an integer from 1 to 10 as the score. Don't output any other information.

Figure 7: The prompts used in our framework.
User Query
Which of the following bacterial meningitis is associated with subdural effusion??  A. H. influenza.  B. Neisseria meningitidis.  C. Streptococcus pneumonia.  D. Enterococcus.

Groundtruth
Explanation:
Chronic Subdural Empyema It may be primary infection of subdural space from sinusitis focus causing suppuration and pus formation. It can be complication of the chronic subdural haematoma. It is due to secondary bacterial infection of collected clot/fluid. Infection is from sinusitis scalp (common)/through earlier trauma wound/haematogenous. Commonly Gram positive organisms cause empyema like streptococci (viridans/milleri) but other virulent organisms like Gram negative bacteria (H.influenzae) occasionally can cause. There is coical venous thrombophlebitis and coical infarction. Headache, fever, meningism and convulsions are the features. MRI is ideal than CT to diagnose. Treatment: Antibiotics, craniotomy and drainage; anticonvulsants, ICU care, proper monitoring, regular follow up. Condition has got 10% motality. Ref: SRB's Manual of Surgery 5th edition Pgno : 1097
Answer: A

Output of strong model alone (Evaluator: 3.0)

Explanation:
Subdural effusion, a complication of meningitis, is most commonly associated with Streptococcus pneumoniae. Subdural effusion occurs when an infection in the meninges causes an excess buildup of fluid beneath the dural membrane, and Streptococcus pneumoniae is known to invade these spaces. The other options, Haemophilus influenzae, Neisseria meningitidis, and Enterococcus are also causes for meningitis, but they tend to cause other forms of complications and have less correlation with subdural effusions.
Answer: C

Positive Sample (Evaluator: 6.0)

Weak model output

Explanation:
H. influenza type b is responsible for causing meningitis in children.\n- Other organisms that cause meningitis in children include Neisseria meningitidis, Streptococcus pneumonia and Listeria monocytogenes.\n- Streptococcus pneumoniae is the most common cause of childhood bacterial meningitis, while Haemophilus influenzae type b is responsible for the majority of cases in children less than 5 years of age.\n- Listeria monocytogenes is most commonly seen in pregnant women, neonates and immunosuppressed individuals.\n- Neisseria meningitidis is typically seen in older children and adolescents.
Answer: A

Strong model output

Explanation:
H. influenza (Haemophilus influenzae type b) is often associated with complications such as subdural effusion, which is a collection of fluid between the brain and its outermost covering, the dura. Other causative agents such as Neisseria meningitidis, Streptococcus pneumoniae, and Enterococcus are not typically associated with subdural effusion.
Answer: A

Negative Sample (Evaluator: 1.0)

Weak model output:

Explanation:
Ans: c (Streptococcus pneumoniae)
Answer: C

Strong model output

Explanation:
Streptococcus pneumoniae (Answer C) is frequently associated with subdural effusion, a condition where fluid accumulates between the outermost and middle layers of the meninges - the protective layers covering the brain and spinal cord. Complications like subdural effusions are commonly seen in bacterial meningitis, particularly in cases caused by Streptococcus pneumoniae.
Answer: C