Fast and Slow Generating: An Empirical Study on Large and Small Language Models Collaborative Decoding

## Abstract

Abstract Large Language Models (LLMs) exhibit impressive capabilities across various applications but encounter substantial challenges such as high inference latency, considerable training costs, and the generation of hallucinations. Collaborative decoding between large and small language models (SLMs) presents a promising strategy to mitigate these issues through methods including speculative decoding, contrastive decoding, and emulator or proxy fine-tuning. However, the specifics of such collaborations, particularly from a unified perspective, remain largely unexplored. Inspired by dual-process cognitive theory, we propose a unified framework in this paper, termed Fast and Slow Generating (FS-GEN). Within this framework, LLMs (sometimes along with SLMs) are categorized as System 2 (slow and deliberate), while independent SLMs are designated as System 1 (fast and intuitive). We provide a comprehensive analysis of these collaborative methodologies, elucidating their common properties and shedding light on the differential knowledge capabilities of System 2 versus System 1 through the FS-GEN framework. Our findings indicate that only a small proportion of collaborative interactions (approximately less than 20% in most instances) are necessary across various methods. These interactions between System 1 and System 2 conform to a scaling law related to the parameter ratios, enabling predictable collaboration. Furthermore, we explore the specific conditions under which collaboration proves most effective, particularly from an uncertainty perspective, offering novel insights that may guide future optimization efforts. Our research underscores that the fundamental distinction between System 1 and System 2 lies in the uncertainty of next token predictions, where interventions by System 2 are crucial to support System 1.

## 1 Introduction

Figure: Figure 1: In the fast-and-slow generation framework, we conceptualize small language models as System 1 and a mixed-scale compound system of large and small language models as System 2. This compound system incorporates various collaborative decoding methods, such as speculative decoding, contrastive decoding, and emulator tuning. In this paper, we investigate the optimal conditions and contexts for applying System 1 and System 2 at the token-level decoding stage.
Refer to caption: x1.png

In recent years, the landscape of artificial intelligence has been dramatically reshaped by the development of Large Language Models (LLMs) like GPT-4 (Achiam et al., ), Gemini (Team et al., ), Llama (Touvron et al., ; ; AI@Meta, ), and Qwen (Bai et al., ). These models have emerged as versatile tools, enhancing software development (Qian et al., ; Yang et al., ), and accelerating scientific discovery (AI4Science & Quantum, ; Qi et al., ) in different domains (Boiko et al., ; Romera-Paredes et al., ).
Despite their advancements, these models face significant challenges that affect their practical application.
Key issues include managing inference time with constraints such as low-latency requirements (Miao et al., ; Zhou et al., ),
and addressing concerns related to hallucinations in generated content (Huang et al., ).
Furthermore, the training of these models entails considerable resource consumption (Hu et al., ; Hoffmann et al., ; Ding et al., ), particularly during fine-tuning and in achieving domain specialization (Bommasani et al., ).

A novel trend in mitigating these challenges involves the collaborative decoding of large and small models.
This approach leverages the strengths of both model sizes to improve efficiency and effectiveness across various downstream applications.
In these methods, the next token during generation depends on both LLMs and SLMs, whose logits are combined to form the final predictions.
For instance, speculative decoding (Leviathan et al., ; Kim et al., ; Sun et al., ; Xia et al., ) allow small models to swiftly generate initial drafts, which are then refined by large models, significantly cutting down on inference time.
Similarly, in contrastive decoding (Li et al., ; O’Brien & Lewis, ; Sennrich et al., ), small models contribute amateur tokens as negative signals to enhance the decoding.
Proxy tuning (Mitchell et al., ; Liu et al., ) aims to reduce training costs by applying the delta logits from proxy small models to the base large models, rather than directly tuning the large models. This method significantly reduces training costs.

However, while the use of large and small models in tandem is increasingly common, the research community lacks a unified framework to analyze and understand the dynamics of their collaboration.
This gap in knowledge motivates our current study, which seeks to formalize the collaboration between these models from the perspective of Systems 1 and System 2—a framework typically applied to modeling human cognitive processes (Kahneman, ).
As shown in Figure 1, within the human brain, System 1 operates quickly and intuitively, automatically addressing everyday problems such as pattern recognition. Conversely, System 2 functions in a slower, more deliberate manner, tackling complex issues that require long-term planning.
The integration of System 1 and System 2 in daily human activities achieves maximum efficiency and demonstrates robust intelligence.
Notably, System 1 handles approximately 95% of everyday tasks, while System 2 is responsible for the remaining 5% (Daniel, ).
The fast and slow thinking paradigm has been applied in various LLM applications (Hagendorff et al., ; Lin et al., ; Qi et al., ), which focus on improving the abilities of System 2 and the inference efficiency of LLMs.
This paradigm provides a human-like mechanism for model optimization toward advanced artificial intelligence.
Similarly, large and small models exhibit the natural characteristics of slow and fast processing. Previous works (Kim et al., ) on speculative decoding indicate that only a small percentage of generations require intervention by LLMs, while SLMs can independently complete most of the remainder.

However, many unknowns persist in these collaborations within this broader paradigm, including questions about when and where System 2 should be utilized.
To the best of our knowledge, no previous work has attempted to unify the collaborative decoding methods of SLMs and LLMs from this perspectives.
In this paper, we explore the fundamental questions of model collaboration under the fast and slow generation framework: 1) What is the essential difference between Systems 2 and 1 from the perspective of decoding? 2) Why does System 1 under-perform compared to System 2? 3) How can we improve effiency of the large and small compound system?

To investigate these problems, we focus on analyzing the frequency and position characteristics of collaboration between large and small models, employing both qualitative and quantitative methods.
Our research examines theoretical limits to collaboration frequency and employs scaling laws to predict how often different model combinations might collaborate based on the disparity in their parameters.
Additionally, we explore the position tendencies of these collaborations, particularly in terms of where in the generation process they are most effective and how they correlate with the inherent uncertainty of small models.
Our preliminary findings are summarized as follows:

- •
A comparison between the collaboration dynamics in large and small models and human dual-system cognitive processes reveals a shift from the expected 95:5 intuitive reasoning ratio to an 80-20 distribution ($\S$ 4.1.1).
This finding also suggests a strong correlation between the frequency of interventions and the size differences in parameters between models ($\S$ 4.1.2), in accordance with scaling laws.
- •
Collaborative interactions among models are most critical at the beginning of the generation process, underscoring the adage that “Well begun is half done.” ($\S$ 4.2.1) The uncertainty levels in next token prediction of smaller models play a pivotal role, serving as a reliable indicator of collaboration necessity ($\S$ 4.2.2).

## 2 Preliminary

### 2.1 Text Generation

Text generation, also referred to as decoding (Welleck et al., ), is the process whereby a language model $\mathcal{M}$ selects the next token in a sequence based on the context.
Mathematically, this involves estimating the conditional probability of a token given its preceding sequence.
For a given source context with $N$ tokens and a target response with $T$ tokens, noted as $X=(x_{1},x_{2},\cdots,x_{N})$ and $Y=(y_{1},y_{2},\cdots,y_{T})$, the probability of the response sequence is expressed as:

$$ $P(Y|X)=\prod_{i=1}^{T}p(y_{i}|X,y_{1},y_{2},\cdots,y_{i-1})$ $$

**Table 1: Motivation, roles of SLMs and LLMs in different collaborative decoding methods.**
| Method | Speculative Decoding | Contrastive Decoding | Proxy Tuning |
| --- | --- | --- | --- |
| Motivation | Inference efficiency | Factuality and Reasoning | Training efficiency |
| Role of SLMs | Drafter<br>(generates candidate tokens) | Amateur<br>(generates suboptimal tokens) | Domain expert<br>(provides delta logits) |
| Role of LLMs | Verifier of draft tokens | Expert generator | Base logits provider |
| System 1 | SLMs | SLMs | SLMs |
| System 2 | LLMs | LLMs + SLMs | LLMs + SLMs (FT/Base) |
| Collaborations | Generation-verification | Logits fusion | Logits fusion |

During the decoding phase, at each step $i$, the model $\mathcal{M}$ computes the probability distribution $p(y_{i}|X,y_{1},y_{2},\cdots,y_{i-1})$ over the entire vocabulary, based on the input sequence provided so far.
The selection of the next token $y_{i}$ is then based on this distribution, typically employing methods such as greedy search (Parsing, ), beam search (Vijayakumar et al., ; Freitag & Al-Onaizan, ), or various sampling strategies (Fan et al., ; Holtzman et al., ).

### 2.2 Definition of FS-GEN Methods

Beginning with the fundamental definition of text decoding, we extend this concept to encompass collaborative decoding.
In order to provide a unified definition of the collaborations employed in various FS-GEN methods, we consider setups involving large language models, denoted as $\mathcal{M}_{l}$, and small models, denoted as $\mathcal{M}_{s}$.
The respective input prompts for $\mathcal{M}_{l}$ and $\mathcal{M}_{s}$ are given as $X_{l}$ and $X_{s}$, which are identical in most settings.
As illustrated in Figure 1, the logits for the next token generated by LLMs and SLMs are denoted as $p_{l}$ and $p_{s}$, respectively, with the reference golden logits noted as $p_{f}$.
We categorize the outputs into three types: golden or collaborative outputs $\mathcal{O}_{g}$, outputs $\mathcal{O}_{l}$ from $\mathcal{M}_{l}$, and outputs $\mathcal{O}_{s}$ from $\mathcal{M}_{s}$.
Note that the term golden logits $p_{f}$ and outputs $\mathcal{O}_{g}$ refers to the results of collaboration, which may not always equate to the golden answers in some instances.
These variables are assigned differently across each FS-GEN method, varying according to the fusing strategy and underlying motivation:

Speculative Decoding. This approach leverages predictions from SLMs that are subsequently validated or refined by LLMs. In this setting, the outputs from LLMs are considered golden outputs, denoted as $p_{f}=p_{l}$ and $\mathcal{O}_{g}=\mathcal{O}_{l}$, where $\mathcal{M}_{l}$ signifies the model chosen to select tokens, reflecting the trust in the outputs $\mathcal{O}_{s}$ from $\mathcal{M}_{s}$. Speculative decoding represents the most essential case.

Contrastive Decoding. This method employs differential strategies to reconcile the disparities between the predictions from large and small models, aiming to enhance accuracy. In the typical implementation (Li et al., ), the reference logits are calculated as $p_{f}=p_{l}+\beta(p_{l}-p_{s})$, and $\mathcal{O}_{g}$ outperforms both $\mathcal{O}_{l}$ and $\mathcal{O}_{s}$, indicating that $\mathcal{O}_{g}$ leverages the strengths of both models.

Proxy Tuning. This strategy involves tuning SLMs to emulate the behavior of LLMs, thus minimizing training resource usage without significantly compromising output quality.
In this configuration, SLMs are differentiated into base and chat versions, with their logits denoted respectively as $p_{s}$ for the base version and $p_{s}^{\prime}$ for the chat version.
The fused logits are represented by $p_{f}=p_{l}+(p_{s}^{\prime}-p_{s})$, suggesting that $\mathcal{O}_{g}$, incorporating insights from both $\mathcal{O}_{l}$ and the delta change between the tuned and untuned states of SLMs, approaches the performance of the fully tuned LLMs.

As shown in Table 1, in speculative decoding, SLMs serve as the base models for fast generation, while LLMs manage the more deliberate and slow aspects, providing interventions to the fast process.
In contrast, there is difference in contrastive decoding and proxy tuning, with SLMs introducing interventions into the LLM generation process.
In addition to the three typical methods, further exploration of other FS-GEN methods is displayed in Appendix B.

### 2.3 Definition of Collaboration Frequency

As shown in Figure 1 and as discussed previously, not all steps in the fast system’s generation require collaborations or interventions from the slow system.
We denote the number of collaborations in FS-GEN methods as $N_{l}$ and total length of generated content as $N_{total}$.
We defined collaboration frequency as $CoF=\frac{N_{l}}{N_{total}}$, which is typically set to 1 in contrastive decoding and proxy tuning.
We can establish a lower bound of collaboration frequency, denoted as $CoF_{\text{lower}}$, which has been preliminarily proven in prior research (Kim et al., ) and is further discussed in $\S$ 4.1.1.
The term “lower bound” denotes the minimal collaboration rate required for the text produce by SLMs (System 1) to exactly match the output generated solely by SLMs and LLMs (System 2).

## 3 Empirical Study

### 3.1 Experiment Setup

Models.
Our selection of language models primarily includes the Qwen (Bai et al., ) series and Pythia series (Biderman et al., ), which feature the broadest range of model parameters.
The Qwen series spans from 0.5B to 72B, while the Pythia series covers from 70M to 12B.
For the FS-GEN methods, model combinations are strategically grouped based on size to facilitate collaborative decoding, such as pairing 0.5B with 14B, and 7B with 72B.
To validate and observe the original behavior of models, we utilize the base models in addition to implementing proxy tuning, which involves combining base and chat models.
For the Pythia series, we fine-tuned the base models using the hkust-nlp/deita-10k-v0 dataset (Liu et al., ) to develop a chat version.

Datasets.
To explore the common features of collaborative decoding across various tasks, our experiments utilize datasets tailored to distinct domains: MMLU-STEM (Hendrycks et al., ) for general knowledge,
GSM8k (Cobbe et al., ) for math, and MBPP (Austin et al., ) for coding. Analysis is primarily conducted on the test sets to mitigate the risk of training data leakage; however, we also perform statistical analyses on the training data for threshold setting.

Our experiments are conducted exclusively using greedy decoding, which effectively captures the fundamental uncertainty associated with tokens in both large and small models (Wang & Zhou, ).
Drawing on previous studies involving speculative and contrastive decoding, greedy search could potentially be adapted to sampling decoding through temperature rescaling. (Leviathan et al., ; Chen et al., ; ).
Additional implementation details are provided in Appendix C.

### 3.2 Experiment Design

Our empirical study investigates two key aspects of collaboration in FS-GEN: frequency and position.
Collaboration frequency evaluates the minimum necessary interactions between large and small language models, enabling predictions of interaction frequency across different model scales.
Collaboration position identifies specific interaction points, predicting when collaboration is needed.
For collaboration frequency, we first analyze lower bound of frequency across various model scales and tasks to identify trends ($\S$ 4.1.1). Next, we examine a scaling law in FS-GEN that links models scale ratios with optimal interaction frequency for prediction ($\S$ 4.1.2).
For collaboration position, we initially study mismatch points relative to generated content length ($\S$ 4.2.1), followed by an analysis based on token uncertainty in SLMs ($\S$ 4.2.2).

## 4 Experimental Results

### 4.1 Exploring Collaboration Frequency

We explore collaboration frequency through analyses of the lower bound and scaling laws.
In the former analysis, we compute the lower bound of collaboration frequency for models of different scales across tasks.
In the latter analysis, we examine the relationship between the lower bound frequency and the models’ parameters ratios, which aids in building predictable collaborations.

Figure: Figure 2: Collaboration frequency between models across various tasks (Qwen Series).
Refer to caption: x2.png

Figure: Figure 3: Collaboration frequency between models across various tasks (Pythia Series).
Refer to caption: x3.png

#### 4.1.1 What is the Lower Bound of the Number of Collaborations in FS-GEN Methods?

As illustrated in Figure 2 and 3, the collaboration frequency varies with the difference between the task and model combination. Notably, for the specific SLMs, the frequency of collaboration increases as the size of LLMs increases, while models of similar sizes exhibit lower frequencies.
When the parameters of the models are more closely aligned, the ${CoF}_{\text{lower}}$ is reduced, generally not exceeding 20% across various tasks.
This observation suggests that the parameter gap between models does not conform to traditional scaling laws, where task performance is typically emergent (Wei et al., ).
It also indicates that scaling laws may be influenced by the continuity of metrics (Schaeffer et al., ), which are affected by token uncertainty.
The primary distinction between large and small models lies in the selection of essential tokens.
Small models can be significantly enhanced by a few key tokens from larger models, particularly those that are difficult for them to select.
This phenomenon is evident not only in oracle decoding, a form of speculative reasoning, but also in other collaborative decoding methods such as contrastive and proxy decoding.

#### 4.1.2 What is the Relationship Between Models’ Ratios and Lower Bounds?

Figure: Figure 4: The relationship between models’ parameters ratios and the lower bounds of collaboration frequency (Qwen Series).
Refer to caption: x4.png

Figure: Figure 5: The relationship between models’ parameters ratios and the lower bounds of collaboration frequency (Pythia Series).
Refer to caption: x5.png

Inspired by previous works on the “Scaling Law” (Kaplan et al., ; Ruan et al., ; Besiroglu et al., ; Hu et al., ), we aim to develop a scaling law relating the models’ parameter ratios to the lower bound of collaboration frequency (${CoF}_{\text{lower}}$).
Let number of parameters of $\mathcal{M}_{l}$ and $\mathcal{M}_{s}$ are denoted as $\mathcal{N}_{l}$ and $\mathcal{N}_{s}$ respectively.
Intuitively, the ${CoF}_{\text{lower}}$ is inversely proportional to the scale ratio $R=\frac{\mathcal{N}_{l}}{\mathcal{N}_{s}}$.
The relationship between the variables can be expressed by the equation:

$$ ${CoF}_{\text{lower}}=\gamma\cdot R^{-\alpha}+\beta$ $$

where $\gamma$ is a negative constant that determines the sensitivity of the ${CoF}_{\text{lower}}$ to the scale ratio, $\alpha$ is the scaling exponent, $\beta$ is a fitting parameter.

As the scale ratio $R$ increases, the collaboration frequency also increases. This is because the distance between the output distributions of the larger model and the smaller model increases (as “Scaling Law” works), necessitating a higher proportion of collaboration ${CoF}_{\text{lower}}$ from the smaller model.
Figure 4 and 5 illustrate the fitting curves for different $R$ across various tasks and FS-GEN methods. The results indicate that the lower bound of collaboration frequency is related to the scale ratio between LLMs and SLMs, as well as the specific tasks and methods involved.

### 4.2 Exploring Collaboration Position

Building on previous analysis, we examine the mismatch tokens and key collaboration positions.
This includes analyzing the mismatch positions within sequences in $\S$ 4.2.1 and the relationship between mismatch positions and token uncertainty in $\S$ 4.2.2.

#### 4.2.1 Which Part of the Generation Most Requires Collaboration or Intervention?

Figure: Figure 6: Percentage of mismatch positions relative to length for model combinations across various datasets (Qwen Series, Oracle Decoding).
Refer to caption: x6.png

Figure: Figure 7: Percentage of mismatch positions relative to length for model combinations across various datasets (Qwen Series, Contrastive Decoding).
Refer to caption: x7.png

Figure: Figure 8: Percentage of mismatch positions relative to length for model combinations across various datasets (Qwen Series, Proxy Tuning).
Refer to caption: x8.png

Figure: Figure 9: Percentage of mismatch positions relative to length for model combinations across various datasets (Pythia Series, Oracle Decoding).
Refer to caption: x9.png

As discussed, interventions from LLMs are necessary only at mismatch positions in generations between SLMs and LLMs.
As depicted in Figure 6, 7, 8, and 9, we quantify these positions using a mismatch rate for all positions, analyzed with a 10% split where deeper colors indicate higher percentages and lighter colors represent lower percentages.
Our initial analysis of mismatch positions across the entire generated content corroborates the importance of the generation’s start.
Prior research on reasoning tasks demonstrates that the pass rate for smaller models can be significantly improved by substituting the initial steps with those generated by larger models (Jain et al., ; Wang & Zhou, ).
This phenomenon is similarly observed during the collaborative decoding between large and small models.
The findings reinforce the adage that a good start is crucial, largely due to the presence of more reliably conditioned tokens from larger language models.
This effect is particularly pronounced in GSM8k and MBPP tasks, illustrating that steps are more critical in reasoning tasks than in common knowledge tasks.
Additionally, this finding could also support privacy protection in cloud-device collaborations, as highlighted in recent work (Zhang et al., ).

Figure: Figure 10: Visualization of token uncertainty in SLMs and LLMs. Red signifies mismatch tokens, while Blue indicates match tokens; darker shades represent higher confidence, where only with interventions from LLMs on red tokens can SLMs provide the correct answers.
Refer to caption: x10.png

#### 4.2.2 Which Position in Fast Generation Most Requires Collaboration/Intervention?

Beyond a macro analysis of whole-length position discrepancies, we also delve into the mismatch token logits and internal uncertainty within the vocabulary.
Next token prediction (Radford et al., ; ; Brown et al., ; Achiam et al., ), a cornerstone of the “Scaling Law”, further highlights differences between large and small models.
As depicted in Figure 10, this illustration shows that only a few tokens during the generation of a problem in the GSM8k task by small models require guidance from larger models, which are identified as red tokens.
Compared to large models, small models often struggle with uncertain tokens, which leads to incorrect responses.
These tokens exhibit a similar distribution, characterized by high uncertainty at each step, with particularly low confidence in the top token, which helps identify crucial collaboration points.
These findings have become fundamental principles for routing draft models in previous works (Kim et al., ; Du et al., ) on speculative decoding.
We also observe their occurrence in various FS-GEN methods (i.e., contrastive decoding and proxy tuning), as detailed in Appendix D.1.

To further investigate the uncertainty of token logits, we conduct a visual analysis of the top-k token logits at each generative step, noting whether they match.
The resulting correlation plots are displayed in Figures 11, 12 and  13.
There is a high correlation between the mismatch positions and token uncertainty, particularly when more context tokens are considered.
Additional cases and implementation details are provided in Appendix D.1.

Figure: Figure 11: KDE and scatter visualization of feature distribution for mismatch/uncertain tokens (1 & 5 tokens) (Qwen Series).
Refer to caption: extracted/5949255/assert/fig7.jpg

Figure: Figure 12: KDE and scatter visualization of feature distribution for mismatch/uncertain tokens (1 & 5 tokens) (Pythia Series).
Refer to caption: extracted/5949255/assert/fig-pythia-2/fig7-py.jpg

Figure: Figure 13: 3D visualization of feature distribution for mismatch/uncertain tokens logits (1 & 5 tokens) (Qwen Series).
Refer to caption: extracted/5949255/assert/fig6.jpg

## 5 Discussion

Different Model Series and Architecture.
In this paper, we exclusively conduct experiments on Qwen-series and Pythia-series models, which utilize transformer-based architecture. While further analysis could extend to different model series such as Llama and Yi (AI et al., ) models, the parameter scales of these models are not as extensive as those of the Qwen models. Additionally, recent state space models (SSM) (Gu et al., )
like Mamba (Gu & Dao, ; Dao & Gu, ) and hybrid architectures combining SSM and attention mechanisms, such as Jamba (Lieber et al., ), are garnering researcher interest.
Speculative decoding, when applied to multimodal models (Gagrani et al., ), allows for the expansion of FS-GEN methods across various multimodal applications (Zhu et al., ; Team, )
Our findings can be verified and further investigated on these models.

Cost-aware Collaboration Optimization.
In our experiments, we illustrate the lower bound of collaboration frequency; however, achieving this in practice is not fully feasible.
We can optimize the collaboration frequency by examining token uncertainty, employing heuristic rules (Kim et al., ) that prove challenging to scale automatically across various scenarios.
Additionally, few studies consider inference costs. Future research could explore strategies to balance inference costs and performance within a defined budget (Shen et al., ).
We hope that these empirical findings will further inform more FS-GEN applications.

## 6 Conclusion

In conclusion, our study provides a comprehensive analysis of collaborative decoding techniques under the fast and slow generation paradigm. By examining speculative decoding, contrastive decoding, and emulator fine-tuning, we identify key performance factors and conduct empirical analyses on collaboration frequency and positions. Our results demonstrate that the essential difference between large and small models lies in the uncertainty confidence of next token predictions during decoding.
We hope this work will inspire further research and innovation in model collaboration.

## Appendix A Related Works

### A.1 Collaborative Decoding

Speculative Decoding.
Leveraging the high performance of LLMs and the low inference cost of SLMs, speculative decoding (Leviathan et al., ; Kim et al., ; Sun et al., ) has become an effective method to accelerate the generation process in LLMs.
This approach involves two main steps: drafting and verification (Xia et al., ). In the drafting step, SLMs generate a candidate span comprising several tokens.
Subsequently, in the verification step, LLMs review all draft tokens in a single forward pass. The verified tokens are then fed back to the SLMs as input for subsequent generations.
Current efforts in this area can be categorized into two primary streams: drafting and verification.
In drafting, researchers focus on developing robust drafters that aim to generate as many tokens as possible that are likely to be verified successfully (Xia et al., ; Liu et al., ).
For verification, methods such as lossless and approximate approaches (Stern et al., ; Xia et al., ; Fu et al., ) have been proposed for both greedy decoding and nucleus sampling.
Nearly all the aforementioned studies employ draft tokens of a fixed length, adjusted as hyper-parameters. However, recent efforts have begun to determine the candidate length dynamically (Mamou et al., ; Shen et al., ), with more in-depth empirical studies also currently under exploration.

Contrastive Decoding.
Compared to the efficiency gains in speculative decoding, contrastive decoding (Li et al., ) focuses on improving factuality (Sennrich et al., ) and reasoning abilities (O’Brien & Lewis, ) through collaborative decoding.
In this approach, the next token logits from expert and amateur models are fused comparatively.
The underlying assumption is that tokens generated by the amateur model are less reliable and should be less frequently utilized in expert models, implying that greater scrutiny is required for tokens in expert models.
Recent studies also explore contrasting models with and without context, designated as expert and amateur (Jin et al., ), which helps reduce hallucinations.
Similar to speculative decoding, these efforts primarily examine how to fuse the logits, without addressing the extent and specific needs for collaboration.

Emulator or Proxy Tuning and others.
Unlike enhancements in inference time that involve two models, emulator tuning (Mitchell et al., ) and proxy tuning (Liu et al., ) employ three models to improve training efficiency.
In these methods, researchers fine-tune only SLMs which generate logits from both base and chat model versions.
By using the delta logits, the effectiveness of large chat models can be approximated through logits interpolation with large base models.
Additionally, Shen et al. () propose learning to collaboratively decode with multiple LLMs, including both general and specific domain models, to achieve domain generalization. Zhang et al. () suggest a collaborative generation framework leveraging large and small models under cloud-device scenarios that focuses on secure context-aware instruction following.

Scaling Law and Emergent Ability.
The experience of a “Scaling Law” has been successfully applied across various scenarios, including model size (Kaplan et al., ; Hoffmann et al., ), data size (Sorscher et al., ), and reward modeling (Gao et al., ; Rafailov et al., ), serving as a pivotal driver of LLMs development.
Research indicates that emergent abilities (Wei et al., ) manifest in larger models as dictated by the scaling law, where certain capabilities appear exclusively in large models, with smaller models failing to solve the same problems effectively.
In contrast, recent studies suggest that emergent abilities are significantly influenced by the continuity of indicators (Schaeffer et al., ; Hu et al., ).
When continuous metrics such as token-level match are employed, minimal divergence is observed between large and small models.
However, employing discrete metrics like accuracy in mathematical or reasoning tasks and pass rates on coding assignments reveals pronounced variations across models of different scales.
This paper delves into the differences in abilities between large and small models from the decoding perspective, which can be quantified using continuous metrics. Through this lens, we explore the key distinctions driven by token uncertainty.

Totally, deeper empirical analysis is still lacking, and the quest for further efficiency optimization continues.
This paper delves into the essential differences between LLMs and SLMs from the perspective of collaborative decoding.
Our exploration provides new insights into various methods and significantly enhances the efficiency and quality of the aforementioned techniques.

## Appendix B Definition of More FS-GEN methods

In accordance with specific definitions for various FS-GEN methods in $\S$ 2.2, we introduce additional variants as follows:

Mitchell et al. () propose emulator tuning, which, like proxy tuning, is motivated similarly but differs in implementation.
In this configuration, the SLMs are also divided into a base version and a chat version, with their logits represented as $p_{s}$ for the base version and $p^{\prime}_{s}$ for the chat version. The fused logits are represented by $p_{f}=p_{l}\cdot\left(\frac{p^{\prime}_{s}}{p_{s}}\right)$, indicating that $O_{g}$ combines the insights of the incremental changes between the tuned and untuned states of the SLMs, approaching the performance of a fully tuned LLMs.

Zhang et al. () propose CoGenesis for secure, context-aware instruction following. In the logits-level variant of CoGenesis, the inputs for LLMs and SLMs differ, denoted as $X_{l}\neq X_{s}$. This approach utilizes predictions from both SLMs and LLMs, where SLMs generate personalized content and LLMs produce outlines.
In this configuration, the reference logits are calculated as $p_{f}=f(p_{l}-p_{s})$, where $f$ represents a trained model that combines features from both, and $\mathcal{O}_{g}$ outperforms both $\mathcal{O}_{l}$ and $\mathcal{O}_{s}$.
This indicates that $\mathcal{O}_{g}$ effectively leverages the strengths of both models while preserving the privacy of user data.

## Appendix C Implementation Details

In speculative decoding, the number of tokens generated by draft small models is set as a hyperparameter, denoted as $K$.
In our empirical study, we define $K=1$ as representing oracle decoding rather than speculative decoding. Although the experimental results are equivalent, implementing and analyzing $K=1$ is simpler.

For each task across all methods, we adapt in-context few-shot to enhance generation performance under base models, using 5-shots for MMLU, 10-shots for MBPP, and 8-shots for GSM8k. The few-shot examples are unified into a chain-of-thought format (Fu et al., ).

Note that we regard the outputs from collaborative decoding as references; therefore, we do not compute accuracy for each task. By default, results from collaborative decoding are generally better than those from either smaller or larger models alone. Our primary focus is on exploring the differences between larger and smaller models, rather than comparing them to the ground truth.

## Appendix D More Empirical Results

### D.1 Results on Collaboration Position

We provide heatmaps of mismatch positions relative to Pythia model combinations for contrastive decoding and emulator fine-tuning in Figures 14 and 15 respectively. These heatmaps reveal findings similar to those obtained from oracle decoding and support the adage that “well begun is half done”.

Figure: Figure 14: Percentage of mismatch positions relative to length for model combinations across various datasets (Pythia Series, Contrastive Decoding).
Refer to caption: x11.png

Figure: Figure 15: Percentage of mismatch positions relative to length for model combinations across various datasets (Pythia Series, Proxy FineTuning).
Refer to caption: x12.png

The difference between Figures 11 and  13 lies in the selection of tokens within the generated content. Specifically, we select only one token for the former and five continuous tokens for the latter.
This relationship between the top-k logits for each position and mismatches with large models is clearly evident in the visualization.
The results further indicate that identifying mismatch positions benefits from additional context at each step.

We provide additional model combinations for visualization of token-level uncertainty, as shown in Figure 16, 17, 18, 19, 20, 21.
This figure also demonstrates that the uncertainty in tokens of SLMs results in incorrect answers.
Examples illustrating uncertainty in SLMs using contrastive decoding and proxy tuning are presented in Figures 22 and 23, respectively.

Figure: Figure 16: Visualization of Token Contributions. Red signifies mismatch tokens, Blue indicates match tokens, with darker shades representing higher weights.
Refer to caption: x13.png

Figure: Figure 17: Visualization of Token Contributions. Red signifies mismatch tokens, Blue indicates match tokens, with darker shades representing higher weights.
Refer to caption: x14.png

Figure: Figure 18: Visualization of Token Contributions. Red signifies mismatch tokens, Blue indicates match tokens, with darker shades representing higher weights.
Refer to caption: x15.png

Figure: Figure 19: Visualization of Token Contributions. Red signifies mismatch tokens, Blue indicates match tokens, with darker shades representing higher weights.
Refer to caption: x16.png

Figure: Figure 20: Visualization of Token Contributions. Red signifies mismatch tokens, Blue indicates match tokens, with darker shades representing higher weights.
Refer to caption: x17.png

Figure: Figure 21: Visualization of Token Contributions. Red signifies mismatch tokens, Blue indicates match tokens, with darker shades representing higher weights.
Refer to caption: x18.png

Figure: Figure 22: Visualization of Token Contributions in Contrastive Decoding. Red signifies mismatch tokens, Blue indicates match tokens, with darker shades representing higher weights.
Refer to caption: x19.png

Figure: Figure 23: Visualization of Token Contributions in Proxy Tuning. Red signifies mismatch tokens, Blue indicates match tokens, with darker shades representing higher weights.
Refer to caption: x20.png

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: How generalizable do you believe your findings are to other language tasks or domains?
How do you think the collaborative patterns might change, If different sampling technique is used.
soundness: 3
strengths: Originality:- The paper introduces a novel FS-GEN framework.
Quality:- The tables and figures are very well used.
The paper is written with a great clarity. 
significance:- The paper compares from smaller models to larger ones, based on the number of parameters.
summary: The paper explores collaborative decoding strategies between large language models (LLMs) and small language models (SLMs). The authors introduce the FS-GEN framework, categorizing LLMs as System 2 (slow and deliberate) and SLMs as System 1 (fast and intuitive). The research focuses on decoding methods like speculative decoding, contrastive decoding, and proxy tuning to improve efficiency and mitigate issues like high inference time.
weaknesses: Could provide more discussion of practical applications.
Trade-offs between the inference time and cost can be a great addition. 
The experiments focused on only few tasks like:- MMLU-STEM, GSM8k, and MBPP, Having experiments over domain specific datasets can give a better understanding.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Related work is pushed to the Appendix. This is a strange choice. I understand there might have been a space crunch, but Related Work makes much more sense to be in the main paper.
soundness: 2
strengths: Paper studies a relatively under explored but important and emerging area of research.
The findings are interesting, particularly the 2:8 law, collaborations being most necessary at the beginning of decoding and that high uncertainty tokens within System 1 are more likely to require collaboration.
Some of the findings could spur targeted research in the field of collaborative decoding.
Experimental benchmarks cover different capabilities like knowledge, math and coding, as well as two LLM families.
summary: The paper studies collaborative decoding, where small language models and large language models work together in the decoding process. In particular, the paper offers a unifying perspective on 3 different collaborative decoding techniques: proxy tuning, speculative decoding and contrastive decoding. Authors categorize the larger model as System 2 and smaller model as system 1.
The paper studies the 3 techniques, their commonalities and differences through their framework FS-GEN (Fast and Slow Generating).
They find that only small fraction of decoding steps require collaboration and that System 1 and 2 follow a scaling law related to parameter ratios.
weaknesses: The System 1 and System 2 analogy is not well fleshed out, to the point where it feels more like a distraction from the main contributions.

The line fits on the param ratio scaling plot aren't very convincing.

The uncertainty analysis is only qualitative - quantitative metrics to support this hypothesis (covering different tasks and model families) are missing. Without them its hard to have confidence in this finding.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: - It is not clear what exactly is being illustrated in Figures 11, 12, and 13. What are the different features? 
- How does one use the insights from this paper for contrastive decoding and proxy tuning?
- Currently, greedy decoding is used to establish whether collaboration is required or not. I wonder if the next token perplexity could be another measure.
soundness: 3
strengths: - Proposes a new framework to analyze the collaborative behavior between models
- Empirical results shed new light on this collaborative behavior. In particular, the scaling law for collaboration and frequent positions of collaboration are quite interesting.
summary: The paper analyzes the patterns of collaboration between SLMs and LLMs when used in a collaborative decoding/training setup. By analyzing this behavior across multiple collaboration setups, tasks, and model families, the authors draw the following conclusions:
- The collaboration frequency peaks at about 20%, with the maximum collaboration happening when there's the biggest gap in the model sizes. In fact, there's an inverse scaling law connecting the model size ratio and the collaboration frequency (more clearly evident for Qwen models than Pythia). 
- Most of the LLMs/System 2 interventions are required at the start of the decoding and for tokens for which SLMs are uncertain.
weaknesses: - The paper analyzes speculative decoding, contrastive decoding, and proxy tuning. Except for speculative decoding, it's not clear if the analysis provides any executable insights for the other two setups. 
Drawing questionable analogies with human cognitive processes just because one model runs fast and the other slow and then commenting about how the collaborative distributions are different (L127-L129) is extremely flawed reasoning. The analogy doesn't make sense, except for the fact that one model is faster and the other is slower.   

Comments about writing:
- Why is O_g being used and not O_f for p_f (fused logits) in Section 2.2
- L053: "allow" -> "allows"
- L195: "produce" -> "produced"

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 1
questions: Could you elaborate on the motivation of using system 1 - system 2 reasoning for collaborative decoding with SLMs and LLMs, specifically?
soundness: 1
strengths: The paper asks an interesting question and presents several findings. The idea to take inspiration from system 1 and system 2 is interesting.
summary: This paper presents an investigation into collaborative decoding between small and large language models, attempting to formalize it from the perspective of a system 1 / system 2 collaboration, where system 1 operates quicly and intuitively, while system 2 functions in a more slow and deliberate manner. The paper focuses on the differences between system 1 and 2 in the context of decoding, when system 1 would underperform compared to system 2 and how efficiency of the compound system can be improved. For their investigation, the authors use the Qwen and Pythia series. To evaluate the system, they consider MMLU-STEM, GSM8k and MBPP. The analysis focusses on two aspects of collaboration: frequency and position, where the former refers to how often the models should interact, where as the second one refers to the specific points of interaction. They find thta collaborative interactions are most critical at the beginning of the generation, and that the optimal frequency is around
 80-20, depending on the task.
weaknesses: My main qualm with the work is the presentation of the paper, which almost reads like a slide deck: plenty of conclusions and graphics, but little to no details about how the experiments are actually set up or how the conclusions are drawn. I also don't see any evidence of how well the collaborative decoding actually works (that is, there are no accuracy scores reported), and how that may depend on the frequency or place of collaboration). The many figures are hardly described. There is also no discussion of how the results are different between the benchmarks and whether that may make sense given the topics.

Lastly, while I like the idea of interpreting collaborative decoding as a system-1 system-2 scenario, but the current work does not really convince me that it makes sense to explore collaborative decoding with SLMs and LLMs in this way. Wouldn't LLMs be better both at the intuition and the deliberate reasoning?

In sum, it could be that the paper contains many interesting results, but if so, the current presentation does not do them justice.

NB: the related work section is in the appendix and is not even referred to

Meta-review: The paper examines collaborative decoding, where small and large language models work together during the decoding process. It unifies three techniques: proxy tuning, speculative decoding, and contrastive decoding, framing them through the FS-GEN (Fast and Slow Generating) framework. The larger model is characterized as System 2, which operates slowly and deliberately, while the smaller model is System 1, functioning quickly and intuitively. The study finds that only a small fraction of decoding steps require collaboration and identifies a scaling law related to parameter ratios. Using the Qwen and Pythia series, evaluated across datasets like MMLU-STEM, GSM8k, and MBPP, the research highlights that collaborative interactions are most critical at the start of the generation process, with an optimal interaction frequency around an 80-20 ratio, depending on the task.

My decision is to reject the paper, as it requires substantial revisions, particularly in areas highlighted by Reviewer VTMo. Additionally, the authors need to clarify the terminology used throughout the paper.

Decision: Reject