Mixture of Inputs: Text Generation Beyond Discrete Token Sampling
Authors: Yufan Zhuang
Liyuan Liu
Chandan Singh
Jingbo Shang
Jianfeng Gao
Venue: NeurIPS 2025 poster ()
Keywords: language model
continuous representation mixture
Primary area: deep_learning

Abstract: In standard autoregressive generation, an LLM predicts the next-token distribution, samples a discrete token, and then discards the distribution, passing only the sampled token as new input. To preserve this distribution’s rich information, we propose Mixture of Inputs (MoI), a training-free method for autoregressive generation. After generating a token following the standard paradigm, we construct a new input that blends the generated discrete token with the previously discarded token distribution. Specifically, we employ a Bayesian estimation method that treats the token distribution as the prior, the sampled token as the observation, and replaces the conventional one-hot vector with the continuous posterior expectation as the new model input. MoI allows the model to maintain a richer internal representation throughout the generation process, resulting in improved text quality and reasoning capabilities. On mathematical reasoning, code generation, and PhD-level QA tasks, MoI consistently improves performance across multiple models including QwQ-32B, Nemotron-Super-49B, Gemma-3-27B, and DAPO-Qwen-32B, with no additional training and negligible computational overhead.

Mixture of Inputs:
Text Generation Beyond Discrete Token Sampling
Yufan Zhuang'. Liyuan Liu', Chandan Singh', Jingho Shang', and Jianfeng Gao
I UC San Diego
Microsoft Research
Abstract
In standard autoregressive generation, an LLM predicts the next-token distribution, samples discrete token, and then discards the distribution, passing only the sampled token as new input. To preserve this distribution's rich information, propose Mixture of Inputs (Mol), training-free method for autoregressive generation. After generating a token following the standard paradigm, we construct new imput that blends the generated discrete token with the previously discarded token distribution. Specifically, we employ Bayesian estimation method that treats the token distribution as the prior, the sampled token as the observation, and replaces the conventional one-hot vector with the continuous posterior expectation the new model input. Mol allows the model to maintain richer internal representation throughout the generation process, resulting in improved text quality and reasoning capabilities. On mathematical reasoning, code generation, and PhD- level QA tasks, Mol consistently improves performance across multiple models including QwQ-32B, Nemotron-Super-490, Gemma-3-27B, and DAPO-Qwen- 32B, with no additional training and negligible computational overhead.
Introduction
Large language models (LLMs) are trained predict the full distribution of the next token given an input context. To generate desirable sequences of text, various methods have been proposed to sample discrete tokens from these iterative next-token distributions 2 After the sampling process, only the discrete token is passed as the new input, and the rich predicted distribution is discarded. This process forces the model to commit to single path in its reasoning, potentially abandoning valuable alternatives that could lead to better solutions.
On the other hand, human thinking first occurs in high-dimensional and fluid manner before being articulated as natural language. Inspired by this cognitive process, we explore methods to enable LLMs to utilize only articulated natural language but also partially-frmeed ideas, competing possibilities, and conceptual associations that exist in a probabilistic space before crystallizing into words.
Specifically, we propose Mixture of Inputs (Mol), novel approach that takes as input not only discrete, sampled token but also the sampled token's distribution. This preserves the model's uncertainty and allows to conduct inner speech in high- nigh-diimesional space. We employ a Bayesian estimation method, treating the token distribution as the prior and the sampled token as the observation, then replacing the conventional one-hot vector with the continuous posterior expectation. With this posterior expectation, weighted average embedding is passed as the new input subsequent prediction steps.
Code available
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Regular Generation
Diacreaa takens are surrpled from ditt busón udaongresstecly
Mixture of Inputs (Mol)
Incuus are aad entoeddings from outpus and dist busion
joy
all
him
Output
joy
but
leurs
Language Model
Language Model
Output Disribution
Ougut Distribuiion
>,
00;0;
bad
sad
awe
mad
Joy
bad
sad
awe
mad
joy
Irput
soon
felt
joy
all
over
Input
He
soon
felt
h1
hg
Prompe
Discrata Takoma
Pronst
Mised Emboddinge
Comparison of the regular autoregressive generation pipeline (left) and our proposed Mixture of Inputs (MoI) strategy (right). In regular generation, only the discrete sampled token is fed back at each step, whereas Mol preserves the full sampling distribution by computing a blended embedding h Σ; wye,, with weights w; interpolating embeddings {er) letting the model consider several plausible tokens simultaneously within single forward pass.
Figure 1:
Mol is conceptually intuitive and requires no additional training architectural changes, making immediately applicable to existing models. We implemented our method in modern LLM serving frameworks and found it to have negligible computational overhead and minimal deployment effort.
We evaluate Mol across range of tasks includinn mathematical reasoning, code generation, and graduate-level question auswering-wher maintaining uncertainty can play crucial role in step-by- step inference. Across these domains, Mol brings consistent performance improvements for multiple models including QwQ-32B,
- Nemotron-SSuper-993. Gemma-3-27B, and DAPO-QWen-32B.
Related Work
Linearity of Embedding Representations The foundation of our work builds upon emerging research on the continuous nature of language model embedding spaces. Semantic linearity has been observed in embedding spaces dating back word embedding models [ and has been shown in various ways in modern LLMs 47 A more recent work demonstrates that transformer language models naturally learn to process inputs and outputs as smooth, continuous functions rather than merely as discrete tokens [8] This finding suggests that models inherently operate in continuous latent space, even when traditionally constrained to discrete token processing. Similarly, Vector- ICL [9] shows that LLMs can effectively decode and process projected text embeddings via linear projection layers when provided with in-context demonstrations, While Vector-ICI. projects external continuous data into the embedding space, our Mol directly leverages the linearity of the existing embedding space, demonstrating that meaningful representations can be created through lincar combinations of token embeddings. Our work extends these insights by applying them specifically to preserve distributional information during the generation process, showing that this approach can enhance reasoning capabilities without model modifications.
Continuous Chain of Thought Chain-of-thought (CoT) prompting and related works improve language model performance by encouraging step-by-step reasoning through natural language [TO]- 12]. However, these approaches rely on discrete text tokens, which can become inefficient and lengthy. More recently, COCONUT (Chain of Continuous Thought) [13] addresses this limitation by operating directly the model's hidden state space rather than generating explicit text. By feeding the model's 's hidden state back as input, COCONUT enables more efficient reasoning that condenses lengthy thoughts into single tokens without the overhead of explicit thought generation. While COCONUT manipulates hidden states during multi-step reasoning processes, our Mol similarly leverages continuous representations but focuses specifically on the input embedding space during token generation. This key difference allows our approach to achieve improved reasoning without requiring architectural changes or model retraining, making a more lightweight and accessible intervention.

---

Prompt and Weight Merging Linearity of LLM representations has been explored in few related applications. Motivated by the success of methods that improve performance by ensembling multiple LLM calls [14)16], learning an ensemble of soft prompts or compressing a large prompt have been studied to enable strong performance without increasing computational cost [17] Similarly, mechanistic methods for steering have proposed adding different latent vectors to elicit desired LLM behaviors [20-22). The concept of linearity in neural networks extends beyond input representations to model parameters themselves. Recent work demonstrates that when two language models with shared initialization are combined through linear interpolation their weights, their internal representations blend produce stronger model [23]. This discovery has enabled various merging techniques, from basic weight averaging to more sophisticated approaches [24] 25]. Mol applies similar linearity principles but at the level of individual tokens rather than full prompts or model weights.
Methods: Mixture of Inputs
When humans think, they often use natural language as an internal dialogue, but thinking is more fluid and multidimensional than just discrete words and sentences. Our cognition includes partially-forman ideas, competing possibilities, and conceptual associations that exist in probabilistic space before crystallizing into specific language.
Our proposed method mirrors this cognitive reality by enabling LLMs to take as inputs both discrete tokens (representing specific linguistic choices) and token distributions (capturing the uncertainty, nuance, and competing possibilities that exist human thought). By combining both as the model input, obtain richer representation that better reflects how human thinking operates balancing the concrete and the probabilistic aspects of cognition.
Specifically, we introduce Mixture of Inputs (Mol). The core idea is to reinterpret token mixing as probabilistic inference under Bayesian model. This formulation enables principled mechanism to reconcile the model's prior belief (the output distribution) with its observed evidence (the sampled tokens), resulting in a more robust and statistically grounded method for input blending.
3.1 Token Generation and Embedding Aggregation
A key strength of Mol lies in its simplicity and modularity: it enhances the input representation without altering the model architecture or the underlying sampling algorithm. Mol operates after the language model produces its output distribution and before the next token is fed back into the model for the subsequent generation.
Token Generation Let {e:) E Rd be embedding weights, with hidden dimension d and vocabulary size V. At each decoding timestep I, the language model outputs a probability distribution P: YV over the vocabulary. This is typically followed by sampling step that selects token y, (c.g.. via top-k, nucleus sampling, or temperature scaling), In conventional approaches, the model would retrieve the embedding ey, corresponding to the sampled token and feed into the next layer as the sole input.
Mol does not modify the sampling process itself: the sampled token y: is still used as the output token. This design makes Mol fully compatible with any decoding strategy and seamlessly integrable into existing autoregressive generation pipelines.
Embedding Aggregation Mol first uses both the sampled token y₁ and the distribution {pr,s) to compute ( as in Section 3.2 then uses {w,,+} to construct mixed embedding vector h
h
Wi where WiF 2 O. Σ Wtni
This representation allows the model to reason over distribution of plausible next tokens rather than committing to a single discrete choice, effectively enabling a form of "inner speech" with richer representational capacity.
3

---

3.2 Bayesian Input Construction with Mol
To capture the distribution information, a naive idea might simply be to directly mix the inputs according to the output distribution, setting Wa, pt,s. However, this approach only treats the token distribution as the imput and neglects the sampled next token. In Section we experiment with this approach (referred to Direct Mixture) and find that it leads to performance degradation in most cases.
Instead, Mol combines two sources of information: (1) the output distribution P,. representing the model's prior belief over possible next tokens, and (2) the sampled token y/, representing a concrete observation drawn from this belief.
To reconcile these sources, Mol treats the sampling process as probabilistic evidence and formulates the blending of representations as Bayesian inference problem. Specifically, it constructs a posterior-weightee mixture over token embeddings by computing a new weight vector w, {wr,i)==1 V/ that incorporates both the uncertainty in p, and the evidence from y!-
The resulting mixed embedding h, is given by Equation [ and it replaces the embedding for the discrete token as the input to the next decoding step (i.e., replaces ey, with h2). What changes the internal representation passed into the model, allowing the decoder to reason over both the chosen token and the context of plausible alternatives.
4 Mixing Weight Estimation
Here, elaborate our proposed Bayesian estimation method for w, (
4.1 Dirichlet Mixture Model
In probabilistic modeling, prior encodes belief before observing new data. Accordingly, we begin by constructing a prior distribution over token choices based on the model's output logits. Specifically, we assume the prior distribution to be Dirichlet, with concentration parameter a.
We view y as the output of the sampling process and assume the sampled token comes from a multinomial distribution parametrized by Then, estimate the mixing weight w by conducting the posterior estimation.
4D Di)( y Multinomial(un)
where = H(p) PP
H(p) is the normalized entropy of p
α
w
This formulation ensures that tokens with higher model confidence (i.e., lower entropy) exert stronger influence on the posterior, while still respecting the sampled outcome. We will go over each part of the Bayesian model in the following sections.
4.2 Estimating Mixing Weight
Let P, E - be the next-token distribution step and let Vr., {0, 1) indicate the sampled token (yr.; = iff token is chosen). We estimate the mixing weights w, by Bayesian posterior inference in Dirichlet-Multtnomill model.
Entropy-scaled prior.
Define the normalized entropy H as the following
H = H(P,)
Pt. log
H e E
(2)
log V
1=1
We place Dirichlet prior
we Dir(a);
a ==H(p,) Pr:
so that the total concentration Σ, 0; H(p,) grows with uncertainty and vanishes when the model confident. So when uncertainty is high, the prior distribution will be more widespread over P and vice versa.

---

Pseudo-count observation. increases as confidence rises:
The sampled token contributes a single pseudo-count whose weight
G = 13+11-177 y1,s,
with hyperparameter 3.
(4)
The hyperparameter 3 controls the concentration over mixing weight. Smaller values emphasize more on output distributions, while larger values highlight more the sampled output token. The effect of [ is easier to observe in Eq. (5) We also conduct an analysis of I's empirical effect in Section 7.1
Posterior mean. Dirichlet conjugacy yields the posterior mean of we, and we use that estimation as our mixing weights:
11, Σ, a +N
a, + C
H Pt.i + (8+1-1H V1.i
with N=
Σ C
(5)
3+1
Behavior of w. Eq. smoothly interpolates between the distribution (w., Pt,s when H + 1) and the one-hot token (W', - yr, when + 0), thereby reconciling distributional and discrete evidence in single principled estimator.
The complete procedure for computing the mixture of inputs is summarized in Algorithm |
5 Experimental Setup
Algorithm 1: Mixture of Inputs
We evaluate MoI across diverse suite of bench- marks spanning competition mathematics, com- binatorial problem solving, program synthesis, and graduate-level question answering. These tasks vary widely in structure and domain, allow- ing us assess Mol's 's generality and effective- ness across distinct application settings.
Require: Sampling distribution P,, sampled token
y, hyperparameter P. and embeddings {ei)
1. Compute entropy H with Eq. 2
2. Compute mixing weight wh.s with Eq 5 return h, E. We.
5.1 Tasks and Metrics
To ensure a comprehensive evaluation, we select four challenging benchmarks that span distinct reasoning domains and require different cognitive skills, from symbolic manipulation procedural generation and scientific comprehension:
AIME [26] consists of complex high-school level mathematical problems that often require multiple stages of symbolic reasoning, algebraic manipulation, and geometric insight. We use the official AIME datasets from 2022 to 2024 and evaluate models based exact match accuracy, reflecting their ability to arrive at precise, correct solutions.
Count Down 4 (27) is synthetic numerical reasoning task that presents models with arithmetic puz- zles. It requires deriving a target number by applying sequence of operations (addition, subtraction, multiplication, division) fixed set of four input numbers. This benchmark emphasizes procedural and combinatorial reasoning. We report the success rate, indicating whether the model arrives at the correct. final equation.
LiveCodeBench (28) is a dynamic and realistic code generation benchmark that includes tasks ranging from simple string manipulations advanced data structures and algorithms. Each prob- lem specifies a goal in natural language, and the model must generate executable code that meets functional correctness criteria. We use pass@1 1 the proportion of correct solutions on the first attempt as the primary evaluation metric.
GPQA 29 highly challenging multiple-choice question answering benchmark drawn from graduate-level science and engineering exams. Its diamond subset features the most difficult ques- tions that demand domain-specific knowledge, long-range reasoning, and the integration of multiple concepts. We evaluate models based on multiple-choice accuracy.
5.2 Models
We evaluate Mol using 4 state-of-the-art open-source LLMs with advanced reasoning capabilities.
5

---

QwQ-32B 33 is optimized for mathematical and logical reasoning through curriculum of instruc- tion tuning on symbolic tasks, math word problems, and chain-of th-uuht datasets.
1aama-..--Nmooronnee
[3] is derived from Meta's s Llama 3.3 70B model [32 The model
underwent neural architecture search to optimize for inference efficiency, followed by supervised
fine-tuning and reinforcement learning. These techniques were applied to enhance the model's reasoning abilities, instruction following capabilities, and tool-calling performance.
Gemma-3-27B [33] is part of Google's s Gemma 3 family muniiioalll (text + image) models with 128 K token context windows and an integrated SigLIP vision encoder. The 27B variant instruction- tuned for chat and reasoning.
DAPO-Qwen-32
[4] is a customized version of Qwen2.5-32BB [35 that incorporates Decou- pled Clip and Dynamic Sampling Policy Optimization (DAPO). which stabilizes and scales RL for long chain-of-thougle reasoning. This model is designed to encourage faithful and step-consistent reasoning trajectories.
5.3 Baselines
To quantify the benefit of MoI, we compare it with two decoding schemes that keep the underlying model architecture and sampling mechanism fixed. The primary baseline (Standard) is the widely used nucleus sampling with lemperature scaling M. It represents the default inference recipe shipped with each model. Our second baseline (Direct Mixture) constructs the imput representation weighted sum of token embeddings using the softmax probabilities as coefficients, i.e., computing the value of h, as E Pt.je,. Unlike Mol. it performs no Bayesian reconciliation between the distribution and the sampled token, providing a stringent ablation for assessing the value of our posterior estimator. We also tried directly feeding the mixed output hidden states, but we found that the models cannot make sense of the hidden states without retraining.
simple
5.4 Hyperparameter Settings
We perform 5 runs for all experiments and report the average. For AIME and Count Down 4, we perform hyperparameter grid search on baselines, Direct Mixture and Mol with / E { 3. 1, 4, 8). T E {0.6, 0.8, }} and top-p E (0.4, 0.6, 0.8, 0.95]. We report the mean result of the best configura- tion for all three methods. We investigate the importance of these hyperparameters in Section 6.2 For GPQA Diamond and LiveCodeBench, we use the universal hyperparameter for all of them with T = 0.6, top-p 0.95, B more details can be found in Appendix F
6
Main Results
6.1 Mol Boosts Capabilities of LLMs
Table reports accuracy on four reasoning intensive benchmarks for four open-source LLMs. Across all 16 model-task pairs, our approach Mol either matches or outperforms the Standard autoregressive baseline, with an average absolute gain of 1.8%. In contrast, the ablation that removes distribu- tion-smoothing (Direct Mixture) degrades performance in most cases, underscoring the importance of our Bayesian smoothing.
Consistency across model scales. Mol achieves gains for both medium-sized (Gemma-3-27B and larger (32 to 49 B-parameter) models. The largest improvement appears on Somnoron-SSupe-4990 where Mol adds up to +4.1% GPQA-Diamond and +2.6% on Count Down 4, lifting the overall average to 55.45% (+2.36%). These results indicate that mmixture-of-inputt remains beneficial even when the underlying model already possesses strong zero-shot reasoning abilities.
Task-specific trends. Improvements are most pronounced on benchmarks requiring extended symbolic manipulation. Count Down 4 benefits the most (+3.7% mean gain), suggesting that explicitly representing uncertainty over arithmetic operations mitigates the compounding error typical in multi-step numerical reasoning. Gains on AIME and GPQA-Diamond further show that Mol generalizes from high-school mathematics graduate-level science QA, while BiveCodeBench sees more modest but still positive changes.

---

Table Main results on four benchmarks with four large language models. The "Input Info." column indicates the source of input passed into the model: Output Token uses only the sampled discrete token, Output Dist. uses the full output probability distribution, and Token Dist. combines both, Accuracy (%) is reported on AIME, Count Down 4, GPQA-Diamone, and pass @ is used on LiveCodeBench. Standard uses conventional sampling, that is temperature scaled nucleus sampling, Direct Mixture removes the posterior estimation, and Mol is our full approach. Shaded cells highlight Mol and its
performance gain (absolute difference over the conventional generation).
Model
Method
Input Info.
AIME
CounDDownd
GPQA-D LiveCodeRench
Avg
Standard
Outgut Token
77.78
79.25
58,08
76.32
72.86
Direct Mixture
Output Dist.
72.00
66.88
51.52
53.42
60.96
QwQ-32B
Mol
80.00
80.01
60.10
76.51
74.15
Token Dist.
Gain Standard
+2.22
+0.76
+2.02
+1.29
Standard
Output Token
54.89
56.93
60.60
39.92
53.09
Nemotron-Super-P99
Direct Mixture
Output Dist.
60.00
51.72
60.10
16.04
46.97
Mol
57.11
59.53
64.65
40.50
55.45
Token Dist.
Guin Standard
+2.22
+2.60
+4.05
+2.36
Standard
Output
56.51
46.97
40.09
Direct Mixture
Output Dist.
26.44
55.47
51.52
31.99
41.36
Gemma-3-27B
Mol
26.89
59.38
47.47
32.87
41.65
Token Dist.
Standard
+2.87
+4.50
+1.56
Standard
Output
64.67
42.42
58.28
Direct Mixture
Output Dist.
62.67
67.19
37.88
23.87
47.90
DAPO-Qwen-32B
Mol
64.44
78.75
42.93
55.18
60.33
Token Dist.
Gain Standard
+6.72
+0.51
+2.05
Expected by Best-of-N-Shots Tuning
Random Forest Feature Importances
Performance Gain
Importance
0.3
5
0.2
Parameter Beta
3
Temperature
0.0
4
12
14
Beta
Temperature
Figure 2: Hyperparameter Importance Analysis. Comparison of three key hyperparameters (B in Mol, top-p, and temperature) across four L.LMs on two mathematical reasoning tasks. Left: Expected performance gain (%) when optimizing each hyperparameter individually through best- of-N-shots tuning. The graph shows B consistently outperforms other parameters as N increases. Right: Relative feature importance derived from random forest regression analysis, confirming 3's strong influence (0.41) on model performance compared to top-p (0.32) and temperature (0.27). These results demonstrate that is highly influential for effectively controlling input mixing during thain-of-thought reasoning.
Role of autoregressive inputs. Feeding back the full output distribution alone is insufficient: Direct Mixture often harms accuracy (e.g., -22.9% LiveCodeBench for Nemotron Nemotron-Super-499 The combination of the sampled token and its distributional context lets the model retain a discrete anchor while preserving alternative hypotheses, yielding the best of both worlds.
Together, these findings demonstrate that Mol offers principled and consistently effective way to enhance multi-step reasoning. By marrying discrete choices with probabilistic context, it improves accuracy without sacrificing decoding efficiency or requiring model-specific fine-tuning.
6.2 Hyperparameter Importance Analysis
To understand which factors most strongly influence reasoning performance, we analyze three key hyperparameters: 8, top-p, and temperature. This analysis spans four LLMs and two mathematical

---

reasoning tasks, with multiple runs and grid search over the hyperparameter space, as described in Section 5.4
Fig. 2 provides two complementary perspectives on hyperparameter importance. The left plot tracks expected performance gain when optimizing each parameter individually through best-of-N-shon tuning, with experiment setup explained in Appendix[E.] As N increases from to 15, 8 consistently yields the highest gains, reaching nearly 7.5% improvement N=15, while top-p and temperature plateau at approximately 6.0-6.5%. This separation becomes particularly pronounced after N=10, suggesting that fs's impact grows with more extensive search.
The right panel quantifies each parameter's importance through random forest regression analysis, with experiment setup explained in Appendix [E.]] With inputs hyperparameters and accuracy as the target, this reveals ,3 as the dominant factor (importance score of 0.41), followed by top-p (0.32) and temperature (0.27).
7 Analysis
7.1
Task-Dependent Optimal Mixing Strategies
Different reasoning tasks may benefit from varied degrees of distribution mixing. To investigate this phenomenon, we analyze the parameter sensitivity of two distinet benchmark types: AIME (requiring advanced mathematical reasoning) and Count Down 4 (demanding extensive combinatorial enumeration). Fig- 3 visualizes how performance varies with the mixing parameter (3 across four LLMs, showing the deviation from each task's global mean accuracy.
The results reveal an interesting inverse relationship between task type and optimal values. AIME performance peaks at low 3 values (3 S 1), with accuracy dropping sharply when 3 > 1. In contrast, Count Down shows the opposite pattern, performing substantially below average at low values but excelling when / > 1. This divergence suggests fundamental differences in how distribution mixing affects distinct reasoning processes.
For reasoning-innensive AIME problems, low 33 values promote greater consideration of alternative solution paths while maintaining focus on the most promising directions. Conversely, for enumeration- intensive Count Down 4 problems, higher 3 values increase concentration on the most probable combinations, effectively pruning the vast search space.
These findings highlight the importance of approopproe 33 calibration when deploying MoL. Lower values suit open-ended reasoning, while higher values suit systematic enumeration an adaptability that fixed decoding strategies lack.
7.2 Case Study: Linear Prompt Blending with Various Lengths
Although our main experiments focus on loken-by-token blending at generation time, we also investigate whether a similar blending strategy applied to instruction prompts of varying lengths can boost performance. To this end, we perform 10-shot in-context learning on five sentiment analysis benchmarks using three medium-sizee sized LLMs, building on prior work showing that prompt wording and structure have major impact on classification accuracy 15
We assembled three prompt pools: (1) binary sentiment analysis Rotten Tomatoes [36], SST2 37 and IMDB [38], consisting of 96 prompts of length 3-16 words (mean 7.57); (2) 6-class emotion classification the Emotion dataset (39], with 32 prompts of length 3-15 words (mean 7.27); and
(3) 3-class financial sentiment on the Financial Phrasebank [40]. comprising 40 prompts of length 2-14 words (mean 6.51).
Our blending procedure first linearly extrapolates each prompt's embedding to the maximum length in its pool and then averages these fixed-length embeddings to form single "blended" prompt representation. This approach integrates semantic nuances from all constituent prompts while preserving their instructional intent.
Table reports 10-shot accuracy under the expectation over randomly drawn single-prompt, the blended-prone promm,, and the absolute gain over baseline. Across most benchmarks and models, linear prompt blending consistently outperforms random ingle-prompt selection, further demonstrating that embedding-sperd space mixing can be highly effective for boosting LLMs' capacity.
8

---

Table 10-shot in-context learning accuracy (%) for three LLMs (Llama3 8B 32 Mistral 7B 41 Gemma 7B (42)) on five sentiment analysis benchmarks. We compare the expectation of single-
prompt baseline against embedding-spaan prompt blending via linear interpolation of prompts.
Model
Method
Rotten Tomatocs
SST2
IMDB
Emotion
Financial Phraschank
Average
Single Prompt
91.58
94.12
87.26
53.82
68. 76
79.11
Llama3
Linear Interpolation
92.68
94.49
95.40
51.75
72.34
81.33
+1.10
+0.37
+8.14
-2.07
+3.58
+2.22
Single Prumpt
91.11
85.06
54.87
70.75
78.22
Mistral 7B
Lineur Interpolation
92.21
94.03
92.82
51.60
73.42
80.82
Gain
+2.89
+2.92
+7.76
-3,27
+2.67
+2.59
Single Prompt
86.66 66
87.18
87.31
50.77
72.63
76.91
Gemma 7B
Linear Interpolation
92.30
93.34
93.88
50.30
74.39
80.84
Gain
+5.64
+6.16
+6.57
-0.47
+1.76
+3,93
7.3 Throughput Analysis
The mixing weight calculation is lightweight and efficient. We perform throughput analysis, shown in 3 to examine the runtime overhead added by Mol. We measure generation statistics for solving the Count Down task, and we record the average input and output throughput over 5 runs. To better compare the throughput, we picked the benchmarks where the generation length is about same. The median difference between Mol-generated text baseline-generated text 1.7%.
Table Throughput analysis (tokens/s) for QwQ-32B with and without Mol in vLLM.
Accuracy Deviation from Taok Mean
Method
Input Speed
Output Speed
Standard Mol
62.87
1,143.31
61.36
1,101.44
AIME
Overhead
2.40%
3.66%
Count
-1
8 Discussion
Limitations and Future Work While Mol demonstrates consistent gains on a wide range of benchmarks, its current scope is intentionally focused on tasks that can be objectively evaluated. As result, applications such as open-ended generation or creative writing, where ob- jectives are less formally defined, remain outside the current scope and present promising directions for further study. Additionally, observe that the hyper- parameter 3 exhibits task-dependent be- havior. This suggests that different task types benefit from varying degrees dis- tributional mixing, phenomenon wor- thy of deeper theoretical exploration. Fu- ture work could investigate adaptive test-time 3 tuning strategies.
Figure 3: Task-dependent Optimal Mixing Strate- gies. The plot shows accuracy deviation from task mean across different values for AIME (reasoning- heavy) and Count Down 4 (enumeration-heavyy) eraged across four LLMs. Lower 3 values (3 < 1) significantly benefit AIME's performance while higher B values (3 > 1) improve Count Down 4. This diver- gence demonstrates how Mol's impact varies based on task characteristics: reasoning-intensti tasks perform better with stronger distribution mixing (low (i) to be more creative, while nnumeration-intensie tasks bene- fit from higher distribution mixing (high () that helps explore the combinatorial search space with more focus.
Conclusion We presented Mixture of Inputs (Mol), a training free enhancement to autoregressive generation that preserves distributional information. By treating input mixing as a Bayesian inference problem, Mol maintains richer internal representation throughout the generation process, allowing models to conduct a form inner speech beyond discrete tokens while requiring no architectural changes or additional training.

---

Our evaluation across LLMs and benchmarks demonstrates consistent performance improvements. Mol's conceptual simplicity, negligible computational overhead, and immediate applicability to existing models make it practical enhancement that bridges the gap between the high-dimensional nature of thought and the discrete nature of language.
Acknowledgement
Our work is sponsored in part by NSF CAREER Award 2239440, NSF Proto-OKN Award 2333790, Sponsored Research Projects from companies like Cisco and eBay, as well as generous gifts from Google, Adobe, and Teradata. Any opinions, findings, and conclusions or recommendations expressed herein are those of the authors and should not be interpreted as necessarily representing the views, either expressed or implied, of the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for government purposes not withstanding any copyright annotation hereon.
References
[1] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint 190400990, 2019.
[2] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. Advances in neural information processing systems, 27, 2014.
[3] Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies, pages 746-751,
2013.
[4] Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.036566 2023.
[5] Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models. arXiv preprint arXiv:2309.0094/ 2023.
[6] Yibo Jiang, Goutham Rajendran, Pradeep Ravikumar, Bryon Aragam, and Victor Veitch. On the origins of linear representations in large language models. arXiv preprint 200::2203300867, 2024.
[7] Jack Merullo, Noah A Smith, Sarah Wiegreffe, and Yanai Elazar. On linear representations and pretraining data frequency in language models. arXiv preprint arXiv:2504. 12459,, 2025.
[8] Samuele Marro, Davide Evangelista, X Angelo Huang, Emanuele La Malfa, Michele Lom- bardi, and Michael Wooldridge. Language models are implicitly continuous. arXiv preprint vrXiv:2504.03933 2025.
[9] Yufan Zhuang, Chandan Singh. Liyuan Liu, Jingbo Shang, and Jianfeng Gao. Vector-icl: In-context learning with continuous vector representations. arXiv preprint arXiv:2410.05629, 2024.
[10] Jason Wei. Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[11] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint ar0iv:2305. 100000 2023.
[12] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang. Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self- Refine: Iterative refinement with self-feedback. 2023.
10

---

[13] Shibo Hao, Sainbayar Sukhbaatar, Dilia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space. arXiv preprint arXiv:24/2.06766 2024.
[14] Amanda Bertsch, Alex Xie, Graham Neubig, and Matthew R Gormley. It's mbr all the way down: Modern generation techniques through the lens of minimum bayes risk. arXiv preprint arXiv:2310.01387. 2023.
[15] John X Morris, Chandan Singh, Alexander M Rush, Jianfeng Gao, and Yuntian Deng. Tree prompting: efficient task adaptation without fine-tuning. arXiv preprint arXiv:2310.1400 2023.
[16] Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint (rXi: 2022.
[17] Guanghui Qin and Jason Eisner. Learning how ask: Querying Ims with mixtures of soft prompts. arXiv preprint arXiv:2104.06599 2021.
[18] Huiqiang Jiang, Qianhui Wu, Chin-Yev Lin, Yuging Yang, and Lili Qiu. LImlingua: Compressing prompts for accelerated inference of large language models. arXiv preprint arXiv:2310.05736, 2023.
[19] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM computing surveys, 55(9):1-35, 2023.
[20] Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering language models with activation engineering. arXiv preprint arXiv:2308. 10248, 2023.
[21] Nishant Subramani, Nivedita Suresh, and Matthew Peters. Extracting latent steering vectors from pretrained language models. arXiv preprint arXiv: 2022.
[22] Liu Yang, Zigian Lin, Kangwook Lee, Dimitris Papailiopoulos, and Robert Nowak. Task vectors in in-context learning: Emergence, formation, and benefit. arXiv preprint 47X2v.2501.09240. 2025.
[23] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, pages 23965-23999 PMLR, 2022.
[24] Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging Resolving interference when merging models. Advances in Neural Information Processing Systems, 36:7093-7115, 7115, 2023.
[25] Gabriel Ilharco, Mareo Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.0400 04409999, 2022.
[26] Mathematical Association of America. American invitational mathematics examination (aime), 2024. htt.o///to///////////////aaaaaaa
[27] Jiayi Pan, Junjie Zhang, Xingyao Wang, Lifan Yuan, Hao Peng, and Alane Suhr. Tinyzero. http://giihbb.coom//aaaaaaa////i
2025. Accessed: 2025-01-24.
[28] Naman Jain, King Han, Alex Gu, Wen-Ding Li. Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and lon Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint 2rXiv:2403.07974 2024.
[29] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference Language Modeling, 2024.

---

[30] Qwen. Qwq-32b: Embracing the power of reinforcement learning, March 2025.
[3]] NVIDIA. Llama-nemotron: Efficient reasoning models, 2025.
[32] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
[33] Gemma, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma technical report. arXiv preprint arXiv:2503.19778 2025.
[34] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng. Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei- Ying Ma, Ya-Qin Zhang. Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-source IIm reinforcement learning system at scale, 2025.
[35] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint 2rXiv:2412.15115 2024.
[36] Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the ACL, 2005.
[37] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631-16422 Seattle, Washington, USA, October 2013. Association for Computational Linguistics.
[38] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguisties: Human Language Technologies, pages 142-150, Portland, Oregon, USA, June 2011. 1. Association for Computational Linguistics.
[39] Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. CARER: Contextualized affect representations for emotion recognition. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pages 3687-3697, Brussels, Belgium, October-Novembee 2018. Association for Computational Linguistics.
[40] Malo, A. Sinha, P. Korhonen, J. Wallenius, and P. Takala. Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65, 2014.
[41] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b, arXiv preprint arXiv:2310.06825 2023.
[42] Gemma, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08225, 2024.
[43] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang. Joseph E. Gonzalez, and lon Stoica. Judging -uue- with MT-hench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023,
12

---

[44] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and F. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
[45] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2300.00574, 2023.
[46] Anthropic. Claude 3.5. https: www.anthropic.com 2023, Accessed: 2025-05-15.
[47] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
[48] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024.
13

---

A Comparing over Hyperparameter Grid Search
We perform a head-to-head evaluation between our method (MoI) and the standard text generation with lemperature-scalte nucleus sampling (baseline), under two complementary regimes:
Best-case: each method is run with its single best-performing hyperparameter configuration, Fig. summarizes the results.
Grid-average: performance is averaged across all combinations in the hyperparameter grid (see details in Section 5.4). Fig. A22 provides these averages and confirms that the gains are not an artifact of cherry-pun therry-picking one lucky setting.
AIME
Baseline
68.0
81.0
00.0
67.0
83.5
83.0
50.0
05.0
29.5
56.0
$4.0
52.9
62.0
OwQ-328
Huma-33.-lemsttn-4468
Gemma3-278
DAPO-Qwwn-328
Count Down
Daseline
Mol
84.0
62.0
62.0
$1.0
82.0
76.0
74.9
50.0
57.0
72.0
96.0
56.0
70.0
76.0
68.
Hlmma33..Nemstron-eee
Gemma3-278
MMPO-QWen-328
Figure A1: We show comparison of distributions of evaluation results across the best top-p and temperature hyperparameter for baseline and with Mol. The results indicate strong performance gain brought by incorporating the sampling distribution in the generation process.
B Generalization of single hyper-parameter setting across tasks
A practical concern when adding new decoding hyperparameters is whether the values tuned one task will transfer to others. To investigate this, we conduct grid search over the hyperparameters described in Section 5.4 For each (method, model) pair, we keep the configuration that maximizes the average accuracy on AIME & CountDown and freeze for the remainder the study.
Table Al shows the results on the held-out
and LiveCodeBench. The single-tune tuned setting consistently outperforms Standard nucleus sampling on 15 of the 16 (model, task) pairs.
Overall, these results suggest that Mol requires only modest tuning effort: once the hyperparameters calibrated small proxy set, they generalize robustly new tasks and domains without much further adjustment.
C Results on Instruction-followig Tasks
We conducted additional experiments on MT-Bench 43 using the four larger models, and the
two smaller models, Llama-3.1-88 [32] and Qwen-2.5-14B (5). Below, we report the average performance for standard decoding (temperature=0.6. top-p=0.95) and our Mixture of Inputs (Mol)
14

---

AIME, Average Hyperparameter Comparison
Baseline
Mol
27.0
56.3
61.0
78.0
56.0
63.0
27.0
$5.0
24.0
62.0
75.0
5355
22.0
21.0
QwQ-128
Gemma33276
DAPD-Dwee-118
Count Average Hyperparameter Comparison
Baseline
Mol
59.0
78.0
57.
35.0
72.9
54.
52.0
68.0
52.9
66.0
QwQQ328
Gemma3-278
DAPO-Qeen-328
Figure A2: We show a comparison of distributions of evaluation results across all top-p and tem- perature hyperparameters. The results indicate almost universal performance gain across average hyperparameter settings.
Table Al: Additional results on four benchmarks with four large language models. For every (method, model) we the decoding hyperparameters on AIME and CountDown 4, fix the best setting, and reuse it when assessing
Diamond and LiveCodeBench. Accuracy (%) is reported on AIME, Count Down 4, GPQA A-Dimoodd,,, and pass used on LeveCodeBench. Standard uses conventional sampling, that is cenperature-scaled nucleus sampling and Mol is our full approach.
Model
Method
Input Info.
AIME
CounDown4
GPQA-D
LiveCodeRench
Avg
Standard
Output Token
76.89
78.04
58.08
76.48
72.37
QwQ-32B
Mol
78.44
79.22
61.41
77.85
74.23
Token + Dist.
Gain Standard
+1.55
+1.18
+3.33
+1.37
+1.86
Standard
Output Token
54.67
56.77
63.64
40.35
53.86
Nemotron-Saper-499
Mol
57.11
59.22
64.95
40.90
55.55
Token Dist.
Gain vs.
+2.44
+2.45
+1.31
+0.55
+1.69
Standard
Output Token
24.44
55.21
45.45
32.37
39.37
Gemma-3-277
Mal
26.67
57.19
46.26
32.92
40.76
Token
Gain vs. Standard
+2.23
+1.98
+0.81
+0.55
+1.39
Standard
Output Token
64.67
70,31
42.93
56.91
58.70
DAPO-Qwen-32B
Token + Dist.
75.00
43.94
57.50
59,50
Gain Standard
3.11
+4.69
+1.01
+0.59
+0.80
decoding (3=1,2,4, same decoding configuration), together with the relative improvements brought by Mol (A). As summarized in Table A2 Mol consistently outperforms the baseline for all models.
We also noticed Mol reduced repetition in the response. Particularly, we measured the 3-gram and 4-gram repetition rate in the response for MT-bench and found Mol reduces the repetition on 72.9% of responses, reducing the average 4-grams repetition rate from 4.27% to 3.1%%. also provide side-by-side comparison coding task as below (QwQ-32B),
With standard sampling (repetitive around HCA definition):
To address the problem of finding the highest common ancestor (HCA) of two nodes a binary tree, we need to clarify the definition. Based on common terminology, the lowest common ancestor (LCA) is the deepest node shared by both paths from
15

---

the nodes to the root. However, since the problem specifies "not LCA," we assume the HCA refers to the shallowest common ancestor (closest the root) [content continues...
With mixture of inputs sampling (reduced repetitiveness);
To find the highest common ancestor (HCA) of two nodes in binary tree, utilize a recursive approach similar to finding the Lowest Common Ancestor (L.CA.. The HCA is the deepest node that is ancestor of both given nodes. Here's 's how to implement this: [content continues...
Table A2: Comparison of conventional sampling (Baseline) and Mol scores across six large language models. For each model, report the baseline score, Mol score, and the relative improvement (A%) of Mol over the baseline.
Model
Baseline Score
MoI Score
A 5% (Mol Baseline)
QwQ-32B
9.25
9.51
+2.81
Nemotron-Super--ee Super-49B
9.41
9,48
+0.74%
Gemma-3-27B
9.05
9.38
+3.65 %
DAPO-Owen-2BB
8.96
9,46
+5.54%
Llama-3.1-88
8.24
8.65
+4.98%
Qwen-2.5-
8.87
9.32
+5.07% %
D Statistical Robustness of the Results
We conducted formal significance testing. Specifically, we applied McNemar's 's test to compare MoI against the baseline across all four benchmarks for each model. Table reppors the resulting p-values. These results validate that the observed gains are statistically sigmificant most cases.
To further address concerns about variance, we conducted an extensive 64-run evaluation on the AIME dataset across four models with the same configuration in experiments of Table [ The results in Table Table A4] consistent improvements from Mol.
Table A3: McNemar's test y-values comparing model performance across four benchmarks. Lower values indicate statistically significant differences between baseline and Mol.
Model
CountDown4
AIME
GPQA-Diamon LiveCodeBench
QwQ-32B
0.001
0.001
0.003
0..4
Nemotron-SSpper-48e
< 0.001
< 0.001
< 0.001
0.02
Gemma-3-27B
< 0.001
0.001
0.04
< 0.001
DAPO-Qwen-32D
<< 0.001
0.8
0.02
0.001
Table A4: Comparison of baseline and Mol performance across four large language models over 64
runs on AIME. Each result reports the mean (μ), standard deviation (a) and min-max score range. We observe consistent performance gain brought by Mol.
Model
Baseline (μ + ())
Baseline Range
Mol (μ + g)
Mol Range
Gain
QwQ-32B
76.02 2.51
12.22-82.22
77.66 + 2.06
73.33-82.22
+1.64
Nemotron-Super-4e
54,80 + 2.93
48.89-61.11
55.03 + 3.54
44.44-62.22
+0.23
Gemma-3-278
21.80 + 2.34
16.67-27.78
24.91 + 2.30
20.00-30
+3.11
DAPO-Qwen-32B
61.16 2.50
56.67-66.67
62.57 1 2.49
57.78-68.85
+1.41
16

---

Table A5: Hyperparameter configuration by task. AIME and Count Down use grid-search ranges; and LiveCodeBench share single universal setting.
Hyperparameter
ATME
Count Down 4
GPQA-D
LiveCodeBench
8
{ 3 2,2,4,8)
{ 3. 2,
Top-p
{0.40, 0.60, 0.80, 0.95) {0.6, 0.8, 1.0)
40.40, 0.80, 0.95)
0.95
0.95
Temperature T
{0.6, 0.8. 0,.00.8.10. 1.0) 8,192
0.6
0.6
Max generation length
32,768
16,384
16,384
Chat template
Default Templates
Default Templates
No Chat Templates'
Default Templates
"Except for Cemma-3-27B. the performance degradation significant without chat template.
E Additional Setups
E.1 Best-of-N Analysis Setup
To measure how quickly limited tuning budget yields performance gains, simulate best-of-N For every model-task pair in AIME and Count Down we start
random search for N 1. : 15. from the complete Cartesian grid of hyperparameters in Table 4. At each Monte-Carlo replicate uniformly draw distinct values for a single target hyperparameter (3, top-p, or temperature) while keeping the other two their default settings, retrieve the corresponding validation accuracies that were pre-computed during the main grid search, and record the improvement of the best sampled configuration over the initial draw. Repeating this procedure 256 times and averaging across the four LLMs produces the curves in Fig.
F,2 Random-Forest Regression Analysis Setup
Every completed grid-search run - defined by a specific model, task, and random sced-serv as one training example for model-agnostic importance analysis. Each example is encoded by the triple (3, top-p. T) and labeled with its accuracy. We fit RandomForesttogressoo from SCIKIT-LEARN 44 with 100 trees that have unrestricted depth. Impurity baaedd Gini importances rank the hyperparameters as (0.41), top-p (0.32), and temperature (0.27), as shown in Fig. 2
E.3 Setup for Case Study
Section 7.2
investigates linear prompt blending on five sentiment benchmarks with three 7B-sized
LLMs: Llama3 8B 32). Mistral 7B [41 and Gemma 1.1 7B (2)
For each dataset, sampled from GPT4o 45 and Claude 46 curate diverse pool of task prompts-96 for binary sentiment, 32 for six-class emotion, and 40 for financial sentiment-verifyiing that each prompt forms a syntactically valid query when concatenated with the input sentence. Letting Lmax denote the length of the longest prompt in a pool, every prompt-embedding matrix E B IR Lxa extended length Lmax via linear interpolation, and then combined by simple averaging across prompts to form the blended prompt.
During inference, we prepend this blended vector each of the 10-shot demonstrations and feed the embeddings directly to the model. For the single-promp baseline we calculate the average accuracy of the prompts the expectation of randomly choosing any single prompt, whereas the blended prompt is evaluated once; consequently, Table Lable[rreflcctt an identical number of forward passes per model. Because all other factors remain fixed, any performance difference isolates the effect of the prompt representation itself.
F Hyperparameters
Table A5 lists the full search space for AIME and COUNT DOWN 4, along with the universal settings used for GPQA-DIAMOND and LIVECODEBENCH, All searches use five random seeds; reported results seed-averaged.
17

---

G Implementation Details
We implement Mol on top of the vLLM framework 47. which supports efficient tensor parallelism. Mixing weights are computed from both the output token and the associated logits after each generation step. The resulting mixed inputs are cached and used input for the subsequent decoding step.
For GPQA evaluations, are evaluated using the default configuration with "thinking mode" enabled. The only exception is Gemma-3-27B, which requires additional prompt elicit multiple-choice outputs in the format of (A, B. D).
the Language Model Evaluation Harness framework [48] All models
LiveCodeBench evaluations follow its official implementation 28 using the default setup and test suite corresponding to the period from May 2023 to May 2024. At the time of writing, generation templates were not officially available for Sanaa333-Nemmttoo-Supeeee DAPO-Qwen-32B. We manually created a template for Llama-3.3 3.3- Suirron--u based on its official documentation, including the required system prompt to activate its thinking mode. For Gemma-3-27B and DAPO-Qwen-3215 we adopt the Generi cBase and Bode@venInstruee templates, respectively.
Gemma-3-27B, and
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We propose Mol, our main claim is using both the sampled output and the distribution can boost language models' generation capabilities. is accurately reflected in the title and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have a limitation section in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
19

---

Answer: [Yes]
Justification:
We provided derivations and model formulations in Section 4
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification:
We include the hyperparameter settings in Appendix F details implementa- tions in Appendix G that are sufficient for reproducing our results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
20

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provide our implementations along with our submission.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ipss.cc/pill wttp:ccppppcccceeeeeeeeee..c.ett httpa///ggiieee/eeeeebiiisssssss (Codeooomiissssseeeeeeee ssiiiiiisiiiyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We discuss the hyperparameters in Section 5.4 and provide detailed table in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We provide the error bar analysis in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
21

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We include runtime analysis in Section 7.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer:
[Yes]
Justification:
We conducted our research conforming the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
We explore a technique that has no direct relationship or impact on society.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: We have no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
We have provided appropriate citations to code, data and models used in our research.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
23

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We have provided a readme in our implementation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cood/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
25


Reviews:
[{"review_id": "VhqNf4Usgv", "reviewer": "Reviewer_3KDf", "summary": "This paper proposes Mixture of Inputs (MoI), a method that blends the discrete one-hot vector and the continuous next-token probability distribution in autoregressive language modeling. The authors uses a dirichlet distribution to interpolate between the mixture between the predicted one-hot distribution and the full probability distribution. MoI allows the model to maintain richer internal representation compared to standard autoregressive modeling. Experiments spans multiple downstream tasks and models show the effectives of the method compared to standard decoding.", "questions": "1. Would the authors be able to show a few generated examples with both standard decoding and the proposes method side by side to see where does the proposed method actually improve?\n\n2. Can the authors explain how to integrate their method to vLLM?\n\nFormatting issue: Figure 3 on page 9 is a bit too large, consider making it smaller.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n- The method is novel and interesting: too much information is discarded by just feeding the predicted token back to the model. \n- Experiments are comprehensive, covering many downstream tasks and model families. The performance improvement is quite large given that this method does not incur too much overhead to standard decoding.\n- The paper is clearly written, with details and hyperparameters that enhances the reproducibility of the work.\n\nWeaknesses:\n\n- The experiments are mostly conducted on verifiable benchmarks. However, a lot of the LM use cases are chatbots. It would make the paper much more solid if the author can conduct experiments on an instruction following benchmark (e.g. WildBench), or show a few (not cherry picked) examples of the generations to check the quality.\n\n- The significance of the work is questionable: my understanding is that this method requires a good estimate of the next token distribution (log probs). But modern inference engines (e.g. vLLM, sglang) uses a lower precision, and that the full probability distritbution from these inference engines might not be as accurate. If this method is incompatible with inference engines, I don't know how much would the community adopt this method.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback. We appreciate your comments and will address concerns regarding inference engine compatibility, instruction-following capacity, qualitative study in this rebuttal, with further elaborations to be included in the final paper. \n\n**Modern inference engine compatibility**  We agree modern inference engines play an important role in LLM serving, and accordingly build our implementation on top of vLLM, despite it requiring more engineering efforts. In fact, all experiments in our submission are conducted on our vLLM-based implementation, and the baseline method in the throughput analyses is the original vLLM. As to the precision issue, the vLLM V1 engine’s sampler casts the softmax logits to fp32 for sampling (as in L47 of the `sampler.py` in the vLLM repo), and we followed the same strategy and didn’t notice any significant issues. \n\n**Additional results on instruction-following benchmarks and more models** We conducted additional experiments on MT-Bench, using the four models from our original paper, and adding two smaller models, Llama-3.1-8B and Qwen-2.5-14B. Below, we report the average performance  between standard decoding (temperature=0.6, top-p=0.95) and our Mixture of Inputs (MoI) decoding (beta={1,2,4}, same decoding config), together with the relative improvements brought by MoI (Δ). As summarized in Table A, MoI consistently outperforms the baseline for all models. \n\n**Table A: MT-Bench Scores**\n| Model            | Baseline Score | MoI Score | Δ % (MoI vs Baseline) |\n|------------------|:--------------:|:---------:|:---------------------:|\n| QwQ‑32B          | 9.25 | 9.51 | **+2.81 %** |\n| Nemotron‑49B     | 9.41 | 9.48 | **+0.74 %** |\n| Gemma‑27B        | 9.05 | 9.38 | **+3.65 %** |\n| DAPO‑Qwen‑32B    | 8.96 | 9.46 | **+5.54 %** |\n| Llama‑3.1‑8B     | 8.24 | 8.65 | **+4.98 %** |\n| Qwen‑2.5‑14B     | 8.87 | 9.32 | **+5.07 %** |\n\n\n**Qualitative Evaluation on Repetitiveness** We conducted additional qualitative analyses on MoI sampling outputs and noticed MoI reduced repetition in the response. Particularly, we measured the 3-gram and 4-gram repetition rate in the response for MT-bench and found MoI reduces the repetition on 72.9% of responses, reducing the average 4-grams repetition rate from 4.27% to 3.13%. We also provide a side-by-side comparison on a coding task as below (QwQ-32B).\n\n*With standard sampling (repetitive around HCA definition):*\n>To address the problem of finding the highest common ancestor (HCA) of two nodes in a binary tree, we need to clarify the definition. Based on common terminology, the lowest common ancestor (LCA) is the deepest node shared by both paths from the nodes to the root. However, since the problem specifies \"not LCA,\" we assume the HCA refers to the shallowest common ancestor (closest to the root) [content continues...]\n\n*With mixture of inputs sampling (reduced repetitiveness):*\n>To find the highest common ancestor (HCA) of two nodes in a binary tree, we can utilize a recursive approach similar to finding the Lowest Common Ancestor (LCA). The HCA is the deepest node that is an ancestor of both given nodes. Here's how to implement this: [content continues...]\n\nWe hope our responses have adequately addressed your concerns and further highlighted the innovations and potential impact of our study. If you have any further questions or need additional information, please do not hesitate to ask."}, {"review_id": "HHaLSMh260", "reviewer": "Reviewer_Py1S", "summary": "This paper proposes a novel generative sampling method. It utilizes Bayesian estimation to combine the predicted probability distribution from the previous step with the actually sampled token, creating a new input representation. This approach requires no additional training and has minimal computational overhead. Experimental results show that MOI significantly improves the performance of various LLMs on multiple tasks, including mathematical reasoning, code generation, and PhD-level question answering.", "questions": "1. Why were there no experiments conducted for the case where \\beta = 0 ?\n\n2. In terms of model selection, why was the Qwen series of models not included in the evaluation, while QwQ were?\n\n3. In Equation 5, what would happen if we were to remove H and only use a hyperparameter to control the proportion between p_t and y_t?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1.  The method is simple and plug-and-play, and it demonstrates strong performance across a wide range of tasks.\n\n2.  The experimental analysis is thorough, and the method is well-motivated with a clear theoretical foundation.\n\n**Weaknesses:**\n\n1.  Although the authors have already mentioned this in the limitations section, I still have questions regarding the method's generality. The paper lacks a systematic experimental comparison on open-domain dialogue. In fact, numerous evaluation benchmarks are available, such as MT-Bench, AlpacaEval, and the LMSYS Chatbot Arena, which can be used to assess human preferences. For the completeness of the paper, conducting such experiments seems essential.\n\n2.  The sensitivity to the hyperparameter \\beta is indeed a concern. It would be beneficial if the authors could provide suggested values for \\beta for a broader range of general-purpose tasks.\n\n3.  The paper lacks qualitative analysis or case studies. Observing the generation trajectories of specific examples and providing a comparative analysis would be an interesting experiment. This would offer valuable insights into how the method works in practice.", "was_revised": true, "final_justification": "The additional experiments have addressed all my concerns.", "rebuttal": "We thank you for the constructive feedback.  We appreciate your comments and will address concerns regarding instruction-following capacity, hyper-parameter selection, and qualitative study in this rebuttal, with further elaborations to be included in the final paper.\n\n**Additional results on instruction-following benchmarks and more models** We conducted additional experiments on MT-Bench, using the four models from our original paper, and adding two smaller models, Llama-3.1-8B and Qwen-2.5-14B. Below, we report the average performance for standard decoding (temperature=0.6, top-p=0.95) and our Mixture of Inputs (MoI) decoding (beta={1,2,4}, same decoding config). together with the relative improvements brought by MoI (Δ). As summarized in Table A, MoI consistently outperforms the baseline for all models. \n\n**Table A: MT-Bench Scores**\n| Model            | Baseline Score | MoI Score | Δ % (MoI vs Baseline) |\n|------------------|:--------------:|:---------:|:---------------------:|\n| DAPO‑Qwen‑32B    | 8.96 | 9.46 | **+5.54 %** |\n| QwQ‑32B          | 9.25 | 9.51 | **+2.81 %** |\n| Nemotron‑49B     | 9.41 | 9.48 | **+0.74 %** |\n| Gemma‑27B        | 9.05 | 9.38 | **+3.65 %** |\n| Llama‑3.1‑8B     | 8.24 | 8.65 | **+4.98 %** |\n| Qwen‑2.5‑14B     | 8.87 | 9.32 | **+5.07 %** |\n\n\n**Hyperparameter Robustness** In our experiments, we observed that it generally improves the performance across tasks by setting beta to less aggressive values (0.5-2), while tuning the hyperparameter has the potential to further push the performance for each task. To demonstrate the robustness of the hyperparameter, we conduct a hyperparameter transfer experiment as below. \n\nIn particular, we first choose the hyperparameter (beta/temperature/top-p) based on the performance on AIME & CountDown-4, and test it on GPQA-Diamond and LiveCodeBench. As summarized in Table B, MoI consistently outperforms the baseline for all models. \n\n**Table B: Hyperparameter Robustness (hyperparameters used in this experiment is picked based on their AIME & CountDown-4 performance)**\n| Model                  | Method   | Input Info    |    GPQA-D |       LCB |   Avg |\n| ---------------------- | -------- | ------------- | --------: | --------: | ----: |\n| **QwQ-32B**            | Standard | Output Token  |     58.08 |     76.48 | 67.28 |\n|                        | **MoI**  | Token + Dist. | **61.41** | **77.85** | 69.63 |\n|                        | Gain     | —             |     +3.33 |     +1.37 | +2.35 |\n| **Nemotron-Super-49B** | Standard | Output Token  |     63.64 |     40.35 | 52.00 |\n|                        | **MoI**  | Token + Dist. | **64.95** | **40.90** | 52.92 |\n|                        | Gain     | —             |     +1.31 |     +0.55 | +0.92 |\n| **Gemma-3-27B**        | Standard | Output Token  |     45.45 |     32.37 | 38.91 |\n|                        | **MoI**  | Token + Dist. | **46.26** | **32.92** | 39.59 |\n|                        | Gain     | —             |     +0.81 |     +0.55 | +0.68 |\n| **DAPO-Qwen-32B**      | Standard | Output Token  |     42.93 |     56.91 | 49.92 |\n|                        | **MoI**  | Token + Dist. | **43.94** | **57.50** | 50.72 |\n|                        | Gain     | —             |     +1.01 |     +0.59 | +0.80 |\n\n\n\n**Qualitative Analyses on Repetitiveness**\nWe conducted additional qualitative analyses on MoI sampling outputs and noticed MoI reduced repetition in the response. Particularly, we measured the 3-gram and 4-gram repetition rate in the response for MT-bench and found MoI reduces the repetition on 72.9% of responses, reducing the average 4-gram repetition rate from 4.27% to 3.13%. We also provide a side-by-side comparison on a coding task as below (QwQ-32B).\n\n*With standard sampling (repetitive around HCA definition):*\n>To address the problem of finding the highest common ancestor (HCA) of two nodes in a binary tree, we need to clarify the definition. Based on common terminology, the lowest common ancestor (LCA) is the deepest node shared by both paths from the nodes to the root. However, since the problem specifies \"not LCA,\" we assume the HCA refers to the shallowest common ancestor (closest to the root) [continues...]\n\n*With mixture of inputs sampling (reduced repetitiveness):*\n>To find the highest common ancestor (HCA) of two nodes in a binary tree, we can utilize a recursive approach similar to finding the Lowest Common Ancestor (LCA). The HCA is the deepest node that is an ancestor of both given nodes. Here's how to implement this: [continues...]\n\n**Response to Question 1: what about Beta=0?** We think beta=0 would be similar to the case of the direct mixture baseline. Beta=0 would fully emphasize the predicted distribution and ignore the sampled token. We found this often leads to less grounded and unstable generations, so we focused on more practical beta values that blend both signals.\n\n**Response to Question 2:  why not Qwen?** QwQ-32B is the leading open-source mode at its 32B scale from the Qwen family at the time of our submission. In Table A, we included a Qwen-2.5-14B model in our MT-bench experiments and found MoI consistently brings performance to this model.\n\n**Response to Question 3: what about removing H?** In our early phase experiments, we had explored replacing H with a hyperparameter, which causes QwQ-32B to drop 7.07% on GPQA-Diamond. After some analyses, we found it is necessary to balance the output distribution and one-hot output token dynamically, for which we found entropy to be a natural choice. \n\nWe hope our responses have adequately addressed your concerns and further highlighted the innovations and potential impact of our study. If you have any further questions or need additional information, please do not hesitate to ask.\n\n[1] Zhu, Rui-Jie, et al. \"A survey on latent reasoning.\" arXiv preprint arXiv:2507.06203 (2025)."}, {"review_id": "cxBcyWZDjX", "reviewer": "Reviewer_gMqW", "summary": "The paper proposes Mixture of Inputs, where instead of passing in a one-hot encoding the sampled token, they instead pass in a linear mixture of the sampled one-hot and the predicted next token distribution. The method is motivated by the intuition that models might perform better if they can explore multiple reasoning paths instead of being forced into a single path. The authors evaluate MoI on a wide range of models and tasks, finding that it consistently improves performance.", "questions": "(1) For the direct mixture baseline, to make things fair with the standard sampling baseline, I would be curious about how much things improve if one also does grid search on nucleus top-p and temperature.\n\n(2) Why do you do grid search on aime and countdown, but not on gpqa and livecodebench?\n\n(3) How important is it to have entropy-dependent mixture weights? How well would the method perform if one simply did a linear mixture with some fixed weight? And what if we normalize it to [0,2] instead of [0,1]? Why was this particular normalization chosen?\n\n(4) The Dirichlet model makes sense in that the posterior mean just ends up being a linear combination, but I don't really understand the motivation or intuition behind this probabilistic model. For example, why do we not see w, when it is observed in practice? What are we uncertain over?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n(1) The method is interesting, simple, and requires no additional training. It also has strong results across a wide range of models and datasets.\n\n(2) The paper has thorough analysis and ablations, with interesting intuition about the mixture weight beta across different tasks.\n\nWeaknesses: no major weaknesses; some minor ones are discussed in the questions section.", "was_revised": true, "final_justification": "The new experiments look good and most of my questions were answered. I still find the Dirichlet model a bit unnatural but the authors' explanation was helpful. I will keep my score the same (5 - Accept).", "rebuttal": "Thank you for your constructive feedback. We appreciate your comments and will address concerns regarding evaluation setup, the role of entropy in MoI, and the intuition of the Dirichlet model. \n\n**Response to Question 1 on baseline setup** Both the vanilla sampling and the direct mixture baseline were also tuned over grid search for top-p and temperature. We will add an explanation in the paper to clarify this.\n\n**Response to Question 2 on evaluation setup** We deliberately choose not to do grid search on GPQA and LiveCodeBench to demonstrate that MoI does not rely heavily on hyperparameter tuning. \n\n**Response to Question 3.a on the importance of entropy** In our early phase experiments, we had explored replacing entropy with a hyper-parameter, which causes QwQ-32B to drop 7.07% on GPQA-Diamond. After some analyses, we found it is necessary to balance the output distribution and one-hot output token, for which we found entropy to be a natural choice. \n\n**Response to Question 3.b on the design of normalization** We chose the [0,1] range to ensure all token weights remain non-negative and interpretable as probabilities. Extending the range or allowing negative weights is possible, though we suspect this may complicate optimization and introduce instability. Exploring more flexible formulations would be an interesting direction for future work.\n\n**Response to Question 4.a on the intuition behind the Dirichlet model** At the input, the model has both the predicted distribution and the sampled discrete token. To capture the semantic meaning of both, we need to “adapt” the predicted distribution so that the resulting distribution incorporates the information of the sampled discrete token. To this end, we find it is a natural idea to use the Dirichlet model. As widely used in topic modeling (LDA), the Dirichlet distribution is the natural prior over multinomial probabilities and gives closed-form posteriors (due to conjugacy).\n\n**Response to Question 4.b on the predicted distribution p** The predicted distribution p represents the semantics of model outputs *before* we commit to a sampling trajectory. Using p directly as w would ignore the valuable signal from the sampling decision. In our experiments, we refer to this approach as *Direct Mixture* and find MoI to outperform this baseline consistently (as in Table 1 of the submission).\n\n **Response to Question 4.c on the uncertainty** Although we observed both (1) the model's predicted distribution, and (2) the sampled token, it is still an open question to find a distribution that incorporates the semantics of both the distribution and the token. Here, we assume such an underlying distribution exists, and propose to use the Dirichlet method to estimate it. \n\n**Additional results** We would like to highlight the additional experiments presented in our rebuttal: \n\n\n1. Table A (in our reply to Reviewer Py1S) summarizes instruction following evaluation results on MT-Bench across 6 models, including 2 additional models that are smaller in size (i.e., Qwen-2.5-14B and LLaMA-3.1-8B), where MoI brings consistent performance gain. \n\n2. Table B (in our reply to Reviewer Py1S) explores the robustness of Hyperparameter tuning. It tunes hyperparameters on AIME/CountDown, and then directly transfers the resulting hyperparameters to GPQA and LiveCodeBench. MoI also leads to consistent performance gain under this setting. \n\n3. In the *Qualitative Analyses on Repetitiveness* part of our reply to Reviewer Py1S, we observe that MoI helps to reduce the repetitiveness in the model generations.\n\nWe hope our responses have adequately addressed your concerns and further highlighted the innovations and potential impact of our study. If you have any further questions or need additional information, please do not hesitate to ask."}, {"review_id": "VfhXFv5FSO", "reviewer": "Reviewer_Ev8i", "summary": "This paper introduces a method that uses continuous tokens as input for large language models (LLMs), named Mixture of Inputs (MOI), to improve model performance. Specifically, it passes both the predicted distribution and the selected token information from the current step to the next step's input, employing a Dirichlet mixture model for mixing. Experimental results show that this method provides minor improvements in reasoning benchmarks across multiple models.", "questions": "See weaknesses", "limitations": "The paper lacks deeper analysis on why using continuous embeddings improves performance as described in Weaknesses.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n- The paper demonstrates that correctly mixing token embeddings can enhance LLM performance, whereas using direct linear interpolation of token embeddings leads to degradation.\n\n### Weaknesses\n\n- It is reasonable to build a one-hot token embedding and linearly interpolate it as model input. However, using a complex model like the Dirichlet mixture model lacks theoretical justification. Could a simpler interpolation achieve similar effects? For example, an interpolation formula such as $\\alpha e_{\\text{chosen}} + (1-\\alpha) \\sum w_i e_{i}$, where $\\alpha$ is a hyperparameter, $e_{\\text{chosen}}$ is the sampled token, and $w_i$ is the predicted distribution.\n\n- The average improvement across four datasets is only around 2%, which raises two concerns:\n  - Randomness in Small Sample Sizes: For datasets with fewer samples, such as AIME (with less than a hundred samples), single-generation randomness can introduce significant errors. The paper does not specify how many times AIME scores were averaged.\n  - Lack of Significance Analysis: While the appendix (Appendix A) seems to provide distributions of multiple experimental results, it does not clearly state the number of experiments conducted. For all improvements listed in Table 1, variance and significance test results (e.g., t-tests) should be provided.\n\n- The paper lacks deeper analysis on why using continuous embeddings improves performance. Models are trained using discrete one-hot tokens converted to embeddings as inputs, but this paper proposes a training-free approach by directly switching to continuous embeddings. Beyond accuracy, additional evidence supporting this improvement would be beneficial. Case studies would also provide valuable insights.\n\n- Table 3 only reports changes in throughput. Does using this method affect the length of generated text?", "was_revised": true, "final_justification": "Although I am not fully persuaded regarding why this method brings improvements (and I still believe this should be a point that the paper needs to explore), the authors have provided fairly solid empirical results demonstrating the effectiveness of the approach. I will raise my score to 4.", "rebuttal": "Thank you for your constructive feedback. We appreciate your comments and will address concerns regarding the Dirichlet model, intuition behind MoI, evaluation setup, and qualitative study in this rebuttal, with further elaborations to be included in the final paper.\n\n**Linear Interpolation and Dirichlet Posterior Mean** It is worth mentioning that the posterior mean of Dirichlet is a linear interpolation of the prior mean and the observation, and the mixing scheme proposed by the reviewer ($t \\cdot e_{\\mbox{chosen}} + (1 - t) \\sum p_i e_i$, we replace $\\alpha$ with $t$ since $\\alpha$ has special mearnings in Dirichlet) can be viewed as a reparameterization of the posterior mean. Particularly, we assume $e_i$ to be the one-hot vector of $i$ for simplicity, then $t \\cdot e_{\\mbox{chosen}} + (1 - t) \\sum p_i e_i$ is the posterior mean of the Dirichlet model $\\alpha \\to w \\to e_{\\mbox{chosen}}$, where $w \\sim \\mbox{Dir}(\\frac{1-t}{t} p)$ and $e_{\\mbox{chosen}} \\sim \\mbox{Multinomial}(w)$. \n\nCompared to the plain linear interpolation, the posterior mean yields the same results, but with a clearer interpretation. Specifically, the linear interpolation is viewed as the posterior mean, and the interpolation ratio is decided by the concentration factor. We believe this interpretation should be clean and understandable for the NLP community, where the Dirichlet prior is widely used, and we will add writing to make the connection to linear interpolation as simple and widely accessible as possible.\n\n\n\n**Length of generated text in throughput analysis** To better compare the throughput, we picked the benchmarks where the generation length is about the same. The median difference between MoI-generated text and baseline-generated text is 1.7%.\n\n**Significance of the improvements** We report the average over 5 independent runs for each decoding configuration (as stated in Section 5.4, line 213). In addition, we visualized performance distributions in Appendix A to give insight into variability across seeds. \n\n**Rational on why mixing continuous embeddings helped** Intuitively, continuous embedding is capable of capturing more information in the continuous state than the discrete token; there have been some recent theoretical analyses such as [1]. They effectively showed that continuous states can enable models to explore multiple generation paths at the same time, where discrete tokens need to collapse on one path at a time, hence can be more effective in finding a correct solution. \n\nWe believe this continuous blending helps preserve distributional information that would otherwise be lost during hard sampling, enabling the model to produce more coherent and contextually consistent generations. Our empirical findings support this interpretation, particularly on tasks that benefit from structured reasoning (e.g., math and code).\n\n**Qualitative Analyses on Repetitiveness**\nWe conducted additional qualitative analyses on MoI sampling outputs and noticed MoI reduced repetition in the response. Particularly, we measured the 3-gram and 4-gram repetition rate in the response for MT-bench and found MoI reduces the repetition on 72.9% of responses, reducing the average 4-grams repetition rate from 4.27% to 3.13%. We also provide a side-by-side comparison on a coding task as below (QwQ-32B).\n\n*With standard sampling (repetitive around HCA definition):*\n>To address the problem of finding the highest common ancestor (HCA) of two nodes in a binary tree, we need to clarify the definition. Based on common terminology, the lowest common ancestor (LCA) is the deepest node shared by both paths from the nodes to the root. However, since the problem specifies \"not LCA,\" we assume the HCA refers to the shallowest common ancestor (closest to the root) [content continues...]\n\n*With mixture of inputs sampling (reduced repetitiveness):*\n>To find the highest common ancestor (HCA) of two nodes in a binary tree, we can utilize a recursive approach similar to finding the Lowest Common Ancestor (LCA). The HCA is the deepest node that is an ancestor of both given nodes. Here's how to implement this: [content continues...]\n\n**Additional results** We would like to highlight the additional experiments presented in our rebuttal: \n\n\n\n1. Table A (in our reply to Reviewer Py1S) summarizes instruction following evaluation results on MT-Bench across 6 models, including 2 additional models that are smaller in sizes (i.e., Qwen-2.5-14B and LLaMA-3.1-8B), where MoI brings consistent performance gain. \n\n2. Table B (in our reply to Reviewer Py1S) explores the robustness of Hyper-parameter tuning. It tunes hyperparameters on AIME/CountDown, and then directly transfers the resulting hyperparameters to GPQA and LiveCodeBench. MoI also leads to consistent performance gain under this setting. \n\nWe hope our responses have adequately addressed your concerns and further highlighted the innovations and potential impact of our study. If you have any further questions or need additional information, please do not hesitate to ask.\n\n[1] Zhu, Hanlin, et al. \"Reasoning by Superposition: A Theoretical Perspective on Chain of Continuous Thought.\" arXiv preprint arXiv:2505.12514 (2025)."}]

Author rebuttal:

Meta-review: The paper presents a new method that combines the token distribution and the sampled token for better autoregressive generation. The reviewers in general value the proposed idea and find the empirical evaluation to be fairly solid. They raised concerns such as its compatibility with modern inference engines, the performance on the instruction following benchmarks, and significance and insights on the improvement etc. 

The author provided additional results and clarification on these concerns, and the rebuttal convinced the negative reviewer **Ev8i** towards positive. The AC agrees with the reviewers and values the proposed model and its strong empirical validation, thus recommending acceptance of the paper. The final version should, as suggested by the reviewers, include the results in the rebuttal and provide well-articulated arguments on the motivation of using the Dirichlet model as well as insights on the model improvement.
Decision: Accept (poster)