Title: Mixture of Inputs: Text Generation Beyond Discrete Token Sampling
Venue: NeurIPS 2025
Accepted: yes
Keywords: language model

continuous representation mixture
Abstract: In standard autoregressive generation, an LLM predicts the next-token distribution, samples a discrete token, and then discards the distribution, passing only the sampled token as new input. To preserve this distribution’s rich information, we propose Mixture of Inputs (MoI), a training-free method for autoregressive generation. After generating a token following the standard paradigm, we construct a new input that blends the generated discrete token with the previously discarded token distribution. Specifically, we employ a Bayesian estimation method that treats the token distribution as the prior, the sampled token as the observation, and replaces the conventional one-hot vector with the continuous posterior expectation as the new model input. MoI allows the model to maintain a richer internal representation throughout the generation process, resulting in improved text quality and reasoning capabilities. On mathematical reasoning, code generation, and PhD-level QA tasks, MoI consistently improves performance across multiple models including QwQ-32B, Nemotron-Super-49B, Gemma-3-27B, and DAPO-Qwen-32B, with no additional training and negligible computational overhead.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method that uses continuous tokens as input for large language models (LLMs), named Mixture of Inputs (MOI), to improve model performance. Specifically, it passes both the predicted distribution and the selected token information from the current step to the next step's input, employing a Dirichlet mixture model for mixing. Experimental results show that this method provides minor improvements in reasoning benchmarks across multiple models.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- The paper demonstrates that correctly mixing token embeddings can enhance LLM performance, whereas using direct linear interpolation of token embeddings leads to degradation.\n\n### Weaknesses\n\n- It is reasonable to build a one-hot token embedding and linearly interpolate it as model input. However, using a complex model like the Dirichlet mixture model lacks theoretical justification. Could a simpler interpolation achieve similar effects? For example, an interpolation formula such as $\\alpha e_{\\text{chosen}} + (1-\\alpha) \\sum w_i e_{i}$, where $\\alpha$ is a hyperparameter, $e_{\\text{chosen}}$ is the sampled token, and $w_i$ is the predicted distribution.\n\n- The average improvement across four datasets is only around 2%, which raises two concerns:\n  - Randomness in Small Sample Sizes: For datasets with fewer samples, such as AIME (with less than a hundred samples), single-generation randomness can introduce significant errors. The paper does not specify how many times AIME scores were averaged.\n  - Lack of Significance Analysis: While the appendix (Appendix A) seems to provide distributions of multiple experimental results, it does not clearly state the number of experiments conducted. For all improvements listed in Table 1, variance and significance test results (e.g., t-tests) should be provided.\n\n- The paper lacks deeper analysis on why using continuous embeddings improves performance. Models are trained using discrete one-hot tokens converted to embeddings as inputs, but this paper proposes a training-free approach by directly switching to continuous embeddings. Beyond accuracy, additional evidence supporting this improvement would be beneficial. Case studies would also provide valuable insights.\n\n- Table 3 only reports changes in throughput. Does using this method affect the length of generated text?\n\nQUESTIONS:\nSee weaknesses\n\nLIMITATIONS:\nThe paper lacks deeper analysis on why using continuous embeddings improves performance as described in Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

comments: SUMMARY:
The paper proposes Mixture of Inputs, where instead of passing in a one-hot encoding the sampled token, they instead pass in a linear mixture of the sampled one-hot and the predicted next token distribution. The method is motivated by the intuition that models might perform better if they can explore multiple reasoning paths instead of being forced into a single path. The authors evaluate MoI on a wide range of models and tasks, finding that it consistently improves performance.

STRENGTHS AND WEAKNESSES:
Strengths:

(1) The method is interesting, simple, and requires no additional training. It also has strong results across a wide range of models and datasets.

(2) The paper has thorough analysis and ablations, with interesting intuition about the mixture weight beta across different tasks.

Weaknesses: no major weaknesses; some minor ones are discussed in the questions section.

QUESTIONS:
(1) For the direct mixture baseline, to make things fair with the standard sampling baseline, I would be curious about how much things improve if one also does grid search on nucleus top-p and temperature.

(2) Why do you do grid search on aime and countdown, but not on gpqa and livecodebench?

(3) How important is it to have entropy-dependent mixture weights? How well would the method perform if one simply did a linear mixture with some fixed weight? And what if we normalize it to [0,2] instead of [0,1]? Why was this particular normalization chosen?

(4) The Dirichlet model makes sense in that the posterior mean just ends up being a linear combination, but I don't really understand the motivation or intuition behind this probabilistic model. For example, why do we not see w, when it is observed in practice? What are we uncertain over?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel generative sampling method. It utilizes Bayesian estimation to combine the predicted probability distribution from the previous step with the actually sampled token, creating a new input representation. This approach requires no additional training and has minimal computational overhead. Experimental results show that MOI significantly improves the performance of various LLMs on multiple tasks, including mathematical reasoning, code generation, and PhD-level question answering.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1.  The method is simple and plug-and-play, and it demonstrates strong performance across a wide range of tasks.\n\n2.  The experimental analysis is thorough, and the method is well-motivated with a clear theoretical foundation.\n\n**Weaknesses:**\n\n1.  Although the authors have already mentioned this in the limitations section, I still have questions regarding the method's generality. The paper lacks a systematic experimental comparison on open-domain dialogue. In fact, numerous evaluation benchmarks are available, such as MT-Bench, AlpacaEval, and the LMSYS Chatbot Arena, which can be used to assess human preferences. For the completeness of the paper, conducting such experiments seems essential.\n\n2.  The sensitivity to the hyperparameter \\beta is indeed a concern. It would be beneficial if the authors could provide suggested values for \\beta for a broader range of general-purpose tasks.\n\n3.  The paper lacks qualitative analysis or case studies. Observing the generation trajectories of specific examples and providing a comparative analysis would be an interesting experiment. This would offer valuable insights into how the method works in practice.\n\nQUESTIONS:\n1. Why were there no experiments conducted for the case where \\beta = 0 ?\n\n2. In terms of model selection, why was the Qwen series of models not included in the evaluation, while QwQ were?\n\n3. In Equation 5, what would happen if we were to remove H and only use a hyperparameter to control the proportion between p_t and y_t?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Mixture of Inputs (MoI), a method that blends the discrete one-hot vector and the continuous next-token probability distribution in autoregressive language modeling. The authors uses a dirichlet distribution to interpolate between the mixture between the predicted one-hot distribution and the full probability distribution. MoI allows the model to maintain richer internal representation compared to standard autoregressive modeling. Experiments spans multiple downstream tasks and models show the effectives of the method compared to standard decoding.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The method is novel and interesting: too much information is discarded by just feeding the predicted token back to the model. \n- Experiments are comprehensive, covering many downstream tasks and model families. The performance improvement is quite large given that this method does not incur too much overhead to standard decoding.\n- The paper is clearly written, with details and hyperparameters that enhances the reproducibility of the work.\n\nWeaknesses:\n\n- The experiments are mostly conducted on verifiable benchmarks. However, a lot of the LM use cases are chatbots. It would make the paper much more solid if the author can conduct experiments on an instruction following benchmark (e.g. WildBench), or show a few (not cherry picked) examples of the generations to check the quality.\n\n- The significance of the work is questionable: my understanding is that this method requires a good estimate of the next token distribution (log probs). But modern inference engines (e.g. vLLM, sglang) uses a lower precision, and that the full probability distritbution from these inference engines might not be as accurate. If this method is incompatible with inference engines, I don't know how much would the community adopt this method.\n\nQUESTIONS:\n1. Would the authors be able to show a few generated examples with both standard decoding and the proposes method side by side to see where does the proposed method actually improve?\n\n2. Can the authors explain how to integrate their method to vLLM?\n\nFormatting issue: Figure 3 on page 9 is a bit too large, consider making it smaller.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-04"}