Title: Hogwild! Inference: Parallel LLM Generation via Concurrent Attention
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM

reasoning

parallel generation
Abstract: Large Language Models (LLMs) have demonstrated the ability to tackle increasingly complex tasks through advanced reasoning, long-form content generation, and tool use. Solving these tasks often involves long inference-time computations. In human problem solving, a common strategy to expedite work is collaboration: by dividing the problem into sub-tasks, exploring different strategies concurrently, etc. Recent research has shown that LLMs can also operate in parallel by implementing explicit cooperation frameworks, such as voting mechanisms or the explicit creation of independent sub-tasks that can be executed in parallel. However, each of these frameworks may not be suitable for all types of tasks, which can hinder their applicability. In this work, we propose a different design approach: we run LLM "workers" in parallel , allowing them to synchronize via a concurrently-updated attention cache and prompt these workers to decide how best to collaborate. Our approach allows the instances to come up with their own collaboration strategy for the problem at hand, all the while "seeing" each other's partial progress in the concurrent cache. We implement this approach via Hogwild! Inference: a parallel LLM inference engine where multiple instances of the same LLM run in parallel with the same attention cache, with "instant" access to each other's generated tokens. Hogwild! inference takes advantage of Rotary Position Embeddings (RoPE) to avoid recomputation while improving parallel hardware utilization. We find that modern reasoning-capable LLMs can perform inference with shared Key-Value cache out of the box, without additional fine-tuning.


Peer reviews:
comments: SUMMARY:
The paper studies a method that has multiple LLM instances to collaboratively generate answers. Instead of pre-defining a framework to decompose a task into sub-tasks and having multiple LLM instances to solve sub-tasks in parallel, the proposed method has multiple LLM instances to figure out the collaboration strategies themselves. The instances collaborate through KV cache. By having different LLM instances to view the same KV cache and update KV cache, they can see each other's progress, potentially avoid redundant works and generate better results.

STRENGTHS AND WEAKNESSES:
Strengths:
* Even though the basic idea seems simple, the paper couples multiple interesting ideas to make it work. The ideas are interesting and quite novel.
* The method has been evaluated on multiple reasoning benchmarks (ranging from easy benchmarks and challenging benchmarks) and has been shown good performance improvement on these benchmarks.

Weaknesses:
* The presentation can be improved. For example, fonts and lines in the figures are too small. What is the forward pass?
* The evaluation may not be completely fair. If a forward pass means a token generated by an LLM instance,  Hogwild! Inference with 2 workers should generate twice as many tokens as baseline methods.

QUESTIONS:
What is the cost of this method (how many more tokens are generated?)
In figure 4, 5, 6, when 2 workers are used, does the method use twice as more tokens as the baselines?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces “Hogwild! Inference”, a novel parallel inference protocol enabling multiple LLM instances to collaborate dynamically using a shared, concurrently-updated Key-Value (KV) attention cache. This design leverages Rotary Position Embeddings (RoPE) to enable efficient real-time collaboration without predefined strategies. Through extensive empirical evaluation on various reasoning benchmarks (e.g., LIMO, OlympiadBench, LiveCodeBench, AIME’25), the authors demonstrate significant performance improvements, highlighting faster inference and enhanced reasoning accuracy when compared to existing parallel inference strategies.\n\nSTRENGTHS AND WEAKNESSES:\n**Disclaimer:** I am not an expert in this field, but I tried my best to evaluate the work.\n\n**Strengths**\n \n**Novel and Effective Parallelization Strategy** \nThe approach of dynamically sharing and updating KV caches among parallel inference threads is innovative and practical. It leverages intrinsic LLM reasoning capabilities without additional fine-tuning, representing an impactful advancement in LLM inference techniques.\n\n**Robust Empirical Results:** \n\nEmpirical validation is thorough, demonstrating substantial accuracy and inference-time improvements across diverse and challenging benchmarks. Results convincingly illustrate the efficacy and practical potential of the proposed method.\n\n**Clear, Detailed, and High-Quality Presentation:** \n\nI find the paper to be well-written in general.\n\n**Practical Utility and Efficiency Gains:** \n\nThe paper includes explicit and detailed evaluations of practical inference-time performance gains, highlighting near-linear scalability in parallel token generation, which strongly supports real-world applicability.\n\n**---XXX---**\n\n**Weaknesses** \n\n**Limited Theoretical Justification for Collaboration Dynamics:** \n\nAlthough empirically successful, the paper lacks theoretical grounding or deeper insights into why allowing LLM instances to synchronize dynamically via shared caches universally improves reasoning.\n\n**Potential Scalability and Generalization Limitations:**\n\nThe method's effectiveness seems dependent on task-specific factors such as task complexity and the intrinsic capability of the LLMs to \"self-coordinate.\" It remains unclear how well the approach generalizes to significantly larger models or vastly different reasoning domains.\n\n**Dependence on LLM’s Intrinsic Collaborative Ability:** \n\nThe paper implicitly assumes that LLMs possess sufficient \"intrinsic reasoning and collaborative instincts\" without clearly specifying or demonstrating the limitations or conditions under which this assumption might fail.\n\nQUESTIONS:\nPlease see the weakness section\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper introduces Hogwild! Inference, a method that allows an LLM to explore multiple execution traces for the same problem while enabling synchronization between them. The traces share the attention cache, and the properties of RoPE are exploited to further enhance efficiency.

STRENGTHS AND WEAKNESSES:
Strengths:

1. The idea is very interesting, and the algorithm is supported by both solid math foundations and system design.

2. The authors conduct experiments using large-scale (32B) models such as QwQ, covering a diverse set of tasks including GSM8K, LIMO, LiveCodeBench, and OlympiadBench.

3. They provide illustrative examples, such as the expected collaboration patterns, to help validate their hypothesis.

Weaknesses:

1. The approach is currently for evaluation only. It would be interesting to explore fine-tuning the model toward the desired behavior to evaluate whether further improvements can be achieved.

QUESTIONS:
How does this method compare with long-CoT-based reasoning? When would be a better scenario to use Hogwild! Inference?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe main contributions are the three-tier cache structure, isolation of page to cache block mapping, and the approach to rotate token queries at an angle rather than all previous keys. The authors demonstrate some evidence that LLMs have the capacity to reason without post-training by collaborating without pre-specified roles or explicit cooperation strategies. The paper is very readable and a joy to read.\n\nSTRENGTHS AND WEAKNESSES:\nThe prompting of LLM workers to decide next actions seems analogous to producing states and offloading actions to the LLM in an RL setup. The approach the authors take seems more difficult to generalize. The four cases of “solving parallel sub-tasks, cross-verifying each other, discussing strategy, or pivoting to a new plan” may not have much coverage. The authors also acknowledge the need to tune prompts in 4.2. An analysis of routing strategy between the alternatives that are based on voting, roles, and speed; and showing evidence of these approaches not having better compute and accuracy profiles would make the case stronger. The statement “However, when asked directly, they can spot redundancy and change their approach. Overall, we found that when prompted this way, LLMs often (but not always) detect redundancies in their actions and can determine the optimal course of action.” requires data to justify the claim. In Figure 5. The baselines (dotted lines) for the 32B models have more slope than Hogwild. Given the observation that the authors acknowledge the approach can replace correct answers with mistakes at higher token counts, the results would be more convincing if the forward passes were extended. As a positional paper, the work is novel to the best of my knowledge and significant.\n\nQUESTIONS:\nMy interpretation of figure 1. Is not what is described as keeping KV memories. Rather than stitch together in different orders, why not use a head node to orchestrate with fewer ops? Other routing-like approaches have shown good results by treating a router as a head node orchestrator.\nWhere in Appendix B do you discuss ways to scale further? Are you referring to the section on distributed inference? \nA more comprehensive discussion of the scalability challenges described in the limitations section would be helpful.\nDo you have logs that show the distribution of parallel threads in SoT?\n\nLIMITATIONS:\nIt’s not clear that the approach addresses the limitation of alternative methods that is acknowledged: “... at least some of the agents have to solve the entire problem sequentially, and process (re-encode) each other’s progress. This creates additional computational overhead, which presents challenges for both runtime and memory efficiency Wang et al.”.\nThe stitching together still grows computation. An analysis should be presented if the authors believe it represents a net improvement. \nThe evaluation on 2-4 parallel workers does not seem sufficient to generalize results and in particular is below a threshold where communication problems of distributed systems will emerge. The foundations of the LLM-as-a-Judge paradigm are not very convincing, especially for the older GPT-4o model. A more convincing evaluation would be human-evals on this particular task. Authors acknowledge the lack of direct correlation between base accuracy and collaborativeness. Could you show that collaborativeness is not correlated to output length?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}

comments: SUMMARY:
When solving complex questions, LLMs take a long time. In this study, LLMs were run in parallel to synchronize via a concurrently-updated attention cache, implemented using prompting strategy for these workers to decide how best to collaborate. The evaluation results showed improved performance.

STRENGTHS AND WEAKNESSES:
Strengths:
Increased token-generation speed
The comparison results showed improved model performance

Weaknesses:
It would be helpful to formulate the problems and solutions formally
It is unclear if it is standard using LLM in parallel or with theory innovation
It would be helpful to compare the proposed model with paralle models (with multiple LLMs/agents)

QUESTIONS:
1) To formulate the problems and solutions formally
2) It would be helpful to compare the proposed model with parallel models (with multiple LLMs/agents) 
3) It would be important to evaluate more and larger LLMs (the used models are small with limited performance), especially larger models, considering the Chain-of-Thought designs.
4) It would be important to evaluate the models using more datasets (with complex problems). The model evaluation (better use more and larger models and more datasets) is limited to show the stability and reliability of the strategy.
5) Though the token generation speed is faster, how about the average token generation speed, or the total time needed (combination of a set of workers) for solving the challenging problems.

LIMITATIONS:
The theoretical innovation is limited.
It would be helpful to compare the proposed model with parallel models (with multiple LLMs/agents)
The model evaluation (better use more and larger models and more datasets) is limited to show the stability and reliability of the strategy.
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03