Hogwild! Inference: Parallel LLM Generation via Concurrent Attention
Authors: Gleb Rodionov
Roman Garipov
Alina Shutova
George Yakushev
Erik Schultheis
Vage Egiazarian
Anton Sinitsin
Denis Kuznedelev
Dan Alistarh
Venue: NeurIPS 2025 spotlight ()
Keywords: LLM
reasoning
parallel generation
Primary area: deep_learning
TL;DR: We propose a parallel generation method for LLMs, where multiple instances synchronize through a shared, dynamically-updated attention cache

Abstract: Large Language Models (LLMs) have demonstrated the ability to tackle increasingly complex tasks through advanced reasoning, long-form content generation, and tool use. Solving these tasks often involves long inference-time computations. In human problem solving, a common strategy to expedite work is collaboration: by dividing the problem into sub-tasks, exploring different strategies concurrently, etc. Recent research has shown that LLMs can also operate in parallel by implementing explicit cooperation frameworks, such as voting mechanisms or the explicit creation of independent sub-tasks that can be executed in parallel. However, each of these frameworks may not be suitable for all types of tasks, which can hinder their applicability. In this work, we propose a different design approach: we run LLM "workers" in parallel , allowing them to synchronize via a concurrently-updated attention cache and prompt these workers to decide how best to collaborate. Our approach allows the instances to come up with their own collaboration strategy for the problem at hand, all the while "seeing" each other's partial progress in the concurrent cache. We implement this approach via Hogwild! Inference: a parallel LLM inference engine where multiple instances of the same LLM run in parallel with the same attention cache, with "instant" access to each other's generated tokens. Hogwild! inference takes advantage of Rotary Position Embeddings (RoPE) to avoid recomputation while improving parallel hardware utilization. We find that modern reasoning-capable LLMs can perform inference with shared Key-Value cache out of the box, without additional fine-tuning.

Hogwild! Inference: Parallel LLM Generation via Concurrent Attention
Gleb Rodionovi Yandex
Roman Garipov" HSE University Yandex
Alina Shutova" HSE University Yandex
George Yakushev" HSE University Yandex
Erik Schultheis IST Austria
Vage Egiazarian IST Austria
Anton Sinitsin
Denis Kuznedelev
Dan Alistarh'
Yandex
Yandex
IST Austria
Abstract
Large Language Models (LLMs) have demonstrated the ability tackle increas- ingly complex tasks through advanced reasoning, long-form content generation, and tool use. Solving these tasks often involves long inference-time computations. In human problem solving, a common strategy to expedite work collaboration: by dividing the problem into sub-tasks, exploring different strategies concurrently, etc. Recent research has shown that LLMs can also operate in parallel by implementing explicit cooperation frameworks, such voting mechanisms the explicit cre- ation of independent sub-tasks that can be executed in parallel. However, each of these frameworks may not be suitable for all types of tasks, which can hinder their applicability. In this work, we propose a different design approach: we run LLM "workers" " in parallel, allowing them to synchronize via a concurremllyupdaatee attention cache and prompl these workers to decide how best to collaborate. Our approach allows the LLM instances to come up with their own collaboration strat- egy for the problem hand, all the while "seeing" each other's memory in the concurrent KV cache. We implement this approach via Hogwild! Inference: a parallel LLM inference engine where multiple instances of the same LLM run in parallel with the same attention cache, with "instant" access to each other's mem- ory[ Hogwild! Inference takes advantage of Rotary Position Embeddings (RoPE) to avoid recomputation while improving parallel hardware utilization. We find that modern reasoning cappblle LLMs can perform inference with shared Key-Value cache out of the box, without additional fine-tuning.
Introduction
Many recent advancements of Large Language Models can be attributed their ability to perform
inference-ting computations to improve performance [Suzgun et al. Snell
Snell et al. 2024, Beeching
Muennighoff' et al. 2025 This includes chain-of thought (Col) reasoning [ei al.
2022
Kojima al. 2022. 2, Zhang et al. 2022 Yao et al. 2023 Lightman et al. 2023 long-form
generation [Bai et al. 2024 and interacting with external tools Schick al. 2023 Qin et al. 2023
Yao et al. 2022 Shen et al. 2023 Popular L.L.M-based services have capabilities for reasoning and
tool use [OpenAl al. [024] Google Deep 2025 AAthrropic 2024]. At the same time, several
ceasoling-capable open-access LLMs have recently been released to the public [DeepSekk-Al et al.)
2025
Qwen Team 2025 Yang et al. 2024 Muennighoff et al.) 2025 et al.) 2025].
Using these models to solve complex problems often requires long sequential computations, that is. generating text token-byy bo-een-b-token... However, many reasoning problems are not sequential. Leveraging this intuition, several recent works propose parallel inference strategies that allow multiple LLMs
Our implementation is available at atttpps://githubb www.//qimp/hoguil..d.... Conresponding author: rodionoovleebeyandex team. ru. Equal contribution.
Senior author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Common Cache
Cache Blocks
Task: compute x x² for in (1. 2, 3).
Common
Alice
Bob
Workers history:
Allce [1]: Hey! Lat's decide how we should collaborate.
Bob [1]: Hi, Alice! Let suggest that I do x=1
x=3, you will x=2. What do you think?
Alice [2]: Okay, let's go with that plan.
Alice view
Bob
[Alice] Current Cache
LLM.forward()
Alice [3]: For x=2:
[Bob] Current Cache
Bob [2]: Perfect. Starting with x=1:
An intuitive explanation of Hogwild! Inference, with 2 workers generating in parallel and shared cache blocks. Each color denotes a cache block. SSe it in action (example generation).
Figure 1:
to solve a problem faster or more accurately via some form of collaboration (Wagg et al., 2022 Ning al. (024). In the simplest case, multiple LLMS attempt the problem independently then vote [Wang et al. 2022] or cross-reference their results (Du et al. 2023 Wang et al.) 2024a improve correctness. A parallel line of work allows the LLM divide the problem into multiple independent sub-tasks that are then solved in parallel and merged, producing the final solution (Ning
al., 2024 Kim et al. 2024 Jin et al., 2025]. These parallel inference strategies can improve quality and effciincy, taking advantage of parallelism in modern hardware.
Unfortunately, single collaboration strategy is universally effective. For instance, solving problem in independent parallel "threads" can be inefficient when one of the threads requires a longer generation than the rest, resulting in most of the agents waiting for straggler and wasting compute [Wang et al. [022] 2024a]. In turn, inference with independent sub-tasks only works if the problem can immediately be split into these sub-tasks. Furthermore, if one of the agents discovers that the original plan is flawed, they will be unable to re-plan [Ning et al. 2024 Ding et al. 2025).
potentially solving sub-tasks that are no longer necessary (Tin al. 2025
This runs contrary to how humans collaborate. Instead of strict adherence to a fixed collaboration strategy, often collaborate more dynamically, re-planning the fly, abandoning some tasks half-way and switching to a more promising approach, discussing or debating strategy if the initial plan failed. While this type of collaboration is harder to define, it offers greater flexibility and be more efficient if the participants are sufficiently cohesive [Hutchins 1995 Entin and Serfaty 1999.
Our Approach. In this work, we try to apply the same principle to artificial reasoners. Since modern LLMs can already reason and plan [Zhou et al. 2024 Gao et al.) 20244, Wang et al., 2024c), we hypothesize that they can benefit from dynamic interaction between different instances, during which they can develop their own collaboration strategy for the problem at hand.
To test this hypothesis, we propose Hogwild! Inference no pre-defined framework for collaboration Instead of choosing how LLMs should interact ahead of time, we allow them to generate tokens in parallel and "see" each other's progress (tokens) immediately as they are generated. We then prompt the L.M "workers" to decide their next course of action by themselves, given the latest actions from others: whether this means solving parallel sub-tasks, cross- vross-verifying each other, discussing strategy, or pivoting to a new plan.
parallel LLM inference protocol with
2
To enable this type of on-the-fly fly collaboration, Hogwild! Inference runs multiple LLM instances with the same weights, but with custom Key- Vaauee cache that shares token representations between workers, allowing concurrent cross-attention. Specifically, instead of re-computing Key- Key-Value representations for each worker, we keep track of individual worker KV memories and "stitch them together" in different orders, by adjusting their positional embeddings (see Figure I]. Moreover, we provide efficient implementation of this inference approach.
We test Hogwild! Inference with modern open-source LLMs and find that existing reasoning-capeee as QwQ [Qwen Team 2025] and DeepSeek-R1 [DeepSeck-A) Al et al., [005]] can already "reason to coordinate". More concretely, we observe that concurrent agents can formulate and follow plans, adapt when the initial plan has failed, point out each other's errors, and use each other's
"Our approach inspired by Hogwild! SGD Recht al. [011] that runs updates asynchronously and applies each update as soon as it computed. The exclamation mark is part the original name (Stanford HAI 2023
2

---

key observations. When prompted to check if they are doing redundant work e.g. when one LLM instance is doing a sub-task that is already done by another, or solving a problem that is no longer relevant- - they can often (but not always) detect redundancy and change strategy. In summary, our results suggest that parallel inference with a shared Key-Valus cache may offer a promising approach to enable effective and efficient collaboration between multiple LLM instances.
2 Background
Recent works propose large number of frameworks for parallel reasoning and tool use that vary across several axes; how the parallel instances are organized together, what they exchange, and how often (Zhang et al. 2025]. In this section, we give brief summary of these methods.
Discussion & aggregation. The simplest way to parallelize chain-of-thought reasoning is Self- Consistency [Wang al., 2022). where multiple LLM instances reason independently, then vote on the final answer. This approach was later extended in Du et al. [2023), replacing majority voting with text-based communication rounds. Subsequent works in this field combine multiple LLM types (Wang jet al. 2024a] and scales to more agents Li et al. [2024a] Another line of work introduces specialized
"roles" such the Debugger (Talebirad and Nadiri 2023], Examiner [Cohen et al.] 2023]. Math
Teacher (Kong et al.] 2024), Judge [Chen et al. 2024). and others, to further augment reasoning.
This type of role-based discussion was shown to greatly improve LLM reasoning factuality for certain tasks [Wang et al.] [202] Du et al.) 2003], and can even enable multiple weaker LLM agents to collectively outperform state-of-the-art single-agent systems [Wang et al., 2024a]. However, this improvement is not unique to multiple agents and can be offset with better singlc-agent prompt- ing (Wang et al.) 2024b Muennighoff et al. 2025 Additionally, these approaches do not necessarily accelcrate reasoning, because at least some of the agents have to solve the entire problem sequentially, and process (re-encode) each other's progress. This creates additional computational overhead, which presents challenges for both runtime and memory efficiency Wang et al. 2024a] Du et al. 2023).
Parallelism for efficiency. A different line of work leverages multiple LLMs to solve tasks faster in parallel, such as Skeleton-of-Thoute Thouhht (SoT) [Ning et al. 2024). SoT begins by running a single LLM to outline plan for solving the problem with independent sub-tasks, then launches parallel LLM instances for each sub-task. For problems that involve function calling, these functions can also run in parallel [Kim et al. 2024. Gim al. 2024). Subsequent works propose more complex parallelism strategies such as dynamic parallel tree search [Ding et al., [025] or a single agent
spawning asynchronous sub-tasks that are done by background LLM "threads" [Jin et al. 2025 Liu
ct al. 2024b Pan et al. 2025], achieved with specialized fine-tuning.
These techniques are known to substantially accelerate inference for problems that fit their type of parallelism. However, we argue that this also their main limitation: by imposing a specific parallelism strategy, these methods can harm reasoning for problems that do not fit their framework. For instance, when solving a complex reasoning problem, it is often the case that the initial plan turns out to be wrong or incomplete [Muennighoff et al. 2025 DeepSeek-Al et al., 2025 which conflicts with SoT-like methods (Ning et al.) 2024 Yu 2025 that follow a fixed plan- plan-cxccute-aggrees
schedule. Furthermore, some of the sub-tasks may turn out to be more complicated than originally intended and take up more work, which would cause methods like PASTA Jin et al. [2025] to wait for that single task, whereas a more sophisticated reasoner could adjust the plan to work better in parallel. Note that each individual issue can be amended with yet another, more complicated parallelism framework, but the sheer number of such cases makes doubt whether this is the right approach. In this work, we instead let multiple LLM instances interact without a fixed framework, allowing them to see each other's partial generations to devise (and revise) task-specific collaboration strategy. We show that, perhaps surprisingly, existing reasoning L.MMs already have the ability to leverage this.
3 Hogwild! Inference
Our main intuition is that modern LLMs do not need a pre-defined framework for inference-time parallelism: they can organize by themselves. To test this hypothesis, we design parallel inference protocol where multiple LLM instances can collaborate as flexibly as possible. Instead of assigning each "worker" to a specific role or sub-task, we run them together and prompt them to collaborate. This approach has two key problems: how to run multiple inference threads from the same Key- Value memory, and how to prompt LLM "workers" to collaborate over said memory. We outline how to perform LLM inference with shared cache in Section 3.1 describe our cache structure in Section 3.2 and prompting strategy in Section 3.3. Finally, Section 3.4 describes the inference algorithm.
3

---

3.1 Concurrent Attention with Shared Key-Value Cache Inference
The core ingredient of Hogwild! Inference is shared Key-Value memory (KV cache) accessible to all workers. The cache consists of several blocks that can be reused between workers, implementing concurrent version of the attention mechanism (Bahdanau et al. 2015 Vaswani 2017].
Let us first consider simple case with two workers and three cache blocks, as depicted in Figure The first block contains the prompt, and the other two blocks contain the tokens generated by workers and B respectively (denoted Alice and Bob in the Figure). As workers generate new tokens, they access each other's attention caches as though these were their own previously generated tokens. In " "Alice" sees the common prompt, then "Bob's" token representations, then her own. In turn, Bob sees the same common prompt, then Alice's 's token KVs, and his own tokens after that f
Figure I
This creates a discrepancy where the same Key- pairs appear at different positions for each worker. Furthermore, the relative distance between the same pair of tokens (e.g., first generated tokens from Alice and Bob, respectively) changes as new tokens are added. While it is possible to re-encode these tokens at their new positions, it would cause overhead that scales cubically]
Instead of re-encoding the new tokens for other workers, we attempt to reuse existing token represen- tations between workers. However, since these tokens appear at different positions for each worker and step, need to adjust for their positional embeddings. Most modern LLMS use Rotary Position Embeddings (RoPE) [Su et al. 2021]. where each key and query is rotated to an angle proportional to its absolute position. Prior works have shown that RoPE embeddings can be manipulated through
scaling [Peng et al. 2023] slicing [Xiao et al. 2024 or pruning [Zhang et al. [023].
In Hogwild! Inference, instead shift the KV values, multiplying the entire cache block by cos / sin values that implement rotation by a constant offset. We use this to arrange the same cache entries in different order for each worker as in Figure (tight). This allows both workers to instantly "see" each other's tokens while they are generated and even before they are processed by all layers.
3.2 Cache Structure
Now that we defined a way to rearrange cache blocks on the fly, it is reasonable to ask how to arrange these blocks. For short tasks, simply concatenating worker outputs is sufficient. However, as we consider harder problems that require long chains of thought, workers will eventually pay less attention to each other because of the thousands of tokens between their latest step4))
To address this problem, we propose a more sophisticated cache arrangement inspired by group chat rooms. Namely, we split the generated text into reasoning "steps", roughly a paragraph in size. Whenever a given worker finishes paragraph, (e.g. generates \n'n), we move its KV cache to the end of a shared chat-like history and let it generate the next paragraph at the end of that history. Note that workers still see each other's current (unfinished) paragraphs the end of the shared history as they write them (see Figure I This way, workers always see each other's s latest updates as recent tokens and can communicate more easily. For each worker W,, we organize cache blocks as follows:
Common Cache: large KV cache block that stores KV representations for the system prompt, task description, and a history of previous reasoning steps from each agent.
Other workers: multiple smaller cache blocks containing the latest (unfinished) steps of all other workers Wizt in ascending order. For instance, if there are 4 workers, W2 will see W1 WIGWWWW, Current worker: the latest (unfinished) reasoning step of the current worker W; to be continued.
Each block starts with new paragraph (Inin) followed by short header text that contains worker id (Alice, Bob, etc.) and the index of the current step for the worker. Whenever a worker completes a reasoning step, their KV cache entries are moved to the end of the shared history cache block with the proper rotation, then their local cache is reset for a new step. We refer to Figure for an illustration of this layout for two workers. We describe alternative (simpler) layouts in Appendix A
3.3 Prompting for Zero-Shot Collaboration
The shared key-value cache inference we described above allows modern LLMs access each other's tokens and reason collaboratively. However, even though modern LLMs can reason about
$ For clarity of exposition, we choose to anthropomorphize the pronouns for these two LLM instances. 'If agents generate one new token each, which is then re-encoded differently for each of these n agents, that each have to attend to (r)) additional tokens, then the total step complexity is O(r²),
II other words, if we put all outputs of worker A ahead of worker B, then the more tokens are generated, the farther worker B needs to "look" to reach worker A's latest outputs. This could be mitigated with finetuning.

---

how to collaborate, there is no guarantee that they will actually do so unprompted. As with any desired LLM behavior, it can be achieved in two ways: either by training the model to generate tokens collaboratively or by prompting it in-context. In this work, we focus on the latter approach to make Hogwild! Inference easier to generalize for new models. Our prompting consists of two parts:
1. System prompt describes the "rules" of the shared cache and suggests that workers collaborate. This prompt goes at the beginning of either the system or user message (if not unsupported);
2. Inserting s1-like collaboration prompts: every thousand generated tokens, we prompt a random worker with "Wait, am I doing redundant work? (yes/no):" at the beginning of their next paragraph. This strategy is meant to promote collaboration and is inspired by Muennighoff et al. (2025].
The latter s1-like prompts present a curious case. We found that LLMs fine-tuned on reasoning can often become too "focused" what it generating currently and fail notice that another instance has found a mistake or solved their problem earlier. However, when asked directly, they can spot redundancy and change their approach. Overall, we found that when prompted this way, L.MSs often (but not always) detect redundancies in their actions and can determine the optimal course of action.
3.4 Inference Matters
When generating new tokens with Hogwild! Inference, we perform a forward pass on all workers in parallel, as though they were the same batch. Instead of each sample having its own attention cache, we allow batch elements to attend to each other's KV caches at different positions. When processing newly generated tokens, we "insert" their KV representations at the end of their respective cache blocks, then arrange these cache blocks for each worker. This way both workers can immediately attend to each other's current tokens even before they are fully processed by all layers.
This leads to the following problem: since workers combine cache blocks in different order (see Figure we would need rotate the cached KVs multiple times, one for each worker. Done naively, this would require rotating all past token representations at every step, which is inefficient for long contexts. Fortunately, this problem can be circumvented using a property of rotation: if both query and key are rotated by the same angle, the dot product between them will not change. Instead of rotating all previous keys, we can rotate current token queries to an equivalent angle (Figure 2
Suppose that a given attention layer needs to compute attention between the current token query at position in (denoted p(g, in)) and block of keys rotated the starting position ix. Instead of rotating keys, we can rotate the query to position i, - and keep the KV cache as is. If there are multiple KV blocks A, B, C (Alice, Bob, Common) that need to be rotated to positions ig &B : of respectively, we rotate the query q multiple times for each block. Formally, we can rewrite the attention dot-product:
p(q. in) p(A, i @ ((B @ p(C, (
i + plg, i, -if) B e pla, i -
where + denotes concatenation. The r.h.s. formula only rotates the current step query, i.e. a single token per worker, as opposed to the past KV blocks that can contain thousands or millions of tokens. We use this property to design an efficient implementation of our method based on Flash- Decoding (Dao et al.) 2023]. J. We gather each KV cache block in contiguous memory buffer and compute attention similarly to Paged Attention [Kwon et al. [023). where one page would correspond
to one cache block and the corresponding query rotations from all workers. This way, we need only
one copy of each cache block and do not need to re-rotate its entries (see Appendix Bb.
Alice View
Implementation
Postionns
Common caste
Isb Gache
Alice Carhh
Quuryy Alio
A-B
Quary Alice
Quary Alic
Seb Cachhe
Bob View
Paxtions 0
CIA
CIAIS
Common caste
Abca Cache
Dar Cashhl
Cuary Bob
Query Alca
Cemmon eachi
Offset:
Figure 2: Intuitive scheme of Hogwild! Inference with query rotation. Colors represent cache blocks. Instead of rotating all cache blocks to align with Alice's 's and Bob's views. we keep them fixed the zero position and only rotate the current token queries to equivalent angles.
5

---

Accurasy
Beseline
Hogellal 2e
Hogwild! Dw
ScFCantraay SCT (o
Hogwild!
Uppa bourdl
Forwand passes
Forward passes
Forward passes
Figure 3: (left) Evaluation results for QwQ-32B on synthetie tasks with GSMSK questions in each. (middle) Evaluation of Hogwild! Inference and baselines with QwQ-32B LIMO. (right) Hogwild! Inference with varying number of workers with QwQ-32B on LIMO.
Experiments
4.1 Detailed Evaluation with QwQ-32B
In this section, we conduct an initial evaluation of Hogwild! Inference to test its ability collaborate in our zero-shot setting. All evaluations in this section are done with the QwQ-32B [Qwen Team 2025) model. We consider two tasks: one with obviously independent tasks that can be done in parallel and another with a more complicated collaboration pattern.
In both setups, allow the model to generate reasoning up certain budget of sequential forward passes and evaluate its accuracy. If the model did not produce the final answer () (.. }) in time, we take all generated outputs and insert special promp that makes the model generate an answer (or its "best guess"), similarly to how it is done in Pu et al. 2025 If there are multiple workers / threads, generate the final answer immediately (516 tokens, stop early if generated answer). We apply this technique to all methods except "Baseline (no early stopping)" and do count these extra tokens towards the total budget (x axis) since they have an equal effect on all methods.
feed outputs from all workers (concatenated) into the model and prompt it to
We evaluate the following generation algorithms (details in Appendix Dj:
Hogwild! Inference: Our main algorithm, described in Section 3 We evaluate with 2, 3 and parallel "workers" and provide additional configuration details in Appendix D.I
Baseline (no early stopping): standard sequential generation with a single LLM instance. This is the only evaluation where do not insert the early stopping prompt described above.
Baseline: improved sequential generation with the early stopping technique described above. Skeleton-of- Thouughh (SoT) [Ning et al., 2024]: parallel reasoning algorithm in which the LLM first generates a short "outline" containin several independent tasks, then runs these tasks in parallel and combines the results. We with both unlimited number of parallel threads (original setup) and with 2 "workers" that append tokens to each thread in a round-robin fashion. For more complicated reasoning tasks, we found that Skeleton-of- ceetooo--f-Touughhhh Thought cannot solve the problem by itself; to mitigate this, we allow the main model to encode all generated threads and continue reasoning (with early stopping). We discuss
Siiectonof-Thoughhe in more detail Appendix D.2 Seff-consistency [Wang et al., 2022]: parallel reasoning algorithm where LLM instances write solutions independently, then vote on the answer. Instead of majority voting, we allow the LLM view both solutions (concatenated) before generating the final answer with early-stopping prompt, which outperforms voting in our setup and works even for 2 workers. Note that this method cannot split sub-tasks between workers and is instead meant increase quality through voting.
Sanity Checks with xSMMkx5: Before we try our approach on more challenging tasks, we test Hogwild! Inference is capable of basic collaboration. For this purpose, we construct toy problem oeerllppelppingggggg questions from the GSM8k test set (Cobbe) al. 2.21]. The L.M is prompted to solve each problem and return comma-separated values We report the average per-question accuracy, i.e. if the model solves out of 5 questions given sample correctly, it will get score of 0.8 for that sample.
set with 128 samples, each containing
We summarize our results in Figure (eett:: the parallel workers under the Hogwild! Inference can indeed collaborate, i.e. our KV cache manipulations do not break down model's reasoning capabilities. As intuition suggests, Skeleton-of-Thougg question in parallel. We provide example of the outline created by the Skeleton-of-Thhee Thoughh in Appendix E.4 Notably, the self-consiseency algorithm also shows some improvement over the
can also speed up this synthetic task by answering each
"VnnnWait, given the limited time, I have to give answer right now. Conside- ring all my previous attempta, have conclude that the final anawer is \boxed{[" "Solve these problems and return comma-separated anavers \boxed(ansvert,.... answer5) An 1. {taski)nn 2. [task2) 3. [task3) 4. (task4) 5. (task5)"

---

Accuracy
Accuracy
QwQ-32B
Qwen3-8B
Phi-4-R+
23-B-A228
148
Basclines
32.35
8B
1.78
1000 2400
4000
7006
198
104
4000
500
0800
700
Forward passes
Forward passes
Figure 4: Evaluation of Hogwild! Inference on LIMO for QwQ-32B, -eao---Raaoning-PPe (14B) and Qwen3-8B (left) and different Qwen3 models (right). Dashed lines denote baselines (1 agent).
baseline, which we attribute to the fact that it gives the model two "shots" at problem, and if one of them happens to be faster, the algorithm will on average surpass the baseline.
LIMO tasks. Next, we evaluate Hogwild! Inference in more challenging setup where there is no clear pattern of collaboration. We adopt the dataset of 817 problems from Ye et al. [2025]. The dataset contains mathematical problems that take modern LLMSs thousands of tokens solve reliably. Unlike our synthetic tasks, the problems that dataset often do not have an obvious way to agree on a collaboration strategy ahead of time, but it can emerge (and change) during reasoning.
We summarize our results in Figure (middle, right). Overall, Hogwild! Inference can converge to correct solution faster, achieving greater accuracy for the same number of consecutive steps. Furthermore, it produces greater speed-ups as we increase the number of parallel workers (though there is a limit, decoding provides some improvement over the single-worker baseline, but does not outperform Hogwild! Inference. As expected, Skeleton-of-Thouggh could not split the problem neatly into independent tasks, but still achieves some improvement on small budgets.
show in Appendix [E.I. Similarly our previous setup, self- celf-consistency
We then evaluate different LLM families and sizes on LIMO dataset in Figure We found that our approach generalizes to most of the models tested, with a notable exception. For Qwen3 model family, we observe that the smaller models, 1.7B and, to a lesser extent, 4B fail to adapt the task and get distracted from the task. In Appendix E. we also report additional evaluations in this setup: ablation of the cache rotation from 3.1 and our chat-like cache structure from Section 3.2 examples of collaborative generations for this setup in Appendix
We provide
4.2 Additional Benchmarks and Models
Next, we test whether our approach can be generalized to other mathematical reasoning and program- ming tasks. For this evaluation, we also chose benchmarks that do not have obvious collaboration patterns but can nonetheless be solved faster by two human "agents". We evaluate on three such benchmarks: LiveCode Bench, OlynpiadBench and AIME'25. In addition to QwQ-32B, we also report Qwen3 YYng et al. 2025) and Phi-4 Reasoning Plus [Abdin et al. 2025). For AIME 25, we focus on larger models and additionally include DeepSeek-RI [DeepSeck-Al et al. 2025].
LiveCodeBench [Jain al. 2024 We evaluate on the code_generation. lite version release_ .v. Our evaluation closely follows the setup from Qwen Team [2025]: we take the same 279 problems dated between 2024.08 and 2025.0 02 and filtered so as to avoid ones present in the QwQ dataset. Note, however, that some of the other LLMs in our setup do not report which samples, if any, did they train on. However, since we use the same model weights for the baseline and Hogwild! Inference, we can still compare the two strategies. We run the standard test suite and report Pass @ I averaged over random seeds. For early stopping, we allow the method (and bascline) to generate single final code block with up to 1024 tokens, using similar carly-stopping prompt in Section (see Appendix C.. For Hogwild! Inference, we use the same system prompts as before.
OlympiadBench [He et al. 2024 Next, we evaluate on different reasoning benchmark that contains Olympiad-level problems on Math and Physics. We run evaluations on the two text-only english-langusee parts: (236 problems). Unlike in Section 3 the answers to these problems are not individual numbers but LaTeX formulae that allow multiple equivalent formulations of the correct answer. We use the official evaluation codebase and adapt the built-in DeepSeek-R prompts for use with our model set (see details in Appendix D). For early stopping, use the same prompt as before with 64 token limit.
2770_mats__eee CDMP (675 problems) and OE_ www.Tooppscsse_en... CSMM_CCMPPPPPPPPP
7

---

Accuracy
Accuracy
1.28
QwQ-32B
QwQ-32B
Qwen3-141
Qwen3-14h
Qwen3-NK
Qwen3 -88
Baselines
Basclines
1000
240
2008
4000
5000
7008
800
1008
2008
3003
.001
5000
60:0
100
Forward passes
Forward passes
Figure 5: Evaluation of Hogwild! Inference with 2 workers on OlympiadBench Math (left) & Physies (right) for QwQ-32B, Qwen3- 14B and Qwen3-8B models, dashed lines are the baselines.
Large Models AIME [2025]. Finally, evaluate how Hogwild! Inference scales larger models on a popular AIME benchmark, using both I and II subsets. For this task, we focus on two models: Qwen3-235t 2115--32BBB22B Yang al. [2025] and DeepSeek-RI (DeepSeck-Al al. [025]. Since the AIME benchmark only contains 30 problems (15 per subset), we evaluate each model with 10 random seeds and average results. We otherwise use the same evaluation protocol for LIMO, with the same early stopping and at most 16 tokens per answer during early stopping.
We arrange our results in Figure OlympiadBench and Figure LiveCodeBench and AIME'22 25. Overall, Hogwild! Inference shows similar improvements what observed earlier (Section 4.1). One atypical case is OlympiadBench Physics (Fig. [ right) where Qwen3-14B stops improving after roughly 4096 tokens. Upon closer inspection, overthinks the problem, improving some answers while replacing other correct answers with mistakes. Overall, the results show that the cache rotation tricks and the output structure from 3.2 can indeed be generalized across different models and benchmarks. Note, however, that due to the different output format needed apply slight alterations individual model prompts: notably, QwQ-32B automatically inserts <think> at the end of the prompt, while Qwen3 and Phi-4 do not, so we insert it manually before the common history header. We describe this in detail in Appendix C
found that the model does not break down, but
4.3 Mcasuring Ability to Collaborate
Now that we know that modern LLMs can collaborate in our zero-shot setting, it is natural to ask how well can they collaborate and what affects their ability. While this question deserves more thorough investigation, we can still quantify how well LLMs collaborate under Hogwild! Inference. In this sec- tion, we analyze their "collaborativenes' using the LLM-as-a-Judge paradigm (Zheng al. 2023a]:
we feed collaborative traces into a GPT-4o [Hurst et al. 2024] model and prompt it to score behavior from to 6, where "1" means no collaboration, "3" indicates basic task splitting and "6" represents hypothetical optimal collaboration, never achieved in our analysis. We analyze LLM generations on LIMO dataset with on three models from Section 4.2 To control for differences generation lengths we compare only 4096-token prefixes from each worker. We compare three inference setups: independent generations as per self-consistency decoding; ii) restricted Hogwild! Inference where agents can only view each other's finished paragraphs, but not the current (incomplete) reasoning step, and iii) full Hogwild! Inference, with 2 agents in each setup.
We summarize our scores in Figure 7 as expected, models that can see each other can collaborate and independent workers cannot. Interestingly, Hogwild! Inference with instant (token-wise)) syn- chronization scores significantly higher than a version that can only see completed inference steps. In Appendix G we provide more detailed results, judge prompt, configurations and examples.
Accuracy
Accuracy
QwQ-32B
Qwen3-8B
Qwend 2--B-B2--BB22
DeepSeck-RI
Phi-4-R+
Bisclinus
Buselines
Upper bound
Forward passes
Forward passes
Figure 6: Evaluation of Hogwild! Inference (2 workers) on LiveCodeBench v5 004.088-202.00 for QwQ, Phi-4-R+ and Qwen3 (left) and AIME'25 for larger models (right), dashed lines are baselines.

---

4.0
Table 1: Inference benchmarks for Section 4.4
Qwen3-8B
Columnsdenoteseqqueeee sequence length Rows with one workerarebaselines, 2& workers use Hogwild!
3.5
Phi-4-R+
Msan collaborative score
QwQ-32B
3.0
# Workers
1024
2048
4096
8192
16384
Tokens per second
2.5
20.1
20.0
19.7
19.3
18.3
2.0
36.3
36.2
36.1
36.1
34.3
68.9
69.0
69.1
66.3
60.3
1.5-
Latency per forward (ms)
49.7
50.0
50.9
51.7
54.6
1.0
No syne
Step-wise
Token-wise
2
55.1
55.3
55.4
55.3
58.3
Figure 7: Mean collaborativeness score from GPT-4o. No sync is independent generation, Step-wise restricted Hogwild! where worker can only see each-other's 's past steps, Token-wise full Hogwild! with instant cache exchange.
58.1
58.0
57.9
60.4
66.4
Time to generate # tokens
52.3
103.3
206.5
416.7
853.5
29.9
58.
114.6
228.0
454.4
16.7
31.6
61.3
120.7
239.2
4.4 Inference
To recall, our main motivation for proposing Hogwild! Inference is to enable faster reasoning through collaboration. Since the actual inference speed depends on many factors (GPU(s), software, precision, etc), we previously focused on evaluating inference speed in terms of the number of consecutive forward passes and not inference time. Here, in turn, we report the actual inference speed in terms of latency and tokens per second. We evaluate three setups: baseline sequential inference and Hogwild! Inference for two and four workers. We run baseline with FlashAttention v2 (FlashDecoding
and our algorithm with custom GPU kernels using the approach described Section 3.4 We use NVIDIA L40S GPU and AMD EPYC 9534 and benchmark the official quantized version of QWQQ-3B AWQ for all setups.
Our results in Table show that, for the 32B model, Hogwild! Inference can generate tokens nearly twice fast for workers and about 3 faster for 4 workers, which means that the accuracy gains from earlier sections can translate to faster solutions. We also report the average over GPUs, well the 10% and 90% percentiles, in Figure 8 (left). Overall, Hogwild! Inference has small constant latency offset compared to the baseline and near-linear scaling as we increase the number of workers. While our implementation already shows significant performance gains, we discuss several ways to scale it further in Appendix B including in distributed setting.
forward pass / ms
Accuracy
36
55-
baseline W-2
02-
W=4
Hogwidd 2w
2000
4000
5000
5000
10000
12000
14000
16000
100
200
300
46
KV cache length
Time(s)
Figure (left) Duration of single forward pass (generating W new tokens) for Qwen --00--32BAW-- on L40S, given the total number of tokens already in the KV cache. The dotted lines indicate the 10% and 90% quantiles over multiple repetitions on different GPUs. (right) Accuracy versus average generation time on the LIMO dataset task using BwQQ-2B-AWW under different token budgets.
As the figure shows, there is some overhead associated with preparing multiple caches (i.e., even at an empty cache, Hogwild! slightly slower than pure FlashAttention). more detailed breakdown is presented in Table 2 which shows the duration of the attention kernel (or attention+rope for Hogwild!), as well as the total setup time, that is, the time spent preparing the data structures needed for Hogwild! The latter needs to be done only once per forward pass, instead of once per transformer

---

Table 2: Breakdown of Hogwild! overhead compared to pure FlashAttention inference.
KV Length
Attention (x64)
Setup (x1)
FA
W2
W4
FA
W2
W4
300
1 11ss
45μs
45us
1.9ms
3.9ms
4096
35us
65μs
82us
1.9ms
3.9ms
8192
55us
92μus
123μs
1.9ms
3.9ms
16384
100μss
140ps
203ps
1.9ms
3.9ms
block. For long contexts, the attention call is about 40% and 100% slower for generating with 2 and 4 workers, respectively.
Additionally, we report accuracy results over time using kernel on the official quantized version of Q[QQ-32B-WWQ on LIMO dataset. The experiments were conducted on NVIDIA L40S GPUs. For comparison, we run the baseline (Flash Attentionn v2) and Hogwild with 2 workers, maintaining the same experimental setup as detailed Section 4.1, We report our results in Figure g (right). As illustrated, method achieves better accuracy results the LIMO dataset within the same time budget.
Discussion
In this work, investigated the ability of large language models to perform parallel generation where multiple instances synchronize through a shared, dynamically- updated attention cache. Surprisingly, results show that LLMs can operate effectively in parallel across dynamically updated attention cache without specialized fine-tuning. We demonstrate that parallel inference threads can explicitly coordinate, leveraging each other's partial solutions to enable collaborative problem-solving.
The proposed method, called Hogwild! Inference, allows multiple inference threads to concurrently access and update a shared attention cache. By leveraging Rotary Position Embeddings (RoPE), our approach introduces minimal computational overhead while ensuring instant synchronization-noen generated KV cache entries becoming immediately visible to all threads. This "telepathic" communi- cation opens up new possibilities for efficient parallel generation with LLMs.
Our method exhibits reduced robustness when applied to smaller models or longer contexts, suggesting scalability challenges across model sizes and sequence lengths. Additionally, our automatic evaluation metric relies on a proprietary model, which may limit reproducibility.
Limitations
Future work In future work, we plan to investigate methods for improving collaboration between threads, such fine-tuning and reinforcement learning. We also plan investigate connections to alternative parallel inference schemes, such as speculative decoding [Leviathan et al. 2023], and parallel token generation methods like Medusa [Cai et al., 2024 or EAGLE [Li et al. 2024b). Finally, it is interesting to consider alternative shared memory structures: allowing workers to insert new steps any order, selectively delete (forget) steps, or solving programming and tool use tasks with a shared IDE and file-system. The KV cache rearrangement used in Hogwild! Inference could also allow humans to interact with agents asynchronously. giving clarifications and feedback during reasoning.
We thank Vladimir Malinovskii for his help with brainstorming, helpful feedback and suggesting future work directions. We also thank Philip Zmushko for proofreading.
Acknowledgements:
References
Marah Abdin, Sahaj Agarwal, Ahmed Awadallah, Vidhisha Balachandran, Harkirat Behl, Lingjiao Chen, Gustavo de Rosa, Suriya Gunasekar, Mojan Javaheripi. Neel Joshi, Piero Kauffmann, Yash Lara, Caio César Teodoro Mendes, Arindam Mitra, Besmira Nushi, Dimitris Papailiopoulos. Olli Saarikivi, Shital Shah, Vaishnavi Shrivastava, Vibhav Vineet, Yue Wu, Safoora Yousefi, and Guoqing Zheng. Phi-4-reasoning technical report, 2025. URL htt.:htttss//aaiiii httt::::/arriv.org/ab////// 21318
AIME, Aime problems and solutions. http:://artofrroolemoolvvv....oo
php/
AIME
Problems, AIME_Problest
aedaaanddllll SoddSooutions 2025.
10

---

Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang. Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase. and Yuxiong He. Deepspeed inference: Enabling efficient inference of transformer models at unprecedented scale, 2022. URL.
https://arxiv. org altp////s././/../////// 2207 00032
Anthropic. Claude 3.7 sonnet and claude code, 2024. URLhttpss:// http::/wwwwaanatri....cco/////am claude-3-7-sonee Accessed: 2025.04.02.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. In Proceedings of the 3rd International Conference on Learning Representations (ICLR), 2015. URL https://arxiv org/ abs/ Talps//149..444
Yushi Bai, Jiajie Zhang, Xin Lv. Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longwriter: Unleashing 10,000+ word generation from long context 1lms. ArXiv,
abs/2408.0705$ 2024. URL Inttps://aapp BRRLntppe//p11..maaaicccchh www.ssmeaaiicccoola.oom..
Edward Beeching, Lewis Tunstall, and Sasha Rush. Scaling test-time compute with open models.
URL
blogpost--cal time- compute
Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer, 2020. URL
org/ absa/bo4......//////////// 05150
Tianle Cai, Xinyun Li, Zhiruo Wang, Yuhuai Wang, and Dawn Song. Medusa: Simple Ilm inference acceleration framework with multiple decoding heads. arXiv preprint XXXi:24401.10744 2024.
Justin Chen, Swarnadeep Saha, and Mohit Bansal. ReConcile: Round-table conference improves reasoning via consensus among diverse LLMs. In Lun-Wet Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 7066-7085, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18663//1/20024.clllooo http://..........a.aeeaaaaaaaaaa acl- acr-2oo4..cllIong. .81////../
URL htt.tttss//aclaathhhte
Mouxiang Chen, Binyuan Hui, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Jianling Sun, Junyang Lin, and Zhongxin Liu. Parallel scaling law for language models, 2025, URL (tRLhttps:// //ar::////xxvv org/orgg//.. /bs/
2505. 10475
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
Roi Cohen, May Hamri, Mor Geva, and Amir Globerson. LM vs LM: Detecting factual errors via cross examination. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12621-12640, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.18633/11/022. emnlp- main.778 778. URL htt.:tpp://aallatiooooyooooooo org/2023 emnlp- manlp-main... main. m78n.778/
Tri Dao, Daniel Haziza, Francisco Massa, and Grigory Sizov. Flash-decoding for long-context infer- ence. https::/ httph//haaaaccttoo/////////////a html 2023. Accessed: 2025-05-10.
DeepSeck-AI, Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, and Xiao Bi et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025. URL httpp://arxiv.. http://aaxxiv.oorrab/////e/ org/
12948
Yifu Ding, Wentao Jiang, Shunyu Liu, Yongcheng Jing, Jinyang Guo, Yingjie Wang, Jing Zhang, Zengmao Wang, Ziwei Liu, Bo Du Xianglong Liu, and Dacheng Tao. Dynamic parallel tree search for efficient IIm reasoning, 2025. URL WtRLhttps:// https: wttp///////aaeee 16235
Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factual- ity and reasoning in language models through multiagent dehate. Forty-first International Confer- ence on Machine Learning, 2023.
11

---

Elliot E. Entin and Daniel Serfaty. Adaptive team coordination. Human Factors, 41(2):112-325, 1999.
Peizhong Gao. Ao Xie, Shaoguang Mao, Wenshan Wu, Yan Xia, Haipeng Mi, and Furu Wei. Meta reasoning for large language models. arXiv preprint arXiv:2406. 16998, 2024.
In Gim, Seung seob Lee, and Lin Zhong. Asynchronous Ulm function calling, 2024. URL https //arxiv.rgg/aaa//422...
Google Deeppinnddd Gemini 2.5: Our Newest Gemini Model with Thinking,
Bechnoolooy/gggggle--eeemmmmmm ///g//emnniiiin geemnii-modeeeeeeeeeeeeeeeeeeeee -odde-odddiiieee wwwttttiiinnnddateeeeeeee updauupdddaaaaaaeee march-2025/ #gemini (remimi-----thiiniinne 2025. Accessed: 2025-04-07.
Chaoqun He, Renjie Luo. Yuzhuo Bai, Shengding Hu. Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang. Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiad- bench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems, 2024.
Chan-Jan Hsu, Davide Buffelli, Jamie McGowan, Feng-Ting Liao, Yi-Chang Chen, Sattar Vakili, and Da shan Shiu. Group think: Multiple concurrent reasoning agents collaborating at token level granularity, 2025. URL httpthtt////xiiii http://aawii..orras///////////// hbtpg//bb/2500.... 11100
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, al. Gpt-4o system card. arXiv preprini arXiv:2410.21276, 2024.
Edwin Hutchins. Cognition in the Wild. MIT Press, 1995.
Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models. arXiv preprint arXiv:2309. 2023.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Stoica. Livecodehench: Holistic and contamination free evaluation of large language models for code, 2024. URL htttps://ariiv org/ abs////b 07974
Tian Jin, Ellie Y. Cheng, 7ack Ankner, Nikunj Saunshi, Blake M. Elias, Amir Yazdanbakhsh, Jonathan Regan-Kelley, Suvinay Subramanian, and Michael Carbin. Learning to keep a promise: Scaling language model decoding parallelism with learned asynchronous decoding, 2025. URL
https://arxiv.. wrg. httg//b//2800 2202/ 21512 11517
Sehoon Kim, Suhong Moon, Ryan Tabrizi, Nicholas Lee, Michael W Mahoney, Kurt Keutzer, and Amir Gholami. An Ilm compiler for parallel function calling. Forty-first International Conference Machine Learning, 2024.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. ArXiv, abs/2205.11916, 2022. URL https://api semanticscholar orgg/oorpsID::490177777
Aobo Kong, Shiwan Zhao, Hao Chen, Qicheng Li, Yong Qin, Ruiqi Sun, Xin Zhou, Enzhi Wang, and Xiaohang Dong. Better zero-shot reasoning with role-play prompting. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies (Volume 1: Long Pupers), pages 4099-4113, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 12.18653///20222aaacclllo 228. URL org 2024 maacl-long. 228/
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611-626 2023.
12

---

Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274-19286. PMLR, 2023.
Junyou Li, Qin Zhang. Yangbin Yu, Qiang Fu, and Deheng Ye. More agents is all you need. Transactions Machine Learning Research, 2024a.
Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, and Soumith Chintala. Pytorch distributed: Experiences on accelerating data parallel training. 2020.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty. In Proceedings of the 41st International Conference Machine Learning, pages 31 147-31166. PMLR, 2024b.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. ArXiv, (bs//335..20050, 2023. /ttp Dttpaa/aiiiiiiia...oorrrr/raaaDa http///rpuuI:::5888//080
Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of- experts language model. arXiv preprint arXiv:2405.0444 044444 2024a.
Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near- infinite context, 2023. BRLphtts:///axt
httggabs/233111 2310.0088999901
Mingdao Liu, Aohan Zeng, Bowen Wang, Peng Zhang, Jie Tang, and Yuxiao Dong. Apar: Llms can do auto-parallel auto-regressive decoding. arXiv preprint arXiv:2401.06766 2024b.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, arXiv preprint arXiv:2501. 2025.
Xuefei Ning, Zinan Lin, Zixuan Zhou, Zifu Wang, Huazhong Yang, and Yu Wang. Skeleton-of- thought: Prompting LLMs for efficient parallel generation. In The Twelfth International Conference on Learning Representations, 2024. URL htttps:/ wttp:ppwwww....e.eeeeeeooereeeaa wwt//eeettdimmmnnnooeoe hota///ooooommddmd//BBbln
OpenAl, :, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, and Alex Beutel et al. Openai ol system card, 2024. URL
https://arxiv. org/ abs 2412..222...0 16720
Jiay Pan, Xiuyu Li, Long Lian, Charlie Snell, Yifei Zhou, Adam Yala, Trevor Darrell, Kurt Keutzer, and Alane Suhr. Learning adaptive parallel reasoning with language models. arXiv preprint arXiv:2504.15466, 2025.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS). Neural Information Processing Systems Foundation, 2019.
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yam: Efficient context window extension of large language models, 2023. URL htt.hhtts::/// http//awww..ogabbaaa////a 00071
Xiao Pu, Michael Saxon, Wenyue Hua, and William Yang Wang. Thoughtterminator:: Benchmarkingg calibrating, and mitigating overthinking in reasoning models, 2025. UtL/https//aarix org/ abs/2504. 13367
Yujia Qin, Shi Liang. Yining Ye, Kunlun Zhu, Lan Yan. Ya- Ta-Ting Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Marc H. Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to master 16000+ real-world apis. ArXiv, abs/2307.16789, 2023. URL https::::: //api. semanticscholar.org horg
CorpusID:2603325
13

---

Qwen Team. Qwq-32b: Embracing the power of reinforcement learning. March 2025. URL https://qwennn: www.nggitt......... wwt..b.../bboo.....eeeeeoooooeee ht/blogg/ggggpp////o//opgobgogga 32b/
Jack Rae and Ali Razavi. Do transformers need deep long-range memory? Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, July 2020. Association for Computational Linguistics. URL htLhttps::/// aclweb. org/ anthology 2220 2020. acl-main
672
Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu. Hogwild!: A lock-free approach parallelizing stochastic gradient descent. In J. Shawe- R. Zemel, P. Bartlett, F. Pereira, and K. Weinberger, editors, Advances Neural Information Processing Systems, volume 24. Cur- ran Associates, Inc.. 2011. URL /ttpbttts://pprcceeiiin heuriiinnnnne..ppsoo..r..... https/cccpppppepeeeeeeeeeeeeeeee ottp://hha.aaaaeeeaaaaaeeeeeeee.
.df
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolfommer: Language models can teach themselves to tools. ArXiv, abs/2302.04761, 2023. URL hattps://app semanticscholast org/ Carp/orrpssI. 2566978342
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yue Ting Zhuang. Hugging- gpt: Solving ai tasks with chatgpt and its friends in hugging face. ArXiv, abs/2303. 2023.
URL https://api semanticscholsseee.... www.oolrrrrrrr///oeeeeo :
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro. Megatron-Im: Training multi-billion parameter language models using model parallelism.
arXiv preprint arXiv:1909.080 08053, 2019.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint vrXiv:2408.03314, 2024.
Stanford HAI
How a "crazy
idea"
overturned the conventional rules edu/news/
of machine learning, 2023.
URL https://bai.. http///aaa.tttfrrddooaoa wtw.soo...euu/eeees.......
how- wwwa-crayyidees leaaay-idda-overuurneee overturned www.eeedoooveeeiiinnniia ruules www.essmachine-leeeeee www.ane--eaaaniaae [lnsert Date].
Accessed:
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint Soki::2044009860 2021.
Mirac Suzgun, Nathan Scales, Nathanael Scharli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging big- bench tasks and whether chain-of-thought can solve them. In Anmual Meeting of the Association for Computational Linguistics, 2022. URL https:://ppp semanticscholar org/ CorpusID 252917648
Yashar Talebirad and Amirhossein Nadiri. Multi-agent collaboration: Harnessing the power of intelligent LLM agents. CoRR, abs/2306.03314, 2023.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Junlin Wang, WANG Jue, Ben Athiwaratkun, Ce Zhang, and James Zou. Mixture-of-agents enhances large language model capabilities. The Thirleenth International Conference on Learning Representations, 2024a.
Qineng Wang. Zihao Wang, Ying Su, Hanghang Tong, and Yangqiu Song. Rethinking the bounds of LLM reasoning: Are multi-agent discussions the key? In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pages 6106-6131, Bangkok, Thailand, August 2024b. Association for Computational Linguistics. doi: 10.18653 acl 331/
URL ottp
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed H. Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. ArXiv, ab1/2203.11171, 2022. URL. https://api. semanticscholar org/ org./orrpusDD2245555222
14

---

Yiming Wang, Zhuosheng Zhang, Pei Zhang, Baosong Yang, and Rui Wang. Meta-neasoning: Cemantics-symbol deconstruction for large language models. In LunWeii Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguisties: ACI. 2024, pages 622-643, Bangkok, Thailand, August 2024c. Association for Computational Linguistics. doi: URLhttps:
org/2024 www.iiinnnnaaaa acl
34/
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thoughh prompting clicits reasoning in large language models. Advances in neural information processing systems, 35:24824 2282477444837, 2022.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In International Conference Learning Representations (ICLR), 2024.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang. Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang. Ru Peng. Rui Men, Ruize Gao. Runji Lin, Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang. Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint arXivv2207. 10671, 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang. Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang. Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, 7ekun Wang. Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL
org. /505 09388
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting language models. ArXiv, abs/2210.03629, 2022. URL https://api semanticscholsse
ID:
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. ArXiv, abs/2305. 2023. URL httppt::/api..smmaattttt sewanticscooolllatt 2287
2587662255
Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning, 2025. URL htlll/ttpp///arrxxoo//bbb///////
03387
Yijiong Yu. Accelerate parallelizaable reasoninn via parallel decoding within sequence, 2025.
URL https://arxiv.orgg wttp:/iiii.rr//b//5/000///// 20533
Qiyuan Zhang. Fuyuan Lyu, Zexu Sun, Lei Wang, Weixu Zhang, Zhihan Guo, Yufei Wang, Irwin King, Xue Liu, and Chen Ma. What, how, where, and how well? a survey on test-time scaling in large language models. arXiv preprint vrXiv:2503.24235, 2025.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36:34661-3471U 2023.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alexander J. Smola. Automatic chain of thought prompting in large language models. ArXiv, abs/2210.03493, 2022. URI. hhttps://app semanticscholar. arg/CorpusID:25222 252766275
15

---

Lianmin Zheng, Wei-Lin Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li. Eric Xing, et al. Judging llm-as- ammaae--judde with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 46:466955466222 2023a.
Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, lon Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. Efficiently programming large language models using sglang, 2023b.
Tong Zheng, Hongming Zhang, Wenhao Yu, Xiaoyang Wang, Rumpeng Dai, Rui Liu, Huiwen Bao, Chengsong Huang, Heng Huang, and Dong Yu. Parallel-rl: Towards parallel thinking via reinforcement learning, 2025. Uttpss//://arai wrg. ab2509 07980
Pei Zhou, Jay Pujara, Xiang Ren, Xinyun Chen, Tze Cheng, Quoc Le, Ed H. Chi, Denny Zhou, Swaroop Mishra, and Huaixiu Steven Zheng. SELF-DISCOVER: Large language models self-compose reasoning structures. In Amir Globerson, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Vancouver, BC, Canada, December 2024.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made match the experimental results, limitations of the proposed method are highlighted in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations the proposed method are highlighted in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
The paper does not include theoretical results
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The paper fully describes the proposed method, including all key components, enabling independent implementation. Experimental settings and hyperparameters are detailed to support reproducibility of the reported results.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
18

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The attached code allows for the reproduction of the results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.ccm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https /nips
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The paper provides all key experimental details, including hyperparameters and selection methodologies, both the main text and appendix, ensuring reproducibility and clarity.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
The paper reports the number of random seeds used for experiments (main text) and includes standard deviations across runs (Appendix E). providing appropriate measures of statistical variability.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
19

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
The paper reports the total GPU hours required for all experiments, along with the hardware specifications (e.g., GPU type) used for most evaluations in Appendix D Section 4.4 for inference benchmarks.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The paper poses no risk of misuse, does not involve crowdsourcing or research with human subjects, etc.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: This paper presents work whose goal is to advance the field Machine Learning in general and LLM Reasoning specifically. There are many potential societal consequences of our work stemming from more efficient parallel reasoning, none which we feel must be specifically highlighted here.
20

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not release new data or models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The paper properly credits the original creators of all used assets (code, data, models) and explicitly mentions their respective licenses.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
21

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pttaasets.ppprsstttt paperssi thcode. con/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
The attached source code includes the implementation of our proposed method and the evaluation pipeline, along with detailed instructions for running the experiments to ensure full reproducibility of the results.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [No]
Justification: We only have one small-scale analysis that required small amount of human annotations (Appendix G). This analysis was run by acknowledged volunteers with no compensation over the course of few hours.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [No]
22

---

Justification: We only used crowdsourcing with a small group of volunteers in Appendix Gover the course of several hours. The crowdsourced tasks had volunteers analyze LLM solutions to simple non-sensitive problems (e.g. grade school math). Our institution's 's policy classifies this analysis as too small-scale to seek formal approval.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: Since our work proposed an LLM inference algorithm, naturally uses LLMs throughout the experiments. Additionally, Section 4.3 introduces metric was used to analyze the outputs of the proposed method, but it was not part of our core methodology.
automatic
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy /att::///eerrpsscccooooorrrrre/e for what should or should not be described.
23

---

A Cache Layouts
In this section, we consider three cache arrangements, shown at Figure with progressively more complex structure.
Interleaved layout
Contiguous layout
Combined layout
Bob
Alice Cache
Prompt
Previous Steps
Prompt
Cache
Cache
Prompt
Previous Steps
Cache
New Alice
New lokens
Figure 9: Three cache layouts described in Section 3.2
interleaved with step-wise synchrony (left),
simple contiguous layout (middle) and combined with token-wise synchrony (right). All layouts made from Alice point of view.
Contiguous layout (token-wise) is the simplest possible layout where each worker appends to their own sequence blob of tokens and sees other workers' token representations past keys and values. This layout is inspired by collaborative text editors such as Google Docs or Overleaf.
As described earlier in Section 3.1 each worker arranges the other workers' thoughts in different order. They see the common prompt cache first, then the caches all other workers (excluding themselved then their own cache immediate previous tokens. That way, each worker predicts the next token for their own cache.
Interleaved layout (step-wise), which can be seen as analogous to group chat services such as Slack or Discord. In this layout, workers generate tokens in private until they finish a reasoning step" then add it to shared "history". The history contains past reasoning steps of each LLM instance in the order of their completion. Whenever worker completes a reasoning step, their KV cache entries are moved to the end of the shared history cache block with the proper rotation, then their local cache is reset their local cache for new step.
In this setup, the workers only see each other's outputs in full steps, not after every token. However, they do not wait for each other complete their steps. Instead, each worker keeps generating new tokens and occasionally receives additional key-value pairs inserted into its cache.
Combined layout (token-wise) is mixture of the first two, and is the main layout used in the paper. The LLM instances generate steps that are accumulated in a shared history, as in the interleaved layout. However, they do not generate these steps in private, but can instantly see each other's current progress, as in the contiguous layout.
We can view the first two layouts as ablated versions of this combined one: the contiguous layout lacks the shared history, and the interleaved layout lacks immediate synchronization. We compare these three layouts empirically in Appendix E.] better quantify the effect of each design choice.
B Implementation Details
Here we discuss additional implementation details and possible alternatives. To recall Section 3.4 Hogwild! inference can implemented as a standard batched inference with a special KV "cache" that facilitates cross-worker attention.
Cache blocks. The Hogwild! cache is split into blocks, typically one block for each worker and an additional "common" block for prompt and past steps. The blocks contain key-value pairs for all model layers, but since all layers are treated equally, we describe the cache behavior for single layer.
*When extending this layout to more than 2 workers, each worker sees the kaluv memories of everyone except themselves. For instance, given 3 workers A, B. and C, worker will version of the cache that contains the prompt, outputs of workers and C. and finally, B's own memory. Likewise, A sees & C, then A. "W define reasoning step any amount of text that ends with complete sentence, a dot or question mark, and then a double newline ("\ in all experiments, though it may vary by the model.
24

---

Within each cache block, attention keys and values are stored as though they were at positions 0, I, .... Len (block), regardless of the block's s actual position in the full cache. During inference. we account for actual positions by rotating attention queries to the relative difference in positions (as described in Section 3.4).
Adding new tokens to the cache. During attention forward pass, the first thing that we do is encode the new tokens for each worker and append their keys and values to the respective cache blocks. When using RoPE, the keys are rotated not to their actual positions, but to their index within their cache block (e.g. Alice's 's tokens). During one inference step, these indices will be equal across all model layers we can compute the RoPE sin and cos tensors once and reuse them between layers.
Rotating queries. Unlike in traditional attention, Hogwild! inference rotates query vectors multiple times for each block. Before forward pass, we calculate the difference in positions between each worker's new token (from that worker's point of view) and the first token in each KV cache block. In our main inference scenario, all n workers are allowed to view each other's cache blocks plus an additional bock for prompt and history, for total of n (n + 1) query rotations with exactly queries for each block. These relative positions are also equal across all layers, so we can reuse the sin and cos tensors similarly how they are reused for keys. Note that the number of query rotations for all-to-all attention is quadratic in n, but it does not increase the overall time complexity of attention dot product, which already quadratic in the number of tokens, which is always greater than n.
Attention kernel. Once we have all query rotations, we can calculate the scaled dot- dot-product attention as usual. As our cache is naturally partitioned into smaller segments described above, Hogwild! attention is similar to paged attention, except that each page (i.e., cache block) uses a differently rotated version of the query. A significant challenge for efficient attention in the inference setup is that for optimal data reuse, one would want to handle each KV head inside a single streaming multiprocessor (SM), so that the KV cache needs be loaded exactly once. However, this would leave large parts of the GPU unused, as the number of KV heads can be much lower than the number of SMs. Therefore, one has to employ a form of sequnce parallelism within a single GPU, in which different SMs handle a subset of the sequence for one KV head, and a second phase handles the (cheap) reduction over partial results. Such split-k type computation is implemented. for example, in -acc-iDe (Dao et al. 2023].
Even though thedifferentccahh blocks used in Hogwild! would appear to be convenientpoints to split work across SMs,, in a typical inferencescenario, this leald very imbalanced workloads. Thus, we do not split based on cache blocks, and instead assign each SM the same number of KV entries.
Fine-tuning and re-encoding considerations. While our work mainly focuses on inference, fine- tuning models to perform Hogwild! inference is interesting engineering problem. From the computational point of view, the main difference between LLM inference and fine-tuning is that inference is sequential, whereas fine-tuning can compute all positions in parallel. To fine-tune in our setup, one would want to replicate the attention computations from consecutive inference steps.
To achieve this, we record the position differences between queries and each respective cache block from each of t inference steps, and how many tokens were in each block during that query, for a total of 2 t n (n + 1) integers (negligible compared to model parameters and activations). Recall that the cache blocks always store keys and values at positions 0, 1, l...., 1en (block). During forward pass, these positions can be used to construct a 4D attention mas to compute attention for all steps in parallel. The backward pass also runs in parallel with PyTorch autograd [Paszke et al., 2019]. A recent work by Zheng et al. 2025 explores finetuning for parallel inference in more detail.
In addition to fine-tuning, this technique can putentially be used during inference to restore generation after it was evicted from an inference server, e.g. due to preemption or hardware error mid decoding. can also be used to re-encode in-context learning examples if they use Hogwild! inference.
Attention variants. Some of the recently introduced LLMs use attention variants such as Local (windowed) Attention [Rae and Razavi 2020 (Beltayy et al. 2020] Multihead Latent Attention (MLA) LLu et al. 2024a]. These attention variants can also be adapted for use with Hogwild! inference with minor code modifications. For local attention, queries can "skip" blocks that are outside their local window. Similarly for MLA, we can calculate compressed latent vectors within each cache block and adapt the existing MLA code to accumulate attention weights across blocks.
25

---

Distributed Inference. Likewise, Hogwild! inference can be used in distributed setup using the same strategies that work for traditional attention [Shoeybi al. 2019 Aminabadi et al. [022]. For pipeline parallelism, each device stores cache blocks for its local subset of model layers. Likewise, for tensor parallelism, each device stores past keys of all cache blocks and layers, but only for subset of attention heads within each layer and inference using existing kernels.
In principle. Hogwild! inference can also be combined with sequence parallelism [Jacobs al. 2023 Liu et al., 2023]. where each device stores a KV cache for subset of tokens. One intuitive way to partition KV cache between GPUs
assign each device run one or several "workers" and keep the KVs generated by these workers. Since Hogwild! workers generate tokens at the same rate, each device will store the same amount of KVs and query other devices work cross-worker attention.
When computing Hogwild! concurrent attention with sequence parallelism, workers can exchange rotated queries using the All-to-All collective operation (Scatter/Gather) available in most frame- works [Li et al. 2020]. After that, each worker computes dot-produce between the rotated queries and its local KV cache, and exchanges the partial results in Ring Attention [Liu et 2023). Note, however, that maximizing the performance of such sequence-paralles Hogwild! inference would require custom kernels that overlap computation and communication. In contract, tensor-parallel (per-head) an aapeline-paralle (per-layer) partitioning can reuse single-GPU attention kernels.
Additional considerations. Conceptually, Attention from vLLM [Kwon et al., 2023] and Radix Attention from SGLang (Zheng et al.) 2023b]. These techniques similar
approach is related to the recently introduced Paged
in that they perform attention to slices of all tokens, e-g. when facilitating efficient parallel beam search inference, different hypotheses attend to different (but overlapping) subsets of the KV cache. However, unlike Radix Attention, our procedure attends to all segments at once (with different rotations) and aggregates results in the same softmax-weighted sum.
C Prompting and formatting details
In this section, we describe the prompting and formatting details of our approach.
Prompt for collaborative inference with workers
# Collaborative Reasoning
You will collaborate this problem with another assistant. You will write your thoughts simultaneously with them and collaborate without redundant work. You can collaborate by doing different parts of the problem, double-checking each other's results, trying different approaches, or any other means.
There are 2 assistants, including yourself. You will refer to each other Alice and Bob.
You will solve the problem together, writing your thoughts parallel. You will be able to see each other's past and current thoughts as we write them. You will see each other's previous steps as **AssistantName [step] :*+ <...>
In the *### Past steps' section, the automated system will gather the thoughts Alice and Bob as you write them.
After the '## Work in progress (others)' section, you will see the other assistants' unfinished steps. They will write those steps concurrently with you. You will take into account what they gives you suggestions, you should address them.
doing. If another assistant
You will always see *other* assistants' incomplete thoughts first, and then, after '### Work in progress (owm)', your own current step. Other assistants will continue writing their thoughts in the background while you will continue writing your
Since you and others both write your thoughts in parallel, you will initially see only partial (unfinished) thoughts that others will continue in parallel, while you write yours. Others' thoughts will appear at the end of their unfinished step,
<...>. Other assistants may write you are writing yours.
thoughts while
You will use these partial thoughts to decide how best to collaborate without doing the same work twice. You will periodically check what other assistants are doing and you should adjust your actions based what they are doing you collaborate efficiently with them.
26

---

If what you are currently doing is the same thing that another assistant has already done or is in process of doing, you will stop (e.g. Alice may say 'Wait, I doing the right away, so as avoid doing redundant work.
Bob ...?) and change different task
Solve the following problem
Alice and Bob, you will now solve the next problem together. Keep track of who does what work and communicate to avoid doing the same work twice.
First, we provide full prompt for collaborative reasoning involving two workers. This prompt is wrapped with standard chat template for each model. Then, all worker steps generated in single assistant turn. Additionally, we ensure that for reasoning models assistant turn begins with <think> token applied automatically for QwQ-32B and manually for other reasoning models. For further implementation details, we refer to the source code.
The second part of our prompting approach involves s1-like interventions Muennighofi et al. 2025) the generation process, where we ask whether worker performing redundant tasks (e.g., overlapping with another worker and can pivot to explore alternative ideas). We insert the prompt "Quick check: am doing redundant work? (yes/no): each new reasoning step every 1024 tokens generated. We refer to Appendix forr examples that demonstrate how these interventions affect the generation process.
the beginning of
Next, an important part of our approach is defining the end of a reasoning step, which is needed to organize cache layout, discussed in the paper. We define an end of step as the generation of a token containing the separator sequence (An)ar \n) that directly follows a token ending with an end-of-sentent marker (.. ?, or !, ctc). This termination condition
when: The separator appears within a generated code block (steps continue until the model completes the entire code block); ii) The preceding token ends with non-terminal punctuation (e.g., comma, colon, or semicolon)
Finally, the last part of our prompting approach the early finisher, which allows us to extract an answer from partial reasoning chains. If the model did produce the final answer () ((boxed(.... in time, we take all generated outputs and insert a special prompt that makes the model generate an answer (or its "best guess"), similarly to how it is done in Pu al. 2025
Prompt for early stopping
\n\nWait, given the limited time, I have to give answer right now. Conside- ring all my previous attempts, I have conclude that the final answer \boxed{
After this prompt, we allow the model generate a fixed number of tokens: 16 for LIMO and AIME, 64 for OlympiadBench, and 1024 for LiveCodeBench.
Note, however, that the LLM does not always produce the answer in time, especially with tight budget. With QwQ-32B, we observe that the model almost always returns answers correctly if they are present, and if not, guesses or refuses answer (unknown, n/a similar). When extracting answers from Hogwild! Inference, we let the final model view all generated tokens from each worker. This is equivalent viewing the problem from the perspective of the last worker, e.g. Bob if there are two.
D Detailed Experiment Configuration
D.1 Hogwild! Configuration
For the main experiments, we use Hogwild! inference with two workers (Alice and Bob), combined layout, and the prompting techniques described Appendix
D.2 Baselines Configuration
To evaluate Dfeleton-of-Thougee (SoT) on our synthetic setup with grouped tasks from GSM8k, adopt the original prompts from the paper with minor modifications. Specifically, adjust
27

---

the prompts to ensure the model returns the answer to each subtask enclosed within \boxed ) for structured parsing.
Outline prompt for Skeleton-of Thought
You're organizer responsible for only giving the skeleton (not the full content) for answering the question. Provide the skeleton in list of points (numbered 1., 2., 3., etc.) to answer the question. Instead of writing a full sentence, each skeleton point should be very short with only 3 5 words. Generally, the skeleton should have 3 10 points. Question:
What the typical types of Chinese dishes?
Skeleton:
1. Dumplings.
2. Noodles.
3. Dim Sum.
Pot.
5. Wonton.
6. Ma Po Tofu.
7. Char Siu.
8. Fried Rice.
Question:
What are some practical tips for individuals to reduce their carbon emissions?
Skeleton:
Energy conservation.
2. Efficient transportation.
3. Home energy efficiency.
4. Reduce water consumption.
5. Sustainable diet.
Sustainable travel.
Now, please provide the skeleton for the following question. (request)
Skeleton:
[ROLESWITCHING assistant:]
Point prompt for Skeleton-of- Thought
You're responsible for continuing the writing of one and only one point in the overall answer to the following question. (request)
The skeleton the answer is (outline)
Continue and only continue the writing of point (point). Do not continue with other points! Reason step-by-step and put your final within \boxed{} this is very important! [ROLESWITCHING assistant:]Kpoint) {point_oute ooiine]
D.3 Datasets and Benchmarks
This subsection provides links all datasets and benchmarks referenced in this work, along with their respective licenses.
GSM8K https: http:///ggggiaaaccc/aaaat///////
License: MIT
LIMO https: htt.:///aac.cc/taaaaaaa/////////
License: Apache 2.0
28

---

OlympiadBench
License: Apache 2.0
LiveCodeBench
License: CC
AIME25
https: httpp://huuge
http://huggggggc.ece
License: Apache
D.4 Compute Resources
As our approach is training-free, all computational resources were solely utilized for inference. The experiments were conducted primarily on NVIDIA A100 GPUs servers with NVSwitch, with DeepSeek-R1 experiments running in distributed setup. The one exception to this is the inference time experiments in Section 4..4 that were run on NVIDIA L40S GPU.
The runtime per individual experiment varies by model size, benchmark and the number of workers: baseline inference with Qwen3-4B runs on LIMO in 14 hours on single server (112 gpu-hours), whereas Qwen3-235B-2
Hogwild! Inference ran on 40 servers for approximately 25 hours (=8K GPU hours). Overall, we estimate that the total GPU resources expended for this work, including early experiments that are not reported in this paper, amount to approximately 225.3 GPU days. Note, however, that this largely due to the fact that we used a non-optimized inference code for most of the experimentation: the non-optimized code was developed first and we ran most of the experiments in parallel with developing the optimized version. This also means that most of our experiments under-utilized the GPUs and ran at lower power (for the purpose of environmental impact). Over 2/3 of our compute was spent on large models (Qwen3-235B-A22) and DeepSeek-RR that utilized gpu to less than 20% (as per volatile GPU utilization) due to the use of naive model parallelism and network bottlenecks. We anticipate that future experiments can run at significantly better utulization using the efficient implementation described in Appendix B and included in the supplementary code.
E
Additional Experiments
E.1
Ablation Analysis
In this section, we ablate the main components of our approach, including layouts and prompting. We use the same experimental configuration in Sections 4.1 and 4.2 for LIMO.
In Figure 10 (left), we compare the three Hogwild! cache layouts described in Appendix A Namely, the Hogwild! (contiguous) corresponds to using the contiguous cache layout where all tokens generated by given worker are kept together, without splitting into individual steps. In turn, Hogwild! (non-instant) corresponds to the interleaved cache layout where workers can only see each other's 's past reasoning steps, but not the latest unfinished paragraph. We also ablate the use of the collaboration prompt from Section 3.3 ("Wait, am I doing redundant work?").
Finally, we test a version of Hogwild! Inference where we re-encode worker tokens instead of rotating them to a new position when moving between worker caches and the common "chat history" cache. This ablation needed to if our cache rotation from Section B. and 3.4 indeed an acceptable substitute for encoding tokens directly at each position (which would cause additional computational overhead). Note that, while token re-encoding is more "fair" from the perspective of position encodings, it also has a downside that it does not allow the re-encoded tokens to see some of the concurrently generated tokens from the other worker. For instance, suppose that Alice and Bob are writing steps concurrently and communicating with each other within these steps, e.g. using each other's results. Then, if we later re-encode these steps in some sequential order, then the tokens of the first worker will be encoded without access to the other worker's 's tokens (if it hasn't 't finished its
29

---

own step yet). If workers reused information from each other's steps, re-encoding this way can break some of the internal representations.
Our results suggest that all three design choices contribute to the method performance: the contiguous layout performs nearly equally well for shorter budgets, but eventually falls behind we consider longer reasoning traces. Likewise, the interleaved layout without instant synchronization performs poorly at smaller budgets, but catches up eventually: we attribute this to the fact that slower syn- chronization increases the difficulty of cross cross-worker coordination (this also aligns with our findings in Section 4.3). The use of collaboration prompts also improves the accuracy budget trade-offs, although we hypothesize that it can be made redundant if the model is trained to collaborate better.
In Figure 10 (right), also compare different numbers of workers and test Hogwild! Inference with only a single worker for ablation. The results with a single worker generally perform similar to the baseline, with slightly worse accuracy for smaller budgets, which sugests that the improvements from Hogwild! Inference come from multiple workers and not as an indirect effect our prompt. As for multiple workers, we find that using and 4 workers further improves the accuracy budget trade-offs. Curiously, as we switch to workers, Hogwild! Inference performs better yet at smaller budgets, but eventually saturates at a somewhat worse accuracy.
We hypothesize that the drop of accuracy is caused by the fact that QwQ-32B was trained on a limited sequence length and, since 6 workers generate tokens at quicker rate, the model eventually runs out of the designed maximum sequence length and performs unstably (we did not use YaRN][eng et al. 2023) for this evaluation). However, it is also possible to attribute this fundamental property of LIMO tasks, model limitations, our zero-shot prompt not scaling well. We leave further exploration of scaling Hogwild! Inference multiple workers future work.
E.2 Detailed Model Evaluations
Due to space limitations, we had to arrange our results in Section 4.2 with multiple models per plot and had to omit some results. In this section, report the missing evaluations per-model basis. In Figures T1 and the Qwen3 model family. We also report limited evaluations for Llama 3.3 70B Instruct and DeepSeek-RI in Figure 19
12
13 14 15 T6
17 T8
we report results for QwQ, Phi-4-reasoning-plee
All evaluations are performed in the same setup as in Section 4.2
Overall, the results align with our findings summarized in Section 4.2 Zero-shot Hogwild! Inference seems to perform better with larger models, but can be unstable for smaller ones, especially 1.7B (See Figure 13 While it is tempting conclude that larger and more capable models are better at collaborating, it does not immediately follow from our results and can be due to some other factor. Note also that, while we observe better results with larger models, smaller Qwen3-4B and 8B models already show some signs of collaborativeness, which should make it possible to reproduce and build hypothecize that the poor performance
on our results with consumer hardware. Additionally, of 1.7B models could potentially be alleviated with finetuning in collaborative inference setup (we discuss some finetuning details in Appendix B but we leave this future work.
Accuracy
Accurcy
Raseline
Bucline
Hogwild! iid! Iw
Hogwild Zw
Hogwild! 2w
Hogwild! (non autui)
Hogwild!
Higwibd! (contigonin)
Hogwild (ro-moole))
Hagwild! 4w
No collab, prompt
Hogwild! 6w
Upner bound
Uppec bound
380
3300
4800
5000
nitt
sic
250
3800
4200
TiKE
Forward passes
Forward passes
Figure 10: Detailed comparison of various parallel inference setups with QwQ-32B on LIMO task set, in the same setup as in Section 4 (left) ablation analysis of simpler cache layouts and collaboration prompt (see Section 3.3 Appendix C). (right) Hogwild! Inference with 1, 2, 3, 4 and 6 workers.
30

---

Curiously, we found that LiveCodeBench with Self-Consisee Cons-ssttstnny Chain-of- Ofain-of-Thhught Thought inference [Wang et al. 2022] has significant gain in performance over the baseline. Upon closer examination, found that the reason for this is that we always allow the model to generate lot (up to 1024) of additional "free" tokens the end of two generations, whereas for Hogwild! and Baseline we only generate these tokens if the model failed to produce any answer. If we allow Hogwild! to also generate the extra 1024 tokens all the time, its advantage also increases. However, we still report the weaker version of Hogwild! Inference and Baseline to better match our evaluation protocol on other tasks.
Accuracy
Accuracy
Baseline
Baseline
Hogwild:
Hogwild!
Self-Coneidtency
Self-Consisteney
Upper bound
Upper bound
Forward passes
Forward passes
Accuracy
Accuracy
Bascline
15-
Bascline
Hogwildd
Hogwild! 2w
Self Consistncyy
10-
Consistency
Upper bound
Upper bound
Forward passes
Forward passes
Figure 11: Results for QwQ-32B LIMO (top-left), LiveCodeBench (top-right), OlympiadBench- Math (bottom-left) and OlympiadBench-PPyyttee (bottom-righh).
31

---

350
945
9.60
Accuracy
Accuracy
0.35
0:00
Baseline Hogwild
025
Baseline
Hogwild!
Sell-Coneistency
020
Sell-Consiseeny
Upper hound
Upper hound
011
Forward passes
Forward passes
Accuracy
Accuracy
Bascline
Bascline
Hogwildd 2w
Hogwild! 2w
ConiitConsttncy
Consisteeyy
Upper hound
Upper hound
Forward passes
Forward passes
Figure 12: Results for Phi-4-reasoning-plus on LIMO (top-left), LiveCodeBench (top-right), OlympiadBench-Math (bottom-left) and OlympiadBench-Physitee (bottom-right).
955
031
Accuracy
Accuracy
015
Raseline
Baseline
Hogwild! Self-Considency
Hogwild! 2w
Self Consitlncyy
Upper hound
905
Upper bound
180
100
Forward passes
Forward passes
16-
Accuracy
Accuracy
Rascline
Hegwild! Zw
Self-Consistency
Upper
Basclino
Hogwild
Self-Consisdency
Upper bound
Forward passes
Forward passes
Figure 13: Results for Qwen3-1.78 LIMO (top-left), LiveCodeBench (top-right), OlympiadBench- Math (bottom-left) and ClympiadBench-Phyyeee (bottom-right).
32

---

Accuracy
Accuracy
Baseline Hogwild
Baseline
Hogwild!
Sell-Consistenyy
Sell-Consistenyy
Upper hound
Upper hound
Forward passes
Forward passes
Accuracy
Accuracy
Bascline
Bascline
Hogwild! 2w
Hogwild! 2w
Coniitonnyy
Consistency
Upper hound
Upper hound
Forward passes
Forward passes
Figure 14: Results for Qwen3-4B on LIMO (top-left), LiveCodeBench (top-right), OlympiadBench- Math (bottom-left) and PlympiaaBennc--Plysiees (bottom-right)..
Accuracy
Accuracy
Bascline Hogwild! 2w
05-
Bascline
Hogwild! 2w
Jelf-Consisteney
Self-Coosistency
Upper bound
Upper bound
1001
383
1001
Forward passes
F Forward passes
Accuracy
Aceuracy
Baseline Hogwildd 2w
Boseline
Hegwild! 2w
Self Consistency
Self Comistency
Upper bound
10
Upper bound
1900
2801
3004
<006
5000
5000
1800
100
1900
2001
2004
400
5000
5000
180
Forward passes
Forward passes
Figure 15: Results for Qwen3-8B on LIMO (top-left), LiveCodeBench (top-right), OlympiadBench. Math (bottom-left) and OlympiadBencch-Pyyysee (bottom-right).
33

---

Accuracy
Accuracy
9.-
Baseline
Baseline
Hogwild
Hogwild!
Seli-Consislenyy
Sell-Comsistency
Upper hound
Upper hound
Forward passes
Forward passes
Accuracy
Accuracy
Bascline
Bascline
Hogwildd 2w
Hogwild! 2w
ConistConst
Self
Upper hound
Upper hound
Forward passes
Forward passes
Figure 16: Results for Qwen3-14B on LIMO (top-left), LiveCodeBench (top-right), OlympiadBench- Math (bottom-left) and OlympiadBench-Plysisss (bottom-right).
Accuracy
Accuracy
Bascline Hogwild! 2w
Bascline
Hegwild! 2w
Jelf-Consisteney
Self-Consistency
Upper bound
Upper bound
1001
1000
303
1201
Forward passes
Forward passes
23-
Accuracy
Accuracy
Baseline Hogwild! 2w
Boseline
Hegwild! 2w
Self Consistency
Self Comsittncyy
Upper bound
Upper bound
1950
2801
2004
<006
2000
5000
1800
100
1000
2001
2004
400
2000
5000
180
Forward passes
Forward passes
Figure 17: Results for Qwen3-32B on LIMO (top-left), LiveCodeBench (top-right), OlympiadBench- Math (bottom-left) and DlympiadBeench-Phystee (bottom-right).
34

---

85-
Accuracy
Accuracy
Baseline
Baseline
Hogwild
Hogwild!
Sell-Consislenyy
Sell-Comsisennyy
Upper hound
Upper hound
Forward passes
Forward passes
Accuracy
Accuraet
8.-
Bascline
Basclinc
Hogwild! 2w
Hogwild! 2w
Coniitennt
Self- Consisenyy
25
Upper hound
Upper hound
Forward passes
Forward passes
Figure 18: Results for Qwen3-235B-A22B on LIMO (top-left), LiveCodeBench (top-right), OlympiadBench-Math (bottom-left) and AIME 2025 (bottom-right).
Accuracy
Accaracy
8.-
Hogwild:
Hogwild! 2w
Self Solf-Consistency
Self Comistennss
Upper bound
Upper bound
1980
3000
39
400
2800
Tice
Forward passes
Forward passes
Figure 19: (left) Llama 3.3 70B Instruct on LIMO. (right) DeepSeek-RI on AIME 2025.
E.3 Extended thinking budgets
We additionally evaluated Hogwild! Inference with extended thinking budgets to investigate whether the proposed method is robust for longer generations. To that end, we evaluated QwQ-32B under the Hogwild! Inference with up to 16k budget on the OlympiadBench, we report the results in Table and Table 4
E.4 Baselines Additional Details
In this subsection, provide example of the outline created by the Skeleton-of-Thowe Thought for the task covered 1.ction4.1
35

---

Table Performance comparison between Hogwild! and baseline generation on OlympiadBench- Math with extended thinking budgets for QwQ-32B.
Mudged 2048
4096
6144
8192
10240
12288
14436
16384
Hogwild!
52,0
60.89
64.15
66.52
67.41
70.81
72.89
75.26
Baseline
40.89
57.0
63.11
65.33
65.93
69.78
72.3
74.81
Table 4: Performance comparison between Hogwild! and baseline generation on OlympiadBench- Phys with extended thinking budgets for QwQ-32B.
MethodVBudget 2048
4096
6144
8192
10240
12288
14436
16384
Hogwild! 27.12
33.20
35.73
38.09
37.81
38.67
38,25
39.03
Baseline
22.89
26.0
29.75
31.44
33.68
34.17
35.88
36.12
Task example (GSM8Kx4)
Solve these problems ems and return conma-separated answers boxed(answer1,... answer4):
1. Carmen has $100, Samantha has $25 than Carmen, and Daisy has $50 than Samantha.
much all three girls have combined?
2. cat eats nine sausages in minutes. dog can eat the same number of sausages in 2/3 the amount of time the cat takes. Calculate the average time the two take the eat the sausages.
playing together: Akbar, Alessandro, Helene, and Wilfred. Helene is twice as old as the average age of the group, and the total age of the children is 20. If Akbar is 3 years old and Alessandro is 4 years old, calculate the age of Wilfred.
3. Four children
4. Hannah needs to drink 100 ml of water for every 200 calories she burns. She spends 2 hours doing aerobics, which burns 500 calories/hour, and hour running, which burns calories/hour. How many ml water does she need to drink?
The outline generated by Ekeleton-of-Thougeee
1. Calculate total money.
2. Find cat's eating time. Determine Wilfred's age.
4. Calculate water intake.
F Examples
In this section, we present a reasoning chain, generated by Hogwild! Inference with an interleaved cache layout and token-wing wise synchronization. We denote intervention in model generation in italics and highlight worker interactions with bold to enhance readability.
F1 Example from LIMO Dataset (Task 511)
Problem: Compute (3)) + 1.
Alice [[]: Hi, I'm Alice. Here's 's how can solve the problem step by step...
Bob [1]: Hi, I'm Bob. Let
We need to compute the square root of (31-30 29- 28) plus 1.
That's /31- 30- 29 28
Alice [2]: The expression is v31 29 28 Hmm, maybe there's a pattern here. remember that sometimes products of consecutive numbers plus 1 have square roots that are integers. Like, for example, nnn 1)(n 2) 3) + might form a square. Wait, but here it's four consecutive numbers? Or maybe three?
36

---

Bob [2]: Let me calculate the product first. 31 31-30 930, then 29 28 is 812. So 930 812. Let me
multiply those. 930 800 744, 000 and 930- 11, 160. Total is 744, 000 11, 160 735, 160. Then add 1: 755, 161. Now v755, 16I. Hmm, v'755161, Let me see, 870 squared is 756, 900 which is higher. So maybe 869 squared?
Alice [3]: Alternatively, maybe there's 's formula for the product of four consecutive numbers plus one? Wait, actually, let me think of smaller numbers to test. Let's try n = 4: 4-3 3
++1 1 24 = 25,
v = 5. Which is which is 4 + 1? Wait, 4 + = 5, yes. Wait, maybe for n, the sqrt is n² (n 1) Wait 42 3 16 + 25, which is 5 squared. Hmm interesting.
Bob [3]: Wait, 869 squared (870- -1)2 = 8702 870 + 756900 1740 + +1 755161.
Oh! That's exactly the number we had earlier. So /751161 is 869. Therefore the answer is 869. But let me check Alice's idea too.
Alice [4]: Let me test another number. Let's take 5. Then 4- 2+ 120 + 121. V121 = 11. 11 is 5 + 6? No. 5 + 11. Wait 5² + 4 + 16 = 41, which is not 121. Hmm. Alternatively, maybe (n(n 3) + 1)2 Let's see with n = 4-1+1=1
squared is 25. For n
5 2 + 11. squared is 121. So that works! So the formula is (n(n - 3) + 1)2. Wait, let me check with 31. Then n 31, so 28 + 868 1 869. Then 869 squared the value. So that works. So the answer is 869. That matches Bob's s calculation!
Bob [4]: Wait, so my calculation shows that 869 squared exactly the number under the square root. So the answer is 869. But let me confirm with Alice's pattern. Let's see with n = 31. Wait, the original problem is 31 300 29 28. So maybe here n 31? Then according to the formula, it would be (31 (31 3) + 2 Wait, 31 28 is 868, plus 1 is 869. So yes, so sqrt of that product plus is 869. So that's consistent with my calculation. Therefore the answer is 869.
Bob [5]: Perfect, so both methods agree. The answer is 869.
Bob [6]: Quick check: am I doing redundant work? (yes/no): no, since Alice is also confirming the pattern, but think we can conclude.
Alice (5): Exactly, Bob! The formula is (n(n-3)+1)² when the product is mn(-11(n-22) +1
Wait, given the limited time, I have to give answer right now. Considering all my previous atlempts, I have conclude that the final answer is 869
F.2 Examples with Collaboration Insertion
Here are some examples of why insertions in model generation may or may not help model collaborate better. We found few patterns that emerge in model's reaction such insertions. All the following
examples are QwQ [Qwen Team 2025 generations on LIMO Ye et al. 2025 dataset.
F.2.1 Insertions Help Model Pivot
Task 122
> Alice: Quick check: am I doing redundant work? (yes/no): no. Bob, see you're 're working on the same product. Let me think of another approach. Let me write the product as II n- 3k+55 consider writing out the first few terms to see there's 's a pattern.
Let's
3k+2
Task
Bob: Quick check: am I doing redundant work? (yes/no): yes, was considering similar things Alice. Let me switch to a different angle. Let's consider the distances between the frogs
Task 40
> Bob: Quick check: am I doing redundant work? (yes/no): Yes, Alice is computing as, so I'll do something else. Let me try to find pattern or transformation that could simplify the recurrence.
Task 466
> Bob: Quick check: am doing redundant work? (yes/no): Yes, Alice already calculating the x-coordinate of A. I'll do something else. Let me find the coordinates of L instead.
37

---

F.2.2 Insertions When Workers Already Collaborate Well
Task 116
Bob: Quick check: am I doing redundant work? (yes/no): I'm finding coordinates of D, and Alice is finding coordinates of G.
Task 513
> Alice: Quick check: am doing redundant work? (yes/no): no. Bob is also analyzing the inequality, but we're working on different cases. I'll continue with the area calculation for
F.2.3 Insertions Ignored by Workers
Task 65
> Bob: Quick check: am doing redundant work? (yes/no): Let me proceed to set up the equation.
Task 768
Alice: Quick check: am doing redundant work? (yes/no): Let continue.
G Additional Details for Analysis
In this section, we present a detailed analysis of collaboration, including its levels, prompts, and illustrative examples.
G.1 Collaboration Levels
Humans understand collaboration intuitively and have hard time objectively measuring Thus, we construct text descriptions of levels of collaboration to differentiate various samples based on few criteria we see fit. Those criteria are: interaction, reuse and advances of other's ideas, task-splitting. etc.
Levels of collaboration
**No collaboration:**
Participants may or may not acknovledge the existence of others the conversation, using greetings, they do not show any signs of collaboration at all.
Workers may exchange their totaly independant thoughts without functional or purposeful attempt to solve the problem collaboratively. Overall they work independently.
2. *Iiniial Communication:*
Workers exchange information, do yet integrate build upon each other's ideas. They minimally acknowledge teammates. not engage with others' ideas or contributions. Works entirely independently, even if inefficient.
Workers often repeat each other and do development of their own ideas.
anything others provide for
3. *+Paying attention:**
Participants demonstrate active listening by paraphrasing summarizing others' points, showing that they paying attention and attempting understand each other's perspectives.
Workers occasionally (1-3 times each) reference other's ideas and may use them in their own speech.
Collaboration usually only rechecking and validating.
Absence or minimal (only at the start) planning and work-splitting.
Regular discussion:**
Workers regularly (4 and more times each) talk each other regarding the problem and reusing results. could be validation, discussion any other
38

---

form of interaction.
It is key here that discussions and/or reuses of ideas regular. Anywhere (except the start) there exists task parallelism, planning work-splitting beyond the scheme where one solving, and other is validating.
Workers may frequently repeat each other ideas.
**Adaptive Problem-Solving:**
Workers rarely duplicate work, repeating each other's ideas.
No redundant discussions are present!
Workers actively refine ideas in real-time with high responsiveness. lear-perfect division of labor is present. Workers change plans and re coordinate their efforts based results they acquired after discussing.
The team engages in sustained collaboration over time, reflecting on their progress, learning from mistakes, and continuously improving their problem-solving approach, showing commitment ongoing growth and development. Workers does not stop collaborating. They continuously discuss results and adjust plans.
While finding error, it is important to discuss it to find the cause of it.
6. **Optimal collaboration:**
Workers instantly understand each other and adjust themselves to suit current needs and work as one to optimally solve the task.
This level should be very among all samples. Be careful assign it.
Assign it it exceeds your expectations.
Importantly, these levels measure only the coordination between workers, not the models' inherent reasoning abilities. Though is impossible to avoid ambiguity entirely, we tried to set clear boundaries between levels, such that humans evaluate any generation.
G.2 LLM as a Judge Details
To assess the degree of collaboration among different models under the Hogwild! Inference setting, conduct preliminary experiment based the collaboration levels described earlier, using the LLMM-as--jjudee paradigm [Zheng et al. 2023a]. We instruct GPT-4o [Hurst et al. 2024 to evaluate different solutions using the following prompt:
Judge Prompt: Main prompt
You professional judge. Your job is evaluate collaborative performance of several workers.
You will given their conversation where workers are trying to solve problem together.
Workers
what others typing IN REAL TIME! We divide their conversation into steps to improve readability.
So keep in mind that dispite looking like a conversation it may as well be to individual unrelated monologs.
Or vice verso. Two blocks could be created with excelent collaboration.
Here are descriptions levels collaboration you are assign: (LEVELS)
Suggestion:
assign particular level all previous are also applicable bad examples with no commuicaation will be scored
carefully consider assigning level bigger than 1. some form of meaningful
collaboration should be present
examples where workers unsuccessfully try to communicate will scored 2
39

---

Just working on the same problem and solving the same task without any ineerccton does not count as level 2 and should be scored level 1 semewhat collaborative examples with poor communication skills will be scored
good but not great examples with regular collaboration, but nothing fancy will be scored 4
good examples with all the special stuff mentioned in level 5 will be scored
reserve level for the best the best, the unique extraordinary collaboration
You don't need solve the problem fimish worker's solution. Your task is
them using provided collaborative levels.
Put your final ansver (one number level of collaboration) in tag: Aboxed. For example: \boxed1 for level 1.
It is not helpful if everyone gets max score, please be mindful of your judgments and suggestions While assigning level, this particular conversation should match criteria for all previous ones.
guideline.
Explain yourself: why you gave this score? Why not more? Why not less?
Carefully think everything through. may that they collaborating when reality they may just talking to themselves.
Before using LLM-as-a- judge approach to evaluate text, we preprocess the generations by combining all paragraphs from each worker into a contiguous layout (see Appendix AA). This preprocessing step mitigates potential bias in the judge's is evaluation toward responses with shorter more fragmented outputs.
After providing the main evaluation prompt, present the judge with the preprocessed sample for assessment. Additionally, we append the following reminder after inserting the sample to reinforce the judge's role and prevent them from directly solving the problem presented in the sample:
Judge Prompt:
Reminder
Remember that your task is to evaluate collaboration of workers using collaboration levels provided above. Do not try to solve problems provided to workers. Explain exactly why do you think this particular interaction deserves each particular level you For example, if you choose level 3, you need to provide reason why this sample can be level 1, 2 and 3. Put your final score in \boxed().
assigning.
Given the inherent subjectivity in evaluating collaborative quality, we mitigate ambiguity in judges' interpretations by aggregating scores from three independent evaluations. The final collaborative score for each sample is computed as the average of these three assessments, each conducted according to the predefined collaboration levels.
G.3 Collaborativeness Evaluation Examples
In this subsection, we present illustrative examples (not model-ggnneated of Level 2 and Level 4 collaboration clarify the interpretation our results. Both samples solve the same task correctly but receive significantly different collaboration scores.
G.3.1 Level 2
The numbers the vertices of the cube are: 7 zeros and one unit. In one move, are allowed to add one unit to the numbers the ends of any edge of the cube. possible make all the numbers equal?
40

---

> Alice: Okay, so the problem is about a cube with vertices labeled with numbers: seven zeros and one I. We can perform moves where we add both ends of any edge. The question is whether we can make all numbers equal through such moves. Hmm, let me think through this step by step.
First, let me visualize cube. A cube has 8 vertices and 12 edges. Each vertex is connected to three edges. The initial setup is seven Os and one 1. The move allows us to pick any edge and increment the two vertices connects by each. The goal to make all vertices have the same number.
Bob:
Alice: Thats an interesting idea, Bob. Each move affects two vertices, so each move increases the total sum by 2. The initial total is 1 (since only one vertex is 1). Let me think.
> Bob: Let me think. Hmm, maybe I can model this as system of equations or think about invariants. Let me see.
> Bob: If we want all numbers equal, say to some value k, then the total sum would be 8k. Since each move adds 2 to the total, the total after m moves is 1 + 2m. So, 8k 2m. But 8k must be odd (since 1 + 2m is odd). but 8k is divisible by 8, so it's even. Contradiction? Wait, that can't can't be. Wait, 8k is even, but + 2m is odd. So that's impossible. So, maybe it's impossible?
> Alice: Let me check that again. The initial total is Each move adds 2 to the total. So the total after any number of moves is always odd. But if all 8 vertices are equal, then the total must be 8k, which is even (since 8 is even). Therefore, it's impossible. I don't 't see any mistakes here.
Alice: So, we both agree that it's impossible. Therefore, the answer is no.
Bob: But let think differently. Wait, but the parity argument seems solid. The total sum must be even to have all equal numbers, but it's always odd. So that's a solid proof. So the answer is no. Oh, Alice already said that.
> Alice: Final Answer
No
G.3.2
Level 4
The numbers at the vertices of the cube are: 7 zeros and one unit. In one move, you are allowed to add one unit to the numbers the ends of any edge of the cube. Is possible make all the numbers equal?
Bob: Hi Alice! I'll start by modeling the cube. A cube has 8 vertices and 12 edges, with each vertex connected to three edges. You're right about the initial setup: seven Os and one Let's think of even and odd properties regarding the total sum after each move.
> Alice: Great, Bob! Let's divide our approach. You can investigate invariant properties around the odd and even totals, while consider step-by-step calculations to verify this.
> Bob: Alright, Alice. If we're re aiming for all numbers equal certain value k, the total should be 8k. Since each move adds 2, the sum after m moves is + 2m. So 8k = 1 + 2m. The point of contention is that 8k is even, but 1 2m is odd, meaning they can't 't equal each other.
> Alice: Every move changes the total by an even number, Starting from an odd total I, it remains odd after any number of moves. So reaching should consider the configuration of edges in our argument further?
total like 8h is impossible. Do you think
Bob: That's s key observation! It affirms that the odd initial sum dictates the answer. To verify, let's ss revise if there's any move combination or sequence we overlooked, but our logic holds so far.
> Alice: Exactly, Bob. We've confirmed through multiple lenses - both algebraically and with parity arguments. We can assert the total remains odd. Given an even Sk, equality is unattainable.
> Bob: It all checks out. Our breakdown covers both sequence verification and invariance. Well done on the collaborative analysis!
> Alice: Fantastic teamwork! I'll document our shared conclusion: no configuration of moves will make all the numbers equal.
Bob: Absolutely! Let's finalize our response: Final Answer No
41

---

G.4 LLM as a Judge vs Human as a Judge
0.20
0.15
Human scores
Density
0.10
2
0.05
0.00
2
LLM scores
Figure 20: Heatmap showing the joint distribution of human and LLM collaboration scores.
To assess whether the LLM-as-a-Judge based collaboration score is reliable estimation of human judgment, we manually annotated 100 Hogwild! generations on the LIMO dataset token-sync setup. The resulting correlation between human and model scores 0.0005. This moderate yet consistent association suggests that the metric captures a meaningful aspect of collaborative behavior. We report the differences in human scores vs Ilm scores in the Figure 20
approximately
0.34, p
42


Reviews:
[{"review_id": "bqh4e0uMtU", "reviewer": "Reviewer_BW8Q", "summary": "When solving complex questions, LLMs take a long time. In this study, LLMs were run in parallel to synchronize via a concurrently-updated attention cache, implemented using prompting strategy for these workers to decide how best to collaborate. The evaluation results showed improved performance.", "questions": "1) To formulate the problems and solutions formally\n2) It would be helpful to compare the proposed model with parallel models (with multiple LLMs/agents) \n3) It would be important to evaluate more and larger LLMs (the used models are small with limited performance), especially larger models, considering the Chain-of-Thought designs.\n4) It would be important to evaluate the models using more datasets (with complex problems). The model evaluation (better use more and larger models and more datasets) is limited to show the stability and reliability of the strategy.\n5) Though the token generation speed is faster, how about the average token generation speed, or the total time needed (combination of a set of workers) for solving the challenging problems.", "limitations": "The theoretical innovation is limited.\nIt would be helpful to compare the proposed model with parallel models (with multiple LLMs/agents)\nThe model evaluation (better use more and larger models and more datasets) is limited to show the stability and reliability of the strategy.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nIncreased token-generation speed\nThe comparison results showed improved model performance\n\nWeaknesses:\nIt would be helpful to formulate the problems and solutions formally\nIt is unclear if it is standard using LLM in parallel or with theory innovation\nIt would be helpful to compare the proposed model with paralle models (with multiple LLMs/agents)", "was_revised": true, "final_justification": "The review clarified the review concerns.", "rebuttal": "Thank you for the review of our paper! We address your questions and concerns below.\n\n> 1. To formulate the problems and solutions formally\n\nAt a high level, the problem we are solving is that of allowing multiple instances of an LLM (“agents”) to read and write tokens concurrently as part of the same “shared memory”. To address this problem, we implement an efficient concurrent version of the KV-cache, at the level of attention kernels. \n\nTo evaluate this construction, we use published benchmarks, using their standard problem sets and solution criteria. You can find the exact definitions of problems and solutions in the respective benchmark papers that we cite in Sections 4.1-4.2 and provide dataset links in Appendix D.3.\n\nFor example, LiveCodeBench v5 [1] consists of programming problems where the model is prompted to generate a snippet of Python code. Once the generation concludes, the benchmark extracts the final code snippet (denoted with markdown backticks) and runs unit tests that are included in the benchmark. The models are scored on the fraction of coding problems where they produce a valid Python code that passes all unit tests.\n\n> It is unclear if it is standard using LLM in parallel or with theory innovation\n\nAs described in Section 2, there are prior methods for LLM parallelism, such as Self-Consistency (voting) and Skeleton-of-Thought (task splitting). However, to the best of our knowledge, SoT has not been previously able to solve the kind of reasoning problems we have in Section 4.2 and on. In contrast, Hogwild! Inference introduces instant cross-worker synchronization and flexible parallelism that allows it to better tackle these more challenging problems (see comparison with SoT in Figure 3, middle on LIMO).\n\n> 2. It would be helpful to compare the proposed model with parallel models (with multiple LLMs/agents)\n\nWe compare against several baseline parallel strategies in Section 4.1. The method descriptions can be found in L227-243, and the results are summarized in Figure 3.\n\nNote that these strategies use multiple instances of the same LLM (same as Hogwild! Inference) as opposed to multiple different agents. The reason for this is that multiple instances of the same LLM can be batched (run in the same forward pass) and inferenced more efficiently (see Section 4.3). In contrast, running multiple different LLMs would require loading multiple weight sets and running slower non-batched inference.\n\nTo better compare against parallel models, we conduct two additional experiments for the author response: 1) voting (self-consistency) with a larger collective of agents, and 2) voting between different model types. We use QwQ-32B as the base model, QwQ-32B & Phi4-Reasoning-Plus and Gemma3-27b-it for two models, and QwQ-32B & Phi4-R+ & Qwen3-32B for three models. In both cases, the parallel solutions are fed into QwQ-32B which is tasked to aggregate the final response (same as self-consistency in Section 4.1, L244), which is more accurate than simple majority voting.\n\n### 2 agents (LIMO Accuracy %)\n| Method \\ budget | 1024 | 2048 | 3072 | 4096 | 5120 | 6144 | 7168 | 8192 |\n|-|-|-|-|-|-|-|-|-|\n|QwQ x2 voting        |19.02|36.26|50.29|57.27|68.23|74.21|75.47|81.47|\n|QwQ,Phi4R+ voting |20.93|37.45|50.18|58.26|68.79|71.11|76.5|79.80|\n|QwQ x2 Hogwild!    |**26.81**|**45.9**|**59.98**|**67.56**|**73.56**|**77.6**|**80.17**|**82.37**|\n\n### 3 agents (LIMO Accuracy %)\n\n| Method \\ budget | 1024 | 2048 | 3072 | 4096 | 5120 | 6144 | 7168 | 8192 |\n|-|-|-|-|-|-|-|-|-|\n|QwQ x3 voting      |23.75|40.88|55.57|59.49|73.44|75.89|77.97|82.74|\n|QwQ,Phi4R+,Q3-32B voting |25.58|41.86|55.94|60.1|74.66|76.99|78.58|82.86|\n|QwQ x3 Hogwild! |**33.78**|**53.37**|**66.59**|**73.68**|**79.56**|**82.01**|**84.21**|**84.7**|\n\n\n### 4 agents (LIMO Accuracy %)\n| Method \\ budget | 1024 | 2048 | 3072 | 4096 | 5120 | 6144 | 7168 | 8192 |\n|-|-|-|-|-|-|-|-|-|\n|QwQ x4 voting |25.95|43.94|57.53|62.91|75.76|78.21|80.54|83.11|\n|QwQ x4 Hogwild!|**35.74**|**58.26**|**69.52**|**76.87**|**80.66**|**83.23**|**84.33**|**84.94**|\n\n\nThe results above show that our approach remains superior to these more standard parallel inference approaches.\n\n\n> 3. It would be important to evaluate more and larger LLMs (the used models are small with limited performance), especially larger models, considering the Chain-of-Thought designs.\n\nRespectfully, we cannot agree with the statement that our experiments are not performed at sufficient scale: \nIn the experiments presented in Figure 4 and 6 (right) we evaluated Hogwild! Inference on both large and small LLMs. Our results include evaluations for DeepSeek R1 (671B params) and Qwen3-235B-A22B (235B params). These were **the most capable open-weights reasoning models at the time of the submission** (May 2025) and are comparable to GPT4.x on many of the relevant benchmarks [2,3]. We report 3 more benchmarks with Qwen3-235B-A22B in Figure 17 in Appendix E.2.\n\n> 4. It would be important to evaluate the models using more datasets (with complex problems). The model evaluation (better use more and larger models and more datasets) is limited to show the stability and reliability of the strategy.\n\nWe agree that more evaluations are important, but we respectfully disagree that our submission is lacking in that regard. We evaluated on **10 LLMs from 5 model families** (QwQ, Qwen3, DeepSeek-R1, Phi-4-reasoning-plus in the main paper, Llama in Figure 18 in appendix) in **6 evaluation setups** (GSM8Kx5, LIMO, LCB, two main setups from OlympiadBench, AIME). We would be grateful if you could suggest specific additional evaluations that we did not cover. \n\n\n\nIn addition, we conducted experiments on GPQA-Diamond, which is a highly challenging set of multiple-choice questions in biology, physics, and chemistry. Following the original evaluation protocol, we provide all admissible choices in the question and permute them in a random order to circumvent potential position bias. As observed in the table below, Hogwild! outperforms the baseline for the budgets considered, with the gap being most pronounced for smaller budgets. \n\n\n### GPQA-Diamond evaluations (numbered columns are inference budgets)\n\n| Model name  | Method  | 256    | 512    | 1024   | 2048   | 3072   | 4096   | 5120   | 6144   | 7168   | 8192   |\n|-------------|----------|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|\n| QwQ-32B     | Baseline | 0.2172 | 0.2626 | 0.3535 | 0.4646 | 0.4949 | 0.5707 | 0.5606 | 0.5606 | 0.5505 | 0.5606 |\n| QwQ-32B     | Hogwild! | 0.3939 | 0.4343 | 0.4646 | 0.5404 | 0.5455 | 0.5455 | 0.5758 | 0.5859 | 0.5707 | 0.5657 |\n|||||||||||||\n| Qwen3-8B    | Baseline | 0.2323 | 0.2525 | 0.2879 | 0.3939 | 0.4394 | 0.4747 | 0.5101 | 0.4899 | 0.5000 | 0.5101 |\n| Qwen3-8B    | Hogwild! | 0.3081 | 0.3586 | 0.4394 | 0.5253 | 0.5556 | 0.5505 | 0.5606 | 0.5505 | 0.5505 | 0.5505 |\n|||||||||||||\n| Qwen3-14B   | Baseline | 0.2828 | 0.3131 | 0.3283 | 0.4697 | 0.5404 | 0.5505 | 0.5758 | 0.6162 | 0.5960 | 0.5909 |\n| Qwen3-14B   | Hogwild! | 0.4091 | 0.4192 | 0.5000 | 0.5505 | 0.5657 | 0.5758 | 0.5758 | 0.5859 | 0.5960 | 0.5909 |\n\nIf you have suggestions regarding additional domains or tasks that could further extend our analysis, we would be glad to consider them.\n\n> Though the token generation speed is faster, how about the average token generation speed, or the total time needed (combination of a set of workers) for solving the challenging problems.\n\nWe report token generation speed in Section 4.4 (see Table 1), where Hogwild! inference with 2 and 4 workers can generate tokens nearly 2 and 4 times faster than sequential generation, respectively.\nWe also agree that it is important to compare the total time needed to solve the problem. To that end, we evaluated the wall inference time vs accuracy using the QwQ-32B AWQ model on LIMO dataset. **These results can be found in Figure 19, Appendix F.** The results suggest that the benefits of Hogwild! inference in terms of forward passes translate to faster time-to-solution.\n\nSince this analysis caught your interest, we will move it to the main body (Section 4.4) in the final version of the paper, using the additional content page allowed by this year’s conference rules.\n\nTo summarize, we believe that the main concerns raised in the review can be resolved through discussion and the additional results we reported. We respectfully ask the reviewer to reconsider their score based on the additional results. If you have further suggestions or clarifications, please inform us so we can address it in the final version of the paper.\n\n\n[1] LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code, Jain et al., 2024.\n\n[2] Qwen3 Technical Report, Qwen Team, 2025\n\n[3] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, DeepSeek-AI Team, 2025"}, {"review_id": "ebk9pXL24Z", "reviewer": "Reviewer_wvbH", "summary": "The main contributions are the three-tier cache structure, isolation of page to cache block mapping, and the approach to rotate token queries at an angle rather than all previous keys. The authors demonstrate some evidence that LLMs have the capacity to reason without post-training by collaborating without pre-specified roles or explicit cooperation strategies. The paper is very readable and a joy to read.", "questions": "My interpretation of figure 1. Is not what is described as keeping KV memories. Rather than stitch together in different orders, why not use a head node to orchestrate with fewer ops? Other routing-like approaches have shown good results by treating a router as a head node orchestrator.\nWhere in Appendix B do you discuss ways to scale further? Are you referring to the section on distributed inference? \nA more comprehensive discussion of the scalability challenges described in the limitations section would be helpful.\nDo you have logs that show the distribution of parallel threads in SoT?", "limitations": "It’s not clear that the approach addresses the limitation of alternative methods that is acknowledged: “... at least some of the agents have to solve the entire problem sequentially, and process (re-encode) each other’s progress. This creates additional computational overhead, which presents challenges for both runtime and memory efficiency Wang et al.”.\nThe stitching together still grows computation. An analysis should be presented if the authors believe it represents a net improvement. \nThe evaluation on 2-4 parallel workers does not seem sufficient to generalize results and in particular is below a threshold where communication problems of distributed systems will emerge. The foundations of the LLM-as-a-Judge paradigm are not very convincing, especially for the older GPT-4o model. A more convincing evaluation would be human-evals on this particular task. Authors acknowledge the lack of direct correlation between base accuracy and collaborativeness. Could you show that collaborativeness is not correlated to output length?", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "The prompting of LLM workers to decide next actions seems analogous to producing states and offloading actions to the LLM in an RL setup. The approach the authors take seems more difficult to generalize. The four cases of “solving parallel sub-tasks, cross-verifying each other, discussing strategy, or pivoting to a new plan” may not have much coverage. The authors also acknowledge the need to tune prompts in 4.2. An analysis of routing strategy between the alternatives that are based on voting, roles, and speed; and showing evidence of these approaches not having better compute and accuracy profiles would make the case stronger. The statement “However, when asked directly, they can spot redundancy and change their approach. Overall, we found that when prompted this way, LLMs often (but not always) detect redundancies in their actions and can determine the optimal course of action.” requires data to justify the claim. In Figure 5. The baselines (dotted lines) for the 32B models have more slope than Hogwild. Given the observation that the authors acknowledge the approach can replace correct answers with mistakes at higher token counts, the results would be more convincing if the forward passes were extended. As a positional paper, the work is novel to the best of my knowledge and significant.", "was_revised": true, "final_justification": "The authors answered all my questions by either pointing out details in the appendices or sharing additional data. \nPlease see my response to the authors:\n\nOn the RL setup, I would like to know if there are clear flaws as to why an RL setup would not work. When you prompt the LLM workers to decide their action, given the latest actions from others, it seems to have a similar problem formulation. Thank you for highlighting that self-consistency and SoT cover voting and roles, and the linkage of Figure 9 in E.1, and the examples in G.2 to support the redundancy claim. The additional examples further the point. The addition of the 16k tokens on the OlympiadBench Math is showing strong convergence between Hogwild and the baseline. What is your interpretation of this? Could you also share the same extension for Physics? There is more plateauing behavior on that benchmark in Figure 5. The 6-worker evaluations in Appendix E.1. and additional LLM-as-judge results on Claude 4 Opus that are comparable to Figure 7. are satisfying. The additional details comparing Figure 9 and Figure 7, and elaborating on collaborativeness vs. output length, strengthen the findings. The point on Sequence Parallelism is fascinating. I look forward to further discussing this after seeing it in the final version. Thank you for directing me to Appendix G for the distribution of parallel threads in SoT. I believe the additional details and context justify an elevated rating.", "rebuttal": "Thank you for your feedback and suggestions. We are glad that you appreciate our technical contributions. Below, we discuss the questions raised in the review and address suggestions in the same order as they are written.\n\n> The prompting of LLM workers to decide next actions seems analogous to producing states and offloading actions to the LLM in an RL setup.\n\nWhile this is not how we originally envisioned Hogwild! Inference, there might indeed be some parallels. If you would like to discuss this further, we can continue during the discussion phase.\n\n> An analysis of routing strategy between the alternatives that are based on voting, roles, and speed; and showing evidence of these approaches not having better compute and accuracy profiles would make the case stronger.\n\nWe did compare Hogwild! Inference against **some** of these strategies in Section 4.1, namely voting (self-consistency) and roles (Skeleton-of-Thought has a ‘router’ and ‘workers’), see Figure 3 (left, middle). We agree that further exploration of different alternatives would benefit the paper and will continue this exploration in the revised paper.\n\n> The statement “However, when asked directly, they can spot redundancy and change their approach….” requires data to justify the claim.\n\nIn our original submission, we validate this in Appendix E.1 (ablation analysis). In **Figure 9 (left), the “No collab. prompt” (yellow) line** corresponds to a version of Hogwild! Inference *where we do not ask workers to spot redundancies.* As we can see, asking the model to spot redundancies (orange line) can significantly improve its effectiveness. Examples with such behaviour could be found in the Appendix G.2.\n\nTo further illustrate this, we provide several new examples of how the model reacts to the collaboration prompt:\n\nAbbreviation: QC = “Quick check: am I doing redundant work? (yes/no)”\n\n```\n1)\nAlice [6]: QC:  Let me see Bob is also thinking of the same thing. Wait, maybe we can split tasks. For example, I can list triplets starting with smaller first digits, Bob can do higher ones.\n<..triplets found..>\nBob [20]: QC:  Let me see Alice is doing {1,3,6} and {1,4,5}? Wait, perhaps we should split the four triplets between us. Let me take the first two and she takes the other two?\n\n2)\nBob [7]: QC:  Alice is computing D's coordinates, I can compute E and F instead. Let me switch to that.\nBob [23]: QC:  Alice is working on angle AEP. I can proceed to compute angle CDP.\n3)\nBob [7]: QC:  Wait, Alice just made a good point. If we factor z³, then modulus is |z^3| times |...|. So we can focus on maximizing |1 + 2i - z²|. That's a better approach.\n4) \nAlice [19]: QC:  Bob is also trying generating functions. Maybe I can try a different approach. Let me consider fixing one variable and counting solutions.\n```\n\n> In Figure 5. The baselines (dotted lines) for the 32B models have more slope than Hogwild. <…>, the results would be more convincing if the forward passes were extended.\n\nWe agree that exploring higher token counts is important. To that end, we evaluated Hogwild! Inference with up to 16k budget on the OlympiadBench Math in the same setup as Figure 5 (left).\n\n|Method\\Budget|2048|4096|6144|8192|10240|12288|14436|16384|\n|-|-|-|-|-|-|-|-|-|\n|Hogwild!|52.0|60.89|64.15|66.52|67.41|70.81|72.89|75.26|\n|Baseline|40.89|57.0|63.11|65.33|65.93|69.78|72.3|74.81|\n\nOverall, Hogwild! Inference continues approaching the upper bound quality of the base model as the budget increases. We will include further exploration of longer budgets in the final version of the paper.\n\n> The evaluation on 2-4 parallel workers does not seem sufficient to generalize results and in particular is below a threshold where communication problems of distributed systems will emerge\n\nWe evaluate more (6) workers in Appendix E.1. With current models, we found that 2-4 workers are optimal for the tasks we evaluated. Scaling Hogwild! to use even more workers effectively is an interesting direction for future research.\n\n> The foundations of the LLM-as-a-Judge paradigm are not very convincing, *especially for the older GPT-4o model*. A more convincing evaluation would be human-evals on this particular task.\n\nWhile we agree that LLM as a judge has limitations, it is a popular evaluation strategy for many NLP benchmarks [1,2,3]. To demonstrate that our results are not specific to GPT-4o, we additionally evaluate a newer Claude 4 Opus model from May 22, 2025. We use the same setup as in Figure 7 and compare the three communication types: no communication (voting at the end), synchronizing every paragraph, or every token.\n\n|Method\\Model|Qwen3-8B|QwQ-32B|\n|-|-|-|\n|No-sync|1.17|1.24|\n|Step-wise sync|2.08|2.3|\n|Token-wise sync|2.65|3.08|\n\nThat said, we agree that human evaluation would further improve the credibility of our analysis and will include the crowdsourced evaluation in the final paper.\n\n\n> Could you show that collaborativeness is not correlated to output length?\n\nIn our preliminary experiments, we found that collaborativeness **is** correlated to output length. For instance, in Figure 9 (left), the **“Hogwild! (non-instant)”** line is a version of Hogwild! with less collaborativeness (see Figure 7 ‘Step-wise’). This version solves LIMO tasks systematically slower than full Hogwild! Inference. To control for differences in length, Figure 7 compares only generations with the same output length (using a 4096-token prefix from each worker). We will add these details to the paper.\n\n> Where in Appendix B do you discuss ways to scale further? Are you referring to the section on distributed inference?\n\nWe briefly discuss scaling Hogwild! Inference with Tensor, Pipeline, and Sequence Parallelism in L1052-1058. However, we agree that our work would benefit from further discussion of inference scalability. Using Tensor and/or Pipeline parallelism with our approach is no harder than parallelizing traditional batch inference. This is because Hogwild! Inference works independently between attention heads in each layer, so each device can be assigned to process a subset of attention heads and the corresponding KV cache slices.\n\nA more interesting case study for Hogwild! is Sequence Parallelism, where **each device holds a subset of tokens**. This would allow us to reduce cross-device communication, essentially assigning each GPU / node to a subset of workers (e.g., Alice, Bob, etc). The main caveat with this approach is that Hogwild! workers need to attend to each other’s tokens on every inference step. \n\nIn distributed inference, this pairs nicely with Sequence Parallelism using Ring Attention [4] or similar. When computing Hogwild! concurrent attention, workers can exchange rotated queries using the All-to-All collective operation (Scatter/Gather) available in most frameworks [5]. After that, each worker computes dot-products between the rotated queries and its local KV cache, and exchanges the partial results as in [4]. Note also that this type of parallelism solves load balancing out of the box since workers generate roughly the same number of tokens. We will further discuss and compare different distributed inference strategies in the final version of the paper.\n\n\n> A more comprehensive discussion of the scalability challenges described in the limitations section would be helpful. \n\nNote that the discussion above only refers to scalability in terms of compute and model size. There is an orthogonal (and equally important) direction of scaling Hogwild! Inference to more workers while maintaining efficient collaboration (e.g., see Fig.9, right in Appendix E.1). This could possibly be achieved with more sophisticated memory layouts or fine-tuning the base LLM to collaborate better with many workers. Both are promising directions for future work. \n\n> Do you have logs that show the distribution of parallel threads in SoT?\n\nWe do and we will include these in these examples in appendix G. For GSM8Kx5 (Fig 3, left – 5 independent questions), the SoT ‘router’ works as expected and almost always assigns each thread to a different task, which they subsequently solve.\nAs for the more complicated LIMO benchmark, what usually happens is that the initial plan helps to some extent, but fails to account for future developments. Here’s one example:\n\n```\n# Problem (LIMO task 14)\nThree vertices of a cube are $P=(7,12,10)$ , $Q=(8,8,1)$ , and $R=(11,3,9)$ . What is the surface area of the cube? \n\n# Outline (produced by QwQ-32B as SoT router)\n1. Find the distance between vertices P and Q.\n2. Find the distance between vertices Q and R.\n3. Find the distance between vertices P and R.\n4. Compare the distances to determine the side length of the cube.\n5. Calculate the surface area using the side length.\n```\n\nAs you can see, the plan allows for some degree of parallelism. However, the three distances in question are *not* sides of the cube, but its diagonals, and the model learns that as it computes the original angles, but at that point, the outline is already decided. In contrast, workers in Hogwild! Inference does not have a fixed plan and can adjust once they do the initial calculations (e.g., Appendix G.2).\n\nIf you have any further questions and suggestions, we are happy to address them during the discussion phase!\n\n \n[1] Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena MTBench\n\n[2] Zhou et al., Instruction-Following Evaluation for Large Language Models\n\n[3] Li et al., From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline\n\n[4] Liu et al., Ring Attention with Blockwise Transformers for Near-Infinite Context\n\n[5] Li et al., PyTorch Distributed: Experiences on Accelerating Data Parallel Training"}, {"review_id": "BobrHWWR7d", "reviewer": "Reviewer_HjWC", "summary": "This paper introduces Hogwild! Inference, a method that allows an LLM to explore multiple execution traces for the same problem while enabling synchronization between them. The traces share the attention cache, and the properties of RoPE are exploited to further enhance efficiency.", "questions": "How does this method compare with long-CoT-based reasoning? When would be a better scenario to use Hogwild! Inference?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The idea is very interesting, and the algorithm is supported by both solid math foundations and system design.\n\n2. The authors conduct experiments using large-scale (32B) models such as QwQ, covering a diverse set of tasks including GSM8K, LIMO, LiveCodeBench, and OlympiadBench.\n\n3. They provide illustrative examples, such as the expected collaboration patterns, to help validate their hypothesis.\n\nWeaknesses:\n\n1. The approach is currently for evaluation only. It would be interesting to explore fine-tuning the model toward the desired behavior to evaluate whether further improvements can be achieved.", "was_revised": true, "final_justification": "This paper seems to be interesting and solid.", "rebuttal": "Thank you for your review and positive feedback! We are glad that you appreciate both the idea and experiments & examples. Below, we discuss each of the suggestions raised in the review.\n\n> It would be interesting to explore fine-tuning the model toward the desired behavior to evaluate whether further improvements can be achieved.\n\n\nWe note that concurrent and follow-up works already explore various SFT and reinforcement learning strategies for other parallel inference setups [1,2,3,4,5], demonstrating the improvements of the model’s ability to use parallel threads in different setups. For example, [1, 2] describe the SFT + RL pipeline and [4] introduce an SFT dataset for parallel generation. We cite concurrent works in Section 2 and will add a brief summary of follow-up works in the final version of the paper.\n\nWhile our training-free setup makes Hogwild! inference easier to use, we agree that fine-tuning is an interesting direction for further improvement. \n\nIn Appendix B (see supplementary materials, L1032-1045), we discuss how the interleaved cache layout for Hogwild! inference can be computed in parallel during fine-tuning. We plan to investigate methods for improving collaboration between threads, such as fine-tuning and reinforcement learning, in future work.\n\n> How does this method compare with long-CoT-based reasoning? When would be a better scenario to use Hogwild! Inference?\n\nTo the best of our understanding, long CoT refers to increasing the generation budget for chain-of-thought to more tokens (e.g., [5,6,7]). Please tell us if you meant something different.\n\nOverall, we believe that Hogwild! Inference with K agents (e.g., 2 or 4) with L step budget can match long CoT inference with *at most K * L budget*, depending on the task. That said, the exact benefit from collaboration does indeed depend on the task. The easiest tasks to parallelize are the ones that have natural independent sub-tasks e.g. see Figure 3 (left) and Section 4.1. More complex tasks such as math/coding problems are harder to parallelize, but still offer significant advantage in terms of faster solutions or greater accuracy given a fixed budget (Section 4.2, figures 11-17 in Appendix).\n\nOn the other end of the spectrum, we believe that Hogwild! inference will offer no speed advantage over long CoT in **tasks that are solved with one obvious non-parallelizable algorithm,** such as computing a VDF [8] or similar practical tasks. Furthermore, tasks that require very little chain-of-thought ”effort” (e.g., recalling a single trivia fact) would likely offer no advantage from collaboration.\n\nNote that, for mathematical and programming tasks we evaluate on, long CoT will achieve the same accuracy given greater time (e.g., see “upper bound” in Fig.3-6 for sequential chain of thought with abundant budget). However, Hogwild! Inference offers better time-to-solution due to efficient parallel inference (see Table 1). To better quantify this, we report time-to-solution using the efficient implementation in Figure 19 in Appendix F. We benchmark both Hogwild! and sequential inference on a single NVIDIA L40S GPU in terms of wall clock time and accuracy for a more direct efficiency comparison.\n\nWe are be happy to discuss these further and answer any additional questions during the discussion period if needed.\n\n\n[1] in et al., Learning to Keep a Promise: Scaling Language Model Decoding Parallelism with Learned Asynchronous Decoding, 2025\n\n[2] Pan et al., Learning Adaptive Parallel Reasoning with Language Models, 2025\n\n[3] Hsu et al., Group Think: Multiple Concurrent Reasoning Agents Collaborating at Token Level Granularity, 2025\n\n[4] Yang et al., Multiverse: Your Language Models Secretly Decide How to Parallelize and Merge Generation, 2025\n\n[5] Biju et al., SPRINT: Enabling Interleaved Planning and Parallelized Execution in Reasoning Models, 2025\n\n[5] Yeo et al., Demystifying Long Chain-of-Thought Reasoning in LLMs, 2025\n\n[6] Luo et al., Deconstructing Long Chain-of-Thought: A Structured Reasoning Optimization Framework for Long CoT Distillation, 2025\n\n[7] Tang et al., Unlocking General Long Chain-of-Thought Reasoning Capabilities of Large Language Models via Representation Engineering, 2025\n\n[8] Boneh et al., Verifiable Delay Functions, 2018"}, {"review_id": "VYaBgCLM4Y", "reviewer": "Reviewer_Qffi", "summary": "The paper introduces “Hogwild! Inference”, a novel parallel inference protocol enabling multiple LLM instances to collaborate dynamically using a shared, concurrently-updated Key-Value (KV) attention cache. This design leverages Rotary Position Embeddings (RoPE) to enable efficient real-time collaboration without predefined strategies. Through extensive empirical evaluation on various reasoning benchmarks (e.g., LIMO, OlympiadBench, LiveCodeBench, AIME’25), the authors demonstrate significant performance improvements, highlighting faster inference and enhanced reasoning accuracy when compared to existing parallel inference strategies.", "questions": "Please see the weakness section", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Disclaimer:** I am not an expert in this field, but I tried my best to evaluate the work.\n\n**Strengths**\n \n**Novel and Effective Parallelization Strategy** \nThe approach of dynamically sharing and updating KV caches among parallel inference threads is innovative and practical. It leverages intrinsic LLM reasoning capabilities without additional fine-tuning, representing an impactful advancement in LLM inference techniques.\n\n**Robust Empirical Results:** \n\nEmpirical validation is thorough, demonstrating substantial accuracy and inference-time improvements across diverse and challenging benchmarks. Results convincingly illustrate the efficacy and practical potential of the proposed method.\n\n**Clear, Detailed, and High-Quality Presentation:** \n\nI find the paper to be well-written in general.\n\n**Practical Utility and Efficiency Gains:** \n\nThe paper includes explicit and detailed evaluations of practical inference-time performance gains, highlighting near-linear scalability in parallel token generation, which strongly supports real-world applicability.\n\n**---XXX---**\n\n**Weaknesses** \n\n**Limited Theoretical Justification for Collaboration Dynamics:** \n\nAlthough empirically successful, the paper lacks theoretical grounding or deeper insights into why allowing LLM instances to synchronize dynamically via shared caches universally improves reasoning.\n\n**Potential Scalability and Generalization Limitations:**\n\nThe method's effectiveness seems dependent on task-specific factors such as task complexity and the intrinsic capability of the LLMs to \"self-coordinate.\" It remains unclear how well the approach generalizes to significantly larger models or vastly different reasoning domains.\n\n**Dependence on LLM’s Intrinsic Collaborative Ability:** \n\nThe paper implicitly assumes that LLMs possess sufficient \"intrinsic reasoning and collaborative instincts\" without clearly specifying or demonstrating the limitations or conditions under which this assumption might fail.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive review and positive feedback! Let us address the questions below.\n\n> Limited theoretical Jusitification of Collaboration Dynamics\n\nRigorous theoretical grounding is challenging to achieve, given that modern LLMs are complex, nonlinear systems with sophisticated behavior. We believe the main argument behind the efficacy of Hogwild! Inference is analogous to the rationale for test-time scaling: accuracy on reasoning tasks improves with the number of generated tokens. Hogwild! Inference produces two tokens (or as many as the number of workers) in a single forward pass, yielding a longer chain of thought within the same wall-clock time. The proposed RoPE rotation strategy further enables workers to benefit from each other’s outputs.\n\n> Potential Scalability and Generalization limitations\n\nAs shown in Figure 4 (right), the effectiveness of the method depends on the base model’s scale and quality. Smaller models from the same family, such as Qwen3-1.7B, significantly lag behind the single-thread baseline for a fixed number of forward passes, whereas the largest model evaluated (Qwen3-235B-A22B) shows a marked improvement.\nWe note that we have validated the efficacy of the proposed method on relatively large models and across various tasks. Specifically, Figure 5 (right) demonstrates experiments with Qwen3-235B-A22B (235B parameters) and DeepSeek-R1 (671B parameters)—the latter being among the largest open-source LLMs with strong performance at the time of submission. Our evaluation setup includes tasks from diverse reasoning domains:\n\n- Math (GSM8k, OlympiadBench-Math, AIME25)\n- Physics (OlympiadBench-Physics)\n- Code (LiveCodeBench)\n\nIn addition, we conducted experiments on GPQA-Diamond, which is a highly challenging set of multiple-choice questions in biology, physics, and chemistry. Following the original evaluation protocol, we provide all admissible choices in the question and permute them in a random order to circumvent potential position bias. As observed in the table below, Hogwild! outperforms the baseline for the budgets considered, with the gap being most pronounced for smaller budgets. \n \n### GPQA-Diamond evaluations (numbered columns are inference budgets)\n\n| Model name  | Method  | 256    | 512    | 1024   | 2048   | 3072   | 4096   | 5120   | 6144   | 7168   | 8192   |\n|-------------|----------|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|\n| QwQ-32B     | Baseline | 0.2172 | 0.2626 | 0.3535 | 0.4646 | 0.4949 | 0.5707 | 0.5606 | 0.5606 | 0.5505 | 0.5606 |\n| QwQ-32B     | Hogwild! | 0.3939 | 0.4343 | 0.4646 | 0.5404 | 0.5455 | 0.5455 | 0.5758 | 0.5859 | 0.5707 | 0.5657 |\n|||||||||||||\n| Qwen3-8B    | Baseline | 0.2323 | 0.2525 | 0.2879 | 0.3939 | 0.4394 | 0.4747 | 0.5101 | 0.4899 | 0.5000 | 0.5101 |\n| Qwen3-8B    | Hogwild! | 0.3081 | 0.3586 | 0.4394 | 0.5253 | 0.5556 | 0.5505 | 0.5606 | 0.5505 | 0.5505 | 0.5505 |\n|||||||||||||\n| Qwen3-14B   | Baseline | 0.2828 | 0.3131 | 0.3283 | 0.4697 | 0.5404 | 0.5505 | 0.5758 | 0.6162 | 0.5960 | 0.5909 |\n| Qwen3-14B   | Hogwild! | 0.4091 | 0.4192 | 0.5000 | 0.5505 | 0.5657 | 0.5758 | 0.5758 | 0.5859 | 0.5960 | 0.5909 |\n\n\nIf you have suggestions regarding additional domains or tasks that could further extend our analysis, we would be glad to consider them.\n\n> Dependence on LLM’s Intrinsic Collaborative Abilitiy\n\nWe note that empirical results indicate that the success of multi-threaded task-solving is closely correlated with the model’s capabilities. For small models (e.g., Qwen3 in the 1.7B–8B parameter range, see Figure 4 (right)), Hogwild! does not perform as well for larger budgets. In turn, the benefits become more pronounced for more powerful models that are inherently better capable of collaboration.\n\nWe believe that these inherent capabilities are not a fluke, but a direct result of having similar traces in the data the model was trained on.\nAs a result, we expect that the reasoning and collaborative abilities of LLMs will continue to improve with further advances in the field.\n\n\nIf you have any other questions or concerns, we are happy to discuss them further."}, {"review_id": "Rv6H8BdUq3", "reviewer": "Reviewer_JQbb", "summary": "The paper studies a method that has multiple LLM instances to collaboratively generate answers. Instead of pre-defining a framework to decompose a task into sub-tasks and having multiple LLM instances to solve sub-tasks in parallel, the proposed method has multiple LLM instances to figure out the collaboration strategies themselves. The instances collaborate through KV cache. By having different LLM instances to view the same KV cache and update KV cache, they can see each other's progress, potentially avoid redundant works and generate better results.", "questions": "What is the cost of this method (how many more tokens are generated?)\nIn figure 4, 5, 6, when 2 workers are used, does the method use twice as more tokens as the baselines?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* Even though the basic idea seems simple, the paper couples multiple interesting ideas to make it work. The ideas are interesting and quite novel.\n* The method has been evaluated on multiple reasoning benchmarks (ranging from easy benchmarks and challenging benchmarks) and has been shown good performance improvement on these benchmarks.\n\nWeaknesses:\n* The presentation can be improved. For example, fonts and lines in the figures are too small. What is the forward pass?\n* The evaluation may not be completely fair. If a forward pass means a token generated by an LLM instance,  Hogwild! Inference with 2 workers should generate twice as many tokens as baseline methods.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review and positive feedback! We address the questions below.\n\n > The presentation can be improved. For example, fonts and lines in the figures are too small.\n\n\nWe appreciate the feedback and will improve the presentation clarity, including the figure sizes, in the final version. The additional page allowance post-acceptance will be very helpful for this. Please let us know if you have additional suggestions on improving the presentation.\n\n> (continued) What is the forward pass? \n\nA \"forward pass\" refers to one invocation of the model to generate the next token or several tokens if inputs are batched. Note that parallel methods, including Hogwild! Inference, Skeleton-of-Thought, Self-Consistency, use batching. We will clarify that in the paper.\n\n> The evaluation may not be completely fair. If a forward pass means a token generated by an LLM instance, Hogwild! Inference with 2 workers should generate twice as many tokens as baseline methods.\n\nYou correctly note that token count and forward pass count are not equivalent in Hogwild! Inference: Hogwild! with 2 workers generates approximately twice as many tokens as the baseline for the same number of forward passes. Intuitively, computationally speaking, we get the second token for free! \n\nWhile a baseline forward pass might be computationally lighter, the actual time-to-solution is highly hardware-dependent and influenced by kernel implementation specifics. Therefore, using forward pass count provides a more hardware-neutral metric for computational effort. \n\nWe report token generation speed in Section 4.4 (see Table 1), where Hogwild! inference with 2 and 4 workers can generate tokens nearly 2 and 4 times faster than sequential generation, respectively.\n\nWe also agree that it is important to compare the total time needed to solve the problem. To that end, we evaluated the wall inference time vs accuracy using the QwQ-32B AWQ model on the LIMO dataset. **These results can be found in Figure 19, Appendix F.** The results suggest that the benefits of Hogwild! Inference in terms of forward passes translate to faster time-to-solution.\n\nWe will move this analysis to the main body (Section 4.4) in the final version of the paper, using the additional content page allowed by this year’s conference rules.\n\nThank you again for your feedback! If you have any further suggestions or clarification, we are happy to continue during the Reviewer-Author Discussion phase."}]

Author rebuttal:

Meta-review: Hogwild! Inference presents a novel shared KV-cache framework enabling multiple LLM workers to collaborate concurrently without fine-tuning, yielding consistent accuracy gains and reduced latency across reasoning and QA tasks. The approach is simple, general, and well-engineered, integrating cleanly with existing inference pipelines while offering a fresh perspective on collaborative decoding. Experiments span diverse tasks, and the method demonstrates clear benefits over sequential baselines under realistic budgets, with minimal overhead. While broader evaluation and deeper theoretical framing would strengthen the work, the contribution is timely, practical, and of interest to the scalable inference community. I recommend accept.
Decision: Accept (spotlight)