Plato: Plan to Efficient Decode for Large Language Model Inference
Authors: Shuowei Jin
Xueshen Liu
Yongji Wu
Haizhong Zheng
Qingzhao Zhang
Atul Prakash
Matthew Lentz
Danyang Zhuo
Feng Qian
Zhuoqing Mao
Venue: COLM 2025 (2025)
Keywords: Efficient LLM Inference
TL;DR: Plan to exploit parallelism structure to break the autoregressive nature of LLM inference.

Abstract: Large language models (LLMs) have achieved remarkable success in natural language tasks, but their inference incurs substantial computational and memory overhead.
To improve efficiency, parallel decoding methods like Skeleton-of-Thought (SoT) decompose prompts into sub-problems for concurrent processing. However, these methods significantly compromise answer quality by treating semantically linked sub-problems as independent.
We propose Plato, a novel approach that co-designs algorithms and systems for semantic-aware parallel decoding. Plato leverages LLMs to organize sub-problems into a dependency graph based on logical and causal relationships, enabling concurrent decoding of non-dependent nodes while preserving answer coherence and quality.
To further enhance efficiency, Plato pipelines planning and node decoding stages, implements a global context cache, and carefully structures node inference prompts to maximize key-value cache reuse and minimize overhead. Our evaluations show that Plato improves throughput by up to 68% over autoregressive decoding while achieving a 40% net win rate in answer quality. Compared to SoT, Plato demonstrates a remarkable 90% quality net-win rate. Ablation studies reveal that our pipeline design improves speedup by 29%, while our KV cache reuse optimization reduces overhead by 75%.

Published as a conference paper at COLM 2025
Plato: Plan to Efficiently Decode for Large Language Model Inference
Shuowei Jin'; Xueshen Liu Yongji Wu?t Haizhong Zheng Qingzhao Zhang',
Atul Prakash", Matthew Lentz", Danyang Zhuo', Feng Qian Z. Morley Mao
" of Michigan UUniversity California, Berkeley
CCrrnegie Mellon University *Duke University
SUniversity of Southern California
{jinsw, liuxs, qzzhang, aprakash, zmao @amicheeemccheeee. yongji wu@berkeley. edu zhozheazendr cmu.edu {mlentz, danyang duke.edu fengqian@usc.ed edu
Abstract
Large language models (LLMs) have achieved remarkable success in nat- ural language tasks, but their inference incurs substantial computational and memory overhead. To improve efficiency, parallel decoding methods like Skeleton-of- Thought (SoT) decompose prompts into sub-proble ceob-probless for concurrent processing. However, these methods significantly compromise answer quality by treating semantically linked sub-problems as indepen- dent. We propose Plato, a novel approach that co-designs algorithms and systems for semantic-aware parallel decoding. Plato leverages LLMs to ganize sub-problems into a dependency graph based on logical and causal relationships, enabling concurrent decoding of non-dependent nodes while preserving answer coherence and quality. To further enhance efficiency, Plato pipelines planning and node decoding stages, implements a global context cache, and carefully structures node inference prompts to maximize key-value cache reuse and minimize overhead. Our evaluations show that Plato improves throughput by up to 68% over autoregressive decoding while achieving 40% net win rate in answer quality. Compared to SoT, Plato demonstrates remarkable 90% quality net-win rate. Ablation studies reveal that our pipeline design improves speedup by 29%, while our KV cache reuse optimization reduces overhead by 75%.
1 Introduction
Large language models (LLMs) (Guo et al., 2025; Grattafiori et al., 2024; Jaech et al., 2024) have demonstrated remarkable performance across wide range of tasks (Nam et al., 2024; Cascella al., 2023), becoming increasingly important in daily life. A key contributor to the success of LLMs is the transformer architecture, which efficiently models relationships among a large number of context tokens and generates responses token-by-token in an autoregressive manner. However, this autoregressive decoding process inherently limits generation throughput and increases latency, ultimately affecting the user experience (Wang et al., 2024).
Various approaches (Leviathan et al., 2023; Chen et al., 2023; Cai et al., 2024; Fu et al., 2024) have been proposed to improve generation throughput through parallel sampling which generates multiple consecutive tokens at the same time. However, research exploring an orthogonal approach--xppoithi parallelization opportunities directly within the semantic
"Equal contribution.
t "Corresponding author: Yongji Wu. Work done while the University of Michigan.

---

Published as a conference paper at COLM 2025
ppace-remains limited. When humans compose responses to complex questions, they typically begin by creating an outline that defines the answer structure, then develop the full response following this framework (Hadi et al., 2024). This approach enhances logical flow and ensures comprehensive coverage of the topic. Such outlines naturally decompose autoregressive generation tasks into distinct sub-problems in the semantic space, creating opportunities to increase efficiency by generating independent components in parallel.
A sta state-of-the-art contribution in this direction is Skeleton of Thought (SoT) (Ning et al., 2023), which uses prompting techniques to decompose the original problem into indepen- dent sub-problems (i.e., the skeleton) and then inferences all sub-problems parallel. While SoT improves generation efficiency, it often sacrifices response quality because its inde- pendence assumption frequently fails in practice. Dependencies commonly exist between sub-problems, particularly when addressing questions requiring strong logical reasoning in domains such as mathematics and programming. For instance, as illustrated in Figure 1, SoT produces five cub--probees where only the first two are truly independent, while the others have dependencies on previous node results to correctly reason through the answer. When these interdependent nodes are inferenced independently, in SoT's approach, the coherency and correctness of the final answer are compromised.
Contribution. This paper presents Plato (Plan to Efficiently Decode), an inference framework that leverages effective paral- lelism in the semantic space to enhance in- ference efficiency. Unlike previous methods, Plato first prompts the LLM to create struc- tured plan that decomposes the original problem into nodes (sub-problems) while explicitly identifying the dependencies be- tween these nodes. Subsequently, Plato constructs a Directed Acyclic Graph (DAG) based on these dependencies and exploits the parallelism within the graph structure to perform batched inference. Nodes with- out unfinished dependencies are submitted to inference asynchronously, enabling par- allel execution while maintaining logical co- herence. This design ensures high-quality answers while significantly boosting effi- ciency.
If the endpoints of line segment are (2. -2) and (10, 4), what is the lengtn of the segment?
Skeleton Thought
Plato
delta
Find deltt y- - (-21) 8. Square each Yos should calculata (daatia xy22 and (delta y/2 Add squared results->> Taking the sum idelts x)"2 and (dalta y?2 which are the is from 3.0 Taking the square the of squared dallo daita giva longth the line segment.o
1. Calculate x2 -x1-> 10-2-0,
2. Calculata y2 y2- yt- 44(2)=6.v
3. Calculate (x2-x1)"2-
>012-64. +
4. Calculate -y2-1
>6P2=36, <
Calculate (1)-x1) ty2
21) 100. of
Calculate root of sum->*The square
Results Format: (Nade Node Description-> Inference results.)
Additionally, we co-designed the inference system to optimize for the Plato workflow. Our system pipelines the planning and node inference stages, significantly reduc- ing latency by ensuring planning doesn't block node inference. Nodes are immediately added to a waiting queue as they are parsed from the planning inference stream. For the nodes in the waiting queue, once all their dependencies are satisfied, they are submitted for inference with results from previously completed nodes as context. To further enhance efficiency, we implement global context cache and strategically design node inference prompts to maximize key-value cache reuse across requests, substantially reducing computational overhead and improving throughput.
Figure 1: An example to demonstrate the dif- ference between SoT and Plato.
We conducted extensive experiments on nearly 300 questions across 28 skill types to eval- uate Plato performance compared with SoT and Autoregressive (AR) inference baselines. Our results show that Plato achieves the optimal balance between speedup and quality, Compared AR baselines, Plato improves throughput by up 68% while achieving a 40% net win rate in answer quality. When compared to SoT, Plato demonstrates remarkable 90% quality net-win rate. Through detailed ablation studies, we show that our pipeline design improves speedup by 29%, while our KV cache reuse optimization reduces overhead by 75,, resulting in just 1.33% total overhead.
2

---

Published as a conference paper at COLM 2025
In summary, we propose Plato, novel inference framework that efficiently exploits paral- lelism within the semantic space to enhance LLM inference efficiency without compromising response quality. Our approach is complementary to existing multi-token consecutive par- allel decoding methods (Leviathan et al., 2023; Chen et al., 2023; Cai et al., 2024; Fu et al., 2024), offering orthogonal optimization dimension. Through the co-design of algorith- mic and system components, Plato effectively maximizes throughput while minimizing computational overhead, advancing the state-of-the-art in LLM inference systems.
2 Related Work
2.1 Efficient LLM Inference Algorithm
Consecutive multi-token parallel decoding. Various sampling algorithms have been pro- posed to enhance generation throughput by parallel sampling multiple consecutive tokens simultaneously, breaking the token-by-token autoregressive generation nature of LLMs. Speculative Decoding algorithms (Leviathan et al., 2023; Chen et al., 2023; Li et al., 2024) employ a smaller model to generate a draft of multiple tokens, which are then verified in parallel by a larger target model, effectively improving the throughput of the sampling process. Similarly, multi-token parallel decoding methods (Cai et al., 2024) decode multi- ple consecutive tokens simultaneously by fine-tuning the base LLM to generate multiple prediction heads. Lookahead decoding methods (Fu et al., 2024) concurrently extract and verify n-gram tokens in parallel. These approaches collectively improve LLM decoding throughput by generating multiple consecutive tokens at once.
Parallelism opportunities in semantic space. Research exploring parallel opportunities directly within the semantic space to enhance inference efficiency remains limited. A notable work in this direction is Skeleton of Thought (SoT) (Ning et al., 2023), which improves LLM generation efficiency by decomposing the original problem into several independent sub- problem nodes that can be decoded in parallel as batch. However, SoT does not account for potential dependencies among these sub-problem nodes, leading to quality degradation in complex reasoning tasks. Our work further pushes the boundary of this direction by considering necessary dependencies across semantic space, thereby efficiently utilizing semantic parallelism during decoding.
2.2 Prompting for Higher Quality
To better exploit the reasoning ability of LLMs, many methods have been proposed to construct more informative and helpful prompts for LLMs to solve complex problems. Chaun-of-Thought (CoT) (Wei et al., 2022; Wang al., 2023) introduces intermediate steps between inputs and outputs as demonstrations in prompts, improving generation quality on complex tasks. Tree-of-Thought (ToT) (Yao et al., 2023) advances LLMs' reasoning capabilities by representing the reasoning process as a tree, with nodes representing different solutions. Graph-of-Thought (GoT) (Besta et al., 2023) extends ToT structure by considering graph-based approach, introducing aggregation operations on different sub-solutions to further improve reasoning. These methods focus on improving response quality for complex prompts, but ignore generation efficiency; in contrast, work focus improving the generation efficiency while not sacrificing quality by only parallelizing independent subtasks by only parallelizing independent subtasks.
3 Background
3.1
Attention and KV cache
The attention mechanism (Vaswani et al., 2017) is fundamental component of LLMs that helps models understand relationships between tokens in a sequence. KV cache is an optimization technique that makes LLM inference faster.
The attention mechanism transforms input sequences into query (Q), key (K), and value (V) representations through learned weights: Q XWo, K = XW*, V XWy. It then
3

---

Published as a conference paper at COLM 2025
Planning
Global Context
Node 1
Node 2
Node
Node Dependency
Cache
Update
Enqueue
<N1 Descr: <N1 Output>
Waiting Queue
Queuing
<N2 Desco: <N2 Output>
Dequeue
<NG Desco:
Node Inference
Node
Infer
Node Infer
Reuse
Node Infer
30
Inference Timeline
Figure 2: Plato begins with planning phase where the LLM decomposes the original question into nodes (sub-problems) with their logical dependencies. As each node is generated, it enters waiting queue. Nodes become eligible for inference when all their dependencies have been satisfied. For example, in the figure, Node and Node have no dependencies, they immediately launched for inference upon generation. Node however, depends on both Node and Node 2, so must remain in the waiting queue until both dependency nodes complete their inference. This dependency-aware scheduling ensures generation quality while maximizing parallel execution opportunities.
OKT Váp
computes attention scores as Attention(Q.K, = softmax
V, where dy is a scaling
factor for stability.
Without a KV cache, models inefficiently recalculate keys and values for all previous tokens at each generation step. The KV cache optimization stores these computed values, requiring only calculation of the query for the new token, computing its key and value, and updating the cache. This significantly reduces redundant computations during inference.
Additionally, current state-of-the-art LLM inference engines and services (Zheng et al., 2023b; Kwon al., 2023; OpenAI, 2024b; Anthropic, 2024; Deepseek, 2024) have implemented prompt caching mechanisms. These systems store previously computed KV caches from prior conversations in GPU, CPU memory, or disk (Jin et al., 2024). When subsequent requests contain identical prompt prefixes, the system can reuse these cached KV values, eliminating redundant computation and reducing latency. In Plato design, we leverage this KV cache reuse opportunity to improve overall throughput.
3.2 Prefill and Decode Stages in LLM Inference
Current state-of-the-art inference systems (Kwon et al., 2023; Zheng et al., 2023b) divide LLM inference into two distinct stages: the prefill stage and the decode stage. The prefill stage processes the entire input prompt at once, computing and storing the KV cache for all prompt tokens in parallel. This highly parallelizable computation efficiently utilizes GPU resources, though it becomes computation- decode stage generates tokens autoregressively, incrementally updating the KV cache with each new token. This process primarily memory-bound as it requires repeatedly loading the KV cache and model weights into GPU memory for each token generation, preventing full utilization of the GPU's computational capacity.
with very long sequences. In contrast, the
To improve efficiency, systems can batch multiple independent requests together, allow- ing shared model weights to reduce per-request memory I/O load and enhance inference throughput. The design of Plato exploits these characteristics by identifying and decoding multiple parallelizable sub-problems in a batch, significantly improving throughput com- pared to traditional autoregressive approaches while maintaining logical coherence between dependent components.

---

approach Published as a conference paper at COLM 2025
4
Design of Plato
Plato co-designs its algorithm and inference system maximize efficiency. As illustrated in Fig. 2, Plato employs planning step that exploits parallel opportunities in the semantic space by decomposing the original question into subquestions together with their depen- dencies on the previous nodes. It then builds dependency graph these subquestion nodes and performs parallel batched inference based on this graph structure. To further enhance performance, we design specialized inference system that improves throughput by overlapping the planning and node inference phases. Additionally, to reduce prefill over- head during node inference, design global context cache together with cache-aware prompting technique that maximizes the utilization of previously generated KV cache.
4.1 Core Stages of Plato
Plato operates through two primary stages: a planning stage that decomposes the original question into sub-question nodes and constructs a dependency graph, followed by node inference stage that exploits the graph's parallel structure to process multiple independent nodes concurrently.
In the planning stage, Plato decomposes complex answering tasks into multiple nodes (subtasks) while preserving their logical dependencies. Prior work (Ning al., 2023), which treats all subtasks as independent, often fails to capture the intricate relationships between reasoning steps. This can lead to inconsistent or incorrect answers, illustrated in Fig. 1. To address this challenge, Plato models subtask relationships as a Directed Acyclic Graph (DAG). During the planning stage, the LLM generates both the content and corresponding dependencies for each subtask node. This dependency ensures that information flows correctly through the reasoning process.
In the node inference phase, Plato improves throughput while ensuring quality. Plato enforces logical dependencies by processing nodes in the proper order. A node launched for inference only after all its dependencies have been satisfied. For instance, if node B depends on node A, Plato first completes node A, then incorporates A's results into B's context before performing inference on B. This structured approach maintains consistency, significantly improving answer quality compared to when all nodes are treated as indepen- dent. Additionally, within this graph structure, nodes whose dependencies are sa satisfied will be launched for inference asynchronously in parallel to improve the system's 's overall throughput.
4.2 Inference System Optimization
Beyond leveraging the inherent parallelism in the graph structure, Plato implements spe- cialized system-level optimizations that exploit the unique characteristics of its workflow to significantly enhance efficiency.
4.2.1 Pipelining Planning and Node Inference
One key insight in Plato is that we can improve throughput and reduce end-to-end latency by pipelining the planning and node inference stages rather than executing them sequentially. Traditional approaches like SoT execute these stages strict sequence, where node inference only begins after the planning stage is completely finished. This sequential execution creates substantial bottleneck, as the planning stage introduces non-negligible latency and negatively impacts the system's 's overall throughput.
To eliminate this bottleneck, we implement a pipelining architecture that overlaps planning with node inference. This is possible due to the autoregressive nature of node generation: earlier nodes cannot depend on later nodes (which haven't been generated yet), while later nodes may depend earlier ones. This unidirectional dependency pattern allows us to begin processing early nodes without waiting for the complete plan.
Our implementation uses real-time token streaming during the planning stage. As tokens are generated, Plato continuously analyzes the output buffer, identifying completed nodes that match our structured format:
5

---

Published as a conference paper at COLM 2025
Node ID. Node Content. [Dependency List]
As soon as complete node is detected in the buffer, it is immediately extracted and dis- waiting for the entire planning phase to complete. Nodes in the waiting queue whose dependencies have already been satisfied are immedi- ately dequeued and launched for inference, creating a pipeline where planning and node inference overlap to hide latency.
patched to the waiting
This pipelined execution design transforms a sequential process into a concurrent one, where early nodes in the dependency graph can begin inference while later nodes are still being generated. This results in a substantial reduction in overall latency and more efficient utilization of computational resources throughout the inference process.
4.2.2 KV Cache Reuse Optimization
During the node inference stage, each node requires a comprehensive prompt containing the original question and all relevant context information. However, the prefill operations needed to generate KV cache for these lengthy prompts introduce non-negligible compu- tational overhead. To address this challenge, we carefully design the prompt structure combined with a global context cache module. This approach maximizes KV cache reuse across different nodes while minimizing redundant prefill computations by leveraging the prefix caching capabilities of modern LLM inference engines (as described in $ 3.1).
We carefully structure the prompt to facilitate efficient KV cache reuse across the node inference requests. It includes four critical components, arranged to maximize caching benefits: (1) the original question, (2) instructions, (3) the global context cache containing all previously generated context, and (4) the specific subtask the current node addresses. By placing the shared contents (1,2,3) at the prefix of the prompt, we ensure the KV cache corresponding to these elements are cached and reused actoss nodes, significantly reducing redundant computations. The details of the prompt are described in Fig. 6.
Additionally, we design the global context cache module to maintain a consistent context prefix across nodes, maximizing the KV cache reuse opportunities. As each node completes its inference, its output is appended to this cache. An example of the cache content is illustrated in Fig. 2. Subsequent nodes incorporate relevant context from this cache, ensuring accurate information flow and efficient KV cache reuse.
Counterintuitively, using all previous node results (global context cache) in the context is more efficient than only including dependent nodes' results. To illustrate, consider sequence of nodes A + B - C + D, with an additional dependency B-D. The original question and instructions prompts together are represented as P. When Plato starts inference on A, the prefix prompt will be P. After prefill, P's corresponding KV cache will be stored in memory. Once node A finishes, its output A is stored in the global context cache, and the system starts Node B's inference. The prompt for Node B has prefix P+*'. Since P's KV cache is already computed, the inference engine loads it and only needs to perform prefill on A", generating and saving s'ss KV cache for future reuse. Similarly, when node completes, its output B* is added to the cache, and node C's prompt becomes P+ BB The system can reuse the KV cache for P+ and only needs to perform prefill computation on B'. When node D's inference begins, its prompt prefix will be *+A++B" +C*, and the system already has KV cache for * requiring prefill computation only for C*.
In contrast, if we were to include only the direct dependencies in each node's context (e.g., for node D, only including outputs from nodes B and C), the prompt would be +C* This approach would significantly reduce KV cache reuse, as KV cache values depend on tokens' positions in the sequence. Since B* and C* would appear at different positions compared to their locations in the global context, the system would need to perform additional prefill operations, generating entirely new KV caches. This selective context approach would be less efficient than our global context strategy.
From this analysis, can that the global context cache maintains consistent prefix order across nodes, maximizing KV cache reuse opportunities. Each node's output only needs to be prefilled once, after which its KV cache can be reused. In contrast, without our

---

Published as a conference paper at COLM 2025
cache aligmment, generating the i-th node would typically require re-prefilling all outputs from nodes through i -I, incurring a cumulative overhead of 1 + + ... (N - -1) = O(N2) for nodes. By ensuring each node's output is added to the cache exactly once and reused thereafter, our approach reduces prefill computation to O(N), significantly minimizing overhead.
5 Experiments
We comprehensively evaluate our proposed method aiming to answer the following research questions: (1) How much does Plato affect generation efficiency and quality compared to baseline methods in general? (2) How does Plato perform across different question categories? Which categories benefit most from Plato, and which are less suitable? (3) What benefits does our pipelining planning and node inference optimizations provide? (4) How much overhead does Plato incur?
5.1 Experiments Setups
Dataset. We evaluate our method using two comprehensive datasets. The primary dataset is Vicuna (Chiang et al., 2023), which contains 80 diverse questions across nine categories. Additionally, we evaluate Plato on WizardLM (Xu et al., 2023), a larger dataset with 218 questions spanning 28 different skill types including academic writing, code debugging, historical analysis, and more. Since the evaluation results on both datasets yield similar conclusions, we focus our discussion on the Vicuna results in the main evaluation section, while WizardLM results are presented in Appendix C.
Models. We evaluate three state-of- open-source models of different sizes: Phi-4 (Ab- din et al., 2024), Qwen2.5-32B (Team, 2024), and Llama3. 1-70B (Meta, 2024). Details regarding the model sizes, API endpoints, and model repositories can be found in Appendix A.
System Setups. We build Plato inference system on top of SGLang (Zheng et al., 2023b) v0.4.3 and running inference two 80 GB NVIDIA A100 GPUs with tensor parallelism.
Baselines. We compare our solutions to three different baselines: autoregressive generation (AR), cfain-of-thought prompting (CoT) (Wei et al., 2022), as well as the state-of-the-art semantic-space parallel decoding technique: Skeleton-of- ofoooohhhtttttttt (SoT) (Ning et al., 2023). Other consecutive multi-token parallel decoding methods (Leviathan et al., 2023; Chen et al., 2023; Cai et al., 2024; Fu et al., 2024) are orthogonal and complementary to our work as they focus on token-level parallelism rather than semantic-space parallelism, and can be applied to improve the efficiency of either our planning stage or node inference stage.
Evaluation Metrics. We evaluate answer quality using the LLM-as-a-judge frame- work (Zheng et al., 2023a). For each evaluation, the judge is presented with the question and pair of answers generated by two different decoding techniques on the same model. To mitigate position bias, we swap the order of the two answers and call the judge twice for each comparison. We use GPT-4o-mini (OpenAI, 2024a) as the judge, which achieves similar judging performance
GPT-4o (Hurst et al., 2024) but is 20x cheaper (Tan et al., 2024). We report net win rates (Ning et al., 2023) for each model, calculated as (#wins - #loses) / #questions, where 0% indicates similar answer quality and higher values indicate better answers. For efficiency evaluation, we measure throughput speed-up by calculating the total number of generated tokens divided by the total processing time across all questions. This metric provides a fairer assessment of inference system performance (Kwon et al., 2023; Zhong et al., 2024; Sheng et al., 2023) compared to end-to-end latency, which can be affected by randomly varying answer lengths.
5.2 Overall Evaluation
We evaluate different decoding approaches using Autoregressive (AR) generation as our baseline. Figure presents the throughput speed-up and answer quality for each model compared to AR. Our results demonstrate that Plato achieves optimal balance between quality and efficiency. It not only produces higher quality answers than AR with an average net win rate of 40%, but also delivers inference speed-ups of up to 1.68x.
7

---

Published as a conference paper at COLM 2025
100%
Win (Lett
(Right
Net Win Rate
Plato
SoT
0%
AR
50T
Plato
Owen2
-50%
SoT
Llama3
COT
Phi4-14B
Plato
AR
-100%
0.0x
1.0x
2.0x
3.0x
0%
20%
40%
60%
80%
100%
Speedup
Percentage
Figure 3: [Overall Evaluation]: Answer quality and speed-up compared to normal autoregressive generation (AR) Vicuna.
Figure 4: [Overall Evaluation): Quality of answers across models between differ- ent methods Vicuna.
Model
Phi6-118
Qwenn 5-326
Llama..1 1-708
Model
Pho1-146
Owen255:22 5-36
Liamas.1 1-70B
Method
Sol
Flato
SaT
Plato
Sol
Flato
Method
Sol
Flato
Plato
Plato
codling
-100%
14%
-100%
-43%
-56%
-57%
coding
287
1.79
2.49
1.34
3,02
1.25
common--eerse
-10%
80%
-90%
50%
-80%
30%
common-sense
3.89
1.70
3.17
2.05
3.64
2.00
counterfactual
-00%
100%
-100%
50%
-80%
100%
counterfactua
382
1.90
3.14
1.85
3,50
1.46
genetic
-10%
100%
-90%
60%
-90%
80%
generic
3.87
1.51
3.39
2.01
3.87
1.75
knowledge
-50%
90%
-80%
70%
-80%
80%
knowledge
4.14
1.34
3.23
1.88
3.91
1.74
math
-100%
-67%
-100%
-100%
66%
47%
math
2.79
2.49
1.25
2.71
1.14
roleplay
roleplay
wrtting
-90%
10%
-0%
-20%
-90%
10%
writing
3.29
1.98
3.42
1.50
4.04
1.34
Table 1: |Category Breakdownl: Net win rate of answer quality over AR across all models on each question category.
Table 2: [Category Breakdown]: Speedup over AR across all models on each question category.
In contrast, while SoT achieves the greatest speed-up by processing sub-question nodes in parallel, its answer quality is significantly worse than AR methods. This quality degradation expected due to SoT's assumption of complete independence between nodes, which substantially reduces answer coherence and effectiveness.
We also observe that CoT prompting actually degrades performance compared to standard AR generation. This aligns with previous findings that CoT is an emergent ability highly dependent on model scale, primarily benefiting models with approximately 100B parameters or larger (Wei et al., 2022).
Figure 4 further illustrates the win/ tios/ rates across all models when comparing methods pairwise. The results are striking: Plato thoroughly outperforms SoT with 90% net win rate, while AR achieves a 76% net win rate over SoT. Overall, Plato consistently delivers the highest quality responses, while SoT produces the lowest quality outputs despite its speed advantages.
5.3 Category Breakdown
We further analyze performance across different question categories to understand which types of questions are most suitable for Plato and SOT. Table 1 breaks down answer quality and Table shows the speed-up. Both methods are compared against the baseline AR decoding.
From Table 1, we observe that Plato improves quality across most question categories, while SoT decreases quality across all categories. Plato performs worst on tasks with strong sequential dependencies that require holistic outputs, such as coding and math. these categories, Plato also achieves the least speedup these tasks offer fewer parallel inference opportunities, with solutions that must be developed step by step.
Upon closer examination of these challenging categories, find that for coding tasks, Plato tends to produce code fragments with detailed explanations rather than generating complete, cohesive code examples. For math questions, Plato often produces verbose explanations
8

---

Published as a conference paper at COLM 2025
Model
Phi6-145
Owenn255 3-26
1ammaa1-005
Optimization
Pipeline [secords]
KV [Hlolenss
Metric
TP
PD
1E
TP
PD
1E
TP
PD
1E
Model
14
02.5
L31
74
02.5
L3.1
307.81
29.5
7,86
75.10
9.35
Befone Opt.
16.21
17.30
4580
Plato
136.46
3,87
55.26
67.05
2.01
33.36
31.57
2.37
13.32
After Opt.
11.49
12.68
Reduction
29%
27%
29%
73%
76%
77%
Table 3: [Compare with SoT] Average throughput (TP: token /sc),, parallel degree (PD) and inference efficiency (IE: TP/ (PPD) of Plato and SoT over all questions in Vi- cuna.
Table 4: [Ablation Studyl: Overhead reduc- tion after applying optimization in $4.2.1 and $4.2.2. P4, Q2.5, L3.1 are abbreviations of Phi-4, Qwen2.5 and Llama3.1.
Model
Phio-145
Llamaã1-775 Ow2
Pha4-44
Owen2.5-326
Plato
SoT
AX
63.5%
95.5%
94.5%
94.0%
56.0%
65.0%
55.0%
93.0%
5.5%
Plato
301
2.10x
1.16x
2:11x
1.2xx
2.31x
1.12x
Table 5: Correctness rate of AR, SoT, and Plato on GSM-8K.
Table 6: Speedup of AR and SoT on GSM- 8K over AR.
where several steps could ideally be condensed into one. The LLM judge penalizes this redundancy, resulting in lower scores even when the final answer is correct.
To validate this, we conducted additional experiments on the first 200 samples of the GSM- 8K dataset (Cobbe et al., 2021). We directly compared the correctness of the numerical results with the ground-truth answers using Deepseek- V3-03244 (DeepSeek-a AI, 2024), rather than assessing overall answer quality. As shown Table 5, while SoT's accuracy drops sharply on math problems, Plato maintains correctness rate comparable to AR. Furthermore, Table 6 shows that, despite the highly sequential nature of math solutions, Plato can still exploit limited parallelism, achieving up to 1.23x speedup over AR.
Additionally, Plato excels in both quality and efficiency for questions requiring breadth rather than sequential reasoning. For these tasks, Plato can generate more comprehensive answers in parallel, incorporating more examples and covering broader aspects of the topic, thus simultaneously improving both quality and speedup.
We provide three case studies Appendix for deeper analysis of the differences in generation results between AR, SoT, and Plato.
5.4 System-level Inference Efficiency Comparison
To understand the overall system-level inference efficiency, we compare Plato with SoT in Table 3, reporting throughput, parallel degree, and inference efficiency.
For Plato, we define the parallelism degree as the total number of nodes divided by the maximum path length in the DAG graph generated during the planning phase. For SoT, the parallelism degree equals the number of nodes in the skeleton. We also introduce an inference efficiency (IE) metric, calculated speed-up divided by parallelism degree, to evaluate how efficient the inference is independent of parallelism benefits. A higher IE indicates a more optimized system.
The results show that Plato significantly outperforms SoTT in terms of inference efficiency, with improvements ranging from 24% 108%. This demonstrates the effectiveness of Plato's s approach in co-designing algorithms and systems to achieve superior efficiency.
5.5 Ablation Study on System Optimization
We evaluate the impact of our system optimizations on both inference speedup and KV cache prefill overhead.
Pipeline Optimization. The left side of Table shows the inference latency before and after applying our pipeline design that overlaps planning and node inference stages. This optimization yields a consistent 27-29% speedup (latency reduction) across all models, demonstrating the effectiveness of our pipeline approach.

---

Published as a conference paper at COLM 2025
KV Cache Reuse. A significant overhead in Plato comes from prefill computations during multi-node inference. The right side Table shows that our KV cache reuse strategy reduces the number of additional prefill tokens by approximately 75% across all models. This substantial reduction in prefill overhead contributes significantly to the overall efficiency of Plato while maintaining answer quality.
Additionally, we report the exact overhead percentage caused by prefilling additional tokens. Thanks to the KV cache reuse optimization in $4.2.2, Plato introduces minimal latency overhead-less than 1.6% of the total generation time across all tested models. The detailed overhead analysis for different models presented in Table 10.
Conclusion
In this paper, we propose Plato, which pushes the boundary of semantic space parallelism to enhance inference efficiency. Plato codesigns the inference algorithm together with the system, significantly improving both answer quality and generation speed. Our comprehen- sive evaluations demonstrate that Plato achieves an optimal balance between speedup and quality, with 90% quality net-win rate over So'T while improving throughput by up to 68% compared to autoregressive baselines. Our pipeline design improves speedup by 29%, and KV cache reuse optimization reduces overhead by 75%, demonstrating the effectiveness of our algorithm-syssem co-design approach.
Looking ahead, Plato's principles can extend to multi-agent systems, complex reasoning, and collaborative AT workflows. By modeling dependencies between sub-problems and leveraging semantic space parallelism, Plato advances LLM inference efficiency and ef- fectiveness, opening new research avenues at the intersection of algorithmic design and systems optimization.
Acknowledgments
We would like to thank anonymous reviewers for their valuable comments and feedback. This work was supported in part by NSF under grants CMM1-20382155, CNS-2321532, CNS- 2323174, OAC-2503010, CNS-24026%,, CNS-2238665 and National AI Institute for Edge Computing Leveraging Next Generation Wireless Networks, Grant-2112262.
10

---

Published as a conference paper at COLM 2025
References
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. arXiv preprint ar axXio:2412.08905, 2024.
Anthropic. Prompt caching, 2024. URL httpp::/wwwwnntthoop....co http://wwwnrrpppc..ccco///////pe http:////////////////oooooooooo/ http://///.////////oo//aaacccaaa
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. arXiv preprint arXiv:2308.0940 2023.
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason Lee, Deming Chen, and Tri Dao. Medusa: Simple IIm inference acceleration framework with multiple decoding heads. ar preprint ar Xiv:2240 10774, 2024.
Marco Cascella, Jonathan Montomoli, Valentina Bellini, and Elena Bignami. Evaluating the feasibility of chatgpt in healthcare: an analysis of multiple clinical and research scenarios. Journal of medical systems, 47(1):33, 2023.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, and John Jumper. Accelerating large language model decoding with speculative sampling. Xrv preprint arxii::3022001311 2023.
Lespiau, Laurent Sifre,
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Éric P. Xing. Vicuna: An open- chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
URL http://ssssoog///oo///-/3------/ otg
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. ar preprint ar drXiv:2110.14168, 2021.
Deepseek. Prompt Prompt caching api, 2024. URL httptps//pplafffomddeeeeee. coppppppeeekcmm/ppdpooo///eeee ws/news0802/.
DeepSeek-AL. Deepseek-v3 technical report, 2024. URL.htps://arxii.orr/////1111 37.
Yichao Fu, Peter Bailis, lon Stoica, and Hao Zhang. Break the sequential dependency of llm inference using lookahead decoding, ar Xiv preprint ar xrXi::2402.02057, 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. XrXiv e-prints, pp. arXiv-2400 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arv preprint XxXo::2501.12948. 2025.
Nurul Farhanah Abdul Hadi, Nur Afifah Diyanah Shaharudin, Nurul Dalila Faris, Wan Sarah Wan Zainal Abidin, Nurin Erdiani Mhd Fadzil, and Siti Fazlina Isnin. Understand- ing writing difficulty and writing process among pre-university students. Malaysian 2024.
Journal of Social Sciences and Humanities (MJSSH), (19):e002335--02220
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arxio preprint aro::24400.212 2024.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. ar Xiv preprint ar XrXit:2412.16720 2024.
11

---

Published as a conference paper at COLM 2025
Shuowei Jin, Xueshen Liu, Qingzhao Zhang, and Z Morley Mao. Compute or load kv cache? why not both? Xiv preprint ar arXiv:2410.03065, 2024.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp. 611-626, 2023.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274-19286. PMLR, 2023.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint arXio:240..15077, 2024.
Meta. Introducing Ilama 3.1: The next generation of open models. https:://a http://aaamaaaa....// ai.meta.com/bl -aae---1mmmm1-1111 July 2024. Accessed: 2025-03-29.
Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. Using an I1m to help with code understanding. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pp. 1-13, 2024.
Xuefei Ning, Zinan Lin, Zixuan Zhou, Huazhong Yang, and Yu Wang. Skeleton-of-thought Large language models can do parallel decoding. arXiv preprint arXio:2307.153377 2023.
OpenAL. GPT-4o mini: Advancing cost-efficient intelligence, 2024a. URL https://open http://indxxx/ttteeeeee
ai. wwt.coooiiiiii wtt.//ooe///////////e//o/// gpt 40 mini- efffiiiiei--------iiiii--------e intelllgenneelleeeeee.... Accessed: 2025-03-29.
OpenAL. Prompt caching, 2024b. URL /Rtp::t:://plaaffmm openai. .pweaa..com/occ//geeeiiiieeooeee comm//bcss/guides/prmmea -caching.
Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, lon Stoica, and Ce Zhang. Flexgen: High-throughput generative inference of large language models with single gpu. In International Conference on Machine Learning, pp. 31094-31116 PMLR, 2023.
Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y Tang, Alejandro Cuadron, Chen- guang Wang, Raluca Ada Popa, and lon Stoica. Judgebench: A benchmark for evaluating based judges. arXiv preprint ar arXiv:2410.12788 2024.
Qwen Team. Qwen2.5: party of foundation models, September 2024. URL https: //qwenlm. github. tt/blog/b///ooo /wwe2/./weee.mmmmmmm 5/.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: http:////oceeeees..e/mmm neurips. httpps.cc/ppppeee httpppr...lleeepppp///////////// e91 191fbd053c1c4484440 .
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Repre- sentations, 2023.
Zhibin Wang, Shipeng Li, Yuhang Zhou, Xue Li, Rong Gu, Nguyen Cam-Tu, Chen Tian, and Sheng Zhong. Revisiting slo and goodput metrics in Ilm serving. arx preprint ar axXi::2400.14257 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thougg prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-2444 2022.
12

---

Published as a conference paper at COLM 2025
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint ar arXie:2304.12244, 2023.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. arv preprint arXiv:2305. 10600, 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Bric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. ar arv preprint ar Xiv:23006.0000 05685, 2023a.
Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. Efficiently programming large language models using sglang. ar Xiv preprint ar aXX1o:2312.07104, 2023b.
Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. Distserve: Disaggregating prefill and decoding for goodput-optimizer large language model serving. arXiv preprint ar XrXiv:2401.08 096600, 2024.
13

---

Published as a conference paper at COLM 2025
Model
Hugging Face Path or API Model Name
GPT-4o-mini (OpenAI, 2024a)
gpt-4o-mini
Phi4-14B (Abdin et al., 2024)
microsoft/phi phi-4 Owwen/Qwen2.5-22B-22 Meteallama//MMtaaLLaaa3....
Qwen2.5-32B (Team, 2024)
Llama3.1-70B (Meta, 2024)
Table 7: Hugging Face or API Model Name of all used models.
A Models Details
We summarizes Hugging Face Hubs' repos of the open-source models and the GPT model we used in Table 7. We run the open-source models using SGlang (Zheng et al., 2023b) v0.4.3 with 2xA100 80G.
For all LLM-as-a-judge (FastChat) (Zheng et al., 2023a) evaluations, we use GPT-4o-mini as the judge LLM with the API model name gpt-4o-mini.
B Prompt Details
Plan Generation Prompt We demonstrate our plan generation prompt for Plato in Figure
To ensure the generation of node dependency graphs that are both informative and straight- forward to parse, we define the expected output format at the beginning of the prompt. This specification serves to align the Large Language Model (LLM) output with our parsing requirements, thereby ensuring that the generated skeletons are easy to parse. Then we provide guidelines instruct the LLM to generate comprehensive and cohesive plan while maximizing the parallelism of decoding. To complement this specification, we provide an illustrative example that serves as one-shot learning instance.
We also observe that models like Qwen2.5-32B and Phi-4 follow the instructions well. Meanwhile, Llama3.1-70B produces unstable outputs from time to time.
Node Inference Prompt We demonstrate our node inference prompt for Plato in Figure 6.
The node inference prompt helps the LLM focus on the subtask it assigned to, and provides detailed and in-depth answer to it, making the overall response better.
The template begins by reminding the LLM of the main question, which helps to keep the answer focused and on track. Then we state the requirements for answering the subtask. Following this, we append the description and results of all finished subtasks at this point. Because the generation waits until its dependent subtasks are finished, the relevant information has been included in the context. Instructions and contexts are only prefilled once, their KV cache will be reused repeatedly during parallel node inference. After the context, we add the description of the subtask.
14

---

Published as a conference paper at COLM 2025
Planning Prompt of Plato.
<|im_start |2syste
Decompose the given problem into subtasks with dependencies, optimizing for parallel execution (shortest critical path). Format each subtask as (each line is subtask):
<id> Subtask description.(deeendnnc..ii
Guidelines:
Conciseness:
Keep descriptions short and action-oriented (e-g., "Calculate X", "Summarize Y").
Dependencies: Only specify dependencies if a subtask requires outputs from others.
Readability: Ensure the final concatenated answer flows logically (e.g,, analysis follows data generation).
Parallelism: Maximize independent subtasks; minimize critical path length.
Comprehensiveness:: Explore any relevant aspects of the given problem in parallel.
Structure: Ensure a clear logical flow between subtasks and avoid redundancy and fragmentation.
Completion: End with a synthesis subtask to reach a *very short* and cohensive final answer.
Example:
Problem:
"Compare the average GDP growth of Country and over years, then recommend investments.
Response:
Fetch GDP data for Country A (2019-2023).
2. Fetch GDP data for Country B (2019-2023).
3. Calculate average GDP growth for Country A.[1]
4. Calculate average GDP growth for Country B.[2]
5. Compare growth rates and identify higher-performing country.[3,4]
6. Recommend investment strategy based on comparison.[5]
Note: Subtasks 1-2 and 3-4 can run in parallel; 1/2-3/4-5-6 form the critical path. 
 < start >uuser
[Insert Question Here] < im end]>
<| <mi_starrt paasisstan
Figure 5: Plan generation prompt of Plato.
15

---

Published as a conference paper at COLM 2025
Node Inference Prompt of Plato.
</insstart |>system
You are an AI assistant that can answer questions.< im end|>
<|intttarti >>user Your team is solving the question: [Insert Question Here]
Your task is to complete one specific subtask in processing plan.
You are an expert solving subtask as part a larger workflow. Follow
these rules:
Conciseness: Answer in 1-3 sentences if possible.
Depth: Include key eesights/calculatie (e.g. formulas, reasoning, examples).
Consistency: Consider the logic flow from previous results and the question and be cohensive.
Previous results: [Global Contexts]
Directly state your answer to the subtask: [Subtask Descrip- tion].< im_end|> <I im.starrt/ Saterr//aasian >>assistant
Figure 6: Node inference prompt of Plato.
16

---

Published as a conference paper at COLM 2025
Model
Phi4-14B
Method
SoT
Plato
Result
Win
Tie
Lose
Net win rate
Win
Tie
Lose
Net win rabe
Academic Wriking
-50%
3
D
50%
Art
-40%
4
0
80%
Biology
50%
Chemistry
-100%
100%
Code Dehug
-20%%
0
60%
Code Generation
16
-89%
4
46% 5
Common-Sense
Complex Format
-83%
33%
Computer Science
12
-80%
10
2
53%
Counterlactual
0%
100%
Econceny
Entertainmentt
80% 0
Ethics
-33%
67%
History
-29%
0
30%
Law
100%
Literature
-00%
60%
Math
15
-20%
8
10
30% 2
Medicine
-80%
60%
Multilingual
Music
-00%
80%
Physics
-90%
1
0%
Reasoning
-54%
54%
Roleplay
Sport
-80%
60%
Technology
-67%
4
2
0
67%
Tusicity
0%
100%
TruthfulQA
Writting
-89%
13
50%
Table Answer quality of SoT and Plato compared with auto-regressive (AR) across different categories on WizardLM. The answers are generated by Phi-4.
C Evaluation on WizardLM
In addition the Vicuna dataset, we also evaluate Plato on a larger dataset, WizardLM (Xu et al., 2023). It contains 218 questions covering wide range of skills, such as academic writing, code-debug, history, etc. We use standard autoregressive (AR), Skeleton-of-Thought (SoT), and Plato to answer these questions and evaluate the output using LLM-as-a-judge (FastChat) (Zheng et al., 2023a). We present the results in Table 8 and Table 9 with detailed breakdown across different categories.
Across all the subjects and models, Plato achieves positive total net win rate over AR while SoT suffers from significant quality drop, showing that Plato effectively maintains answer quality while improving the token generation speed.
17

---

Published as a conference paper at COLM 2025
Model
Qwen2.5-326
Method
SoT
Plato
Result
Win
Tie
Lose
Net win rate
Win
Tie
Lose
Net win rafe
Academic Writing
-100%
0
0
0%
Art
-60%
40%
Biology
Chemistry
-100%
30%
Code Debug
10
-100%
4
0
40%
Code Generation
-100%
12
-59%
Comman-Sense
Complex Format
-9%%
58%
Computer Science
15
-10%%
4
7%
Counterfactual
-8%%
30%
Ecencny
Entertainment
80%%
0%
Ethies
-33%
0
83%
History
1000%
50%
Literature
-00%
0%
Math
17
-85%
2
12
6
-20%
Medicine
-10%%
60%
Multilingual
Music
-60%
60%
Physics
-07%
0%
Reasoning
31%
Roleplay
Sport
-107%
0%
Technology
-50%
0
83%
Toxicity
-75%
Writtig
-72%
11
30%
Overall
-83%
44
19%
Table Answer quality of SoT and Plato compared with auto-regressive (AR) across different categories on WizardLM. The answers are generated by Qwen2.5-32B.
18

---

Published as a conference paper at COLM 2025
Model
Phi4-14B
Qwen2.5-32B
Llama3.1-70B
Avg. prefill time [s]
0.13
0.19
0.47
Avg. total time [s] Overhead
11.49
12.68
35.51
1.17%
1.52%
1.32%
Table 10: Overhead of Plato after applying KV cache reuse optimization. Time is averaged over all questions in Vicuna.
D Overhead Analysis
We present the latency overhead results across different models in Table 10.
E
Case Studies
We present three case studies to compare answers from AR, SoT, and Plato generated by different models in details.
E.1
Comparison of Answers Generated by Phi4-14B
Question: What are the main differences between Python and JavaScript programming languages?
(Concise) AR Answer: Python and JavaScript are both popular programming languages, but they have distinct characteristics and use cases. Here are some of the main differences between the two:
Design and Syntax: Python: Known for its clean and readable syntax, Python uses indentation to define code blocks, which enhances readability. It follows philosophy of simplicity and minimalism. JavaScript: Uses curly braces '0' to define code blocks and semicolons ' to end statements. Its syntax can be more verbose and less intuitive for beginners compared to Python.
Typing: Python: Dynamically typed, but with optional type hints (introduced in Python 3.5), allowing for some static type checking JavaScript: Also dynamically typed, but with the addition of TypeScript, a statically typed superset that provides optional static type checking.
Execution Environment: Python: Primarily server-side language, but also used for scripting, data analysis, and machine learning. It executed in variety of environments including standalone scripts, web servers (e.g., Django, Flask), and data science platforms. JavaScript: Originally designed for web browsers to add interactivity to web pages. It is now widely used on both the client-side (browsers) and server-side (Node.js) with the rise of full-stack JavaScript development.
Concurrency: Python: Uses Global Interpreter Lock (GIL) that can be a bottleneck for CPU-bound multithreading. However, it supports asynchronous programming with 'asyncio' and can leverage multiprocessing for parallelism. JavaScript: Utilizes an event- driven, non-blocking 1/O model, making it highly suitable for asynchronous operations. uses Promises and ''async/awaar aawii'' to handle asynchronous code more easily.
Use Cases: Python: Commonly used in web development, data science, artificial intel- ligence, scientific computing, automation, and scripting. JJvaScript: Primarily used for front-end web development, but also for back-end development with Node,js. It is essential for building interactive web applications.
Libraries and Frameworks: Python: Rich ecosystem with libraries like NumPy, Pandas, TensorFlow, and frameworks like Django and Flask. JavaScript: Extensive library ecosystem with tools like React, Angular, Vue.js for front-end development, and Ex- press.js, Koa.js for server-side development.
19

---

Published as a conference paper at COLM 2025
Community and Support: Both languages have large, active communities and extensive documentation, but the focus of their communities differs based on their primary use cases.
These differences highlight the distinct roles and strengths of Python and JavaScript, making them suitable for different types of projects and development needs.
(Concise) SoT Answer:
Language type. key concept: Python primarily an interpreted, high-level programming language focused on code readability and simplicity, often used for scripting and rapid application development, JavaScript is a lightweight, interpreted or Jl'T-compiled pro- gramming language with first-class functions, primarily used for client-side scripting in web browsers.
Execution model. key concept: Python is an interpreted language, typically executed line- by-line, which can make debugging easier but may result in slower execution. JavaScript, originally designed for web browsers, is also interpreted but often compiled just-in-time (ITT) by modern engines like V8, enhancing performance.
Syntax differences. key concept: Python emphasizes readability with its use of indentation to define code blocks, while JavaScript uses curly braces and semicolons, allowing for more flexible syntax styles.
Use cases. key concept: Python is widely used in data science, machine learning, and scien- tific computing due to its extensive libraries like NumPy and Pandas, while JavaScript is predominantly used for web development, enabling dynamic and interactive web pages through frameworks like React and Angular.
Typing system. key concept: Python dynamically typed with optional type hints, allowing more flexibility, while JavaScript is also dynamically typed but has introduced TypeScript, a statically typed superset, to enhance type safety and development experience.
Concurrency model. key concept: Python uses the Global Interpreter Lock (GIL.) which limits execution to one thread at time, whereas JavaScript employs an event-driven, non- blocking I/O model with an event loop, allowing for efficient handling of asynchronous operations.
Libraries and frameworks: Python boasts a rich ecosystem with libraries like NumPy and frameworks like Django for web development, while JavaScript is renowned for its extensive libraries such as React and frameworks like Angular and Vue.js, catering primarily to front-end development.
Community and support: Both Python and JavaScript have large, active communities, but Python is often praised for its emphasis readability and simplicity, which attracts beginners, while JavaScript's community is heavily focused on web development and rapidly evolving with frameworks like React and Node.js.
Performance characteristics. key concept: Python is generally slower than JavaScript due to its interpreted nature and dynamic typing, while JavaScript, especially with the V8 engine, optimized for high performance web environments.
Browser support. key concept: JavaScript is natively supported by all web browsers, making it essential for client-side web development, while Python is not directly supported in browsers but can be used server-side or through transpilers like Brython or Pyodide for client-side scripting.
(Concise) Plato Answer (dependent nodes are in I]):
Identify core use cases of Python []: Python is primarily used for web development, data analysis, artificial intelligence, scientific computing, and automation scripting. Its sim- plicity and readability make it popular in data science and machine learning. Python's extensive libraries, like NumPy, Pandas, and TensorFlow, support its use in scientific and analytical applications. Additionally, frameworks such as Django and Flask facilitate web development, while its versatility allows use in automation and backend server applications.
20

---

Published as a conference paper at COLM 2025
Identify core use cases of JavaScript [I: JavaScript is primarily used for web development, enabling dynamic and interactive content on websites through client-side scripting. It is essential for front-end development, interacting with web browsers via the Document Object Model (DOM), and is also used in back-end development with platforms like Node.js. Additionally, JavaScript is utilized in building mobile applications through frameworks like React Native and in desktop applications using Electron.
Analyze symlax differences between Python and JavaScript []: Python and JavaScript have distinct syntax differences. Python uses indentation to define code blocks, promoting readability, whereas JavaScript relies on curly braces '(). Python's syntax is generally more concise and clean, with dynamic typing and simpler function definitions, while JavaScript supports both functional and object-oriented programming with a more com- plex syntax including semicolons and explicit type declarations. Additionally, Python uses 'def' to define functions, whereas JavaScript uses the 'function' keyword arrow functions ("=:²).
Compare performance characteristics of Python and JavaScript []: Python and JavaScript have distinct performance characteristics due to their design and use cases. Python is typically slower than JavaScript in execution speed because it is an interpreted language, often requiring optimization or using extensions like Cython for performance-crttccl applications. In contrast, JavaScript, especially with modern engines like V8, is designed for fast, efficient execution in web browsers and server environments (Node.js), benefiting from just-in-tine tnme (JIT) compilation. While JavaScript can handle asynchronous operations efficiently, particularly with its non-blocking I/O model, Python requires additional frameworks (such as asyncio) to achieve similar asynchronous capabilities.
Evaluate ecosystem and library support for Python II: Python boasts a robust ecosystem with extensive library support, particularly strong in data science, machine learning, and scientific computing. Libraries such as NumPyy Pandas, and TensorFlow provide powerful tools for numerical analysis, data manipulation, and deep learning. For web development, frameworks like Django and Flask offer comprehensive solutions for building complex web applications, and libraries such Requests facilitate network operations. Python's package manager, pip, simplifies the installation and management of these libraries, ensuring a rich and versatile development environment. Overall, Python's 's ecosystem is well-integrated and continuously evolving, making it preferred choice for a wide range of applications.
Evaluate ecosystem and library support for JavaScript D: JavaScript has a robust ecosystem and extensive library support, primarily driven by its use in web development. Key libraries and frameworks include React, Angular, and Vue.js for front-end development, which enable the creation of dynamic user interfaces. For back-end development, Node.js allows JavaScript to run on the server-side, supported by libraries like Express.js for building web applications. Additionally, Electron provides tools for building cross- platform desktop applications, and React Native extends JavaScript's 's reach to mobile app development. The npm (Node Package Manager) registry further bolsters JavaScript's 'ss ecosystem, offering one of the largest collections of open-source libraries and tools, facilitating extensive community support and rapid development.
Compare learning curves and communitty support for both languages []: Python and JavaScript both have extensive community support and resources, but their learning curves differ due to syntax and use cases. Python is often considered easier for beginners due to its simple, readable syntax and emphasis on code readability, making it ideal for learning programming fundamentals. It has a large community, with comprehensive documen- tation and numerous educational resources available, which aids in learning. On the other hand, JavaScript's 's learning curve can be steeper due to its more complex syntax, including asynchronous operations and prototypal inheritance, which may require deeper understanding of web technologies.
Synthesize findings into a concise comparison of Python and JaonScript [3,4,5,6,7]: Python and JavaScript are both versatile languages with distinct core use cases. Python excels in data analysis, scientific computing, and artificial intelligence, supported by libraries like NumPy and TensorFlow, and is favored for its readability and simplicity. JavaScript, on the other hand, is primarily used for web development, enabling dynamic content
21

---

Published as a conference paper at COLM 2025
websites and supporting both front-end and back-end development through frameworks like React, Node.js, and React Native...
Findings: We find that the AR answer provides clear, structured overview of Python and JavaScript differences but lacks synthesis and in-depth technical insights (e.g, JIT com- pilation, browser support). SoT answer is more comprehensive but is overly granular and fragmented. The Plato answer systematically breaks down key aspects (syntax, performance, ecosystems) and synthesizes them into cohesive comparison, avoiding redundancy while maintaining depth.
E.2 Comparison of Answers Generated by Qwen2.5-32B
Question:
Can you explain the basics of quantum computing?
Certainly! Quantum computing is type of computing that uses quantum mechanics to process information. It leverages the principles of superposition and entanglement to perform operations on data.
(Concise) AR Answer:
Qubits: Unlike classical bits, which are the basic unit of information in classical computing and can be either 0 or 1, quantum bits or qubits can exist in multiple states simultaneously thanks to superposition. This means qubit can be 0, 1, or any quantum superposition of these states.
Superposition:
This principle allows quantum system to be in multiple states at the same time. For example, a qubit can be in a state that is a combination of both and 1 until it measured. The act of measurement collapses the qubit into one of its possible states, which could be 0 or 1.
This is a phenomenon where quantum particles become interconnected and the state of one (no matter the distance) can depend on the state of another. For entangled, the state of one qubit instantly influences the state
Entanglement:
instance, if two qubits of the other. This property is crucial for many quantum computing operations and can be used to perform certain computations much faster than classical computers.
Quantum Gates: In classical computing, logic gates (like AND, OR, NOT) manipulate bits to perform logical functions. Quantum gates manipulate qubits in similar ways but use quantum operations that can take advantage of superposition and entanglement. These gates can perform complex operations on multiple qubits simultaneously. which is one of the reasons quantum computers have the potential be significantly more powerful than classical computers for certain tasks.
Quantum Algorithmss Algorithms like Shors algorithm for factoring large numbers and Grovers algorithm for searching unsorted databases are examples of algorithms that exploit quantum mechanical phenomena to solve problems much faster than any classical algorithm could.
Challenges: Quantum computing faces significant challenges, such as decoherence (loss of quantum behavior due to interaction with the environment) and error rates, which are higher than those in classical computing. Quantum error correction techniques are being developed to address these issues.
Quantum computers are particularly promising for solving problems in cryptography, optimization, and simulation of quantum systems, which are currently infeasible for classical computers. However, practical large-scale quantum computers are still in the early stages of development and research.
(Concise) SoT Answer:
Quantum bits (qubits).: Qubits are the fundamental units of quantum information, analo- gous to classical bits, but capable of existing in multiple states simultaneously due to the principle of superposition.
Superposition principle. key concept: The superposition principle allows qubits to exist in multiple states simultaneously, represented as combination of both 0 and 1, enabling
22

---

Published as a conference paper at COLM 2025
quantum computers to process vast amounts of information more efficiently than classical
computers.
Entanglement phenomenon. key concept: Entanglement occurs when qubits become corre- lated in such a way that the state of one (whether it's or 1) can depend on the state of another, no matter the distance between them, allowing for instantaneous state changes that are crucial for quantum computing operations.
Quantum interference. key concept: Quantum interference allows constructive or destructive interference of probability amplitudes, enabling quantum algorithms to amplify correct answers and cancel out wrong ones, thus optimizing computational outcomes.
Quantum gates. key concept: Quantum gates are operations performed on qubits to manip- ulate their states, analogous to classical logic gates, but they can also create superposition and entanglement, enabling quantum computations.
Quantum algorithms. key concept: Quantum algorithms, such Shor's and Grover's, s, leverage quantum properties to solve problems more efficiently than classical algorithms, offering exponential speedups for specific tasks.
Decokerence challenges. key concept: Decoherence refers to the loss of quantum behav- ior in qubits due to interaction with the environment, leading to errors in quantum computations and posing a significant challenge in maintaining quantum state integrity. Quantum supremacy. key concept: Quantum supremacy refers to the point where a quantum computer can solve problem that classical computer cannot solve within reasonable time frame, demonstrating the superior computational power of quantum systems.
(Concise) Plato Answer (dependent nodes are in I):
Define quantum bits (qubits) []: Quantum bits, or qubits, are the fundamental units of quantum information, analogous to classical bits but capable of existing in superposition of states, representing both 0 and 1 simultaneously until measured. This property stems from the principles of quantum mechanics, enabling qubits to perform multiple computations at once, potentially offering significant speedups for certain tasks.
Explain superposition [1]: Superposition in quantum computing allows qubit to exist in Superposo multiple states simultaneously, not just binary 0 or 1 as in classical computing. Mathemat- ically, a qubit can be represented as a linear combination al0) + B[1), where and & are complex numbers such that |a|2 + | 1, encapsulating the probabilities of measuring the qubit in state (0) or [1).
Describe entanglement [1]: Entanglement is quantum phenomenon where qubits be- come interconnected such that the state of one (whether it's 0 or 1) cannot be described independently of the state of another, no matter the distance separating them. This inter- connection allows operations on one qubit to instantaneously affect another, enabling complex quantum algorithms and secure quantum communication protocols.
Introduce quantum gates [1]: Quantum gates are the basic building blocks of quantum circuits, analogous to classical logic gates but operating on qubits. These gates manipulate the state of qubits using unitary transformations, enabling the creation of superpositions, entanglements, and interference, which are crucial for quantum algorithms. Examples include the Hadamard gate, which creates superposition, and the CNOT gate, which generates entanglement.
Summarize quantum algorithms [2,3,4] Quantum algorithms leverage the principles of superposition, entanglement, and quantum gates to perform specific computational tasks more efficiently than classical algorithms. Notable examples include Shor's algorithm for factoring large integers exponentially faster than the best-known classical algorithms, and Grover's 's algorithm for unstructured search problems, providing a quadratic speedup. These algorithms exploit quantum parallelism and interference to achieve their speedups.
Compare quantum computing with classical computing [5]: Quantum computing funda- mentally differs from classical computing in its ability to leverage superposition and entanglement, allowing qubits to exist in multiple states simultaneously and become interconnected in ways that classical bits cannot. This enables quantum computers to perform certain operations in parallel and process complex computations more efficiently
23

---

Published as a conference paper at COLM 2025
than classical computers for specific tasks, such factoring large numbers and searching unstructured databases.
Outline potential applications of quantum computing [5]: Quantum computing has the po- tential revolutionize several fields by solving complex problems more efficiently. Key applications include cryptography, where quantum computers could break current en- cryption methods but also enable quantum key distribution for secure communication; optimization in logistics and finance, where quantum algorithms can find optimal so- lutions faster; and simulation of quantum systems chemistry and materials science, enabling the development of new drugs and materials.
Findings: We find that all three methods give a comprehensive introduction to quantum computing, while Plato uses concrete examples to elaborate concepts. Plato also provides a clear and structured logical flow with comparison and summarization, making the whole answer coherent, while the answer from SoT is fragmented.
E.3 Comparison of Answers Generated by Llama3.1-70B
Question: How many times does the average human blink in a lifetime? Try to explain your answer. Your explanation should take the reader through your reasoning step-by-step.
(Concise) AR Answer:
Step 1: Delermine the average blink rate per minute: Research suggests that the average person blinks around 12-15 times per minute. Let's use conservative estimate of 13 blinks per minute.
Step 2: Calculate the number of blinks per hour: There are 60 minutes in an hour, so we can multiply the average blink rate per minute by 60 to get the number of blinks per hour: 13 blinks/minute minute x 60 minutes/hour
780 blinks/hour
Step 3: Calculate the number of blinks per day: Assuming an average of 16 hours of wakeful- ness per day (accounting for 8 hours of sleep), we can multiply the number of blinks per hour by the number of hours spent awake: 780 blinks blinks/hour /hour 16 hours/ /day 12,480 blinks/ day.
Step 4: Calculate the number of blinks per year: There are 365.25 days in year (accounting for leap years). We can multiply the number of blinks per day by the number of days year: 12,480 blinks/ day x 365.25 days/ dear x 4,555,200 blinks tiiks/year. year.
Step 5: Estimate the average human lifespan: According the World Health Organization (WHO), the average human lifespan is approximately 72 years (based on global data from 2019).
Step 6: Calculate the total number of blinks in a lifetime: Finally, we can multiply the number of blinks per year by the average human lifespan: 2,5 blinks/year x 72 years ss 327 027,971,2000 blinks / lifetime.
Therefore, the average human blinks approximately 327,971,200 times in lifetime, assuming an average blink rate of 13 blinks per minute, 16 hours of wakefulness per day, and an average human lifespan of 72 years. Keep in mind that this is a rough estimate and actual blink rates may vary from person person. Nonetheless, this calculation should provide a reasonable ballpark figure.
Conclusion:
(Concise) SoT Answer:
Average blinks per minute: According to various studies, the average person blinks around 12-15 times per minute. This rate can vary depending on factors such as attention, environment, and eye health, but 12-15 blinks per minute is commonly cited estimate.
Minutes per day: Assuming an average person sleeps for 8 hours day, that leaves 16 hours of wakefulness, which translates to 960 minutes per day.
Days per year: Assuming non-leap year, there are 365 days in year, and for leap year, there are 366 days, so we can take the average of the two, which is approximately 365.25 days per year.
24

---

Published as a conference paper at COLM 2025
Years in lifetime: Assuming an average human lifespan of around 80 years, can use this as a rough estimate for our calculation.
Calculate total blinks: To calculate the total number of blinks, we multiply the average blinks per minute (point 1) by the minutes per day (point 2) and then by the days per year (point 3) and finally by the years in lifetime (point 4). This gives us a preliminary estimate of total blinks in lifetime.
Consider blinking variations: We must account for variations in blinking rates due to factors such age, activity level, and environment, which can significantly impact the average blink rate.
Estimate lifetime blinks: Based on these calculations, the estimated total number of blinks in a lifetime is approximately 415 million, assuming an average lifespan of 80 years. However, this is a rough estimate and actual numbers may vary significantly from person to person.
(Concise) Plato Answer (dependent nodes are in I]):
Determine the average human lifespan []: Based on global data, the average human lifespan is approximately 72 years (70 years for males and 74 years for females), according to the World Health Organization (2019). This estimate takes into account factors such as mortality rates, life expectancy, and population demographics. I will use this value baseline for subsequent calculations.
Calculate the average number of blinks per minute I]: The average number of blinks per minute is approximately 15-20 blinks per minute (bpm). This rate can vary depending on factors like age, environment, and activity level, but 17.5 bpm is a commonly cited average. This rate will serve the basis for further calculations determine the total number of blinks in a lifetime.
Estimate the average number of minutes spent awake per day []: Based on typical adult sleep patterns, estimate the average number of minutes spent awake per day to be around 960 minutes (16 hours). This assumes 7-9 hours of sleep per night and 24 hours in day.
Calculate the average number of blinks per day [2,3]: Based on the estimated average number of minutes spent awake per day (960 minutes) and the assumed average blink rate (17.5 blinks per minute), the average number of blinks per day can be calculated as: 960 minutes/ day 17.5 blinks/ minute = 16,800 blinks/ day.
Calculate the average number of blinks per year [4]: To calculate the average number of blinks per year, we can multiply the average number of blinks per day (16,800 blinks /day) by the number of days in year (365.25, accounting for leap years). This results in: 16,800 blinks/day 365.25 days/ year R 6,133,800 blinks/yeat
Multiply the average number of blinks per year by the average human lifespan [1,5]: Multiplying the average number of blinks per year (6, (6,133,800 blinks/ year) by the average human lifespan (72 years), we get: 6,133,800 blinks/ /year 72 years = 441,652,800 blinks in lifetime. This represents the estimated total number of blinks an average human would make over the course of their life.
We find that all three answers are close. However, Plato answer delivers the most rigorous and structured reasoning process, leveraging logic dependency ensure coherence and transparency. For SoT answer, although the value makes sense, doesn't 't have a cohesive explanation process, which directly retrieves the final results.
Findings:
25


Reviews:
[{"review_id": "3llZuAaQWx", "reviewer": "Reviewer_NYGo", "summary": "This paper presents a novel method for prompt decomposition into sub-tasks. The authors evaluate its performance in terms of speed and quality against direct autoregressive decoding (AR) and Skeleton-of-Thought (SoT). The method features parallel processing capabilities and a queuing architecture for asynchronous sub-task processing, while maintaining awareness of inter-task dependencies. The implementation includes an optimization using a global KV cache for the attention layer. Experimental results demonstrate Plato's superior performance across multiple metrics compared on a selection of four LLMs.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The global KV cache is an interesting optimization approach.\n- The paper is well structured and follows a logical flow.\n- The experimental setup is designed well and thoroughly explained.\n- The \"Related Work\" section provides comprehensive context and embeds the paper in the research environment. \n- The proposed method demonstrates significant improvements over both the AR and SoT approaches on the mentioned benchmark datasets.", "reasons_to_reject": "- The name \"Plato\" (derived from \"Plan to Efficiently Decode\") appears somewhat arbitrary and could benefit from a more intuitive justification.\n- In the \"Design of Plato\" section:\n  - The pipeline architecture description contains redundant explanations of the concurrent approach.\n  - The prompting strategy for problem decomposition deserves more detailed discussion. \n  - Better referencing to appendix. The prompt templates should be integrated into the main body of the paper with comprehensive explanations, rather than being demoted to the appendix.\n- Some smaller literal and  grammatical mistakes.\n- The experimental analysis would benefit from addressing limitations such as the small sample size. Maybe, including confidence intervals or a discussion of reproducibility.\n- The consideration of the complexity of the prefill computation needs more explanation. O(N^2) in line 271 is not so clear to me. I also was wondering about prefill for nested graphs.", "questions_to_authors": "Figure 3: please move (outside?) and change the legend; it's really hard to understand.", "extra_scores": {}}, {"review_id": "1RTsUGdqBr", "reviewer": "Reviewer_qy58", "summary": "The core idea of structuring parallel decoding through semantic-aware dependency graphs is intuitive and addresses a known limitation in existing approaches like SoT. The combination of planning with logical/causal relationships and efficient decoding is interesting. This paper is well written.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper co-designs planning with system optimization to improve efficiency, while demonstrating better answer quality than other baselines.\n2. The motivation of the proposed method is well articulated and the paper is well written.", "reasons_to_reject": "1. Coding and math are two important tasks, but the current framework doesn't perform well on these two types of tasks.\n2. The planning should be highly related to the models and the quality of planning should be very important in order to apply Plato. Plato mainly does experiments on two instruction tuning datasets Vicuna and WizardLLM. It's skeptical how Plato can solve more complex and realistic tasks that need more complex reasoning.", "questions_to_authors": "1. What is the principle to select these two datasets for the experiment? Is there any reason except that there are different categories?", "extra_scores": {}}, {"review_id": "ohNCtCjtdz", "reviewer": "Reviewer_JH4e", "summary": "The authors introduce “Plato”, a novel approach that integrates algorithmic and system-level innovations for semantic-aware parallel decoding. Plato utilizes LLMs to construct a dependency graph that captures logical and causal relationships among sub-problems. This enables concurrent decoding of independent nodes while maintaining answer coherence and quality. In addition, to further optimize efficiency, their approach pipelines the planning and decoding stages, employs a global context cache and carefully designs node prompts to maximize key-value (KV) cache reuse and reduce computational overhead. Empirical evaluations show promising results against standard autoregressive decoding and against Skeleton-of-Thought (SoT), also a parallel decoding technique that enhances efficiency by breaking prompts into sub-problems for simultaneous processing.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors introduce a novel approach for decomposing prompts into sub-problems that carefully preserves the semantic dependencies between them. Unlike methods such as Skeleton-of-Thought (SoT), which treat each sub-problem as entirely independent and therefore risk fragmenting the overall coherence of the response, their method explicitly models logical and causal relationships among sub-components. This semantic-aware design allows for parallel processing without sacrificing the consistency or quality of the final answer.\n\n- By maintaining these dependencies, the approach not only avoids the degradation in answer coherence seen in existing parallel decoding strategies but also enhances computational efficiency through selective concurrency. Empirical results demonstrate that this method achieves superior answer quality compared to prior models, highlighting its effectiveness in balancing performance and accuracy.", "reasons_to_reject": "From my perspective, this work provides substantial value to the research community. I believe it merits acceptance.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s0wpzir6YG", "reviewer": "Reviewer_5cB7", "summary": "This work point out that the current parallel decoding methods divide the problem to subproblem for parallel processing. This work suggests that such parallel deocing may be harmful for the answer because the subproblems are solved independently. Thereofore, this work propose Plato  for semantic-aware parallel decoding.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This work proposes to build a relation for the subproblems to improve the performance, which mitigation the limitation of previous parallel decoding.\n* The work conducted a sufficient experiment to prove the effectiveness of Plato. This work evaluates Plato with different datasets, different models. These experiments support the effectiveness of Plato.\n* The paper is easy to read.", "reasons_to_reject": "* The Plato needs more time because some decoding may have to wait for another.\n* There may be more datasets used for evaluation. For example, there is no math dataset evaluation.\n* The work maximum model size 70B, which may be larger for further validation.", "questions_to_authors": "N/A", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents Plato, a training-free method for efficient LLM inference using semantic-aware parallel decoding. It builds a dependency graph to preserve logical structure and applies system optimizations to improve speed and output quality.

Reviewers initially found the idea promising and the experiments solid, though they raised concerns about performance on math and coding tasks, some unclear technical details, and generalization. The authors provided detailed responses and additional experiments, which addressed most concerns.

Overall, I recommend acceptance. The method is practical and well-supported. The final version should clearly state limitations and include more implementation details in the main text.
Decision: Accept