Title: Parallel Scaling Law for Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models

Scaling Law
Abstract: It is commonly believed that scaling language models should commit a significant space or time cost, by increasing the parameters (parameter scaling) or output tokens (inference-time scaling). We introduce another and more inference-efficient scaling paradigm: increasing the model's parallel computation during both training and inference time. We apply $P$ diverse and learnable transformations to the input, execute forward passes of the model in parallel, and dynamically aggregate the $P$ outputs. This method, namely parallel scaling (ParScale), scales parallel computation by reusing existing parameters and can be applied to any model structure, optimization procedure, data, or task. We theoretically propose a new scaling law and validate it through large-scale pre-training, which shows that a model with $P$ parallel streams is similar to scaling the parameters by $\mathcal O(\log P)$ while showing superior inference efficiency. For example, ParScale can use up to 22$\times$ less memory increase and 6$\times$ less latency increase compared to parameter scaling that achieves the same performance improvement. It can also recycle an off-the-shelf pre-trained model into a parallelly scaled one by post-training on a small amount of tokens, further reducing the training budget. The new scaling law we discovered potentially facilitates the deployment of more powerful models in low-resource scenarios, and provides an alternative perspective for the role of computation in machine learning. Our code and 67 trained model checkpoints are publicly available at https://github.com/QwenLM/ParScale and https://huggingface.co/ParScale.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Parallel Scaling (PARSCALE), a novel scaling strategy for language models that aims to increase parallel computation during both training and inference without significantly increasing the number of model parameters. The core idea is to execute multiple parallel streams of computation with dynamic aggregation of their outputs. The authors propose a theoretical scaling law and demonstrate through large-scale pretraining that PARSCALE achieves performance comparable to parameter scaling while offering substantial reductions in memory usage and latency. This makes the method particularly appealing in resource-constrained environments.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. Innovative Scaling Strategy: PARSCALE presents a fresh approach to model scaling by increasing computational parallelism instead of parameter count.\n2. Solid Theoretical Foundation: The proposed scaling law is well-formulated and empirically validated, providing strong theoretical support for the method.\n3. Efficiency Gains: PARSCALE demonstrates significant improvements in memory and latency efficiency, which is particularly beneficial for deployment on edge devices.\n4. Cost-Effective Training: The two-stage training strategy effectively reduces training costs, making large-scale model training more accessible.\n5. Strong Empirical Results: The experimental results are compelling, especially for reasoning-heavy tasks such as code generation and mathematical problem solving.\n5. Strong Experimental Results: The paper presents compelling experimental evidence, showing significant improvements in performance for reasoning-intensive tasks, particularly in code generation and math.\n\n### Weaknesses\n1. Scalability on Larger Models: The paper lacks an in-depth evaluation of PARSCALE’s scalability on very large models and across diverse architectures such as GPT and LLaMA.\n2. Hardware Variability: While memory and latency improvements are reported, the paper does not discuss how these benefits hold across different hardware environments (e.g., GPU types or configurations).\n3. Multi-GPU Scaling: There is no exploration of how PARSCALE performs with varying numbers of GPUs, which is crucial for understanding its hardware scalability.\n\nQUESTIONS:\n1. How does PARSCALE’s scalability and efficiency vary with the number of GPUs in a cluster?\n2. Has PARSCALE been evaluated on more complex models like GPT or LLaMA, and is it compatible with MoE (Mixture-of-Experts) architectures?\n3. Does PARSCALE introduce additional communication overhead in distributed training or inference, and how does this overhead vary with different parallelization strategies (DP, TP, PP) ?\n\nLIMITATIONS:\n1. Lacking analysis of how PARSCALE performs across different GPU setups or hardware configurations.\n2. Lacking analysis of PARSCALE's communication volume and its scalability across different cluster scales or parallelisms.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a brand-new way of thinking about Scaling Laws, focusing more on expanding a model's capabilities (as seen in the loss curve) by increasing the amount of computation rather than the computational space (i.e., the number of model parameters).\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThis paper dedicates significant resources to discussing the Parallel Scaling Law, which I believe is a very promising approach. Recently, the decoding and training speeds of LLMs are often bottlenecked by communication rather than raw computation speed. This makes scaling up the amount of computation particularly important, especially in a context where GPU computational performance is growing rapidly while communication bandwidth and GPU memory remain significant constraints. I was impressed by the clear and detailed experiments in this paper that introduce this novel scaling method.\n\n\n**Weaknesses**\n\nThe paper lacks a more in-depth comparison with methods like Self-consistency. Although Self-consistency is, to some extent, a form of sample-level scaling, it has shown benefits in many areas, including reasoning and question answering. On the other hand, when considering a universal method like Universal Self-Consistency for Large Language Model Generation, I find it difficult to think of a generative AI scenario where the Parallel Scaling Law could be applied, aside from real-time streaming generation.\n\nThe paper explores, at most, a scenario with P=8. As far as I know, methods like Self-consistency often have an upper limit; their performance growth slows down significantly once the number of samples reaches around 1000. I believe it is too early to determine the curve's trend for the Parallel Scaling Law with P=8.\n\nFrom a cost perspective, the computation in this paper is equivalent to P x L x T, whereas for inference scaling law, it is L x T'. When the computational cost at inference time is comparable—for instance, generating 1000 words with P=8 versus generating 8000 words with P=1 (the latter might have a longer wait time, but I believe this setup is a reasonable approximation when considering resource consumption and throughput in high-concurrency scenarios)—I am curious which would perform better. Considering that in the MATH dataset, the Qwen3-235B-A22B model improved from a base of 71.84 to 98 through inference scaling law, or that the same Qwen3-235B-A22B model, using a \"thinking\" mode, improved its MATH score from 91.2 (non-thinking) to 98, and its CodeForces score from 1387 to 2056, I think it's difficult for the Parallel Scaling Law, based on Table 4, to demonstrate this level of performance improvement.\n\nI am not sure if the scaling method proposed in this paper is effective enough, and the paper itself lacks corresponding comparative experiments. For example, if I use a baseline model with P=1, I could use a Dropout-like approach where I input the same P=8 input texts during each forward pass and then, in the final layer of the decoding process, merge these eight parallel Hidden States (e.g., by averaging them) to predict the next new token. This method requires no training, but I believe it could still yield similar results.\n\nAs another example, one could also test this by directly evaluating a Mixture of Experts (MoE) model. For instance, the DeepSeek-V3/R1 model activates 8 experts per token. Therefore, one could sequentially activate P=1, P=2, P=4, P=8, P=16, P=32 experts, which would also allow for scaling without the need for training.\n\nQUESTIONS:\n1. Under current application conditions, can Parallel Scaling demonstrate a sufficient performance advantage over MoE and Inference-Time Scaling? If not now, is there hope for it in the future?\n\n2.  Under current conditions, is the Parallel Scaling method robust enough? Compared to sample-level scaling like self-consistency, and considering the method's huge resource consumption (even just in post-training), is the proposed prefix extension more cost-effective than no-training alternatives (like using Dropout) or using MoE for parallel scaling?\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel scaling paradigm for language models called Parallel Scaling (PARSCALE), which serves as an alternative to traditional parameter scaling and inference-time scaling. The core method involves applying P diverse, learnable transformations to an input, processing these P versions in parallel through a single shared-weight model, and then dynamically aggregating the outputs to produce a final result. This approach effectively scales the amount of parallel computation during both training and inference by reusing existing model parameters, thereby avoiding the significant space or time costs associated with conventional scaling methods.\n\nThe primary contribution is the proposal and empirical validation of a new \"parallel scaling law,\" which finds that scaling computation by P parallel streams is analogous to scaling the model's parameters by a factor of O(log P). The authors demonstrate that for an equivalent performance gain, PARSCALE is substantially more efficient during inference, showing significant reductions in memory and latency increases compared to parameter scaling. Furthermore, the paper introduces a practical two-stage training strategy to mitigate high training costs and shows that PARSCALE can be effectively applied to off-the-shelf models, enabling dynamic adjustment of model capability at deployment.\n\nSTRENGTHS AND WEAKNESSES:\nStrength：\n\n* The paper is supported by a comprehensive suite of experiments. The authors have done a thorough job of validating the proposed parallel scaling law and demonstrating the method's effectiveness across a wide range of model sizes, datasets, and training regimes.\n* The proposed method is elegant in its simplicity and directness. By promoting parameter reuse, it effectively improves the compute-to-memory-access ratio, a critical bottleneck for LLM inference. This characteristic makes the approach highly practical and gives it significant potential for deployment in resource-constrained scenarios, such as on-device and edge computing.\n\nWeakness：\n\n* Lack of Comparison with Alternative Parallel Architectures: The paper compellingly demonstrates the benefits of PARSCALE's \"end-to-end\" parallel design. However, the analysis could be strengthened by comparing it against other potential parallel computation strategies. For instance, what would be the effect of introducing information exchange between the P streams at intermediate layers? A more tightly-coupled design could involve expanding hidden states into P branches within each layer, performing parallel computations, and then fusing them before the next layer. A discussion or empirical comparison—analyzing both model performance and hardware efficiency (e.g., latency under different batching scenarios)—would help clarify whether PARSCALE's approach of P independent forward passes is inherently more advantageous than these more integrated alternatives.\n* Absence of a \"Brute-Force\" Upper Bound: The paper compares PARSCALE to parameter-scaled models with better inference efficiency. However, a crucial baseline is missing: a standard dense model that is P-times wider, which would have a roughly equivalent computational budget (FLOPs) to a PARSCALE model with P streams. While this \"wide\" model would be highly inefficient due to its low compute-to-memory-access ratio, including it as a theoretical \"upper bound\" would be highly informative. It would help quantify the performance trade-off inherent in reusing parameters versus simply adding more, providing a clearer picture of how effectively PARSCALE closes the gap with a brute-force scaling approach for a given computational cost.\n\nQUESTIONS:\n* Clarification on \"Similar Effects as Parameter Scaling\": The paper claims that PARSCALE has \"similar effects as parameter scaling.\" I would appreciate clarification on this point. Does this mean that for an equivalent increase in total computation (FLOPs), scaling P yields a comparable performance improvement to scaling parameters? Or does it primarily mean that the performance improvements from scaling P can be modeled by a scaling law with a similar mathematical form (i.e., a power law) to that of parameter scaling?\n* Optimizing KV Cache Memory Efficiency: The PARSCALE approach can be viewed as increasing the effective batch size of computation over a shared set of parameters, which successfully improves the compute-to-memory-access ratio for the model weights. However, as implemented, the KV cache size scales linearly with P, which could become the next memory bottleneck, especially for long contexts. Have the authors considered or explored strategies to optimize this? For example, could the P streams potentially share or contribute to a common KV cache, particularly for the initial shared context tokens, to further improve memory efficiency?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes ParScale, a scaling strategy that improves language-model performance by re-using the same parameters for P parallel forward passes on differently-transformed inputs via learnable prefixes and then aggregating the outputs with a lightweight MLP. Extensive pre-training experiments (~10^6 GPU-h) on 42B and 1T tokens confirm a log-linear scaling law: P-way parallel computation yields the same loss reduction as multiplying parameter count by O(log P), while adding only ~0.2 % parameters and incurring modest memory/latency overhead. A two-stage training recipe, i.e., pre-train normally on 1T tokens, then PARSCALE-fine-tune on 20B (2%), eliminates most of the extra cost and allows off-the-shelf models (e.g., Qwen-2.5) to be retro-fitted via parameter-efficient fine-tuning, supporting dynamic P-switching at inference.\n\nThis work provides both theoretical grounding and compelling empirical evidence that parallel computation can substitute for parameters, especially on reasoning-heavy benchmarks (code/math). \nCompared with dense scaling, MoE, or inference-time scaling, PARSCALE is architecture-agnostic, training-time compatible, and memory-efficient, which is highly attractive for edge deployment.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n* Strong empirical grounding, with 10^6 GPU-h scaling-law experiments yielding an R² ≈ 0.998 fit that cleanly isolates the effect of parallel computation versus parameters.\n* The proposed two-stage training (1 T normal + ParScale) can still give large downstream gains.\n* ParScale matches a 4.4B dense model with 22× less memory and 6× less latency at batch 1, which could potentially address resource-constrained deployment.\n\n**Weaknesses**\n\n* Even with the two-stage recipe, ParScale multiplies FLOPs by P during fine-tuning and still doubles total compute for P=8; this burden grows linearly with P and could outweigh benefits at larger scales.\n* All latency/memory claims assume batch ≤8; at server-scale batch sizes the memory bottleneck evaporates and ParScale's compute overhead dominates, possibly erasing its edge over dense scaling.\n\nQUESTIONS:\n1. Could you elaborate more on the details of how to conduct prefix tuning as input transformation.\n2. In Table 4, can the authors add the actual number of FLOPs for the baseline and ParScale, I'm curious about the results under fair FLOPs comparisons.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-12"}