Parallel Scaling Law for Language Models
Authors: Mouxiang Chen
Binyuan Hui
Zeyu Cui
Jiaxi Yang
Dayiheng Liu
Jianling Sun
Junyang Lin
Zhongxin Liu
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
Scaling Law
Primary area: deep_learning

Abstract: It is commonly believed that scaling language models should commit a significant space or time cost, by increasing the parameters (parameter scaling) or output tokens (inference-time scaling). We introduce another and more inference-efficient scaling paradigm: increasing the model's parallel computation during both training and inference time. We apply $P$ diverse and learnable transformations to the input, execute forward passes of the model in parallel, and dynamically aggregate the $P$ outputs. This method, namely parallel scaling (ParScale), scales parallel computation by reusing existing parameters and can be applied to any model structure, optimization procedure, data, or task. We theoretically propose a new scaling law and validate it through large-scale pre-training, which shows that a model with $P$ parallel streams is similar to scaling the parameters by $\mathcal O(\log P)$ while showing superior inference efficiency. For example, ParScale can use up to 22$\times$ less memory increase and 6$\times$ less latency increase compared to parameter scaling that achieves the same performance improvement. It can also recycle an off-the-shelf pre-trained model into a parallelly scaled one by post-training on a small amount of tokens, further reducing the training budget. The new scaling law we discovered potentially facilitates the deployment of more powerful models in low-resource scenarios, and provides an alternative perspective for the role of computation in machine learning. Our code and 67 trained model checkpoints are publicly available at https://github.com/QwenLM/ParScale and https://huggingface.co/ParScale.

Parallel Scaling Law for Language Models
Mouxiang Chen 2: Binyuan Hui Zeyu Cui', C Yang", Dayiheng Liu?, Jianling ,un', Junyang Lin?, Zhongxin Liu 'Zhejiang University, *Qwen Team, Alibaba Group fchenmx, 1iu caie,,llll__x___l (u__xx@ejju edu. en, binyuan. hby@al bbaba-innc inn..ccommmmmmmccccccccc com
Abstract
It is commonly believed that scaling language models should commit a significant space or time cost, by increasing the parameters (purameter scaling) or output tokens (inference-time scaling). We introduce another and more inference-efffcient scaling paradigm: increasing the model's s parallel computation during both training and inference time. We apply /' diverse and learnable transformations to the input, execute forward passes of the model parallel, and dynamically aggregate the P outputs. This method, namely parallel scaling (PARSCALE), scales parallel computation by reusing existing parameters and can be applied to any model structure, optimization procedure, data, or task. We theoretically propose scaling law and validate it through large-scale pre-training, which shows that a model with P parallel streams is similar scaling the parameters by Ollog P) while showing superior inference efficiency. For example, PARSCALE can use up 22x less memory increase and 6x less latency increase compared to parameter scaling that achieves the same performance improvement. It can also recycle off-the-shelf pre-trained model into parallelly scaled one by post-training on a small amount of tokens, further reducing the training budget. The new scaling law we discovered potentially facilitates the deployment of more powerful models in low-resource scenarios, and provides an alternative perspective for the role of computation in machine learning. Our code and 67 trained model checkpoints are publicly available at https: https:://giii //gistu///gtub... com/ Btte//onnLMMPaaaaaa and/httts::/hugginge
co/ParScale
Introduction
Recent years have witnessed the rapid scaling of large language models (LLMs) 10 narrow the gap towards Artificial General Intelligence (AGI). Mainstream efforts focus on parameter scaling [40]. practice that requires substantial space overhead. For example, DeepSeck- V3 [5]] scales the model size up to 672B parameters, which imposes prohibitive memory requirements for edge deployment. More recently, researchers have explored inference-time scaling [65] to enhance the reasoning capability by scaling the number of generated reasoning tokens. However, inference-
64 55
to
scaling is limited to certain scenarios and necessitates specialized training data (20 68 and typically imposes significant time costs. For example, Chen et al. [13] find that the most powerful models can generate up 900 reasoning tokens for trivial problems like "2+3=?" This motivates the question: Is there a universal and efficient scaling approach that avoids excessive space and time costs?
We draw inspiration from classifier-free guidance (CFG) 34 widely used trick during the inference phase of diffusion models [35), with similar concepts also developed in the NLP community [75] 50). Unlike traditional methods that use a single forward pass, CFG utilizes two forward passes during inference: first performs a normal forward pass to obtain the first stream of output, then perturbs
Work is partially done during internship in Qwen Team, Alibaba Group.
'Corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Parameter Scaling
Inference- Scaling
Model
Model
Reasoning
Tokens
Next Token
Input
Input
Taken
High Space Cost!
High Time Cost!
Parallel Scaling
Learnable Transformation
Parallel Forward
Learnable Aggragation
Model
Input
Next
Token
Input
(1) Illustration of Three Scaling Approaches
Number of Parallel Streams
x
*
*
Parameter Scaling
>+
Parameter Scaling
Scaling Law:
10
Parallel Scaling
Parallel Scaling
1.1 10"
1.6B
1.6B
Loes 0.7
Parameters
(0.4log
1.08-
1.08
0.58
106-
1.06
LOss
2.8B
LO55
2.8B
112
1.1B
1:04
1.04
LO55
1:10
7x less memory
4.48
.1.7x
4.48
102
1,02
2.8B
increase
latency
4.4B
increase
-45%
parameters
1000
1.0
0.98
000 085
1.5
Parameters (Non-Embedding) (2) Parallel Scaling Law
Latency (s) (3) Scaling with Inference Space and Time Cost
GPU Memory (GB)
Figure I: (1) Illustrations of our proposed parallel scaling (PARSCALE). (2) Parallel scaling laws for pre-training models on 42B tokens from Stack-V2 (Python subset). (3) Loss scaling curve with inference cost. Results are averaged from batch size E {1, 2, 8} and input + output tokens {128, 256, 512, 1024].
the input (e.g., by discarding conditions in the input) to get second stream of output. The two streams are aggregated based on predetermined contrastive rules, yielding superior performance over single-pass outputs. Despite its widespread use, the theoretical guarantee of CFG remains open question. this paper, we hypothesize that the effectiveness of CFG lies its double computation. We further propose the following hypothesis:
Hypothesis 1.
Scaling parallel computation (while maintaining the nearly constant parameters) enhances the model's capability, with similar effects as scaling parameters.
We propose proof-of-concept scaling approach called parallel scaling (PARSCALE) to validate this hypothesis on language models. The core idea is increase the number of parallel streams while making the input transformation and output aggregation learnable. We propose appending P different learnable prefixes to the input and feeding them parallel into the model. These P outputs are then aggregated into single output using dynamic weighted sum, as shown in Figure I]). This method efficiently scales parallel computation during both training and inference time by recycling existing parameters, which applies to various training algorithms, data, and tasks.
Our preliminary theoretical analysis suggests that the loss of PARSCALE may follow power law similar to the Chinchilla scaling law [36.. We then carry out large-scale scale pre-training experiments on the Stack- V2 (58) and Pile (26 corpus, by ranging P from to 8 and model parameters from 500M to 4.4B. We use the results to fit new parallel scaling law that generalizes the Chinchilla scaling law, as depicted in Figure
shows that parallelizing into streams equates to scaling the model parameters by (log P). Results on comprehensive tasks corroborate this conclusion. Unlike parameter scaling, PARSCALE introduces negligible parameters and increases only little space overhead. It also leverages GPU-friendly parallel computation, shifting the memory bottleneck

---

Table 1: Comparisons of mainstream LLM scaling strategies. We subdivide parameter scaling into
traditional Dense Scaling and Mixture-of-Expee Expert (MoE) Scaling (23] for comparison. Inference- Time Scaling: Enhancing the reasoning ability through large-scale reinforcement learning (RI.) to scale reasoning tokens during inference.
Method
Inference Time
Inference Space
Training Cost
Specialized Strategy
Dense Scaling
Moderate
High
Pre-training only
No
MoE Scaling
Low
High
Pre-training only
Load balancing
Inference- Time Scaling
High
Moderate
Post-training
RL / reward data
Purallel Scaling
Moderate
Moderate
Pre- or Post-training
No
in L.M decoding to a computational bottleneck and, therefore, does not notably increase latency. For example, for 1.6B model, when scaling P = 8 using PARSCALE, it uses 22x less memory increase and 6x less latency increase compared to parameter scaling that achieves the same model capacity (batch size 1, detailed in Section 3.3) Figure (3) illustrates that PARSCALE offers superior inference efficiency.
Furthermore, we show that the high training cost of PARSCALE approach: the first stage employs traditional training with most of the training data, and PARSCALE is applied only in the second stage with small number of tokens. Based this, we train 1.8B models with various P and scale the training data to IT tokens. The results of 21 downstream benchmarks indicate the efficacy of this strategy. For example, when scaling to P 8, it yields 34% relative improvement for GSM8K and 23% relative improvement for MMI.U using exactly the same training data. We also implement PARSCALE on off-the-shelf model, Qwen-2.5 [70), and demonstrate that PARSCALE is effective in both full and parameter-efficien fine-tuning settings. This also shows the viability of dynamic parallel scaling, which allows flexible adjustment of I during deployment while freezing the backbone weights, to fit different application scenerios.
be reduced by two-stage
Table compares PARSCALE with other mainstream scaling strategies. Beyond introducing efficient scaling approach for language models, our research also tries to address a more fundamental question in machine learning: Is model's capacity determined by the parameters or by the computation, and what is their individual contribution? Traditional machine learning models typically scale both parameters and computation simultaneously, making it difficult to determine their contribution ratio. The PARSCALE and the fitted parallel scaling law may offer a novel and quantitative perspective on this problem.
We posit that large computing can foster the emergence large intelligence. We hope our work can inspire more ways to scaling computing towards AGI and provide insights for other areas of machine learning. Our key findings in this paper can be summarized as follows:
Scaling P times of parallel computation is similar to scaling parameters by a ratio of Olog P), and larger models reap greater benefits from PARSCALE.
Reasoning-intensive tasks (e.g., coding math) benefit more from PARSCALE, which suggests that scaling computation can effectively push the boundary of reasoning.
PARSCALE offers superior inference efficiency compared to parameter scaling due to the effective use of memory, particularly suitable for low-resource edge deployment.
The training cost PARSCALE can be significantly alleviated through two-stage training strategy.
PARSCALE remains effective with frozen main parameters for different P. This illustrates the potential of dynamic parallel scaling: switching P to adapt model capabilities during inference.
2 Background and Methodology
Classifier-Free Guidance (CFG) CFG 34 has become a de facto cnfference-time trick in diffusion
models (35], with similar concepts also developed NLP 75| 50). At a high level, these lines of work can be summarized follows: given an input E R4: and trained model So Rd, - Rd. where 0 is the parameter and di, d, are dimensions, we transform x into "bad" version x' based on some heuristic rules (c.g., removing conditions), obtaining two parallel outputs fo fo(x) and fo
3

---

The final output (x( is aggregated based on the following rule:
(I) Here, w 0 is a pre-set hyperparameter. Intuitively, Equation 4 can be seen starting from a "good" prediction and moving w steps in the direction away from a "bad" prediction. Existing research shows that go (z) can perform better than the vanilla fx()) in practice [73].
90 (a: w (x) fo(x'))
Motivation In Equation I x' is simply a degraded version of æ, suggesting that go( does not gain more useful information than fa(x). This raises the question: why is So (=) unable to learn the capability of go (z) during training, despite both having the same parameters? We hypothesize that the fundamental reason lies in ge) having twice the computation fz)... This inspires to further expand Equation
(T into the following form:
(z) (o(o) + w2f0(x2) + .... Wp fo
where P denotes the number of parallel streams. T1,"". Fr P distinct transformations of æt and W1, , up are aggregation weights. We term Equation (2) as parallel scaling (PARSCALE) of the model fe with P' streams. This scaling strategy does not require changing the structure of fe and training data. In this paper, we focus on Transformer language models [86] 10]. and regard the stacked Transformer layers f.).)
Implementation Details and Pivot Experiments We apply Equation 2 in both training and inference time, and perform series of pivot experiments to determine the best input transformation and output aggregation strategies (refer to Appendix [A]. The findings revealed that variations in these strategies minimally affect model performance; the significant factor is the number of computations (i.e., P). Finally, for input transformation, we employ prefix tuning [49] as the input transformation, which is equivalent For output aggregation, we employ a dynamic weighted average approach, utilizing an MLP to convert outputs from multiple streams into aggregation weights. This increases about 0.2% additional parameters for each stream.
using different KV-caches distinguish different streams.
3 Parallel Scaling Law
This section focuses on the in-depth comparison of scaling parallel computation with scaling param- eters. In Section 3.1 we theoretically demonstrate that parallel scaling is equivalent to increasing parameters by a certain amount. In Section 3.2. we validate this with a practical scaling law through large-scale experiments. Finally, in Section 3.3 we analyze latency and memory usage during inference to show that parallel scaling is more efficient.
3.1 Theoretical Analysis: Can PARSCALE Achieve Similar Effects as Parameter Scaling?
From another perspective, PARSCALE can be seen ensemble of multiple different next token predictions, despite the ensemble components sharing most of the parameters. Existing theory in literature finds that the ensembling performance depends 8,57. In this section, we further validate this finding by theoretically proposing a new scaling law that generalizes existing language model scaling laws, and demonstrate that PARSCALE achieve similar effects as parameter scaling.
the diversity of different components
We consider special case that wi w2 = ... 1/p to simplify our analysis. This is degraded version of PARSCAL.F, therefore, we can expect that the full version of PARSCALE is at least not worse than the theoretical results we can obtain (See Appendix A for further numeric comparison). Let (: x) = fe (Ix) denote the next token distribution for the input sequence æ predicted by the i-th stream. Based on Equation (2). the final prediction $(- x) g=(z) is the average across p.. i.e.
p( x) = 1/P >, (: (. x). Chinchilla (36) proposes that the loss L of a language model with N parameters is function of N after convergence. We assume the prediction of each stream adheres to the Chinchilla scaling law, as follows:
Lemma 3.1 (Chinchilla Scaling Law (36)). The language model cross-entropy loss for the i-th stream prediction (with N parameters) when convergence is:
C
E,
S
(3)
N

---

where {A, a.a are some positive constants. E is the entropy of natural text
Based Lemma 3.1 we theoretically derive that after aggregating streams, the prediction follows a new type of scaling law, as follows:
Proposition 1 (Theoretical Formula for Parallel Scaling Law). The loss for PARSCALE (with P streams and N parameters) is
A
a
L . DIVERSITY
+E.
(4)
We define DIVERSITY as:
DIVERSITY = [ ++]1] -1
where p is the correlation coefficient between random variables Ap; and Ap, (i + j), and Aps is the relative residuals for the i-th stream prediction, iee, Ap; = (ralll))-lolllllppg ply x)) is the real next token probability.
Proof for Proposition Iis elaborated in Appendix
B From it, we can observe two key insights:
1. When p 1, predictions across different streams are identical, which point can validate that Equation (4) degenerates into Equation (3). Random initialization on a small number of parameters introduced (i.e., prefix embeddings) is sufficient avoid this situation in experiments, likely due to the impact being magnified by the extensive computation of LLMs.
2. When p * 1, L is inversely correlated to P. Notably, when p 0, residuals are independent between streams and the training loss exhibits a power-law relationship with P (i.e., L x P-1). This aligns with findings in Lobacheva et al. [57]. When p negative, the loss can further decrease and approach zero. This somewhat demystifies the effectiveness of CFG: by widening the gap between "good" input z and "bad" input z', force the model to "think" from two distinct perspectives, which can increase the diversity between the two outputs.
Despite the difficulty in further modeling p. Proposition suggests that scaling P times of parallel computation is equivalent scaling the model parameter count, by factor of "P//'.DIVERSITYY This motivates us to go further, by empirically fitting a practical parallel scaling law to validate Hypothesis [
3.2 Practical Parallel Scaling Laws
Experiment Setup To fit a parallel scaling law in practice, we pre-train Transformer language models with the Qwen-2.5 dense architecture and tokenizer (70) from scratch on the open-source corpus. We primarily focus on the relationship between parallel scaling and parameter scaling. Therefore, we fix the training data size at 42 billion tokens without data repea We introduce the results for more training tokens in the next section, and leave the impact of data scale on the scaling law for future work. Most of our settings follow existing works 63). detailed in Appendix
Our pre- pre-training is conducted on two widely utilized datasets: Stack V2 (Python subset) Pile (26). Pile serves as general corpus aimed enhancing common sense and memorization skills, while Stack-V V2 focuses on code comprehension and reasoning skills. Analyzing PAR PARSCALE across these contexts assess how parameters and computations contribute to different skills.
58 and
Parametric Fitting We plot the results Figure where each point represents the loss a training run, detailed in Appendix We observe that increasing P yields benefits following logarithmic trend. Similar gains are seen when raising P from 1 to 2, 2 to 4, and to 8. Thus, we preliminarily try the following form:
(= +E,
A A
log N=
L= C
5
*Chinchilla also considers the limited training steps. In this paper, we focus on the impact of computation and parameters model capacity and assumes that the model has been trained to convergence. "In Appendix D we test PARSCALE with repeated data on a small-scale corpus, Web 288, and found that PARSCALE helps mitigate the overfitting when data is limited, We leave further exploration on the data-constrainee parallel scaling law for future work.
5

---

Stack-V2-Python
1.20
0.58
P=1
2:15
0.58
P-1
1.15
0.78
P=2
2:10
0.78
P
1.18
*
P P-4
1.1B
P
Loss
1.6B
P=8
2.00
1.B
P=
1.95
2.8B
,.8
E= 0.6912
0.3933
10
4.4B
1.05
E= 1.2080
A=1.974 10
k= k-0.3345
|44
0.1894
== 0.199963
Goodness: R² 0.9978
Goodness: R 0.9987
18 Parameters (Non-Embedding)
18 Parameters (Non-Embedding)
Figure 2: Loss of LLMs scaled parameters and number of parallel streams P trained 42B tokens. Each point depicts the loss from training run. The fitted scaling law curve from Equation (5)
displayed, with annotated fitted parameters (E, A, k, a)) and the goodness of fit R²a
Stack-V2-Pythor
Scaling Computation
Scaliing Computation
2.1
1.10
2,0
P=
1.05
P=4
10009
1.00
P=2
188
0.78
1.68
2.88
4.48
0.78
1.68
2.88
4.48
Scaling
Scaling Parameters
Figure 3: Predicted loss contours for PARSCALE. Each contour line indicates combination of (parameter, P) with similar performance.
where we assume that Pt/a DIVERSITY k log P in Equation 4 based on the finding of the logarithmic trend. (A, k E) are parameters to fit, and we use the natural logarithm (base e). We
follow the fitting procedure from 33 63), detailed Appendix E
Figure 2 iillustrates the parallel scaling law fitted for two training datasets. It shows high goodness of fit (R² up to 0.998), validating the effectiveness of Equation (5). Notably, we can observe that the k value for Stack-V2 (0.39) is higher than for Pile (0.33). Recall that k reflects the benefits of increased parallel computation. Since Stack- V2 emphasizes coding and reasoning abilities while Pile emphasizes memorization capacity, we propose an intuitive conjecture that model parameters mainly impact the memorization skills, while computation mainly impacts the reasoning skills. This aligns with recent findings on inference-time scaling (27). Unlike those studies, we further quartitatively assess the ratio of contribution model performance between parameters and computation through our proposed scaling laws.
Recall that Equation 5 implies scaling equates to increasing parameters by O(N P). suggests that models with more parameters benefit more from PARSCALE. Figure 3 more intuitively displays the influence of computation and parameters on model capacity. As model parameters increase, the loss contours flatten, showing greater benefits from increasing computation.
Downstream Performance Tables 2 and 3 illlustrate the average performance on downstream tasks (coding tasks for Stack-V2-Python and general tasks for Pile) after pre-training, with comprehensive results in Appendix G It shows that increasing the number of parallel streams P consistently boosts performance, which confirms that PARSCALE able enhance the model capabilities and similar to scale the parameters. Notably, PARSCALE offers more substantial improvements for coding tasks compared to general tasks. For example, as shown in Table 2 for P 8 aligns with the 4.4B model, while Table Table3indicate: indicates that such setting performs comparably the 2.8B model on general tasks that focusing on common-sens, memorization.
the coding ability of the 1.6B model
3.3 Inference Cost Analysis
We further compare the inference efficiency between parallel scaling and parameter scaling at equivalent performance levels. Although some work uses FLOPS measure the inference cost

---

Table 2: Average performance (%) on two code generation tasks, HumanEval(+) and MBPP(+), after pre-training on the Stack V2-Python dataset.
Table Average performance (%) on six general am-evvlluation-hane tasks after pre-training on the Pile dataset.
N
0.5B
0.7B
1.IB
1.6B
2.8B
4.4B
0.5B
0.7B
1.1B
1.6B
2.8B
4.4B
P 1
26.7
28.4
31.6
33.9
36.9
39.2
p
49.1
50.6
52.1
53.1
55.2
57.2
P p= 2
30.3
32.4
33.6
37.4
39.4
42.6
P
49.9
51.0
52.4
54.4
57.0
58.5
p
30.1
32.5
34.1
37.6
40.7
42.6
P
50.6
51.8
53.3
55.0
57.8
59.1
P 8
32.3
34.0
37.2
39.1
42.1
45.4
P
50.7
51.8
54.2
55.7
58.1
59.6
1.0
110
1.30
1.30
1.0-
P-2
3.68
1.02
memory
22x less
102
loss latency
45
1.52
1.00
1.03
1.00
1.50
0.98
0.98
0.58
10
0.8
12
1.4
0.5
1.5
25
Momory (GB)
GPU Memory (GB)
Lanency (S)
Latency (5)
(a) batch size
(b) batch size
(c) batch size
(d) batch size
Figure 4: Loss scales on the inference space-time cost, with three parameters (1.6B, 2.8B, and 4.4B) and two batch sizes. Results are averaged from input output tokens E {64, 128, 256, 512]. Blue arrows indicate parameter scaling; gray arrows represent parallel scaling.
argue that this is not ideal metric. Most Transformer operations are bottlenecked by memory access rather than computation during the decoding stage [39]. Some work (such as flash attention (18) incurs more FLOPS but achieves lower latency by reducing memory access. Therefore, we use memory and latency to measure the inference cost, based on the -nnallslssis framework [48). We analyze the inference cost across various inference batch sizes. It is worth mentioning that all models in our experiments feature the same number of layers, differing only in parameter width and parallel streams (detailed in Appendix C). This enables a more fair comparison of the efficiencies.
36 76,
Space Cost Figures ((a) and 4(b) compare the inference memory usage of two scaling strategies, where utilize loss Stack P2yPhn as indicator for model capacity. It shows that PARSCALE only marginally increases memory usage, even with larger batch sizes. This is because PARSCALE introduces negligible amounts of additional parameters (i.e., prefix tokens and aggregation weights, about 0.2% parameters per stream) and increases KV cache size (expanded by times with streams), which generally occupies far less GPU memory than model parameters. As the input batch size increases, the KV cache size also grows; however, PARSCALE continues to demonstrate significantly better memory efficiency compared parameter scaling. This suggests that PARSCALE maximizes the utility of memory through parameter reusing, while parameter scaling employs limited computation per parameter and cannot fully exploit computation resources.
Time Cost Figures[4(c) and (dd) compare the inference time of two scaling strategies. shows that PARSCALE adds minimal latency at smaller batch sizes since the memory bottlenect is converted tothe computation bottleneck. Given that parallel computation introduced by PARSCALE friendly to GPUs, it will not significantly raise latency. As batch sizes increase, decoding shifts from memory to computation bottleneck, resulting in higher costs for PARSCALE, but remains more efficient than parameter scaling up to batch size of 8.
The above analysis indicates that PARSCALE is ideal for low-resource edge devices like smartphones, smart cars, and robots, where queries are typically few and batch sizes are small. Given limited memory resources in these environments, PARSCALE effectively utilizes memory and latency ad- vantages at small batch sizes. When batch size is 1, for 1.6B model and scaling to P = 8 using PARSCALE, it uses 22x less memory increase and 6x less latency increase compared to parameter scaling that achieves the same performance. We anticipate that the future LLMSs will gradually shift from centralized server deployments to edge deployments with the popularization of artificial intelligence. This suggests the promising potential of PARSCALE in the future.
7

---

Table 4: Performance comparison of the 1.8B models after training on IT tokens from scratch using the two-stage strategy. We incorporate recent strong baselines (less than 2B parameters) as a comparison validate that our P 1 baseline is well-trained. The best performance and its comparable performance (within 0.5%) is bolded. Appendix G elaborates the evaluation details.
Average
General
Tokens
Data
General
Math Code
MMLU
WinoGrande Hellaswag OBQA
PIQA
ARC
gemma-3-18
2T
Private 53.4
1.9
14.9
26.4
61.4
63.0
37.8
75.6
56.2
Llama-3.2 3.2-1B
1ST
Private
54.8
4.7
30.1
30.8
62.1
65.7
39.2
75.9
55.3
Qwen2.5-1.50
18T
Private
63.6
52.3
55.8
61.0
65.6
68.0
42.6
76.6
67.9
Smoll.M-1.. M-1.78
IT
Public
57.0
37.9
29.7
61.8
67.3
42.8
77.3
63.3
SmolLM2-11.75
12T
Public
63.3
24.3
41.6
50.1
68.2
73.1
42.6
78.3
67.3
Buseline (P 1)
1T
Public
56.0
25.5
45.6
28.5
61.9
65.0
40.6
75.2
64.8
PARSCALE (P² = 2)
IT
Public
56.2
27.1
47.4
29.0
62.4
64.7
42.0
74.9
64.3
PARSCALE (P-
IT
Public
57.2
30.0
48.6
30.0
63.4
65.9
42.0
75.6
66.3
PARSCALE (P² 8)
IT
Public
58.6
32.8
49,9
35.1
64.9
42.6
76.1
66.0
Math
Code
Tokens
Data
GSM8K
Minerva Math
HumanEval= HvaanEval @ @10
MBPP
MBPP+
GSM8K
+CoT
@1
@ @10
@1
@
@ @0
@1
@ 10
gemma-3-1B
2T
Private
1.8
2.3
1.5
6.7
15.9
6.1
14.6
13.0
29.1
10.8
23.0
Llama-3.2-1B
15T
Private
7.2
1.8
27.4
14.0
25.0
33.1
54.5
43.4
Qwen2.5-1.5B
18T
Private
61.7
67.2
28.1
36.0
62.8
31.1
55.5
61.9
79.6
50.8
68.5
SmolLM-1.7BB
IT
Public
6.4
8.3
3.2
20.1
35.4
15.9
32.3
40.7
66.4
34.7
57.4
SmolLM2-1.7B
12T
Public
30.4
30.8
11.8
23.8
44.5
18.9
37.8
45.2
68.5
36.0
57.9
Baseline (P
Public 28.7
35.9
12,0
26.8
44.5
20.7
38.4
75.9
43.9
62.7
PARSCALE (P=22
IT
Puhlic 32.6
35.6
13.0
26.2
50.0
20.1
42.1
52.9
77.0
45.0
65.6
PARSCALE (P =P=
IT
Public 34.7
40.8
14.5
27.4
47.6
23.8
43.9
55.3
77.0
47.1
66.7
PARSCALE
Public 38.4
43.7
16.4
28.7
50.6
24.4
44.5
56.3
79.1
48.1
67.2
4
Scaling Training Data
Due to our limited budget, our previous experiments on scaling laws focus on pre-training with 42 billion tokens. In this section, we will train 1.8B model (with 1.6B non- embeddinng parameters) and scale the training data to IT tokens, to investigate whether PARSCALE is effective for production-level training. We also apply PARSCALE to an off-the-shelf model, Qwen-2.5 (70 (which is pre-trained on 18T tokens), under two settings: continual pre-training and parameter-efficient fine-tuning (PEFT).
4.1 Two-Stage Pretraining
While PARSCALE efficient for the inference stage (as we discuss in Section 3.3 it still introduces about P times of floating-point operations and significantly increases overhead in the computation- intensive training processes. To address this limitation, we propose two-stage strategy: in the first stage. we use traditional pre-training methods with I trillion tokens; in the second stage, we conduct PARSCALE training with 20 billion tokens. Since the second stage accounts for only 2% of the first stage, this strategy can greatly reduce training costs. This two-stage strategy is similar to long-coe long-context fine-tuning 22]. which also positions the more resource-intedd inteenee- phase at the end. In this section, we will examine the effectiveness of this strategy. Pre-training setup is elaborated in Appendix
Training Loss Figure 5(a) demonstrates the loss curve during our two-stage training. At the beginning of the second phase, the loss for P > 1 initially exceed those for P 1 due to the introduction of randomly initialized parameters. However, after processing small amount of data (0.0002T tokens), the model quickly adapts to these newly introduced parameters and remains stable thereafter. This proves that PARSCALE can take effect rapidly with just little data. We can also find that in the later stages, PARSCALE yields similar logarithmic gains. This aligns with previous scaling law findings, suggesting that our earlier conclusions for from-scratch pre-training - parallelism with P streams equates to O(N log( P)) increase in parameters - also applies to continued pretraining. Additionally, larger P (such as P = 8) can gradually widen the gap compared to smaller P values (such P 4). This demonstrates that parallel scaling also benefit from data scaling.
Downstream Performance In Table we report the downstream performance of the model after finishing two-stage training, across 7 general tasks, 3 math tasks, and 8 coding tasks. It can be

---

Stage 1: normal pre-training
P
Pass@10
(1T takens)
53.0
Performance (%)
78.2
Stage perellel scaling training tokens)
76.4
1.62
50.1
75.4
47.4
73.1
2-
0.07
0.27
0.47
0.67
0.8T
1.0T
Training Tokens
(a) Two-Stage Training
(b) PEFT for Qwen-2.5
Figure 5: (a) Loss for two-stage training, smoothing using an exponential moving average with a weight of 0.95. (b) Code generation performance after fine-tuning Stack-V? (Python), averaged from HumanEval(+)) and M+PP(+). We only fine-tune the introduced parameters (prefix tokens and aggregation weights), with different P sharing exactly the same @wen2.5-3B pre-trained weights.
observed that with the increase of P. the performance presents an upward trend for most of the benchmarks, which validates the effectiveness PARSCALE trained on the large dataset. Specifically, when P increases from 1 to 8, PARSCALE improves by 2.6% on general tasks, and by 7.3% and 4.3% on math and code tasks, respectively. It achieves 10% improvement (34% relative improvement) on GSM8K. This reaffirms that PARSCAL. can more effectively address ceasoning-intensin Moreover, in combination with CoT, achieves about 8% improvement on GSM8K, suggesting that parallel scaling can be used together with inference-time scaling to achieve better results.
tasks.
To validate the applicability of PARSCALE to other training stages and data, use instruction tuning for the base models as detailed in Appendix
4.2 Applying to the Off-the-Shelf Pre- Trained Model
We further investigate applying PARSCALE to off-the-shelf models under settings: continual pre-training and parameter-efficiens fine-tuning (PEFT). Specifically, we use Pile and Stack-V2 (Python) continue pre-training the Qwen-2.5 (3B) model. The training settings remain consistent with Appendix C with the only difference being that we initialize with Qwen2.5-3B weights and adjust the RoPE base to the preset 1,000,000.
The results for full continual pre-training is elaborated in Appendix J We further utilize PEFT to fine-tune the introduced parameters while freezing the backbone. Figure (5b) shows that this strategy can still significantly improve downstream code generation performance. Moreover, this demonstrates the promising prospects of dynamic parallel scaling: switch between different numbers of parallel streams in various scenarios (e.g.. high throughput and low throughput), which enables quick transitions between different levels of model capacities.
deploy the same backbone and flexibly
Related Work
Beyond language modeling, our work can be connected to various machine learning domains. First, scaling computation while maintaining parameters also the core idea inference- time scaling. Second, previously mentioned, PARSCALE can be viewed guidance. Third, our method can be seen as a special case of model ensemble. Lastly, the parallel scaling law explore is generalization of the existing language model scaling laws.
dynamic and scalable classifier- -free
Inference- Time Scaling The notable successes of reasoning models, such as GPT-ol [65] DeepSeck-RI (20), QwQ [71, and Kimi kl.5 [4 have heightened interest in inference-time scaling.
These lines of work 91
59 [02 focus on scaling serial computation to increase the length of
the ofain-of--houglh 191. Despite impressiveness, it results in inefficient inference and sometimes
exhibits overthinking problems [3 g3
Other lines of approaches focus on scaling parallel computation. Early methods such beam search (94), self-consistency [90], and majority voting [T] require no additional training. We also provide an experimental comparison with Beam Search and Majority Voting in Appendix which shows the

---

importance of scaling parallel computing during the training stage. Recently, the proposal-verifier paradigm has gained attention, by employing a trained verifier to select from multiple parallel outputs 19, 95 101. However, these methods are limited to certain application scenarios (i.e., generation tasks) and specialized training data (i.e., reward signals).
More recently, Geiping et al. [27] introduce training LLMs to reason within the latent space and scale sequential computation, applicable to any application scenarios without needing specialized datasets. However, this method demands significant serial computation scaling (e.g., 64 times the looping) and invasive model modifications, necessitating training from scratch and complicating integration with existing trained LLMs.
Classifier-Free Guidance Classifier-Fres Guidance (CFG) stems from Classifier Guided Diffusion [21], which uses an additional classifier to guide image generation using diffusion models [35]. By using the generation model itself as a classifier, CFG [34 further eliminates dependency on the classifier and leverage two forward passes. Similar concepts have emerged in NLP, such as
Coherence Boosting 600, PREADD 66), Context-Aware Decoding (78), and Contrastive Decoding
[50] Recently, Sanchez et al. 775 proposed transferring CFG to language models. However, due
to constraints of human-designed heuristic rules, these techniques cannot leverage the power of training-time scaling 40 and the performance is limited.
Model Ensemble Model ensemble a classic research field in machine learning and is also em- ployed in the context of LLMs [1]. In traditional model ensembles, most ensemble components do not share parameters. Some recent work consider setups with partially shared parameters. For example, Monte Carlo dropout [25] employs multiple different random dropouts during the inference
phase, while BatchEnsemble 993 85 and LoRA ensemble [89] use distinct low-rank matrix factoriza- tions for model weights to differentiate different streams (we also experimented with this technique as input transformation in Appendix AA. Weight sharing 996 is another line of work, where some weights of model are shared across different components and participate in multiple computations. However, these works have not explored the scaling law of parallel computation from the perspective of model capacity. As discuss in Appendix [ we find that the specific differentiation technique had a minimal impact, and the key factor is the scaling in parallel computation.
Scaling Laws for Language Models Many researchers explore the predictable relationships between LLM training performance and various factors under different settings, such as the number of parameters and data [33 40 36 19
24, data repetition cycles [63 32]. data mixing 197 69). and fine-tuning [100]. By extending the predictive empirical scaling laws developed from smaller models to larger models, we can significantly reduce exploration costs. Recently, some studies have investigated the scaling effects during inference (76). noting a log-linear relationship between sampling number and performance work extends the Chinchilla scaling law (36) by introducing the intrinsic quantitative relationship between parallel scaling and parameter scaling. Existing literature has also identified a pawer-law relationship between the number of ensembles and loss in model ensemble scaling laws [56]. which can be considered a special case of Proposition when p p= O.
80 But they are limited certain application scenarios. Our
Conclusions
In this paper, we propose new type of scaling strategy, PARSCALE, for training LLMs by reusing existing parameters for multiple times to scale the parallel computation. Our theoretical analysis and extensive experiments propose a paraliel scaling law, showing that a model with N parameters and Px parallel computation can comparable to a model with ON log P) parameters. We scale the training data to ITT tokens to validate PARSCALE in the real-world practice based on the proposed two-stage strategy, and show that PARSCALE remains effective with frozen main parameters for different P. We also demonstrate that parallel scaling is more efficient than parameter scaling during the inference time, especially in low-resource edge scenarios. We elaborate the discussion and future work in Appendix K
10

---

Acknowledgement
This research/projeet was partially supported by the National Natural Science Foundation China (No. 62202420) and Zhejiang Provincial Natural Science Foundation of China (No. LZ25F020003), The authors would like thank Jiquan Wang, Han Fu, Zhiling Luo, and Yusu Hong for their early idea discussions and inspirations, well as Lingzhi Zhou for the discussions efficiency analysis.
References
[1] Loubna Ben Allal, Anton Lozhkov, and Elie Bakouch. Smollm blazingly fast and remarkably powerful. https::/huggigggaaeeeeee co/ /wlo//lo/////////o///oo smollm 2024.
[2] Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydliček, Agustín Piqueres Lajarín, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Suan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathiuu Morlon, Colin Raffel, Leandro von Werra, and Thomas Wolf. Smollm2: When smol goes big data-centric training of a small language model. arXiv preprint ar arXiv:2502.02731 2025. UtLpt:ss://artt org/ abs/2502..n 02737
[3] Jacob Austin, Augustus Odena, Maxwell Nye. Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. URLhttps
//arxiv.or org/ attpa////////../// 07732
[4] Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra. Cosmopedia, February 2024. URL UtL.tt httph:tt////t http:///hggggaa//c/e/e co/datasets/
[5] Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. BitFit: Simple parameter-efficient fine- tuning for transformer--at masked anguagg-moddde
Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1-9, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 1226553/1/2022.acllsssl https://aclanthologg org/2022. 2022. acl short.
URL
[6] Stella Biderman, Hailey Schoelkopf. Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, Anthony DiPofi, Julen Etxaniz, Benjamin Fattori, Jessica Zosa Forde, Charles Foster, Jeffrey Hsu, Mimansa Jaiswal, Wilson Y. Lee, Haonan Li, Charles Lovering, Niklas Muennighoff, Ellie Pavlick, Jason Phang. Aviya Skowron, Samson Tan, Xiangru Tang, Kevin A. Wang, Genta Indra Winata, François Yvon, and Andy Zou. Lessons from the trenches reproducible evaluation of language models. arXiv preprint arXiv:2405. 14782, 2024. URIphttss h//htttss///axiiiii //arxiv. org/ abs/ 2405. 14782
[7] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432-7439 AAAI Press, 2020. doi: 00.1609/AAALV34100 66299. URL URL
aaai. v34105.0.... 6239
[8] Leo Breiman. Random forests. Mach. Learn., 45(1):5 October 2001. ISSN 0885-6125. doi: 10. 10.1033/A100009334444222 URL http:://///aa/ oaw.lorrg11.. 1023/A: 1010933443344
[9] Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoe V. Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.217887 2024. URI. https:ttp/// htttp:://arxiv. wwwp.xiw.../bb////00000
21787

---

[10] Tom B. Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger: Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv preprint arXiv:2005. 2020. URL htttps:/aare htwpv/axvvvvv.a org/
14165
[11] Lingjiao Chen, Jared Quincy Davis, Boris Hanin, Peter Bailis, Ion Stoica, Matei Zaharia, and James Zou. Are more LLM calls all you need? towards the scaling properties of compound Al systems. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL httttt::///ppenrriiiieie wwtprriiwwiineeeeeeeeeeeeeeeeeee ii 1 06RRLgx
[12] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray. Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Paviov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Tang, Igor Babuschkin, Suchir Balaji. Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, llya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. arXiv preprint ar arXiv:2107.0337 177777777334 2021. URL httthttts//aaatt wwwpexivvv.rrg/// abs/ 2107 03374
[13] Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang. Zhaopeng Tu, Haitao Mi, and Dong Yu. Do not think that much for 2+3=? on the overthinking of ol-like Ilms. arXiv preprint 21187
arXiv:2412.21187 2024. URL wwtpt:ttt///atttt
[14] Zhijun Chen, Jingzheng Li, Pengpeng Chen, Zhuoran Li, Kai Sun, Yuankai Luo, Qianren Mao, Dingqi Yang. Hailong Sun, and Philip S. Yu. Harnessing multiple large language models: A survey on Ilm ensemble. arXiv preprint arXiv:2502. 18036, 2025. (RIttttps
wwwxiii.org/abb//// 2200/ 18036
[15] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint arXiv:1803.054: 2018. URL /httts:///axiiii org abs/ 1803 05457
[16] Aaron Clauset, Cosma Rohilla Shalizi, and M. E. J. Newman. Power-law distributions in empirical data. SIAM Review, 51(4):661-703, 2009. doi: 10.1137/070710111 URL
1137/00070/10
[17] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.1416. 2021. htttpp:://// www.a/iwivooggab//////aa 14166
[18] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. arXiv preprint arXiv:2205.14135. 2022. URL htt.hhttt////xxivv org/ http:ab.//....111/ 14135
[19] DeepSeek-AN Deepseek Ilm: Scaling open-source language models with longtermism. arXiv
preprint arXiv:2401.02954 2024. URL https://: org/ atsp//bbs/4411111 02011.029544
[20] DeepSeek-Al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement
learning. arXiv preprint arXiv:2501. 2025. URL htt.ttt:://arzi..rgg//bbba////
12948
12

---

[21] Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs on image synthesis. In A. Beygelzimer, Y. Dauphin, P. Liang, and Wortman Vaughan, editors. Advances in Neural Information Processing Systems, 2021. URI. httpss:///pppereeieeee
[22] Yiran Ding. Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending Ilm context window beyond 2 million tokens. arXiv preprint arXiv:2402.135:
2024. URL hwtpthhttt///aateee .wg.ziii..rr.ooooo aasp/////222220 13753
[23] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: scaling trillion parameter models with simple and efficient sparsity. J. Mach. Learn. Res., 23(1), January 2022.
ISSN 1532-4435.
[24] Elias Frantar, Carlos Riquelme Ruiz, Neil Houlsby, Dan Alistarh, and Utku Evei. Scaling laws for sparsely-connected foundation models. In The Twelfth International Conference on Learn- ing Representations, 2024. httpst/s: /ttpb:::///pperreieee meet/Eorun?idd11K22ZNN
[25] Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings The 33rd International Conference on Machine Learning, volume 48 Proceedings of Machine Learning Research, pages 1050-1059, New York, New York, USA, 20 22 Jun 2016. PMLR. URL https::t https::/proocedding.. mlr
htt.:iigs....ii...es/sese///m press/ htmlmml.l
[26] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe. Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.0000 2021.
www.rxiiv...rrraaee abspg//as/21011.. 00027
[27] Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling test-time compute with latent reasoning: recurrent depth approach. arXiv preprint urXiv:2502.05171,
2025, htt.:/ttp:://axxiiorrr//baaaae 200.b///22220517111 05171
[28] Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. Openwebtext corpus. http:///kyylon0077.gttbbbbb http://ghhh..oaoppop/oooooooooro
2019.
[29] Zeyu Han, Chao Gao, Jinyang Liu, Jeff' Zhang, and Sai Qian Zhang. Parameter-efficient fine- tuning for large models: A comprehensive survey, Transactions on Machine Learning Research, 2024. ISSN 2835-8856 URL hatppss//ppparrrreeeeeee//rrrres
[30] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. URL httptttss./ppeeeveee...ttt id=d7KBjmIIGml
[31] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
[32] Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan. Tristan Hume, Scott Johnston, Ben Mann, Chris Olah, Catherine Olsson, Dario Amodei, Nicholas Joseph, Jared Kaplan, and Sam McCandlish. Scaling laws and interpretability learning from repeated data. arXiv preprint arXiv:2205. 10487, 2022. URL http::: org/ htw.b/205.... 10487
[33] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00400 2017. URL https://arxiv
org/ wwwa////2.... 00409
[34] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv:2207.12598 2022. /ttp::t:////arii.oorr
arXiv preprint
12598
13

---

[35] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6840-6851. Curran As- sociates, Inc., 2020. URL httt:::///po neurips cctpppee_fille//peeeeeeeeeeeeeee pdf
[36] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models. arXiv preprint arXiv: 2022. https://aaaxx org/ hwtgg//b///// 2203. 15556
[37] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. httpp:tsss//openree set.fforumiid-nZeWeeees
[38] Shengding Hu, Yuge Tu, Xu Han, Ganqu Cui, Chaoqun He, Weilin Zhao, Xiang Long, Zhi Zheng. Yewei Fang, Yuxiang Huang, Xinrong Zhang, Zhen Leng Thai, Chongyi Wang. Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, dahai li, Zhiyuan Liu, and Maosong Sun. MiniCPM: Unveiling the potential of small language models with scalable training strategies. In First Conference on Language Modeling, 2024. URL htttps::/openrevieer net/ wtt.ffrrmmm-d-22222000
[39]
Andrei Ivanov, Nikoli Dryden, Tal Ben-Nun, Shigang Li, and Torsten Hoefler. Data movement is all you need: A case study on optimizing transformers. In A. Smola, A. Dimakis, and I. Stoica, editors, Proceedings of Machine Learning and Systems, volume 3, pages 71 11 732, 2021. URL URLAttpss://rroceeeing..... .aa.aaggsmmsssss....... http:///pppeee.ee//a///////////0 bo6666006632225115644117666----3 pdf
[40] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint (xxiv:2001.08611, 2020. URIpttps: http///xxivv.rrr//e /abg/b//////
[41] Kimi Team. Kimi k1.5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501. 12599, 2025. URL httpt:tt//s/xxi..... http://awww...rrr/////5../5/0 12599.
[42] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedingss 2015. URL attt:://arxis: orggeaaaaao abs/ wwwp////......... 111111112100 6980
[43] Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE: Large-scale ReAding comprehension dataset from examinations. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785-794, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/11/D17-100882 URL https http:hhhooooo.ooo//7--110000
[44] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint MeXivv/909) 1942, 2019. URL wtt.:tttt///rrttt. org/ abs/ 11942
[45] Minh Le, Chau Nguyen, Huy Nguyen, Quyen Tran, Trung Le, and Nhat Ho. Revisiting prefix-tuning: Statistical benefits of reparameterization among prompts. In The Thirteenth International Conference on Learning Representations, 2025. https://:::// htttt:::///peereviee net/ set/Eoorum"id=0)TS5aFXX22
[46] Brian Lester, Rami Al-Rfou, and Noah Constant, The power of scale for parameter-efficient prompt tuning. In
Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau
14

---

Yih, editors. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational I Linguistics. doi: 10.18653/v 12655///1222...mmmmmmaaiall https://aclanthoolpg. org.ggg...r.r/o. org/ 2221. emmlp-main. 243/
URL
[47] Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. arXiv preprint 22X.:::200..48866 2022. URL https
//arxiv. abs wwwpg//bs/220..../ 221./22266..1 14858
[48] Cheng Li. Lim-analysis: Latency and memory analysis of transformer models for training and inference. https: pttps://githube com c1199/11m 1at99/mmmmaaaalliiiia.aaa 2023.
[49] Xiang Lisa Li and Percy Liang. Prefix- tuning:
Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 4582-4597, Online, August 2021. Association for Computational Linguistics. doi: 10. (0.18533/v1/2211 acl-long.353. URL https: hatc://cllllc http::///caaathhloooooo anthology org/2021. 22/110021...0 aallllong.355
[50] Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. In Anna Rogers, Jordan Goyd-Graber and Naoaki Okazaki, editors, Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 12286-12 12886-12310 12, Toronto, Canada, July 2023. Association for Computational 2023
Linguistics. doi: (0.18653/v1/2000 acc.o/0....oooo.66633333 687. URL acl- 687/
[51] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412. 19437, 2024.
[52] Dong C. Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimiza- tion. Math. Program., 45(1-3):503-523 August 1989. ISSN 0025-5610.
[53] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang. and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https
// /oopnnreview.oo wwwparrvveee.ee/fooeo
1qvx600uu
[54] Zechun Liu, Changsheng Zhao, Forrest landola, Chen Lai, Yuandong Tian, Igor Fedorov, Yunyang Xiong, Ernie Chang, Yangyang Shi, Raghuraman Krishnamoorthi, Liangzhen Lai, and Vikas Chandra. Mobilellm: Optimizing sub-billion parameter language models for device use cases. arXiv preprint arXiv:2402. 14905, 2024. 2402. 14905]
httpp::////xvvvo/oo www.:.org/bbb.. /
[55] Llama Team. The Ilama 3 herd of models, arXiv preprint arXiv:2407.2178: 2024. URL https://arxa
[56] Ekaterina Lobacheva, Nadezhda Chirkova, Maxim Kodryan, and Dmitry Vetrov. On power laws in deep ensembles. arXiv preprint urXiv:2007.08483, 2020. URL https://arxiv
org/ 202//002/00 08483
[57] Ekaterina Lobacheva, Nadezhda Chirkova, Maxim Kodryan, and Dmitry P Vetrov. On power laws in deep ensembles. In Larochelle, M. Ranzato, Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 2375-2385 Curran Associates, Inc., 2020. URL httptttt//pooceeeedaieeee http:://ppooceednnngneeerrrrreee www.ine.uuccmmm cc. acc/paper__. saaer_filest files/ .dff
[58] Anton Lozhkov Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov,
15

---

Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii, Nii Osae Osae Dade, Wenhao Yu, Lucas Krauß, Naman Jain, Yixuan Su, Xuanli He, Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff, Xiangru Tang, Muhtasham Oblokulov, Christopher Akiki, Mare Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebaze, Olivier Dehaene, Nicolas Patry, Canwen Xu, Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet. Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa Patwary, Nima Tajbakhsh, Yacine Jernite, Carlos Muñoz Ferrandis, Lingming Zhang. Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402. 19173, 2024. URL pattss//arxivv
[59] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegr- effe. Uri Alon, Nouha Dziri, Shrimai Prabhumoye. Yiming Yang, Shashank Gupta, Bod- hisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. In seventh Conference on Neural Information Processing Systems, 2023, URL httpps:/ wwt.w/eeee.rrrm 1d=S37hUerQLB
[60]
When your pretrained language model is not paying enough attention. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages $214-8236, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 10.18653/1/20222aac acl--ong..55.. URL https://accanthhlloog org/ 2022. acl-llngg 565/
Nikolay Malkin, Zhen Wang, and Nebojsa Jojic. Coherence boosting:
[61] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016. URL https: org/ /bb9/109/99/ 1609 07843
[62] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? ? new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier. and Jun ichi Tsujii, editors, Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pages 2381-233 123911, Brussels. Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/ D18- 1260. http:/ttts/:/aaaattttteee
org/ DD8- 1260/
[63] Niklas Muennighoff, Alexander M Rush, Boaz Barak. Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models. Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL /
[64] OpenAl. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023. URL https
1/arxiv. Drrxiv.orggab//3333.ou
[65] OpenAl. New reasoning models: Openai ol -rreeviwww and ol-mini. https://openai com/
research/ o1 research/o1-preve h1tp:/011pprreeeaeee and- minni 2024.
[66]
Jonathan Pei, Kevin Yang, and Dan Klein. PREADD: Prefix-adaptive decoding for controlled text generation. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings Toronto,
of the Association for Computational Linguisticx: ACL 2023, pages Canada, July 2023. Association for Computational Linguistics. doi: 10.1866 1111186/1/2222. findings-acl.666. URL htttp:::/aclattthooooo hwwpoology.orgg///00 org/2023 findings - acl. 636/
[67] Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. arXiv preprint arXiv:2406. 17557, 2024. URL. https: /ttpas////avvv.eee
httgg/bb/244000 17557
[68] Yiwei Qin, Xuefeng Li, Haoyang Zou, Yixiu Liu, Shijie Xia, Zhen Huang, Yixin Ye, Weizhe Yuan, Hector Liu, Yuanzhi Li, and Pengfei Liu. 01 replication journey: A strategic progress report - part arXiv preprint arXiv:2410. 18982, 2024. URI httttps://aaxi..ooo abs/
2410. 18982
16

---

[69] Haoran Que, Jiaheng Liu, Ge Zhang, Chenchen Zhang, Xingwei Qu, Yinghao Ma, Feiyu Duan, Zhiqi Bai, Jiakai Wang, Yuanxing Zhang, Xu Tan, Jie Fu, Wenbo Su, Jiamang Wang. Lin Qu, and Bo Zheng. D-cpt law: Domain-specific continual pre-training scaling law for large language models. arXiv preprint arxiv:2400.01375, 2024. URIptttss: /ttp:::////xvvv/vo org abs/
[70] Qwen Team. Qwen2.5 technical report. arXiv preprint arXhv:2412.151133 2024. URL attps://arzie www.ivv.rrggbb////2222 15115
[71] Qwen Team. Qwq: Reflcc deeply on the boundaries of the unknown.
Setps::/qwenllee
2025.
[72] Qwen Team. Qwen3 technical report. arXiv preprint arXiv:2505.09338 2025. URL https
arxzvv.rgg/ab//555.0002
[73] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to- image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487, 2022. https://aaaxi http://aaxivvoooaaaaa// abwaabas/2220 11487
[74] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: adversarial winograd schema challenge at scale. Commun. ACM, 64(9):99-100 August 2021. ISSN 0001-0782. doi: 10.1145/3474381. httpt//ps:// org/ 10.
[75] Guillaume V Sanchez, Alexander Spangher, Honglu Fan, Elad Levi, and Stella Biderman. Stay on topic with classifier- free guidance. In Proceedings of the 41 International Conference on Machine Learning, pages 43197-43234, 2024.
[76] Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. Beyond chinchilla- optimal: accounting for inference in language model scaling laws. In Proceedings of the 41st International Conference on Machine Learning, ICMI:22. JMLR.org, 2024.
[77] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely- mixture-of- experts layer. arXiv preprint arXiv: 100:0070..6638, 2017. URL [htph::p:: h/apsi://rxxxv org/ abs/
1701. 06538
[78] Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Wen-tau Yih. Trusting your evidence: Hallucinate less with context-aware decoding. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers). pages 783-791, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: org/2024. 2024. www.laallllaolll short. 69/
URL
[79] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Imi Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.0805: 2019. URL /ttttt:tt//aiiii wwwp:/wwwii.rgg/////oaaaa
08053
[80] Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In The Thirteenth International Conference Learning Representations, 2025. URL https:// openreview http://iwwwne/ffruneee.. net:/forum/dd-4FWW///
[81] Benedikt Stroebl, Sayash Kapoor, and Arvind Narayanan. Inference scaling flaws: The limits of I1m resampling with imperfect verifiers. arXiv preprint arXiv:2411.17501, 2024. URL
https://arxiv..or /ttps/o00r//bbb/111....00.
[82] Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2 204 102.104.00060 09864,
2021. https://:://art www.vorgg/oooo abs 22044 09864
17

---

[83] Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong. Hanjie Chen, and Xia Hu. Stop overthinking: A survey on efficient reasoning for large language models. arXiv preprint arXiv:2503. 16419, 2025.
abs 16419
[84] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the Inception Architecture for Computer Vision In 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 2818-2826, Los Alamitos, CA. USA, June 2016. IEEE Computer Society. doi: 10. 109/CVPR 10.1100902...201...308 2016..333 Sttps://dol: :ttpss///abb.ieeeccopteeeee www.eooottttreessicittyyy org/ 1109/ C210/VVPR... 2016.00220000 308
URL
[85] Dustin Tran, Jeremiah Liu, Michael W. Dusenberry, Du Phan, Mark Collier, Jie Ren, Kehang Han, Zi Wang, Zelda Mariet, Huiyi Hu, Neil Band, Tim G. J. Rudner, Karan Singhal, Zachary Nado, Joost van Amersfoort, Andreas Kirsch, Rodolphe Jenatton, Nithum Thain, Honglin Yuan, Kelly Buchanan, Kevin Murphy, D. Sculley, Yarin Gal, Zoubin Ghahramani, Jasper Snoek, and Balaji Lakshminarayanan. Plex: Towards reliability using pretrained large model extensions. arXiv preprint urXiv:2207.074111 2022. URL https: //axiiv.//xxvv..m//m wrg...oo../....o.oooooooooo /bw. .0007707413
[86] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 30. Curran Asso- ciates. Inc., 2017. URL https: neur htcp/appr_fflle///eeeeeee 200//ffii0/55e222557666661118822
pdf
[87] Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. Will we run out of data? limits of IIm scaling based on human- generated human-generated data. arXiv preprint arXiv:221 1.04225 2022. URL httpt:ttttssiiii.t//t// org/ /a///bb///2/// 2211. 221122422111..51
[88] Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew J. Nelson, Eric Jones, Robert Kern, Eric Larson, C Carey, ilhan Polat, Yu Feng. Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald. Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SeiPy 1.0: Fundamental Algorithms for Scientific Computing Python. Nature Methods, 17:261-272 2020. doi:
[89] Xi Wang. Laurence Aitchison, and Maja Rudolph. Lora ensembles for large language model fine-tuning, arXiv preprint arXiv:2310.00035, 2023. URL /tttps://axxi...e org/ Barrg//bb//aee abs/2310 00035
[90] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference Learning Representations, 2023. URL http::////ppmeerieeeeeefftooooo
[91] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. arXiv preprint urXiv:2201.11903 2022. URL https://arxiv... www.aiiwww.....oeeeoooooooooooot abs/ 2201
11903
[92] Johannes Welbl, Nelson Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In Leon Derezynski, Wei Xu, Alan Ritter, and Tim Baldwin, editors, Proceedings of the 3rd Workshop on Noisy User- Eser-generated Text, pages 94-106, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. 118853/11/11-4444 URI htw.:atppseaaaaanttlllo..e wtwp:/gg..rg/////// 4413/
[93] Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: An alternative approach to efficient ensemble and lifelong learning. arXiv preprint arXiv:2002.06715 2020. URL.
https://arxiv. www/org........ooo.... ht///bb//200 00tp//00000015
18

---

[94] Sam Wiseman and Alexander M. Rush. Sequence-tor to-nueeee---suuncee learning as beam-search optimization. In Jian Su, Kevin Duh, and Xavier Carreras, editors, Proceedings of the 2016 Conference Empirical Methods in Natural Language Processing, pages 1296-1306, Austin, Texas, November 2016. Association for Computational Linguisties. doi: 10. 10.18653/v/ D16-1137, httpsts::////// hatt:::://acllanhoooogyy -tt.:/annnnoooooooo///..------ org/D16-1137
[95] Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optiman inference for LLM problem-solving. In The Thirteenth International Conference on Learning Representations, 2025. URL https
//openreviev. www.:/eefffooooo hhet/forrmnidd-Wckpp77
[96] Shuo Yang, Le Hou, Xiaodan Song, Qiang Liu, and Denny Zhou. Speeding up deep model training by sharing weights and then unsharing. arXiv preprint 10.v:21100038844 2021. URL
https: attps://arxiv> org/ abs/ 2210. 03848
[97] Jiasheng Ye, Peiju Liu, Tianxiang Sun, Jun Zhan, Yunhua Zhou, and Xipeng Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. In The Thirteenth International Conference on Learning Representations, 2025. URL https
//openreviev.o www//eetooouueee Forumnid=jjCR21
[98] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Mårquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguis- tics, pages 4791-4800, Florence. Italy, July 2019. Association for Computational Linguistics. doi: 10.18653//1/P19-1110 1472. URL http:p///aacaathooooy.rr/r/////
[99] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transform- ers. arXiv preprint arXiv:2106.04500 040600000446.6... 2021. URL httphhttt////tttee org abs/ 2106 04560
[100] Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. When scaling meets LLM fine- tuning: The effect of data, model and finetuning method. In The Twelfth International Conference on Learning Representations, 2024. URL
wwwpaarvviiieeeefffruueee net/forum?
id=5HCnKDeTwss
[101] Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. In The 4th Workshop Mathematical Reasoning and Al at NeurlPS 24, 2024. URL https pettps://opene /ttp::///peeeeieieeeeee net/
Forum?id-CCxHooTlV
[102] Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representations, 2023. IRRLhttps: / //tp///pp///vi.e/////////// nor/forum?
1d-WZH7099tgfM
[103] Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. eestruction-folowiinn evaluation for large language models. arXiv preprint urXiv:2311.0790 2023. URL https://arxx: wwtp://wrg.bbbbb......o 221.//111111111911
19

---

Appendix
Contents
A Implementation Details and Pivot Experiments
21
B Proof for Proposition] 1
22
Training Details
24
D Training Loss for OpenWeblText with Repeating Data
25
E Parametrie Fitting for the Parallel Scaling Law
26
Training Loss for Pile and Baack-V2-Pythnn
28
Downstream Datasefs
29
Compared with Inference-time Parallel Scaling
30
Instruction Tuning
30
Continual Pre-training Qwen-2.5 3B Model
31
Discussion and Future Work
31
L Broader Impacts
32
M Visualization for Different Parallel Streams
32
20

---

A Implementation Details and Pivot Experiments
Input Transformation We expect that the transformations applied the input embedding x can significantly influence the output, which avoids excessively similar outputs across different parallel streams. This can be achieved through the Soft Prompting technique [46]. Specifically, Lester et al. [46] introduced trainable continuous embeddings, known as soft prompts, which are appended to the original sequence of input word embeddings. Freezing the main network while only fine-tuning these soft prompts can be comparable to full fine-tuning. Building on this concept, prefix-tuning [49 incorporates soft prompts into every attention layer of the Transformer model and appends them to the key and value vectors, showing superiority performance to soft prompts.
We utilize prefix-tuning to implement input transformation. To be specific, first duplicate the input x into P parallel copies, distinguishing them with different prefixes in each attention layer. This can be implemented initializing the prefixes is sufficient to ensure diverse outputs across different streams. We also which is theoretically proved effectiveness by
using different KV caches for different streams. We found that randomly
leverage the prefix reparameterization trick 44 29 Le et al. 45].
As comparison, also compared using other parameter-effiiient fine-tuning strategy for discrimi- nating the input, including LoRA [37 and BitFit (5). Notably, LoRA has been also applied to the model ensemble scenario in the literature [89), but only used in the fine-tuning setting while freezing the main parameters in their experiments.
Output Aggregation We found that using dynamic aggregation weights performs better than static ones. Specifically, we concatenate each output together and use MLP h RRxP + R.P to convert it into vector of length /' as aggregation weights. The process can be formalized as:
11 up Softmax (h (f))cat)f((11): :fo(zr)))),
(6)
where Softmax ensures aggregation weights are normalized. It can be seen as dynamically weighting different parallel streams during forward process for each token. We observed that, in the early stages of training, the model may assign nearly all weight to few streams, leaving others with near-zero weights. It prevents the prefix parameters of these unlucky streams from receiving gradients and updates. This is similar to the load imbalance phenomenon in sparse MoE architectures 23] (77), where most tokens are sometimes assigned to few experts. To address this, we apply label smoothing 84 to set non-zero minimum for each weight, formulated
- wi x (1 +)+ P'
where e denotes the smoothing parameter and we set it to 0.1 in our experiments.
As comparison, we also compare using Linear layer to aggregate different outputs and directly average the outputs.
Results We trained 0.5B model on Stack- Btack-V2-Pythenn V2-hhnn Table compares the impact of different strategies on final performance. For output aggregation, a dynamic weighted sum with label smoothing proved most effective. The differences between methods for input transformation were minor (around 0.1%), much less than the benefits obtained from changing P. Therefore, we opt for the simplest strategy, prefix tuning. Unlike LoRA and BitFit, it requires minimal changes to the model, only necessitating adjustments to the KV-ccche.
21

---

Table 5: Comparisons of different strategies for input transformations and output aggregation.
P Input Transformation
Output Aggregation
Loss
Rel, Improvement
1.1518
0.00%
Pretix (48 tokens)
Dynamic Weighted Sum (c 0.1)
1.1276
2.10%
2 Prefix (48 tokens)
Dynamic Weighted Sum (e 0.0)
1.1284
2.03%
2 Prefix (48 tokens)
Average
1.1288
2.00%
Prefix (48 tokens)
Lineur Layer
1.1323
1.69%
Prefix tokens) shared KV eache
Dynamic Weighted Sum (e 0.1)
1.1282
2.05%
Prefix (48 tokens)
Dynamic Weighted Sum (f = 0.1)
1.1276
2.10%
Prefix (96 tokens)
Dynamic Weighted Sum (c 0.1)
1.1278
LoRA (r
Dynamic Weighted Sum (e =0.1) 1.1281
2.06%
Prefix (48 tokens) LoRA 2)
Dynamic Weighted Sum (c (0..)) 1.1263
2.21%
2 Prefix (48 tokens) + LoRA (r 2) BitFit
Dynamic Weighted Sum (c (0.1)
1.1263
2.21%
2 Prefix (48 tokens) + LoRA (r
Dynamic Weighted Sum (e =0.1)
1.1260
2.24%
4 Prefix (48 tokens)
Dynamic Weighted Sum (E 001))
1.1145
3.24%
Prefix tokens)
Dynamic Weighted Sum (c 0.1)
1.1019
4.33%
B Proof for Proposition 1
Proof. We first decompose the individual loss 4. Based on the definition of language model loss, we have:
Li = Er
ply a)) P: x)]]
VEV
Eg
p(y x)) log [ply a)) (1 Api(y =))}
VEV
Ex
ply x ply x)] + Ex Σ
ply x) log (1 + Apr x)),
VEV
VEV
entropy of natural text
approximation error for the language model
where V is the vocabulary. In Chichilla scaling law, the entropy of natural text is E and the approximation error is (A/N)o. Therefore, we have:
ES
Σ ply | x:) log (1 + (p:(v/1))
A
(8)
N
VEV
Based on the Taylor series expansion, log(1 x) Equation (8), we have:
Os+²).
Apply this expansion
A
Apr(y xx) Ap.(u x)
E
ply x))
+ +0 (Aps x){3)
N
2
NEV
Apr(y/ x²²
E
(pi(y) x)) p(y x))
+ Er
Σ
plu (x)
+ 0 (Api(y)) xx)3)
2
NEV
NEV
Apr(y xx)
E
Pily x) + Σ p(y xx)
+ 1 (Apily) x)))
NEV
BEV
Ap.(v
x>)
E
where the higher-order terms O (Apr(y) xz)²) are omitted in the last line. The results suggest that minimizing the approximation loss of a language model is equal to minimizing the mean square error (MSE) of relative residuals. After fitting the data, an MSE estimator is usually assumed to be
unbiased, meaning that E,,y Ap(yy) x) = 0. Here we follow this unbiased assumption simplify the following derivation.
22

---

We next consider the aggregated loss L. Let Ap(y | x) denote the new relative residual after the
aggregation:
f x) ply x))
Ap(y x))
p(y x)
1 Ei Pz(y x))
p(y x))
p(y x))
Paly x)) p(y x))
P 1=1
p(y x))
Σ P Ap: x).
==1
Let denote the correlation coefficient between any two relative residuals Ap: (v | x) and (v x) for * j. Repeating the above process decomposite the aggregated loss we have:
C - Ex Σ p(y x)) pog x)] Eg
Σ ply x) log + Ap(y x))
VEV
VEV
entropy natural text, equal
approximation error
Ap(y x)
E + Ex,y
2
P
E + Enyy
1
Σ
Api(y 2x)
2
P
i=1
E +
Bryy
App (v) (x) +2 Σ
Apr(y x)) (1)x)
2P2
i2
P
(p x))
Apr(y
x) x)
E +
Ex Be
+ 2
E
Exy
p2
2
Based on the Corollary of Chinchilla Scaling Law (Equation
A 9h, for the first term:
Apr(y
xx)21
Ex,y
2
N
for the cross terms:
Eryyy (y p) (y xx)]
App(u ax)
Ap3 x))
A
PV
Er
Ex.
P
2
2
2
N
Combining the results, obtain the desired result:
A
E +
+ +P 1)
ip
p2
N
N
1 (P- 1)p
= E++ +
N
P
A
1
E+
N
Di/a
(P
1)p + 1]
1/2
A
E+
NP [[P
1)p + 1]
1111 1/a
23

---

C Training Details
Setup for Scaling Law Experiments Our training is based Megatron-LM [79]. We use batch size of 1024 and sequence length of 2048, resulting in 20K training steps. Models have up to 4.7 billion parameters (with 4.4B non-embedding parameters) and 8 parallel streams. For models with P 1, we incorporate prefix embeddings and aggregation weight, as introduced in Appendix A No additional parameters are included for P 1 models to maintain alignment with existing architectures. We report the last step training loss using exponential moving average, with a smoothing weight of 0.95.
For other hyperparameters, the learning rate undergoes linear warm-up over 2,000 steps, reaching a peak of 3 x 10-4 before decreasing to minimum of 1 x 10-5 according to cosine decay schedule. The models are trained using batch size of 1,024 and sequence length 2, 2,048, alongside RoPE base of 10,000 [82] We utilize bfloat 6 precision and the Adam optimizer [4], setting the epsilon to 1e-8, 31 0.9, and 32 to 0.95. All parameters, including backbones we' ve introduced, are initialized with Gaussian distribution having a standard deviation of 0.02. Furthermore, we maintain dropout rate of O, enforce weight decay rate of 0. 1,1, and clip gradients
additional ones
at 1.0. The hyperparameter choices mostly adopted from existing research [63 36 70
Model Architectures The model architectures are mostly based on the dense model of Qwen- 2.5 [70]. Recent work has indicated that the number of layers still significantly impacts the final performance smaller models [ [4]. To eliminate the influence of the number of layers and derive a cleaner scaling law, we utilize the architecture of Qwen-2.5-3B (comprising 36 layers, 16 attention heads, and KV groups) and vary the hidden size intermediate size within this framework. By keeping the number of layers constant and increasing the parameter width, compare the latency of parallel scaling and parameter. The final model structure is presented in the Table
more fairly
Table 6: Model architectures.
P Parameters (Non Embedddn)d
Hidden Size
Intermediate Size
535,813,376
896
4,864
2
538,195,842
896
4,864
4
540,577.412
896
4,864
8
545,340,552
896
4,864
693,753,85
1,024
5,504
2
696,738,81
1,024
5,504
4
699,722,756
1,024
5,504
8
705,690,632
1,024
5,504
1,088,376.228
1,280
6,912
2
1, 1,092,762,889
1,280
6,912
4
1,097,148, 164
1,280
6,912
8
1,105,918,728
1,280
6,912
1
1,571,472,384
1,536
8,320
2
1,577.522,690
1,536
8,320
4
1,583,57
1,536
8,320
8
1,595,669,00
1,536
8,320
1
2,774,773,760
2,048
11,008
2
2,784,937,98%
2,048
11,008
4
2,795,100,16
2,048
11,008
8
2,815,424,500
2,048
11,008
2,53,203,200
2,560
13,824
2
4.368,529,922
2,560
13,824
4
4,383,854,084
2,560
13,824
8
4,414,502,408
2,560
13,824

---

Setup for Two-Stage Training We follow Allal et al. (2] and use the Warmup Stable Decay (WSD) learning rate schedule [38] [9]. In the first stage, employing a 2K step warm-up followed by fixed learning rate of 3e-4. In the second stage, the learning rate is annealed from 3e-4 to le-5. The rest of the hyperparameters remain consistent with the previous experiments.
In the first phase, we do not employ the PARSCALE technique. We refer to the recipe proposed by Allal et al. [2] construct our training data, which consists of 370B general data, 80B mathematics data, and 50B code data. We train the model for two epochs to consume IT tokens, Among the general text, there are 345B from FineWeb- Web-Edu 667 and 28B from Cosmopedia 4 the mathematics data includes 80B from FineMath [2] and the code data comprises 47B from Stack- V2- V2-Python and 4B from Stack Baack-Python-Edie
In the second phase, we use the trained model from the first phase as the backbone and introduce additional parameters from PARSCALE, which randomly initialized using standard deviation of 0.02 (based on the initialization of Qwen-2.5). Following (2). in this phase, we increase the proportion of mathematics and code data, finally including total of 7B general text data, 7B mathematics data, and 7B Baack-Python-ddu data.
D Training Loss for Open Text with Repeating Data
3.0
3.0
N P
38 PP1
N 38 P
Validation Loss
2.9
N 38 P
Training Loss
2.8
= 58 1
N 58 PP=
2.8
2.6
2.7
2.4
2.6
2.5
2.2
08
108
20B
308
408
0B
10B
20B
308
408
Training Tokens
Training Tokens
(a) Training Loss
(b) Validation Loss
Figure 6: Loss for training on Open for repeating several epochs. On the fifth epoch, the validation loss suddenly increases, while the model with more computations (N = 3B, P = 2) shows a stronger resistance overfitting compared the model with more parameters (N 5B, P= 1).
Both Stack- Baock-V2-PA V2-Python and Pile datasets contain more tokens than the total number used in our experiment (42 billion), and therefore previous scaling law experiments did not involve data reuse. Muennighoff et al. [63] noted that the performance of scaling laws tends to change when training data repeated. In this section, we explore how PARSCALE performs on smaller dataset, WebTett 28 with repeating data.
Figure 6 shows a comparison of training loss and validation loss, with WebText repeated over four cycles. At the transition from the end of the fourth epoch to the beginning of the fifth epoch, we observe significant decrease in training loss and a notable increase in validation loss, indicating overfitting. This aligns with the optimal number of epochs being four for training language models, as reported by Muennighoff et al. [63]. Comparing parallel scaling and parameter scaling, we observed an intriguing phenomenon: parallel scaling results in a smaller decline in performance when overfitting occurs, while parameter scaling leads to a larger decline. Specifically, by the time overfitting occurs, the validation loss of 3B parameter model with two-way parallel scaling matched that of a 5 billion parameter model. This suggests that PARSCALE may alleviate the risk of overfitting, possibly due to having fewer parameters. As we increasingly face the depletion of pre-training data 887, further research into the scaling laws of computation in data-constrained scenarios presents a compelling future direction.
25

---

E Parametric Fitting for the Parallel Scaling Law
To obtain the practical parallel scaling law in Equation (5), based on the 24 runs (i.e., four P six N) we obtain for each dataset, we follow the strategy from Hoffmann et al. [36] and Muennighoft et al. to use the LBFGS algorithm and Huber loss for curve fitting. This process can be fomulated as:
63
1HITI Σ E,k.
HUBERS (log Fipeed log fimes)
run
where Li is the i-th observed final loss obtained from experiments and Fiped is the corresponding prediction based on the corresponding observations {N, P) and parameters {A, k, E, a.. We use 5 = 0.001 for the Huber loss to avoid overfitting.
al. 163, Hoffmann al. [36. utilize the LBFGS algorithm [52] via SciPy (88] to locate local minima of the objective. The initialization grid is defined by: E E
Following Muennighoff
J A {e-1, e-= e., ee2 e², 10 a E {0, 0.5, 1, 1.5, 2), k E (0.2, 0.4. 0.6, 0.9). All parameters are constrained to be positive. After fitting, the optimal initialization found to be away from the boundaries of our sweep.
{e- e")
Table Fitting results of the logarithmic seal- ing law (Equation (5) for Stack V2 (Python).
Table Fitting results of the logarithmic scal- ing law (Equation (5) for Pile.
A
1.130616 10"
A
1.973520 10
k
0.393463
0.334456
E
0.691 237
E
1. 1.288766
08189371
aK
0. 0.196333
Fitting Huber Loss J.
3.677 10
Fitting Huber Loss J.
1.814 x 10
Fitting R²2
0.9978
Fitting Ri2
0.9987
Table 9: Prediction of the logarithmic scaling law (Equation (5)) for Stack V2 (Python).
Table 10: Prediction of the logarithmic scaling law (Equation (5) for Pile.
P
Parameters
Lped
Lime
Error
P
Parameters
Epeat
Leue
Error
1
535,813,376
1.1728
1.1722
0.0006
535,813,376
2.1107
2.1113
-0.0006
1
693,753,856
1.1498
1.1496
0.0002
693,753,856
2.0701
2.0671
0.0030
I
2,088.376.320
1.1123
1.1131
-0.0008
1.088,376.300
2.0039
2.0027
0.0012
1
1,571,472,384
1.0840
1.0817
0.0023
1,571,472.384
1,9542
1.9539
0.0003
1
2,774.773,76
1.0439
1.0451
-0.0012
2,774,773,766
1.8839
1.8876
-0.0037
1
4.353,203,200
1.0151
1.0213
0.0062
4,353,203,200
1,8335
1.8451
-0.0116
2
195,89
1.1509
1.1507
0.0002
538,195,847
2,0770
2.0772
-0.0002
2
696,738,818
1.1290
1.1262
0.0028
2
696,738,818
2,0381
2.0363
0.0018
2
1,092,762,882
1.0932
1.0940
-0.0008
2
1,092,762,882
1.9747
1.9730
0.0017
2
1.577,522,690
1. 1.0662
1.0623
0.0039
1,577,522,690 2
1,9270
1.9266
0.0004
2
2,784,937,988
1,0280
1.0244
0.0036
2,784,937,96
1.8596
1.8610
-0.0014
2
529..229.922
1.0005
1.0025
-0.0200
4,368,529,92
1.8113
1,8137
-0.0024
4
540.577.412
1.1340
1.1354
-0.0014
540,577.412 4
2,0501
2.0544
-0.0043
699,722,756 4
1.1129
1.1124
0.0005
699,722,756 4
2.0125
2.0128
-0.0003
(497,148,6, 4
1.0784
1.0808
-0.0244
1,583,571,460 4
1,097,148,16, 164
1,9514
1,9509
0.0005
4
1,583,571,466
1.0524
1.0490
0.0034
1.9053
1,9040
0.0013
4
2,795 164
1.0156
1.0126
0.0030
2,795,100, 164
1.8402
1.8394
0.0008
4
3383,854,088
0.9891
0.9906
-0.0015
4,383,854.088 4
1.7936
1.7938
-0.0002
545.340,552 8
1.1198
1.1231
-0.0033
545.340,552 8
2.0272
2.0364
-0.0092
705.690.632 8
1.0994
1.0997
-0.0003
705.690.632
1,9908
1.9933
-0.0025
8
1.105.918.728
1.0661
1.0688
-0.0027
1.105.918,728
1,9315
1,9318
-0.0003
8
1.595.669.000
1.0410
1.0383
0.0027
8
1,595.669 600,00
1.8869
1.8856
0.0013
8
2,815.424.57
1.0053
1.0016
0.0037
8
424.5.224.520
1.8238
1.8218
0.0020
8
4,414,502,408
0.9797
0.9794
0.0003
8
4,414,502,408
1.7785
1.7772
0.0013
Tables and present the fitted parameters and evaluation metrics. Tables and TO show the prediction results based on the fitted parameters.
26

---

Table 11: Fitting results of the theoretical scal- ing law (Equation (4 for Stack- V2 (Python).
Table 12:
Fitting results of the theoretical scal-
ing law (Equation (4) for Pile.
A
1.187646 x 10"
A
2.150890 x 10s
p
0.891914
P
0.899475
E
0.660016
E
1.272178
0.175036
0.190100
Fitting Huber Loss +
5.259 x 10-5 0.9959
Fitting Huber Loss +
4.545 x 10-5 0.9968
Fitting R
Fitting R²
Table 13: Prediction of the theoretical scaling law (Equation (4)) for Stack V2 (Python).
Table 14: Prediction of the theoretical scaling law (Equation (4) ) for Pile.
P
Parameters
pred
Live
Error
P
Parameters
Cyral
Leue
Error
1
535,813,376
1.1734
1.1722
0,0012
535,813,376
2.1129
2.1113
0.0016
693.753,856 I
1.1507
1.1496
0.0011
693,753,856
2.0726
2.0671
0.0055
1,088.376.320
1.1135
1.1131
0.0004
1,088,376.300
2.0069
2.0027
0.0042
1,571,472,384
1.0853
1.0817
0.0036
1,571,472,384
1.9574
1.9539
0.0035
1
2,774,773,70 760,760
1.0450
1.0451
-0.0001
2,774,773,760
1.8872
1.8876
-0.0004
4,353,203,000 1
1.01588
1.0213
-0.0055
4,353,203,
1.8367
1.8451
-0.0084
198,195,842 2
1,1453
1.1507
-0.0554
2
138,195,842
2.0700
2.0772
-0.0072
696.738,818 2
1.1238
1.1262
-0.0024
696,738,818
2.0317
2.0363
-0.0046
2
1,092.762,822
1.0887
1.0940
-0.0033
2
1,092,762.88
1,9695
1.930
-0.0035
2
1,577,522,699 690
1.0620
1.0623
-0,0033
1,577,522,69 690 2
1,9225
1.9266
-0.0041
2
2,784,937,98 937,986
1,0239
1.0244
-0,0005
2
2,784,937, 986
1.8559
1.8610
-0.0051
2
4,368,529 1,68,529,999 529,922
0.9964
1.0025
-0.001
4,368,529 ,22
1. 1,8080
1.8137
-0.0057
540.577,412 4
1.1310
1.1354
-0.0044
540,577.41 412
2.0482
2.0544
-0.0062
699,722,756 4
1.1102
1.1124
-0.0022
699,722,756 756
2,0110
2.0128
-0.0018
1097,148,16 4
1.0762
1.0808
-0.0046
4
1,097,148, 197,448,164
1.9505
1.9509
-0.0004
4
1,583,571, 1,,60
1.0503
1.0490
0.0013
1583,571,460 4
1.9048
1.9040
0.0008
2,795,100. 100.
1.0133
1.0126
0.0007
1,0, 164
1.8400
1.8394
0.0006
4,383.854,084 4
0,9866
0.9906
-0.0400
4,383,854.084
1.7935
1.7938
-0.0003
545,340,552 8
1.1234
1.1231
0.0003
545,340,552 8
2.0364
2.0364
-0.0000
695,690,632 8
1.1030
1.0997
0.0033
705.690,632 8
1,9998
1.9933
0.0065
8
1,105,918,725 728
1,0695
1.0688
0.0007
1,105,918,78 728
1,9403
1.9318
0.0085
1.595.669,00
1.0440
1.0383
0.0057
1,595,669 669,0066900 ,000
1.8953
1,8856
0.0097
8
224,5,424.520
1.0077
1.0016
0.0061
8
2,8.5,424.520
1.8315
1.8218
0.0097
8
4.414.502,408
0.9814
0.9794
0.0020
8
4,414,502.408
1.7857
1.7772
0.0085
We also test our derived theoretical parallel scaling law (Equation 44)) in which the correlation coefficient is treated as constant to fit. Tables [1 and 12 present the fitted parameters and evaluation metrics, while Tables [3 and [4] show the fitted results. evident that, whether for Stack or Pile, treating as a constant yields fitting accuracy that is inferior to the previously proposed logarithmic parallel scaling law.
27

---

F Training Loss for Pile and Stack- Baakk-V--Python
Figure Iillustrates the curve of loss versus data size in our scaling law experiments. It clearly shows that scaling P yields benefits, regardless of the data scale.
Stack-v2-PPthho (0.58 Model)
Stack-V2-Pytie V2-t-too (0.78 Model)
Stack-V2-Python (1.18 Model)
1=1
1.35
P=1
1:30
Training Loss
2=2
Training Loss
1.30
Pm
Training Loss
==
1.30
1.25
1.25
P=8
P
1.20
1,20
1.15
1.15
1.15
1.10
108
208
308
408
06
108
208
300
408
108
208
306
408
Training Tokens
Training Takens
Training Tokens
(a)
(b)
(c)
Stack-V2-Python (1.68 Model)
Stack-V2- P2-thhnn (2.88 Model)
Stack-V2-Python (4.4B Model)
P-1
1.25
Training Loss
1.25
2
Training Loss
1.20
P=2
Training Loss
1.20
1.20
1.15
P=8
1.15
P
8
1.15
1.10
1.10
1.05
1.05
1.0
1.
08
106
208
308
408
06
108
208
308
406
08
108
206
306
406
Training Tokens
Training Tokens
Training Tokens
(d)
(c)
Pile (0.5B Model)
Pile (0.7B Model)
Pile (1.18 Model)
2.40
P-1
P-1
2.30
2.35
2.30
P= 4
2.25
2.20
P 4
Training
2.25
2.20
P= 8
2.20
P=
2.15
2.10
2.15
2.10
2.10
2.05
2.05
2.00
1.95
08
10B
208
308
408
08
10B
208
308
408
08
106
208
308
408
Training Tokens
Training Tokens
Training Tokens
(h)
(i)
Pile (1.6B Model)
Pile (2.88 Model)
Pile (4.4B Model)
2.20
2.15
2.20
P 2
Training 5
2.15
P
2.15
2.10
2.05
Training
2.10
P 8
2.00
2.05
Training
8
1.95
1.90
1.95
1.90
1.85
1.85
1.80
1.80
105
208
308
108
308
408
108
208
308
Training Tokens
Training Tokens
Training Tokens
(k)
Figure 7: Training loss for the Stack- V2-Python and the Pile, smoothing with 0.98 exponential moving average.
28

---

G Downstream Datasets
Table 15: HumanEval Pass @ (%)
Table 16: HumanEval+ Pass (%)
0.5B
0.7B
1.1B
1.6B
2.8B
4.4B
N
0.5B
0.7B
1.1B
1.6B
2.8B
4.48
P=1
12.8
15.9
17.7
18.3
18.3
19.5
P=
11.0
13.4
15.9
15.9
15.9
16.5
P
15.9
20.7
20.1
19.5
18.9
P=2
13.4
17.7
16.5
16.5
21.3
P
4
15.2
17.1
18.3
18.3
22.0
20.7
P=4
14.0
14.6
15.9
15.9
18.9
18.3
P=8
8
18.9
18.3
21.3
18.3
21.3
25.0
P=8
15.9
15.2
18.3
16.5
19.5
20.7
Table 17: MBPP Pass I (%)
Table 18: MBPP+ Pass (%)
0.5B
0.78
1.1B
1.6B
2.8B
4.4B
0.5B
0.7B
1.1B
1.6B
2.8B
4,48
P
27.8
30.2
31.5
36.0
40.5
45.8
P
23.5
26.2
25.7
31.2
34.7
38.4
36.8
42.6
47.4
P
27.0
27.8
30.7
35.4
39.4
P
35.2
33.9
39.4
40.7
45.5
50.3
P
P=
4
28.0
27.2
32.3
32.5
37.3
40.5
P
33.3
36.0
39.9
45.5
47.4
48.4
P
8
29.1
29.1
33.6
38.9
40.5
42.3
Table 19:
HumanEval Pass @ 10 (%)
Table 20: HumanEval+ Pass 10 (%)
N
0.5B
0.7B
1.1B
1.6B
2.8B
4.4B
N
0.5B
0.7B
1.1B
1.6B
2.8B
4.4B
P=
25.0
21.3
29.9
29.9
32.9
37.2
P=
23.8
21.3
26.8
26.8
29.9
33.5
27.4
28.0
35.4
34.8
p 2
22.0
25.6
24.4
31.7
32.3
36.0
26.2
30.5
28.0
36.0
38.4
p p=
24.4
27.4
33,5
33.5
34.8
P
29.9
32.3
31.7
37.2
38.4
47.0
P 8
27.4
29.9
29.9
32.9
36.0
42.7
Table 21:
MBPP Pass 10 (%)
Table 22: MBPP+ Pass @ 10 (%)
N
0.5B
0.7B
1.IB
1.6B
2.8B
4.4B
N
0.5B
0.7B
1.1B
1.6B
2.8B
4.4B
P=1
49.2
54.0
57.7
61.4
68.3
66.4
P
40.2
44.7
47.9
51.6
54.5
56.3
P=
57.9
57.7
60.3
64.0
67.7
p p=
46.8
48.4
54.2
58.2
60.6
P
59.8
61.1
66.9
70.9
p p=4
43.7
49.5
52.1
56.6
59.0
62.7
P
57.7
60.3
66.1
67.5
72.8
75.1
P P
46.0
50.8
56.6
56.1
60.6
62.2
Table 23: WinoGrande Performance (%)
Table 24: Hellaswag Performance (%)
N
0.5B
0.7B
1.IB
1.6B
2.8B
4.4B
N
0.5B
0.7B
1.1B
1.6B
2.8B
4.4B
P=1
$1.9
51.2
51.6
52.2
53.5
54.7
P=
35.7
37.4
40.1
42.6
46.7
49.3
P 51.0
5144
53.0
53.3
56.0
$7.4
P 2
36.8
38.4
41.3
44.5
48.4
51.9
P=4
53.0
53.5
54.5
p P=4
37.4
39.4
45.7
50.0
53.8
P
51.7
53.6
55.0
53.4
55.6
56.9
P=8
38.6
40.6
44.1
46.8
51.0
54.6
Table 25: OpcnBookQA Performance (%)
Table 26: PiQA Performance (%)
N
0.5B
0.7B
1.IB
1.6B
2.8B
4,4B
N
0.5B
0.7B
1.1B
1.6B
2.8B
4.4B
p=1
26.0
28.8
28.2
28.0
29.0
32.4
P 1
65.0
65.8
66.9
67.5
68.8
69.5
P 26.8
26.6
27.8
29.8
30.6
29.8
P 2
65.7
66.8
67.4
68.5
70.5
71.3
29.0
29.8
29.4
31.0
P
66.7
70.6
72.1
P 26.8
27.2
29.4
31.0
31.6
30.6
P 66.5
66.9
67.8
69.5
70.9
71.5
Setup for 42B token experiments For HumanEval(+)) [12] and MBPP(+) 33, we use the EvalPlus framework (53) for evaluation, where Pass@ employs greedy decoding and Pass @ 10 employs temperature of 0.8. Considering that the pretrained base model cannot follow instructions, we use the direct completion format. For general tasks, we employ Im-eval harness [6] and report normalized accuracy when provided. The number of few-shot mostly follows existing research configurations. Benchmarks include: WinoGrande (5-shot, Sakaguchi et (74). Hellaswag (10-shot, Zellers et al.
198). OpenBookQA (5-shot, Mihaylov et al. 622 PiQA (5-shot, Bisk et al. 77). ARC-Fasy and
ARC-Challenge (25-shot, Clark et al. [15]; we reporting the average of both), and SeiQ (3-shot, Welbl et al. (92).
29

---

Table 27: ARC Performance (%)
Table 28: SciQ Performance (%)
0.7B
1.IB
1.6B
2.8B
4.4B
N
0.5B
1.1B
1.6B
2.8B
4.4B
36,9
40.5
42.1
44.7
P
78.9
81.7
86.1
88.5
91.0
38.6
40.9
43.2
45.9
49.1
P=2
80.8
83.2
87.3
90.5
91.4
P 39.1
39.9
41.1
44.2
47.4
48.8
P=
4
82.3
82.7
84.4
87.2
91.2
91.7
P
39.4
39.8
42.0
44.8
48.2
49.4
P
8
81.3
83.0
86.9
88.9
91.2
94.2
Setup for token experiments In the token experiments, we use more challenging datasets for comprehensive testing, including MMLLU (5-shot, Hendrycks et al. (30] and RACE (4-shot, Lai et al. 43) In addition, we introduce mathematics reasoning datasets, including GSM8K (4-shot, Cobbe al. (17), GSM8K-CoT (8-shot), and MATH (4-shot, Hendryeks et al. [1]; using the Minerva evaluation rules ). When evaluating the Instruct model, we also use IFEval (0-shot, Zhou et al. and report the average among four metrics provided by Im-eval harness.
103
Tables 15 to show the downstream task performance in the 42B token experiments. We report average of these performances in the main text (Tables 2 and
H
Compared with Inference-time Parallel Scaling
Some training-frees inference-time scaling methods also expand parallel computation during the inference phase, with Beam Search and Majority Voting being representative methods. We applied Beam Search and Majority Voting to the trained Baseline-1.88 and compared it with PARSCALE to emphasize the importance of expanding parallel computation during the training phase. It is worth mentioning that these methods cannot be applied to the general tasks in Table because these tasks primarily evaluate next-token prediction.
Table 29: Comparison with Beam-Search Search.
Number of Parallels
Method
GSM8K
GSMSK-CoT
Minerva MATH
28.7
35.9
12.0
2
PARSCALE
32.6
35.6
13.0
2
Beam-Search
29.3
37.8
13.6
2
Majority Voting (temp=0.2)
28.2
12.0
Majority Voting (temp=0.8)
19.7
7.5
PARSCALE
34.7
40.8
14.5
4
SearhS
27.7
37.8
12.5
Majority Voting (temp=0.2)
31.5
12.6
4
Majority Voting (temp=0.8)
25.7
8.5
8
PARSCALE
38.4
43.7
16.4
8
Deam-Searde
22.5
30.1
10.0
Majority Voting (temp=0.2)
32.8
14.5
8
Majority Voting (temp=0.8)
32.0
10.8
The results are shown in Table 29
can be observed that Beam Search is only effective when beabems As r__eemms increases, the performance of Beam Search actually decreases. This indicates that, without strong verifier, it is difficult for LLM alone to select the correct result from multiple sampling results. This aligns with the finding in [8I. This experiment further emphasizes the importance of expanding parallel computation in both the training stage and inference stage.
I Instruction Tuning
We follow standard practice to post-train the base models, to explore whether PARSCALE can enhance performance during the post-training stage. We conducted instruction tuning on four checkpoints (P E (1,2, 4, 8)) from the previous pre-training step, increasing the sequence length from 2048 to 8192 and the RoPE base from 10,000 to 100,000, while keeping other hyperparameters constant. We
30

---

Table 30: Comparison of the performance
of different Instruct models, where the few- shot examples are treated as a multi-turn
conversation.
IFEval
MMLU GSM8K
0-shot
5-shot
4-shot
SmolLM-I,7B-Ing
16.3
28.4
2.0
Buseline-Inst (P 1)
54.1
34.2
50.3
PARSCALE-InG Inst (P 55.8
35.1
55.3
PARSCALE-InS (P 4)
58.4
38.2
54.8
PARSCALE-InG (P 5)
59.5
41.7
56.1
used I million SmolTalk (2 the instruction tuning data and trained for epochs. We refer these instruction models as PARSCALE-Inst.
The experimental results in Table 30 show that when P increases from I to 8, method achieves 5% improvement on the instruction daarucioo-followinn benchmark IFEval, along with substantial gains in the general task MMLU and the reasoning task GSM8K. This demonstrates that the proposed PARSCALE performs effectively during the post-training phase.
J Continual Pre-training Qwen-2.5 3B Model
Stack-V2-Python
Pile
1.85
1.07
Training Lose
Training Loss
4
1.75
1.0
0.98
1,70
1.65
08
108
208
308
408
06
108
208
308
408
Training Tokens
Training Tokens
(b)
Figure 8: Loss for continual pre-training the Qwen-2.5-33 model on the two datascts.
Figures 8(a) and 8(b) illustrates the training loss after continuous pre-training on Stack- V2 (Python) and Pile. Notably, Qwen2.5 has already been pre-trained on 18T of data, which possibly have significant overlap with both Pile and Stack-V2. This demonstrates that improvements can still be achieved even with thoroughly trained foundation model and commonly used training datasets.
K Discussion and Future Work
More Interpretation of Scaling Law Claims Our analysis relics on fitting a log plwer-law relationship to empirical scaling data using Huber loss, which is less sensitive to outliers than standard squared loss. While this approach improves robustness, it does not, by itself, constitute rigorous statistical evidence for the existence of true power-law distribution in the underlying data. As emphasized by Clauset et al. [16], identifying power laws in empirical data is notoriously challenging and requires careful model comparison, goodness-of-ff testing, and consideration of alternative functional forms (e.g., log-normal or exponential). Future work could strengthen these conclusions by applying more rigorous scaling law detection protocols outlined in the statistical literature.
Training Inference-Optimal Language Models Chinchilla 36 determine the training-optimal amounts for parameters and training data under training FLOP budget. On the other hands, modern L.LMs are increasingly interested on inference-optimal models. Some practitioners use much more data than the Chinchilla recommendation to train small models
explored the scaling law to
31

---

due to their high inference efficiency 770 2 76]. Recent inference petreence-tmeeeeee scaling efforts attempt to provide computation-opiina strategy during the inference phase [95] 80], but most rely on specific scenarios and datasets. Leveraging the proposed PARSCALF, determining how to allocate the number of parameters and parallel computation under various inference budgets (e.g., memory, latency, and batch size) to extend inference-optimal scaling laws [76] is a promising direction.
Further Theoretical Analysis for Parallel Scaling Laws One of our contributions quantitatively computing the impact of parameters and computation on model capability. Although we present some theoretical results (Proposition [T), the challenge of directly modeling DIVERSITY limits us to using extensive experiments to fit parallel scaling laws. Why the diversity is related to log P. is there a growth rate that exceeds (log P), and whether there is a performance upper bound for P > remain open questions.
Optimal Division Point of Two-Stage Strategy Considering that PARSCALE inefficient the training phase, introduced two-stage strategy and found that LLMs can still learn to leverage parallel computation for better capacity with relatively few tokens. We currently employ a IT vs. 20B tokens as the division point. Whether there is more optimal division strategy and its trade-off with performance is also an interesting research direction.
Application to MoE Architecture Similar to the method proposed by Geiping al. 27 PARSCALE is a computation-heut (but more efficient) strategy. This somewhat complementary to sparse MoE 23 777. which is garameter-heavy. Considering that MoE is latency- -rrency-friendly while PARSCALE memory-friendly, exploring whether their combination can yield more efficient and high-performing models is worth investigating.
Application to Other Machine Learning Domains Although we focus on language models, PARSCALE is more general method that can be applied to any model architecture, training algorithm, and training data. Exploring PARSCALE in other areas and even proposing new scaling laws is also a promising direction for the future.
L Broader Impacts
LLMs present certain potential risks, such as generating offensive language, perpetuating social biases, and leaking private information. While the public release all our models and the insights we provide to scaling LLMs could inadvertently contribute to the proliferation of these harms, it important to note that there are already larger and more capable models freely available that can also be used in harmful ways. We believe that the benefits of open-source releasing our models and research can outweigh the associated risks.
M Visualization for Different Parallel Streams
In the output aggregation, assign different dynamic weights to different streams. These weights indicate the contribution ratio different parallel streams to the next token prediction. We visualize the stream that contributes the most to each token (i.e., the stream with the highest aggregated weight), based on the 4.4B model pre-trained on the Pile. Tokens of the same color indicating that these tokens primarily contributed by the same stream. Tables 31 33 visualize the results. An interesting observation is the locality of the colors: tokens in close proximity are often primarily contributed by the same stream, especially when relatively small (c.g., P 2).
32

---

Table 31: Parallel stream visualization, where paragraphs are sampled from wikitext 61 Tokens with the same color indicate they are primarily contributed by the same stream.
Somerset progressed the second round of the competition after Trinidad and Tobago beat Deccan in the final group match, but lost Trescothick, who flew home after recurrence of his illness. Wes Durston, who replaced Trescothick in the side top scored for Somerset in their next match, making 57 Only two P=2 other players reached double figures for the county and the Diamond Eagles chased down the total with eight balls to spare Somerset went into their final match against the New South Wales Blues with a slim mathematical chance of progressing. but a strong bowling display from Brett Lee and Stuart Clark restricted Somerset to 111, which the Australian side reached with ease
Somerset progressed to the second round of the competition after Trinidad and Tobago beat Deccan in the final group match but lost Trescothick who flew home after recurrence of his illness. Wes Durston, who replaced Trescothick in the side top scored for Somerset in their next match, making 57 Only two P=4 other players reached double figures for the county and the Diamond Fagles chased down the total with eight balls to spare Somerset went into their final match against the New South Wales Blues with a slim mathematical chance of progressing, but a strong bowling display from Brett Lee and Stuart Clark restricted Somerset to 111, which the Australian side reached with ease
Somerset progressed to the second round of the competition after Trinidad and Tobago beat Deccan in the final group match but lost Trescothick who flew home after a recurrence of his illness. Wes Durston, who replaced Trescothick in the side top - scored for Somerset in their next match, making 57 Only two P=8 other players reached double figures for the county and the Diamond Eagles chased down the total with eight balls to spare Somerset went into their final match against the New South Wales Blues with a slim mathematical chance of progressing. but a strong bowling display from Brett Lee and Stuart Clark restricted Somerset to 111, which the Australian side reached with ease
Table 32:
Parallel stream visualization, where paragraphs are sampled from GSM8K with the same color indicate they are primarily contributed by the same stream.
Tokens
Question: A baker is making bread according to a recipe that requires him to use 3 eggs for every 2 cups of flour. the baker wants to use up the 6 cups of flour
P=2 2 he has remaining in his pantry, how many eggs will he need to use?. Answer: If he uses cups of flour, the baker will be making 6/2 <</23> times the normal amount that the recipe describes. Thus, he must use 3#3 = =3*=99 eggs. #####
Question: A baker is making bread according to recipe that requires him to use 3 eggs for every 2 cups of flour. If the baker wants to use up the 6 cups of flour he has remaining in his pantry, how many eggs will he need to use?. Answer: If he uses cups of flour, the baker will be making 6/2 = <6/2=> times the normal amount that the recipe describes. Thus, he must use 3#3 = <3*3=9 eggs. ####
P=4 4
Question: A baker is making bread according to a recipe that requires him to use 3 eggs for every 2 cups of flour. If the baker wants to use up the 6 cups of flour he has remaining in his pantry, how many eggs will he need to use?. Answer: If he uses cups of flour, the baker will be making 6/2 6/2=3> times the normal amount that the recipe describes. Thus, he must use 3#3 = <3#3=99 eggs. #####: 9
P=8
33

---

Table 33: Parallel stream visualization, where paragraphs are sampled from RACE 443 Tokens with the same color indicate they are primarily contributed by the same stream.
The Mysterious Universe By Ellen Jackson and Nic Bishop How did the universe begin? How big is it? What is dark matter? Cosmologist and expert supernova hunter Alex Filippenko hopes that supernovas can help us answer some of these questions, But first we' 'e got to find them! Join Alex and his team as they go on the hunt with huge telescopes and banks of computers. The Time and Space of Uncle Albert By Russell Stannard What would you say your uncle asked you whether you would like go into space? You'l 'd say, "When do leave?", just like the girl this story, Gedanken speeding across the universe trying help her uncle answer some questions, such "How big is space?" and "Where does gravity come from?" Along the way she also discovers how get heavier without getting fat, how live forever without knowing it, and the strange things that can happen when you go really fast, George's Secret Key to the Universe By Lucy Hawking and Stephen Hawking When George chases his pet pig through a hole in the fence, little does he expect that he will soon be riding a comet around Saturn But just as he discovers the joys of space exploration with the computer Cosmos, which can open doors any anywhere in the universe, everything starts to go wrong.
P=2
The Mysterious Universe By Ellen Jackson and Nic Bishop How did the universe begin? How big is it? What is dark matter? Cosmologist and expert supernova hunter Alex Filippenko hopes that supernovas can help us answer some of these questions. But first we' we've got to find them! Join Alex and his team as they go on the hunt with huge telescopes and banks of computers. The Time and Space of Uncle Albert By Russell Stannard What would you say if your uncle asked you whether you would like to go into space? You'd say. "When do I leave?", just like the girl in this story. Gedanken is speeding across the universe trying to help her uncle answer some questions, such as "How big is space?" and "Where does gravity come from?" Along the way she also discovers how to get heavier without getting fat, how to live forever without knowing it, and the strange things that can happen when you go really fast. George's 's Secret Key to the Universe By Lucy Hawking and Stephen Hawking When George chases his pet pig through a hole in the fence, little does he expect that he will soon be riding a comet around Saturn. But just as he discovers the joys of space exploration with the computer Cosmos, which can open doors anywhere in the universe, everything starts go wrong.
P 4
The Mysterious Universe By Ellen Jackson and Nic Bishop How did the universe begin? How big is it? What is dark matter? Cosmologist and expert supernova hunter Alex Filippenko hopes that supernovas can help us answer some of these questions. But first we've ve got to find them! Join Alex and his team they go on the hunt with huge telescopes and banks of computers. The Time and Space of Uncle Albert By Russell Stannard What would you say if your uncle asked you whether you would like go into space? You't 'd say. "When do leave?" just like the girl in this story. Gedanken is speeding across the universe trying help her uncle answer some questions, such as "How big is space?" and "Where does gravity come from?" Along the way she also discovers how to get heavier without getting fat, how to live forever without knowing it, and the strange things that can happen when you go really fast. George's ' Secret Key to the Universe By Lucy Hawking and Stephen Hawking When George chases his pet pig through a hole in the fence, little does he expect that he will soon be riding a comet around Saturn But just as he discovers the joys of space exploration with the computer Cosmos, which can open doors anywhere in the universe, everything starts to go wrong.
P=8 8
34

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See the Abstract and Introduction sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
35

---

Justification:
See Appendix B
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
36

---

Answer: [Yes]
Justification: We only use public training data. The training details are in Appendix C and the code is available in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https /nips.ccc/pulli//guu guwde//ddee/ee/ Cot//ooooeeeiiseeeeeee hwtpbbbmsssiooooooooiooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Training LLMs is expensive, especially we conducted large-scale scaling law experiments. However, our scaling law fits show high accuracy (Appendix E), indicating that the impact of noise is minimal.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
37

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments? A
Question:
Answer: [Yes]
Justification:
We provide the number of GPU hours in our paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We follow all the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification: See Appendix
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
38

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
See Appendix
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite all the assets properly.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
39

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We document our released code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
40

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
We use LLMs for writing, editing, or formatting purposes.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy fattps://neur//: //tee/rrpp...c///ofeeeerr//////s for what should or should not be described.
41


Reviews:
[{"review_id": "ZeQHF263Wt", "reviewer": "Reviewer_Qbt2", "summary": "This paper proposes ParScale, a scaling strategy that improves language-model performance by re-using the same parameters for P parallel forward passes on differently-transformed inputs via learnable prefixes and then aggregating the outputs with a lightweight MLP. Extensive pre-training experiments (~10^6 GPU-h) on 42B and 1T tokens confirm a log-linear scaling law: P-way parallel computation yields the same loss reduction as multiplying parameter count by O(log P), while adding only ~0.2 % parameters and incurring modest memory/latency overhead. A two-stage training recipe, i.e., pre-train normally on 1T tokens, then PARSCALE-fine-tune on 20B (2%), eliminates most of the extra cost and allows off-the-shelf models (e.g., Qwen-2.5) to be retro-fitted via parameter-efficient fine-tuning, supporting dynamic P-switching at inference.\n\nThis work provides both theoretical grounding and compelling empirical evidence that parallel computation can substitute for parameters, especially on reasoning-heavy benchmarks (code/math). \nCompared with dense scaling, MoE, or inference-time scaling, PARSCALE is architecture-agnostic, training-time compatible, and memory-efficient, which is highly attractive for edge deployment.", "questions": "1. Could you elaborate more on the details of how to conduct prefix tuning as input transformation.\n2. In Table 4, can the authors add the actual number of FLOPs for the baseline and ParScale, I'm curious about the results under fair FLOPs comparisons.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n* Strong empirical grounding, with 10^6 GPU-h scaling-law experiments yielding an R² ≈ 0.998 fit that cleanly isolates the effect of parallel computation versus parameters.\n* The proposed two-stage training (1 T normal + ParScale) can still give large downstream gains.\n* ParScale matches a 4.4B dense model with 22× less memory and 6× less latency at batch 1, which could potentially address resource-constrained deployment.\n\n**Weaknesses**\n\n* Even with the two-stage recipe, ParScale multiplies FLOPs by P during fine-tuning and still doubles total compute for P=8; this burden grows linearly with P and could outweigh benefits at larger scales.\n* All latency/memory claims assume batch ≤8; at server-scale batch sizes the memory bottleneck evaporates and ParScale's compute overhead dominates, possibly erasing its edge over dense scaling.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough and insightful review of our manuscript. We are grateful for your positive feedback, particularly recognizing our rigorous empirical grounding with large-scale scaling-law experiments and the significant efficiency gains of parallel scaling. Below are our responses to your concerns:\n\n> W1: Even with the two-stage recipe, ParScale multiplies FLOPs by P during fine-tuning and still doubles total compute for P=8; this burden grows linearly with P and could outweigh benefits at larger scales.\n\nWe agree that the ParScale fine-tuning stage introduces a computational overhead. We see this as a strategic trade-off, and we would like to clarify that this concern is also applicable to other established scaling strategies. For example, \n\n- Parameter scaling increases FLOPs proportionally with model size, impacting the entire training pipeline and inference.\n- Inference-time scaling (e.g., scaling output tokens) also linearly increases FLOPs, often through less efficient serial computation.\n\nIn contrast, ParScale strategically localizes the computational increase to the second, shorter fine-tuning stage. This one-time investment unlocks significant and persistent efficiency gains during inference (e.g., superior memory and latency). By shifting the computational burden away from inference and the initial pre-training phase, ParScale offers a more targeted approach to achieving the capabilities of larger models.\n\n> W2: All latency/memory claims assume batch ≤8; at server-scale batch sizes the memory bottleneck evaporates and ParScale's compute overhead dominates, possibly erasing its edge over dense scaling.\n\n-  Thanks for pointing it out. We respectfully suggest that **ParScale's value is best understood by its targeted strengths in low-resource environments and its adaptability, rather than judging it solely on its performance in high-throughput, large-batch scenarios.** Our work positions ParScale primarily as a solution for memory-limited and small-batch applications (like *edge computing*), where it demonstrably excels in both latency and memory efficiency.\n- Additionally, ParScale is compatible with Parameter-Efficient Fine-Tuning (PEFT) methods. This opens up possibilities for dynamic resource allocation in *server environments*. For instance, a model could operate in a low-FLOPs, low-cost mode (i.e., low P) during peak demand and scale up its computational budget (i.e., large P) with ParScale during off-peak hours for more powerful processing.\n    \n\n> Q1: Could you elaborate more on the details of how to conduct prefix tuning as input transformation.\n\n- A detailed description of our prefix tuning method is provided in Appendix A. To be specific, we first duplicate the input x into P parallel copies, distinguishing them with different prefixes in each attention layer. This can be implemented as using different KV caches for different streams. We found that randomly initializing the prefixes is sufficient to ensure diverse outputs across different streams. We also leverage the prefix reparameterization trick.\n    \n\n> Q2: In Table 4, can the authors add the actual number of FLOPs for the baseline and ParScale, I'm curious about the results under fair FLOPs comparisons.\n\n- Thank you for your suggestion. Assuming the input sequence length is S, as P increases, the FLOPs are 3.14S (P=1), 6.28S (P=2), 12.56S (P=4), and 25.12S (P=8), respectively, in units of GFLOPs. \n- However, we wish to clarify that our primary goal here is not a direct FLOP-for-FLOP \"fair\" comparison. Instead, the experiment is designed to demonstrate a new scaling dimension: showing that for a *fixed parameter budget*, model performance scales directly with increased *parallel computation*. **Even with larger FLOPs, ParScale could achieve lower memory consumption and latency, shown in our cost analysis.**\n\n---\n\nThank you again for your invaluable feedback. We hope our response could effectively address your concerns, and we would be grateful if you would reconsider your evaluation of our manuscript."}, {"review_id": "AJ5AxrQiJp", "reviewer": "Reviewer_8hYJ", "summary": "This paper introduces a novel scaling paradigm for language models called Parallel Scaling (PARSCALE), which serves as an alternative to traditional parameter scaling and inference-time scaling. The core method involves applying P diverse, learnable transformations to an input, processing these P versions in parallel through a single shared-weight model, and then dynamically aggregating the outputs to produce a final result. This approach effectively scales the amount of parallel computation during both training and inference by reusing existing model parameters, thereby avoiding the significant space or time costs associated with conventional scaling methods.\n\nThe primary contribution is the proposal and empirical validation of a new \"parallel scaling law,\" which finds that scaling computation by P parallel streams is analogous to scaling the model's parameters by a factor of O(log P). The authors demonstrate that for an equivalent performance gain, PARSCALE is substantially more efficient during inference, showing significant reductions in memory and latency increases compared to parameter scaling. Furthermore, the paper introduces a practical two-stage training strategy to mitigate high training costs and shows that PARSCALE can be effectively applied to off-the-shelf models, enabling dynamic adjustment of model capability at deployment.", "questions": "* Clarification on \"Similar Effects as Parameter Scaling\": The paper claims that PARSCALE has \"similar effects as parameter scaling.\" I would appreciate clarification on this point. Does this mean that for an equivalent increase in total computation (FLOPs), scaling P yields a comparable performance improvement to scaling parameters? Or does it primarily mean that the performance improvements from scaling P can be modeled by a scaling law with a similar mathematical form (i.e., a power law) to that of parameter scaling?\n* Optimizing KV Cache Memory Efficiency: The PARSCALE approach can be viewed as increasing the effective batch size of computation over a shared set of parameters, which successfully improves the compute-to-memory-access ratio for the model weights. However, as implemented, the KV cache size scales linearly with P, which could become the next memory bottleneck, especially for long contexts. Have the authors considered or explored strategies to optimize this? For example, could the P streams potentially share or contribute to a common KV cache, particularly for the initial shared context tokens, to further improve memory efficiency?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength：\n\n* The paper is supported by a comprehensive suite of experiments. The authors have done a thorough job of validating the proposed parallel scaling law and demonstrating the method's effectiveness across a wide range of model sizes, datasets, and training regimes.\n* The proposed method is elegant in its simplicity and directness. By promoting parameter reuse, it effectively improves the compute-to-memory-access ratio, a critical bottleneck for LLM inference. This characteristic makes the approach highly practical and gives it significant potential for deployment in resource-constrained scenarios, such as on-device and edge computing.\n\nWeakness：\n\n* Lack of Comparison with Alternative Parallel Architectures: The paper compellingly demonstrates the benefits of PARSCALE's \"end-to-end\" parallel design. However, the analysis could be strengthened by comparing it against other potential parallel computation strategies. For instance, what would be the effect of introducing information exchange between the P streams at intermediate layers? A more tightly-coupled design could involve expanding hidden states into P branches within each layer, performing parallel computations, and then fusing them before the next layer. A discussion or empirical comparison—analyzing both model performance and hardware efficiency (e.g., latency under different batching scenarios)—would help clarify whether PARSCALE's approach of P independent forward passes is inherently more advantageous than these more integrated alternatives.\n* Absence of a \"Brute-Force\" Upper Bound: The paper compares PARSCALE to parameter-scaled models with better inference efficiency. However, a crucial baseline is missing: a standard dense model that is P-times wider, which would have a roughly equivalent computational budget (FLOPs) to a PARSCALE model with P streams. While this \"wide\" model would be highly inefficient due to its low compute-to-memory-access ratio, including it as a theoretical \"upper bound\" would be highly informative. It would help quantify the performance trade-off inherent in reusing parameters versus simply adding more, providing a clearer picture of how effectively PARSCALE closes the gap with a brute-force scaling approach for a given computational cost.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and constructive feedback on our manuscript. We are delighted that you found our work to be supported by a `comprehensive suite of experiments` and that you appreciated the `elegant simplicity and directness` of our proposed method. We are particularly encouraged by your recognition of its potential for improving the compute-to-memory-access ratio, which makes it `highly practical for resource-constrained scenarios`. Below are our responses to your concerns:\n\n> W1: Lack of Comparison with Alternative Parallel Architectures: The paper compellingly demonstrates the benefits of PARSCALE's \"end-to-end\" parallel design. However, the analysis could be strengthened by comparing it against other potential parallel computation strategies. For instance, what would be the effect of introducing information exchange between the P streams at intermediate layers? A more tightly-coupled design could involve expanding hidden states into P branches within each layer, performing parallel computations, and then fusing them before the next layer. A discussion or empirical comparison—analyzing both model performance and hardware efficiency (e.g., latency under different batching scenarios)—would help clarify whether PARSCALE's approach of P independent forward passes is inherently more advantageous than these more integrated alternatives.\n\n- Thank you for this excellent and thought-provoking suggestion. We would like to first clarify that our primary goal is to demonstrate that scaling parallel computation is a new, effective dimension for improving a model's intrinsic capability. Given our computational budget constraints, we chose the most direct implementation — scaling based on the CFG-like techniques —to validate this concept. **Whether other parallel architectures are effective does not affect the core contribution and conclusion of this paper.** \n\n- In fact, regarding alternative parallel architectures, we did explore several variations in Appendix B (Table 5), which showed minimal performance differences. This initial finding suggested that **the amount of parallel computation, rather than the specifics of the architecture, was the dominant factor.**\n\n- Furthermore, to directly test your insightful hypothesis of a more \"tightly-coupled\" design, we implemented a new variant that averages the K/V outputs at each layer before branching out again, based on 0.6B model pre-trained on StackV2 for 42B tokens, and reported the training loss.\n\n\n| P | ParScale | ParScale (shared KV-cache) |\n| --- | --- | --- |\n| 1 | 1.1518 | \\- |\n| 2 | 1.1276 | 1.1282 |\n| 4 | 1.1145 | 1.1135 |\n| 8 | 1.1019 | 1.1034 |\n\nThis new result strongly reinforces our central thesis: **it is the act of scaling parallel computation itself, not the specific architectural implementation, that drives the performance gains.** We will add this valuable comparison to the paper (Appendix B, Table 5) to make our contribution clearer.\n\n> W2: Absence of a \"Brute-Force\" Upper Bound: The paper compares PARSCALE to parameter-scaled models with better inference efficiency. However, a crucial baseline is missing: a standard dense model that is P-times wider, which would have a roughly equivalent computational budget (FLOPs) to a PARSCALE model with P streams. While this \"wide\" model would be highly inefficient due to its low compute-to-memory-access ratio, including it as a theoretical \"upper bound\" would be highly informative. It would help quantify the performance trade-off inherent in reusing parameters versus simply adding more, providing a clearer picture of how effectively PARSCALE closes the gap with a brute-force scaling approach for a given computational cost.\n\nThis is an excellent point, and we are happy to clarify that our paper already includes this \"brute-force\" upper bound comparison. The baseline you suggested—a wider dense model—is exactly what we refer to as our \"parameter scaling\" baseline (explained in Line 887). Our scaling law analysis provides a direct answer to your question about the performance trade-off. We quantified this gap with the following capacity ratio:\n\n$$\\frac{\\text{Capacity of ParScale}}{\\text{Capacity of brute-force upper bound}}=\\frac{N(k\\log P+1)}{NP}=\\frac{k\\log P+1}{P}$$\n\n> Q1: Clarification on \"Similar Effects as Parameter Scaling\": The paper claims that PARSCALE has \"similar effects as parameter scaling.\" I would appreciate clarification on this point. Does this mean that for an equivalent increase in total computation (FLOPs), scaling P yields a comparable performance improvement to scaling parameters? Or does it primarily mean that the performance improvements from scaling P can be modeled by a scaling law with a similar mathematical form (i.e., a power law) to that of parameter scaling?\n\n- Thank you for your question. We would like to clarify that your second understanding is correct: scaling P is similar to scaling the model parameter by $O(\\log(P))$, as modeled by the scaling law.\n\n> Q2: Optimizing KV Cache Memory Efficiency: The PARSCALE approach can be viewed as increasing the effective batch size of computation over a shared set of parameters, which successfully improves the compute-to-memory-access ratio for the model weights. However, as implemented, the KV cache size scales linearly with P, which could become the next memory bottleneck, especially for long contexts. Have the authors considered or explored strategies to optimize this? For example, could the P streams potentially share or contribute to a common KV cache, particularly for the initial shared context tokens, to further improve memory efficiency?\n- Thank you for your suggestion. As mentioned above (in our response to W1), we conducted this experiment, where the KV cache is averaged and shared across different streams. The results show that ParScale can further reduce KV cache usage while maintaining similar performance.\n---\nThank you again for your invaluable feedback. We hope our response could effectively address your concerns. Given the extend of our evaluation, we would be grateful if you would reconsider your evaluation of our manuscript."}, {"review_id": "D1OPD818ZV", "reviewer": "Reviewer_UM4q", "summary": "This paper presents a brand-new way of thinking about Scaling Laws, focusing more on expanding a model's capabilities (as seen in the loss curve) by increasing the amount of computation rather than the computational space (i.e., the number of model parameters).", "questions": "1. Under current application conditions, can Parallel Scaling demonstrate a sufficient performance advantage over MoE and Inference-Time Scaling? If not now, is there hope for it in the future?\n\n2.  Under current conditions, is the Parallel Scaling method robust enough? Compared to sample-level scaling like self-consistency, and considering the method's huge resource consumption (even just in post-training), is the proposed prefix extension more cost-effective than no-training alternatives (like using Dropout) or using MoE for parallel scaling?", "limitations": "yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nThis paper dedicates significant resources to discussing the Parallel Scaling Law, which I believe is a very promising approach. Recently, the decoding and training speeds of LLMs are often bottlenecked by communication rather than raw computation speed. This makes scaling up the amount of computation particularly important, especially in a context where GPU computational performance is growing rapidly while communication bandwidth and GPU memory remain significant constraints. I was impressed by the clear and detailed experiments in this paper that introduce this novel scaling method.\n\n\n**Weaknesses**\n\nThe paper lacks a more in-depth comparison with methods like Self-consistency. Although Self-consistency is, to some extent, a form of sample-level scaling, it has shown benefits in many areas, including reasoning and question answering. On the other hand, when considering a universal method like Universal Self-Consistency for Large Language Model Generation, I find it difficult to think of a generative AI scenario where the Parallel Scaling Law could be applied, aside from real-time streaming generation.\n\nThe paper explores, at most, a scenario with P=8. As far as I know, methods like Self-consistency often have an upper limit; their performance growth slows down significantly once the number of samples reaches around 1000. I believe it is too early to determine the curve's trend for the Parallel Scaling Law with P=8.\n\nFrom a cost perspective, the computation in this paper is equivalent to P x L x T, whereas for inference scaling law, it is L x T'. When the computational cost at inference time is comparable—for instance, generating 1000 words with P=8 versus generating 8000 words with P=1 (the latter might have a longer wait time, but I believe this setup is a reasonable approximation when considering resource consumption and throughput in high-concurrency scenarios)—I am curious which would perform better. Considering that in the MATH dataset, the Qwen3-235B-A22B model improved from a base of 71.84 to 98 through inference scaling law, or that the same Qwen3-235B-A22B model, using a \"thinking\" mode, improved its MATH score from 91.2 (non-thinking) to 98, and its CodeForces score from 1387 to 2056, I think it's difficult for the Parallel Scaling Law, based on Table 4, to demonstrate this level of performance improvement.\n\nI am not sure if the scaling method proposed in this paper is effective enough, and the paper itself lacks corresponding comparative experiments. For example, if I use a baseline model with P=1, I could use a Dropout-like approach where I input the same P=8 input texts during each forward pass and then, in the final layer of the decoding process, merge these eight parallel Hidden States (e.g., by averaging them) to predict the next new token. This method requires no training, but I believe it could still yield similar results.\n\nAs another example, one could also test this by directly evaluating a Mixture of Experts (MoE) model. For instance, the DeepSeek-V3/R1 model activates 8 experts per token. Therefore, one could sequentially activate P=1, P=2, P=4, P=8, P=16, P=32 experts, which would also allow for scaling without the need for training.", "was_revised": true, "final_justification": "I sincerely thank the authors for their detailed experiments. The provided experimental results have fully addressed all of my concerns. I appreciate that the authors have offered clear, empirical evidence. This is a strong paper, and I recommend it for acceptance.", "rebuttal": "We would like to express our sincere gratitude for your thorough and insightful review of our manuscript. We are particularly encouraged that you found our work on the Parallel Scaling Law to be a promising approach for `addressing the communication bottlenecks` in modern LLMs. We appreciate you recognizing the value of our `clear and detailed experiments` and acknowledging our contribution in introducing this `novel scaling method`. Below are our responses:\n\n> W1: The paper lacks a more in-depth comparison with methods like Self-consistency. Although Self-consistency is, to some extent, a form of sample-level scaling, it has shown benefits in many areas, including reasoning and question answering. On the other hand, when considering a universal method like Universal Self-Consistency for Large Language Model Generation, I find it difficult to think of a generative AI scenario where the Parallel Scaling Law could be applied, aside from real-time streaming generation.\n\n- Thank you for your suggestion. The table below presents the performance comparison on GSM8K and MATH using ParScale-1.8B (P=1) with self-consistency (SC), showing that ParScale consistantly performs better than SC.\n\n|  | ParScale | SC (temperature = 0.2) | SC (temperature = 0.8) |\n| --- | --- | --- | --- |\n| **GSM8K** |  |  |  |\n| P=2 | **32.6** | 28.2 | 19.7 |\n| P=4 | **34.7** | 31.5 | 25.7 |\n| P=8 | **38.4** | 32.8 | 32.0 |\n| **MATH** |  |  |  |\n| P=2 | **13.0** | 12.0 | 7.5 |\n| P=4 | **14.5** | 12.6 | 8.5 |\n| P=8 | **16.4** | 14.5 | 10.8 |\n\nWe would also like to provide a more in-depth comparison of these two methods:\n\n- **Foundational Method vs. Inference Technique**: ParScale is a foundational method that enhances a model's intrinsic capacity, making it directly comparable to parameter scaling. In contrast, SC is an inference-time sampling technique applied as a post-processing step. They operate at different levels.\n- **Broader and Distinct Applicability**. We respectfully clarify that ParScale can be used for critical generative AI scenarios where methods like SC are impractical, such as real-time streaming generation and open-ended dialog. Additionally, unlike SC, ParScale improves the single-shot generation quality. This is crucial for tasks requiring a single, high-quality output, or in specific architectural configurations (e.g., when P=2) where a simple majority vote is not applicable.\n- **Generalizable Principle**. The concept of parallel scaling is a general architectural principle with the potential to extend beyond LLMs to other domains like diffusion models and MLPs, particularly in memory-limited contexts. SC is limited for text generation.\n\n> W2: The paper explores, at most, a scenario with P=8. As far as I know, methods like Self-consistency often have an upper limit; their performance growth slows down significantly once the number of samples reaches around 1000. I believe it is too early to determine the curve's trend for the Parallel Scaling Law with P=8.\n\n- Thank you for your insightful comment. We agree that the performance for P >> 8 is an interesting open question and we consider it a direction for future work due to our computation budget (our experiments already take up above 10^6 GPU-h). We have mentioned this in line 1007 (Future work).\n\n> W3: From a cost perspective, the computation in this paper is equivalent to P x L x T, whereas for inference scaling law, it is L x T'. When the computational cost at inference time is comparable—for instance, generating 1000 words with P=8 versus generating 8000 words with P=1 (the latter might have a longer wait time, but I believe this setup is a reasonable approximation when considering resource consumption and throughput in high-concurrency scenarios)—I am curious which would perform better. Considering that in the MATH dataset, the Qwen3-235B-A22B model improved from a base of 71.84 to 98 through inference scaling law, or that the same Qwen3-235B-A22B model, using a \"thinking\" mode, improved its MATH score from 91.2 (non-thinking) to 98, and its CodeForces score from 1387 to 2056, I think it's difficult for the Parallel Scaling Law, based on Table 4, to demonstrate this level of performance improvement.\n\n- Thanks for you valuable and interesting analysis! First, we kindly argue that **direct controlling cost for comparison is challenging.** The computational cost of inference time scaling is unpredictable due to the variable token count (T'), whereas our method's cost is deterministic and controllable via the number of parallel instances (P).\n- Second, we wish to emphasize that parallel scaling and serial scaling (e.g., extending reasoning tokens) are **not mutually exclusive**. As shown in our Table 4, combining CoT with ParScale yields further improvements on GSM8K. For future work, we plan to investigate the integration of ParScale with reasoning models like Qwen3.\n- Furthermore, we would like to clarify the concept of \"cost.\" **While FLOPs is a key metric, it does not capture the full picture of practical resource utilization.** ParScale fully leverages hardware parallelism, which serial generation inherently underutilizes. This provides a crucial efficiency advantage and this benefit will grow with more powerful parallelization hardware.\n- Finally, from a generality perspective, our method is a universal technique that applies to any model structure, unlike the inference-time scaling which is only applicable to text generation. This offers broader applicability than approaches relying on specialized training data or architecturally-ingrained thinking modes.\n\n> W4: I am not sure if the scaling method proposed in this paper is effective enough, and the paper itself lacks corresponding comparative experiments. For example, if I use a baseline model with P=1, I could use a Dropout-like approach where I input the same P=8 input texts during each forward pass and then, in the final layer of the decoding process, merge these eight parallel Hidden States (e.g., by averaging them) to predict the next new token. This method requires no training, but I believe it could still yield similar results.\n\n- Thank you for your suggestion. Regarding the mentioned baseline (i.e., MC Dropout), **it is shown in the literature [1] that it yields only marginal improvements and is significantly underperformed by training-based methods.** To further address your concern, we trained a 0.6B model on Stack V2 using MC Dropout and recorded its loss, as shown in the table below:\n\n| P | ParScale (no dropout) | MC Dropout (dropout=0.1) |\n| --- | --- | --- |\n| 1 | 1.1518 | 1.2064 |\n| 2 | 1.1276 | 1.2016 |\n| 4 | 1.1145 | 1.1923 |\n| 8 | 1.1019 | 1.1909 |\n\nThe results above once again demonstrate the importance of scaling parallel computation during the training phase. It is noteworthy that the pre-training recipes for most modern LLMs (like Qwen and Llama) have omitted dropout, making it difficult to apply MC Dropout.\n\n\n> W5: As another example, one could also test this by directly evaluating a Mixture of Experts (MoE) model. For instance, the DeepSeek-V3/R1 model activates 8 experts per token. Therefore, one could sequentially activate P=1, P=2, P=4, P=8, P=16, P=32 experts, which would also allow for scaling without the need for training.\n- Thank you for your suggestion. We conducted this experiment on Qwen3 30B-A2B MoE model, which has 128 experts and similarly activates 8 experts per token. The table below presents its performance on GSM8K, MATH, and EvalPlus:\n\n| Activated Experts | GSM8K | MATH | EvalPlus |\n| --- | --- | --- | --- |\n| 8 (original) | 91.2 | 32.7 | 71.3 |\n| 16 | 91.2 | 29.4 | 71.1 |\n| 32 | 84.9 | 28.7 | 66.1 |\n\nAs can be seen, performance actually degrades as the number of activated experts increases. This demonstrates that **the best results are achieved only when the training and inference setups are kept consistent.**\n\n> Q1: Under current application conditions, can Parallel Scaling demonstrate a sufficient performance advantage over MoE and Inference-Time Scaling? If not now, is there hope for it in the future?\n\n- As stated in the response to W3, Parallel Scaling and Parameter/Inference-Time Scaling are orthogonal and can be combined. In our paper, we demonstrate that Parallel Scaling offers a greater inference cost advantage, particularly when the batch size is small. We also hope that future work could further raise the value of $k$ (i.e., the performance gain for ParScale) and even break the bound of $O(\\log P)$.\n\n> Q2: Under current conditions, is the Parallel Scaling method robust enough? Compared to sample-level scaling like self-consistency, and considering the method's huge resource consumption (even just in post-training), is the proposed prefix extension more cost-effective than no-training alternatives (like using Dropout) or using MoE for parallel scaling?\n\n- We have used extensive experiments (including Scaling Laws and downstream performance) to demonstrate the robustness of Parallel Scaling. Regarding the other baselines you mentioned, we have already established the importance of scaling parallel computation during the training phase in our response to W4 and W5.\n\n---\n\nThank you again for your invaluable feedback. We hope our response could effectively address your concerns. Given the extend of our experimental effort, we would be grateful if you would reconsider your evaluation of our manuscript.\n\n[1] LoRA Emsembles for Large Lanuage Model Fine-Tuning."}, {"review_id": "W0cO2ew4tF", "reviewer": "Reviewer_T6Gx", "summary": "This paper introduces Parallel Scaling (PARSCALE), a novel scaling strategy for language models that aims to increase parallel computation during both training and inference without significantly increasing the number of model parameters. The core idea is to execute multiple parallel streams of computation with dynamic aggregation of their outputs. The authors propose a theoretical scaling law and demonstrate through large-scale pretraining that PARSCALE achieves performance comparable to parameter scaling while offering substantial reductions in memory usage and latency. This makes the method particularly appealing in resource-constrained environments.", "questions": "1. How does PARSCALE’s scalability and efficiency vary with the number of GPUs in a cluster?\n2. Has PARSCALE been evaluated on more complex models like GPT or LLaMA, and is it compatible with MoE (Mixture-of-Experts) architectures?\n3. Does PARSCALE introduce additional communication overhead in distributed training or inference, and how does this overhead vary with different parallelization strategies (DP, TP, PP) ?", "limitations": "1. Lacking analysis of how PARSCALE performs across different GPU setups or hardware configurations.\n2. Lacking analysis of PARSCALE's communication volume and its scalability across different cluster scales or parallelisms.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "### Strengths\n1. Innovative Scaling Strategy: PARSCALE presents a fresh approach to model scaling by increasing computational parallelism instead of parameter count.\n2. Solid Theoretical Foundation: The proposed scaling law is well-formulated and empirically validated, providing strong theoretical support for the method.\n3. Efficiency Gains: PARSCALE demonstrates significant improvements in memory and latency efficiency, which is particularly beneficial for deployment on edge devices.\n4. Cost-Effective Training: The two-stage training strategy effectively reduces training costs, making large-scale model training more accessible.\n5. Strong Empirical Results: The experimental results are compelling, especially for reasoning-heavy tasks such as code generation and mathematical problem solving.\n5. Strong Experimental Results: The paper presents compelling experimental evidence, showing significant improvements in performance for reasoning-intensive tasks, particularly in code generation and math.\n\n### Weaknesses\n1. Scalability on Larger Models: The paper lacks an in-depth evaluation of PARSCALE’s scalability on very large models and across diverse architectures such as GPT and LLaMA.\n2. Hardware Variability: While memory and latency improvements are reported, the paper does not discuss how these benefits hold across different hardware environments (e.g., GPU types or configurations).\n3. Multi-GPU Scaling: There is no exploration of how PARSCALE performs with varying numbers of GPUs, which is crucial for understanding its hardware scalability.", "was_revised": true, "final_justification": "This is a very solid paper. The rebuttal has removed all my concerns. I agree to accept this paper.", "rebuttal": "We would like to extend our sincere gratitude for your thoughtful and constructive feedback on our manuscript. We are pleased that you recognized the core contributions of our paper. We appreciate your acknowledgment of the `Solid Theoretical Foundation` of our proposed scaling law, which is both well-formulated and empirically validated. We are also grateful for your positive feedback on the significant `Efficiency Gains` in memory and latency that ParScale provides, its potential for enabling `Cost-Effective Training`, and the `Strong Empirical Results` we demonstrated. \n\nBelow are our responses to your concerns: \n\n> W1: Scalability on Larger Models: The paper lacks an in-depth evaluation of PARSCALE’s scalability on very large models and across diverse architectures such as GPT and LLaMA.\n\n> Q2: Has PARSCALE been evaluated on more complex models like GPT or LLaMA, and is it compatible with MoE (Mixture-of-Experts) architectures?\n\nThank you for your valuable comments.\n- Larger Model: Our study on models from 0.6B to 4.4B aligns with standard practices in scaling law research. Given the substantial computational cost (over 10^6 GPU hours), we defer training even larger models to future work, which will be guided by our proposed scaling law. As stated in Line 196, we believe that larger models will benefit even more from ParScale.\n- Architectures (GPT/Llama): We chose Qwen as it is a representative modern Transformer, sharing its core design with GPT and Llama. This architecture is not less complex than GPT and Llama. Our conclusions are thus expected to be broadly applicable across these standard architectures.\n- MoE Compatibility: ParScale is fully compatible with MoE and we plan to investigate their combination, as we stated in the future work section (Line 1014).\n\n> W2: Hardware Variability: While memory and latency improvements are reported, the paper does not discuss how these benefits hold across different hardware environments (e.g., GPU types or configurations).\n\n- Thank you for this important point on hardware variability. The memory and latency benefits of ParScale stem from fundamental principles—improving arithmetic intensity (the ratio of compute to memory access) and reducing the overall memory footprint. **Since these principles are universal to all modern parallel processors, the advantages are not tied to a specific GPU architecture.** While our experiments were conducted on industry-standard NVIDIA A100 GPUs, the observed efficiency gains are expected to be robust and generalizable across different hardware environments, including various GPU types and configurations.\n\n> W3: Multi-GPU Scaling: There is no exploration of how PARSCALE performs with varying numbers of GPUs, which is crucial for understanding its hardware scalability.\n\n> Q1: How does PARSCALE’s scalability and efficiency vary with the number of GPUs in a cluster?\n\n> Q3: Does PARSCALE introduce additional communication overhead in distributed training or inference, and how does this overhead vary with different parallelization strategies (DP, TP, PP) ?\n\n- Thank you for these insightful questions regarding hardware scalability. ParScale is specifically designed to be highly efficient in multi-GPU settings because it operates on the Data Parallelism (DP) dimension. It works by partitioning a single data sample into independent parallel streams that are processed without any cross-communication (except for the input transformation and output aggregation).\n- Consequently, unlike TP or PP which introduce significant communication overhead, ParScale adds nearly no additional communication cost. Its scalability profile is therefore similar to that of standard DP, which is well-known for its high efficiency and near-linear scaling with the number of GPUs.\n\n---\n\nThank you again for your encouraging feedback. We hope our response could fully address your concerns."}]

Author rebuttal:

Meta-review: All reviewers agreed this paper should be accepted: it contains a comprehensive suite of experiments, the proposed method is elegant, and the results are impressive. The main concerns were around framing: e.g., reviewer Qbt2 worried that all experiments were in a regime with a small batch size (≤ 8). The authors responded by clarifying that they are targeting low-resource scenarios. This an important, often unexplored regime for language models. Given this and the author responses to other reviewer concerns, this paper is a clear accept.

One aspect that did not come up in the discussion but I would urge the authors to consider is the strength of the claim of power law discovery. Something that has not received enough attention in the LM power-law research community is that it is notoriously difficult to identify such power laws, see e.g.,

Clauset, Aaron, Cosma Rohilla Shalizi, and Mark EJ Newman. "Power-law distributions in empirical data." SIAM review 51.4 (2009): 661-703.

Using the Huber loss is a standard trick to mitigate the sensitivity of the squared loss to outliers, however it is not at all obvious that this fitting procedure identifies a ground-truth power law. I encourage the authors to take this into account when making changes to the manuscript for the camera-ready version. Thank you!
Decision: Accept (poster)