Capability Salience Vector: Fine-grained Alignment of Loss and Capabilities for Downstream Task Scaling Law
Authors: Qiming Ge, Shuhao Xing, Songyang Gao, Yunhua Zhou, Yicheng Zou, Songyang Zhang, Zhi Chen, Hang Yan, Qi Zhang, Qipeng Guo, Kai Chen
Accepted at: acl25

Capability Salience Vector: Fine-grained Alignment of Loss and Capabilities for Downstream Task Scaling Law

Qiming Ge*, Shuhao Xing**, Songyang Gao◊, Yunhua Zhou◊, Yicheng Zou†, Songyang Zhang◊, Zhi Chen◊, Hang Yan◊, Qi Zhang*, Qipeng Guo◊, Kai Chen◊
◊ Shanghai AI Laboratory
*College of Computer Science and Artificial Intelligence, Fudan University
qmge22@m.fudan.edu.cn, zouyicheng@pjlab.org.cn

Abstract
Scaling law builds the relationship between training computation and validation loss, enabling researchers to effectively predict the loss trending of models across different levels of computation. However, a gap still remains between validation loss and the model’s downstream capabilities, making it untrivial to apply scaling law to direct performance prediction for downstream tasks. The loss typically represents a cumulative penalty for predicted tokens, which are implicitly considered to have equal importance. Nevertheless, our studies have shown evidence that when considering different training data distributions, we cannot directly model the relationship between downstream capability and computation or token loss. To bridge the gap between validation loss and downstream task capabilities, in this work, we introduce Capability Salience Vector, which decomposes the overall loss and assigns different importance weights to tokens to assess a specific meta-capability, aligning the validation loss with downstream task performance in terms of the model’s capabilities. Experiments on various popular benchmarks demonstrate that our proposed Capability Salience Vector could significantly improve the predictability of language model performance on downstream tasks.

1 Introduction
Large language models have demonstrated impressive performance across a wide range of tasks, but this achievement comes with the trade-off of significant computational demands. To mitigate the computational burden during the training of large models, researchers have developed the scaling law (Bahri et al., 2024; Hoffmann et al., 2022; Kaplan et al., 2020; Muenninghoff et al., 2024), a framework that predicts how variations in model size and data scale impact the validation loss of LLMs. This allows researchers to use smaller models to predict the validation loss of larger ones, reducing the cost of hyperparameter tuning and trial-and-error experimentation. However, in practical applications, a model’s capabilities are typically assessed using a variety of downstream benchmarks, and validation loss does not always correlate directly with performance on these tasks. It is not uncommon to encounter instances where models with similar validation losses exhibit significantly different downstream task performance (Liu et al., 2023). Consequently, validation loss alone is not always a reliable indicator of a model’s true capabilities.

To address this issue, some researchers have chosen to bypass validation loss, focusing instead on directly establishing the scaling relationship between computational resources and task performance (Gadre et al., 2024). To minimize the effects of data distribution shifts, they adjusted the amount of computation and model sizes while keeping the pre-training dataset fixed across all experimental settings. However, we argue that the impact of data distribution shifts cannot be overlooked, as they may substantially influence model performance on downstream tasks. As demonstrated by our experiments (Section 3.1), the same amount of computation can yield markedly different outcomes when models are trained on distinct data distributions.

Another line of research focuses on developing alternative metrics that better correlate with downstream task performance. For instance, Ruan et al. (2024) proposed the Observational Scaling Law (OSL), leveraging open-source benchmark results and applying principal component analysis (PCA) to explore various dimensions of model capabilities. While these approaches encompass a broad range of model families that are pre-trained on different data distributions, they depend heavily on the diversity and scope of the benchmarks used, which can incur substantial evaluation costs.
From the discussions above, we identify two important but underexplored questions that this work seeks to address:

• Is it reasonable to predict downstream capabilities directly based on computational cost?

• Why is it challenging to establish a clear relationship between validation loss and a model’s capabilities?

To address the first question, we explored the impact of different pre-training data distributions on the ability to model downstream tasks. Our experiments show that, under the same computational budget, different pre-training data distributions can lead to varying downstream capabilities. Therefore, it is not reasonable to predict downstream task performance directly based on computational cost. To answer the second question, we revisit the concept of validation loss, a metric used to evaluate a model’s cumulative penalty for next-token predictions. This metric inherently assumes that all tokens contribute equally to the model’s training process. However, previous studies have demonstrated that tokens may differ in their learnability and difficulty (Lin et al., 2024; Xia et al., 2022). Inspired of this, we introduce the concept of Capability Salience Vector (CSV), which quantifies the capabilities required for downstream tasks by assigning different importance weights to tokens. By computing a weighted average of the loss for these tokens, this metric can replace the raw validation loss and demonstrate a stronger correlation with downstream task performance. Furthermore, we propose an optimization algorithm to automatically derive the Capability Salience Vector for any given text. Experiments conducted on six popular benchmarks show that our approach consistently achieves a high correlation across different model series and significantly enhances the predictability of language model performance on downstream tasks.

Overall, our contributions are threefold:

• Our experiments indicate that downstream scaling laws cannot be directly modeled using computation or the average token loss on the validation set.

• We introduce the Capability Salience Vector and present an algorithm to automatically derive it. This allows us to quantify a model’s capabilities by assigning different importance weights to the loss of specific tokens, thereby establishing a strong correlation between validation loss and downstream task performance.

• Through experiments on six benchmarks for different model capabilities, we show that our method consistently achieves high correlation across various model series and significantly improves the predictability of model performance on these benchmarks. The predicted mean squared error was maintained within the range of 1e-3.

2 Related work

2.1 Scaling Law for Downstream Tasks

Scaling law establishes the power-law relationships between computation \( C \), model parameters \( N \), the number of training tokens \( D \), and cross-entropy loss on the validation set. However, there is still a gap between validation loss and downstream task performance. Recent studies have tried to investigate the scaling laws governing downstream task performance. Owen (2024) investigated the predictability of model scaling on BBH (Srivastava et al., 2022) and MMLU (Hendrycks et al., 2020) tasks, proposing that downstream task performance follows a sigmoidal function and can be predicted. Their findings also indicate that prediction errors decrease as more models are incorporated into the fitting process. Ruan et al. (2024) introduced an alternative observational approach, applying low-rank decomposition to publicly available model evaluation results to extract sub-dimensions representing the model capabilities. These sub-dimensions were then used to establish scaling laws that relate computational cost to downstream task performance. Gadre et al. (2024) examined the scaling laws of overtraining models on specific pre-training data distributions. They identified a power-law relationship between validation loss and downstream task performance, demonstrating predictability after pre-training on a given data distribution.

However, the aforementioned studies primarily focused on the relationship between computational cost and downstream task performance, without addressing the fact that different pre-training data distributions can result in different downstream performances for the same computational cost. Isik et al. (2024) explored the impact of pre-training data distribution on downstream performance in translation tasks. Their findings revealed that different
pre-training distributions significantly influence the scaling behavior of downstream tasks, indicating that data distribution should be considered when evaluating model scaling laws.

2.2 Efficient Benchmark Prediction
As large language models continue to advance, numerous benchmarks have been developed to assess their diverse capabilities. To comprehensively evaluate the performance of a pre-trained model, it is necessary to measure its scores across various downstream tasks. However, conducting a full downstream evaluation is often time-consuming, prompting recent efforts to improve its efficiency. For example, Ye et al. (2023) and Perlitz et al. (2023) have explored reducing the number of task examples in benchmarks like BBH (Srivastava et al., 2022) and HELM (Liang et al., 2022) to streamline the evaluation process.

Recent studies, such as Polo et al. (2024) and Kipnis et al. (2024), have explored leveraging Item Response Theory (Lord and Novick, 2008; Baker, 2001) to extract model features from open-source evaluations and compress validation sets. This allows for the reconstruction of benchmark scores using these compressed subsets. Similarly, Pacchiardi et al. (2024) applied a feature extraction approach and trained an evaluator to directly predict performance on individual samples. Zhang et al. (2024) demonstrated that incorporating task performance across different model families, alongside additional model and task information, can enhance prediction accuracy. However, these methods still rely on accuracy metrics for part of the data, and it remains uncertain whether downstream task performance (e.g., accuracy) can be directly predicted based on downstream scaling laws.

3 Method
In this section, we first explore whether downstream task performance can be directly predicted using computation or average token loss. Then, we introduce Capability Salience Vector (CSV), which automatically assign different importance weights to validation token loss. This aligns the loss of tokens with the specific capabilities required for downstream tasks.

3.1 Impact of Pre-training Data Distribution on Modeling Downstream Scaling Law
Previous work on downstream scaling laws has primarily explored whether the performance of large models on downstream tasks can be predicted using pretrained small models. These studies follow a common setting, where small models trained on the same pretraining data distribution are used to predict the performance of larger models. However, during the pretraining process, researchers may need to continuously adjust the data composition to compare the impact of different data distributions on model performance. In this framework, applying downstream scaling laws would require training multiple small models under different configurations with the sa data distributions, making the experiments significantly more costly.

In this section, we first investigate whether downstream scaling laws can be directly modeled using computational cost. Specifically, we use InternLM2.5 as the base architecture and train two series of models on significantly different pretraining data distributions which is the same pretraining data sources as InternLM2.5. These sources include a diverse mix of web data, code data, math data, and other domain-specific datasets. We only adjusted the proportions of different subsets within this pretraining corpus. We then select checkpoints at different levels of computational cost to evaluate downstream performance.

Figure 1 illustrates the performance of two models with different training data distributions on various downstream tasks under different computational budgets. We observe that for tasks such as

![Relationship between computation (FLOPs) and performance (accuracy) across various downstream tasks is depicted, with red and blue points indicating model checkpoints trained on two different data distributions. Under the same computational budget, models trained on different data distributions exhibit varying downstream task performance.](page_768_120_627_496.png)

Figure 1: Relationship between computation (FLOPs) and performance (accuracy) across various downstream tasks is depicted, with red and blue points indicating model checkpoints trained on two different data distributions. Under the same computational budget, models trained on different data distributions exhibit varying downstream task performance.
Figure 2: The relationship between average token loss and downstream task performance.

Hellaswag and BBH, models trained with the same computational cost exhibit differing downstream task performance. This indicates that a direct function from computation to downstream task performance cannot be reliably modeled. Therefore, when predicting downstream task performance, we must account for differences in the models’ training data distributions. Moreover, language models exhibit emergent capabilities in certain tasks, making it difficult to predict their performance using smaller models trained on the same pretraining data distribution. Before reaching the emergence threshold—when computational resources are insufficient and loss remains relatively high—the performance of small models on downstream tasks becomes difficult to measure (Du et al., 2024).

This raises an important question: Is there a suitable metric that satisfies the following conditions?

• The metric provides continuous measurements and is independent of emergent capabilities.

• The metric reflects differences in data distribution and can be predicted during pretraining.

Inspired by traditional scaling laws, we consider validation loss as a potential candidate. Validation loss is continuous and serves as an indicator of model capability. Therefore, we aim to model the relationship between validation loss and downstream performance. To explore this, we measure the downstream performance of different open-source models and their average token loss on task validation datasets. To avoid biases introduced by different tokenizers across models, we normalize the loss calculation at the character level.

Figure 2 illustrates the relationship between validation loss and downstream performance in MMLU and BBH. We observe that models can exhibit varying downstream performance across different tasks with the same validation loss. We revisit the computation of validation loss. Traditional scaling laws calculate validation loss by averaging it at the dataset level, treating all loss values with equal importance. However, different loss values may contribute unequally to measuring a model’s task specific capability. This indicates that a more fine-grained approach may be necessary for modeling loss effectively. In the next section, we introduce the concept of the Capability Salience Vector (CSV). This method assigns different importance weights to token-level loss values in the validation set, aiming to establish a more predictable relationship between validation loss and downstream performance.

3.2 Modeling Downstream Scaling Law

Inspired by the work of Arora and Goyal (2023), we consider that predicting different token fragments requires different meta-capabilities. These meta-capabilities ultimately determine the performance of downstream tasks. Traditional scaling laws treat all tokens equally when modeling validation loss. While the expectation of validation loss provides a rough estimate of model capability, it does not accurately capture the relationship between loss and downstream performance. To solve this problem, we propose the Capability Salience Vector. By applying a simple linear weighting to loss calculation, we can better model the relationship between loss and downstream performance. Given a validation set \( S = \{X_s\}_{s=1}^{|S|} \), we first compute the validation loss of each sample \( X_s = (x_1, x_2, ..., x_{|X_s|}) \). Next, we use the Capability Salience Vector, denoted as \( W = \{w_{s,i}\} \), to apply linear weighting to each token’s loss. This allows us to compute the capability score \( C_m \), which represents the model’s ability based on the validation text.

\[
C_m = -\frac{1}{N_c} \sum_{s=1}^{|S|} \sum_{i=1}^{|X_s|} w_{s,i} \log p(x_i \mid x_{<i}).
\]

where \( N_c \) is the number of validation text characters. Then we use the sigmoidal function to model the functional relationship between capability score \( C_m \) and the model’s downstream task performance (e.g., accuracy):

\[
A_{t,m} = \gamma + \frac{1-\gamma}{1+\exp(-\alpha(C_m-\beta))}.
\]

where \( \alpha \) and \( \beta \) is the parameters to be fitted, \( \gamma \) represents the expected performance of the task under random guessing.
Figure 3: Capability salience vector establishes the relationship between cross-entropy loss and downstream task performance. It quantifies a model’s capabilities as a weighted sum of token losses, with each token contributing differently to the overall performance.

3.3 Modeling Capability Salience Vector

Since the weights of the Capability Salience Vector and the parameters \( \alpha, \beta \) are unknown, we use three steps to get the weights of capability Salience vector.

Extract Weights of Capability Salience Vector
For a given validation set, we first apply a loss mapping algorithm to align losses across different vocabulary spaces. Specifically, we average the token-level losses over their corresponding characters to obtain character-level losses, and then aggregate these character-level losses according to the tokenization result of the target tokenizer. Then we use language model with a scoring head to obtain capability Salience Vector weights \( W = \{w_{s,i}\} \) of each token:

\[
w_{s,i} = f_\theta(x_{s,i}|x_{s,<i}).
\]

where \( \theta \) is the parameters of scoring head. The scoring head is implemented as a linear layer applied to the final hidden states of a the language model and we froze the other parameters of language model during optimization. This step can be seen as assigning a score to each token, reflecting its contribution to the overall capability representation.

Fitting Downstream Scaling Law Function
After getting capability Salience Vector, we collect token losses on the given validation set and downstream task performance of different models to fitting downstream scaling law function. To do this, we fix \( \theta \) and estimate the parameters \( \alpha \) and \( \beta \) of the downstream scaling law function. We minimize the MSE loss between the predicted and observed downstream task performance by using the Levenberg-Marquardt algorithm.

\[
\min_{\alpha,\beta} \sum_{\text{Models } m} [A_{t,m} - \hat{A}_{t,m}]^2.
\]

Optimize Capability Salience Vector
In this stage, we use the downstream scaling law function to optimize Capability Salience Vector parameters \( \theta \). This is done by minimizing the MSE loss between predicted and observed downstream task performance using SGD:

\[
\min_{\theta} \sum_{\text{Models } m} [A_{t,m} - \hat{A}_{t,m}]^2.
\]

We run multiple iterations of the three-steps optimization process to select the parameters that achieve the best downstream performance prediction. These optimal parameters are saved to obtain the Capability Salience Vector and fit downstream scaling law function. Finally, our algorithm workflow is illustrated in Appendix B.

4 Experiment

To validate the effectiveness of our approach, we divide the experiment into two parts. First, we evalu-
ate the performance of the Capability Salience Vector using open-source models. Next, we use checkpoints from our own models to assess whether the Capability Salience Vector, optimized from open-source models, can accurately predict downstream task performance for models trained on different data distributions.

4.1 Settings

Dataset To explore the scaling capabilities of models across various downstream tasks requiring different abilities, we conducted our experiments on six popular benchmarks that cover a wide range of skills, including knowledge, reasoning, commonsense, and mathematics. For knowledge capabilities, we choose MMLU (Hendrycks et al., 2020). For reasoning capabilities, we choose BBH (Suzgun et al., 2022). For commonsense capabilities, we choose Hellaswag (Zellers et al., 2019). For mathematics capabilities, we choose Gsm8k (Cobbe et al., 2021). For Chinese language capabilities, we selected the CMMLU (Li et al., 2023) and Ceval (Huang et al., 2024) benchmarks.

Evaluation Models Our experiments were conducted under two different evaluation setups:

• Open-source Model Evaluation: We validate the effectiveness of our method on over 50 open-source models across different series, including LLAMA2 (Touvron et al., 2023), LLAMA3 (Dubey et al., 2024), Gemma2 (Team et al., 2024), Qwen1.5 (Bai et al., 2023), Qwen2 (Yang et al., 2024), Yi, Yi1.5(Young et al., 2024), InternLM2, and InternLM2.5 (Cai et al., 2024). In this setup, we use smaller models from the same series as the training set, while larger models are treated as the validation and test sets. The specific splits are detailed in Appendix A.

• Closed-source Model Evaluation: We train two different series of models under varying data distributions and aim to predict their downstream capabilities using the Capability Salience Vector. The train data source is the same as InternLM2.5. For Capability Salience Vector optimization, we use all the open-source models along with early checkpoints (the first 50k steps) from both series. One series of our own model is used as the validation set, while the other serves as the test set.

Evaluation Details We use the OpenCompass (Contributors, 2023b) tool to evaluate the downstream scores for the aforementioned models. OpenCompass is a python package that supports various large language models and datasets for evaluation and benchmarking. The detailed evaluation results for the open-source models are provided

Figure 4: Downstream task prediction results of open-source models using capability score derived from Capability Salience Vector.
Figure 5: Downstream task prediction results of closed-source models using capability score derived from Capability Salience Vector.

in Appendix C. To obtain the token cross-entropy loss on specific validation sets, we utilize LMDeploy(Contributors, 2023a) framework. To ensure the validation loss better reflects the different capabilities of the models, we randomly sample 100 examples from the six validation sets and mixed them. Additionally, we include 50 examples with chain-of-thought related to BBH. We treated CMMLU and CEval as reflecting the same capability and only sampled from one of these datasets. In total, we gathered 550 examples for optimizing the Capability Salience Vector. Our method can use only 550 examples as validation set for modeling downstream scaling law.

Baseline

• All token loss : This method simply calculate the average of all token loss in validation set. Then We use the metric to model downstream scaling law by fitting a sigmoidal function.

• Label token loss : Following the same setup as Llama3 (Dubey et al., 2024), we compute the negative log-likelihood of the correct answers on each task’s test set and calculate its expectation. We then use this metric to fit a sigmoidal function between fitting performance and downstream task performance.

Training Details We use InternLM2.5 1.8B (Cai et al., 2024) as the scoring model, replacing its language head with a linear layer followed by a nonlinear activation function. In practice, we found that increasing model complexity has little impact on the final results. To optimize efficiently, we freeze the language model parameters and only train the linear layer. We train with a learning rate of 1e-3, and most tasks converged to the best performance within 300 steps(training for 1-3 hours on a single A800.), demonstrating the practical efficiency of our method.

<table>
  <tr>
    <th rowspan="2">Task</th>
    <th colspan="3">Test Model Prediction MSE</th>
  </tr>
  <tr>
    <th>CSV</th>
    <th>All token</th>
    <th>Label token</th>
  </tr>
  <tr>
    <td>mmlu</td>
    <td><span style="color:red">1.45e-3</span></td>
    <td>2.40e-2</td>
    <td>3.31e-2</td>
  </tr>
  <tr>
    <td>bbh</td>
    <td><span style="color:red">2.26e-3</span></td>
    <td>5.81e-2</td>
    <td>8.27e-2</td>
  </tr>
  <tr>
    <td>gsm8k</td>
    <td><span style="color:red">6.32e-3</span></td>
    <td>7.46e-2</td>
    <td>9.88e-2</td>
  </tr>
  <tr>
    <td>hellaswag</td>
    <td><span style="color:red">4.49e-3</span></td>
    <td>3.15e-3</td>
    <td>3.33e-2</td>
  </tr>
  <tr>
    <td>ceval</td>
    <td><span style="color:red">3.71e-3</span></td>
    <td>2.67e-2</td>
    <td>3.57e-2</td>
  </tr>
  <tr>
    <td>cmmmlu</td>
    <td><span style="color:red">3.32e-3</span></td>
    <td>2.69e-2</td>
    <td>3.67e-2</td>
  </tr>
</table>

Table 1: MSE of Different loss to predict downstream performance.

4.2 Main Results

Downstream Task Prediction for Open-source Models Figure 4 shows the optimization results for different downstream tasks across various model families. It can be observed that by identifying a shared Capability Salience Vector across
Figure 6: Relationship between capability scores and downstream task performance across different steps of the Capability Salience Vector optimization process. As the optimization progresses, the capability scores of different model series gradually align, leading to improved predictive performance on the test set.

different model families on validation text, we are able to align the capabilities across different models for downstream tasks. Furthermore, the sigmoidal function fitted using the Capability Salience Vectors can effectively predict the downstream task performance of models with larger parameters. This indicates that the downstream task capabilities of models also follow a scaling law, which makes their performance predictable. Table 1 and Figure 7 present the prediction MSE of different loss calculation methods. The capability salience vector significantly improves the predictability of downstream task performance by assigning different importance weights to different token losses. In contrast, the label token loss does not perform well. We believe this is because the loss at the position of the correct answers in the downstream task test set is susceptible to biases in the pretrain data distribution. As a result, this metric fails to align across different model families. We further validated this on checkpoints from our own trained models.

Figure 7: Visualization of prediction results of our method compared to baseline

Downstream Task Prediction for Closed-source Models The results in figure 5 demonstrate that by using the Capability Salience Vector as a proxy, we can predict downstream performance during the model training process, even when models are trained with different data distribution. During training, we were able to maintain a prediction accuracy for downstream task performance with a mean squared error of 1e-4. This demonstrates that our method provides strong guidance for model training and effectively predicts downstream performance.

4.3 Convergence Trend of Capability Salience Vector Optimization

Figure 6 depicts the optimization trend of model capabilities during the process of Capability Salience Vector optimization. The figure shows that the model capabilities progressively converge toward a universal curve shared across different models. This convergence demonstrates that the LM cross-entropy loss reliably reflects model capabilities and that a functional relationship exists between LM cross-entropy loss and downstream task performance. Our algorithm successfully models the downstream scaling law across various model families, improving the predictability of downstream performance based on LM cross-entropy loss.

4.4 Analysis Studies

Impact of Validation Set Data Distribution on Capability Salience Vector Optimization We found that as the diversity of the validation set distribution increases, the range of meta-capability combinations captured by the model from LM cross-entropy loss also expands. This leads to more accurate capability predictions for downstream scaling law. Therefore, when using loss to measure model capabilities, we recommend considering token losses from a more diverse set of data sources, rather than relying solely on the loss from a single data distribution. The result is shown in Figure 8
Case Study   In this section, we visualize the Capability Salience Vectors identified by the automatic optimization algorithm. Figure 9 shows the results of the Capability Salience Vector optimized for the BBH downstream task. The intensity of the green color represents the importance weight assigned by the Capability Salience Vector, with darker shades indicating larger weights. We observe that the optimized Capability Salience Vector highlights the tokens which can measure the model’s reasoning capabilities. The model needs to summarize contextual information and apply reasoning to predict these tokens. These examples demonstrate that our method effectively extracts meta-capabilities that capture the model’s scaling behavior on specific tasks.

![Impact of varying validation set distributions on Capability Salience Vector optimization.](page_186_573_1077_420.png)

Figure 8: Impact of varying validation set distributions on Capability Salience Vector optimization.

Let’s trace the sequence of ball sw aps to determine which ball Eve ends up with at the end of the game : 1 . ** C la ire and Dave swap balls ** - Claire now has the yellow ball . - Dave now has the brown ball . 2 . ** E ve and Alice swap balls ** - Eve now has the blue ball . - Alice now has the black ball . 3 . ** G er tr ude and Eve swap balls ** - Ger tr ude now has the blue ball . - Eve now has the purple ball . 4 . ** C la ire and Alice swap balls ** - Claire now has the black ball . - Alice now has the yellow ball . 5 . ** Dave and Eve swap balls ** - Dave now has the purple ball . - Eve now has the brown ball . 6 . ** Alice and Ger tr ude swap balls ** - Alice now has the blue ball . - Ger tr ude now has the yellow ball . 7 . ** F red and Bob swap balls ** - Fred now has the pink ball . - Bob now has the red ball . After following all the sw aps , Eve ends up with the brown ball . Therefore , the correct answer is : (C ) brown ball .

Figure 9: Visualization of the importance weights assigned to different tokens by the Capability Salience Vector after optimization on the BBH task. Tokens with higher importance weights are highlighted in green. This shows that the Capability Salience Vector effectively identifies tokens that require reasoning by the model, capturing the model’s reasoning capabilities.

loss and downstream task performance is a meaningful and valuable area for future research.

6 Limitation

In this section, we will discuss some limitations of our method. First, while our approach can optimize any given text to identify its Capability Salience Vector, the effectiveness varies across different texts. Therefore, selecting suitable optimization texts remains an open area for future research. Second, since the optimization needs to be performed simultaneously on all texts, the computation cost increases with the length of the text. Thus, improving the efficiency of the optimization algorithm or exploring simpler modeling approaches is a topic for future investigation. Finally, our experiments are limited to six popular objective evaluation benchmarks. Exploration of more challenging tasks or subjective evaluations is left for future work.

Acknowledgements

The authors wish to thank the AC and anonymous reviewers for their constructive comments. This work was supported by the China Postdoctoral Science Foundation under Grant Number 2023M741851.

References

Sanjeev Arora and Anirudh Goyal. 2023. A theory for emergence of complex skills in language models.
arXiv preprint arXiv:2307.15936.

Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. 2024. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, 121(27):e2311878121.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Owen technical report. arXiv preprint arXiv:2309.16609.

Frank B Baker. 2001. The basics of item response theory. ERIC.

Zheng Cai, Maosong Cao, Haojing Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. 2024. Internlm2 technical report. arXiv preprint arXiv:2403.17297.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.

LMDeploy Contributors. 2023a. Lmdeploy: A toolkit for compressing, deploying, and serving llm. https://github.com/InternLM/lmdeploy.

OpenCompass Contributors. 2023b. Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/opencompass.

Zhengxiao Du, Aohan Zeng, Yuxiao Dong, and Jie Tang. 2024. Understanding emergent abilities of language models from the loss perspective. arXiv preprint arXiv:2403.15796.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, et al. 2024. Language models scale reliably with over-training and on downstream tasks. arXiv preprint arXiv:2403.08540.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556.

Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, et al. 2024. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. Advances in Neural Information Processing Systems, 36.

Berivan Isik, Natalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii, and Sanmi Koyejo. 2024. Scaling laws for downstream task performance of large language models. arXiv preprint arXiv:2402.04177.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.

Alex Kipnis, Konstantinos Voudouris, Luca M Schulze Buschoff, and Eric Schulz. 2024. metabench – a sparse benchmark to measure general ability in large language models. arXiv preprint arXiv:2407.12844.

Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2023. Cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint arXiv:2306.09212.

Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110.

Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Ye-long Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, et al. 2024. Rho-1: Not all tokens are what you need. arXiv preprint arXiv:2404.07965.

Hong Liu, Sang Michael Xie, Zhiyuan Li, and Tengyu Ma. 2023. Same pre-training loss, better downstream: Implicit bias matters for language models. In International Conference on Machine Learning, pages 22188–22214. PMLR.

Frederic M Lord and Melvin R Novick. 2008. Statistical theories of mental test scores. IAP.

Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scão, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. 2024. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36.

David Owen. 2024. How predictable is language model benchmark performance? arXiv preprint arXiv:2401.04757.

Lorenzo Pacchiardi, Lucy G Cheke, and José Hernández-Orallo. 2024. 100 instances is all you need: predicting the success of a new llm on unseen data by testing on a few instances. arXiv preprint arXiv:2409.03563.
Yotam Perlitz, Elron Bandel, Ariel Gera, Ofir Ariviv, Liat Ein-Dor, Eyal Shnarch, Noam Slonim, Michal Shmueli-Scheuer, and Leshem Choshen. 2023. Efficient benchmarking (of language models). arXiv preprint arXiv:2308.11696.

Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. 2024. tinybenchmarks: evaluating llms with fewer examples. arXiv preprint arXiv:2402.14992.

Yangjun Ruan, Chris J Maddison, and Tatsunori Hashimoto. 2024. Observational scaling laws and the predictability of language model performance. arXiv preprint arXiv:2405.10938.

Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615.

Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. 2022. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261.

Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Husseidot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajijwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Mengzhou Xia, Mikel Artetxe, Chunting Zhou, Xi Victoria Lin, Ramakanth Pasunuru, Danqi Chen, Luke Zettlemoyer, and Ves Stoyanov. 2022. Training trajectories of language models across scales. arXiv preprint arXiv:2212.09803.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671.

Qinyuan Ye, Harvey Yiyun Fu, Xiang Ren, and Robin Jia. 2023. How predictable are large language model capabilities? a case study on big-bench. arXiv preprint arXiv:2305.14947.

Alex Young, Bei Chen, Chao Li, Chenge Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. 2024. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830.

Qiyuan Zhang, Fuyuan Lyu, Xue Liu, and Chen Ma. 2024. Collaborative performance prediction for large language models. arXiv preprint arXiv:2407.01300.
A Appendix

A Open-source Model Evaluation Setting
In this section, we present the specific model splits used for Capability Salience Vector optimization in the evaluation of open-source models. Larger models within the same series were used for validation and testing, as detailed in Table 2. This split ensures that our method is evaluated across models of varying sizes while maintaining consistency within each series for training and validation.

B Algorithm Workflow
Based on the derivation in Section 3.3, we summarise the algorithmic flow of Capability Salience Vector modeling in Algorithm 1 and 2.

C Open-source Model Evaluation Results
In this section, we present the evaluation results in Table 3 for various tasks on open-source models using OpenCompass.
Table 2: Model split in open-source model evaluation

<table>
  <tr>
    <th>train</th>
    <th>validation</th>
    <th>test</th>
  </tr>
  <tr>
    <td>Llama-2-{7, 13}B</td>
    <td>Llama-2-70B</td>
    <td>Yi-1.5-34B</td>
  </tr>
  <tr>
    <td>Llama-2-{7, 13}B-chat</td>
    <td>Llama-2-70B-chat</td>
    <td>Llama-3-70B</td>
  </tr>
  <tr>
    <td>Qwen1.5{1.8, 4, 7, 14}B</td>
    <td>Qwen1.5-{32, 72}B</td>
    <td>Llama-3-70B-Instruct</td>
  </tr>
  <tr>
    <td>Qwen1.5{1.8, 4, 7, 14}B-chat</td>
    <td>Qwen1.5-{32, 72}B-Chat</td>
    <td>Yi-1.5-34B-Chat</td>
  </tr>
  <tr>
    <td>Llama3-8B</td>
    <td>internlm2-chat-20B</td>
    <td>gemma-2-27B</td>
  </tr>
  <tr>
    <td>Llama3-8B-Instruct</td>
    <td>internlm2-20B</td>
    <td>gemma-2-27B-it</td>
  </tr>
  <tr>
    <td>Qwen2{1.5, 7}B</td>
    <td>internlm2-chat-20B-sft</td>
    <td>Qwen2-72B</td>
  </tr>
  <tr>
    <td>Qwen2{1.5, 7}B-Instruct</td>
    <td>Yi-{34}B</td>
    <td>Qwen2-72B-Instruct</td>
  </tr>
  <tr>
    <td>Yi-{6}B</td>
    <td>Yi-{34}B</td>
    <td>internlm2_5-20B</td>
  </tr>
  <tr>
    <td>Yi-{6}B-chat</td>
    <td>Yi-{34}B-chat</td>
    <td>internlm2_5-20B-chat</td>
  </tr>
  <tr>
    <td>Yi1.5-{6, 9}B</td>
    <td></td>
    <td>Yi-1.5-34B-Chat</td>
  </tr>
  <tr>
    <td>Yi1.5-{6, 9}B-chat</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Gemma2-2-{2, 9}B</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>internlm2-{1.8, 7}B</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>internlm2-{1.8, 7}B-chat</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>internlm2-{1.8, 7}B-chat-sft</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>internlm2.5-{1.8, 7}B</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>internlm2.5-{1.8, 7}B-chat</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>internlm2.5-{1.8, 7}B-chat-sft</td>
    <td></td>
    <td></td>
  </tr>
</table>

Algorithm 1: Loss Mapping Between Different Tokenizers

Require: Input text \( C = (c_1, c_2, \ldots, c_n) \)
Require: Source tokenizer \( T_s \) and target tokenizer \( T_t \)
Require: Source token losses \( L_s = (\ell_1^s, \ell_2^s, \ldots, \ell_m^s) \)

1: Tokenize input with source tokenizer: \( T_s(C) = (\tau_1^s, \ldots, \tau_m^s) \)
2: Initialize character-level losses \( \ell_j^c = 0 \) for each character \( c_j \in C \)
3: for each token \( \tau_i^s \) in \( T_s(C) \) do
4:   Let \( S_i \) be the indices of characters spanned by \( \tau_i^s \)
5:   for each \( j \in S_i \) do
6:     \( \ell_j^c = \frac{\ell_j^s}{|S_i|} \)
7:   end for
8: end for
9: Tokenize input with target tokenizer: \( T_t(C) = (\tau_1^t, \ldots, \tau_k^t) \)
10: Initialize target token losses \( L_t = (\ell_1^t, \ldots, \ell_k^t) \)
11: for each token \( \tau_j^t \) in \( T_t(C) \) do
12:   Let \( T_j \) be the indices of characters spanned by \( \tau_j^t \)
13:   \( \ell_j^t = \sum_{i \in T_j} \ell_i^c \)
14: end for
15: return Target token losses \( L_t \)
Algorithm 2: Optimization of Capability Salience Vector and Downstream Scaling Law

Require: Validation set \( S = \{ X_s \}_{s=1}^{[S]} \)
Require: Language model with score head \( f_\theta \)
Require: Model Set \( M = \{ m \} \)
Require: Observed downstream performance of different models \( \hat{A}_{t,m} \)

1: Collect token losses on the validation set of different model \( \{ L_m \} \) and mapping the losses from source tokenizer to target tokenizer Using algorithm 1
2: for each epoch do
3:     Step 1: Extract Weights of Capability Salience Vector
4:     for each sample \( X_s = (x_1, x_2, ..., x_{|X_s|}) \in S \) do
5:         for each token \( x_{s,i} \) in \( X_s \) do
6:             Compute \( w_{s,i} = f_\theta(x_{s,i} \mid x_{s,<i}) \) using the language model scoring head.
7:         end for
8:     end for
9:     Obtain initial Capability Salience Vector \( W = \{ w_{s,i} \} \).
10:    for each model \( m \) in Model Set \( M \) do
11:       Step 2: Fit Downstream Scaling Law Function
12:      Using equation 1 and 2 to predict downstream task performance \( A_{t,m} \).
13:      Fix \( \theta \) and estimate \( \alpha, \beta \) by minimizing the MSE loss:
\[
\min_{\alpha, \beta} \sum_m [A_{t,m} - \hat{A}_{t,m}]^2
\]
14:     Use the Levenberg-Marquardt algorithm to solve for \( \alpha, \beta \).
15:    Step 3: Optimize Capability Salience Vector
16:    Using equation 1 and 2 to predict downstream task performance \( A_{t,m} \).
17:    Update \( \theta \) by minimizing:
\[
\min_{\theta} \sum_m [A_{t,m} - \hat{A}_{t,m}]^2
\]
18:     Use Stochastic Gradient Descent (SGD) to optimize \( \theta \).
19: end for
20: Repeat Steps 1-3 until convergence.
21: end for
22: return Optimized Capability Salience Vector \( W \) and Downstream Scaling Law parameters \( \alpha, \beta \).
Table 3: Downstream Tasks Evaluation Results(1)

<table>
  <tr>
    <th>model</th>
    <th>bbh</th>
    <th>ceval-test</th>
    <th>cmmlu</th>
    <th>gsm8k</th>
    <th>hellaswag</th>
    <th>mmlu</th>
  </tr>
  <tr>
    <td>Llama-2-7b-hf</td>
    <td>38.27</td>
    <td>30.13</td>
    <td>32.75</td>
    <td>16.76</td>
    <td>29.29</td>
    <td>46.78</td>
  </tr>
  <tr>
    <td>Llama-2-13b-hf</td>
    <td>45.68</td>
    <td>37.38</td>
    <td>38.81</td>
    <td>29.87</td>
    <td>45.06</td>
    <td>55.76</td>
  </tr>
  <tr>
    <td>llama-2-70b-hf</td>
    <td>64.78</td>
    <td>49.53</td>
    <td>53.05</td>
    <td>63.53</td>
    <td>55.91</td>
    <td>69.87</td>
  </tr>
  <tr>
    <td>Qwen1.5-1.8B</td>
    <td>27.01</td>
    <td>58.67</td>
    <td>57.45</td>
    <td>34.87</td>
    <td>42.32</td>
    <td>47.14</td>
  </tr>
  <tr>
    <td>Qwen1.5-4B</td>
    <td>34.81</td>
    <td>66.55</td>
    <td>66.38</td>
    <td>47.61</td>
    <td>55.89</td>
    <td>57.03</td>
  </tr>
  <tr>
    <td>Qwen1.5-7B</td>
    <td>39.87</td>
    <td>72.49</td>
    <td>71.86</td>
    <td>54.36</td>
    <td>68.51</td>
    <td>62.15</td>
  </tr>
  <tr>
    <td>Qwen1.5-14B</td>
    <td>50.38</td>
    <td>76.93</td>
    <td>76.95</td>
    <td>63.53</td>
    <td>83.86</td>
    <td>69.1</td>
  </tr>
  <tr>
    <td>Qwen1.5-32B</td>
    <td>67.47</td>
    <td>82.5</td>
    <td>81.58</td>
    <td>72.71</td>
    <td>87.28</td>
    <td>73.88</td>
  </tr>
  <tr>
    <td>Qwen1.5-72B</td>
    <td>58.81</td>
    <td>83.03</td>
    <td>83</td>
    <td>79.53</td>
    <td>90.41</td>
    <td>77.02</td>
  </tr>
  <tr>
    <td>Qwen2-7B</td>
    <td>54.28</td>
    <td>82.22</td>
    <td>83.67</td>
    <td>73.77</td>
    <td>73.74</td>
    <td>70.27</td>
  </tr>
  <tr>
    <td>Qwen2-72B</td>
    <td>74.52</td>
    <td>89</td>
    <td>90.82</td>
    <td>89.54</td>
    <td>93.46</td>
    <td>83.73</td>
  </tr>
  <tr>
    <td>Yi-6B</td>
    <td>44.82</td>
    <td>70.78</td>
    <td>74.01</td>
    <td>39.58</td>
    <td>66.83</td>
    <td>64.08</td>
  </tr>
  <tr>
    <td>Yi-34B</td>
    <td>66.37</td>
    <td>80.93</td>
    <td>82.79</td>
    <td>67.7</td>
    <td>83.83</td>
    <td>76.26</td>
  </tr>
  <tr>
    <td>Yi-1.5-6B</td>
    <td>57.55</td>
    <td>66.93</td>
    <td>69.68</td>
    <td>61.33</td>
    <td>70.79</td>
    <td>64.88</td>
  </tr>
  <tr>
    <td>Yi-1.5-9B</td>
    <td>71.09</td>
    <td>72.71</td>
    <td>74.11</td>
    <td>74.53</td>
    <td>76.64</td>
    <td>71.52</td>
  </tr>
  <tr>
    <td>Yi-1.5-34B</td>
    <td>75.06</td>
    <td>82.24</td>
    <td>83.55</td>
    <td>79.98</td>
    <td>84.92</td>
    <td>77.7</td>
  </tr>
  <tr>
    <td>Qwen1.5-1.8B-Chat</td>
    <td>27.03</td>
    <td>55.19</td>
    <td>48.3</td>
    <td>29.57</td>
    <td>42.32</td>
    <td>45.39</td>
  </tr>
  <tr>
    <td>Qwen1.5-4B-Chat</td>
    <td>43.19</td>
    <td>61.37</td>
    <td>58.22</td>
    <td>46.02</td>
    <td>60.74</td>
    <td>56.01</td>
  </tr>
  <tr>
    <td>Qwen1.5-7B-Chat</td>
    <td>35.19</td>
    <td>68.18</td>
    <td>67.98</td>
    <td>55.88</td>
    <td>69.8</td>
    <td>61.78</td>
  </tr>
  <tr>
    <td>Qwen1.5-14B-Chat</td>
    <td>55.58</td>
    <td>74.67</td>
    <td>75.29</td>
    <td>64.82</td>
    <td>80.03</td>
    <td>68.08</td>
  </tr>
  <tr>
    <td>Qwen1.5-32B-Chat</td>
    <td>68.55</td>
    <td>80.66</td>
    <td>80.17</td>
    <td>79.15</td>
    <td>88.42</td>
    <td>75.12</td>
  </tr>
  <tr>
    <td>Qwen1.5-72B-Chat</td>
    <td>71.73</td>
    <td>81.37</td>
    <td>82.15</td>
    <td>79.68</td>
    <td>88.99</td>
    <td>77.1</td>
  </tr>
  <tr>
    <td>Qwen2-7B-Instruct</td>
    <td>64.56</td>
    <td>82.11</td>
    <td>80.84</td>
    <td>87.29</td>
    <td>78.57</td>
    <td>70.68</td>
  </tr>
  <tr>
    <td>Qwen2-72B-Instruct</td>
    <td>82.07</td>
    <td>87.02</td>
    <td>89.88</td>
    <td>92.42</td>
    <td>93.7</td>
    <td>82.73</td>
  </tr>
  <tr>
    <td>Yi-6B-Chat</td>
    <td>30.98</td>
    <td>70.69</td>
    <td>72.74</td>
    <td>42.46</td>
    <td>64.27</td>
    <td>63.22</td>
  </tr>
  <tr>
    <td>Yi-34B-Chat</td>
    <td>60.47</td>
    <td>78.58</td>
    <td>80.33</td>
    <td>72.71</td>
    <td>78.69</td>
    <td>73.91</td>
  </tr>
  <tr>
    <td>Yi-1.5-6B-Chat</td>
    <td>58.68</td>
    <td>68.05</td>
    <td>68.59</td>
    <td>75.36</td>
    <td>75.66</td>
    <td>64.76</td>
  </tr>
  <tr>
    <td>Yi-1.5-9B-Chat</td>
    <td>69.61</td>
    <td>73.19</td>
    <td>74.51</td>
    <td>79.23</td>
    <td>83.14</td>
    <td>71.35</td>
  </tr>
  <tr>
    <td>Yi-1.5-34B-Chat</td>
    <td>73.63</td>
    <td>80.83</td>
    <td>81.05</td>
    <td>85.82</td>
    <td>87.33</td>
    <td>75.95</td>
  </tr>
  <tr>
    <td>internlm2-1_8b</td>
    <td>36.03</td>
    <td>44.79</td>
    <td>45.27</td>
    <td>30.4</td>
    <td>44.86</td>
    <td>45.99</td>
  </tr>
  <tr>
    <td>internlm2-7b</td>
    <td>63.56</td>
    <td>63.54</td>
    <td>66.17</td>
    <td>69.98</td>
    <td>89.52</td>
    <td>65.84</td>
  </tr>
  <tr>
    <td>internlm2-20b</td>
    <td>71.29</td>
    <td>67.28</td>
    <td>68.28</td>
    <td>76.8</td>
    <td>91.41</td>
    <td>67.58</td>
  </tr>
  <tr>
    <td>internlm2-chat-1_8b</td>
    <td>37.31</td>
    <td>47.06</td>
    <td>46.19</td>
    <td>39.2</td>
    <td>59.71</td>
    <td>47.72</td>
  </tr>
  <tr>
    <td>internlm2-chat-1_8b-sft</td>
    <td>38.23</td>
    <td>47.21</td>
    <td>46.17</td>
    <td>37.83</td>
    <td>60.86</td>
    <td>47.7</td>
  </tr>
  <tr>
    <td>internlm2-chat-7b</td>
    <td>59.96</td>
    <td>58.85</td>
    <td>62.66</td>
    <td>72.1</td>
    <td>84.44</td>
    <td>63.51</td>
  </tr>
  <tr>
    <td>internlm2-chat-7b-sft</td>
    <td>59.35</td>
    <td>58.99</td>
    <td>62.57</td>
    <td>70.51</td>
    <td>85.01</td>
    <td>63.51</td>
  </tr>
  <tr>
    <td>internlm2-chat-20b</td>
    <td>69.53</td>
    <td>63.03</td>
    <td>65.85</td>
    <td>76.19</td>
    <td>88.16</td>
    <td>67.36</td>
  </tr>
  <tr>
    <td>internlm2-chat-20b-sft</td>
    <td>69.31</td>
    <td>63.2</td>
    <td>66.04</td>
    <td>77.03</td>
    <td>88.68</td>
    <td>67.28</td>
  </tr>
  <tr>
    <td>internlm2_5-1_8b-chat</td>
    <td>42.58</td>
    <td>61.25</td>
    <td>62.3</td>
    <td>53.3</td>
    <td>49.97</td>
    <td>50.98</td>
  </tr>
  <tr>
    <td>internlm2_5-7b</td>
    <td>72.64</td>
    <td>77.21</td>
    <td>78.95</td>
    <td>74.83</td>
    <td>73.49</td>
    <td>71.5</td>
  </tr>
  <tr>
    <td>internlm2_5-7b-chat</td>
    <td>74.8</td>
    <td>77.38</td>
    <td>78.04</td>
    <td>84</td>
    <td>67.74</td>
    <td>71.82</td>
  </tr>
  <tr>
    <td>internlm2_5-20b-chat</td>
    <td>79.77</td>
    <td>79.75</td>
    <td>79.18</td>
    <td>88.1</td>
    <td>66.53</td>
    <td>72.49</td>
  </tr>
  <tr>
    <td>internlm2_5-1_8b</td>
    <td>42.65</td>
    <td>63.97</td>
    <td>65.5</td>
    <td>38.82</td>
    <td>56.64</td>
    <td>53.44</td>
  </tr>
  <tr>
    <td>internlm2_5-20b</td>
    <td>79.83</td>
    <td>81.94</td>
    <td>82.25</td>
    <td>82.64</td>
    <td>79.35</td>
    <td>74.2</td>
  </tr>
</table>
Table 4: Downstream Tasks Evaluation Results(2)

<table>
  <tr>
    <th>model</th>
    <th>bbh</th>
    <th>ceval-test</th>
    <th>cmmlu</th>
    <th>gsm8k</th>
    <th>hellaswag</th>
    <th>mmlu</th>
  </tr>
  <tr>
    <td>Meta-Llama-3-8B</td>
    <td>59.69</td>
    <td>48.83</td>
    <td>50.95</td>
    <td>54.28</td>
    <td>50.86</td>
    <td>66.43</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-70B</td>
    <td>79.16</td>
    <td>66.56</td>
    <td>68.36</td>
    <td>69.98</td>
    <td>80.6</td>
    <td>79.35</td>
  </tr>
  <tr>
    <td>Llama-2-7b-chat-hf</td>
    <td>41.51</td>
    <td>30.2</td>
    <td>32.62</td>
    <td>28.13</td>
    <td>48.94</td>
    <td>44.56</td>
  </tr>
  <tr>
    <td>Llama-2-13b-chat-hf</td>
    <td>49.61</td>
    <td>33.76</td>
    <td>37.01</td>
    <td>42.08</td>
    <td>61.43</td>
    <td>52.1</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-8B-Instruct</td>
    <td>52.5</td>
    <td>49.93</td>
    <td>51.89</td>
    <td>79.3</td>
    <td>73.29</td>
    <td>67.16</td>
  </tr>
  <tr>
    <td>Meta-Llama-3-70B-Instruct</td>
    <td>80.45</td>
    <td>66.91</td>
    <td>70.11</td>
    <td>90.22</td>
    <td>87.72</td>
    <td>80.52</td>
  </tr>
  <tr>
    <td>gemma-2-2b-it</td>
    <td>46.44</td>
    <td>41.82</td>
    <td>44.86</td>
    <td>55.65</td>
    <td>62.43</td>
    <td>58.39</td>
  </tr>
  <tr>
    <td>gemma-2-9b-it</td>
    <td>75.07</td>
    <td>57.32</td>
    <td>59.49</td>
    <td>86.66</td>
    <td>70.29</td>
    <td>73.14</td>
  </tr>
  <tr>
    <td>gemma-2-27b-it</td>
    <td>80.05</td>
    <td>61.71</td>
    <td>62.5</td>
    <td>89.99</td>
    <td>75.96</td>
    <td>76.45</td>
  </tr>
  <tr>
    <td>gemma-2-27b</td>
    <td>74.77</td>
    <td>60.84</td>
    <td>61.53</td>
    <td>80.97</td>
    <td>75.55</td>
    <td>75.97</td>
  </tr>
  <tr>
    <td>gemma-2-2b</td>
    <td>42.12</td>
    <td>39.44</td>
    <td>39.31</td>
    <td>33.51</td>
    <td>65.91</td>
    <td>54.35</td>
  </tr>
  <tr>
    <td>gemma-2-9b</td>
    <td>70.63</td>
    <td>57.63</td>
    <td>59.02</td>
    <td>72.86</td>
    <td>72.93</td>
    <td>72.44</td>
  </tr>
  <tr>
    <td>Llama-2-70b-chat-hf</td>
    <td>59.94</td>
    <td>32.21</td>
    <td>43.07</td>
    <td>60.65</td>
    <td>74.18</td>
    <td>59.93</td>
  </tr>
  <tr>
    <td>Qwen2-1.5B-Instruct</td>
    <td>38.31</td>
    <td>68.27</td>
    <td>68.12</td>
    <td>63.53</td>
    <td>55.38</td>
    <td>55.73</td>
  </tr>
  <tr>
    <td>Qwen2-1.5B</td>
    <td>35.75</td>
    <td>69.32</td>
    <td>70.58</td>
    <td>59.06</td>
    <td>46.76</td>
    <td>57.45</td>
  </tr>
</table>

Reviewer paper summaries: This paper introduces the Capability Salience Vector (CSV), a novel method designed to bridge the gap between validation loss and downstream task performance in language models. The authors argue that traditional validation loss, which treats all tokens equally, fails to capture the nuanced capabilities required for specific downstream tasks. The CSV assigns different importance weights to tokens based on their contribution to task-specific capabilities, allowing for a more accurate prediction of model performance on downstream tasks. The paper demonstrates the effectiveness of CSV through experiments on six popular benchmarks, showing significant improvements in predictability.
Summary of strengths: 1. Novel Methodology: The introduction of the Capability Salience Vector is a novel approach. It addresses the issue of the gap between validation loss and downstream task performance by assigning differential weights to tokens, which is a new way of thinking about loss calculation in the context of performance prediction.
2. Comprehensive Experiments: The experiments are comprehensive, covering a wide range of benchmarks and different model series. Using both open-source and closed-source models for evaluation provides a more complete picture of the method's effectiveness.
3. Clear Literature Review: The paper has a well-organized literature review. It clearly presents the existing research on downstream scaling laws and efficient benchmark prediction, highlighting the limitations of previous studies and positioning its own work in the context of the broader research area.
Summary of weaknesses: 1. Comparison with State-of-the-Art Methods. The paper could benefit from a more comprehensive comparison with other state-of-the-art methods for predicting downstream task performance, such as those based on item response theory or other meta-learning approaches.
2. Generalization across different datasets. If the parameters (like alpha and beta) need to be retrained for each dataset, I have some doubts about the generalization performance of this method. If one set of the parameters can be generalized to different datasets, it should be more valuable in practical engineering applications.
Comments, suggestions and typos: 1. Typos. There should be no space between "function" and "." in L367, and a missing "." in L368.

If there is anything I have understood incorrectly, please feel free to offer your valuable opinion. I will respond very fast and I am more than willing to increase the initial score.
Overall assessment scores: 3
Metareview score: 4.5
Metareview: This paper proposes an improved way to construct scaling predictions for downstream tasks. The key insight is that one should not merely look at overall pretraining perplexity, as different downstream tasks depend on different skills. Instead, the paper learns a linear projection from the hidden state of a (separate) LM that assigns a weight to each token. These weights are trained for each task to match the scaling trend on a "training set" of smaller models. The fit is then evaluated on a "test set" of larger models. The paper shows good predictive capability across a variety of tasks and model families.

The reviewers were all somewhat positive about this paper. One main weakness raised is that this method introduces additional overhead compared to a "traditional" scaling law. But this seems to be a necessary trade-off to get better task-specific fits, which is a valuable goal. So I do not find any significant weaknesses in the paper and believe the contribution to be substantial.
Summary of reasons to publish: * New way of creating task-specific scaling laws
* Strong results showing extrapolation to larger models
* Method is intuitive