Can a Crow Hatch a Falcon? Lineage Matters in Predicting Large Language Model Performance
Authors: Takuya Tamura
Taro Yano
Masafumi Enomoto
Masafumi Oyamada
Venue: COLM 2025 (2025)
Keywords: LLM
Performance Estimation
Matrix Factorization
Neural Collaborative Filtering
TL;DR: Explicitly modeling LLM **lineage** boosts performance prediction accuracy. Our lineage-regularized matrix factorization leverages ancestry to outperform standard methods when predicting new or merged models with minimal evaluation data.

Abstract: Accurately forecasting the performance of Large Language Models (LLMs) before extensive fine-tuning or merging can substantially reduce both computational expense and development time. Although prior approaches like scaling laws account for global factors such as parameter size or training tokens, they often overlook explicit lineage relationships—i.e., which models are derived or merged from which parents. In this work, we propose a novel Lineage-Regularized Matrix Factorization (LRMF) framework that encodes ancestral ties among LLMs via a graph Laplacian regularizer. By leveraging multi-hop parent--child connections, LRMF consistently outperforms conventional matrix factorization and collaborative filtering methods in both instance-level and benchmark-level performance prediction. Our large-scale study includes 2,934 publicly available Hugging Face models and 21,000+ instances across 6 major benchmarks, 
showing that the introduction of lineage constraints yields up to 0.15–0.30 higher correlation coefficients with actual performance compared to baseline methods.
Moreover, LRMF effectively addresses the cold-start problem, providing accurate estimates for newly derived or merged models even with minimal data. This lineage-guided strategy thus offers a resource-efficient way to inform hyperparameter tuning, data selection, and model combination in modern LLM development.

Published as a conference paper at COLM 2025
Can a Crow Hatch a Falcon? Lineage Matters in Predicting Large Language Model Perfor- mance
Takuya Tamura, Taro Yano, Masafumi Enomoto & Masafumi Oyamada NEC Corporation
(tamura-takyya, taro yano, masafumi-enomoto, oyamada (enaddd)@eee.coom com
Abstract
Accurately forecasting the performance of Large Language Models (LLMs) before extensive fine-tuning or merging can substantially reduce both com- putational expense and development time. Although prior approaches like scaling laws account for global factors such as parameter size or training tokens, they often overlook explicit lineage relationships-ice which mod- els are derived or merged from which parents. In this work, we propose a Matrix Factorization (LRMF) framework that encodes ancestral ties among LLMs via graph Laplacian regularizer. By leveraging multi-hop parent-chile connections, LRMF consistently outper- forms conventional matrix factorization and collaborative filtering methods both instance-level and benchmark-leve performance prediction. Our large-scale study includes 2,934 publicly available Hugging Face mod- els and 21,000+ instances across 6 major benchmarks, showing that the introduction of lineage constraints yields up to 0.15-0.30 higher Pearson correlation coefficients with actual performance compared to baseline meth- ods. Moreover, LRMF effectively addresses the cold-start problem, providing accurate estimates for newly derived or merged models even with minimal data. This Eheegge-guded strategy thus offers resource-efficient way to inform hyperparameter tuning, data selection, and model combination in modern LLM development.
novel
1 Introduction
Building and refining Large Language Model (LLM) is an increasingly expensive venture. Developers face critical decisions such as model size, the amount and type of training data, and how to allocate parameters for fine-tuning. Moreover, model merging (or model "souping") is emerging as a promising strategy: models models or checkpoints are combined at the parameter level to create variants with complementary strengths (Wortsman et al., 2022). Each of these steps-training fine-tuning, and merging -oaegnnggtteenn often demands large-scale exploration and incurs high computational cost.
If one could accurately predict the performance of a proposed LLM variant-even before running large-scale optimization-it would profoundly reduce the search space. Such foresight would streamline decisions about (1) the right model size and data volume for pre-training, (2) fine-tuning hyperparameters and data composition (e.g., domain mix), and (3) whether combining certain model checkpoints is worthwhile. Indeed, prior work has sought to provide partial solutions. Classic scaling laws (Kaplan al., 2020; Hoffmann et al., 2022) predict an LLM's loss on the training corpus as function of parameter size and token count. These laws help developers extrapolate the compute needed to reach desired perplexity training loss. Subsequent studies extend these ideas to specific bench- marks, e.g., MMLU (Hendrycks et al., 2021a; Wang et al., 2024), BBH (Suzgun et al., 2023), or MATH (Hendrycks et al., 2021b), by fitting regression models or matrix factorization frameworks to empirical performance data (Zhang et al., 2024; Polo et al., 2024; Kipnis et al., 2025).
1

---

Published as a conference paper at COLM 2025
However, real-world LLMs rarely arise in isolation. In practice, single base model often spawns multiple variants via continual pre-training, domain-specific fine-tuning, preference tuning (e.g., RLHF), parameter-level merging of multiple checkpoints. Such derivation or lineage relationships typically imply that "child" models inherit certain performance characteristics from their parents. Despite this, previous perffrmance-prediction work has largely ignored the explicit structure of these relationships, focusing instead on coarse factors such as model size training tokens. As a result, the potential advantage of modeling multi-hop ancestry-iee, "who was the parent's parent?"-remann underexplored.
In this paper, we we propose a novel way to incorporate such lineage information into perfor- mance prediction. Specifically, we introduce a (LRMF) framework that treats the known derivation paths among models as a lineage graph. Within a standard matrix factorization procedure (where models form the rows and benchmarks/ tasks form the columns), we add a graph Laplacian regularization term that explicitly constrains models with direct or multi-hop parent-child ties to lie "close" in the latent space. The resulting method can more accurately propagate performance signals from well-evaluated parent models to their lesser-known descendants.
Matrix Factorization
We conduct a large-scale empirical study encompassing 2,934 publicly available Hugging Face models and 21,000+ instances across 6 major benchmarks. We demonstrate that incorporating lineage constraints- i.e. explicitly modeling which models were fine-tuned or merged from which parents connnts--nnsstentlly boosts predictive accuracy. In fact, compared to standard matrix factorization baselines, our method improves the Pearson correlation coefficient with true performance by up to 0.15-0.30 on certain tasks, affirming that lineage is a key factor in understanding and forecasting LLM behaviors.
Moreover, by leveraging explicit lineage relationships (parent-child relations or merges), our method LRMF tackles the cold-start problem, enabling accurate performance estimates for newly derived or merged models without costly full re-benchmarking. Indeed, our experiments reveal that even with as few as 10 labeled instances per model, we can achieve near state-of-the-ar Pearson correlation with fully evaluated baselines- remarkable im- provement over standard collaborative filtering scaling- based approaches. Taken together, our findings suggest that preserving and exploiting the lineage graph is vital for guiding resource-efficient decisions in LLM development.
2 Related Work
Performance Estimation via Scaling Laws. Early studies on large language models (LLMs) have shown that core metrics like cross-entropy loss exhibit approximate power-law relation- ships with key design factors such as model size, dataset scale, and overall computational budget (Kaplan al., 2020; Hoffmann et al., 2022; Hernandez et al., 2022; Gordon et al., 2021). While these classical "scaling laws" provide valuable insights into how performance grows with parameters and tokens, their estimates typically assume a single model family or narrowly defined training setup. Consequently, scaling relationships learned from one architecture (e.g., Transformer-base language models) may not directly generalize to another.
Recent work expands scaling laws beyond a single family. For instance, Ye et al. (2023) investigate whether past performance records from multiple LLM variants can predict new model settings; Owen (2024) show that aggregated benchmark scores (e.g., BIG-Bench) follow smooth scaling trends across diverse model families, yet individual tasks are more difficult to predict. Ruan et al. (2024) explore observational rather than purely experimental scaling, leveraging over a hundred pretrained L.MSs to fit parametric curves without training each from scratch. These lines of research challenge the assumption that scaling laws transfer seamlessly across all model families. Indeed, Choshen et al. (2025) highlight large predictive errors when applying scaling trends derived from one family to different, structurally distinct family. Overall, while scaling laws are fundamental to LLM research, their reliance on homogeneous
wear-homogeneour settings can limit predictive accuracy, especially when addressing newly merged or fine-tuned models for which classical scaling metrics (e-g., total FLOPs or training tokens) are not strictly comparable.
2

---

Published as a conference paper at COLM 2025
Predicting Downstream Task Performance from Observational Data. A growing body of work aims to bypass some limitations of traditional scaling laws by using observational or empirical records from existing models. Rather than modeling only cross-entropy loss, these methods directly estimate how well a new or partially evaluated LLM will perform on downstream benchmarks. One common strategy is to reduce the high cost of comprehensive benchmarking by sampling only a small subset of instances. For example, Polo et al. (2024), Kipnis et al. (2025), and Pacchiardi et al. (2024) show that evaluating each new LLM on 100-600 carefully chosen samples can often predict that LLM's accuracy on thousands of held-out benchmark items with minimal error. Such "tiny benchmarks" mitigate the expense of model-by podell-y-model, full-scale evaluations.
Beyond sample-efficint evaluation, another theme is to learn joint representations of both models and tasks. Zhang et al. (2024) combine historical LLM results with model and task descriptors (e.g., parameter size, instruction-tuning data) to improve prediction via matrix factorization or neural collaborative filtering. A similar philosophy underlies the works of Ruan et al. (2024) and Choshen et al. (2025), who incorporate both scaling metrics and observed performance to predict complex "emergent" or "agentic" behaviors of new models. Although these observational approaches can generalize better across different LLM architectures, they seldom exploit fine-grained lineage information- as which parent checkpoints were merged or how a model was fine-tuned even even though such lineage frequently constrains or correlates performance across tasks.
Routing, Ensemble Methods, and Embedding-Based Prediction. A complementary direction addresses how to select or roule queries among multiple candidate LLMs that each has known strengths. In kNN-based routing (Shnitzer al., 2024; Hu et al., 2024), for instance, the system uses prompt embeddings or input features to decide which model is most likely to excel on a particular query; Lu et al. (2024) similarly train routing functions imp
based on reward signals. Such approaches implicitly rely on predicting individual models' performance from prior records, so improved performance estimation methods can feed directly into these routing pipelines. Meanwhile, Ong et al. (2025) propose factorization approach (combining matrix factorization with abbel-enhancement techniques) that learns dynamic "router" capable of choosing between smaller larger models at inference time to optimize cost-performance trade-offs. These routing-oriented works share a common challenge: how to reliably predict an LLM's performance on new input. While embedding- based or factor-based methods capture prompt similarity, they again treat each model as a separate entity without leveraging the relationship between parent and child or among merged "soups."
Motivation for Lineage-Aware Estimation. Merging or fine-tuning from existing check- points is pervasive in LLM development (Wortsman et al., 2022). Two "child" models derived from the same "parent" frequently behave similarly across tasks, and naive averag- ing of parent checkpoints can sometimes preserve or even enhance performance in certain domains. Nevertheless, conventional scaling laws or standard matrix factorization rarely encode such genealogical constraints. As result, performance estimates for newly merged or fine-tuned models often rely on expensive direct evaluations or broad assumptions. Our work addresses this gap by building on empirical performance-baser frameworks and injecting explicit lineage regularization, thereby offering a more reliable way to predict the behavior of newly derived models without exhaustive re-benchmarking.
3 Method
Our goal is to predict the performance of Large Language Models (LLMs) on specific tasks or benchmarks by leveraging partial observations and model relationships. In particular, we address the cold-start setting, where we estimate the performance of LLMs that have not yet been constructed or evaluated.
3

---

Published as a conference paper at COLM 2025
3.1 Problem Statement
Let M be the set of LLMs and X' be the set of problems. Define Zu Zu,i E {0, } as the evaluation score for problem i X' solved by LLM u E M. Here, Zu,i if the LLM correctly solves the problem, and 0 otherwise. For some LLM-problem pairs, their true scores are already known, and using these observed data, our goal is accurately estimate the scores for unobserved pairs.
Existing Methods for LLM Performance Prediction
Based Methods. I'erformance-based prediction methods gen- erally rely on historical performance data. These methods assume that if two models or problems exhibit similar observed performance, their unobserved performances will also be similar. Formally, they define general function f(-) that estimates performance using latent embeddings or features of models and instances:
auj f (mu, xi)
where ma R" and x; Rd are embeddings for the model and instance i, respectively. Specific examples include:
First, in the Matrix Factorization (MF) approach, the prediction is computed
u,, = 5 (,,,,))
where my and are latent vectors representing the user and item, respectively, and o is the sigmoid function.
Second, in the Item Response Theory (IRT) model, the prediction is given by
2uj (a,0, b;),
where 0n denotes the ability parameter of user u, and a; and b, are the discrimination and difficulty parameters item i, respectively.
Finally, the Neural Collaborative Filtering (NCF) model estimates the interaction as
Eu,i o (MLP( 0)),
where MLP(-) is multi-layer perceptron parameterized by 9, applied to the concatenation of my and X₃.
To learn these embeddings, methods typically minimize binary cross-entropy (BCE) loss on observed performance scores, combined with an L2 regularization term:
OBCE = [
Σ (ui))n
IZu,i log( + (1 Zu,i) log(1 11 Eu,)) + A12 [
where is the set of observed LLM-instance pairs, M and X are the matrices containing embeddings my and as rows, respectively, and || || denotes the Frobenius norm.
The methods described above face significant limitations when dealing with new LLMs that have no observed performance data the well-known cold-start problem in recommenda- tion systems. This issue is particularly acute in LLM development, where developers must often make decisions about newly created models without having the resources to evaluate them extensively.
Factor-Enhanced Method. Further improvements can be achieved by explicitly incor- porating model and task design factors, such parameter size, dataset scale, and task characteristics, into the prediction (He et al., 2017; Zhang et al., 2024). The enhanced NCF (NCF with factors) includes these additional factors embedding vectors ex ezy
6 MLP (pi 9j. Co Ez
4

---

Published as a conference paper at COLM 2025
Specifically, Zhang et al. (2024) considered model-related factors including Model Family (e.g,, LLaMA 2, Pythia), Pretraining Dataset Size (tokens), Parameter Size, GPU Time, FLOPs, Context Window Size, Batch Size, Number of Layers, Number of Attention Heads, Key/Value Size, Bottleneck Activation Size, and Carbon Emission. They also included task-related factors such Ability (e-g,, reasoning), Task Family (e.g., ARC), Output Format (e.g., binary classification), and Few-Shot Setting.
Shapley value analysis indicated that among model factors, Prefraining Dataset Size, Model Family, and Batch Size were most critical, while for task factors, Ability was most influential. These results underscore that factors beyond traditional scaling laws significantly influence prediction accuracy.
Notably, in a homogeneous model set derived from the same model family, factors such as Parameter Size, Context Window Size, and Batch Size often remain identical or highly similar across models.
3.3 Proposed: Lineage-Based Prediction
We address the cold-start problem by leveraging an underutilized source of information: the lineage relationships between LLMs. When new model is derived through fine-tuning or merging existing models, its behavior is likely to be inherited from its "ancestors" or "parents." This intuition forms the foundation of our approach. While factor-based methods (e.g. NCF with factors) attempt to transfer information indirectly by identifying models with similar characteristics (e.g,, similar parameter sizes, training data, or architectural features), our lineage-based approach enables direct information transfer along known derivation paths. This distinction is crucial: rather than inferring similarities from potentially noisy or incomplete metadata, we utilize the explicit knowledge of how models are derived from one another, capturing relationships that may not be apparent from factors alone. To leverage lineage information effectively, Model Lineage Averaging, directly utilizes lineage relationships in their simplest form to make predictions. The second approach, Eineag-Reggllaiize Matrix Factorization (LRMF), integrates lineage constraints into the collaborative filtering framework, allowing for more sophisticated modeling of performance transfer patterns between related models.
propose two distinct approaches. The first approach,
Fine-tuned models and merged models inherently inherit the performance characteristics of their base models. We can estimate performance on an instance i for a new or sparsely evaluated LLM u from the results of its lineage-related (neighbor) LLMs:
Proposed-1:
Model Lineage Averaging
Σ MENM(()
[M((U)
Zuf,i-
Here, the neighborhood (u) is defined based lineage relationships. Specifically, an LLM u is derived from v through a short lineage path (fine-tuning, merging, etc.), treat a a neighbor of u. This simple yet effective approach directly addresses the cold-start problem by transferring performance knowledge from established models to their derivatives.
Proposed-2: Regula-Ree Matrix Factorization (LRMF). While derived mod- els fundamentally inherit capabilities from their base models, subtle differences emerge in practice during fine-tuning or merging processes. We account for these nuances by drawing inspiration from Collaborative Filtering approaches. Following the framework of Tag Informed Collaborative Filtering (TagiCoFi) (Zhen et al., 2009), which integrates additional information (e.g,, user-generated tags) into the matrix factorization procedure to Matrix Factor-
improve recommendation performance, we propose Baaneage-Regulartte ization (LRMF). LRMF extends traditional matrix factorization by explicitly encouraging similarity between embeddings of neighboring LLMs and neighboring tasks.
Specifically, let A (M) and A (X)) be adjacency matrices indicating lineage connections among LLMs and similarity relationships among tasks, respectively. For models, A (M) directly
5

---

Published as a conference paper at COLM 2025
encodes the lineage relationships: edge exists between models if one is derived from the other through fine-tuning or merging. For example, if model v is fine-tuned from
(M) (M)
model u, or if model created by merging models u and v, then Aip Aöj
and
(M)
A
We introduce regularization terms to ensure neighboring embeddings remain close. In particular, for models, the regularization term that enforces similarity between feature vectors of lineage-related models is given by:
O (.))
Σ
A
(M) u,
my
mall?.
2
This term penalizes large differences between the embeddings of models that share a lineage connection. Expanding and simplifying this expression yields:
O (M)
m
ma D
(M) M,
Em m. A (M) u,
Tr(M L MMM
M2
where the diagonal degree matrix ((M) is defined by D (M Laplacian is L ((M) (M) A (M),
Duu
E.
Amp
(M) Aue
and the graph
Similarly, for tasks, we construct A (x) using cosine similarity between task embeddings, retaining the top k similar tasks, and define a corresponding regularization term O Tr(XT x)
The overall objective function then formulated as:
O OBCE + Ax 0 + AM AMOMM.
Here, the regularization parameters Åx and AM control the influence of instance similarity and model lineage constraints, respectively. A detailed analysis of these hyperparameters and their effects on model performance is provided in Appendix B. To optimize this objective function, we employ a stochastic gradient descent method with adaptive learning rates (Adam optimizer), which efficiently handles the non-convex nature of our formulation while providing fast convergence rates. This approach addresses data sparsity and significantly alleviates cold-start issues for newly derived models.
Experiments
We evaluate the effectiveness of our proposed Lineage-Based Prediction methods through extensive empirical experiments using publicly available data from the Hugging Face Open LLM Leaderboard v2 (Fourrier al., 2024). Our experiments focus on two critical scenarios encountered in real-world LLM development: (1) predicting performance among homogeneous models (e.g., those derived from merging or fine-tuning within the same model family) and (2) predicting performance across more diverse, heterogemeous models (e.g., different architectures or training methods).
4.1 Experimental Setting
We conducted large-scale experiments using evaluations from the Hugging Face Open LLM Leaderboard, focusing on 6 benchmark tasks (BBH (Suzgun et al., 2023), GPQA (Rein et al., 2024), IFEval (Zhou et al., 2023), MATH (Hendrycks et al., 2021b), MMLU-Pro (Wang et al., 2024), MuSR (Sprague et al., 2024)) comprising 39 sub-benchmarks in total. Our dataset includes up to 2,934 publicly available models with partial or complete lineage information and 21,065 instances, though specific subsets were used depending on the experimental condition.
We compare three methods in our experiments:

---

Published as a conference paper at COLM 2025
Baseline: Neural Collaborative Filtering with Factors (NCF with factors) Among our baselines, this is the only approach that estimates cold-start performance while explicitly leveraging historical results. We focus cold-start methods that do not assume any target- model evaluations.! We use the same network architecture as prior work (Zhang et al., 2024)): a two-layer MLP with 128 hidden units per layer. Owing to practical constraints when collecting data from the leaderboard, we restrict model factors three attributes: architecture type (50 categories; e.g., qwen2forcaauall1m, gemma2forcasualle (three categories: fine-tuned, merged, and includinn base models), and parameter size. For task factors, we include benchmark identity (six majar benchmarks decomposed into 39 hob-benchmarks). All factors are embedded and concatenated with the model and instance embeddings before being fed to the MLP.
model type
Proposed-1; Model-Lineage Averaging (MLA) We construct lineage relationships by extracting metadata from Hugging Face model cards. For fine-tuned models, we identify parent models via the base model: finetune: tag; for merged models, we use the base model: merge: tag and, when available, the merge config. file. To ensure consistency across the corpus, we restrict model factors to those reliably present in the model-card "ach-"archifecture type," " "model type," and "parameter size." " We omit factors such as dataset size batch size because they are not reported consistently; in the homogeneous setting, such omitted factors are likely similar across models and thus have limited impact on our conclusions.
Proposed-2: Lineaae-Regularizee Matrix Factorization (LRMF) We construct the instance-similarity graph from semantic embeddings of the prompts. By default, we use Snowflake/ wwwakk/esnoolaakeeeaaeee seaee--rcci-eemed-------- 0 to embed each prompt, compute cosine sim- ilarities, and connect each instance to its top-k nearest neighbors (k=20 unless otherwise noted). The model-lineage graph reuses the lineage information described in the Model Lineage Averaging baseline, with an edge indicating a direct derivation between models. The model is implemented in PyTorch and optimized with Adam (learning rate 3 x 10 30-) We train for at most 10,000 epochs with early stopping based on the validation set (patience of 100 epochs).
Hyperparameter Tuning Hyperparameters A12, AM, and x were selected via grid search to maximize AUC-ROC on the development set, yielding the optimal values A1 = 10 5 Am = 10-4, and x 10-5 (see Appendix B for details).
We evaluate each method using two complementary metrics that assess different aspects of prediction ability:
Evaluation Metric
AUC-ROC (Individual Performance Estimation): Measures the ability to accurately estimate whether single LLM's performance on specific task exceeds a threshold, distinguishing correct (1) vs. incorrect (0) predictions the instance level.
Pearson Correlation (Relative Performance Ranking): Measures the ability to correctly preserve the relative performance hierarchy among multiple candidate LLMs on a given task, computed as the Pearson correlation coefficient between predicted and true average scores across benchmarks.
Pearson correlation is crucial for our goal of identifying the most promising model variants without exhaustive benchmarking, as it quantifies how well our method maintains the correct ranking of LLM performance. We report both metrics for each benchmark to provide a comprehensive assessment of individual score estimation accuracy and ranking fidelity.
MMethods such Polo et al. (2024) and Kipnis al. (2025) assume the target model has been evaluated on least subset of instances, and are therefore inapplicable under our strict cold-start setting. Likewise, scaling-law approaches (e.g., Ruan et al. (2024)) extrapolate from factors like parameter count or pretraining data volume- quantities that remain essentially unchanged under post-training or model perging-making them tend to be less informative for post-training model merging for the present objective.
7

---

Published as a conference paper at COLM 2025
4.2 Experiment 1: Predicting Performance among Homogeneous Models
Scenario Setup We first examine scenarios involving homogeneous models, i.e., those derived from common base model through multiple fine-tunings merges. This setting simulates practical use cases such as determining which model merges or fine-tunings are most promising.
We use 145 models derived from Qwen 2.5-7B, dividing them into 105 models for train- ing, 20 for validation, and 18 for testing. (The detailed configuration of fine-tuned and merged models is listed in the Appendix.) We sample 1,000 instances from the leaderboard benchmarks and ensure that every model both trained and evaluated on the same set of 1,000 instances. This choice follows findings that performance on 100-600 samples reliably predicts overall benchmark performance (Polo et al., 2024; Kipnis et al., 2025). For this experiment, repeat the sampling procedure five times with different random seeds and report the mean and standard deviation across these runs.
1.00
0.75
Pearson Correlation
0.50
AUIC Score
0.25
-0.5
Method
Method
L Lineage-Be Regulorized Mr
-0.50
ineage-Regulaiiee Regularired MF
Madel Lineage Avaraging
NCF factors
-0.75
Modal Lineage Averaging
with
BBH
DPG4
iPEval
MATH
MULL-PPO
MusH
BBH
GPGA
i'Eval
MATH
MALLL-PP Pro
MUSH
Benchmark
Benchmark
(a) AUC-ROC
(b) Pearson correlation
Figure 1: Performance evaluation results for a homogeneous set of models derived from Qwen 2.5-7B across various benchmarks. (a) AUC-ROC scores show that all methods achieve similar absolute performance estimation capability (.83-..86). (b) Pearson cor- relation reveals significant differences in relative performance ranking ability, with our proposed Lineage- Regulaize MF achieving the highest overall correlation (0.642), sub- stantially outperforming NCF with factors (0.490) and Model Lineage Averaging (0.301). This demonstrates the effectiveness of combining lineage information with collaborative fil- tering for preserving model performance hierarchies. Note the benchmark-speciff patterns: LRMF excels instruction-followiing (IFEval) and general reasoning tasks, while Model Lineage Averaging performs better on specialized knowledge (GPQA) where performance characteristics tend to be more directly inherited from parent models.
Results Figure reports both AUC-ROC and Pearson correlation between the actual and predicted performance of each LLM across different benchmarks. All methods achieve high AUC-ROČ scores (approximately 0.83-0.86), with our proposed leading marginally. However, when evaluating relative performance via Pearson correlation, LRMF attains a substantially higher score (0.642) compared to NCF with factors (0.490) and Model Lineage Averaging (0.301). This discrepancy indicates that while existing methods can reasonably estimate absolute performance (as reflected by AUC-ROC), they struggle to distinguish the fine fine-grained performance ordering among multiple candidate models. In contrast, LRMF's incorporation of lineage information enables accurate prediction of model hierarchies beyond individual performance estimation.
MF
Benchmark-specific results reveal meaningful contrasts in method effectiveness. On IFEval, which measures instruction-folloiing capability, fine-tuning typically induces substantial behavioral changes. Consequently, methods relying primarily on ancestor scores- such underperform (correlation: 0.175), whereas LRMF, which exploits collaborative patterns beyond direct lineage, achieves sub- stantially higher accuracy (correlation: 0.702). Conversely, on specialized knowledge bench- marks (GPQA) and complex reasoning tasks (MATH, MuSR), the Qwen-2.5-7B family demonstrates limited performance most models solve very few problems- booolems-mmkingg rela- tive ranking inherently challenging and resulting in lower correlations across all methods.
as Model Lineage Averaging (MLA)--ystematiial
8

---

Published as a conference paper at COLM 2025
4.3 Experiment 2: Predicting Performance among Heterogeneous Models
Scenario Setup To address realistic model merging scenarios, such as Frankenmerge (Wortsman et al., 2022), we examine performance prediction across diverse models with different architectures and parameter sizes. Specifically, we test whether our method remains effective compared scaling-law-basee approaches like NCF with factors, which typically benefit from heterogeneous training sets.
We train using 2,534 models, with 200 models each for validation and testing. As in Experiment 1, we sample 1,000 instances from the leaderboard benchmarks.
Results Figure/ 2 reports both AUC-ROC and Pearson correlation in the heterogeneous setting. All methods again exhibit similarly high AUC-ROC ( 0.84-0.88), while Pearson correlation shows clear separation: LRMF reaches 0.615, Model Lineage Averaging 0.504, and NCF with factors 0.301. This mirrors the homogeneous results-whie traditional approaches can estimate absolute accuracy, lineage-aware modeling in LRMF most reliably captures the relative ordering of diverse models.
Compared with Experiment 1, NCF with factors shows relative decline. This underperfor- mance likely reflects the factor-importance bias observed in prior work (Zhang et al., 2024), where SHAP analyses indicated that coarse factors (e.g,, model family) dominate predic- tions. When such strong explanatory factors exist, the model tends to collapse predictions within each family and fails to resolve within-family correlation in homogeneous settings but does not transfer to heterogeneous ecosystems.
cetenntions-bbhaveee that may inflate
Benchmark-specific trends are largely consistent with Experiment Matrix Factorization continues to excel on BBH, IFEval, MATH, and MMLU-Pro, whereas Model Lineage Averaging remains stronger on GPQA and MuSR. This stable pattern across settings reinforces our view of how distinct capabilities propagate along model lineage. Notably, MLA becomes relatively stronger this heterogeneous scenario, and the gap between LRMF and MLA narrows. This suggests that greater model diversity requires the latent space to encode more complex relationships; in this regime, direct lineage information serves a comparatively stable signal, whereas embedding-basee methods can struggle to represent nuanced differences across architectures and training recipes within unified latent space.
RegulaiRe
0.75
Pearson Correlation
Score
0.25
-0.25
Method
Method
Lineage Reguiarized MF
Lineage Requlbrized HF
Model Unnegee Averaging
-0.75
Model Averaging
BRH
GPOA
irEval
MATH
MALL--Pr
MuSR
BBH
GPGA
iFEal
MATH
MMLLL- Pro
MusR
Benchmark
Benchmark
(a) AUC-ROC (absolute evaluation)
(b) Pearson correlation (relative evaluation)
Figure 2: Performance evaluation results for heterogeneous models from the Hugging Face Open LLM Leaderboard. (a) AUC-ROC scores remain similar across methods. (b) Pearson correlation shows LRMF achieving the highest overall correlation (0.615), demonstrating effectiveness across diverse architectures and training strategies.
4.4 Experiment Routing
To evaluate the practical applicability of our performance predictions, we consider instance-level routing scenario where each input must be dynamically assigned to the LLM expected to perform best. We compare five routing strategies: (i) LRMF-based Routing (ours),

---

Published as a conference paper at COLM 2025
(ii) Model Lineage Averaging (MLA)-based Routing, (iii) Neural Collaborative Filtering (NCF) with factors-based Routing, (iv) Random Routing, and (v) Best Model, which always assigns all instances to the single model with the highest average true performance.
As shown in Figure 3, LRMF-based routing consistently outperforms all baselines in both homogeneous and heterogeneous settings. Crucially, our method exceeds the Best Model baseline- the minimum threshold for effective instance-level routing demonstrating that dynamic model assignment based on predicted performance can improve overall benchmark results beyond using a single optimal model.
These findings indicate that explicit modeling of lineage relationships substantially improves model selection in practical routing scenarios, enabling more effective dynamic assignment of inputs maximize overall performance.
1.0
USP7 ussed Racing
LMM Connn Pouting
0.2
ait futtor bosst fouting
0.5
weceeeotrroneeee anedddedddd feeting
kardom heating
Rendon soutin
0.0
80H
iroail
HATTY
0.
Hatt
HOLL-NO
Benchmark
Benchmark
(a) Routing performance in homogeneous setting
(b) Routing performance in helerogeneous setting
Figure 3: Instance-level routing performance across benchmarks in (a) homogeneous and (b) heterogeneous settings. Each test instance is dynamically assigned to the model with the highest predicted score according to different methods. We compare LRMF-based routing against MLA, NCF with factors, Random assignment, and Best Model (always using the single top- per-erronn model per benchmark). LRMF routing consistently outperforms all baselines in both settings, demonstrating that incorporating lineage information enables more effective dynamic model selection that exceeds the performance of using a single optimal model for all instances.
Conclusion
Our work demonstrates that lineage relationships- explicit parent-child connections between derived signiiiggif improve performance prediction accuracy. Across experiments with 2,934 models and six major benchmarks, our dineage-Regularizee Matrix Factorization (LRMF) framework achieved up to 0.15-0.30 increases in correlation with actual performance by effectively modeling ancestral connections between models.
Our analysis revealed that while instruction-folloiing abilities undergo substantial changes during fine-tuning or merging, specialized knowledge and complex reasoning capabilities remain more closely tied to parent models. These insights have practical implications for LLM development, allowing researchers to efficiently explore model variants without ex- haustive re- b-benchharkingg thus reducing computational costs and accelerating development cycles.
Limitations include reduced effectiveness for novel architectures without clear lineage and uniform treatment of all lineage connections regardless of modification type. Future work could explore weighted connections and deeper analysis of how specific fine-tuning techniques affect capability inheritance.
As the LLM ecosystem continues expanding through derivatives rather than training from scratch, tracking lineage information becomes increasingly valuable. Our work shows that understanding "where a model comes from" "m is just as important as knowing "what it is" when predicting L.LM performance.
10

---

Published as a conference paper at COLM 2025
References
Leshem Choshen, Yang Zhang, and Jacob Andreas. A hitchhiker's guide to scaling law estimation, 2025. URL
Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Open Ilm leaderboard v2. https://hugginngfaan co/ ssppsspppcppnnnneeee/ee 11m-m Laapeeen-m----eeerbooaoee leaderboard/ open 1.m.leadebboar 2024.
Mitchell A Gordon, Kevin Duh, and Jared Kaplan. Data and parameter scaling laws for neural machine translation. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pp. 5915-5922, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2000 emnlp-main.4788 URL https: http::///caaaaahaaacooo org/2021 momlp-main.44.
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collaborative filtering. In Proceedings of the 26th International Conference on World Wide Web, www '17, pp. 173-182, 2017. doi: 1145/3038912.30522 3052569.
URL https://dol org/10
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference Learning Representations, 2021a. URL d7KB GGmQ.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021b. URL https:://ppeeeeiiee// (tter/oorrmmm1d//yt2mmmm3m3
Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El- Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, Scott Johnston, Ben Mann, Chris Olah, Catherine Olsson, Dario Amodei, Nicholas Joseph, Jared Kaplan, and Sam McCandlish. Scaling laws and interpretability learning from repeated data, 2022. URL https: //ax:i///iivv///aa wwtpvvv...r/gb///22555. 10487.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de las Casas, Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack William Rae, and Laurent Sifre. An empirical analysis of compute-optimal large language model training. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyur Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: /tpp://///ereeiew... net/ net/ forum?)d==1BBRRJJAAP
Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, and Shriyash Kaustubh Upadhyay. Routerbench: A benchmark for multi- LLM routing system. In Agentic Markels Workshop at ICML 2024, 2024. URL https:
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL httph: //axxiv. org/abs/200 083611.
Alex Kipnis, Konstantinos Voudouris, Luca M. Schulze Buschoff, and Eric Schulz. metabench sparse benchmark of reasoning and knowledge in large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https: wtwprrrviwee.ne/tooommi--------
Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, and Jingren Zhou. Routing to the expert: Efficient -wiidddguided ensemble of large language models.
11

---

Published as a conference paper at COLM 2025
In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 1964-1974, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653 /tt////1122..maalllooo/// URL htttpp::/aalaantholgg/. org 2024. naanl-long.... 109/.
Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and Ion Stoica. RouteLLM: Learning to route LLMs from preference data. In The Thirteenth International Conference on Learning Representations, 2025. URL
https://openrewee wttp://wweeefooommmmddsssssaaaa
David Owen. How predictable is language model benchmark performance? CoRR, abs/2401.047555 2024. doi: 10.48550/ 401.0772 URL. https: https://doi.org/1/
48550/arXiv. 2401. 04757.
Lorenzo Pacchiardi, Lucy G. Cheke, and José Hernández-Orallo 100 instances all you need: predicting the success of new Ilm unseen data by testing few instances. In Proceedings of the Workshop on Evaluation and Trustworthiness of Generative Al Models KDD 2024, Barcelona, Spain, August 2024. ACM.
Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. tinybenchmarks: evaluating LLMs with fewer examples. In Forty-first Inter- national Conference on Machine Learning, 2024. URL /ttps:://ppereev/// wew. fort/forrmidd qAm13FpfhG.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: graduate-level google- proof q&a benchmark. In First Conference Language Modeling, 2024. URL https: //tpperreiieee.eeeooorrrddddddde wtt.:///ooormm///11177755553090
//openreview
Yangjun Ruan, Chris J. Maddison, and Tatsunori Hashimoto. Observational scaling laws and the predictability of language model performance, 2024. URL https:st/////xxi.orrr abs/2405. 10938.
Tal Shnitzer, Anthony Ou, Mírian Silva, Kate Soule, Yuekai Sun, Justin Solomon, Neil Thompson, and Míkhail Yurochkin. Large language model routing with benchmark datasets. In First Conference on Language Modeling, 2024. URL https://openreview.rr net/ forum?id=ZboajiZ7At
Zayne Rea Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. MuSR: Testing the limits of chain-of-thought with multistep soft reasoning. In The Twelffh International Conference on Learning Representations, 2024. URL https: http:://openreview net net/forum?id= jenyYQzue1.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. Challeng- ing BIG-bench tasks and whether chain-of-thoughe
can solve them. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Com- putational Linguistics: ACL 2023, pp. 13003-13051, Toronto, Canada, July 2023. Asso- ciation for Computational Linguistics. doi: 10. 20553/v1/2223.. findings-acl.82. URL https://aclanthhologg. org/2023. findings- 824/
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL. https:// htwp///pppreevi...ee.ee.. pet//formm2id-y100MG22
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo- Lopes, Ari Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models
12

---

Published as a conference paper at COLM 2025
improves accuracy without increasing inference time. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 23965-23998 PMLR, 17-23 Jul 2022. URL https: /ttpse//proceeding. mlr. pur.press/vll wttpss/vvv/wwoorteee 62/wortsman222 htmll.
Qinyuan Ye, Harvey Yiyun Fu, Xiang Ren, and Robin Jia. How predictable are large language model capabilities? case study on BIG-bench. In The 2023 Conference Empirical Methods Natural Language Processing, 2023. URL https://openreiiew.nee
Qiyuan Zhang, Fuyuan Lyu, Xue Liu, and Chen Ma. Collaborative performance prediction for large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Pp. 2576-2596, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 2024. emn.p-malppppp... main. 150/.
URL https: /acastthooon org/
Yi Zhen, Wu-Jun Li, and Din Yeung. Tagicofi: tag informed collaborative filtering, In Proceedings of the Third ACM Conference on Recommender Systems, RecSys '09, pp. 69-76, New York, NY, USA, 2009. Association for Computing Machinery. ISBN 9781605584355. doi: 02.1145/1339114.10
URL https: org/ /144/16397711 1639727.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra,, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models, 2023.
URL
A Detailed Model Statistics
We present more extensive information about the models used in each experiment. This includes both the partition splits (train / dev / test) and breakdown by model type: whether a given model checkpoint is (1) a Fine-Tuned (FT) model, derived from a single parent model via continued training additional or specialized datasets; (2) Merged model, created through parameter-level averaging or "souping" of multiple parent models; or (3) Other model, which encompasses base models released without any further fine- tuning or merging, as well as models for which lineage information is incomplete or unavailable.
A.1 Experiment 1: Homogeneous Models (Qwen-2.5-7B Family)
Table 1 lists all 143 models in the Qwen 2.5-7B family used in the homogeneous setting, along with their assigned partition (105 for training, 20 for development, 18 for testing) and their lineage type (Merged / FT Other). This family traces back the publicly available Qwen 2.5-7B checkpoint, with each descendant model created via single multiple merges and/or domain-specific fine-tuning steps.
Lineage Analysis of Qwen 2.5-7B Family We investigated the lineage distribution within this homogeneous group, discovering that the number of lineage connections roughly follows a logarithmic distribution (Figure 4a). Furthermore, we visualized the complete lineage network among the Qwen models (Figure 5), revealing that the model with the most lineage connections O. I-nBtttstt
A.2 Experiment 2: Heterogeneous Models (All Models)
In the heterogeneous scenario (Experiment 2), we used a total of 2,934 models from the Hugging Face Open LLM Leaderboard that included partial or complete lineage metadata. These were split into 2,534 for training, 200 for development, and 200 for testing. Due to the diversity in architectures, sizes, merge strategies, and fine-tuning procedures, do
13

---

Published as a conference paper at COLM 2025
not list each model name. Instead, Table2 summarizes the approximate proportions of base, merged, and fine-tuned models.
Lineage Analysis of Heterogeneous Models We also conducted a lineage analysis across the full heterogeneous set, uncovering again roughly logarithmic distribution in the number of lineage connections (Figure 4b). Approximately 60% of these models are either fine-tuned or merged variants, suggesting lineage information is widely applicable for modeling relationships and improving predictions across broad spectrum of publicly available models.
Table 1: Composition of the homogeneous model set (Qwen 2.5-7B family) used in Experi- ment 1.
Partition
Total
FT Models
Merged Models
Other (including Base)
Train
105
45 (42.9%)
60 (57.1% 1%)
0 (0.0%)
Dev
20
(40.0%)
12 (60.0%)
0 (0.0%)
Test
18
(50.0%)
9 (50.0%)
0 (0.0%)
Table 2: Composition of the heterogeneous model (All models from the Hugging Face Open LLM Leaderboard) used in Experiment 2.
Partition
Total
FT Models
Merged Models
Other (including Base)
Train
2,534
856 (33.8%)
707 (27.9%)
971 (38.3%)
Dev
200
(28.0%)
54 (27 .0%)
90 (45.0%)
Test
200
54 (27.0%)
66 (33.0%)
80 (40.0%)
10'
102
Number of Model
of
10
107
10
20
30
40
50
60
10
20
40
50
Number of Lineage
Number of Lineage
(a) Qwen 2.5-7B family models
(b) All models
Figure 4: Distribution of lineage connections among models. Both distributions approximate a logarithmic trend, indicating many models have relatively few lineage connections, with only minority having extensive lineages. (a) Homogeneous Qwen 2.5-7B family. (b) All models from the Hugging Face Open LLM Leaderboard.
14

---

Published as a conference paper at COLM 2025
Figure Lineage network visualization among the Qwen 2.5-7B models. The most densely connected model is Qwen- 2. S5-7-Innstrue
B Hyperparameter Tuning and Ablation Study
Our experiments involve performance data collected from six benchmark tasks provided by the Hugging Face Open LLM Leaderboard, totaling approximately 21,000+ instances, For each instance, we have binary outcomes indicating whether given LLM correctly solved the instance or not. The task is thus formulated as predicting these binary instance-level outcomes for model-instance pairs for which no evaluation available.
B.1 Experimental Protocol
All hyperparameter optimization was performed on the validation set using two comple- mentary metrics:
Primary metric: Instance-level AUC-ROC, which captures the model's ability to distinguish between correct and incorrect predictions
Secondary metric: Pearson correlation coefficient, which measures how well the method preserves relative performance superiority relationship among models
For all methods, we performed grid searches over relevant regularization parameters and selected the configuration that maximized AUC-ROC the development set. Addition-
15

---

Published as a conference paper at COLM 2025
ally, conducted comprehensive ablation studies using Pearson correlation to better understand the contribution of each component.
B.2 Lineage-Re MF
Lineage-Regularized MF extends the conventional matrix factorization by including addi- tional Laplacian regularization terms:
AM Tr [M (w0 + Ax Tr (
where ((M) and ((²) are the graph Laplacians for models and instances, respectively, and an L2 penalty with coefficient 212 is also applied. The hyperparameters AL2, AM, Ax were tuned over the range (10-9, 10-8 ...., ,,,, using the development set.
B.2.1 AUC-ROC Analysis
Figure 6 illustrates the effect on AUC-ROC when varying AM (lineage regularization) and xx (instance similarity regularization)) while fixing 412 10-5.
For the model lineage regularization parameter A, we observe a clear improvement as the regularization strength increases, with AUC-ROC rising from approximately 0.795 at Am 10 to peak of approximately 0.895 at Am 10-4, demonstrating the critical importance of lineage information for performance prediction.
For the instance similarity regularization parameter Åx, we observe that performance remains highly stable (AUC-ROC 0.895) for values at 10-5, and then decreases to approx- imately 0.86 when x 2 10-3, indicating over-regulariization.
These results indicate that model lineage regularization (Am) has more pronounced impact on performance, with optimal values around 10 10-4, while instance similarity regularization (Axr) shows broader stability across multiple orders of magnitude.
Lineegt-Reguuleraee MP
0.88
0.88
0.86
0.90
0.82
0.82
0.78
0.78
Unnaae Regularzied
10
10-
10
10
(a) Varying AM (model lineage) with fixed Ax (b) Varying Ax (instance similarity) with fixed 10
Am = 10
Figure 6: Development-set AUC-ROC Lineage-Regularized MF as function of the lineage regularization parameter AM and the instance regularization parameter xx. Each plot fixes one parameter while sweeping the other. Optimal performance is achieved with
moderate regularization (AM Rs 10 Axx = 10 5.).
B.2.2 Comprehensive Ablation Study
To more thoroughly understand the interaction between regularization components and their individual contributions, conducted an extensive ablation study measuring Pearson correlation coefficients across the full hyperparameter space (Figure 7).
Critical observations from the ablation study:
16

---

Published as a conference paper at COLM 2025
1. Baseline failure in cold-start: When both AM and Ax 0 (equivalent to standard Matrix Factorization/ the method completely fails in cold-start scenarios with correlation z confirming that vanilla collaborative filtering cannot handle unseen models.
2. Model lineage as the dominant factor: Setting AM > while keeping Ax
0 achieves Pearson correlation coefficients of 0.2-0.5, demonstrating that model lineage alone provides substantial predictive power even without instance similarity information. This shows that most of the correlation improvement stems from the model lineage regularization term.
3. Synergistic effects: The optimal configuration (AM R 10 Ax ~ 10 achieves correlation 0.5, with the heatmap revealing a clear "sweet spot" (highlighted by the red box) where both regularization terms work synergistically.
4. Robustness of model lineage: The correlation remains stable across wide range of Å values (10 to 1 indicating that lineage information is inherently robust and does not require precise tuning.
Heatmap of Pearson Correlation
1e+04
0.5
1e+02
Le+00
bvr (model lambala gggaph)))
Pearson's Conclation
1e:02
16-04
1e-06
1e:08
1c:10
0.1
16-12
0e+00
0.0
0es00
1e-11
1e-09
1e-07
1e-03
1e-01
Le+01
le+03
1e+05
1e+07
1e+09
le+11
10413
(task (task_lama www._lmmbdbbppoo ggrash)
Figure 7: Heatmap of Pearson correlation coefficients for all combinations of AMM (model lineage) and Λx (task/instann similarity) regularization parameters. The red box highlights the optimal region. Note that Λy = 0, = 0 (bottom-left) corresponds to standard MF and fails completely in cold-start prediction (correlation = 0). The visualization clearly shows that model lineage regularization (AM > 0) is the primary driver of performance improvement.
These ablation results validate our core hypothesis that explicit lineage relationships provide crucial signals for performance prediction, with the model lineage regularization term contributing the majority of the improvement. Once the best hyperparameter values are identified on the development set, we retrain
MF on the combined training and development sets and report the final performance the test set.
B.3 Ablation: Instance-graph construction
Embedding backbones. We evaluated several widely used embedding models for Stt.:/owwwaaeeessooooeeeeeerrre-
0, constructing the instance-similarity all-mpnet-base-v2 For each model, we embedded httit//e--mmistaall---iiiee
and a
the prompt text, computed cosine similarities, and formed a k-NN graph. Across bench- marks, observed no material differences in downstream metrics (AUC-ROC and Pearson correlation), and the relative ordering of methods remained stable. Given this robustness and for consistency, use Snowf Snowflakee walellaee/aaaee looke/ssoofll www.lkaaatteeeeeeeeeaee - 1-v2.0 0 in the main results.
17

---

Published as a conference paper at COLM 2025
Neighborhood size We seset k E 1,, 100) on the development set, selecting k by AUC-ROC. Performance was largely insensitive to k across tasks; we therefore k= in all reported experiments stable sparsity-connective trade-off.
C Robustness to Lineage Noise
To quantitatively assess the impact of noise/ seaaee/icooppeeeess. lineage data, we conducted additional experiments by randomly modifying the lineage information (either adding or removing lineage links) in the heterogeneous setting. Notably, we observed distinct responses between the two methods:
Lineage-Regularized MF: Robust to incomplete (missing) lineage data (removal 40% of links led to only 10% decrease in correlation), but highly sensitive to incorrect lineage additions (adding 40% false links caused 50% decrease in correlation). For LRMF, accuracy is better preserved by omitting uncertain lineage information rather than risking incorrect additions.
Model Lineage Average: Demonstrated robustness against both random additions and removals of lineage data. This resilience likely stems from random lineage connections averaging out model-specific variations and pulling results toward global mean.
Noise Level Pearson Correlation for LRMF and MLA
LIME
0.5
MLA
0.4
Pearson's Corrlection
0.3
Randomly remove the Ineages
Randomly Ineages
0.1
40
-20
20
40
60 80 100
Level
Figure 8: Effect of noise in lineage data on model prediction accuracy in the heterogeneous setting, measured by Pearson correlation. Noise is introduced either by randomly removing lineage links (negative values) or randomly adding spurious links (positive values). LRMF shows robustness to missing links but suffers significant degradation when incorrect lin- eages are added, while MLA remains comparatively stable under both scenarios.
D Impact of the Number of Observed Instances per Model on Prediction Performance
We investigate how varying the number observed instances per model (t) affects predictive performance. By sampling {5, 10,20, 50, 100, 200,500, 1000} instances for each model in the training data, we simulate scenarios ranging from minimal evaluation (resource- constrained) to comprehensive benchmarking.
Figures and 10 summarize the test-set AUC-ROC across two critical evaluation scenarios for both the homogeneous (Qwen 2.5-7B family) and heterogeneous (all models) settings:
18

---

Published as a conference paper at COLM 2025
In-distribution prediction: Estimating performance unobserved instance-model pairs, where the models themselves are included in the training set but certain instances remain unobserved (shown in Figures 9a and 10a for the homogeneous and heterogeneous settings, respectively). While this differs from the main cold- start scenario, it requires imputing missing data due to limited observations per model.
Cold-start prediction: Zero-shot performance on entirely new models unseen during training, representing the main cold-start scenario of this work (shown in Figures 9b and 10b for the homogeneous and heterogeneous settings, respectively).
Known LLMs x known Problems
New LLMs x Known Problems
1.0
Lineaqe Regular and MY
Model Uneage fueraging
0.9
0.9
6.0
0.0
0.7
0.7
0.6
Linzgge Roquirized NF
Model Averaging
NCF
10
10a
10a
10'
10
102
Number Osserved Irstances Model (1)
Number of Clasaved Instances Model
(a) In-distribution prediction: AUC-ROC for with- (b) Cold-start prediction: AUC-ROC for in- held mnstance-mode pairs within training models Maacee-moel pairs unseen test models
Figure 9: AUC-ROC as a function of the number of observed instances per model (f) in the homogeneous Qwen 2.5-7B family: (a) in-distribution prediction and (b) cold-start prediction.
LLMs Instances
LLMs Known
1.0
1.0
Uneage Reçularized NF
Model Lireago Averaging
0.9
0.5
NCF with factors
0.6
Uneage Reguartzed HF
Linnage Averoging
0.5
NCT with factors
0.5
10'
1.0
10'
30*
Number Observed Instances per Model (f)
Number of Observed instances per Wodel to
(a) In-distribution prediction: AUC-ROC for with- (b) Cold-start prediction: AUC-ROC for in- held instance-model pairs within training models stance-model pairs of unseen test models
Figure 10: AUC-ROC as a function of the number of observed instances per model (t) in the heterogeneous Hugging Face model set: (a) in-distribution prediction and (b) cold-start prediction.
The results align with our main findings in Section 4: outperforms baselines, with its advantage becoming more pronounced in the cold-start scenario and with fewer observed instances per model. This confirms our hypothesis that lineage relationships provide valuable signals for predicting the performance of new or sparsely evaluated models.
MF consistently
Impact of Limited Observations To quantify the robustness of our approach under data- constrained scenarios, we systematically reduced the number of observed instances per model from 1,000 down to as few as 5 instances. As shown in Figure 11, MF maintains impressive prediction accuracy even with drastically reduced observations,
19

---

Published as a conference paper at COLM 2025
demonstrating significant resilience compared to Model Lineage Averaging and NCF with factors.
The performance gap between methods is especially pronounced in the homogeneous MF achieves with just 50-100 instances
setting (Figure 11a), where per model what baseline methods require 500+ instances to match. This 5-10x reduction in required evaluations translates to substantial computational savings during model develop- ment. In the heterogeneous setting (Figure 11b), while all methods benefit from the diversity of models, our approach still maintains a consistent advantage across all observation levels.
1.0
VCF with factors
NC1 with factors
Middl Uneage
Nodel Lineage
0.8
Regulariadd
Commlation
Concaatonn
302
112
10a
10'
Number Observed instances, Model !!)
Number Observed Instances Model (1)
(a) Homogeneous setting (Qwen 2.5-7B family): (b) Heterogenaous setting (all models): Pearson Pearson correlation for cold-start prediction
relation for cold-start prediction
Pearson correlation coefficient for cold-start prediction as function of the number of observed instances per model (t): (a) homogeneous Qwen 2.5-7B family and (b) heterogeneous Hugging Face model set.
Figure 11:
These results have significant practical implications: developers can reliably predict new model's performance across multiple benchmarks after evaluating on just 50-100 carefully selected instances, provided that lineage information is properly incorporated. This dramatically reduces the computational overhead typically associated with comprehensive benchmarking during iterative LLM development.
E Routing Details
Below we provide detailed analysis of the routing assignments produced by each method. For both the heterogeneous model set (all models) and the homogeneous Qwen-2.5-7B family, we report:
1. The top-20 models most frequently selected across all test instances (Figures 18-22, Figures 12-16).
2. The top-20 models most frequently selected per benchmark (Figures 19-23, Figures 13-17).
In both the homogeneous (Qwen-2.5-7B family) and heterogeneous (all models) settings, we observe the following common routing behaviors:
LRMF-Based Routing. LRMF consistently leverages lineage information to select diverse set of nigh-pperforming variants. When aggregated across all tasks, instruction-tuned and merge-derived ("soup") models dominate the top selections. On per-benchmark basis, the method dynamically adapts-foo example, favoring instruction-tuned models on IFEval and large base models on reasoning benchmarks such as BBH and MMLU-Pro.
Model Lineage Averaging-Based Routing. The kNN-style lineage averaging approach
concentrates almost exclusively immediate ancestor checkpoints. It shows minimal variability across benchmarks, typically defaulting to whichever parent model achieved the highest raw score.
20

---

Published as a conference paper at COLM 2025
NCF with Pactors-Based Routing. The factor-based MLP exhibits overwhelming bias toward a single model across both settings. This behavior indicates overfitting to dominant metadata signals and lack of task-specific discrimination.
E.1 Homogeneous Qwen-2.5-7B Family setting
E.1.1 LRMF-Based Routing
LRMF-based Bloutnng Top 20 Nost Frequently Selected Models (All Tasks)
evreaho spacied
Notel
Intralloe: Spans-
Henday Owend
Number of lasts Assigned
Figure 12: Top 20 most frequently routed-to models by LRMF-based routing in the homoge- neous experiment.
301 Ba
Chok Bo 13.956
ITNAL Medees
SATE Maders
Figure 13: Top 10 most frequently routed-to models by LRMMF-based routing in the homoge- neous experiment. This demonstrates that this routing approach assigns different models as optimal destinations depending on the benchmark.
21

---

Published as a conference paper at COLM 2025
E.1.2 Model Lineage Searaaiin-Bassee Routing
MLA based Routing
Top 20 Mcst Frequently Selected Models (All Tasks)
Number of livts Assigned
Figure 14: Top 20 most frequently routed-to models by MLA-based routing in the homoge- neous experiment.
Routing
24 Model
ore No aae
Figure 15: Top 10 most frequently routed-to models by MLA-based routing in the homoge- neous experiment. This demonstrates that this routing approach assigns different models as optimal destinations depending on the benchmark.
22

---

Published as a conference paper at COLM 2025
E.1.3 NCF with Factors-Bass Routing
NCF with factors based Routing Top 20 Most Frequently Selected Models (All Tasks)
Model
Number of Tanks Assignedd
Figure 16: Top 20 most frequently routed-to models by NCF with factor-based routing in the homogeneous experiment. This result indicates that the routing strategy assigns all tasks exclusively to two models.
NCF with factorr based Routhng Hodel Selection by Task
Figure 17: Top 10 most frequently routed-to models by NCF with factors-based routing in
the homogeneuus experiment. This result indicates that the routing strategy assigns all tasks exclusively to two models.
23

---

Published as a conference paper at COLM 2025
E.2 Heterogeneous Models setting
E.2.1 LRMF-Based Routing
IRMF-based Bouting Top Most Frequently Selected Models Taskss
Eo an, Lee 148
273
and Aareoo 1i
Helooeon Oneer Lociii ladi-ie
tomesies07 indo ROVOUSUZE- ang UXX
Sie now, Lockeation undt
Caters Owend 534--tpeeion nonv
Hotel I
Hylbyee,, Owerr 3DVI0 See Jem
The 200-
perttifevers Oedendgenvi
Wille Utee 100 Bnt
Jo Zwe Day
130
15
325
30
Axsigeed
Figure 18: Top 20 most frequently routed-to models by LRMF-based routing in the heteroge- neous experiment.
UIW -amad Bazting Model Delection
The
Figure 19: Top 10 most frequently routed-to models by LRMF-based routing in the heteroge- neous experiment. This demonstrates that this routing approach assigns different models as optimal destinations depending on the benchmark.
24

---

Published as a conference paper at COLM 2025
E.2.2 Model Lineage Searagiin-Baseee Routing
MLA based Routing Top 20 Most Frequently Selected Models (AlI Tesks)
Tod__
Modal
TaE 2016- 43V22
Medeetheaal,
TwntoeettnoCreen instect
Uemo Homotoo 106 instrntttt
flvi-
Saval: Oownt
reperrne dancey 85 48 i0tts
Inchow Jin
Namber Tasks Assigeed
Figure 20: Top 20 most frequently routed-to models by MLAAbbsed routing in the heteroge- neous experiment.
Souting
Ed1 by NuMe
OPOR Mo.0
PHALL Mu-30
Figure 21: Top 10 most frequently routed-to models by MLA-based routing in the heteroge- neous experiment. This demonstrates that this routing approach assigns different models as optimal destinations depending on the benchmark.
25

---

Published as a conference paper at COLM 2025
E.2.3 NCF with Factors Factors-Based Routing
NCF with factors based Routing Top 29 Mest Frequently Selected Models (All Tasis!
Model
Namber Tasks Ancigeed
Figure 22: Top 20 most frequently routed-to models by NCF with factors-based routing in the heterogeneous experiment. This result indicates that the routing strategy assigns all tasks exclusively to one model.
WCF with factors based Roattig Model Selection by Tosk
CUAR
Figure 23: Top 10 most frequently routed-to models by NCF with factors-based routing in the heterogeneous experiment. This result indicates that the routing strategy assigns all tasks exclusively to one model.
26


Reviews:
[{"review_id": "iHIvmvif2E", "reviewer": "Reviewer_vHug", "summary": "This paper studies the problem of large language model performance prediction, and proposes to incorporate lineage information (e.g., the model is fine-tuned from a base model, or merged from two models) into the performance prediction model. \nThe proposed methods are validated with model performance on the Open LLM benchmark, in two different settings, (1) homogeneous, with 145 models derived from Qwen 2.5 7B; and (2) heterogeneous, with 2534 models based on various training recipes.\nThe proposed lineage-regularized matrix factorization (LRMF) approach outperforms the baseline of collaborative filtering with factors substantially, with up to 7-10 percentage point increases.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper explores an important research question and provide a reasonable solution to improve existing methods.\n* Simple and effective method that leads to substantial performance improvement over the baseline.", "reasons_to_reject": "* Evaluation could be strengthened with additional baselines and ablation studies.\n  * The paper currently uses _one_ baseline method (NCF with factors). This is concerning since there are multiple prior works on model performance prediction (e.g., Ruan et al., 2024; Zhang et al., 2024; Polo et al., 2024;  Kipnis et al., 2025; etc as discussed in Section 2). In the case that methods in prior work are not applicable to the problem here, the reasons should also be explained.\n  * The proposed method LRMF uses both “lineage regularization” and “task regularization”, with two separate loss terms in the final loss (line 244). It’s unclear whether the performance improvements come from “lineage regularization” or “task regularization”. If the paper claims that the former is important (as done in the title), I would expect an ablation study on the two loss terms to show that.\n* Confusing claim in the abstract and confusing paper organization\n  * An improvement of “up to _7-10_ percentage points” is described in the abstract and conclusion. The proposed method improves from 0.460 to 0.719 in Table 1 and 0.256 to 0.579 in Table 2. It’s confusing how the number of 7-10 was derived.\n  * The paper claims that the method addresses the cold-start problem in the abstract and Line 251, however I only find the relevant results in appendix C. The paper should guide the reader more explicitly if it's a main advantage of the method.", "questions_to_authors": "Question\n* Line 302: Clarification question about evaluation metrics. \n  * Did you predict instance-level accuracy (i.e., either 0 or 1), aggregate them to dataset-level accuracy, and compute the correlation (using multiple test models)? \n  * Also by “across various benchmarks”, is it referring to experimenting with each benchmark independently?\n* Line 314: Is there one fixed train/test split or do you sample multiple times? \n* Line 341: I don’t quite understand the reasoning of “this limited factor diversity explains …” Could you please elaborate on this?\n\nDiscussion\n* Line 393: I’m curious about your thoughts on how to incorporate weighted connections (i.e., edge information) and maybe multi-hop relations on the model fine-tuning/merging graph. What are methods you feel suitable for this extension?", "extra_scores": {}}, {"review_id": "xhszxgRGsA", "reviewer": "Reviewer_zxPT", "summary": "The paper studies the effect of lineage on model predictive performance -- before running large-scale optimisation and prediction. The authors show that various techniques, such as finetuning and model merging, fail to take into account the lineage relationships between various models that are derived from their ancestors. Taking lineage features into account would help in improving the correlation between actual and predictive performance. Existing techniques make use of scaling laws or an adaptation that considers parameter size and token count. The authors pose a novel lineage-based matrix factorization technique that considers known graph paths as lineage dependencies accross original and derived models. This technique also addresses the cold start problem of performance estimation of LLMs.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper addresses the cold start problem with very few predictions from the model. \n2. The paper provides a large-scale evaluation of the LMRF technique. \n3. Accounting for linear information does seem like an interesting idea.", "reasons_to_reject": "The paper demonstrates incorporating lineage, it doesn't extensively analyze the sensitivity of its LRMF model to varying degrees of noise or incompleteness in this foundational lineage data. A critical aspect would be understanding at what threshold of data inaccuracy the benefits of lineage regularization begin to diminish or even become detrimental. Furthermore, over smaller subsets of data this effect might be more pronounced. \n\nFor both homogeneous and heterogeneous  cases how does one decide to use model lineage averaging or Lineage-Regularized Matrix Factorization? The paper does not discuss in depth about the failure nodes and provide insights into the emperical analysis. Further, the paper still needs to shed light on the poor performance of GPQA", "questions_to_authors": "Please read weaknesses.", "extra_scores": {}}, {"review_id": "652AyHuJtg", "reviewer": "Reviewer_adW9", "summary": "This paper proposes to leverage lineage relationship between models to predict performance of fine-tuned or merged LLMs. It demonstrates improvement over a scaling-laws-based method and provide interesting observations with practical implications on the effect of the lineage on the model performance.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The use of the readily available model lineage information for performance prediction is a unique and novel approach to an important problem in current LLM research. In addition, the paper highlights intriguing characteristics of the relationship between model lineage and its performance, which would be interesting to explore further.", "reasons_to_reject": "A few clarifications and additions would be helpful:\n* When a new model is to be evaluated, does the entire optimization problem have to be re-run or is there a more efficient approximation that can be done?\n* How stable are the obtained model embeddings? When adding a new model or several models for example, how different will the results be?\n* The comparison is done only with one previous work that relies on scaling laws. Comparing to another performance prediction technique, e.g. from the model routing works, would help highlight the importance of the model lineage information and could increase the impact of the paper.", "questions_to_authors": "* Are the 1,000 random samples the same for all models or are they randomly selected for each model? What is the resulting size of the instance matrix?", "extra_scores": {}}, {"review_id": "Lg4TmVEiBb", "reviewer": "Reviewer_WqPU", "summary": "This paper proposes **Lineage-Regularized Matrix Factorization (LRMF)**, which incorporates ancestral model information into a matrix factorization framework. LRMF models how well a particular model performs on a specific task using latent embeddings, but adds graph Laplacian regularization terms based on model lineage and instance similarity. \n\nI found the method to be well-grounded and novel enough to warrant publication. The experimental evaluation is comprehensive, covering different model sets, cold-start scenarios, and data sparsity impacts. The paper reads well and has a logical structure. Their results back up their claims nicely. The practical value here is significant. By showing how important lineage is, the paper offers a more efficient approach to model selection and tuning.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This work addresses the high cost of LLM evaluation, a practical bottleneck faced by the LLM community, by developing tools to predict performance beforehand.\n* The core idea of leveraging lineage information through graph regularization is both novel for LLM performance prediction and intuitively makes sense. It fills a gap in existing work that typically ignores these relationships.\n* The research claims are adequately backed by experiments on representative datasets. I appreciate that they considered diverse scenarios including mixed model types, cold-starts, and varying data availability.\n* The gains shown over relevant baselines are meaningful - LRMF clearly outperforms both NCF with factors and their simpler lineage averaging method.\n* The methodology is explained clearly enough that I could probably reproduce this work if needed, with sufficient details on data, implementation, and hyperparameters.", "reasons_to_reject": "* The NCF with factors baseline only uses a limited set of factors (architecture type, model type, param size). While I understand the challenges with collecting comprehensive factors (and they cite Zhang et al., 2024 on issues with dominant factors), I would've liked to see comparison against a baseline with richer factors where possible - things like training tokens, dataset mix, or detailed architecture information where available. This would give us a more confident sense of LRMF's advantage.\n\n* The quality of lineage information feels like a potential weak spot. The approach relies on lineage data extracted from Hugging Face metadata (using \"base model:\" tags and config files). How reliable and complete is this metadata? The authors acknowledge this limitation but don't really quantify how missing or incorrect lineage data might impact results.\n\n* The current LRMF model treats all lineage connections the same way - whether it's fine-tuning, merging, different degrees of fine-tuning - and handles different path lengths implicitly through the graph Laplacian. This seems like an oversimplification of what can be complex inheritance patterns. The authors do mention this as future work, but it's still a limitation of the current approach.\n\n* The negative correlation on GPQA needs addressed. LRMF consistently shows poor (negative) correlation on this benchmark in both experiments (Tables 1 & 2), even performing worse than their simple Model Lineage Averaging baseline. This suggests the LRMF assumptions might break down for certain task types. This deserved deeper discussion and analysis than just reporting the number.\n\n* The construction of the instance similarity graph (A^(X)) depends on a specific embedding model and a fixed k=20. I would've liked to see some exploration of how sensitive the results are to these choices.\n\n* The evaluation focuses on relative performance (using Pearson correlation) to rank models correctly. While this makes sense, I think some discussion on predicting absolute performance scores would be valuable for certain applications.\n\n* The paper is missing citations for several models they mention - Llama 2 (which they also capitalize incorrectly), Pythia, Qwen, etc. And when they mention specific models like Qwen-2.5-7B-Instruct, hyperlinks to the Hugging Face models would be helpful.", "questions_to_authors": "* Why are we seeing negative correlation by LRMF on the GPQA benchmark across both experiments. Even the simpler Model Lineage Averaging baseline does better here. Do you have any insights into why LRMF might struggle specifically with this benchmark?\n\n* Could you tell us more about the specific factors used in the NCF baseline and how you chose them? Did you try incorporating potentially stronger factors, even if they were only available for some models, to further assess what lineage regularization uniquely contributes?\n\n* I'm wondering how sensitive the LRMF results are to how you constructed the instance similarity graph A^(X). Have you experimented with different embedding models for prompts or different values of k when defining neighbors?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies the task of predicting performance of LLMs on downstream tasks without running the evaluation. The key insight is to use information about lineage, i.e., what base model the LLM in question is derived from. The proposed lineage-aware method based on matrix factorization substantially improves prediction accuracy. All four reviewers agreed that the paper was insightful and achieved substantial gains on this task. The authors should incorporate their comprehensive rebuttal materials into the final version of the paper.
Decision: Accept