A Study on PAVE Specification for Learnware
Authors: Hao-Yu Shi
Zhi-Hao Tan
Zi-Chen Zhao
Yang Yu
Zhi-Hua Zhou
Venue: ICLR 2026 Poster (2026)
Keywords: Learnware
Specification
Parameter Vector
Learnware Identification
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We formalize the Parameter Vector (PAVE) specification, which encodes model capabilities for efficient learnware identification, eliminating costly per-model evaluations and outperforming fine-tuned pre-trained models in limited-data scenarios.

Abstract: ``Learnware = Model + Specification''. A learnware comprises a submitted model paired with a specification sketching its capabilities. For a Learnware Dock System (LDS) which accommodates numerous models, these specifications are essential to enabling users to identify helpful models, eliminating the requirement for prohibitively costly per-model evaluations. Recently, Parameter Vector (PAVE) specification, which utilizes the changes in pre-trained model parameters to inherently encode the model capability and task requirements, shows promising capabilities in enabling identifying useful learnwares for high-dimensional, unstructured text data. In this paper, we present a comprehensive study of PAVE specification for learnware identification. Theoretically, from the neural tangent kernel perspective, we establish a tight connection between PAVE and prior specifications, providing a theoretical explanation for their shared underlying principles. We further approximate PAVE in a low-rank space and analyze the approximation error bound, highly reducing the computational and storage overhead. Extensive empirical studies demonstrate that PAVE specification excels at identifying CV and NLP learnwares even from heterogeneous learnware repository with corrupted model quality. Reusing identified learnware to solve user tasks can even outperform user-fine-tuned pre-trained models in data-limited scenarios.

Published as a conference paper at ICLR 2026
A STUDY ON PAVE SPECIFICATION FOR LEARNWARE
Hao-Yo Shi, Zhi-Hao Tan, Zi-Chen Zhao, Yang Yu, Zhi-Hua Zhou
National Key Laboratory for Novel Software Technology, Nanjing University, China School of Artificial Intelligence, Nanjing University, China (shihy, tanzh, zhaoze, yuy, zhouzh } @ (laadda... nju. edu..edd..omm cn
ABSTRACT
"Learnware Model + Specification". A learnware comprises a submitted model paired with a specification sketching its capabilities. For a Learnware Dock System (LDS) which accommodates numerous models, these specifications are essen- tial to enabling users to identify helpful models, eliminating the requirement for prohibitively costly per-model evaluations. Recently, Parameter Vector (PAVE) specification, which utilizes the changes in pre- trained model parameters to in- herently encode the model capability and task requirements, shows promising capabilities in enabling identifying useful learnwares for high-dimensional, un- structured text data. In this paper, we present a comprehensive study of PAVE specification for learnware identification. Theoretically, from the neural tangent kernel perspective, we establish a tight connection between PAVE and prior specifi- cations, providing a theoretical explanation for their shared underlying principles. We further approximate PAVE in a low-rank space and analyze the approximation error bound, highly reducing the computational and storage overhead. Extensive empirical studies demonstrate that PAVE specification excels at identifying CV and NLP learnwares even from heterogeneous learnware repository with corrupted model quality. Reusing identified learnware to solve user tasks can even outperform user-fiiettuned pre-trained models data-limited scenarios.
INTRODUCTION
"Learnware = Model + Specification" (Zhou 2016 Zhou and Tan] 2024). Consider a scenario where many machine learning developers are willing to share their trained models but are reluctant to disclose their private training data. In this context, we can envision a Learnware Dock System (LDS) (Tan et al. 2024b) that accommodates a diverse array of models. These models may be trained on different data, for different tasks, and based on different learning algorithms, w.r.t. different objectives, appearing as different models (not limited to neural networks), contributed by developers all over the world. Then, with the LDS accommodating huge amount of models, future user, who plans to build her own machine learning model, can submit her task requirement to the LDS, and LDS will then identify some helpful model user, such that she does not need to build her own model from scratch. Note that, during the whole process, neither model developers nor the user disclose their training data to the LDS. Given the fact that the LDS does not have access to the original training data, and that many models are blackbox ones whose functionality can hardly be understand, how can the LDS identifies helpful models to the user task? how can the various models, even seemingly-irreleva ones, be identified and reassembled to work together for the user?
reassemble sets of models to be helpful to the
"Learnware" makes this possible. A key ingredient is the "specification" which enables a trained model to be adequately identified to reuse according the requirement of new user who knows nothing about the model, while model developers' training data are preserved. Briefly speaking, when each model submitted to I.DS, specification generated by a machine learning process will be assigned to the model, such that the model becomes a learnware. The specification is able to roughly characterize the functionality of the model to some extent, without getting access to its training data. Future user's task requirement will also be submitted in the form of task specification. Once the LDS has accommodated a huge amount of learnwares, it is expected that many fancy abilities can
*Coreespnddenc:: Zhi-Hao Tan <tanzh @ lamda.nju.cdu..

---

Published as a conference paper at ICLR 2026
1. stng Leamosi (3) Trilning Taek (Ph,
Tradking Taae Dt
Lass Fundion
Fidiition
Pro
PAUE
Submit
Skoethhing (P%,,
leartlinng
Skanching Use Thank
Identily
Parsinee Videa Space
Shildin Monel Capetar
Fineshed Part P.
Innos Fuincl on Uier Tosk
Search
Spediicriin
Model
Spacilial
Previouree blooee
Dar lock
Hogi
Uniiecki
LAAMANN
Figure 1: Identifying helpful learnwares based on parameter vector similarity. 1. The developer trains model with large amount data in D, and generates the model vector TĄ based on the model prediction y, then submits them to the system as a learnware. 2. The user generates the task vector Tu from a few samples D.. 3. The larger the cosine similarity between the model and task vectors means that the more likely the model capability is to fulfill the user task requirements.
emerge, such "small models do big" (Zhou and Tan 2024 Tan et al. 2025), and models can
exhibit abilities of doing things even beyond their original development purposes (Liu et al. 2024). Existing studies (Tan et al. 2224a Lei et al. 2024) have shown success in tabular scenarios by using a provided explicit kernel to generate a sketch the data distribution where models excel. But when facing high-dimensional, unstructured
reduced set serving as a specification to
data like images or text, the required sample complexity based on classic universal kernels tends to be unaffordable (Phillips and Tai 2020).
Recently, by fine-tuning a shared pre-trained model to fit p()]x) with the task's customized loss function, the changes in the pre-trained model's parameters, referred as Parameter Vector (Tan et al. 2025), provide an effective way to represent the model capabilities and user task requirements. Fundamentally, trained model can be viewed as mapping from input r to output ÿ. which is represented by p(ù x). thus the most straightforward way to sketch the conditional probability p(ū(x) to fit a model to it directly. During the process, information about p()]x) essentially leads and only leads-to changes in the pre-trained model's parameters, and is thus incorporated into the changes, which can characterize models and tasks in a unified way. Changes in model parameters have also been explored in Model Merging (Ilharco et al. 2023 Ortiz-Jiménez et al. 2023) and Gradient Matching (Zhao et al. 2021 Shi et al. 2022).
We find that the parameter vector also addresses some key difficulties for learnware identification in open and complex real-world scenarios. Specifically, in such scenarios, it is very challenging to identify helpful learnwares for task due numerous diverse task types and semanties and a lack of assurance regarding model quality. (a). Due to the diversity of tasks, the output y usually has different meanings and is difficult encode universally, making their comparison intractable. For instance, models trained on a single face dataset can address diverse tasks, such as age prediction and emotion classification. However, it's hard to tell which model best suited for hair color recognition. (b). Regarding model quality, some low-quality models that might not be adequately trained on their respective training datasets yet are still submitted. Evaluating model quality using unified test datasets is unrealistic in open, real-world scenarios due to task diversity and data privacy concerns. For the above issues, the parameter vector shows a promising solution by sketching the model capability x())x) and capture both the task semantics and model quality in unified way. Though, directly utilizing the parameter vector will result unaffordable computational and storage cost. This motivates us approximate the parameter vector in low-rank space, enabling efficient identification of helpful learnwares from a large number of candidates for user tasks.
In this paper, present a comprehensive study of Parameter Vector (PAVE) specification for identifying learnwares proficient in user tasks by measuring the vectors similarity. shown specifically in Figure showing its potential to quantify cross-task model-task cespaatbility-sspeciin in
2

---

Published as a conference paper at ICLR 2026
kaing Genten frstrained
Fre- Saire
Menitynn Lesemnnnts und
Logerad
Ar
Figure 2: Advantages over fine-tuning pre-trained models. With limited samples and the restricted capacity of single model, even a well pre-trained model Fre may fail to adapt to unusual user task 72. In contrast, a large collection of learnwares offers stronger overall capability, allowing more user tasks be addressed by identifying appropriate models for further adaptation. Arrow length (Manhattan distance in blocks) denotes the required number of samples when adaptation is needed.
scenarios with heterogeneous tasks, involving diverse semantics and varying output spaces, across different model qualities. Our main contributions can be summarized as follows:
We first formalize PAVE specification which constructs a compact representation based on the changes in pre-trained model parameters, inherently encoding the model capability and task requirements. Then we theoretically establish tight connection between PAVE and RKME specification (Zhou and Tan 2024) with specific kernel, providing a theoretical explanation for their shared underlying principles.
We further approximate the parameter vector in a low-rank space and analyze the approxi- mation error bound, substantially reducing computational and storage overhead.
Extensive experiments spanning CV and NLP reveal that the PAVE specification effec- tively identify quality learnwares from heterogeneous and corrupted repositories, and these identified learnwares can even outperform eser-fine-tuned models in data-limited scenarios.
PROBLEM SETUP
In this section, we summarize the learnware identification process for solving machine learning tasks. In general, we assume that each machine learning task (D, L) consists of dataset D C X' x 3 and loss function L (h, Th) consists of model h X ) and a specification Th for sketching the model capability. A learnware is helpful for the user task (D., C..) if the corresponding model achieves a low loss L,, on dataset D,,, thereby performing well on the data distribution p.. (x, y) from which D,, is drawn.
the input space and the output space. And learnware -> E. where
The entire identification process of learnwares can divided into three parts, shown Figure
1. For the i-th developer, they train the model h; on their local training dataset Dr, using the loss function L,, and construct the corresponding specification Th, The learnware (h, Th,)) is then submitted to the learnware dock system, platform for sharing learnwares.
2. For user aiming solve the task (D,., L.). they construct a specification Ta based the few-shot dataset D., and the task-specific loss function L.. Instead of training a model from scratch, the user submits тu to the platform to find suitable learnwares for their task.
3. The learnware dock system identifies learnwares that best match the user's task and mini- mizes the loss L. on Pu based on the specifications submitted by both users and developers.
3 METHODOLOGY
In this section, we formalize the PAVE specification for identifying helpful learnwares via parameter vector similarity. Section I.I the formalization of sketching the model capability by fitting the distribution p(ğ]x) |x) for learnware identification. Section 3.2 then establishes the theoretical
connection between PAVE and prior specifications (Zhou and Tan 2024 Wu et al. 2023). showing
3

---

Published as a conference paper at ICLR 2026
the underlying shared principles. Finally, Section 3.3 shows how parameter vectors can be optimized by compressing them into a low-rank representation. substantially reducing storage and computational overhead, and analyzes the approximation error bound theoretically.
3.1 FORMALIZATION OF PARAMETER VECTOR SPECIFICATION
To sketch model capability identify high-quality learnware that closely aligns with user tasks, the parameter vector, serving as the specification, is constructed by fitting the distribution p(g x)), integrating both task semantics and model quality.
To underscore the importance of sketching a model's capabilities, we begin with an intuitive example. Consider two models: one for age prediction and another for sentiment classification. If user wants to recognize hair color, which model is better suited? With output spaces differing in structure and meaning. direct alignment and comparison become challenging. Moreover, if the models vary in quality, which should the system recommend? To protect data privacy, real user data cannot be submitted online for evaluation, and requiring users to evaluate multiple models locally and provide feedback is impractical. Next, we present solution to address both challenges simultaneously.
The capability of model is characterized by the relationship between the input x and its prediction h(x), formalized as p(h( It consists of two components: task semantics and model quality, The former is determined by the input and output space, while the latter arises from the alignment between the predicted value and the true label y. Meanwhile, the requirements of user task can be formalized as the conditional probability pa ()) reflecting the required capability to solve it.
In this paper, parameter vectors categorized into two types based on their functionality: model vectors to represent the model capability, and task vectors to represent the user task requirements. For the model h trained task (Ct, Dt), the corresponding model vector can be constructed through |x). In some cases, the output not J but rather a feature space Z C ROO And, it is common
fine-tuning the pre-trained model f to represent its capability, p(h(x)|x))
space of the pre-trained model practice to bridge the feature space Z to the output space y using function
Z + y, such that go f X + y. Typically. the function g can be constructed without training via Prompt-Tunnigg (Lester et al. 2021 In general, we can obtain the model vector Th. for the model / through
Th = arg min (x.))ED:
Σ
Li(gr o f(x; On T), M(x)).
And for the task vector T., the prediction h(x) should be replaced with the true label y, reflecting the required capability solve the user task. Then, the alignment between the model capability and the user task requirements can be assessed by the PAVE similarity
cos(Th, Tw) = (p((x))))) (x(xx))).
(2)
Accordingly, we identify the learnware (h, Th) to solve the user task with specification Tn if cos( The Ta) is high. In practice, we identify the one with the highest similarity.
A model is identified only if its capability p(h(x)lx) aligns closely with the user task requirements (v())x). Besides, if model is of poor quality, its capability p(h(x)|x) deviates significantly from the true task semantics p(y|x) on the training dataset. Therefore, even if the training dataset accurately
reflects the user task-i.e., p(y (x) = pu(y)x)~a x) low-quality model will still not be identified.
3.2 THEORETICAL CONNECTION BETWEEN PAVE AND RKME
To clarify the relationship between PAVE and prior specifications speciffcaly RKME (see Ap- pendix F.I for details)- note that, under the NTK assumption, parameter vector can be viewed as an accumulation of gradients over the task data. This implies that the inner product between PAVES corresponds to a kernel function over the tasks' sample spaces, with each parameter vector representing a kernel mean embedding. Consequently, PAVE similarity and the maximum mean dis- crepancy (MMD) employed by RKME exhibit order consistency in evaluating model-task alignment.
Using gradient descent, the model update vector Ty can be expressed as an accumulation of gradients:
V Lilo D:) V.Le(ge f(xi, Oo T), M(x:)) Vo Vigh VoJ(x; 0o) E R10 (3)
4

---

Published as a conference paper at ICLR 2026
where y/ 9t f(x,, 6 +)) and R/ is the parameter space. The second equality in Equation holds if the fine-tuning process satisfies the neural tangent kernel (NTK) behavior (Lee et al. 2019), where the gradient function evolves with accumulated updates but remains approximately fixed under linearized dynamics. The same formulation applies to the task vector Tu, with úJ replaced by Yj-
Lemma Suppose the fine-tuning process lies in the neural tangent kernel regime. Then the parameter update vector is the accumulation of gradients
VuLi (s) v.C.t V29t VoJ(xi, 0o) E TR
Moreover, the inner product between samples from the training task (De, Li) of model h and from the user task (Dus Lu) defines implicit kernel function kg,c...
(V~Lille Bc). Lulli v:))
-EK Kc.a)r kF,c.o (x;, Xj, Ny.. T,
(5)
where the empirical neural tangent kernel K (Xi, X3) Vof(x,, 0o) V f(x,, Go) Mohamadi et al. 2023), and (h,)(h(x,), 93) LV (V Cu Vagu).
The component Kc., is dynamically constructed from the task semantics induced by the loss functions Lt, Lu and mappings 9t. 9u- acts as a consistency weight matrix applied element-wiss to Ky, which encourages similarity between pairs of features sharing the same correlation with solving the task and penalizes the similarity of others. Therefore, the implicit kernel function kg,c.o in Lemma 1 jointly encodes input data model predictions h(x), and task semantics (Domingos/ 2020). The more consistent the onput-output relationships between two samples, the larger the kernel value. The detailed derivation of the concrete construction of the kernel kg,C.g given in Appendix A
Lemma 2. For the model vector Th and the task vector Tu their inner product can be expressed as
(Th, Tu)
Σ
Σ
k h(x;), X;: U))
With normalization constant Z chosen such that (T, T) 1, 1, the PAVE T be expressed in form analogous to an empirical kernel mean embedding (KME), up to a rescaling factor:
Kg,coglis )), where =(Xi, y()-
T
7
To eliminate the bias introduced by dataset size and optimization trajectory length, normalize the gradient sum when defining т. With the definition of T in place, we are now ready to present an important result showing the order-consistency between PAVE similarity and MMD.
Theorem 3. Suppose the fine-tuning process operates in the neural tangent kernel (NTK) regime, under which the parameter vector admits an interpretation as a kernel mean embedding. Then, for any user task, the similarity between PAVEs and the MMD between the corresponding distributions with respect to the kernel C.g are order-consistent, inducing the same preference between models.
kf,c.g
Building on Lemmas 1 and 2 Theorem is natural consequence, as both measures clearly induce the same ordering. From the PAVE perspective, this ordering reflects the similarity between the directions of two fine-tuning trajectories in function space, whereas from the MMD perspective, captures the distance between the corresponding distributions in RKHS (Gretton et al. 2012).
3.3 APPROXIMATION OF PARAMETER VECTOR SIMILARITY
The weight space of modern pre-trained models is typically in the hundreds of millions, making direct construction and comparison of parameter vectors derived from the weight space impractical. We propose a method to approximate parameter vectors in a low-rank space, reducing their size to less than 1% of the pre-trained model's parameters often fewer than 1M IM-thereby significantly enhancing efficiency construction, comparison, and storage. Furthermore, we provide theoretical guarantee that approach preserves similarity with high probability, as illustrated in Figure
5

---

Published as a conference paper at ICLR 2026
(A) Sasiii simiarit cailts lot with pa semitaa Tre Laring
Casine unilarly cas Buck Roket
FO- en divitarity SE
Figure 3: Consistency of the cosine similarity between parameter vectors, with the diagonal colors omitted to enhance visual contrast. (a) shows the exact similarity of the parameter vectors with full parameter fine-tuning. (h) shows the approximating similarity in the low-rank space after expanding parameter vectors to (BA) in full size. (c) shows the approximating similarity computed using B alone for improved storage and computational efficiency, which is the method we propose.
Specifically, we approximate the parameter vector in LoRA (Hu et al. 2022) manner that = + - AW AW AW"1 = (B'AA) (B²A3) (B*A*) e BA,
(8)
where the pre-trained model consists of L weight matrices [W').. To simplify notations, we assume that A - [A' A A1 and likewise, where A is initialized randomly and B is initialized with zeros. As improvement to Equation (I), we replace the parameter vector t with low-rank approximated BA, leading to a significant reduction in construction and storage overhead.
With constructed model and task vectors, it's still challenging to compare their similarity efficiently. we expand the metrics B' and A A to (B' D')))mmx in full size, it costs the same computational overhead for computing cosine similarity as without approximation. To avoid this, we propose to approximate cos(fi, +2) by computing Cos(BI, B2) only, where the parameter vectors are implicitly flattened for the inner product and cosine similarity calculations. We gonna give some key ideas here and the rigorous proof is presented in Appendix
B First, it's easy to verify that in expectation
E[A'(A))
sI and B(A +2) B( B2A)] s(B,, B2),
where A ~ w1 3s n, + /s/nn/ element-wised with a constant factor s. Note that it is Kaiming initialization (He et al. 2015) with proper Then, for the cosine similarity,
prove that
cos(Fi, +2) x cos(B), B2).
(10)
And more details are presented in Theorem In summary, the exact parameter vector similarity cos(Tr Tı, T2) is equal to cos B1, B2) in the low-rank space approximately.
4 EXPERIMENTS
In this section, we evaluate our method on natural language processing and computer vision tasks. We first outline the experimental setup, including the dataset, compared methods, and experiment configuration. Our results show that identified learnwares outperform fine-tuning pre-trained models on user tasks and surpass the previous methods, particularly in solving tasks beyond the original functionality of the learnwares. Furthermore, ablation studies confirm the effectiveness and necessity of our method in preserving model quality while maintaining computational efficiency.
4.1 EXPERIMENTAL SETUP
We give a brief description of the evaluation dataset, the compared methods, and the experiment configuration. And, more specific information can be found in Appendix C
Datasets. We evaluate our proposed method on 15 classic natural language processing datasets, medical L.M benchmarks and 12 computer vision datasets, 9 medical SFT datasts and 9 medical

---

Published as a conference paper at ICLR 2026
Table 1: Performances NLP datasets beyond the original functionality.
User Task
Finetune Pro-trained Models
Identify Learneaee
Dataset
Metric
BERT-B
BERT-L
RoBERTa-B
RoBERTa-I.
Random
RKME.
PAVE
Oracle
CoLA
Mec
0.049
0.012
0.086
0.064
0.095
0.083
0.100
0.162
MNLI
Acc
0.498
0.519
0.594
0.646
0.582
0.628
0.677
0.677
MRPC
Acc
0.537
0.703
0.700
0.718
0.782
MRPC
F1
0.609
0.694
0,749
0.776
0.782
0.799
0.854
0.854
QNLI
Acc
0,552
0.542
0.636
0.682
0.643
0,580
0.732
0.749
QQP
Acc
0.574
0.512
0,653
0.664
0.646
0.696
0.631
0.715
QQP
F1
0.542
0.558
0.648
0.664
0.637
0.686
0.657
0.686
RTE
Ace
0,527
0.560
0.523
0,686
0.596
0.635
0.560
0.668
SNLI
Acc
0,515
0.575
0.676
0.753
0.671
0.702
0.755
0.755
SST-2
Acc
0.811
0.872
0.896
STS-B
Poc
0.332
0.536
0.740
0.620
0.603
0.440
0.718
0.778
STS-B
See
0,312
0.529
0.734
0.615
0.593
0.424
0.722
0.781
WNLI
Ace
0.408
0,408
0,465
0.423
0.458
0.437
0.479
0.521
or
Ace
0.873
0.881
0,906
0.920
0. 0.886
0.888
0.888
0.902
mpqa
Ace
0.789
0.822
0.812
0.810
0.814
0.826
0.838
Acc
0.757
0.802
0.849
0.877
0.828
0,850
0.850
0.850
subj
ACE
0.858
0,909
0.894
0.885
tree
Ace
0.752
0.728
0,740
0.824
0.732
0.702
0,750
0,770
(1).(1)
0.572
0.610
0.682
0.699
0.668
0.659
0.709
0,739
Avg. rank(()
6.389
6,000
3,333
2.444
4,056
3.389
2.111
PAVE (winttic/los)
17/0/1
17/1/0
13/055
10/0/8
15/0/3
12/3/3
0/6/12
LLM benchmarks. The natural language processing tasks are mainly from the GLUE benchmark (Wang et al. 2019a) and some other datasets. And the 12 computer vision tasks incorporate diverse
task semantics, such as satellite images in EuroSAT AT (Helber et al. 2019). traffic signs in GTSRB (Stalkkamp et al., 2012) and scene recognition in SUN397 (Xiao et al. 2010). And the medical LLM
benchmarks come from the Open Medical-LL.MM (PalerboarddPal et al. 22024) following the setup in
(Tan et al. 2025). The details of the datasets are described in Appendix
C.I
Compared methods. We evaluate our method, denoted as PAVE, against famous pre-trained models fine-tuned for the user task, including RoBERTa (Liu et al.) 2019), ) BERT (Devlin et al. 2019), and CLIP (Radford et al., [021). Additionally, we include RKME (Tan et al. (024bb from previous studies as a compared method and introduce several baseline methods. Specifically. 'Random" refers to the random identification of learnwares, representing the average capability of the learnwares, while "Oracle" denotes the selection of learnwares that perform best on the user task, representing the optimal capability. A specific description of the compared methods can be found in Appendix C.2
Experiment configuration of learnware set and user tasks. Each dataset consists of two subsets: training dataset and test dataset. In this experiment, the training dataset is further divided into two parts: the dataset DD for developers to constructe the learnwares and the k-shot dataset D,, as part of the user task (Du; fu). To construct the learnware set, the developers trained the models on the dataset D., paired them with the corresponding model vectors as specifications, and added them to the set. Specifically, for NLP and CV tasks, each dataset provided two versions of learnware: well-trained version and a corrupted version (consisting of low-quality models). And for models of the same type, whether corrupted or not, they are trained using identical hyper-parameters (e.g.. learning rate, number of epochs) to ensure generality. For medical L.LM tasks, we only use high-quality models as the inherent output degeneracy (e.g., self-repetition) triggered by destructive fine-tuning precludes any meaningful evaluation; thus, model vectors are directly obtained by fitting the training data to characterize model capabilities. Additionally, both the construction of task vectors and the fine- tuning of pre-trained models on the user tasks are performed within the few-shot learning framework. Finally, the identified learnware is evaluated on the corresponding test dataset. The detailed process of learnware training and the hyper-paramette configurations can be found in Appendix
4.2 PERFORMANCES
We show that learnwares identified by our proposed method outperforms fine-tuned pre-trained models on user tasks, which is the most popular approach for solving new deep learning tasks in the past. Moreover, our proposed method significantly improves upon the previous RKME method. We
7

---

Published as a conference paper at ICLR 2026
also verified that in many tasks, our proposed method can identify learnware that efficiently solve tasks beyond their original function and exhibit strong generalization capabilities.
4.2.1 ALIGNING MODELS WITH USER TASKS
This experiment confirms that PAVE specification can identify the model from the candidate set that best aligns with the user task requirements. And to ensure objectivity and completeness, we evaluated PAVE on both natural language processing and computer vision tasks.
Classic NLP. The results of our evaluation are presented in Table A5 and Figure A7 demonstrating the advantages of PAVE. And the evaluation metrics follow the GLUE benchmark evaluation settings. The performance of PAVE specification significantly surpasses the random selection and is close to the optimal capability, demonstrating better results than RKME. Remarkably, PAVE is tied with Oracle and identifies the optimal learnware 14 out of 18 user tasks.
Medical LLMs. The evaluation results on Open Medical LLM Leaderboard (Pal et al. 2024; Singhal et al. 2022) are presented in Table A4 demonstrating the effectiveness of PAVE specification in identitying large language models. We compare our approach with three popular LLMs and other baseline methods, showing that our method offers a clear overall advantage. Additionally, a visual illustration of the identification results provided in Figure A6 for further insights.
Computer vision. The evaluation results presented in Table And
and Figure A8 PAVE effectively identifies the most proficient candidate learnware for the user task. Furthermone, it demonstrates clear advantages over fine-tuning pre-trained models and previous RKME methods.
4.2.2 BEYOND THE ORIGINAL FUNCTIONALITY
This experiment demonstrates that the identified learnwares can be utilized beyond their original functionality, meaning that models can be repurposed assist with tasks not originally intended by their developers. For each user task, remove all learnwares in the candidate set that have been trained on the corresponding training dataset, presenting results Table [ and Figure A9] where "PAVE"" represents the performance after fine-tuning the identified learnware on the user task.
It shows that fine-tuning the identified learnware achieves better performance compared to fine-tuning pre- ire-trained models. This finding suggests that learnwares can effectively tackle tasks beyond their original functionality, even outperforming popular pre-traine pre-trained models. One possible explanation that the learnware candidate set, as a whole, encompasses a broader range capabilities than a single pre-trained model, shown in Figure 2 Furthermore, the PAVE specification outperforms the previous method, RKME, by an average of 7.59% in Table
4.3 ABLATION STUDY
We carry out ablation studies demonstrate the necessity of the components in our method. First, we verify the necessity of sketching the model capability p(ğ(x) for capturing model quality, instead of the data distribution p(y|x). Second, we show that approximating the parameter vectors in the low-rank space truly leads to great speedups and is almost harmless for the performance.
4.3. MODEL QUALITY
To verify that fitting the model capability p()]x) instead of the data distribution p(y[x) (x) is necessary for filtering out low-quality learnwares, we construct a corrupted version of each learnware that has not been trained on the training dataset and include them in the candidate set. The results of the ablation experiments are presented in Tables and A8 where "PAVE represents the performance of model vectors derived from fitting the data distribution p(y|x). serving as the ablation variant. Notably, the ablation variant "PAVE*" performs significantly worse than PAVE, highlighting the necessity of fitting the model capability p(o(x) when the model's quality is not guaranteed.
4.3.2 PARAMETER VECTOR APPROXIMATION
We confirm that approximating parameter vectors in low-rank spaces significantly improves storage and computational efficiency. The computational overhead comprises two primary components:
8

---

Published as a conference paper at ICLR 2026
Table 2: Performances computer vision datasets with corrupted learnwares.
User Task
Finetune Pre-trained Models
Identify Learnware
Dataset
ResNet-152
ViT-B-32
VIT-L-14
Random
PAVE
PAVE
Oracle
CIFAR10
0.100
0.899
0.956
0.839
0.894
0.890
0.979
CIFAR100
0.008
0.663
0.777
0,585
0.764
0.885
0.885
Cars
0.006
0.604
0.784
0,545
0.639
0.680
0,680
DTD
0.022
0,440
0.556
0,422
0.568
0.698
0.698
EuroS AT
0.161
0.541
0.536
0.465
0.742
0.991
0.991
Food101
0.008
0.828
0.924
0.762
0.856
0.886
0.886
GTSRB
0.030
0.330
0.511
0.338
0.656
0.987
0.987
MNIST
0.117
0,503
0.764
0,517
0.740
0.997
0,997
RESISC45
0.019
0,611
0,713
0,565
0.770
0.934
0.934
STLI0
0.060
0.973
0.994
0.949
0.978
0.985
0.985
SUN397
0.003
0.648
0.694
0.611
0.686
0.739
0.739
SVHN
0.096
0.329
0.584
0.357
0.643
0.970
0.970
Avg.(T)
0.052
0.614
0.733
0,580
0.745
0.887
0.894
Avg. rank(L)
6.000
4.000
2.167
4.750
2.583
1.500
PAVE (win/tic/loss)
12/0/0
11/0/1
8/0/4
12/0/0
11/0/1
0/11/1
PAVE* (win/tie/loss)
12/0/0
11/0/1
5/0/7
12/0/0
1/0/11
0/0/12
constructing the specifications and computing their similarity. Similarly, storage overhead refers to the space required store them. In Figures 3 A13 and [A14] the results demonstrate that the approximation preserves the relative relationships terms of cosine similarity. Table (Able(A9) (abble/Assummarzzes summarizes the computational and storage overheads associated with each method, along with their performance. These results underscore the benefits of approximating parameter vectors in low-rank spaces.
4.4 ANALYSIS
In order to gain deeper understanding of PAVE specification, we perform some further analysis in Appendix E We perform Hierarchical Linear Regression (Woltman al. 2012) show the significant positive correlation between similarity and performance. Moreover, we apply Exact Multinomial Test (Read and Cressie 2012) to verify that PAVE able to identify the helpful learwware with statistical significance. These findings support the effectiveness of PAVE strongly.
RELATED WORK
This section reviews related topics and illustrates their relevance to our work. First, our work contributes to Learmware paradigm by enhancing its identification for deep learning models and characterizes model capability the model's Transferability to tasks without unscalable per-model evaluations while ensuring data privacy. We also advance the understanding of Parameter Vector by offering insights into the relationship between changes in model parameters and the model capabilities.
more generalizable and efficient way. Second, we evaluate
Learnware. The learnware paradigm (Zhou 2016: Zhou and Tan 2024) leverages specifications to identify and reassemble multiple heterogeneous models for emergent capabilities without accessing original training data. A key method for identifying suitable learnware is the Reduced Kernel Mean Embedding (RKME) specification (Zhou and Tan 2024). which matches the user's data distribution with the model's 's original data distribution. Its unsupervised version has been demonstrated effective various tasks (Wu al., 2023). Recently, there have been various subsequent works that reinforce the feasibility of the learnware paradigm. Lei et al.) (2024) theoretically prove its ability of the learnware paradigm for developers' raw data. Liu et al. (2024) introduce the Evolvable Learnware Specification with Index (ELSI), which increasingly improves model capability character- ization as the number of learnware increases. Tan et al. (2023) 2024a) propose learnware search and reuse algorithms leverage learnwares from heterogeneous feature spaces, Guo et al. (2023) try to utilize learnwares from heterogeneous label spaces. Xie et al. (2023) explore the methods for identifying learnware efficiently by constructing anchor learnwares. In order to facilitate research

---

Published as a conference paper at ICLR 2026
on learnware, Beimingwu learnware research platform has been released (Tan et al. 2024b) and have further contributed to related research in this area. [Ta et al. (2025) first apply the learnware paradigm to SLMs with parameter vector specification, demonstrating performance superior LLMs across specialized tasks. Our work contributes to Learnware by presenting a comprehensive study of PAVE specification and enhancing learnware identification for deep models by characterizing model capability in the parameter space.
Transferability. The transferability of pre-trained model (PTM) is critical for its effective adaptation from a source to a target task. Many methods estimate transferability via a forward pass on the target task. leveraging different metries to approximate performance. Notable approaches include NCE (Taan et al. 2019). LEEP (Nguyen et al. 2020). LogME (You et al. 2021), PACTran (Ding et al. 2022). , and TransRate (Huang et al. 2022). , which estimate negative conditional entropy, log expectation, marginalized likelihood, PAC- PAC-Bayesian bounds, and mutual information, respectively. While these methods reduce the need for full fine-tuning. they incur high computational costs due to the complexity of modern PTMs and are limited by PTM-specific embeddings. Additionally, transferability depends on the relatedness between pre-training and downstream tasks (Ben-David and Schuller 2003 Wang et al.. 2019b; Lu et al.] 202 I Lin et al. 2024). Various approaches assess graphs (Dwivedi) and Roig. 2019), optimal transport between label spaces (Lu et al. 2022), and representation-level similarities (Dwivedi et al.) 2022). Recently, Model Spider (Zhang et al. 2023) leveraged historical data learn task and PTM similarities, enhancing model selection. However, these methods generally require direct access to user data, and computational constraints hinder per-mo per-model evaluations on user devices. These challenges can be addressed naturally based on specifications learnware paradigm, advancing transferability research by leveraging broader real-worke model platform.
task relatedness, including full fine-tuming (Zamir et al., 2018 ) example-base
Parameter Vector.
The change in model parameters, which we term the parameter vector, has been explored in gradient matching and, more recently, as task vectors. Gradient matching is used for distilling large datasets into smaller synthetic ones (Zhao et al. 2021 Cazenavette et al., 2022) or domain generalization (Shi et al.
(022). More recently. task vectors have been increasingly used to merge models for multi-task learning (Ihaacoo et al. 2023 Ortiz-Jiménez et al. 2023: Tang et al. 2024a Yadav et al. [2023] Yang et al. 2024. Xu et al. 2025a Zhang and Zhou| 2025 Yoshida et al.
2025 Ji et al., 2025] Gargiulo et al.) 2025). Besides, Li et al. (2025) analyze the generalization
ability of task vectors and demonstrate the feasibility of low-rank approximation, while RRialli et al.)
(2025) investigate how to use task vectors to transfer fine-tuned knowledge to new backbones without retraining. Further studies have proposed advanced merging strategies avoid parameter conflicts (Lai et al. (025)) or measured shifts in internal activations for model selection (Xu et al., (0256). This research primarily focuses on combining model capabilities. In contrast, our work explores a different direction: we investigate the relationship between parameter vector and model capability similarities, not for merging models, but for identifying the best models for a specific user task.
6 CONCLUSION AND DISCUSSIONS
In this paper. we provide a comprehensive study of PAVE specifications, which sketch model ca- pability and task requirements with parameter vectors to efficiently capture reusability relations, to identify and reuse high-quality learnware that effectively solves the user task. Theoretical analysis and extensive experiments verify its effectiveness, and shed light on its underlying mechanism. The method outperforms user-fine-tuned pre-trained models, exhibiting remarkable generalization capabilities and successfully solving tasks beyond their original functionality. Furthermore, the ap- proach demonstrates robustness by selecting the good learwwares even from repositories of corrupted quality. Finally, we verify that approximating parameter vectors in the low-rank space enhances both storage and computational efficiency with minimal impact on performance. These findings show the parameter vector's 's potential in the learnware paradigm, suggesting future work on efficient generation for resource-constrainge settings (Tang et al. 2024b).
LCKNOWIED
This work was supported by Jiangsu Science Foundation Leading-edke Technology Program (BK20232003) and Collaborative Innovation Center of Novel Software Technology and Industrial-
10

---

Published as a conference paper at ICLR 2026
ization. Zhi-Hao Tan was supported by JiangsuSF (BK20251215) and the Postdoctoral Fellowship Program of CPSF under Grant Number G2B20250396. We thank Tian-Zuo Wang, Han-Jia Ye, Lan-Zhe Guo, Peng Tan, Jia-Wa Wei Shan, Hao- Yi Lei and Xin- Yu Zhang for their helpful discussions. We are also grateful to the anonymous reviewers for their helpful comments.
REFERENCES
Anthropic. Introducing the model context
odel-coo wwwtootttttt--oe -ooettopppoooooo 2024.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li. Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang, Jian Yang. Shusheng Yang. Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv:2309. 12009.1666 16609, 2023.
Shai Ben- David and Reba Schuller. Exploiting task relatedness for multiple task learning. In Proceed- ings of the 16th Annual Conference on Learning Theory and 7th Kernel Workshop (COLT/Kemnl)), pages 567-580, 2003.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4749-4758, 2022.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 17th Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 4171 4186, 2019.
Nan Ding, Xi Chen, Tomer Levinboim, Soravit Changpinyo, and Radu Soricut. Pactran: Pac-bayesian metrics for estimating the transferability of pretrained models to classification tasks. In Proceedings of the 17th European Conference on Computer Vision Computer Vision (ECCV), pages 252 268. 2022.
Pedro Domingos. Every model learned by gradient descent is approximately a kernel machine. arXiv:20/2.00152. 2020.
Kshitij Dwivedi and Gemma Roig. Representation similarity analysis for efficient task taxonomy & transfer learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12387-12396, 2019.
Kshitij Dwivedi. Jiahui Huang. Radoslaw Martin Cichy, and Gemma Roig. Duality diagram similarity: A generic framework for initialization selection in task transfer learning. In Proceedings of the 16th European Conference on Computer Vision Computer Vision (ECCV), pages 497-513, 2022.
Antonio Andrea Gargiulo. Donato Crisostomi, Maria Sofia Bucarelli, Simone Scardapane, Fabrizio Silvestri, and Emanuele Rodola. Task singular vectors: Reducing task interference in model merging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18695-18705, 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and et al. The llama herd of models. arXiv:2407.2178. 2024.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf. and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
Lan-Zhe Guo, Zhi Zhou, Yu-Feng Li, and Zhi-Hua Zhou. Identifying useful learnwares for heteroge- neous label spaces. In Proceedings of the 40th International Conference on Machine Learning (ICMI), pages 12122-12131, 2023.
11

---

Published as a conference paper at ICLR 2026
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The 10th International Conference Leurning Representations (ICLR), 2022.
Long-Kai Huang, Junzhou Huang, Yu Rong, Qiang Yang, and Ying Wei. Frustratingly easy trans- ferability estimation. In Proceedings of the 39th International Conference on Machine Learning (ICML), pages 9201-9225, 2022.
Gabriel Ilharco, Marco Túlio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In The 11th International Conference on Learning Representations (ICLR), 2023.
Ruochen Jin, Bojian Hou, Jiancong Xiao, Weijie J Su, and Li Shen. Fine-tuning attention modules only: Enhancing weight disentanglement in task arithmetic. In The 13th International Conference on Learning Representations (ICLR), 2025.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In The 3rd International Conference on Learning Representations (ICLR), 2015.
Kunfeng Lai, Zhenheng Tang, Xinglin Pan, Peijie Dong, Xiang Liu, Haolan Chen, Li Shen, Bo Li, and Xiaowen Chu. Mediator: Memory--fficien IIm merging with less parameter conflicts and uncertainty based routing. urXiv:2502.04411, 2025.
Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In Advances in Neural Information Processing Systems 32 (NeurIPS), pages 8570-8581, 2019.
Hao-Yi Lei, Zhi-Hao Tan, and Zhi-Hua Zhou. On the ability of developers' training data preservation of learnware. In Advances in Neural Information Processing Systems 37 (NeurIPS), pages 36471- 36513, 2024.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for pframeter-cfficient prompt tuning. In Proceedings of the 26th Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3045-3059, 2021.
Hongkang Li, Yihua Zhang. Shuai Zhang, Meng Wang. Sijia Liu, and Pin- Yu Chen. When is task vector provably effective for model editing? a generalization analysis of nonlinear transformers. arXiv:2504. /0957, 2025.
Shuxia Lin, Miaosen Zhang, Ruiming Chen, Xu Yang, Qiufeng Wang, and Xin Geng. Linearly decomposing and recomposing vision transformers for diverse-scale models, Advances in Neural Information Processing Systems, 07:33188-33214 2024.
Jian-Dong Liu, Zhi-Ho Shi Tan, and Zhi-Hua Zhou. Towards making learnware specification and market evolvable. In Proceedings of the 38th AAAI Conference on Artificial Intelligence (AAAI), pages 13909-13917, 2024.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. 190. 2019.
12

---

Published as a conference paper at ICLR 2026
Su Lu,, Han-Jia Ye, and De-Chuan Zhan. Tailoring embedding function to heterogeneous few-shot tasks by global and local feature adaptors. In Proceedings of the 35th AAAI Conference on Artificial Intelligence (AAAI), pages 8776-8788 2021.
Su Lu, Han-Jia Ye, and De-Chuan Zhan. Selective cross-task distillation. arXi::2204. 2022.
Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timmit Gebru. Model cards for model reporting. In Proceedings of the 1st Conference Fairness, Accountability, and Transparency (FAT*), pages 220-229, 2019.
Mohamad Amin Mohamadi, Wonho Bae, and Danica J. Sutherland. fast, well-founded approxima- tion to the empirical neural tangent kernel. In Proceedings of the 40th International Conference on Machine Learning (ICMI., pages 25061-25081, 2023.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning. The MIT Press, 2012.
Cuong V. Nguyen, Tal Hassner, Matthias W. Seeger, and Cédric Archambeau. LEEP: A new measure to evaluate transferability of learned representations. In Proceedings of the 37th International Conference on Machine Learning (ICML), pages 7294-7305, 2020.
OpenAI. Chatgpt (gpt-3.5), 2023. Large language model.
Guillermo Ortiz-Jiménez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. In Advances in Neural Information Processing Systems 36 (NeurIPS), pages 66727-66754, 2023.
Ankit Pal, Pasquale Minervini, Andreas Geert Motzfeldt, Aryo Pradipta Gema, and Beatrice Alex. co/ http/pacee//peenifeessee
Open medical 11m leaderboard. https://hugginggttt mm leaderboard, 2024.
nceai/ cecee__mediiall
Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. Advances Neural Information Processing Systems 37 (NeurIPS), pages 126544-1222 26544-1225655, 2024.
Jeff M. Phillips and Wai Ming Tai. Near-optimal coresets of kernel density estimates. Discrete Comput. Geom,, 63(4):867 887, June 2020.
Yujia Qin, Shi Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toollim: Facilitating large language models to master 16000+ real-world apis. In The 12th International Conference on Learning Representations (ICLR), 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML), pages $748-8763, 2021.
Timothy RC Read and Noel AC Cressie. Goodness-of-f statistics for discrete multivariate data, Springer Science Business Media, 2012.
Filippo Rinaldi, Giacomo Capitani, Lorenzo Bonicelli, Donato Crisostomi, Federico Bolelli, ELISA FICARRA, Emanuele Rodolà, Simone Calderara, and Angelo Porrello. Update your transformer to the latest release: Re-basin of task vectors. In Proceedings of the 42nd International Conference on Machine Learning (ICML), pages 51688-51700 2025.
Bernhard Schölkopf and Alexander Johannes Smola. Learning with kernels: Support vector machines, regularization, optimization, and beyond. MIT press, 2002.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems 36 (NeurIPS), pages 381 3800, 2023.
13

---

Published as a conference paper at ICLR 2026
Yuge Shi, Jeffrey Seely, Philip H. S. Torr, Siddharth Narayanaswamy, Awni Y. Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. In The 10th International Conference Learning Representations (ICLR), 2022.
Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. 2rXiv:2212.131133 2022.
Stallkamp. M. Schlipsing, J. Salmen, and C. Igel. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural Networks, 32:323-332, 2012.
Peng Tan, Zhi-Hao Tan, Yuan Jiang, and Zhi-Hua Zhou. Handling learnwares developed from heterogeneous feature spaces without auxiliary data. In Proceedings of the 32th International Joint Conference on Artificial Intelligence (IJCAI), pages 4235-4244 2023.
Peng Tan, Hai-Tian Liu, Hai-Hao Tan, and Zhi-Hua Zhou. Handling learnwares from heterogenenus feature spaces with explicit label exploitation. In Advances in Neural Information Processing Systems 37 (NeurIPS), pages 12767-12795, 2024a.
Zhi-Hao Tan, Jian-Dong Liu, Xiao-Dong Bi, Peng Tan, Qin-Cheng Zheng, Hai-Tian Liu, Yi Xie, Kiao-Chuan Zou, Yang Yu, and Zhi-Hua Zhou. Beimingwu: A learnware dock system. In Proceedings of the 30th ACM SIGKDD Conference Knowledge Discovery and Data Mining (KDD), pages 5773-5782, 2024b.
Zhi-Hao Tan, Hao Yu Shi, Zi-Xuan Chen, and Yuan Jiang. Learnware reduced kernel mean embedding specification based on neural tangent kernel. Chinese Journal of Computers, 1236-11232- 2024c.
Zhi-Hao Tan, Zi-Chen Zhao, Hao-Yu Shi, Xin- Yu Zhang, Peng Tan, Yang Yu, and Zhi-Hua Zhou. Learnaare of language models: Specialized small language models can big. arXiv:2505. 2025.
Anke Tang, Li Shen, Yong Luo, Yibing Zhan, Han Hu, Bo Du, Yixin Chen, and Dacheng Tao. Parameter-efficien multi-task model fusion with partial linearization. The 12th International Conference on Learning Representations (ICLR), 2024a.
Ningyuan Tang, Minghao Fu, Ke Zhu, and Jianxin Wu. Low-rank attention side-tuning for parameter- efficient fine-tuning. WrXiv:2402.040000 2024b.
Anh Tuan Tran, Cuong V. Nguyen, and Tal Hassner. Transferability and hardness of supervised classification tasks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1395-1405, 2019.
Joel A. Tropp. User-friendly tail bounds for sums of random matrices. Foundations of Computational Mathematics, 12(4):389-434, 2012.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In The 7th International Conference on Learning Representations (ICLR), 2019a.
Zirui Wang, Zihang Dai, Barnabás Póczos, and Jaime G. Carbonell. Characterizing and avoiding negative transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11293 115 11302, 2019b.
Heather Woltman, Andrea Feldstain, J Christine MacKay, and Meredith Rocchi, An introduction to hierarchical linear modeling. Tutorials in Quantitative Methods for Psychology, 8(1):52-69, 2012.
Xi-Zhu Wu, Wenkai Xu, Song Liu, and Zhi-Hua Zhou. Model reuse with reduced kernel mean embedding specification. IEEE Transactions on Knowledge and Data Engineering (TKDE), 35(1): 699-710, 2023.
Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large-scale scene recognition from abbey to zoo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3485-3492, 2010.
14

---

Published as a conference paper at ICLR 2026
Yi Xie, Zhi-Hao Tan, Yuan Jiang, and Zhi-Hua Zhou. Identifying helpful learnwares without examining the whole market. In Proceedings of the 26th European Conference on Artificial Intelligence (ECAI), pages 2752-2759, 2023.
Jing Xu, Jiazheng Li, and Jingzhao Zhang. Scalable model merging with progressive layer-wise distillation. In Proceedings of the 42nd International Conference on Machine Learning (ICML), pages 69451-69477 2025a.
Zhiqiu Xu, Amish Sethi, Mayur Naik, and Ser-Nam Lim. Delta activations: A representation for finetuned large language models. wrXiv:2509.04444 2025b.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A. Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. In Advances in Neural Information Processing Systems 36 (NeurIPS), pages 7093-7115, 2023.
Enneng Yang, Zhenyi Wang. Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging; Adaptive model merging for multi-task learning. In The 12th International Conference on Learning Representations (ICLR), 2024.
Kotaro Yoshida, Yuji Naraki, Takafumi Horie, Ryosuke Yamaki, Ryotaro Shimizu, Yuki Saito, Julian McAuley, and Hiroki Naganuma. Mastering task arithmetic: SitauSjp key indicator for weight disentanglement. The 13th International Conference on Learning Representations (ICLR), 2025.
Kaichao You, Yong Liu, Mingsheng Long, and Jianmin Wang. Logme: practical assessment of pre-trained models for transfer learning. In Proceedings of the 38th International Conference Machine Learning (ICML), pages 12133-12143, 2021.
Amir R. Zamir, Alexander Sax, William B. Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 3712-3722, 2018.
Haobo Zhang and Jiayu Zhou. Unraveling lora interference: Orthogonal subspaces for robust model merging. In Proceedings of the 63rd Anmual Meeting of the Association for Computational Linguistics (ACL.), pages 26459-26472, 2025.
Yi-Kai Zhang, Ting-Ji Huang, Yao-Xiang Ding, De-Chuan Zhan, and Han-Jia Ye. Model spider: Learning to rank pre-trained models efficiently. In Advances in Neural Information Processing Systems 36 (NeurIPS), pages 16692-1379, 2023.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In The 9th International Conference on Learning Representations (ICLR), 2021.
Zhi-Hua Zhou. Learwware: the future of machine learning. Frontiers of Computer Science, 10(4): 589-590, 2016.
Zhi-Hua Zhou and Zhi-Hao Tan. Learnware: small models do big. Science China Information Sciences, 67(1):11210,, 2024.
15

---

Published as a conference paper at ICLR 2026
A DETAILS ON THE CONNECTION BETWEEN PAVE AND RKME
In this section, we present a detailed discussion on why the cosine similarity between parameter vectors effectively captures the alignment between model capabilities and task requirements, as illustrated in Equation (2). Furthermore, we demonstrate that this similarity corresponds to an implicit kernel function grounded in task semantics and the empirical Neural Tangent Kernel (NTK).
The model vector Th with respect to the training task (De, Le) and model h can be obtained by solving this optimization formula:
The arg min
Σ
Clge f(x, 0o + T), h(x)).
(11)
(x.))ED
When we optimize it with gradient descent, the model vector we obtain is the accumulation of the gradients of the loss function L(g: o f(x, 0o + T), h(x)) with respect т. Accordingly, the gradient of the loss function on the sample (x,, y:) is
ViCe 9e) V.Li(gi o f(x, 8u + T), h(xx)) = Df V.9t Vef(x,, 0o) R (12)
where y/₁ g fxx, Gn 7) and R is the weight space. Note that the second equality in Equation 12 holds the fine-tuning process satisfies the neural tangent kernel behaviour (Lee et al. 2019).
And, the model vector Ty, essentially the accumulation of gradients
Σ
V.L(s, h(x.)) = Σ V Vof(x, 0o).
(13)
(x.0)ED.
(X.))ED,
This also holds for the task vector Tu with the user task (D,, L.) and the true label Then, the inner product between model vector and task vector is defined
(Thy Σ
Σ
V.Lalhh M(x;)) Vj)
(x.. (E).
(14)
Σ
Σ
V V 6o) [V 00) Vo) Iu D. Vy
(xD (X)
We can define the similarity in the summation equation kernel function k,,c, that k. h(x;), U;) = V (v). ((x,)) Cu/l://// Us)
(15)
(V b/ V =9)) (x;, xj) Lu
R.
Note that Ky xz) is the empirical neural tangent kernel Mohamadi et al. 2023
(x,, xy) Vof (x,, 0o) VJ fxx, 0o) 1xo, where O denotes the dimension of the feature space Z.
(16)
To understand this kernel function k (xi, h(xx), xj, Ui) even further, we can decompose it into two parts (x, Xÿ) and K Ke.g (h(x,)) yÿ) with some trace tricks:
(x, h(x;), 2;) -Tr
(V (. K xx) (V (u
TT
K k x3) (V So V=gu) (V
Tr
K , (x., X,y V. Iu ,/ Lu Vo V=g(]
(17)
1 x3) RT. (l(x;), no>]
0
Ky x1) (1(x))))))) ui)
16

---

Published as a conference paper at ICLR 2026
Therefore, the kernel function ,,,Ccy h(xx), Xj. Xz.3y) can be viewed as a kernel function implic- itly constructed based on the task type and semantics. consists of two parts, Ky(x,, Xj) and (k.,(((X,)) yi), which correspond to the similarity between the inputs x of the samples and the importance of different pairs of similarities for solving the task respectively, Specifically,
Ky,xi, x,) is matrix of shape O O as
(x))( voli(x)) (Vo)2(xi), Voli(x,))
(V)l((x)) Valo(x)))
(Vo/2(xi), vofo(x;))
xy(x;,,;))
(18)
((Vooox)), Vofi(x,))
(Vofo(x;) Vofo(x;))
where fo(r) denotes the o-th output of the neural network f(x, 0) E R and O denotes the dimension of the feature space Z. The elements
[x xy)re (x,), Vefa(x,)),, for any r, E [O]
(19)
characterize the similarity between x; and x, from the perspective of pairs of features, O pairs in total, through empirical neural tangent kernel.
Va 2y The gradient V of L
is a vector such that
/// C V.
L==1
Dl
Oge d2i
ac Öge
Σ Oga. 8zo
D k=1
8C1 ag..
E IR0
(20)
where f(x,) E RD. And the gradient components
Oc Ögk
VV LV
for any E [0].
(21)
0gn Ozo"
1=1
describe the relationship between the features 2o and solving the task, with some features being more helpful the larger they are and others the opposite.
Finally, for the matrix Kc. (h(x))) Wi) = (V Le 2
(V
vs Lu V Venu,
Vi k=
DC., 8g.. Back De1
Ke.9
D'al
aC. Og..k Bya.k 821
Σ k= Hya.k Bso
ac. Og
E Roxo
8g..k V4 k=
Oc,
Bark Azo
(22)
encourages similarity between pairs of features sharing the same relationship with solving the task and penalizes the similarity of the opposite.
Therefore, the kernel function simultaneously incorporates input data x, model predictions f(x), and task semantics. The greater the consistency in the input-output relationship between two samples, the higher the kernel function value. For the model vector Ty, and the task vector Tu. we have
(Th, Tu)
Σ
Σ
kg,c., h(x;), Xx, My).
(23)
(N (M)
To eliminate the bias introduced by dataset size and optimization trajectory length, we normalize T and T, unit vectors. Consequently, (Th, Tu) precisely corresponds cos Thi Tw) in Equation (
17

---

Published as a conference paper at ICLR 2026
B DETAILS ON THE APPROXIMATION OF PARAMETER VECTOR SIMILARITY
In this section, we establish an error bound for approximating the similarity between parameter vectors, stated in Theorem and provide complete proof.
Before delving into the theorems and their proofs, we first define the necessary notation. Consider a pre-trained model, f(x. 8g 7) which consists of L weight matrices. Without loss of generality, assume that each weight matrix has the shape m x n. The corresponding approximated parameter vector is expressed
x F E AWW AW AW41 = (B'AA) (B²A²) ...
(B"A4) BA,
where A is initialized randomly, and B is initialized with zeros. To clarify, the dimensions of the relevant matrices are as follows:
AW Ruxn A E B B' E'er, for any E [A].
Accordingly, the parameter vector T E Rmx(nL) the concatenation of matrices { (W') and B E Rmx(rL) is the concatenation of matrices (B'}. And we obtain the parameter vector via
B arg min
Σ
C(go f(x, 9o BA), h(x)).
(24)
(X.))DD
For simplicity, in this paper, implicitly flatten matrices from the matrix space to the vector space when computing the inner product and cosine similarity between them. Then we can begin our proof.
Theorem 4. In order compute the cosine similarity between the parameter vectors 7 BA obtained by the low-rank approximation following Equation (24). the cosine similarity between B only is sufficient approximate it efficiently. Specifically, we can prove that for any 0,
-2c + min
cos(BI, B2), 1-c cos(BI, B2) cos(Ti, 72)
I+c
(25)
1 cos(BB B2). I+e cos cos B2) 2e,
S max
with probability of 8, when nL = O0 (n(g)) and Inr S 36 p for each I E [N] and the matrices
2m
A A~U(-V3s//n + +/3s/n)
element-wisely with constant factor which is Kaiming initialization (He et al. 2015) with proper s.
Proof. Our proof is divided into three main steps. First, we demonstrate that the random matrix A' (A")) approximates scalar matrix 8 8I xr for any 1 E [T]. Next, we show that the squared norm of B', denoted as |B'] |2, is approximately equal to | A 1 for any E [T.]. Finally, we establish that cos B, B2) closely approximates cos(#; f1, +2) with high probability.
Step We begin by analyzing the random matrix A' It can be decomposed into the sum of independent random matrices follows:
A'(A'T=
Σ aaa E Rr
*E(m)
where A
[a1: ........ an is element- wise uniformly distributed as (4(-/3s/10 1/sssnnn/////..... 3s/n). /3//n) In expectation, the random matrix satisfies:
B[a;a,] $ Txxrs and sI =I<E[A(AA)]] sI.
(26)
n
Moreover, for any vector v, holds that:
/35 S.T
5 u (a,a) y (o aT)) ||011||a,|1
v.
(27)
n
Thus, the spectrum of a is bounded, and we have:
3s I, almost surely.
0 3 a,a < n
(28)
Next, apply tail bound for the sum of random matrices, stated in Lemmal
(ropp
201 2
18

---

Published as a conference paper at ICLR 2026
Lemma 5. Let X1. ..... N be independent real symmetric matrices with 0 X; R f
some R E R. Let pmin 3 Ei=1 B I Mmax I For any E
2 Pmax
Pr
Amax
Σ x
2 (1 + e)
repp
3R
i=l
2 Munin
Pr
Amin
Σx1 X
S (1
cmin
rrexp
2R
By choosing R and /mmx = Pmin
8, we obtain:
e
Pr (1- A
(') I (1 + c)sI 2 1 -1-2rexp
(29)
9s
any 6 0. This shows that A is approximately equal to si with high probability
1 - 2r exp(- e'n) for any E [Z].
Step 2. We gonna show that the squared norm of B', denoted as [B is approximately equal to B a for any E [L]. Using the decomposition |B'A 1|? we have:
B = (B'A B'A))
Σ by (A' iE(ml
(30)
From the previous result, it follows that:
(1 - c)
Σ
b, b, < BB (1++) Σ 6,6b..
(31)
kE(n]
<Ein
or equivalently we have:
(1 - [ < | s (1 (1+e)||B')
(32)
with high probability of it follows that
2r expepp(=2
(-m) 9e
2m) By moving r into the exp function and assuming that
2m In 1/
t 2m
Pr B
B | 2 f] <xe
lur
9s B||
(33)
t2m
<2exp
12s (-
B
||
It implies that |B'A - |B'I22 is sub sub-gaussian random variable with parameter /6s/m |B'l
Lemma 6. Suppose that the random variables {X+)-1 are independent and [[X]] p4, and X, is tub--Gussiaan random variable with parameter oy Then for all t 2 0, we have
n (X+ vi) 21 < 2exp
+2
Pr
Σ
With Lemma layers BA
Mohri et al. 2012). the sum of sub-gaussian random variables B.A, in all N satisfies
Pm
Pr[||| 12 2- B 2 1]
2exp
(34)
12s Σ/_ |B
In general, for real numbers Ti, , Tna we have the inequality:
(I1 + D2 + + xx,)² 2 2 + 2 + ... + x < (x1 + x2 + ... xx)
(35)
19

---

Published as a conference paper at ICLR 2026
where the first inequality takes equality when 2; Σ. 24 and the second inequality takes equality when some 24 Ei Jy.- In practice, find that |B|| is nearly uniformly contributed by ||B') in each layer. Therefore, we can assume that there exists a constant с < L such that
B IBI|
[B +... |B"|| < 7 (1I
C
| +.. | 1B1
L |||
(36)
Finally, summing across all layers, the L2 norm of the task vector satisfies:
(1 (1-c)
[BA] < (1 +c]
(37)
with high probability of 2 nL/12se
Step 3. We will establish that cos(BB, B2) closely approximates cos F2) with high probability.
First, by substituting B B1 -B and B = B1 + B2 into Equation 337 we obtain the following inequalities:
(1 c) B1 B₂ll² < |BA B 2 S (1 (1+6) B1 BBlla
(38)
(1 - BB + B₂|l2 < B + B 2 S (1 (1++) B B₂|12
Next, bound the inner product, note that
13A,BBA4 1 B |B,A- - B2Al²) s (B,,
(I 2 [B
(39)
and for the other direction,
B2A) 1 [IB |B - -B2)l?) 2 (BI, B2) (I 12 +
[Ba
12)l))
(40)
Now, consider the case (BIA, B2A)
0. To bound the cosine similarity, we have
A B2A)
(Bi B2)
B 2| [ 12
(41)
B& A
(1 - <) B B2ll
2 (1 - <) B B
and similarly,
B2A, B A B
BI,
B 1 2 + B
(42)
(1+ B B2ll
2 (1++ + e) B B2
For the case (B, B,A) < O, we establish similar bounds:
B B2A)
(B, B
B || + B 1
(43)
A A BAA
(1+O
B
B2l
2 2 <) B
B
and,
BJA,B2A)
(B, B2)
B
|| + B2
(44)
B A B
B2
2 2
B
B2l
By combining these bounds, and letting A() Bill. |Ba]] upper bound:
Bi
Ball B.
we derive the following
B2
1
1
cos B A, B,A) max
1- cos(BB, B1, B2),
cos CBs B2)
1+e
max
A B [B2l),
A( [
2(1
2(1 +
1
max
cos(B1, B2),
cos cos(B,, B2) 2(1 e)
A( A [B2]))
1+c
e)
(45)
The lower bound can be proved in similar manner.
20

---

Published as a conference paper at ICLR 2026
In summary, establish that
min 1
Cos(BI,, B2),
I-e cos cos(BI,
A [2))
(+E
2(1 +
I cos(BB
B2A)
(46)
< max
---e cos Cos B1, B2),
1
cos BI, B2)
+
c 2(1
A B BBll)
1te
Since we only care about the direction of the parameter vector, without loss of generality we let B
B3
and S 0.5, the result simplifies to
cos cos(B1, B2), 1 l+e
1
-2 min
cos B1, B2) cos
(47)
cos Cos BB) 2e,
max
cos (B, B2); 1+6
with high probability of 1 nL
21

---

Published as a conference paper at ICLR 2026
C EXPERIMENTAL SETTINGS
C.I DATASET DESCRIPTIONS
Classic NLP. The classic natural language processing datasets we use are mainly from the GLUE benchmark datasets. Specific information about these datasets is listed below:
CoLA (Corpus of Linguistic Acceptability): A dataset for grammatical acceptability classi- fication, where each English sentence labeled either grammatically correct incorrect, sourced from linguistic literature.
MNLI (Multi-Genre Natural Language Inference): A natural language inference dataset where models determine if hypothesis entails, contradicts, or is neutral with respect to given premise. It includes diverse genres of text.
MRPC (Microsoft Research Paraphrase Corpus): A dataset for paraphrase identification, containing sentence pairs annotated for semantic equivalence, useful for measuring sentence similarity.
QNLI (Question Natural Language Inference): dataset derived from SQuAD, reformu- lated as a binary classification task to determine if a sentence contains the answer to a given question.
QQP (Quora Question Pairs): A paraphrase detection dataset consisting of question pairs from Quora, labeled as either duplicate (same meaning) or not.
RTE (Recognizing Textual Entailment): natural language inference dataset where mod- els classify whether given premise entails hypothesis, sourced from various textual entailment challenges.
SNLI (Stanford Natural Language Inference Corpus): A large-scale dataset for natural language inference, with sentence pairs labeled as entailment, contradiction, neutral.
SST-2 (Stanford Sentiment Treebank Binary Classification): sentiment analysis dataset where sentences from movie reviews are labeled as either positive or negative.
STS-B (Semantic Textual Similarity Benchmark): dataset for evaluating the semantic similarity between sentence pairs, with similarity scores ranging from to
WNLI (Winograd Natural Language Inference): A challenging dataset derived from the Winograd Schema Challenge, requiring commonsense reasoning to resolve ambiguous pronoun references.
We also evaluate our method on some other natural language processing datasets:
(Customer Reviews Dataset): sentiment analysis dataset containing customer reviews from various products, labeled as positive or negative.
mpqa (Multi-Perspectivve Question Answering Opinion Corpus): A dataset focused on opinion mining, containing news articles annotated for subjective expressions and sentiment.
mr (Movie Reviews Dataset): A dataset for sentiment analysis, where movie reviews are labeled either positive or negative.
subj (Subjectivity Dataset): dataset for classifying sentences as subjective (expressing opinions) or objective (stating factual information).
trec (Text REtrieval Conference Question Classification Dataset): A dataset for question classification, where questions are categorized into different types such as location, person, number, and more.
Computer vision. We evaluated our method several computer vision datasets. These datasets presented as follows
A widely used image classification dataset containing 60,000 small (32x32) color images across 10 categories, including airplanes, cars, and animals.
CIFAR-10:
22

---

Published as a conference paper at ICLR 2026
RESISC45 (Remote Sensing Image Scene Classification): A dataset with 31,500 high- resolution aerial images categorized into 45 scene types (e.g., airport, forest, residential areas), designed for remote sensing applications.
MNIST (Modified National Institute of Standards and Technology Database): A classic dataset 70,000 grayscale handwritten digits (0-9) 28x28 resolution, commonly used for benchmarking digit classification models.
SUN397 (Scene UNderstanding Dataset): A large dataset with 108,754 images covering 397 different scene categories, used for scene recognition and understanding tasks.
SVHN (Street View House Numbers): A dataset containing over 600,000 real-world images of house numbers from Google Street View, designed for digit recognition in natural scenes.
GTSRB (German Traffic Sign Recognition Benchmark): A dataset of over 50,000 images containing 43 different German traffic sign categories, used for traffic sign classification tasks. Food-101: large dataset containing 101,000 images across 101 different food categories, such sushi, pizza, and spaghetti, used for food recognition.
EuroSAT: A satellite image dataset with 27,000 RGB images categorized into 10 land use classes, such residential areas, forests, and industrial zones, based on Sentinel-2 satellite data.
DTD (Describable Textures Dataset): A dataset containing 5,640 texture images categorized into 47 different texture descriptions, such "cracked" "striped", used for material recognition.
Stanford Cars (Cars Dataset): A dataset with 16, 185185 images of 196 different car models, categorized by make, model, and year, used for fine-grained vehicle classification.
CIFAR-100: An extension of CIFAR-10, containing 60,000 images classified into 100 fine-grained categories, grouped into 20 superclasses (e.g., mammals, vehicles, insects).
STL-10: dataset similar CIFAR-I0 but with larger (96x96) images and fewer labeled examples, designed for semi-supervised learning tasks.
Medical LLM. We construct learnwares based on the following supervised fine-tuning (STF) datasets for solving the medical LLM benchmarks:
The AlpaCare-Medlnsruuttt dataset, available under Aivita/AlpaCare MedInstruct-52k. consists of 52,000 instruction-basee medical dialogues. is designed to improve medical language understanding and response generation, focusing on clinical inquiries and patient care instructions.
AlpaCare:
ChatDoctor: The ChatDoctor- HealthCareMagicc 100k dataset. referenced as 100,, contains 100,000 medical dialogue inter- actions. The dataset is collected from HealthCareMagii, providing extensive doctor-patient conversational data for training medical conversational Al models.
medalpaca cleaned: This dataset combines multiple medical text sources, including: This dataset combines multiple medical text sources, including:
medical meadow wikidoc: meadow_wikidoo: Medical encyclopedic knowledge from WikiDoc.
Structured medical knowledge in flashcard
format.
medical medical_meadoww
Patient-oriented medical content ex-
tracted from WikiDoc.
medical meadow Beceddowppbmed_caae causal: Causal medical information extracted from PubMed.
medical _ewadlwweeawaaaaaaa soaudow_mediqa. Medical question- dataset designed for clinical inquiries.
General medical guidance and advice for patient
care.
medical flasheards The medical meadow medical flashcards dataset provides structured medical knowledge a flashcard format, aiding in medical education and knowledge retention.
medical_faasccad
23

---

Published as a conference paper at ICLR 2026
medqa train The medicallmeedoow Lecooa_aaaaaaaae dataset focuses multiple-choice question answering in the medical domain. contains curated medical questions designed to assess and enhance medical reasoning capabilities.
pubmed causal The medical_ meudow_ wwwddow_puueedddd_emeeee causal dataset is derived from PubMed, focusing on causal relationships in medical research literature. It useful for developing models that require causal inference in healthcare.
medmeqa train The medmoqa instruct dataset (chenhaodev perooddvv//eedeee /tt////aaaaaaaee/eeo/ee//eaaaa iew__nnstuuuee is a large-scale medical uuestion-nnswerigg dataset. contains multiple-choice questions covering various medical subjects, aiding in Al-driven medical education and assessment.
medqa train & pubmed causal This dataset combination includes both medi- cal meadow medqa and medical meadow pocooedd_pp____eeee causal, integrating medical multiple- choice question answering with causal inference from PubMed literature.
AlpaCare & ChatDoctor This dataset set includes:
linhDuonggccaaddottr-Skk A small-scale medical conversational dataset. ChatDoctor MeetrorrrretttthhttCeaeaa------e 100k: Large-scale medical dialogue dataset. AlpaCare-Mednnttruct Instruction-baset medical response dataset.
medalpaca cleaned & AlpaCare & ChatDoctor This dataset aggregation merges medal- paca_ cerru_ceddd,,, AppaCare-eednstruut-ae and ChalDootoo-HeelliCCaaMMaag-- offer- ing a comprehensive training set
These evaluation datasets are from the Open Medical- Leaderboard et al. 2024). and they have been used to evaluate the performance of our method for identifying large language models.
MedMCQA: A large-scale Multiple-Choice Question Answering dataset designed to ad- dress real-world medical entrance exam questions. It comprises over 194,000 high-quality multiple-choie questions covering 2,400 healthcare topics and 21 medical subjects, col- lected from AIIMS and NEET PG entrance exams. Each sample includes a question, correct answer(s), and other options, requiring deep language understanding and reasoning across a wide range of medical subjects and topics.
MedQA 4-Options: benchmark dataset derived from medical licensing exam questions, formatted as multiple-choice questions with four answer options. It is used to evaluate the performance of language models in understanding and answering medical questions accurately.
Anatomy: A subset of the MMLU (Massive Multitask Language Understanding) benchmark focusing on anatomical knowledge. assesses model's 's understanding of human anatomy through various question formats, including multiple-choice questions.
Clinical Knowledge: Part of the MMLU benchmark, this subset evaluates a model's s grasp of clinical concepts and practices. It includes questions that test understanding of clinical scenarios, diagnoses, and treatments.
College Biology: Another subset of the MMLU benchmark, focusing on biology at the college level. It assesses knowledge in areas such genetics, ecology, and cellular biology through various question types.
component of the MMLU benchmark that tests knowledge in medical subjects typically covered in college-level courses. It includes questions on topics like pharmacology, pathology, and physiology.
College Medicine:
Medical Genetics: This subset of the MMLU benchmark focuses on genetics within the medical field. It evaluates understanding of genetic principles, disorders, and their clinical implications.
Professional Medicine:
Part of the MMLU benchmark, this subset assesses advanced medical knowledge expected of practicing professionals. It includes questions on complex medical cases, treatments, and ethical considerations.
PubMedQA: A question-answeriin dataset collected from PubMed abstracts, focusing research articles. It contains questions, corresponding abstracts, and answers, designed to evaluate model's ability to comprehend and extract information from biomedical literature.
24

---

Published as a conference paper at ICLR 2026
C.2 COMPARED METHODS DETAILS
To demonstrate the generality and effectiveness of our proposed PAVE specification, we selected representative pre-trained models from the corresponding domains for comparison.
Pre-trained language models. We compare these following pre-trained models for solving natural language processing tasks. Their specific information is listed below:
BERT (Devlin al. 2019) is transformer-baste language model developed by Google, designed to capture bidirectional context by processing text from both left to right and right to left simultaneously. It is pre-trained using masked language modeling (MLM) and next sentence prediction (NSP), allowing it to understand word relationships in context. BERT comes in multiple sizes, including BERT-base (12 layers. 768 hidden units, 12 attention heads) and BERT-large (24 layers, 1024 hidden units, 16 attention heads), providing different levels of computational efficiency and model complexity.
RoBERTa (Liu al. 2019), developed by Facebook AI, is an optimized variant of BERT that removes the next sentence prediction task and trains with larger batch sizes and more data. It refines the masked language modeling objective by dynamically changing the masking pattern across training epochs, leading to improved contextual representations. Like BERT, RoBERTa is available in different sizes, including RoBERTa-baa (same architecture as BERT-base) and RoBERTa-large (same architecture as BERT-large), offering enhanced performance in various NLP tasks due to its more extensive pretraining regimen.
Pre-trained vision models. These are some common pre-trained visual models for solving computer vision tasks as our compared methods. Their specific information is presented as follows:
CLIP (Radford et al. 2021) is a powerful multimodal model introduced by OpenAI, trained using contrastive learning to align images and text in shared embedding space. Unlike traditional supervised learning models that rely on labeled datasets, CLIP is trained on large-scale internet data, allowing generalize across a wide range of visual tasks. It supports different backbone architectures, including Vision Transformers (ViT), which are defined by patch sizes such as ViT-B/16 and VTT-B/32-VV size used for processing images. A smaller patch size (e.g., 16) allows for finer granularity but requires more computation, while a larger patch size (e.g.. 32) reduces computational cost at the expense of detail. CLIP has demonstrated strong zero-shot learning capabilities, making it highly adaptable across various domains.
the number refers to the patch
ResNet (He et al. 2016), short for Residual Network, is deep convolutional neural network architecture designed to address the vanishing gradient problem in very deep networks. By introducing residual connections, ResNet allows gradients to bypass multiple layers, ensuring stable training even with hundreds of layers. The architecture comes in various depths, including ResNet-50, ResNet-101, and R5sNet-152, where the number indicates the total number of layers in the model. Deeper variants, such as ResNet-1522 152,, tend to capture more complex features but require more computational power. ResNet has been a foundational model in computer vision, excelling in image classification, object detection, and feature extraction across various datasets, including ImageNet.
Large language models. We also compare our method with some popular large language models on medical LLM benchmarks. These large language models are described as follows:
Developed by OpenAI, ChatGPT-3.5 Turbo (OpenAl 2023) is an optimized version of GPT-3.5, designed balance efficiency, speed, and response quality. is trained using a mixture of supervised fine-tuning and reinforcement learning from human feedback (RLHF), making it adept at conversational AI, coding, and general problem-solving. Compared to its predecessors, it is more cost-effective and faster, often deployed in applications requiring reliable and coherent text generation.
Meta's 1ama--3-Insruue (Grattafiori et al. 2024) is a fine-tuned variant of the Llama-3 series, optimized for enstruction-folloviii tasks. It builds on Llama-3's transformer-baser architecture, with improvements in context understanding and reasoning capabilities. This model is designed to be highly efficient, leveraging compact architecture perform well
25

---

Published as a conference paper at ICLR 2026
in knowledge-intensiie tasks while maintaining lower computational costs. is particularly useful for chatbots, research, and enterprise Al applications.
An iterative improvement over Llama-3, instruction-followe accuracy and response relevance. It enhances finentuning techniques to provide better alignment with user queries, ensuring more precise and context-aware outputs. Compared to its predecessor, demonstrates improved robustness in handling complex queries while maintaining high efficiency. This model is suitable for scenarios requiring nuanced responses, such as content generation, tutoring, and technical assistance.
(Grattafiori et al. 2024 refines
We also compare some learnware identification methods to illustrate the advantages of the PAVE specification. Next, we will introduce these methods and the reasons for comparing with them.
Random. The method uniformly and randomly picks one of the candidate learnwares and return the user. The performance of method reflects the average capability of the candidate learnwares and can be considered as a baseline.
Oracle. The method always returns to the user the best performing learnware on their user task, and is a method that exists ideally. It reflects the upper limit of what a learnware identification method can achieve and the optimal capability of the candidate learnwares,
RKME. This is the most dominant approach in previous research on Learnware and has had a profound impact in the field Wu et al.) 2023; Tan et al. (024c). The implementation of the method is based on the released platform Beimingwu (Tan et al. 2024b).
C.3 EXPERIMENT CONFIGURATION
Following the original dataset split, we divide the data into a training set and a test set, merging the validation set into the training set if it exists. The original test set remains unchanged, while the training set is further split into two parts: one for users and one for developers. For user tasks, we adopt the k-shot setting from few-shot learning, where each class contains i samples for the classification task. The regression task contains the same number of samples as the binary classification task. The remaining training samples are allocated to developers for model training and learnware construction. For natural language processing datasets, le is set to 16, while for computer vision datasets, it is set to 2, as they typically contain more classes.
In this experiment, we conducted two separate stages of training: one for learnware models and another for task vector construction, as outlined below.
For learnware training, all models were trained using the Adam optimizer (Kingma and Ba 2015) and followed the hyperparameters listed in Table A3 Specifically, the learning rate set to 2 for natural language processing tasks and lc-5 for computer vision tasks, with a weight decay of 0.1 for all tasks. The batch size was set to 8 for NI.P tasks and 128 for computer vision tasks. The maximum number of epochs was set to 5 for computer vision tasks, while the maximum number of steps was capped at 1000 for some tasks. In cases where models identified from user tasks were fine-tuned, the same set of training hyperparameters was used to ensure faimess and generalizability across all tasks. This consistent approach in fine-tuning ensures that the models adapted specific user tasks without introducing variability due to differing training configurations.
For task vector training, the hyperparameters largely aligned with those used for learnware training to maintain consistency in how task vectors represent the models' capabilities. The learning rate for task vector training was set to 2e-5 for natural language processing tasks and le-5 for computer vision tasks, matching the learning rates used in learwware training. Similarly, the batch size was set to for NLP tasks and 128 for computer vision tasks, consistent with the batch sizes used during learnware training. Unlike learnware training, we did not apply weight decay for NLP and computer vision tasks in task vector training, which aligns with the explanation provided in Section 3.2 Task vectors were constructed using pre-trained models appropriate for each task type. ReBERTa-bar was used for encoder-only natural language processing tasks, CLIP-B-32 for computer vision tasks, and q.5n-O.5B (Bai et al. 2023) for decoder-only medical L.LMs.
26

---

Published as a conference paper at ICLR 2026
Table A3: Hyper-parameters for training learnwares.
Hlper--paameee
Classic NLP
Computer vision
Medical LLM
Learning rate
2e-5
le-5
[le-5, 2e-5, le-4, 2e-4)
Weight decay
0.1
0.1
0.0 (32. 64)
Batch size
128
Number of epochs (max)
5
Number of steps (max)
1000
model
RoBERTa-hase
ViT-B-32
Llama-3. --8-ttrrutut
C.4 COMPUTATION RESOURCES
Experiments were conducted using Tesla A100 80GB GPU, two Intel Xeon Platinum 8358 CPUs with 32 cores each (base clock 2.6 GHz, turbo boost 3.4 GHz), and 512 GB of RAM.
D EXPERIMENTAL RESULTS
This appendix provides additional details and supplementary materials complement the experimen- tal results presented in Section 4 additional visualizations that further demonstrate the effectiveness of our method across diverse task types. The results are structured to highlight the performance under various experimental settings.
includes extended performance metrics, ablation studies, and
D.1 PERFORMANCES
Similarity
Persrmonte
02
30
ciccomac
meaccorac
HRPCaccrery
Merca
bre-arcoracy
His Hisfaccoray
STS4 patrtann
5TS apatammnt
ogg cyparcanacy
sublercurccy
SRU SRU--CCHHHY
CotA mittmatthen cormeetteo
RTE RTE-accurory
Figure A4: Visualization of performance on classic NLP datasets.
27

---

Published as a conference paper at ICLR 2026
10.
Pertorance
26
52
Allrearmerss Simdartt
SU80977
5WW
GTORd
Poad1
EurmSA
DTD
CRANTiO
Use lasks
Figure A5: Visualization of performance on computer vision datasets.
Table A4: Performances on medical LLM benchamrks.
User Task
Large Langunge Models
Identify Leamware
Dataset
G.T Turbo
Llama. 3 Instruct
Llama Instruct
Random
RKME
PAVE
Oracle
Anatomy
70.37
Clinical knowledge
74.71
60.41
78.87
76.78
77.36
78.49
79.09
College biology
72.91
63.70
81.25
80.88
79.86
84.03
85.28
College medicine
64.73
71.70
68.79
66.85
69.07
67.92
70.23
Medical genetics
74.00
74.31
77.00
78.68
76.50
78.00
82.33
McdMCQA
53.79
57.08
59.26
58.03
56.25
60.55
60.55
MedQA
57.71
$7.08
63.63
60.94
64.24
64.13
64.24
Professional medicine
63.01
77.57
PubMadQA
72.66
82,00
74,80
74.59
76,20
76,50
76.50
Avg.(t)
67.69
65.52
72.15
71.19
71.53
72.78
74.14
Avg. rank(1)
3.44
4.56
2.44
3.44
3.00
2.11
(winhiefloss)
28

---

Published as a conference paper at ICLR 2026
Simlarity top
Smmlarll top
75
Accracy
85
60
55
MedBMCIA
Medii
Anstory
Colegg blinng
Cantral prowinngge
Callege medicier
Medical genetess
Professional mediche
(Banchmarks!
Figure A6: Visualization performance on medical LLM benchmarks.
PAVE Tricuun Pro treireiiedd Model
Mandly Loumnare
Figure A7: Performances on classic NLP datasets. And to achieve better visualization, perfor- mance metrics are normalized relative the Oracle.
29

---

Published as a conference paper at ICLR 2026
Table A5; Performances on classic NLP datasets.
User Task
Finetune Pre-trained Models
Identiy Learwware
Dataset
Metric
BERT-B
BERT-L
RoBERTa-B
RoBERTI-L
Random
RKME
PAVE
Oracle
CoLA
Mee
0.049
0.012
0.086
0.064
0.019
0.460
0.460
0.460
MNLI
Acc
0.498
0.519
0.594
0.646
0.520
0.731
0.731
0.731
MRPC
Acc
0.537
0.500
0.662
0.703
0.576
0.743
0.743
0.743
MRPC
FI
0.609
0.694
0,749
0.776
0.669
0.811
0.811
0.827
QNLI
Acc
0.552
0.542
0.636
0.682
0.559
0.817
0.817
0.817
0.540
0.771
QQP
F1
0.542
0.558
0,648
0.664
0.569
0.737
0.737
0.737
RTE
Acc
0,527
0.560
0,523
0.686
0.585
0.690
0.690
0.690
SNLI
Ace
0.515
0.575
0.676
0.753
0.519
0.785
0.785
0.785
SST-2
Ace
0.811
0.864
0.889
0.937
0.796
0.908
0.908
0.917
STS-B
0.332
0.536
0,740
0.620
0.317
0.859
0.859
0.859
STS-B
See
0.312
0.529
0,734
0.615
0.354
0.860
0.860
0,860
Ace
0.408
0,408
0.423
0.432
0.268
0.268
Acc
0.873
0.881
0.906
0.920
0.806
0.921
0.921
0,921
mpqa
Ace
0,789
0.775
0.822
0.812
0.700
0.869
0.869
0.869
Ace
0.757
0.802
0.849
0.877
0.756
0,856
0.856
0.872
subj
Acc
0.858
0.883
0.910
0.909
0.617
0.940
0.940
0.940
0.752
0.728
0,740
0.824
0.301
0.230
0.948
Avg-()
0.572
0.610
0.682
0.699
0.535
0.736
0.776
0.791
Avg. rank(
5,889
5.444
3.722
3,000
5.833
1.722
1.389
PAVE (winttie/lsss)
17/01
17/01
17/01
15/0/3
17/0/1
1/17/0
0/14/
HAVE va Niittnn Pro 2ww.
Nirriiyy Loar nward
Figure A8: Performances on computer vision datasets. And to achieve better visualization, performance metrics are normalized relative to the Oracle.
Table A6: Performances computer vision datasets.
User Task
Finetune Pre-trained Models
Identify Learwware
Dataset
R5sNet-152
ViT-B-32
ViT-L-14
Random
RKME
PAVE
Oracle
CIFARIO
0.100
0.899
0.956
0.780
0.511
0.890
0.979
CIFARIO0
0.008
0.663
0.777
0.527
0.226
0.885
0.885
Cars
0,006
0.604
0.784
0,492
0.462
0.680
0.680
DTD
0.022
0.440
0.556
0.406
0.293
0.698
0.698
EuroSAT
0.161
0.541
0,536
0.437
0.112
0.991
0.991
Food101
0.008
0.828
0.924
0.699
0.455
0.886
0.886
GTSRB
0.030
0.330
0.511
0,351
0,295
0.987
0.987
MNIST
0.117
0.503
0.764
0.551
0.811
0.997
0.997
RESTSC45
0.019
0.611
0.713
0.524
0.338
0.934
0.934
STLI0
0.060
0.973
0.994
0,927
0.831
0,985
0.985
SUN397
0.003
0.648
0.694
0,590
0.616
0.739
0.739
SVHN
0.096
0.329
0.584
0.398
0.970
0.970
0.970
(T).(1)
0.052
0.614
0.733
0,557
0.493
0.887
0.894
Avg. rank(J)
5.917
3,250
1.917
4,000
4.417
1.417
PAVE (win/tie/loss)
12/0/0
11/0/1
8/0/4
12/0/0
11/1/0
0/11/1
30

---

Published as a conference paper at ICLR 2026
24
6A
Performances classic NLP datasets beyond the original functionality. And to Figure A9:
achieve better visualization, all performance metrics are normalized relative to the Oracle.
D.2 BEYOND THE ORIGINAL FUNCTIONALITY
Alllearwwoce Simdurky top
Similarky top
SST2 actuaayy
MaPCacurary
HRPCR
Inc.rcurcyy
ONd lacraaay
HiL Lcccarcy
575-8 prisnson
ST5- pppmmaa
Ogp apPaaccurccy
subiaccuracy
SH SNLlacuracy
Araattiieess coratadonn
cracoumecy
User Tasks
Figure A10: Visualization of performance on classic NLP datasets beyond the original functionality.
31

---

Published as a conference paper at ICLR 2026
Table A7: Performances computer vision datasets beyond the original functionality.
User Task
Finetune Pre-trained Models
Identify Learnware
Dataset
ResNet-152
ViT-B-32
ViT-I-114
Random
RKME
PAVE
Oracle
CIFARIO
0.100
0.899
0.956
0,795
0.577
0.897
0,916
CIFARI00
0,008
0,663
0.777
0,617
0,410
0.693
0.698
Cars
0.006
0.604
0.784
0.573
0.555
0.613
0.613
DTD
0.022
0.440
0.556
0.405
0.334
0.461
0.461
EAroSAT
0.161
0.541
0.536
0.605
0.182
0.782
0.782
Food101
0.008
0.828
0.924
0,786
0,714
0.828
0.833
GTSRB
0.030
0.330
0.511
0.368
0,361
0.370
0.426
MNIST
0.117
0.503
0.764
0.627
0.864
0.864
0.864
RESISC45
0.019
0.611
0.713
0,569
0.481
0,515
0.657
STLI0
0.060
0.973
0.994
0,932
0.853
0.931
0.975
SUN397
0,003
0,648
0.694
0,617
0,628
0.638
0.638
SVHN
0.096
0.329
0.584
0,436
0.538
0.346
0.538
Avg-(1)
0.052
0.614
0.733
0,611
0.541
0.662
0.700
Avg. rank(J)
6.000
3.083
1.417
3.583
4,250
2.583
PAVE (win/tic/loss)
12/0/0
7/0/5
2/010
9/0/3
10/1/1
0/5/7
Figure A11: Performances computer vision datasets beyond the original functionality. And to achieve better visualization, all performance metrics are normalized relative to the Oracle.
32

---

Published as a conference paper at ICLR 2026
Allenenerres Simitrity
07
26
35
0.
03
RESICCAD
HuST
Sww
orsea
Iedd
Estolarr
uro
STuu
CPASSO
User Taoks
Figure A12: Visualization of performance computer vision datasets beyond the original function- ality.
33

---

Published as a conference paper at ICLR 2026
D.3 ABLATION STUDY
We provide supplementary results for the ablation study, including performance tables where "PAVE"" represents the performance of task vectors derived from fitting the data distribution p(y|x) x). serving as the ablation variant. Additionally, we present similarity matrices across tasks to show the effectiveness of parameter vector similarity approximation in the low-rank space. These materials offer insights into the contribution of key components and the robustness of our approach.
Table A8: Performances on classic NLP datasets with corrupted learnwares.
User Task
Finetune Pre-trained
Identify Learnware
Dataset
Metric
BERT-B
BERT-L
RoBERTa-B
RoBERTa-L
Random
PAVE
PAVE
Orscle
CoLA
Moc
0.049
0.012
0,086
0.064
-0,006
0.215
0.460
0.460
MNLI
Acc
0.498
0.519
0.594
0.646
0.505
0.611
0.731
0.731
MRPC
FI
0.609
0,694
0.749
0.776
0.604
0.675
0.811
0.827
QNLI
Acc
0.552
0.542
0,636
0.682
0.533
0.662
0.817
0.817
QQP
Ace
0,574
0.512
0.653
0.664
0.472
0.587
0.771
0.771
QQP
0.542
0.558
0.648
0.664
0.543
0.627
0.737
0.737
RTE
Acc
0.527
0,560
0.523
0.686
0.558
0.610
0.690
0.690
SNLI
Ace
0,515
0.575
0.676
0.753
0.502
0.635
0.785
0.785
STS-B
Poc
0,332
0,536
0.740
0.620
0.230
0.501
0.859
0.859
STS-B
See
0.312
0.529
0,734
0.615
0.274
0.527
0,860
0.860
WNLI
Ace
0.408
0.408
0.465
0.423
0.420
0.338
0,268
0.493
Ace
0.873
0.881
0.906
0.920
0.804
0. 862
0,921
0.921
mpqa
Acc
0.789
0,775
0.822
0.812
0.706
0.790
0.869
0.869
mr
Ace
0.757
0.802
0,849
0,877
0.745
0.795
0.856
0.872
subj
Acc
0,910
0.909
0.736
Acc
0,752
0,728
0.740
0.824
0,275
0.599
0.948
0.948
Avg.()
0.572
0.610
0,682
0.699
0.502
0.622
0.776
0.791
Avg. rank(.)
5,500
4.667
3.056
2.222
6,356
4,500
1.444
PAVE (win/tce/oss)
15/03
17/0/1
PAVE (winticsloos)
14/04
8/0/10
4/0/14
1/0/17
17/0/1
I/V117
0V0/18
Table A9: In mode (a), parameter similarity is computed exactly using the full parameter vectors obtained through complete fine-tuning. mode (b), similarity is approximated in the low-rank space after expanding the parameter vectors to their full size as (BA)mxx In mode (c), similarity is approximated both the low-rank space and computed using only the matrix B, which significantly reduces storage and computational costs-this is the method proposed in our approach.
Computer Vision
Medical LLM Benchmarks
Abs.
Ratio
Ratio
Abs.
Ratio
(a)
120.6IM
147.08M
472.55M
Trainable Parameters
(b)
0.42M
0.3%
1.4IM
1.0%
0.42M
0.1%
(c)
0.42M
0.3%
1.41M
1.0%
0.42M
0.1%
(a)
120.61M
147.08M
472.55M
Computing Similarity
(b)
120.6IM
100.0%
147.08M
100.0%
472.55M
100.0%
0.42M
0.3%
1.4IM
1.0%
0.42M
0.1%
(a)
120.61M
147.08M
472.55M
Storage Cost
(b)
0-42M
0.3%
1.41M
1.0%
0,42M
0.1%
(c)
0.42M
0.3%
1.41M
1.0%
0.42M
0.1%
(a)
0.7807
0,8868
71.47
Performance
(b)
0.7762
99.4%
100.0%
72.78
101.8%
(c)
0.7762
99.4%
0.8868
100.0%
72.78
101.8%
34

---

Published as a conference paper at ICLR 2026
Figure A13: Consistency of the cosine similarity between parameter vectors on computer vision datasets, with the diagonal colors omitted to enhance visual contrast.
Figure A14: Consistency of the cosine similarity between model vectors medical LLM bench- marks.
35

---

Published as a conference paper at ICLR 2026
E ANALYSIS
E. HIERARCHICAL LINEAR REGRESSION
To validate the effectiveness of our proposed method, we conduct a quantitative analysis to examine the relationship between similarity and performance. Specifically, we employ hierarchical linear regression to assess whether higher task vector similarity correlates with improved model performance on user tasks. Additionally, use the superscripts "4> and in the tables denote different experimental settings, introduced in Section 4.2 and Section 4.3 respectively. These symbols help distinguish the specific configurations under which method were evaluated. Note that although "PAVE" E denotes the ablation variant in Section 4.3 it here represents the standard PAVE similarity in scenarios involving corrupted learnwares (i.e., those associated with low-quality models)
We conduct this analysis on both natural language processing (NLP) and computer vision (CV) datasets, treating the user task as a grouping variable (Woltman et al. 2012). The results, presented in Table A10 and Table A1l for NLP and CV, respectively, show statistically significant positive correlation between task vector similarity and performance metrics. This indicates that models with higher similarity to the given task tend to achieve better results.
Table A10: Hierarchical linear regression on natural language processing datasets.
Coef.
Std Eir.
P>lø
CI Lower
CI Upper
Intercept
0.508
0.053
9.557
0.000 000
0.404
0.612
Similarity
0.788
0.063
12.482
0.000
0.664
0.912
Intercept 0.667
0.053
12.644
0.000
0.564
0.771
Similarity
0.122
0.044
2.785
0.005 2as
0.036
0.208
Intercept*
0.499
0.057
8.688
0.000
0.386
0.611
Similarity
0.708
0.047
15.050
0.000
0.616
0.801
In the regression Table A10 and Table A11 the P>lzl column reports p-values, indicating the likelihood of the observed relationship under the null hypothesis. Lower values provide stronger evidence against the null, with common significance thresholds of 0.05 and 0.01" T significant p-values further validate the observed correlation.
In our analysis,
Table Al1: Hierarchical linear regression on computer vision datasets.
Coef.
Std Err.
P>lzl
CI Lower
CI Upper
Intercept
0.497
0.051
9.703
0.000
0.396
0.597
Similarity
2.988
0.282
10.612
0.000 **..
2.436
3.539
Intercept
0.592
0.049
12.138
0.000 *.00
0.496
0.687
Similarity o
1.738
0.572
3.040
0.002 ***
0.617
2.858
Intercept 0.557
0.055
10.089
0.000
0.449
0.665
Similarity 2.248
0.227
9.911
0.000
1.804
2.693
These findings provide strong empirical evidence for our approach, demonstrating that task similarity a valuable criterion for selecting the most suitable learnware for a given user task. Furthermore, the consistency of results across different domains reinforces the generalizability of our method.
F.2 EXACT MULTINOMIAA. TEST
We employ the Exact Multinomial Test (Read and Cressie 2012 rigorously evaluate our proposed method against a null hypothesis of random selection. The Exact Multinomial Test is a non-parametric
Significance levels: p 0.1 (*), p S 0.05 (+*), S 0.01 (*+*), p S 0.001 (****).
36

---

Published as a conference paper at ICLR 2026
statistical procedure used to determine whether the observed frequencies of categorical outcomes differ significantly from the frequencies expected under a given null hypothesis. our context, each user task represents an independent trial where the outcome is defined by whether the learnware identified by our method is the optimal one for a given user task among those candidate learnwares.
Under the null hypothesis Ho, assume that each learwware has equal probability of being identified for a given user task, which corresponds to the baseline method Random in our experiments. Therefore, we formally present the distribution Ri, which represents the number of top & learnwares in representative performance on the user task among the top k learnwares identified based on similarity from candidate set of size m. The distribution H; given as
k
h m
T.
for 0 sisk
m
where is the number of user tasks where our method is tied with Oracle in the evaluation results. And the variable x the observation of the experiment, where x; is the number of times the identification result with probability occurs. Then, we have
Pr (x))
|x|
By calculating the exact probability of obtaining the observed evaluation results (or a more extreme outcome), the p-value is given as
p(sig]
E
Py
y Pr(y), Pr(x),
The Exact Multinomial Test allows us to determine if the performance of our method is statistically significant. This test is particularly useful in scenarios with discrete outcomes and limited sample sizes, ensuring robust inference without relying on large-sample approximations.
Table A12: The statistical significance of our method: p-values.
Natural language processing Datasets
Computer Vision Datasets
PAVE PAVE &
0.00000
0.00000***
0.00081 ....
0.00193
PAVE
0.00000
0.00000 *..0
Our analysis revealed that the frequency with which our method identifies the most suitable model significantly higher than would be expected by chance, where the & is set to Specifically, in both natural language processing and computer vision tasks, the p-values obtained from the Exact statistically significant, presented in Table A12 Note that statistical significance is commonly inferred when the p-value is less than 0.05 or 0.01 These findings provide strong evidence that our method is effective in selecting superior models for user tasks, thereby validating its practical utility across different domains.
Multinomial Test indicate that the improvements
Significance levels: p 0.1 (*),
< 0.05 (**), p 0.01 (*+*), p 0.001 (*s0a).
37

---

Published as a conference paper at ICLR 2026
F DISCUSSION ON RKME
In this section, we first provide a brief overview the Reduced Kernel Mean Embedding (RKME) specification. Then highlight its similarities and differences with the Parameter Vector (PAVE) specification, emphasizing the unique strengths of our approach for deep learning models.
F.1 REDUCED KERNEL MEAN EMBEDDING (RKME)
Specification is the central part of the learnware, capturing the model ability. We briefly introduce the Reduced Kernel Mean Embedding (RKME) specification (Zhou and Tan 2024), which sketches the data distribution in the sample space x y inputs and outputs with kernel methods.
We start by introducing the Kernel Mean Embedding (KME) (Schölkopf and Smola 2002). which offers a novel representation for distributions. KME transforms distribution into a reproducing kereel Hilbert space (RKHS). Given distribution D defined over space N, the KME is defined as:
Pn (D) A / k(x, (dD(x))
(48)
where k X x X - R is symmetric and positive definite kernel function, and its associated RKHS H. For a data set (x: J sampled from D, the empirical estimate of KME is given by:
(D)
k(x,,))
(49)
m
KME is considered as potential specification due to several favorable properties. Accessing the raw data, however, compromises the necessary privacy concerns of the specification. Based on KME, the RKME specification is proposed to use a synthetic dataset of minor weighted samples ((3,,4)))==+ ni & m to approximate the empirical KME of the original dataset with model pseudo- outputs {4 ]= {(x; m where û = hx(x;) ) the model prediction. The synthetic dataset generated through minimizing the Maximum Mean Discrepancy (MMD) (Gretton et al. 2012):
m l(g ))
12
min 3,
Σ B,k(tz,
(50)
m
H
with the non-negative coefficients (3,))-1- The RKME (() = Σj= Byk(l,,-) H acts as the specification, and the RKHS H is referred to as the specification space. This specification effectively captures the major information of the distribution 'D without exposing raw data.
F.2 COMPARISON OF RKME AND PAVE
The RKME is initially designed for models on tabular data, whereas our method focuses on deep learning models. To protect data privacy, RKME generates synthetic dataset to represent the data distribution rather than the original data via minimizing Maximum Mean Discrepancy (MMD) (Wu al. 2023). However, for nigh-dimensional, unstructured data such as texts and images, generating meaningful synthetic data by minimizing MMD is still challenging and computationally expensive. In contrast, our method identify learnwares via the parameter vector similarity and has certain advantages in capturing the capabilities of deep learning models. Similarly, we compressed the parameter vector in low-rank space, which intuitively prevents the recovery of the private original data. From a high-level perspective, both methods generate a model's representation specification-in the representation space induced by a helper function, f. In RKME, f is a kernel function, and the specification is reduced set representation in the RKHS spanned by f. In contrast, our proposed PAVE employs a pre-trained model as f. yielding vector representation in the function space spanned by f. Below is comprehensive breakdown of the differences.
Application Scenarios. RKME is primarily designed for tabular data, where kernel-based methods can effectively capture the underlying distribution. Its reliance kernel mean embeddings makes well-suited for structured datasets where feature relationships can be explicitly modeled. In contrast, our method is tailored for deep learning tasks, where high-dimensional features and complex
38

---

Published as a conference paper at ICLR 2026
relationships between inputs and outputs are the norm. By leveraging pre-trained models and parameter vectors, our approach naturally adapts to a wide range of deep learning applications, making it more versatile in real-world deep learning tasks.
Representation. The two methods differ significantly in how they store and represent information. RKME implicitly encodes data distributions within reproducing kernel Hilbert space (RKHS), approximating the empirical kernel mean embedding through synthetic data. This approach ensures privacy while retaining key distributional properties. Our method, on the other hand, directly represents model specifications in H using parameter vectors derived from pre-trained models. This direct upproximation enables more efficient encoding of model capability and task information while circumventing the challenges of constructing high-quality synthetic data by minimizing MMD.
Privacy. Both methods prioritize data privacy but achieve it through different mechanisms. RKME protects raw data by generating a synthetic dataset, ensuring that the original data remains inaccessible while preserving statistical properties through maximum mean discrepancy minimization. Our method, instead, compresses information via low-rank approximations, making the reconstruction of the original dataset impossible. This not only safeguards privacy but also reduces storage and computational costs, making our approach more efficient for large-scale applications.
Generalization. RKME'S reliance on symmetric kernel functions introduces limitations in capturing task semantics and model quality. When comparing tasks with different output spaces- praces-Such-a binary classification versus multi-class classification
becomes difficult to define a reasonable kernel function that respects symmetry while preserving meaningful task information. The problem is further exacerbated when comparing model outputs, as user tasks involve raw data while model predictions represent probability distributions, leading to inconsistencies in how similarity is measured. Our method overcomes these issues by constructing parameter vectors with the task's customized loss function, which provide a general way to generate specification across diverse task type and semantics. By measuring the similarity of parameter vectors, the compatibility between any model and task can be evaluated, regardless of divergences in task types and semantics.
Expressiveness and efficiency. RKME struggles to effectively capture sample similarities in text and image datasets due to the limited expressiveness of traditional kernel methods. While neural tangent kernels (NTKs) could theoretically enhance synthetic data generation, they come at a high computational cost. contrast, our method leverages the structural advantages of pre-trained models, inherently benefiting from NTKs without added complexity. By operating directly in the pre-trained model's 's parameter space, our approach eliminates the inefficiencies of constructing synthetic datasets to approximate kernel mean embeddings. This enables a more expressive and scalable solution for deep learning tasks such as natural language processing and computer vision.
In summary, while RKME offers pnivacy-preservinn kernel-based specification for tabular data, our method is better suited for deep learning models. It provides a more effective and universal representation of task semantics and model quality while maintaining greater computational efficiency.
G DISCUSSION ON MODEL DESCRIPTION
In the current landscape of LLMs, agents, and world models, model selection and coordination primarily rely on semantic-level model descriptions such as Model Cards (Mitchell et al.) 2019) or the Model Context Protocol (MCP) (Anthropic 2024). For instance, HuggingGPT (Shen et al.) 2023) leverages LLMs as controllers to parse user requests and select expert models from Hugging Face based on their textual task descriptions. Extending this paradigm tool use, Gorilla (Patil et al.) 2024) and ToolLLM (Qin et al. 2024) align user instructions with API documentation to facilitate tool selection and invocation via retrieval- retrieval-ugmmented auggmeteed generation or instruction tuning. While such semantic descriptions support high-level coordination through natural language or structured metadata, they remain inherently extrinsic and often fail to faithfully reflect actual model capabilities, leading to "vocabulary gap" between textual specifications and empirical behaviors. To bridge this gap, PAVE characterizes models at the functional level as conditional distributions, providing an intrinsic representation that bypasses the ambiguity of textual metadata. This formulation enables mathematically grounded identification of helpful models across diverse manifestations, offering a level of precision and reliability that semantic descriptions fundamentally lack.
39

---

Published as a conference paper at ICLR 2026
H LLM USAGE DISCLOSURE
Large Language Models were used solely to aid in polishing and editing the writing of this manuscript. Their role was limited to language refinement, such improving clarity and grammar. No LLMs were employed in idea generation, experimental design, data collection, analysis, or code development. All scientific contributions the original work of the authors.
40


Reviews:
[{"review_id": "9oer5wRYaF", "reviewer": "Reviewer_KAPE", "summary": "The paper considers a setup that we have access to a large number of models. Each model consists of a model and a \"specification\" that describes its capabilities. The central problem is how to create a specification that allows a user to efficiently identify the most helpful model from a vast repository, without the costly process of evaluating every single model.\n\nThe main idea is simple: assume we have access to a trained model h. To create its specification, they take a shared, public pre-trained model (e.g., BERT, CLIP) and fine-tune it to mimic the predictions of their model h. The resulting change in the pre-trained model's parameters is saved as the model vector.  A user with a new task provides a small, few-shot dataset. To create their task specification, they fine-tune the exact same shared pre-trained model on their few-shot data to fit the true labels. This parameter change becomes the task vector. Finally the selection is based on measuring the similarity between task vector and the finetuning vector.", "questions": "see above", "limitations": "", "rating": 4, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The PAVE method's primary strength is its effectiveness with high-dimensional, unstructured data like images and text, a scenario where prior specifications failed. The authors conducted extensive experiments across Natural Language Processing (NLP),", "weaknesses": "Reliance on a Shared Pre-trained Model: The entire system fundamentally relies on both the developers (creating learnwares) and the users (sketching their tasks) using the exact same shared pre-trained model as a common basis to generate the parameter vectors. I think the authors should study the robustness of their finding s to this assumptions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p3HkdnsXpD", "reviewer": "Reviewer_bGit", "summary": "The paper proposes a new way, called PAVE (Parameter Vector), to describe and identify reusable models in the Learnware framework. Instead of using reduced data samples to represent a model’s capability (as in prior work like RKME), PAVE uses how a model’s parameters change during fine-tuning. This parameter-based “signature” helps match models (learnwares) with user tasks more efficiently and works even when data are high-dimensional (e.g., text or images). The authors also derive a theoretical link between PAVE and RKME through the neural tangent kernel (NTK) and propose a low-rank approximation to make computation feasible. Experiments on NLP and CV tasks show that PAVE can identify suitable learnwares better than baselines and sometimes outperform fine-tuned pre-trained models", "questions": "Please refer to the weakness section", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Using parameter changes instead of data samples to represent a model’s capability is original and intuitive, especially for privacy-sensitive or unstructured data.\n2. The connection between PAVE and RKME under the NTK assumption provides a sound theoretical explanation for why the approach should work.\n3. The low-rank approximation (similar to LoRA) reduces the storage and computational cost while maintaining similarity accuracy\n4. The experiments cover multiple domains (NLP, CV, medical LLMs) and show clear and consistent improvements over RKME and fine-tuning baselines.", "weaknesses": "1. While the paper introduces PAVE as a general solution, it’s not entirely clear how broadly this parameter-vector representation generalizes beyond fine-tuned models based on shared backbones.\n2. The method assumes all learnwares are fine-tuned from a common pre-trained model. It’s unclear how it performs when models come from different architectures or pre-training distributions.\n3. The experiments mainly compare to RKME and basic fine-tuning baselines. It would strengthen the evaluation to include more data-centric reuse or transferability estimation methods, such as LEEP, LogME, or task2vec, to contextualize PAVE’s advantage.\n4. The paper doesn’t discuss how PAVE specifications would be stored, shared, or updated in a real learnware repository, or how they interact with privacy and security constraints.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h8p9Mr98dX", "reviewer": "Reviewer_iVnw", "summary": "This paper introduces the Parameter Vector (PAVE) specification to identify and select high-quality learnwares for reuse in solving new user ML tasks. The PAVE approach formalizes the use of parameter vector changes (from fine-tuning a shared pre-trained model) as a representation of both a model’s capability and the requirements of the user’s task. Theoretically, the work connects PAVE to kernel mean embedding approaches via the neural tangent kernel regime, derives error bounds for low-rank approximations of the parameter vectors, and demonstrates through extensive empirical studies that PAVE achieves superior learnware selection, often outperforming conventional fine-tuning and previous learnware identification methods.", "questions": "The PAVE specification, as described in L51-53, appears to rely on a shared pre-trained model and architecture to ensure the comparability of parameter vectors. Could the authors clarify if this is a necessary constraint? Furthermore, could the authors provide discussion on the potential for generalizing PAVE to a more heterogeneous setting, where learnwares in a repository might originate from different base models or possess distinct architectures?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper tackles the practical challenge of efficiently reusing high-performing models by providing a specification (PAVE) that allows users to identify suitable models without direct per-model evaluation.\n- The mathematical exposition is detailed and generally clear.\n- The proposal for low-rank approximation of parameter vectors significantly reduces memory and compute, a nontrivial contribution given the scale of modern check-pointed models.\n- The experimental validation is exceptionally thorough and is a major strength of the paper.", "weaknesses": "Generally, I did not identify any major limitations or weaknesses in its core contributions. This is a high-quality paper that makes a practical contribution to the field of model reuse and the Learnware paradigm. The proposed PAVE specification is novel, well-motivated, and supported by both theoretical analysis and a comprehensive set of experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)