Handling Learnwares from Heterogeneous Feature Spaces with Explicit Label Exploitation
Authors: Peng Tan
Hai-Tian Liu
Zhi-Hao Tan
Zhi-Hua Zhou
Venue: NeurIPS 2024 poster (2024)
Keywords: learnware
heterogeneous feature spaces
subspace
Primary area: evaluation
TL;DR: This paper solves the problem of managing and reusing models developed from heterogeneous feature spaces by explicitly utilizing label information.

Abstract: The learnware paradigm aims to help users leverage numerous existing high-performing models instead of starting from scratch, where a learnware consists of a well-trained model and the specification describing its capability. Numerous learnwares are accommodated by a learnware dock system. When users solve tasks with the system, models that fully match the task feature space are often rare or even unavailable. However, models with heterogeneous feature space can still be helpful. This paper finds that label information, particularly model outputs, is helpful yet previously less exploited in the accommodation of heterogeneous learnwares. We extend the specification to better leverage model pseudo-labels and subsequently enrich the unified embedding space for better specification evolvement. With label information, the learnware identification can also be improved by additionally comparing conditional distributions. Experiments demonstrate that, even without a model explicitly tailored to user tasks, the system can effectively handle tasks by leveraging models from diverse feature spaces.

Handling Learnwares from Heterogeneous Feature Spaces with Explicit Label Exploitation
Peng Tan, Hai-Tian Liu, Zhi-Hao Tan, Zhi-Hua Zhou
National Key Laboratory for Novel Software Technology. Nanjing University, China School of Artificial Intelligence, Nanjing University, China {tanp, liuht, tanzh, 2houzh)00lamma nju.. edu. cn
Abstract
The learnware paradigm aims to help users leverage numerous existing high- performing models instead of starting from scratch, where a learnware consists of a well-trained model and the specification describing its capability. Numerous learnwares accommodated by learnware dock system. When users solve tasks with the system, models that fully match the task feature space are often rare or even unavailable. However, models with heterogeneous feature space can still be helpful. This paper finds that label information, particularly model outputs, is helpful yet previously less exploited in the accommodation of heterogeneous learnwares. We extend the specification to better leverage model pseudo-labed and subsequently enrich the unified embedding space for better specification evolvement. With label information, the learnware identification can also be improved by additionally comparing conditional distributions. Experiments demonstrate that, even without a model explicitly tailored to user tasks, the system can effectively handle tasks by leveraging models from diverse feature spaces.
1
Introduction
The current machine learning paradigm has achieved remarkable success across various domains. This success, however, hinges several critical factors: access to abundant high-quality labeled data, expensive computational resources, and deep expertise in feature engineering and algorithm design. These requirements pose significant challenge for ordinary individuals aiming to build high-quality models from scratch. Moreover, issues such as data privacy, the difficulty model adaptation, and catastrophic forgetting complicate the reuse or adaptation of trained models across different users.
Indeed, most efforts have focused on these issues separately, paying less attention the fact that these problems are entangled. To address these challenges simultaneously. the learnware paradigm was proposed by Zhou [2016). The learnware paradigm [Zhou and Tan 2024) aims to assist users in solving their tasks by leveraging existing high-performing models, through the establishment of learnware dock system. One important purpose of learnware paradigm is to enable high-pere perffrrrrooinggggg models, submitted by developers, to be used "beyond- wead-wwhttwweee was-submitted." This means that models can be repurposed to assist with tasks not originally targeted by developers. To achieve this, learnware is designed as gerghperforming model with a specification describing its capability and utility. The specification, a central component for learnware management and identification, can implemented by sketching the data distribution in which the model is proficient (Zhou and Tan 2024 Recently, to facilitate research on the learnware paradigm, the learnware dock system, Beimingwu, has been released (Tan al. 2024a
Previous research Liu et al. 2024 Xie et al. 2023 Zhang et al. 2021 focuses on the homogeneous case where models and user tasks share the same feature space. However, in real-world scenarios, the feature spaces of models often differ due to varied feature engineering. As an example, we
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

consider the widely used clinical database, the OMOP Common Data Model [Biedermann et al. 202T]. as illustrated in Figure This model manages healthcare data from various sources through several standardized tables, such as demographic information, diagnoses, laboratory results, and medications. Experts across different hospitals often use different tables for feature engineering, even when working on the same clinical task, leading to the development of heterogeneous models.
In order manage and exploit models developed from het- erogeneous feature spaces, it is essential to build connections between these different spaces. Existing related techniques for exploiting relationships between feature spaces either rely on raw data [Wang and Sun 2022] Zhu al.) 2023] of the
drug safety analysis
fx
model or utilize additional co- co-occurrence data [ et al. 2013
Huang et al. 2023]. However, with model specifications, the learnware dock system can determine the relationships through subspace learning without the need for raw data or extra aux-
feature engineering
iliary data [Tan et al. 2023]. To effectively accommodate heterogeneous learnwares, a unified subspace is constructed based on specifications of all submitted models, which helps to evolve the specification to have the capabilities of meeting requirements across different feature spaces. This paper finds that, without label information, subspace learning tends to yield suboptimal results, causing embeddings with entangled class representations in the subspace, or even rendering them meaningless when feature spaces are only weakly correlated. Additionally, without exploiting label information, the system can only identify models with marginal distributions similar to the user's task, ignoring models' capabilities.
person
diagnose
laboratory
drug
standard tables database
Figure 1:
Heterogeneous feature space models in real-world scenario.
This paper explicitly leverages label information for managing and utilizing heterogeneous models. We extend the specification to better incorporate model pseudo-labels, enabling the transition from unsupervised to supervised subspace learning for better specification evolvement. The extended specification also allows for additional comparison of conditional distributions using label information, thereby improving the learnware identification. The contributions
follows:
This paper proposes to exploit the model outputs to evolve specifications into a unified space during heterogeneous learnware accommodation. Specifically, the unified space is constructed based on the specifications of all models. By exploiting model outputs encoded in the specification, the resulting subspace exhibits improved properties, with less entangled class representations and more coherent embeddings,
This paper extends the specification implementation to more effectively leverage label infor- mation by encoding both marginal and conditional distributions. This extended specification provides more accurate label information during subspace learning to better evolve specifi- cations. Additionally, it also allows for additional comparison of conditional distributions, thereby improving the learnware identification.
Experiments demonstrate that, even without model explicitly tailored the user's task, the system can effectively handle the task by leveraging models from diverse feature spaces.
2 Preliminary
Specification is the central part of the learnware, capturing the model ability. This section briefly introduces the Reduced Kernel Mean Embedding (RKME) specification [Zhou and Tan [202]) which sketches the joint distribution of task features and model outputs with kernel methods.
We start by introducing the Kernel Mean Embedding (KME) (Schölkopf and Smola 2002 which offers novel representation for distributions. KME transforms distribution into reproducing kernel Hilbert space (RKHS). Given a distribution D defined over a space X, the KME is defined Mr (D) := Jx k( (xx),, where X x X + E is symmetric and positive definite kernel function, and its associated RKHS is H. For data set {x+), sampled from D, the empirical estimate of KME given by (D) T7 Σn k (a);.).
KME is considered as potential specification due to several favorable properties. Accessing the raw data, however, compromises the necessary privacy concerns of the specification. Based

---

on KME, the RKME specification is proposed to use reduced set of minor weighted samples {(5,,t)) nn & m to approximate the empirical KME of the original dataset with model pseudo-outputs {q: 14:1e = {x, 0.) 0.))=r where di = f(x;) is the model prediction. The reduced set is generated by:
min B,t
Σ ==1
k (g
Σ 3yk(1
7
with the non-negative coefficients {3,} The RKME - E3 Byk (t,.-) E acts as the specification, and the RKHS H is referred to the specification space. This specification effectively captures the major information of the distribution D without exposing raw data and explicitly encodes the model capability based its outputs. Notably, in simple cases where the features are sufficient to represent the model capability, the sketch solely on the features { J/m can also be used as the
model specification [Wu al., [023]. In this paper, further extend the specification generation process to more effectively encode the model's 's outputs.
3 Problem setup
This paper addresses the challenge constructing a heterogeneous learnware dock system and leverag- ing it to assist users who have only limited labeled data such that training model by themselves will lead to poor performance. Without loss of general- ity, we consider the underlying full feature space, denoted as Xan, as composite of Q distinct blocks, Xq. The feature spaces for developers, X and for users, Xuser are rep- resented as Cartesian products of specific blocks Xiec where C refers to block indices.
Ai NBs ... Xa
learnware dock system
Xail
mocel & specificaioo
developer task 1
fi
submit
O (f..a))
i... Nan X1
developer task
fx
submit
x
(frr axe)
The overall procedure consists of two stages: the submission stage and the deployment stage. In the submission stage, the developer trains a well-performing model Ja on the dataset D.
recommend
user task
ta requirment
((aig. )= and generates a developer-level specification sdev which captures the model's 's per- formance without exposing raw data. After receiv- ing all heterogeneous models and their developer-
An illustration of the learnware paradigm with heterogeneous feature spaces
Figure 2:
level specifications, the learnware dock system assigns a system-level specification 8; to each model fú. based all submitted specifications {a ) j==1- The heterogeneous learnware dock system is then constructed as {(fi, s:) / In the deployment stage, the user has unlabeled data Dj {wos = and limited amount of labeled data Dú2 { (@oo, )o:)) (the unlabeled data cover labeled data features, ie.. (ã:}: S (x; }.) The user generates a user-level task requirement syer and submits it to the learnware dock system. The system then identifies the most helpful model(s) for reuse tackle the user task.
4 Methodology
This section outlines our methodology for accommodating heterogeneous models under the learnware paradigm and assisting user tasks, emphasizing the importance and utilization of label information, which remains unexplored learnware paradigm when dealing with heterogeneous feature spaces.
4.1 Improve managing heterogeneous models with label information
To handle learnwares with heterogeneous feature spaces, it is helpful to exploit the relationships between these spaces. A common approach to learn a unified subspace. However, without label information, the resulting subspace may produce entangled embeddings of samples from different
3

---

classes, and when feature blocks are weakly correlated, the subspace may become meaningless (see Section B.. for detailed discussion). Since subspace learning based on all learwaare specifications. incorporating label information into the model specification is highly beneficial.
We rewrite the RKME specification represented by R (B, T) = (,) RKME, represented by R, = ZB, = {(3g. 2,, by splitting the sample t, into the feature 2j and the pseudo label Vj. emphasizing label information. Given existing model specifications {s(tV [(3g. 2zj Miy) )= }. the learnware dock system can learn unified subspace Xuub with encoding functions {h+ Xx + Xsub 1 and decoding functions {9: Xsub + Xx X by optimizing
L= Lrccossrrctiiiiionnnoo + 022 Lsimilar + ag Lupeerered over mapping functions (hx, gk 1 The objective function has three components: the reconstruction loss, which trains mapping functions (/g, 9x) to map and reconstruct data in Xx: the similarity loss, which makes embeddings of different slices of zj similar: and the supervised loss, which uses label information to improve subspace learning by making class embeddings more separable or aligning samples within the same class. After subspace learning, the mapping functions (hx- 91 ) can project data from any combination of feature space blocks to the subspace. When reusing heterogeneous models, these functions can also fill in missing parts needed for model predictions. Such a framework can be implemented by existing subspace learning methods, such self-supervised learning [Ucar et al.) 2021 Bahri et al., [022], matrix factorization [Xu and Gong 2004, Wang et al. 2016]. When the system receives learnwares from unseen feature spaces after subspace generation, the system can update the subspace during idle time.
4.2 Improve matching model and user task with label information
Matching with only marginal dis- tribution Px is not enough. We first review previous methodolo- gies for matching a user's task with a model in the homogeneous case, where all models and user tasks share the same feature space [Wu al., [0033 Zhang et al.) (2021]. These methods recommend the model with the most similar marginal distribution Px. To avoid exposing raw data, they use RKME to sketch the marginal distributions of the model task and user task, serving as the model specifications and user requirements. To illustrate the deficiency, we refer to Figure 3 which presents five tasks with uniform distributions. Among these tasks, four have circular support sets and one has a square support set. The two problems are: 1) Models with the same Px but different Pxjy are indistinguishable. In Figure 3 Models 1, 2 and 4 are all recommended, but model 2 is unsuitable for the user task. 2) Models with different Px are rarely considered, despite their potential usefulness. Model 3 in Figure 3 though suitable, are excluded because its Px is square instead of circle.
(6) model
(b) model 2
(s) modal
(d) model 4
(+) user lask
unitorm distribttion (positive diass)
unifomm distribution (nogative class)
Figure 3: Label information is beneficial for matching.
Enhance matching by incorporating the conditional distribution Pxiy. Matching solely on the marginal distribution is insufficient for model identification. To better recommend models to user tasks, we propose additionally considering the conditional distribution Pxir. which helps exclude the model with dissimilar conditional distributions (Model 2) and include the model with similar ones (Model 3). While the user's task can estimate the conditional distribution from labeled data, a key question arises for the model task: should we use true labels or model-generatte pseudo labels? Using True labels results in comparing the user's task distribution P(X, Y) with the model's 's original task distribution, while pseudo labels results in comparing the model-generated joint distribution P(X, Y) with new tasks, allowing the model to be reused beyond its original purpose. As shown in Figure 3 Model 4 would be recommended using pseudo labels but not using true labels. In conclusion, considering both marginal and conditional distributions improves model identification, with model-generated pseudo labels being helpful for encoding model capabilities.
4.3 Summary
To accommodate and identify models developed from heterogeneous feature spaces, it advantageous to utilize pseudo-label information generated by the models. To better incorporate this information,

---

we propose to integrate both marginal and conditional distributions into the model specification and user requirements, represented as {.. z,, y/ 1ー By comparing these distributions, we improve learnware identification. The inclusion of label information enhances subspace learning, and the framework can be applied across various subspace learning methods.
Detailed solution
In this section, we provide the detailed procedure for the heterogeneous learnware problem based on the aforementioned methodology, which consists of model specification generation, heterogeneous learwwares accommodation by the system, and system exploitation for solving user tasks.
5.1 The developer generates the model specification
The model specification sketches task distribution and model capabilities with reduced set. Instead of sketching the joint distribution of task features and outputs (Eq. (.). propose to generate feature and label part separately to balance label and feature information. This includes a unified mechanism for classification and regression, and specialized mechanism for classification.
Unified mechanism for classification and regression tasks. Given a dataset = {(2;, yi }
and model trained it, first generate reduced set ((3) solely RKME via Eq. (T) with = 24 which sketches the marginal distribution of the task feature. To encode the model's ability, pseudo labels can be assigned to the reduced set using yy = f(z,). resulting in
)= based
the labeled reduced set Rg (3, Z, Y) = {(3,, 2j: {).B))}}= serving as the developer-leve model specification gdev.
Specialized mechanism for classification tasks. For classification problems, given the finite label space V. we propose the mechanism to directly sketch the model's capacity, characterized by the conditional distribution P(X of the model f. We first obtain the model predictions {Ø: F on its "skilled" marginal distribution, i.e., its training data {-, Then, the pseudo-labeled dataset {(x; (:) H=) which encodes the model's conditional distribution, can be sketched by a labeled using the following objective:
reduced set Rr. (3, 2, = ((3,, Fzz W5)]
lo (I
- Σ j=1
Byk (zz,
Σ PET
ke (x;,))
Σ (zj, JEI
(2)
H2
where Ie and I' represent the indices of samples æ; and zj belonging to class c, respectively. is the parameter used to balance the marginal distribution distance and conditional distribution distance. The labeled reduced set Rg should approximate both the marginal distribution Σ 1 1 with Σ 350=; and the conditional distribution given the c class Σiet, n 1,5m; with Σjer, 8502, simultaneously. Here, 6-)) is the Dirichlet function, which describes the probability mass at single point. The objective Eq. (2) can be optimized by alternating optimization, the details are showed in E.1 The optimized reduced set R2 is served as the developer-level model specification dev
The first unified mechanism sketches the marginal distribution and then encodes the model informa- tion, while the second specialized mechanism directly sketches the model's conditional distribution.
5.2 The system accommodates heterogeneous learnwares
After the developer-level specification dev is generated, the developer submits the model f with specification gdev to the learnware dock system. The system exploits the relationship of different feature spaces and manages heterogeneous models by assigning system-level specification sy
Subspace learning After the learnware dock system receives heterogeneous models with their developer-level specifications, it generates unified subspace Xuh to connect different feature blocks {x based all developer-level specifications {a := [(Big. Zij, " During subspace learning, the learnware dock system learns 2Q mapping functions: {h+ Xx + Xsub 12
1.V.
5

---

and {gs= Kub + & For particular sample Zzy. it can be split into several blocks (=/) IRE, according to the feature split Xall 21 x ... Xq- The encoding function h; produces the embedding of sample slice zij as (h) and the decoding function g% reconstructs it to X'. as 2
(8)
The loss for subspace learning is implemented as follows: 1) The reconstruction loss, Crecommtuct I lff: penalizes the difference between the original sample 2 and the reconstructed sample =!) weighted by the sample importance Biz. 2) The supervised loss J.N. ) where Viy mean( (me) )LEC,), and calculating the prediction loss to make the embeddings of different classes more separable. 3) The contrastive loss aims to make the embeddings ( heee. of a single sample Zij similar, while ensuring that embeddings of different samples are dissimilar. The contrastive loss Leontrasive = Ex 2i=1 1 includes N terms, each term being weighted loss extended from the Self-VPCL loss [Wang (k)
ΣN E
(h) Zij
involves building a simple classifier on {[(mg) Viy))
(8)
(8)
and Sun. 2022], calculated on the embeddings (fub bREC m of one specification s!o:: 14 = ex t (vi) ()))
Σnt Big Exec. Evec. t log E Extec, exp & .01")
where vp the cosine similarity
function. In the logarithm term, the numerator represents the similarity of the positive pair, while the denominator is the sum of all pairs. The loss L is optimized with gradient descent.
Heterogeneous learnware accommodation. After subspace learning, the learnware dock system builds a unified subspace and corresponding mapping functions {g. h4 ) The dock system then assigns system-level specification 8, {(Bej. Vij. Miy) J for each model based on its developer- level specification x!0² == {(5bg, Zy: yig) ) During the system-level specification generation, the sample žij is projected the unified subspace as Miy, while the coefficient Bij and the label Viş remain unchanged. The projection Maj is calculated as follows: tij
[. Σkec. C. h& (2(g))
(K)). The is
whole procedure of learnware dock system construction is described in Algorithm
5.3
The user exploits the learnware dock system
After the learnware dock system accommodates heterogeneous learnwares, users can submit their task requirements to receive recommended models and the toolkit used for feature transformation. They can then directly reuse the model or combine with self-training model.
User requirement generation. As described in Section 4.2, comparing both the marginal distribu- tion Px and conditional distribution Pxly based on the model specification and user requirement helps better learnware identification. Similar to RKME, specification encoding both distributions, the RKME, requirement of the user generated similarly reflect both. In details, given the unlabeled data D = {x;} } Th and some labeled data D' = {(ē,, y:) W (the unlabeled data cover labeled data features, i.e., [i,)t S (x, }:), the user can generate RKME, requirement presented by labeled reduced set Rg- = {(8), zj Uj) j=1 to sketch the task distribution.
For the classification case, the reduced set Rt, can be generated by minimizing the following distance:
na
(x Σ (z).)) 1=1
Σ IE
(ā.,
B,k(zz))
TM
H
JEI
This equation is similar to specification generation in Eq. (2). where the specification sketches the lseudo-labeled dataset, while the requirement sketches semi- sunniiiueeviieed data. The first term calculates the distance between the marginal distributions of the unlabeled dataset Σ ne, 8m, and the reduced set プラ 3,02,. The second term calculates the distance between the conditional distributions of the
-
labeled dataset Eier, 1 ōR, and the reduced set ΣjET; 3,02,. where I. and I denote the sample indices of the labeled dataset and the reduced set with label c, respectively. The optimized reduced set R2 becomes the user-level requirement sfter The optimization is described in E.2
For regression, the requirement is generated by sketching the marginal distribution and applying a self-trained model for pscudo-labeling, similar unified specification generation in Section 5.1

---

Learnware identification. After the user submits the user-level task requirement sher {(hne: zok. You ||= to the dock system, the dock system transforms it into the system-level task requirement 80 = [[Pok; fok, Yos k== by projecting Zok into the subspace vok- The dock system then calculates the distance between the system-level specification 8; = (19g. Mijs Dir)) and the system-level task requirement 80 as follows:
Σ
mo
Book(VoRs :) - Bis((u)))))
m.
+a Σ
Σ |ETa.c
Book(VoR.))
- Σ Bigk(uss (4) jEI,cc
which measures both the conditional distribution distances and marginal distribution distances. Where Iicc represents the indices Vsj with the class c. The learnware dock system then recommends the learwware with minimal distance.
Learnware reuse. Once the user receives the recommended model fú and the dock system toolkit (hx. gk 12 they can apply the model to their task. The toolkit helps bridge the gap between different feature spaces. For example, if the user's task is Xi X2 and the model is X2 X3 x X4, the user can project their data using æı and h22 then decode it to A3 and N4 with Ya and J4. The user can the recommended model's predictions directly ensemble them with self-trained model.
5.4 Overall procedure
In the submission stage, the dock system receives models with developer-level specifications that sketch model capabilities and assigns system-level specifications by learned unified subspace. In the deployment stage, users submit task requirements detailing marginal and conditional distributions to receive recommended learnware. This learnware be integrated with their self-trained models to significantly enhance performance. The overall process is summarized in Appendix D.I
Experiments
6.1 Experiment setup
We tested our methods on 30 datasets from the Tabzilla benchmark [McElfresh et al. 2023). excluding tiny datasets. These include 23 classification tasks and 7 regression tasks. For classification tasks, the sample sizes range from 1,000 to 58,310, feature space dimensions from 7 to 7, 7,200, and the number of classes from 10. For regression tasks, the sample sizes range from 418 to 108,000, and feature space dimensions from 8 to 128.
Datasets.
Compared methods. As the heterogeneous learnware problem, where the user has some labeled data, is a new problem, we first compare our approach with two basic methods that train models from scratch: lightgbm (Ke al. 2017), widely used tree-based method for tabular datasets, and TabPFN (Hollmann et al. 2023). a recently proposed prior-data fitted network capable of training and inference small classification datasets in less than one second. When seeking assistance from the model repository, one simple but inefficient approach is to fetch all models, conduct heterogencous transfer learning, and select the best one. AligHumiabo Tann et al. 2024a] aligns the feature space and uses the aligned model directly, while Alignlabel (Tan et al. 2024a goes a step further by fine- tuning through training new model with augmented features that include aligned model predictions. Another method for reusing knowledge from heterogeneous tasks involves pre-training unified tabular network on different tables and fine-tuning on the downstream user tasks: Transtab (Wang and Sun) 2022) and Xtab [Zhu et al., 2023). However, these methods require access to raw task data, whereas our method protects user privacy. Next, we compare with Hetero [Tan al. 2023), an initial attempt to address the heterogeneous learnware problem without using label information. Finally, we substitute the specification in our method with the RKME specification from (Zhou and Tan 2024
Ourbasic and conduct comparison with the proposed method.
Experiment configuration. The feature space is randomly divided into four equal blocks, creating feature spaces from three-block combinations for developer tasks and two-block combinations for user tasks. For user tasks, 100 labeled data points are sampled from the training set. All experiments repeated five times. For more details, please see Appendix F.I
7

---

Table 1: Accuracy (%) (mean + std) on user data true labels. The best performance is in bold.
Dataset name
Lightghm
TabPEN
Ald
Transtab
Xtab
Hetero
OuT
Our
Ouraa
credit-g
674
70.5
58.3 2:
69.1
69 6
70.3 2:
67.5
71.2 1
semeion
54.8 2
634
6.8 119
59.6 1:
52.1 g::
27.0 g:14
40.2 a.e
53.2 p.
59.9 211
63.7 =:
mifeat-kathinenn
66.1
72.6
73.3
67.2
36.0
71.7
73.5
77.6 :13
splice
64.
63.2
42.5 u:1
56.2
72.8
56.0
72.2
gina agnostic
76.6
45.1 1:7
74.5 1
78.4 2:22
67.8 2:13
82.0 2:0
88.9 1
87.3 1:35
89,9 :13
Biorespomse
64,2
57.4
48.4
64.0
51.6
56.4
66.7
67.1
67.9
71.5
sylvine
=
714 201
47.2 2:41
69.2
69.6
66.6 277
75.1 200
1:17
2:1
christine
61.7 204
65.4 13
46.6 5:47
66.9 1m
$4.6
61.8 5:12
72.2 2.00
72.0 1:12
72.6 2:12
72.7 11
thereem- prowinn
42.1
42.5
27.0
39.5
22.0
40.8
44.3
51.2
50.4
51.0=1
satimage
80.5 1:
83.3 PM
6.3 1:44
81. 1.
83.5 1.
75.4 2:33
48.8 2.08
82.8 1:13
86.9 1:10
87.1 112
Fabert
23.4 o.
24.0 4:4
14.0 6-35
32.0 4:44
28.4 5:54
22.9 6:45
34,2 00
41.2 1A
41.0 1-30
46.0 22
gesture segmentation
40.7
37.7 134
2:34
38,6 2:27
robert
22.5
24.3 14
9.4 1:77
24.6 A
12.0 20
13.4 6:
45.5 4:08
44.8 9:00
45.3 1:42
45.7
112
ortffiia--chrneteer
25
31.6
8.5
20.2
21.2
21.1
414
40.1
41.4
41.8
eyc_ movements
42.8
42.1
213
39.7 1
39.2 2:13
nursery
59.1 mm
60.3 628
20.4 1:76
57.9 11
60.2 11
58.4 2:13
$7.9 2:00
62.2 PM
60.5 1:1
62.0 =1
cog-eye-stali
59.2
47.2
55.3
$3.7
58.4
63.2
64.4
63.1
65.9
magic
73.6
76.2 12
48.3 1:49
72.1 4:25
73.6 8:35
72.4 8:33
1:11
riccardo
73.1 1:
75.2 8.04
74,9 8:11
72.3 1:20
74,9 8:22
75.0 8:03
83.2 8:48
81.8 1:
83.1 1:11
81.9 tt
guillermo
$7.7
59.9
58.6
60.0
58.9
59.9
63.6
63.0
62.5
68.4
nomso
85.8 25
86.3
57.5
86.7 1:14
89.0 2:
82.2 2:13
79.9 2.0
89.6 2:11
2:11
89.5 211
click prediction
14
83.2 um
80.1 2:26
83.0 1M
83.2 AM
82.6 8:14
82.4 0:00
80.8 20
83.2 13
83.2 11
volkert
20
42.1
38.1
2:
47.9
46.0
Avg
0.577
0.599
0.353
0.581
0.572
0.530
0.593
0.652
0.061
0.674
Ouraniy w/Vl
23/0/0
20/1/2
23/00
25/000
22/1/0
23/0/0
17/1/5
13/0/10
4/1/18
Ouras willl
23/00
22/1/0
22/W11
LS/VV5
Avg. rank
6.826
4.826
9.826
6,565
6.174
7.696
5.261
3.391
2.522
1.435
Table 2: RMSE (mean std) on user data true labels. The best performance is emphasized in bold.
Dataset name
Lightgbm
Alig
Aligmahs
Transtab
Xtab
Hetero
Outbesic
Our
phc
0.411 1:00
0.507 1001
0.402
0.491
0.434
0.396
0.396
0.395
colleges
0.194 1m0
0.234 suee
0.194 ore
0.249 1:00
0.199 pses
0.232 1:00
0.196 c004
0.188 xmn
epu- gmulsmall
0.172 1:
0.229
0.168
0.238
2.614
0.211
0.132
0,152
kin8nm
0.178 ses
0.204 uuan
0.170 2004
0.242 1:000
0.179 sses
0.170 ==mm
0.172 cue
0.170 x.00
dataset-sales
0.072
0,086
0.068
0.464
0.079
003
0.061
0.058
califormia
0.172 6:1
0.251 2o0e
0.163 exda
0,217
a 0.00
0.169 6004
0.197 4:000
0.190
0003
0.163 4.822
aloi
0.303
0.327
0.301
0.322
0.301
0.249
0.224
0,235
Avg
0.215
0,263
0,209
0,318
0.568
0,223
0.196
0.194
Ouranity
7A00
5/2/0
7/V0
7/0N0
6/1/0
5/W2
Avg. rank
4.286
7.143
2,571
7.429
5,286
4,286
2.714
1.286
6.2 Performance on user tasks
Tables [aale][an]]] and compare the performance of our proposed methods with other contenders on classifica- tion and regression tasks. Ourunity refers to the performance of the overall procedure with the unified specification, while Our refers to the specialized specification designed for classification tasks, Our approach, Our Ourwiity- outperforms the competitors in most cases. While Lightgbm and TabPFN use self-training, their performance is limited by the small amount of labeled data. It is showed that TabPFN performs better than Lightgbm under these conditions. This highlights the importance of leveraging well-trained models, even with heterogeneous feature spaces, to improve performance.
Examining uniabbe shows that heterogeneous transfer learning with only aligning feature spaces without labels is less effective than self-training. However, further fine-tuning enables Al AligM1abel to outperform self-training methods. Nevertheless, without leveraging knowledge across different tasks, Alig" still performs worse than our approach. Transtab and Xtab attempt to create a unified backbone across different tables to leverage cross-task knowledge, but they fail to reuse the nigh-performing model on each developer task, leading to worse performance than ours. These methods also require training on raw developer data, whereas our method only accesses model specifications without exposing raw data.
Hetero performs worse than our methods due to its lack of modeling the conditional distribution of submitted models and its reliance on unsupervised subspace learning. Compared to Our our proposed specification outperforms the RKME specification, as it alleviates the issue of label infor- mation being overshadowed by feature information during specification generation and comparison. Notably, for classification tasks, our specialized model Our outperforms Ourunity due to its ability to encode conditional distribution of the model more effectively.
8

---

Info fofaakkaiinnnnnn
gna agrosiic
Biorespanise
Small dataset (x5k)
0.6
0.-
100
Dainoe Coire
0.6-
0.7
377
250
Selttrariig
Sellcrairing
Sellcninng
025
6.5
0.5
300
10
50
120
520
16
30
50
103
200
14
10
10
100
500
14
26
10
50
200
500
28
26
(Ammorr lididd
Namber uboled
of aboled dala
Number abcled dale
christine
eye movements
nursery
Mediam dataset (Se11.kk
0.50
0.55
0.6-
10.0
0.50
0.5
0.45
0.4
2.40
Eroembl
303
500
120
320 data
14
libieed
Number abeled data
magic
nomaoo
volkort
Large dataset (2. 6k-k)
0.60
0.33
5.65
0.4
2.50
Selicriring troembie
Ersamble
6.75
Erseno
100
120
Figure 4: User performance curve for classification tasks.
small
kinBnm
20602
0.43
Ensemblle
0.22
0.20
Ensemble
0:15
Ermemate
6.18
Sefiruingg Ensemble
0.40
50
10c
200
10
1:0
502
S
10
S
1:0
5op
1k
2k
24 Nurbber labe dota
50
50c
Number lbbe data
Number ofiabele data
Number of labeled data
dataset sales
ca lifooriia
aloi
Average performance
Geiimining Eisorible
Se Poriiiingg
Selt traritr Ensenilc
Ensentle
0.07
0.775
0.05
0.6
0.255
0.29
50
100
3k
120
50
102
Figure 5: User performance curve for regression tasks.
6.3 Evaluation on users with different size of labeled data
In the previous section, we showed that using a single learnware with heterogeneous feature spaces outperforms training models from scratch when labeled data is limited. Now, we analyze how performance changes as users train models and ensemble their predictions with learnware across different amounts of labeled data. Figures and tasks. These figures indicate that ensemble methods consistently outperform self-training with 100 labeled data points. With 500 labeled data points, the ensemble method still performs better in nearly 80% of cases. Additionally, learnware continues to enhance performance even with 5000 labeled samples, improving 21 % of classification cases and 50% of regression cases. For certain datasets like kin8nm in regression tasks, even when users use their entire training dataset, the recommended heterogeneous learnware still significantly boost performance.
these trends for classification and regression
7 Conclusion
This paper evolves specifications to unified subspace with explicit exploitation of model outputs under the heterogeneous learnware scenario. The specification is extended by additionally encoding conditional distribution to better encode the model capability, which can be further evolved by more effective subspace learning enriched by label information. The extended specification also improves learnware identification by additionally matching conditional distributions. We present the complete workflow of the learnware dock system accommodating heterogencous learnwares and validate the effectiveness of the proposed methods through extensive experiments.

---

Acknowledgments
This research was supported by NSFC (62250069) and the Program for Outstanding PhD Candidates of Nanjing University (202401 The authors would like to thank Jian-Dong Liu and Jia- Wei Shan for helpful discussions. We are also grateful for the anonymous reviewers for their valuable comments.
References
Zhi-Hua Zhou. Learnware: on the future of machine learning. Frontiers of Computer Science, 10: 589-590, 2016.
Zhi-Hua Zhou and Zhi-Hao Tan. Learnware: small models do big. Science China Information Sciences, 67(1):112102, 2024.
Zhi-Hao Tan, Jian-Donn Liu, Xiao-Dong Bi, Peng Tan, Qin-Cheng Zheng, Hai-Tian Liu, Yi Xie, Xiao-Chuan Zou, Yang Yu, and Zhi-Hua Zhou. Beimingwu: learnware dock system. In Proceedings of the 30th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pages 5782- 2024a.
Jian-Dong Liu, Zhi-Hao Tan, and Zhi-Hua Zhou. Towards making learnware specification and market evolvable. In Proceedings of the 38th AAAI Conference on Artificial Intelligence, pages 13909-13917, 2024.
Yi Xie, Zhi-Hao Tan, Yuan Jiang, and Zhi-Hua Zhou Identifying helpful learnwares without examining the whole market. In Proceedings of the 26th European Conference on Artificial Intelligence, pages 2752-2759 2023.
Yu-Jie Zhang, Yu-Hu Yan, Peng Zhao, and Zhi-Hua Zhou. Towards enabling learnware to handle unseen jobs. In Proceedings of the 35th AAAI Conference on Artificial Intelligence, pages 10964- 10972, 2021.
Patricia Biedermann, Rose Ong, Alexander Davydov, Alexandra Orlova, Philip Solovyev, Hong Sun, Graham Wetherill, Monika Brand, and Eva-Maria Didden. Standardizing registry data to the omop common data model: experience from three pulmonary hypertension databases. BMC Medical Research Methodology, 21:1-16, 2021.
Zifeng Wang and Jimeng Sun. Transtab: Learning transferable tabular transformers across tables. In Advances in Neural Information Processing Systems 35, pages 2902-2915 2022.
Bingzhao Zhu, Xingjian Shi, Nick Erickson, Mu Li, George Karypis, and Mahsa Shoaran. Xtab: Cross-table pretraining for tabular transformers. In Proceedings of 40th International Conference on Machine Learning, pages
2023.
Chang Xu, Dacheng Tao, and Chao Xu. A survey on multi-view learning- arXiv: 2013.
Weitian Huang, Sirui Yang, and Hongmin Cai. Generalized information-theoreet multi-view cluster- ing. In Advances in Neurul Information Processing Systems 36, pages 58752-58764, 2023.
Peng Tan, Zhi-Hao Tan, Yuan Jiang, and Zhi-Hua Zhou. Handling learnwares developed from heterogeneous feature spaces without auxiliary data. In Proceedings of the 32nd International Joint Conference Artificial Intelligence, pages 4235-4244 2023.
Bernhard Schölkopf and Alexander Johannes Smola. Learning with kernels: Support vector machines, regularization, optimization, and beyond. MIT press, 2002.
Xi-Zhu Wu, Wenkai Xu, Song Liu, and Zhi-Hua Zhou. Model reuse with reduced kernel mean embedding specification. IEEE Transactions on Knowledge and Data Engineering, 35(1):699-710. 2023.
Talip Ucar, Ehsan Hajiramezanali, and Lindsay Edwards. Subtab: Subsetting features of tabular data for self-supervised representation learning. In Advances Neural Information Processing Systems 34, pages 18853-18888 2021.
10

---

Dara Bahri, Heinrich Jiang, Yi Tay, and Donald Metzler. Scarf: Self-super contrastive learning using random feature corruption. In The 10th International Conference on Learning Representa- lions, 2022.
Wei Xu and Yihong Gong. Document clustering by concept factorization. In Proceedings of the 27th Anmual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 202-209, 2004.
Hao Wang, Yan Yang, and Tianrui Li. Multi-view clustering via concept factorization with local manifold regularization. In Proceedings of the 16th International Conference Data Mining, pages 1245-1250, 2016.
Duncan McElfresh, Sujay Khandagale, Jonathan Valverde, Vishak Prasad C, Ganesh Ramakrishnan, Micah Goldblum, and Colin White. When do neural nets outperform boosted trees on tabular data? In Advances in Neural Information Processing Systems 36, pages 76336-76366 2023.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems 30, pages 31 146-3154, 2017.
Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. Tabpfn: A transformer that solves small tabular classification problems in second. In The I 1ith International Conference on Learning Representations, 2023.
Yazheng Yang, Yuqi Wang, Guang Liu, Ledell Wu, and Qi Liu. Unitabe: Pretraining unified tabular encoder for heterogeneous tabular data. arXiv:2307.09249 2023.
Siyuan Guo, Jonas Bernhard Wildberger, and Bernhard Schölkopf. Out-of-variable generalisation for discriminative models. In The 12th International Conference on Learning Representations, 2024.
Peng Tan, Zhi-Hao Tan, Yuan Jiang, and Zhi-Hua Zhou. Towards enabling learnware to handle heterogeneous feature spaces. Machine Learning, 113(4)1139-1160, 2024b.
Lan-Zhe Guo, Zhi Zhou, Yu-Feng Li, and Zhi-Hur Zhou, Identifying useful learnwares for heteroge- neous label spaces. In Proceedings of the 40th International Conference on Machine Learning. 2023.
pages
David Alvarez-Melis and Nicolò Fusi. Geometric dataset distances via optimal transport. In Advances in Neural Information Processing Systems 33, 2020.
Facundo Mémoli. Gromov-wasserstenn distances and the metric approach to object matching.
Foundations of Computationall Mathematics, 11(4))417 487, 2011.
Lixin Duan, Dong Xu, and Ivor W. Tsang. Learning with augmented features for heterogeneous domain adaptation. In Proceedings of the 29th International Conference Machine Learning, 2012.
Chang Wang and Sridhar Mahadevan. Heterogeneous domain adaptation using manifold alignment. In Proceedings of the 22th International Joint Conference on Artificial Intelligence, 2011.
Oscar Day and Taghi M Khoshgoftaar. A survey on heterogeneous transfer learning. Journal of Big Data, 4:1 4,1-42,220 2017.
Han-Jia Ye, De-Chuan Zhan, Yuan Jiang, and Zhi-Hua Zhou. Rectify heterogeneous models with semantic mapping. In Proceedings of the 37th International Conference on Machine Learning. pages 5630-5639, 2018
Han-Jia Ye, De- De-Chuan Zhan, Yuan Jiang, and Zhi-Hua Zhou. Heterogeneous few-shot model rectifi- cation with semantic mapping. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(11))3878-388 2020.
Yao- Xiang Ding and Zhi-Hua Zhou. Boosting-based reliable model reuse. Proceedings of the 12th Asian Conference on Machine Learning, pages 145-160, 2020.

---

Yao-Xiang Ding, Xi-Zhu Wu, Kun Zhou, and Zui-Hua Zhou. Pre-trained model reusability evaluation for small-data transfer learning. Advances in Neural Information Processing Systems 35, pages 37389-37400, 2022.
Yi-Kai Zhang, Ting-Ji Huang, Yao-Xiang Ding, De-Chuan Zhan, and Han-Jia Ye. Model spider: Learning to rank pre-trained models efficiently. Advances in Neural Information Processing Systems 36, pages 13692-13711 2023.
Chao Yi, De-Chuan Zhan, and Han-Jia Ye. Bridge the modality and capacity gaps in vision-language model selection. arXiv:2403. 13797, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, The 3rd International Conference on Learning Representations, 2015.
12

---

Table of contents
Section
presents the key notations used throughout this work.
Section
offers an in-depth discussion on the proposed methods.
Section
reviews the related literature and techniques.
Section
D
describes the detailed algorithm of the overall procedure.
Section
B
explains the optimization during specification generation and user requirement
generation.
Section provides additional information on the experiments.
A
Notations
The major notations this paper are summarized Table
Category
Notations
Description
Xan
Nu
x XQ
the overall feature space and its Q blocks, the corresponding dimen- sions are d, d..... .dg.
T kinds of feature spaces for developers' task, each of them is the Cartesian product of several feature blocks. The index of blocks that 2 has is Ck. i.e., Ander XieG, X;. the feature space of the user's task, the Cartesian product of several feature blocks, the corresponding index of blocks is Co. i.e., u XxeG, the label space.
basic
Aueer
y
D. := {( Boy 1=
the labeled dataset of the i-th developer defined on X: x y where di E [1,.... , 7" represents the developer feature space index. the model of the i-th developer trained on D,.
developer
fi A + y
(Bag #cg) ymm
the developer-levef specification of the model generated from RKME..
D3 = amos Ja
the unlabeled dataset of the user.
user
(mos you) [i=]
the user's laheled dataset, which is of limited size the user-level requirement of the user generated from Do via RKME,
8 (1o0, 201) j==1
= (Bis, Masil 1
the system-level specification of the i-th model assigned by the learn- ware dock system, which is generated by adjusting the developer- level specification seev
learnware dock system
4 == (fi, 8;)
the i-th learnware accommodated by the learwware dock system. the heterogeneous learnware dock system. the system-level requirement of the user task generated by the learn- ware dock system.
(1) iml
so (18b), Voy mo
Xau h2 2 + Xauh
the learned subspace with dimension d the mapping function which projects the data on the k-th feature block X₂ to the subspace Xaa- the mapping function which reconstructs the data on the subspace Nur to the k-th feature block Xt.
subspace
Xanh + X
Table 3: Major notations of this work.
B
More discussion
B.1
Superiority of the learnware paradigm for handling heterogeneous models
Difficulties of managing models with heterogeneous feature spaces. To manage models de- veloped from heterogeneous feature spaces, it is essential to exploit the relationship among the corresponding feature spaces {_xd Jiz1. While multi-view learning [Xu et al. 2013) can be benefi- cial if co-occurrence data across entire feature space Xai is available, obtaining such data is nearly impossible in real-world scenarios. Alternatively, if the data of the model task is accessible.
13

---

training a unified tabular network on heterogeneous tables Wangg and Sun 2022 Zhu et al. 2023
Yang et al., 2023] offers another approach to reusing knowledge from heterogeneous tasks. However, a unified model often struggles to perform well across all source tasks due to complex and sometimes conflicting internal patterns. Additionally, sensitive areas like medicine, data sharing is restricted, and privacy concerns prevent access to raw data. In our problem, raw data is inaccessible to protect the model provider's privacy, and we do not use hard-to-collect auxiliary data.
Exploit feature space relationship with model specifications. Under the learnware paradigm, each model is submitted with a specification that describes its abilities. This specification can be naturally used to explore the relationships between feature spaces. [Tan et al. 2023] generates unified subspace Xua and linear projection functions linking it to all feature blocks { 1 by leveraging model RKME specifications generated solely on features. However, the specification lack of label information often leads unsatisfactory performance of subspace learning. like entangled classes of embeddings, it also performs poorly when feature blocks are weakly dependent. This paper better incorporates label information into the specification improve subspace learning for better heterogeneous learnware accommodation.
B.2 Exploit label information to handle models with heterogeneous feature spaces
To handle learnwares with heterogeneous feature spaces, it's crucial to exploit the relationship between different feature spaces. When the overall feature space divided into disjoint blocks and each task data is an arbitrary Cartesian product of several blocks, this exploitation can be divided into two parts: learning relationships between co-occurring feature spaces using data of specific specification, and learning relationships between non-co-occurring feature spaces across all specifications. The first part involves subspace learning to identify a unified subspace, while the second ensures that embeddings of heterogencous data with intersecting features are closely aligned in the subspace. The first part lays foundation for the second, which is key to managing models with heterogeneous feature spaces.
For the foundational step of subspace learning, if label information is not available, the embeddings of different slices of the same data may not align correctly within the subspace. This misalignment can result in embeddings from different slices having entangled or mixed classes. Furthermore, in extreme cases where feature blocks are jointly independent, subspace learning becomes meaningless due to the irrelevance of features. However, when label information is available, the feature blocks are no longer independent, as all information is used to generate the labels. More discussion that the label information is useful for building connections between independent feature blocks can be found in [Guo et al. 2024].
In summary, label information is essential for subspace learning, as it mitigate entangled classes of embeddings, which critically affects the learnware identification and reuse. It also help ensure performance even when feature blocks are weakly dependent.
C Related work
The learnware paradigm. The learnware paradigm [Zhou] 2016 Zhou and Tan 2024 offers a systematic approach to managing well-trained models and leveraging their capabilities to assist users in solving their tasks, rather than training model from scratch. A learnware consists of a well-trained model accompanied by specification that describes its capabilities. with this specification being the central component of the learnware, /W et al. [0033] proposed the RKME specification, which uses reduced set to sketch the distribution of the task data. Based on the RKME specification, Wu et al. 2023] proposed to match the data distribution for learnware identification, while Zhang et al. [202]] extended it to handle user tasks with unseen parts. To efficiently recommend Iearnwares among numerous learnwares, Liu et al. (2024) suggested evolving the specification with other learnwares for more accurate identifications and construct the specification index for managing learnwares for efficient learnware search, Xie et al. [2023) proposed using minor representative learnwares as anchors speed up learnware identification without traversing the whole system.
Previous research has primarily focused on the homogeneous case, where all models and user tasks share the same feature space. However, in real-world applications, the feature spaces of developer models and user tasks often differ. [Tan et al. [2024b) was the first to consider the heterogeneous feature space scenario, but it assumes that the original training data is accessible, and auxiliary data
14

---

across the entire feature space is collected. To relax this strong assumption of data accessibility, Tan al. [023] investigated the organization and utilization of heterogeneous learnware dock system without requiring access to the original data or auxiliary data across the feature space. While this approach is more realistic, its lack of effective use of label information leads to unsatisfactory performance. This paper examines the importance of label information and integrates it throughout the entire process of the heterogeneous learnware dock system. As a broader impact, the detailed implementation of incorporating label information into the learnware specification can help enhance various aspects of the learnware paradigm. In addition to research on heterogeneous feature spaces, Guo et al. 2023 considered scenarios involving heterogeneous labels.
Based on above research, the first learnware docking system, Beimingwu (Tan al., 2024a], was recently released. The system streamlines the entire learnware process and provides a highly scalable architecture, facilitating future algorithm implementation and experimental research.
Related techniques. To measure the distance between two labeled datasets in the same feature space, Alvarez- and Fusi (2020) proposed the Optimal Transport Dataset Distance (OTDD). This approach separately calculates feature and label distances using optimal transport and then combines them. The label distance is derived from the feature distances of partial samples with specific labels. This method aligns with the proposed loss for sketching the labeled dataset generate specifications and requirements, where the loss consists of feature and label components based on MMD, with the label loss defined by the conditional distributions P(X YY) Comparing simply concatenating feature and label, separately tackling feature and label can better measure the distance without label information overwhelmed by the longer feature information. For measuring the distance between two distributions in different feature spaces, Mémoli [201] proposed the Gromov-Wassersee distance, which aggregates all distances of tetrads to measure the distances between two points. In our work, we introduce method to measure the distance between two labeled datasets in heterogeneous feature spaces using subspace learning and maximum mcan discrepancy (MMD) over marginal and conditional distributions.
Existing studies on heterogeneous feature spaces, such as heterogeneous domain adaptation (Duan 2011]. heterogeneous transfer learning (Day and Khoshgoftaar 2017]. and heterogeneous model reuse et al. 2018 [020], generally map different feature spaces to an intermediate subspace. This process typically requires original data from both domains or co-occurrence data to establish the relationship between different spaces. However, in the learnware paradigm, managing models developed from different feature spaces without auxiliary data becomes feasible due to the existence of RKME specifications associated with each model. Based on this paradigm, we can accommodate, identify, and reuse heterogencous models of any type without accessing original data or additional co-occurrence auxiliary data. Recently, Guo et al. [2024] explored the relationship between two intersecting feature spaces from causal perspective, showing that residuals from model predictions can provide information into unobserved variables, specifically, the partial derivative of the true generating function with respect to these unobserved variables. This finding aligns with our approach, where we leverage label information to explore the relationship of different feature spaces.
et al. 2012 Wang and Mahadevan
Recently, some works have focused on identifying and reusing models from a model hub. Ding and Zhou| (2020) select models based on an anomaly detector associated with each model, which helps determine whether a feature sample is appropriate for prediction with that model. Ding et al. [2022] propose selecting models using task-model metric that requires only minimal interaction with data providers. Zhang et al. [2023] propose selecting pre-trained models by calculating the similarity between learned model embeddings and task embeddings, both of which are obtained through the ranking loss. Yi et al. [2024] explore the model selection specifically for visual language models. However, it is important to note that none of these approaches can directly address scenarios involving heterogeneous feature spaces.
D Algorithm details
D.1 Summary of overall procedure
The overall procedure of the heterogeneous learnware dock system consists of two stages. In the submission stage, the dock system receives models with developer-level specifications sketching
15

---

model capabilities and assigns system-level specifications using learned unified subspace. In the deployment stage, users submit task requirements detailing marginal and conditional distributions to receive the recommended learnware. This learnware can be integrated with their self-trained models to significantly enhance performance. 'he detailed procedures for each stage are outlined in Algorithm and 2 respectively.
Algorithm 1 Submitting stage (Jearnware accommodation by the system)
1: Each developer trains a model fi and generates the developer-level specification 6!t" on the dataset D, defined on Xddev y.
2: Each developer submitts both the model and the developer-level specification (fs, sf to the learnware dock system.
3: The learnware dock system generates the projection function h; : X₂ Xaub reconstruction function Auu > X; for each feature block X; and the system-level specification s; for each model f; based on all developer-level specifications {ster) Timl-
4: The model f: is accommodated by the learnware dock system with system-level specifications as learnware L; == [ft.s:):
5:
The heterogeneous learnware dock system is established as (4:)
Algorithm Deploying stage (system exploitation by the user)
1: The user generates the user-level task requirement sü based on the unlabeled data Dø and limited labeled data Dj defined on Xpver x V
2: The user submits the user-level requirement s/) to the learnware dock system.
The learnware dock system uses the projection functions (h,} to generate the system-level requirement 8g.
4: The learwware dock system recommends one learnware 4 based on the system-level requirement Sy and the system-level specifications { and provides the toolkit (g., h4) to the user.
5: The user reuses recommended learnware I with the toolkit {g:, h: on the task.
D.2 Subspace learning and system-level specification generation
After the developer-level specification dev generation, the developer submits the model with specification gdv to the learnware dock system. The system exploits the relationship of different feature spaces and manages heterogeneous models by assigning system-level specification s The details of subspace learning and system-level specification generation described in Algorithm
E
Optimization
E.1
Optimization of specification generation specialized for classification tasks
The RKME, specification represented by Rg (,, Z, [(8m, 2m Mo km))) sketches both the marginal distribution Px of the training data and the conditional distribution Pxty of the model prediction, which is obtained through minimizing the following objective over R,;
N
M Σ m=1
(Ins))
Bmk (Zm:
+
Σ
Σ MET
()
Σ Bmk (Z mEI'
(5)
He
Hu
This objective consists of two parts: the first term sketches the marginal distribution Px of the training data, while the second term sketches the conditional distribution Pxxy the model prediction. This objective can be optimized by iterative optimization, which is detailed in the following.
16

---

Algorithm 3 System-level specification generation
Input: All developer-level specifications { N
Hyper-Parammetrs: batch size B, temperature т, corruption rate c, max iteration T, trade-off parameters a, 022, ag 04
Output: System-level specifications "= learnware dock system toolkit {t; == (his 9i) system engine F-)-
1: while Max epoch not achieved do
2: Initialize the system engine F() and dock system toolkit {t; : (h4, 9:))
3: for each specification 8 in developer-level specifications {8 Ji=1 do
4: for sampled
{ (z yy )};= do 5: split æj according to the feature blocks {X. {ay let o
(8) JKEC.
(k) h4 (x() (.)). # embeddings of each blocks.
6:
exp vi.vt')
7:
define Leont E. &
Σ ky
log
where vs is the
2j-1 Eitzr exp vi (v2.))))
cosine similarity function.
let { 9:(0) (4)) and concatenate them as ã.. reconstructed samples.
8:
9:
define Freconstructed Et EKEc $ B31|a, - 2 111
2|2
let ún F(u,) where Fi = mean( (0) (k) /EE) # simple classifier prediction. B
11:
define Lsupervised Σラ= By[(yj. új) where / is cross entropy for classification problem
and squared error for regression problem.
12:
== + 0: Creconstruct + Ory Countaatiievee + ayySsueeriieedd
13:
update the encoder networks { decoder networks {g and system engine F(-) according to using gradient descent.
14:
end for
15:
end for
16: end while
17: generate system-level specification s²" = {B,, (w). based on the developer-level spec- ification eye {3,, (z, - by replacing the raw sample æ, with the embedding Vy.
18: return System-level specifications { with corresponding toolkit {t: == (h2 9,)) consists of encoder networks and decoder networks, system engine F(.).
Denote B = (B1, ,B,) Z = f=1, ,,M) and Y (y1,... .yM ), expanding Eq. gives
N
M
N
M
(18, = Σ
k (In, 3mm
Bmpm k (Z,: 2m) -
k (x 2m)
N2
N
n,m
m,m==
n= m=
3m ls (Inr 2m))
Σ
(Iny Rm) +
Σ
Br8mk (2ng Σ
Σ
N2
N
a.MEZ.
n.mEZ
BEZ. MEI
The distance F(B,2,Y also be rewritten as
a T KKank + 8 KzaB 8- 2a Kr.3 +0 Σ
T Kagn oe B Ka 20 3o)
aml
-K + +KK Ja 3T (K TT 0Kh)3 )3 20 T(Kaa OKK.))
=a + 3T K 8 2a K
where is the vector containing N elements of
Next, we address the optimization of Eq. (5) First, we generate Y while preserving the class ratio of the original labels {yn 1R-1- Then, proceed to optimize 3 and Z.
17

---

Fix Z and update 3. Suppose vectors in Z are fixed, setting 2F(3.2) 8p
0 obtains the closed-form
solution of B3 using pseudo-inverse of K2,-
B = (K 22)
)
'' a.
Fix 3 and update Z. When f is fixed, {zı, =MM in Z. are independent in Eq. 5) can iteratively run gradient descent on each Zm as
therefore we
@F(B77)
(t) 2(t-1) 11
Ozm
We first review the gradient 2G(8.2)
(G(
Ōk(zn: 3m))
N
Bm ak(In, 2m)
8m
0zm
Ozm
N
Ozm
n=1
n=1
M Σ
N
Bm
Bmom (-27k((zn 3m (2m -2n))
2
22 2m) (2 - xnn)
N
n=1
n=
N
1
Bak(zn; Zm) - =nn
k(@n, 2m (2m vkn))
N
==
then, we consider the gradient in our problem ar(3,2) 0=m
which is calculated as follows:
0k(zn: 2n, 2m
Bm
Ok( Ok( Dn: Zm)
2
Bm
2zm
N
Ozm
Ōk(zn Zn; Zmm)
Bm (n 2m)
+ 0
2
3zm
N
2zm
nET.
NEI.
M
N
Bm
= 2
(-2gkk (mm 2n)) - Σ
(-2rkk(an 2m (Zm xxn))
N
n=1
n=1
Bm 2m2 zn))
+
2
2mm 2n)) -
N
nEI.
nET
1
4m
B Enz Σ
2m 2mm
2n)
Σ
k(an; Zm
(2m Zmm Inn)
N
+
Bnk (Zn: Zm 2m) (2m -xx)
1
Σ
klØns 3m m2m Inn
N
nEZ'
M
4m
Σ
Ba(1 + 01(y.,, = c)) Zm (2m - nn)
n=1
+ Ollyya.
= c)
N Σ
k(æn, 2m (zm xnn)
N
n=1
where I, is the sample indices of class of sample 2m..
E.2 Optimization of user requirement generation
When the user has major unlabeled data n ( and limited labeled data DJ = (( y/: No (We adjust the notation slightly for clearer description of the optimization process), we can also
Na
18

---

find a weighted labeled set of points R = (B,Z.Y) = {m Zm. Vm m=1 to sketch the user data as task requirement.
The weighted labeled set can be generated by minimizing the following distance:
N
M
Σ
Bmk (Zm;")
Σ nEZc
Ni b
(Tn,
Σ
Bmk (Zmi
(6)
Nu
PdN
MEI.
Hh
which can be optimized by iterative optimization. The details are described follows.
Denote
.Ba)). (31,-..
Z {-1.--- ..M} and Y {y11-.. .yM expanding Eq. 66 gives
Nu
M
No
M
Bm
F(8, ZYY)
Σ
Nu 2
(Din,
Fmm) +
Bunmk & (Zn, 2m) - 2
Nu
k (Din 2mn)
n,m==
n,
== m=
Σ
( 3m)
Σ
BaBmk (Z 2m) 2m) Σ
Bun k (#ns Zm)
No
n,,ELL
n.MEZ.
nEI, MET.
The distance ((3,2,Y) can also be rewritten as
a a 3 K 2a K 3 Σ (ā Ka. S ā B Kr 2ā K K Bc)
=aa (an +0 K Kb + 3 (K₃z + 0k* 2(a Kgz + 0ā kg -aa"Kan 0ā a) + 3d (K 2(a² Kaz 0ā K22))
where a is the vector containing N, elements of 1, ā the vector containing N₁ elements of 1.
Then, we discuss the optimization of Eq. (6). We first generate Y, which will keep the class ratio of Then we optimize B, Z.
{yn I.
Fix Z and update Suppose vectors in Z are fixed, setting 0F(3.2) 8g
obtains the closed-form
solution of B using pseudo-inverse of Kzs:
B (K ( (K + OK a).
Fix 3 and update Z. When B3 is fixed, f21, [MM] } in Z are independent in Eq. 6 therefore we can iteratively run gradient descent on each 2m as
(t- (F(B2Z)
3zm
The gradient in our problem
(F(3.2) iz..
can be calculated as:
0F(3,2)
M
Ōk(zn: 2m)
Nr
Bm ake (alaann Zm
0zm
3zm
N
3zm
n=1
n=1
-
Σ
Ba (1+ + 01l(y)z = c))k(zz 2 (2m - 2n)
===1
0
k(#n, Zm ) Fnn)
k(@n Zm 2mm) mmn) Σ
N
NEI.
where I. represents the indices of labeled samples ã,, belonging to the same class as 2m.
E.3
Summary
The generation mechanism for model specification and user requirement is summarized in Figure
19

---

Specification generation
sketch Px
{(ims Amo mi reduced set
original data
model
model
capsert
(at,, 8hs Ja
(10m Ares Bos mus labeled reduced set
pacudo-labode
sketch Px. Pxt
Requirement generation
(megrescion)
sketch Ps.
[[5m Amm)) mm) reduced
D (Zt Iict DV { Jum1
model f trained on D' 0
user data
sketch Px. Pxix
{(Pm fim Mon Mf labeled reduced
(clasiificatoo)
Figure 6: summarized mechanisms for RKME, generation.
F Experiments
F.1
More details for basic information
Dataset details.
The basic information of used datasets is summarized in Table and Table 5
Table 4: Details for classification tasks
Dataset name
#elnsses #features
#instances
openml openmi__credit _credittgg___... 31
20
1000
openml semeion 9964
10
256
1593
openml mieat karhumen
10
64
2000
openmi splice
openml
ama__agnoticc 3891
970
3468
openml
Rioresponse 9910
1776
3751
openml
sylvine
openml christine 168908
1636
5418
openmul first theoreee 9985
6
51
6118
openmi satimage 2074 2074
36
6430
openml fabert 168910
7
800
8237
openml
GesttnnPinneSggmenaiiiooeeeeeee 14969
5
32
9873
openmi robert 168332
10
7200
10000
openmi churiiiiall-cuuaaeeers.... 14964
openml movements.......m 3897
3
27
10936
openml nursery 9892
4
8
12958
openml eeg eye state
openmi magic 146206
1
10
19020
openmil riccardo 168338
1
4296
20000
openmi guillermo 168337
4296
20000
openml _ooaao 9977
118
34465
openml Sellk__redictton_ssn small 190408
11
39948
opennil volkert 168331
10 180
58310
Experiment configuration. Each dataset is split into training and test sets with 4:1 ratio MccE fresh et al. [023). The output of regression tasks scaled to [0,1]. The feature space is randomly divided into four equal blocks. We create four feature spaces for developer tasks from all three-blocd combinations and six feature spaces for user tasks from all two-block combinations. Our encoder, decoder, and system classifier are two-layer ResNets (He et al. 2016) for tabular data, with subspace and hidden layer dimensions set to 16 and 32, respectively. We optimize using Adam (Kingma and Ba 2015 For user tasks, sample 100 labeled data points from the training set, using stratified
20

---

Table 5: Details for regression tasks
Dataset name
#classes
#features
#instances
openml
pbc 4850
4850
19
418
openml
colleges 359942
44
7063
opennl
12
8192
openml
kin8nm 2280
8192 8
openml
dataset 19041__190418 190418
14
10738
openml california 361089
8
20640
openml aloi 12732
128
108000
sampling for classification and binning for regression. The coefficients for contrastive, reconstruction and supervised losses set to 100, and 1, respectively. Developers train the model using LightGBM Ke et al. 2017] with grid search. All experiments are repeated five times.
Model training details. Tree-based models provide strong performance and high efficiency for supervised tabular tasks, so we use the popular efficient tree ensemble method LightGBM (Ke et al.) 2017] to train developer models and include it as a comparative method. The hyper-parameter search space used for the developer and user LightGBM models consists of list of specific combinations over parameters learning_res rate, num leaves and max max_depth: (0.015, 224, 66), (0.005, 300, 50), (0.01, 128, 80), (0.15, 224, 80), and (0.01, 300, 66).
Baseline details. More details of deep tabular network are described as follows:
TabPFN: For TabPEN, use the official checkpoint pre-trained model and further fine-tune it on downstream datasets. When testing TabPFN on datasets with more than 1,000 instances or 100 features, we randomly sample up to 1,000 instances and 100 features from the full training set, repeating this process three times. The final output is obtained by averaging the predictions from these trials.
Xtab: We utilize the pre-trained backbone with the most iterations from XTab's s official implemen- tation. Since this model is based on the Trannossfrrmer from AutoGluon, which currently does not support fine-tuning on tasks with more than 300 features, we generate random subsets of up 300 features from the overall training dataset, repeating this procedure three times. The final prediction is derived by averaging the results across three evaluations on the target datasets, using XTab's lightweight fine-tuning approach over 15 epochs.
Transtab: We employ the official version of Trans Tab v0.0.5, testing it in a contrastive learning setting. This involves initial contrastive pre-training the developers' training data, followed by fine-tuning user tasks. We adopt supervised contrastive learning objective during pre-training which has been shown to outperform the unsupervised version as noted in (Wang and Sun 2022]. For regression tasks, linear regressor is integrated during the fine-tuning phase. To manage memory constraints with large datasets, we subsample the developers' training sets 20,000 data points and 100 features.
Computation resources. Experiments were conducted using a Tesla A100 80GB GPU, two Intel Xeon Platinum 8358 CPUs with 32 cores each (base clock 2.6 GHz, turbo boost 3.4 GHz), and 512 GB of RAM.
F.2
All user curves for classification tasks
The all user performance curves for classification tasks shown in Figure
F.3
Ablation study
Ablation study for subspace learning loss by adding loss in sequence. We undertake an ablation study to evaluate the efficacy of our subspace learning procedure, as depicted in Tables 6 and 7 Initially, we utilize only the contrastive loss as the baseline. This helps in learning encoding functions for different feature blocks, ensuring consistency in embeddings across all sample slices. However, the reconstruction functions learned in this manner subpar. By adding reconstruction loss,
21

---

credit-g
semelon
infeat-kafiuunen
splice
0.77
0.5
0.70
Sattoraning Ensemmtle
Aocuncy
0.6
0.4
Accurey
0.5-
ACCURDYY
Selttrariing Creemble
Gelorriing Ensembtee
Sebtira Ring
Freembre
50
503
53
200
500
20
50
120
500
50
103
500
20
Nimberr baskled cwis
Simber ofleeeed dats
Number of wbeled diis
Number oflabeled dets
ginsa ainnossiic
Bioresporis
sylvine
christine
0.3
07-
0.
Satt-raanng Ensemile
Sel-tra Ersemble
Sattraiing Ensemitle
Set-tranto ring Ersenbre
0.5
50
120
sto
2k
50
100
102
20
50
120
500
1x
50
500
2t.
2
Number laseled daca
Nimber of data
fumber abeled dira
Number of dats
firs--rddor th thoieen-proinng
satimage
Tabert
0.-
0.5-
0.59
Sirriiinn Enperttte
Accuracy
0.
0.45
Set-orahin
Se8-tra
0.4
Set-tra rino Creembre
0.43
Ensemblee
Creemble
5t0
50
24
520 Number uieled
100
24
Number lasehed dosa
Number oflibeee data
Number oflieeleeee data
robsrt
artificia-charrctees
movements
nursery
Setttraning Ersembre
Accuayy
Seleyanns
Sefrrniing Ersomitle
Seltmarin Freemtre
Ensemnnle
100 520
520
Number lasefes duta
Number lobe
Number ubeled
Number allubeted date
reg-eye-statee
riccardo
guillermo
0.2
1.5
Seftraiiing Bnsemble
0.7
Accuneey
0.5
Settratiing Ensemile
0.70
Sel-trarinng
0.5
0.42
Set-tra
Ersembee
Ersembre
0.5
100
520
520
0.53
prediction small
volkert
0.00
0.53
Seltoraning Ereembre
0.3-
Sel-raiiin
Getorannng Eruencle
Eroemble
100
520
Figure 7: All user performance curve for classification tasks.
the performance of decoding functions is matched with the encoding functions. Despite the initial effectiveness of unsupervised subspace learning, further enhancements are achieved by incorporating label information through our proposed specification. With the addition of supervised loss, we ultimately attain optimal performance for both classification and regression tasks.
Table 6: Ablation study for subspace learning loss by adding loss in sequence for classification tasks.
Table 7: Ablation study for subspace learning loss by adding loss in sequence for regression tasks.
contrastive
reconstruction
+ supervised
contrastive
+ reconstruction supervised
acc(%)
61.1
66.2
67.4
rmse
0.208
0.200
0.194
F,4
Code availability
The code can be at htt.l/tt.o....cc.......o....e.ee
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims made in the abstract and introduction accurately reflect the paper's contributions supported by the remaining sections.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: This paper assumes that all models with heterogeneous feature spaces share the same label space. However, this assumption be further extended include heterogeneous label spaces as well. For example, basic way is to recommend multiple learnwares by distribution matching and reuse learnwares by dynamic classifier selection proposed in previous work [Wu et al. 2023).
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
23

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide a detailed description of the experimental setup Appendix F.1 The specifics of our algorithms are thoroughly explained in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
24

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
The link to the code can be found in Appendix F.4
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
public/guides
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7/nips cc/ wtc/publicc/ hoolllccggiiess. gwt.///ieee//eee// Codeeobmiiiie /esoc/uuuuie SanSSuubissss iolbbmmsssioo ssisnnoiccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: detailed description of the experimental setup is provided in Appendix E.T Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We report the standard deviations for performance comparison in our experi-
ments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
25

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report our compute resources in Appendix F.I
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https::/neurrpps.coom https//sssss//////oppooooooooaaa
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the
NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
26

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: This paper clearly cites the original paper that produced the code package or dataset.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
27

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
28

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "vjkLsu8d28", "reviewer": "Reviewer_NnRb", "summary": "The paper focuses on the learnware paradigm and finds that label information plays an important role in it, which is both practical and interesting. It proposes a new specification that enhances subspace learning and improves learnware management. Extensive experiments demonstrate the superiority of the proposed methods.", "questions": "Please see weaknesses.", "limitations": "The authors could clarify the limitations and future work for better understanding.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper is well-written and easy to follow.\n\n2.The paper addresses a significant gap in the learnware paradigm by proposing a method for handling heterogeneous feature spaces, a common real-world scenario.\n\n3.The experiments are thorough, covering both classification and regression tasks, and comparing the proposed method against a wide range of baseline methods.\n\n4.The code is provided in the appendix, enhancing the reproducibility of the results.", "weaknesses": "1.The method seems complex. Thus, a running time comparison is necessary to show its efficiency.\n\n2.I wonder if the method can be used for larger data to show its scalability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The method seems complex. Thus, a running time comparison is necessary to show its efficiency.**\n\nThank you for your suggestion. In the author rebuttal, we provided a brief discussion; here is a more detailed explanation.\n\n**1. Evaluation Setup**\n\nThe evaluation setup includes the preparation and utilization phases. Since the preparation phase is done once by non-user entities, we focus on the utilization phase time required for user tasks. The evaluation setup for each method is as follows:\n\n- **lightgbm**: Utilization phase only, training from scratch on user-labeled data.\n- **TabPFN, Transtab, Xtab**: Both phases. Preparation involves obtaining a pre-trained model; utilization involves fine-tuning on user data.\n- **Align_unlabeled, Align_labeled, Hetero, Our_unify, and Our_cls**: Both phases are involved. Preparation includes constructing and organizing a model hub, while utilization involves generating task requirements, recommending models, and reusing models on user data. \n  - Our_unify requires a self-trained model for pseudo-labeling during specification generation and is influenced by the model type and training parameters. Therefore, $\\textbf{Our}_{\\textbf{unify}}$ records the time assuming a pre-existing self-trained model, whereas $\\textbf{Our}\\_{\\textbf{unify}}^{\\text{lightgbm}}$ includes the entire process, including model training time using the time-consuming lightgbm method. Users can opt for a simpler model to reduce time costs.\n\n**2. Summary of Evaluation Results**\n- Classification Tasks:\n  - $\\textbf{Our}\\_{\\textbf{unify}}$ ranks 1st, $\\textbf{Our}\\_{\\textbf{cls}}$ ranks 3rd\n  - $\\textbf{Our}\\_{\\textbf{unify}}^{\\text{lightgbm}}$ ranks 7th (lightgbm ranks 5th).\n- Regression Tasks:\n  - $\\textbf{Our}\\_{\\textbf{unify}}$ ranks 2nd\n  - $\\textbf{Our}\\_{\\textbf{unify}}^{\\text{lightgbm}}$ ranks 4th (lightgbm ranks 3rd).\n\n**3. Analysis of Utilization Phase**\n\nCompared to other methods with $\\textbf{Our}_{\\textbf{unify}}$:\n- **lightgbm**: Requires training a model from scratch, whereas our methods reuse relevant heterogeneous models, making them faster.\n- **TabPFN, Transtab, Xtab**: Our methods do not need the entire labeled user data for fine-tuning, making them quicker.\n- **Align_unlabeled, Align_labeled**: Our methods do not traverse the model hub but directly find the most suitable model, making them faster.\n- **Hetero**: Similar process, but Hetero might recommend multiple models, whereas our methods recommend only one, making them faster.\n\nAmong our methods, $\\textbf{Our}\\_{\\textbf{unify}}$ is faster than $\\textbf{Our}\\_{\\textbf{cls}}$. $\\textbf{Our}\\_{\\textbf{unify}}$ compresses X first, then uses the model to predict on the reduced set, generating the specification quickly if user model is already prepared. $\\textbf{Our}\\_{\\textbf{cls}}$ predicts first, then compresses X and Y distributions simultaneously, making it slower. \n\n**By efficiently generating user task requirements and quickly reusing heterogeneous models, our method reduces the utilization phase time compared to other methods.**\n\n**4. Analysis of Preparation Phase**\n- **TabPFN**: Pre-training from simulated datasets generated by structure causal model, learning the posterior distribution $P(y\\_{test}|x\\_{test},D\\_{train})$ and applying it to small-scale real tasks. Pre-training time: 20 hours (8*RTX2080Ti).\n- **Xtab**: Learns a shared backbone network from many tasks; pre-training time unknown.\n- **Transtab**: Uses feature descriptions and values from related tasks to generate a unified subspace, training a shared Transformer backbone. Average pre-training time: 672 seconds (classification) and 1425 seconds (regression).\n- **Align_unlabeled, Align_labeled**: No model library organization time.\n- **Hetero**: Organization time: 2.44 seconds (classification) and 2.55 seconds (regression).\n- **Our_unify**: Organization time: 41.39 seconds (classification) and 40.78 seconds (regression).\n- **Our_cls**: Organization time: 45.69 seconds (classification).\n\nTimes are based on a single A100 GPU.\n\n**Compared to methods using pre-trained models, our methods significantly reduce preparation time by not requiring training on complete datasets but using specifications, which are smaller. We also do not train a unified backbone network, only performing simple subspace learning.**\n\n---\n\n**Q2: I wonder if the method can be used for larger data to show its scalability.**\n\n\nThank you for your question. Indeed, our experiments test datasets of various sizes and show that even with more labeled data, our paradigm enhances user performance. We also discuss the scalability of the learnware dock system with numerous heterogeneous learnware in the author rebuttal.\n\n**1. Our experiments test datasets of various sizes**\n\nWe tested datasets with varying sizes. For classification tasks, sample sizes range from 1,000 to 58,310, feature dimensions from 7 to 7,200, and classes from 2 to 10. For regression tasks, sample sizes range from 418 to 108,000, with feature dimensions from 8 to 128.\n\n**2. Our method boosts performance even with more labeled data**\n\nTables 1 and 2 in our paper show that with only 100 labeled samples, using the recommended heterogeneous model significantly outperforms user self-training. Figures 4 and 5 illustrate that combining the heterogeneous model with user self-trained models improves performance across different labeled data amounts. **On average, with 500 labeled samples, reusing the recommended model still outperforms self-training. With 2,000 labeled samples, the heterogeneous model continues to enhance performance. Even with 5,000 labeled samples, learnware improves 21% of classification cases and 50% of regression cases.** In some cases, like kin8nm, even using the entire training dataset, the heterogeneous model improves user performance.\n\n**3. The cost of organizing a learnware dock system with numerous heterogeneous learnware is relatively small**\n\nPlease refer to the author rebuttal for details.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3PjrhmyHbh", "reviewer": "Reviewer_8wdE", "summary": "This paper introduces a novel approach for utilizing heterogeneous models with diverse feature spaces in the learnware paradigm. Key innovations include incorporating label information to improve model specification and subspace learning, and constructing a heterogeneous learnware dock system that uses pseudo labels to encode model capabilities. Extensive experiments show that the recommended heterogeneous learnwares outperform user self-training with limited labeled data and improve task performance as more labeled data becomes available.", "questions": "None", "limitations": "None", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Addresses a practical and important problem of handling heterogeneous models with diverse feature spaces in the learnware paradigm, which is often encountered in real-world scenarios.\n2. Innovative approach of leveraging label information, including model predictions and user's minor labeled data, to enhance the model specification and subspace learning, going beyond previous methods that relied on raw data or auxiliary co-occurrence data.\n3. Thorough experimental evaluation on real-world tasks, demonstrating the significant performance improvements over user self-training with limited labeled data.", "weaknesses": "1. The potential challenges or limitations of the proposed model are suggested to be given.\n\n2. The paper does not provide a thorough analysis of the computational complexity and scalability of the proposed framework, which could be important for large-scale, real-world deployments.\n\n3. The experiments are conducted on a single dataset, and a more diverse evaluation of different types of tasks and datasets would strengthen the generalizability of the findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The potential challenges or limitations of the proposed model are suggested to be given.**\n\nThanks for your advice. We discuss the limitations of the proposed method in the checklist (see Q2: limitations).\n\nIn this paper, we consider scenarios where the feature spaces of models in the learnware dock system are heterogeneous. However, real-world applications may also have heterogeneous label spaces. We will add more discussions on this in the revised version and please see the author rebuttal for a brief discussion.\n\n---\n\n**Q2: The paper does not provide a thorough analysis of the computational complexity and scalability of the proposed framework, which could be important for large-scale, real-world deployments.**\n\nThe process of the learnware paradigm is outlined as follows:\n\n* **Building the Learnware Market:** Developers generate specifications, package them with models into learnware, and submit them to the learnware dock system. This system organizes the learnware and assigns system-level specifications to the models.\n* **Utilizing the Learnware Market:** Users generate requirements and submit them to the system, which then recommends appropriate learnware. Users can then reuse the learnware.\n\nBelow, we analyze the complexity of each process. The relevant symbols are defined as follows: the size of the raw dataset is $ n $, the number of categories is $ c $ (1 for regression), the dimension is $ d' $, and the size of the specification is $ m $. There are a total of $ N $ learnwares in the learnware dock system, and the complete feature space contains $ K $ sub-blocks with a total dimension of $ d $.\n\n**1. Specification/Requirement Generation**\n\n* **Initialization:** k-means clustering, with a complexity of $ O(nmcd') $.\n* **Iterative Optimization:** Achieved through alternating optimization, with the bottleneck being the inverse matrix computation of the kernel matrix, which has a complexity of $ O(cn^3) $.\n\nThe total complexity is $ O(c(nmd' + T\\_s n^3)) $. Typically, the number of iterations $ T\\_s $ is small, around 10, and the size of the specification $ m $ is much smaller than the dataset size $ n $. When the sample size $ n $ is large, the complexity of specification generation is $ O(cn^3) $, scaling cubically with the size of the raw dataset $ n $. However, **with GPU acceleration, the time required for specification generation is quite small. For the dataset openml__volkert__168331 with shape (58310, 180), the generation time of specification is 2.826s (A100 80G*1).**\n\n**2. Subspace Learning**\n\n* **Single Epoch, Single Specification:** The complexities for calculating the contrastive loss, reconstruction loss, and supervised loss are $ O(m^2 K^2 d) $, $ O(mK d^2) $, and $ O(mcd + md^2) $ respectively, totaling $ O(m^2 K^2 d + mK d^2) $. The complexity of updating the model using the loss functions (both are two-layer fully connected networks) is $ O(m d^2) $. Thus, the complexity for subspace learning with a single specification is $ O(m^2 K^2 d + mK d^2) $.\n* **All Epochs, All Specifications:** Considering all specifications and the number of iterations, the complexity for subspace learning is $ O(N T\\_{sub} (m^2 K^2 d + mK d^2)) $.\n\n**This complexity is linearly related to the number of learnwares $ N $ in the market and quadratically related to the size of the specifications $ m $，the dimension $ d $ of the feature space and the number of feature blocks $K$.** \n\n**3. Learnware Recommendation** \n\nThe time complexity is $ O(Ncm^2) $, **linearly related to the number of learnwares $ N $ in the market**.\n\n**4. Model Reuse** \n\nWhen reusing a model, it only requires mapping the raw data through a two-layer fully connected network to the feature space corresponding to the heterogeneous learnware and making predictions using the learnware, which has low time complexity.\n\n**5. Summary**\n\n**The most time-consuming procedure of the learnware paradigm is the subspace learning for heterogeneous learnware recommendation. Even with a large number of learnwares $ N $, the total training data $ Nm $ for subspace learning remains manageable due to the small size of each specification (typically $ m = 50 $). Furthermore, our methods train an encoder and decoder for each feature block instead of each model. Although the number of models can be quite large, the number of feature blocks is relatively small.** An extreme case involves $ K $ blocks, resulting in $ 2^K $ feature spaces and $ 2^K $ models for each feature space, our method trains $K$ encoders and decoders, not $2^{K}$.\n\nBesides the theoretical analysis, we will also provide the actual running time in the attached PDF to demonstrate the efficiency of the proposed framework. Our method have much less time for preparation (construct the learnware market) compared to the pre-training methods, and the time for utilization (recommend and reuse learnware) is also efficient. Please see author rebuttal for details.\n\n---\n\n**Q3: The experiments are conducted on a single dataset, and a more diverse evaluation of different types of tasks and datasets would strengthen the generalizability of the findings.**\n\nWe have conducted experiments on real-world projects, please see author rebuttal for details. Thanks for your suggestion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CP1sQ55wsU", "reviewer": "Reviewer_PBMH", "summary": "In this submission, authors focus on the learnware paradigm, which aims to help users leverage numerous existing high-performing models instead of starting from scratch. They find that label information, including model prediction and user’s minor labeled data, is crucial and previously unexplored and then explicitly explore this information to address the problem in handling learnwares from heterogeneous feature spaces.", "questions": "Please clarify the weaknesses mention above.", "limitations": "In this paper, authors discussed the limitations as follows:\n\nThis paper assumes that all models with heterogeneous feature spaces share the same label space. However, this assumption can be extended to include heterogeneous label spaces as well, through the use of multiple learnware recommendations proposed in previous work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Learnware is useful learning paradigm and handling learnwares from heterogeneous feature spaces has more wide application scenarios.\n\n2. To my knowledge, for handling learnwares from heterogeneous feature spaces, it might be the first attempt towards explicitly exploiting label information, including model prediction and user's minor labeled data. \n\n3. For this new setting, comparative studies are designed, and the experimental result validate the effectiveness of the proposed method.", "weaknesses": "1. To my knowledge, there have been some works aiming at handling learnwares from heterogeneous feature spaces:\n\n[a] Peng Tan, Zhi-Hao Tan, Yuan Jiang, and Zhi-Hua Zhou. Handling learnwares developed from heterogeneous feature spaces without auxiliary data. In Proceedings of the 32nd International Joint Conference on Artificial Intelligence, pages 4235–4243, 2023.\n\n[b] Peng Tan, Zhi-Hao Tan, Yuan Jiang, and Zhi-Hua Zhou. Towards enabling learnware to handle heterogeneous feature spaces. Machine Learning, 113(4):1839–1860, 2024.\n\nHowever, these works are only cited in reference list while not adequately discussed.\n\n2. The main contribution might be handling learnwares from heterogeneous feature spaces via explicitly exploiting label information. The current abstract does not focus on this contribution. Moreover, RKME is the most commonly-used specification in learnware, this paper is also based on RKME, so discussion on the newly proposed specification should be given in detail, especially the differences with existing RKME specification. If I misunderstand something, authors can rely in the rebuttal phase. \n\n3. It is mentioned that \"the recommended heterogeneous learnware significantly outperforms user self-training with limited labeled data\", it is similar to semi-supervised learning. Is it fair to compare the proposed method with user self-training with limited labeled data?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: To my knowledge, there have been some works aiming at handling learnwares from heterogeneous feature spaces. However, these works are only cited in reference list while not adequately discussed.**\n\nThanks for your careful reading. We will discuss more related papers in the revised version. Here is a brief discussion:\n\nThese works focus on **constructing and utilizing heterogeneous learnware markets in specific scenarios** with relatively fixed feature combinations, such as relational databases constructed from multiple related tables. For a specific task, the complete feature space consists of several blocks, and the user and learnware feature spaces are combinations of these blocks.\n\n- **MLJ 2024:** The **first work** to consider heterogeneous learnware. **The original training data is accessible, and auxiliary data across the entire feature space is collected.** The paper assigns specifications to heterogeneous models, learns the subspace using the model's original data and auxiliary data, and then generates the RKME specification based on the projection results.\n\n- **IJCAI 2023:** This study explores **organizing and utilizing a heterogeneous model hub without accessing the original data or auxiliary data across the feature space.** It performs subspace learning via the RKME specification. Developers upload models and specifications to the learnware market. The market learns the subspace and mapping function based on the specification, aligning learnware specifications with user requirements.\n\n- **This paper:** This work **identifies the limitations of the RKME specification,** noting its inability to effectively support subspace learning and model recommendation due to the lack of label information. Unsupervised subspace learning can mix data from different tasks and categories, leading to poor model recommendation and reuse. The RKME specification only describes the marginal distribution, matching models based solely on this without considering their capabilities. **This paper embeds label information and model capabilities into the specification** to improve the heterogeneous learnware market mechanism. This **allows subspace learning to incorporate supervised information**, better aligning samples from each class of different tasks. In model recommendation, the model's classification boundary is matched with the user distribution, enhancing the performance of the heterogeneous model hub.\n\n---\n\n**Q2: Discussion of newly proposed specifications**\n\nThank you for your advice. We will revise the abstract and provide a detailed discussion on the newly proposed specification in the revised version. Below are the major differences between the two specifications:\n\n- **Summary:** Specifications describe a model's ability without exposing raw data. $\\text{RKME}$ (TKDE 2023) sketches the marginal distribution of the training data but lacks label information, which is insufficient for handling heterogeneous learnwares. We extend $\\text{RKME}$ to $\\text{RKME}_\\text{L}$ by adding label information to better encode the model's ability through its conditional distribution.\n- **Representation Form:** The original $\\text{RKME}$ specification includes minor weighted samples $\\\\{\\beta\\_m,\\boldsymbol{z}\\_m\\\\}\\_{m=1}^M$. The extended $\\text{RKME}\\_\\text{L}$ specification includes minor *labeled* weighted samples $\\\\{\\beta\\_m,\\boldsymbol{z}\\_m,y\\_m\\\\}\\_{m=1}^M$, where $\\beta\\_m$ is the weight, $\\boldsymbol{z}\\_m$ is the generated sample, and $y\\_m$ is the label.\n- **Generation Mechanism:** $\\text{RKME}$ outlines the marginal distribution $P(X)$ of a dataset. In contrast, $\\text{RKME}\\_\\text{L}$ includes label information, outlining both the marginal distribution $P(X)$ and the conditional distribution $P(Y|X)$. $\\text{RKME}$ uses minor weighted samples to approximate the kernel mean embedding of the original data, with distance measured by MMD distance. $\\text{RKME}\\_\\text{L}$ can either compress the marginal distribution $P(X)$ first and then generate labels or compress both $P(X)$ and $P(Y|X)$ simultaneously based on model predictions.\n- **Impact on Subspace Learning:** With $\\text{RKME}$, the lack of label information can result in poor model recommendation and reuse as different task categories may overlap in the subspace. In contrast, $\\text{RKME}\\_\\text{L}$ allows for better alignment of samples from different tasks within the same class, leading to improved performance of the heterogeneous model hub.\n- **Impact on Recommendation Mechanism:** $\\text{RKME}$ only recommends models with a similar marginal distribution to the user task, ignoring the model's ability. $\\text{RKME}\\_\\text{L}$ enables the system to match the model boundary with the user task boundary by comparing the conditional distribution $P(Y|X)$, resulting in better model recommendation outcomes.\n\n---\n\n**Q3: It is mentioned that \"the recommended heterogeneous learnware significantly outperforms user self-training with limited labeled data\", it is similar to semi-supervised learning. Is it fair to compare the proposed method with user self-training with limited labeled data?**\n\nWe apologize for any confusion caused and appreciate your careful reading. We will address this in the revised version.\n\n**Our problem setup aligns with supervised learning; however, the user's training data may be insufficient.** In such cases, training a model with a small expected loss is challenging due to the lack of labeled data. By leveraging the heterogeneous learnware dock system to identify and reuse existing well-trained models, users can significantly enhance task performance, as demonstrated in the paper experiments.\n\n---\n\n**Q4: Extension for heterogeneous label space**\n\nThanks for your suggestion. Please see author rebuttal for a brief discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bWEPVKTH6Z", "reviewer": "Reviewer_HLyF", "summary": "The paper designs a learnware system to handle the heterogeneous feature spaces. It applies the label information to manage the heterogeneous models. Then it also proposes a strategy to match the model with the additional conditional distribution. The paper also conducts experiments to compare with other learnware systems.", "questions": "In Eq.(2), there is a term $L_{similar}$,  but in Section 5.2, there is a term called $L_{contrastive}$. Are them the same? Is this a typo or are they two different losses?", "limitations": "The paper claims the limitations in Checklist.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The observation that matching with only the marginal distribution is not enough is interesting and novel. The paper improves the matching with the conditional distribution to tackle this problem.\n\n2. The experimental results are good.\n\n3. The paper is well presented and easy to follow.", "weaknesses": "1. The paper focuses on the heterogeneous feature spaces in learnwares. However, it seems that the paper uses subspace learning to handle the heterogeneous feature spaces. Subspace learning is a standard and commonly used method to handle heterogeneous features. Therefore, the contributions to handle heterogeneous features of this paper seem limited. The proposed technology, such as to use the conditional distribution, seems to be irrelative to handling the heterogeneous features because it can also be used in a homogeneous setting. The paper should clarify the relations between each technical contribution and the heterogeneous feature spaces.\n\n2. In the experiments, the paper should compare with more SOTA methods which handle the heterogeneous features. In current version, most compared methods are not designed for heterogeneous features.\n\n3. In the experiments, the heterogeneous feature spaces are randomly divided from the original data sets. However, in real applications, the distribution of heterogenous features is not in this case. For example, in real-world applications, each feature space has its own semantics and is not random. Therefore, it would be better to conduct experiments on at least one real-world heterogeneous data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Clarify the relations between each technical contribution and the heterogeneous feature spaces.**\n\nThank you for your comments. We will provide a more detailed discussion in the revised version. The following is a brief explanation:\n\n**Subspace learning is a standard method for handling heterogeneous features. The critical aspects are from what to learn and how to learn it.** The learnware framework protects model providers' privacy without accessing their raw data and avoids collecting auxiliary data across feature spaces. **Without raw and auxiliary data, how can the learnware dock system effectively learn a unified subspace?**\n\nIn the learnware paradigm, subspace learning can be performed based on the RKME specification of the model. The RKME specification compresses the marginal distribution $P(X)$ through a small number of weighted sample points. However, **previous methods that lack label information rely on unsupervised learning techniques, leading to suboptimal subspace learning results**. Without label information, categories from different tasks may overlap in the subspace, impacting model recommendation and reuse effectiveness.\n\nTo address this, **we introduce supervised information into subspace learning to better align samples from each class of different tasks within the subspace**, enhancing the heterogeneous model hub's performance. Thus, we **extend the original RKME specification representation $\\\\{\\beta\\_m,\\boldsymbol{z}\\_m\\\\}\\_{m=1}^M$ to include label information**, obtaining $\\text{RKME}\\_\\text{L}$, represented as $\\\\{\\beta\\_m,\\boldsymbol{z}\\_m, y\\_m\\\\}\\_{m=1}^M$. We propose two methods: compressing the marginal distribution first and then generating the label, or **simultaneously compressing both the marginal distribution $P(X)$ and the conditional distribution $P(Y|X)$**. By incorporating label information, we **add supervised term for subspace learning**, obtaining a subspace with better properties and corresponding mapping functions to original feature spaces.\n\nIn summary, our main techniques are designed to enhance subspace learning.\n\n---\n\n**Q2: In the experiments, the paper should compare with more SOTA methods which handle the heterogeneous features. In current version, most compared methods are not designed for heterogeneous features.**\n\nThank you for your comments. **Indeed, most of the compared methods are designed for heterogeneous features and are SOTA methods.** We will describe the contenders more clearly in the revised version.\n\n**This paper explores using knowledge from tasks with heterogeneous feature spaces to improve user task performance.**\n\nFirst, we evaluate user self-training performance using LightGBM with the same training setup as the learnware preparation for fairness.\n\nNext, **we compare our approach with recent SOTA methods that use the heterogeneous model hub for the user task**:\n\n- **Align_unlabel (KDD 2024):** Minimizes MMD distance between the user task and learnware task based on RKME. We modified it to traverse the best model in the hub, improving performance.\n- **Align_label (KDD 2024):** Builds on Align_unlabel by using user-labeled data for feature augmentation, further improving performance. We also traverse the hub for the best model.\n- **Hetero (IJCAI 2023):** Generates a subspace for model recommendation/reuse using basic RKME without label information, creating a linear mapping function.\n\nThese solutions learn feature space mapping without label information. Our paper fully utilizes label information, resulting in significantly better performance.\n\n**We also compare with other SOTA methods that use raw heterogeneous task data, compromising privacy**:\n\n- **TabPFN (ICLR 2023):** Uses a Transformer to fit the posterior predictive distribution from synthetic tasks and applies it to real-world tasks. It outperforms other deep methods in the small data paradigm.\n- **Transtab (NeurIPS 2022):** Uses feature descriptions and values to generate a unified subspace and trains a shared Transformer backbone, pioneering deep network training across heterogeneous tasks.\n- **Xtab (ICML 2023):** Trains a shared backbone for different tasks with a specific input processor and prediction head, without using feature descriptions, offering flexibility in real-world scenarios.\n\n**In summary, the contenders are highly relevant to heterogeneous feature spaces. We compared not only with SOTA methods that exploit the heterogeneous model hub but also with other SOTA methods that use raw data from heterogeneous tasks for new user tasks.**\n\n---\n\n\n**Q3: Conduct experiments on at least one real-world heterogeneous data.**\n\nThanks for your suggestion, we add **two real-world projects** to demonstrate the efficacy of proposed method, please see author rebuttal for details.\n\n---\n\n**Q4: In Eq.(2), there is a term Lsimilar, but in Section 5.2, there is a term called Lcontrastive. Are them the same? Is this a typo or are they two different losses?**\n\nWe apologize for any confusion caused. Thanks for your careful reading. \n\n**In short, $L\\_{\\text{similar}}$ is a general term for similarity loss, and $L\\_{\\text{contrastive}}$ is a specific implementation.**\n\nIn Section 4.2, we define the general objective for subspace learning as Equation (2), based on all model specifications $\\\\{\\boldsymbol{s}\\_i^{\\text{dev}}:=\\\\{(\\boldsymbol{\\beta}\\_{ij},\\boldsymbol{z}\\_{ij},y\\_{ij})\\\\}\\_{j=1}^{m\\_i}\\\\}\\_{i=1}^N$:\n$$\\min\\_{\\\\{h\\_k,g\\_k\\\\}\\_{k=1}^Q} L=\\alpha\\_1L\\_{\\text{reconstruction}} + \\alpha\\_2L\\_{\\text{similar}} + \\alpha\\_3L\\_{\\text{supervised}}.$$\nThe similarity loss ensures that embeddings of different slices of the sample $\\boldsymbol{z}_j$ are similar. This can be implemented in various ways, such as through contrastive loss, manifold loss, etc.\n\nIn Section 5.2, we provide a detailed implementation of the loss function, specifically using contrastive loss to measure similarity, which is a quite popular loss for self-supervised learning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

**Please see the attached one-page PDF with a summary of additional experimental results regarding time analysis and performance on real-world projects.**

We would like to thank all reviewers for their constructive feedback, which has greatly improved our paper. We are encouraged by the reviewers' positive remarks, including:

* The **practical and important problem** of handling learnwares from heterogeneous feature spaces (Reviewer NnRb, 8wdE) and the recognition of learnware as a useful learning paradigm (Reviewer PBMH).
* The **novel observation** that matching with only the marginal distribution is insufficient (Reviewer HLyF).
* The well-written and **easy-to-follow** paper (Reviewer NnRb, HLyF), with thorough (Reviewer NnRb) and good (Reviewer HLyF) experiments .

---

We have diligently worked on addressing your suggestions and have summarized the additional experiments and analysis below.

**1. Running Time and Scalability**

**Running Time.**  Our experiments demonstrate the efficiency of our method through comparative analysis (see Tables 1 and 2 in the attached PDF). The evaluation is divided into the preparation phase (pretraining a model/building the model hub) and the utilization phase (fine-tuning the pre-trained model/recommending and reusing models), with a primary focus on the latter as it involves user tasks. **When a user has a self-trained model for pseudo-labeling a reduced set during specification generation, our method** $\textbf{Our}\_{\textbf{unify}}$ **ranked 1st in classification tasks and 2nd in regression tasks.**  Even when including our method with a self-training procedure using the time-consuming LightGBM, $\textbf{Our}\_{\textbf{unify}}^{\text{lightgbm}}$, it is still much quicker than most pretrained methods (Transtab, Xtab) and the model hub method that requires fine-tuning the model $\textbf{Align}\_{\textbf{label}}$. **The efficiency of our method is heavily based on specifications, which are much smaller than the original dataset.**  Specifications quickly sketch the user task and help match the relevant heterogeneous learnware, while the mapping functions learned during the learnware dock system construction help quickly reuse heterogeneous learnware. **All procedures avoid using the whole dataset for fine-tuning.** In the preparation phase, our method is much quicker than pre-trained methods (detailed results in the Response to Reviewer NnRb), because our method only reuses specifications for subspace learning with simple mapping functions, rather than training a shared, complicated Transformer-based backbone with raw data.

**Scalability.**  In this part, we talk about the case when the learnware dock system have too many heterogeneous learnware. The most time-consuming aspect of the learnware paradigm is the subspace learning required for heterogeneous learnwares organization. **Even with a large number of learnwares $N$ ​, the total training data $Nm$ for subspace learning remains manageable due to the small size of each specification (typically $m=50$​ ).** Additionally, our methods **train an encoder and decoder for each feature block rather than for each model**.  Despite the potentially large number of models, the number of feature blocks is relatively small. For $K$ blocks resulting in $2^K$ feature spaces and models, our method trains only $K$ encoders and decoders, not $2^K$. This also helps address user tasks with combinations of feature blocks not present in the learnware dock system.

**2. Performance on Real-World Projects**

We tested our method on two real-world projects: a hot sales-forecasting competition (Predict Sales Forecasting, short for PFS) on Kaggle, a regression task with raw data containing 6 tables, and a widely used large clinical database of critical care units called MIMIC-III [Johnson etal., 2016], which contains 26 tables. 

For PFS, we selected a popular feature engineering in the competition, and for MIMIC-III, we used the MIMIC-III benchmark [Harutyunyan et al., 2019]'s preprocessing and selected the in-hospital mortality task, a binary classification task. For processed data, we split the feature space according to its semantics for PFS and randomly split for MIMIC-III due to its large feature space dimension (714). For PFS, we further split the data by location, as sales forecasting is evaluated locally. The data for each location is at the hundred thousand level. 

**Our methods ranked first, demonstrating superior performance over other contenders in both real-world medical classification and business regression tasks.**

---

Finally, we discuss the extension of our method. Although our method is designed for heterogeneous feature spaces, it can also be naturally extended to handle both heterogeneous feature and label spaces. We give a brief discussion as follows:

**3. Extension for heterogeneous feature & label spaces**

We assume that the overall feature space and label space for a task are represented as $\mathcal{X}=\mathcal{X}\_i\times\cdots\times\mathcal{X}_Q$ and $\mathcal{Y}$, respectively. When the learnwares have heterogeneous feature spaces $\mathcal{X}\_i=\times\_{k\in C\_i}\mathcal{X}\_k$ and label spaces $\mathcal{Y}\_i\subseteq\mathcal{Y}$, our method can be naturally extended to address this problem. We present the following extension as an example. The subspace can be learned in the same way, **with the supervised term playing an important role in label space alignment within the subspace**. For learnware recommendation, the learnware dock system can recommend the most suitable model **for each class** of the user's task using the MMD distance. For the recommended learnwares, reuse can be achieved through **dynamic classifier selection**, as demonstrated in previous work (TKDE 2023).

---

For other questions, please refer to our reviewer-specific feedback for further details. We hope our responses are satisfactory and can address your problem.

Meta-review: The proposed methods have been shown to improve SOTA on tasks with heterogeneous feature spaces, availability of some labelled data, and requirement for non-exposure of raw data of the model developers. While this is a relatively specific setting, the manuscript together with the rebuttal have sufficiently convincingly explained why this is a relevant task for the community.

The submitted version of the paper did not consider a realistic heterogeneous feature space setup, but instead was randomly dividing the data sets. However, during the rebuttal phase, the authors added experiments about two real-world scenarios, and in one of those the split was designed to be non-random and more realistic.

All 4 reviewers have stated that the presentation was good. All reviewers also see the paper as sound, except for some concerns of one reviewer, but these were alleviated during the rebuttal.

Further analysis of method complexity was requested by one of the reviewers, and this was provided during the rebuttal.

Overall, the paper can be recommended for acceptance.
Decision: Accept (poster)