Residual Feature Integration is Sufficient to Prevent Negative Transfer
Authors: Yichen Xu
Ryumei Nakada
Linjun Zhang
Lexin Li
Venue: ICLR 2026 Poster (2026)
Keywords: deep transfer learning theory
negative transfer
provable safe transfer mechanism
Primary area: learning theory
TL;DR: We prove that residual feature integration is sufficient to eliminate negative transfer in deep transfer learning, providing the first general theoretical guarantee with supporting empirical validation.

Abstract: Transfer learning has become a central paradigm in modern machine learning, yet it suffers from the long-standing problem of negative transfer, where leveraging source representations can harm rather than help performance on the target task. Although empirical remedies have been proposed, there remains little theoretical understanding of how to reliably avoid negative transfer. In this paper, we inves­tigate a simple yet remarkably effective strategy: augmenting frozen, pretrained source-side features with a trainable target-side encoder that adapts target features to capture residual signals overlooked by models pretrained on the source data. We show this residual feature integration strategy is sufficient to provably prevent negative transfer, by establishing theoretical guarantees that it has no worse con­vergence rate than training from scratch under the informative class of target dis­tributions up to logarithmic factors, and that the convergence rate can transition seamlessly from nonparametric to near-parametric when source representations are informative. To our knowledge, this is the first theoretical work that ensures protection against negative transfer. We carry out extensive numerical experiments across image, text and tabular benchmarks, and empirically verify that the method consistently safeguards performance under distribution shift, label noise, seman­tic perturbation, and class imbalance. We additionally demonstrate that this resid­ual integration mechanism uniquely supports adapt-time multimodality extension, enabling a pretrained single-cell foundation model to incorporate spatial signals for lymph-node anatomical classification despite the source model being trained without them. Our study thus advances the theory of safe transfer learning, and provides a principled approach that is simple, robust, architecture-agnostic, and broadly applicable.

Published as a conference paper at ICLR 2026
RESIDUAL FEATURE INTEGRATION IS SUFFICIENT TO PREVENT NEGATIVE TRANSFER
Yichen Xu
Ryumei 2 Linjun Zhang
Lexin Li'
University of California, Berkeley Harvard University
RRuggers University
{yichen_xu, www.nnij@berreeey...
1z4120stat rutgers edu
ryumei i na makada@hms harvard.
ABSTRACT
Transfer learning has become central paradigm in modern machine learning, yet it suffers from the long-standing problem of negative transfer, where leveraging source representations can harm rather than help performance on the target task. Although empirical remedies have been proposed, there remains little theoretical understanding of how reliably avoid negative transfer. In this paper, we inves- tigate a simple yet remarkably effective strategy: augmenting frozen, pretrained source-side features with trainable sarget-side encoder that adapts target features to capture residual signals overlooked by models pretrained on the source data. We show this residual feature integration strategy is sufficient to provably prevent negative transfer, by establishing theoretical guarantees that it has no worse con- vergence rate than training from scratch under the informative class of target dis- tributions up to logarithmic factors, and that the convergence rate transition seamlessly from nonparametric to near-pan wear-parametric when source representations are informative. To our knowledge, this is the first theoretical work that ensures protection against negative transfer. We carry out extensive numerical experiments across image, text and tabular benchmarks, and empirically verify that the method consistently safeguards performance under distribution shift, label noise, seman- tic perturbation, and class imbalance. We additionally demonstrate that this resid- ual integration mechanism uniquely supports adapt-time multimodality extension, enabling a pretrained single-cell foundation model incorporate spatial signals for lymph-node anatomical classification despite the source model being trained without them. Our study thus advances the theory safe transfer learning, and provides principled approach that is simple, robust, architecture-agnssic and broadly applicable.
INTRODUCTION
Transfer learning provides a fundamental paradigm in modern machine learning, where knowledge acquired from one task (source domain) is leveraged to enhance performance on another related task (target domain). It encompasses wide range of applications, from adapting models across different sources or domains, to distilling knowledge from large, pretrained models into smaller, task-specific models. Yet, a critical and persistent challenge is negative transfer: the phenomenon where transferring knowledge degrades performance compared to simply training on the target data from scratch. This issue, which arises from mismatches between source and target distributions, has been documented across numerous scenarios [34
40), It is especially concerning in high-stakes applications such as healthcare, where transferring from broad datasets like ImageNet to medical imaging can be detrimental [38 6 Despite its prevalence, there remains little theoretical understanding of how reliably avoid negative transfer.
6 28
20
49
46
"Equal contribution Corresponning author

---

Published as a conference paper at ICLR 2026
In this article, we identify and validate simple yet remarkably effective strategy that provably prevents negative transfer, i.c., augmenting frozen, pretrained source-side features with a trainable target-side encoder that adapts target features to capture residual signals overlooked by models pre- trained on the source data. We call this strategy Residual Feature Integration (REFINE). Its imple- mentation is straightforward after obtaining the transferred representation f (2:) from the source domain, instead of relying solely on frep(z), we further introduce residual connection with a train- able feature encoder k(z) that learned from the target domain. We then combine frep (1) and h(x), and fit a shallow neural network on the concatenated representation (fre(az), h(z)). Intuitively, while (1)((r) captures transferable features, may omit target-specific signals that are critical for accurate prediction in the target domain. The residual connection via l((z)) compensates for this omission, ensuring that key information in the target domain is preserved. Furthermore, because (x)((r) already encodes substantial portion of the predictive signal, learning from the joint repre- sentations (fre(zz) (z),, h(z)) can potentially be achieved with a much simpler class of functions than learning from or h(z) alone. We demonstrate, both theoretically and empirically, that this strategy is sufficient to prevent negative transfer across a broad range of settings.
Our contributions are threefold. First, we identify the residual connection, widely adopted struc- tural component originally devised to address optimization challenges in deep neural networks 15 [2], as powerful mechanism for provably avoiding negative transfer. This strategy in turn offers a lightweight, robust, architecture-agnosiic, and broadly applicable enhancement to trans- fer learning pipelines. We further identify an under-explored form of negative transfer that arises when source models lack modalities available only at adaptation time, and demonstrate that RE- FINE uniquely enables such adapt-time multi-modality extension on a single-cell foundation model for lymph-node domain classification. Second, we formally justify this simple yet remarkably effec- tive approach through rigorous theoretical analysis, which is the main contribution of this article. Specifically, we show that augmenting any frozen frep with trainable hxx) guarantees that the resulting predictor achieves a convergence rate of prediction risk that is never worse than that ob- tained by training from scratch on the target data alone. In other words. REFINE is inherently robust against negative transfer in the worst-case scenario. Morcover, our prediction risk bound seamlessly transitions from nonparametric convergence rate to near-parametric rate when source represen- tations are informative. Finally, we conduct extensive experiments on benchmark datasets spanning image, text, and tabular domains, and compare REFINE with multiple alternative solutions. We em- pirically verify that our method consistently mitigates negative transfer, especially under significant representational mismatch task divergence.
2 RELATED WORK
Transfer learning. Linear probing (26] and adapter-based feature extraction 18] are two of the most widely used transfer learning approaches. Both methods operate by extracting penultimate- layer features from a pretrained model in the source domain, followed by fine-tuning the final layer using data in the target domain. The main difference between the two is that linear probing employs a linear layer, while the adapter method uses a shallow neural network. Both are computation- ally efficient, but both are vulnerable negative transfer. Knowledge distillation is another widely used transfer learning technique, where a large pretrained foundation model (the teacher) transfers knowledge to a simpler model (the student) that is typically fine-tuned in the target domain with substantially reduced complexity (16). However, distillation remains vulnerable to negative trans- fer, especially when the teacher is poorly aligned with the target domain or when the transferred knowledge is too complex for the student to absorb effectively [10]. Our approach is applicable not only to knowledge transfer in foundation models, but also to general transfer learning settings.
Negative transfer mitigation. To mitigate negative transfer, various empirical remedies have been proposed, most of which focus on developing metrics that estimate similarity between source and target domains [1] 29 47 I]. Yet in practice, such similarity measures are often difficult to quan- tify, and sometimes require specialized loss functions or architectures, which limits their applicabil- ity [17. [27] proposed SAFEW, which constructs an ensemble of source-domain models using min-max max framework. While theoretically sound, this method is computationally intensive and relies on the assumption that the optimal predictor can be expressed as convex combination of source classifiers. [45] introduced DANN-GATE, a state-of-the-ee solution that reduces negative transfer by combining adversarial training with a gating mechanism to filter out misleading source samples.
2

---

Published as a conference paper at ICLR 2026
While practically effective, this method requires direct access source data and is primarily pirical, lacking theoretical guarantees. In contrast, our method does not require access to original training data in the source domain and comes with rigorous theoretical guarantees. We also exam- ine a largely overlooked form negative transfer in which the source model lacks modalities that become available only adaptation time. This setting is seldom discussed in multimodal learning (2). where it is typically assumed that all modalities are present during source-model training. Exist- ing approaches cannot exploit such missing- missing-modality information without retraining on source data. REFINE uniquely enables adapt-time multimodality extension without access to source data.
Residual learning, stacking, and parameter-effiiient fine-tuning. Several methods are conceptu- ally related to REFINE, although they do not explicitly target negative transfer in transfer learning. Residual learning, core idea in architectures such as ResNet [5]] and algorithms like gradient boosting [22]. was originally developed to ease optimization challenges or improve prediction. Its potential for addressing negative transfer, however, remains unexplored. Stacking is an ensemble technique that combines predictions from multiple base models through meta-learner trained on validation outputs. This approach is generally more robust than simple model averaging 44, but it assumes that all external models reliable [9] [2], and requires aligned output spaces, which restricts its applicability across different types of tasks. Parameter-efficiee fine-tuning methods, such LoRA [19), insert lightweight, trainable modules into pretrained models to enable domain adaptation without modifying the original weights. Such approaches are effective and significantly reduces parameter costs, but struggles when source representations misalign with the target domain. Besides, it requires access to pretrained model weights and computational graphs, limiting their flexibility, particularly in the multi-source transfer setting.
PROBLEM FORMULATION AND ALGORITHM
Transfer learning aims to leverage knowledge from a source task to improve performance on a related target task. A common practice is use representation function frep learned from large source dataset D under source distribution IPa as an extracted feature for the target task. However, feep does not align well with the target distribution P', naively reusing can lead to negative transfer, resulting in degraded performance compared to using the target data alone.
Residual Feature Integration (REFINE) approach. construct a method such that, when See aligns well with the target distribution, we
We formalize the to
The objective is
effectively leverage transferred
knowledge and outperform mod- els trained from scratch on tar- get data only, and when fue mis- aligns with the target distribution, safeguard against negative transfer and outperform models that rely solely fre((x).. We focus the supervised learning task. Let D. {(x4, );)) Ipt denote the labeled dataset from the target task. Assume access to a frozen extracted feature fxe X + R2 trained on an external source data D'. Define class H of trainable feature encoders h X' + R and class W of trainable adapters w RR++y - IR on top of (fe(()), Let un be the trained adapter top of the baseline model, and let Åve be the model trained from scratch on æ. We seck to learn both the encoder h and the adapter úù such that the expected excess risk of v o (frep fi) over the target distribution is bounded by the minimum of the excess risks of the two baselines: ih frep and dsc-
LinearProbing/Aee Adapte Finetune Frozen
ReFine
frep Max)
Frep
Figure schematic overview of REFINE.
Algorithm outlines the REFINE approach. extracts (x) from the penultimate layer of a frozen pretrained model, and combines it with the residual connection h(x)). The concatenated features
3

---

Published as a conference paper at ICLR 2026
(Fre(()), h(x)) are passed linear classifier for prediction, where only h(x) and the adapter updated, whereas the pretrained model and f1e(()) remain unchanged. This design allows REFINE to efficiently complement transferred knowledge with adapted features from the target data, and thus recover potentially lost information during the forward pass in the frozen source model. Figure gives schematic overview of REFINE.
4 THEORETICAL ANALYSIS
We provide a theoretical analysis to prove that REFINE is robust to negative transfer. The intuition and core insight is that the residual connection provides natural transition: if the external repre- sentation feep is uninformative, the residual network & can still learn the target function from the raw input, recovering the performance of training from scratch. Conversely, if frep is informative, only needs to learn the simpler residual function, reducing the effective complexity of the problem and accelerating the learning. This intuition formalized two ways: antee showing that, under mild growth conditions model capacity, REFINE is never worse than either training from scratch or using a linear probe on Sme and risk bound showing that its con- vergence rate smoothly interpolates between the standard nonparametric rate and paramparaettric rate depending on the quality of the external representation.
mneaatvee-rranseee guar-
We formalize this intuition within the framework of nonparametric regression. We consider the model with trainable residual feature encoder h::
g(x) wh(x)) + +o = Scp((a)
where h (z) is (clipped) ReLU network over raw input, combined with linear probe on the feature (xp(()). We establish the risk bound demonstrating that, for moderate capacity of h. REFINE'S excess risk is no worse than the excess risk of the model trained from scratch or the linear probe on fiop(x). Furthermore, when the capacity of f is tuned to the difficulty of the residual task, the rate adapts and improves, showcasing its ability to effectively leverage useful prior information from (x)((z).
Formal Setup. We consider the nonparametric regression setup adopted in the statistical analysis of deep neural networks [2 39 24) Specifically, we observe i.idd pairs (X, Xi)ee()) ~ IP' with support on [0, 1] IR following the model
Y; = J'X,) + 6
(1)
where f [0, 1]2 + [-1,1] is the ground regression function, (Xi)se(a) are ii.d, samples
from the marginal distribution Pxx on X, and (Ei)ægjn are i.i.d. Gaussian with variance o² = @(1), independent of (X.)vejn)- We assume PX admits positive continuous density on [0. 1] upper bounded by an absolute constant. Under this set-up, the expected loss for given function g
(g) = -x ((xx))))((((()))->>>)))()))>))))
To facilitate the theoretical analysis, following the standard setup of nonparametrie regression, we consider f to be Hölder smooth. Specifically, for non-integer 3 > 0, the Hölder norm for f that
are [time differentiable on [0, 1]d is
f(x) 80 f(x')) | - 1'11--10 S
[Iflle= max
sup 10a f(x)). LEN ( =0
max
max aEN [8] xft
sup
Algorithm The residual feature integration (REFINE) method.
1: Input: Training data Diaiin = (X,, Yi) test data Dest pretrained model f. loss function
2: Output: Prediction of the label ((x)) for 2o D
3: Training Phase:
4: (a) Extract ( from the penultimate-laye of frozen pretrained model f.
5: (b) Construct the concatenated features Cx(x) == (frep(a), h(x)).
6: (c) Let (w, ft) be the minimizer of E, (lww(((,,)) (X;)), Y)) while freezing frep.
7: Prediction Phase:
8: (a) Compute Cn (2v) with the frozen f-
9: (b) Obtain the final prediction j(zn) based on w(C, (2o)).
4

---

Published as a conference paper at ICLR 2026
The unit ball is Ci (f [0. RR [ -times differentiable and lifllca I 1}.
Further, we assume the residual connection h R + R is realized by a ReLU network with width at most W, depth at most L, and weight magnitude at most B:
h(z) A1 byz, x0 (A b,) (e IL' 1)), x1 Z. (2)
for some L' I L, where do = d. dr 1, and de W. Here (2) ma 2} is applied element-wise, A₂ E [-B. B, B) didex and by E [- BJ The class is Ha(W, L, B), and we use
its clipped counterpart (W, B) == for min max - h(x))) h E Ha(W, B)).
Empirical risk minimization for REFINE. We consider squared loss {v. v/) = (y 5/)² Let frep [0, 1 + (1) be an external representation with (R) {u R2 : ||l S RJ. Define the REFINE class
(W, L, B; sap) = {9: [0, 1J" + R g(x) = v² (ap(() uh(()), ul S 1, ||l|| S L, h Ra L, L)
We train 9 via empirical risk minimization.
Σ ((g(X,), Y.). KEIm]
9 =
arg min
(3)
g LW,L,
The effectiveness of REFINE depends on the quality of frep- We quantify this by defining the best possible linear probe and the corresponding residual. Specifically, for any Step [0, 1| + (I),,, the best linear probe is defined as
2² arg min E[ (x) (X1)) VERR
The difficulty of learning the residual is then captured by its Hölder norm, which we denote as p² ||0* Srep f'll A small p² indicates that Step is highly informative for the target task.
We state a theorem that provides an upper bound on the generalization error of the empirical risk minimizer in equation 3 when the model capacity is chosen appropriately.
Theorem 4.1 (Generalization Error of REFINE). Assume v* fmp - f C Let 0 be tuning parameter, which serves as a proxy for the residual norm, and choose the network parameters for
L1 W = 02 max{ n (28++d)) 22/(22+4) ,1).
B (p" V maxmaxxmm (np²,1} (4)
where G1, C22 03 > depend on B, and 7. Let 9 be the empirical risk minimizer in parameter specified as in d4). Then there exists C >> O, which depends on 3, d, such that
(3) with the
E R( (x(())) of (p (221(28)))) loga + p² (0081(22+4)))) -8/(2244) + plog n
(5)
n
The bound in 5 splits into a parametric term p log n/n for learning v² on top of freps and a non- parametric term with the standard minimax rate n (2/((2++d) for learning the residual modulated by the tuning parameter p and the residual difficulty p" The tuning radius p controls the effective capacity of via W and in (4). That is, a larger increases the approximation power, achieving a smaller bias, but worsens the estimation, resulting in a larger variance factor 2d/((2++)) On the other hand, smaller p regularizes h, which is preferable when the residual is genuinely small.
Proof sketch of Theorem 4.1 For any U. decompose
(" = (x) (rep(x)
Srep
residual
lincar in foop (2t)
The first term is fit by h and the second by linear probe on frep Approximation results for ReLU networks over 8 functions give the residual term rate n (2/(2)44) with capacity-dependen multiplier governed by p. A standard linear estimation yields the p/n term for v. Choosing (W, I., B) in (4) implements this bias- trade-off. The full proof is deferred to Appendix
5

---

Published as a conference paper at ICLR 2026
We remark that theoretical results derived under the squared-loss objective, following long line of work that analyzes classification problems through regression surrogates [1]]50] This approach aligns with common practice in the machine learning theory community, where regression surrogates are employed to derive insights for classification algorithms.
We further diseuss two direct implications of Theorem 4.1
Corollary 4.2 (Fixed p). Under the same conditions as in Theorem > the bound implies that
4.1 for any fixed choice of
BiRr Rsd (f*) = Ō(n -0nn22//25t0))
28/(284)) n2 ")
This corollary indicates that, by introducing additional residual connection h, REFINE never has worse rate than 22/(22+-)) for fixed which is the standard minimax-optimal rate when training from scratch on (X,, Xe)eem for 3-Hölder f (See, for example, Theorem 3.2 in Györfi et al. (3).
Corollary 4.3 (Tuned p). Under the same conditions as in Theorem 4.1 balancing (5) by choosing P. 4 p² yields
- 20110000))) -2
-2/02244)) n2 -
(6)
This corollary indicates that, when feep is well aligned with the target, i.e., small p", choosing p" effectively regularizes the residual network h via the parameter choice in 4 which shrinks the nonparametric term so that the bound dominated by the near-parametrie p/n term. Conversely, when frep is misaligned, i.e., a large p², the nonparametric component dominates and the rate reverts the classical 3-Hööler minimax rate (2)(())
We now provide a corollary for no-negative-transe guarantee. The key idea is to define a class of functions that be approximated in the 3-Hölder norm by linear combination of fiep up error 7 0:
J (Jopp 7) = ( [0. 1j R
min ||0 - St). e:leel
This class captures the functions that can be learned from the residual connection h and the linear probe frep. and thus serves target for the empirical risk minimization in equation 3 In special case where feep not informative, i.e., Seep = O, the class reduces to the standard Hölder ball with radius y: {f" Illf llee 5.).
guarantee). Fix d N_ and 3 >> O. Also fix frep : [0, 1j + Iº satisfying T frep E C. for any unit vector E Sy-1, Consider the model trained from scratch and the linear probe on frep with comparable capacity:
Corollary 4.4 (No-negative-tran
1 Σ E(w Fie( Y.).
Esc
arg min gE JW..LW
Σ ((g(X.),
ún arg min WE3F
iE[n]
sE(m]
Then,
sup (CFFO(Jmmn)
E(Rx(() (p()))))
E[R Srep Rx( R
- sup
(( - RRr(")) sup (EFF(())))))
(EF)((
holds for any Y 10.
Specifically, when 7 0, i.c., when f lies exactly in the linear span of fiep- the excess risk of RE- FINE is, up to logarithmic factors, no worse than that of the linear probe on frep- When 7 > 0, RE- FINE attains the standard nonparametrie rate for estimating 3-Hölder functions; this rate improves upon that of the linear probe on fuep which suffers from bias due to representational misalignment. Hence REFINE provably avoids negative transfer for this class of target functions.
We also provide the asymptotic no-negative transfer guarantee for a fixed f" under any mild model capacity in Appendix A.3

---

Published as a conference paper at ICLR 2026
5 NUMERICAL EXPERIMENTS
5.1 EXPERIMENT SETUP
We demonstrate that REFINE consistently mitigates negative transfer through extensive numerical experiments across image, text, and tabular modalities, using benchmark datasets including CIFAR- 10, CIFAR-100 [25], STL [SI Clipart, Sketch [36], USPS, MNIST, Books, Kitchen, DVD, and Electronies 3. We evaluate performance using classification accuracy, area under ROC (AUC), F1 score, and minimum class accuracy.
We also compare REFINE with a number of alternative solutions. In particular, NoTrans serves as a no-transfer baseline, reusing pretrained features without any adaptation. LinearProbe (26) trains only a linear classifier on top of frozen features, offering lightweight baseline. Adapter [T8] inserts small trainable module into pretrained models, enabling efficient adaptation with limited parameters. Distillation [[6] transfers knowledge from a frozen teacher to a student model through a combination of hard labels and soft predictions. LoRA [19] applies low-rank adaptations weight matrices, achieving cerammter-effiiiee fine-tuning. DANN-Gate [45] combines adversarial training with gating to encourage domain-invariant invariant representations.
We consider a variety of experimental settings. In Section 5.2 we evaluate REFINE on datasets exhibiting natural distribution shift. In Section construct challenging scenarios stress-test robustness under controlled perturbations. In Section we examine an adapt-time multimodality extension setting based on spatial transcriptomics, where a new modality becomes available only after pretraining. Finally, in the Appendix, we include additional studies on source-free multi-source transfer (Appendix C.I. and tabular benchmark evaluations (Appendix C.4.
In our implementations, we train all models using stochastic gradient descent with learning rate 0.01 and momentum 0.9, with pretraining for 60 epochs and fine-tuning for 30 epochs. We consider both CNNs and transformer architectures for the pretrained model frep and the encoders h. We also carry out an ablation study in Appendix C.5 regarding the complexity of the encoder h. showing that REFINE remains effective across different choices of the model parameters for h.
We provide more details about the experiment setup and implementations in Appendix D
5.2 SINGLE-SOURCE TRANSFER WITH NATURAL DISTRIBUTION SHIFT
In the first experiment setting, we evaluate REFINE on datasets that exhibit natural distribution shift. To provide a comprehensive assessment, we consider transfer tasks spanning both image and language, thereby covering cross-domain we include CIFAR-10, CIFAR-100, and STL-10, which offer complementary object recognition tasks with varying class granularity and image resolution. We further incorporate artistic domains, specifically, Clipart and Sketch, to capture substamtial stylistic diversity, along with digit recogni- tion benchmarks, USPS and MNIST, which provide structured and well-currted handwritten digits. For text, we adopt the datasets, Books, DVD, Electronics, and Kitchen, which span heterogeneous product categories and exhibit rich linguistic variations. We process the image datasets using con- volutional neural networks (CNNs), and process the text datasets using transformers. This design allows us assess transfer across distribution and domain shifts, and also under cross-modality and mooss settings. Collectively. these datasets constitute a broad and rigorous benchmark for evaluating transfer learning methods.
well as cross-modality adaptation. For image,
We use the notation A + B to denote transfer learning from source domain A to target domain B. Our evaluation covers diverse scenarios. Specifically, CIFAR100-110 and CIFARI0->100 test transfers across datasets with overlapping but non-identical class spaces and label granu- larity; CIFARIO-STL reflects natural distribution shift due to resolution and dataset construc- tion; Clipart-Skettch represents cross-style adaptation between artistic domains; USPS->MNIST examines digit recognition under handwriting and design difference; and Books~>Ktehhe and DVD- capture cross-topic sentiment transfer, where vocabulary and linguistic style vary considerably. We exclude knowledge distillation [16] in this comparison, as it requires identical class spaces across source and target, which do not apply here.

---

Published as a conference paper at ICLR 2026
Dataset
Method
Accuracy
AUC
F1
Min CAce
NoTrans
56,5820 + 0.3659
0.9005 0.0012
1.5634000004
37.2000 + 3.4117
LinearProb
38.9260 +: 0.5463
0.8284 0.0017
0.3815 0.0051
16,9400 + 3.7441
CIFAR100-110
Adapter
38.2320 + 0.3111
0.8217 + 0.0016
0075440.0077
16,4600 5.4544
LoRA
43.1360 +: 0.3239
0.8603 + 0.0003
0,4237 + 0.0046
20.1400 + 4.1002
DANN-Gate
43.2220 +: 0.1295
0.8605 + 0.0005
0,4214 0.0040
17,4000 +: 4.7755
REFINE
54.4000 0.3336
0.8942 0.0026
0.5406 0.0051
33.6200
2.8273
NoTrans
18.3200 +: 0.5254
0.8140 0.0050
0.1774 0.0052
1.0000 0.8944
LincarProbe
7,0140 0.3347
0.7489 + 0.0011
0.0496 + 0.0034
0.0000 + 0.0000
Adapter
6.5640 = 0.2875
0.7499 + 0.0008
0.0459 + 0.0026
0.0000 + 0.0000
CIFARIO-100
LoRA
6.8240 0.1037
0.7558 0.0010
0.0463 : 0.0015
0.0000 1. 0.0000
DANN-Gate
5.1980 0.3924
0.7341 0.0055
0.0285 + 0.0033
0,0000 0.0000
REFINE
18,5880 + 0.5494
0.8276
0.0053
0.1787 1 0.0057
1,4000 + 0.800
NoTrans
48.6925 +: 0.6338
0.8683 + 0
0.4831 + 0.0089
26.8000 + 4.9006
LincarProbe
50.2725 +: 0.3016
0.8795 0.0015
0.4955 + 0.0067
18.9250 + 6.1546
Adapter
49.2900 0.7344
0.8773
0.4865 0.0096
15.6750 6.6340
CIFARIO-STL
LoRA
50.7550 + 0.3793
0.8613 + 0.0016
0.4930 = 0.0040
5.6750 + 2.6933
DANN- Gate
47.7050 + 0.6586
0. 0.8659 + 0.0013
0.4712 0
13.9250 # 5.3424
REFINE
53.4175 + 0.3628
0.8944 0.0013
0.5301 = 0.0053
25.9750 + 3.5093
NoTrans
18,8804 +: 1.3709
0.7170 0.0117
0.1828 + 0.0119
0.0000 + 0.0000
LinearProhe
18,3430 + U.8649
0.7290
0.0065
0.1727 + 0.0087
0.0000 0.0000
Ckipart-tSetch
Adapter
18.2356 + 0.5807
0.7369
0.0059
0.1549 + 0.0040
0.0000 + 0.0000
LORA
16.9010 + 0.6906
0.6937
0.0043
0.1671 + 0.0069
0.0000 + 0.0000
DANN-Gar Gate
16.5786 + 0.4868
0.6942
+.044400000$
0.0000 0.0000
REFINE
20.3403 = 0.4768
0.7338
0.0043
0.1968 + 0.0059
0.5263 + 1.0526
NoTrans
62.0740 + 8.7771
0.9566 0.0073
0.5967 + 0.0969
9.2863 12.1512
LinearProbe
66.9960 + 1.111.00 1.0095
0.9469 + O. 0.0050
0.6563 + 0.0086
9,1576 + 3.5478
Adapter
61.8600 + 3.0334
0.9375 0. 0.0085
8,8750 + 7.2427
USPS-MMNIST
LoRA
64.8240 +: 0.8520
0.9333 0.0045
0.6435 + 0.0135
29.3265 + 13.5652
DANN-Gate
52,2080 +: 36666
0.9012 + 0.0185
0.4853 + 0.0482
0.0198 + 0.0396
REFINE
70.0460 + 2.1721
0.9582 0.0053
0.6954 + 0.0194
31.6157 + 14.5527
NoTrans
71.6600 + 1.3632
0.7848 0.0155
0.7161 + 0.0137
68.6000 +: 2.9719
LincarProbe
66.7400 + 3.1455
0.7568 0.0278
0.6571 + 0.0401
51.6600 9.7336
Adapter
71.3400 + 0.1356
0.7839 + 0.0008
0.7111 = 0.0015
62,8800 = 2.9027
Books -Kitchen
LoRA
66.9600 + 0.2154
0.7279 + 0.0018
0.6695 + 0.0022
65.6100 : 0,4079
DANN-Gate
66.6000 + 0.0894
0.7330 0.0006
0.6659 + 0.0009
61.6800 + 0.6997
REFINE
72.7200 + 1.6522
0.8147 0.0133
0.7248 0.0189
65.5200 6,4778
NoTrans
68.5200 +: 2.8979
0.7585
0.0304
0.6806 + 0.0338
59.8000 9.8298
LineurProbe
66.0600 + 0.5122
0.7266 + 0.0017
0.6580 + 0.0072
58.3600 + 4.5379
DVD-EElectronics
Adapter
65.8600 + 0.3200
0.7206 0.0008
0.6577 + 0.0037
61.4400 + 2.5935
LoRA
66.5000 + 0.3555
0.7170 0.0013
0.6656 + 0.0036
65.4000 : 0.4899
DANN- MAte-Gate
66.9000 + 0.1897
0.7196 + 0.0013
0.6686
+ 0.0019
63.5600 # 0.2653
REFINE
70.3400 + 0.9972
0.7886 + 0.0115
0.6995 + 0.0122
6L.7200 + 7,5181
Table 1: Single-source transfer learning with natural distribution shift.
Table Treports the results. REFINE consistently achieves competitive superior performance com- pared to alternative methods across all scenarios. On transfers with large label-space difference, including CIFARI00~>1 10 and CIFAR 100, REFINE improves accuracy by over 10 - 15% rel- ative to Adapter, LoRA, and DANN- Gate, substantially narrowing the gap to the no-transfer base- line while remaining robust to negative transfer. On transfers under natural resolution or stylistic shift, including CIFAR I0LSLL, Clipart-tSketch, REFINE achieves 3 - 4% accuracy gains over the strongest alternative, along with consistent improvements in AUC and FI. On transfers with digit benchmarks, including USPS-MMIIST, it yields 5 - 10% accuracy gains, and much higher mini- mum class accuracy, indicating stronger preservation of performance on underrepresented classes. On transfers with cross-topics, including 2 4% improvements across all metrics. Overall, REFINE not only avoids the severe degradation observed in other methods, but also provides reliable accuracy lifts of 5 15% across image and text domains under diverse settings of distribution shifts.
DVD-=llectronie. REFINE delivers
5.3 SINGLE-SOURCE TRANSFER UNDER LABEL NOISE, SEMANTIC PERTURBATION, AND CLASS IMBALANCE
In the second experiment setting. we deliberately construct challenging scenarios to stress-test vari- transfer learning methods. Using CIFAR-10 with CNNs, we introduce four types of challenges
8

---

Published as a conference paper at ICLR 2026
Datuset
Setting
Method
Acc
AUC
F1
MinCAce
NoTru
56.05 + 0.64
0.9037 + 0.0028
0.5580 = 0.0080
32.40
5.84
LinearProbe
65.54 0.06
0.9378
0.6561
42.82
1.45
Adapter
65.78 0.19
0.9376 + 0.0007
0.6581 = =0.0024
45.20
- 2:29
40% flips
Distill
57.01 4: 0.58
0.9115 +: 0.0016
0.5674 = 0.0033
34.84
4.53
LoRA
65.47 0.12
0.9874
0.6545 0.0018
42.38
0.89
DANN-Gate
05.40 +: 0.15
0.9353 + 0.0006
0.6539 = 0.0016
43.40
2.22
REFINE
66.25 = 0.32
0.9388 + 0 0006
6625 0. 0036
43.941 3.78
NoTrans
56.37 +: 0.64
0,9037 0.0033
0.5622 = 0.0055
33.60 3.04
LineurProbe
19.46 0.75
0,6895 0.0011
0.1177 0.0108
0,00 + 0.00
Adapter
18.49+40.44
0.6906 + 0.0006
0.1219 = 0.0156
0.00 + 0.00
80% flips
Distill
53.51 0.79
0.8982 0.0021
0.5269 = 0.0091
26. 2.49
LoRA
22.92 1.73
0.7202 0.0079
0,1911 0.0330
0.76 + L.52
DANN-Gate
20.83 1.32
0.7097 + 0.0084
0.1341 = 0.0253
0.00 + 0.00
REFINE
56.58 0.33
0.9067 0.0019
0.5655 = 0.0041
36.90 2.94
NoTruns
56.53 0.77
0,9006 0.0021
0.5639 = 0.0056
35.76 2.75
LincarProbe
18.51
0.8987 + 0.0008
0.4757 = 0.0016
18.11 = 7.89
CIFAR-10
Adapter
47.17 +: 0.82
0.8998 + 0.0006
0.4479 = 0.0148
7.42 + 6.47
Schematic confusion
Distill
57.80 0.44
0.9068 0.0009
0.5772 0.00ST
35.92 3.00
LoRA
19.96 +: 0.26
0.9039 + 0.0005
0,4861 = 0.0116
16.34 9.91
DANN-Gate
49.04 + 0.33
0,9028 + 0.0006
0.4719 0.0059
11.40 1.53
REFINE
58.65 0.47
0.9034 0.0011
0.5861
0.0048
38.40
NoTruns
56,44 0.48
0,9055 0.0019
0.5599
0.0051
32.80 4.51
LinearPrbbe
53.15 + 1.04
0.8883 + 0.0145
0.5238 + 0.0215
28.36 + 14.04
Adapter
51.64 0.99
0.8960 0.0022
0.5130 = 0.0150
19.52 8.32
Class imbalance
Distill
5-1899
0,9063 0.0013
0.5192 0.0065
41.96 3.43
LoRA
53:21 + 0.19
0.8975 + 0.0005
0.5338 - 0.0022
33.76 5.38
DANN-Gate
53.05 +: 0.28
0.8964 0.0009
0.5281 = 0.0055
32.62 3.60
REFINE
56.34 0.73
0.9103 0.0012
0.5619
0.0103
31.38 + 10.31
Table 2: Single-source transfer learning with label noise, semantic perturbation, and class imbalance for CIFAR-10 using CNNs.
in the pretraining data while keeping the target domain fixed: (i) heavy label noise with 40% random label flips, (ii) extreme label noise with 80% flips, (iii) semantic perturbation created by paired-class flipping combined with additive image noise, and (iv) class imbalance induced by resampling to a long-tailed distribution. In addition, we repeat the experiments on CIFAR- and also evalu- ate both CIFAR-10 10 and CIFAR-100 100 with transformer-based models. We report the corresponding results in Appendix
Table 2 summarizes the results. REFINE consistently mitigates severe degradation and outperforms competing methods across all stress-test scenarios. In the moderate noise setting with 40% label flips, it achieves the best overall balance, improving accuracy and Fl by about 1% over Adapter and LoRA, while maintaining competitive minimum class accuracy. the more extreme noise setting with 80% flips, most baselines collapse, with LincarProbe, Adapter, and DANN-Gate drop below 25% accuracy, whereas REFINE remains close to the no-transfer baseline, improving accu- racy by nearly 35% over the strongest adaptive alternative. In the semantic confusion setting, with paired-class flips plus image noise, REFINE gains 1-2% in accuracy and F1 over NoTrans, while all other adaptive baselines perform worse, highlighting the robustness of REFINE perturbed label semantics. In the class imbalance setting, it surpasses LinearProbe, Adapter, and LoRA by 3-5% in accuracy and F1, achieving the strongest overall results aside from a slightly lower minimum class accuracy than Distillation. Overall, REFINE avoids the catastrophic failures common to exist- ing transfer strategies under noise, semantic perturbation, and class imbalance, while consistently delivering performance gains across all stress-test conditions.
We also briefly remark that, an important advantage of REFINE is that its complexity can be flex- ibly tuned through the choice of the encoder h. Such design keeps it comparable in parameter efficiency to methods such as Adapter and Distillation. For instance, in this setting, for REFINE, the number of trainable parameters is 4.88% of the total number of parameters in the frozen source 5.46%, and for Distillation, 4.68% Thus REFINE achieves a compara-
model, for Adapter, ble parameter efficiency, but clearly outperforms in mitigating negative transfer. The ablation study in Appendix [.3] further shows that the performance of REFINE remains stable across different pa- rameter choices of h, indicating that the overall parameter complexity has relatively little impact.

---

Published as a conference paper at ICLR 2026
ACC
F1
AUC
Figure 2: Metric comparison across labeled target sizes for Adapter, LinearProbe, NoTrans, and REFINE.
By contrast, increasing Adapter's complexity fails resolve negative transfer, suggesting that its limitation stems from design rather than capacity.
5.4 ADAPT-TIME MULTI-MODALITY EXTENSION: SPATIAL-OMICS EXAMPLE
Spatial transcriptomics provides a clean setting to study a phenomenon that is becoming increas- ingly common in biological data analysis: pretrained model is strong, but a crucial modality appears only adaptation time. In the SpatialGiue human lymph node dataset [30,, cells come with both transcriptomes and spatial coordinates, and the major anatomical domains cortex, medulla cords, follicles, capsule, pericapsular adipose tissue, and others form well structured spatial patterns. Because expert annotation of these domains is costly, only small subset of cells typically receives labels. Importantly, scGPT [7], like most foundation models for single cell data, is pretrained purely on dissociated RNA and therefore never observes spatial information. This naturally creates what we refer to as an adapt-time time multimodality extension problem, where the model must incorporate modality that was entirely absent during pretraining. Conventional fine-tuning PEFT cannot reliably supply information the pretrained model has never learned.
The empirical results reflect this challenge. As shown in Figure 2 both LinearProbe and Adapter exhibit negative transfer when applied directly to scGPT representations. With 1000 labeled cells, their Fl scores remain near 0.24 0.29, far below a simple GNN trained from scratch, denoted as NoTrans, which already reaches about 0.47 by leveraging spatial structure directly. Even with more labeled data, their AUC and F1 improvements stagnate. In contrast, REFINE adds a lightweight residual spatial encoder that complements the frozen scGPT features without modifying the back- bone. This allows the model to integrate the missing spatial modality at adaptation time. The gains are substantial: at 1000 labels, REFINE raises Fl to roughly 0.52, and surpasses 0.70 by 3000 labels, with consistently stronger AUC across all training sizes.
Figure S2 Ilustratte the qualitative impact of this adapt-time modality gap. LinearProbe and Adapter compress the cortical region, miss several follicular and trabeculae regions. REFINE reconstructs these domains much more faithfully, recovering cortical extent, follicle structure. and peripheral regions that the other approaches systematically miss. These results suggest a broader message. When modality is entirely missing during pretraining, fine-tuning alone is insufficient, but resid- ual mechanism that injects the missing information at adaptation time can bridge the gap effectively and without imposing additional engineering burdens practitioners.
ETHICS STATEMENT
This research does not involve human subjects, personally identifiable information, sensitive data. The datasets used are publicly available and widely used in the community. We are not aware of with any machine
direct applications of method that raise ethical concerns. Nevertheless, learning system, there is a potential risk of misuse deployed in contexts where faimness or bias are critical. We encourage future work to examine these dimensions before deployment in such settings.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We have made efforts to ensure the reproducibility of our results. Detailed descriptions of datasets, preprocessing steps, and hyperparameters, optimizers provided in Section 5 and Appendix [D] All proof' for theoretical claims are provided in Section Appendix[A] An anonymized version of our source code is included in the supplementary materials and will be released publicly upon acceptance.
ACKNOWLEDGMENT
The research of LZ was partially supported by NSF CAREER 2340241 and Renaissance Philanthropy "Al for Math" Fund. The research of LL was partially supported by NIH grants UG3NS140730 and AGO80043.
REFERENCES
[1]
Muhammad Jamal Afridi. Arun Ross, and Erik M. Shapiro. On automated source selection for transfer learning in convolutional neural networks. Pattern Recognition, 73:65-75, 2018. ISSN 0031-3203. doi: htt.://nt........oo.oooo...toaea
URL wttp::ttts/////t //ttttts://ww..
occieneddo rect. wwt.eettt.ceeeeeeeoe www.ccoccccciiee
[2] Tadas Baltrusaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learn- ing: A survey and taxonomy. IEEE Trans. Puttern Anal. Mach. Intell., 41(2):423 443, February 2019. ISSN 0162-8828. doi: 20009/TAM..2018.79988600 URL https: 2018
[3] John Blitzer, Mark Dredze, and Fernando Pereira. Biographies, Bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Annie Zaenen and Antal den Bosch (eds.), Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pp. 440-447, Prague, Czech Republic, June 2007. Association for Computational Linguistics. URL https: aacaaaaaaaatt www.cllnnnoolooyy.oooooooooo P07-
[4] Aurélien Bibaut Cheng Ju and Mark van der Laan. The relative performance of ensemble methods with deep convolutional neural networks for image classification. Journal of Applied Statistics, 45(15)22800 2818, 2018. doi: 10. 1080000666473.22000 72..444776338888.. 2018.1444388 URL https: //doi otddi.org//1 1080 0200626666443 2018. 1441383
PMID: 31631918.
[5] Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsu- pervised feature learning- In Geoffrey Gordon, David Dunson, and Miroslav Dudík (eds.), Proceedings of the Fourleenth International Conference on Artificial Intelligence and Statis- tics, volume 15 of Proceedings Machine Learning Research, pp. 215-223, Fort Lauderdale, FL. USA, 1 1313 Apr 2011. PMLR. URL https: http::::/ppocccediingg mlr wtteeesm.rr.rrrr. wwt.pprsss/vi.... coates html
[6] Rhys Compton, Lily Zhang, Aahlad Puli, and Rajesh Ranganath. When more is less: Incor- porating additional datasets can hurt performance by introducing spurious correlations, 2023.
httss::::: arxiv... www.///.///........ abs 2308 04431
[7] Huazhe Cui, Chen Wang, Haitham Maan, Jason D. Buenrostro, Nir Yosef, Carolina Caldas, Rui Sun, and Bing He. segpt: toward building a foundation model for single-cell multi omics using generative AI. Nature Methods, 2109):1470-148) September 2024. doi: 10.1038/ s41 141592-024-02201
UtL.https: htw.ddoooorggg////
592 024 024-0221--00000
[8] Jing Gao, Peng Li, Zhikui Chen, and Jianing Zhang. A survey on deep learning for multimodal data fusion. Neural Computation, 32(5):22-8664 2020. doi: 10.1162/neco a_01273.
[9] Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning, 2019. httpstss::////// //axivv.//rivv.///aa//ia org/ http//a/s1190 0286100444888888
[10] Jianping Gou, Baosheng Yu, Stephen Maybank, and Dacheng Tao. Knowledge distillation:
A survey. International Journal of Computer Vision, 129(6): 18891 2021.

---

Published as a conference paper at ICLR 2026
[11] Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. J. Mach. Learn. Res., 13(mull):723-7733 March 2012. ISSN 1532-4435.
[12] Pratham Grover, Kunal Chaturvedi, Xing Zi, Amit Saxena, Shiv Prakash, Tony Jan, and Mukesh Prasad. Ensemble transfer learning for distinguishing cognitively normal and mild cognitive impairment patients using mri. Algorithms, 16(8), 2023. ISSN 1999-4893. doi:
10.3390/a16080:777
URLhttps::/ www. wwwpi.mdpp.... com 1 999 199-999999999////77777
[13] László Györfi, Michael Kohler, Adam Krzyžak, and Harro Walk. A distribution-free theory of nonparametric regression. Springer, 2002.
[14] XY Han, Vardan Papyan, and David L Donoho. Neural collapse under mse loss: Proximity to and dynamics the central path. arXiv preprint Stti::2006000077. 2021.
[15] Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recogni- rion, pp. 770-778, 2016.
[16] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network, 2015. URL httpltttt//saseestt org/ aabs/1500 02531
[17] A. Hosna, E. Merry, J. Gyalmo, Z. Alom, Z. Aung, and M. A. Azim. Transfer learn- ing: a friendly introduction. Journal of Big Data. 9(1):102, 2022. doi: 10.1186/
$40537-022-0065 -022-00552-.. URL https: //t:://doo..ooo Epub 2022 Oct 22.
www.:oi........oooooo.oooooo.... org. 10. 1186/ $40537 2222-00652-2 111.1.111///0057777
[18] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficient transfer learning for nlp, 2019. htttt:ttp//aaa/t org. abs/ 1902.007555
[19] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL
http:/iv..rr//b//11....... 09685
[20] Rohan Jha, Charles Lovering, and Ellie Pavlick. Does data augmentation improve generaliza- tion in nip?, 2020. URL
[21] Yuling Jiao, Huazhen Lin, Yuchen Luo, and Jerry Zhijian Yang. Deep transfer learning: Model framework and error analysis, 2025. URL
a241//00bbb//.0...o/// 09383
[22] Guolin Ke. Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma. Qiwei Ye, and Tie- Yan Liu. Lightgbm: highly efficient gradient boosting decision tree. In Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gareett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: wtw.:/cceee.nngg...eii..oe.e www.nneeuii..c. cc/paper files//aper/2017/ E
[23] Ronny Kohavi and Barry Becker. Adult income dataset. https://ww/ kaggle. com/
batassess/ccigg laadulll-censuu
1996. Originally from the UCI Machine
Learning Repository, Kaggle version shared by user 1251, updated 2016.
[24] Michael Kohler and Sophie Langer. On the rate of convergence of fully connected deep neural network regression estimates. The Annals of Statistics, 49(4):2231-222 2021.
[25] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical re-
port, University of Toronto, 2009. URL https: http:///www www. cs wwt.:cs...tttttooooooooooo.
learning- featt-ffuuursss 2009- TR. pdf Technical Report.
[26] Ananya Kumar. Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution, 2022. URL https:
7/axxiv.oc wwt.r/iwworggbbb//////o 10054
12

---

Published as a conference paper at ICLR 2026
[27] Yu-Feng Li, Lan-Zhe Guo, and Zhi-Hua Zhou. Towards safe weakly supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(1):334- 346, 2021. doi: 10.1 (0.109/TPAMM222099222333
[28] Chi-Heng Lin, Chiraag Kaushik, Eva L. Dyer, and Vidya Muthukumar. The good, the bad and the ugly sides of data augmentation: An implicit spectral regularization perspective. Mach. Learn. Res., 25:91:1-91:85, 2022. URL o semantisccoooalrrr
2D28115755
[29] Y. P. Lin and T. P. Jung. Improving beg-based emotion classification using conditional transfer learning. Frontiers in Human Neuroscience, 11:334, 2017. doi: 10.3389/fnhum.20 20...001.0000000 2017.00334. URL https: ::::://doo../ wwo.o...ogggooo 3389/ Enhum, 2017 00334 2017.
Published on June 27,
[30] Yichen Long, Ka Shing Ang, Ritambhara Sethi, Guanghua Xiao, and Guo-Cheng Yuan. De- ciphering spatial domains from spatial multi-omics with spatialglue. Nature Methods, 21(9): 1668-1667, September 2024. doi: 19.10388/44152--224--020
URL https://doi.
[31] Zheda Mai, Ping Zhang, Cheng- Tu, You Chen, Li Zhang, and Wei-Lun Chao. Lessons and insights from unifying study of parameter--fffcient fine-tuning (peft) in visual recognition, 2025. UKLAtttp.//axxxi.ccc///////22...
[32] Shunya Minami, Kenji Fukumizu, Yoshihiro Hayashi, and Ryo Yoshida. Transfer learning with affine model transformation. Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023, Curran Associates Inc.
[33] Ryumei Nakada and Masaaki Imaizumi. Adaptive approximation and generalization of deep neural network with intrinsic dimensionality. Journal of Machine Learning Research, 21(174); 1-38, 2020.
[34] Thao Nguyen, Gabriel Ilharco, Mitchell Wortsman, Sewoong Oh, and Ludwig Schmidt. Qual- ity not quantity: On the interaction between dataset design and robustness of clip, 2023. URL
https: http://axxvvvvoo wwwpiivwww.r...eeooooo..eoooerre wttp:////aba/.....////// 2200. 05516
[35] Rohan Paris. Credit score classification.
https:/ www www.ka.gleeee. com/ datasets/ 2022. Kaggle Dataset, CCO: Public
Domain.
[36] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE International Con- ference on Computer Vision, pp. 1406-1415, 2019.
[37] Philipp Petersen and Felix Voigtlaender. Optimal approximation of piecewise smooth functions using deep relu neural networks. Neural Networks, 108:296-330. 2018.
[38] Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio. Transfusion: Understand- ing transfer learning for medical imaging. In Advances in Neural Information Processing Systems (NeurIPS 2019), pp. 3347-3357, Red Hook, NY, USA, 2019. Curran Associates, Inc.
[39] Johannes Schmidt-Hieber. Nonparametric regression using deep neural networks with relu activation function. The Annaly of Statistics, 2020.
[40] Michael J. Sorocky, Siqi Zhou, and Angela P. Schoellig. Experience selection using dy- namies similarity for efficient multi-source transfer learning between robots. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 3739-2745 2020. doi: 10.1 (0099//CRRR444.2000.0.9999
[41] Sören Richard Stahlschmidt, Benjamin Ulfenborg, and Jane Synnergren. Multimodal deep learning for biomedical data fusion: review. Briefings in Bioinformatics, 23(2):bbab669, 01 2022. ISSN 1477-4054. doi: 10. 10.1093/bibbbbab569, URL CRL/attps://// :/ttps:://o.... http:////o...oo.1/10/0..0 bib bbbbbbab699
13

---

Published as a conference paper at ICLR 2026
[42] Taiji Suzuki. Adaptivity of deep relu network for learning in besov and mixed smooth besov spaces: optimal rate and curse dimensionality. arXiv preprint arXi::18/0.0803)) 2018.
[43] Tedo. Students performance: analysis and classification.
btttps:/wwww.aagg/ com/
htte://edd/sttuuuee -ew.ottraaprrroaeeeeeeooea analysis and classiifiitcttooo 2018. Kaggle Notebook, Version 4, Apache 2.0 License.
[44] Karthik Chowdary Tsaliki. Diabetes classification (pima indians diabetes database). https:
7 //www. www..kaggle...........o....o com/ Soon
2019. Kag-
gle Community Prediction Competition.
[45] Zirui Wang, Zihang Dai, Barnabás Póczos, and Jaime Carbonell. Characterizing and avoiding negative transfer. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11285-11299 2019. doi: 10. 10.1109/CVP222211901155..
[46] Dongrui Wu. Online and offline domain adaptation for reducing bei calibration effort. IEEE Transactions on Machane Systems, 47(4):550-563, 2017. doi: 1091109/HHMS 2016. 2608931.
[47] Ge Xie, Yu Sun, Minlong Lin, and Ke Tang. A selective transfer learning method for concept drift adaptation. In Fengyu Cong, Andrew Leung, and Qinglai Wei Networks ISNN 2017, pp. 353-361, Cham, 2017. Springer International Publishing. ISBN 978-3-319-590811
in Neural
[48] Yi Yao and Gianfranco Doretto. Boosting for transfer learning with multiple sources. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1855- 1862, 2010. doi: 101100/CV..20000538577.
[49] Wen Zhang, Lingfei Deng, Lei Zhang, and Dongrui Wu. A survey on negative transfer. IEEF/CAA Journal of Automatica Sinica, 10(2):00--329, 2023. doi: 10.1 10.1109//S.202.. 106004.
[50] Jinxin Zhou, Xiao Li, Trayyu Ding, Chong You, Qing Qu, and Zhihui Zhu. On the optimization landscape of neural collapse under mse loss: Global optimality with unconstrained features. In International Conference on Machine Learning, pp. 27179-27202. PMLR, 2022.
14

---

Published as a conference paper at ICLR 2026
APPENDICES
In the appendices, we provide additional technical and empirical details. Appendix A provides the proof of the main theorem, supported by auxiliary lemmas Appendix B Appendix C expands the empirical evaluations, including additional results more benchmark data, tabular data, and an ablation study. Appendix D documents the experiment setup and implementation details for reproducibility. Together, they offer a complete account of the theory, validation, and practical details underlying our work.
A PROOFS OF MAIN RESULTS
In this section, we prove the main results in Section
Additional Notation. Let || lle,, denote the Ty norm under the probability measure Pxx for any E oo), where Px is the distribution of X; in the training data. For a, E IE we define A minfa, b} and V max(a, b}.
In addition, we would like to recall (g) E E(x,y)) (((()) - r).]. As result,
Re - Rr = (ixx)~-((((( - (X) - e) o²
=
11g f 122
(S.1)
where the last equivalence x is due to the fact that we assume X has positive continuous density on [0, 1] bounded by an absolute value. As [0, 1] is compact space and the density function of X is continuous, this implies that the density function is both upper and lower bounded by absolute constants.
A.1 GENERALIZATION ERROR UPPER BOUND
We now prove the main theorem on the prediction risk of REFINE. The results of the two corollaries can be obtained straightforwardly, and thus omit their proofs.
Proof of Theorem 4.1
Recall that o² is defined as the optimal linear probe of frep. i.e.,
arg min [[ - v frep())33]. vERr
We begin by observing that the difficulty of the estimation problem is governed by the residual r" ;= f - frep. since fee is assumed to be known, and 8 frep can be seen as a linear function of the known quantity. By appropriately choosing the parameters W, L, and B, we control the complexity of the neural network, and the bias of estimating r".
Specifically, choose
L = (2+ [log $1) 11 2) W = -GE B (0" 14,
(S.2)
where e, cs 0 are constants appearing in Lemma B.2 Define p" := ||r lces. Set
é ::= n -B((22+))) 23/(28+d) A
(S.3)
where P > 0 is some tuning parameter. The choices in (4) are realized by taking = n (2( 23/(28+d) and setting e₁ = (2 [log, (1) + B/d), c2 = d, C3 == cg.
Note that
1 lax < 2. From Lemma B.3 with the choice 8 1/n, have
log log Ga.p (W. L, B; fep): (1.0)) 1 1
Blllg f* 11,,]
<
inf llg
f
gEg
Next compute the first term and the second term separately.
15

---

Published as a conference paper at ICLR 2026
Part 1: Bounding the first term.
Suppose for that p² 0. Rescale the residual by noting
that (1/0")r* E Ce. Then, by Lemma B.2 there exists neural network INN Ha(W, I,, B) such that
INN (1/D")r"lla> S
(S.4)
This inequality provides the approximation error of the ReLU network class. To translate this result the bias term [|g ff llig we proceed as follows. Write
TNN INN,L 7-N,L-1
(2),
where (x)) = (A + be) for & E [L - 1] and (N), = A + br.- Define ( = (p" (0 - (p*br) to approximate o² TNN. Then, it follows that the function
gx(x) == 1 (A(((1))))))))))) N, LNN,L-1 ....o (x)) + on T fre((r)
belongs to Gd.p W, L, B: frep) since ||0*|| - L. Moreover, we can write d as
= ((-))= V. + T fre((x).
Using S.4.
we have
-X) ((x))3
11 (( ((-1) p'NNN) TT Srep
-111
1
1
p"
V
VINN
0
p
1.2
S p²
INN =p²
L2
2 p'e,
where we used the fact that ||r* /p0 - || / /0" lla S 11/0". Thus,
inf gW Biyp))
Bllgg f 1221] s Blllg 1'1l2a)) S 222
(S.5)
instead P² = O, then we can simply choose a ReLU network in H. B) with all weights and biases set to zero. By taking 9" = 0 u* fiep- the bound in equation S.5 triviilalyy holds.
Part 2: Bounding the second term. The covering number bound from Lemma B.4 with the choice of W, L, in (S.2), we have
log.N(1 (1/n,, (W, frep), 11e < C' (E d/B +p) log -
(6-1//
n1
where C' a constant depending on d and B.
Part 3: Balancing terms. Finally, we combine the results from part 1 and part Recalling the choice of e in (S.3), we consider two cases depending on the value of
When 1/v7 < p. we have = (np²) 8/(28+d), In this case, the bound becomes
Blllg [ S p²pp 40/////22202 (2)())))) -28 22/((2+)) + C² (22d/(22+d)) (28 (3/(2++d) +
P log n n
-48/(2840) + 2d/(20+d) log n)n 23/(28+d) + p log n s (C' + 1) 102
(S.6)
When p S 1//n (so that c 1), the bound becomes
x2 c² log ploog
(C' +1) ( -000000001) -28/00++)) p plogn
Ellg f fd
(S.7)
n
Combining the bounds in (..6) and
S.7 with (S.1 we obtain the desired result.
This completes the proof of Theorem 4.1
16

---

Published as a conference paper at ICLR 2026
A.2 WORST CASE NO-NEGATIIV- GUARANTEE
Proof of Corollary 4.4 By a similar argument as in the proof of Theorem given any Srep satis- fying T Srep Cu for all unit vector S the prediction risk can be upper bounded
R RR (f*) y20-an 20t4 log n + plog (F
sup
(S.8)
n
For the linear adapter, for any estimator iú, the prediction risk with respect to f satisfies
E (w X1 (X1))) (w X1 (*()))) + El " X -ww
where w" arg wen El(w" X (X)) This follows by the normal equation
(xx( w w x)] = 0,
which imply that the cross term vanishes. Under the model in equation the estimation term is of order (ppnn). Hence we have
sup f- (=
[1 fre) (r)
sup inf Ellly Tip((1)) (x1)) P (EFF(Mmm1)
VERY
n
(S.9)
For the training from scratch, note that yCl C (Frepp 7). By Theorem 3.2 in Györfi et al. [3
we have
sup ((mm))
B[ - RB Rx( z inf sup rxcc"
B[(((X)) - - (X()) / 24/(2 (28
(S.10)
Therefore, combining equation
S.9 and equation S.10 with equation concluue the proof.
A.3 ASYMPTOTIC NO-NEGATIVE- GUARANTER
Here we compare the performance REFINE with two natural baselines: training from scratch with comparable model capacity, and fitting a linear probe on frep. without any parameter tuning.
Proposition A.1 (Asymptotic No-negative No-negative-transfer transfer Guarantee). Assume 2² frep - f C². Sup- pose that the parameters (W, L,, B) satisfy log(nBB'-(W + +))) o(n) and that p plogn = o(n) n oc. Consider the model trained from scratch and the linear probe on frep with compa- rable capacity:
Dsc arg min 1 Σ ((g(X)); Yt), Wi = arg min gEHa(W.L.B)
Σ iE(n]
f(w fww Smp.)))X (S.11)
kE(m)
wEBP WERP.||w]]
Then,
R (G) -2- s (1 o(1) ((PP RR (r()"),, Re (10 frep) Rx: Rx)(()]] o(1),
(S.12)
as n DC.
Proposition A. shows that, provided the model capacity increases slowly enough with n so that the estimation error vanishes, the excess risk of REFINE is bounded, up to multiplicative and an additive o(1) term, by the smaller of the two alternatives: training a comparable model from scratch or fitting linear probe frep- In particular, REFINE is asymptotically moderate choice of (W, L, B). We also note that the choice of (W, L, B) the following theorem satisfies the conditions of Proposition A.I
worse than either baseline for any
We now prove the proposition that REFINE avoids negative transfer asymptotically, which provides result when either Rr. (v frep) Ryx (f.c) is bounded from below co.
17

---

Published as a conference paper at ICLR 2026
Proof of Proposition A.7 Consider the hypothesis classes for training from scratch and linear prob- ing:
9so (,,,, = + h(z): h HW,,,
On: := (x=+0* (x)(()) ||e|| - 1).
Let 9'c and In be the empirical risk minimizers over (w, L, B) and Gn. respectively. To case notation, we write 9 = (W, , L, B; fre). By construction, we have In U Gsc CG. Hence
E[R inf Rp( I minf inf (g),), inf RR(( S min (inx gEG
(Gn)))
(S.13)
8E9c
gE9n
By assumption,
| I 1 and sup 1 s 2. Lemma B.3
with the choice +- 1/n and
F 4 gives
(og.V(Inn 9. (1 1
-Z ( S (1 + x)) Ra (g) Rx)) + Ci
-
+
=
gEg
11
(S.14)
for some universal constant C1 > 0, where we used Rx(g) - Re((f*) Bl(g({,) - (X1))
Lemma B.4 with the choice 8 <- 1/n gives
log/N(1/nn 9, (lr..)) S C2f Co(Wlog Wogg (nLB" logg(nLnB
+ 1)4) + log n
(S.15)
for some universal constant C2 > 0. Combining equation S.13 and equation S.15 into the right hand side of equation
EE [ ([*) S (1 + x)² min [E ((x)) Rr(S"), E[ (ōn) Ry (J*))
W log (W + 1)
plog
C
n
n
where C' > 0 is some universal constant. Since W log(nLB' (W + +1) o(n) and that p log
o(n) as n + oo, we have
R
R (g) R2 (f*) s (1 +++) R²² min (in (fw)) - (F( R[[p - (F*))*)} +
Ki
where R = o11 as + co. Since E (0, 1] is arbitrary, we choose x VRA 1(= o(1)) to conclude the proof.
B AUXILIARY LEMMAS
In this section, we provide some auxiliary lemmas.
The next lemma is about the entropy bound for Ha(WW L, B).
Lemma B.1 (Lemma 21 from Nakada & Imaizumi 33].. Fix any W, L, and B > 0. Then, we have the covering number bound
log.N(e, Ha L, B), |I 11 S W log 2LB"(W> + 1) +1)²
The next lemma is modified from Petersen & Voigtlaender (37], adapted to consider L2 approxima- tion error with respect to the probability measure Fx over the domain [0, 1]d, rather than the original L2 error with a uniform measure on 1-1/2,1/2]*
Lemma B.2 (Modification of Theorem 3.1 from Petersen & Voigtlaender 37D. Fix d N+ and 3 > Suppose that Fxx has a density bounded by O(1). Then, there exist constants d, 0, depending and B, such that for any E (0, 1/2), if one chooses W, L, and B satisfying
3 =)
S (2 + [ogg B7)
11
W W See 4/8, BSC
then
inf s#ech L)..L
sup
Il - f S
18

---

Published as a conference paper at ICLR 2026
The next lemma provides bound the prediction risk of the empirical risk minimizer in terms of the covering number of the function class and the approximation error.
Lemma B.3 (Modification to Lemma 4 from Schmidt-Hieber (39 ). Let G be a function class, and let 9 be the minimizer of the empirical risk (L/n) Eie(n ((),,,, over 9 under the data generating process introduced in Section 4 Suppose that {fr Lg C {0,0 1] + [-F, F), for some F 2 Then there exists a universal constant Co > such that
( g. [km) + 8F }
(F2logN
Bllg- 1 (1+
122l
inf llg - f' + Co 6E9
nK
for all n, 5 E (0, 1}.
The next lemma provides bound on the covering number the REFINE class Ga (W, L, B; frp). Lemma B.4. Fix W E N4, E N4, B O, and 8 > Then, there exists a universal constant C O such that
lo, Ga, B; frep) (1x)))) S cf log LBB(W + +11²) + P log (
Proof. We next bound the covering number (6,, Gap (W, L, B; ftep): || .ll,,)) Note that for any
5 > 0, have
log,V(5, Gan (W, L,, B; frep): (1x,))
8
< Nog.N N
for + uh(x)) [-1,1]], he Ha(W, L, B)}, 112..
2
+ log N
2. fa + v fie(() | u E B,(1)), -l1xx)
(S.16)
Recall that Srep : [0, 11 + (1). Since || T Srep - Tt Sreplle> S ||Ie o'll for any v, of E BB,()..
standard argument shows that
N (ar + T frop(x)) UE (1)},, I1.x)) S S,N - (
(S.17)
Furthermore, since ||uj - uzhelle I l1/1 hellew + lui - u2l for any U1 U2 E [-1, 1] and h h2 HaW L, B), we have
c ta + uu((x) 1-E,-1,11, HW,, B)}, -1x
NG.
s N (....111111)) (.111.1111))
Hal(W L, B), 11,11-
11....
$ 1 N ( HaW L, B), -1
(S.18)
Note that clipping does not increase the covering number of mHa(W, L, B). Using 1616) (S.17) and (.118) combined with Lemma B. we obtain
LB" + 1))
log.N(8, Ga, L, B; fee). (I..)) S W Wlog
+ plog 5)
8
This completes the proof of Lemma B.4
C MORE NUMERICAL EXPERIMENTS
In this section, we present additional results that complement Section 5
19

---

Published as a conference paper at ICLR 2026
Nalur
0.52
Noisy Acc
Noisy AUC
Noisy F1
0.55
6:75
Low Ir Acc
Low Ir AUC
Low F1
Figure S1: Results of multi-source transfer learning under noisy and low-learning-rate conditions.
C.I MULTI-SOURCE TRANSFER
In the third experiment setting, we investigate multi-source transfer, important yet underexplored setting where knowledge is drawn from multiple heterogeneous sources to achieve better general- ization than any single source alone. Despite its practical relevance, most existing approaches, such as LinearProbe, Adapter, and Distillation, are designed for single-source transfer and do not natu- rally extend to the multi-source case. To provide fair comparison, we implement a Naive bascline that assigns each source domain its own feature extractor, concatenates the resulting representa- tions, and trains classifier top of the joint embedding. This straightforward strategy captures the most natural way of leveraging multiple sources in the absence of specialized methods. For our experiments, we partition CIFAR-I0 into eight disjoint subsets of 2000 samples each, treating them as distinct source domains and training separate CNNs on each. REFINE then integrates the corresponding penultimate representations through its modular structure, mimicking multi- source transfer while keeping inference overhead modest. This setup enables direct evaluation of princi- pled multi-source integration against naive concatenation.
Figure S1 reports the results under two stress conditions, a noisy case with 50% label corruption, testing robustness to unreliable label supervision, and low learning rate case, testing training sta- bility and efficiency. In the noisy case, REFINE significantly outperforms both Naive and Nrara as more external sources are integrated. With all eight sources, REFINE achieves classification ac- curacy 52.5%. AUC 0.8962, and F1 0.5242, compared to Naive's 's 48. 48.2%. 0.8773, and 0.4744, and NoTrans's s 49.3%, 0.8803, and 0.4871. Notably, Naive consistently performs worse than NoTrans, indicating negative transfer when external information is not integrated effectively. In the low learn- ing rate case, REFINE again improves steadily over NoTrans as the number of sources increases, while Naive suffers severe degradation. With all eight sources, REFINE reaches 34.09% classifica- tion accuracy, surpassing NoTrans's ' 30.16% and Naive's 's 22.53%. Overall, these results demonstrate that REFINE effectively integrates multiple sources, and remains robust under adverse supervision and training conditions. It avoids the pitfalls of naive concatenation and provides stable approach for multi-source transfer.
C.2 DISCUSSION ABOUT MULTIMODALITY EXTENSION
Baltrusaitis et al. survey multimodal machine learning from general taxonomy perspective, organizing existing methods into representation. translation, alignment, fusion, and co-learning paradigms. Gao et al. 88] focus specifically on deep multimodal learning techniques that emphasize neural joint representation learning and fusion strategies, assuming that all participating modalities
20

---

Published as a conference paper at ICLR 2026
Ground
UneerProbe
capsule
capsulle
cortex
cortcx
tollicee
tollice
filum
filum
medulla
madulla
madulla
per capsular adipsse tisue
pericapsular odipose Bisua
subcapsulter snus
subcapsura sinus
trabeculan
trabec Jine
Adapter
REFINE
falicia
Mulm
Mum
medulls cords
medulls cords
medulla sinuses
madulla sinuses
modulla
madulla
capsulr adipose subcapsular
pericapsuler edipose
subcapsula
trabeculze
trabeculbe
Figure S2: Spatial predictions at target cell.
known and available during training. Stahlschmidt al. [1 review biomedical multimodal fusion approaches that similarly rely on paired multimodal data and end-to-end or coordinated train- ing with all modalities present beforehand. In contrast to these settings, we define an adapt-time multimodalityy extension regime in which foundation model is pretrained on single modality, the backbone remains fixed, upstream data are inaccessible, and previously unseen modality becomes available only at adaptation time; to our knowledge, this problem formulation is not explicitly iden- tified studied in prior multi-source transfer multimodal learning literature.
Figure S2 provides additional qualitative comparisons of spatial domain predictions on the human lymph node dataset, showing ground truth alongside results from LinearProbe, Adapter, and RE- FINE, as discussed in Section 5.4
C.3 SINGLE-SOURCE TRANSFER UNDER CHALLENGING SCENARIOS
Similar to the setting considered in Section 5.3 for CIFAR-10, we run the experiments on CIFAR- 100. Moreover, in addition to CNNs, we also evaluate both CIFAR-I0 and CIFAR-100 with transformer-basee models.
Table S1 reports the results on CIFAR-100 with CNNs. Similar to CIFAR-IO, REFINE consis- tently outperforms the baseline methods under all four stress scenarios. In particular, in the extreme noise setting with 80% label flips, most competing methods collapse to near-random performance, whereas REFINE remains stable and comparable to the no-transfer baseline. In the semantic con- fusion and class imbalance settings, REFINE achieves strongest improvements classification accuracy and F1, highlighting its ability to mitigate negative transfer even when pretraining data is severely perturbed.
Table S2 and Table $3 report the results CIFAR-10 and CIFAR-100, respectively, with transformer-bate based models. Similar to CNNs, existing adaptation methods degrade sharply under noisy or imbalanced pretraining, whereas REFINE maintains stable and superior performance in accuracy, AUC, and F1.
Together, these results demonstrate that the advantages of REFINE not tied to specific model architecture or dataset size. By design, it reliably suppresses negative transfer and delivers consistent gains under challenging pretraining conditions.
21

---

Published as a conference paper at ICLR 2026
Dataset
Setting
Method
Acc
AUC
F1
MinCAcc
NoTruns
17.82 0.36
0.8259 +: 0.0068
0.1684 + 0.0039
0.60 0.49
LinearProbe
17.35
0.0015
0,1472
0.0043
0.00 0.00
Adapter
16.19 0.33
0.8578 + 0.0019
+ 0.0037
0.00 0.00
40% flips
Distill
0.22
0.8605 +: 0. 0.0035
0,161 +: 0.0024
0.00 = 0.00
LoRA
17.24
0.8568
0,1463 010053
0.00
DANN-Gate
15.02 0.39
0.8472 + 0.0020
0,1239 + 0.0041
0.00 0.00
REFINE
19.28 0.34
0.8535 + 0.00-12
O. 1805 - + 0.0043
0.10 0.80
NoTrans
17.32
0.8252 + 0.0059
O. 0.1663 + 0.0047
0,60 0 0.49
LineurProbe
1.00
0.6740 + 0.0019
0,0002 0,0000
0.00 0.00
Adapter
1.00 0.00
0.5250 + 0.0058
0,0002 + 0.0000
0.00 + 0.00
80% flips
Distill
1..11
0.8174 + 0.0069
0.1227 + 0.0039
0.00 0.00
LoRA
2.01
0.6251 + 0.0032
0,0026 0.0006
0.00 0.00
DANN-Gate
1.00 0.00
0.5754 +: 0.0113
0,0002 + 0.0000
0.00 0.00
REFINE
1.00
0.8239 + 0.0060
0.1641 + 0.0109
0.20 0.40
NoTruns
0.71
0.8129 + 0, 0.0044
0.1747 +: 0.0073
1.20 0.75
LinearProhe
20.81 +.11-+0013
0.8316 + 0.0003
0,2006 + 0.0038
0.60 0.80
CIFAR-100
Adapter
19.99 0.24
0.8308 + 0.0012
0,1895 + 0.0052
0.00 = 0.00
Schematic confusion
Distill
20.06 30.06:4..99
0.8361 + 0.0077
0.1959 + 0.0080
1.00 0.63
LoRA
20.05 0.18
0.8246 + U.0017
0.1953 0.0035
0.60 0.80
DANN-Gate
17.56
0.8122 + 0.0023
0.1720 + 0.0032
0.00 = 0.00
REFINE
21.76=0.60
0.8308 + 0.0072
0.2139 + 0.0067
2.00 1.10
NoTrans
17.58 0.24
0.8271 + 0.0033
0.1656 0.0046
LinewProbe
0.44
0.8687 + 0.0011
0,2133 + 0.00488
0.00 0.00
Adapter
2.366
0.8676 + 0.0014
0.2102 + 0.0025
0.00 0.00
Class imhalance
Distill
19.59
0.8639 0.0034
0.0072
0.00
LoRA
0.8535 + 0.0009
0,2129 + 0.0022
0.00 0.00
DANN-Gate
20.72
0.8432 + 0.0021
0.1966 + 0.0031
0.00 0.00
23.31
0.0010
0.2264 0.0032
0.40 0.49
Table S1: Single-source transfer learning with label noise, semantic perturbation, and class imbal- ance for CIFAR-100 using CNNs.
Dataset
Settng
Method
Acc
AUC
F1
MinCAce
NoTruns
45.17 + 1.39
0.8678 +: 0.0028
0.4391 = 0.0183
16.21 4.52
LinearProbe
20.65
0.6826
0.1410
0.0083
0.00
Adapter
17.88 + 0.73
0.6682
0.0066
0.1248 = 0.0111
0.00 + 0.00
80% flips
Distill
10.19 +: 0.57
0,8445
0.0022
0.3827 = 0.0068
8.00 + 5.22
LoRA
21.69 +: 0.49
0.6831
0.0010
0.1511 0.0059
DANN-Gate Gate
21.37 + 0.27
0.6829 0.0015
0.1468 0.0075
0.00 + 0.00
REFINE
15.53
0.95
0.8694 + 0 0040
0.4463 = 0.0105
18. 68 : 4.97
NoTrans
44.37 + 0.74
0.8628 0.0035
0.4375 = 0.0055
20.80 4.86
LincaPProbe
16.04
0,8643 0.0015
0.4544
23.46 4.74
Adapter
44.87 0.55
0,8514 + 0.0029
0.4445 = 0.0059
26.74 + 1.89
Domain mismatch
LoRA
47.74 0.38
0.8752 0.0015
0.4750 0.0032
27.96 2.61
DANN-Gate
47.79 0.40
0,8750
0.4733
28. 4.52
REFINE
44.855 0.38
0.8524 0.0011
0.4174 = 0.0035
29. 69.68 1.78
CIFAR-10
NoTrans
45.36 0.59
0,8662 + 0.0033
0.4455 = 0.0081
18.98 7.49
LincuProbe
53.45 0.44
0,9090 0.0002
0.5259 = 0.0078
26 6.59
Adapter
52.67 + 0.33
0,9089
0.0008
0.5195 0.0050
30.84 4.96
Schematic confusion
Distill
46.01 +: 1.11
0.8736
0.0028
0.4435 = 0.0143
14.00 6.94
LoRA
52.35 0.42
0,9024
0.0008
0.5176 = 0.0053
32.50 0.97
DANN-Gate
52.13 0.35
0,9021 0.0009
0.5141 0.0036
33.
4.33
REFINE
54.62 - 0.45
0.9134 : 0.0010
0.5431 = 0.0056
33.90 3.34
NoTrums
45.36 1.33
0.8678 0.0028
0.4391 = 0.0183
16.24 4.52
LineaProbe
18.44
0. 0.0008
0.4805
25.911 6.98
Adapter
47.57 0.27
0,8678 + 0.0029
0.4689 = 0.0045
25.26 4.53
Class imhalanace
Distill
42.25 0.63
0.8650 0.0085
0.3996 0.0051
3.86 + 0.82
LoRA
48.99 0.30
0.8759
0.0007
0.4866
0.0036
30.92 3.71
DANN-Gate
48.94 4: 0.41
0.8766
0. .0000
0.4860 = 0.0051
31.62 1.52
REFINE
47.81 + 0.23
0.8691
0.00U7
0.4755 0.0026
29.44 3.26
Table S2: Single-source transfer learning with label noise, semantic perturbation, and class imbal- ance for CIFAR-10 using transformers.
C.4 TABULAR DATA
We demonstrate that REFINE is equally effective in handling tabular data. We consider three binary- class datasets, Adult 233. Credit [35] Diabetes 44 and multi-class dataset, Performance 43
22

---

Published as a conference paper at ICLR 2026
Dataset
Setting
Method
Acc
AUC
F1
MinCAcc
NoTruns
15.32 +: 0.33
0.8449 +: 0.0021
0.1358 + 0.0041
0.00 0.00
LinearPrabe
6.70 0.27
0.7377
0,039
0.0014
0.00 0.00
Adapter
6.51
0.7405 + 0.0011
8,0340 + 0.0009
0.00 0.00
80% flips
Distill
11.83:0.20
0.8130 +: 0. 0.0027
0,8015 + 0.0024
0.00 = 0.00
LoRA
6.97 0.07
0.7390 0.0015
0,4288 (10014
0.00 0.00
DANN-Gate
6.91 0.23
0.7392 + 0.0016
0,0129 + 0.0014
0.00 0.00
REFINE
=
0.8437 + 0.0041
0.1378 - 0.0067
0.00 0.00
NoTrans
1..2
0.8023 + 0.0034
0.0884 + 0.0033
0.00 0.00
LineurProbe
0.55
0.8186 + 0.0015
0,1175 0.0049
0.00 0.00
Adapter
12.64++0.32
0.8267 + 0.0006
0.1082 + 0.0030
0.00 0.00
Domain mismatch
LoRA
0.22
0.8466 + 0.0010
0.1289 + 0.0028
0.00 0.00
DANN-Gate
14.08
0.8465 + 0.0012
0,1280 0.0023
0.00 0,00
REFINE
14.38 -..44
0.8291 + 0.0032
0.1329 + 0.0039
0.00 0.00
CIFAR-100
NoTrans
=6.24=0.58
0.8471 + 0.0036
0.1485 + 0.0075
0.00 0.00
LineurProbe
1..88
0.7950 + 0.0016
0.1067 + 0.0015
0.00 0.00
Adapter
11.17++0.43
0.7936 + 0.0027
0,0918 0.0040
0.00 0.00
Schematic confusion
Distill
15.01:00.64
0.8266 + 0.0028
0.1260 + 0.0081
0.00 = 0.00
LoRA
11.36
0.7899 + 0.0013
0.0991 + 0.0015
0.00 0.00
DANN-Gate
11.46 0.221
0.7893 + 0.0013
0.0989
0.0017
0.00 0.00
REFINE
14.94
0.8282 + 0.0026
0.1402 + 0.0026
0.00 0.00
NoTrans
0.8474 + 0.0025
U.1386 + 0.0012
0.00 0.00
LinearProbe
0.28 25.82
0.8877 +: 0.0010
0,2529
3.60 0.80
Adapter
21.18
0.88-17 + 0.0010
0,2320 + 0.0027
0.00 0,80
Class imbalance
Distill
16.01 1.011:0..0 0.13
0.8721 + 0.0017
0.1252 + 0.0021
0.00 = 0.00
LoRA
23.52
0.09
0.8669 0.0015
0.2250 0.0023
0.00
DANN-Gate
23.48 +
0.8671 + 0.0018
0,2264 + 0.0019
0.00 0.00
REFINE
25.54 0.433
0.8879 + 0.0013
0,2524 + 0.0039
4.80 = 0.75
Table S3: Single-source transfer learning with label noise, semantic perturbation, and class imbal- ance for CIFAR-100 using transformers.
Classifier
Dataset
Metric
MLPI
MLP2
Raw
DirectAug
REFINE
DirectAug
REFINE
Accuracy
0.807 + 0,008
0,831 + 0,006
0.821 +: 0.004
0.800 0.011
0.833 + 0.005
0,814 = 0.010
Adult
AUC
0.832 + 0.008
0,878 +: 0.006
0.852 ::52:00 0.008
0.833 = 0.010
0.883 = 0.005
0.854 = 0.008
0,547 + 0.037
0.619 0.015
0.595 0.030
0.570 0.028
0.627 = 0.021
0.612 = 0.021
Accuracy
0.723 1 0.028
0.740 0.022
0.717 0.027
0.732 0.015
0.726 0.020
Credit
AUC
0,730 + 0,024
0.738 + 0.013
0,745 + 0.018
0.725 = 0.025
0.754 = 0.020
0.736 = 0.023
0.490 + 0.043
0.524 + 0.030
0.520 : 0.038
0.515 0.041
0.541 = 0.030
0.535 = 0.037
Accuracy
0.565 +: 0.015
0.573 +: 0.008
0.571 0.008
0.561
0.5% 0.007
0.572
Diabetes
AUC
0.582 : 0.019
0.597 + 0.008
0.591 1 0.012
0.576 0.018
0.626 = 0.008
0.593 = 0.013
F1
0,505 + 0.028
0.533 + 0.014
0.523 + 0.022
0,501 0.029
0.534 0.017
0,522 = 0.027
Accuracy
0.684 + 0.019
0.724 + 0.011
0.711 0.014
0.683 0.018
0.668
0.084
0.702 = 0.022
Performance
AUC
0.857 + 0.011
0,878
: 0.009
0,869 0.009
0.858 0.011
0.830 0.070
0.865 0.011
F1
0.478 1 0.025
0,557 + 0.024
0.521 + 0.029
0.478 0.027
0.494 = 0.090
0.507 = 0.035
Table S4: Single-source transfer learning with original tabular data.
Each raw training data contains K x 100 samples, where K is the number of classes. To assess model complexity, we design two multilayer perceptron (MLP) architectures: MLP1 with lower complexity, and MLP2 with a more complex structure. We also compare to DirectAug, which refers to directly combining additional data with the raw data to train the classifier.
Table S4 reports the results using the original data, and Table S5 reports the results using the noisy data with 80% flips of class labels. both settings, REFINE consistently improves accuracy, AUC, and Fl over using the raw data alone. Although DirectAug can sometimes perform better through full data merging, REFINE surpasses it on several datasets, including Credit and Performance, con- firming its ability to exploit useful auxiliary information without over-relying on data merging. In the presence of heavy label noise, DirectAug suffers severe degradation, whereas REFINE maintains or slightly improves performance. Overall, these results show that REFINE is effective tabular data, and offers a safe and reliable mechanism for leveraging additional data compared to direct augmentation.
23

---

Published as a conference paper at ICLR 2026
Classifier
MLPI
MLP2
DirectAug
DirectAug
REFINE
Accuracy
0.808 +: 0.007
0.615 + 0.046
0.805 0.008
0.800
0.010
0.641 0.052
0,791 = 0.016
Adult
AUC
0.834 + 0.009
0.612 +: 0.051
0.832 0.010
0.834 = 0.013
0.639 = 0.052
0.828 = 0.014
0.549 + 0.046
0.383 + 0.039
0.555 0.029
0.564 0.032
0.395 0.047
0,562 = 0.027
Accuracy
0.723 +: 0.027
0,581 +: 0.035
0.705
0.028
0.716 0.027
0.599 0.045
0,705 0.028
Credit
AUC
0.728 + 0.027
0,578 + 0.045
0.705 + 0.028
0.720 0.026
0.599 = 0.045
0,687 = 0.028
0.483 + 0.049
0.417 + 0.048
0.481
0.035
0.512 0.041
0.433 + 0.041
0.493 0.034
Accuracy
0.587 +: 0.007
0.516 + 0.007
0.575 0.006
0.614
0.006
0.551 = 0.016
0.609 = 0.004
Diabetes
AUC
0.580 + 0,020
0,516 + 0.014
0,554 1:44
0.577
0.017
0.585 0.019
0.567 = 0.020
F1
0.503 + 0.032
0.489 + 0.025
0.498 + 0.022
0.503 0.026
0.483 = 0.037
0,514 = 0.025
Accuracy
0.682 + 0.020
0.637 + 0.088
0.696 0.023
0.684
0.018
0.650 0.079
0.69 = 0.023
Performance
AUC
0.857 + 0.011
0.805 + 0.074
0.862 0.011
0.859 0.010
0.814 0.068
0.863 0.012
F1
0,476 + 0.028
0,464 + 0.096
0.499 + 0.036
0.480 0.029
0,472 0.088
0,500 + 0.035
Table S5: Single-source transfer learning with noisy tabular data.
C.5 ABLATION STUDIES
We conduct an ablation study to investigate the effect of complexity of the encoder h in REFINE, by varying the width and depth of the neural network models used. Figure S3 reports the performance of REFINE under five different models with increasing complexity for h. The left panel reports the total number of trainable parameters, the middle panel reports the classification accuracy using the original data, and the right panel using the noisy data. On the original data, REFINE consistently outperforms NoTrans across all levels of complexity by considerable margin, demonstrating its ability to leverage useful pretrained features. On the noisy data, REFINE performs on par with NoTrans regardless of the complexity of h, confirming its robustness to negative transfer. Together, these results show that REFINE offers robust and reliable safeguarding against negative transfer.
Jac.
auturn
Parameter Number
Accuracy (Original data)
Accuracy (Heavy noise)
Figure S3: Ablation study for the encoder h with varying complexity.
related ablation on adapter size further confirms that negative transfer not due to insufficient parameter count. Here, 1x corresponds to the same adapter size used in the main experiment in Table As shown in Fig. S4 enlarging the adapter from 1x to 500x yields only minor fluctuations around 65-66 percent accuracy, 0.72 AUC, and 0.65 F1, and never approaches the NoTrans baseline at 68.5 percent accuracy or 0.76 AUC. In contrast, REFINE reaches 70.3 percent accuracy and 0.79 AUC, clearly surpassing both NoTrans and all adapter scales. These results show that increasing capacity alone cannot overcome the source-tate target mismatch responsible for negative transfer, while REFINE remains the only mechanism that reliably corrects it.
D MORE DETAILS ON EXPERIMENT SETUP AND IMPLEMENTATIONS
We provide additional details experiment setup and implementations for better reproducibility. All experiments are conducted on an NVIDIA A10G (Ampere) GPU with 23 GB of GDDR6 mem- ory, driver version 535.183.01, and CUDA 12.2. For semantic confusion in CIFAR-10 10 and CIFAR- 100, we construct and 47 pairs of related classes, respectively, and flip 50% each pair's samples to its counterpart, while also injecting white noise into image attributes with 0.2. For class
24

---

Published as a conference paper at ICLR 2026
Acotter
Adoeter
M6X
100x
Acaplor
Accuracy
AUC
FI
Figure S4: Performance of Adapter under varying parameter count multipliers.
imbalance, create each imbalanced pretrained subset by first sampling 10,000 images from the full training split with a fixed seed (42). In CIFAR-10, classes 0-9 are sampled with proportions [0.35, 0.30, 0.10, 0.07, 0.06, 0.045, 0.03, 0.02, 0.015, 0.01], yielding 3,500 100 images per class. In CIFAR-100, 100, the first 10 classes are designated as majority, with 400 images each, and the re- maining 90 as minority, with 100 images each, truncated total of 10,000 samples. Table S6 summarizes the experiment settings.
Dataset
Pretrained Model
Base Model
Pretrain Size
Fine-tune Size
Adapter Para (%)
REFINE Para (%%
CIFAR-CNN-relted
CNN
CNN
10000
4000
5.46
4.88
CIFAR-TF-related
Transformer
Transformer
10000
4000
CIFAR-10-FS
CNN
CNN
10000
4000
5.46
4,88
Clipart-
ResNet18 CNN
ResNet10
3000
1000
1.36
44.2
CNN
5000
100
5.46
4,88
Kitchen
Transformer
former
2000
400
2.25
96.58
DVD-+Electronies
Transformer
Transformer
2000
2.25
Table S6: Experiment settings for all data examples.
We clarify the exact model architectures used. For CNN experiments, the finetuned model a standard three-block convolutional network with channels {32, 64, 64), where each block consists of a 3 x 3 convolution (padding 1), ReLU activation, and 2 x max pooling, followed by a 512- dimensional fully connected layer and linear classifier. The pretrained CNN is a larger backbone with convolutional stages {80, 160, 320, 640, 640, 768). followed by a 2560-dimensional projection layer and linear classifier. For transformer experiments, the finetuned model lightweight vision transformer with patch size 4, embedding dimension 128, two encoder layers, a 3112dimensional MLP head, and a linear classifier. The pretrained transformer uses patch size 2, embedding dimen- sion 512, six encoder layers, 2560-dimensions projection head, and a linear classifier. For the DomainNet experiments, following standard practice, we use ResNet- 10 as the finetuned model and ResNet- 18 (from torchvision sion) the pretrained model.
E FURTHER DISCUSSION ABOUT RELATED WORK
Transfer learning. The affine model transformation (AMT) approach [32] is fundamentally dif- ferent from our setting. AMT only applies output-level update of the form fr(x) a fs(x) +b) which corresponds to a global scale and bias correction on the pretrained model. Such a transfor- mation cannot address representation-leve errors, nor can it introduce new features or modalities. In contrast, REFINE modifies adaptation at the representation level by keeping the pretrained encoder fixed and introducing an additive resid- ual encoder that corrects the internal representation. This allows the predictor to change its entire functional form rather than merely rescale outputs. The residual structure also provides a natural safety property: when the pretrained model is helpful, the residual remains small; when it is harm- ful, the residual can override it, yielding performance no worse than training from scratch. AMT does not provide this fallback guarantee and cannot accommodate new modalities, whereas REFINE can incorporate additional sources of information at adaptation time (c.g., spatial encoders added atop scGPT in spatial-omics experiments).
mismatch, nonlinear domain shift, structured encoder
25

---

Published as a conference paper at ICLR 2026
While the deep transfer learning (DTL) framework [21] also introduces auxiliary component be- yond the base representation, its goals and assumptions differ substantially from ours. DTL retrains the representation using all upstream domains jointly with Wasserstein and distance-covariine penalties, requiring full access to multi-domain source data and a fixed set of domains and modalities during pretraining. Only after this upstream retraining is completed is the target-domain predictor then fitted under an independence constraint. In contrast, REFINE assumes a fixed pretrained model from the outset and introduces residual encoder only at adaptation time correct the frozen rep- transfer guarantee and fallback to the target-only estimator properties not provided by DTL. Moreover, because DTL assumes that no new modalities appear after upstream training, it cannot handle scenarios such as spatial-omics where new sources of information become available exclusively at adaptation time, precisely the regime targeted by REFINE.
resentation on the target distribution. This design enables our no- -ereeiie---rrnserr
Baseline selection for negative-transfer evaluation. Our selection of baselines follows re- cent recommendations from studies on negative transfer (NT) and parameter-efficient fine-tuning (PEFT). Importantly, our goal is not to benchmark raw target accuracy against the newest domain- alignment algorithms,, but evaluate robustness to negative transfer, for which the modern literature identifies only small set of meaningful baselines, Recent PEFT analyses [31] show that most con- temporary PEFT variants behave similarly under distribution shift and provide little or no protection against negative transfer; thus, LoRA serves as a representative and widely used PEFT baseline for NT evaluation. Likewise, the NT survey literature emphasizes that very few modern transfer- learning methods are explicitly designed with safety objectives in mind; accordingly, adversarial domain adaptation approaches such as DANN remain the standard benchmarks used in NT studies [49]. Many newer transier-learning methods primarily target domain alignment or feature matching but lack any mechanism for safety or fallback, so including additional variants would not meaning- fully strengthen the evaluation. Consistent with the NT literature (49), we therefore adopt bascline set that directly probes safety: NoTrans, feature-base adaptation (LinearProbe and Adapter), ad- versarial DA (DANN), and one representative PEFT method (LoRA). These baselines provide the appropriate lens for assessing whether REFINE achieves its intended property avoiding negative transfer rather than merely improving average accuracy.
Source-free multi-source transfer. Our multi-source experiment operates under a source-free, adaptation-ting setting in which the pretrained model is fixed and no upstream source data are ac- cessible during adaptation. Under this constraint, classical multi multi-source transfer algorithms that rely on joint training over all source domains, full access to source datasets, re-optimization of shared encoder, or traditional boosting [88] cannot be applied, including multi-source domain alignment methods, -fftuue-of-eeperts training frameworks, and multi-source adversarial domain adaptation approaches. These methods fundamentally assume retraining with all sources present and therefore fall outside the feasible operation regime of our setting. At adaptation time, we only receive a small number of target-like auxiliary sources, often with mismatched structure, and have no ability visit any upstream data. Consequently, the only baselines that are valid in this source-free scenario NoTrans and simple data concatenation. These baselines reflect the operations that a practitioner can realistically perform when upstream data cannot be accessed and isolate the negative-transfer phenomenon that we aim study, namely how safely incorporate multiple heterogeneous sources without degrading downstream performance.
26


Reviews:
[{"review_id": "QSANKsnJNM", "reviewer": "Reviewer_VFA3", "summary": "The authors identify residual connections as a powerful mechanism for provably avoiding negative transfer, and based on this insight, propose a simple yet effective method termed Residual Feature Integration (REFINE). Theoretically, the authors formally justify this simple yet remarkably effective approach through a rigorous analysis. Empirically, the authors conduct extensive experiments across image, text, and tabular benchmarks, and demonstrate that REFINE consistently mitigates negative transfer under diverse scenarios, including distribution shifts, label noise, semantic perturbations, class imbalance, and multi-source transfer settings.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The manuscript is well written, logically structured, and easy to follow.\n- The authors provide theoretical analysis.\n- According to the empirical evaluations, the proposed simple approach achieves good performance across image, text, and tabular benchmarks, and under challenging conditions such as distribution shift, label noise, semantic perturbation, and class imbalance.", "weaknesses": "- The current setting is restricted to supervised fine-tuning. The proposed method requires access to labeled target-domain/task data, which is a relatively strong assumption even in conventional transfer learning [1]. Could the authors provide insights into whether the method would still work under unsupervised domain adaptation (UDA) [1] (does not require target-domain labels) or source-free domain adaptation (SFDA) settings [2-3] (does not require access to either the source data or target-domain labels)? Empirical validation or theoretical discussion in such scenarios would substantially enhance the generality of the method.\n- The manuscript does not investigate how the performance varies with different amounts of target-domain data used for transferring.\n- This manuscript lacks clarification of model architectures and size. The specific architectures of the CNN and Transformer models used in experiments are not clearly described. This makes it difficult to assess how architectural design and parameterization influence the effectiveness of the proposed method. \n- For intentional and extreme negative transfer, such as non-transfer learning [4-6] (where the goal is to enforce strong negative transfer for model intellectual property protection), will the proposed method can still work? It would be insightful if the authors could discuss or empirically investigate the applicability or limitations of the method in such settings.\n\n[1] Unsupervised domain adaptation by backpropagation. ICML 15\\\n[2] Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation. ICML 20\\\n[3] A Comprehensive Survey on Source-free Domain Adaptation. TPAMI 23\\\n[4] Non-Transferable Learning: A New Approach for Model Ownership Verification and Applicability Authorization. ICLR 22\\\n[5] Your Transferability Barrier is Fragile: Free-Lunch for Transferring the Non-Transferable Learning. CVPR 24\\\n[6] Toward Robust Non-Transferable Learning: A Survey and Benchmark. IJCAI 25", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QEx1zv9D3n", "reviewer": "Reviewer_PRLw", "summary": "The paper proposes a transfer learning algorithm designed to prevent negative transfer. If the source model provides useful information, the algorithm leverages it; otherwise, its performance does not fall below that of training solely on the target data. It assumes the existence of a pretrained source model $f_{\\text{rep}}$. A residual trainable network $h$ is then added, and a linear trainable model $w$ is applied on top of the concatenated representation  $(f_{\\text{rep}}(x), h(x))$.", "questions": "What is the practical justification for the assumption that $||v^*||\\leq 1$ ?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper derives an error bound for the proposed algorithm, comparing its error to that of the ground truth regression function. Furthermore, the authors conduct numerical experiments demonstrating that their approach is not affected by negative transfer.", "weaknesses": "1. The paper claims that its approach avoids negative transfer and that Equation (1) holds for their method, stating that the error of the proposed approach is less than the minimum of the errors obtained by using only target data and by training on top of the source model $f_{\\text{rep}}$. However, the theorem they provide does not justify this claim. In fact, Theorem 4.1 only establishes a rate guarantee relative to $f^*$, not an inequality comparing $\\hat{g}$—the algorithm’s output—with $\\min \\\\{\\text{only target}, w \\circ f_{\\text{rep}}\\\\}$. In particular, Theorem 1 does not imply this claim. Therefore, the main claim of the paper remains unproven.\n\n2. A major difficulty with the algorithm lies in tuning the hyperparameter $\\rho$, yet the paper provides no method for adaptively selecting this parameter.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KKJi2zib71", "reviewer": "Reviewer_fQBT", "summary": "This paper introduces a transfer learning method called REFINE for mitigating the phenomenon of negative transfer. Specifically, the authors employ an additional term in addition to composing a linear probe based on the learned representation from the source domain. They provide several experiments to demonstrate the effectiveness of the proposed method. Moreover, a upper bound of excess risk regarding fine-tuning is also offered by this study.", "questions": "The authors claimed that they desire to learn $h$ and $w$ such that \n$$\n\\mathcal{R}\\_{\\mathbb{P}^t}(w \\circ (f\\_{rep}, h)) \\leq \\min\\\\{\\mathcal{R}\\_{\\mathbb{P}^t}(w_{ft} \\circ f_{rep}), \\mathcal{R}\\_{\\mathbb{P}^t}(g\\_{sc})\\\\}\n$$\nI wonder did authors demonstrate this holds for ERM $(\\hat{w}, \\hat{h})$ at somewhere of this paper? Or this is just a desire? If this can be justified, what kind of requirements are necessary for the risk $\\mathcal{L}$?\n\n**Summary of review:** I think this is a solid paper overall, considering its comprehensive and sound experiments as well as the theoretical guarantees provided. However, the originality of the idea is not clear, along with some experimental concerns mentioned above. Therefore, I would like to set my initial rating at 4. If author successfully resolve my concerns, I will be pleasure to raise my rating.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "1. This study is well-written and easy to follow. The motivation behind their method is reasonable.\n2. They conducted comprehensive transfer learning experiments to demonstrate the effectiveness of their method in transfer learning scenarios, which are sound.\n3. Meanwhile, the theoretical results obtained by this work are rigorous.", "weaknesses": "1. The core idea of this paper is similar to that of the following papers. Both add extra terms to model the potential differences between the source and target tasks that are not successfully captured by the learned representation, in addition to the linear probe on the learned encoder. I sincerely suggest that the authors discuss the differences between their work and these studies in the related works section.\n\n[1] Transfer learning with affine model transformation. Advances in Neural Information Processing Systems, 36, 17296-17329.\n\n[2] Deep transfer learning: Model framework and error analysis. arXiv preprint arXiv:2410.09383.\n\n2. Some of the presented accuracy results seem a little bit weird. For example, the results transferring from CIFAR100 to CIFAR10 are lower than expected. Based on my experience, it should be around 80%, rather than the figure given in the paper (around 40%). The reason for this phenomenon seems to be that they only adopt 10,000 as the pretraining size. I don't understand the rationale behind such a design. Can the authors elaborate on that? This raises a concern about whether the conclusion still holds under sufficient pretraining.\n\n3. The experiments conducted in this study are all focused on small-scale datasets. But I am totally okay with that, given that they have a solid theoretical guarantee.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YPhlwxtWh1", "reviewer": "Reviewer_cD6H", "summary": "This paper studies transfer learning and proposes a residual feature integration method to mitigate negative transfer. The authors also introduce an upper bound for the generalization error under a nonparametric regression setup. However, the theoretical contribution is somewhat limited, and the empirical comparisons lack depth in several aspects.", "questions": "1. The proposed model, \\hat{g}(x)=v⊤frep(x)+uh(x) can be viewed as a ReLU network directly according to the descriptions on P5 . The approximation analysis for such architectures has been extensively studied in prior works, and the theoretical guarantees provided here do not appear to deviate significantly from existing results.\n2.The baseline methods selected for comparison—such as DANN and LoRA—are relatively outdated. Several influential and more recent transfer learning strategies are not included, which limits the relevance and comprehensiveness of the experimental evaluation.\n3. A detailed comparison of the number of parameters across different methods is missing. This makes it difficult to assess the efficiency of the proposed approach relative to existing alternatives.\n4. In the multi-source transfer experiments, only two simple baselines (NoTrans and Naive) are used. This is insufficient to demonstrate the advantages of the proposed method in multi-source settings, where more competitive or state-of-the-art approaches should be considered.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.The paper identifies a simple, intuitive, and model-agnostic method (ReFine) to address the persistent problem of negative transfer. The strategy of augmenting a frozen pre-trained feature extractor with a trainable residual encoder is straightforward to implement and integrates easily into existing transfer learning pipelines.\n2.The work provides a rigorous theoretical analysis demonstrating that ReFine provably prevents negative transfer. \n3.The method is evaluated across a wide range of settings, including image, text, and tabular data. Experiments cover natural distribution shifts and challenging stress tests (e.g., high label noise, semantic perturbation, class imbalance), consistently showing that ReFine mitigates performance degradation where other methods fail.", "weaknesses": "1.While the theoretical analysis is rigorous, the core architecture—a linear combination of a pre-trained feature map and a ReLU network—is structurally similar to models previously analyzed in the deep learning theory literature. The resulting generalization bounds, though well-derived, may not represent a significant conceptual advance over existing approximation theories for neural networks.\n2.The selection of baseline methods lacks contemporary relevance. Compared approaches like DANN and LoRA are established but not state-of-the-art. The empirical evaluation would be more compelling if it included more recent and influential transfer learning strategies, providing a clearer view of the method's current competitive standing.\n3.The paper does not provide a detailed comparison of the number of trainable parameters across all methods. This omission makes it difficult to assess the true efficiency of ReFine and to perform a fair comparison with parameter-efficient fine-tuning baselines like Adapter and LoRA.\n4.The multi-source transfer experiments are evaluated against only two simple baselines (NoTrans and a Naive concatenation). This setup is not sufficient to demonstrate a clear advantage, as it lacks comparison with dedicated and potentially stronger multi-source transfer learning algorithms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)