Title: A High-Dimensional Statistical Method for Optimizing Transfer Quantities in Multi-Source Transfer Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: multi-source transfer learning

K-L divergence

high-dimensional statistics
Abstract: Multi-source transfer learning provides an effective solution to data scarcity in real-world supervised learning scenarios by leveraging multiple source tasks. In this field, existing works typically use all available samples from sources in training, which constrains their training efficiency and may lead to suboptimal results. To address this, we propose a theoretical framework that answers the question: what is the optimal quantity of source samples needed from each source task to jointly train the target model? Specifically, we introduce a generalization error measure based on K-L divergence, and minimize it based on high-dimensional statistical analysis to determine the optimal transfer quantity for each source task. Additionally, we develop an architecture-agnostic and data-efficient algorithm OTQMS to implement our theoretical results for target model training in multi-source transfer learning.  Experimental studies on diverse architectures and two real-world benchmark datasets show that our proposed algorithm significantly outperforms state-of-the-art approaches in both accuracy and data efficiency. The code is available at https://github.com/zqy0126/OTQMS.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to answer the question of “what is the optimal sample size for each sources when conducting transfer learning.” The authors proposes a parametric model-agnostic framework to find out the optimal size for each source by minimizing the expected KL divergence between true target distribution and the pooled-version estimated distribution w.r.t. source sample sizes. Practical algorithm is presented and the effectiveness of the proposed algorithm is evaluated through experiments.\n\nSTRENGTHS AND WEAKNESSES:\n## Strength\n\n1. To my knowledge, while most existing transfer learning work focus on finding the optimal sources, this paper goes with another direction by focusing on finding the optimal sample size the algorithm should use from each source.\n2. The methodology is theory-driven.\n\n## Weakness\n\n1. **Terminology**: In theoretical statistics or machine learning community, the term “high-dimensional” usually refer to the cases where the dimension of the data fast excess the sample size, i.e., $d \\gg n$, or the dimension grows as sample size increase. In such regime, many theoretical results in fixed dimension no longer hold and require different techniques or mathematical tools. For example, the asymptotic normality, i.e., eq (2), could fail under high-dimensional regime. I believe what authors consider is a fixed dimension $d$, which aligns with the classical multivariate analysis. Using “high-dimensional” is rather misleading. \n2. **The set-up**: Although I don’t appreciate the idea that the authors consider the data are generated from a specific network parameterized by $\\theta$ cuz it can produce some specification problem, it generally does not affect the core idea of the paper. However, the assumption that all source tasks and the target task share the same parametric model is worrying, which is practically impossible. This assumption is essential for the derivation in Section 4, as if it does not hold, the proposed methodology does not stand.\n3. **The estimator**: The transfer learning estimator considered in this manuscript is rather simple, which is the minimizer of the pooled log-likelihood. It has been too simple to provide helpful insight into the current transfer learning paradigm, i.e., pre-training and fine-tuning. Pooling both source and target data to obtain an estimator is never effective and efficient, as has been long demonstrated in the community. \n    \n    Generally, I can understand why authors need to consider such estimator. This pooled version allows it corresponding distribution be the weighted combination of all source and target distribution (where the weights are determined by the sample size ratio). This allows the authors to derive the expression of expected KL divergence easier under the (1) same parametric model assumption (point 2) and (2) sufficient close models assumption (line 139). However, with so many restriction, again, it is too hard to have the theory provide insights.\n    \n4. **Assumption**:  The bounds in this paper are asymptotic and the authors require a “sufficient small models” assumption (line 139 or line 145) when derive these asymptotic results. However, in asymptotic regime, the “sufficient small models” actually indicates the source and target tasks/models are identical. Therefore, it is not surprised why the assumptions on same parametric model and pooled estimator are needed.\n5. **Practical algorithm**: The practical algorithm (QTAMS) replaced the true source parameters by their empirical version. While this is a compromise with reality where the true parameters are unknown in practice, we have to remember that, since the bounds are asymptotic, there could be error terms induced by the approximation that are not lower order of $1/(N_0 + n_1)$ in eq (8), (12), (14). This will make the optimal solution for $n_{k}$ no longer follows the solutions derived from the objective in eq (8), (12), (14).\n6. **Experiments**: I do not closely work on applications, so I typically have no idea what SOTA results would be, therefore, please view my comments on this point dialectically. For me, QTQMS only improved very limited compared to existing approaches/baselines, i.e., the improvement over the runner-up is rather small. What is more interesting to me is that the efficiency improved by QTQMS.\n\nQUESTIONS:\nThe authors provide a numerical approach in Appendix F to get the solution to objective (14). The authors proposed to do the grid search by first fixing $s$ in some feasible domain of $s$. However, what is the feasible domain of $s$ in general? What is it order? If no prior knowledge kicks in, the range of the feasible domain of $s$ can go quite large and ruin the computation.\n\nLIMITATIONS:\nYes, although authors only put limitations in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper discusses the optimal quantity of source samples from each source task. A generalization error measure based on KL divergence is introduced. The quantity of source samples can be solved based on high-dimensional statistical analysis. Experiment results demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n1. The task that estimates the optimal quantity of source samples holds broad application scenarios. And the theoretical results are intuitive yet not trivial.\n\n2. The derivation of the method is reasonable, and the training process is clear and is easy to generalize to other tasks.\n\n3. The authors have conducted sufficient evaluation and analysis experiments, demonstrating the effectiveness and efficiency of the proposed method.\n\nWeaknesses:\n\n1. The main theorems are derived based on the MLE estimation of the model parameters. In learning problems with other objectives, the effectiveness of the results is unclear.\n\n2. Optimizing the quantities of source samples should be considered as part of the whole transfer learning paradigm. Therefore, further discussions about the relation with sample selection and domain alignment will be helpful.\n\nQUESTIONS:\nThe questions are consistent with the \"Weaknesses\".\n\n1.\tAs the learning objective of many problems are not limited to negative log likelihood, I am interested in the optimal quantities with general learning objectives, such as cross-entropy loss in classification problems and mean square error in regression problems, or at least some error analysis when the proposed method is applied in general settings.\n\n2.\tHow can the proposed method combine with other parts of the transfer learning? For examples, the sample selection and domain alignment. I understand these may be a bit  out of the scope of this paper, so it's only a suggestion and the answer will not decrease the score.\n\nLIMITATIONS:\nThe authors have mentioned the sampling method in the discussion of limitations, but the limitations in the general training objective have not been discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel theoretical framework for determining the optimal number of samples to transfer from each source task in a multi-source transfer learning setting. The authors formulate the problem as a parameter estimation task and leverage high-dimensional statistical analysis to derive the optimal transfer quantities by minimising a generalisation error measure based on Kullback-Leibler (K-L) divergence. This framework is implemented in a practical, architecture-agnostic algorithm named OTQMS. The method is empirically evaluated on standard benchmarks, demonstrating superior performance in both accuracy and data efficiency compared to existing approaches.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n- The paper addresses a critical and often overlooked question in transfer learning: not just which sources to use, but how much data to use from each. \n- At its core, the paper proposes a solid theoretical analysis. It introduces the expected K-L divergence as a measure of generalisation error, which is well-justified by its connection to cross-entropy loss. The use of high-dimensional statistical analysis to derive the optimal transfer quantities is rigorous, and the proofs are clear.\n- The theoretical findings are successfully translated into a practical algorithm, OTQMS. A key feature is the dynamic strategy that iteratively updates the estimate of the target parameter $\\theta_0$​ and re-computes the optimal quantities, mitigating the issue of having limited target data. The algorithm is architecture-agnostic nature and its demonstrated compatibility with both full model training and parameter-efficient methods like LoRA.\n- The authors conduct extensive experiments on multiple real-world datasets (DomainNet, Office-Home, Digits) and across various settings. OTQMS consistently outperforms a range of state-of-the-art baselines in accuracy.\n\n# Weaknesses\n\n- The theoretical derivation for the optimal transfer quantity in Theorem 4 and its extensions relies on the assumption that the parameter distance between source and target tasks is small, i.e., $||\\theta_i-\\theta_0||=O(1/\\sqrt{N_0})$. The paper argues this is a reasonable assumption supported by related work and that its conclusions hold even when the distance is large. However, this is a strong condition that may not hold in cases of significant domain shift. The theoretical guarantees are strongest for closely related tasks, and the robustness in practice does not fully alleviate this theoretical limitation.\n- The framework models the difference between tasks solely as a parametric distance, $\\theta_i-\\theta_0$, within a given model class $P(X|\\theta)$​​. While this general formulation is a strength, it also prevents a deeper analysis of why tasks differ. The framework does not distinguish between different transfer learning scenarios, such as covariate shift (change in the input distribution $P(Z)$) versus concept shift (change in the conditional distribution $P(Y|Z)$). Consequently, it cannot provide insights into how these distinct types of domain shifts affect the optimal transfer quantity. The parameter vector $\\theta$​ conflates these potentially disparate effects.\n- The OTQMS algorithm requires computing the optimal transfer quantities $(s,\\alpha)$ in each training epoch. This is achieved by solving a quadratic programming problem for $\\alpha$ within a numerical search for $s$. While the paper reports a net reduction in training time due to using fewer samples, it does not analyse the computational overhead of this selection step itself. In scenarios with a very large number of source tasks (large $K$), this step could introduce a non-trivial computational burden that might offset the gains from training on a smaller dataset.\n- The dynamic strategy for estimating the target parameter $\\theta_0$ begins with a model trained only on the (often scarce) target samples. The accuracy of this initial estimate can be poor, potentially influencing the subsequent iterative optimisation process. While the dynamic approach is shown to be superior to a static one, the sensitivity of the final performance to this initial, potentially noisy, estimate of $\\theta_0$ is not explored.\n\n# Conclusion\n\nThe formulation of a method to optimise transfer quantity is a significant contribution, and the empirical results are compelling. However, the theoretical claims are narrower than the empirical results might suggest due to restrictive assumptions. I believe that the paper deserves acceptance for its novelty and practical impact, but its limitations should be clearly acknowledged. The theoretical framework is more of a good motivation for an optimisation strategy than a general theory of transferability.\n\nQUESTIONS:\nMy questions are related to the weaknesses mentioned in the critical review above. I will summarise my main concerns in the form of questions for the authors:\n- The theoretical framework relies on the assumption that task parameters are asymptotically close ($|\\theta_{0}-\\theta_{1}|=O(\\frac{1}{\\sqrt{N_0}})$). While the empirical results appear robust, what are the theoretical implications when this assumption is strongly violated, for instance, in cases of far-domain transfer? How does the framework behave, and could it be extended to formally account for larger parameter distances?\n- The framework models task divergence as a unified parametric distance ($\\underline{\\theta}_i - \\underline{\\theta}_0$), which aggregates different sources of domain shift. Can the framework be decomposed to provide more granular insights into how different types of shift (e.g., covariate shift vs. concept shift) individually impact the optimal transfer quantity?\n- The dynamic OTQMS algorithm initializes the target parameter $\\underline{\\theta}_0$ using only the limited target data before iteratively refining it. How sensitive is the final model performance to the quality of this initial, and potentially noisy, estimation? Have you investigated whether a poor initialization could lead the optimization to a suboptimal solution?\n- The paper highlights significant reductions in training time. However, the OTQMS algorithm introduces a selection step in each epoch that involves solving a constrained quadratic program. Could you provide an analysis of the computational overhead of this selection process itself, particularly as the number of source domains ($K$) increases?\n- In the high-dimensional case, the task discrepancy term $t$ is an average over the parameter dimension $d$. Does this averaging potentially mask the impact of large discrepancies in a small subset of critical parameters? Would a dimension-weighted approach offer further improvements?\n- On page 4, you state the assumption $||\\theta_i-\\theta_0||=O(1/\\sqrt{N_0})$ is \"supported by related studies [20]\". Could you elaborate on how the cited work [20] specifically justifies this scaling law for parameter distance in the context of transfer learning?\n\nLIMITATIONS:\nMy main comments on the limitations of this work are detailed in the \"Weaknesses\" section of the review above. To summarise, the primary limitations are: (1) the restrictive theoretical assumption that source and target task parameters are asymptotically close, which may not hold in cases of significant domain shift; (2) the framework's inability to distinguish between different types of transfer (e.g., covariate vs. concept shift), limiting its explanatory power; and (3) unanalyzed sensitivities in the practical OTQMS algorithm, namely its dependence on a potentially noisy initial estimate of the target model and the unexamined computational overhead of the sample selection step.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles a fundamental problem in multi-source transfer learning (MSTL): how to eliminate negative transfer from a sample-level perspective. The authors aim to determine the optimal number of samples to transfer from each source task to best benefit the target task. Unlike existing approaches that transfer all available data or rely on heuristic selection strategies, the paper proposes a theoretically grounded framework for transfer quantity optimization.\n\nSpecifically, the authors introduce a generalization error measure based on the expected Kullback-Leibler (KL) divergence between the true target distribution and the learned model. They leverage asymptotic high-dimensional statistical analysis to derive closed-form expressions for the optimal transfer quantities in both single-source and multi-source settings. This theoretical foundation leads to the development of OTQMS, a data-efficient and architecture-agnostic algorithm. Furthermore, the paper proposes a dynamic training strategy that iteratively updates both the estimation of the target model parameters and the corresponding optimal transfer quantities during training.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n\nThe paper presents a theoretically grounded formulation for optimizing transfer quantities in multi-source transfer learning. By leveraging KL-divergence-based generalization error and asymptotic high-dimensional statistical analysis, it offers a novel and mathematically rigorous approach. A key strength of the work lies in its focus on sample-level optimization, which allows for fine-grained control over transfer and directly addresses the problem of negative transfer, unlike prior approaches that operate at the domain or model level. The proposed algorithm, OTQMS, is both architecture-agnostic and data-efficient, enabling effective knowledge transfer while minimizing unnecessary sample usage. \nEmpirically, OTQMS demonstrates strong performance on challenging benchmarks such as DomainNet and Office-Home with notable improvements in accuracy. The method exhibits robustness across a wide range of shot settings from 5-shot to 100-shot.\n\nWeakness\n\nWhile the proposed framework is theoretically well-grounded, it relies on asymptotic analysis, and the paper does not offer non-asymptotic guarantees or finite-sample bounds. This limits our understanding of how well the method performs when the number of target samples is extremely limited. A central step in the algorithm is estimating the Fisher information matrix using a small set of target samples. While this is a practical design choice, it raises concerns about the variance and numerical stability of the estimate—especially in few-shot settings where data scarcity may result in unreliable curvature information. Furthermore, although the authors argue that transferring too many samples can lead to negative transfer, the experiments do not clearly show this behavior. The paper does not quantify what the optimal number of transferred samples actually is, nor does it provide empirical evidence of performance degradation when that number is exceeded. This weakens the practical insight into when and how negative transfer occurs. In addition, the optimization problem in Theorem 7, involving the total transfer quantity \\$s^*$  and the proportion vector \\$\\alpha^∗$, is said to be solved numerically, but Appendix F provides insufficient detail about how this is actually done. It is unclear whether a unique or stable solution always exists, and the lack of discussion on the solvability or initialization sensitivity may hinder reproducibility and theoretical completeness.\n\nOne minor notation issue: the paper uses underlines to denote high-dimensional vectors. It is more standard and clearer to use boldface notation to indicate vector quantities.\n\nQUESTIONS:\n1. Fisher Information Estimation:\nGiven that the Fisher information matrix is estimated using only a small number of target samples, how do the authors ensure the stability and reliability of this estimate? Have they evaluated the sensitivity of the algorithm to noise or variance in the empirical Fisher?\n\n2. Negative Transfer Evidence:\nWhile the theoretical analysis suggests that transferring too many source samples may lead to negative transfer, the experimental results do not explicitly show this. Can the authors provide empirical evidence demonstrating performance degradation when the transfer quantity exceeds the theoretical optimum?\n\n3. Optimal Transfer Quantity in Practice:\nThe framework is designed to compute the optimal number of samples to transfer from each source, but the paper does not state what these optimal quantities actually are in practical settings. Could the authors report or visualize the learned transfer quantities to enhance interpretability?\n\n4. Existence and Solvability of the Optimum:\nHas the existence and uniqueness of the solution to the optimization problem in Theorem 7 been formally analyzed or empirically verified? Are there conditions under which the solution may not exist?\n\nLIMITATIONS:\nOne key limitation of the proposed framework is its reliance on asymptotic statistical assumptions, which may not hold in extreme few-shot regimes. In such cases, the empirical estimation of the Fisher information matrix based on a small number of target samples can suffer from high variance and numerical instability, potentially affecting the reliability of the computed transfer quantities", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-05"}