Title: Mixed-Sample SGD: an End-to-end Analysis of Supervised Transfer Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Transfer learning

domain adaptation
Abstract: Theoretical works on supervised transfer learning (STL)---where the learner has access to labeled samples from both source and target distributions---have for the most part focused on statistical aspects of the problem, while efficient optimization has received less attention. 
We consider the problem of designing an SGD procedure 
for STL that alternates sampling between source and target data, while maintaining statistical transfer guarantees without prior knowledge of the quality of the source data. 
A main algorithmic difficulty is in understanding how to design such an adaptive sub-sampling mechanism at each SGD step, to automatically gain from the source when it is informative, or bias towards the target and avoid negative transfer when the source is less informative. 
 
 We show that, such a mixed-sample SGD procedure is feasible for general prediction tasks with convex losses, rooted in tracking an abstract sequence of constrained convex programs that serve to maintain the desired transfer guarantees. 
 We instantiate these results in the concrete setting of linear regression with square loss, and show that the procedure converges, with $1/\sqrt{T}$ rate, to a solution whose statistical performance on the target is adaptive to the a priori unknown quality of the source. Experiments with synthetic and real datasets support the theory.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a theoretical analysis of the supervised transfer learning settings between the source and target distributions of a linear regression model. This paper shows that the returned solution is extrapolated between the source and target datasets, leaning towards the beneficial one, under the mixed-sample SGD setting. Experiments confirm the theoretical analysis\n\nSTRENGTHS AND WEAKNESSES:\nThis paper has several strengths:\n- A new theoretical result on the transfer learning setting with a linear regression (classification) model.\n- Experiments confirm theoretical results.\n\n\nWeaknesses:\n- This paper considers a simple setting where the source and target datasets share the same domain $\\mathcal{X}$ and $\\mathcal{Y}$. This setting is too strict and less applicable in practice since the transfer learning is often done between domains or datasets with the same input domain but have different labels.\n\n- The \"usefulness\" of the source dataset is indicated by the sample size, which is not always the case. In my opinion, it is chosen since this paper only focuses on the source and target datasets from the same domain for both input and output. \n\n- The presentation of this paper causes confusion, and it is not ready to appear at this venue. I recommend that authors carefully revise the manuscript to make it more readable.  For instance, Section 4.1. provides a series of lemmas and starting with an algorithm for warming up, the motivation behind the warm-up algorithm should be discussed, and the interpretation of lemmas should be also discussed carefully to understand why it is important. In addition, the importance of $\\rho$, the norm between the initial and the calculated point, should be discussed. \n\n- What is the motivation behind the use of mixed SGD?\n\n- The related works should be discussed further why this work is important in the literature instead of listing lines of works.\n\nQUESTIONS:\nPlease see the weaknesses\n\nLIMITATIONS:\nPlease see the weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes mixed-sample sgd which adaptively alternates between sampling from the source and target datasets at a rate that automatically adjusts based on the source data's quality. This approach avoids computationally expensive tuning the source data's importance weight while tracking a sequence of constrained convex programs to maintain strong theoretical guarantees. The authors prove that their method converges at a rate of $1/\\sqrt{T}$ to a solution that automatically performs as well as a model trained on either the source or target data alone, whichever is better.\n\nSTRENGTHS AND WEAKNESSES:\nDisclaimer: I am not expert in optimization so most of my reviews are educated guess. If there are something wrong, please correct me. \n\n## Strengths\n- The paper provides rigorous proof for the proposed algorithm's convergence at a standard $O(1/\\sqrt{T})$ rate and, more importantly, its adaptivity. \n\n- The paper is logically structured and does an excellent job in the introduction of motivating the problem, highlighting a clear gap in the existing literature. \n\n- The paper tackles the core, practical challenge of safely and efficiently leveraging source data in transfer learning. \n\n- The proposed Mixed-Sample SGD algorithm is original. Its mechanism of using a dynamic, learnable sampling rate to approximate a constrained optimization problem is a new approach compared to prior methods that rely on expensive cross-validation or direct projections.\n\n## Weaknesses\n\n\n\n\n- The formal guarantees are derived for convex loss functions. This is a significant limitation, as it doesn't cover the non-convex nature of modern deep learning models. The authors acknowledge this, but the gap between the convex theory and some of the deep-learning-based experiments remains.\n\n- The detailed proofs contain minor  but recurring errors in the calculation of constants, particularly regarding factors of 2 and 4 when bounding gradient norms. Since empirical risk is  $\\frac{1}{n} \\sum_{ (x_i, y_i) \\in S_Q} (\\theta^\\top x_i - y_i)^2$, \nthe gradient should be \n$$\\nabla\\hat{R}_Q(\\theta) = 2\\hat{\\Sigma}_Q(\\theta-\\hat{\\theta}_Q)$$  \n\nin lemma 8. Thus, the minimum gradient norm should be $4\\epsilon \\lambda^{+}_{\\min}(\\hat{\\Sigma}_Q)$. Similarly, the factor of 2 is omitted in line 582.\n\n- A minor typo: unversal -> universal in line 148.\n\nQUESTIONS:\nWhy is $\\hat{\\theta}_Q$ unknown? Since we're performing linear regression with finite samples, I would expect a closed-form solution to be available.\n\nLIMITATIONS:\nplease see the weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provided an analysis of the expected error when using mixed-sample SGD algorithm in the context of supervise transfer learning. The main result is presented in Theorem 1, that provided an upper bound of the expected loss at the target distribution. More particular, it includes $\\varepsilon_Q$ (best achievable rate from $Q$ target distribution) and some quantities measure the discrepancy between source and target distributions. Other theorems (2,3) and Lemmas (1,2,3,4,5,6,7) were stated to support the proof of Theorem 1. \nIn the work, the author presented an hybrid algorithm that use both source and target data to train the model, meanwhile, the ratio between their sizes was  adjusted along the training procedure. The main reason for mixing data in the SGD is that the source-target problem has two minimization procedures at the same time, that is connected through a parameter in the Lagrangian form. The authors proposed to use mixed data with adaptive rate to deal with the situation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. I think the work is concrete, when results are stated, algorithm was derived, the motivation is well-stated.\n2. The flow of the proofs is clear, in which the order and link between steps were explained.\n\nWeaknesses:\n\n1. The linear regression setting is too simple. Two distributions are differed only by their covariance matrices. \n2. Poor presentation of mathematical results. It seems to me that the author assumed that readers know the problem in advance. I have a list of questions below. \n3. I am not certain that the proof is novel. The main idea is common technique in analysis and probability, consider a quantities in its local neighborhood and for others outside the neighborhood bounding it by using other properties.\n4. I am not certain that the upper  bound is Theorem 1 is optimal? The RHS of the inequality in Theorem 1 has two terms: one with respect to target distribution, other measures the discrepancy between source and target distribution. This is not surprising, since the estimator comes from mixed data. \n5. Experiments were carried out on small datasets, synthetic data and school data with very small sample sizes.  It seems to me that the authors consider it is a theoretical work. Meanwhile, the more important thing is if the proposed algorithm actually works/ outperforms other methods in various settings and deployed to help solving problems.\n\nQUESTIONS:\nWhy is there $6\\epsilon_Q$ in the constraint (2), I mean the factor $6$?\n\nDoes $\\lambda_{\\max}  (\\Sigma_p^{-1} \\Sigma_Q)$ exist?\n\nNotation $\\eta = \\frac{c_\\eta}{\\sqrt{T}}$ in Theorem 1, so $\\eta$ depends on $c_\\eta$?\n\nHow to check all conditions do not violate each other?\n\nShould have a table for all quantities to check? \n\n$\\theta_{Q,t}$ what does $t$ means here, step $t$?  equation (2)\n\nLine 46, what does $\\tilde{\\nabla}$ mean? \n\nLine 59, what does ``converge in risk\" mean? \n\nLine 142, what is $\\theta_0$?\n\nIn Theorem 2, why do we need condition $M_x \\geq 1$? Because we could be in the situation when $\\|x\\| \\leq 1$.\n\nTheorem 1 includes $\\epsilon_Q, \\epsilon_P$ but no  definition for those terms. When I track back line 72, some words describe them, still no mathematical definition. \n\nWhat is the limit of $\\lambda_t$ as $t\\rightarrow \\infty$?\n\nLIMITATIONS:\nYes\n\nI would like to reject the paper since the poor presentation of its mathematics and the results seem to be not technically novel and challenging to prove. However, I am not familiar with the previous theoretical works in this domain. Hence, I reserve the decision when more things will be clear.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies a class of supervised transfer learning algorithms that utilize stochastic gradient descent (SGD) to construct a regressor by minimizing the empirical risk on a source domain while constraining the empirical risk on a target domain. The authors provide a theoretical analysis of the generalization error for this approach, showing that the SGD solution converges to that of a specific hypothesis transfer learning algorithm with a well-established generalization rate. Notably, the resulting generalization error adapts between the rates achieved by training solely on the source or solely on the target data. The authors support their theoretical findings with experiments, demonstrating that their proposed algorithm achieves adaptive performance, matching the rates of either solo source or solo target training depending on the scenario.\n\nSTRENGTHS AND WEAKNESSES:\nI lean toward accepting this paper primarily due to the benefit of proposing a practical algorithm for transfer learning with rigorous theoretical error guarantees.\n\nThis paper is clearly written and addresses the relevant topic of error analysis for the SGD method in transfer learning. The theoretical and experimental contributions are well-highlighted, making the main results easily understandable. The theoretical results are supported by rigorous mathematical proofs that appear sound upon review. The clarity of the paper is generally high, with the exception of comparisons with existing work.\n\nThe paper provides error upper bounds for the mixed-sample SGD algorithm in a transfer learning setup. While the obtained error bounds match those shown by Hanneke et al. (2024), the originality of this paper lies in analyzing these bounds for the more practical mixed-sample SGD algorithm. The significance of this paper hinges on the extent of advancement beyond Hanneke et al. (2024), but unfortunately, the authors do not provide detailed comparisons with this prior work. They merely mention that Hanneke et al. (2024) focuses on classification settings, when in fact that work provides a general framework applicable to both classification and regression. Indeed, the theoretical framework of Hanneke et al. (2024) has wider applicability than this paper, and regarding error bounds, the results are equivalent. This lack of detailed comparison diminishes the clarity regarding the paper's significance.\n\nThe primary contribution may lie in providing a concrete implementation to find a predictor within the confidence set defined by Hanneke et al. (2024), effectively offering a tractable implementation of Algorithm 1 from that work. Developing such a tractable algorithm represents a significant contribution to transfer learning. However, as mentioned, the paper lacks rigorous comparisons with Hanneke et al. (2024), including clarification of the relationship between the algorithms. A more thorough comparison would substantially improve the paper.\n\nI have concerns about the experimental quality. First, the accuracies of the HTL method remain constant regardless of source sample size. The authors attribute this to HTL's difficulty in finding suitable regularization parameters through cross-validation due to limited target data. However, the relatively small error bars suggest limited impact from small sample sizes, contradicting this explanation. Without implementation details or code for HTL, I cannot verify the experimental design's adequacy. Second, runtime results are highly implementation-dependent, yet the authors provide no implementation details or code for HTL, PSGD, or their proposed methods. This raises concerns about potential unfair comparisons, such as the proposed algorithm potentially using fewer iterations than comparison algorithms. Providing implementation details would help establish the fairness of the runtime comparisons.\n\nQUESTIONS:\n1. Could the authors please provide a more detailed comparison with Hanneke et al. (2024)? Specifically, I would appreciate clarification on how your theoretical framework differs from theirs, and what novel contributions your paper makes beyond their work. A clearer articulation of these distinctions would significantly strengthen my evaluation of the paper's originality and significance.\n\n2. Regarding the experimental evaluation, could the authors provide more details about the implementation of all compared methods? In particular, I noticed that the HTL method shows constant accuracy regardless of source sample size, which seems counterintuitive. Additionally, the runtime comparisons would be more convincing with implementation details to ensure fair comparison (e.g., number of iterations, optimization parameters). Addressing these concerns would improve my assessment of the paper's experimental quality.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}