Title: Co-Regularization Enhances Knowledge Transfer in High Dimensions
Venue: NeurIPS 2025
Accepted: yes
Keywords: High Dimensional Statistics

Transfer Learning

Generalized Linear Models

Statistical Learning
Abstract: Most existing transfer learning algorithms for high-dimensional models employ a two-step regularization framework, whose success heavily hinges on the assumption that the pre-trained model closely resembles the target. To relax this assumption, we propose a co-regularization process to directly exploit beneficial knowledge from the source domain for high-dimensional generalized linear models. The proposed method learns the target parameter by constraining the source parameters to be close to the target one, thereby preventing fine-tuning failures caused by significantly deviated pre-trained parameters. Our theoretical analysis demonstrates that the proposed method accommodates a broader range of sources than existing two-step frameworks, thus being more robust to less similar sources. Its effectiveness is validated through extensive empirical studies.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper discusses the multi-source transfer learning for high-dimensional generalized linear models (GLMs). The Author proposed a co-regularization process that simultaneously learns source and target hypothesis with target domain and source domains that are similar to the target. Theoretical results shows that the proposed CoRT has a tighter error bound and a narrower confidence interval than target-only Lasso. Empirical studies validated its effectiveness.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The method has clear modeling logic and is easy to follow.\n\n2. The theoretical analysis provide guarantee to the convergence of the optimization process.\n\nWeaknesses\n\n1. The paper only discussed method and theory for the GLMs, while even the small datasets may have more complicated hypothesis space. Thus, further discussion for more general model is necessary. \n\n2. The objective of the CoRT, i.e. Eq. (1), seeks for optimal solution of both source and target model, which is harder than only training the optimal target model. The gradients may cause negative impact to each other.\n\n3. The evaluation experiments are insufficient and further numerical analysis of the method is lacked.\n\nQUESTIONS:\n1. Discussion for more general model is suggested. Will the CoRT still work for more complicated non-linear model? What about the convergence?\n\n2. The idea of Co-Regularization seems to be similar to the multi-task learning. The similarities and differences between them should be discussed. \n\n3. Further, I am curious about whether the training of target model will be disturbed by the gradient of the source models, making it harder to reach the optimal solution.\n\n4. Since all models are known in the simulation experiment, numerical validation of the error bounds and the convergence behavior will more strongly support the main theorems of the paper.\n\nIf the authors can address most of the questions, I am willing to raise the score.\n\nLIMITATIONS:\nDiscussion for more general model is lacked.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates high dimensional (dimension larger than the number of data in target domain) transfer learning with General Linear Models. Unlike standard two steps approach, pretraining on the source data and fine tuning on the target data, this paper proposed a co-regularization learning scheme where source data act as a regularization for learning in the target domain. Specifically, the paper proposes co-regularization transfer (CoRT), which estimates the target domain parameter $\\beta$ and parameter contrast $\\delta$ simultaneously. Moreover, the authors propose an adaptive algorithm that identifies potential outlier sources to avoid transferring \"bad\" sources. Simulations show the favorable performance of CoRT compared to two-step training schemes.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. The paper is overall well written and easy to follow.\n2. Strong statistical analysis is provided that shows the clear advantage of the proposed CoRT framework vs existing baselines such as Trans-GML.\n\nWeakness:\n\n1. Even though the specific combinations of components are unique, the idea of one step learning for both source and target domain are not exactly brand new in the literature of domain adaptation.\n2. The practicality of the paper is also of concern, as even though simultaneously training on source and target domain would intuitively leads to better estimation error, having access to both source data at the time of learning in target domain is a rather restrictive assumption. As the establishment of two-step training paradigm is partially conditioned on the difficulty of training on both source and target data. For example, the proposed approach would make switching to new target domain extremely expensive, as the source data need to be involved all the time.\n3. There are some minor presentation issues, including really small font size in Figure 1, making them hard to read.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Co-Regularization Transfer (CoRT) framework for multi-source transfer learning in generalized linear models (GLMs). Instead of the traditional two-step approach of pre-training and fine-tuning, CoRT jointly estimates the target parameter and source deviations via co-regularized optimization. The paper contributions also include i) an adaptive knowledge transfer algorithm using the majority vote to detect outlier source domains; ii) an asymptotic element-wise confidence interval for the estimate of the target domain parameter; and iii) an extensive theoretical analysis proving robustness to heterogeneous source domains.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The proposed CoRT framework is novel and avoids the bias introduced by the usual pre-training + fine-tuning paradigm.\n- The paper presents solid theoretical foundations and the provided convergence bounds compare favorably with the existing methods.\n- The proposed adaptive algorithm for identifying outlier sources eliminates the need for hyperparameter tuning, which is a significant practical advantage over existing methods.\n- The experimental results both in synthetic and real world data confirm the effectiveness of the method\n\nWeaknesses:\n- Although the authors claim that the methodology can be extended to other types of models, the paper is focused on GLMs, so it becomes unclear how broadly CoRT can generalize beyond this family.\n- There’s no discussion on computational complexity of the derived algorithms.\n- The real world dataset (COVID-19 mortality) is insightful but relatively small.\n- The authors claim that CoRT can identify which specific covariates in the source domain are transferable, enhancing interpretability. While I agree with this observation, it would be nice to see this analysis in the experiments.\n\nQUESTIONS:\n- Please elaborate on runtime complexity and scalability to large K (number of sources) or p (dimensionality).\n- Regarding the derived theoretical guarantees, how do these relate to or differ from discrepancy-based domain adaptation methods in terms of assumptions and guarantees?\n- Are you willing to release the source code implementing your methods and experiments?\n\nLIMITATIONS:\nThe limitations are only marginally addressed in the Discussion section. The paper would benefit from having a dedicated Limitations section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a two-step framework for multi-source transfer learning. The proposed method combines Lasso and Co-Regularized Transfer (CoRT) to prevent overfitting and selects the most suitable source dataset ($\\hat{S}$) among multiple sources, providing predictions optimized for the target data.\n\nSTRENGTHS AND WEAKNESSES:\n$\\textbf{Strengths}$\n\n1. The authors' efforts to provide mathematical interpretations as evidence and theoretical support for the proposed method are noteworthy.\n2. By splitting the target data to calculate errors, the method minimizes overfitting to specific sources in transfer learning.\n3. Compared to conventional ML techniques, it offers automation advantages, such as reducing the need for parameter tuning.\n\n$\\textbf{Weaknesses}$\n\n1. The explanation of the proposed method is somewhat difficult to understand. For example, in line 49 of the introduction, it should be clarified how adaptive identification of outlier sources is achieved and how this information is utilized during training.\n2. The improvement in task performance by the proposed method is somewhat limited.\n3. The analysis of experimental results is not described in depth.\n4. Minor: The text and captions in the figures are too small. Careful review by the authors is needed.\n\nQUESTIONS:\n1. An explanation is needed for what \"E\" and the \"acute\" symbol in line 94 mean.\n2. The effectiveness of the proposed method could be further emphasized by presenting results for each source without the adaptive technique.\n3. In what scenarios is the proposed method more effective (excluding the size of the target data)?\n4. The authors mention that their method would be effective when applied to neural networks. What considerations should be made when applying it to DNNs?\n5. Instead of dropping specific source domains, is there a way to maximize the knowledge that can be obtained from each source domain (e.g., sample-wise)?\n\nLIMITATIONS:\n1. Some parameters still require tuning.\n2. The observed improvement in target task performance is relatively modest. This may indicate that the framework struggles in cases where source-target alignment is weak.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-07-03"}