Efficient Source-Free Time-Series Adaptation via Parameter Subspace Disentanglement

Authors: Gaurav Patel, Christopher Michael Sandino, Behrooz Mahasseni, Ellen L. Zippi, Erdrin Azemi, Ali Moin, Juri Minxha
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Time Series, Source-Free Domain Adaptation, Efficiency

Abstract:
In this paper, we propose a framework for efficient Source-Free Domain Adaptation (SFDA) in the context of time-series, focusing on enhancing both parameter efficiency and data-sample utilization. Our approach introduces an improved paradigm for source-model preparation and target-side adaptation, aiming to enhance training efficiency during target adaptation. Specifically, we reparameterize the source model's weights in a Tucker-style decomposed manner, factorizing the model into a compact form during the source model preparation phase. During target-side adaptation, only a subset of these decomposed factors is fine-tuned, leading to significant improvements in training efficiency. We demonstrate using PAC Bayesian analysis that this selective fine-tuning strategy implicitly regularizes the adaptation process by constraining the model's learning capacity. Furthermore, this re-parameterization reduces the overall model size and enhances inference efficiency, making the approach particularly well suited for resource-constrained devices. Additionally, we demonstrate that our framework is compatible with various SFDA methods and achieves significant computational efficiency, reducing the number of fine-tuned parameters and inference overhead in terms of MACs by over 90\% while maintaining model performance.

Decision: accept (poster)
Reviewer scores: 6, 10, 5, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1729690430302

thread_type:
official_review

<thread_object>
creator: reviewer_g1yq

<content>
summary:
This paper investigates source-free domain adaptation problem. The authors introduce a straightforward yet efficient framework that decomposes the parameter matrices using Tucker-style factorization, allowing the model to adjust only a subset of parameters during target adaptation. Extensive experiments conducted across various datasets and settings demonstrate the superiority of the proposed method over existing baseline models and high sample efficiency.

strengths:
1. The proposed metric decomposition-based approach for efficient target domain adaptation is simple and efficient. 
2. The experiments are very detailed and thorough, clearly demonstrating the effectiveness and efficiency of the proposed method. That is appreciated. 
3. The manuscript is well-structured and clearly written, facilitating easy comprehension for readers.

weaknesses:
1. The connection between the proposed method and the characteristics of time-series data is somewhat weak. The method appears to be a general SFDA approach rather than being specifically tailored for time-series data.
2. The novelty of the approach is limited, as parameter decomposition for domain adaptation is a prevalent technique.
3. The description of target domain adaptation is insufficient. Specifically, which loss function is utilised? And how to optimise the parameters?

questions:
1. Could the author explain the rationality of replacing $P$ (the prior distribution) and $S$ (sampled data) with $\mathcal{W}{src}$ (the pre-trained network) and $\mathcal{W}_{trg}$ (fine-tuned network), respectively, in the context of PAC-Bayesian generalization theory? The original theory is designed to measure the generalization error between the empirical and expect loss, but for the same distribution.
2. Could the author include an ERM baseline in Table 1 to provide readers with a clearer understanding of the performance improvements achieved by the proposed method?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730663644449

thread_type:
official_review

<thread_object>
creator: reviewer_kbwb

<content>
summary:
This paper focuses on the topic of Source-Free Time-Series Adaptation. To make the model adaptation more efficient, this paper proposes to decompose the model into core tensor and factor matrices.  When adaptation, only core tensors are selectively fine-tuned. The method can reduce the overall model size and enhance inference efficiency by decomposing and selective fine-tuning.

strengths:
The paper proposes to use Tucker-style factorization for the model decomposition. This decomposition raises a new perspective (at least for me) beyond the conventional feature selection and  BN adaptation methods. 

The results are solid. A PAC-Bayes generalization bound is provided for the selective fine-tuning stage. The method is evaluated on the real-world dataset such as AdaTime benchmark.

weaknesses:
I am only curious about some points listed below:

1) Some domain adaptation methods only adapt the BN layers [1],  normalization [2], or only feature modules [3]. They also selectively fine-tune the model, what is the relations between them and the proposed one. Can the proposed method explain or unify these methods in some cases?

2) Can the proposed method be transferred to the general DA task? If not, could you please explain the reasons?

3) It seems that the proposed theorem can be applied to any method with selective fine-tuning. What is the advantage of the proposed decomposition method analyzed from this theorem? 

[1] Evaluating prediction-time batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006.10963, 2020.
[2] Tent: Fully Test-time Adaptation by Entropy Minimization. ICLR 2021.
[3] Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. ICML 2020.

questions:
Please refer to the weaknesses part. Overall, I think it is a good paper with new thoughts.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730966628158

thread_type:
official_review

<thread_object>
creator: reviewer_vywa

<content>
summary:
The paper introduces a novel domain adaptation approach for time series modeling using tensor decomposition. It decomposes source model weights and only finetunes the tensor core components for target data adaptation. The authors provide PAC-based generalization bounds for the adaptation task and demonstrate computational efficiency through experiments.

strengths:
- New application of tensor decomposition for domain adaptation in time series, building upon concepts from Monarch architecture and block tensor train methods used in NLP transformer models
- Comprehensive experimental results across multiple datasets clearly demonstrate computational benefits

weaknesses:
1. While the theoretical analysis provides valuable insights into the trade-off between target data fit and source-target distribution divergence, the assumptions have limitations:

- The posterior $Q(S)$ is assumed to be an isotropic Gaussian with variance matching the prior $P$. This assumption doesn't reflect real-world scenarios and is more like a typical variational inference solutions. A more realistic upper bound should incorporate: (1) Weight differences; (2) Distance between isotropic Gaussian posterior and true posterior. While such an extension appears feasible, the distance metric between isotropic Gaussian and true posterior remains undefined

-  The performance of SFDA compared to LoRA aligns with expected outcomes based on existing work in Monarch and Tensor Train architectures, making it somewhat incremental

questions:
Related Work:
The paper would benefit from discussing connections to recent NLP transformer architectures, particularly:

- Dao et al. (2022), "Monarch: Expressive Structured Matrices for Efficient and Accurate Training"
- Qiu et al. (2024), "Compute Better Spent: Replacing Dense Layers with Structured Matrices"
</content>
</thread_object>

</thread>
<thread>
creation date:
1731020421825

thread_type:
official_review

<thread_object>
creator: reviewer_ggv1

<content>
summary:
This paper considers SFDA using Tucker style decomposed weights and selective fine tuning on target data. Some bounds are presented for this strategy and experimental results are shown.

strengths:
The proposed method can be integrated with several baselines SFDA methods to improve adaptation efficiency.

weaknesses:
1. Cite and compare with:
- Tang et al, Source-Free Domain Adaptation with Frozen Multimodal Foundation Model, CVPR 2024
- Litrico et al, Guiding pseudo-labels with uncertainty estimation for source-free unsupervised domain adaptation, CVPR 2023
- Tang et al, Source-free domain adaptation via target prediction distribution searching, ICLR 2023
- Gao et al, UNITS: A Unified Multi-Task Time Series Model, NeurIPS 2024
- Ding et al, Source-Free Domain Adaptation via Distribution Estimation, CVPR 2022
It's important to compare against foundation model style approaches since they offer SoTA adaptation efficiency.

2. Writing needs to be improved. The tables with results and plots are very very tiny, and zoom 200% is needed to read them and evaluate. Please choose the most important results to present.

3. The loss functions for target adaptation are not presented, please be specific and self-contained.

4. The method is not time series specific. Why not compare with vision models and datasets to show universality of the method?

5. The authors do not compare against Lokra, Lora with the same style of parameter block freezing and tuning. This would improve fairness of comparisons.

questions:
See weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1731036526398

thread_type:
official_review

<thread_object>
creator: reviewer_bzkd

<content>
summary:
This work introduces a method for source-free domain adaptation (SFDA) in the domain of time-series data. The task poses a challenge due to two main reasons: 1) the model has access only to unlabeled data and the source-pretrained model but not the source data, and 2) finetuning with small amounts of unlabeled data in this domain often results is overfitting. This papers explore the method of applying Tucker-style factorization, by breaking the parameters into two parts: core-tensors and factor matrices. By doing so and selectively finetune only on the core-tensor matrix, the model shows little signs of overfitting and improved generalization. Empirically, experiments from changing the rank fo the decomposed tensor to comparisons with current popular adaptor methods show the proposed method is effective at reducing overfitting and improving generalization with fewer parameters, achieving the best of both worlds. Finally, the authors provided a PAC-Bayes generalization bound, showing that the test loss is upper bounded by the empirical loss plus the regularization effect from the matrices learned.

strengths:
The problem presented at this paper is of great significance, with considerations about the challenges of finetuning the full model and application of on-device settings. The authors have done a great job in both the introduction and method motivating the problem and stating the importance and relevance of this research problem. The method is clear, with definitions and equations clearly defined and laid-out. The method proposed itself is sound also. The authors also try to draw connections with related works to Lottery Ticket Hypothesis, as an attempt to justify why their method works well and is able to achieve the best of both worlds; This connection is a nice touch and welcomed. Experiments-wise, the paper has done a thorough job in demonstrating the effectiveness of their method, with sufficient number of datasets and models, and comparisons with state-of-the-art methods that show clear improvements.

weaknesses:
A weakness of this work can be characterized as a missing link between the entire method and the time-series aspect of the problem. Specifically, it’s not entire clear why the method here is specifically applied to time-series, as the authors have setup here. Perhaps the authors can clarify how their method has leverage properties in time-series data to their advantage. If this connection is indeed not mentioned, then it seems reasonable to add a sentence or a section in the Appendix justifying this.

questions:
1. Can the authors comment on the time for inference before and after factorization? Does the smaller number of parameters reduce the time required for inference? Or does the increased number of matrix factorization increase the time required?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731053700977

thread_type:
official_review

<thread_object>
creator: reviewer_56c7

<content>
summary:
This work presented a Tucker-style tensor factorization as a low-rank decomposition of source-pretrained models for time-series data. This method enables a selective fine-tuning of the core tensor on the target side, effectively improving the efficiency.

strengths:
I think the topic of the article is cutting-edge. The article as a whole uses rich methods such as PAC-Bayes to expound the effectiveness of the proposed method. The supplementary materials are also very detailed in explaining the article, such as the derivation of the formal bound parameter distance. The experiments are sufficient, and the effects shown in the experiments are good. The elaboration and analysis of limitations are also very sufficient.

weaknesses:
1. In my opinion, in the Related Work section, what is shown is more accurately ablations rather than the so-called observations in the title. Compared to this experimental result, it is more necessary to simply explain why Tucker-style decomposition is used instead of other decomposition methods you have exemplified.
2. I think the insights in lines 201-204 need a more complete explanation. From Equation 1, I can only understand that the linear operation in deep learning is the so-called sequence of linear suboperations in the forward process. However, I cannot directly confirm whether such a description is also accurate for the backward process. If there are relevant formula derivations or proofs, references can be provided. If not, considering the space limitation, it is best to prove it in the Appendix. Because this claim is the premise of Equations 4, 5, and 6 later, which is necessary and important for understanding this method.
3. A small mistake. In the sentence after Eq3, on line 221, after “where”, it should be an explanation of the symbol V in Equation 3 instead of U.
4. On line 238, “Building on these insights, we decompose the pre-trained source model weights using Tucker decomposition”. Although the expression here is rigorous. Logically, the former is a necessary condition for the latter. From the beginning of the article until now, there is no sufficient reason given for why Tucker decomposition is used instead of other decomposition methods. Even if it is mentioned earlier, it is better to emphasize it again here.

questions:
Lines 248-252, why is fine-tuning for 2-3 epochs? Is this number closely related to the dataset and model scale? I am not very familiar with this field and hope that relatively rigorous evidence can be given here in the text.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732610710100

thread_type:
official_comment

<thread_object>
creator: area_chair_kmnk

<content>
title:
Encouragement to Actively Participate in the Discussion Phase

comment:
Dear Reviewers,

Thank you for your valuable contributions to the review process so far. As we enter the discussion phase, I encourage you to actively engage with the authors and your fellow reviewers. This is a critical opportunity to clarify any open questions, address potential misunderstandings, and ensure that all perspectives are thoroughly considered.

Your thoughtful input during this stage is greatly appreciated and is essential for maintaining the rigor and fairness of the review process.

Thank you for your efforts and dedication.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733286265953

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response

comment:
We would like to thank all of the reviewers for their evaluation of our paper. Their insights and suggestions have been incorporated (where possible) in the updated manuscript. Importantly, the review process has helped us better define our contribution within the broader research landscape. In what follows, we summarize aspects of our work that were generally praised by the reviewers, and expand on some of the commonly raised concerns. 


**Recognizing Innovation and Theoretical Rigor**  
We are encouraged that **Reviewer vYWA** identified this as a **cutting-edge topic**, acknowledging its relevance to the rapidly evolving landscape of domain adaptation. Similarly, **Reviewers bZkd and KBWb** acknowledged the innovative methodology of decomposing model parameters into core tensors and factor matrices, enabling selective fine-tuning. This technique not only reduces overfitting and enhances generalization but also provides a fresh perspective beyond conventional approaches such as feature selection or BatchNorm adaptation. Moreover, **Reviewers 56c7 and G1Yq** noted our use of PAC-Bayes generalization bounds, which we used to provide theoretical support. This alignment between theoretical underpinnings and empirical validation forms a cornerstone of our work. Collectively, these remarks underscore the novelty of our contribution.

**Experimental Strength and Versatility**  
The experimental results were widely praised for their **comprehensiveness and robustness**, as highlighted by **Reviewers 56c7, bZkd, and G1Yq**. Our approach demonstrates effectiveness across diverse datasets, including the AdaTime benchmark, showcasing reduced overfitting, improved generalization, and computational efficiency with fewer parameters. Furthermore, **Reviewer gGV1** commended the method's **versatility**, particularly its ability to seamlessly integrate with existing SFDA baselines to enhance adaptation efficiency. **Reviewer G1Yq** also appreciated the manuscript's **clear and structured presentation**, bolstered by detailed supplementary materials. These comments collectively underscore the practical and theoretical impact of our work.

**Connections to the Lottery Ticket Hypothesis and Dual advantage**  
We are especially grateful to **Reviewer bZkd** for drawing a parallel between our approach and the *lottery ticket hypothesis*. Moreover, identifying our method's dual advantage: achieving both parameter and sample efficiency, we effectively strike a balance between these two critical objectives,i.e. realizing "the best of both worlds."
</content>
</thread_object>

</thread>


Metareview:
metareview:
(a) Summary of Scientific Claims and Findings
This paper introduces a novel framework for Source-Free Domain Adaptation (SFDA) in time series, leveraging a Tucker-style tensor decomposition to improve parameter and sample efficiency. Key contributions include:
Decomposed Model Representation: The source model's weights are reparameterized using Tucker decomposition, enabling selective fine-tuning of core tensors while freezing peripheral components.
Efficiency Improvements: The approach significantly reduces the number of parameters requiring fine-tuning (by over 90%) and improves inference efficiency.
Theoretical Justification: A PAC-Bayesian framework is employed to provide generalization bounds and insights into the effectiveness of selective fine-tuning.
Versatility Across Methods: The framework integrates seamlessly with various SFDA methods and demonstrates competitive performance on five datasets from the AdaTime benchmark, showcasing robustness in diverse domain adaptation scenarios.

(b) Strengths of the Paper
Innovative Approach: The use of tensor decomposition for efficient SFDA is novel and addresses practical challenges in resource-constrained environments.
Theoretical Rigor: The PAC-Bayesian analysis provides a solid theoretical foundation for understanding the framework’s generalization capabilities.
Empirical Validation: Extensive experiments demonstrate significant parameter and computational efficiency gains while maintaining or improving model performance.
Broad Applicability: The framework is compatible with a variety of SFDA methods and tasks, highlighting its versatility.
Thorough Rebuttal and Improvements: The authors addressed reviewer concerns with additional analyses, comparisons, and improved clarity, significantly strengthening the submission.

(c) Weaknesses of the Paper
Lack of Time Series-Specific Focus: While the method leverages inter-channel dependencies in time series, it is not inherently specific to this domain and could be generalized to other data modalities.
Limited Exploration of Hyperparameters: The sensitivity of key parameters, such as decomposition ranks, to different datasets and tasks requires further exploration.
Writing and Presentation: Although improved, the clarity of some sections (e.g., detailed loss function descriptions and comparisons) could be further enhanced.

(d) Reasons for Acceptance
Significant Efficiency Gains: The proposed framework delivers substantial reductions in fine-tuning and inference costs, which is critical for deploying models on resource-constrained devices.
Theoretical and Empirical Synergy: The combination of PAC-Bayesian generalization bounds and strong experimental results provides a compelling case for the framework’s effectiveness.
Versatility Across Methods: By integrating seamlessly with diverse SFDA techniques, the framework serves as a valuable addition to the domain adaptation toolkit.
Strong Rebuttal: The authors demonstrated a clear commitment to addressing reviewer concerns through meaningful revisions and additional analyses.
Practical Relevance: The framework bridges a critical gap in SFDA for time series, offering a robust solution for real-world applications in computationally constrained environments.