Efficient Source-Free Time-Series Adaptation via Parameter Subspace Disentanglement
Authors: Gaurav Patel
Christopher Michael Sandino
Behrooz Mahasseni
Ellen L. Zippi
Erdrin Azemi
Ali Moin
Juri Minxha
Venue: ICLR 2025 Poster (2025)
Keywords: Time Series
Source-Free Domain Adaptation
Efficiency
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Efficient source-free domain adaptation framework for time series that improves parameter and sample efficiency of the adaptation process.

Abstract: In this paper, we propose a framework for efficient Source-Free Domain Adaptation (SFDA) in the context of time-series, focusing on enhancing both parameter efficiency and data-sample utilization. Our approach introduces an improved paradigm for source-model preparation and target-side adaptation, aiming to enhance training efficiency during target adaptation. Specifically, we reparameterize the source model's weights in a Tucker-style decomposed manner, factorizing the model into a compact form during the source model preparation phase. During target-side adaptation, only a subset of these decomposed factors is fine-tuned, leading to significant improvements in training efficiency. We demonstrate using PAC Bayesian analysis that this selective fine-tuning strategy implicitly regularizes the adaptation process by constraining the model's learning capacity. Furthermore, this re-parameterization reduces the overall model size and enhances inference efficiency, making the approach particularly well suited for resource-constrained devices. Additionally, we demonstrate that our framework is compatible with various SFDA methods and achieves significant computational efficiency, reducing the number of fine-tuned parameters and inference overhead in terms of MACs by over 90\% while maintaining model performance.

Published as a conference paper at ICLR 2025
EFFICIENT SOURCE-FREE TIME-SERIES ADAPTATION VIA PARAMETER SUBSPACE DISENTANGLEMENT
Gaurav Patel" Chris Sandino', Behrooz Mahasseni', Ellen Zippi Erdrin Azemi", Ali Moin', Juri Minxha Purdue University?, Apple
ABSTRACT
In this paper, propose framework for efficient Source-Free Domain Adap- tation (SFDA) in the context of time-series, focusing on enhancing both parame- ter efficiency and data-sample utilization. Our approach introduces an improved paradigm for source-model preparation and target-side adaptation, aiming to en- hance training efficiency during target adaptation. Specifically, we reparameterize the source model's weights in a Tucker-style decomposed manner, factorizing the model into a compact form during the source model preparation phase. During target-side adaptation, only subset of these decomposed factors is fine-tuned, leading to significant improvements in training efficiency. We demonstrate using PAC Bayesian analysis that this selective fine-tuning strategy implicitly regular- izes the adaptation process by constraining the model's learning capacity. Fur- thermore, this re- parameterizzation reduces the overall model size and enhances inference efficiency, making the approach particularly well suited for resource- constrained devices. Additionally, we demonstrate that our framework is compati- ble with various SFDA methods and achieves significant computational efficiency, reducing the number of fine-tuned parameters and inference overhead in terms MACs by over 90% while maintaining model performance.
INTRODUCTION
In typical Source-Free Domain Adaptation (SFDA) setup, the nource-pretraings model must adapt to the target distribution using unlabeled samples from the target domain. SFDA strategies have become prevalent due to the restrictive nature of conventional domain adaptation methods (Li et al., 2024) which require access both source and target domain data simultaneously and therefore may not be feasible in real-world scenarios due to privacy and confidentiality concerns. Although SFDA techniques have been extensively investigated for visual tasks (Liang et al., 2020; 2021; Li al., 2020; Yang et al., b02;;;a; 2022; 2023; Kim et al., 2021; Xia et al., 2021; Kundu et al., 2021; 2022b), their application time series analysis remains relatively nascent (Ragab et al., 2023b; Gong et al., 2024). Nevertheless, time-series models adaptation is crucial due to the nonstationary and heterogeneous nature of time-series data (Park et al., 2023), where each user's data exhibit distinct patterns, necessitating adaptive models that can learn idiosyneratic features.
Despite growing interest in SFDA, sample and parameter efficiency during adaptation is still largely unexplored (Karim al., 2023; Lee al., 2023). These aspects of SFDA are of particular impor- tance in situations where there is a large resource disparity between the source and the target. For instance, courcc-pretrained model may be deployed to resource-constrainee target device, render- ing full model adaptation impractical. Additionally, the target-side often has access to substantially fewer reliable samples compared to the volume of data used during source pretraining, increasing the risk of overfitting-
In response to these challenges, we revisit both the source-model preparation and target-side adapta- tion processes. We demonstrate that disentangling the backbone parameter subspace during source- model preparation and then fine-tuning a selected subset of those parameters during target-side adaptation leads to a computationally efficient adaptation process, while maintaining superior pre- dictive performance. Figure IB illustrates our proposed framework. During source-model prepa-
"Work done during internship at Apple. Contact: gpatel 00purduee edu, mi www.aapple......m..mp

---

Published as a conference paper at ICLR 2025
ration, re-parameterize the backbone weights in low-rank Tucker-style factorized (Tucker, 1966; Lathauwer et al., 2000) way, decomposing the model into compact purameter subspaces. Tucker-style factorization offers great flexibility and interpretability by breaking down tensors into core tensor and factor matrices along each mode, capturing multi-dimensional interactions while independently reducing dimensionality. This reparameterization results in model that is both parameter- and computation-efficcen significantly reducing the model size and inference overhead in terms of Multiply-Accumulat Operations (MACs).
A.
B.
The re-parameterized source-model then deployed the target side.
Subipace
Source
Source
During target-side adaptation, we perform selective fine-tuning (SFT) within a selected subspace (i.e., the core tensor) that constitutes very small fraction of the total num- ber of parameters in the backbone. Our findings show that this strat- egy not only enhances parameter ef- ficiency but also serves as form of regularization, mitigating overfit- ting to unreliable target samples by restricting the model's learning ca- pacity. We provide theoretical in- sights into this regularization effect using PAC-Bayesian generalization bounds (McAllester, 1998; 1999; Li & Zhang, 2021; Wang et al., 2023) for the fine-tuned target model.
Backbone
Backoone
Weight Intinalcation
Weiign Iviiallatioo
Target
Target
Backoono
e
Selective Fineturing
Tramnabe
Model
Model
Figure 1: Prior Paradigm vs. Ours. A. Existing ap- proaches (Liang et al., 2020; Yang et al., 2021a; 2022; Ragab al., 2023b) adapt the entire model to the target distribution. B. Our method disentangles the backbone pa- rameters using Tucker-style factorization. Fine-tuning small subset of these parameters proves both parameter- and sample- sample-efficient (cf. Section 3), while also being robust against overfitting (cf. Figure 2B and Section 3), leading to superior adaptation to the target distribution.
Empirical results demonstrate that low-rank weight disentanglement during source- prepara- tion enables parameter-efficiee adaptation the target side, consistently improving performance across various SFDA methods (Liang et al., 2020; Yang et al., 2021 a; 2022: Ragab al., 2023b) and time-series benchmarks (Ragab et al., 2023a;b). It is important to emphasize that our contribution does not introduce a novel SFDA method for get adaptation process more parameter- and sample-efficient, and demonstrating that our framework be seamlessly integrated with existing, and potentially future, SFDA techniques. In summary, our key contributions are:
data. Instead, we focus on making the tar-
We propose a novel strategy for source-model preparation by reparameterizing the back- bone network's s weights using low-rank Tuucer-style factorization. This decomposition into a core tensor and factor matrices creates compact parameter subspace representation, leading to a parameter- and computation-effiiie model with reduced size and inference overhead.
2. During target-side adaptation, introduce selective fine-tuning (SFT) approach that adjusts only a small fraction of the backbone's parameters (i.e., the core tensor) within the decomposed subspace. This strategy enhances parameter efficiency and acts an implicit regularization mechanism, mitigating the risk of overfitting to unreliable target samples by limiting the model's 's learning capacity.
3. We ground the regularization effect of SFT using the PAC Bayesian generalization bound. Empirical analyses demonstrate that our proposed framework improves the parameter and sample efficiency of adaptation process of various existing SFDA methods across the time- series benchmarks, showcasing its generalizability.
RELATED WORK AND OBSERVATIONS
Unsupervised Domain Adaptation Time Series. Unsupervised Domain Adaptation (UDA) for time-series data addresses the critical challenge of distribution shifts between the source and methods, such as those of Cai et al. (2021), Liu & Xue (2021)
target domains. Diserepancy-bais and He et al. (2023), align feature representations through statistical measures, while adversarial ap- proaches (Wilson al., 2020; 2023; Ragab al., 2022; Jin al., 2022; Ozyurt et al., 2023) focus on
2

---

Published as a conference paper at ICLR 2025
A.
B.
HHAR
SSC
Tg ADGC Loe Viaine
1:00
Oripenal Der
Dilectve Rant
138-
Cfledive Park
Orginnl
target adaptation loss
2.14
magnitude
158
2.13
10
02-
40-
22-
Cin layer-2
Cou
Cin
Cout
Cin
Cons
Cin layer-3
Cont
layer-3
layer-2
epochs
Figure 2: A. Original Dimensions vs. Effective Rank: Comparison of input/output channel di- mensions (Cin, Cou) in the last two layers of the source model trained on the HHAR (Stisen al., 2015) and SSC (Goldberger et al., 2000) datasets, alongside their effective rank computed via VBMF (Nakajima et al., 2013). B. Regularization Effect: Training dynamics illustrate the regularization effect of our source decomposition and selective fine-tuning (SFT) compared to the SHOT (Base- line) (Liang et al., 2020) the SSC dataset.
minimizing distributional discrepancies through adversarial training. The comprehensive study by Ragab al. (2023a) provides a broad overview of domain adaptation in time series. However, SFDA assumes access to only the source-preee nource-pretrained model (Liang et al., 2020; 2021; Li et al., 2020; Yang al., aa; 2022; 2023; Kim et al., 2021; Xia et al., 2021). In SFDA, one of the most commonly used strategies is based on unsupervised clustering techniques (Yang et al., 2022; Li et al., 2024), promoting the discriminability and diversity of feature spaces through information maximization (Liang et al., 2020), neighborhood clustering (Yang et al., 2021a) or contrastive learning objectives (Yang et al., 2022). Recent advances incorporate auxiliary tasks, such masking and imputation, to enhance SFDA performance, as demonstrated by Ragab al. (2023b) and Gong et al. (2024), taking inspiration from Liang et al. (2021) and Kundu et al. (2022a) to include masked reconstruction as an auxiliary task. However, exploration of SFDA in time series contexts remains limited, especially with regard to parameter and sample efficiency during target-side adaptation (Ragab et al., 2023a; Liu et al., 2024; Gong et al., 2024; 2025; Furqon et al., 2025).
Parameter Redundancy and Low-Rank Subspaces. Neural network pruning has demonstrated that substantial reductions in parameters, often exceeding 90%, can be achieved with minimal ac- curacy loss, revealing significant redundancy in trained deep networks (LeCun et al., 1989; Hassibi & Stork, 1992; Li et al., 2017; Frankle & Carbin, 2018; Sharma et al., 2024). Structured prun- ing methods, such as those of Molchanov et al. (2017) and Hoefler et al. (2021), have optimized these reductions to maintain or even improve inference speeds. Low-rank models have played crucial role in pruning. with techniques such as Singular Value Decomposition (SVD) (Eckart & Young, 1936) applied to fully connected layers (Denil et al., 2013) and tensor-train decomposition used to compress neural networks (Novikov et al., 2015). Methods developed by Jaderberg et al. (2014), Denton et al. (2014), Tai et al. (2016), and Lebedev et al. (2015) accelerate Convolutional Neural Networks (CNNs) through low-rank regularization. Decomposition models such as CAN- DECOMP/PARAFAC (CP) (Carroll & Chang, 1970) and Tucker decomposition (Tucker, 1966) have effectively reduced the computational complexity of CNNs (Lebedev et al., 2015; Kim al., 2). Recent works have extended these techniques to recurrent and transformer layers (Ye et al., 2018; Ma et al., 2019). broadening their applicability. In Figure 2A, we identifyy significant parameter redundancies in source-pretrained models in terms of effective parameter ranks, revealing low effee- tive ranks, prompting us to employ Tucker decomposition for reparameterizing the weights. Tucker decomposition offers several advantages; naturally accommodates the multi-modal structure of convolutional weight tensors by disentangling different modes (e.g., temporal and channel dimen- sions), allows for mode-specific rank selection to capture varying complexities across dimensions, and enhances interpretability by modeling interactions within the weight tensors effectively. Com- pared to other methods like CP decomposition, which impose uniform ranks and lack flexibility, Tucker decomposition better aligns with our goals of parameter and sample efficiency during tar- get adaptation. We utilize low-rank tensor factorization (Tucker-style decomposition) to disentangle weight tensors into compact disentangled subspaces, enabling selective fine-tuning during target adaptation. This strategy enhances the parameter efficiency of the adaptation process and implicitly mitigates overfitting. As shown in Figure 2B, the validation Fl score for baseline methods declines over epochs, whereas our proposed SFT method maintains consistent performance, demonstrating its robustness against overfitting.
3

---

Published as a conference paper at ICLR 2025
0
R.
R.
vil
Am
Core Tensor
A
Gu
W
X3
X
Matrices
Figure 3: Illustration of Tucker-style factorization (Tucker, 1966; Lathauwer et al., 2000).
PROPOSED METHODOLOGY
In this section, we discuss our proposed methodology through several key steps: (1) We begin by discussing the SFDA setup. (2) We discuss the Tucker-style tensor factorization on the weight ten- sors of the pre-trained source-model, decomposing them into a core tensor and mode-specific factor matrices; where we introduce the rank factor (RP) hyperparameter to control the mode ranks in the decomposition, allowing for flexible trade-offs between model compactness and capacity. (3) Dur- ing target-side adaptation, selectively fine-tune the core tensor while keeping the mode factor matrices fixed, effectively adapting to distributional shifts in the target domain while reduced com- putational overhead and mitigating overfitting. (4) Finally, we offer theoretical insights grounded in PAC-Bayesian generalization analysis, demonstrating how source weight decomposition and target- side selective fine- fine-tuning implicitly regularize the adaptation process, leading to enhanced general- ization in predictive performance while rendering the process parameter- and sample-efficiee
SFDA Setup. SFDA aims to adapt pre-trained source model to a target domain without access to the original source data. Specifically, in classification problem, we start with source dataset D. = [(T,, y,); E Xs, U/ E Cg). where X, denotes the input space, and Cg represents the set of class labels for the goal task in deteed-set setting. On the target side, have access to an unlabeled target dataset D. = { 2z E &eJ. where X, is the input space for the target domain. The aim of SFDA is to learn target function J. X₂ Cg that can accurately infer the labels Yt E Cg for the samples Dr. fe obtained using only the unlabeled target samples in Dr, and the source-model f. X. - Cg. which is pre-trained on the source domain. Each æ (either from the source or target domain) is a sample of multivariate time series, i.e., x E X where L is the number of time steps (sequence length) and M denotes number of observations (channels) for the corresponding time step.
Tucker-style Factorization. After the supervised source pre-training (cf. Appendix A.9 for de- tails on source pre- training),, we adopt Tucker-style factorization (Tucker, 1966; Lathauwer et al., 2000) of the weight tensors for its effectiveness in capturing multi-way interactions among differ- ent mode-independent factors. The compact core tensor encapsulates these interactions, and the factor matrices in the decomposition offer low-dimensional representations for each tensor mode. As illustrated in Figure 3A, the rank-(R1, R2,Rs) R3) Tucker-style factorization of the 3-way tensor A xaxxaa is represented as:
R1 Σ
Ra
A 9 x ( X2 ( X3 U(3) or, A
9 Srirair
Un j.r kirs
U(2
(2) U(3)
(1)
1==1
where G E xRRxxRaxRs is referred to the core tensor, and U E R4XR U E Ra and U(3) E R as factor matrices.
Furthermore, the linear operations involved in Tucker-style representation (Equation (1)), which primarily mode-indeppndent matrix multiplications between the core tensor and factor matrices, simplify both mathematical treatment and computational implementation. This linearity ensures that linear operations (e.g.. convolution linear projection) in deep networks using the complete tensor can be efficiently represented as sequences of linear suboperations on the core tensor and factor matrices.
For instance, in one-dimensional convolutional neural networks (ID-CNNs), the convolution oper- ation transforms an input representation I X into an output representation O E RC using a kernel W E RRC C. Here, Cin- Cout> K, L,, and L/' denote the number of input chan- nels, output channels, kernel size, input sequence length, and output sequence length, respectively.
4

---

Published as a conference paper at ICLR 2025
The operation is mathematically defined as:
C O₂ =
Given that the dimensions of the channels (Cin and Cout) typically much larger than the size of the kernel (K), we restrict the decomposition to modes I and 2 only, focusing on the input and output channels. The 2-mode decomposition is represented by:
Run
Ru
W V() X2 V or, Wiji =
Σ
T 11
(.) v(2 f. J.22?
(3)
r==11==1
where T E R X is the 2-mode decomposed core tensor, and V E R and V
IR x represent the respective factor matrices, also illustrated in Figure 3B. With this decomposed form, the convolution operation can be represented as sequence of the following linear operations:
Cia
Zr
(Channel Down-Projection)
(4)
j=1
Ra
T Kr
(Core Convolution)
(5)
r2=1 Am
Rest
O
Σ =1
v!l) 1,2 M
(Channel Up-Projection)
(6)
where both the Channel Down-Projection and the Channel Up-Projection operations are imple- mented as unit window-sized convolution operations.
Building these insights, we utilize Tucker-style decomposition to reparameterize the weights, as it allows us to express convolution operations as sequences of linear operations that can each be implemented using standard convolutional layers. This reparameterization simplifies integration into existing architectures and leverages efficient convolution computations, facilitating both ease of implementation, computational efficiency, and overall reduction in the total number of parameters. We decompose the pre-trained source model weights using Tucker decomposition, optimized via the Higher- Orthogonal Iteration (HOOI) algorithm (De Lathauwer et al., 2000; Kolda & Bader, 2009), an alternating least squares method for low-rank tensor approximation (detailed algorithm in Appendix A.1). The optimization problem is defined as:
T (1),, (2)+
I|W- X1 v() X2 ( T. vV
arg min
and mode factor matrices y(I) E C x Rea and ( E R Ru,, as formulated in Equations (4), (5), and (6).
The weight tensor is then re-parameterized with the core tensor T E IR
However, the ce-parameterization is performed after minimizing the linear reconstruction error of the weights (Equation 7), which may degrade the predictive performance on the source domain (Kim et al., 2016). To mitigate this effect, we fine-tune the core tensor and factor matrices using the source data for an additional 2-3 epochs. prior to deploying the model on the target side. This fine- tuning effectively restores the original predictive performance, preserving the model's performance while leveraging the benefits of fewer parameters (ef. Figure 11). This behavior strongly aligns with the Lottery Ticket Hypothesis (Frankle & Carbin, 2018), which suggests that compressing and retraining pre-trained, over-parameterized deep networks can result in superior performance and storage efficiency compared to training smaller networks from scratch. Similar observations have been made by Zimmer et al. (2023) in the context of large language model training.
Moreover, introduce the rank factor (RF), hyperparameter standardize the mode rank anal- Co
ysis. The mode ranks are set (Pin; Ram))
[
Cun |), where higher R.F results in lower
5

---

Published as a conference paper at ICLR 2025
mode ranks, and vice versa. Although RF can be set independently for the input and output chan- nels, for simplicity our analysis, we control both Rim and Rear with a single hyperparameter, RF. This tunable parameter allows for flexible control over the trade-off between parameter reduction and model capacity.
Source
Target (negated)
Target Side Adaptation. The core tensor in our setup plays a pivotal role by capturing multi-way interactions between different modes, encoding essential inter-modal relationships with the factor matrices. Additionally, its sensitivity to temporal dynamics (cf. Equation (5)) makes it particularly well-suited for addressing distributional shifts time-series data, where discrepancies often arise due to temporal variations (Fan et al., 2023). There- fore, by selectively fine-tuning the core tensor during tar- get adaptation, we can effectively adapt to these shifts while maintaining a compact parameterization, Figure 4 presents a toy experiment that demonstrates how fine- tuning the core tensor suffices in addressing domain shift, aligning the model more effectively with the target do- main. This approach not only ensures that the model aligns better with the target domain but also enhances pa- rameter efficiency by reducing the number of parameters that need be updated, minimizing computational over- head. Moreover, selective fine-tuning mitigates the risk of overfitting, providing more robust adaptation process, shown in Figure 2B. This strategy leverages the expres- siveness of the core tensor to address temporal domain shifts while maintaining the computational benefits of structured, low-rank parameterization.
Input
layer
Core
Core adaptationi
layer
Core
Core fafter adiptationn
layer 3
Core
Core latter adaptation)
output festure
Figure 4:
The two columns show the
inputs from MNIST-ID (Greydanus & Kobak, 2024) and the core tensors for the source and target domains. The tar- get domain is generated manually by vertically flipping (i.e. negating) the source. Both Rin and Rout are set to (cf Figure 3B) to facilitate visualization of the core tensors.
output features are achieved as the core tensor adapts the target samples to mitigate the domain shift (negation).
Computational and Parameter Efficiency. eter count from Cout x Oi x K to:
The decomposition significantly reduces the param-
Rout X Rin x
Cour x Rout
Cin x
Rin
(8)
Core Tensor (T)
Factor Matrix (V())
Factor Matrix (V())
Cin- ensuring substantial parameter savings. Fur- thermore, since we selectively finetune the core tensor, the parameter efficiency is further enhanced.
where Rost = Con
R.F < Cout and Rm
Co RF
<
In terms of computational efficiency, the factorized convolution reduces the operation count from (C X Cm x K x L') to series of smaller convolutions with complexity:
(C x Rin L. + O(Ruu x R. x K x L') + O x Row I.)
Equation (4)
Equation (5)
Equation (6)
resulting in notable reduction in computational cost, dependent on the values of Rou and Run (see Appendix A.2 for detailed explanation).
Robust Target Adaptation. this subsection, we aim to uncover the underlying factors con- tributing to the robustness of the proposed SFT strategy, particularly in terms of sample efficiency and its implicit regularization effects during adaptation. To achieve this, we leverage the PAC- Bayesian generalization theory (McAllester, 1998; 1999), which provides principled framework for bounding the generalization error in deep neural networks during fine-tuning (Dziugaite & Roy, 2017: Li & Zhang, 2021; Wang et al., 2023). We analyze the generalization error on the network parameters W = (W()) 11=1 ie., L(W)- L(W), where L(W) is the test loss, L(W) is the empirical training loss, and D denotes the number of layers.
Theorem 1. (PAC-Bayer generalization bound for fine-tuning) Let W be some hypothesis class (network parameters). Let P be a prior (source) distribution on W that independent of the target training set. Let QS)) be posterior (target) distribution on W that depends on the target training set S. consisting of n. number of samples. Suppose the loss function L()) is bounded by C. If we set

---

Published as a conference paper at ICLR 2025
the prior distribution P
N(W src, I). where W sre are the weights of the pre-traimed network.
The posterior distribution Q(S) centered at the fine-tuned model as N(W ug, o2 Then with probability 1 - 8 over the randomness of the training set, the following holds:
E Q(()) [L(W] E(~Q
[(W. s)]
+ +CV
El
I
mg
W 1
are
kln 7 +1
(10)
202n
n
for some 8, k. > O, where W ug number of layers.
E W irg. and W
E W see
AA si D, D denoting the total
arc
Proof. See Appendix A.5 (Theorem 1)
SSC
MFD
HHAR
In Equation (10), the test error (LHS) is bounded by the empirical training loss and the divergence between the source pre-trained weights (W ()) and the target fine-tumed weights (W ng)) measured using the layer-wise
Wog Weel
CAFT PnC
8
10
(01)
2001
Frobenius norm. Motivated by this theoretical insight, empirically analyze the layer-wist parameter distances between the source and tar- get models in terms of the Frobenius norm. Fig- ure illustrates these distances for both vanilla fine-tuning (Baseline: Fine-tuming all parame- ters) and our SFT approach which selectively fine-tunes the core subspace, across various time series datasets, including SSC (Goldberger et al., 2000), MFD (Lessmeier et al., 2016), and HHAR (Stisen et al., 2015), for different RF values. The results show that SFT naturally constrains the parametric distance between the source and target weights, with higher RF exhibit- ing greater constraining ability, thus regulating the adaptation process in line with the distance- based regularization hypothesis of Gouk et al. (2021). Furthermore, in Appendix A.4 (Lemma 2), we provide a formal bound on the parameter distance in terms of the decomposed ranks of the weight matrices. This observation reveals a trade-off in the source-target parameter distance; while smaller distance tightens the generalization bound, it also limits the model's 's capacity, potentially regularizing or, in extreme cases, hindering adaptability. In Section 4, we use this bound to further discuss SFT's sample efficiency.
layer index
layer index
layer Index
Figure 5: Layer-wise parameter distance be- tween the source- pretrcceee model and the target- adapted model using the SFT strategy for different rank-factor values (RF E {2, 4, 8)) on the SSC (Goldberger et al., 2000), MFD (Lessmeier al., 2016), and HHAR (Stisen et al., 2015) datasets. The values represent the average parameter dis- tances across all source-target pairs provided for the respective datasets. Lower values indicate smaller parameter distances.
4 EXPERIMENTS AND ANALYSIS
Datasets and Methods. We utilize the AdaTime benchmarks proposed by Ragab et al. (2023a;b) to evaluate the SFDA methods: SSC (Goldberger et al., 2000), and MFD (Lessmeier et al., 2016), HHAR (Stisen et al., 2015), UCIHAR (Anguita et al., 2013), WISDM (Kwapisz et al., 2011). Here each dataset involves distinct domains based on individual subjects (SSC), devices (HHAR, UCI- HAR, WISDM) or entities (MFD). For comprehensive dataset descriptions and domain details, refer to the Appendix A.6. To assess the effectiveness and generalizability of our decomposition frame- work, we integrate it with prominent SFDA methods: SHOT (Liang et al., 2020), NRC (Yang al., 2021a), AAD (Yang et al., 2022), and MAPU (Ragab et al., 2023b) within time-series contexts. For more details the adaptation methods we direct readers to Appendix A.7 and A.8.
Experimental Setup. Our experimental setup systematically evaluates the proposed strategy (SFT) alongside contemporary SFDA methods evaluated by Ragab et al. (2023b); Gong et al. (2024), including SHOT (Liang et al., 2020), NRC (Yang et al., 2021a), AAD (Yang et al., 2022), and MAPU (Ragab et al., 2023b). In SFT, the backbone fine-tuning is restricted to the core subspace of the de- composed parameters, and the hyperparameters for source training and target adaptation are kept consistent between each vanilla SFDA method and its SFT variant. In addition, we assess the im- pact of different ranking factors (RF). For values of RF greater than 8, we observed significant underfitting, leading us to limit our evaluation to RF E (2, 4, 8). The experiments are carried out using the predefined source-target pairs from the AdaTime benchmark (Ragab al., 2023a).
7

---

Published as a conference paper at ICLR 2025
SSC
MFD
HHAR
WISDM
UCIHAR
94
SHOT
207.5
60
85.0
90
12.5
B1OT (W9) Newiin))
66
66
E40 Sep (991
NRC
(3)
366.0
92.5
80
90.0
62.5
67.5
50
-.000
1PC (FT (91)
pe (01
100
100
5.10
100
5.0
100
AAD
80
Store
40
60
AAB.1 301 boit
50
20
6.5
5.0
100
0.5
5.0
100
0.5
6.0
100
0.5
5.0
100
0.5
5.0
100
95
70
MAPU
80
65
Score
75
80
04.81 p1%)
Mihi- (911 55
85
Unitsbeled Semples
% Unnlabeled Samples
Liitabeled Samples
5. L.niisbelled Samples
LLntebhe led Samples
Comparison of the predictive performance of our selective fine-tuning (SFT) strategy W.r.t. Fl score (%) at different target sample ratios against baseline methods (SHOT, NRC, AAD, MAPU). Adaptations are conducted using 0.5%, 5%, and 100% of the total unlabeled target samples, randomly sampled in a stratified manner. The figure demonstrates performance differences across methods as the amount of target data varies, highlighting the sample efficiency of the proposed SFT strategy. Results averaged over three runs.
Figure 6:
The predictive performance of the methods is evaluated by comparing the average F1 score of the target-adapted model across all source-target pairs for the respective dataset in the benchmark. Each experiment is repeated three times with different random seeds to ensure robustness. More details provided in Appendix A.9.
Table 1: Performance and efficiency comparison on SSC dataset across SFDA methods, reported as average F1 score (%) at target sample ratios (0.5%, 5%, 100%), inference MACss (M), and fine-tuned parameters (K). Highlighted rows show results for SFT, where only the core tensor is fine-tuned at different RF values. Green numbers represent average percentage improvement, while Red numbers indicate reduction in MACs and fine-tuned parameters. In Appendix A.9 (Table 4), we extend this analysis to MFD, HHAR, WISDM, and UCIHAR datasets.
Score (%)
Methods
RF
0.5%
5%
100%
Average
MACs (M)
Params, (K) J
SHOT (Llong al., 2020)
62.32 1.57
63.95 1.51
67.95 1.04
64.74
12:92
83.17
62.53 0.46
66.55
0.46
10.50
1.33
61.53
0.80 (93.819)
1.38
SHOT SFT
62.71 0.57
66
68.56
66.14 (2. 16%)
(84.60%)
5.32 (93.60%)
63:05 + 0.32 2
65 44 + 0.8
67.48
+ 0.89
65.32 (0.90%)
5.54 (57.12%)
20.88 (74.89%)
(YCn al., 2021a)
59.92 + 1.19
56 + 1.35
0.55
(5.23
62,90
12:92
83.17
60.65 + 1.37
63.60
1.43
65.05
1.66
63.10 (0.
0.80 (93.8198)
1.38 34%)
NRC SFT
61.95 : 0.62
65.11
1.34
67.19 0.14
64.75 (2 94%))
1.99 (846)
5.32 (93.60%)
60.60 + 0.58 2
0.24
66.85
+ 0.51
64.16 (2.00%)
5.54 12%)
20.88 (74.89%
AAD(Yangg ot al,, 2022)
59.39 + 80
63:21
1,53
63.71
2.06
62.10 63.93 (2
12.92
83.17
1.40
(93.8198)
1.38 (98.34%%
AAD + SFT
61.92 : 0.68
66.59
0.95
67.14 2: 0.57
65.22 (5 (2%)
1.99 (84.6095)
5.32 (93.60%)
60.59 + 0.59 2
96.6
2.04
64.39 + 1.45
62.98 (1 42%)
5.54 (57. 12%)
20.88 (74.89%
MAPU (Ragub et al., 20236)
60.35 à 2.15
62.48
1.57
66.73 0.85
63.19
12.92
$3.17
63.52 1 1.24
64.77
66.09 0.19
64.79 (2 (2.53%%
(93.8198)
1.38 (98.34%%
MAPU + SFT
62.21 1 0.88
20
1,25
67.40 2: 0.59
64,94 (2.77%)
1.99 (84.6095)
5.32 (93.60%)
2
62.25 + 1.74
66.19 + 1.02
67.85 + 0.62
63.43 (3 54%)
5.54 12%)
20.88 (74.89%)
Sample-efficiency across SFDA Methods and Datasets. We assess the sample efficiency of the proposed SFT method by evaluating its predictive performance in the low-data regime. To conduct this analysis, we randomly select 0.5% and 5% of the total available unlabeled target samples in a stratified manner, utilizing fixed seeds for consistency. These sampled subsets serve exclusively for the adaptation process. In Figure 6, we present a comparative analysis of the average Fl score post-adaptation, contrasting SFT with baseline methods (SHOT, NRC, AAD, and MAPU) across
8

---

Published as a conference paper at ICLR 2025
LoRA-2
LoRA-4
LoRA-8
LoRA-16
LoKrA
87.55
67.6
67.5
67.6
SHOT
Ft Bonna
65.0
65.0
60.0
65.0
65.0
62.6
62.0 5
62.5
62.5
62.5
00.0
5.
6.0
100
0.6
6.0
100
60.0
0.5
5.0
aod Ldkc
100
0.5
6.0
100
0.6
6.0
1407 Lock
100
67.0
07.6
AAD
66.0
65.0
65.0
65.0
85.0
82.5
62.5
62.5
82.5
00.00
00.0
00.0
00:0
60.0
Boore (%)
NRC
MB
14 Lib
000
140 Lock
6.5
6.0
100
0.5
6.0
100
0.5
5.0
100
0.5
5.0
100
0.5
5.0
100
Ft Store 1%3
MAPU
85
65
85
85
85
A/A an aru
60
60
60
60
0.6
6.0
100
6.0
100
0.5
5.0
100
0.5
6.0
100
0.6
5.0
100
Samples
Samples
Uniisbeldd Samples
Untiebeled Samples
Unisbaleed Sumples
Method
Baseline
SET (RF-2)
SFT (RF-4)
SFT (RF-8)
LoRA-2
LoRA-4
LoRA-8
LoRA-16
LoKrA
MACs (M).
12.92
5.54
1.99
0.80
12.92
12.92
12.92
12.92
12.92
# Paramss (K) 4
83.17
20.88
5.32
1,38
0.81
1.94
5.19
15.63
1.84
Figure 7: Comparison of LoRA (Hu et al., 2021) and LoKrA (Edalati et al., 2022; Yeh et al., 2024) (in Purple) against baseline methods (SHOT, NRC, AAD, MAPU) and our proposed approach (SFT) on the SSC dataset, evaluated across varying target sample ratios used during adaptation. The table at the bottom shows the target model's inference overhead after adaptation in terms of MACs and the number of parameters finetuned at the time of adaptation.
multiple datasets: SSC, MFD, HHAR, WISDM, and UCIHAR. Notably, our empirical observa- tions are grounded in the theoretical framework established by Theorem 1, which is encapsulated in Equation (10). While all SFDA methods aim to minimize the empirical loss on the target do- main (the first term on the right-hand side of Equation (10)) through their unsupervised objectives (ef. Appendix A.8), the second term term- dependenn on both the number of samples and the distance between parameters- -llas critical role in generalization. In low-data regimes, the number of target samples reduces, this second term increases, thereby weakening the generalization bound. As result, baseline methods exhibit a noticeable drop in predictive performance, distinctly observed for the MFD and UCIHAR datasets (Figure 6), when adapting under the low sample ratio setting. In contrast, SFT effectively mitigates the adverse effects of small number of samples by implicitly constraining the parameter distance term (the numerator of the second term on the RHS), as empir- ically observed in Figure 5, therefore, balancing the overall fraction to not loosen the bound on the RHS. This renders SFT significantly more sample-efficient, preserving robust generalization even in low-data regimes. We extend this analysis in Appendix A.9.
Parameter-efficiency across SFDA Methods and Datasets. In addition, Table demonstrates consistent improvements in the Fl score alongside enhanced parameter efficiency during the adapta- tion process, as evidenced by the reduced parameter count (# Params.) of the parameters fine-tuned. As the rank factor (RF) increases, the size of the core subspace decreases (cf. Equation (8)), leading to fewer parameters that require updates during adaptation. This reduction not only enhances effi- ciency but also lowers computational costs (MACs) (cf. Equation (9)). The observed performance gains and improved efficiency are consistent across different SFDA methods and datasets, underscor- ing the robustness, generalizability, and efficiency of our approach across different SFDA methods and benchmark datasets. In Appendix A.9, we extend this analysis to MFD, HHAR, WISDM and UCIHAR datasets and observe similar trends.
Comparison with Parameter-efficient Tuning Methods. To further substantiate the robustness of SFT, we benchmark its performance against Sarameter-Effiiien Fine-Tuning (PEFT) approaches (Han et al., 2024; Xin et al., 2024). PEFT methods primarily aim to match the performance of full model fine-tuning while substantially reducing the number of trainable parameters. Among the most prominent approaches, Low-Rank Adaptation (LoRA) (Hu et al., 2021) has gained significant trac- tion in this domain. In Figure 7, we analyze the performance of LoRA at varying intermediate ranks
9

---

Published as a conference paper at ICLR 2025
Table 2: Ablation studies finetuning different parameter subspaces during target-side adaptation on MAPU.
Mehod
RF
Pacmmazee Fincuned
SSC
(M)
HHAR
No Adietation
Nose Backhone
56.N 0
12:02
6667 = 1,03
9:04
MAPU
66:23 8.0.5
92.07
52:32 1.15
604
198.21
Paeter Martens (V1) Vialy
66.11 0:275
08
5.5
50.42 1:51
0.53
7:18
Care (T) Ficter Mitricns (V11) y 1 Oue Tanor (T)
06.09 2..1
088
1.38
0.53
3.19
00.5
088
4.1
40:29 1/04
0.53
10.37
Core Treser (7)
67 00 4.0.0
1:94
50:34 1.55
MAPU SPT
Fozter Maancoo (V'1, Vidly Dun Teroue (7)
67.82 0.221
1.94
11.48
29634 1.6
26.88
Pictor Matticos VVll. Villa
62:13::0.0
5:54
13:31
80.51 2:41
3.79
28.68
Core (7)
67.85 0.0
5:34
20.88
y
(f2, 4, 8, 16)), and Low-Rank Kronecker Adaptation (LoKrA) (Edalati et al., 2022; Yeh al., 2024) under its most parameter-efficient configuration, alongside SFT, on the SSC dataset (additional re- sults are provided in Appendix Figure 12). Remarkably, SFT consistently outperforms across SFDA tasks, demonstrating superior adaptability and performance. This advantage arises from the struc- tured decomposition applied during source model preparation, which explicitly disentangles the pa- rameter subspace. In contrast, LoRA-style methods introduce a residual branch over the pretrained weights that relies on the fine-tuning objective to uncover the underlying low-rank subspaces (Hu et al., 2021), which results in weaker control over the adaptation process. It is important to highlight For instance, LoRA or LoKrA-style adaptation frameworks can be applied to the core tensor, further enhancing the efficiency of the adaptation process, which we discuss in detail in Appendix A. However, while A-syle methods can effectively adapt model weights to the target domain, they do not reduce inference overhead, achieving this requires explicit decomposition and reparameterization of the weights.
that PEFT methods can also be seamlessly integrated with SFT during
Ablation Studies on Fine-tuning different Parameter Subspaces. In Table 2, we present an ab- lation study evaluating the impact of fine-tuning different components of the decomposed backbone for MAPU. We compare against baseline methods: (1) fine-tuning the entire backbone and (2) tun- ing only Batch-Norm (BN) parameters. For our decomposed framework, we assess: 1. Fine-tuning only the factor matrices: Here, we update the factor matrices while keeping the core tensor fixed, adjusting directional transformations in the weight space. 2. Fine-tuning only the core tensor: We freeze the factor matrices and tune only the core tensor, the smallest low-rank subspace, capturing critical multi multi-dimensional interactions. This is efficient, especially with a high RF values, as the core tensor remains compact, improving performance with fewer parameters and enhancing sample efficiency. 3. Fine-tuning both the core tensor and factor matrices: Both are updated, offering more flexibility but introducing more parameters, risking overfitting. Primarily tuning the core tensor is advantageous due to its compact representation, yielding better generalization, and freezing the fac- tor matrices preserves mode-specific transformations while optimizing key interactions. We observe that core tensor fine-tuning strikes an optimal balance between adaptation and parameter efficiency, delivering strong performance with a minimal computational footprint. Appendix Tables 11, 12, and 13 show the ablation analysis for SHOT, NRC and AAD, respectively.
5 CONCLUSION
In this work, we presented a framework for improving the parameter and sample efficiency of SFDA methods in time-series data through a low-rank decomposition of cource-pretrained models. By leveraging Tucker-style tensor factorization during the source-model preparation phase, we were able to reparameterize the backbone of the model into compact subspace. This enabled selective fine-tuning (SFT) of the core tensor on the target side, achieving robust adaptation with significantly fewer parameters. Our empirical results demonstrated that the proposed SFT strategy consistently outperformed baseline SFDA methods across various datasets, especially in resource-constrained and low-data scenarios. Theoretical analysis grounded in PAC-Bayesian generalization bounds pro- vided insights into the regularization effect of SFT, highlighting its ability to mitigate overfitting by constraining the distance between source and target model parameters. Our ablation studies fur- ther reinforced the effectiveness of selective finetuning, showing that this approach strikes a balance between adaptation flexibility and parameter efficiency. Appendix A.13, we discuss the limita- tions of the presented work. Overall, our contributions positively complement the SFDA methods, offering framework that can be seamlessly integrated with existing SFDA techniques improve adaptation efficiency. This lays the groundwork for future research into more resource-efficient and personalized domain adaptation techniques.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
We would like to express our sincere gratitude to Sheikh Shams Azam for his invaluable feedback, thoughtful suggestions, and insightful advice throughout the drafting process of this work.
REFERENCES
Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, Jorge Luis Reyes-Ortiz, al. A public domain dataset for human activity recognition using smartphones. In Esann, 2013.
Peter L Bartlett, Dylan Foster, and Matus Telgarsky. neural networks. In NewrIPS, 2017.
normalized margin bounds for
Peter L. Bartlett, Nick Harvey, Christopher Liaw, and Abbas Mehrabian. Nearly-tight ve-dimension and pseudodimension bounds for piecewise linear neural networks. JMLR, 2019.
Mathilde Bateson, Hoel Kervadec, Jose Dolz, Hervé Lombaert, and Ismail Ben Ayed. Source-free domain adaptation for image segmentation. Medical Image Analysis, 2022.
Ruichu Cai, Jiawei Chen, Zijian Li, Wei Chen, Keli Zhang, Junjian Ye. Zhuozhang Li. Xiaoyan Yang, and Zhenjie Zhang. Time series domain adaptation via sparse associative structure align- ment. In AAAI, 2021.
Giuseppe Calvi, Ahmad Moniri, Mahmoud Mahfouz, Qibin Zhao, and Danilo P Mandic. Com- pression and interpretability of deep neural networks via tucker tensor layer: From first principles to tensor valued proppappoaagtionn arXiv preprint arXiv: /903.06/33 /3,3, 2019.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In ECCV, 2018.
J Douglas Carroll and Jih-Jie Chang, Analysis of individual differences in multidimensional scaling via an n-way generalization of "eckart-young" decomposition. Psychometrika, 1970.
Wei Chen, Zichen Miao, and Qiang Qiu. Inner product-basee neural network similarity. NeurIPS, 2024a.
Wei Chen, Zichen Miao, and Qiang Qiu. Parameter-effict tuning of large convolutional models. arXiv preprint wrXiv:2403.00269, 2024b.
Mingyue Cheng, Jiqian Yang. Tingyue Pan, Qi Liu, and Zhi Li. Convtimenet: A deep hierarchical fully convolutional model for multivariate time series analysis. arXiv preprint 233 2024.
Xiuyuan Cheng, Zichen Miao, and Qiang Qiu. Graph convolution with low-rank learnable local filters. In ICLR, 2021.
Tri Dao, Beidi Chen, Nimit Sohoni, Arjun Desai, Michael Poli, Jessica Grogan, Alexander Liu, Aniruddh Rao, Atri Rudra, and Christopher Ré. Monarch: Expressive structured matrices for efficient and accurate training. In ICML. 2022.
Lieven De Lathauwer, Bart De Moor, and Joos Vandewalle. On the best rank-1 1 and rank-(rl .22 .,rn) approximation of higher-order tensors. SIAM Journal on Matrix Analysis and Applications, 2000.
Misha Denil, Babak Shakibi, Laurent Dinh, Mare' Aurelio Ranzato, and Nando De Freitas, Predict- ing parameters in deep learning. In NeurIPS, 2013.
Emily L Denton, Wojeiech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In NewrIPS, 2014.
Ning Ding, Yixing Xu, Yehui Tang, Chao Xu, Yunhe Wang, and Dacheng Tao. Source-free domain adaptation via distribution estimation. In CVPR, 2022.
"https://shant" pai rm1 comm

---

Published as a conference paper at ICLR 2025
Luo Donghao and Wang Xue. ModemTCN: A modern pure convolution structure for general time series analysis. ICLR, 2024.
Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. In UAI, 2017.
C. Eckart and G. Young. The approximation of one matrix by another of lower rank. Psychometrika, 1936.
Ali Edalati. Marzich Tahaci, Ivan Kobyzev, Vahid Partovi Nia. James J Clark, and Mehdi Rezagholizadeh. Krona: Parameter efficient tuning with kronecker adapter. arXiv preprini arXiv:2212. 10650, 2022.
Wei Fan, Pengyang Wang, Dongkun Wang, Dongjie Wang, Yuanchun Zhou, and Yanjie Fu. Dish-ts: general paradigm for alleviating distribution shift in time series forecasting. In AAAI, 2023.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR, 2018.
Muhammad Tanzil Furqon, Mahardhika Pratama, Ary Shiddiqi, Lin Liu, Habibullah Habibullah, and Kutluyil Dogancay. Time and frequency synergy for source-free time-series domain adaptations. Information Sciences, 2025.
Shanghua Gao, Teddy Koker, Owen Queen, Thomas Hartvigsen, Theodoros Tsiligkaridis, and Marinka Zitnik. UniTS: unified multi-task time series model. In NeurlPS, 2024.
Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Chung- LaanngKKaggaaa Peng, and H Eugene Stanley. Physiobank, physiotoolkit, and physionet: components of new research resource for complex physiologic signals. Circulation, 2000.
Peiliang Gong, Mohamed Ragab, Emadeldeen Eldele, Wenyu Zhang, Min Wu, Chuan-Sheng Foo, Daoqiang Zhang, Xiaoli Li, and Zhenghua Chen. Evidentially calibrated source-free time-series domain adaptation with temporal imputation. arXiv preprint arXiv:2406.020 2024.
Peiliang Gong, Mohamed Ragab, Min Wu, Zhenghua Chen, Yongyi Su, Xiaoli Li, and Daoqiang Zhang. Augmented contrastive clustering with uncertainty-aware prototyping for time series test time adaptation. arXiv preprint arXiv:2501.0177,, 2025.
Henry Gouk, Timothy M Hospedales, and Massimiliano Pontil. Distance-based regularisation of deep networks for fine-tuning. In ICLR, 2021.
Sam Greydamus and Dmitry Kobak. Scaling down deep learning with mnist- 1d. In /CML, 2024.
Dimitrios Halatsis, Grigorios Chrysos, Joao Pereira, and Michael Alummoottil. TUCKER DECOM- POSITION FOR INTERPRETABLE NEURAL ORDINARY DIFFERENTIAL EQUATIONS. In ICLR Workshop on A14DifferenialEEuattoss In Science, 2024.
Zeyu Han, Chao Gao, Jinyang Liu, Sai Qian Zhang, et al. Parameter-efffcient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403. 14608, 2024.
Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. In NewrIPS, 1992.
Huan He, Owen Queen, Teddy Koker, Consuelo Cuevas, Theodoros Tsiligkaridis, and Marinka Zitnik. Domain adaptation for time series under feature and label shifts. In ICML, 2023.
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. JMLR, 2021.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2021.
M Jaderberg, A Vedaldi, and A Zisserman. Speeding up convolutional neural networks with low rank expansions. In BMVC, 2014.
12

---

Published as a conference paper at ICLR 2025
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, 2021.
Yiding Jiang*, Behnam Neyshabur*, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. ICLR, 2020.
Xiaoyong Jin, Youngsuk Park, Danielle Maddix, Hao Wang, and Yuyang Wang. Domain adaptation for time series forecasting via attention sharing. In ICML, 2022.
Nazmul Karim, Niluthpol Chowdhury Mithun, Abhinav Rajvanshi, Han-pang Chiu, Supun Sama- rasekera, and Nazanin Rahnavard. C-sfda: A curriculum learning aided self-training framework for efficient source free domain adaptation. In CVPR, 2023.
Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, and Dongjun Shin. Com- pression of deep convolutional neural networks for fast and low power mobile applications. In ICLR, 2016.
Youngeun Kim, Donghyeon Cho, Kyeongtak Han, Priyadarshini Panda, and Sungeun Hong. Do- main adaptation without source data. IEEE TAI, 2021.
Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv: 2014.
Tamara G. Kolda and Brett Bader. Tensor decompositions and applications. SIAM Review, 2009.
Jogendra Nath Kundu, Naveen Venkat, and R Venkatesh Babu. Universal source-free domain adap- tation. In CVPR, 2020.
Jogendra Nath Kundu, Akshay Kulkarni, Amit Singh, Varun Jampani, and Venkatesh Babu. Gen- eralize then adapt: Source-free domain adaptive semantic segmentation. In ICCV, 2021.
Jogendra Nath Kundu, Suvaansh Bhambri, Akshay Kulkarni, Hiran Sarkar, Varun Jampani, and R Venkatesh Babu. Concurrent subsidiary supervision for unsupervised cource-free domain adap- tation. In ECCV, 2022a.
Jogendra Nath Kundu, Akshay R Kulkarni, Suvaansh Bhambri, Deepesh Mehta, Shreyas Anand Kulkarni, Varun Jampani, and Venkatesh Babu Radhakrishnan. Balancing discriminability and transferability for source-free domain adaptation. ICMI, 2022h.
Jennifer R. Kwapisz, Gary M. Weiss, and Samuel A. Moore. Activity recognition using cell phone accelerometers. SIGKDD Explor. Newsl., 2011.
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long- and short-term temporal patterns with deep neural networks. The 41st International ACM SIGIR Conference on Research Development Information Retrieval, 2017.
Lieven De Lathauwer, Bart De Moor, and Joos Vandewalle. multilinear singular value decompo- sition. SIAM Journal on Matrix Analysis and Applications, 2000.
V Lebedev, Y Ganin, M Rakhuba, I Oseledets, and V Lempitsky. Speeding-up convolutional neural networks using fine-tuned cp-decomposition. In ICLR, 2015.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. In NeurIPS, 1989.
Suho Lee, Seungwon Seo, Jihyo Kim, Yejin Lee, and Sangheum Hwang, Few-shot fine-tuning is all you need for source-free domain adaptation. arXiv preprint arXiv:2304.00792, 2023.
Christian Lessmeier, James Kuria Kimotho, Detmar Zimmer, and Walter Sextro. Condition mon- itoring of bearing damage in electromechanica drive systems by using motor current signals of electric motors: A benchmark data set for data-driven classification. In PHM Society European Conference, 2016.
Dongyue Li and Hongyang Zhang. Improved regularization and robustness for fine-tuning in neural networks. In NeurIPS, 2021.
13

---

Published as a conference paper at ICLR 2025
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. efficient convnets. ICLR, 2017.
Pruning filters for
Jingjing Li, Zhiqi Yu, Zhekai Du, Lei Zhu, and Heng Tao Shen. A comprehensive survey on source- free domain adaptation. IEEE TPAMI, 2024.
Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Si Wu. Model adaptation: Unsupervised domain adaptation without source data. In CVPR, 2020.
Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In ICML, 2020.
Jian Liang, Dapeng Hu, Yunbo Wang, Ran He, and Jiashi Feng. Source data-absent unsupervised domain adaptation through hypothesis transfer and labeling transfer. IEEE TPAMI, 2021.
Mattia Litrico, Alessio Del Bue, and Pietro Morerio. Guiding pseudo-labels with uncertainty esti- mation for source-free unsupervised domain adaptation. In CVPR, 2023.
Qiao Liu and Hui Xue. Adversarial spectral kernel matching for unsupervised time series domain adaptation. In IJCAI, 2021.
Ya Liu, Yingjie Zhou, Kai Yang, and Xin Wang. Unsupervised deep learning for iot time series. IEEE Internet of Things Journal, 2023.
Yu Liu, Duantengchuan Li, Jian Wang, Bing Li, and Bo Hang. Mdlr: A multi-task disentangled learning representations for unsupervised time series domain adaptation. Information Processing & Management, 2024.
Yuang Liu, Wei Zhang, and Jun Wang. Source-free domain adaptation for semantic segmentation. In CVPR, 2021.
Aitian Ma. Dongsheng Luo, and Mo Sha. Mixlinear: Extreme low resource multivariate time series forecasting with 0.1 k parameters. arXiv preprint arXiv:2410.02081 2024.
Xindian Ma, Peng Zhang, Shuai Zhang, Nan Duan, Yuexian Hou, Ming Zhou, and Dawei Song. A tensorized transformer for language modeling, In NeurlPS, 2019.
David McAllester. Simplified pac-bayesian margin bounds. In COLT, 2003.
David A. McAllester. Some pac- muc-bayesian theorems. In COLT, 1998.
David A. McAllester. Pac-bayesian model averaging. In COLT, 1999.
Zichen Miao, Ze Wang, Xiuyuan Cheng, and Qiang Qiu. Spatiotemporal joint filter decomposition in 3d convolutional neural networks. In NeurlPS, 2021.
Zichen Miao, Ze Wang, Wei Chen, and Qiang Qiu. Continual learning with filter atom swapping. In ICLR, 2022.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. In ICLR, 2017.
Rafael Müller, Simon Korbllith, and Geoffrey E Hinton. When does label smoothing help? In NewrIPS, 2019.
Shinichi Nakajima, Masashi Sugiyama, S. Derin Babacan, and Ryota Tomioka. Global analytic solution of fully-observed variational bayesian matrix factorization. JMLR, 2013.
Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A pac-bayesian approach to opectrally-normalizee margin bounds for neural networks. In ICLR, 2018.
Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov. Tensorizing neural
networks. In NeurlPS, 2015.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint 1 2018.
14

---

Published as a conference paper at ICLR 2025
Yilmazean Ozyurt, Stefan Feuerriegel, and Ce Zhang. Contrastive learning for unsupervised domain adaptation of time series. ICLR, 2023.
Yeon Park, Kichang Lee, Sungmin Lee, Mi Zhang, and JeongGil Ko. Attfl: A personalized federated learning framework for time-series mobile and embedded sensor data processing. In Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., 2023.
Shikai Qiu, Andres Potapczynski, Mare Anton Finzi, Micah Goldblum, and Andrew Gordon Wilson.
Compute better spent: Replacing dense layers with structured matrices. In ICMI., 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Mohamed Ragab, Emadeldeen Eldele, Zhenghua Chen, Min Wu, Chee- Keong Kwoh, and Xiaoli Li. Self-supervised autoregressive domain adaptation for time series data. IEFE TNNIS, 2022.
Mohamed Ragab, Emadeldeen Eldele, Wee Ling Tan, Chuan-Sheng Foo, Zhenghua Chen, Min Wu, Chee-Keong Kwoh, and Xiaoli Li. Adatime: A benchmarking suite for domain adaptation on time series data. ACM TKDD, 2023a.
Mohamed Ragab, Emadeldeen Eldele, Min Wu, Chuan-Sheng Foo, Xiaoli Li, and Zhenghua Chen. Source-free domain adaptation with temporal imputation for time series data. KDD, 2023b.
Cristiano Saltori, Stéphane Lathuiliére, Nicu Sebe, Elisa Ricci, and Fabio Galasso. Sf-uda 3d: 3d object detection. In 3DV, 2020.
Source-free unsupervised domain adaptation for
Pratyusha Sharma, Jordan T. Ash, and Dipendra Misra. The truth is in there: Improving reasoning in language models with layer-selective rank reduction. In ICLR, 2024.
Allan Stisen, Henrik Blunck, Sourav Bhattacharya, Thor Siiger Prentow, Mikkel Baun Kjærgaard, Anind Dey, Tobias Sonne, and Mads Møller Jensen. Smart devices different: Assessing and mitigatingmobile sensing heterogeneities for activity recognition. In ACM Conference on Embedded Networked Sensor Systems, 2015.
Cheng Tai, Tong Xiao, Yi Zhang, Xiaogang Wang, and E Weinan. Convolutional neural networks with low-rank regularization. In ICLR, 2016.
Song Tang, An Chang, Fabian Zhang, Xiatian Zhu, Mao Ye, and Changshui Zhang. Source-free domain adaptation via target prediction distribution searching. IJC,, 2024a.
Song Tang, Wenxin Su, Mao Ye, and Xiatian Zhu. Source-free domain adaptation with frozen multimodal foundation model. In CVPR, 2024b.
L. R. Tucker. Some mathematical notes on three-mode factor analysis. Psychometrika, 1966.
Ze Wang, Seunghyun Hwang. Zichen Miao, and Qiang Qiu. Image generation using continuous filter atoms. In NeurIPS, 2021.
Zifan Wang, Nan Ding, Tomer Levinboim, Xi Chen, and Radu Soricut. Improving robust general- ization by direct pac-bayesian bound minimization. In CVPR, 2023.
Garrett Wilson, Janardhan Rao Doppa, and Diane J Cook. Multi-source deep domain adaptation with weak supervision for time-series sensor data. In KDD, 2020.
Garrett Wilson, Janardhan Rao Doppa, and Diane Cook. Calda: Improving multi-source time series domain adaptation with contrastive adversarial learning. IEEE TPAMI, 2023.
Haifeng Xia, Handong Zhao, and Zhengming Ding. Adaptive adversarial network for source-free domain adaptation. In ICCV, 2021.
Yi Xin, Siqi Luo, Haodi Zhou, Junlong Du, Xiaohong Liu, Yue Fan, Qing Li, and Yuntao Du. Parameter-efficient fine-tuning for pre-trained vision models: A survey. arXiv preprint arXiv:2402.02242, 2024.
Lin Xiong, Mao Ye, Dan Zhang, Yan Gan, and Yiguang Liu. Source data-free domain adaptation for faster r-cnn. Pattern Recognition, 2022.
15

---

Published as a conference paper at ICLR 2025
Shiqi Yang, Joost Van de Weijer, Luis Herranz, Shangling Jui, et al. Exploiting the intrinsic neigh- borhood structure for source-free domain adaptation. In NeurIPS, 2021a.
Shiqi Yang, Yaxing Wang, Joost Van De Weijer, Luis Herranz, and Shangling Jui. Generalized source-free domain adaptation. In ICCV, 2021b.
Shiqi Yang, Shangling Jui, Joost van de Weijer, et al. Attracting and dispersing: A simple approach for source-free domain adaptation. In NeurIPS, 2022.
Shiqi Yang, Yaxing Wang. Joost Van de Weijer, Luis Herranz, Shangling Jui, and Jian Yang. Trust your good friends: Source-free domain adaptation by reciprocal neighborhood clustering. IEEE TPAMI, 2023.
Jiexia Ye, Weiqi Zhang, Ke Yi, Yongzi Yu, Ziyue Li, Jia Li, and Fugee Tsung. A survey of time series foundation models: Generalizing time series representation with large language model. arXiv preprint arXiv:2405.02358, 2024.
Jinmian Ye, Linnan Wang, Guangxi Li, Di Chen, Shandian Zhe, Xinqi Chu, and Zenglin Xu. Learn- ing compact recurrent neural networks with block-term tensor decomposition. CVPR, 2018.
Shih- Ying Yeh, Yu-Guan Hsieh, Zhidong Gao, Bernard BW Yang, Giyeong Oh, and Yanmin Gong. Navigating text-to-image customization: From lycoris fine-tuning to model evaluation. In ICLR, 2024.
Wenda Zhou, Victor Veitch, Morgane Austern, Ryan Adams, and Peter Orbanz. Non-vacuous generalization bounds at the im-agenet scale: A pac- pac-bayesian compression approach. In ICLR, 2019.
Max Zimmer, Megi Andoni, Christoph Spiegel, and Sebastian Pokutta. Perp: Rethinking the prune- retrain paradigm in the era of 1lms, arXiv preprint arXiv:2312.152380 2023.
16

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 HIGHER ORDER ORTHOGONAL. ITERATION
Algorithm 1: The higher-order orthogonal iteration (HOOI) algorithm. (De Lathauwer et al., 2000; Kolda & Bader, 2009)
Input: Tensor A R Truncation R2, .... Ry),
.,,,,, Initial guess
(U) n = 1,2,,...NN
Output: Core tensor g, Factor matrices l :n 1,2, 1....... N}
k - 0:
while not converged do
for all E {1,2,.... N do
B A xi (U+)) (+1)
T X2.. X ( Xn+1 (Ulnt)))T... XN )
(m) B in matrix format;
U.E, V truncated rank- R. SVD of (())
k+1 U;
k +1;;
9 + EVT tensor format;
A.2 PARAMETER AND COMPUTATIONAL EFFICIENCY WITH TUCKER DECOMPOSITION
A.2.1 TUCKER FACTORIZATION FOR CNN WEIGHTS
Consider 3D weight tensor W E RCaxCu for ID convolutional layer, where Cau and Cin represent the output and input channels, and is the kernel size. Using Tucker factorization, W can be decomposed into a core tensor T x Rlx and factor matrices ( Cour ( C x R2, such that:
W = T X1 V() X2 (()
(11)
Parameter Efficiency.
The number of parameters before factorization is: ParamSoriginal
Cout x Ci x K
(12)
After 2-Mode Tucker factorization, the number of parameters become:
ParamStucker Rout X Rin K Cout
Rout + Cin x Rin
(13)
Given that R. < Cin, and R < Cous the reduction in the number of parameters is significant, leading to a parameter-efficient model.
Computational Efficiency. The convolution operation requires OCCour x X X x L') operations, where L' is the length of the output feature map. After Tucker factorization, the convolutional operations become a sequence of smaller convolutions involving the factor matrices and the core tensor, reducing the computational complexity
OCCin x Rm x L.) O x Rm, x K x L') OCCou x R X Lt)
(14)
where L denotes the length of the input sequence. This reduction depends on the rank Ry and Routs leading to lower computational costs compared to the original convolution.
A.2.2 TUCKER FACTORIZATION FOR FULLY CONNECTED WEIGHTS
Consider a 2D weight matrix W E R in fully connected (FC) layer, where M is the input dimension and N is the output dimension.
17

---

Published as a conference paper at ICLR 2025
Using Tucker factorization, W can be decomposed into matrix G IR R and factor matrices U(1) RMxR, and U E TR,,R such that:
W G(UR)T
(15)
Parameter Efficiency.
The number of parameters before factorization is:
ParamSoriginal M x N
(16)
After Tucker factorization, the number of parameters becomes:
ParamSTucker R4 x Rg + M x R1 + + Rz
(17)
Again, with R < M and Rg < N, this leads to substantial reduction in the number of parame- ters.
Computational Efficiency. The original matrix multiplication requires O(M N) operations.
After Tucker factorization, the computation broken down into smaller matrix multiplications:
O(M x R1) OR x R2) + O x N)
(18)
This decomposition reduces the computational cost, particularly when R₁ and R2 are much smaller than M and N.
A.2.3
COMPUTATION ANALYSIS
Original Convolutional Layer: Consider convolutional layer with the following dimensions:
Input Channels (Cin): 128
Output Channels (Com): 256
Kernel Size (K):
Input Length (L): Length of the input sequence
Output Length (Z'): Length of the output feature map
simputationnSoiginall OCCous Cin K I!' 256 128 x 8 x T' 262,144 x I! (19)
After Tucker Decomposition: We apply Tucker decomposition along the channel dimensions (in- and output channels) of the weight tensor. The decomposition factorizes the original weight tensor into smaller tensors, introducing two rank parameters:
Input Rank (Rn):: Reduced dimension for input channels
Output Rank (Row): Reduced dimension for output channels
The computational complexity after Tucker decomposition becomes:
Computations,uster = OC'nn x Rm L) + O(Rour X Rm x K x L') + OCCout x Rout x L')
Input Projection + Core Convolution Output Projection
(20)
Components Computation Explanation:
Input Projection: Project the input feature maps from Cjn channels to Ri channels:
Input Projection Cin x Rin x L 128 x Rin x L
(21)
Core Convolution: Perform convolution using the core tensor of size Row. R. K:
Core Convolution Rout x Ri K T'' Ruu Rm 8 x I'
(22)
18

---

Published as a conference paper at ICLR 2025
Output Projection: Project the result from Rout channels to Cout output channels: Output Projection Cout Roa x I 256 x Rom T
(23)
Computational Efficiency Demonstration: For demonstration purposes, let us assume equal duced ranks for input and output channels: Rin = Rom = R. where Rt 32
Computations After Decomposition:
Input Projection:
Input Projection 128 32 L 4,096 T.
(24)
Core Convolution:
Core Convolution 32 x 32 8 X L' 8,192 L'
(25)
Output Projection:
Output Projection - 256 x 32 x L = 8,192 x L'
(26)
Total Computations After Decomposition:
Computatioon&gcomppsd 4,096 x L + 8,192 x L' 8,192 x L' = 4,096 x L+ 16,384 x L' (27)
Comparing Computational Costs: Assuming the input and output lengths are approximately equal (L s L'), we can directly compare the total computations.
Total Computations Before Decomposition:
Computations original 262,144 L'
(28)
Total Computations After Decomposition:
ComputationSnukkr 4,096 L' 16,384 L' 200,480 L'
(29)
Computational Reduction:
Reduction Ratio ComputationsTucee
20,480 L'
~ 0.078
(30)
Computations original
262,144 x L'
This indicates a reduction of approximately 92% in computational cost after Tucker decomposition. The reduced ranks Rm and Rour lead to lower computational complexity compared to the original convolutional layer. The substantial decrease in computations directly results in faster inference times, as the model performs significantly fewer operations.
In sum, Tucker factorization significantly reduces both the number of parameters and the computa- tional cost for ID CNN and FC weights, making it an effective technique for achieving parameter efficiency and computational efficiency in deep neural networks.
A.3 FORWARD- AND BACKWARD- PASS EQUIVALENCE WITH TUCKER DECOMPOSITION
In the interest of simplicity and ease of explanation, demonstrate equivalence by conduct- ing the Tucker decomposition for fully connected weights, i.e., a 2-D matrix, such that, W = U G(U())).
Forward Propagation.
Case I: Using W = U(G(((2) directly Let the input to the layer be x. Forward propagation through W is represented as:
Wr U
Case Using U G, (U(2)) T separately. We compute the forward pass in steps:
1. Compute y1 (U()))Tz
2. Compute 3/2 Gly = G(
3. Compute
U 12 u ( (G (G((t()) x)).
By associativity of matrix multiplication, this is equivalent to: ( G(U(2)
Wr
Hence, forward propagation through U@)) G. (U()) T in sequence is equivalent to propagating through W directly.
19

---

Published as a conference paper at ICLR 2025
Backward Propagation. Let the gradient of the loss L with respect to the output be OC The task is to propagate this gradient backward. Case I: Using W U G(U()) directly. The gradient of the loss with respect to æ is:
0= 9x-
9x WT J= (U(G(())) 9: G (T 92.
Case 2: Using U G, (U(2)) T separately. Backward propagation proceeds
1. Compute Syz (U()) Dz:
2. Compute Py1 = G 9v2 = G ((U())T
3. Compute Ix U By1 U (GT ((U()) g.)).
By the associativity of matrix multiplication, this simplifies to:
9x = U (Uw)
= g. = W 92-
Gradients with respect to U@ G, (U(2))T.
The gradients of L with respect to the parameters
are:
Gradient with respect to U());
dc
9.32 9= (G xT)
aU())
2. Gradient with respect to G:
dc
((VW) 9.) (0)0)))))(111))) (I))))=T)*
= 91
0G
3. Gradient with respect to (U(2))
0C
Py1 x = (GT ( 9z 9z)) x
((UR))
In sum, the output z is identical whether computed or via U()) G. (U())) or via its composed reconstructed weight W separately. The gradient Ia also renders identical whether computed via UO)) G, (T())T via W separately. Thus, forward and backward propagation through the decom- posed factors U(1, G, (U(2))T is mathematically equivalent to propagating through W directly.
A.4 LEMMAS
Lemma 1. Let Wu E M be the initial weight matrix, and W. R N denote the weight matrix after t iterations of gradient descent with fixed learning rate n > 0. Suppose the gradient
of the loss function L(W) is bounded, such that for all iterations k 0, 0,1,...,4 1, we have ||V (((W)))lee < G where G > O is a constant.
Then, the Frobenius norm of the difference between the initial weight matrix wo and the weight matrix we after iterations is bounded by:
IW, WollF MNV.NG.
Proof. The gradient descent algorithm updates the weight matrix according to the rule:
W = Wr - VW
Starting from the initial weights W. o. the weights after / iterations are:
20

---

Published as a conference paper at ICLR 2025
W. Wo " (-1
Σ VL(W&))
Taking the Frobenius norm of the difference between and we have:
W. Woll= = Σ VL(W))
Using the triangle inequality for norms:
W. Wolle S VL(We)ll.
Since the Frobenius norm || || is sub-multiplicative and satisfies [[V((W))]e S
V and by the assumption [VL[(&))lle G. it follows that:
Substituting this into the earlier expression, we obtain the bound:
Ww - WollF & MNV..
Lemma 2. Consider the weight matrices Wo and W, expressed as:
Wo = 0 Go( and Wi=
where 0 E MRMR and U E N are fixed matrices, and Gu E Rrrxxr and G, E Rrrxxzz represent the evolving core matrices. Assume that the entries of U and U are bounded by constants Cn > and C2 > 0, respectively.
Let the Frobenius norm of the difference between G2 and Gu after t iterations of gradient descent be bounded as:
|G, - GollF S nt Ri RrR
where 1) >> 0 is the learning rate, and G₁ > 0 is constant.
Then, the Frobenius norm of the difference between W. and Wo is bounded by:
W. - W.-WollF S Retoy MN C.G&Co.
M.N
Proof. Given the case where Wo (u( and W, (U( with U and U being fixed in both W, and W,. Where U and u are matrices of dimensions M Ri and N x R2, respectively, and Go and G; are matrices of dimensions R₁ x R2-
The Frobenius norm of the difference between W, and Wa is:
W. Woll || ( U. (
21

---

Published as a conference paper at ICLR 2025
Since Ul and U are common in both Wo and W,, we can factor them out:
W. WollF F | 8 (G. Go)
Using the sub-multiplicative property of the Frobenius norm:
| (G. ( TTr S | ||
Golle|lU@ ||
Since U.! and U are of dimensions M x R₁ and N x R2, and (G, - Go) is Rı x R2.
Assuming the maximum entries of uo and ey are bounded by constants C₂ and C.:
|10% F S VM
M Rı Cn.
|G, Gol F ml v Rí R2 Gs (from Lemma 1)
( IF S /N. R2 C.
Thus:
M AM..M Rí Cu Cu-yl: VR R2 Gk VN R2 Cv W Wolls S VM.
This simplifies to:
W. Wolls S By C.G.C
A.5 PROOF OF THEOREM 1
Background. PAC-Bayesian generalization theory offers an appealing method for incorporating data-dependent aspects, like noise robustness and sharpness, into generalization bounds. Recent studies, such as (Bartlett et al.. 2017; Neyshabur al., 2018). have expanded these bounds for deep neural networks to address the mystery of why such models generalize effectively despite possess- ing more trainable parameters than training samples. Traditionally, the VC dimension of neural networks has been approximated by their number of parameters (Bartlett et al., 2019). While these refined bounds mark a step forward over classical learning theory, questions remain to whether they are sufficiently tight or non-vacuous. To address this, Dziugaite Roy (2017) proposed com- putational framework that optimizes the PAC- bound, resulting in tighter bound and lower test error. Zhou et al. (2019) validated this framework in a large-scale study. More recently, Jiang" et al. (2020) compared different complexity measures and found that PAC-Bayes-based tools align better with empirical results. Furthermore, Li & Zhang (2021) and Wang al. (2023) utilized this bound to motivate their proposed improved regularization and genralization, respectively. Conse- quently, the classical PAC-Bayesian framework (McAllester, 1998; 1999) provides generalization guarantees for randomized predictors (McAllester, 2003; & Zhang, 2021). )). In particular, let fw be any predictor (not necessarily a neural network) learned from the training data and parametrized by W. We consider the distribution Q over parameters of predictors of the form fw. where W is random variable whose distribution may also depend on the training data. Given prior distribution P over the set of predictors that is independent of the training data, S. The PAC-Bayes theorem states that with probability at least 1 - å over the draw of the training data, the expected error of fw can be bounded as follows
KL(Q(S) || P) + kln 3 +1
((s) [C(W)] < Ew~q(s) [[(, s)} + +CV
(31)
Then based on the above described bound we reduce it for our case, where the prior distribution on the parameters is centered on source pre-trained weights and the posterior distribution the target-adapted model and define Theorem 1.
for some k, 0.
Theorem 1. (PAC-Bayes generalization bound for fine-tuning) Let W be some hypothesis class (network parameter). Let P be a prior (source) distribution on W that is independent of the target training set. Let QSS) be posterior (target) distribution on W that depends on the target training set S. consisting of n. number of samples. Suppose the loss function L()) is bounded by C. If we set
22

---

Published as a conference paper at ICLR 2025
the prior distribution P The posterior distribution Q(S) centered at the fine-tuned model as N(W ug, o2 Then with probability 1 - 8 over the randomness of the training set, the following holds:
((W sre, I).,, where W sre are the weights of the pre-trained network.
E ((S) [C(W]] Qws) [(W. s)} + CV
E
W (
W
2
k kln
+
(32)
20
n
for some ó, k, 1, > O, where W og E Wirg irg. and W mr E W We 11 sis D, D denoting the total
number of layers.
It is important to note that the original formulation in (McAllester, 1999) assumes the loss function
restricted to values between and 1. In contrast, the modified version discussed here extends
the applicability to loss functions that are bounded between 0 and some positive constant C. This adjustment is made by rescaling the loss function by factor of c which introduces the constant C in the right-hand side of Equation (31).
Proof. We expand the definition using the density of multivariate normal distributions.
(L(Q(S) P) = Ew-q(() log Pr(W ~ Q((S))
Pr(W ~ P)
Substituting the densities of the Gaussian distributions, this can be written
exp
W
W ug 1?2)
KL(Q(S) || P) Ews)(()
log
20 20
(33)
exp
W
W
scc
This simplifies to:
1
KL(Q(S) P)
20 Ew
[ W 2
W
W ug
(34)
Expanding the squared terms:
1 20²
(L((Q() P)
E QWs)
[ W 2(W W ing W W Wsx]].
(35)
Since the expectation Ew~Q(s) W W rg]
(because W is distributed around W Wng). the
cross-term vanishes:
KL(Q(S) || P)
W ing
W sce 1|2.
(36)
2
D Li=l
w 12
= KL(((S) P) = 20
Wug ug W w v
(37)
202
where. ( W Irg: and ( W sr. V1 M D, D denoting the total number of layers. Then, we can obtain Equation (32) by substituting Equation (37) in Equation (31).
We would also like to emphasize that our primary goal is not to introduce novel theorem but to utilize established PAC-Bayesian bounds as a theoretical framework to explain our empirical obser- vations of implicit regularization and sample efficiency (ef. Figure 2B)). The PAC-Bayesian analysis serves tool to provide theoretical insights into why our strategy exhibits improved performance under sample scarce scenario (discussed Section in the SFDA setting. By grounding our find- ings in existing theoretical work, we aim to bridge the gap between empirical results and theoretical understanding in this specific context.
23

---

Published as a conference paper at ICLR 2025
Table 3: Dataset Summary (Ragab al., 2023a).
Dataset
Users/Domains
# Channels
# Classes
Sequence Length
Training Set
Testing Set
UCIHAR
30
128
2300
990
WISDM
36
3
6
128
1350
720
HHAR
9
3
6
128
12716
5218
SSC
20
1
5
3000
14280
6130
MFD
4
5120
7312
3604
A.6 DATASET DESCRIPTION
We utilize the benchmark datasets provided by AdaTime (Ragab et al., 2023a). These datasets exhibit diverse attributes such as varying complexity, sensor types, sample sizes, class distributions, and degrees of domain shift, allowing for a comprehensive evaluation across multiple factors.
Table 3 outlines the specific details of each dataset, including the number of domains, sensor chan- nels, class categories, sample lengths, and the total sample count for both training and testing sets. A detailed description of the selected datasets is provided below:
UCIHAR (Anguita et al., 2013): The UCIHAR dataset consists of data collected from three types of sensors-aaceeeromeett gyroscope, and body sensors-used on 30 different subjects. Each subject participated in six distinct activities: walking, walking upstairs, walking downstairs, standing, sitting, and lying down. Given the variability across subjects, each individual is considered separate domain. From the numerous possible cross-domain combinations, we selected the ten scenarios set by Ragab al. (2023a).
WISDM (Kwapisz al., 2011): The WISDM dataset uses accelerometer sensors gather data from 36 subjects engaged in the same activities as those in the UCIHAR dataset. However, this dataset presents additional challenges due to class imbalance among differ- ent subjects. Specifically, some subjectsonly contribute samples from a limited set of the overall activity classes. As with the UCIHAR dataset, each subject is treated as indi- vidual domain, and ten cross-domain scenarios set by Ragab et (2023a) are used for evaluation.
HHAR (Stisen et al., 2015): The Heterogeneity Human Activity Recognition (HHAR) dataset collected from 9 subjects using sensor data from both smartphones and smart- watches. Ragab et al. (2023a) standardized the use of single device, specifically a Sam- sung smartphone, across all subjects to minimize variability. Each subject is treated as an independent domain, and a total of 10 cross-domain scenarios are created by randomly selecting subjects.
SSC (Goldberger et al., 2000): The sleep stage classification (SSC) task focuses on cate- gorizing electreencephalography (EEG) signals into five distinct stages: Wake (W), Non- Rapid Eye Movement stages (N1, N2, N3), and Rapid Eye Movement (REM). This dataset is derived the Sleep-EDF dataset, which provides EEG recordings from 20 healthy individ- uals. Consistent with prior research (Ragab et al., 2023a), we select a single EEG channel (Fpz-Cz) and ten cross-domain scenarios for evaluation.
MFD (Lessmeier et al., 2016): The Machine Fault Diagnosis (MFD) dataset has been col- lected by Paderborn University identify various types of incipient faults using vibration signals. 'Te data was collected under four different operating conditions, and in our ex- periments, each of these conditions cross-condition scenarios to evaluate the domain adaptation performance. Each sample in the dataset consists of single univariate channel.
treated separate domain. We used twelve
A.7 SFDA METHOOSS
SHOT (Liang et al., 2020; 2021): SHOT optimizes mutual information by minimizing con- ditional entropy (YY])) to enforce unambiguous cluster assignments, while maximizing marginal entropy H (Y) to ensure uniform cluster sizes, thereby preventing degeneracy.
24

---

Published as a conference paper at ICLR 2025
A. Baseline (vanilla) architecture
1D Conv (2 C...
Natchhoom
1D-Ma (2, 26
1D come az O, K
Butth Nar
Baas Norrs
TD 1D-MMaePooo 20
Liner Glasseher
Hel
RaLL
10-Masras
10-Come
10-0am (R A2
1D.Cavv (52, al
"D-daoohio 21
10-0am de 1)
1D-Cam (R. 80
10. Com 1a4 R
Batch Norm
(2, 21
10-Cremv LR. f)
10-Gamm (220. R.2
Liwer Chosol
10--ommv 1.
RaUu
RaLu
RaL
"D-Meschoo
Eatch
B. Tucker-style parametrized architecture
Figure 8: A. Baseline architecture (Ragab et al., 2023a;b) B. Architecture reprametrization after Tucker-style factorization of convolution weights. Cin and K denote the number of input channels and the filter size, respectively. R(1) and R(!)) and R(()] denote the mode ranks for the zth layer.
NRC (Yang et al., 2021a; 2023): NRC leverages neighborhood clustering, with an objective function comprising two main components: neighborhood clustering term for prediction consistency and a marginal entropy term (YY to promote prediction diversity.
AAD (Yang et al., 2022): AAD also utilizes neighborhood clustering but incorporates contrastive objective similar to InfoNCE (Oord et al., 2018), which attracts predictions of nearby features in the feature space while dispersing (repelling) those that are farther apart.
MAPU (Ragab et al., 2023b; Gong et al., 2024): Building on the concepts from Liang et al. (2021) and Kundu et al. (2022a), and focusing time-series context, MAPU introduces masked reconstruction as an auxiliary task enhance SFDA performance.
A.8 SFDA TARGET ADAPTATION OBJECTIVES
SHOT.
The overall objective is given as follows:
(SHoT(gr) (en + Laiv (f(:t)) BE (e.
p-= log (f)
(38)
where,
Lent Cent(fiit)) EE Ex
de dx(fe(())) log (n(fe(())),
(39)
1=1
p. +
(fi Σ pr log DKL
log K,
(40)
in the above equation DRL stands for the KL divergence. f((x) (x() is the K- dimensional output of each target sample, 1x a K K-dimensional vector with all ones, and p = En Ex [8(f((z))] is the mean output embedding of the whole target domain. Recall that (a)a) Σ. explas) represents the k-th element of the softmax output.
explaz)
Moreover, the dt E Dr denotes the pseudo label on for the target samples, based on DeepCluster (Caron et al., 2018), derived as follows. The centroid for each class in the target domain is obtained, similar weighted k-means clustering:
Emex. (e(fe(ze)) (e(2)) c EnExx, du (f(x)))))
(41)
25

---

Published as a conference paper at ICLR 2025
where fi 91 f4 denotes the previously learned target hypothesis. These centroids robustly and reliably characterize distribution different categories within the target domain. Then, pseudo labels are obtained via the nearest centroid classifier:
f/+
arg min D, ( ck )
(42)
k
where Dy b) measures the cosine distance between a and b. Then, the target centroids based on the new pseudo labels are computed:
ExEx, 1(tt =k) (z.)) ExEx. 1 = k)
(43)
Ú. = arg min Dy (0r (x.), (.)
(44)
de denotes the self-supervised pseudo-labels, since they are generated by the centroids obtained in an unsupervised manner. practice, the centroids and labels are updated for multiple rounds.
NRC. The NRC adaptation objective leverages reciprocal neighborhood clustering to guide learn- ing through a combination of losses:
LSRC = Ldiv + LNN + L. + Lrelt-
(45)
The approach maintains memory banks: one for feature representations (F {21, 2...... 2n } and another for prediction scores (S {P1. PPI.PP.....PPo where 2; and p; denote the feature and the probability score of the i-th sample, respectively. These banks are updated per mini-batch by replacing outdated entries with newly computed values, ensuring efficient and consistent updates. Nearest neighbors are identified using cosine similarity, and their predictions, weighted by affinity values, provide supervision signal:
LN = Th Σ Σ
-
A...Sp..s Ps.
(46)
RENK
where AiA reflects the affinity between sample and its k-th neighbor, S& is the stored prediction, and A,K denotes the K-nearest neighbors of 2;. Neighbors are further classified into reciprocal (RNN) and non-reciprocal (nRNN) based on mutual membership in each other's nearest-neighhor sets. Reciprocal neighbors, being more reliable, are assigned higher affinity values:
1, if N AiE AiENT otherwise.
A = +
(47)
This selective weighting ensures that supervision focuses on semantically similar neighbors. To enhance learning, self-regularization loss aligns a sample's current prediction with its stored mem- ory:
Luell
1
Σ ST
ST..
p
(48)
Additionally, expanded neighbors, defined as the M-nearest neighbors of K-nearest neighbors, pro- vide supplementary supervision with reduced affinities to account for potential noise:
CE - Σ The RENK mEFt
T P..
(49)
Finally, the diversity loss Laiv prevents degenerate solutions by encouraging predictions to span di- verse clusters. Together, these components enable robust adaptation by combining reliable neighbor supervision, self- relfaregularization. and expanded relationships.
26

---

Published as a conference paper at ICLR 2025
Dataset (N subjects)
One-to-one
Many-to-one
source
target
source
target
Train
Test
$ x3
Train
Train
Test
train source model
adapt source model (unsupervised)
test performance (supervised)
Figure 9: visual representation of the experimental setup used for evaluating the Source-Free
Domain Adaptation (SFDA) frameworks.
AAD.
The AAD objective pretty straingforward as follows
LAAD EEL,,CC; B:)), with L,(Ci, B.)
Dt Σntr Σ JEC.
T Pi Pm
(5)
Note the gradient will come from both p; and Pm- For this objective, there existstwo sets for each feature 2j: close neighbor set C₂ containing K K-nearest neighbors of (with distances cosine sim- ilarity), and background set B,, which contains the features that are not in C, (features potentially from different classes). To retrieve nearest neighbors for training, two memory banks are main- tained store all targer features along with their predictions similar to NRC (Yang al., 2021 a; 2023), which is efficient in both memory and computation, since only the features along with their predictions computed in each mini-batch are used update the memory bank.
MAPU. In the MAPU framework, the SHOT objective serves as the primary adaptation objective. To further enhance the adaptation process, an auxiliary objective based on masked reconstruction is used, defined follows:
(MAPU(g)) LsHor (g.) Lrecou (gt),
(50)
where the reconstruction loss, Lrecon is given by:
Crecongge = Ex, ([xe - ja (1x with &, = MASK(x)).
(51)
Here, MASK(-) represents a masking function applied to input w,, and j. the imputer module. Importantly, during the adaptation process, the imputer module je remains frozen. This auxiliary reconstruction objective guides the target encoder ht preserve meaningful feature representa- tions while adapting to the target domain, ensuring alignment with the source representations recon- structed by js-
27

---

Published as a conference paper at ICLR 2025
SSC
MFD
HHAR
WISDM
UCIHAR
SSC
HHAR"
SHOT
2,00.
62.6
5.3
0.5
5.0
5.2
9.U
2.3
9.0
5.5
NRC
83.0
AAD
15
8.3
1.2 102
5.0
5.2
5.0
100
5.0
MAPU
45
70.0
35
5.3
100
2.3
102
5.0
100
233
103
5.0
100
5.E
Sanples
Samples
Somples
Figure 10: Same experimental analysis as done in Figure 6 with an additional many-to-on SFDA
adaptation experiment on the SSC (SSC*) and the HHAR (HHAR*), marked with asterisk (*).
A.9 TRAINING DETAILS, EXPERIMENTAL SETUP, AND EXTENDED ANALYSIS
For all datasets, we utilize simple 3-layer ID-CNN backbone following (Ragab et al., 2023b), which has shown superior performance compared to more complex architectures (Donghao & Xue, 2024: Cheng et al., 2024). Figure 8 illustrates both the baseline (vanilla) architecture and the pro- posed Tucker factorized architecture obtained after reparametrized the weights. The filter size (K) for the input layers varies across datasets to account for differences in sequence lengths. Specifically, set the filter sizes 25 for SSC, 32 for MFD, 5 for HHAR, 5 for WISDM, and 5 for UCIHAR, following (Ragab et al., 2023a).
Source Pretraining. Following Liang et al. (2020; 2021), we pre-train deep neural network source model fs X, + C.. where fa = 9s has with ht as the backbone and gs as the classifier. The model is trained by minimizing the standard cross-entropy loss:
(s)) = -EI,...).Xx.x. Ts log (f((
(52)
Σ, explas) represents the k-th element of the softmax output. and q is the one-hot encoding of the label ye- To enhance the model's discriminability, we incorporate label smoothing as described in Müller et al. (2019). The loss function with label smoothing is:
where (a) = exp(ag)
K
cb (fs) -E Jex, xC,
Σ ai log du (ff(x))))
(53)
where of = (1 - a) + aa/K represents the smoothed label, with the smoothing parameter a set to 0.1.
Additionally, MAPU (Ragab et al., 2023b) introduces an auxiliary objective optimized alongside the cross-entropy loss, namely the imputation task loss. This auxiliary task is performed by an imputer network j., which takes the output features of the masked input from the backbone and maps them to the output features of the non-masked input. The imputer network minimizes the following loss:
Crcsun (js) = Re xc, (r.) - js (h. where f. = MASK(x.).
(54)
The backbone and classifier weights are optimized using the Adam optimizer (Kingma, 2014) with learning rate of le-3. We follow Ragah et al. (2023b) for setting all other hyperparameters.
28

---

Published as a conference paper at ICLR 2025
RF-22
RF--4
RF-8
1.50
1.50
source empircal
1.22
SSC
4.23
3.49
6008
epochs
epochs
epocchs
RF-4
sourna ampltcaliss
0.14
HHAR
6004
0100
eepccs
opochs
epacch
RF-2
RF-4
RF-8
MFD
0.7
2.00
03
epochs
opochs
epochs
before decomposition
after decompasition
Figure 11: Evolution of source empirical risk after backbone decomposition, showing the network trained on the source domain as it transitions from the freshly reparameterized state (epoch 0) to progressively recover its original source empirical risk.
Target Adaptation. For target adaptation, we apply the objective functions and training strategies specific each SFDA method (detailed in Appendix A.7 and A.8). The backbone weights are optimized to adapt to the target distribution, with Adam (Kingma, 2014) used as the optimizer to learn the target-adapted weights. We experiment with range of learning rates: (5e-4, le-4, 5e-5, le- 5, 5e-6, 1e-6, 5e-7, le-7} for each method (including the baseline) and report the best performance achieved.
typical SFDA evaluation setting, a source model pre-trained using labeled data from one domain (the source domain) and subsequently adapted to another domain (the target domain), where it evaluated using unlabeled target data. This evaluation strategy, known as One-to-one evaluation, involves single source domain for pretraining and a single target domain for adaptation. In this paper, introduced in (Ragab et al., 2023a;b). Each source-target pair is evaluated separately, and the average performance across all pairs is reported to assess the effectiveness of the employed SFDA methods.
One-to-One Analysis.
multiple source-target pairs derived from the datasets
Many-to-one Analysis. To provide more comprehensive and realistic evaluation, we also con- duct an experiment under the many-to-one setting. In this setting, data from all source domains in the source-target pairs are merged to form a single, larger source domain for pretraining. The pre- trained model is then adapted to target domains that were not part of the combined source domain. This many-to-one approach allows for much robust evaluation. Figure 9 visually describes the over- all evaluation setup. In Figure 10, we extend our analysis from Figure to include results for the Many-to-on setup of the SSC (Goldberger al., 2000) and HHAR (Stisen et al., 2015) datasets, results for the Many-to-on tony-to-one setting are marked with an asterisk (*).
Comparison with Parameter-Efficient Tuning Methods (Extended Analysis). In Table 4, we extend the results from Table 1 by including the outcomes for the WISDM and UCIHAR datasets. Consistent improvements are observed across various sample ratios, along with significant reduc- tion in both inference overhead and the number of parameters fine-tuned during adaptation. Ad- ditionally, we provide the F1 scores for each source-target pair across all the discussed datasets in Tables 5-9, for direct comparison with the number reported by Ragab et al. (2023b).
29

---

Published as a conference paper at ICLR 2025
A.10 COMBINING WITH PARAMETER- EFFICIENT METHODS (EXTENDED ANALYSIS)
Our extended analysis investigates a modified LtRlestyle PEFT method, where we freeze the ran- domly initialized factor and fine-tune only the zero initialized factor; freezing the zero-initialized factor is infeasible as it results in learning nothing due to its multiplicative zero effect. To evaluate its performance against the standard LoRA adaptation and our Source-Free Target (SFT) strategy at different RF values. However, we observe notable underperformance compared standard LoRA adaptation and our proposed method, as shown in Figures 13, 14, and 15. The frozen factor, which is randomly initialized, imposes arbitrary constraints severely restricting its ability to explore the target domain effectively. This leads to a suboptimal discovery of the ideal subspace for adaptation and results in significant underfitting.
the optimization of the fine-tuned factor,
A.11 COMBINING SFT WITH LORA-STYLE PEFT METHODS.
To evaluate the hybrid integration of SFT with Lstle PEFT methods, we conducted an in-depth analysis, applying range of SFDA perroaches-nantnee SHOT, NRC, AAD, and MAPU-acrose the SSC, HHAR, and MFD datasets in the one-to-one and Many-to-one settings (cf. Appendix A.9), as depicted in Figures 16-20. This evaluation explores the efficacy of combining SFT with LoRA- like frameworks under different domain adaptation scenarios, offering insights into the trade-offs between performance and efficiency across diverse set of tasks. Additionally, Table 10 presents a detailed comparison of parameter efficiency and inference overhead for all hybrid combinations tested.
Our results indicate that the parameter efficiency of SFT can be further improved by incorporat- ing LoRA-style PEFTss during the fine-tuning stage. However, we observed a slight degradation in predictive performance compared to the vanilla SFT. This decline can likely be attributed to the compounded effect of low-rank approximations: the initial rank reduction from source model decomposition, followed by an additional low-rank fine-tuning for adgattaiapt,,ionn excessively constrains the model's 's learning capacity. Nevertheless, combining SFT with LoRA-style adapta- tion still outperforms using LoRA alone in terms of predictive performance, while also achieving superior parameter efficiency, demonstrating the advantage of source model decomposition.
A.12 EXTENDED RELATED WORKS, OBSERVATIONS AND MOTIVATION
Unsupervised Domain Adaptation. Unsupervised Domain Adaptation (UDA) for time-series data addresses the fundamental challenge posed by distributional discrepancies between source and target domains, which can lead to significant performance degradation when models are deployed in new environments. Discrepancy-based methods (Cai et al., 2021; Liu & Xue, 2021; He et al., 2023) aim align feature representations by minimizing statistical divergences- the source and target
as Maxi-
mum Mean Discrepancy (MMD) or Kullback-Leibler divergence- feature distributions. Adversarial approaches (Wilson et al., 2020; 2023; Ragab et al., 2022; Jin et al., 2022; Ozyurt et al., 2023) employ adversarial training frameworks where a discriminator is trained to distinguish between source and target features, and the feature extractor is trained to pro- duce representations that the discriminator cannot differentiate, thereby promoting features. The comprehensive survey by Ragab et al. (2023a) provides an extensive overview of these domain adaptation techniques specific time-series data. However, conventional domain adapta- tion methods generally assume access to both source and target domain data during adaptation -n assumption that is often impractical in real-world scenarios. Source data may be inaccessible due to privacy concerns, confidentiality agreements, or intellectual property restrictions (Kundu et al., 2020; 2021). Moreover, transmitting and storing large-scale source datasets on aesource-constraimgg devices, such as IoT devices or mobile platforms, may be infeasible due to limited computational resources and storage capacity (Liu et al., 2023).
innvariat
Source-Free Domain Adaptation. To address the limitations associated with source data acces- sibility, Source-Free Domain Adaptation (SFDA) has emerged as a compelling alternative. SFDA operates under the assumption that only a pre-trained source model is available for adaptation to the target domain, thereby eliminating the need for access source data (Liang et al., 2020; 2021; Li
30

---

Published as a conference paper at ICLR 2025
et al., 2020; Yang et al., 2021a; 2022; 2023; Kim et al., 2021; Xia et al., 2021; Ding et al., 2022; Litrico et al., 2023; Tang et al., 2024b;a). This paradigm has garnered significant attention in com- puter vision tasks, including image classification (Kundu et al., 2020; Liang et al., 2020), semantic segmentation (Liu et al., 2021; Bateson al., 2022), and object detection (Saltori et al., 2020; Xiong et al. 2022). In SFDA, prominent strategies involve leveraging unsupervised clustering techniques to refine feature representations. These methods promote the discriminability and diversity of the feature space through various objectives, such as information maximization (Liang et al., 2020), which encourages the model to produce confident and diverse predictions; neighborhood clustering (Yang et al., 202 which clusters target samples based on feature similarity; and contrastive learn- ing objectives (Yang et al., 2022), which enhance representation learning by contrasting positive and negative sample pairs. Recent advancements incorporate auxiliary self-supervvsed tasks, such as masking and imputation, to improve model adaptation, as demonstrated by Ragab et al. (2023b) and Gong et al. (2024). These approaches build on earlier works (Liang et al., 2021; Kundu al., 2022a) that integrate auxiliary objectives to capture intrinsic data structures and enhance represen- tation learning. Furthermore, the utilization of pre-trained foundation models (Radford et al., 2021; Jia et al., 2021) has been explored to guide adaptation by leveraging the extensive knowledge em- bedded in models trained large-scale datasets (Tang et al., 2024b). However, the applicability of such models to specialized domains like healthcare and agriculture remains limited due to do- main mismatch. Foundation models may fail to capture the specific characteristics inherent in niche datasets, making the adaptation process inefficient, particularly when a substantial amount of infer- ence is required through the large foundation model to represent target samples. While foundational models have recently been developed for time-series data (Gao et al., 2024; Ye et al., 2024), their applicability to SFDA is yet to be verified. Notably, the exploration of SFDA in time-series con- texts, especially concerning parameter efficiency and sample efficiency, remains largely unexplored (Ragab et al., 2023a; Liu et al., 2024; Gong et al., 2024; 2025; Furqon et al., 2025). In this work, we aim to address these limitations by proposing a strategy to conduct SFDA efficiently in the context of time-series data utilizing the established framework by Ragab al. (2023b). We demonstrate that our approach achieves both parameter efficiency and sample efficiency, providing a unified solution for these challenges.
Parameter Redundancy and Low-Rank Subspaces. Neural network pruning has demonstrated that substantial reductions in parameters, often exceeding 90%, can be achieved with minimal ac- curacy loss, revealing significant redundancy in trained deep networks (LeCun et al., 1989; Hassibi & Stork, 1992; Li et al., 2017; Frankle & Carbin, 2018; Sharma et al., 2024). Structured pruning methods, such as those proposed by Molchanov et al. (2017) and Hoefler et al. (2021), have opti- mized these reductions maintain or even improve inference speeds. Low-rank models have played a crucial role in pruning, with techniques such as Singular Value Decomposition (SVD) (Eckart & Young, 1936) applied to fully connected layers (Denil et al., 2013) and tensor-train decomposition used to compress neural networks (Novikov et al., 2015). Methods developed by Jaderberg et al. (2014), Denton et al. (2014), Tai et al. (2016), and Lebedev et al. (2015) accelerate Convolutional Neural Networks (CNNs) through low-rank regularization. Decomposition models such as CAN- DECOMP/PARAFAC (CP) (Carroll & Chang, 1970) and Tucker decomposition (Tucker, 1966) have effectively reduced the computational complexity of CNNs (Lebedev et al., 2015; Kim al., 20. Recent works have extended these techniques to recurrent and transformer layers (Ye et al., 2018; Ma et al., 2019), broadening their applicability. Mcanwhile, Chen et al. (2024b) propose filter subspace decomposition for CNN weights along the channel and spatial dimensions. This filter subspace de- composition has shown effecttivnnes in continual learning (Miao et al., 2022; Chen et al., 2024a), video representation learning (Miao et al., 2021), graph learning (Cheng et al., 2021), and gener- ative tasks (Wang et al., 2021). Moreover, recent advances in structured parameterization, such as Monarch (Dao et al., 2022) and block tensor train (BTT) (Qiu et al., 2024), have further explored parameter-efffcient representations for dense layers. Monarch uses structured matrices to achieve hardware efficiency and expressiveness in NLP transformer layers, while BTT leverages block ten- sor train decomposition for computational efficiency in similar settings. However, these methods primarily target dense layers and remain matrix-based lacking native support for higher-order ten- sors, such as convolution filters. In contrast, our work focuses on source-free domain adaptation (SFDA). Tucker decomposition, central to our approach, natively accommodates higher-ordhe ten- sors, such as those in convolutional layers, by decoupling the core tensor and factor matrices. This decoupling enables fine-grained control over adaptation by selectively tuning the core tensor while freezing the mode factors. For example, demonstrated in Figure 4, adapting only the core tensor
31

---

Published as a conference paper at ICLR 2025
allows the model to effectively adjust to the target domain (negating source influences) while main- taining parameter efficiency. This fine-tuned control over adaptation contrasts with the uniform rank structures imposed by CP decomposition and the block-wise parameterization of BTT, where it is not entirely clear what should be fine-tuned the time of target adaptation.
Compared to Monarch and BTT, our approach offers several unique advantages:
Parameter Efficiency:
By adapting only the tensor core while freezing the mode factors, our method further minimizes the tunable parameters, achieving computational efficiency without sacrificing the quality of adaptation. This design is particularly beneficial in low- resource settings (Ma et al., 2024) often encountered in time-series.
Beyond computational efficiency, our method excels in sample effi- ciency, critical metric in data-scarce scenarios that is central to real-world SFDA applica- tions. This property of Monarch and BTT explicitly needs further investigation.
Sample Efficiency:
Flexibility Across Modes: Tucker decomposition allows independent rank adjustments for each mode (e.g., time steps vs. channels), enabling adaptive modeling across diverse dimensions.
Interpretability: The factors resulting from Tucker decomposition can provide insights into domain-spectt characteristics, such as temporal correlations or channel-specific im- portance. This interpretability could be advantageous for analyzing the adaptation dynam- ics in SFDA (Calvi et al., 2019; Halatsis et al., 2024).
Our Motivation and Link to Time-Series. this work, we aim to address the simultaneous challenges of parameter efficiency and sample efficiency in Source-Free Domain Adaptation (SFDA) for time-series data, leveraging inherent properties of this data type to design generalizable yet powerful approach.
A key insight that guided approach is the observation of significant parameter redundancy along the channel dimension of source models trained on time-series data. Figure 2A demonstrates this redundancy, highlighting the inter-channel dependency inherent in multivariate time-series features as they propagate through deep networks. Prior work (Donghao & Xue, 2024; Lai et al., 2017) has shown that modeling dependencies among variables through convolutions along the variable dimen- sion effectively captures these cross-variable relationships. Inspired by this, our method decomposes the source model along the channel dimension, as described in Equation 3, enabling us to focus on the most significant channels or variables while maintaining model performance.
By leveraging the inter-variable dependencies inherent in time-serics data, our decomposition ap- proach achieves significant parameter efficiency. Instead of operating on the full channel space, we focus on the principal channels, which correspond to the most critical variables in the parameter space. This focus allows for a compact representation of the model, reducing the number of param- eters without sacrificing performance as we observe empirically. The empirical results in Section highlight the effectiveness of this approach across a variety of settings and datasets, demonstrating its versatility.
To ensure the generalizability of our method, we refrained from introducing specific inductive biases, allowing our approach to be applicable across wide range of SFDA methods. We validated our method with both prominent generalized SFDA techniques (e.g., SHOT, NRC, AAD) and state- of-the-art time-series-speciff experiments encompass the five datasets in the AdaTime benchmark (Ragab al., 2023a) where prior works (Ragab et al., 2023b) typically focus on only firee-along with ten source-target pairs- where prior works experiment with five source-target pairs-aineei innsssdinnnggggg many-to-one scenarios where a source model trained on multiple domains is adapted to new targets.
SFDA method (MAPU) introduced by (Ragab et al., 2023b). Our
A.13 LIMITATIONS
The proposed method has limitation in that it is not rank-ada meaning it does not adjust the rank based on the intrinsic low-rank structure of the data or model layers. Each dataset and mode (i.e.. training vs. inference) has its own optimal low rank, and this ideal rank can also vary across different layers of the model (Sharma et al., 2024). When the rank is not appropriately chosen, the model risks either under-fitting-ir the rank is too low and unable to capture the necessary complexity-on over-fitting--t the rank is too high and introduces unnecessary complexity, leading
32

---

Published as a conference paper at ICLR 2025
to poor generalization. Since the current method uses fixed rank for the entire source model, it may result in suboptimal performance, with inefficiencies in both parameter usage and generalization. Addressing this limitation by incorporating rank-adaptiveness could allow the model dynamically adjust its rank based on the properties of the data and model layers, thereby reducing the risk of under- under-fitting and over-fitting and improving overall performance.
Moreover, another limitation of this work is the lack of analysis the interpretability of the factor matrices obtained during decomposition. These factors are presumed to capture representations that generalize across different domains (i.e., source target). However, effort has been made to explicitly analyze or interpret what these factors represent, leaving open the question of whether or how they effectively capture cross-dataset generalization. This important aspect of understanding the model's behavior is left for future work and could provide valuable insights into how the method works across various domains.
A. FUTURE WORK
While our method leverages significant parameter redundancy along the channel (or variable) di- mension in models trained on time-series data (ef. Figure 2), this redundancy highlights substantial inter-channel dependencies time-series data propagate through deep networks. To this end, propose our method that involves decomposing the backbone. Importantly, we acknowledge that the core principles of our approach are not inherently limited to this domain, and we plan to explore whether such properties are also exhibited in other modalities.
In future work, we aim to extend our method to other areas, such as computer vision, adapting it to handle the distinct characteristics and dependencies of visual data. This exploration will assess the universality of our method, enhance its broader applicability, and contribute to general adaptation strategies across various domains beyond time-series data.
Furthermore, promising direction involves integrating our method with time-series foundation models (Gao et al., 2024; Ye et al., 2024), which have shown significant potential in capturing generalized representations across diverse domains. By combining their ability to leverage broad pretraining with our fine-tuning efficiency approach, we aim to explore how such synergies can address domain-specifie adaptation challenges while maintaining computational feasibility. Inves- tigating the application of time-series foundation models in source- frece domain adaptation would also provide valuable insights into the role of generalized features in facilitating efficient adaptation across diverse time-series scenarios.
33

---

Published as a conference paper at ICLR 2025
SSC
SSC
LoFA-E
SMST
ADD
14
HHAR
HHAR LoRA-S
18860
LoRAN
LoRA-4
LOKA
AVD
MFD
LoRV2
LOSA-4
LODV-
LOKG
SHOT
AAD
Dataset
Method ->
Baseline
(RF2)
(RI4)
(RF8)
LaRA-16
LoKrA
SSCISC*
MAC's (M)
12.92
5.54
1.99
0.80
12.92
12.92
12.92
12.92
12.92
Params. (K).)
83.17
20.88
5.32
1.38
1.94
15.63
1.84
HHAR/HHAR*
MAC's (M)
9.04
3.79
1.34
0.53
9.04
9.04
9.04
9.04
9.04
à Params, (K) |
198.21
49.63
12,53
3.19
1.11
2.4
5.46
13.62
3.33
MFD
# Params. (K) }
199.3
50.18
12.8
3.33
1.22
2.82
7.18
20.50
3.46
Figure 12: Comparison LoRA (Hu et al., 2021) and LoKrA (Edalati et al., 2022; Yeh al., 2024)
(in Purple) against bascline methods (SHOT, NRC, AAD, MAPU) and our proposed approach (SFT) on the SSC, HHAR and MFD datasets in the One-to-one setting and in the Manu-to-one setting for SSC (SSC*) ) and HHAR (HHAR*), evaluated across varying target sample ratios used during adaptation. The table at the bottom shows the target model's inference overhead after adaptation in terms of MACs and the number of parameters finetuned at the time of adaptation. Extension of the analysis presented in Figure 7.
34

---

Published as a conference paper at ICLR 2025
Table 4: Performance and efficiency comparison on SSC, MFD, HHAR, WISDM, and UCIHAR datasets across SFDA methods, reported as average FI score (%) at target sample ratios (0.5%, 5%, 100%), inference MACs (M), and fine-tuned parameters (K). Highlighted rows show results for SFT, where only the core tensor is fine-tuned at different RF values. Green numbers represent average percentage improvement, while Red numbers indicate reduction in MACs and fine-tuned parameters.
Methosk:
0.5%
5%
Aseruge
MACS (M)
(K)
SSC
SHEFT di king et at, (0100)
61.0%
11444
12.95
N5.17
SHOT SPT
62.71 0055
62.16
(2.16)2)168)
199
(54.00%)
63.05 0.30
63.46
0.8
67.45 10.50
(3.12009955
5.54 (SP 229)
30.85 (74.39%)
NRC (Yasg et 2021a)
59.92 1.11
63:56 1.35
65.23 0.559
62,90
12:92
$5.17
60165
63:60 1.43
65.05
(99.8199
60.00 0.58
66.83
(2.000))
(57.129))
(74.89%)
AAD (Ying en 2003)
59.30 = 1.50
63.31 1.53
63.71 2.66
62.10
12.98
15.17
00002 1:60
1d 0-055
60 00:00+111 1.17
1993
1 COR 3415)
AAD
61.92
66.59
67.14:
(5.22850088
199
(54
5.32
0.59
(74.59%))
MAPU (Ragab at. 2023b)
60.35 2:15
62.48 1.57
06.73 0
63,19
12.92
$7.17
63.52 1.24
64.77 0.22
66.09
66.09-10.10
1..19
64.79 (2.55R)
0.80
193
1.36 COR 3496)
MAPU + SET
01:21 00
20:20: 1:25
67
(2778)
1.99 (S4
9.32 0O15)
02.25
1.00
4003
(57)129))
MFD
SHOT (Liang et al. 2020)
90.74 1.22
91.53 1.44
91.03 = 1.37
91ADD
$8.15
199.3
1:77
91-16
91.8 A393
(9939993
1.19 ST6E)
SHOT SPT
02.00 0.779
0:15
2064
(84.879)
(74.82%)
NRC (Ror etall 202160
W9.05 3.84
91.74 76 1:31
52:30 1:33
W1.9
56.15
199.3
92.14 1.77
00:36 0.51
03
92.88 (2,66))
3.52
(9335583
3.33 (98
NRC SET
92.91
03.25
92.97
93.04
(2.2583
(54.979))
AAD (Yan ot 2020)
80.03 3.60
a1.06:22.31
50.65 2.44
90.25
58.15
199.3
01.15 = 1:78
01.35 1111---000 0.35
3.52
3.35 (ERS)
AAD + SET
91.00 1:17
91.0% 6-011
97.07
+0:41
830
(54.579)
12.80 (9)) 750%)
92.07 0.79
93.45 0.53
94.41
0.19
24.56
(57.6196)
(74.8256)
MAPU (Ragab 20030)
05.32
50.49
1.50
91.71
$7,84
58.18
154.3
91.07 = 1.76
3 1.14
92 11: 1.56
3.52 (00.8593)
3:33 (98.339))
MAPU + SPT
01.01 0.07
92.10 1,02
91 18:11:111
91.3 (U00PI)
8.80 (54.9793)
12,80 (00 58%5)
91.87 2.88
91.84
1.44
91.841 + 188
91 28
24.56
CST (11669
(74.83)))
SHOT (Liang et al. 2020)
22.08 =: 1.20
70.80 1.70
50 50.12:20.2
77.35
9:04
198.21
73,65 2.39
79307 & 1.38
$1.23 1.47
0.53
(94LL499)
3.19 (98339533
SHIT SPT
79:06
1045
1.54
(85 1893
(9) ARI)
75.14 1.85
77.70
1.37
$2.920033
78.59 (1.638)
(58.088))
(74.9665)
NRC ENing stall 2001))
23.18 00.0
75.52 1.55
10:80-1 0.16
75.25
9.04
196, 21
21:41 00000
T6.15
4 0.09
50.00
74 0000
1.19 COR NOSS)
NRC + SET
76.60
2 1.97
80.27
20.55
76.55
1.34
(B5.LBR))
12.55 (83,685)
73.34 0.14
75.5%
19.45 1:51
3.79
49.6% (74
YYau
20.18 2.700
76.55
62.25
99,66
9:04
195.21
57.83 0.00
78.57
0.74
$2.92 8 1866
73.14
(22.999
0.53 (94LLAR))
3,19 (08L30
AAD - SET
54.63 1.04
78.57
0.744
65.15 6.88%
73.05
cuu 79965
1.54
(8S 1998)
12.4% (UT) ARERO
40:40 2.20
79.11 6:37
53.75 0.34
7000 49663
3.20
49.01 (74
MAPU
21.18
9:04
198.21
25.79 2:33
75.56 2:37
10:16-5 2.58
J.MM 1915)
MAPU SPT
2..82
50.04 2.04:20044 042
$0.24 1.22
78.47 (22666)
1.34 (S5.L89)
12.53 (00.68%))
76.06 5
78.95
2.04
$0.69
78.57 (2AUS)
3.79
(58L08R))
40.63 (74.9095)
WISDM
SHOTT al, 2010))
$7.90 2:00
90.50
1.05
60:40 + 1:51
59.5
9:04
196.31
60.57 0.79
61.25
a 1.91
62.08
2000
0.53 (N4L148)
3,19 (98,395))
SHOTT - SPT
58.06 3.58
60.39
1:56
61
13
0:33
W0,00 1I 2000
5.54
183-1893)
12.55 (00) A80
60.01 0.73
60.05
62.04
125
69 (40088))
3.79
(58,084))
49 (74-901))
NRC at, 20211)
53.58 1.24
0:85
55.57
9:04
198.21
57.08 0.94
57.11
1:94
0:5 (9AL199)
NRC SPT
55.00 7.00
56.82
1.57
1.6
1.94
(65
12.5% (V7) ARTIO
2.
0.95
59.21 = L.18
56.88 (2.96))
3.79
(58,08R)
40.63
(74.908)
AAD 2001)
56.65 00.50
$7.59
$7.69
57.31
(94L44))
AAD + SFT
55,90 2.67
56.71
1.47
58,57
2.3
(4.8668
1.34 (85.1896)
12.53 (03,680)
55.83 2.22
$4.85
1:35
59.10
4.7.5
46.75 (1393)
5.0 (S8
40.05 (995
MAPU (Rupab al,, 2003b)
56.33
42.42.3
108.21
3.73
55
6L08 (4L27R)
(84L449)
MAPU + SPT
02.00 4.00
00 1.37
66
360
01332611333
1:54 (SS
(93
59.36 5.74
63.25
1.01
62.45
a 2.66
6L69
3.79 (59,088))
49.63 (74.96%)
UCINAR
Aang sl. 20700)
89.06 1.17
1:36
80.81
9.28
300L15
86-52 0.44
08.05
0.69
90.00
1 1999
9:41 mOATR
0.57
(93.36933
3.4% 2096)
SHOT SPT
90.02 0.56
39.41
0.56
92.96 = 1.02
1.41 (84.119)
13.01 (X2 50S)
90061 0.7.6
90:38 1.081..0000 0.78
01.65-100 1.75
3.92 (50.7693)
50.59 (74.726)
NRC CYang et 202200
45.13
200L13
0.00
91.86
85.36
NRC SPT
85.21 1:35
K6 14 1.84
05 1.01
84
ST
641 93643
1-41 (543119)
13.01 (91 50%
84.20 1.21
18 3,00
01.51 1.65
ST. N0
640 00945
5.93
(89.9693)
50.99 194.91947220)
AAD (Yiu et al,, 2023)
57.27 6.775
S6.21
0.44
200.13
AAD
96.73
60
(54.819)
(93.505)
85.51 = 1.09
85.53 63.33&2344 2.62
0.16
87.55 1299
3.92 (57.7696)
50.59 (74.2%))
MAPU (Ragah et al, 2003b)
N1.05 3.10
M1 1.46
50196 + 1.62
89.35
9:24
200.15
89.57 0.11
89.10 0:98
91.93
(2.12
(90 (03.8699)
MAPU
80 94 1.74
96.05
91 0.49
90:78 (1.311
3.92
(57.77693
50.59 (24-229)
35

---

Published as a conference paper at ICLR 2025
Table 5: Fl scores for each cource-target pair, with domain IDs established by Ragab et al. (2023a), on the SSC dataset across different SFDA methods.
Nakel STOT
Aecrgg
(M).
6000
1531
4:344
4008
2:36
106
525
2003 2.35
07.55
0.90
1.6
SHOT -5T
5401
0481 6.20
15.1
5.99
1.29
126
21A25
31.95
NRC
NRC
00169
8:3
08.70
62.0
(2.23
71.2
14.5
(191
532
1057
63.55.12
700 31
1355-6816
75.30
23.81
AAD
14:0440
5071 2/1
7151
66.48 1601
A
13.33 6337
03.7 2.00
13.03
$3.7
AMD+3F1 3F1
10.12 12:0
1.11
1333 2112
7547 1111
139
330
5301 886
0115
a337
23194333
15.51 4037
7401 331
60.000019
0.80
136
539/1033
71.88
54.01
2034
6459 1350
3435 120
1340 100
11.00 2:34
0008 1:55
15.37 1061
7562 133
01.30 0.82
3.3
33.85
Table 6: Fl scores for each source-targe pair, with domain IDs as established by Ragab et al. (2023a), on the MFD dataset across different SFDA methods.
WAR M65
12.9
08:15
24.06
240
NBC-
8.9
12.00
180010-0
10:00030
9145-205
10:200.6
34
MANU
1:514-13
B.
1000
1818.187
34.06
Table 7: Fl scores for each source-target pair, with domain IDs as established by Ragab et al. (2023a), on the HHAR dataset across different SFDA methods.
Make SHOP
62
2
3-455
4
1-
5-45
Arange: (901
[MAES (MI | |#fmmms IK]
18
11
$1.0 2:10
17.22-000
5150-044
134
12
-B0.1
SHOT SF7
18.57 C:48
64060 339
03 005
67.0-6.1
618-6611
- 13.54
8:05
1.36
12:33
13
00 0055
28 0:00
07.00 1:20
613)-635
0.5
119
14.99 0.11
6580 4477
11.00
11.01
34312300
1131-021
1634-00
006
131
5.5
4000
AAD
6.33
05 RA1
66 6M1
63.
1.
6151-0M
1:34
-2.31
7760 1010
7410 1943
2026
71.21 2.331
15:55144 648
1135-06)
113)-034
1632-021
9000 2011
13151095
1.36
11:3
7400 1824 1824
19.1 1031
2028
10 11131
21342433
1959-060
18:18:1322
81101130
0.30
119
41-111
1735-13
4435-D1
56 55
(961
Table 8: Fl scores for each source-target pair, with domain IDs as established by Ragab et al. (2023a), the WISDM dataset across different SFDA methods.
Miteed SHOT
15
Arcrue (S)T
MING (MLL
2.34
31.35 1135
196
13.31
SHOT SFT
69-0046777 75
72.00 2005
68 18:70003 32
1.3
13:5)
1
11 30
1501-000
1679--11 10.34
7:3
13.20 k01
-8.08-11 106
68 3.55
0.5
1401 11M
1.50
RAD
500
12:4
3039 2-13
4133--633
28.35.1.0
3.36
106.11
AAD 3U7
54.77 A43
0542 5575
48.77
47.22
65.3 4430
44.33 2.5
26.5742.99
50.
13.35
4121
3.79
460
19631
5300
3508
4331
62.542277
3.9
MAPU-SFT
5FT
5140 19:11
2125
2432
1.5
Table 9: Fl scores for each source-target pair, with domain IDs as established by Ragab et al. (2023a), on the UCIHAR dataset across different SFDA methods.
Metue
[
Aeee MAC' (Mi]
Foree (K0
100.00 2.53
1003 1440
D00LE DAC
JOLA JAMA
$400 3.30
07.32 120
3.5
20113
SHOT 12T
12.48
$5.00
110.00 9.03
1.0
180R
122
129
1000-20000
3.52
SL36
16.15
4.3
01.83
201.3
072900 2.76
110.90 3.33
10.27-20
19110 1110
18100 11122
1901 1110
1010
54 14:2521:11
1.41
(300
AAB
1430
102.90 3:0
07.22 4000
19 133
1040 80AS
1000 0/0
14.33 0400
11.78
01 555000000
0.7
20
01
1110
2016
1110
142
MAFU
01:0111.3)
1318
2801-108 201
3006 11055
2000 2010
10180-11100
00
10
9%
2013
MARTI SFT
1447 0.24
1910 +100
0921+006
-15
1:11
1.000.10
33.5 2355
L00
1000
$7/8
SL9
36

---

Published as a conference paper at ICLR 2025
SSC
SHOT LSRK-6
NRC
Lo0k-4
Lof4--4
LORA-10
LaKVA
LOWAN22
LOGA
Lokilett
LAW-A
35-3
19-4
BFG
MAPU
LocA--
LORAS
1152
Comparison of the vanilla LoRA-style PEFT method, fine-tuning all components (in Purple) and fine-tuning while freezing the randomly initialized factors (in Magenta), against baseline methods (SHOT, NRC, AAD, MAPU) and our proposed approach (SFT) at different RF values on the SSC dataset.
Figure 13:
HHAR
Lo0A-E
Lit44-4
L-RA-8
LURk-115
25-2
RP-4
AXO
MAPU Luti-
Lolk-2
LuGiA
LUGA-15
Comparison of the vanilla LoRA-style PEFT method, fine-tuning all components (in Purple) and fine-tuning while freezing the randomly initialized factors (in Magenta), against baseline methods (SHOT, NRC, AAD, MAPU) and our proposed approach (SFT) different RF values on the HHAR dataset.
Figure 14:
37

---

Published as a conference paper at ICLR 2025
MFD
NBC
27-4
MAPU LoF4-3
LoSA-2
LoGA-E
L:RA-D
L2RA-22
Lotoo
LoRA-
LaGA-15
LaKo
DF-4
Figure 15: Comparison of the vanilla LoRA-style PEFT method, fine-tuning all components (in Purple) and fine-tuning while freezing the randomly initialized factors (in Magenta), against baseline methods (SHOT, NRC, AAD, MAPU) and our proposed approach (SFT) at different RF values on the MFD dataset.
MFD
SHO
NRC
Lo0k-6
LoSA- 15
LOWA
Latoo
RP
AAD L
MAPU
LOOA-g
LoBk-
LaRk
LoRA-B
LSRA-22
LOKUA
Figure 16: Performance comparison of SFT with LoRA-style PEFT methods on the MFD dataset at different sample ratio of target samples used for finetuning, on SHOT. NRC, AAD, and MAPU. The results highlight the trade-offs between parameter efficiency and predictive performance across different adaptation methods.
38

---

Published as a conference paper at ICLR 2025
HHAR
SHOT LoftA- df
NRC LeRA-a
L
Lufik-4
LoftA-
LORN-4
LoRt-nt
LOKO
IP-2
DF- TENIEEN
IF-
AND LeRh-n
MARU
LoRNo
LORN
Loki
LORA
LORA
LeRk-16
04
Figure 17: Performance comparison of SFT with -tyle PEFT methods on the HHAR dataset at different sample ratio of target samples used for finetuning, on SHOT, NRC, AAD, and MAPU. The results highlight the trade-offs between parameter efficiency and predictive performance across different adaptation methods.
SSC
MARU
LoGAu
LoBA-g
LoRA-B
LoRA-at
Laki
35-4
0F-4
Figure 18: Performance comparison of SFT with LtRe- PEFT methods on the SSC dataset at different sample ratio of target samples used for finetuning, on SHOT, NRC, AAD, and MAPU. The results highlight the trade-off's between parameter efficiency and predictive performance across different adaptation methods.
39

---

Published as a conference paper at ICLR 2025
HHAR"
SHOT LoRt-
NRC LuttA
LaRk-a
Lohh-g
Loble1t
LeNia
LU4IA-2
Lotti
LokiA
AAD
MASU
LORA-4
LONA
RF-
OF-
UP4
10-00
Figure 19: Performance comparison of SFT with LoRA-style PEFT methods on the HHAR dataset, for the Many-to-one setting (noted with an asterisk (*)). at different sample ratio of target samples used for finetuning, on SHOT, NRC, AAD, and MAPU. The results highlight the trade-offs between parameter efficiency and predictive performance across different adaptation methods.
SSC
NRC
LORA-D
Lotas
Figure 20: Performance comparison of SFT with LoRA-style PEFT methods the SSC dataset, for the Many-to-one setting (noted with an asterisk (*)), at different sample ratio of target samples used for finetuning, on SHOT, NRC, AAD, and MAPU. The results highlight the trade-offs between parameter efficiency and predictive performance across different adaptation methods.
40

---

Published as a conference paper at ICLR 2025
Table 10: Comparison of inference overhead (measured in MACs) and the number of parameters fine-tuned during adaptation (# Params.) for various combinations of SFT and LoRA-style PEFT methods. The table highlights the trade-offs between parameter efficiency and computational cost across different hybrid configurations.
Method
MFD
HHAR/HHAR
SSC/SSC*
MACs (M)
# Params. (K)
MACs (M) 1
Params, (K)
MACs (M) 4
Params. (K).
Baseline
198.21
12.92
SFT (RF8)
3.52
3.33
0.53
3.19
0.80
1.38
SFT (RF8) LoKrA
3,52
0.41
0.53
0.34
0.80
0.26
(RES)
LoRA-2
3.52
0.80
SFT (RF8)
LcRA-4
3,52
1.03
0.53
0.60
0,80
0.82
SFT (RFS) LoRA-8
3.52
3,59
0.53
1.88
0.80
2.95
16
3.52
13.33
0.80
SFT (RF4) SET (RF4) LoKrA
8.80
12.80
1.34
12.53
1.99
5.32
8.80
0.93
1.34
0.86
1.99
0,51
SFT (RF4) LoRA-2
8,80
0.45
1.34
0.35
1.99
0.33
SET (RF4) LoRA-4
8.80
1.28
1.34
0.86
1.99
0.98
SFT (RF4)+ LoRA-8
8.80
4.10
2.39
1.99
3.27
SFT (RF2) LoRA-16
8,80
14.35
7.47
1.99
11.79
SFT (RF2)
24.66
50.18
3.79
49.63
5.54
20.88
SET (RF2) LoKrA
24.66
1.38
3.79
1.24
5.54
0.92
SET (RF2) LoRA-2
24.66
0.71
3.79
0.60
5.54
0,49
SFT (RF2)+ LoRA-4
24.66
1.80
3.79
1.37
5.54
1,30
SET (RF2) LoRA-8
24.66
3.79
3,42
5.54
3,91
SFT (RF2) LoRA-16
24.66 66
16.40
3.79
9.52
5.54
13.07
LoKrA
58.18
3.46
9.04
3.33
12.92
3.46
LoRA-2
12.92
LoRA-4
58.18
2.82
9.04
2.40
12.92
2.82
LoRA-8
58.18
7.18
9.04
5.46
12.92
7.18
LoRA-16
58.18
20.5
13.62
12.92
20.50
Table 11: Ablation studies on finetuning different parameter subspaces during target-side adaptation for SHOT. The asterisk (*) top of the dataset names denote the Many-to-one setting (ef, Appendix A.9 for reference).
13:00
15.21
Pule Maliar 1il yil Seau
6.00 km
630
631
$135
6.59
301
100
641
113
15:0
138
1.3
Pine Vaturs 3016 (7
Table 12: Ablation studies on finetuning different parameter subspaces during target-side adaptation for NRC. The asterisk (*) top of the dataset names denote the Many-to-one setting (cf. Appendix A.9 for reference).
Platroin
M822 444
1250
12.0
615
K00
6.5
K00
Cle (7)
M6-7-
5.00
1.14
19 84-44-33 3.00
6:30
3.1
Fite
94230 344
150
110
11:09
170
1.3
Pole Maaioan (1ae
130
301
3119
379
179
1601
Table 13: Ablation studies on finetuning different parameter subspaces during target-side adaptation for AAD. The asterisk (*) on top of the dataset names denote the Many-to-one setting (ef. Appendix A.9 for reference).
11/0me/211 2011
Fltmes KG.
MADI
Edte Bucdess
MS
84804304 384
1346
645
13:05
645
K0n
6.4
M-12
4.0
6D6
Cos
120
130
118
630
111
05 1148111
6.00
111
Pore Vatur 7V% Smar:11
140
114
11.
3.40
170
35.00
179
2101
41

---

Published as a conference paper at ICLR 2025
Table 14: Ablation Studies on finetuning different parameter subspaces during target-side adapta- tion on MAPU. The asterisk (*) on top of the dataset names denote the Many-to-one setting (cf. Appendix A.9 for reference).
#Fram 8X1
#ftres
MICHV9
#hher
5405 1X0
1140
0117
12204123
12:5
$112
3021
X:531
35
Pale Malica W01 Da
0..8
411
M128
6.5
0135
650
6.2
3001
Fice Cle
0330
14.2
1231
43122331
135
15114430 3m
1310
M0004241
179
139
208
42


Reviews:
[{"review_id": "YQxtdsC95B", "reviewer": "Reviewer_56c7", "summary": "This work presented a Tucker-style tensor factorization as a low-rank decomposition of source-pretrained models for time-series data. This method enables a selective fine-tuning of the core tensor on the target side, effectively improving the efficiency.", "questions": "Lines 248-252, why is fine-tuning for 2-3 epochs? Is this number closely related to the dataset and model scale? I am not very familiar with this field and hope that relatively rigorous evidence can be given here in the text.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I think the topic of the article is cutting-edge. The article as a whole uses rich methods such as PAC-Bayes to expound the effectiveness of the proposed method. The supplementary materials are also very detailed in explaining the article, such as the derivation of the formal bound parameter distance. The experiments are sufficient, and the effects shown in the experiments are good. The elaboration and analysis of limitations are also very sufficient.", "weaknesses": "1. In my opinion, in the Related Work section, what is shown is more accurately ablations rather than the so-called observations in the title. Compared to this experimental result, it is more necessary to simply explain why Tucker-style decomposition is used instead of other decomposition methods you have exemplified.\n2. I think the insights in lines 201-204 need a more complete explanation. From Equation 1, I can only understand that the linear operation in deep learning is the so-called sequence of linear suboperations in the forward process. However, I cannot directly confirm whether such a description is also accurate for the backward process. If there are relevant formula derivations or proofs, references can be provided. If not, considering the space limitation, it is best to prove it in the Appendix. Because this claim is the premise of Equations 4, 5, and 6 later, which is necessary and important for understanding this method.\n3. A small mistake. In the sentence after Eq3, on line 221, after “where”, it should be an explanation of the symbol V in Equation 3 instead of U.\n4. On line 238, “Building on these insights, we decompose the pre-trained source model weights using Tucker decomposition”. Although the expression here is rigorous. Logically, the former is a necessary condition for the latter. From the beginning of the article until now, there is no sufficient reason given for why Tucker decomposition is used instead of other decomposition methods. Even if it is mentioned earlier, it is better to emphasize it again here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bseYY9dJCu", "reviewer": "Reviewer_bZkd", "summary": "This work introduces a method for source-free domain adaptation (SFDA) in the domain of time-series data. The task poses a challenge due to two main reasons: 1) the model has access only to unlabeled data and the source-pretrained model but not the source data, and 2) finetuning with small amounts of unlabeled data in this domain often results is overfitting. This papers explore the method of applying Tucker-style factorization, by breaking the parameters into two parts: core-tensors and factor matrices. By doing so and selectively finetune only on the core-tensor matrix, the model shows little signs of overfitting and improved generalization. Empirically, experiments from changing the rank fo the decomposed tensor to comparisons with current popular adaptor methods show the proposed method is effective at reducing overfitting and improving generalization with fewer parameters, achieving the best of both worlds. Finally, the authors provided a PAC-Bayes generalization bound, showing that the test loss is upper bounded by the empirical loss plus the regularization effect from the matrices learned.", "questions": "1. Can the authors comment on the time for inference before and after factorization? Does the smaller number of parameters reduce the time required for inference? Or does the increased number of matrix factorization increase the time required?", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The problem presented at this paper is of great significance, with considerations about the challenges of finetuning the full model and application of on-device settings. The authors have done a great job in both the introduction and method motivating the problem and stating the importance and relevance of this research problem. The method is clear, with definitions and equations clearly defined and laid-out. The method proposed itself is sound also. The authors also try to draw connections with related works to Lottery Ticket Hypothesis, as an attempt to justify why their method works well and is able to achieve the best of both worlds; This connection is a nice touch and welcomed. Experiments-wise, the paper has done a thorough job in demonstrating the effectiveness of their method, with sufficient number of datasets and models, and comparisons with state-of-the-art methods that show clear improvements.", "weaknesses": "A weakness of this work can be characterized as a missing link between the entire method and the time-series aspect of the problem. Specifically, it’s not entire clear why the method here is specifically applied to time-series, as the authors have setup here. Perhaps the authors can clarify how their method has leverage properties in time-series data to their advantage. If this connection is indeed not mentioned, then it seems reasonable to add a sentence or a section in the Appendix justifying this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "znJV2HZGiY", "reviewer": "Reviewer_gGV1", "summary": "This paper considers SFDA using Tucker style decomposed weights and selective fine tuning on target data. Some bounds are presented for this strategy and experimental results are shown.", "questions": "See weaknesses", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The proposed method can be integrated with several baselines SFDA methods to improve adaptation efficiency.", "weaknesses": "1. Cite and compare with:\n- Tang et al, Source-Free Domain Adaptation with Frozen Multimodal Foundation Model, CVPR 2024\n- Litrico et al, Guiding pseudo-labels with uncertainty estimation for source-free unsupervised domain adaptation, CVPR 2023\n- Tang et al, Source-free domain adaptation via target prediction distribution searching, ICLR 2023\n- Gao et al, UNITS: A Unified Multi-Task Time Series Model, NeurIPS 2024\n- Ding et al, Source-Free Domain Adaptation via Distribution Estimation, CVPR 2022\nIt's important to compare against foundation model style approaches since they offer SoTA adaptation efficiency.\n\n2. Writing needs to be improved. The tables with results and plots are very very tiny, and zoom 200% is needed to read them and evaluate. Please choose the most important results to present.\n\n3. The loss functions for target adaptation are not presented, please be specific and self-contained.\n\n4. The method is not time series specific. Why not compare with vision models and datasets to show universality of the method?\n\n5. The authors do not compare against Lokra, Lora with the same style of parameter block freezing and tuning. This would improve fairness of comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mqrLswvUVL", "reviewer": "Reviewer_vYWA", "summary": "The paper introduces a novel domain adaptation approach for time series modeling using tensor decomposition. It decomposes source model weights and only finetunes the tensor core components for target data adaptation. The authors provide PAC-based generalization bounds for the adaptation task and demonstrate computational efficiency through experiments.", "questions": "Related Work:\nThe paper would benefit from discussing connections to recent NLP transformer architectures, particularly:\n\n- Dao et al. (2022), \"Monarch: Expressive Structured Matrices for Efficient and Accurate Training\"\n- Qiu et al. (2024), \"Compute Better Spent: Replacing Dense Layers with Structured Matrices\"", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- New application of tensor decomposition for domain adaptation in time series, building upon concepts from Monarch architecture and block tensor train methods used in NLP transformer models\n- Comprehensive experimental results across multiple datasets clearly demonstrate computational benefits", "weaknesses": "1. While the theoretical analysis provides valuable insights into the trade-off between target data fit and source-target distribution divergence, the assumptions have limitations:\n\n- The posterior $Q(S)$ is assumed to be an isotropic Gaussian with variance matching the prior $P$. This assumption doesn't reflect real-world scenarios and is more like a typical variational inference solutions. A more realistic upper bound should incorporate: (1) Weight differences; (2) Distance between isotropic Gaussian posterior and true posterior. While such an extension appears feasible, the distance metric between isotropic Gaussian and true posterior remains undefined\n\n-  The performance of SFDA compared to LoRA aligns with expected outcomes based on existing work in Monarch and Tensor Train architectures, making it somewhat incremental", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mr8xwPPDKM", "reviewer": "Reviewer_KBWb", "summary": "This paper focuses on the topic of Source-Free Time-Series Adaptation. To make the model adaptation more efficient, this paper proposes to decompose the model into core tensor and factor matrices.  When adaptation, only core tensors are selectively fine-tuned. The method can reduce the overall model size and enhance inference efficiency by decomposing and selective fine-tuning.", "questions": "Please refer to the weaknesses part. Overall, I think it is a good paper with new thoughts.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper proposes to use Tucker-style factorization for the model decomposition. This decomposition raises a new perspective (at least for me) beyond the conventional feature selection and  BN adaptation methods. \n\nThe results are solid. A PAC-Bayes generalization bound is provided for the selective fine-tuning stage. The method is evaluated on the real-world dataset such as AdaTime benchmark.", "weaknesses": "I am only curious about some points listed below:\n\n1) Some domain adaptation methods only adapt the BN layers [1],  normalization [2], or only feature modules [3]. They also selectively fine-tune the model, what is the relations between them and the proposed one. Can the proposed method explain or unify these methods in some cases?\n\n2) Can the proposed method be transferred to the general DA task? If not, could you please explain the reasons?\n\n3) It seems that the proposed theorem can be applied to any method with selective fine-tuning. What is the advantage of the proposed decomposition method analyzed from this theorem? \n\n[1] Evaluating prediction-time batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006.10963, 2020.\n[2] Tent: Fully Test-time Adaptation by Entropy Minimization. ICLR 2021.\n[3] Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. ICML 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CSxVU61NBw", "reviewer": "Reviewer_G1Yq", "summary": "This paper investigates source-free domain adaptation problem. The authors introduce a straightforward yet efficient framework that decomposes the parameter matrices using Tucker-style factorization, allowing the model to adjust only a subset of parameters during target adaptation. Extensive experiments conducted across various datasets and settings demonstrate the superiority of the proposed method over existing baseline models and high sample efficiency.", "questions": "1. Could the author explain the rationality of replacing $P$ (the prior distribution) and $S$ (sampled data) with $\\mathcal{W}{src}$ (the pre-trained network) and $\\mathcal{W}_{trg}$ (fine-tuned network), respectively, in the context of PAC-Bayesian generalization theory? The original theory is designed to measure the generalization error between the empirical and expect loss, but for the same distribution.\n2. Could the author include an ERM baseline in Table 1 to provide readers with a clearer understanding of the performance improvements achieved by the proposed method?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "1. The proposed metric decomposition-based approach for efficient target domain adaptation is simple and efficient. \n2. The experiments are very detailed and thorough, clearly demonstrating the effectiveness and efficiency of the proposed method. That is appreciated. \n3. The manuscript is well-structured and clearly written, facilitating easy comprehension for readers.", "weaknesses": "1. The connection between the proposed method and the characteristics of time-series data is somewhat weak. The method appears to be a general SFDA approach rather than being specifically tailored for time-series data.\n2. The novelty of the approach is limited, as parameter decomposition for domain adaptation is a prevalent technique.\n3. The description of target domain adaptation is insufficient. Specifically, which loss function is utilised? And how to optimise the parameters?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)