Low-Rank Tensor Transitions (LoRT) for Transferable Tensor Regression
Authors: Andong Wang
Yuning Qiu
Zhong Jin
Guoxu Zhou
Qibin Zhao
Venue: ICML 2025 poster (2025)
Keywords: tensor regression
tensor singular value decomposition
tensor recovery
tensor nuclear norm
Primary area: general_machine_learning->transfer_multitask_and_metalearning
TL;DR: We propose LoRT, a theoretically grounded and modular method for robust tensor regression under limited data and heterogeneous distributions.

Abstract: Tensor regression is a powerful tool for analyzing complex multi-dimensional data in fields such as neuroimaging and spatiotemporal analysis, but its effectiveness is often hindered by insufficient sample sizes. To overcome this limitation, we adopt a transfer learning strategy that leverages knowledge from related source tasks to improve performance in data-scarce target tasks. This approach, however, introduces additional challenges including model shifts, covariate shifts, and decentralized data management. We propose the Low-Rank Tensor Transitions (LoRT) framework, which incorporates a novel fusion regularizer and a two-step refinement to enable robust adaptation while preserving low-tubal-rank structure. To support decentralized scenarios, we extend LoRT to D-LoRT, a distributed variant that maintains statistical efficiency with minimal communication overhead. Theoretical analysis and experiments on tensor regression tasks, including compressed sensing and completion, validate the robustness and versatility of the proposed methods. These findings indicate the potential of LoRT as a robust method for tensor regression in settings with limited data and complex distributional structures.

Low-Rank Tensor Transitions (LoRT) for Transferable Tensor Regression
Andong Wang' Yuning Qiu
Zhong Jin
Guoxu Zhou Qibin Zhao
Abstract
Bader, 2009). Building on this foundation, fensor regression extends classical regression directly model relationships between tensor-valued inputs and outputs (Zhou et al., 2013;
Tensor regression is powerful tool for analyzing complex multi-dimensional data in fields such neuroimaging and spatiotemporal analysis, but its effectiveness is often hindered by insufficient sample sizes, To overcome this limitation, we adopt a transfer learning strategy that leverages knowledge from related source tasks to improve performance in data-scarce target tasks. This ap- proach, however, introduces additional challenges including model shifts, covariate shifts, and de- centralized data management. We propose the Low-Rank Tensor Transitions (LoRT) framework. which incorporates a novel fusion regularizer and two-step refinement to enable robust adaptation while preserving cow-tubal-rant structure. To sup- port decentralized scenarios, we extend LoRT D-LoRT, distributed variant that maintains sta- tistical efficiency with minimal communication overhead. Theoretical analysis and experiments on tensor regression tasks, including compressed sensing and completion, validate the robustness and versatility of the proposed methods. These findings indicate the potential of LoRT ro- bust method for tensor regression in settings with limited data and complex distributional structures.
Sun & Li, 2017; Liu et al., 2021; Papadogeorgou et al., 2021;
Taki et al., 2023; Wang et al., 2024a; Luo & Zhang, 2024;
Zhou et al., 2024; Billio et al., 2024), preserving multi-way structure and unifying tasks like tensor completion and com- pressed sensing under common framework (Wang et al., 2021; Lu al., 2018). This modeling paradigm has abled advances across neuroimaging (Zhou et al., 2013), social networks (Romera-Patee Parerrs et al., 2013), computer vision (Llosa- Mota-Vite& & Maitra, 2022), phenotype prediction (Dos Santos et al., 2023), and climate science (Yu & Liu, 2016).
Despite notable progress, tensor regression still faces a fun- damental limitation: limited sample size (Challenge / in Fig. 1), especially when applied to complex structured ten- sor data (Liu et al., 2021; Zhou et al., 2013; 3; Lock, 20. Here, the sample size refers to the number of available in- put-output pairs, where each input is a high-dimensional tensor and the output in functional magnetic resonance imaging (IMRI) studies (Song & Lu, 2017), tensor predictors encode spatiotemporal brain activity, while responses reflect behavioral or clinical measures. Due to high data acquisition costs, the number of subjects-i.e.. training samples-is often small. Similarly, in tensor compressed sensing and completion, the sample size is defined by the number of observed entries, which typically much smaller than the total number of tensor ele- ments (Wang et al., 2021). These scenarios create severe underdetermined problem, as the number of model parame- ters increases rapidly with tensor order and size. This leads to a central challenge:
response variable. For instance,
1. Introduction
The increasing complexity of data in fields such as neu- roimaging, chemometrics, and spatiotemporal analysis has amplified the need for expressive modeling tools. Ten- con--multiddimessional generalizations of matrices-have become essential for representing structured data (Kolda &
How can tensor regression models remain effective under severe sample scarcity?
RIKEN ATP, Tokyo, Japan. Department of Computer, China University of Petroleum Beijing at Karamay, Karamay, China. 'School of Automation, Guangdong University of Technology, Guangzhou, China. "Key Laboratory of Intelligent Detection and the Internet of Things in Manufacturing, Ministry of Education, Guangdong University of Technology, Guangzhou, China. Corre- spondence to: Qibin Zhao <qibin.zhao @riken.jp>.
To address this challenge, transfer learning provides a promising strategy by enabling models to leverage knowl- edge from related tasks datasets (Zhuang et al., 2020; Sun et al., 2019; He et al., 2024a). Specifically, information can be transferred from "source tasks" with abundant or complementary data to the "target task" with insufficient samples, offering the potential to mitigate data scarcity and enhance model performance.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Low-Rank Tensor Transitions (LoRT)
A novel framework for transferable tensor learn- ing: LoRT' introduces new framework that enhances transfer learning in tensor regression by leveraging low- tubal-rank structures with a novel fusion regularizer designed to identify shared patterns across tasks. By employing a two-step estimation strategy, LoRT first in- tegrates information from both source and target tasks, addressing the challenge of limited sample size (Chal- lenge 1). The refinement step further ensures accurate estimation of target tensor parameters, mitigating the impact of model shifts (Challenge 2).
covariate X
noise (0)
Challenge 1:
Limited sample size dydgda 2 Ny
Target task
Estimate para. W!
observation {M Jami
(0) Ny
Challenge 2: Model W!! y ?W?"
Challlnge 1
Challenge
Challlenge Covariate shift F(2" (
Challeege
covariato X
noise !M)
Challenge 4: Decentralized data
parameter W!
observation Es Jami
( Nie
management
Source tank c [K Challenge 4
Figure 1: Illustration of the transferable tensor regression problem, featuring four key challenges: Limited sample size (Challenge 1), model shifts between tasks (Challenge 2), covariate shifts (Challenge 3), and decentralized data management (Challenge 4). The objective is to estimate the target task parameter w/ by leveraging information from K source tasks while addressing these challenges, as detailed in Section 3.
This paper provides compre- hensive theoretical analysis of the LoRT framework, deriving non-asymptotic error bounds that establish the conditions under which LoRT can effectively leverage information from source tasks. These results demon- strate LoRT's robustness to model and covariate shifts (Challenges 2 & 3).
Theoretical insights:
Decentralized data management: To tackle the chal- lenge of decentralized data management (Challenges 4), the LoRT framework is extended to a distributed set- ting with the development of D-LoRT. This distributed variant retains the statistical efficiency of the central- ized LoRT while significantly reducing communica- tion overhead, making it particularly suitable for large- scale, privacy-sensitive, or distributed environments.
However, in the context of tensor regression, transfer learn- ing also introduces additional challenges due to the inherent structure of tensor data and the presence of distribution shifts between source and target samples (He et al., 2024a;b; Duan & Wang, 2023). These shifts manifest as model shifts (Challenge 2 in Fig. 1), where the relationship between tensor predictors and responses changes across domains, and covariate shifts (Challenge 3 in Fig. 1), which occur when the distribution of tensor predictors differs. For in- stance, in tensor regression tasks like fMRI analysis, model shifts might reflect differing brain activity-respone patterns across populations, while covariate shifts could arise from variations in imaging conditions or demographic factors. Similarly, in tensor sensing and completion, covariate shifts might represent differences in the sampling pattern of obser- vations across the source and target tasks, and model shifts could indicate changes in the underlying low-rank struc- tures. These distribution shifts can significantly degrade the performance of transfer learning models, complicating the already challenging task of tensor regression.
To the best of our knowledge, this work is the first to system- atically address the four key challenges of transfer tensor regression through the innovative LoRT framework. The structure of the paper is as follows: we start with notations and preliminaries in Section 2, followed by the problem formulation in Section 3. The LoRT framework and its dis- tributed extension, D-LoRT, are then introduced in Section 4, along with their implementation and theoretical analysis. We proceed with empirical results in Section 5 to validate the proposed methods and conclude with a discussion of lim- itations in Section 6. Related work, proofs and experimental details can be found in the Appendix.
Beyond distribution shifts, another critical challenge in ap- plying transfer learning to tensor regression is decentralized data management (Challenge 4 in Fig. 1) (He et al., 2024a). Data often cannot be centralized due to privacy concerns, logistical constraints, or its large scale. This necessitates developing efficient strategies for implementing transfer learning in distributed settings, with a focus on minimizing communication overhead.
2. Notations and Preliminaries
Notations. For any positive integer d, let [d] := {1, ,d}. We use lowercase bold letters (e.g.. a) for vectors, a upper- case bold letters (e.g., A) for matrices, underlined uppercase letters for 3-way tensor (e.g., A). We use c, and its vari- ants like c1, C, etc., to denote positive constants whose value may vary between lines. For 3-way tensors of size dj x d2 x dg, we assume d1 2 d2 without loss of generality. In this paper, "with high probability (w.h.p.)" means with probability at least - - C3e -c logíd de +d2ds) where { are constants and N7 is the sample size of the target task.
To address the above-mentioned challenges (summarized in Fig. 1), this paper proposes the Low-Rank Tensor Tran- sitions (LoRT) framework for transferable tensor learning with the following contributions:
2

---

Low-Rank Tensor Transitions (LoRT)
For any matrix A, its spectral norm |p and nuclear norm ||A||, are defined as its maximum and sum of its singular values, respectively. Given a tensor A, its f--norm is de- fined || [|)ec(A)]lp and its F-norm is defined as | := (1D]l2 where vee(-) denotes the vector- ization operation of a tensor (Kolda & Bader, 2009). The inner product between two tensors A and B is defined as (A, B) = (Ac vveceeB vec(A)"vec(B) (B). For A E R Rddixdaxdy xda we use A... to denote its i-th frontal slice.
The tuaal rank of I is defined as the number of non-zero tubes in tensor 8 in the t-SVD in Eq. (2), i.e.,
(i()) == # ( S 7 0, i S min min(d1,44) d2])-
To further quantify the low-rank structure of tensors in the transformed domain, we introduce the concepts of tensor tubal nuclear norm and tensor t-spectral norm:
Definition 2.3 (Tensor tubal nuclear norm, tensor t-spectral norm (Lu et al., 2019b)). The tubal nuclear norm (TNN) and tensor spectral norm of any tensor J E IR x x under any M Eq. (I) are defined as the sum of the nuclear norms and the maximum of the spectral norms the frontal slices of (T), respectively, i.e..
The t-SVD Framework. The tensor singular value de- composition (t-SVD) framework is built upon the t-product operation under an invertible linear transform M (Kernfeld et al., 2015). This approach stems from the observation that certain linear transformations can enhance the low-rank characteristics of tensors, effectively exploiting intrinsic cor- relations within the data (Zhang & Ng. 2021; Wang et al., 2021). Recent research has focused on utilizing orthogonal matrices to define the transform M, due to their advan- tageous properties (Lu, 2021; Wang et al., 2023a). This convention is also adopted in this paper. Formally, given an orthogonal matrix M E Rdaxd, we define the associated linear transform (.) and its inverse M (.) on any tensor T E Reh xd as follows:
|
Σ (M
|| == max (M sE|ds]
The significance of tubal rank and TNN lies in their ability to capture and exploit the low-rank structure of tensors in the transformed domain. Tubal rank quantifies the degree of low-rankness by counting non-zero tubes in the t-SVD representation, while the tubal nuclear norm serves as its convex surrogate. These concepts have found wide-ranging applications in tensor estimation tasks, enabling efficient representation and processing of multi-dimensional data (Lu, 2021; Zhang & Ng, 2021; Hou al., 2021).
M(J)= X3 M, and M (T) =J X3 M (1)
where X3 denotes the mode-3 tensor-matrix product (Kern- feld et al., 2015).
Building upon this framework, we can now introduce the fundamental concepts of t-SVD:
3. Problem Formulation
Definition 2.1 (t-product (Kernfeld et al., 2015)). The t- product of any A RRdx xdxxds and B E RRd2xdxx6 xds under the transform M in Eq. (1) is denoted and defined as A * B e E R4 xd such that
This section proposes a problem setting that aligns with and addresses the four challenges illustrated in Fig. 1. We first introduce the target task as follows:
Target Task. For the target task?, we observe Ny input- output pairs (2:(0) ,y; generated from:
((e) ((A) O M()
in the transformed domain, where @ denotes the tensor frontal-slice-wine product.
y 00)
(2c( w(o) + + ie [N]
(3)
where wi()) E Rd xd is our primary parameter of in- terest and (0) is the observation noise. The primary goal is to estimate the target parameter tensor W)()) from noisy observations, as commonly encountered in tasks like tensor compressed sensing and completion (Wang al., 2021).
Following the definitions of t-transpose, t-identity tensor, orthogonal tensor, and f-diagonal tensor given' by (Kernfeld et al., 2015), we can now introduce the t-SVD:
Definition 2.2 (t-SVD, tensor tubal rank (Kernfeld al., 2015)). The tensor Singular Value Decomposition (t-SVD) of any J E Rdd1 xdg dd3 under the invertible linear transform M in Eq. (1) is given as follows:
A key difficulty in this setting is the limited sample size (Challenge 1), where the total number of parameters D = di did2d3 far exceeds the available target samples Nr, making reliable estimation of W!()) inherently challenging.
J
*M A T
where U Rdixd, xd, xd and V. E xddxdax are orthogonal, and S E Rd xdx xds is f-diagonal.
Motivated by the empirical and theoretical success of low- rank models in tensor completion (Wang et al., 2021; Lu
Due to space constraints, please refer to the appendix for detailed definitions.
Following He et al. (2024a), we the superscript 0 to denote the target task, i.e., task 0.
3

---

Low-Rank Tensor Transitions (LoRT)
al., 2019b; Liu et al., 2020), we assume that W has low tubal rank, i.e.,
Assumption 3.2 (Gaussian designs), For any 0 S S
K, the entries of the covariate x(()) E RRdxxd for the k-th task are i.i.d drawn from NO of Further- more, there exists a universal constant &r such that oz S minogAs { S maxo
r r: (:WW) < d2.
(4)
This assumption helps reduce the effective number of pa- rameters and enables statistically efficient estimation under limited sample sizes. It is also supported by prior studies showing the effectiveness of t-SVD-based low-rank struc- tures in capturing intrinsic correlations in tensor data (Zhang & Ng, 2021; Qiu et al., 2022a; Lu, 2021; Wang et al., 2023a; 2021).
S Cz.
This assumption allows for different covariate distributions across tasks while ensuring vell-behaved tail properties. While Gaussian design assumed for simplicity, Assump- tion 3.2 can be extended to sub- Gaub-Ga designs under mild conditions. Finally, for the observation noise, we assume:
Assumption 3.3 (Gaussian random noises). For all 0 < k S K, the noises c!")) are independent Gaussian with zero mean and variance uniformly upper bounded by universal constant e², and are independent of the covariates x((A)
To further address the limited sample size issue, we adopt transfer learning, utilizzig data from related source tasks to enhance target task performance.
Source Tasks, tentially in decentralized scenarios, has Ns samples:
Similarly, each of the K source tasks, po-
Problem Objective. Given the task models, we may ask:
y ( (2)(
(k) w( + (M) ie [Ns], e [K]
(5)
Q1:
How can estimate the target parameter W using both target and source samples while facing Challenges 1-3 in Fig. 1?
where ( E Rd xd2xx is the k-th source task parameter.
While transfer learning offers a promising solution to miti- gate the limited sample size issue, also brings additional challenges. Notably, differences between source and tar- get chsaatcharatterized by model shift and covariate shift (Challenges 2 and 3 in Fig. 1) be-bec significant hurdles.
Q2: How can we further address the decentralized data management challenge (Challenge 4 in Fig. 1) given potentially decentralized source tasks?
Answering this question is the primary focus of our subse- quent sections, where we introduce our proposed method, LoRT, and its distributed variant, D-LoRT.
First, model shift is quantified by the difference between each source model and the target model, i.c., (x(k) w(()) for & E [K] To formalize the notion of informative source tasks, we introduce the parameter space for all the tasks:
4. Low-Rank Tensor Transitions
W(r,)) h)
(6)
((W0))) a (( S - S hr}
We introduce LoRT (Low-Rank Tensor Transitions) and its distributed variant, D-LoRT, to address Q1 and Q2. An illustration of both methods provided in Fig. 2.
with h (li,...,he Nx)TT Here, h 0 quantifies the informative level of the k-th source task. In W(r.)), we posit that the model shift between source and target tasks follows an approximately low-rank structure imposed by TNN to capture shared patterns across tasks, facilitating effective knowledge transfer.
4.1. LoRT for Centralized Tensor Transfer Learning
Our method is motivated by the recent TransFusion frame- work (He et al., 2024a), which studies fusion-based transfer learning for sparse vector regression under covariate shift. In TransFasion, a shared low-complexity component is jointly estimated across tasks using f fi-based fusion regularization, enabling robustness to distributional changes. LoRT extends this idea to the tensor regression setting, where both the pre- dictors and parameters are tensors. This extension poses new challenges. such as modeling low-rank structure along multiple tensor modes, handling tensor-specifie regulariza- tion, and ensuring identifiability under limited samples.
Remark 3.1 (Rationale of approximately low-rank model shift). The approximately low-rank shift in our parameter space W(r, h) is motivated by empirical findings in Low- Rank Adaptation (LoRA) (Hu et al., 2022) and its extensions (Agiza et al., 2024; Wang et al., 2024b). In LORA, it is as- sumed that the target model parameters differ from those of the pre-trained or source model by low-rank increment. This assumption aligns conceptually with our approximately low-rank constraint in Wrr, h). which captures shared pat- terns while accommodating task-specific differences.
To address these, LoRT introduces novel fusion regularizer (Eq. (8) based on the TNN, which promotes low tubal-rank structure in both the target parameter and its differences with the source parameters to leverage the parameter structures
Then, to account for covariate shift, we make the following assumption on the distribution of covariates ,c(4)

---

Low-Rank Tensor Transitions (LoRT)
LoRT
D-LoRT
Task I
Target Task Data
Task-Speciffic Refinement
Task
Target Task Data
Task-Specific Refinement
Task
Task 2
Joint Low-rank
Task K
Weighted Averaging
Task X
Model Aggregation
Weighted Averaging
Minimization
Source Aggregation
Source Data
Parameter
Distributed
Estimation
Transmit Data
Transmit
(a)
(b)
Figure 2: Illustration of the proposed LoRT and D-LoRT frameworks: (a) LoRT: centralized method that combines joint low-rank learning and target-specific refinement address challenges such as limited sample sizes, model shifts, and covariate shifts in tensor regression. (b) D-LoRT: A distributed extension of LoRT that handles decentralized data by aggregating locally estimated models from source tasks, ensuring statistical and communication overhead.
in W(r, h). More specifically, LoRT adopts two-step estimation framework:
xg(4) Rd and y; E R denote the i-th input- output pair from task k, and (p(h) is the correspond- ing task-specific parameter. The optimization vari- able Problem (7) is W = (W(() W(1)) (o("),...,ww()) and W (w vi (()) denotes the corre- sponding solution.
(k)
a joint learning step that integrates source and target data under low-rank constraints to alleviate sample scarcity, and
target-specific refinement step that adapts the solution to the target domain and mitigates negative transfer caused by heterogeneity.
The regularizer R(W) enforces both low-rank struc- ture in the target and structured similarity across tasks:
This design enables LoRT to systematically address the key challenges of transferable tensor regression, including limited target data, model shift, and covariate shift.
R(W) |n
Σ an 1
- 1p(k) (8)
Step 1: Joint Low-rank Learning. It consists of three sub-steps:
where ||-||+ denotes the TNN. The hyperparameter An controls overall regularization strength, while {ax} J== modulate the contribution of each source task in the fusion.
Data Aggregation: We collect .y input-output pairs for the target task and Ns pairs for each of the K source tasks, resulting in total sample size of N .T KNg. This aggregation enables the integration of information from both source and target domains.
Problem (7) integrates average squared loss term ensure good fit across all tasks, and fusion regular- ization term R(W) with two primary functions. First, [|>p(0) ||- encourages a low-rank structure the target task parameters, addressing the high-dimensionality challenge. Second, Σk=1 04 ()()) -w(())), cap- tures the contrast between the target and source tasks, enabling effective knowledge transfer while account- ing for potential model shifts. The use of the TNN is crucial as it effectively captures low-rank structures in tensor parameters and model shifts.
$2 Joint Low-Rank Estimation: Given the structure of the parameter space W(r, h) in Eq. (6), our goal is to estimate the ground truth parameters W. (w)()) w(1) (()) by minimizing regularized empirical loss across all tasks. Specifically, we solve the following optimization problem:
K
(10(()),((()))) x + R(W), yi
(k)
min W
S3 Weighted Averaging: Define the first-step estimator:
2N
k=0 8m1
(7) where N₂ Nr for the target task (k: = 0) and N₂ = Ng for each source task (k [K]. Here,
wV = N-1
Ns k=
w (k) + Ny vo
(9)
5

---

Low-Rank Tensor Transitions (LoRT)
This averaging step combines information from all tasks, weighting each task's contribution by its sample size. It potentially reduces variance and improves over- all accuracy by leveraging the collective knowledge from both source and target domains. The resulting W serves robust initial estimate that balances the influence of all tasks.
across source tasks with the metric
K
N IINs (W - w S 8,.
A small 6, indicates that the source tasks [W(()) ] are well-aligned with the target task W!O) facilitating effective knowledge transfer. We now present our main theoretical
Step 2: Target-Specific Refinement. While joint low- rank learning leverages information from all tasks, it may introduce bias due to differences between source and target tasks. To mitigate this, we introduce refinement step focused solely on the target task:
results, starting with the convergence rate for the one-step estimator. Recall that we have assumed d1 I d2 without loss of generality.
Theorem 4.2 (One-step LoRT), Under Assumptions 3.2 and 3.3, if Ng > rdjds dg. then by choosing do Co / /d,/N and 0g = 61 Ns/N /N for some constant co and C1 we huve
Want w + é
(0)
(10)
with correction è computed by
I w
rd1ds N
d1 +11 (1++
+ + (12)
Nr
è E argmin 2NT
lol - ( W + +))² + xljel.
Ns
w.h.p., where Un := hK / di/Ns. quantifying the impact of task heterogeneity on the estimation error.
This refinement step can be interpreted as a denoising pro- cedure. It allows us to fine-tune our initial estimator using only the target sample, potentially mitigating any negative transfer effects from the source tasks. The use of the TNN regularization in this step ensures that the refinement main- tains the low-rank structure of the estimator.
This result reveals how the estimation error depends on the sample sizes, task differences, and source task alignment. The first term represents the standard error rate for estimat- ing a low-rank tensor, while the latter terms capture the impact of transfer learning.
Parameter Selection. The performance of LoRT depends on the appropriate selection of tuning parameters. Our anal- ysis suggests the following parameter selection strategy:
For the refined estimator, we have the following guarantee:
Theorem 4.3 (Refined LoRT). Under the assumptions of Theorem 4.2, if Nr 2 rdids, Ns 2 K3rd, d2 and hy/d/NNT + KKdd da/Ns = o()). then by choosing pa- rameters as specified in Eq. (11), the solution of the two-step method satisfies w.h.p. that
do =0f11 ax c152 L-c2Vdi/Ns (11)
Here, E and & are adaptive factors for regulariza- tion and source task weighting, respectively, defined as: E = /di/NTl + /di/NsTa- and &2 = /Ns/NTA + (Ns, where lA and lA-- are indicator functions for the event A and its complement. The event is defined as A {rd]ds ds/Ns 2 / with h == N - Σx=1 hy representing average task heterogeneity (i.e., the differences between the model parameters of the source tasks and the target task). This strategy balances source and target influ- ences, enabling LoRT to leverage informative source tasks or rely on target data when source tasks are less relevant.
1iv fart - w < rdids
d1 h
+
(13)
NT
Ng
This theorem demonstrates that the two-step method can effectively mitigate the impact of non- non-diverse source tasks, as the dependence on 8, in Eq. (12) is removed.
These results underscore key strengths of LoRT, effectively addressing Q1. When source tasks are both informative (small h) and closely aligned with the target task (small 8.), LoRT achieves significantly smaller errors compared to the O(rd dg N bound of I-SVD-based tensor sensing (Wang et al., 2021), which depends solely on the target task data. The two-step refinement design further ensures robust- ness by protecting against negative transfer, maintaining reliable performance even when source tasks are uninfor- mative or poorly aligned. Additionally, the dependence on rd dg rather than the full dimension di d2dg demonstrates efficiency in handling of biggh-iimessioall data through low- rank structure.
4.2. Theoretical Analysis of LoRT
We provide theoretical guarantees for LoRT, demonstrat- ing its effectiveness in addressing the challenges of high- dimensional tensor regression in transfer learning settings.
We first introduce a key concept that quantifies the alignment of source tasks with the target task:
Definition 4.1 (Source task alignment). Given the ground truth parameter W. E (r, h), we quantify the alignment
6

---

Low-Rank Tensor Transitions (LoRT)
4.3. D-LoRT: A Distributed Variant of LoRT
Theorem 4.5 (Refined D-LoRT), Under the conditions of
Theorem 4.4, and assuming Nr 2 rd /1/1//r
While LoRT effectively addresses Q1, it assumes central- ized data access, thus fails to address the decentralization challenge in Q2. To overcome this, we introduce D-LoRT, a distributed variant of LoRT.
(11), then, for some chosen parameters", w.h.p.
rd1ds N
d1
I/ diort w0
h
(16)
NT
Algorithm Overview. D-LoRT adapts the joint learning step of LoRT to operate on summary statistics from source tasks, proceeding as follows:
This theorem shows that the two-step D-LoRT method achieves a statistical rate comparable to the centralized LoRT under specific conditions. It effectively balances sta- tistical efficiency and communication cost, making it ideal for high-dimensional tensor regression in decentralized en- vironments, thereby addressing Q2.
S1 Distributed Estimation: Each source node & computes (k)
local estimator W using its data (Xc(4), Mi )
in a distributed manner.
S2 Model Aggregation: Source nodes transmit their lo- cal estimators the target node and the target node computes Wa via the following minimization problem:
5. Experiments
We evaluate the proposed methods, LoRT and D-LoRT, both synthetic and real-world datasets in the context of transferable tensor regression'. The objective is to evaluate whether transfer learning can significantly improve tensor regression performance under limited sample conditions, using tensor compressed sensing and tensor completion as test scenarios.
K
Wa E arg min W
Ns
((
(k)
-
(14)
2N
Nr
+ \OR(W
+
(0)
(2c(0) w())
2N
5.1. Synthetic Data for Tensor Compressed Sensing
S3 Weighted Averaging: The target node computes the averaging parameter Wa based on Wa using Eq. (9).
In the synthetic experiments, we investigate the impact of transfer learning on tensor compressed sensing under Gaus- sian design tensor regression models. Following the setup in $ 3, we generate low-tubal-rank target task parameters wi()) using Nr 200 target samples and NNs 2000 source samples per task. We vary the number of source tasks (K), model shift magnitude (h&),, and covariate shift level (og) to evaluate the robustness of the proposed methods. Due to space constraints, we present only the key results here; comprehensive experimental setups can be found in the Appendix A.I.
$4 Pask-Specific Refinement: The target node refines W
locally via Eq. (10) to obtain the estimator W (0) diort
Theoretical Guarantees for D-LoRT. We establish the following statistical guarantees for D-LoRT. Recall that we have assumed di < d2 without loss of generality.
Theorem 4.4 (One-step D-LoRT). Suppose Assump- tions 3.2 and 3.3 hold. For each source task, the debi- ased estimator is defined in Eq. (E.2) and satisfies stan- dard approximation conditions. Assume Ns >> drdd,dz, Ns Z (h² v K² K2)rd,dg, and by x h = O()). Then, for some chosen parameters', w.h.p.
Performance Across Source Tasks. Fig. 3(a) demonstrates that as the number of source tasks K increases from to 9, LoRT exhibits consistent performance gains, particularly in Step 2. This highlights the effectiveness of the refinement step in leveraging diverse source data. D-LoRT also benefits from additional source tasks but plateaus earlier, likely due to the limitations of decentralized learning. In contrast, baselines TNN (Lu et al., 2018) and k-Sup (Wang et al., 2021) show no improvement as they rely solely on the target
S rdid3 N
di
+ h
Ns + +
(15)
This theorem demonstrates that D-LoRT achieves statistical accuracy comparable to centralized LoRT, particularly when source tasks are similarly informative and sample sizes are sufficiently large. The error bound captures the influence of task heterogeneity (h), sample sizes (N and Ns), and source task alignment (8.).
"See Appendix for detailed parameter settings. 'A simulated implementation is available at: https:// g thub. com hwtp//pngaaaaa alrigg/ww//////////// /LORT, including an example for simulating distributed computation.
"While we do not explicitly ablate individual terms in the fusion regularizer, their effects are indirectly reflected through varying the number of source tasks (see Fig. 3 for tensor compressed sensing and Table (a) for tensor completion). These variations serve empirical ablations.
For the two-step D-LoRT method, we obtain:
*See Appendix for detailed parameter settings.
7

---

Low-Rank Tensor Transitions (LoRT)
task.
dataset, the PSNR gradually improves with more source tasks, reflecting LoRT's ability to aggregate complementary information from diverse sources. LoRT-Step2 consistently shows stronger trends compared to D-LoRT and TNN, indi- cating the importance of refinement in utilizing source-task knowledge.
Robustness Model Shifts. Fig. 3(b) shows the impact of model shifts. While both steps of LoRT experience in- creasing errors with larger shifts, Step 2 maintains superior accuracy, underscoring the benefits of refinement in mitigat- ing model discrepancies. D-LoRT follows a similar trend but with slightly higher errors, while TNN and k- k-Sup re- main stable yet suboptimal due to their inability to leverage source task data.
Impact of Sampling Rates. Tables 1(b) and I(c) reveal how source and target sampling rates (SR) affect performance on the akiyo dataset. Higher source SR leads to more stable improvement in PSNR, while higher target SR amplifies the reconstruction accuracy. These trends suggest LoRT's robustness in handling varying data availability and its ca- pacity to balance source and target contributions effectively.
Impact of Covariate Shifts. Fig. 3(c) highlights the robust- ness of LoRT to covariate shifts. As as varies from 0.3 to 1.8, LoRT remains stable, with Step 2 outperforming Step in most cases, D-LoRT exhibits higher sensitivity to covari- ate shifts but still outperforms the baselines, which show consistently high errors.
Table I: Performance comparison of LoRT, D-LoRT, and TNN on the akiyo dataset. Results are reported in PSNR. (a) Impact of the number of source tasks (K).
5.2. Real-World Data for Tensor Completion
Tensor completion can be viewed as a special case of tensor regression where the design tensor consists of standard ten- sor bases (Qiu al., 2022b). In real-world experiments, we investigate the impact of transfer learning on tensor comple- tion tasks. We evaluate LoRT and D-LoRT using YUV RGB video datasets (akiyo, bridge, grandma, and hall). where each video frame is represented as 128 x 128 x tensor. Completion performance is measured using Peak Signal-to- Noise Ratio (PSNR)? 7 The target task aims to reconstruct video frame from a very limited number of observed entries. To enhance performance, preceding frames are treated as re- lated source tasks, enabling the model to transfer and utilize information from these tasks for improved reconstruction of the target frame. TNN (Lu et al., 2019b) applied solely to the target task is used as the baseline Detailed setups and additional results are provided in Appendix A.2.
K
LoRT Step1
LoRT Step2
D-LoRT
TNN
2
28.60
28.67
26.64
14.49
4
35.44
35.59
27.12
14.50
6
37.58
37.61
27.18
14.30
8
40.29
40.32
27.37
14.24
(b) Effect of source sampling rates (SR).
SR
LoRT Step1
LoRT Step2
D-LoRT
TNN
50%
31.88 34.43
31.92
27.41
14.50
70%
34.51
27.56
14.20
90%
36.42
36.60
27.65
14.38
(c) Effect of target sampling rates (SR).
SR
LoRT Step1
LoKT Step2
D-LoRT
TNN
5%
35.10
35.31
27.50
14.50
10%
37.13
37.22
27.51
17.91
15%
37.35
37.71
27.51
19.70
Performance Across Source Tasks. Table I(a) illustrates the trend that LoRT benefits from an increasing number of source tasks (K = 2 to 8). For instance, on the akiyo
Summary of Experimental Results. The experiments demonstrate that the proposed transfer learning framework, LoRT, effectively enhances tensor compressed sensing and completion tasks under limited sample conditions. By utiliz- ing multiple source tasks and a two-step refinement design, LoRT achieves consistent improvements over baseline meth- ods across various settings, showcasing its potential practical tool for tensor compressed sensing and completion in data-constrained scenarios.
"Other perceptual metrics such SSIM (Wang et al., 2004) and LPIPS (Snell et al., 2017) were also examined. Since our focus is on transfer effectiveness rather than perceptual fi- delity, we report PSNR and RE. Preliminary checks indicated that SSIM trends largely mirrored PSNR across Table 7 and Ta- ble 8, offering limited additional insight. Given this and the nontrivial cost of full evaluation, we omit these metrics in the current version. Additional results will be made available at
https: : /////g ///ittiii thu.. wwwc/iinnnnneeeeeeceeeeeeeeeeeme wwpppiinaaiiia waangg/iwnn/// LLRT.
6. Conclusion and Extensions
"To knowledge, no existing transfer learning methods di- rectly apply to our setting. which involves tensor completion under extreme sparsity and distribution shifts. In this regime. target- only basclines across tensor formats (c.g., t-SVD, Tucker, CP, TT) generally perform poorly due to insufficient observations. As these methods show no substantial difference or comparative value under such conditions, we report TNN (Lu et al., 2019b) as rep- resentative baseline to illustrate the henefit of transfer. Additional comparisons are provided in Table 6.
To address the challenge of limited sample sizes in tensor regression, we propose LoRT, framework that leverages transfer learning to enhance target tasks by utilizing knowl- edge from related source tasks. LoRT tackles key challenges such as model and covariate shifts through a novel fusion regularizer and a two-step refinement process that adapts to distributional differences. Furthermore, we extend LoRT to
8

---

Now-Rank Tensor Transitions (LoRT)
Performance Comparison 150, os
Performance Compatison with dy
Performance Compatison
08-
08-
Averer Rebotive Eate
0.
LofT Sxap TNN (aassinn
Loii shap 2- x-Sup K-Supp 200 Dull
06-
06
DIoRT
basskin)
Averoge Relative
06
0.
D
LURT
LORT
Suplbesetnne
02
011
01-
0.
12
Numbee Scurcr Tsnics (8C)
Maddl Sti8 (6)
(a)
(b)
(c)
Figure 3: Comparison of average relative error for LORT, D-LoRT, TNN, and k-Sup under varying conditions. (a) Average Relative Error vs. Number of Source Tasks (K) for LoRT, D-LoRT, TNN, and k-Sup (h 150, os = 0.9). (b) Average Relative Error vs. Model Shift (h) for LoRT, D-LoRT, TNN, and k-Sup (K 3, ag 0.9). (c) Average Relative Error vs.
Covariate Shift (as) for LoRT, D-LoRT, TNN, and S-pup (K' 3, h 150).
decentralized settings with D-LoRT, which reduces commu- nication costs while retaining statistical efficiency. Through rigorous theoretical analysis and empirical validation on tasks like tensor compressed sensing and completion. LoRT proves to be a robust and effective solution for advancing tensor regression in data-scarce and complex scenarios.
Real-world TCS upplications. Our synthetic TCS ex- periments follow theoretical convention. Applying LoRT to real sensing setups (e.g.. MRI, hyperspectral imaging) requires modeling domain-specife structures and noise, which we leave for future work. We provide preliminary results in Table 8.
Limitations and Extensions. This work offers first theo- retical framework for transfer tensor regression under low- sample and distribution-shifted settings. Several directions merit further exploration:
Real-world data without ground truth. A key gap in study is the lack of experiments real-world ten- sor data without ground truth. We follow common practice in theory-driven work (Lu et al., 2018; Zhang et al., 2020; Wang et al., 2021) by focusing on con- trolled settings to directly validate theoretical predic- tions. Nonetheless, evaluating LoRT on real datasets without known parameters is valuable direction for future work.
Computational complexity. The LoRT framework in- volves repeated applications of the TNN proximal operator (Lu et al., 2019b). Each such call requires Od3- min d1, d2} d1d2) operations due to per-slice SVDs, making large-scale deployment computation- ally demanding. Further work may explore randomized tensor decompositions to reduce runtime.
Adaptive control of negative transfer. Our theoret- ical analysis ensures improvement under moderate heterogeneity, consistent with the observation in He et al. (2024a) that effective transfer generally re- quires adaptation. The modular fusion mechanism in LoRT enables task-specific weighting based on align- ment metrics, thereby downweighting uninformative or misaligned sources. This design naturally supports eaterogeneity-awarr regularization strategies such as He et al. (2024b); Duan & Wang (2023), which in- terpolate between full-source fusion and target-only estimation.
Extension to broader tensor formulations. While LoRT is instantiated with the t-SVD, its design is broadly compatible with alternative tensor decompositions, in- cluding Tucker, Tensor Train, and Tensor Ring (Liu et al., 2013; Imaizumi et al., 2017; Qiu et al., 2022b). Beyond the t-product family, formulations based on HOSVD (De Lathauwer et al., 2000) Tensor Re- gression Networks (TRN) (Kossaifi et al., 2020) offer complementary modeling paradigms that can be inte- grated within the LoRT transfer pipeline.
A Acknowledgements
Beyond the i.i.d. setting. The current analysis assumes i.i.d. sampling within tasks, which simplifies estima- tion bounds. Real-world tensor data often involve de- pendencies (e.g., spatiotemporal structure), motivating future extensions to non-i.i.d regimes. See preliminary experiments in Table 7.
The authors sincerely thank the Area Chair and the four anonymous reviewers for their detailed and constructive feedback. Their suggestions have greatly enhanced the qual- ity and clarity of this paper. This work was supported in part by the National Natural Science Foundation of

---

Low-Rank Tensor Transitions (LoRT)
China under Grant 62203124, and the JSPS KAKENHI
environmental trials. arXiv preprint arXiv:2301.03655 2023.
Grant Numbers JP25K21283, JP24KK0849, JP23K2811 109, JP24K03005, JP25K21288, and RIKEN Incentive Research Project 20847-20230106200 Yuning Qiu was supported by the RIKEN Special Postdoctoral Researcher Program.
Du, S. S., Hu, W., Kakade, S. M., Lee, J. D., and Lei, Q. Few-shot learning via learning the representation, provably, arXiv preprint arXiv:2002.0%4 09434, 2020.
Impact Statement
Duan, Y. and Wang, K. Adaptive and robust multi-task learn- ing. The Annals of Statistics, 51(5):2015-2039, 2023.
This paper develops the Low-Rank Tensor Transitions (LoRT) framework, which systematically addresses the chal- lenges of transfer learning in tensor regression, particularly under limited data and distributional shifts, While the pro- posed methods offer potential benefits for applications in scientific domains such as neuroimaging and spatiotemporal analysis, the primary contribution theoretical. Given its focus on methodological advancements rather than direct application deployment, the societal impact of this work is expected to be indirect, primarily influencing future research in censor-based learning and transfer learning theory.
Hamneke, S. and Kpotufe, S. A no-free- frree-lunch theorem for multitask learning. The Annals of Statistics, 50(6):3119) 3143, 2022.
Harshman, R. A. et al. Foundations of the parafac procedure: Models and conditions for an "explanatory" multi-modal factor analysis. UCLA Working Papers in Phonetics, 16 (1):84, 1970.
He, Z., Sun, Y., and Li. R. Transfusion: Covariate-shie robust transfer learning for high-dimensional regression. In International Conference on Artificial Intelligence and Statistics, pp. 703-711. PMLR, 2024a.
References
He, Z., Sun, Y., Liu, J., and Li, R. Adatrans: Feature- wise and sample-wise adaptive transfer learning for high- dimensional regression. arXiv preprint arXiv:2403. 13565, 2024b.
Aghbalou, A. and Staerman, G. Hypothesis transfer learn- ing with surrogate classification losses: Generalization bounds through algorithmic stability. In International Conference on Machine Learning, pp. 280-303. PMLR, 2023.
Hou, J., Zhang, F., Qiu, H.,, Wang. J., Wang, Y., and Meng, D. Robust cow-tubal-rank tensor recovery from binary measurements, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Agiza, A., Neseem, M., and Reda, S. Mtlora: Low-rank adaptation approach for efficient multi-task learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16196-16205, 2024.
Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adap- tation of large language models. In International Confer- ence on Learning Representations, 2022. URL https: / //openreview wwt/eeouuneeeee www.rrummiinneeeeeeeeeeeeee
Billio, M., Casarin, R., and lacopini, M. Bayesian markov- switching tensor regression for time-varying networks. Journal of the American Statistical Association, 119(545); 109-121, 2024.
Imaizumi, M., Maehara, T., and Hayashi, K. On tensor train rank minimization: Statistical efficiency and scalable algorithm. Advances Neural Information Processing Systems, pp. 3930-3939, 2017.
Carroll, J. D. and Chang, J.-J. Analysis of individual dif- ferences in multidimensional scaling via an n-way gener- alization of "Eckart-Y Y Yuung decomposition. Psychome- trika, 35(3):283-319 1970.
Kernfeld, E., Kilmer, M., and Aeron, S. Tensor-tensor products with invertible linear transforms. Linear Algebra and its Applications, 485:545-570, 2015.
De Lathauwer, L., De Moor, B., and Vandewalle, J. A multilinear singular value decomposition. SIAM journal on Matrix Analysis and Applications, 2144):12538 2000.
Kolda, G. and Bader, B. W. Tensor decompositions and applications. SIAM Review, 51(3):555-50 2009.
Deng. Y., Kuzborskij. I., and Mahdavi, M. Mixture weight estimation and model prediction in multi-source multi- target domain adaptation. In Advances in Neural Infor- 2023.
Kong, H., Xie, X., and Lin, Z. t-Schatten-p norm for low- rank tensor recovery. IEEE Journal of Selected Topics in Signal Processing, 12(6): 1405-149,, 2018
mation P Processing Systems, pp.
Konstantinov, N., Frantar, E., Alistarh, D., and Lampert, C. On the sample complexity of adversarial multi-source pac learning. In International Conference on Machine Learning, pp. 5416-5425. PMIR, 2020.
Dos Santos, A. A., Sarti, D. A., Moral, R. A., and Parnell, A. C. Bayesian additive main effects and multiplica- tive interaction models using tensor regression for multi-
10

---

Low-Rank Tensor Transitions (LoRT)
Kossaifi, J., Lipton, Z. C., Kolbeinsson, A., Khanna, A., Furlanello, T., and Anandkumar, A. Tensor regression networks, Journal of Machine Learning Research, 21 (123):1-21, 2020.
Lu, C., Peng, X., and Wei, Y. Low-rank tensor completion with new tensor nuclear norm induced by invertible linear transforms. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 199966000 2019b.
Kuzborskij, I. and Orabona, Stability and hypothesis transfer learning. In International Conference on Ma- chine Learning, pp. 942-950. PMLR, 2013.
Luo, Y. and Zhang, A. Tensor-on-tensor regression: Rie- mannian optimization, wver-parameterivaaton,, statistical- computational gap and their interplay. The Annals of Statistics, 52(6):2583-200 2024.
Kuzborskij. and Orabona, F. Fast rates by transferring from auxiliary hypotheses. Machine Learning, 106:171- 195, 2017.
Mu, Y., Wang, P., Lu, L., Zhang, X., and Qi, L. Weighted tensor nuclear norm minimization for tensor completion using tensor-SVD. Pattern Recognition Letters, 130:4-11, 2020.
Li, X., Xu, D., Zhou, H., and Li, L. Tucker tensor regression and neuroimaging analysis. Statistics in Biosciences, 10 (3):520-545, 2018.
Papadogeorgou, G., Zhang, Z.,, and Dunson, D. Soft lensor regression. Journal of Machine Learning Research, 22(219):1-53 2021.
Lin, H. and Reimherr, M. Smoothness adaptive hypoth- esis transfer learning. In International Conference on Machine Learning, 2024.
Qiu, H., Wang, Y.. Tang, S., Meng, D., and Yao, Q. Fast and provable nonconvex tensor RPCA. In International Con- ference on Machine Learning, pp. 1821 1-1249. PMLR, 2022a.
Liu, J., Musialski, P., Wonka, P., and Ye, J. Tensor comple- tion for estimating missing values in visual data. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 35(1):208-22) 220, 2013.
Qiu, Y., Zhou, G., Zhao, Q., and Xie, S. Noisy tensor completion via low-rank tensor ring. IEEE Transactions on Neural Networks and Learning Systems, 35(1):1127- 1141, 2022b.
Liu, J., Zhu, C.,, Long, Z., Liu, Y., et al. Tensor regression. Foundations and Trends@ in Machine Learning, 14(4): 379- 565,55555220 2021.
Qiu, Y., Zhou, G., Wang, A., Zhao, Q.. and Xie, S. Balanced unfolding induced tensor nuclear norms for high-order tensor completion. IEEE Transactions on Neural Net- works and Learning Systems, 2024.
Liu, X., Aeron, S., Aggarwal, V., and Wang, X. Low-tubal- rank tensor completion using alternating minimization. IEEE Transactions on Information Theory, 66(3):1714 1737, 2020.
Raskutti, G., Wainwright, M. J., and Yu, B. Minimax rates estimation for high-dimensional linear regression over Ey Ca-balls. IEEE Transactions on Information Theory. 57 (10):6976-6994, 2011.
Llosa-Vite Vite, C. and Maitra, R. Reduced-rank tensor-on- tensor regression and tensor-variate analysis of variance. IEEE Transactions on Pattern Analysis and Machine In- telligence, 45(2):2282-2296 2022.
Raskutti, G., Yuan, M., and Chen, H. Convex regulariza- tion for high nigh-dimensional multiresponse tensor regression. The Annals of Statistics, 47(3):1554-1555 2019.
Lock, E. F. Tensor-on-tenan regression. Journal of Compu- lational and Graphical Statistics, (7(3):638-647 2018.
Comera-Paredes, B., Aung, H., Pontil, M. Multilinear multitask learning. In Interna- tional Conference on Machine Learning, pp. 1444-1452. PMLR, 2013.
N., and
Lu, C. Transforms based tensor robust pea: Corrupted low-rank tensors recovery via convex optimization. In JEEE/CVF International Conference on Computer Vision, pp. 1145-1152, 2021.
Snell, J., Ridgeway, K., Liao, R., Roads, B. D., Mozer, M. C., and Zemel, R. S. Learning to generate images with perceptual similarity metrics. In IEEE International Conference on Image Processing, pp. 4277-428 IEEE, 2017.
Lu, C., Feng, J., Lin, Z., and Yan, S. Exact low tubal rank tensor recovery from gaussian measurements. In International Joint Conference on Artificial Intelligence, pp. 1948-1999 1954, 2018.
Lu, C., Feng, J., Liu, W., Lin, Z., Yan, S., et al. Tensor robust principal component analysis with a new tensor nuclear norm. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019a
Song, X. and Lu, H. Multilinear regression for embedded feature selection with application to fmri analysis. In Pro- ceedings of AAAI Conference on Artificial Intelligence, pp. 2562-25666 2017.

---

Low-Rank Tensor Transitions (LoRT)
Sun, Q., Liu, Y., Chua, T.-S., and Schiele, B. Meta-transfer learning for few-shot learning. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 403- 412, 2019.
Wang, A., Qiu, Y., Bai, M., Jin, Z.,, Zhou, G., and Zhao, Q. Generalized tensor decomposition for understanding multi-output regression under combinatorial shifts. In Annual Conference on Neural Information Processing Systems, 2024a.
Sun, W. W. and Li, L. Store: Sparse tensor response gression and neuroimaging analysis. Journal of Machine Learning Research, 18( (1133))1-37,2000
Wang, J. and Zhao, X. Functional transform-based low-rank tensor factorization for multi-dimensional data recovery. In European Conference on Computer Vision, pp. 39-56. Springer, 2024.
Taki, B. A., Sarwate, A., and Bajwa, W. U. Structured low-rank tensors for generalized linear models. Transac- tions on Machine Learning Research, 2023. ISSN 2835- 8856. URL
Wang. X., Oliva, J. B., Schneider, J. G., and Póczos, Nonparametric risk and stability analysis for multi-task learning problems. In International Joint Conference on Artificial Intelligence, pp. 2146 2146-2152, 2016.
httpeeerwwwwweeeeeeeeeeeeeeeeeee forum?
id 3L441
Tan, Z., Huang, L., Cai, H., and Lou, Y. Non-convex ap- proaches for low-rank tensor completion under tubal sam- pling. In IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 1-5. IEEE, 2023.
Wang. X., Ye, F., and Zhang, Y. Task-aware low-rank adaptation of segment anything model. arXiv preprint 3XXI::2403,19777 2024b.
Tian, Y. and Feng, Y. Transfer learning under high- dimensional generalized linear models. Journal of the American Statistical Association, 1186544)2284-2697) 2023.
Wang. Z., Bovik, A. C., et al. Image quality assessment: from error visibility to structural similarity. IEEE Trans- actions on Image Processing, 13(4):600-61) 612, 2004.
Wimalawarne, K., Sugiyama, M., and Tomioka, R. Multi- task learning meets tensor factorization: Task imputation via convex optimization. pp. 2825-2833, 2014.
Tian, Y., Gu, Y., and Feng, Y. Learning from similar linear representations: Adaptivity, minimaxity, and robustness. arXiv preprint arXiv:2303.17755, 2023.
Wu, S., Zhang, H. R., and Ré, C. Understanding and im- proving information transfer in multi-task learning. arXiv preprint (rXiv:2005.00944 2020.
Tripuraneni, N., Jin, C., and Jordan, M. Provable meta-
learning of linear representations. In International Con- ference Machine Learning, pp. 10434-10443. PMLR, 2021.
Yu, R. and Liu, Y. Learning from multiway data: Simple and efficient tensor regression. In International Conference on Machine Learning, pp. 373-381. PMLR, 2016.
Tucker, I. R. Some mathematical notes three-mode factor analysis. Psychometrika, 31(3)2779-311 1966.
Zhang, A. Cross: Efficient low-rank tensor completion. arXiv preprint arXiv: 11t.16...111....11111111111 2016.
Wainwright, M. J. High-dimensional statistics: A non- asymptotic viewpoint, volume 48. Cambridge university press, 2019.
Zhang, F. Wang. W., Huang, J., Wang, J., and Wang, Y. Rip- based performance guarantee for low-tubal-rank tensor recovery. Journal of Computational and Applied Mathe- matics, 374:1 12767, 2020.
Wang, A., Jin, Z., and Tang, G. Robust tensor decomposition via t-SD: Near-optimal statistical guarantee and scalable algorithms. Signal Processing, 107:19 2020. ISSN 0165-1684. doi: 107319.
Zhang, X. and Ng, M. K.-P. Low rank tensor completion with poisson observations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Wang, A., Zhou, G., Jin, Z., and Zhao, Q. Tensor recovery via *t-Specctall k-Support norm. IEEE Journal of Se- lected Topics in Signal Processing, 15(3):522-534 2021.
Zhou, H., Li, L., and Zhu, H. Tensor regression with applica- tions in neuroimaging data analysis. Journal of the Amer- ican Statistical Association, 108(502):540-522 2013.
Wang, A., Li, C., Bai, M., Jin, Z., Zhou, G., and Zhao, Q. Transformed low-rank parameterization can help robust generalization for tensor neural networks. In Advances in Neural Information Processing Systems, 2023a.
Zhou, Y., Wong, R. K., and He, K. Broadcasted nonpara- metric tensor regression. Journal of the Royal Statistical Society Series B: Statistical Methodology, 86(5):1197- 1220, 2024.
Wang, A., Zhou, G., Jin, Z., and Zhao, Q. Noisy tensor completion via orientation invariant tubal nuclear norm. Pacific Journal of Optimization, 19(2):273-313 2023b.
Zhuang, F., Qi, Z., Duan, K., Xi, Zhu, Y., Zhu, H., Xiong, H., and He, Q. A comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1):43776 2020.
12

---

Low-Rank Tensor Transitions (LoRT)
Supplemental Material for Low-Rank Tensor Transitions (LoRT) for Transferable Tensor Regression
This supplemental material provides a comprehensive set of theoretical derivations, proofs, and extended experimental results to support the main findings of our paper on Low-Rank Tensor Transitions (LoRT) for transferable tensor regression. The content organized to offer thorough understanding the proposed methodology, its theoretical underpinnings, and its empirical performance across diverse scenarios. Below, we outline the structure of the appendix.
Section A presents extensive set of additional experimental results validate the performance of LoRT and its distributed variant, D-LoRT, on both synthetic and real-world datasets. These experiments evaluate the methods' robustness to model and covariate distribution shifts, the impact of varying source task numbers, and performance under different sampling rates in tensor compressed sensing and completion tasks. Comprehensive comparisons against baselines, such as INN-based regression, highlight the advantages of LoRT and D-LoRT in addressing transfer learning challenges in tensor regression. Additionally, this section includes detailed experimental settings, results on further datasets (e.g., UCF-101 video sequences), and comparisons with other tensor completion methods.
Section B reviews related work, positioning our contributions within the broader context of tensor regression, transfer learning, and multitask learning. This section discusses advancements in low-rank tensor methods and their applications, emphasizing how LoRT integrates transfer learning principles to handle distribution shifts and limited sample sizes effectively.
Section C introduces essential preliminaries, providing a detailed explanation of the t-SVD and key tensor operations that underpin our theoretical framework. This section ensures clarity for readers by defining fundamental concepts and properties critical understanding the subsequent theoretical analysis.
Section D delves into the theoretical analysis of LoRT, presenting rigorous proofs of the main theorems and supporting lemmas. This section establishes performance guarantees for LoRT, demonstrating its effectiveness in high-dimensional tensor regression under model and covariate shifts, with focus on the joint low-rank learning and target-specific refinement steps.
Section E provides the theoretical analysis of D-LoRT, including proofs of convergence for the distributed framework. highlights the method's ability leverage local model aggregation while maintaining performance comparable centralized LoRT, particularly in srivacy-sensitive or bundwidth-constrange settings.
Section F details the algorithmic design and comparison of LoRT and D-LoRT, focusing on the proximal gradient descent (PGD) implementations for both methods. This section also compares their computational complexity and communication overhead.
comprehensive list of notations and symbols used throughout the paper and appendix is provided at the end, ensuring consistency and accessibility for readers.
Ongoing Challenges and Future Directions. While this appendix consolidates the basic theoretical and empirical analysis of LoRT and D-LoRT, we emphasize that this work marks only initial step toward transferable tensor learning. Developing a fully mature framework in this direction requires addressing several open challenges, such as the design of tighter generalization bounds under more adversarial shifts, adaptive task selection mechanisms in large-scale heterogeneous environments, and communication-efficiee protocols for federated tensor optimization. We hope this work lays the groundwork for further research at the intersection of tensor modeling, transferability, and decentralized learning.
13

---

Low-Rank Tensor Transitions (LoRT)
Contents
Introduction
2
Notations and Preliminaries
3 Problem Formulation
4 Low-Rank Tensor Transitions
4
4.1 LoRT for Centralized Tensor Transfer Learning
4
4.2 Theoretical Analysis LoRT
4.3 D-LoRT: A Distributed Variant of LoRT
5
Experiments
5.1 Synthetic Data for Tensor Compressed Sensing
5.2 Real-World Data for Tensor Completion
6
Conclusion and Extensions
8
Additional Experimental Results
15
A.1 Synthetic Data for Tensor Compressed Sensing
15
A.2 Real-World Data for Tensor Completion
16
A.3 Additional Datasets, Baselines, and Experimental Settings
21
Related Work
24
Additional Preliminaries
25
C.I Preliminarics of t-Singular Value Decomposition
25
C.2 Additional Notations
26
C,3 Restricted Strong Convexity (RSC) and Restricted Smoothness (RSM)
27
D Theoretical Analysis of LoRT
28
D.1 Analysis of the Joint Low-rank Learning Step of LoRT
28
D.2 Analysis of the Sarget-Specific Refinement Step of LoRT
31
D.3 Proof of Supporting Lemmas in the Analysis of LoRT
35
E Theoretical Analysis of D-LoRT
39
E.1 Analysis of Step D-LoRT
41
E.2 Proof of Lemmas in the Analysis of D-LoRT
42
F
Algorithm Design and Comparison for LoRT and D-LoRT
44
E1 Optimization Formulation and PGD Implementation
44
F.2 Algorithmic and Communication Comparison between LoRT and D-LoRT
46
G List of Symbols and Notations
49
14

---

Low-Rank Tensor Transitions (LoRT)
A. Additional Experimental Results
This section provides comprehensive experimental evidence supporting the effectiveness of LoRT and its distributed variant, D-LoRT. We assess their robustness to model and covariate shifts, performance under varying source task numbers and sampling rates, and their advantages over classical baselines in both tensor compressed sensing and completion tasks. The results span synthetic and real-world datasets, including YUV and UCF-101 video sequences, and include detailed implementation settings and extended comparisons with representative tensor completion methods.
A.1. Synthetie Data for Tensor Compressed Sensing
In this section, we evaluate the performance of our proposed methods, LoRT and D-LoRT, from the perspective of tensor compressed sensing. The experiments aim to assess their ability to recover low-rank tensor structures and leverage transfer learning to improve performance under limited sample size. We also compare them against two baselines: TNN- based regression (Lu et al., 2018) and k-Support-norm-baase based (k- = 2) regression (Wang et al., 2021), referred to as TNN (baseline) and S-pup (baseline).
The target parameter tensor (( E R4 xdgxda with a tubal rank r is generated as w P AM 2, where P Rd xrxds and E R xds are i.i.d. samples from (N0,, 1). We consider high-dimensional tensor regression problem with dimensions d1 = d2 20, d3 3. and a low-rank level = 2. We generate N₂ = 200 independent target samples (y, ) using y;
(0) (x:"), (( + f
(0) where vea ~ NO I) and c; (V
The source sample size is set to Ns- 2000, with the number of source tasks K varying from 1 to 9. The parameter hy is chosen from values ranging between 10 and 200. To simulate model and covariate shifts, the source tasks are configured as follows:
Model Shift: Model shifts are simulated by setting w w! + {(k) for ls E [K], where g(k) Pk *M 2k= with Pk E RRd xrxds and 2s E xds sampled ii.d. from (o 1). If || (e(() ||~ > hR, then (k)) is rescaled as
{(,)_ h (*)) (1))
Covariate Shift: To assess robustness to covariate shifts, a heterogeneous design is used where (2c ( ~ NOO, o3) for all k E [K]. The value of as is selected from the set (0.3, 0.6, 0.9, 1.2, 1.5, 1.83. For each source task, Nss independent samples are generated.
Experimental Results. The experimental results are shown in Fig. 3. Fig. 3-(a) demonstrates the effect of the number of source tasks on method performance. As the number of tasks increases from one to nine, LoRT exhibits significant improvement, with Step 2 consistently outperforming Step 1. The widening gap between Step and Step 2 indicates that the refinement step becomes more effective with diverse source data. D-LoRT also shows improvement with increasing tasks but plateaus faster than LoRT, possibly due limitations in decentralized learning. TNN and k-Sup, which only utilize target task data, are unable to leverage multiple source tasks, resulting in no improvement.
The impact of model shifts is illustrated in Fig. 3-(b). As the model shift increases, both steps of LoRT show increasing error, with Step 2 maintaining superior performance. The growing gap between Step and Step 2 at larger shifts underscores the increased benefit of the refinement step under significant model discrepancies. D-LoRT follows similar trend to LoRT but with higher error rates, while maintaining relative stability across shifts. TNN and k-Sup exhibit stable yet suboptimal performance due to their inability to harness information from multiple source tasks.
Fig. 3-(c) showcases the influence of covariate shifts. As the covariate shift parameter ranges from 0.3 1.8. LoRT demonstrates stable performance, with Step 2 outperforming Step most of the time. This stability highlights LoRT robustness to changing data distributions. D-LoRT displays more sensitivity to covariate shifts, with increasing error the shift parameter grows, but still outperforms TNN and k-Sup for most shift values. TNN and k-Sup show the highest and most stable error across all covariate shifts.
Summary of Findings. The experimental results demonstrate the performance characteristics of LORT, D-LoRT, TNN and k-Sup under various conditions:
Performance across tasks: LoRT and D-LoRT demonstrate the advantages of transfer learning as the number of source tasks increases, with LoRT showing more consistent gains. In contrast, TNN and k-Sup maintain stable but suboptimal
15

---

Low-Rank Tensor Transitions (LoRT)
performance due their inability to utilize information from multiple source tasks.
Robustness to distribution shifts: LoRT and D-LoRT effectively manage both model and covariate shifts, with LoRT showing more consistent performance across different conditions. D-LoRT also demonstrates robustness, though may experience slight variability in stability under certain shifts.
Centralized vs. Distributed learning: The centralized LoRT method generally outperforms its distributed counterpart, D-LoRT. However, D-LoRT still shows improved performance over TNN and k-Sup, potentially offering a balance between performance and data privacy.
A.2. Real-World Data for Tensor Completion
Tensor completion can be regarded as significant special case of tensor regression, characterized by covariates that are standard tensor basis elements (Qiu et al., 2024). this setting, the relationship between tensor covariates and a response is modeled as
yi (X,, W-) + Ex,
(A.1)
where X; represents i.idd random tensor basis elements as covariates, W, is the parameter tensor encoding the complete data, and denotes the observed entries of the incomplete tensor. The objective is to recover W. using only limited number of observed responses ys-
This formulation highlights the structural similarities between tensor completion and general tensor regression while focusing on the unique challenges posed by missing data and sparse observations. By explicitly leveraging the low-rank structure of W. tensor completion provides an ideal platform for exploring the advantages of transfer learning. Specifically, when the sample size is limited, incorporating information from related tasks (e.g., similar incomplete tensors) can substantially improve the estimation accuracy of WW. This transfer learning approach not only addresses data sparsity but also enhances robustness to distributional shifts, thereby demonstrating its potential in tensor regression scenarios.
Experiment Setup. In our real-world experiments, we investigate the effect of transfer learning on tensor completion tasks, focusing on YUV RGB video datasets (akiyo, bridge, grandma, and hall). Fach frame of the videos represented as 128 x 128 x tensor. The target task involves reconstructing the current video frame from a very sparsely observed subset of its entries. To enhance this reconstruction process, transfer learning is leveraged by treating prior video frames as related source tasks. These source frames share similar spatiotemporal structures with the target frame, providing valuable contextual information. The LoRT and D-LoRT models utilize this information by transferring knowledge from the source tasks to the target task. This enables the models to integrate patterns and relationships learned from earlier frames, resulting in more accurate and robust completion of the target frame, even under conditions of sparse observations. A series of experiments were designed to investigate the impact of different factors on the tensor completion task, including the number of source tasks, sampling rates on source tasks and target tasks, and noise levels.
To evaluate the experimental results, adopt the widely used relatively error (RE) and Peak Signal-to-Noise Ratio (PSNR) metrics, where the RE is defined as
J/ - W- IF
RE
(A.2)
||W
where W and W. are the estimated tensor and ground truth tensor, respectively, and PSNR is given by
MAX
PSNR 10 logı0
(A.3)
MSE
where MAX and MSE denote the maximum element and the mean squared error between the ground truth and reconstructed tensors.
Low-Rank Inductive Bias of the Tubal Nuclear Norm. The tubal nuclear norm (TNN), defined as the convex surrogate of the tubal rank under the t-SVD framework (Definition 2.3), effectively promotes low-rank structure in third-order tensors. By minimizing the sum of nuclear norms across frontal slices the Fourier (or DCT) domain, TNN induces spectral sparsity and suppresses uninformative components. This behavior is analogous the matrix nuclear norm in classical low-rank
e Availaabe at http://taac.cem http://aacccc..aass.dddduuuu asu.cdu/yuv//
16

---

Low-Rank Tensor Transitions (LoRT)
recovery, but adapted to preserve the multi-dimennional correlations inherent in tensor data. Empirical results (e.g., Fig. 4) demonstrate that TNN regularization leads to sharp decay in singular values and energy concentration among small number of spectral components, validating its role powerful inductive bias for modeling low-tubal-rank structure. This property underlies the effectiveness of our approach in compressive sensing and transfer learning settings, where sample efficiency and structural fidelity are critical.
Distribution of Tensor Singular Values
Cumulative Energy Curve
Tensor Singular Value
Accumulated Energy
0.
0.2
0.
Index of Singular Value
Percentage Sinular Values
(a)
(b)
Figure 4: Empirical evidence of the low-tubal-rank structure induced by the tubal nuclear norm. The plots correspond to the recovered tensor (ninth frame of the Akiyo video) and exhibit clear low-rank characteristics: (a) a sharp spectral decay of the tensor singular values, and (b) a rapid saturation the cumulative energy curve.
1. EXPERIMENTAL. RESULTS ON THE VARYING NUMBER OF SOURCE TASKS
We conducted series of controlled experiments investigate the performance of our algorithms under various different source tasks. The number of tasks is selected from the candidate set {1, 3, 4, 5, 6, 7,8}. For simplicity, we let the first K frame be the source tasks tensor and the (K + 1)-th frame as the target tensor. The sampling rate source tensors and target tensor are set to 80% and 5%, respectively.
The experimental results are reported in Table 2 and 3. The results demonstrate the efficacy of the proposed Low-rank Tensor (LoRT) method in leveraging multiple source tasks for improved tensor completion. As the number of source tasks K increases, LoRT exhibits a consistent performance improvement, with PSNR values showing substantial gains and RE values decreasing significantly across all video datasets. This trend underscores the method's ss ability to harness information from multiple source tasks effectively. Notably, LORT, particularly in its LoRT-Step2 implementation, consistently outperforms other methods, including D-LoRT and the TNN baseline. The akiyo video demonstrates the most pronounced improvement with LORT, showcasing remarkable PSNR increase from approximately 22dB at K=/ to over 40dB at K=8. While D-LoRT generally surpasses the TNN baseline, it falls short of LoRT's performance, suggesting potential for refinement in its fusion strategy. The TNN baseline, as expected, maintains relatively constant performance across different K values due its sole reliance on target task data. These findings not only validate the proposed LoRT approach but also highlight its superiority in transfer tensor completion scenarios.
Figure illustrates the reconstruction quality achieved by various algorithms, including LoRT-Stepl, LoRT-Step2, LoRT-D, and TNN approaches, under K = and K settings. The visual results demonstrate the efficacy of the proposed LoRT approach in recovering missing data across diverse vidco sequences, highlighting the impact of different K values on the completion accuracy.
A.2.2. EXPERIMENTAL RESULTS ON VARYING SOURCE TASKS SR
We designed a comprehensive set of experiments to evaluate the performance of our algorithms under varying source sampling rates (SR). The SR values for source tasks were selected from the set (50%, 60%, 70%, 80%, 90%), and the SR for the target task is fixed to 5%. In our experimental framework, we designated the first 4 frames as the source task tensor and the 5th frame as the target tensor.
The experimental results presented in Tables and 5 demonstrate the effectiveness of the proposed LoRT method in
17

---

Low-Rank Tensor Transitions (LoRT)
Table 2: PSNR value comparison akiyo, bridge, grandma, and hall videos under varying source tasks.
akiyo
bridge
K
LoRT Stepl
LoRT Step2
D-LoRT
TNN (baseline)
LoRT Stepl
LoRT Step2
D-LoRT
TNN (baseline)
1 22.15
22.19
25.71
14.49
33.03
33.04
30.33
23.59
28.60
28.67
26.64
14.49
34.99
35.04
31.08
23.82
3 32.90
33,01
26.97
14,49
36.06
36.14
31.22
23.73
35,44
35.59
27.12
14.50
36.53
36.62
31,38
23.68
5 35.60
35.77
27.13
14.51
36.96
37.06
31.31
23.45
6
37.58
37.61
27.18
14.30
37.31
37.42
31,46
23.34
7
39.73
39.77
27.29
14.38
37.63
37.76
31.61
23.65
8
40.29
40.32
27.37
14.24
37.57
37.70
31.60
23.52
grandma
hall
K
LoRT Stepl
LoRT Step2
D-LoRT
INN (baseline)
LoRT Step1
LoRT Step2
D-LoRT
TNN (baseline)
1 29.10
29.10
26.60
17.45
28.52
28.52
25.65
16.10
32.59
32.59
27.24
17.44
31.88
31.89
26.47
16.08
3 34.88
34.88
27.47
17.43
33.04
33.06
26.68
16.02
36.60
36.61
27.62
17.42
35.06
35.08
27.04
15.95
5
37.79
37.80
27.65
17.34
35.75
35.77
27.11
16.11
38.84
38.85
27.79
17.48
36.40
36.44
27.28
16.17
7 39.61
39.63
27.72
17.24
36.73
36.77
27.28
16.12
40.37 8
40.39
27.78
18.13
36.24
36.29
27.17
16.13
Groudd Inah
Mteewal
Lett Strge
Lefs Shap?
Lanit
Ground Truhh
Obeaved
Lastl Stee
Luft lobeet
LINTI
K-4
K= 8
Figure 5: Comparison of completion performance across different methods for four video datasets under (a) K 4 and (b)
K = settings.
18

---

Now-Rank Tensor Transitions (LoRT)
Table 3: RE value comparison on akiyo, bridge, grandma, and hall videos under varying source tasks.
akiyo
bridge
K
LoRT Stepl
LoRT Step2
D-LoRT
INN (baseline)
LoRT Step1
LoRT Step2
D-LoRT
TNN (baseline)
0.1593
0.1388
0.1058
0.3851
0.0292
0.0291
0.0398
0.0865
2 0.0759
0.0753
0.0951
0.3850
0.0233
0,0231
0.0365
0.0842
3
0.0462
0.0457
0.0915
0.3851
0.0206
0.0204
0.0360
0.0851
4
0.0345
0.0339
0.0899
0.3848
0.0195
0,0193
0.0353
0.0857
5
0.0339
0.0332
0.0899
0,3839
0.0186
0.0183
0.0356
0.0879
6 0.0270
0.0269
0.0893
0.3935
0.0178
0,0176
0,0350
0.0890
7
0.0211
0.0210
0.0882
0.3901
0.0172
0.0169
0.0344
0.0860
8
0.0197
0.0197
0.0875
0.3962
0.0173
0.0171
0.0344
0.0872
grandma
hall
K
LoRT Stepl
LoRT Step2
D-LoRT
TNN (baseline)
LoRT Step1
LoRT Step2
D-LoRT
TNN (baseline)
0.1259
0.1260
0.1679
0.4818
0.0659
0.0659
0.0918
0.2757
2
0.0845
0,0846
0.1566
0.4834
0.0448
0,0448
0,0836
0.2765
0.0647
0.0647
0.1518
0.4824
0.0392
0.0391
0.0815
0.2779
4
0.0532
0,0531
0.1494
0.4836
0.0311
0,0310
0,0782
0.2806
0.0464
0.0464
0,1492
0.4891
0.0287
0,0286
0.0775
0.2754
6 0.0412
0,0412
0.1472
0,4827
0.0266
0,0265
0,0761
0.2732
0.0376
0.0375
0.1477
0.4936
0.0256
0.0255
0.0760
0.2748
8
0.0345
0.0345
0.1471
0.4469
0.0271
0.0270
0.0771
0.2746
leveraging source tasks with varying SR for improved tensor completion. As the SR of the source task increases from 50% to 90%, LoRT exhibits a consistent performance improvement across all video datasets, with PSNR values showing substantial gains and RE values decreasing significantly. Notably, LoRT, particularly its LoRT-Step2 implementation, consistently outperforms other methods, including D-LoRT and the TNN baseline, across all SR values. The TNN baseline, expected, shows relatively constant performance across different SR values due to its sole reliance on target task data. This highlights the advantage of LoRT in leveraging source task information.
Table 4: PSNR value comparison on akiyo, bridge, grandma, and hall videos under different SR.
akiyo
bridge
SR
LoRT Stepl
LoRT Step2
D-LoRT
TNN (baseline)
LoRT Stepl
LoRT Step2
D-LoRT
TNN (baseline)
50%
31.88
31.92
27.41
14.50
30.44
30.56
31.60
23.79
60%
33.27
33.33
27.47
14.23
31.71
31.89
31.69
23.14
70%
34.43
34.51
27.56
14.20
32.93
33.21
31.75
23.63
80%
35,41
35.54
27.58
14.56
33.94
34.31
31.80
24.01
90%
36.42
36.60
27.65
14.38
34.68
35.08
31.85
23.69
grandma LoRT Step2
hall
LoRT Stepl
D-LoRT
TNN (baseline)
LoRT Stepl
LoRT Step2
D-LoRT
TNN (baseline)
50%
34.95
34.98
27.87
17.61
30.72
30.77
27.43
16.05
60%
35,76
35.81
27.89
17.40
31.62
31.71
27.42
16.17
70%
36.38
36.43
27.90
17.26
32.60
32.71
27.49
16.17
80%
37.26
37.32
27.96
17.62
33,50
33.63
27.56
16.11
90%
37.92
37.99
28.00
17.40
34.31
34.50
27.59
15.91
A.2.3. EXPERIMENTAL RESULTS ON VARYING TARGET TASK SR
We conducted series of comprehensive experiments to evaluate the performance of algorithms under varying target SR. The SR values for the target task were selected from the set (5%, 8%. 10%, 12% 15% 18% 20% representing range from sparse to relatively dense sampling of the target task. In our experimental framework, we consistently used the first 4 frames as the source task tensors and the 5th frame as the target tensor across all datasets. For the source tasks, we maintained a fixed sampling rate 80% for tensors.
The experimental results presented in Figures demonstrate the effectiveness the proposed LoRT method in leveraging
19

---

Low-Rank Tensor Transitions (LoRT)
Table RE value comparison on akiyo, bridge, grandma, and hall videos under different SR.
akiyo LoRT Step2
bridge
SR
LoRT Stepl
D-LoRT
TNN (baseline)
LoRT Step1
LoRT Step2
D-LoRT
TNN (baseline)
50%
0.0520
0.0518
0.0870
0,3848
0.0393
0.0388
0.0344
0.0846
60%
0.0443
0.0440
0.0864
0.3968
0.0340
0.0333
0.0341
0.0912
70%
0.0388
0.0384
0.0855
0.3982
0.0295
0.0286
0.0338
0.0862
80%
0.0346
0.0341
0.0853
0.3819
0.0263
0.0252
0.0336
0,0825
90%
0.0309
0.0302
0.0846
0.3901
0.0241
0.0231
0.0335
0.0856
grandma
hall
SR
LoRT Stepl
LoRT Step2
D-LoRT
TNN
LoRT Step1
LoRT Step2
D-LoRT
TNN
50%
0.0643
0.0641
0.1452
0,4729
0.0512
0.0509
0.0748
0.2773
60%
0.0586
0.0582
0.1448
0.4849
0.0462
0.0457
0.0749
0.2735
70%
0.0545
0.0542
0.1447
0,4925
0.0412
0.0407
0.0743
0.2736
80%
0.0493
0.0489
0.1437
0.4724
0.0372
0.0366
0.0737
0.2753
90%
0,0456
0.0453
0.1431
0,4848
0,0339
0.0331
0.0734
0,2819
#kyo
bridge
grandima
hal
L:RT-Swa ORT-SSupt LcHT-Shep LofT-C
PSNA (09)
PBNR (09)
PENA (0B)
TNV toaselinc)
13NR (o
-0 LATT-Stept
LoAT-Step2
Laff--Swpt
-Bhi ToRT--Sppp LoPT-Swep
LoAT
LoRT-0
TNN
TNN (coseline)
17
6.14
3.30
6.16
800
81
8.16
016
876
SR
SR
SR
SR
PSNR
skiy
bridge
grandima
Step!
LoRT Stap1
Shapp
4.50
(bascline)
Step?
042
0.08
LaRT Stup2
TNN (tsoeline)
T-D
LoRT Stop2
Inc)
8:36
6.13
0.04
0.0
2.38
0.1
0.18
8.0
016
020
3R
BR
SR
8R
RE
Figure 6: Comparison of completion performance across different methods for four video datasets under (a) PSNR and (b) RE settings.
20

---

Low-Rank Tensor Transitions (LoRT)
target tasks with varying SR for improved tensor completion across different video datasets. As the SR of the target task increases from 5% to 20%, LoRT exhibits consistent performance improvement across all video datasets, with PSNR values showing substantial gains and RE values decreasing significantly. Notably, LoRT, particularly in its LoRT-Step2 implementation, consistently outperforms other methods, including LoRT-D and the TNN baseline. across all SR values. For instance, in the akiyo dataset, LoRT-Step2 achieves the highest PSNR values, ranging from approximately 35 dB at 5% SR to nearly 40 dB at 20% SR.
Figure visually corroborates these quantitative findings, showcasing the superior reconstruction quality of LoRT-Stepl, LoRT-Step2, and LoRT-D compared the baseline, especially lower SR values (10%). The visual quality improvement particularly noticeable in complex scenes like hall and detailed facial features in akiyo and grandma datasets. In conclusion, these results highlight the significant advantage of the LoRT method, especially its two-step implementation, in effectively leveraging source task information for improved tensor completion across various sampling rates and video datasets.
Ground Trch
Obeaved
LefT Ghep!
LofIT Step?
LofT
TVN Dossind
Ground Trsh
Obearved
Loft D
THNN hoed od
SR 10% for Target Tensor
SR 20% for Target Tensor
Figure 7: Comparison of completion performance across different methods for four video datasets under (a) SR 10% and
(b) SR = 20% settings.
A.3. Additional Datasets, Baselines, and Experimental Settings
To better demonstrate the robustness and applicability of the proposed LoRT framework, we provide additional experimental results and implementation details beyond the main paper. These additions aim to offer a more complete understanding of the proposed method's ' empirical behavior across diverse application contexts, especially in visual tensor recovery tasks.
In particular, we incorporate two additional video sequences-AA AAply Eye Make-up and Blowing Candles-free from the UCF-101 benchmark, as used in Wang Zhao (2024), evaluate the generalization performance of LoRT in spatiotemporal tensor completion settings. These datasets widely used for benchmarking and exhibit diverse motion dynamics and textural structures, making them suitable for testing the adaptability of transfer-based tensor learning algorithms.
A.3.1. ADDITIONAL BASELINES
To the best of our knowledge, no existing transfer learning methods directly apply to our setting, which involves tensor completion under extreme sparsity and inter-task distribution shifts. In such regimes, target-only baselines across all tensor formats-including t-VD, Tucker, CP, and TT TT-consistently perform poorly due insufficient observations. Due to the lack of substantive differences among target-only baselines under extreme sparsity, we choose to report the t-SVD-based TNN method (Lu et al., 2019b) representative baseline highlight the benefit of transfer in the main experiments.
To further support this design choice, we also compare LoRT against several representative tensor completion methods proposed. These include Tusker-based methods (Convex in Raskutti et al. (2019) and Tucker in Li et al. (2018)), CP-based methods (Zhou et al., 2013), mixed t-SVD/TR- approache (Balanced TNN in Qiu et al. (2024)), the iesentaiio-invarian TNN method (OITNN in Wang et al. (2023b)), and the tensor f - l2 regularized approach (Tan et al., 2023). However, their target-only nature limits their effectiveness in highly sparse and heterogeneous transfer settings. In fact, their empirical
21

---

Low-Rank Tensor Transitions (LoRT)
performance closely resembles that of TNN under identical sampling conditions, offering limited distinction, comparative value, or practical insight from the perspective of transfer learning. Due this and the associated computational overhead, we report only representative results rather than repeating comparisons across all datasets. Detailed results are provided in Table 6.
Table 6: PSNR results UCF-101 video clips (Apply Eye Make-up up and Blowing Candles) with varying K.
LoRT Step
LoRT Step 2
Balanced TNN
Dataset
K
D-LoRT
TNN
OITNN
E 82
CP
Tucker
Convex
25.09
25.16
24.41
16.01
17.26
16.19
14.73
14.19
14.79
14.73
Eye Make-up
29.16
29.18
24.98
16.10
18.01
16.93
15.33
13.68
14.43
15.33
29.92
29.94
25,50
16.62
18.07
17.14
15.56
14.65
14.23
15.56
25.63
25.64
23.14
13.59
14.49
13.65
12.01
14.04
14.96
12.01
Blowing Candles
25.97
25.98
23.86
13.27
14.87
13.77
12.33
14.15
13.19
12.33
28.72
28.74
25.17
13.36
14.92
13.96
12.19
12.93
13.87
12.19
A.3.2. NON-L.LD. SAMPLING SETTINGS
We further investigate LoRT's robustness under non-i.i.d. tensor sampling patterns, including mixture of tube-wise and element-wise missingness. These experiments are designed to validate the performance of LoRT in less idealized settings, where tensor observations may be sampled in a structured or nonuniform fashion. Results are summarized in Table 7 and show that LoRT retains advantage over conventional baseline TNN even under these more realistic sampling regimes. In particular, we observe that LoRT Step consistently yields the highest PSNR and SSIM values across all configurations. The performance gap becomes especially pronounced the number of source tasks increases, suggesting that LoRT effectively leverages auxiliary information even when source observations are non-i.i.d Notably, while D-LoRT offers improvements over TNN, it trails behind centralized LoRT, underscoring the benefit of joint optimization when data can be aggregated.
Table 7: Preliminary results on tensor completion under non-i.i.d. source task measurements. Each source tensor is partially observed with a 40% sampling rate combining tube-wise and element-wise missing patterns. The target tensor is observed at a 5% sampling rate. PSNR and SSIM are reported for each method.
Dataset
LoRT Step 1
LoRT Step 2 SSIM
D-LoRT
TNN
PSNR
SSIM
PSNR
PSNR
SSIM
PSNR
SSIM
24.38
0.6468
24.47
0.6507
21.89
0.6138
16.01
0.2045
Eye Make-up
28.48
0.7591
28.49
0.7593
23.70
0.6472
16.47
0.2131
3
29.58
0.7854
29.60
0.7861
24.55
0.6545
16.26
0.1641
24.15
0.8984
24.15
0.8985
14.39
0.5854
13.22
0.2690
Blowing Candles
25.17
0.9213
25.18
0.9216
18.34
0.7689
13.43
0.3097
3
27.62
0.9517
27.63
0.9519
19.25
0.8073
13.46
0.2987
A.3.3. PRELIMINARY RESULTS ON TENSOR COMPRESSED SENSING
To further evaluate the effectiveness of LoRT beyond tensor completion, we conduct preliminary experiments on tensor compressed sensing. In this setting, each video frame modeled as a third-order tensor, and the entries are sensed via random Gaussian measurements. Each source task contains 3000 measurements, and the target task is measured with only 1500 observations, reflecting highly undersampled regime.
We adopt two video sequences (Apply Eye Make-up and Blowing Candles), with each frame rescaled to 40 x 40 x 3 due to computational constraints. PSNR and SSIM metrics are used to evaluate reconstruction performance across different values of K, the number of source tasks. The results Table show that LoRT Step consistently outperforms both D-LoRT and TNN across settings K and on both datasets. D-LoRT, despite its decentralized nature, achieves competitive performance, validating the effectiveness of local model aggregation. contrast, TNN trained solely on the target
22

---

Low-Rank Tensor Transitions (LoRT)
Table 8: Preliminary results on tensor compressed sensing using UCF-101 video clips. Each source tensor is sensed with 3000 Gaussian projections, and the target tensor is sensed with 1500. PSNR and SSIM are reported.
Dataset
K
LoRT
Step
LoRT
Step SSIM
D-LoRT
TNN
PSNR
SSIM
PSNR
PSNR
SSIM
PSNR
SSIM
18.29
0.5762
18.33
0.5774
14.48
0.3868
13.54
0.2832
Eye Make-up
29.87
0.8461
30.13
0.8472
16.12
0.4681
13.82
0.2942
33.97
0.9202
34.69
0.9126
16.96
0.5183
13.88
0.2942
15.86
0.6775
15.9
0.679
11.88
0.4397
9.53
0.2326
Blowing Candles
26.41
0.9531
26.72
0.9549
13.03
0.5225
9.32
0.1945
31.42
0.986
32.03
0.9861
13.99
0.5959
9.3
0.2008
task suffers from substantially lower PSNR and SSIM, especially under limited measurement conditions, underscoring the advantages of leveraging source task information through transfer.
23

---

Low-Rank Tensor Transitions (LoRT)
B. Related Work
We categorize related work into three key areas: tensor regression and recovery, transfer learning, and multitask learning. Our approach intersects these domains by introducing a theoretically grounded, low-rank tensor regression framework capable of addressing both limited samples and distributional heterogeneity.
Tensor Regression and Recovery. Tensor-based regression and recovery methods have been extensively developed to model multi--dimensinall structured data (Qiu et al., 2022b; Hou et al., 2021; Lu et al., 2019a; Zhang al., 2020; Zhang & Ng. 2021; Zhang, 2016). Early work established foundational tensor decompositions, including CANDECOMPPARATAS
(CP) (Carroll & Chang, 1970; Harshman et al., 1970) and Tucker decomposition (Tucker, 1966), which were later applied to practical domains such as neuroimaging and chemometrics (Zhou et al., 2013; Sun & Li, 2017). Notably, the STORE model (Sun & Li, 2017) introduced sparsity for efficient estimation, while Raskutti et al. (2019) proposed a convex regularization framework to manage nigh-dimensional multi- respone tensor regression.
In parallel, low-rank tensor recovery has seen both convex and non-convex developments. Convex approaches include tubal nuclear norm (TNN) minimization based on the t-SVD framework (Lu et al., 2019a), as well as theoretical tools like tensor restricted isometry property (T-RIP) (Zhang et al., 2020). Non-convex regularizers have also gained attention, including Schatten-p norms (Kong et al., 20 and weighted TNN strategies (Mu et al., 2020), aiming to balance accuracy with computational scalability.
Our work builds upon these methods by extending low-tubal-rank modeling to a multi-task transfer setting, introducing a principled fusion regularizer and leveraging source-target interactions for improved tensor regression performance under distribution shift. In particular, we adapt the TNN framework to heterogeneous learning across tasks, while preserving its low-rank structural benefits. This contributes new perspective to the low-tubal tubal-rank recovery literature by enabling theoretically justified estimation in transfer settings, without relying on centralized access to all data.
Transfer Learning. Transfer learning for regression has developed along both theoretical and algorithmic lines, especially in high-dimensional or low-sample regimes. Kuzborskij & Orabona (2013; 201 7) established foundational generalization bounds using algorithmic stability, while Wang et al. (2016) analyzed the excess risk of nonparametric transfer across tasks. More recent work has extended to high-dimensional and structured models: Tian & Feng (2023); Tian et al. (2023) developed theory for generalized linear models under covariate and model shift, and Du al. (2020); Tripuraneni et al. (2021) studied few-shot and meta-learning with provable representation transfer.
Refinements include surrogate-loss loss-baedd frameworks (Aghbalou & Staerman, 2023), smoothness-adaptive penalties (Lin & Reimherr, 2024), and distributionally robust optimization under covariate shift (He et al., 2024a). These methods, however, primarily operate on vectorized models or unstructured data. In contrast, our work incorporates transfer learning directly within the tensor regression framework by designing a decomposition-avwe -ware regularizer and two-step refinement strategy, enabling effective transfer under low-rank constraints and structured heterogeneity.
Multitask Learning. Multitask learning has been rich area of study, particularly in understanding the conditions under which joint training improves generalization. Hanneke & Kpotufe (2022) established "no-free-lunch" theorem that characterizes when negative transfer is unavoidable. Tensor-based extensions of multitask learning were proposed by Wimalawarne et al. (2014), 4), who used low-rank tensor factorization to encode task relationships through convex regularization.
In terms of information transfer, Wu et al. (2020) provided theoretical insights into when task information can be beneficially shared, and Duan & Wang (2023) proposed adaptive frameworks that dynamically modulate transfer strength according to task relevance. Multi-source and multi-target challenges have also been addressed theoretically: Konstantinov et al. (2020) studied PAC-learning guarantees in adversarial multi-source settings, while Deng al. (2023) focused on mixture weight estimation in domain adaptation.
Our proposed LoRT and D-LoRT frameworks offer a preliminary step toward structured knowledge transfer in tensor regression, where each task is represented as low-rank tensor and source-targed alignment is incorporated through a fusion regularizer. While modest in scope, this design allows the method to leverage tensor structure under distribution shifts, and provides a scalable implementation for decentralized or heterogencous data environments, with theoretical insights that complement existing multitask and transfer learning approaches.
24

---

Low-Rank Tensor Transitions (LoRT)
C. Additional Preliminaries
C.1. Preliminaries t-Singular Value Decomposition
Due to space limitations, some concepts related to t-SVD were omitted in the main text. We provide additional notions here.
Definition C.1 (Frontal-slice-iins product (Lu et al., 2019a)). The frontal-slice-wis product of any two tensors A Rdxxdaxd and B IR xda denoted by A C B, is defined tensor I such that
5,44 A.... B B.n., e [K]
where denotes the standard matrix multiplication. The frontal-slice-wise product performs matrix multiplication on each frontal slice of the tensors, resulting in new tensor.
Definition C.2 (Mf wel--bo-ddiggnnn matrix). The M-block-diageee matrix of any tensor J IR xdzxda, denoted by T, is the block diagonal matrix whose diagonal blocks are the frontal slices of M(J)= M()):
M (M) Jaa
M(T)
T == bww.ag(aa@@.... (M =
E Reiddxdads,
M
This concept arranges the slices of a tensor in the frequency domain into a block diagonal matrix, facilitating the theoretical analysis of t-SD..
We further provide some definitions and properties related t-SVD:
Definition C.3 ((Kerffeld et al., 2015)). The t-transpose of a tensor T E TRdxxdxxds under the M transform (as shown in Eq. (1)), denoted by J" satisfies
M(J ).... (M(J)
i [K].
In other words, the t-transpose performs transpose each slice in the frequency domain and then transforms back the time domain. This operation is one of the foundations of t-SVD theory.
Definition C.4 (Kerrffeld et al., 2015)). The t-identity tensor I E Rdxdxds under the M transform satisfies that each frontal slice of (J) is an da x da identity matrix, i.e.,
M(J) = I, E [K].
It is easy to verify that T * J = T and I *M T = T hold for appropriate dimensions. The t-identity tensor plays role similar to the identity matrix in t-SVD.
Definition C.5 (Kernfeld et al., 2015)). A tensor Q d is called t-orthogonal under the M transform if satisfies
* M
Q =
Q =
T-orthogonality is an important property of tensor transformations, ensuring that the inner product and norm of tensors remain invariant before and after the transformation.
Decomposability of Tubal Nuclear Norm Consider the reduced t-SVD of w(() given by
O
(0)
U * 8 M T
where U Rd xs and V Rddxxxxs are orthogonal tensors, and 8 xxxxx is f-diagonal tensor. We define the projection operators (-) and P. (-) as follows:
(T) 1 *M u
*M J
+
KM V M V U u U *M J *M V T
(C.1)
P.- (T) = (9 l M U *M
AM
(g
V MM VT).
(C.2)
These operators decompose the tensor J into components aligned with the sub-modules t-spanned by U and V, and their orthogonal complements, respectively.
As shown in the appendix of Wang et al. (2020), the following properties hold:
25

---

Low-Rank Tensor Transitions (LoRT)
a). Any tensor J E Rd xdaxds be uniquely decomposed T = PP.(F) + P.+
b). The inner product between the projections P. (X) and P.. is zero, i.e., (P, (.., -0, for all tensors
X, y E I xd2 xda
c). The tubal rank of the projected tensor PT( is most twice the rank of ! i..., (((,(T)) < m..!( for all J E [ddixd2xe
Additionally, the following properties related to the tubal nuclear norm (TNN) can be established:
a). (Decomposabilityy of TNN) For any tensors X, y E RRd xdxxds satisfying X *M y = O and X *M V = 0, the tubal nuclear norm decomposes additively:
| + MI| II). + (Y)lI.
b). (Norm compatibility inequality) For any tensor T E xdixdxxss the tubal nuclear norm can be related to the tensor Frobenius norm and the tensor rank as follows:
|J S (T( ds |T
C.2. Additional Notations
Throughout this appendix, we adopt the following notations: Total sample size: N = KNs + Ny. Let N₂ = Ng for 1,.... K and Ny for ===0
Following the conventions in Wainwright (2019), adopt the following notations. For any task k = 0, 1,.... K, define the observation vector y noise vector (()), design operator (-))() and the adjoint operator of x+)((-) follows:
(() (v)
(k)
(k)
ERN,
(A) (6
(K)
ERN,
( A) ((00() A)..., (20(1)
A))T E RRM,
VA Rd xd2dd
(C.3)
Ni
x ( (z) = xx(1)
(k) E mddx addxdxxdy
Vz R
We use several asymptotic notations to describe the relationships between functions. For the sake of clarity, provide their definitions here:
The notation ('n) S s(n) means that there exists a positive constant e and a positive integer no such that for all n 2 no. we have f(n) S c. (n). This is equivalent to saying f(n) O(g(n)).
Similarly, f(n) 2 (n) means that there exists positive constant and positive integer no such that for all nu. we have (n) 2 e g(n). This is equivalent to saying g(n) = (()))).
We write fn) = (n) both f(n) $ (n) and f(n 2 gn) hold. This means that f(n) and g(n) are of the same order.
The notation f(n) = o(g(n)) means that for every positive constant there exists a positive integer no such that for all n 2 no. we have [f() & c (g()).
These notations allow us to express the asymptotic behavior of functions concisely, which is particularly useful in our analysis of algorithmic complexity and error bounds.
26

---

Low-Rank Tensor Transitions (LoRT)
C.3. Restricted Strong Convexity (RSC) and Restricted Smoothness (RSM)
Note that according to Assumption 3.2, the design tensors x((a) in each task (k: = 0, l....,KK) is random Gaussian design. We first establish a key property of the Gaussian design tensor, which serves as a fundamental building block for our subsequent analysis. Lemma C.6 demonstrates that the least squares objective function possesses the restricted strong convexity (RSC) and restricted smoothness (RSM) properties.
Lemma C.6 (RSC and RSM). Under Assumption 3.2, any E Rd xd2xds, with probability at least 1 1 exp(-c2Nss)
||
d1 3k
RSC:
x (
|
N₂
NE.
RSM:
|x(()(
(A)
S
Th
A
| ?
N₂.
N
where or 2 20c .7k E C2C₂ and Bk. Th S CzCs, where C1, c22 C3 are universal postive constants.
Proof of Lemma C.6. The results can be simply obtained in a similar manner to the proof of Proposition 1 in Raskutti al. (2011). The only difference to change the changing the & n-orm to tubal nuclear norm, and the upper bound of Elsupeev()) 1²d (E!/2y) by
E sup (H. ob)i S ( S lox(Vd + d22) ΔEV(E)
(C.4)
where H- Rdd1xd2xd xda is a tensor whose entries i.i.d. standard Gaussian, and
V(E) == {A E Rd xdaxda σ | = 1, ||| s S..
(C.5)
27

---

Low-Rank Tensor Transitions (LoRT)
D. Theoretical Analysis of LoRT
D.I. Analysis the Joint Low-rank Learning Step of LoRT
We begin by introducing some key notations and transformations that will facilitate our analysis.
For each source task [ let @ w w denote the difference between the ground truth parameters the k-th source task and the target task, representing the model shift. We define:
+ ( ! ) (w
w -
W W!!)) 12((K+))ddxxxxx
(D.1)
Here, W Rdi xds is the tensor parameter of the target task model, and w Rd is the parameter of the k-th source task model for all & E [K].
We rewrite the loss function as:
(
x100))>(0(000)
Σ Ilykk) x + +
(D.2)
2N
where we use the change of variable:
θ = o (e(,,..,KK))) =
(()))
(WDo) w(1) -w(") wik) wil) E (X
(D.3)
With this change of variables, solving problem (7) is equivalent to solving:
argmin (C(() Ro)))
(D.4)
where ro)( = =K As (x))))
||,, and we set D Agaz for all [K] for simplicity.
We define the estimation error as Á ē 6. E xd++))x with the corresponding k-th block A (4) = (k)
e (k) for all = 0.1,..,,
To quantify the quality of the weighted averaging estimator wV" we further define:
Ns
A
(k) w W
N
as the estimation error of the parameter average W Our goal is to establish an upper bound for |
Supporting Lemmas Theorem 4.2 provides an upper bound on the estimation error of the first step of LoRT. The proof of
Theorem 4.2 relics on three key technical lemmas:
Lemma D.1 (Concentration of Gradient). Under Assumptions 3.2 and 3.3, if Ns Z di, then by choosing
di No = =ccov N
Vd
dR an cD N
for some appropriate constant Co. we have the following upper bound for any Ā = -4 (0) (KK+))x xd2xds, with high probability:
A (1)
A (K)) E
K A II.
LV 6.
(6.)
A)
Σ 2 Jmma
This lemma bounds the inner product of the gradient of the loss function with the estimation error, which is crucial for controlling the first-order term in our analysis.
28

---

Now-Rank Tensor Transitions (LoRT)
Lemma D.2 (Restricted Set of Directions), Under Assumptions 3.2 and 3.3, and the conditions Lemma D.I, if we farther assume A SλO N and Nş Nr, then the averaging estimation error A satisfies the following inequality with high probability:
K
2Ao A
(0)
A 11)
Au
S 8.o0
A
A=
where P.(.) is the operator defined in Eq. (C.1).
This lemma establishes restricted set of directions in which the averaging error A lies, which is essential for our subsequent analysis.
Lemma D.3 (Restricted Strong Convexity). Under Assumptions 3.2 and 3.3 and the conditions of Lemma D.2, the estimation
error A satisfies with high probability:
K U,
C c + +) - - A) (1
Un J.mim A
Akh2
(D.5)
where
Un = 512 3123maxAr max A
rdida
Omiin X2 A (26/(K +1)
N
2563max
d1
K
Un =
X (A//(K + +1) N
Axh
Omin
min Ok: Pinx = max OSAEK
Ar.
OSkS
with RSC constants (ack, B&) defined in Lemma C.6.
This lemma ensures a property analogous to restricted strong convexity for A", which is crucial for establishing the convergence of our estimator.
Now, we proceed with the proof of Theorem 4.2.
Proof of Theorem 4.2 We introduce the function F [K(K+))xd xdgxda - RR defined as:
AoR (6. + A) Ao (6.).
By applying Lemma D.I. we can establish the following inequality with high probability:
F(θ) L (6, + σ) L (6.) Ao (6. + x) - -NOR (6.) (0)
VV 6.) A vec A) 2 (6. + vec(ā) E E (0,
2
K
A
(10! (1)
A
(k)
| (1.) oo ( A (0) Ao !
k=1
(4)
Σ
d
1.
A
A 00)
+vec ( (v)
2
1=1
K
A
A
- ( 1.) 1.+ + No (l
+ (A - pl. (l.)
1=1
(10)
vec(A) Evec
A
d (/p_- (1) -3/12,140))1
A (4)
Σ Axh
k=
29

---

Low-Rank Tensor Transitions (LoRT)
where (i) follows by the mean value theorem with & E (0, 1): (ii) holds as result of Lemma D.I and we also use
O = : IIP.- (I. = 0, and [| + A|, = ! + PP, (A) + (A))l Z | +P. ( (A)l
[IP. |)]. = | Il. + for 1 Σ k S K. Here, Σ represents the Hessian matrix of the loss function ((+) evaluated at the point (6. A), whose explicit form is given by:
- (I) due to the decomposibility of TNN; (iii) holds because | (*) I|. S hi
>x k= E (2) Juce (xc (T) DE vea( (x2)
vec(2 ( (2e)(
Em v (K) (oc(((11))))
vea( (2)
Σ
1 N
Enn vea (2])(((()))
E vec(2 (K) (2)
E BB( (KK
(D.6)
Leveraging the definition of A", we have A (0)
= A
No N
A
(k),
Applying the triangle inequality yields:
kk=1
F(A) 2vec A) Evec A
do IP (A
|l.
1
Ns
P.-
(A
(1)))|
2
N
k=1
3
K
Ns
K
K
Au (A I||
2 Ao
P.
P.
+
(x)
- 2
2
N
2
km
kmm
We select Ap. A such that
de
2
3 No 2
Ag
This choice leads to:
g
N
K
A
(K)
3
A0
K
N3
PA
(3) || - k=1
K
Ns N
P_ (A 11)
2
2
N
k=1
(
K
3
(8)
K
A
(|P.ICA (k)) 011 - P._ (A (k) |1.)
8
3
2
A0
K
No
N
( (A)
||=
-
2
A0
K
N'
N
P. (A (k) |1
1=1
k=
(18)
K
3Ak
[k:
|1
Ax
1
K
Ns
Ped (A (k))
+
8
N
k=
M
0.
Here, (i) follows from the triangle inequality and (ii) holds due to our parameter setting Àe 3 Ne Ao and the fact that ||- 2 |P. (A (8) >|| since orthogonal
| || (P. (1) + P.- (A (1) 2 IP. 11 IP.. ||,
8
2 N
projection does not increase singular values.
Given that θ is the solution problem (D.4), it follows that 0 2 F(A). We can further deduce:
K 2 du [P..) - >
0 2 vec(A) Evec A) E
-2 No (PP.( +
km
Now, we establish an upper bound for the error measured in Frobenius norm, high probability:
Al... Applying Lemma D.3, we obtain with
K Akh
3 do
0 2 (1 2b 0m
|| |
rdz
و 12 + C.
2
km
30

---

Low-Rank Tensor Transitions (LoRT)
where we the fact that |P (A')
0. Rearranging terms, we derive:
K
1
Aorda + (1 + Un)
Aphi
Un Mmin
To complete the proof, we need to show the order of on and prove that un = o(1) under the conditions of Theorem 4.2.
Given the assumptions in Theorem 1 and our choice of Ao. AK. we have:
256
max AD
rdids
rd1ds
= o(1),
Ommin XE. (2g/ + 1))
N
Ng
256,max
di
K
K2d h.
On X A (K
Σ
S
+ +)) N
Ns
k=1
Finally, leveraging the definition of A we have A (0)
K
Ng
(k)
This leads to:
R=1
K
Ns
A
(0) 12
S 2
1
2
(4)
N
k=1
Ns
12) 1
N
rdid3
d1
(1
Un Mi
+ 6,.
N
N₂
This completes the proof of Theorem 4.2.
D.2. Analysis of the Target-Specific Refinement Step of LoRT
Theorem 4.3 provides an upper bound of the estimation error of the target-specific refinement step of LoRT. We begin by defining two key functions:
E(e) 2N -0) (w"" x10) (e) 2
F(A) å (e A) [(e)) + À lle + All. - ax
where e" ()()) W represents the difference between the target parameter and the averaged parameter.
Let A è e. and recall that A
1 - Wr, We now turn to the proof of Theorem 4.3.
Proof of Theorem 4.3
Applying Hölder's inequality and the triangle inequality, obtain:
Vī((") A
1
(0)
[(()) x'((( x² <we + +e- 1e"--WWp -w! se)
Ny
1
(0)
(0)) x(0) - -w:) Ae)
Ny
(0)
(c
| e ||- + 2 vec A ACTTS (0)
vec(A A e) + 2
vecc (A9)T (0)
vec A1)
Ny
1sp
where (0)
ENr vec (2c Juec (2:))
NT
By Lemma D.I, if Ny 2 d,, we can choose X = =c/
for some constant such that Ny
di
1 Nr
(6000) < with high
31

---

Low-Rank Tensor Transitions (LoRT)
probability. This leads to:
A + e*) -(((e)
i a
VV((e ( A0) - vec (A9) E (0)
(
A e vec (A E
(0)
vec
(ec 9)
3vec(A") (0)
1
vec
2
By the optimality condition of e, we have:
0 02 C)
E A e
- -(A + A e + e² - -Alle"1
/
(D.7)
Ć (A e*
- 11. 2X 11
Á e 11. +
2 vec (A (0)
vec A 9 vec (A (0)
1
vec(A*) - 11
2
2
To proceed, need the following auxiliary lemma:
Lemma D.4. Under Assumptions 3.2 and 3.3, if Ns 2 di, Ns > Nr, and we choose do 2 V N. d2 = an ao ~
N V
such that
Ns N
A 2 12\0
5123 m22x A
rd1d3
Un amin X (1 /K + +1))
o(1),
N
K
256 max
di
0n 2 ^ (1 KK + 1)) N
Σ
011)
then with high probability:
K
| S vrdz2o
Σ
dxh
A=1
K duhix
ΣK Auhon
|| || S rdzd0 + vrday
+
2o
The choice of and A depends on the event A defined as:
rdids
di
A
2
hy
Ns
NT
where h = N Ek lik
We now consider two cases:
Case 1: When event A holds, i.e., rdida/Ns 2 di/NN.
In this case, we choose A = ci V N
and a2 12 N
Applying Lemma D.4, we obtain with high probability:
rd1ds N
di h Ns
A F -
(D.8)
di
N
A A
S rds
vdsV
+
rdzh
+
h.
(D.9)
N
No
NNs
Ns
32

---

Low-Rank Tensor Transitions (LoRT)
We further consider two sub-cases:
(i)
(ec( (0)
vec vec(A")
A*)
2 21 ||e Il., then from Eq. (D.7):
+ vec (Ac) D vec(A - -vec(A") "0 vec (*).
2
(D.10)
By Lemma C.6 and the condition
Nr 2 d1, we have:
vec (A') vve((A) 5o 70 |A To
(0)
d1
|
NT
for some constants % and To- Combining this with Eq. (D.10):
d1
A
ll. S vec(
E vec A") 70 || +
||
NT
rdid3 N
rdid3
d1
h+ (rd1dg)²
hia d1 Ng
N
+ (1+
NT V
(D.11)
Ns
Nr
Nr
Given the assumptions in Theorem 4.3 that rd,da = (1) and h d//Nz = o(1), and noting that (N/Ng)h d2 s (K + D)rdida/Ns = O(1,, we conclude that
NT A e || = Op(1). 2
Applying Lemma C.6 to Eq. (D.10) and choosing
ī = cv
di Nr
with > V2 we get:
dj
e
S 70
A
To
| |
NT
This leads to the bound:
rdids
di VV
d1 h NT
e |F S
+
+
N
Ns
Ns
(ii) If (A)) (0)
vec(A")
< 25 ee
||.- we have:
0
2 1 e |1.
+ (50)
T E ae
1
2
implying | 1-11-- ||. < 8 1+ < sh.
Applying Lemma C.6 again yields:
A
e
11.
1
a0l
A
e
_ Ny
d1
1 1112
11
2
2
1
00
1
32
di
ee 1 4 ll
2
NT
This leads to:
64,0
d1
di
d1 h
A
||:
<
ag
Nr
h22 +
か V Nr
+
V
Nr
and
A
|| 8h.
Given the assumption that / di
o(1), we can conclude:
Nr
di Anhh h. Ny
A e
S
VV
33

---

Low-Rank Tensor Transitions (LoRT)
Combining the results from both sub-cases, we have with high probability:
| é w -
<
A
+
++|14
||
F
F
rdids
di
di Nr
di hnh.
VK
+1
h
+
V
N
Ns
Nr
Since A holds and given the condition K K2rd Iidddd/Nss 01), we have:
d1 h
rdid3
(K + 1)2rd d3
rdid3 d3
rd d3
VK +1 1
/K
+
S
S
NT
Ns
Ns.
N
N
This implies:
|: + 1----0
-w
- م
+|4
rd
d1
h
+
di
hAh. h.
N
Ns
NT
Case 2: When event A holds, i.e., rd/d//Ns s d1/NT.
In this case, we choose:
dh Ao co
12Ns
and as
Ns
N
Applying Lemma D.4 again, we have with high probability:
rd Ns
d1 h Ns
||
(D.12)
d1
d1 rdgh
1
vdg
+ h.
(D.13)
Ng
N
Plugging these new bounds into the arguments from Case leads to:
| + é w || S
rdidz Ns
di + +
ht
di NT
di W NT
hAh.
Ns
Recall that in Theorem 4.3 we assume / /d//Nh o(1) and dj/Ny O()). Therefore, in the above bound, the third term has a smaller order compared to the fourth term. Hence, we have:
I + - S
dd
+
V
da h Ns
+
di
hAh
Ns
NT
As assume A holds in this case, we have /dids/Ns I h / //, which further implies:
I + è é ||, $ VVV
di h.
Nr
Combining the results from both cases, we have with high probability:
S V /rdyds
di h. Nr
+
N
This completes the proof of Theorem 4.3.
34

---

Low-Rank Tensor Transitions (LoRT)
D.3. Proof of Supporting Lemmas in the Analysis of LoRT
Proof of Lemma D.1
By the definition of the loss function, have:
VC 2 1
(x0) (x00)))))( (e(0((())))
+R(++)xdxxxxxs
N
Applying Hölder's inequality, we obtain:
K
(VV(()), |
x (k) (E a (4))
| N
Σ
(x) (e (8)
A (0))
N
k=1
k=
(k) (6 (8)
|lspl A
(8)
11.
x (k)
| A 11.
N
k=0
We frist focus bounding (kx** (e(k) llsp. According the definition of tensor spectral norm
No.
(k) (1) (c [1sp
ltsp
Ns
(k)
(8))
max ds)
M
)isie
t=1
1p
N₂
Σ
M (2c(())
max dE
Let Z6,0 = -M(x)())ss Jane for all f [ds]. By the definition of M-transform:
dg
(1.0) i1d2
Jasads Miia v(i1.12) E [di] [dg].
According to Assumption 3.2, (2c(() Disnss are i.i.d. drawn from N(O,of), so (Z(40)) )iir) ~ N()
Now, we need to bound:
Ne
Σ
Z
(f.0)
sp
Let (b) = (c) .... .ENs T E Ns, Define event E as (+) Nc. From Example 2.11 in Wainwright (2019):
(k)
(k)
(1 2c, Nsl <
Let A = EN E (4.6) Let {p. .p...... p.} and { ........ t,f be 1/4-covers in Euclidean norm of gd and gd2-1 respectively, with I < gdi and J < gdz (Wainwright, 2019).
(k)
For any E Sd2-1, we can write qs z for some vector z with & distance at most 1/4 according to the definition of 1/4-cover:
| < sup ||qll2 S sup ||(q,, + +2)112
9
9E8-4-1
1 7 ||A llp
< sup dE[J]
||qoll2 + || Mpp ||z||2 S sup
||Ag +
bE(J]
35

---

Low-Rank Tensor Transitions (LoRT)
Similarly for $d:-1;
1
A S max [P Aqul22 - z A
=E
Thus, we have
| - max max Lo.b bE[J] dEl!]
where Pa.b Pr. Aq". Then, for fixed a, b,
(8) ((.0) Var(p, 4
9b) = E (K) (8.1)
qb - 1p.T (M) 7,5.0 qol
EE [E [1) &. [le (9]
Conditioning event E, obtain:
N2
Var(Labl
E
Var(p 9i
(k) (1.0) %,1E) [E)
i=1
N2
E (c,
(k) 2
Nu
< 2
E
)
2
So Co.b is zero-mean Gaussian with variance at most 2 Ns conditioning on E:
P P|AA s1p 2
leabl
2
E
2
be[ [E
t2
2IJ 2xp(xxp
16c2c2
f2
< 2exp
(16c3c?
(di + d2) Ing
Recall that we assume di > d2. Setting / = CCaCe v Nædi with sufficiently large e, we have Σ/ e
((.0)
sp
|All|a S cc,cc. with probability at least e exp( -2d1) Taking a union bound with E [da]:
(e))
Σ (M)
(.0)
S ScddiNs
max gE|dg)
with probability at least - exp(-c2N1) c3 exp(-cadj + log dz), where c = c' is a universal constant.
Similarly, we can obtain the following bound, whhpp
K
K No
x (k) (A()))
Ilsp
(k) xx
Mup S Vdd..
=
O==
k=
Therefore, choosing 0;Ao co
No and A coo
N
for sufficiently large co gives the desired result.
36

---

Low-Rank Tensor Transitions (LoRT)
Proof of Lemma D.3 This lemma establishes key inequality that is erucial for our convergence analysis. We begin by expanding the difference in the loss function:
+ )) 5
L (6.) (VC -) vec V ( + + vec(A) (7 E (0, 1)).
This expansion uses the mean value theorem, with y representing some point between and 1. Next, we decompose the Hessian term:
K Σ N
vec A V (0 +7 vec(A) ()
Ns
vec
A ( (N)
vec(A
(A))
Σ
Ns
vec A T=
vec(A (A (0)
N
(A)) T
Σ
Ns
(k)
NT (0)
vea(A(") (A)).
vec
N
N
This decomposition separates the contributions from source tasks and the target task. We can simplify the RHS as:
Ns vee (A (k)
+
(0)
D
(k)
(0)
NT
vee (A E (0)
(oc(A())
vee
Now, we apply the Restricted Strong Convexity (RSC) condition from Lemma C.6:
K N
A (k)
(0)
(h)
A
(b)
(0)
Nr
(A T (0) vea (A (0))
vec
+
vec
N
N
Ngak
Nron
(0)
(0)
R'
A
N
2
2
Here, ag are the RSC constants, and R' (A) is remainder term defined
R'
d1
(k)
A
(0)
Nr di NT
A (0)
N
N₂
We can further lower bound this expression using Ommin = minoskgK Og, which leads to:
vec K) T 2 (6, + + vec(A (A) 2
Ohmin
K
Ns
(M)
A
(0)
R' (A)
N
O'iin
R'
(A).
The key to completing the proof is to bound R'
We start by applying the triangle inequality:
K
28
K
23
R'
A
(k)
a (0)
N
N
Next, we use the restricted set of directions from (D.5):
A
Ao
A
{0)
8 |A
I1, +8
Aghi-
This allows to bound R' (A) in terms of |P.(A.)) our final bound:
||, and Σx Aphig After some algebraic manipulation, arrive at
C (6.+ - -())) ( A 2 un)amin | Up
Xuhi
37

---

Low-Rank Tensor Transitions (LoRT)
where
5123 max AB Ormin R A (A// 1))
rd1d3
256 max Up X A ( 1)) N
d1
Un
and
Σ ARheR
This inequality holds with high probability, completing our proof. The terms uh and Un capture the impact of problem parameters our bound, with Un related to the dimension and rank of our problem, and t, related to the heterogeneity across tasks.
Proof of Lemma D.4 This lemma establishes an upper bound on the nuclear norm of the estimation error. First, note that the bound in Frobenius norm follows directly from the proof of Theorem 4.2. We won't I repeat that proof here.
For the nuclear norm bound, we consider two cases based on the relationship between 1 Ao ||al|, and > Aghig-
Case 1: 1 I1, E Agha.
In this case, we have:
7 22o
7
0
ΣA.
Ao
||l
|
- ||l + 2Ao S
4
This implies:
1
1 (A))
7 (P.
Using the relationship between tubal nuclear norm and Frobenius norm for tensors, we get:
A A (|PP. s 2/
< 16 r
Therefore, in this case, have w.h.p.:
K
A rdgdo + Vrd3,
Apha.
Case 2: 1 ||ll, > Aghi-
In this case, we directly obtain:
8 SEK Auhe <
|
Ao
Combining both cases, conclude that w.h.p.:
K
EK Auhis
| S rdzdp + vrds
Akhi
(D.14)
do
This bound captures the impact of the regularization parameters Ag. the task heterogeneity h4, and the problem dimensions on the nuclear norm the estimation error.
38

---

Low-Rank Tensor Transitions (LoRT)
E. Theoretical Analysis of D-LoRT
In this section, we delve into the theoretical analysis of our Distributed Low-Rank Tensor Transitions (D-LoRT) method. We begin with key theorem that establishes the convergence rate of the one-step D-LoRT estimator. Theorem E.1 (One-step D-LoRT), Let Assumptions 3.2 and 3.3 hold. Further, assume:
Ng > Kr²djd3 (sufficiently large source sample size).
Ns (1i² K2) K2)rdidd d3 (sample size dominates task heterogeneity).
hg h = (1) (bounded task differences).
Then, if we choose d = 01 di/N and use the debiased estimator in Eq. (E.2), with high probability:
|
W
rd d3 N
h
Ng + +
di
This theorem provides an upper bound on the estimation error of D-LoRT. The bound consists of three terms: (a) A standard low-rank estimation error term, (b) A term reflecting the impact of task heterogeneity, and (c) A term capturing the alignment of source tasks.
Now, let's examine how we construct our distributed local parameters.
The Choice of Distributed Local Parameters W (8)
crucial step in D-LoRT is the construction of local estimators. We
begin with intuitive approach:
(k) TNN
((())) x (w))
E argmin
(E.1)
2NNs
w
This estimator, based on the tubal nuclear norm (TNN), provides good starting point. Its properties characterized by the following lemma:
(k) Lemma E.2. Under Assumptions 3.2 and 3.3, and rdyda = o(1) if we construct (WIINN I= through Eq. (E.2), with parameters dε cov/ du for some universal constant Co, then we have that for 1,..., K, w.h.p. No
|W (2) w S rds
di
+ hx-
Ns
However, the TNN estimator is biased. Simply aggregating these local estimators would reduce variance but not address the bias. To overcome this limitation, we introduce a debiasing step inspired by He et al. (2024a):
(A
(A) INN
(A) (x)(k))
(5(*)) (A)) (
(k)
+
(E.2)
Ns
Here, θ (k)
pdidddssd approximates Σ
(>))) where ({{) = E,,, (2c (k)
( (2) For this approxima-
(k)
tion to be effective,
must satisfy two key conditions:
Condition 1: |/@ (k) Itsp- C (Bounded operator norm).
Condition 2: | D (b)
Ill I di/Ns (Close approximation of inverse).
To gain insight into this debiasing process, we can rewrite Eq. (E.2) as:
(k(
(p(A))
(k) (k)
(k)
(k)
(k)
(k) +TNN - Wp(k)
(E.3)
Ng
variance term
This decomposition allows us to analyze the variance and bias separately. Let's examine each term:
39

---

Low-Rank Tensor Transitions (LoRT)
Variance term: Using Condition 1, can bound this term follows:
||
(k) x
(E()))
llop S
|6 (k)
lup- (E())
Mlup 2 Ns
Ns
Ns
2. Bias term: Applying Hölder's inequality and Condition 2, get:
(k)
6 (k)
Σ (k)
(1) 1) (W~TNN- -w!
B
W (A)
W+INN - wo!
d1
W (R)
w
- w
Ns
Combining these results with Lemma E.2, we can conclude that with high probability:
(k)
rdids
d1 Ns
B
hw
isp
Ns
This bound demonstrates that our debiasing procedure effectively reduces both the variance and the bias of the local estimators, setting the stage for efficient knowledge transfer in the D-LoRT framework.
Proof of Theorem E.1 To prove Theorem E.1, we start by defining modified loss function that incorporates both the target and source task information:
K
[()) = 2N 1 x0
(0) 112 +
Σ
No
(k) (
(o
2N
Here, + (0) represents the target task parameter, θ (k) are source task parameters, and R(Ō) a regularization term.
Next, need to bound (v (6) ) We introduce auxiliary terms:
6%- rdids
Ns
di
h&
N
N
Ng
Krdids
d1
8u
h.
Ns
These terms help us capture the effects of task heterogeneity and sample sizes. We can now state key lemma: Lemma E.3 (Concentration of Gradient). Under Assumptions 3.2 and 3.3, if Ns Z dy and we choose
Ns di N N
d Ck
+ 8g
dd + 80
Ao co
N
for some appropriate constants Co. ..., CK. then for any w.h.p.
(V)(
(8)
Ao
(0)
2
k=
The key difference between Lemma E.3 and the earlier Lemma D.1 is in the choice of (Ae (Ae) This new choice allows us to leverage Lemma D.4 under the following additional conditions:
Nş >> Krdidg (sample size dominates task complexity),
40

---

Low-Rank Tensor Transitions (LoRT)
2. Ns 2 K2d (sample size dominates number of tasks squared),
3. hy x h for any E Σ K (task heterogeneity is of the same order).
Applying Lemma D.4, we obtain with high probability:
K
| - Wr 2 Vrdado +
Agha
k=1
dl V
K
Ns
di
S Vdda
+
50
+
+
N
Ns
Assuming h4 = h O(), further simplify bound:
(rdd))
d3
rd1dz
dj h Ns
rdz
h2 z
(E.4)
N2
Ns
N
rdid3 Ns
d1
Ns
d1
8Rhi
<
h.
(E.5)
Ns
N
Ns
These simplifications rely the condition Ns (rrd))ddh, which ensures that the sample size is sufficiently large relative to the problem's complexity.
Combining all these results, we arrive at the final bound stated in Theorem E.1, completing our proof.
E.1. Analysis of Step 2 of D-LoRT
In this section, analyze the second step of our Distributed Low-Rank Tensor Transitions (D-LoRT) method. This step refines the initial estimate to achieve better performance. We present a more detailed version of Theorem 4.5, which provides tighter bounds on the estimation error.
Theorem E.4 (Refined D-LoRT). Under the conditions of Theorem 4.4, and assuming:
.y Z rdida (target sample size sufficiently large),
h/d//Nr o(1) (task heterogeneity not too large).
we choose the parameters as follows:
(VNo
Ns d1 N N
A (12 Fen V
+ +
do 8 ( (V
di
di
la +
V
lAA- + 80
Ng
where
rd
Ns
di
56 N
luk
N
N
80 Krdjds d3
N
d1
hi
N
N
Ns
then with high probability:
| diort
w
12
rdh d3
h
d1
NT

---

A>)/ This bound is crucial for establishing the Low-Rank Tensor Transitions (LoRT)
Proof Sketch: We follow similar line of argument in the proof of Theorem 4.3. The key steps follows:
1) Verify conditions: We first verify that if Ns > Krdıd3, Ns 2 K2d,, and h2 x h for any S k K, choice of parameters satisfies the conditions in Lemma D.4.
2) Apply Lemma D.4; This allows us to bound the estimation error in both Frobenius and nuclear norms:
Aalle S Vrdsdo
+
AKhR
rdid3 N
/rdids
d3
K
IA
In. + 1
h +
/rdz0
+
Σ
8gh2
(E.6)
Ns
Ns
A=1
and
K
Σ k==
| r + Vrds
Aphi +
Ao
r dd
rdg 2 dd
d1 rdgh + rdz00 +
K
I+
In
+
VV
rdgōgha + VKhlla + hIA-
(E.7)
No
Ns
Here, IA is the indicator function for event A, which represents the case where source task information is beneficial.
3) Simplify bounds: To complete the proof, we need to show that the terms involving 6g and & in bounds (E.6) and (E.7)
align with other terms. We use the results from (E.4) and (E.5) to show:
dd
d V
0d
5ch2 S
+ +
h.
Ns
4) Conclude: With these simplified bounds, we can follow the same steps as the proof of Theorem 4.3 arrive at our final result.
Interpretation:
The theorem shows that our two-step D-LoRT method achieves estimation enror that scales with two
main terms: 1) rdids; This represents the standard error for low-rank tensor estimation. 2) h de This term captures the impact of task heterogeneity and the target sample size.
Ny
The bound demonstrates that D-LoRT effectively leverages information from source tasks while being robust task heterogeneity, achieving performance comparable to centralized methods under certain conditions.
E.2. Proof of Lemmas in the Analysis of D-LoRT
Proof of Lemma E.3 We aim to bound the inner product K (vā ())
(6)
convergence properties of our D-LoRT algorithm.
First, we need to understand the structure of the noise in our model. For each source task 1...., , K. the noise term given by:
(k)
{(k)_
/ Ns
B ((a)) (k)
Ns
For the target task (k: 0), the noise simply (0), which is the observation noise for the target model.
42

---

Low-Rank Tensor Transitions (LoRT)
Now, we can bound the inner product using Hölder's inequality:
(
6 -
N
(k)
(K)
+
K
s ((0) +
(c000 ) م (0)
N
N
N
K
N
(k)
م (k)
||.
K
Ns
(())
x (E6
|
+
+ N
N
N
k=1
isp
lsp
Substituting the expression for (A,),
we get:
(VC
5) /
(k)
+
Ns (a) N
B
A (6)
K
K
+
hk)
(k)
() + (c0))
+
Σ N
Ns
B
(k)
A (0)
N
tsp
Now proceed by choosing A for source tasks. From our previous analysis, we know that with high probability:
(E())
Ns and
B (k)
x
rdida
+ hx V
di
N
Ns
Ns
To ensure that 0 is large enough to bound the source task terms, we need:
d >
1
(k)
x I
(E(h))
+
Ns
B
(k)
N
N
1sp
sufficient choice is:
Ns di
rd
Ns
d1
An Ck
h&
N N
N
Ns
for some sufficiently large constant GK-
Then we choose A for the target task. For the target task, we need to bound the combined term. Using the triangle inequality:
1
(N) 2 (E(+)) + (E
K
Ns
(k)
B
N
N
Isp
1
(k)
x (E(())
x
(E
(0))
N
B
(k)
N
N
lsp
N
Isp
tsp
dh
ds
K
Ns
di Ns
hk-
N
N
N
V
Therefore, sufficient choice for Ao is:
(V
di N
Krdids
K
Ng
di
Au =
+
h
N
N
Ng
k=1
for some constant co.
With these choices of Ds and Ag. we have established that whi.p.
(v<(G). A
d +
00
2
This bound is crucial for the convergence analysis of our D-LoRT algorithm, as it allows us to control the behavior of the gradient of our loss function.
43

---

Low-Rank Tensor Transitions (LoRT)
F. Algorithm Design and Comparison for LoRT and D-L.RT
This section presents the algorithmic formulation and implementation details of both LoRT and D-LoRT, with a focus on their respective proximal gradient descent (PGD) procedures. We further provide comparative analysis of their computational complexity and communication costs, highlighting the trade-off's between centralized and decentralized design under the low-tubal-rank tensor regression framework.
F.1. Optimization Formulation and PGD Implementation
To solve the proposed low-rank transferable tensor regression model, we adopt a two-step optimization strategy grounded in PGD. This subsection introduces the formal optimization objective and details the corresponding algorithmic procedures. Specifically, we first formulate the joint low-rank estimation problem across multiple tasks with TNN regularization, and then present the PGD implementation used in Step of LoRT. For Step 2 and task-only variants, we similarly derive problem-specific PGD routines adapted to the single-task setting. These implementations enable efficient and scalable optimization while preserving the low-rank structure essential to the transfer learning mechanism.
Optimization Formulation and PGD Implementation for LoRT Step LoRT under the following optimization problem:
We consider the joint low-rank estimation step
K
K + Ao
min (M)))
y(k) - (1))
(0)
+
(()))
E an O
(F.I)
2N
k=0
k=0
where:
② (0) (0 is the target model parameter,
(k) ( Op(0) for k = I,..., K represents the model discrepancy between source task & and the target,
denotes the observations for task & (see Eq. (C.3)),
x (-) is the design operator for task & (see Eq. (C.3)),
|| denotes the TNN (Definition 2.3),
aj E R4 is the regularization weight assigned to task & (with dn 1 for the target),
= N7 + KNg is the total number of training samples.
denote the current iterate for task k. We define the proximal update rule based on the gradient of the loss function and apply soft-thresholding with respect to the tubal nuclear norm:
Proximal Gradient Descent Strategy. Let @!
1. Compute the gradient of the loss function with respect to each e (k);
1 x (e (0) + 2 -y()). N
k = 0 k = I....,K K
V C= C= =
(F22
21) (e + (*)) y(())
2. Take gradient descent step for each θ by z ) V L.
3. Apply proximal operator
(k) prox Agus 1-l1. (k)
(F.3)
where prox, (+) denotes the proximity operator of the tubal nuclear norm (TNN) with threshold 7 20az, Following
the T-SVT scheme in Lu et al. (2019b), this operator can be efficiently computed as follows:
(a) Apply a linear invertible transform M (e.g., DCT) along the third mode of z (k) obtain (1))= = (x)
44

---

Low-Rank Tensor Transitions (LoRT)
(b) For each frontal slice /!a) of (()) compute the SVD: z U
(c) Apply soft-thresholding to singular values: ST diag(((;; - +)+), where (.)+ max(:; 0);
(d) Reconstruct each slice ( U,S and form the tensor O (8);
(e) Apply the inverse transform: O ( (
This procedure corresponds the Tensor Singular Value Thresholding (T-SVT) operator, which is the proximal operator associated with the TNN. It promotes low-tubal-rank structure by shrinking singular values in the transformed domain wise.-wise.
Algorithm
summarizes the above three steps.
Algorithm 1 PGD for Joint Low-Rank Estimation (LoRT Step 1)
Input: Responses (y(k) J== design tensors ( j. regularization weights {a step size 7, max iterations T
Output: Estimated tensors {0r (k)
k==
1: Initialize e ( 0 for all & O,...,K K
2: for t = 0 to T 1 do
3: for k = 0 to K do
Compute gradient V @ L with current (0(²)
4:
( 7 V
5:
m
(k) (+1
prox Agel lll (2(1)
6:
end for
8: end for
Target-Specific Refinement (LoRT Step 2). Given an initial estimate W from LoRT Step 1, we refine the target parameter using target-only data via the following optimization:
è
arg min
y
(2)
V + te)² Rl[lll.
(F.4)
CER Xd2xdy 2NT 1=1
The final refined parameter is given by W + ê. This optimization is solved using PGD with TNN regularization (Algorithm 2).
Algorithm 2 PGD for Speciff Refinement (LoRT Step 2)
Input: Target responses y()), design tensors (2)(0) Nr initial estimate W", step size regularization X, max iterations T
Output: Refined target tensor W Step2
1: Initialize: C. +
2: for t 0 to T
do
Nr (1) -x0(")) W. + c.)) x0
3: Compute gradient: VeL = Nr Li=1
4: Gradient descent step: X + C, VeL
5: Proximal update: C.+1 prox i ||.11. (%)
6: end for
7: Return: Wstep2 + W", + er
Task-Only Tensor Regression. We consider solving the following convex optimization problem using only single-task data, either for the target-only refinement step or for each local source task update D-LoRT. While our theoretical analysis employs a debiased estimator (cf. Eq. (F.2)) to facilitate convergence guarantees, the actual optimization implemented in
45

---

Low-Rank Tensor Transitions (LoRT)
practice uses the standard TNN regularization:
Nk
W E
arg min WERdixK2ee 2N
,,),,,)))) + +1/WW//-.
(F.5)
The algorithmic steps are provided in Algorithm 3.
Algorithm 3 PGD for Task-Only Tensor Regression
Input: Task-specific responses y(8), tensors (xx(*) 11 1 step size y, regularization X, max iterations Output: Estimated target tensor W
1: Initialize: Wo f- 0
2: for = to T do
3: Compute gradient: VwL = 1
N₂ Ei=
(K)
Vi - -xx!! w xx())
4: Gradient descent step: 2 +- W - WC
5: Proximal update: Wi+1 prox || (X)
6: end for
7: Return: W <- WT
E2. Algorithmic and Communication Comparison between LoRT and D-LoRT
In this subsection, we revisit the distinction between the proposed LoRT and its distributed variant D-LoRT, emphasizing that both frameworks can operate in distributed environments where multiple source nodes (or clients) contribute data or models. The key difference lies what is communicated across nodes: LoRT requires the transfer of raw training data to a central optimizer, while D-LoRT limits communication to model-Ievel summaries. This leads to important implications for scalability, privacy, and deployment feasibility.
Algorithmic Design. We begin by comparing how LoRT and D-LoRT structure their learning pipelines under the assumption that source nodes are distributed and cannot share gradients or samples interactively during training.
LoRT aggregates full training data from all nodes-either by uploading raw tensor samples to central server via shared memory system- and performs a joint low-rank optimization over all source and target tasks. In contrast, D-LoRT keeps raw data strictly local: each node performs local regression with tubal nuclear norm regularization, and transmits only the resulting model (optionally debiased) to a coordinating node. This decoupled strategy simplifies communication and protects sensitive data. Table summarizes this design contrast:
Table 9: Algorithm Structure Comparison
Module
LoRT (Data-sharing)
D-LoRT (Model-sharing)
Local Training
No data sent to central optimizer
Yes localocal regression with TNN regu- larization
What Is Shared
Raw tensors (2] 3/ (
(xx()..(())
Local models V only
Fusion Strategy
Global joint optimization
Model aggregation with optional debi- asing
Target Refinement
Centralized update with full gradient cess
Same refinement, but initialized from aggregated models
Algorithmic Complexity. We now analyze the algorithmic complexity LoRT and D-LoRT. First, the TNN proximal operator in Eq. (F.3) plays a central role in both LoRT and D-LoRT via iterative proximal gradient descent (PGD). Its computational complexity arises from two main operations:
46

---

Low-Rank Tensor Transitions (LoRT)
Transform Domain Conversion: The tensor is first transformed along the third mode using either DFT the DCT, typically applied via fast transform with complexity O(did2 log d3)-
Frontal Slice-wise SVDs: After transformation, the tensor is decomposed into d3 frontal slices, each requiring matrix SVD. The cost of computing the SVD of single slice is (mmin(d1, d2 Jdl d2), and thus the total slice- SVD cost is (di d2 (d1d2ds).
Putting these together, the total cost of one TNN proximal operator call is:
(didg log d3 + d3 min(d1, d2) d1d2),
which is typically dominated by the frontal-slice SVD term for large d3 or high spatial resolution.
We now use the above cost to analyze and compare the computational complexity of LoRT and D-LoRT both Step 1 (low-rank estimation) and Step 2 (target refinement).
Step 1: Joint Low-Rank Estimation. In LoRT, Step performs joint optimization over all K tasks. Each PGD iteration involves one TNN proximal step and one gradient computation per task. leading per-iteration cost of:
O ((K + 1) (d1d2 log d3 + d3 min(d1, d2) djd2)).
Since all updates are handled centrally, this complexity must be sustained by the server.
In contrast, D-LoRT distributes this cost across K source nodes. Each node performs local PGD to estimate its model, and the target node runs similar routine. Hence, the per-node cost is:
O (d1d2 log ds + d3 min d2) dzd2),
and the system-wee wide complexity is O(K) times this cost, but fully parallelized.
Step Speciffc Refinement. Both methods use the same proximal TNN update the target node. Thus, the per-iteration cost is:
O (d logdz + dg min(di, d2) d,d2),
with no difference between LoRT and D-LoRT.
Summary. LoRT concentrates the total cost of all (K + 1) TNN updates central server, while D-LoRT spreads the burden across distributed devices. Although both require the same per-task proximal operation, D-LoRT benefits from parallelism and reduced server load, making it more suitable for federated bandwidth-constrriiedd environments.
Communication Overhead. Both LoRT and D-LoRT involve transmitting information from K distributed source nodes coordinating unit. The fundamental distinction lies in the granularity of what is transferred. LoRT requires high-volume transmission of raw training data poote--ltly multiple high-dimensional tensors per node-while D-LoRT reduces this burden by communicating only single third-order tensor per node (e.g., d2 d2 ds parameters). The comparison in Table 10 quantifies this difference:
Table 10: Communication Cost Comparison
Aspect
LoRT (Data-sharing)
D-LoRT (Model-sharing)
Transferred Item
Raw data tensors
One model tensor per node
Transfer Volume per Node
ON
O(D)
Total Uplink Volume
O(KNsD)
O(KD)
Communication Rounds
One (upload before joint training)
One (after local training)
Privacy Level
Raw data exposed
No raw data shared
47

---

Low-Rank Tensor Transitions (LoRT)
Deployment Trade-offs. Finally, contrast the broader implications of LoRT and D-LoRT in real-world distributed systems. While both frameworks target tensor-based transfer learning, their suitability varies based on infrastructure constraints. LoRT is advantageous when raw data transmission is permissible and joint modeling is desired. D-LoRT, in contrast, offers a practical and privacy-preserving alternative for federated or multi multi-institutionall settings. Table 11 synthesizes these trade-offs:
Table 11: Practical Deployment Comparison
Criterion
LoRT (Data-sharing)
D-LoRT (Model-sharing)
Data Visibility
Full access to raw data from all nodes
Only parameter summaries shared High-nen nodes train independently
Scalability
Limited by bandwidth and data central- ization
Task Coupling
Tight-joint gradients across tasks
Loose-models combined post-training
Communication Cost
High proportional to sample size N₂
Low-only model tensors shared
Privacy Sensitivity
Unsuitable for privacy-restricted data
Compliant with federated protocols
Recommended Scenario
Controlled academic or cloud-hosted data settings
Federated learning, healtheare, or multi- site collaboration
48

---

Low-Rank Tensor Transitions (LoRT)
G. List of Symbols and Notations
Symbol/Notation
Description
Rdixdxxds
Space of third-order tensors with dimensions dj d2 d3
d1, d2, d3
Dimensions of the tensor with di 2 d2
D di d2dg
Total dimension of the tensor
K
Number of source tasks
Nr, Ns
Sample sizes for target and source tasks sample size of the k-th task: = 0, then N₂. = Nr, otherwise N₂. = Ng Total sample size
N&
N
Nr. + KNs
W())
True parameter tensor for the target task
(()
True parameter tensor for the k-th source task
Difference between k-th source and target parameters
Collection of all task parameters
W
Weighted average estimator Estimation error of the average parameter
A
Estimator for target task
(1)
Estimator for k-th source task
Collection of all estimated parameters Estimation error for all parameters
Covariate tensors for target and source tasks Response variables for target and source tasks Noise terms for target and source tasks
Empirical covariance matrix for the k-th task Design operator for the k-th task Adjoint of the design operator for the k-th task Projection operators in Eq. (C.1)
()
(+)
(..))..)))
ha
Model shift parameter for the k-th source task
h
Average model shift Source task alignment parameter
8-
h: huk)
Vector of model shift parameters Standard deviation of covariates for the k-th task Universal constant bounding
Ok
Ce
Upper bound on the variance of noise terms Linear transform and its inverse in Eq. (1) Orthogonal matrix defining (.)
M (),M-1L
M
v-c(-)
Vectorization operator Inner product
(
t-product operation
Outer product
Tensor frontal-slice- product
Tubal rank of (0)
ni())
Tubal rank
Frobenius
Tensor (1)-spectral norm
up
Tubal nuclear norm
do, AR. Ā
Regularization parameters
R())
Regularization function
C(-)
Loss function
2(-)
Modified loss function
Oks 3k
Constants related to restricted strong convexity
Th
Constants related to restricted smoothness
Continued next page
49

---

Low-Rank Tensor Transitions (LoRT)
Table 12 Continued from previous page
Symbol/Notation
Description
Way Un
Terms related to the convergence rate
Event defined for parameter selection
Wr, h)
Parameter space
o()
Big O and little o notations
Asymptotic comparisons
P(.)
Probability measure
B
Expectation
w.h.p.
With high probability
50


Reviews:
[{"review_id": "inJnZUpQcl", "reviewer": "Reviewer_nUVW", "summary": "This paper on Low-Rank Tensor Transitions proposes a new tensor regression framework focusing on transferable tensor learning and decentralized data management.\n\nThis work aims to address three challenges. Working with  limited sample sizes,\nshifting modes as well as covariance shift.\n\nTo tackle these challenges, the paper proposed a novel fusion regularizer and a distributed variant for tensor regression. The idea here is that individual nodes compute their estimators, which are sent to a target node and aggregated by solving a minimization problem.\n\nThe proposed framework comes with theoretical guarantees.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive feedback and recognition of our theoretical framework, analysis, algorithmic design, and empirical results. Below we address the specific concerns.\n\n> Hardware implementation for distributed setting \n\nCurrent implementation focuses on validating the theory, but D-LoRT is parallelizable: source gradients are computed independently and aggregated via proximal updates. It is compatible with distributed frameworks like MPI. We will release a version simulating distributed computation in the final submission.\n\n> Ablation study on the fusion regularizer\n\nWhile we do not explicitly ablate individual terms in the regularizer, their effects are indirectly reflected through varying the number of nodes (Tables 1–3, Fig. 4). These experiments serve as empirical ablations. We will clarify this in the final version.\n\n> Empirical evidence for low-rank structure \n\nThe low-rank-inducing property of TNN is well established (e.g., Lu et al., 2019b). Our visualizations (https://anonymous.4open.science/r/LoRT-113D/LowRankFig.png) show rapid spectral decay in recovered tensors, confirming low-rankness.\n\n> Clarification on Line 176 for lack of CP/Tucker/TT experiments\n\nWe respectfully clarify that our paper does not claim these decompositions can be \"easily\" replaced, nor does it mention CP. Line 176 states that \"TNN can also be replaced with other norms, such as Tucker-based (Liu et al., 2013), Tensor Train (Imaizumi et al., 2017), or Tensor Ring norms,\" which aims to highlight the modularity of LoRT.  However, we acknowledge that the word \"replace\" may suggest direct applicability. “Extend to” would be more appropriate. \n\nThis paper focuses on validating LoRT under t-SVD, chosen for its computational and theoretical advantages. CP is excluded due to the NP-hardness of its nuclear norm.\nWhile LoRT can in principle be extended to Tucker or TT, such adaptations require nontrivial redesign and lie beyond the scope of this theory-oriented work. \n\n> Evaluation limited to PSNR \n\nWe use both PSNR and RE, which are standard metrics sufficient to support our core claims. That said, we agree that SSIM and LPIPS could offer complementary insights and will include them in the final version.\n\n> Code documentation and reproducibility \n\nWe will add more docstrings and ensure consistent random seeds for reproducibility.\n\n> Relation to HOSVD\n\nOur method builds on t-SVD, which is structurally distinct but complementary to HOSVD. While HOSVD is not the focus here, our ideas can be extended to it. \n\n> Relation and comparison to TRN\n\nTRN focuses on end-to-end deep learning with tensor layers for single-task prediction, aiming at architectural efficiency. In contrast, our work studies transfer learning across heterogeneous tasks with theoretical guarantees. Due to these fundamentally different goals and settings, a direct comparison is not appropriate. We will cite TRN and discuss its relevance in the final version.\n\n> Potential datasets and UK Biobank\n\nThis work focuses on theoretical development, and the current experiments—based on both synthetic and structured real-world data—are designed to support our theory. We believe this setting is appropriate for the scope and goals of the paper.\n\nFrom a theoretical perspective, LoRT applies to a variety of structured tensor data, such as hyperspectral images, and multi-subject neuroimaging. This includes large-scale datasets like UK Biobank, where transfer across tasks or sites may be beneficial. That said, applying LoRT to such domains requires domain-specific modeling and infrastructure, which are beyond the scope of this work.\n\n> Computational complexity and scalability\n\nLoRT is designed with a focus on theoretical guarantees rather than computational efficiency. As noted in Line 428, it may face scalability challenges in high-dimensional settings, similar to tensor-based LoRA. Such challenges are common in tensor learning and arise from the inherent complexity of modeling high-dimensional structure. Addressing them is beyond the scope of this theory-oriented work, and we will clarify this in the final version. \n\n> Detection of highly heterogeneous source tasks \n\nOur framework assumes moderate heterogeneity, a standard setting in transfer learning. In practice, detecting large distribution gaps is important. Common approaches include computing divergence scores (e.g., MMD), domain classification, or local statistics. We will briefly discuss these options and their integration with LoRT (e.g., adaptive weighting) in the final version.\n\n> Stability of $W$ and norms of $W_k$\n\nThm 4.2 guarantees estimator stability, and Thm 4.3 refines the estimate via target-specific adaptation. While we do not analyze individual norms, this is an interesting extension which we will comment on in the revision.\n\n> Does D-LoRT require more data than LoRT?\n\nYes. Since D-LoRT transmits estimated parameters (not data), local models must be more accurate, requiring more samples per source task.", "claims_and_evidence": "1. The main focus of the paper is theoretical, with theoretical proofs. \n2. I have not found evidence in the supplementary code that the authors actually solved a distributed problem using real hardware nodes.\n3. Generalization with limited data - The paper supports this claim with experimental evidence from synthetic and real-world data. \nPerformance under distribution shifts and covariate shifts: This claim is further experimentally supported by an evaluation of synthetic data.\n4.  “Fusion regulariser enables effective knowledge transfer while accepting for model shits”.: Theoretically, this formulation would allow this, but there is a lack of experimental ablation on the effect of individual terms in the regulariser.", "methods_and_evaluation_criteria": "1. The paper presents Gaussian synthetic experiments and a real-world tensor completion experiment using the YUV RGB video dataset.\nOn the video dataset, the experimental section measures the reconstruction performance of a video tensor is given a limited number of input frames.\n\n- Using real data, the authors observe improved video reconstruction performance of both their local and distributed variants compared to a t-SVD-based approach by Lu et al.", "theoretical_claims": "The supplementary material is extensive. I superficially looked at the proofs, and they seemed okay to me.", "experimental_designs_or_analyses": "1. The theoretical section claims that $\\|W |\\_*$ induces a low-rank structure. Is there any experimental evidence to back up this claim?\n2. The proposed method uses tensor singular value decomposition. Other tensor decompositions, like CP, Tucker, and Tensor-Train, remain underexplored. Line 176 claims they can be easily replaced, but no such experiment is provided in the paper.\n3. Using average relative error seems a logical approach to evaluate the synthetic data results.\n4. In real-world data, the target task is to reconstruct a video frame from a very limited number of previous frames. The evaluation is limited to PSNR. Since this is one of the first works in this domain, it may be wise to incorporate other widely used evaluation metrics like SSIM and LPiPS to strengthen this baseline for future work.", "relation_to_broader_scientific_literature": "The paper relies on the t-SVD from kernfeld et al. While reading the paper, I was uncertain how or if the HOSVD from De Lathauwer et al. ( https://epubs.siam.org/doi/10.1137/S0895479896305696 ) was related?\n\nHow does the presented approach relate to Tensor Regression Networks ( https://www.jmlr.org/papers/volume21/18-503/18-503.pdf )?", "essential_references_not_discussed": "Most people rely on the extensive review by Kolda and Bader for the notation. It's nice that this paper also took inspiration from it, which makes the paper accessible. Overall, to the best of my knowledge, the related work is sufficiently discussed.", "other_strengths_and_weaknesses": "The authors present an interesting and novel theoretical approach to distributed tensor regression. This work is important, especially given the privacy concerns the machine learning community must respect in many countries.\n\nThe idea of transmitting regression parameters instead of data in the decentralized setting is interesting. It is intuitive that parameters have lower memory requirements compared to data. An increase in the model/tensor size or number of source nodes should allow this approach to scale well.", "other_comments_or_suggestions": "Clearly, lots of hard work went into this well-written paper.", "questions_for_authors": "- What potential additional real data sets would this framework allow us to work with?\n- Would it be possible to study the UK Biobank MRI data and compare this paper's approach to Kossaifi et al. ( https://www.jmlr.org/papers/volume21/18-503/18-503.pdf )?\n- What is the overall computational complexity - Similar to tensor-based LoRA [1], does the proposed method suffer from the curse of dimensionality?\n-  Assume the source and target data are highly different, potentially even from different distributions, is there a convenient way to detect such cases in the distributed setting?\n-   Both LoRT and D-LoRT use weighted averaging of the source and target tasks. Is the resulting W in equation 7 stable? Can we measure the norms for individual $W^k$?\n-  Line 340 states, “sample sizes are sufficiently large” for D-LoRT. Does this mean that D-LoRT generally requires more training data than LoRT?\n\n[1] Bershatsky, Daniel, et al. \"LoTR: Low tensor rank weight adaptation.\" arXiv preprint arXiv:2402.01376 (2024).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Az324nKlww", "reviewer": "Reviewer_DU3m", "summary": "In this paper, the authors propose a novel method called Low-Rank Tensor Transitions (LoRT), designed to address issues such as model shift, covariate shift, and decentralized data management in tensor regression for transfer learning. Experimental results demonstrate that LoRT significantly outperforms traditional methods like TNN and k-sup in terms of average correlation error.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and recognition of our contributions to robust tensor regression under distribution shift. We address the reviewer’s key concerns below.\n\n> **Novelty:** The core idea of LoRT lacks significant novelty and seems to be an incremental improvement.\n\nWe respectfully clarify that while LoRT builds on prior transfer learning principles, it tackles a **substantially more challenging and underexplored setting**—that of *tensor-valued regression under shift*. Unlike prior works which focus on vector-valued regression (e.g., TransFusion) or assume shared covariates across tasks, **LoRT generalizes fusion-based transfer learning to the tensor domain**, where:\n\n- The response and predictors are both high-order tensors.\n\n- Low-rank priors must be imposed across multiple tensor modes.\n\n- Task heterogeneity must be handled in decentralized, distribution-shifted settings.\n\nTo address these challenges, we propose a **new fusion regularizer based on the tubal nuclear norm**, a **two-step estimation procedure** specifically designed for tensors, and **estimation error bounds** derived under high-dimensional tensor regression settings.\n\nThese innovations are **not straightforward extensions of previous methods**, and to the best of our knowledge, LoRT is the **first theoretical framework** to jointly address model/covariate shift and low-rank tensor regression under multi-task and decentralized settings.\n\n> **Experimental diversity:** Experimental diversity is limited. More general datasets from computer vision should be used to test robustness.\n\nWe acknowledge the value of broader empirical validation and note that our experiments are carefully designed to support the paper’s theoretical goals. Specifically, we focus on compressed sensing and completion tasks, which provide a controlled setting for directly assessing parameter recovery and validating the theoretical results presented in Sections 4.2 and 4.3. These tasks are intentionally selected to align with our analytical objectives, and the results already provide sufficient evidence supporting our claims.\n\nTo address the reviewer’s suggestion, we have conducted preliminary experiments on two video clips (*Apply Eye Make-up* and *Blowing Candles*) from the UCF-101 dataset, which is a standard dataset for evaluating tensor methods in computer vision [R1]. These experiments follow the setup described in Appendix A.2, and preliminary results are available at (https://anonymous.4open.science/r/LoRT-113D/CVtable.png).\n\n[R1] Wang J, Zhao X. Functional Transform-Based Low-Rank Tensor Factorization for Multi-dimensional Data Recovery. ECCV 2024.\n\n> **Baseline comparisons:** Baseline comparisons only include methods published before 2021. Newer methods should be considered.\n\nWe thank the reviewer for the suggestion. To the best of our knowledge, this paper is the first to consider the proposed transferable tensor learning setting, and no existing methods are directly comparable. As explained in Footnote 6, our goal is to evaluate the benefit of transfer for tensor completion with very limited observations. In such extremely sparse regimes, all tensor formats (e.g., t-SVD, TT, TR, Tucker) are fundamentally limited by data availability, and their performance differences become marginal. Therefore, we selected TNN (Lu et al., 2019b) as a representative baseline to isolate the effect of transfer, rather than focus on fine-grained differences between tensor models.\n\nTo address the reviewer’s suggestion, we have also conducted comparisons with several recently proposed tensor completion methods [R2]–[R4]. Preliminary results are available  at (https://anonymous.4open.science/r/LoRT-113D/CVtable.png), and we will clarify this point in the final version.\n\n[R2] Qiu Y, et al. Balanced unfolding induced tensor nuclear norms for high-order tensor completion. IEEE TNNLS, 2024.\n\n[R3] Wang A, et al. Noisy tensor completion via orientation invariant tubal nuclear norm. PJO, 2023.\n\n[R4] Tan Z, et al. Non-convex approaches for low-rank tensor completion under tubal sampling. IEEE ICASSP, 2023.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Yes. The experimental design is not very reasonable.\n1. The experimental scenarios are somewhat limited, lacking diverse validation. To more comprehensively evaluate the effectiveness of the LoRT method, it is recommended to conduct extensive tensor regression tests on general datasets in the field of computer vision, thereby verifying its universality and robustness across different tasks.\n2. The baseline methods compared in the paper were all published before 2021, which somewhat limits the timeliness and persuasiveness of the experimental results. To more convincingly demonstrate the superiority of the LoRT method, it is advisable to compare it with state-of-the-art methods proposed in recent years, thereby more accurately reflecting its competitiveness in the current research landscape.", "relation_to_broader_scientific_literature": "None", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "In this paper, the authors propose a novel method called Low-Rank Tensor Transitions (LoRT), designed to address issues such as model shift, covariate shift, and decentralized data management in tensor regression for transfer learning. Experimental results demonstrate that LoRT significantly outperforms traditional methods like TNN and k-sup in terms of average correlation error. Although the theoretical proofs in the paper are relatively comprehensive, several issues warrant further exploration:\n1. Although the LoRT method exhibits strong performance in experiments, its core idea lacks significant novelty. The application of tensor regression and transfer learning has already been well-established in prior research, and LoRT appears to be more of an improvement on existing techniques rather than a groundbreaking innovation.\n2. The experimental scenarios are somewhat limited, lacking diverse validation. To more comprehensively evaluate the effectiveness of the LoRT method, it is recommended to conduct extensive tensor regression tests on general datasets in the field of computer vision, thereby verifying its universality and robustness across different tasks.\n3. The baseline methods compared in the paper were all published before 2021, which somewhat limits the timeliness and persuasiveness of the experimental results. To more convincingly demonstrate the superiority of the LoRT method, it is advisable to compare it with state-of-the-art methods proposed in recent years, thereby more accurately reflecting its competitiveness in the current research landscape.", "other_comments_or_suggestions": "None.", "questions_for_authors": "1. Although the LoRT method exhibits strong performance in experiments, its core idea lacks significant novelty. The application of tensor regression and transfer learning has already been well-established in prior research, and LoRT appears to be more of an improvement on existing techniques rather than a groundbreaking innovation.\n2. The experimental scenarios are somewhat limited, lacking diverse validation. To more comprehensively evaluate the effectiveness of the LoRT method, it is recommended to conduct extensive tensor regression tests on general datasets in the field of computer vision, thereby verifying its universality and robustness across different tasks.\n3. The baseline methods compared in the paper were all published before 2021, which somewhat limits the timeliness and persuasiveness of the experimental results. To more convincingly demonstrate the superiority of the LoRT method, it is advisable to compare it with state-of-the-art methods proposed in recent years, thereby more accurately reflecting its competitiveness in the current research landscape.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7N1JWgel9f", "reviewer": "Reviewer_jygk", "summary": "This paper addresses the challenge of data scarcity in the target task within Tensor Regression. The authors propose a novel transfer learning framework called Low-Rank Tensor Transitions (LoRT) to tackle this issue. LoRT employs a two-stage adaptation process to mitigate model shift and enhance adaptation to the target task.\nIn the first stage, Joint Low-Rank Learning (JLL), the framework simultaneously optimizes both the target and source tasks by leveraging a low-rank constraint across multiple datasets. This stage also incorporates a weighted averaging approach to refine parameter estimation.\nIn the second stage, Target-Specific Refinement (TSR), the parameters obtained from the first stage are further adapted by introducing target-specific learning components. A key aspect of this stage is that the low-rank constraint is applied only to the target-specific components, preventing overfitting to the target data while maintaining an optimal balance with the source task information.\nAdditionally, the paper provides theoretical guarantees for LoRT, mathematically delineating the conditions under which source task knowledge can be effectively leveraged. Furthermore, the authors extend LoRT to a distributed data environment by introducing D-LoRT, which minimizes communication overhead by locally performing tensor regression and subsequently integrating the models.\nThe proposed framework is empirically validated on Compressed Sensing and Tensor Completion tasks, demonstrating superior performance compared to conventional approaches.\n\n## update after rebuttal\n\nThank you for the thorough rebuttal. I understand your points regarding the apparent existence of valid applications and the mention of addressing non-i.i.d. cases as future work. I am satisfied with this response, so I will leave my evaluation unchanged. I mistakenly posted my comment in the Official Comment section. My apologies.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful and constructive feedback. We appreciate the recognition of our theoretical framework, decentralized extension, and empirical evaluations. Below, we address the main concerns:\n\n> **Concern 1:** Conducting evaluations on real-world problems where Tensor Compressed Sensing is applicable could strengthen the claim regarding the practical utility of the proposed method. What are some real-world problems related to Tensor Compressed Sensing? Can the proposed method effectively address them?\n\nTensor Compressed Sensing (TCS) arises in various real-world scenarios where acquiring full tensor data is costly or constrained. Prominent examples include:\n\n- Magnetic Resonance Imaging (MRI): Reconstructing high-resolution 3D volumes from a limited number of Fourier measurements.\n\n- Hyperspectral Imaging: Recovering spatial-spectral data cubes under limited sampling due to hardware or bandwidth constraints.\n\nOur method is theoretically well-suited to these scenarios: it leverages both low-rank structures and transferable information from source tasks to enhance reconstruction quality in the presence of data scarcity and distribution shifts.\n\nIn the current paper, we focus on theoretical development and controlled validation. The TCS experiments on synthetic Gaussian measurements are explicitly designed to validate the generalization guarantees and the low-rank transfer mechanism presented in Sections 4.2 and 4.3. These settings enable precise evaluation of recovery error and alignment with theoretical predictions.\n\nMoreover, this design follows a well-established practice in the tensor learning theory literature. Many prior theoretical works (e.g., Lu et al., 2018; Wang et al., 2021) also provide guarantees under compressed sensing assumptions, while using real data primarily for completion tasks.\n\nWe fully respect the reviewer’s suggestion that applying the method to real-world TCS data could further support its practical relevance. However, such experiments are not essential to validate our main theoretical claims, and would require additional modeling assumptions (e.g., real sensing matrices, noise models) and substantial engineering efforts that fall outside the scope of this theory-focused work. Due to the limited rebuttal timeframe, adding real-world TCS experiments is unfortunately not feasible. We will clarify this rationale in the final version and discuss such extensions as promising directions for future research.\n\n> **Concern 2:** Are there cases where the i.i.d. assumption on the data is difficult to satisfy?\n\nYes, and we appreciate the reviewer raising this important point. While our current theory assumes that samples within each task are i.i.d., this is a **standard assumption** adopted in many theoretical works on transfer learning and tensor regression (e.g., Zhang et al., 2020; Qiu et al., 2022a). It facilitates a clear derivation of estimation bounds and highlights the key effects of low-rank structure and distribution shift.\n\nWe acknowledge, however, that real-world data often exhibit dependencies or structured sampling patterns that violate the i.i.d. assumption. Extending our theoretical framework to accommodate such cases is a promising and practically relevant direction for future work.\n\nAs an initial step in this direction, we have conducted preliminary experiments on tensor completion under non-i.i.d. settings, particularly involving mixed tube-wise and element-wise missing patterns (https://anonymous.4open.science/r/LoRT-113D/NIIDtable.png). We will provide a detailed discussion in the final version.\n\n> **Concern 3**: In the definition of t-SVD in Definition 2.1, shouldn't the size of  $C$ be $d_1 \\times d_4$?\n\nThank you for catching this typo. We confirm the correct dimension is $d_1 \\times d_4$ and will make the correction in the final version.", "claims_and_evidence": "The proposed novel transfer learning framework with a two-stage adaptation process is theoretically well-justified based on assumptions about data distribution. Additionally, a reasonable algorithm is presented for application in a distributed data environment. Furthermore, the effectiveness of the proposed method is validated through appropriate evaluation experiments.", "methods_and_evaluation_criteria": "The proposed method is a transfer learning approach designed to leverage knowledge from source tasks to compensate for data scarcity in the target task. Consequently, Tensor Compressed Sensing (TCS), which reconstructs the original tensor from partially observed linear measurements, appears to be a more natural problem setting for evaluating the proposed method compared to Tensor Completion (TC), where only some entries are missing.\nHowever, the evaluation of Tensor Compressed Sensing in this study is based solely on synthetic data. Conducting evaluations on real-world problems where Tensor Compressed Sensing is applicable could strengthen the claim regarding the practical utility of the proposed method.", "theoretical_claims": "I have reviewed the theoretical guarantees of LoRT (Section 4.2) and D-LoRT (Section 4.3). Under the assumption of a Gaussian distribution, the theoretical guarantees appear to be correctly derived. However, there is room for discussion regarding the validity of the i.i.d. assumption across a wide range of tensor data.", "experimental_designs_or_analyses": "As mentioned in the Methods and Evaluation Criteria section.", "relation_to_broader_scientific_literature": "This paper is broadly related to tensor regression, particularly recent tensor learning techniques that incorporate low-rank constraints. Additionally, it is closely connected to the field of transfer learning, providing theoretical guarantees for effectively applying transfer learning to tensor data.", "essential_references_not_discussed": "This paper focuses on transfer learning for tensor regression with low-rank constraints. As far as I am aware, the relevant prior studies for comparison have been appropriately cited.", "other_strengths_and_weaknesses": "As mentioned earlier, this paper has a strong contribution in proposing a novel transfer learning framework with theoretical guarantees to address the issue of data scarcity in Tensor Regression. Additionally, a concern is that Tensor Compressed Sensing, which appears to be a more natural problem setting for evaluating the proposed method, is only assessed using synthetic data. Evaluating it on real-world problems could strengthen the validity of the method. Another point of discussion is the assumption that the data are i.i.d., which may not always be valid across different datasets, and further examination of its appropriateness would be beneficial.", "other_comments_or_suggestions": "In the definition of t-SVD in Definition 2.1, shouldn't the size of $\\underline{C}$ be $d_1 \\times d_4 \\times d_3$?", "questions_for_authors": "1. What are some real-world problems related to Tensor Compressed Sensing? Can the proposed method effectively address them?\n\n2. Are there cases where the i.i.d. assumption on the data is difficult to satisfy?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eOqcuX8W9P", "reviewer": "Reviewer_bSAD", "summary": "This paper proposes the Low-Rank Tensor Transitions (LoRT) framework to address various shift problems and decentralized data management. LoRT employs a novel fusion regularizer to enforce low-tubal-rank solutions, enabling effective integration. Its two-step refinement process mitigates model shifts and ensures robust adaptation to target tasks. For decentralized scenarios, the authors extend LoRT to D-LoRT, a distributed variant that preserves statistical efficiency. The authors provide detailed theoretical analysis for each step. \n\nIn experiments, the authors demonstrate the superiority of the two proposed frameworks in simulated settings and experiments based on YUV RGB video datasets. Overall, this paper explores transfer learning in tensor regression, providing rich content, detailed theory, and numerous experiments. However, the LoRT and D-LoRT frameworks proposed in this paper are largely based on the TransFusion framework [1], with similar algorithmic concepts and theoretical analysis. This significantly diminishes the originality of the paper.\n\n[1] He, Z., Sun, Y., & Li, R. (2024, April). Transfusion: Covariate-shift robust transfer learning for high-dimensional regression. In International Conference on Artificial Intelligence and Statistics (pp. 703-711). PMLR.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the constructive and thoughtful feedback.\nWe are encouraged that the reviewer finds our theoretical analysis and experimental design sound, and recognizes the potential of the proposed LoRT frameworks for addressing distribution shifts in tensor regression. Below, we respond to the reviewer’s concerns point by point.\n\n> On similarity to TransFusion \n\nWhile LoRT is conceptually inspired by TransFusion [1], it addresses a fundamentally different and more challenging setting: **tensor-valued regression**, where both inputs and responses are high-order tensors. This introduces unique challenges:\n\n1. **Multi-mode low-rank structure**: LoRT employs a **tubal nuclear norm-based fusion regularizer** to model low-rank structure along tensor modes, in contrast to the $\\ell_1$-based fusion in vector settings like TransFusion.\n\n2. **Tensor-specific two-step estimation**: Although both methods use two-step strategies, **LoRT’s design is tailored to tensors**, first estimating a shared low-tubal-rank component, then refining task-specific parts using tensor algebra.\n\n3. **Theoretical analysis under tensor structure**: LoRT establishes **estimation error bounds** based on tensor-specific assumptions and tools, differing from the vector-based analyses in TransFusion.\n\nThese aspects make LoRT a **nontrivial and technically distinct extension** of fusion-based transfer learning to the tensor domain.\n\n> On comparison with CP/Tucker/convex baselines\n\nWe have conducted initial comparisons with representative CP-, Tucker-, and convex-based tensor methods under our experimental setting (https://anonymous.4open.science/r/LoRT-113D/CPtable.png) and will include full results in the final version.\n\n> On real data without known parameters\n\n Our current study focuses on settings with known tensor coefficients, which allow precise assessment of parameter recovery and direct validation of our theoretical predictions. This experimental design is a standard and widely accepted practice in the tensor learning literature (e.g., Zhang et al., 2020; Wang et al., 2021), especially for theory-driven studies. By working with controlled synthetic data, we can rigorously examine the behavior of the low-rank transfer mechanism and its consistency with our generalization bounds.\n\nWe respect and appreciate the reviewer’s perspective that experiments on real-world data without ground-truth parameters can demonstrate practical relevance. However, such experiments serve a different purpose and are not necessary for validating the core theoretical contributions of this work. As our current goal is to establish and verify theoretical guarantees, we follow the conventional evaluation protocol in this area. We recognize the value of such experiments for illustrating broader applicability and consider them a promising direction for future work.\n\n> On computational details\n\nWe will include further algorithmic details—such as per-iteration computational cost and implementation notes—in the final version to improve clarity and reproducibility. We note that we have already provided runnable example code in the supplementary material, and we will expand the documentation to make the computational aspects more transparent.\n\n> On notation and typos\n\nMany thanks for your careful reading! We will revise the notation system for better readability and fix all typographical errors. \n\n> On \"no negative transfer\" guarantee\n\nOur current theory guarantees improvement under moderate heterogeneity, which aligns with prior work (e.g., [1]), as universal guarantees are generally infeasible without adaptive mechanisms.\n\nThat said, the structural design of LoRT naturally enables \"no negative transfer\"-style extensions. Specifically, the fusion regularizer allows for task-wise reweighting, and the divergence scores $h_k$ are available during training. This makes it feasible to implement heterogeneity-aware fusion, similar in spirit to [2], where adaptive regularization balances between source pooling and target-only training.\n\nWhile our current focus is on estimation error in favorable regimes, we plan to explore an oracle-type guarantee of the form\n$$\n\\mathbb{E}[\\text{Err}_{\\text{LoRT}}] \\le \\\\min\\\\{\\text{Err}\\_{\\text{target-only}}, \\text{Err}\\_{\\text{fusion}}\\\\} + \\delta,\n$$\nwhere $\\delta$ depends on heterogeneity. We will add a discussion of this future direction in the final version.\n\n> On the assumption $r \\ll d_2$\n\n The assumption aligns with the low tubal-rank setting commonly used in t-SVD-based tensor models (e.g., Qiu et al., 2022a). Note that we assume $d_1 \\ge d_2$ without loss of generality (line 111), so the rate $O(r d_1 d_3 N^{-1})$ can be more precisely written as \n$$O(r \\min\\\\{d_1, d_2\\\\} d_3 N^{-1})$$ Hence, the dependence on $d_2$ is implicitly included via both $\\min\\\\{d_1, d_2\\\\}$ and $r$.", "claims_and_evidence": "The LoRT and D-LoRT frameworks proposed in this paper have been thoroughly validated both theoretically and experimentally, but there are still some shortcomings:\n\n1. **Theoretical Issues**: The authors prove in both LoRT and D-LoRT how the estimation error is affected by the heterogeneous measure $\\bar h$. However, there is an obvious issue: when $\\bar h$ becomes large enough, the current theory cannot guarantee that the transfer learning performance will always outperform using only target data. Referring to the theory in [2], it would be ideal to ensure that in any case, the performance after transfer is at least as good as using only the target data.\n\n2. **Experimental Issues**: The authors have done extensive work and presented many numerical results. In the completion performance, the recovery of the ground truth is quite good. However, the experiments provided are all based on simulations, assuming the true tensor coefficients are known. The authors should include real data analysis using actual tensor covariates $X$ and response $y$ for analysis (without knowing the true $W$). This would make the findings more convincing.\n\n[2] Duan, Y., & Wang, K. (2023). Adaptive and robust multi-task learning. The Annals of Statistics, 51(5), 2015-2039.", "methods_and_evaluation_criteria": "In the experiments, the authors mainly compared TNN and $k$-Sup. However, there are many other comparative methods for tensor regression on a single dataset, such as tensor regression based on CP decomposition/Tucker decomposition [3,4] or tensor regression with convex regularization [5]. The authors did not evaluate the performance of these methods.\n\n[3] Zhou, H., Li, L., & Zhu, H. (2013). Tensor regression with applications in neuroimaging data analysis. Journal of the American Statistical Association, 108(502), 540-552.\n\n[4] Li, X., Xu, D., Zhou, H., & Li, L. (2018). Tucker tensor regression and neuroimaging analysis. Statistics in Biosciences, 10(3), 520-545.\n\n[5] Raskutti, G., Yuan, M., & Chen, H. (2019). Convex regularization for high-dimensional multiresponse tensor regression.", "theoretical_claims": "I have read the proof in the author's appendix, which is very detailed. I believe it is feasible, but the specific details require further reading.", "experimental_designs_or_analyses": "I believe the soundness/validity of any experimental designs or analyses is feasible. The authors have conducted experiments on both simulated and real data. However, the experiments provided so far are all based on simulations, assuming the true tensor coefficients are known. The authors should include real data analysis using actual tensor covariates and response $y$ for analysis (without knowing the true $W$). This would make the findings more convincing.", "relation_to_broader_scientific_literature": "Overall, this paper explores transfer learning in tensor regression, providing rich content, detailed theory, and numerous experiments. However, the LoRT and D-LoRT frameworks proposed in this paper largely follow the TransFusion framework [1], with very similar algorithmic concepts and theoretical analysis. This significantly diminishes the originality of the paper.", "essential_references_not_discussed": "The authors provide a detailed discussion of the literature on transfer learning, tensor regression/recovery, and multi-task learning (MTL), which is very thorough.", "other_strengths_and_weaknesses": "**Strengths**: As discussed above.\n\n**Weaknesses**:\n1. This paper explores transfer learning in tensor regression, providing rich content, detailed theory, and numerous experiments. However, the LoRT and D-LoRT frameworks proposed in this paper largely follow the TransFusion framework [1], with very similar algorithmic concepts and theoretical analysis. This significantly diminishes the originality of the paper. The authors could emphasize the challenges of extending TransFusion to tensor regression and how they address these challenges in theory.\n\n2. The paper does not mention any computational details. Tensor regression is computationally challenging, and the authors should explain the details of the algorithm and its computational cost.\n\n3. The paper uses a large number of symbols, many of which are superscripts and subscripts. This makes the paper difficult to read. I suggest that the authors reorganize the symbols, for example, by representing three-dimensional tensors as $\\mathcal X$ and $\\mathcal W$ to reduce the use of superscripts and subscripts.\n\n4. There are some obvious typos in the paper, such as in Equation 6, where it should be $y^k,X^k$ instead of $y^0, X^0$.\n\n5. In the theoretical analysis, when $\\bar h$ becomes large enough, the current theory cannot guarantee that the transfer learning performance will always outperform using only target data. Referring to the theory in [2], it would be ideal to ensure that in any case, the performance after transfer is at least as good as using only the target data.", "other_comments_or_suggestions": "The authors could reorganize the symbols used in the paper to make it more readable.", "questions_for_authors": "1. In line 163, why is the assumption $ r \\ll d_2 $ made? Is this assumption necessary, and does it have any connection to the final rate $ O(r d_1 d_2 N^{-1}) $, where only $ d_1 $ and $ d_3 $ appear but not $ d_2 $?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This papers introduces a framework of transfer learning in tensor regression. It extends to the tensor setting the work of He et al. (2024) on  covariate-shift transfer learning using regularization strategies promoting solutions with low tubal rank. It also proposes a distributed learning extension of the framework to deal with decentralized data scenarios. This is a solid paper which investigates a topic of theoretical and practical importance and provides nice contributions to the literature on tensor regression. Some concerns have been raised by the reviewers about the originality of the work, the experimental evaluation and the risk of negative transfer. Authors in their rebuttal have provided reasonable explanation and additional results for these points. However, in order to discuss a non-negative transfer extension of the framework, they provided a pdf file with additional text, which is not compliant with the Author Response policy. As this extension is out of the scope of the paper, I recommend acceptance.
Decision: Accept (poster)