Title: Generalized Tensor Decomposition for Understanding Multi-Output Regression under Combinatorial Shifts
Venue: NeurIPS 2024
Accepted: yes
Keywords: multi-output regression

tensor singular value decomposition

tensor completion
Abstract: In multi-output regression, we identify a previously neglected challenge that arises from the inability of training distribution to cover all combinations of input features, leading to combinatorial distribution shift (CDS). To the best of our knowledge, this is the first work to formally define and address this problem. We tackle it through a novel tensor decomposition perspective, proposing the Functional t-Singular Value Decomposition (Ft-SVD) theorem which extends the classical tensor SVD to infinite and continuous feature domains, providing a natural tool for representing and analyzing multi-output functions. Within the Ft-SVD framework, we formulate the multi-output regression problem under CDS as a low-rank tensor estimation problem under the missing not at random (MNAR) setting, and introduce a series of assumptions about the true functions, training and testing distributions, and spectral properties of the ground-truth embeddings, making the problem more tractable.
To address the challenges posed by CDS in multi-output regression, we develop a tailored Double-Stage Empirical Risk Minimization (ERM-DS) algorithm that leverages the spectral properties of the embeddings and uses specific hypothesis classes in each frequency component to better capture the varying spectral decay patterns. We provide rigorous theoretical analyses that establish performance guarantees for the ERM-DS algorithm. This work lays a preliminary theoretical foundation for multi-output regression under CDS.

Generalized Tensor Decomposition for Understanding Multi-Output Regression under Combinatorial Shifts

Abstract

In multi-output regression, we identify a previously neglected challenge that arises from the inability of training distribution to cover all combinations of input features, leading to combinatorial distribution shift (CDS). To the best of our knowledge, this is the first work to formally define and address this problem. We tackle it through a novel tensor decomposition perspective, proposing the Functional t-Singular Value Decomposition (Ft-SVD) theorem which extends the classical tensor SVD to infinite and continuous feature domains, providing a natural tool for representing and analyzing multi-output functions. Within the Ft-SVD framework, we formulate the multi-output regression problem under CDS as a low-rank tensor estimation problem under the missing not at random (MNAR) setting, and introduce a series of assumptions about the true functions, training and testing distributions, and spectral properties of the ground-truth embeddings, making the problem more tractable. To address the challenges posed by CDS in multi-output regression, we develop a tailored Double-Stage Empirical Risk Minimization (ERM-DS) algorithm that leverages the spectral properties of the embeddings and uses specific hypothesis classes in each frequency component to better capture the varying spectral decay patterns. We provide rigorous theoretical analyses that establish performance guarantees for the ERM-DS algorithm. This work lays a preliminary theoretical foundation for multi-output regression under CDS.

1 Introduction

In the realm of Multi-Output Regressions (MOR), systems are designed to predict multiple related outputs from a set of input features, unlike single-output regressions which neglect the relationship between targets. MOR is effective for various scientific problems [55] such as river quality prediction [17], natural gas demand forecasting [5] and drug efficacy prediction [30]. In many MOR applications, it is interactions among multiple features that the predictions are typically derived from. For example, to predict multiple outcomes such as risk scores and health outcomes of patients, the input features usually include healthcare costs and demographic variables (e.g., race) [40]. Similarly, predictions

Mingyuan Bai

RIKEN AIP

Guoxu Zhou ∗

to new climate scenarios such as increased temperature and extreme weather events require new representative concentration pathways [12]. To enhance the generalization capability, it is often necessary to increase the diversity of training samples. However, in some scenarios, data from specific interactions remain inaccessible, such as risk scores of multiple diseases involving novel combinations of factors including age, physiological indicators, and biochemical markers (see Figure 1). Therefore, unseen feature combinations impose a significant challenge to precise predictions, especially when involving multiple outputs.

To this end, in this paper, we address this issue by reducing it to only a couple of input features. This issue is known as the Combinatorial Distribution Shift (CDS) problem [46]. Under CDS, the training data are i.i.d. sampled from the training distribution D train with corresponding label z i , where ( x i , y i ) ⊂ X × Y are the input feature combinations from the feature domains X and Y , and z i = h ⋆ ( x i , y i ) ∈ R K denotes the K outputs determined by the vector-valued ground truth function h ⋆ . Then the problem reduces to predict the new interactions of input combinations ( x, y ) sample from D test , where the probability density or mass of test samples ( x, y ) under D train may be zero. Therefore, the goal of the MOR problem becomes: how can we generalize to the new feature combinations that have never appeared in the training distribution?

It should be noticed that the MOR community has low awareness of CDS and current methods struggle to address these challenges. Traditional models dependent on static datasets, fail to capture the variability in practical applications, making them ineffective with new data combinations. This research gap underscores the urgent need for dynamic MOR models that better adapt to the evolving input landscapes seen in real-world deployments.

To address this bottleneck, we formulate the issue as a generalized tensor completion problem, arranging the predictions into a third-order tensor Z ( x, y, :) = h ⋆ ( x, y ) ∈ R K . Consequently, the prediction of test data can be intuitively interpreted as vector-valued predictions for Missing Not At Random (MNAR) [38] scenarios in multivariate function (see Eq. (4) for more details). While low-rank tensor decomposition techniques, particularly tensor Singular Value Decomposition (t-SVD) [27, 26, 61], have proven effective in recovering missing entries for discrete data, adapting these methods to the complex domain of possibly continuous multivariate functions requires a fundamental

Temperature

Training

Distribution

Test

Distribution

Blood pressure

h

Multiple risk scores

h*(x,y) ∈ IRK

re-establishment of the existing t-SVD framework to address the inherent challenges posed by this transition.

Our contributions. This paper proposes a novel tensor decomposition perspective for multi-output regression under CDS. It analyzes spectral properties that enable handling such shifts, and introduces an algorithm with theoretical guarantees to improve prediction accuracy. The key contributions are:

To the best of our knowledge, we are the first to identify and define the problem of multi-output regression under CDS, revealing the challenges that arise when the training distribution fails to cover all combinations of input features.

We propose a theoretical framework of Functional Tensor Singular Value Decomposition (Ft-SVD), which extends the classical t-SVD to infinite and continuous feature domains, providing a natural tool for representing and analyzing multi-output functions.

Under the Ft-SVD framework, we formulate the multi-output regression problem under CDS as a low-rank tensor estimation problem under the MNAR setting, and introduce a series of assumptions about the true functions, training and testing distributions, and spectral properties of the embeddings, making the problem more tractable.

We develop a tailored Double-Stage Empirical Risk Minimization (ERM-DS) algorithm that uses specific hypothesis classes in each sub-domain to better capture the spectral decay patterns across different sub-domains, and provide theoretical guarantees for the algorithm's performance under CDS.

Brief related work. Our work is related to the areas of multi-output regression, tensor completion, and learning under distribution shift. Multi-output regression has been studied extensively, with approaches ranging from multi-task learning [13, 6] to shared representation learning [39], linear models [11], kernel methods [2], and neural networks [9]. However, these methods often assume that the training and test data come from the same distribution. Tensor completion has seen advancements with various decomposition techniques [24, 50, 41] and methods for handling non-random missing patterns [54, 14], but they do not consider the specific challenges of multi-output regression under CDS. Various types of distribution shifts, such as covariate shift [45], concept drift [19], and domain adaptation [44], have been studied, with recent work addressing distribution shift in multi-output regression by aligning distributions through invariant representations [33]. Recent functional tensor decomposition advancements include spectral tensor-train for high-dimensional function evaluations [7], Tucker-neural network for data recovery [37], Bayesian CP/Tucker approach for uncertainty quantification [18], guaranteed functional CP decomposition for functional data analysis [23], and functional Tensor-Train for efficient multivariate function representation [20]. Equivariant disentangled transformation for domain generalization under combination shift was studied through category theory [59]. Simchowitz et al. [46] proposed the ERM-DS framework for robust learning under CDS. Our work extends the ERM-DS framework to multi-output regression by representing vector-valued functions as embeddings in a Hilbert t-module and leveraging tensor algebra to capture interdependencies among multiple outputs.

Notations. We use lowercase boldface, and uppercase boldface letters to denote vectors, e.g. , a ∈ R m , and matrices, e.g. , A ∈ R m × n , respectively. A 3-way tensor of size 1 × 1 × K is also named a t-scalar , denoted by underlined lowercase, e.g. , t , a 3-way tensor of size d × 1 × K is called a t-vector and denoted by underlined lowercase boldface, e.g. , t , whereas a 3-way tensor of size m × n × K is also called a t-matrix and denoted by underlined uppercase, e.g. , T . For any tensor T ∈ R m × n × K , we use T ( i ) to denote its i -th frontal slice. For A ∈ R m × m , σ j ( A ) ≥ 0 denotes its j -th largest singular value; for symmetric A , λ j ( A ) denotes its j -th largest eigenvalue.

2 Functional t-Singular Value Decomposition for Multi-output Regression

2.1 A t-SVD Perspective of Multi-output Regression

Consider a simpler case where the feature domains X and Y are finite sets. Here, we can represent the vector-valued ground truth h ⋆ as a tensor Z ∈ R |X|×|Y|× K , where each tube Z ( i x , j y , :) = h ⋆ ( x, y ) ∈ R K corresponds to the K outputs for the input feature combination ( x, y ) indexed by ( i x , j y ) . Then, the multi-output regression problem for a new feature combination ( x ′ , y ′ ) becomes a tensor completion task, where the goal is to estimate the missing tube of Z at the index ( i x ′ , j y ′ ) .

To tackle general tensor completion problems, the framework of t-SVD [26, 27] offers an ideal tool [35, 32, 58]. The motivation behind t-SVD stems from the observation that under certain linear transformations M , tensors may exhibit stronger low-rank characteristics than in their original domain. This enhancement of low-rankness often arises due to intrinsic correlations within the data, which these transformations can exploit more effectively. Recent research has focused on using an orthogonal matrix M to define the transform M due to its advantageous properties [34, 51], a convention that this paper also adopts 2 . Given an orthogonal matrix M ∈ R K × K , we define the associated linear transform M ( · ) and its inverse M -1 ( · ) on any tensor T ∈ R m × n × K as follows:

where × 3 denotes the mode-3 tensor-matrix product [25]. In real applications, the choice of the transformation matrix M is often guided by the inherent characteristics of the signal being modeled. Popular choices include the Discrete Cosine Transform (DCT) matrix for smooth and periodic signals [34, 57, 32], the Discrete Wavelet Transform (DWT) matrix for multi-resolution analysis [49], data-dependent transformations that adapt to the dataset's specific characteristics [58], and graph spectral projection matrices for data structured in non-Euclidean spaces [16]. By selecting a transformation matrix that aligns with the signal properties, t-SVD can more effectively uncover the low-rank structure and enable efficient representation and processing of the data.

Based on the linear transform M , the t-product is specifically defined.

Definition 1 (t-product [25]) . The t-product of tensors A ∈ R m × n × K and B ∈ R n × k × K under the transform M in Eq. (1) is denoted by A ∗ M B and defined as the tensor C ∈ R m × k × K such that M ( C ) = M ( A ) ⊙ M ( B ) in the transformed domain, where ⊙ denotes the tensor frontal-slice-wise product (see Definition 4).

This paper also follows the definitions of t-transpose, t-identity tensor, t-orthogonal tensor, and f-diagonal tensor given by [25]. Based on these definitions, the t-SVD is introduced as follows:

Definition 2 (t-SVD, tubal rank [25, 27]) . The tensor singular value decomposition (t-SVD) of T ∈ R m × n × K under the transform M in Eq. (1) is given by:

where U ∈ R m × m × K and V ∈ R n × n × K are t-orthogonal tensors, S ∈ R m × n × K is an f-diagonal tensor, and ( · ) ⊤ denotes the t-transpose. The tubal rank of tensor T is defined as the number of non-zero tubes in S in Eq. (2) , i.e.,

̸

Based on t-SVD, extensive tensor completion models [32, 58, 52] provide robust support for MOR with discrete combinatorial features. However, in many machine learning settings, feature domains X and Y are infinite and potentially continuous sets, posing significant challenges to the traditional t-SVD. Here, t-SVD becomes inapplicable due to the potentially non-discrete nature of these domains, necessitating a novel approach to effectively handle infinite and continuous feature domains. This leads us to consider: how can we extend the powerful t-SVD framework to address multi-output regression problems in the context of infinite and continuous feature domains?

2.2 Functional t-Singular Value Decomposition

To address the above challenge, we propose a novel theoretical framework that extends the foundational principles of t-SVD for infinite and continuous feature domains. By introducing a new theorem, we enable the representation of data and functions defined on these domains while preserving the key properties of t-SVD. This extension allows us to employ t-SVD in a principled way for learning vector-valued functions and tackling related problems, opening new possibilities for applying such methods to a wider range of learning tasks.

Theorem 1 (Functional t-Singular Value Decomposition) . Let F : X × Y → R K be a squareintegrable vector-valued function with Lipschitz-smooth domains X ⊂ R D 1 and Y ⊂ R D 2 . Then, there exist sets of functions { ϕ i } ∞ i =1 ⊂ L 2 ( X ; R K ) and { ψ i } ∞ i =1 ⊂ L 2 ( Y ; R K ) , and a sequence of t-scalars 3 { σ i } ∞ i =1 ⊂ N K with lim i →∞ σ i = 0 , satisfying the functional t-Singular Value Decomposition (Ft-SVD):

where the convergence is in the L 2 sense. The functions ϕ i and ψ i are called the left and right t-singular functions, respectively, and the t-scalars σ i are called the t-singular values. The orthonormality conditions ∫ X ϕ i ( x ) ∗ M ϕ j ( x ) dx = δ ij M -1 (1) and ∫ Y ψ i ( y ) ∗ M ψ j ( y ) dy = δ ij M -1 (1) hold, where 1 ∈ R 1 × 1 × K is the t-scalar with all entries equal to 1, and δ ij is the Kronecker delta.

The proof is provided in Appendix B.2.1. The Ft-SVD theorem provides a principled conceptual framework for decomposing a multivariate function, which maps pairs of inputs from domains X and Y into a vector in R K . Essentially, this theorem states that such a function can be expressed as an infinite series of products of three components: t-singular functions from X and Y ( ϕ i and ψ i , respectively), and a series of t-singular values ( σ i ). This decomposition is analogous to breaking down a complex multivariate relationship into simpler, interpretable modes of variation, where each mode is scaled by its importance, signified by the corresponding t-singular value σ i .

Unlike t-SVD, which factorizes a tensor using a finite number of t-rank-one components, representing the original function exactly using Ft-SVD may require an infinite number of t-rank-one components in Eq. (3). However, in most practical applications, an approximation with a finite number of components is often desired due to computational constraints and the need for a more compact representation. The natural follow-up inquiry is: how can we achieve a finite approximation of the original function using Ft-SVD ?

To address this question, we can leverage principles akin to the Eckart-Young theorem for t-SVD [27], which identifies the best low-tubal-rank tensor approximation by minimizing the Frobenius norm error. Building upon these principles, we demonstrate that an optimal rankr approximation also exists for functions represented using Ft-SVD (see the proof in Appendix B.2.2):

Theorem 2 ( r -term truncated Ft-SVD) . Let F : X × Y → R K be a square-integrable function with the Ft-SVD given by Theorem 1. For any r ∈ N , the r -term truncated Ft-SVD is defined as

Then, F r ( x, y ) is the best r -term approximation to F ( x, y ) in the L 2 sense within Ft-SVD framework. Moreover, the approximation error is given by: ∥ F ( x, y ) -F r ( x, y ) ∥ 2 L 2 ( X×Y ; R K ) = ∑ ∞ i = r +1 ∥ σ i ∥ 2 .

This theorem highlights the optimality of the truncated Ft-SVD in approximating vector-valued functions, making it a potentially useful tool for function compression, denoising, and other applications involving low-rank approximations of functions. However, the error term, an infinite sum ∑ ∞ i = r +1 ∥ σ i ∥ 2 2 , may still be difficult to bound in the worst case. This raises the question: are there situations where the error terms are well-controlled?

In many practical scenarios, the inherent spatial or temporal correlations within data often lead to significant smoothness in functions, a trait crucial for applications in image processing, machine learning, computer vision, climate modeling, and time series analysis, and fluid dynamics [15, 36, 3, 4, 29, 42]. This smoothness property can be leveraged to show that the r -term approximation error of Ft-SVD is indeed well-controlled. The following Theorem 3 shows that if the function components belong to a Sobolev space H s ( Y ) , which captures their smoothness, then the t-singular values of the function decay rapidly. This rapid decay effectively controls the approximation error, keeping it bounded and manageable. The proof can be found in Appendix B.2.3.

Theorem 3 (Spectral decay of smooth functions) . Let Y ⊂ R D 2 be a domain satisfying the strong local Lipschitz condition, and let F : X × Y → R K be a vector-valued function with components F ( i ) ( x, y ) , for all i ∈ [ K ] . Suppose there exists a constant s > 0 such that F ( i ) ∈ L 2 ( X , H s ( Y )) for all i ∈ [ K ] , where H s ( Y ) denotes the s -order Sobolev space on Y . Then, the singular values σ i of F satisfy the polynomial decay rate ∥ σ i ∥ 2 ≤ O ( i -1 -2 s D 2 ) . Moreover, the optimal approximation error of the truncated Ft-SVD with rank r is upper bounded by O (( r +1) -2 s D 2 ) in the L 2 sense.

The rapid spectral decay in the Ft-SVD framework, as guaranteed by the Sobolev smoothness assumption on the output functions, has significant implications for the generalization performance of multi-output regression models. By promoting solutions with rapid spectral decay, the Ft-SVD framework can effectively constrain the complexity of the learned function, striking a balance between fitting the training data and maintaining simplicity. The significance of the spectral decay property in the Ft-SVD framework becomes even more apparent when considering multi-output regression under CDS. In the next section, we will delve into how the approximate low-rankness of the multi-output ground truth, as implied by the rapid spectral decay, provides a key insight into the generalization under CDS. Specifically, we will explore how the spectral decay of the ground truth in the Ft-SVD framework plays a crucial role in enabling effective generalization under combinatorial shifts.

3 An Ft-SVD-based Framework For Multi-output Regression under CDS

3.1 Generalized Tensor Completion with MNAR for MOR under CDS

We propose a theoretical framework based on Ft-SVD to formulate the CDS problem in multi-output regression. The key idea is to represent multi-output functions as t-bilinear embeddings in a Hilbert t-Module, a generalization of Hilbert spaces for handling vector-valued functions. We introduce

a series of assumptions on the ground-truth functions, the training and test distributions, and the spectral properties of the embeddings. These assumptions allow us to characterize the multi-output regression problem under CDS as a low-rank tensor estimation problem under MNAR settings.

To set the stage, we first define the concept of a Hilbert t-Module, which serves as the foundation for our t-bilinear representation.

Definition 3 (Hilbert t-Module) . Let R be the ring of K -dimensional real vectors with t-product. A Hilbert t-Module is a module M over R equipped with an R -valued inner product ⟨· , ·⟩ M , which is complete with respect to the ℓ 2 -norm induced by the R -valued inner product. 4

Intuitively, a Hilbert t-Module extends the classical Hilbert space to accommodate vector-valued functions, allowing us to perform inner products and norm calculations in a way that respects the tensor structure. This provides a natural framework for representing multi-output functions. With the Hilbert t-Module in place, we can now state our key assumption on the ground-truth functions:

Assumption 1 (t-Bilinear representation) . We have the following assumptions on the ground truth:

(I) There is a Hilbert t-Module ( M , ⟨· , ·⟩ M ) and two embeddings f ⋆ : X → M and g ⋆ : Y → M satisfying that h ⋆ ( x, y ) := ⟨ f ⋆ ( x ) , g ⋆ ( y ) ⟩ M is the Bayes optimal predictor on D train and D test , i.e., E D train [ z | x, y ] = E D test [ z | x, y ] = h ⋆ ( x, y ) . Hereafter, we call h ⋆ ( x, y ) the ground truth.

(II) We also assume that for some B > 0 , such that P ( x,y, z ) ∼D train [ ∥ z ∥ 2 ≤ B 2 ] = 1 , and max { sup x ∈X ∥ f ⋆ ( x ) ∥ M , sup y ∈Y ∥ g ⋆ ( y ) ∥ M } ≤ B .

This assumption postulates that the ground-truth functions admit a t-bilinear representation in terms of two embeddings in a Hilbert t-Module, and the inner product of these embeddings gives the Bayes optimal predictor for both the training and testing distributions. The t-bilinear form is a natural extension of the bilinear form in Ref. [46], and it allows us to capture the multi-dimensional structure of the problem. While the t-bilinear representation is expressive, we need additional assumptions on the training and test distributions to make the problem tractable under CDS:

Assumption 2 (Coverage of training and test distribution, Assumption 2.2 in Ref. [46]) . There exist constants κ tst , κ trn > 0 and marginal distributions D X , 1 , D X , 2 over X , and D Y , 1 , D Y , 2 over Y , with product measures D i ⊗ j := D X ,i ⊗D Y ,j , such that for all ( x, y ) ∈ X × Y , the following Radon-Nikodym derivative conditions hold: (I) Training coverage: d D i ⊗ j ( x,y ) d D train ( x,y ) ≤ κ trn for ( i, j ) ∈

In words, this assumption requires that the training distribution covers the key feature combinations in D 1 ⊗ 1 , D 1 ⊗ 2 and D 2 ⊗ 1 , while the test distribution is allowed to include unseen combinations in D 2 ⊗ 2 . The constants κ trn , κ tst quantify the degree of coverage 5 .

While Assumption 2 characterizes the relationship between the training and test distributions, it does not directly control the impact of distribution shifts on the model's performance. For this, we introduce a further assumption on the covariate shift:

Assumption 3 (Controlled covariate shifts) . There exists a κ cov ≥ 1 such that, for any v ∈ M , the following inequalities hold: E x ∼D X , 2 [ ∥⟨ f ⋆ ( x ) , v ⟩ M ∥ 2 ] ≤ κ cov · E x ∼D X , 1 [ ∥⟨ f ⋆ ( x ) , v ⟩ M ∥ 2 ] and E y ∼D Y , 2 [ ∥⟨ g ⋆ ( y ) , v ⟩ M ∥ 2 ] ≤ κ cov · E y ∼D Y , 1 [ ∥⟨ g ⋆ ( y ) , v ⟩ M ∥ 2 ] .

This assumption can be seen as a t-Module variant of Assumption 2.3 in Ref. [46]. It essentially bounds the worst-case impact of covariate shift on the model's performance, ensuring that the error on the unseen distribution D 2 ⊗ 2 is controlled by the error on the training distribution (up to a constant κ cov ) 6 . This is a key ingredient that allows us to provide generalization guarantees under CDS.

Finally, to leverage the spectral structure of the embeddings, we make the following assumptions:

Assumption 4 (Polynomial spectral decay) . Consider the t-covariances Σ f ⋆ := E D X , 1 [ f ⋆ ∗ M ( f ⋆ ) ⊤ ] and Σ g ⋆ := E D Y , 1 [ g ⋆ ∗ M ( g ⋆ ) ⊤ ] . We have the following assumptions:

(I) Balanced embeddings: The ground truth embeddings f ⋆ and g ⋆ reside in an appropriate basis of M such that Σ f ⋆ = Σ g ⋆ =: Σ ⋆ 1 ⊗ 1 ; we also assume λ 1 ( M ( Σ ⋆ 1 ⊗ 1 ) ( i ) ) > 0 holds for all i ∈ [ K ] .

(II) Polynomial spectral decay: Each of the K frequency components M ( Σ ⋆ 1 ⊗ 1 ) ( i ) of the t-covariance Σ ⋆ 1 ⊗ 1 exhibits a polynomial singular value decay pattern, but potentially with different decay rates γ i > 0 , i.e., σ j ( M ( Σ ⋆ 1 ⊗ 1 ) ( i ) ) ≤ Cj -(1+ γ i ) , ∀ j ∈ N , ∀ i ∈ [ K ] .

Assumption 5 (Small low-rank approximation error of ( f ⋆ , g ⋆ ) on D train ) . Let P ( i ) k denote the projection onto the topk eigenspace of M ( Σ ⋆ 1 ⊗ 1 ) ( i ) , which represents the i -th frequency component ( ∀ i ∈ [ K ]) of Σ ⋆ 1 ⊗ 1 induced by the operator M ( · ) in Eq. (1). Define ApxErr ( i ) k ( x, y ) := ( M ( h ⋆ ( x, y )) ( i ) - ⟨ P ( i ) k M ( f ⋆ ( x )) ( i ) , P ( i ) k M ( g ⋆ ( y )) ( i ) ⟩ ) 2 as the error between the ground truth h ⋆ ( x, y ) and the optimal rankk approximations of the ground truth embeddings ( f ⋆ , g ⋆ ) over the training data D train for all rank k ∈ N in each i -th frequency component. We assume that

Assumptions 4 and 5 postulate that the ground-truth embeddings have a favorable spectral structure. Specifically, they assume that the embeddings are balanced in an appropriate basis, and their spectrum decays polynomially (possibly at different rates for different frequency components). Fig. 2 illustrates the varying rates of spectral decay exhibited by the DCT frequency components of the Akiyo video data 7 . This empirical observation serves as a compelling motivation for Assumption 4-(II). Assumption 5 implies that the training distribution D train is sufficiently representative of the true function h ⋆ in each frequency component. It guarantees that if we find embeddings that well approximate h ⋆ on the training data, they will also perform well on the reference distribution D 1 ⊗ 1 .

0

50

100

150

200

250

-2

0

2

4

6

8

10

Formulating MOR under CDS as generalized tensor completion with MNAR. Based on the assumptions, we can formulate the MOR problem under CDS as a tensor completion problem under the MNAR setting. Specifically, given the training data { ( x i , y i , z i ) } n i =1 drawn from D train , we aim to estimate the underlying embeddings f ⋆ and g ⋆ by solving the following problem:

where ℓ ( · , · ) is a suitable loss function. Problem (4) can be interpreted as a tensor completion problem with MNAR: (1) The t-bilinear form ⟨ f ⋆ ( x ) , g ⋆ ( y ) ⟩ M encodes the function h ⋆ ( x, y ) as a (generalized) tensor. (2) The training data only covers a subset of the feature combinations due to Assumption 2. (3) The goal is to estimate the complete tensor from the partially observed tensor entries. By solving Problem (4), we obtain the estimated embeddings ˆ f and ˆ g , which can be used to make predictions on a new test point ( x, y ) via the t-bilinear form ⟨ ˆ f ( x ) , ˆ g ( y ) ⟩ M .

3.2 Algorithms for Robust Generalization of Multi-output Regression under CDS

How does ERM-based multi-output regression perform under CDS? A natural approach to training a multi-output regression model is to fix a target rank r ∈ N and compute an Empirical Risk

where ( · ) ⊤ denotes the t-transpose (see Definition 6), and function classes F r , G r are given in Assumption 8. Assumption 8 ensures that the function classes F r and G r are sufficiently expressive to approximate the true embeddings f ⋆ and g ⋆ , while also having controlled complexity in terms of their covering numbers. With the assumption in place, we can obtain the following generalization guarantee for the ERM solution (see the proof in Appendix D.2):

Theorem 4 (Excess Risk Bound for ERM under CDS) . Suppose the learned embeddings ( ˆ f erm , ˆ g erm ) are α -conditioned and ( ϵ trn , ϵ D 1 ⊗ 1 ) -accurate embeddings satisfying 8 ˘ ϵ ( i ) D 1 ⊗ 1 ≤ ˘ σ ⋆, ( i ) 1 / (40 r ) for all i ∈ [ K ] . Then, under Assumptions 1 to 5 and 8, the excess risk of the ERM solution on D test can be bounded up to a constant factor c = poly( κ cov , κ tst , κ trn ) with probability at least 1 -δ :

where α := max i { α i } , σ := min i { ˘ σ ⋆, ( i ) r } > 0 , tail ⋆ q ( r ) := ∑ K i =1 ∑ j>r (˘ σ ⋆, ( i ) j ) q , q ≥ 1 , ∆ n = B 4 ( N ( r, 2 B/n ) + log 2 δ ) /n with N ( r, ϵ ) = N ( F r , ϵ/ (2 B ) , ∥·∥ ∞ ) · N ( G r , ϵ/ (2 B ) , ∥·∥ ∞ ) . Here, ˘ σ ⋆, ( i ) j := σ j ( M ( Σ ⋆ 1 ⊗ 1 ) ( i ) ) denotes the j -th largest singular value of the i-th frequency component M ( Σ ⋆ 1 ⊗ 1 ) ( i ) of the t-covariance operator Σ ⋆ 1 ⊗ 1 , and N ( F , ϵ, ∥ · ∥ ∞ ) denotes the covering number of a function class at scale ϵ .

Extending the framework of Ref. [46], this theorem bounds the excess risk of the ERM solution for multi-output regression under CDS, incorporating both approximation and statistical errors. Our key contribution, the Ft-SVD for infinite-dimensional tensor completion, addresses multi-output MOR while considering spectral decay across frequency components. However, Theorem 4 has two primary limitations: (1) It assumes that the learned embeddings are α -conditioned and ( ϵ trn , ϵ D 1 ⊗ 1 ) -accurate, which may not be achievable with ERM. (2) The theorem requires that the minimum singular value ˘ σ ⋆, ( i ) r is strictly positive for each frequency component i ∈ [ K ] . This condition may not hold for t-embeddings that exhibit varying spectral decay patterns across different frequency components. In practice, multi-output ground truths often demonstrate diverse spectral decay patterns, making it challenging for the hypothesis classes F r and G r to capture precise low-rank structures across different frequency components (See Fig. 2 for example).

ERM-DS: Addressing the limitations of single-stage ERM. To address the weaknesses of singlestage ERM, we propose the Double-Stage Empirical Risk Minimization (ERM-DS) algorithm. ERMDS uses a two-stage training process with hypothesis classes tailored to each frequency component, better capturing varying spectral decay patterns and balancing model complexity with generalization ability. Our work extends the ERM-DS framework, originally proposed by Simchowitz et al. for robust learning under CDS [46], to address the challenging setting of multi-output regression. By representing vector-valued functions as embeddings in a Hilbert t-module and employing tensor algebra, we capture the complex interdependencies among multiple outputs, enabling robust generalization.

Unlike Assumption 8, the ERM-DS algorithm considers fine-grained hypothesis classes satisfying Assumption 9, which defines function classes specifically for each frequency component to better capture the distinct decay behaviors exhibited by the ground truth in each frequency component. This allows ERM-DS to adapt to the varying spectral properties of the true embeddings across different frequency components, leading to improved generalization performance 9 . The ERM-DS algorithm consists of four main steps:

Overparameterized Training : Train an overparameterized model ( ˜ f , ˜ g ) to approximate the unknown true predictive functions f ⋆ and g ⋆ by choosing their frequency components separately via K parallel ERM sub-problems (Eq. (18)).

t-Covariance Estimation : Estimate the t-covariances of the embeddings using additional unlabeled examples to capture the important directions of variation (Eq. (19)).

Dimension Reduction : Compute low-rank projections using the estimated covariances and obtain reduced-rank embeddings by projecting the overparameterized embeddings onto the low-rank t-subspace (Eq. (20)).

Distillation : Fine-tune the reduced-rank embeddings by approximating their frequency components separately in the transformed domain, minimizing a combination of empirical risk and consistency with the reduced-rank embeddings (Eq. (21)).

The theoretical guarantees for the performance of the ERM-DS algorithm are provided in Theorem 5: Theorem 5 (Excess Risk Bound for ERM-DS under CDS) . Under appropriate conditions on algorithm parameters and sample sizes 10 , for any δ > 0 , the ERM-DS solution achieves an excess risk bound of c ′ ( ϵ 2 + ∑ K i =1 (1+ γ -2 i ) r -2 γ i i, cut ) with probability at least 1 -δ . Here, c ′ = poly( κ cov , κ tst , κ trn ) is a problem-dependent constant, ϵ > 0 is the desired accuracy level, { r i, cut } K i =1 are cutoff rank parameters in ERM-DS, and { γ i } K i =1 are parameters related to spectral decay in Assumption 4.

The proof is given in Appendix D.3. Theorem 5 provides a generalization error bound for the ERMDS algorithm in multi-output regression under CDS. The expected squared error between predicted and true outputs can be bounded by two main terms: the desired accuracy level ϵ 2 , adjustable by algorithm parameters and a term depending on the spectral decay properties of the true embeddings in each frequency component, represented by ∑ K i =1 (1 + γ -2 i ) r -2 γ i, cut , which captures approximation error due to low-rank structure and target function complexity.

Numerical Experiments. As the first theoretical work on the MOR problem under CDS, this paper proposes the novel Ft-SVD theoretical framework, along with related assumptions and algorithm design. The experiments conducted serve solely as a conceptual validation using synthetic data. We consider the settings when X and Y are finite, in which case MOR under CDS naturally degenerates to tensor completion with MNAR tubes. We construct a ground truth tensor X composed of factor tensors 11 A 1 , B 1 , A 2 , B 2 , where range ( A 2 ) ⊂ range ( A 1 ) and range ( B 2 ) ⊂ range ( B 1 ) . The singular values of M ( A 1 ) ( i ) , M ( B 1 ) ( i ) follow a power-law decay σ [1] j = j -(1+ γ ) / 2 , while those of M ( A 2 ) ( i ) , M ( B 2 ) ( i ) are set to σ [2] j = c j κσ [1] j , where c j ∈ (0 , 1) is a uniform random variable, κ controls the covariate shift, and the transform M ( · ) is chosen as DFT. To simulate CDS, we perform two-stage sampling on X : first, we randomly sample entries with rate srall; second, we sample the top block X 11 ( A 1 , B 1 ) with a lower rate sr 11 , leaving the bottom block X 22 ( A 2 , B 2 ) as the unobserved test set. We compare the proposed ERM-DS algorithm with the single-stage one, evaluating their test risks under different CDS intensities (by varying the κ parameter that controls the covariate shift) and different percentages of training data. As shown in Fig. 3, the ERM-DS algorithm consistently outperforms the single-stage ERM across all settings. On the left, we observe that as κ increases, indicating more severe covariate shift, the test risk of both algorithms rises, but the ERM-DS algorithm maintains a significant advantage over the single-stage ERM, with the performance gap widening for larger κ values. On the right, the results demonstrate that the ERM-DS algorithm achieves lower test risks compared to the single-stage ERM under varying training data sizes, highlighting its robustness even with limited training samples. Overall, Fig. 3 shows the ERM-DS algorithm achieves lower test risks in handling the CDS problem across different covariate shift intensities and training data availabilities.

4 Extensions, Conclusion and Limitation

Extension to higher-order Ft-SVD. Our Ft-SVD framework, while designed for 3-order tensors, is not strictly limited to two-dimensional input cases. It is versatile enough to handle multi-dimensional

D

Test RiskOverK

3.5

ERM

3

ERM-DS

Test Risk(log scale)

2.5

2

1.5

0.5

0.5

1

1.5

2

2.5

3

K

TestRiskOver theSizeTrainingData

ERM

ERM-DS

3.5

Test Risk (log scale)

3

2.5

2

1.5

0.5

0.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

PercentageofTrainingData

inputs that can be divided into two distinct sets. Extending Ft-SVD to higher-order cases is indeed non-trivial and we show a preliminary conjecture of a higher-order extension motivated by Ref. [53]:

Let F : ∏ N i =1 X i → R K be a square-integrable vector-valued function 12 , where X i ⊂ R D i for i = 1 , . . . , N . Then there exist sets of functions { U n i } ∞ i =1 ⊂ L 2 ( X n ; R K ) for each n = 1 , . . . , N , and a core function S : ∏ N N → R K , such that F can be represented as F ( x 1 , . . . , x N ) =

i =1 ∑ ∞ i 1 ,...,i N =1 S ( i 1 , . . . , i N ) ∗ M N ∏ ∗ n =1 U n i n ( x n ) . Here, ∏ ∗ denotes the sequential t-product, which applies the t-product operation sequentially to the functions U n i n ( x n ) . The convergence of this infinite sum is in the L 2 sense. We also have

Orthogonality: ∀ n ∈ [ N ] , the functions { U n i } satisfy: ∫ X n U n i ( x ) ∗ M ( U n j ( x )) ⊤ dx = δ ij M -1 (1) .

Core properties: The core function S has two key characteristics:

̸

All-orthogonality: For all 1 ≤ n ≤ N and all α = β , we have ∫ ∏ i = n X i S ( . . . , α, . . . ) ⊤ ∗ M S ( . . . , β, . . . ) ∏ i = n dx i = 0 . This means that slices of the core tensor are t-orthogonal.

̸

ii) Ordering: For all n = 1 , . . . , N , we have ∥ S x n =1 ∥ L 2 ≥ ∥ S x n =2 ∥ L 2 ≥ ··· , where ∥ S x n = α ∥ L 2 denotes the L 2 norm of S with its n -th mode fixed at α . This property ensures a unique ordering of the components.

This decomposition generalizes the concept of Ft-SVD to multilinear functions. It is interesting to generalize the various higher-order variants of t-SVD [22, 31, 43, 1] to functional settings.

Conclusion. The paper addresses the challenge in multi-output regression where training distribution does not cover all input feature combinations, leading to CDS. We introduce a new methodology within a generalized tensor decomposition framework, named Ft-SVD, to tackle this challenge by treating the problem as a tensor completion task under the missing-not-at-random setting. The paper highlights the role of spectral decay of the true embeddings in enhancing model generalization and, through detailed analysis, establishes how multi-output models can manage combinatorial shifts, improving prediction accuracy for new and unseen input combinations.

Limitation. This paper introduces a tensor spectral theory framework to address MOR under CDS, marking an early theoretical exploration in this field. However, several limitations are recognized. First, spectral methods may not fully capture the complexity of real-world data, and the robustness of controlled experimental results remains uncertain. We encourage future research to refine these methods, aiming to develop more effective solutions. Furthermore, an open problem remains in extending the framework to accommodate more than two combinatorial features. Initial investigations suggest that the approach in Ref. [46] does not readily generalize to cases with more than two feature combinations, potentially requiring new mathematical tools to address this challenge.

̸

Acknowledgments

Our sincere appreciation goes to both the area chair and reviewers for their unwavering dedication and meticulous attention given to this paper. This research was supported in part by National Natural Science Foundation of China under Grants 62103110, 62203124, 62073087, and 62071132, in part by RIKEN Incentive Research Project 100847-202301062011, and in part by JSPS KAKENHI Grant Numbers JP24K20849, JP24K03005, JP23K28109.

References

S. Ahmadi-Asl, V. Leplat, A.-H. PHAN, and A. Cichocki. A new tensor network: Tubal tensor train network and its applications, 2024. 4

M. A. Alvarez, L. Rosasco, and N. D. Lawrence. Kernels for vector-valued functions: A review. Foundations and Trends in Machine Learning , 4(3):195-266, 2012. 1, A.2

H. Antil and S. Bartels. Spectral approximation of fractional pdes in image processing and phase field modeling. Computational Methods in Applied Mathematics , 17(4):661-678, 2017. 2.2, B.2.3

H. Antil and C. N. Rautenberg. Sobolev spaces with non-muckenhoupt weights, fractional elliptic operators, and applications. SIAM Journal on Mathematical Analysis , 51(3):2479-2503, 2019. 2.2, B.2.3

H. Aras and N. Aras. Forecasting residential natural gas demand. Energy Sources , 26:463-472, 04 2004. 1

B. Bakker and T. Heskes. Task clustering and gating for bayesian multitask learning. Journal of Machine Learning Research , 4:83-99, 2003. 1, A.2

D. Bigoni, A. P. Engsig-Karup, and Y. M. Marzouk. Spectral tensor-train decomposition. SIAM Journal on Scientific Computing , 38(4):A2405-A2439, 2016. 1

B. Blackadar. Operator algebras: theory of C*-algebras and von Neumann algebras , volume 122. Springer Science & Business Media, 2006. C.1.1

H. Borchani, G. Varando, C. Bielza, and P. Larrañaga. A survey on multi-output regression. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery , 5(5):216-233, 2015. 1, A.2

S. Boucheron, O. Bousquet, and G. Lugosi. Theory of classification: A survey of some recent advances. ESAIM: probability and statistics , 9:323-375, 2005. D.18

L. Breiman. Bagging predictors. Machine Learning , 24(2):123-140, 1996. 1, A.2

C. J. Carlson et al. Climate change will force new animal encounters - and boost viral outbreaks. Nature , 605:20, 2022. 1

R. Caruana. Multitask learning. Machine Learning , 28(1):41-75, 1997. 1, A.2

J. Chen, C. Lu, Y. Li, H. Zhang, and J. Zhang. Nonconvex tensor completion from noisy data with a guaranteed convergence rate. In International Conference on Machine Learning , pages 1489-1498. PMLR, 2020. 1, A.2

L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelligence , 40(4):834-848, 2017. 2.2

L. Deng, X.-Y. Liu, H. Zheng, X. Feng, and Y. Chen. Graph spectral regularized tensor completion for traffic data imputation. IEEE Transactions on Intelligent Transportation Systems , 23(8):10996-11010, 2021. 2.1

S. Džeroski, D. Demšar, and J. Grbovi´ c. Predicting Chemical Parameters of River Water Quality from Bioindicator Data. Applied Intelligence , 13(1):7-17, July 2000. 1

S. Fang et al. Functional bayesian tucker decomposition for continuous-indexed tensor data. arXiv preprint arXiv:2311.04829 , 2023. 1

J. Gama, I. Žliobait˙ e, A. Bifet, M. Pechenizkiy, and A. Bouchachia. A survey on concept drift adaptation. ACM Computing Surveys (CSUR) , 46(4):1-37, 2014. 1, A.2

A. Gorodetsky, S. Karaman, and Y. Marzouk. A continuous analogue of the tensor-train decomposition. Computer Methods in Applied Mechanics and Engineering , 347:59-84, 2019. 1

M. Griebel and G. Li. On the decay rate of the singular values of bivariate functions. SIAM Journal on Numerical Analysis , 56(2):974-993, 2018. B.2.3

J. Han. p -order tensor products with invertible linear transforms. arXiv preprint arXiv:2005.11477 , 2020. 4

R. Han, P. Shi, and A. R. Zhang. Guaranteed functional tensor singular value decomposition. Journal of the American Statistical Association , 119(546):995-1007, 2024. 1

F. L. Hitchcock. The expression of a tensor or a polyadic as a sum of products. Journal of Mathematics and Physics , 6(1-4):164-189, 1927. 1, A.2

E. Kernfeld, M. Kilmer, and S. Aeron. Tensor-tensor products with invertible linear transforms. Linear Algebra and its Applications , 485:545-570, 2015. 2.1, 1, 2.1, 2, 6, 7, 8, C.1.1

M. E. Kilmer, K. Braman, et al. Third-order tensors as operators on matrices: A theoretical and computational framework with applications in imaging. SIAM J MATRIX ANAL A , 34(1):148172, 2013. 1, 2.1, B.1, 9

M. E. Kilmer, L. Horesh, H. Avron, and E. Newman. Tensor-tensor algebra for optimal representation and compression of multiway data. Proceedings of the National Academy of Sciences , 118(28):e2015851118, 2021. 1, 2.1, 2, 2, 2.2, B.2.2

T. G. Kolda and B. W. Bader. Tensor decompositions and applications. SIAM Review , 51(3):455500, 2009. B.1

J. Lai, Z. Li, D. Huang, and Q. Lin. The optimality of kernel classifiers in sobolev space. In The Twelfth International Conference on Learning Representations , 2024. 2.2, B.2.3

H. Li, W. Zhang, Y. Chen, Y. Guo, G.-Z. Li, and X. Zhu. A novel multi-target regression framework for time-series prediction of drug efficacy. Scientific Reports , 7(1):40652, Jan. 2017. 1

S. Liu, X.-L. Zhao, J. Leng, B.-Z. Li, J.-H. Yang, and X. Chen. Revisiting high-order tensor singular value decomposition from basic element perspective. IEEE Transactions on Signal Processing , 2024. 4

X. Liu, S. Aeron, V. Aggarwal, and X. Wang. Low-tubal-rank tensor completion using alternating minimization. IEEE TIT , 66(3):1714-1737, 2020. 2.1, 2.1, 2.1

M. Long, Z. Cao, J. Wang, and M. I. Jordan. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems , pages 1640-1650, 2018. 1, A.2

C. Lu. Transforms based tensor robust pca: Corrupted low-rank tensors recovery via convex optimization. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1145-1152, 2021. 2.1, 2.1

C. Lu, J. Feng, W. Liu, Z. Lin, S. Yan, et al. Tensor robust principal component analysis with a new tensor nuclear norm. IEEE TPAMI , 2019. 2.1, 4

V. Lucarini and M. D. Chekroun. Theoretical tools for understanding the climate crisis from Hasselmann's programme and beyond. Nature Reviews Physics , 5(12):744-765, 2023. 2.2

Y. Luo et al. Low-rank tensor function representation for multi-dimensional data recovery. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023. 1

W. Ma and G. H. Chen. Missing not at random in matrix completion: The effectiveness of estimating missingness probabilities under a low nuclear norm assumption. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. 1

I. Misra, A. Shrivastava, A. Gupta, and M. Hebert. Cross-stitch networks for multi-task learning. In IEEE Conference on Computer Vision and Pattern Recognition , pages 3994-4003, 2016. 1, A.2

Z. Obermeyer, B. Powers, C. Vogeli, and S. Mullainathan. Dissecting racial bias in an algorithm used to manage the health of populations. Science , 366(6464):447-453, 2019. 1

I. V . Oseledets. Tensor-train decomposition. SIAM Journal on Scientific Computing , 33(5):22952317, 2011. 1, A.2

K. Park and A. Stinchcombe. Deep reinforcement learning of viscous incompressible flow. Journal of Computational Physics , 452:110916, 2022. 2.2, B.2.3

W. Qin, H. Wang, F. Zhang, J. Wang, X. Luo, and T. Huang. Low-rank high-order tensor completion with applications in visual data. IEEE Transactions on Image Processing , 31:24332448, 2022. 4

J. Quionero-Candela, M. Sugiyama, A. Schwaighofer, and N. D. Lawrence. Dataset shift in machine learning. The MIT Press , 2009. 1, A.2

H. Shimodaira. Improving predictive inference under covariate shift by weighting the loglikelihood function. Journal of Statistical Planning and Inference , 90(2):227-244, 2000. 1, A.2

M. Simchowitz, A. Gupta, and K. Zhang. Tackling combinatorial distribution shift: A matrix completion perspective. In The Thirty Sixth Annual Conference on Learning Theory , pages 3356-3468. PMLR, 2023. 1, 1, 3.1, 2, 3.1, 3.2, 4, A.2, C.1.3, C.1.3, 6, C.1.3, C.2.2, D, D.1, D.1.2, D.1.4, D.11, D.3, D.3, 20, D.3, D.3, D.3, D.3.2, D.3.2

J. Šimša. The best l 2 -approximation by finite sums of functions with separable variables. Aequationes Mathematicae , 43:248-263, 1992. B.2.2

F. Smithies. The eigen-values and singular values of integral equations. Proceedings of the London Mathematical Society , 2(1):255-279, 1938. B.2.2

G. Song, M. K. Ng, and X. Zhang. Robust tensor completion using transformed tensor singular value decomposition. NUMER LINEAR ALGEBR , 27(3):e2299, 2020. 2.1

L. R. Tucker. Some mathematical notes on three-mode factor analysis. Psychometrika , 31(3):279-311, 1966. 1, A.2

A. Wang, C. Li, M. Bai, Z. Jin, G. Zhou, and Q. Zhao. Transformed low-rank parameterization can help robust generalization for tensor neural networks. In Advances in Neural Information Processing Systems , volume 36, 2023. 2.1

A. Wang, G. Zhou, Z. Jin, and Q. Zhao. Tensor recovery via ∗ l -spectral k -support norm. IEEE Journal of Selected Topics in Signal Processing , 15(3):522-534, 2021. 2.1

Y. Wang and Y. Yang. Hot-svd: higher order t-singular value decomposition for tensors based on tensor-tensor product. Computational and Applied Mathematics , 41(8):394, 2022. 4

W. Xiao, J. Sun, and J. Zhou. Tensor completion with side information: A riemannian manifold approach. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 5051-5058, 2019. 1, A.2

D. Xu, Y. Shi, I. W. Tsang, Y.-S. Ong, C. Gong, and X. Shen. Survey on multi-output learning. IEEE transactions on neural networks and learning systems , 31(7):2409-2429, 2019. 1

K. Yosida. Functional analysis , volume 123. Springer Science & Business Media, 2012. B.2.1

X. Zhang, M. Bai, and M. Ng. Nonconvex-tv based image restoration with impulse noise removal. SIAM Journal on Imaging Sciences , 10(3):1627-1667, 2017. 2.1

X. Zhang and M. K.-P. Ng. Low rank tensor completion with poisson observations. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2021. 2.1, 2.1, 2.1

Y. Zhang, J. Wang, X. Xie, and M. Sugiyama. Equivariant disentangled transformation for domain generalization under combination shift. arXiv preprint arXiv:2208.02011 , 2022. 1

Z. Zhang and S. Aeron. Exact tensor completion using t-svd. IEEE TSP , 65(6):1511-1526, 2017. 10

Z. Zhang, G. Ely, S. Aeron, et al. Novel methods for multilinear data completion and de-noising based on tensor-svd. In CVPR , pages 3842-3849, 2014. 1

Supplementary Materials for

Generalized Tensor Decomposition for Multi-Output Regression with Combinatorial Distribution Shifts

Contents

41

Multi-Output Regression under CDS

Ft-SVD Framework

Theoretical foundation

ERM-DS Algorithm

Tensor completion under MNAR

Motivates

Formulates

Inspires

Addressed by

The appendix provides additional materials, detailed proofs, and extended discussions to support the main content of the paper Generalized Tensor Decomposition for Multi-Output Regression with Combinatorial Distribution Shifts . An overview of the core research content in this paper is shown in Fig. 4. We summarize the notations in Table 1 which provides a comprehensive list of the main symbols, along with their descriptions. The symbols are categorized based on their roles, such as input and output spaces, distributions, embeddings, error terms, hypothesis classes, and algorithmic parameters.

The purpose of this appendix is to offer a comprehensive and rigorous treatment of the theoretical foundations, algorithmic details, and experimental results that underpin the proposed Functional t-Singular Value Decomposition (Ft-SVD) framework for multi-output regression under CDS.

The appendix is organized as follows:

In Appendix A, we discuss potential applications of the proposed Ft-SVD framework, provide a detailed review of related work, and present details in the experiments.

Appendix B delves into the theoretical foundations of the Ft-SVD framework. We start by introducing the preliminaries of t-Singular Value Decomposition (t-SVD) and then present the Functional t-SVD, along with the proofs of its key properties (Theorems 1 and Theorem 2). We also discuss the low-tubal-rank approximability of smooth functions and provide the proof of Theorem 3.

Appendix C provides further explanations and insights into the Ft-SVD framework for multioutput regression under CDS. We offer more details on the Hilbert t-Module, t-bilinear embeddings, and the assumptions used in our analysis. We also present a more in-depth discussion of the proposed algorithms, including the Empirical Risk Minimization (ERM) and the Double-Stage ERM (ERM-DS).

In Appendix D, we present a comprehensive analysis of the algorithms proposed in the main paper. We start by deriving an error decomposition that forms the basis for our theoretical guarantees. We then provide detailed proofs for the performance bounds of the ERM and ERM-DS algorithms, along with the necessary lemmas and change-of-measure results. This appendix also includes a discussion on the risk bound for t-bilinear combinatorial extrapolation, which is a key component of our analysis.

Broader effects. The paper provides a new tensor spectral perspective for the multi-output regression problem due to combinatorial distribution shift. It focuses solely on these technical aspects and does not have potential societal impacts.

A Prevalence of MoR under CDS, Related Work, and Details of Experiments

A.1 Prevalence of MoR under CDS across Domains

The challenge of Multi-output Regression under Combinatorial Distribution Shifts (MoR under CDS) is pervasive across a wide spectrum of real-world applications. The following examples illustrate the ubiquity of this problem in diverse domains:

Healthcare : In medical diagnostics, predicting a patient's risk for multiple diseases (e.g., heart disease, diabetes) based on various patient attributes (e.g., age, blood pressure, heart rate) constitutes a multi-output regression problem. However, training data often comes from specific patient cohorts, covering only a subset of possible attribute combinations. When new patient groups (e.g., different age brackets or geographic regions) emerge, they introduce novel attribute combinations, leading to combinatorial distribution shifts. Diagnostic systems must adapt to these new patient profiles to accurately predict disease risks.

Marketing : Predicting how different user segments respond to various marketing strategies (e.g., ad types, promotions) based on user attributes (e.g., age, income) and strategy attributes (e.g., ad themes, delivery channels) is a multi-output regression task. Training data typically covers common user-strategy combinations, but when new user groups or marketing strategies appear, they create unseen attribute combinations, resulting in combinatorial distribution shifts. Marketing prediction models need to transfer knowledge to these new scenarios to forecast user responses accurately.

Materials Science : Predicting the performance of material formulations across multiple metrics (e.g., hardness, conductivity) based on composition ratios is a multi-output regression problem. Due to experimental constraints, training data usually includes only a subset of known formulations. When researchers explore novel composition ratios, they encounter new formulation combinations, leading to combinatorial distribution shifts. Material performance prediction systems must generalize to these unseen formulations to accelerate materials discovery.

Transportation : Forecasting multiple traffic flow indicators (e.g., vehicle flow, pedestrian flow) for different areas and time periods based on area attributes (e.g., road network, land use) and temporal attributes (e.g., day of the week, holidays) is a multi-output regression task. Training data may cover only typical area-time combinations, but when new areas develop or novel temporal patterns emerge, they create unseen attribute combinations, causing combinatorial distribution shifts. Traffic prediction models must adapt to these new scenarios for effective transportation planning and management.

Environmental Monitoring : Predicting concentrations of various pollutants (e.g., PM2.5, ozone) at different locations and seasons based on location attributes (e.g., terrain, land use) and meteorological conditions (e.g., temperature, humidity) is a multi-output regression problem. Training data often comes from specific monitoring stations and time periods, covering a limited set of location-weather combinations. When new monitoring stations are deployed or unusual weather patterns occur, they introduce new attribute combinations, leading to combinatorial distribution shifts. Environmental monitoring systems must generalize to these unseen scenarios to comprehensively assess environmental quality.

These examples summarized in Figure 5 underscore the prevalence of MoR under CDS across various domains. In these scenarios, multiple prediction tasks share the same input space, but training distribution covers only a subset of attribute combinations. The emergence of new attribute combinations constitutes a combinatorial distribution shift. Transferring knowledge from observed combinations to novel ones is a common challenge in these applications, and addressing this problem is crucial for developing robust intelligent systems.

A.2 Detailed Related Work

Multi-Output Regression. Multi-output regression has been extensively studied in machine learning, with early works focusing on multi-task learning [13, 6] and recent approaches extending to shared representation learning [39], linear models [11], kernel methods [2], and neural networks [9]. However, most of these methods assume that the training and test data come from the same distribution, which may not hold under distribution shifts. In contrast, our work specifically addresses the problem of multi-output regression under CDS by proposing a novel theoretical framework based on Ft-SVD and an extended version of the ERM-DS algorithm.

Tensor Completion. Tensor completion has seen significant advancements with the development of various tensor decomposition techniques [24, 50, 41] and methods that can handle non-random missing patterns [54, 14]. While these works provide a foundation for addressing tensor completion, they do not consider the specific challenges posed by multi-output regression under CDS, such as dealing with infinite and continuous feature domains. Our work aims to address this gap by formulating multi-output regression under CDS as a low-rank tensor estimation problem under the Missing Not At Random (MNAR) setting and developing a tailored ERM-DS algorithm with theoretical guarantees.

Distribution Shift. Various types of distribution shifts have been studied in machine learning, including covariate shift [45], concept drift [19], and domain adaptation [44]. Some recent works have addressed distribution shift in multi-output regression by aligning the distributions through learning invariant representations [33]. However, these methods typically require access to labeled data from the target domain, which may not always be available. In contrast, our work focuses on the specific case of CDS, where the training distribution covers certain marginal distributions of the input features, but the test distribution involves unseen combinations of these features. We address this challenge by leveraging the Ft-SVD framework and the ERM-DS algorithm, which do not require labeled data from the test distribution.

Simchowitz et al. [46] proposed the Double-Stage Empirical Risk Minimization (ERM-DS) framework to address single-output learning under CDS. Our work extends the ERM-DS framework to the multi-output regression setting by representing vector-valued functions as embeddings in a Hilbert t-module and leveraging tensor algebra to capture the interdependencies among multiple outputs. The proposed Ft-SVD theoretical framework, the formulation of multi-output regression under CDS as a low-rank tensor estimation problem, and the tailored ERM-DS algorithm with theoretical guarantees aim to contribute to the existing literature on multi-output regression, tensor completion, and distribution shift, by providing a principled approach to address the specific challenges posed by CDS in multi-output regression.

A.3 Details of Experiments

As the pioneering theoretical study tackling the multi-output regression problem under combinatorial distribution shift, this paper proposes the novel Ft-SVD theoretical framework, along with related assumptions and algorithm design. To validate the proposed approach, we conducted a series of experiments on synthetic tensor data, serving solely as a conceptual proof-of-concept. Considering finite feature domains X and Y , the MOR problem under CDS naturally degenerates to tensor completion with missing-not-at-random (MNAR) tubes. We conducted a series of experiments using synthetic tensor data. The primary focus of our analysis was to compare the empirical risk of three algorithms: Single ERM in Eq. (5), Overparameterized Training (Step 1 of ERM-DS), and the proposed ERM-DS. We generated a synthetic tensor X ∈ R ( m 1 + m 2 ) × ( d 1 + d 2 ) × K . The tensor X was constructed using a combination of factor tensors A 1 ∈ R m 1 × r × K , B 1 ∈ R d 1 × r × K , A 2 ∈ R m 2 × r × K , and B 2 ∈ R d 2 × r × K , where range ( A 2 ) ⊂ range ( A 1 ) and range ( B 2 ) ⊂ range ( B 1 ) . The singular value tubes of A 1 and B 1 followed a power-law decay with rate γ , i.e., M ( σ [ i ] 1 ) = repmat ( i -(1+ γ ) / 2 , K ) ∈ R 1 × 1 × K , and the singular value tubes of A 2 and B 2 are set to be M ( σ [ i ] 2 ) = repmat ( c i κσ [ i ] 1 , K ) ∈ R 1 × 1 × K , and c i ∈ (0 , 1) is the random variable following the uniform distribution. Therefore, the tensor data can be divided into four parts, namely, X 11 ∈ R m 1 × d 1 × K , X 12 ∈ R m 1 × d 2 × K , X 21 ∈ R m 2 × d 1 × K and X 22 ∈ R m 2 × d 2 × K :

Test RiskOverK

4.5

ERM

4

ERM-DS

3.5

Test Risk (log scale)

3

2.5

S2

1.5

0.5

0.5

1

1.5

2

2.5

3

K

TestRiskOvertheSizeTrainingData

4.5

ERM

ERM-DS

3.5

3

2.5

1.5

0.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

Percentage of TrainingData

In this case, the factor tensor can be represented as the features in Eq., that is

and the tube X ( i ′ , j ′ , :) = ⟨ f ⋆ ( x i ′ ) , g ⋆ ( y j ′ ) ⟩ is regarded as the output. Therefore, the data sample can be regarded as ( i ′ , j ′ , z i ′ j ′ ) where z i ′ j ′ = X ( i ′ , j ′ , :)) , i ′ ∈ N m 1 + m 2 and j ′ ∈ N d 1 + d 2 .

To construct D train, the training data are uniformly sampled from three parts, namely, X 11 , X 12 and X 12 . The sampling process was performed in two steps. First, we randomly sampled entries from the subtensor X 11 with sampling rate sr 1 train and from the subtensors X 12 and X 21 with sampling rates of sr 2 train . The block X 22 was left unobserved to be the test set. All experiments are implemented in MATLAB on a Linux server equipped with dual Intel E5 2640v4 and 128GB of RAM. The demo code can be found at https://github.com/pingzaiwang/FtSVD4MORCDS .

To generate the training and test data, we follow the above construction procedure by letting m 1 = d 2 = 80 and m 2 = d 2 = 120 , and the rank r = 15 . The singular value decay γ is fixed to 1 . The sample ratio of X 11 is simply fixed to be sr 1 train = 10% . The transform matrix M is adopted the DFT matrix. In the single training ERM and over-parameter training step, the estimated rank is set to 2 r and 4 r , respectively. For simplicity, we let κ ∈ [0 . 1 , 3] and sr 2 train ∈ [50% , 90%] . For each experimental setting, we randomly ran the experiments for four times and computed the average test risk and standard deviation.

In Figs. 3 and 6, we depict the test risk and standard deviation over κ and the percentage of training data. On the left, we observe that as κ increases, indicating a more severe covariate shift, the test risk of both algorithms rises, but the ERM-DS algorithm maintains a significant advantage over the single-stage ERM, with the performance gap widening for larger κ values. On the right, the results demonstrate that the ERM-DS algorithm achieves lower test risks compared to the single-stage ERM under varying training data sizes, highlighting its robustness even with limited training samples.

Overall, the ERM-DS algorithm achieves lower test risks and stronger generalization capabilities compared to the single-stage ERM approach. It is crucial to note that these experiments serve only as a conceptual validation of the proposed Ft-SVD theoretical framework, aiming to preliminarily verify the effectiveness of the theoretical methods in handling CDS. Future work is still needed to further develop more effective and practical algorithms for improved performance on real-world data.

B Functional t-Singular Value Decomposition

This section provides a comprehensive exploration of the theoretical underpinnings of the Ft-SVD framework. It begins with an overview of the basics of t-SVD, setting the stage for the introduction of Ft-SVD. This section includes detailed proofs of key properties, as outlined in Theorems 1 and 2. Additionally, the appendix examines the low-rank approximability of smooth functions and presents the proof of Theorem 3, highlighting the theoretical contributions and foundational aspects of Ft-SVD.

B.1 Preliminaries of t-Singular Value Decomposition

We first introduce some basic notations and concepts to lay the foundation for the subsequent discussions.

Basic notations. For any positive integer n ∈ N , let [ n ] := { 1 , · · · , n } be the set of integers from 1 to n . We use lowercase bold letters (e.g., a ∈ R m ) for vectors, and uppercase bold letters (e.g., A ∈ R m × n ) for matrices. Following the standard notations in [26], we refer to a third-order tensor of size 1 × 1 × K as a t-scalar, denoted by an underlined lowercase letter, e.g., x ; a third-order tensor of size d × 1 × K as a t-vector, denoted by an underlined lowercase bold letter, e.g., x ; and a third-order tensor of size m × n × K as a t-matrix, denoted by an underlined uppercase letter, e.g., X . For A ∈ R K × K , σ i ( A ) ≥ 0 denotes its i -th largest singular value; for symmetric A , λ i ( A ) denotes its i -th largest eigenvalue, and if A ⪰ 0 , A 1 / 2 its matrix square-root.

Given a tensor T , its ℓ p -norm is defined as ∥ T ∥ p := ∥ vec ( T ) ∥ p , and its F-norm is defined as ∥ T ∥ F := ∥ vec ( T ) ∥ 2 , where vec ( · ) denotes the vectorization operation of a tensor [28]. We also use ∥·∥ to represent the ℓ 2 -norm of vectors, F-norm of matrices and tensors for notation simplicity. For T ∈ R m × n × K , we use T ( i ) or T (: , : , i ) to denote its i -th frontal slice. The inner product between two tensors A and B is defined as ⟨ A , B ⟩ := vec ( A ) ⊤ vec ( B ) . | · | denotes the absolute value of a scalar or the cardinality of a set, and ◦ denotes the function composition operation. For simplicity, let

denote the tensor obtained by applying the M transform to T , as defined in Eq. (1). In the following, we will frequently use ( ˘ · ) and ( ˘ · ) ( i ) to represent M ( · ) and M ( · ) ( i ) , respectively. Here, ( ˘ · ) denotes the tensor or t-embedding after the M transform, while ( ˘ · ) ( i ) refers to its i -th frontal slice, i.e., the i -th frequency component (or the i -th sub-domain).

Concepts related to t-SVD. Due to space limitations, some concepts related to t-SVD were omitted in the main text. We provide additional explanations here, as these concepts are crucial for understanding the properties and operational rules of t-SVD.

Definition 4 (Frontal-slice-wise product [35]) . The frontal-slice-wise product of any two tensors A ∈ R m × n × K and B ∈ R n × k × K , denoted by A ⊙ B , is defined as a tensor T such that

where · denotes the standard matrix multiplication. The frontal-slice-wise product performs matrix multiplication on each frontal slice of the tensors, resulting in a new tensor.

Definition 5 ( M -block-diagonal matrix) . The M -block-diagonal matrix of any tensor T ∈ R m × n × K , denoted by ¯ T , is the block diagonal matrix whose diagonal blocks are the frontal slices of ˘ T := M ( T ) :

This concept arranges the slices of a tensor in the frequency domain into a block diagonal matrix, facilitating the theoretical analysis of t-SVD.

We further provide some definitions and properties related to t-SVD:

Definition 6 ([25]) . The t-transpose of a tensor T ∈ R m × n × K under the M transform (as shown in Eq. (1)), denoted by T ⊤ , satisfies

In other words, the t-transpose performs a transpose on each slice in the frequency domain and then transforms back to the time domain. This operation is one of the foundations of t-SVD theory.

Definition 7 ([25]) . The t-identity tensor I ∈ R m × m × K under the M transform satisfies that each frontal slice of M ( I ) is an K × K identity matrix, i.e.,

It is easy to verify that T ∗ M I = T and I ∗ M T = T hold for appropriate dimensions. The t-identity tensor plays a role similar to the identity matrix in t-SVD.

Definition 8 ([25]) . A tensor Q ∈ R d × d × d 3 is called t-orthogonal under the M transform if it satisfies

T-orthogonality is an important property of tensor transformations, ensuring that the inner product and norm of tensors remain invariant before and after the transformation.

Definition 9 ([26]) . A tensor is called f-diagonal if all its frontal slices are diagonal matrices.

F-diagonal tensors play a central role in t-SVD decomposition, similar to the singular value matrix in matrix SVD.

We present some basic facts about t-product:

(a) The t-product of two t-scalars is commutative:

(b) The ℓ 2 -norm of a t-scalar is defined as

(c) The square of the difference between two t-scalars satisfies ∥ a -b ∥ 2 ≤ 2( ∥ a ∥ 2 + ∥ b ∥ 2 ) .

(d) The square of the sum of three t-scalars satisfies ∥ a + b + c ∥ 2 ≤ 3( ∥ a ∥ 2 + ∥ b ∥ 2 + ∥ c ∥ 2 ) .

(e) For any f , g ∈ R m × 1 × K , we can upper bound ∥ f ⊤ ∗ M g ∥ 2 by ∥ f ∥ 2 ∥ g ∥ 2 due to:

The above inequalities characterize the properties of t-product from different perspectives and serve as the foundation for subsequent theoretical analyses.

Tensor Tubal-rank and tensor multi-rank as measures of low-tankness in the transformed domain. The tensor singular value decomposition (t-SVD) of T ∈ R m × n × K under the transform M in Eq. (1) is given by:

where U ∈ R m × m × K and V ∈ R n × n × K are t-orthogonal tensors, S ∈ R m × n × K is an f-diagonal tensor, and ( · ) ⊤ denotes the t-transpose.

According to Definition 2, the tubal rank of tensor T is the number of non-zero tubes in S in Eq. (2), i.e.,

̸

The tubal rank is also equal to the maximum rank of the frontal slices of ˘ T , i.e.,

which means that the tubal rank is a complexity measure of the low-rankness in the transformed domain. Within the context of t-SVD, we have another rank definition:

Definition 10 (Tensor multi-rank [60]) . The multi-rank of T ∈ R m × n × K under the transform M in Eq. (1) is defined as the vector of matrix ranks of all the frontal slices of M ( T ) , i.e.,

The relationship between the tubal rank and multi-rank is given by

where ∥ · ∥ ∞ denotes the ℓ ∞ -norm of a vector.

The multi-rank can be seen as a fine-grained low-rankness measure in the transformed domain compared to the tubal rank. It captures the notion of structural simplicity by assigning a rank value to each frontal slice of the transformed tensor. A low rank value for a frontal slice suggests that the corresponding frequency component of the data can be well-approximated by a low-dimensional subspace, indicating a strong correlation or pattern within that slice. Conversely, a high rank value implies that the corresponding frequency component is more complex and requires a higherdimensional subspace to capture its information content. The multi-rank provides a more detailed characterization of the low-rank structure of a tensor in the transformed domain. While the tubal rank gives a single value representing the overall low-rankness of the tensor, the multi-rank offers a vector of rank values, each corresponding to a specific frontal slice. This allows for a more nuanced analysis of the tensor's complexity along its different frequency components.

B.2 Functional t-Singular Value Decomposition

B.2.1 Proof of the Ft-SVD Theorem

The Functional t-Singular Value Decomposition (Ft-SVD) in Theorem 1 is an extension of the traditional t-SVD framework to infinite and continuous feature domains. This theoretical approach enables the representation of data and functions defined on these domains while preserving the core properties of t-SVD. By generalizing t-SVD to functional settings, Ft-SVD facilitates the development of efficient and robust algorithms for learning vector-valued functions, addressing complex interdependencies among multiple outputs, and solving related problems in a unified and principled manner. We provide the proof of Theorem 1 as follows.

Proof of Theorem 1. The key idea of proving the Functional t-SVD is to define appropriate Hilbert spaces and linear operators, and then use the spectral theorem for compact operators to establish the existence and properties of the Functional t-SVD.

Step 1: Defining the t-Linear Operators. We first define a R 1 × 1 × K -valued inner product, referred to as the t-inner product, along with the corresponding t-linear operators. Let L 2 ( X ; R 1 × 1 × K ) and L 2 ( Y ; R 1 × 1 × K ) be the set of square-integrable vector-valued functions from X and Y to R 1 × 1 × K ,

respectively. The t-inner product is defined as:

where ∗ M denotes the t-product.

Define the t-linear operator T : L 2 ( Y ; R 1 × 1 × K ) → L 2 ( X ; R 1 × 1 × K ) as:

and its adjoint operator T ∗ : L 2 ( X ; R 1 × 1 × K ) → L 2 ( Y ; R 1 × 1 × K ) as:

To show that T ∗ is indeed the adjoint of T , we verify that ⟨T G,H ⟩ L 2 ( X ; R 1 × 1 × K ) = ⟨ G, T ∗ H ⟩ L 2 ( Y ; R 1 × 1 × K ) :

where we used the commutativity of the t-product for t-scalars to interchange the order of integration and the t-product.

Then, we establish the self-adjointness of the operators T T ∗ and T ∗ T . Consider the operators T T ∗ : L 2 ( X ; R 1 × 1 × K ) → L 2 ( X ; R 1 × 1 × K ) and T ∗ T : L 2 ( Y ; R 1 × 1 × K ) → L 2 ( Y ; R 1 × 1 × K ) . To show that T T ∗ is self-adjoint, we verify that ⟨T T ∗ F, G ⟩ L 2 ( X ; R 1 × 1 × K ) = ⟨ F, T T ∗ G ⟩ L 2 ( X ; R 1 × 1 × K ) :

where we used the adjoint property of T ∗ in the first equality. The self-adjointness of T ∗ T can be shown similarly.

Step 2: Establishing Spectral Properties in the Transformed Domain. The next step is to transform the operators to the transformed domain defined by transform M in Eq. (1) and establish their spectral properties. Define ˘ T := M ( T ) : L 2 ( Y ; R 1 × 1 × K ) → L 2 ( X ; R 1 × 1 × K ) and ˘ T ∗ := M ( T ∗ ) : L 2 ( X ; R 1 × 1 × K ) → L 2 ( Y ; R 1 × 1 × K ) as the transformed operators for any vector-valued functions G ∈ L 2 ( Y ; R 1 × 1 × K ) and H ∈ L 2 ( X ; R 1 × 1 × K ) , respectively:

where M is the invertible linear transform in Eq. (1). We similarly define operators M ( T T ∗ ) : L 2 ( X ; R 1 × 1 × K ) → L 2 ( X ; R 1 × 1 × K ) and M ( T ∗ T ) : L 2 ( Y ; R 1 × 1 × K ) → L 2 ( Y ; R 1 × 1 × K ) by

We proceed to show that:

To verify these identities, consider any H ∈ L 2 ( X ; R 1 × 1 × K ) :

where ⊙ denotes the frontal slice-wise product, and step ( i ) applies the definition of the t-product, with the notation ˘ F ( x, y ) = M ( F ( x, y )) . This verifies that M ( T T ∗ ) = ˘ T ˘ T ∗ . A similar approach confirms M ( T ∗ T ) = ˘ T ∗ ˘ T .

We now characterize the frequency-specific behavior of T T ∗ and T ∗ T by performing eigendecompositions on each of their K frequency 'component' (i.e., the 'frontal slices' of operators ˘ T ˘ T ∗ and ˘ T ∗ ˘ T , respectively). Specifically, for any H ∈ L 2 ( X ; R 1 × 1 × K ) , we define the operator M ( T T ∗ ) ( i ) : L 2 ( X ; R ) → L 2 ( X ; R ) by

as the ' i -th frequency componen' of T T ∗ in the transformed domain induced by transform M ( · ) . Thus, we have:

where · denotes the standard multiplication.

Next, we show that M ( T T ∗ ) ( i ) is a compact and self-adjoint operator on the Hilbert space L 2 ( X ; R ) .

(1) Compactness : Note that one condition of Theorem 1 is that the vector-valued function F ( x, y ) is square-integrable, implying that M ( F ( x, y ) ( i ) ) satisfies the Hilbert-Schmidt condition:

where ∥ · ∥ denotes the ℓ 2 -norm of t-scalars (i.e., vectors), (i) holds since M is an orthogonal matrix in Eq. (1), and (ii) is due to the square-integrability of F ( x, y ) .

(2) Self-adjointness : It can be readily verified that M ( T T ∗ ) ( i ) is also a self-adjoint operator on the Hilbert space L 2 ( X ; R ) , following equations analogous to Eq. (10), and thus we omit it.

Similarly, the operators M ( T ∗ T ) ( i ) also satisfies compactness and self-adjointness.

Then, by the spectral theory of compact linear operator in Hilbert spaces [56], there exist orthonormal eigenfunctions { u i,j } ∞ j =1 and { v i,j } ∞ j =1 and non-descending non-negative eigenvalues { ω i,j } ∞ j =1 such that:

Then, for each i ∈ [ K ] , we can express ˘ F ( x, y ) ( i ) using the eigenfunctions and eigenvalues as follows:

which is equivalent to the slice-wise product formulation:

where ˘ φ j ( x ) , ˘ ς j , ˘ ψ j ( y ) ∈ R 1 × 1 × K satisfy

Step 3: Transforming Back to the Original Domain. In this final step, we apply the inverse transform M -1 from Eq. (1) to return to the original domain and construct the Functional t-SVD of F . The decomposition of F is given by:

where the t-singular values and t-singular functions are defined as follows:

The orthonormality conditions

result from the orthonormality of { u i,j } ∞ j =1 and { v i,j } ∞ j =1 (from Eq. (11)) and the properties of the transform M ( · ) specified in Eq. (1). This concludes the proof.

B.2.2 Proof of the r -Term Truncated Ft-SVD Theorem

Theorem 2 provides an extent of the Eckart-Young theorem for t-SVD [27] to the proposed Ft-SVD framework. It highlights the optimality of the truncated Ft-SVD in approximating vector-valued functions, making it a valuable tool for applications such as function compression, denoising, and other tasks that benefit from low-rank approximations of functions. The proof is given as follows.

Proof of Theorem 2. Let G ∈ L 2 ( X × Y ; R 1 × 1 × K ) be an arbitrary r -term t-product expansion:

where a i ∈ L 2 ( X ; R 1 × 1 × K ) , b i ∈ L 2 ( Y ; R 1 × 1 × K ) , λ i ∈ R 1 × 1 × K .

Our goal is to show that

for all G ∈ K r , where K r is the set of all r -term t-product expansions.

Recall that ˘ F ( i ) = M ( F )(: , : , i ) denotes the i -th ( i ∈ [ K ] ) frequency component of F in the transformed domain induced by Eq. (1). According to the construction of the functional t-singular

value decomposition in the proof of Theorem 1, we have:

where the sequences (˘ σ ( i ) j ) and ( ˘ λ ( i ) j ) are in non-ascending order, and the sets { ˘ ϕ ( i ) j } ∞ j =1 , { ˘ ψ ( i ) j } ∞ j =1 , { ˘ a ( i ) j } r j =1 , and { ˘ b ( i ) j } r j =1 are orthonormal bases in their respective spaces:

Now, let's consider the approximation error:

The last equality holds due to the orthonormality of the bases { ˘ ϕ ( i ) j } ∞ j =1 and { ˘ ψ ( i ) j } ∞ j =1 . The inequality in the sixth line becomes an equality when

This choice of a j , λ j , and b j corresponds to the best L 2 -approximation by finite sums of functions with separable variables [48, 47].

Therefore, we have shown that ∥ F -F r ∥ 2 L 2 ( X×Y ; R 1 × 1 × K ) = ∑ ∞ j = r +1 ∥ σ j ∥ 2 2 , which is the minimum approximation error among all r -term t-product expansions. This completes the proof.

B.2.3 Smooth Functions Exhibit Favorable Low-rank Approximability under Ft-SVD

The Sobolev spaces are a family of function spaces that play a crucial role in the theory of partial differential equations, signal processing and machine learning [3, 4, 29, 42]. They are defined by combining conditions on the function itself and its derivatives up to a certain order.

In our settings, let Y ⊂ R D 2 be a domain satisfying the strong local Lipschitz condition, and let f : X × Y → R 1 × 1 × K be a vector-valued function with components f ( i ) ( x, y ) , i = 1 , . . . , K .

Suppose there exists a constant s > 0 such that f ( i ) ∈ L 2 ( X , H s ( Y )) for all i , where H s ( Y ) denotes the s -order Sobolev space on Y . The function class L 2 ( X , H s ( Y )) is a type of mixed-norm Sobolev space that arises when dealing with functions defined on a product domain X × Y , where X is a domain in R D 1 and Y is a domain in R D 2 satisfying the strong local Lipschitz condition. The strong local Lipschitz condition on a domain Y ⊂ R D 2 requires that, locally around each boundary point y 0 ∈ ∂ Y , there exists a Lipschitz parametrization of the boundary, ensuring a certain level of regularity and avoiding pathological features like cusps, which is crucial for establishing various results in the theory of Sobolev spaces.

For a vector-valued function f : X × Y → R 1 × 1 × K with components f ( i ) ( x, y ) , i = 1 , . . . , K , the condition f ( i ) ∈ L 2 ( X , H s ( Y )) means that each component function f ( i ) belongs to the mixed-norm Sobolev space L 2 ( X , H s ( Y )) . The space L 2 ( X , H s ( Y )) is defined as the set of all vector-valued functions g : X × Y → R 1 × 1 × K with components g ( i ) ( x, y ) , such that for each i = 1 , . . . , K , the following norm is finite:

where ∥ g ( i ) ( x, · ) ∥ H s ( Y ) denotes the Sobolev norm of order s for the function g ( i ) ( x, · ) considered as a function of the y variable alone, with x treated as a parameter. More precisely, the Sobolev norm ∥ g ( i ) ( x, · ) ∥ H s ( Y ) is defined as:

where D α y denotes the weak (or distributional) derivative of order α with respect to the y variable, and the sum is taken over all multi-indices α = ( α 1 , α 2 , . . . , α D 2 ) with | α | = α 1 + α 2 + · · · + α D 2 ≤ s .

Theorem 3 quantifies the relationship between the smoothness of a vector-valued function and the decay of its t-singular values, as well as the approximation error of its truncated t-SVD. It provides a theoretical foundation for using t-SVD to approximate smooth vector-valued functions and highlights the role of the function's smoothness and the domain's dimensionality in the approximation accuracy. The proof of Theorem 3 is given as follows:

Proof of Theorem 3. Let f : X × Y → R 1 × 1 × K be a vector-valued function with components f ( i ) ( x, y ) , i = 1 , . . . , K . The condition f ( i ) ∈ L 2 ( X , H s ( Y )) implies that each component function f ( i ) belongs to the mixed-norm Sobolev space L 2 ( X , H s ( Y )) .

Consider the i -th frequency component ˘ f ( i ) ( x, y ) of f in the transformed domain induced by the transform M defined in Eq. (1). By the definition of M ( · ) , we have

which is a linear combination of the component functions f ( k ) ( x, y ) in the original domain. Since each f ( k ) ∈ L 2 ( X , H s ( Y )) , it follows that ˘ f ( i ) ∈ L 2 ( X , H s ( Y )) as well.

Next, we apply Theorem 3.2 from Ref. [21] to the function ˘ f ( i ) . This theorem states that for a function g ∈ L 2 ( X , H s ( Y )) , the singular values σ j ( g ) satisfy the decay estimate

where C ext ( Y , s ) is the extension constant that depends on Y and s only, and C em ( D 2 , s ) is the embedding constant from ℓ D 2 / ( D 2 +2 s ) , 1 to ℓ D 2 / ( D 2 +2 s ) , ∞ given by

with Γ( · ) denoting the Gamma function.

Applying the estimate (15) to each frequency component ˘ f ( i ) , we obtain

Now, recall from the construction of the functional t-SVD in the proof of Theorem 1 that the t-scalars σ j satisfy

Combining this with the previous estimate, we deduce that

Finally, according to Theorem 2, the rankr approximation error of f can be bounded as follows:

which completes the proof.

C Additional Explanations of Ft-SVD Framework for MOR under CDS

C.1 More Explanations about the Hilbert t-Module, t-Bilinear Embeddings, and Assumptions

C.1.1 About the Definition of Hilbert t-Module

Definition 11 (Ring of K -dimensional vectors with t-product) . Let R := R 1 × 1 × K be the set of all real t-scalars of K -dimensionality which is isomorphic to R K . For any a, b ∈ R , we define their t-product as:

where M : R → R is the linear transform induced by a given K × K orthogonal matrix M in Eq. (1) , and the slice-wise-product ⊙ degenerates to the element-wise product here.

( R , + , ∗ M ) forms a ring with the t-product and standard vector addition, which we call the ring of K -dimensional real vectors with t-product. Specifically, for any a, b, c ∈ R and α, β ∈ R , the following properties hold:

(Associativity of addition) ( a + b ) + c = a +( b + c ) .

(Commutativity of addition) a + b = b + a .

(Additive identity) There exists an element 0 ∈ R such that a + 0 = a for all a ∈ R .

(Additive inverses) For each a ∈ R , there exists an element -a ∈ R such that a +( -a ) = 0 .

(Associativity of multiplication) ( a ∗ M b ) ∗ M c = a ∗ M ( b ∗ M c ) .

(Distributivity of multiplication over addition) a ∗ M ( b + c ) = a ∗ M b + a ∗ M c and ( a + b ) ∗ M c = a ∗ M c + b ∗ M c .

(Multiplicative identity) There exists an element 1 ∈ R such that a ∗ M 1 = a for all a ∈ R .

(Scalar multiplication) ( αβ ) a = α ( βa ) , α ( a + b ) = αa + αb , and ( α + β ) a = αa + βa .

Definition 12 (Hilbert t-Module over R ) . Let R be the ring of K -dimensional real vectors with t-product as defined above. A Hilbert t-Module over R is a module M over R equipped with an R -valued inner product ⟨· , ·⟩ M : M×M→R satisfying the following conditions:

(Conjugate symmetry) ⟨ f , g ⟩ M = ⟨ g , f ⟩ M for all f , g ∈ M , where the overline denotes the element-wise complex conjugate. 13

(Linearity in the second argument) ⟨ f , g 1 + g 2 ⟩ M = ⟨ f , g 1 ⟩ M + ⟨ f , g 2 ⟩ M and ⟨ f , g ∗ M a ⟩ M = ⟨ f , g ⟩ M ∗ M a for all f , g , g 1 , g 2 ∈ M and a ∈ R .

(Positivity) ⟨ f , f ⟩ M ≥ 0 (element-wise) for all f ∈ M , with equality if and only if f = 0 .

The Hilbert t-Module M is required to be complete with respect to the ℓ 2 -norm induced by the R -valued inner product:

where ( ⟨ f , f ⟩ M ) i denotes the i -th component of the vector ⟨ f , f ⟩ M .

Remark. The ℓ 2 -norm used here differs from the traditional C*-algebra norm in the definition of Hilbert C*-modules, as it does not satisfy the C*-equality ∥ f ∥ 2 = ∥⟨ f , f ⟩ M ∥ R [25, 8]. Despite this difference, we still refer to this structure as a Hilbert t-Module to emphasize its similarities with Hilbert C*-modules.

The benefits of using ℓ 2 -norm for multi-output regression instead of the standard C* norm The choice of the ℓ 2 -norm for Hilbert t-modules in the context of multi-output regression offers several advantages over the standard C* norm:

Geometric interpretation and learning theory: The ℓ 2 -norm, as the Euclidean distance between vectors, provides a natural and intuitive way to measure the similarity between vector-valued functions. This geometric interpretation aligns well with the learning theory for vector-valued functions, where the goal is often to find a function that minimizes the expected risk or empirical risk, which are typically defined using the ℓ 2 -norm.

Computational efficiency and scalability: The ℓ 2 -norm is computationally more efficient than the C* norm, especially for high-dimensional vector-valued functions. This computational advantage is crucial for large-scale multi-output regression problems, where the number of outputs and the dimensionality of the input space can be large. The ℓ 2 -norm allows for the development of more scalable and efficient learning algorithms.

Theoretical foundations and connections: The ℓ 2 -norm is closely related to the theory of Hilbert spaces, which provides a rich set of tools and results for studying vector-valued functions. By using the ℓ 2 -norm, Hilbert t-modules can leverage the well-established theory of Hilbert spaces, including concepts such as orthogonality, projection, and spectral decomposition. These theoretical foundations can guide the design and analysis of multi-output regression algorithms.

Practical applications and existing methods: The ℓ 2 -norm is widely used in various practical applications, such as signal processing, computer vision, and control systems. Many existing multi-output regression methods, such as multi-task learning, multi-label classification, and multi-target regression, are based on the ℓ 2 -norm. By adopting the ℓ 2 -norm, Hilbert t-modules can readily benefit from and contribute to these existing methods and applications.

Although the ℓ 2 -norm may not satisfy all the properties of the standard C* norm, it offers significant benefits for multi-output regression. The use of the ℓ 2 -norm in Hilbert t-modules opens up new opportunities for the development of efficient, scalable, and theoretically grounded multi-output regression methods. It also facilitates the integration of multi-output regression with other areas of machine learning and signal processing, where the ℓ 2 -norm is widely used. The name "Hilbert t-module" highlights the connection to the well-established theory of Hilbert spaces while acknowledging the specific choice of the ℓ 2 -norm and the potential for further generalization and exploration in the context of multi-output regression.

Now we have a Hilbert t-Module M over the ring R of K -dimensional real vectors with t-product. The elements of M can be thought of as vector-valued functions or tensors, and we consider a linear transform M induced by an orthogonal matrix M that allows us to represent these elements in the frequency domain. We also define the associated Hilbert space H j as follows:

Definition 13 (Hilbert space H j ) . For each j = 1 , . . . , K , let H j be a Hilbert space over the field of real numbers R . We assume that for any element f ∈ M , its j -th frequency component ˘ f ( i ) := M ( f ) ( i ) belongs to H j , i.e., ˘ f ( i ) ∈ H j . The Hilbert space H j is equipped with an inner product ⟨· , ·⟩ H j : H j ×H j → R satisfying the following properties:

1. (Conjugate symmetry) ⟨ x , y ⟩ H j = ⟨ y , x ⟩ H j for all x , y ∈ H j .

2. (Linearity in the second argument) ⟨ x , y 1 + y 2 ⟩ H j = ⟨ x , y 1 ⟩ H j + ⟨ x , y 2 ⟩ H j and ⟨ x , α y ⟩ H j = α ⟨ x , y ⟩ H j for all x , y , y 1 , y 2 ∈ H j and α ∈ R .

3. (Positivity) ⟨ x , x ⟩ H j ≥ 0 for all x ∈ H j , with equality if and only if x = 0 .

The Hilbert space H j is complete with respect to the norm induced by the inner product: ∥ x ∥ H j = √ ⟨ x , x ⟩ H j for all x ∈ H j .

In essence, H j is a standard Hilbert space associated with the j -th frequency component of the elements in the Hilbert t-Module M . The inner product and norm in H j satisfy the usual properties of a Hilbert space, and the completeness ensures that limits of Cauchy sequences in H j also belong to H j .

This definition allows us to work with the frequency components of the elements in the Hilbert t-Module using the tools and properties of Hilbert spaces. In particular, it enables us to apply the

Cauchy-Schwarz inequality in each H j , which is a key step in proving the inequality between the norm of the inner product and the product of the norms in the Hilbert t-Module.

We also have a Cauchy-Schwarz-like inequality in Hilbert t-Modules as follows.

Lemma C.1 (Cauchy-Schwarz inequality in Hilbert t-Modules) . It holds for any f , g ∈ M that

Proof. Let f , g ∈ M be two elements of the Hilbert t-Module. We begin by expanding the left-hand side of the inequality:

which completes the proof.

C.1.2 One the Expressive Power of Hilbert t-Module and embeddings

The Functional t-SVD theorem states that for a square-integrable vector-valued function F : X×Y → R 1 × 1 × K , there exist orthonormal sets of functions { ϕ i } ∞ i =1 ⊂ L 2 ( X ; R 1 × 1 × K ) and { ψ i } ∞ i =1 ⊂ L 2 ( Y ; R 1 × 1 × K ) , and a sequence of t-scalars { σ i } ∞ i =1 ⊂ R 1 × 1 × K , such that

Now, consider the Ground Truth Representation assumption, which posits the existence of a Hilbert t-Module ( M , ⟨· , ·⟩ M ) and two embeddings f ⋆ : X ↦→ M and g ⋆ : Y ↦→ M , such that the true function h ⋆ ( x, y ) can be expressed as the inner product of these embeddings:

Assuming that the true function h ⋆ is square-integrable, i.e., h ⋆ ∈ L 2 ( X × Y ; R 1 × 1 × K ) , the Functional t-SVD theorem implies that it admits a representation of the form

In this context, the Hilbert t-Module M can be interpreted as the space spanned by the left singular functions { ϕ i } ∞ i =1 , and the embeddings f ⋆ and g ⋆ can be defined as

where the square root operation ( · ) 1 2 is performed element-wisely. With these definitions, the inner product of the embeddings in the Hilbert t-Module M recovers the true function:

This demonstrates that, under the square-integrability assumption, the Functional t-SVD theorem provides a constructive way to obtain a Hilbert t-Module and embeddings that satisfy the t-bilinear representation assumption (Assumption 1). The theorem reveals a deep connection between the spectral decomposition of vector-valued functions and the geometric structure of Hilbert t-Modules, which can be exploited to develop efficient algorithms for multi-output regression and analysis. The use of the t-product ( ∗ M ) highlights the role of the tensor structure in this framework, and the consistent notation for the singular functions ( ϕ i and ψ i ) emphasizes their fundamental importance in the construction of the Hilbert t-Module and the associated embeddings.

C.1.3 More Explanations for the Assumptions

Explanation of Assumption 1. Assumption 1 is the cornerstone of the proposed theoretical framework for multi-output regression under CDS. It establishes a connection between the ground truth function and the geometry of a Hilbert t-Module, which is a generalization of a Hilbert space that accommodates vector-valued functions.

Part (I) of the assumption postulates that the ground truth function h ⋆ ( x, y ) , which maps input pairs ( x, y ) to vector-valued outputs, admits a t-bilinear representation in a Hilbert t-Module ( M , ⟨· , ·⟩ M ) . This means that there exist two embeddings, f ⋆ : X → M and g ⋆ : Y → M , such that h ⋆ ( x, y ) can be expressed as the inner product of these embeddings, i.e., h ⋆ ( x, y ) = ⟨ f ⋆ ( x ) , g ⋆ ( y ) ⟩ M . The t-bilinear form is a natural extension of the bilinear form used (in single-output learning) [46], and it allows us to capture the multi-dimensional structure of the problem.

The assumption further states that this t-bilinear representation is the Bayes optimal predictor on both the training distribution D train and the test distribution D test . This means that among all measurable functions, the t-bilinear representation minimizes the expected loss on both distributions. This assumption provides a clear learning objective and a benchmark for the performance of any learning algorithm.

Part (II) of the assumption imposes regularity and boundedness conditions on the output variable z and the embeddings f ⋆ ( x ) and g ⋆ ( y ) . It assumes that the squared ℓ 2 -norm of z is bounded by B 2 almost surely under the training distribution D train , and the Hilbert t-Module norms of the embeddings are uniformly bounded by the same constant B . These conditions are crucial for deriving theoretical guarantees and ensuring the well-posedness of the learning problem. They prevent pathological cases and ensure that the learning problem is tractable.

Intuitively, Assumption 1 tells us that the ground truth function has a simple and interpretable structure (t-bilinear representation) that is optimal on both the training and testing distributions. Moreover, the output and the embeddings are well-behaved (bounded). This sets the stage for the learning problem and provides a clear target for any learning algorithm.

Explanation of Assumption 2. Assumption 2 characterizes the relationship between the training and testing distributions in the context of CDS. In real-world applications, it is common for the testing distribution to differ from the training distribution. CDS is a specific type of distribution shift where the testing distribution includes novel combinations of features that are not present in the training

distribution. Following Ref. [46], we can also relax Assumption 2 through a probabilistic manner as follows:

Assumption 6 (Coverage of training and test distribution (probabilistic version), Assumption 2.2b in Ref. [46]) . There exist constants κ tst , κ trn > 0 , η tst , η trn ∈ (0 , 1] and marginal distributions D X , 1 , D X , 2 over X , and D Y , 1 , D Y , 2 over Y , with product measures D i ⊗ j := D X ,i ⊗D Y ,j , such that the following conditions in Radon-Nikodym derivatives hold for all ( x, y ) ∈ X × Y :

Assumption 6 formalizes this notion by introducing marginal distributions D X , 1 , D X , 2 over the input space X and D Y , 1 , D Y , 2 over the output space Y . The product measures D i ⊗ j := D X ,i ⊗ D Y ,j represent different combinations of these marginal distributions.

The assumption requires that the training distribution D train covers the key feature combinations ((1,1), (1,2), (2,1)) of these product measures, while allowing the testing distribution D test to include unseen combinations (e.g., (2,2)). The coverage is quantified by the constants κ trn , κ tst , η trn , η tst, which provide some flexibility in the assumption.

Intuitively, this assumption tells us that the training distribution should be diverse enough to cover the key aspects of the marginal distributions, but it doesn't need to cover all possible combinations. The testing distribution, on the other hand, can include novel combinations that are not present in the training distribution. This is a realistic assumption in many practical scenarios.

Explanation of Assumption 3. Assumption 3 is a technical assumption that controls the impact of covariate shift on the model's performance. Covariate shift refers to the situation where the distribution of the input features changes between the training and testing distributions. Motivated by Assumptionn2.3b in Ref. [46] we can also consider a slacked version of Assumption 3 as follows.

Assumption 7 (Controlled covariate shifts, slacked version) . There exists a κ cov ≥ 1 and η cov ∈ (0 , 1] such that, for any v ∈ M , the following inequalities hold:

The assumption states that for any t-vector v in the Hilbert t-Module M , the expected squared norm of the t-inner product between v and the embeddings f ⋆ ( x ) and g ⋆ ( y ) under the unseen distributions D X , 2 and D Y , 2 is bounded by a constant factor κ cov times the same quantity under the training distributions D X , 1 and D Y , 1 , plus a small constant η cov times the squared Hilbert t-Module norm of v .

Intuitively, this assumption ensures that the behavior of the embeddings doesn't change too much under covariate shift. It bounds the worst-case impact of the shift on the model's performance. This is a key ingredient in providing generalization guarantees under CDS.

Without this assumption, the model's performance on the testing distribution could be arbitrarily bad, even if it performs well on the training distribution. The assumption ensures that the performance degradation is controlled, which is essential for learning under distribution shift.

Explanation of Assumption 4. Assumption 4 postulates favorable spectral properties of the ground truth embeddings f ⋆ and g ⋆ . The spectral properties refer to the behavior of the singular values of the embeddings' t-covariance operators.

Part (I) of the assumption, termed 'Balanced embeddings,' posits that the embeddings are balanced in an appropriate basis of the Hilbert t-Module, such that their t-covariances Σ f ⋆ and Σ g ⋆ equal a common t-covariance Σ ⋆ 1 ⊗ 1 . This assumption might initially appear restrictive, but mathematically, it is very mild. Specifically, we can always find a pair of invertible t-linear transformations T f and T g

such that:

In other words, by applying suitable transformations to the embeddings f ⋆ and g ⋆ , their t-covariances can be aligned to a common t-covariance Σ ⋆ 1 ⊗ 1 in a transformed basis of the Hilbert t-Module. Practically, the assumption of balanced embeddings can be viewed as a form of normalization or standardization. By aligning their t-covariances, we ensure that the embeddings have similar scales and orientations in the Hilbert t-Module, facilitating the learning process and simplifying theoretical analysis.

Part (II) of the assumption, termed 'Polynomial spectral decay,' assumes that each frequency component of Σ ⋆ 1 ⊗ 1 in the transformed domain induced by the transform M ( · ) exhibits a polynomial singular value decay, potentially with different decay rates γ i for different components.

Intuitively, this assumption tells us that the information in the embeddings is concentrated in a few principal directions (corresponding to the large singular values), and the importance of the remaining directions decays polynomially. This is a common assumption in many learning problems, and it allows us to effectively approximate the embeddings using low-rank structures.

The polynomial decay assumption is more flexible than the often-used exponential decay assumption, as it allows different frequency components to decay at different rates. This is particularly useful in the context of multi-output regression, where different outputs may have different spectral properties.

Explanation of Assumption 5. Assumption 5 essentially states that the error between the optimal rankk embeddings and the ground truth function h ⋆ over the training data D train is bounded by a constant factor κ apx times the error over a reference distribution D 1 ⊗ 1 . This assumption ensures that the training data provides a good approximation of the ground truth function in each frequency sub-domain.

Intuitively, this assumption implies that the training data D train is sufficiently representative of the true function h ⋆ in each frequency sub-domain induced by the transformation M ( · ) . It guarantees that if we find embeddings that well approximate h ⋆ on the training data, they will also perform well on the reference distribution D 1 ⊗ 1 . More specifically, ApxErr ( i ) k ( x, y ) measures the squared difference between the inner product of the optimal rankk embeddings ( P k M ( f ⋆ ( x )) ( i ) and P k M ( g ⋆ ( y )) ( i ) ) and the i -th frequency component of the true function ( M ( h ⋆ ( x, y )) ( i ) ) for a given pair of inputs ( x, y ) . By taking the expectation of this error over the training distribution D train and the reference distribution D 1 ⊗ 1 , we obtain a measure of how well the rankk embeddings approximate the true function in each frequency sub-domain.

The assumption states that the expected approximation error over the training data is bounded by a constant factor κ apx times the expected error over the reference distribution, for all frequency sub-domains i ∈ [ K ] . This means that the training data provides a good approximation of the true function in each sub-domain, relative to the reference distribution.

The significance of this assumption lies in its importance for generalization analysis under CDS. By ensuring that the training data adequately represents the true function within each frequency sub-domain, it enables the learning of embeddings that effectively capture essential characteristics of the true function. This, in turn, allows the learned model to generalize well to unseen combinations of input features, provided that the spectral properties of the true function are favorable (e.g., exhibiting polynomial decay as assumed in Assumption 4), thereby accommodating the multi-output nature of the problem.

C.2 More Details of the Algorithms

C.2.1 Limitations of ERM for multi-output regression under CDS.

In the ERM model (5), the function classes F r , G r are given as follows:

Assumption 8 (Hypothesis classes in the ERM Model (5)) . For each k ∈ N , there exist function classes F k ⊆ {X → R k × 1 × K } and G k ⊆ {Y → R k × 1 × K } satisfy:

(a) There exist some ( f , g ) ∈ F k ×G k such that ⟨ f ( x ) , g ( y ) ⟩ t := ( f ( x )) ⊤ ∗ M g ( y ) = ⟨ f ⋆ k ( x ) , g ⋆ k ( y ) ⟩ M for all ( x, y ) ∈ X × Y .

(b) The function classes F k and G k have their ϵ -covering numbers bounded as follows: log N ( F k , ϵ, ∥ · ∥ ∞ ) ≤ N F k ( ϵ ) and log N ( G k , ϵ, ∥ · ∥ ∞ ) ≤ N G k ( ϵ ) , where N F k and N G k are non-decreasing functions of k and decreasing functions of ϵ > 0 .

(c) There exists some B > 0 , such that max { sup x ∈X ∥ f ( x ) ∥ M , sup y ∈Y ∥ g ( y ) ∥ M } ≤ B .

This assumption specifies the hypothesis classes F r and G r used in the ERM problem (5). It ensures that these classes are sufficiently expressive to approximate the true embeddings f ⋆ and g ⋆ , while also having controlled complexity in terms of their covering numbers. With these assumptions in place, we can obtain the generalization guarantee for the ERM solution in Theorem 4. However, the single-stage ERM approach has several limitations when applied to multi-output regression under CDS:

Lack of adaptivity to spectral decay patterns : The hypothesis classes F r and G r used in the single-stage ERM approach do not explicitly take into account the potentially varying spectral decay patterns of the ground truth embeddings along different frequency components of the transform M ( · ) . As a result, the learned embeddings may not adequately capture the distinct decay behaviors in different frequency sub-domains, leading to suboptimal approximations.

Global learning and complexity control : The single-stage ERM approach learns the embeddings ˆ f erm and ˆ g erm globally, without considering the specific characteristics of different frequency subdomains. This global learning approach may not be able to adapt to the nuances and variations in the decay patterns across sub-domains. Moreover, the complexity of the hypothesis classes F r and G r is controlled globally through their covering numbers, which may not provide a fine-grained enough complexity control to capture the differences in the decay patterns.

Suboptimal generalization guarantees : Due to the limitations mentioned above, the single-stage ERM approach may result in suboptimal generalization guarantees for multi-output regression under CDS. The learned embeddings may not generalize well to new feature combinations, particularly when the spectral decay patterns vary significantly across different frequency subdomains.

C.2.2 Double-Stage ERM (ERM-DS) for improved multi-output regression under CDS.

To address these weaknesses, we propose a Double-Stage Empirical Risk Minimization (ERM-DS) algorithm. This approach employs a two-stage training process with more specific hypothesis classes to better capture the varying spectral decay patterns across sub-domains. By striking a balance between model complexity and generalization ability, ERM-DS aims to overcome the limitations of the single-stage ERM approach and provide improved generalization guarantees for multi-output regression under CDS.

The framework of Double-Stage Empirical Risk Minimization is first proposed by Ref. [46] to address robust learning under CDS (for single-output learning), our work extends this framework to the more complex setting of multi-output regression with t-bilinear embeddings. By learning vector functions as embeddings in a Hilbert t-module and leveraging tensor algebra, we capture the intricate dependencies among multiple outputs.

To more accurately capture the varying spectral decay patterns of the ground truth along each frequency component of the transform M ( · ) , the algorithm learns the t-bilinear embeddings f ⋆ and g ⋆ by approximating their frequency components ˘ f ⋆, ( i ) and ˘ g ⋆, ( i ) separately in each i -th frequency sub-domain. This is achieved by employing function classes specifically characterized for each sub-domain, allowing the algorithm to adapt to the distinct decay behaviors exhibited by the ground truth in each frequency range. By treating each sub-domain independently and using specialized function classes, the algorithm can better capture the nuances and variations in the decay patterns, leading to a more precise approximation of the t-bilinear embeddings f ⋆ and g ⋆ .

Assumption 9 (Function Approximation in Transformed sub-frequency components Separately) . In any sub-frequency component i ∈ [ K ] , for each k ∈ N , there exist function classes ˘ F ( i ) k ⊆ {X → R k } and ˘ G ( i ) k ⊆ {Y → R k } satisfy:

(a) sup f ∈ ˘ F ( i ) k sup x ∈X ∥ f ( x ) ∥ ≤ B and sup g ∈ ˘ G ( i ) k sup y ∈Y ∥ g ( y ) ∥ ≤ B .

(b) There exist some ( f, g ) ∈ ˘ F ( i ) k × ˘ G ( i ) k such that ⟨ f ( x ) , g ( y ) ⟩ = ⟨ ˘ f ⋆, ( i ) k ( x ) , ˘ g ⋆, ( i ) k ( y ) ⟩ for all ( x, y ) ∈ X ×Y , where ˘ f ⋆, ( i ) k ( x ) , ˘ g ⋆, ( i ) k ( y ) are the projections of ˘ f ⋆, ( i ) ( x ) and ˘ g ⋆, ( i ) ( y ) in the topk eigenspaces of M ( Σ ⋆ 1 ⊗ 1 ) ( i ) , respectively.

(c) The function classes ˘ F ( i ) k and ˘ G ( i ) k have their ϵ -covering numbers bounded as follows: log N ( ˘ F ( i ) k , ϵ, ∥ · ∥ ∞ ) ≤ M ˘ F ( i ) k ( ϵ ) and log N ( ˘ G ( i ) k , ϵ, ∥ · ∥ ∞ ) ≤ M ˘ G ( i ) k ( ϵ ) , where M ˘ F ( i ) k and M ˘ G ( i ) k are non-decreasing functions of k and decreasing functions of ϵ > 0 .

The algorithm consists of the following key steps:

Step 1: Overparameterized Training. In the first stage, ERM-DS trains an overparameterized model ( ˜ f , ˜ g ) with high capacity to approximate the unknown true predictive functions f ⋆ and g ⋆ . This allows the algorithm to handle the uncertainty in the true predictive functions, which are unknown and potentially intricate. The over-parameterized model is trained by minimizing the empirical risk on labeled training data { ( x 1 ,j , y 1 ,j , z 1 ,j ) } n 1 j =1 sampled i.i.d. from D train . Instead of training ( ˜ f , ˜ g ) within a single optimization problem as in Eq. (5), we consider choosing their frequency components in the transformed domains induced by M ( · ) , separately via K parallel ERM sub-problems as follows:

In this equation, ( ˘ ˜ f ( i ) , ˘ ˜ g ( i ) ) represents the i -th frequency component of the overparameterized model ( ˜ f , ˜ g ) in the transformed domain, and ˘ F ( i ) p and ˘ G ( i ) p denote the function classes for f and g , respectively. The objective is to minimize the squared difference between the inner product of f ( x 1 ,j ) and g ( y 1 ,j ) and the i -th component of the transformed target variable M ( z 1 ,j ) , averaged over the training data.

By solving these K parallel ERM sub-problems, the algorithm learns the frequency components of the overparameterized model in the transformed domains. This approach allows for a more flexible and potentially more accurate approximation of the unknown true predictive functions, compared to training the model within a single optimization problem.

Step 2: t-Covariance Estimation. To leverage the assumption that the true predictive functions have a low-rank structure (Assumption 4), the algorithm estimates the t-covariances of the embeddings using n 2 additional unlabeled examples { ( x 2 ,j , y 2 ,j ) } n 2 j =1 sampled from D 1 ⊗ 1 :

These covariance estimates capture the important directions of variation in the learned embeddings and will be used for dimension reduction in the next step.

Step 3: Dimension Reduction. Using the estimated covariances, the algorithm computes low-rank projections ( ˆ Q ˆ r , ˆ r ) using the DimReduce function in Algorithm 1 with a target ranks r cut ∈ N K and cutoff parameters σ cut ∈ R K in each frequency domain:

where µ is a regularization parameter and I is the t-identity tensor. The reduced-rank embeddings are then obtained by projecting the overparameterized embeddings onto the low-rank t-subspace: h red ( x, y ) := ˜ f ( x ) ⊤ ∗ M ˆ Q ˆ r ∗ M ˜ g ( y ) .

The high capacity of the overparameterized model makes it prone to overfitting, especially when the training data is limited. However, the true predictive functions can be effectively approximated using

Algorithm 1 DimReduce ( X , Y , r , σ ′ )

1: Input: Two t-covariances tensors X , Y ∈ R p × p × K , r = ( r 1 , . . . , r K ) ⊤ ∈ N K , σ ′ = ( σ ′ 1 , . . . , σ ′ K ) ⊤ ∈ R K .

2: Initialize Q = 0 ∈ R p × p × K

3: for i = 1 to K do

5: Compute Σ i := W 1 2 i Y i W 1 2 i .

4: Compute W i := X 1 2 i ( X 1 2 i Y i X 1 2 i ) -1 2 X 1 2 i , where X i = M ( X ) ( i ) , Y i = M ( Y ) ( i ) .

6: Set s i ← max { s ∈ [ r i ] : σ s ( Σ i ) ≥ σ ′ i , σ s ( Σ i ) -σ s +1 ( Σ i ) ≥ σ s ( Σ i ) r i } .

7: Let P s i denote the projection onto the top s i eigenvectors of Σ i .

9: end for

8: Compute Q s i := W -1 2 i P s i W 1 2 i and set Q ( i ) ← Q s i .

10: Return ( M ( Q ) , s ) where s = ( s 1 , . . . , s K ) ∈ N K .

fewer dimensions due to their low-rank structure. By projecting the embeddings onto a low-rank subspace, the algorithm reduces the model's complexity and mitigates overfitting. This dimensionality reduction step helps strike a balance between model capacity and generalization ability. It enables the model to focus on the most important patterns in the data while discarding less informative directions. This step is crucial for improving the model's generalization performance and preventing it from memorizing noise or irrelevant details in the training data.

Step 4: Distillation. In the final stage, the algorithm fine-tunes the reduced-rank embeddings ( ˆ f ds , ˆ g ds ) by approximating their frequency components ( ˘ ˆ f ( i ) ds , ˘ ˆ g ( i ) ds ) separately in the transformed domain induced by M ( · ) . This is achieved by minimizing a combination of two loss functions:

where ν = ( ν 1 , · · · , ν K ) ⊤ ∈ R K is a vector of regularization parameters and the function classes ˘ F ( i ) ˆ r i and ˘ G ( i ) ˆ r i have reduced complexity compared to ˘ F ( i ) p and ˘ G ( i ) p .

The first loss ˆ L ( i ) (3) is the empirical risk in the i -th frequency domain on n 3 additional labeled examples { ( x 3 ,j , y 3 ,j , z 3 ,j ) } n 3 j =1 sampled from D train :

By minimizing this loss, the algorithm ensures that the embeddings accurately capture the relationships between the input features and the target outputs.

The second loss ˆ L ( i ) (4) is a regularization term that encourages consistency with the reduced-rank embeddings h red on n 4 unlabeled examples { ( x 4 ,j , y 4 ,j ) } n 4 j =1 sampled from D 1 ⊗ 1 :

By learning to match the reduced-rank embeddings h red on these unlabeled samples, the final model becomes more robust to distribution shifts.

Intuitively, the distillation step allows the model to learn from both labeled and unlabeled data, leveraging the information captured by the reduced-rank embeddings to improve its generalization ability. The regularization term acts as a guide, encouraging the fine-tuned embeddings to maintain the important patterns and structures learned during the previous steps while adapting to new feature combinations. By combining the empirical risk and the regularization term, ERM-DS strikes a balance between fitting the labeled training data and being robust to distribution shifts. This helps the

model generalize well to unseen feature combinations and ensures stable performance even when the test distribution differs from the training distribution.

C.2.3 The Role of Balanced t-Covariance Operator: Capturing Knowledge under CDS

The t-covariance operator Σ ⋆ 1 ⊗ 1 is central to the Ft-SVD framework for multi-output regression under combinatorial distribution shift, capturing the spectral properties and low-rank structure of the predictive functions across K frequency components. Defined as the common t-covariance of balanced embeddings f ⋆ and g ⋆ on D 1 ⊗ 1 , Σ ⋆ 1 ⊗ 1 reflects the intrinsic low-dimensional structure and shared patterns across feature combinations. Its spectral decay, indicating the eigenvalue decay of its frontal slices in the transformed domain, guides the Ft-SVD framework in aligning embeddings with principal eigenfunctions, enabling effective knowledge transfer under distribution shift.

Estimated from D 1 ⊗ 1 := D X , 1 ⊗ D Y , 1 , the t-covariance operator supports three main steps the ERM-DS algorithm:

Estimation : Using unlabeled samples from D 1 ⊗ 1 to identify significant variation directions in embeddings.

Dimension Reduction : Employing spectral decay to compute low-rank projections that balance capacity and generalization.

Distillation : Constructing a regularization term to align fine-tuned and reduced-rank embeddings, enhancing robustness.

At its core, Σ ⋆ 1 ⊗ 1 acts as a form of transferable 'knowledge' for multi-output problems under combinatorial distribution shifts, capturing and transferring crucial patterns across feature combinations. Its spectral properties and principal eigenfunctions encapsulate the low-dimensional structure, guiding the model to focus on the most transferable features and achieve generalization in shifted distributions. Through Σ ⋆ 1 ⊗ 1 , the Ft-SVD framework and ERM-DS enable effective knowledge transfer and address the complexities of combinatorial distribution shift.

D Analysis of the Proposed Algorithms

In this section, we embark on a comprehensive theoretical analysis of the ERM and ERM-DS algorithms for multi-output regression under the challenging setting of CDS. Our primary goal is to establish rigorous generalization guarantees for these algorithms.

Challenges. In the our settings of multi-output regression under CDS, several key challenges arise. Firstly, the algorithms must be able to effectively capture the complex relationships between the input features and the multiple output variables, even when the training data only covers a limited range of feature combinations. Secondly, the spectral decay patterns of the true embeddings, which encode the intrinsic structure of the data, may vary across different sub-domains (i.e., frequency components), necessitating algorithms that can adapt to these variations. Finally, the presence of distribution shift between the training and test domains can significantly impact the generalization performance, requiring algorithms that can mitigate the effects of this shift.

Section organization. To address these challenges and provide a comprehensive analysis of the ERM and ERM-DS algorithms, we organize this section into three key subsections.

First, we introduce a technical tool that allows us to decompose the excess risk of the algorithms into various interpretable terms, such as the approximation error, statistical error, and distribution shift terms in Appendix D.1. This decomposition forms the foundation of our subsequent analysis and provides valuable insights into the factors influencing the generalization performance.

Second, we focus on the theoretical analysis of the ERM algorithm in Appendix D.2. We present the main theorem that provides generalization guarantees for the ERM solution under certain conditions and outline the key steps in proving this theorem. Through this analysis, we highlight the limitations of the ERM approach, such as its lack of adaptivity to the spectral decay patterns and suboptimal generalization guarantees, motivating the need for a more sophisticated algorithm.

Third, we introduce the ERM-DS algorithm as a response to the limitations of the ERM approach in Appendix D.3. We provide an overview of the key steps of the algorithm, including overparameterization, covariance estimation, dimension reduction, and distillation, and explain how each step contributes to the improved generalization performance. Through a high-level sketch of the proof for the main theorem, we highlight the key ideas and techniques used in each step of the algorithm and discuss how the error terms are controlled by carefully selecting the algorithm parameters and sample sizes.

Addtional notations. To facilitate the analysis, we introduce some additional notations. We use a ≲ b to denote a ≤ c · b for some absolute constant c ; we use a ≲ ⋆ b to denote a ≤ c · b for some c that is at most polynomial in the problem constants κ cov , κ trn , κ apx .

Given a probability distribution D on ( x, y ) ∈ X × Y pairs, we define the excess risk of the embeddings ( ˆ f , ˆ g ) as R ( ˆ f , ˆ g ; D ) := E ( x,y ) ∼D [ ∥⟨ ˆ f ( x ) , ˆ g ( y ) ⟩ M -h ⋆ ( x, y ) ∥ 2 ] , which quantifies the expected squared difference between the product of the embeddings and the true relationship h ⋆ ( x, y ) within a Hilbert t-module M . We often omit the function dependence on ( x, y ) in expectations for brevity. For all embeddings f , g ∈ M and distribution D , we decompose the risk as:

In words, the risk R ( f , g ; D ) measures the expected squared difference between the predicted output f ( x ) ⊤ ∗ M g ( y ) and the true output h ⋆ ( x, y ) under the distribution D . It quantifies how well the embeddings f and g approximate the true relationship h ⋆ on average. The key idea in the risk decomposition is to transform the risk into the frequency domain using the transform M ( · ) and then decompose it into a sum of risks over individual frequency components, such that the tools developed in Ref. [46] can be applied. This is achieved by expressing the risk in terms of the frontal slices of the transformed embeddings ˘ f and ˘ g , and the transformed true output ˘ h ⋆ . By defining the risk ˘ R i ( f , g ; D ) for each frequency component i , we can analyze the performance of the embeddings in a more fine-grained manner. This decomposition allows us to study the behavior of the embeddings in different frequency components and identify the sources of error that contribute to the overall risk. Finally, we can express the total risk R ( ˆ f , ˆ g ; D test ) as the sum of the risks ˘ R i ( ˆ f , ˆ g ; D test ) over all frequency components.

We also define some key quantities related to the spectral properties of the embeddings and the t-covariance operators, including the full-multi-rank embeddings, the t-singular values σ j ( f , g ) and σ j ( ˘ f ( i ) , ˘ g ( i ) ) , the tail sums tail ( i ) ⋆ q , tail ⋆ q ( k ) , and tail ⋆ q ( k ) as follows.

Full-multi-rank embeddings: Given a vector r = ( r 1 , . . . , r K ) ⊤ ∈ N K , we say that a pair of R ∥ r ∥ ∞ × 1 × K -embeddings ( f , g ) are full-multi-rank -r if the r i -th singular value σ r i ( ˘ ˆ f ( i ) , ˘ ˆ g ( i ) ) is strictly positive for each frequency component i ∈ [ K ] . This condition ensures that the embeddings have sufficient expressive power and that their spectral properties are well-defined in each frequency component.

t-singular values σ j ( f , g ) and σ j ( ˘ f ( i ) , ˘ g ( i ) ) : The t-singular values σ j ( f , g ) are defined as the t-singular values of the product of the square roots of the covariance operators E D X , 1 [ f ∗ M f ⊤ ] and E D Y , 1 [ g ∗ M g ⊤ ] :

Similarly, σ j ( ˘ f ( i ) , ˘ g ( i ) ) denotes the singular values of the product of the square roots of the covariance operators in the i -th frequency component:

These singular values capture the interactions between the left and right embeddings in each individual frequency component, providing a more fine-grained characterization of their relationship.

Tail sums tail ( i ) ⋆ q ( k i ) , tail ⋆ q ( k ) , and tail ⋆ q ( k ) : The tail sums measure the decay of the singular values in different frequency components and provide a way to quantify the complexity of the embeddings.

(a) tail ( i ) ⋆ q ( k i ) denotes the sum of the q -th powers of the singular values beyond index k i in the i -th frequency component:

where ˘ σ ⋆, ( i ) j := λ j ( M ( Σ ⋆ 1 ⊗ 1 ) ( i ) ) are the singular values of the covariance operator Σ ⋆ 1 ⊗ 1 in the i -th frequency component.

(b) tail ⋆ q ( k ) denotes the sum of the q -th powers of the singular values beyond index k across all frequency components:

(c) Given a vector k = ( k 1 , . . . , k K ) ⊤ ∈ N K , tail ⋆ q ( k ) is a more general version of tail ⋆ q ( k ) , where the summation is taken beyond index k i in each frequency component i :

These tail sums capture the complexity of the embeddings by measuring the decay of their singular values. A rapidly decaying spectrum (i.e., small tail sums) indicates that the embeddings have a low-rank structure and can be well-approximated by a small number of principal components. Conversely, slowly decaying tail sums suggest that the embeddings are more complex and require a larger number of components to capture their variability.

Furthermore, we introduce the multi-rankk t-projection operator P ⋆ k , which projects an embedding onto the top principal directions in each frequency component.

Definition 14 (Multi-rankk t-projection operator) . Let k = ( k 1 , · · · , k K ) ⊤ ∈ N K . The multi-rankk projection operator P ⋆ k : M→M on the range of Σ ⋆ 1 ⊗ 1 satisfies

where ˘ P k i is the orthogonal projection operator onto the topk i eigenspaces of range( M ( Σ ⋆ 1 ⊗ 1 ) ( i ) ) . P ⋆ k projects the embedding f onto the top k i principal directions in each frequency component, serving the purpose of dimensionality reduction and extracting key information.

Definition 14 introduces the concept of the multi-rankk t-projection operator P ⋆ k , which plays a crucial role in dimensionality reduction and information extraction within the Hilbert t-module framework. This operator is designed to capture the most significant components of an embedding f by projecting it onto the top k i principal directions in each frequency component.

The P ⋆ k operator offers a principled approach to tackle the CDS challenge by exploiting the low-rank structure of the multi-output functions in the transformed domain. By projecting the embeddings onto the top k i principal directions in each frequency component, P ⋆ k effectively captures the most significant patterns and correlations present in the data, even in the presence of CDS. The key idea is that the low-rank structure of the multi-output functions, as revealed by the spectral decomposition of the covariance operator Σ ⋆ 1 ⊗ 1 , remains relatively stable across different feature combinations. In other words, the principal directions along which the functions exhibit the highest variability or energy concentration are likely to be shared among different combinations of input features.

Lemma D.1 (Properties of P ⋆ k ) . Let k = ( k 1 , · · · , k K ) ⊤ ∈ N K . The multi-rankk projection operator P ⋆ k : M→M has the following properties:

Self-adjointness: ⟨ P ⋆ k f , g ⟩ M = ⟨ f , P ⋆ k g ⟩ M .

Idempotence: P ⋆ k P ⋆ k f = P ⋆ k f .

Proof. By definition, we have

holds for all i ∈ [ K ] where ⟨· , ·⟩ H i denotes the inner product of Hilbert space defined in Definition 13. Self-adjointness and idempotence reflect the basic properties of projection operators.

Definition 15 below introduces the concepts of the balanced t-embeddings and balancing t-operator. The main idea behind these concepts is to balance the left and right t-embeddings in a way that preserves their spectral properties while simplifying the analysis.

Definition 15 (Balanced t-embeddings, balancing t-operator) . Given a fixed vector of integers r = ( r 1 , · · · , r K ) ⊤ ∈ N K . For a pair of R ∥ r ∥ ∞ × 1 × K -embeddings ( ˆ f , ˆ g ) , if their t-covariance tensors E D X , 1 [ ˆ f ∗ M ˆ f ⊤ ] = E D Y , 1 [ˆ g ∗ M ˆ g ⊤ ] , then ( ˆ f , ˆ g ) is said to be balanced t-embeddings.

For full-multi-rankr R ∥ r ∥ ∞ × 1 × K -embeddings ( ˆ f , ˆ g ) , an operator T : R ∥ r ∥ ∞ × 1 × K → R ∥ r ∥ ∞ × 1 × K is defined as a balancing t-operator if it satisfies: (a) the pair of t-embeddings ( ˜ f , ˜ g ) = ( T -1 ˆ f , T ˆ g ) is

balanced, and (b) σ r i ( M ( E D X , 1 [ ˜ f ∗ M ˜ f ⊤ ]) ( i ) ) = σ r i ( M ( E D Y , 1 [˜ g ∗ M ˜ g ⊤ ]) ( i ) ) = σ r i ( ˘ ˆ f ( i ) , ˘ ˆ g ( i ) ) for all i ∈ [ K ] .

The key property of the balanced t-embeddings is that the r i -th singular values of the t-covariance tensors E D X , 1 [ ˜ f ∗ M ˜ f ⊤ ] and E D Y , 1 [˜ g ∗ M ˜ g ⊤ ] in each frequency component i ∈ [ K ] are equal to the r i -th singular value of the original embeddings ( ˘ ˆ f ( i ) , ˘ ˆ g ( i ) ) . Intuitively, the balancing operator helps to align the spectral properties of the left and right embeddings, making them more compatible and easier to analyze. By balancing the t-embeddings, we can focus on their essential characteristics and ignore any irrelevant or redundant information that may complicate the analysis.

The balancing t-operator plays a crucial role in the definition of aligned k -proxies.

Definition 16 (Aligned proxies for t-embeddings) . Given a fixed vector of integers r = ( r 1 , · · · , r K ) ⊤ ∈ N K . We say ι r : R ∥ r ∥ ∞ × 1 × K →M is a t-isometric inclusion if it preserves t-inner products, i.e., v ⊤ ∗ M w = ⟨ ι r ( v ) , ι r ( w ) ⟩ M . Let ˆ f : X → R ∥ r ∥ ∞ × 1 × K and ˆ g : Y → R ∥ r ∥ ∞ × 1 × K be full-multi-rankr . We say ( f , g ) are aligned k -proxies for ( ˆ f , ˆ g ) if: (a) f = ( ι r ◦ T -1 ) ˆ f , g = ( ι r ◦ T )ˆ g , where ι r : R ∥ r ∥ ∞ × 1 × K → M is a t-isometric inclusion, and T is the balancing t-operator in Definition 15, and (b) for 14 P ⋆ k being the multi-rankk t-projection operator defined by Σ ⋆ 1 ⊗ 1 , we have

The concepts of t-isometric inclusion and aligned k -proxies are essential for establishing the generalization bounds for the ERM-DS algorithm. These concepts allow us to relate the learned embeddings to the true embeddings and control the approximation error.

Definition 17 ( α -Conditioned embeddings) . Given vectors r = ( r i ) K i =1 ∈ N K and α = ( α i ) K i =1 ∈ R K where α i ≥ 1 for all i ∈ [ K ] , we say R ∥ r ∥ ∞ × 1 × K -embeddings ( ˆ f , ˆ g ) are α -conditioned with multi-rank r if

where σ r i ( ˘ ˆ f ( i ) , ˘ ˆ g ( i ) ) denotes the r i -th singular value of the learned embeddings ( ˆ f , ˆ g ) in the i -th frequency component, and σ ⋆ r i ( ˘ f ⋆, ( i ) , ˘ g ⋆, ( i ) ) is the corresponding singular value of the true embeddings ( f ⋆ , g ⋆ ) .

The α -conditioned property ensures that the learned embeddings ( ˆ f , ˆ g ) have singular values that are not too small compared to those of the true embeddings ( f ⋆ , g ⋆ ) , up to a constant factor α i in each frequency component i . This condition guarantees that the learned embeddings capture the essential spectral properties of the true embeddings and have sufficient expressive power to approximate the true relationship between the input features and the output variables. The constants α i ≥ 1 provide flexibility in the condition, allowing for some discrepancy between the singular values of the learned and true embeddings. Smaller values of α i indicate a tighter condition, requiring the learned embeddings to be more closely aligned with the true embeddings in terms of their spectral properties. The α -conditioned property is important in the analysis of multi-output regression algorithms under CDS because it helps control the approximation error in the generalization bounds. By ensuring that the learned embeddings have sufficient expressive power, this condition enables the algorithms to effectively capture the intrinsic structure of the data and achieve good generalization performance.

Definition 18 ( ( ϵ trn , ϵ D 1 ⊗ 1 ) -accurate embeddings) . Given vectors r = ( r i ) K i =1 ∈ N K , ϵ trn = (˘ ϵ ( i ) trn ) K i =1 ∈ R K , ϵ D 1 ⊗ 1 = (˘ ϵ ( i ) D 1 ⊗ 1 ) K i =1 ∈ R K where ˘ ϵ ( i ) trn , ˘ ϵ ( i ) D 1 ⊗ 1 ≥ 0 for all i ∈ [ K ] , we say R ∥ r ∥ ∞ × 1 × K -embeddings ( ˆ f , ˆ g ) are ( ϵ trn , ϵ D 1 ⊗ 1 ) -accurate with tensor multi-rank r if

where ˘ R ( i ) [ s i ] ( ˘ ˆ f ( i ) , ˘ ˆ g ( i ) ; D 1 ⊗ 1 ) is the excess risk relative to ˘ h ⋆, ( i ) s i = ⟨ ˘ f ⋆, ( i ) s i , ˘ g ⋆, ( i ) s i ⟩ , evaluated on the top-block distribution D 1 ⊗ 1 :

The ( ϵ trn , ϵ D 1 ⊗ 1 ) -accurate property ensures that the learned embeddings ( ˆ f , ˆ g ) closely approximate the true embeddings ( f ⋆ , g ⋆ ) in terms of the excess risk on both the training distribution D train and the topblock distribution D 1 ⊗ 1 . The first condition, ˘ R ( i ) ( ˘ ˆ f ( i ) , ˘ ˆ g ( i ) ; D train ) ≤ (˘ ϵ ( i ) trn ) 2 , requires that the excess risk of the learned embeddings on the training distribution is bounded by (˘ ϵ ( i ) trn ) 2 in each frequency component i . This condition ensures that the learned embeddings fit the training data well and capture the underlying patterns in the data. The second condition, inf r ′ ≥ r i ˘ R ( i ) [ r ′ ] ( ˘ ˆ f ( i ) , ˘ ˆ g ( i ) ; D 1 ⊗ 1 ) ≤ (˘ ϵ ( i ) D 1 ⊗ 1 ) 2 , requires that the excess risk of the learned embeddings on the top-block distribution, relative to the best rankr ′ approximation of the true embeddings for any r ′ ≥ r i , is bounded by (˘ ϵ ( i ) D 1 ⊗ 1 ) 2 in each frequency component i . This condition ensures that the learned embeddings generalize well to the top-block distribution and can effectively capture the important spectral components of the true embeddings. The constants ˘ ϵ ( i ) trn and ˘ ϵ ( i ) D 1 ⊗ 1 quantify the degree of accuracy, with smaller values indicating a better approximation. The ( ϵ trn , ϵ D 1 ⊗ 1 ) -accurate property is important in the analysis of multi-output regression algorithms under CDS because it helps control the estimation error in the generalization bounds. By ensuring that the learned embeddings closely approximate the true embeddings on the relevant distributions, this condition enables the algorithms to achieve good generalization performance and mitigate the impact of distribution shift.

D.1 Error Decomposition

In this section, we present a technical method that breaks down the excess risk of the algorithms into distinct, interpretable components, such as the approximation error, statistical variance, and terms capturing the effects of distribution shift. The main idea in this section builds on Ref. [46]. For completeness, we provide a detailed overview of the error decomposition framework used to analyze the generalization performance of multi-output regression algorithms under CDS. Our objective is to establish an upper bound on the population risk R ( ˆ f , ˆ g ; D test ) for the test distribution in terms of the training distribution risk and additional error terms that account for distribution shift.

To achieve this, we introduce key error terms that capture different aspects of the shift between training and test data, defining these terms to address the challenges of CDS in multi-output regression. We then present a series of lemmas that bound the test risk in terms of these error terms, which collectively lead to the final error decomposition. This culminates in Lemma D.2, summarizing the decomposition and providing insights into the algorithm's generalization behavior under CDS.

D.1.1 Key Error Terms

Definition 19. Given functions f : X → M and g : Y → M and k = ( k 1 , · · · , k K ) ⊤ ∈ N K , we define several key error terms as follows:

When it is clear from the context, we will use the shorthand notation ∆ 0 , ∆ 1 , ∆ 2 , ∆ apx , and ∆ train , respectively, for convenience.

These error terms are essential for assessing the performance of embeddings f and g across different distributions:

Weighted Error ∆ 0 ( f , g , k ) : Measures discrepancies between true embeddings ( f ⋆ k , g ⋆ k ) and learned embeddings ( f , g ) under D 1 ⊗ 1 , focusing on the weighted differences along each direction.

Unweighted Error ∆ 1 ( f , g , k ) : Captures unweighted differences between true and learned embeddings under D 1 , assessing how well ( f , g ) approximate ( f ⋆ k , g ⋆ k ) without specific weighting.

D 2 ⊗ 2 -Recovery Error ∆ 2 ( f , g , k ) : Represents discrepancies under D 2 ⊗ 2 , evaluating differences between true and learned embeddings in this setting.

Approximation Error ∆ apx ( k ) : Measures how well the true embeddings ( f ⋆ k , g ⋆ k ) capture the actual relationships under D 1 ⊗ 1 .

Training Error ∆ train ( k ) : Indicates the risk of true embeddings ( f ⋆ k , g ⋆ k ) on the training distribution D train , reflecting their alignment with the training data.

D.1.2 Error Decomposition Lemma

Lemma D.2 (Error decomposition on D test ) . Suppose Assumption 2 and Assumption 3 hold. For any vector k = ( k 1 , · · · , k K ) ⊤ satisfying k ≤ r element-wisely with some fixed vector of integers r = ( r 1 , · · · , r K ) ⊤ ∈ N K , and any aligned k -proxies ( f , g ) of the R ∥ r ∥ ∞ × 1 × K -embeddings ( ˆ f , ˆ g ) , denote ∆ 0 = ∆ 0 ( f , g , k ) and ∆ 1 = ∆ 1 ( f , g , k ) . Let σ ≤ min i ∈ [ K ] σ r i ( ˘ ˆ f ( i ) , ˘ ˆ g ( i ) ) be a lower bound on σ r i ( ˘ ˆ f ( i ) , ˘ ˆ g ( i ) ) , which satisfies σ 2 i ∈ (0 , tail ( i ) ⋆ 2 ( k i ) + ∆ 0 +∆ train ] for all i ∈ [ K ] . Then,

The proof closely follows that of Proposition 4.1b in Ref. [46] and proceeds by incrementally bounding the risk on the test distribution D test through a sequence of lemmas. In these lemmas, rather than applying Assumption 2 and Assumption 3 directly, we employ their relaxed forms as given in Assumption 6 and Assumption 7. This adjustment allows us to handle cases where coverage and covariate shift conditions hold only in a probabilistic or approximate sense, thus broadening the applicability of the result.

First, we apply Lemma D.3 to decompose the risk on D test into the risk on the bottom-right block distribution D 2 ⊗ 2 and the risk on the training distribution D train .

Lemma D.3 (Error decomposition on D test ) . Under Assumption 6 and Assumption 8, the following holds for any f : X → M and g : Y → M :

Next, we focus on bounding the risk on D 2 ⊗ 2 using Lemma D.4. This lemma reveals that the dominant term in the risk decomposition is the weighted error ∆ 0 , while the unweighted errors ∆ 1 and ∆ 2 contribute only quadratically:

Lemma D.4 (Error decomposition on D 2 ⊗ 2 ) . Under Assumption 6, Assumption 7 and Assumption 8, for any f : X → M , g : Y → M , and k ∈ N K :

where above we suppress error term dependence on f , g , k .

To bound the term ∆ 2 , we leverage the assumption that ( f , g ) are aligned k -proxies of ( ˆ f , ˆ g ) , along with the construction in Definition 16. This allows us to apply Lemma D.5 and obtain a bound on ∆ 2 in terms of the training error ∆ train , the weighted error ∆ 0 , and the approximation error ∆ apx ( k ) :

Going forward, recall the construction in Definition 16 that

where the positivity is a consequence of the assumption that ( ˆ f , ˆ g ) are full-multi-rank. We may now bound ∆ 2 .

Lemma D.5 (Decomposition of ∆ 2 ) . Suppose ( f , g ) are aligned k -proxies of ( ˆ f , ˆ g ) . Then, we have

where above we suppress dependence on f , g , k in all error terms.

Lastly, it is noteworthy that the multi-rankk approximation error under the D 1 ⊗ 1 distribution is precisely the tail term tail ⋆ 2 ( k ) . We can exploit this crucial fact by replacing the ∆ apx ( k ) term in the previous error decompositions with tail ⋆ 2 ( k ) .

Lemma D.6. We have ∆ apx ( k ) = R ( f ⋆ k , g ⋆ k ; D 1 ⊗ 1 ) = tail ⋆ 2 ( k ) .

Proof of Lemma D.2. First, we apply Lemma D.3 to decompose the risk on D test into the risks on D 2 ⊗ 2 and D train . We then further decompose the risk on D 2 ⊗ 2 into various error terms using Lemma D.4. Next, we bound ∆ 2 by leveraging Lemma D.5 and the assumption that ( f , g ) is an aligned k -agent of ( ˆ f , ˆ g ) . Substituting the bound on ∆ 2 back into the risk decomposition on D 2 ⊗ 2 , and applying Lemma D.6 to replace ∆ apx ( k ) with tail ⋆ 2 ( k ) . Finally, we combine the error terms using the condition on σ to obtain the final result.

D.1.3 Proof of Lemma D.3

Lemma D.7. Under Assumption 6, we have for any ( i, j ) ∈ { (1 , 1) , (1 , 2) , (2 , 1) }

Proof of Lemma D.7. To bound the risk under D i ⊗ j in terms of the training distribution, we introduce a density-based event and apply it to control expectations.

Define the event E train ,i ⊗ j for any ( i, j ) ∈ { (1 , 1) , (1 , 2) , (2 , 1) } as follows:

This event limits the density ratio between D i ⊗ j and D train , ensuring that D i ⊗ j does not deviate excessively from the training distribution.

Now, for any function h : X × Y → R 1 × 1 × K with ∥ h ( x, y ) ∥ ≤ M , we proceed to bound the expectation under D i ⊗ j :

The first term represents cases where E train ,i ⊗ j does not hold, bounded by M 2 P D i ⊗ j [ ¬E train ,i ⊗ j ] . For the second term, we leverage E train ,i ⊗ j to express the expectation under D train :

Since E train ,i ⊗ j holds in this term, the density ratio is bounded by κ trn, giving:

Substituting P D i ⊗ j [ ¬E train ,i ⊗ j ] ≤ η trn yields:

Now, let h ( x, y ) = ⟨ f ( x ) , g ( y ) ⟩ M -h ⋆ ( x, y ) . Note that

Then, ∥ h ( x, y ) ∥ 2 is bounded by 4 B 4 , giving:

which completes the proof.

Proof of Lemma D.3. To bound the excess risk under Assuption 6, we decompose the expected norm of a function under D test by defining a density-based event and applying it to control expectations.

First, we define the density-based event E test to control the ratio between the density of D test and a mixture of reference distributions D i ⊗ j :

This event ensures that D test does not deviate excessively from the reference distributions, allowing us to manage expectations over D test using properties of the distributions D i ⊗ j .

Next, we bound the expected norm of any function h ( x, y ) : X × Y → R 1 × 1 × K with ∥ h ( x, y ) ∥ 2 ≤ M 2 under D test . We decompose this expectation as follows:

The first term accounts for cases where E test does not hold, bounded by M 2 P D test [ ¬E test ] . The second term, where E test holds, allows us to use the density ratio to write:

Since E test holds in this term, the density ratio is bounded by κ tst, leading to:

We rewrite the integral using the expectations over D i ⊗ j , yielding:

Finally, noting that P D test [ ¬E test ] ≤ η tst, we have:

Now, let h ( x, y ) = ⟨ f ( x ) , g ( y ) ⟩ M -h ⋆ ( x, y ) , where ∥ h ( x, y ) ∥ 2 is bounded by 4 B 4 by Eq. (25). This gives:

By applying Lemma D.7, we further bound the sum over D i ⊗ j for all pairs except (2 , 2) :

̸

Therefore, combining these results, we conclude:

D.1.4 Proof of Lemma D.4

Lemma D.4 establishes bounds on the risk under D 2 ⊗ 2 , highlighting that the primary contribution in the risk decomposition comes from the weighted error term ∆ 0 . In contrast, the unweighted errors ∆ 1 and ∆ 2 have only a quadratic impact. Before proceeding with the proof of Lemma D.4, we first present a lemma that assists in expanding the risk term R ( f , g ; D 2 ⊗ 2 ) by breaking it down into more manageable components.

Lemma D.8. For any function h ⋆ : X × Y → R 1 × 1 × K , and for f 1 , f 2 ∈ X → M and g 1 , g 2 ∈ Y → M , we have

Proof of Lemma D.8. To simplify notation, let h 1 = ⟨ f 1 , g 1 ⟩ M and h 2 = ⟨ f 2 , g 2 ⟩ M . Our goal is to bound the expression ∥ h 1 -h ⋆ ∥ 2 using h 2 and differences between f 1 , f 2 and g 1 , g 2 .

We start by expanding ∥ h 1 -h ⋆ ∥ 2 -∥ h 2 -h ⋆ ∥ 2 as follows:

This identity allows us to split the difference in terms of the inner products of h 1 -h 2 and each of the terms h 1 -h ⋆ and h 2 -h ⋆ .

Next, applying the Cauchy-Schwarz inequality, we get:

Expanding and rearranging terms, we conclude that:

To bound ∥ h 1 -h 2 ∥ 2 , we expand this term as:

Using the Cauchy-Schwarz inequality on each term separately, we obtain:

Finally, applying the Cauchy-Schwarz inequality again to the last term, we have:

Combining this with our earlier bound on ∥ h 1 -h ⋆ ∥ 2 , we arrive at the final result:

This completes the proof.

Proof of Lemma D.4. The proof approach follows the structure of Lemma L2.b in Ref. [46]. The primary distinction is that our proof deals with t-embeddings within the Hilbert t-Module M for multi-output regression, while Lemma L2.b in Ref. [46] addresses single-output learning.

Step 1: Tackling covariate shift under D 2 ⊗ 2 . To analyze the impact of covariate shift under D 2 ⊗ 2 , let's set f 1 = f , g 1 = g , f 2 = f ⋆ k , and g 2 = g ⋆ k . With this setup, we can apply Lemma D.8 to bound the difference in risk as follows:

where each step is justified as follows:

In ( i ) , we apply Lemma D.9 to bound E D 2 ⊗ 2 ∥⟨ f -f ⋆ k , g ⋆ k ⟩ M ∥ 2 and E D 2 ⊗ 2 ∥⟨ f ⋆ k , g -g ⋆ k ⟩ M ∥ 2 . This is possible since ∥ g -g ⋆ k ∥ M ≤ ∥ g ∥ M + ∥ g ⋆ k ∥ M ≤ 2 B and similarly, ∥ f -f ⋆ k ∥ M ≤ ∥ f ∥ M + ∥ f ⋆ k ∥ M ≤ 2 B .

In ( ii ) , we utilize the fact that D 2 ⊗ 2 = D X , 2 ⊗ D Y , 2 is a product measure, allowing us to separate the expectations over D X , 2 and D Y , 2 .

In ( iii ) , we introduce the term (∆ 2 ) 2 = (∆ 2 ( f, g, k )) 2 to further simplify the expression.

Thus, Equation (26) provides the bound on the risk under D 2 ⊗ 2 , accounting for covariate shifts in the multi-output setting.

Step 2: Expanding Terms under D 1 ⊗ 2 and D 2 ⊗ 1 . In this step, we expand the first two terms from Equation (26) to further analyze their contributions.

Starting with ⟨ f -f ⋆ k , g ⋆ k ⟩ M , observe that:

With this, we can bound:

where in ( i ) we use Lemma D.9 to bound terms involving covariate shifts.

Next, we expand and bound the product of norms term as follows:

where we have used the definitions of ∆ and ∆ from Definition 19.

2 1 Summing up these results, we obtain:

Using a similar argument, we can bound the other term as:

Now, define:

Thus, we can combine these results as:

Step 3: Consolidating Results. Combining Equation (27) and Equation (26), we obtain:

Rearranging terms, we get:

Step 4: Final Bound. To finalize, we need to bound Equation (28). By Lemma D.10, we have:

Similarly, for ∆ off , by Lemma D.10:

Thus,

Using Lemma D.7:

we get:

Summing everything up, we obtain:

Dropping constants, we conclude:

D.1.5 Key Change-of-Measure Lemmas

We begin by establishing some important change-of-measure results.

Lemma D.9 (Change of Measure: Factor Estimation Error) . Under Assumption 7 and Assumption 8, the following holds for any i, j ∈ { 1 , 2 } and any ( ˜ f , ˜ g ) ,

The same holds if g ⋆ k (resp. f ⋆ k ) are replaced by g ⋆ > k := g ⋆ -g ⋆ k (resp. f ⋆ > k := f ⋆ -f ⋆ k ) or g ⋆ (resp. f ⋆ ).

Proof of Lemma D.9. We start by proving the first part under Assumption 7; the extension to the second part follows a similar approach. Specifically, we have:

The second part of the lemma follows similarly. To derive analogous bounds for g ⋆ -g ⋆ k , we note that g ⋆ -g ⋆ k = ( I P ⋆ k ) g ⋆ , where I P ⋆ k is also a projection operator. A similar argument applies for bounding f ⋆ -f ⋆ k .

Finally, the bounds for f ⋆ and g ⋆ are simpler to establish, as they do not require commuting the projection operator.

Lemma D.10 (Change of Measure: Approximation Error) . The following bounds hold:

The risks on the product distributions D 1 ⊗ 2 and D 2 ⊗ 1 are bounded by

The risk on the product distribution D 2 ⊗ 2 is bounded by

Proof of Lemma D.10. Introduce the shorthand f ⋆ > k := f ⋆ -f ⋆ k and g ⋆ > k := g ⋆ -g ⋆ k . Note that

which implies that both f ⋆ > k and g ⋆ > k are B -bounded by Assumption 8.

Since P ⋆ k is an orthogonal projection, IP ⋆ k is also an orthogonal projection, meaning it is self-adjoint and idempotent. Therefore, we can write:

Thus, by applying Lemma D.9 and noting that f ⋆ > k is B -bounded, we have:

Similarly, we can bound R ( f ⋆ k , g ⋆ k ; D 2 ⊗ 1 ) as:

Finally, applying Lemma D.9 twice, we find:

This completes the proof.

D.1.6 Proof of Lemma D.5

Proof of Lemma D.5. The lemma states that if ( f , g ) are aligned k -proxies of ( ˆ f , ˆ g ) , then the following inequality holds:

Lemma D.5 expresses that under the aligned proxies condition, the deviation ∆ 2 between ( f , g ) and their rankk approximations ( f ⋆ k , g ⋆ k ) under distributions D X , 2 , D Y , 2 can be controlled by the error ∆ train on the training distribution, the weighted embedding estimation error ∆ 0 , and k -approximation error ∆ apx ( k ) , as long as the estimated embeddings ( ˆ f , ˆ g ) have positive minimum singular values on the corresponding modes. This lemma quantifies the effect of the aligned proxies condition.

The main idea of the proof is to consider E D X , 2 ∥ f ⋆ k -f ∥ 2 M and E D Y , 2 ∥ g ⋆ k -g ∥ 2 M separately and relate them to ∆ 0 , ∆ apx ( k ) , ∆ train using the properties of aligned proxies. Since the arguments for the two parts are symmetric, we focus on how to handle E D Y , 2 ∥ g ⋆ k -g ∥ 2 M .

Our aim is to bound ∆ 2 . We focus on bounding E D Y , 2 ∥ g ⋆ k -g ∥ 2 M , for the bound on E D X , 2 ∥ f ⋆ k -f ∥ 2 M is analogous. Further, let us recall what it means for ( f , g ) to be aligned k -proxies. This means that (a) f = ( ι r ◦ T -1 ) ˆ f , g = ( ι r ◦ T )ˆ g , where ι r : R ∥ r ∥ ∞ × 1 × K →M is an isometric inclusion, and T is the balancing operator for tensors, and (b) for P ⋆ k being the multi-rankk projection defined by Σ ⋆ 1 ⊗ 1 , we have ∀ i ∈ [ K ] :

Let V := range( E D X , 1 [ ˘ f ( i ) ( ˘ f ( i ) ) ⊤ ]) . Since ˆ f , ˆ g are full-multi-rankr , V = range( M ( ι r ) ( i ) ) = range( E D Y , 1 [˘ g ( i ) (˘ g ( i ) ) ⊤ ]) . Moreover, range( E D Y , 1 [˘ g ⋆, ( i ) k i (˘ g ⋆, ( i ) k i ) ⊤ ]) = range( M ( P ⋆ k ) ( i ) ) ⊆ V r i . Hence, by Lemma D.11 , ˘ g ( i ) ( y ) , ˘ g ⋆, ( i ) k i ( y ) ∈ V almost surely, and thus, ˘ g ⋆, ( i ) k i ( y ) -˘ g ( i ) ( y ) ∈ V with probability one. In addition, since V has dimension r i , it follows that for any v ∈ V , and since σ r i ( E D X , 1 [ ˘ f ( i ) ( ˘ f ( i ) ) ⊤ ]) = σ r i ( ˘ ˆ f ( i ) , ˘ ˆ g ( i ) ) > 0 ,

Therefore

Therefore,

-domain)

In other words, we bound E D Y , 2 ∥ g ⋆ k -g ∥ 2 M by relating an expectation involving D X , 1 . Now, we can further expand

Hence,

By Lemma D.9 and the fact that ∥ f -f ⋆ k ∥ M is 2 B -bounded,

By Lemma D.10, R ( f ⋆ k , g ⋆ k ; D 1 ⊗ 2 ) ≤ κ cov R ( f ⋆ k , g ⋆ k ; D 1 ⊗ 1 ) + η cov B 4 = κ cov ∆ apx + η cov B 4 . Finally, by applying Lemma D.7,

Thus, min i { σ r i ( ˘ ˆ f ( i ) , ˘ ˆ g ( i ) ) }· E D Y , 2 ∥ g ⋆ k -g ∥ 2 M ≤ 3 κ cov (∆ 0 +∆ apx )+3 κ trn ∆ train +12 B 4 η cov + η tst . Considering the symmetric argument for E D X , 2 ∥ f ⋆ k -f ∥ 2 M and taking the maximum of the two bounds, we arrive at the conclusion of the lemma:

This completes the proof.

Lemma D.11 (Lemma L.7 in Ref. [46]) . Let D X be a distribution over X , let Σ = E D X [ ff ⊤ ] , and let P be the orthogonal projection on range( Σ ) . Then P f = f D X -almost surely; that is, P D X [ f ( x ) ∈ range( Σ )] = 1 .

D.1.7 Proof of LemmeD.6

Proof of Lemma D.6. Since the t-projection operators are self-adjoint and idempotent, we begin by expressing the risk term as follows:

Rewriting this in terms of projection operators, we have

Next, switching to the M -domain by transforming into individual components, we find

Since expectation and summation are interchangeable, we can separate the components as

Using the trace operator to express this in matrix form, we get

Since E D X , 1 [ ˘ f ⋆, ( i ) ( x ) ˘ f ⋆, ( i ) ( x ) ⊤ ] = M ( Σ ⋆ 1 ⊗ 1 ) ( i ) and E D Y , 1 [ ˘ g ⋆, ( i ) ( y )˘ g ⋆, ( i ) ( y ) ⊤ ] = M ( Σ ⋆ 1 ⊗ 1 ) ( i ) , we obtain

Finally, by summing over the eigenvalues of the transformed covariance matrices, we find

This completes the proof.

D.2 Excess Risk Bounds for the ERM Model

In this section, we analyze the ERM algorithm for multi-output regression under CDS. Our primary objective is to establish a rigorous upper bound on the excess risk R ( ˆ f erm , ˆ g erm ; D test ) of the ERM solution ( ˆ f erm , ˆ g erm ) obtained by Model (5).

Recall that Theorem 4 states the following: Given vectors r = ( r i ) K i =1 ∈ N K , α = ( α i ) K i =1 ∈ R K , ϵ trn = (˘ ϵ ( i ) trn ) K i =1 ∈ R K , ϵ D 1 ⊗ 1 = (˘ ϵ ( i ) D 1 ⊗ 1 ) K i =1 ∈ R K where α i ≥ 1 , and ˘ ϵ ( i ) trn , ˘ ϵ ( i ) D 1 ⊗ 1 ≥ 0 for all i ∈ [ K ] , suppose the learned embeddings ( ˆ f erm , ˆ g erm ) are α -conditioned and ( ϵ trn , ϵ D 1 ⊗ 1 ) -accurate, satisfying ˘ ϵ ( i ) D 1 ⊗ 1 ≤ ˘ σ ⋆, ( i ) 1 / (40 r ) for all i ∈ [ K ] . Then, under Assumptions 1 to 5 and 8, the excess risk of the ERM solution on D test can be bounded up to a constant factor c = poly( κ cov , κ tst , κ trn ) with probability at least 1 -δ :

where α := max i { α i } , σ := min i { ˘ σ ⋆, ( i ) r } > 0 , ∆ n = B 4 ( N ( r, 2 B/n ) + log 2 δ ) /n with N ( r, ϵ ) = N ( F r , ϵ/ (2 B ) , ∥ · ∥ ∞ ) · N ( G r , ϵ/ (2 B ) , ∥ · ∥ ∞ ) .

To prove Theorem 4, we begin by applying Lemma D.2 with the multi-rank parameter r = ( r, . . . , r ) ⊤ ∈ R K , where r is the tubal-rank parameter of the embeddings in the ERM formulation (5). This lemma provides a general decomposition of the excess risk into several key error terms, each capturing a different aspect of the learning problem:

This decomposition lays the groundwork for our subsequent analysis, as it allows us to focus on bounding each error term separately.

Next, we introduce a set of conditions on the ERM solution to ensure its quality and enable more refined bounds. Specifically, we assume that the learned embeddings ( ˆ f erm , ˆ g erm ) are α -conditioned and ( ϵ trn , ϵ D 1 ⊗ 1 ) -accurate, as formalized in Definition 17 and Definition 18, respectively.

Moreover, we impose a constraint on the rank r of the embeddings, requiring that r ≤ ˘ σ ⋆, ( i ) 1 / (40˘ ϵ ( i ) D 1 ⊗ 1 ) for all i ∈ [ K ] . Under these assumptions, we can leverage the powerful Lemma

D.22 to obtain a more refined bound on the excess risk:

where err trn := R ( ˆ f erm , ˆ g erm ; D train ) and err 1 ⊗ 1 := R ( ˆ f erm , ˆ g erm ; D 1 ⊗ 1 ) denote the accuracy on D train and D 1 ⊗ 1 , respectively. This bound reveals the intricate dependencies of the excess risk on various problem parameters, such as the accuracy of the learned embeddings ( err 1 ⊗ 1 and err trn ), the conditioning of the true embeddings ( α i ), and the spectral properties of the true embeddings ( σ ⋆ r +1 , tail ⋆ 1 ( r ) , and tail ⋆ 2 ( r ) ). However, to make this bound more explicit, we still need to control the error terms err trn and err 1 ⊗ 1 .

To this end, we use Lemma D.12 below to establish high-probability bounds on the excess risks R ( ˆ f erm , ˆ g erm ; D train ) and R ( ˆ f erm , ˆ g erm ; D 1 ⊗ 1 ) of the empirical risk minimizers ( ˆ f erm , ˆ g erm ) . These bounds are expressed in terms of the true risk R ( f ⋆ r , g ⋆ r ; D train ) , the tail sum tail ⋆ 2 ( r ) , and the covering number N ( r, 2 B/n ) , which quantifies the complexity of the hypothesis class. By carefully balancing these terms, Lemma D.12 provides the key ingredients needed to control err trn and err 1 ⊗ 1 .

Lemma D.12. Under Assumptions 1 to 5 and 8, let ( ˜ f , ˜ g ) ∈ F p ×G p be empirical risk minimizers on n i.i.d. samples ( x i , y i , z i ) ∼ D train . Then, for any δ ∈ (0 , 1) , the followings hold with probability at least 1 -δ :

Proof of Lemma D.12. The lemma is proved as follows:

The first inequality is a direct consequence of Lemma D.19. Here, we take the function class Φ = F r ×G r . Moreover, by Assumption 8,

Then, we can choose U = 2 B 2 and ϵ = 2 B 2 /n in Lemma D.19. Note that, according to Lemma D.20, it holds that

The second inequality uses Assumption 8 to bound R ( f ⋆ p , g ⋆ p ; D train ) ≤ κ apx R ( f ⋆ p , g ⋆ p ; D 1 ⊗ 1 ) , and noting the fact that R ( f ⋆ p , g ⋆ p ; D 1 ⊗ 1 ) = tail ⋆ 2 ( p ) by Lemma D.6.

The third inequality uses Assumption 2, incurring an addition factor of κ trn .

Now, we can proceed to derive a more explicit bound on the excess risk.

Proof of Theorem 4. By combining the results of Lemmas D.2, D.22, and D.12, and introducing some convenient shorthand notations ( σ = min i ˘ σ ⋆, ( i ) r > 0 , α = max i α i , and ∆ n = B 4 ( N ( r, 2 B/n )+log 2 δ ) n ), we arrive at the following bound:

Limitations of the ERM Model. The ERM Model (5) faces several challenges in multi-output regression under CDS:

Inflexibility in adapting to spectral decay patterns: The hypothesis classes F r and G r in singlestage ERM do not account for varying spectral decay patterns across different frequency components induced by the transform M ( · ) . Consequently, the learned embeddings may fail to capture distinct decay characteristics within each frequency sub-domain, resulting in suboptimal approximations.

Lack of localized learning and complexity control: The ERM approach learns the embeddings ˆ f erm and ˆ g erm globally, disregarding the unique properties of each frequency sub-domain. This global learning strategy can miss the nuanced decay variations across sub-domains. Additionally, controlling the complexity of F r and G r through global covering numbers may be too coarse to effectively capture these variations.

Limited generalization performance: Due to these constraints, the single-stage ERM approach may offer limited generalization guarantees for multi-output regression under CDS. The embeddings may struggle to generalize to new feature combinations, especially when spectral decay patterns differ significantly between frequency sub-domains.

D.3 Excess Risk Bounds for ERM-DS

The ERM-DS algorithm is introduced to address the limitations inherent in the ERM approach. This section provides a comprehensive analysis of the ERM-DS algorithm. The algorithm consists of four main stages: overparameterization, t-covariance estimation, dimension reduction, and distillation. We analyze each stage separately and then combine the results to obtain an overall generalization bound. We first give a detailed version for Theorem 5 as follows.

Theorem 6. For any r i, cut ≳ ⋆ poly( C/σ ⋆ 1 ) and ϵ, δ > 0 , there exists a choice of σ cut > 0 , p ≲ ⋆ (min i r i, cut ) c for some universal c > 0 , and sample sizes n 1:4 satisfy Condition 8, such that with ν i = r 4 i, cut and µ = B 2 /n 1 , the ERM-DS algorithm satisfies with probability at least 1 -δ :

Our proof follows the core idea and approach of Theorem 6 in Ref. [46]. The main steps of the proof are provided as follows.

Analysis of Step 1 (Overparameterization). We begin with a precise analysis of the first phase of the double-stage ERM. We first show that after the first step, we have the following upper bound on the excess risk on D train and D 1 ⊗ 1 .

Lemma D.13 (Guarantee for ERM-DS, Step 1) . Let ( ˜ f , ˜ g ) be the obtained t-embeddings obtained via Step 1 of ERM-DS on n 1 i.i.d. samples ( x 1 ,j , y 1 ,j , z 1 ,j ) ∼ D train . Then, for any δ ∈ (0 , 1) , the followings hold with probability at least 1 -δ :

This lemma establishes upper bounds on the excess risk of the initial embeddings ( ˜ f , ˜ g ) obtained in Step 1 of ERM-DS for both the training distribution D train and distribution D 1 ⊗ 1 . It refines Lemma D.12. The main difference from Lemma D.12 lies in the covering numbers ˘ N ( i ) ( p, 2 B/n 1 ) , which here relate specifically to the function classes ˘ F ( i ) p and ˘ G ( i ) p in Assumption 9, rather than to F and G in Assumption 8. The proof is similar and leverages techniques from statistical learning theory, along with the specific properties of the function classes ˘ F ( i ) p and ˘ G ( i ) p .

Proof of Lemma D.13. As R ( ˜ f , ˜ g ; D ) = ∑ K i =1 ˘ R ( i ) ( ˘ ˜ f ( i ) , ˘ ˜ g ( i ) ; D ) , we prove the result by upper bounding the excess risk in all the transformed sub-domains. For all i ∈ [ K ] , let ( ˘ ˜ f ( i ) , ˘ ˜ g ( i ) ) ∈ ˘ F ( i ) p × ˘ G ( i ) p be empirical risk minimizers on n 1 i.i.d. samples ( x j , y j , ˘ z ( i ) j ) ∼ D train . Then, we will prove that for any δ ∈ (0 , 1) , the followings hold with probability at least 1 -δ/K :

The first inequality is a direct consequence of Lemma D.19. Here, we take the function class Φ = ˘ F ( i ) p × ˘ G ( i ) p . Moreover, by Assumption 9,

Then, we can choose U = 2 B 2 and ϵ = 2 B 2 /n 1 in Lemma D.19. Note that, according to Lemma D.21, it holds that

The second inequality uses Assumption 9 to bound ˘ R ( i ) ( ˘ f ⋆, ( i ) p , ˘ g ⋆, ( i ) p ; D train ) ≤ κ apx ˘ R ( i ) ( ˘ f ⋆, ( i ) p , ˘ g ⋆, ( i ) p ; D 1 ⊗ 1 ) , and noting the fact that ˘ R ( i ) ( ˘ f ⋆, ( i ) p , ˘ g ⋆, ( i ) p ; D 1 ⊗ 1 ) = tail ( i ) ⋆ 2 ( p ) .

The third inequality uses Assumption 2, incurring an addition factor of κ trn .

Analysis of Step 2 (t-Covariance Estimation) and Step 3 (Dimension Reduction). Next, we analyze the effect of the t-covariance estimation and dimension reduction steps in ERM-DS. Let ( ˜ f , ˜ g ) be the empirical risk minimizers from Step 1, and let ˆ Q ˆ r be the balancing projection onto the t-eigenvectors of M ( ˆ Σ ˜ g ) , where ˆ r = (ˆ r 1 , . . . , ˆ r K ) ⊤ ∈ N K . We define the following effective error term in each transformed sub-domain:

To select the rank ˆ r for dimension reduction, we use parameters σ cut = ( σ i, cut ) K i =1 ∈ R K and r cut = ( r i, cut ) K i =1 ∈ N K . Motivated by Definition F.1 in Ref. [46], we consider the following 'good' event E spec (ˆ r , σ cut , r cut ) , under which the choices of σ cut and r cut are sufficient for finding suitable ranks ˆ r i for all the i -th frequency components ( ∀ i ∈ [ K ] ).

Definition 20 (Good spectral event in all the transformed sub-domains, adapted from Definition F.1 in Ref. [46]) . For parameters σ cut = ( σ i, cut ) K i =1 ∈ R K and r cut = ( r i, cut ) K i =1 ∈ N K used in Step 3 of ERM-DS, we define E spec (ˆ r , σ cut , r cut ) as the event that the following inequalities hold in each frequency component i ∈ [ K ] :

The following lemma shows that under appropriate conditions, the excess risk of the reduced-rank embeddings ( ˜ f , ˆ Q ˆ r ∗ M ˜ g ) can be bounded in terms of the effective error ˘ ˜ ϵ ( i ) ( p, n 1 , δ ) . The proof involves a delicate balance between the eigenvalues of the estimated and ground truth covariance operators, as well as careful control of the approximation and estimation errors.

Lemma D.14 (Guarantee for Double-Training, Step 2 & Step 3) . Suppose the parameters σ cut = ( σ i, cut ) K i =1 and r cut = ( r i, cut ) K i =1 satisfy σ i, cut ∈ [2 σ ⋆ i,r cut , 2 3 e σ ⋆ i, 1 ] , for all i ∈ [ K ] . Assume n 1 ≥ p ≥ 2 , n 1 ≥ max i { B 2 /σ 2 i, cut } , µ = B 2 /n 1 , n 2 ≥ 722 max i { r 2 i, cut } n 9 1 log(24 pK/δ ) , and ˘ ˜ ϵ ( i ) ( p, n 1 , δ ) 2 ≤ σ 2 i, cut / (64 r 2 i, cut ) for all i ∈ [ K ] . Then, with probability at least 1 -2 3 δ , we have the following upper bound on the excess risk of h red ( x, y ) to the multi-rankˆ r ground truth h ⋆ ˆ r := ( f ⋆ ˆ r ) ⊤ ∗ M g ⋆ ˆ r on D 1 ⊗ 1 :

Moreover, on this event, sup x,y |⟨ ˘ ˜ f ( i ) ( x ) , ˘ ˆ Q ( i ) ˆ r i ˘ ˜ g ( i ) ( y ) ⟩| ≤ √ 2 n 1 B 2 and E spec (ˆ r , σ cut , r cut ) holds.

The derivation of Lemma D.14 utilizes Proposition F.2 from Ref. [46] across all K subdomains in parallel. This approach extends the proposition's application to our specific multi-output regression context. As the adaptation follows directly, the detailed proof is omitted.

Analysis of Step 4: Distillation After Step 4 of ERM-DS, we obtain embedding ( ˆ f ds , ˆ g ds ) which satisfy the following lemma.

Lemma D.15. Suppose it holds that ∥ ˘ ˆ Q ( i ) ˆ r i ∥ op ≤ √ 2 n 1 , ∀ i ∈ [ K ] . Then, with probability at least 1 -δ/ (3 K ) , the regularized risk in the i -th frequency component satisfy

This lemma provides upper bounds on the training and testing risks of the final embeddings ( ˆ f ds , ˆ g ds ) in terms of the tail sum tail ( i ) ⋆ 2 ( r i ) , the excess risk ˘ R ( i ) [ r i ] ( ˘ ˜ f ( i ) , ˘ ˆ Q r i · ˘ ˜ g ( i ) ; D 1 ⊗ 1 ) , and the covering number N r i , n 4 . The proof of this lemma relies on a careful decomposition of the excess risk and the application of concentration inequalities. The proof of Lemma D.15 is provided in Appendix D.3.1.

We now provide a sufficient condition for parameter settings and sample sizes to improve the bound in Lemma D.15.

Lemma D.16. Suppose that the parameters in ERM-DS are chosen as µ = B 2 /n 1 , and other parameters ( p, σ cut , r cut ) , the sample sizes n 1 , . . . , n 4 , and parameter ν = (˘ ν ( i ) ) K i =1 ∈ R K satisfy that for some C ≲ ⋆ 1 , ∀ i ∈ [ K ]

σ i, cut ∈ [2˘ σ ⋆ r i, cut , 2 3 e ˘ σ ⋆ 1 ] , tail ⋆ 2 ( p ) ≤ σ 2 i, cut Cr 2 i, cut , and p ≥ 2 ;

n 1 ≥ p + C max { 1 , B 4 } max i { σ -2 i, cut r 2 i, cut ( ˘ N ( i ) ( p, 2 B/n 1 ) + log( K/δ )) } ,

n 2 ≥ 722 max i { r 2 i, cut } n 9 1 log(24 pK/δ ) ,

n 4 ≥ n 1 n 3 max { 2˘ ν ( i ) , 3 } .

Then, with probability at least 1 -δ , the event E spec (ˆ r , σ cut , r cut ) in Definition 20 holds and

Lemma D.16 follows from the application of Proposition F.1 in Ref. [46] to our context. The steps align closely with those in the referenced proposition, so the detailed proof is omitted here.

To ensure the successful application of the ERM-DS algorithm, we must carefully select the parameters ˘ ν ( i ) to control the error terms. However, before diving into this crucial step, let us first establish a refined set of sufficient conditions for the algorithm parameters and sample sizes, which will serve as the foundation for our analysis.

First, let us focus on the algorithm parameters, as outlined in Condition 7. We require that the parameters r i, cut , σ i, cut , and p satisfy a series of intricate relationships. These relationships, expressed in parts (a), (b), and (c) of the condition, ensure that the algorithm's components are well-balanced and can effectively capture the underlying structure of the data.

Condition 7 (Algorithm parameters) . Let c i, 1 be some unspecified parameter satisfying 1 ≤ c i, 1 ≲ ⋆ 1 . We stipulate that the algorithm parameters ( σ i, cut , r i, cut , p ) satisfy

Next, we turn our attention to the sample size requirements, as specified in Condition 8.

Condition 8 (Sample size conditions) . Let c i, 2 be some unspecified parameter satisfying c i, 2 ≲ ⋆ 1 . We stipulate that, given δ ∈ (0 , 1) ,

The supervised sample sizes of n 1 , n 3 satisfy

The unsupervised sample sizes n 2 , n 4 satisfy

With these conditions in place, we can now state the main result of our analysis, as presented in Theorem 9. This theorem provides a powerful guarantee for the performance of the ERM-DS algorithm, as measured by the risk function R ( ˆ f ds , ˆ g ds ; D test ) . The theorem asserts that, with high probability, this risk can be bounded by a sum of terms that depend on the parameters r i, cut , σ i, cut , and the tail sums of the singular values, tail ( i ) ⋆ 1 and tail ( i ) ⋆ 2 .

Theorem 9. Suppose the parameters of ERM-DS satisfy σ cut , r cut , p , sample sizes n 1 , . . . , n 4 , and ˘ ν ( i ) = r 4 i, cut , µ = B 2 /n 1 and fix a probability of error δ ∈ (0 , 1) . Then, as long σ cut , r cut , p satisfy Condition 7 and n 1:4 satisfy Condition 8, it holds with probability at least 1 -δ ,

The proof of this theorem relies on a careful analysis of the error terms and their relationships to the algorithm parameters and sample sizes. By leveraging the conditions we have established, along with advanced techniques from tensor algebra and empirical process theory, we can derive the stated bound on the risk function.

Proof of Theorem 9. Our goal is to bound the generalization error of the ERM-DS algorithm in each subdomain after transformation:

The key to the proof is to leverage the properties of each step in the algorithm and recursively bound the generalization error. While the overall proof strategy follows the ideas from [46], we extend these results to the multi-output setting using tensor algebra, which requires handling the additional complexity introduced by the tensor structure and the Hilbert t-Module framework.

Step 1: Based on Lemma D.22, if the learned embeddings ( ˘ ˆ f ( i ) ds , ˘ ˆ g ( i ) ds ) have risk (˘ ϵ ( i ) trn ) 2 on the training set D train and truncated risk (˘ ϵ ( i ) D 1 ⊗ 1 ) 2 on distributio D 1 ⊗ 1 , and satisfy ˘ ϵ ( i ) D 1 ⊗ 1 ≤ min { ˘ σ ⋆, ( i ) 1 / 40ˆ r i , ˘ σ ⋆, ( i ) ˆ r i / 4 } , then their generalization error on the test distribution D test can be

bounded as (with α i ≤ 2 ) for all i ∈ [ K ] :

Step 2: Recall that in the ERM-DS algorithm, we choose ˆ r i ≤ r i, cut and ˘ ν ( i ) = r 4 i, cut . In this case, as long as ˘ ϵ ( i ) D 1 ⊗ 1 ≤ min { ˘ σ ⋆, ( i ) 1 / 40 r i, cut , ˘ σ ⋆, ( i ) r i, cut / 4 } , we can obtain for all i ∈ [ K ] :

Step 3: Conditioned on the occurrence of the good spectral event E spec (ˆ r , σ cut , r cut ) (Definition 20), we can further constrain the generalization error using the spectral properties provided by this event:

Step 4: Substituting the specific forms of (˘ ϵ ( i ) trn ) 2 and (˘ ϵ ( i ) D 1 ⊗ 1 ) 2 from Lemma D.16, and leveraging the high-probability event described in the same lemma, we can deduce:

2

i,

2

σ

cut under Condition 8

This step involves balancing the algorithm parameters, sample sizes, and spectral decay rates across all sub-domains to ensure that the error terms are well-controlled.

Step 5: Substituting the above inequality into the bound from Step 3 yields the result stated in the theorem:

provided that the small tail assumption tail ( i ) ⋆ 2 ( p ) ≤ ( σ i, cut ) 2 r 5 i is satisfied and that the conditions on the algorithm parameters and sample sizes in Lemma D.16 hold, which are ensured by Conditions 7 and 8.

Lemma D.17. Suppose Assumption 4 holds, and that the algorithm parameters σ i, cut , r i, cut , p satisfy σ i, cut ≤ 2 3 e ˘ σ ⋆, ( i ) 1 , and the following (feasible) constraints

Then, Condition 7 holds and

The proof closely follows that of Lemma E.2 in Ref. [46] and is therefore omitted. This lemma establishes the relationship between the algorithm parameters ( σ i, cut , r i, cut , p ) and the error term ERR ( i ) DT ( r i, cut , σ i, cut ) , which plays a central role in bounding the excess risk of the algorithm.

Proof of Theorem 6. Our goal is to achieve the desired accuracy level ϵ . To this end, we let σ i, cut = max { 2 Cr -(1+ γ i ) i, cut , ϵ/ ( K 1 / 2 r i, cut ) } . We then invoke Equation (32), absorb absolute constants into the ≲ ⋆ notation, and arrive at the following bound:

We confirm that both Condition 7 and Condition 8 hold. Specifically, Condition 7 is satisfied as r i, cut is large enough (as a polynomial in C/ ˘ σ ⋆, ( i ) 1 ) and p remains within a constant power of r i, cut . Suitable sample sizes n 1 , n 2 , n 3 , n 4 ensure Condition 8, leading to the result by Theorem 9.

D.3.1 Lemmas From Statistical Learning Theory

Lemma D.18 (Bernstein inequality [10]) . Let Z 1 , . . . , Z n ∈ R be i.i.d. random variables with | Z i | ≤ M and Var[ Z i ] ≤ σ 2 . Then, with probability at least 1 -δ ,

Lemma D.19. Let Φ be a function class with functions ϕ : W → R k , and let ϕ ⋆ ( w ) be a target function defined on W . Assume there exists a constant U > 0 such that sup w ∈W sup ϕ ∈ Φ ∥ ϕ ( w ) -ϕ ⋆ ( w ) ∥ 2 ≤ U , and let N (Φ , ϵ, ∥ · ∥ ∞ ) be the ϵ -covering number of Φ . Let D be a joint distribution on W× R k such that ∥ z -ϕ ⋆ ( w ) ∥ 2 ≤ U and E [ z | w ] = ϕ ⋆ ( w ) . Define R ( ϕ ) := E w ∼ D [ ∥ ϕ ( w ) -ϕ ⋆ ( w ) ∥ 2 2 ] , ˆ L n ( ϕ ) := 1 n ∑ n i =1 ∥ ϕ ( w i ) -z i ∥ 2 2 , and ˆ R n ( ϕ ) := ˆ L n ( ϕ ) -ˆ L n ( ϕ ⋆ ) . Then for any δ ∈ (0 , 1) , with probability at least 1 -δ ,

any ϕ ∈ Φ satisfy:

all empirical risk minimizers ˆ ϕ ∈ arg min ϕ ∈ Φ ˆ L n ( ϕ ) satisfy:

Proof of Lemma D.19. For each ϕ ∈ Φ , first define the loss function ℓ ϕ : W× R k → R as

Also, define the excess loss function ∆ ℓ ϕ : W× R k → R as

We then bound the excess loss function as follows. For any ϕ ∈ Φ and ( w,z ) ∈ W × R k , we have:

Here, the first inequality follows from the Cauchy-Schwarz inequality, and the second inequality uses the assumptions that sup w ∈W sup ϕ ∈ Φ ∥ ϕ ( w ) -ϕ ⋆ ( w ) ∥ 2 ≤ U and ∥ z -ϕ ⋆ ( w ) ∥ 2 ≤ U .

Next, we construct an ϵ -cover of Φ . Let ϵ > 0 be fixed. By the definition of covering numbers, there exists an ϵ -cover Φ ϵ of Φ under the ∥ · ∥ ∞ norm, such that | Φ ϵ | = N (Φ , ϵ, ∥ · ∥ ∞ ) . This means that for every ϕ ∈ Φ , there exists a ϕ ϵ ∈ Φ ϵ such that ∥ ϕ -ϕ ϵ ∥ ∞ ≤ ϵ . We further bound the difference between the true and empirical excess risks for functions in the cover. For each ϕ ϵ ∈ Φ ϵ , define Z i ( ϕ ϵ ) := ∆ ℓ ϕ ϵ ( w i , z i ) . Then we have: (a) E [ Z i ( ϕ ϵ )] = E ( w,z ) ∼ D [∆ ℓ ϕ ϵ ( w,z )] =: R ( ϕ ϵ ) , (b) | Z i ( ϕ ϵ ) | ≤ 3 U 2 almost surely, and (c) E [ Z i ( ϕ ϵ ) 2 ] ≤ 9 U 2 R ( ϕ ϵ ) .

By Bernstein's inequality and a union bound over Φ ϵ , we have with probability at least 1 -δ/ 2 , for all ϕ ϵ ∈ Φ ϵ :

We then extend the bound to all functions in Φ . For any ϕ ∈ Φ , let ϕ ϵ ∈ Φ ϵ be such that ∥ ϕ -ϕ ϵ ∥ ∞ ≤ ϵ . Then, we have:

We will choose ϵ and simplify the bound. Set ϵ = U/n . Then, with probability at least 1 -δ/ 2 , for all ϕ ∈ Φ :

for any a > 0 .

Next, we can bound the risk of the ERM solution. Let ˜ ϕ ∈ arg min ϕ ∈ Φ R ( ϕ ) be a minimizer of the true risk. Then, on the event of the previous step, we have:

Here, ˆ R n ( ϕ ) := 1 n ∑ n i =1 ∆ ℓ ϕ ( w i , z i ) is the empirical excess risk, and the first inequality uses the fact that ˆ R n ( ˆ ϕ ) -ˆ R n ( ˜ ϕ ) ≤ 0 since ˆ ϕ minimizes the empirical risk. Choosing a = 1 / 2 , we get:

Rearranging the terms, we obtain:

Lemma D.20. Let F and G be two classes of embedding functions, where f ∈ F maps from an arbitrary domain X to a Hilbert t-Module M and g ∈ G maps from an arbitrary domain Y to M . Let H be a class of functions of the form h ( x, y ) = ⟨ f ( x ) , g ( y ) ⟩ M , where f ∈ F and g ∈ G . Let N F ( ϵ ) denote the ϵ -covering number of F , N G ( ϵ ) denote the ϵ -covering number of G , and N H ( ϵ ) denote the ϵ -covering number of H . If there exists a constant B such that for all f ∈ F , g ∈ G , x ∈ X , and y ∈ Y , we have ∥ f ( x ) ∥ M ≤ B and ∥ g ( y ) ∥ M ≤ B , then for any ϵ > 0 , the following inequality holds:

Proof of Lemma D.20. Let { f 1 , . . . , f N F } be an ϵ 2 B -cover of F and { g 1 , . . . , g N G } be an ϵ 2 B -cover of G , where N F = N F ( ϵ 2 B ) and N G = N G ( ϵ 2 B ) .

For any h ∈ H , there exist f ∈ F and g ∈ G such that h ( x, y ) = ⟨ f ( x ) , g ( y ) ⟩ M . By the definition of the covering sets, there exist f i and g j such that for all x ∈ X and y ∈ Y , we have ∥ f ( x ) -f i ( x ) ∥ M ≤ ϵ 2 B and ∥ g ( y ) -g j ( y ) ∥ M ≤ ϵ 2 B .

Now, consider the function h ij ( x, y ) = ⟨ f i ( x ) , g j ( y ) ⟩ M . We have:

This shows that the set { h ij : 1 ≤ i ≤ N F , 1 ≤ j ≤ N G } forms an ϵ -cover of H . The cardinality of this set is N F · N G = N F ( ϵ 2 B ) · N G ( ϵ 2 B ) , which implies that N H ( ϵ ) ≤ N F ( ϵ 2 B ) · N G ( ϵ 2 B ) .

Lemma D.21. Let F and G be two classes of vector-valued functions, where f ∈ F maps from an arbitrary domain X to R r and g ∈ G maps from an arbitrary domain Y to R r . Let H be a class of bivariate functions of the form h ( x, y ) = ⟨ f ( x ) , g ( y ) ⟩ , where f ∈ F and g ∈ G . Let N F ( ϵ ) denote the ϵ -covering number of F , N G ( ϵ ) denote the ϵ -covering number of G , and N H ( ϵ ) denote the ϵ -covering number of H . If there exists a constant B such that for all f ∈ F , g ∈ G , x ∈ X , and y ∈ Y , we have ∥ f ( x ) ∥ ≤ B and ∥ g ( y ) ∥ ≤ B , then for any ϵ > 0 , the following inequality holds:

Proof of Lemma D.21. Let { f 1 , . . . , f N F } be an ϵ 2 B -cover of F and { g 1 , . . . , g N G } be an ϵ 2 B -cover of G , where N F = N F ( ϵ 2 B ) and N G = N G ( ϵ 2 B ) .

For any h ∈ H , there exist f ∈ F and g ∈ G such that h ( x, y ) = ⟨ f ( x ) , g ( y ) ⟩ . By the definition of the covering sets, there exist f i and g j such that for all x ∈ X and y ∈ Y , we have ∥ f ( x ) -f i ( x ) ∥ ≤ ϵ 2 B and ∥ g ( y ) -g j ( y ) ∥ ≤ ϵ 2 B .

Now, consider the function h ij ( x, y ) = ⟨ f i ( x ) , g j ( y ) ⟩ . We have:

This shows that the set { h ij : 1 ≤ i ≤ N F , 1 ≤ j ≤ N G } forms an ϵ -cover of H . The cardinality of this set is N F · N G = N F ( ϵ 2 B ) · N G ( ϵ 2 B ) , which implies that N H ( ϵ ) ≤ N F ( ϵ 2 B ) · N G ( ϵ 2 B ) .

Proof of Lemma D.15. Let Φ := { ( x, y ) ↦→⟨ f ( x ) , g ( y ) ⟩ , ( f ( x ) , g ( y )) ∈ ˘ F ( i ) r i × ˘ G ( i ) r i } be a class of bivariate functions for each i ∈ [ K ] . We define the following target functions:

for all i ∈ [ K ] . Let D 3 be the distribution of ( x, y, z ) ∼ D train , and D 4 be the distribution of ( x ′ , y ′ , z ′ ) , where ( x ′ , y ′ ) ∼ D 1 ⊗ 1 and z ′ = ˘ ϕ ( i ) 4 ,⋆ ( x ′ , y ′ ) . We compute the following bounds:

According to Lemma D.21, the covering number of Φ satisfies

We define the following covering numbers:

For k ∈ { 3 , 4 } , let R k and ˆ L k,n k , ˆ R k,n k denote the corresponding population and empirical excess risks as in Lemma D.19. By applying Lemma D.19 with α = 1 / 2 and a union bound over all i ∈ [ K ] , we have with probability at least 1 -δ/ (3 K ) , for all ϕ ∈ Φ :

Now, define the combined population risk R ˘ ν ( i ) ( ϕ ) := R 3 ( ϕ ) + ˘ ν ( i ) R 4 ( ϕ ) . Let

ˆ

ϕ

∈

=

argmin

ϕ

∈

Φ

argmin

ϕ

∈

Φ

ˆ

L

3

ˆ

R

,n

3

3

,n

(

3

ϕ

(

) + ˘

ϕ

ν

(

) + ˘

ν

i

(

)

i

ˆ

L

)

4

ˆ

R

,n

4

4

,n

(

4

ϕ

(

)

ϕ

)

be an empirical risk minimizer, and let ˜ ϕ ∈ argmin ϕ ∈ Φ R ˘ ν ( i ) ( ϕ ) be a population risk minimizer. Then, we have

where the last inequality holds because n 4 ≥ n 1 n 3 max { 2˘ ν ( i ) , 3 } , ∀ i ∈ [ K ] implies N r i ,n 4 ≥ N r i ,n 3 .

Rearranging terms, we get

To conclude the proof, we bound the terms R ˘ ν ( i ) ( ˆ ϕ ) and R ˘ ν ( i ) ( ˜ ϕ ) . First, for the population risk minimizer ˜ ϕ , we have

Second, for the empirical risk minimizer ˆ ϕ , we have

Combining the bounds for R ˘ ν ( i ) ( ˆ ϕ ) and R ˘ ν ( i ) ( ˜ ϕ ) , we conclude

D.3.2 A Lemma on the Risk Bound for t-Bilinear Combinatorial Extrapolation

We now present a lemma on the risk bound for t-bilinear combinatorial extrapolation. This bound depends on the upper bound ϵ trn on the risk of the learned embedding ( ˆ f , ˆ g ) for the training distribution D train , the upper bound ϵ 1 ⊗ 1 on the risk for the top-block distribution D 1 ⊗ 1 , and σ r ( ˆ f , ˆ g ) .

Lemma D.22. Given vectors r = ( r i ) K i =1 ∈ N K , α = ( α i ) K i =1 ∈ R K , ϵ trn = (˘ ϵ ( i ) trn ) K i =1 ∈ R K , and ϵ D 1 ⊗ 1 = (˘ ϵ ( i ) D 1 ⊗ 1 ) K i =1 ∈ R K , where α i ≥ 1 , ˘ ϵ ( i ) trn , ˘ ϵ ( i ) D 1 ⊗ 1 ≥ 0 for all i ∈ [ K ] , suppose ( ˆ f , ˆ g ) are α -conditioned and ( ϵ trn , ϵ D 1 ⊗ 1 ) -accurate R ∥ r ∥ ∞ × 1 × K -embeddings of tensor-multl-rank r , where r i ≤ ˘ σ ⋆, ( i ) 1 / (40˘ ϵ ( i ) D 1 ⊗ 1 ) . If ˘ σ ⋆, ( i ) r i > 0 , then

Lemma D.22 provides an upper bound on the excess risk ˘ R ( i ) ( ˘ ˆ f ( i ) , ˘ ˆ g ( i ) ; D test ) for each i -th frequency component of the learned embeddings on D test . This bound depends on the embedding rank r i , accuracy parameters ˘ ϵ ( i ) trn and ˘ ϵ ( i ) D 1 ⊗ 1 , conditioning parameter α i , and the singular values and tail sums of the true embeddings. The embeddings must be α -conditioned and ( ϵ trn , ϵ D 1 ⊗ 1 ) -accurate, with r i limited relative to ˘ σ ⋆, ( i ) 1 / ˘ ϵ ( i ) D 1 ⊗ 1 .

Proof of Lemma D.22. The proof closely follows the arguments in Theorem 8 of [46], extending them to the multi-output setting using the tensor algebraic tools developed in our work.

where P ⋆ s i is the projection operator onto the tops i eigenspaces of M ( Σ ⋆ 1 ⊗ 1 ) ( i ) .

Suppose ( ˆ f , ˆ g ) are full-multi-rankr . Then there exists an index vector k = ( k 1 , · · · , k K ) ⊤ ∈ N K with k i ∈ [min { r i , s i -1 } ] for all i ∈ [ K ] , and functions f : X → M and g : Y → M such that ( f , g ) are aligned k -proxies of ( ˆ f , ˆ g ) . The construction of ( f , g ) follows the same steps as in the proof of Theorem 8(a) in [46], applied to each component ( ˘ ˆ f ( i ) , ˘ ˆ g ( i ) ) separately. Specifically, we set

where ι r i : R r i →M i is a t-isometric inclusion, M i is the i -th component Hilbert space of M , and ˘ T ( i ) is the balancing operator for ( ˘ ˆ f ( i ) , ˘ ˆ g ( i ) ) as defined in Lemma 15. The index k i is chosen as the largest integer in [min { r i , s i -1 } ] such that ˘ σ ⋆, ( i ) k i -˘ σ ⋆, ( i ) k i +1 ≥ ˘ σ ⋆, ( i ) k i min { r i ,s i -1 } . The existence of such a k i is guaranteed by the gap condition (36) and the fact that ( ˘ ˆ f ( i ) , ˘ ˆ g ( i ) ) are full-rank, as shown in [46].

With the aligned proxies ( f , g ) constructed, we can bound their error terms using the same arguments as in the proof of Theorem 8(b) in [46], applied componentwise. This yields the bounds

Finally, suppose ˘ ϵ 2 i ≤ (1 -α -1 i )(˘ σ ⋆, ( i ) r i ) 2 for some α i ≥ 1 . Then, as shown in the proof of Theorem 8(c) in [46], we have σ r i ( ˘ ˆ f ( i ) , ˘ ˆ g ( i ) ) 2 ≥ (˘ σ ⋆, ( i ) r i ) 2 /α i , and hence ( ˆ f , ˆ g ) are necessarily full-multi-rankr .

Combining these bounds and applying a refined error decomposition in each i -th frequency component induced by the transform M ( · ) in Eq. (1), we arrive at the risk bound

Here, in the last step, we used the fact that s i ≤ r i and ˘ ϵ i ≤ ˘ ϵ ( i ) D 1 ⊗ 1 , and we absorbed the constant factors into the ≲ ⋆ notation. This completes the proof of the lemma.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The paper introduces a tensor spectral theory approach to address multi-output regression challenges under combinatorial distribution shift (CDS), marking a pioneering theoretical exploration in this area.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: It acknowledges several limitations: the potential inadequacy of the spectral methods against real-world data complexity and the uncertain robustness of results from controlled experiments. We suggest future research to refine these methods fostering the development of more effective solutions.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The paper provide the full set of assumptions and a complete (and correct) proof for the theroetical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer:[Yes]

Justification: This paper presents a pioneering theoretical exploration of multi-output regression challenges under CDS using a generalized tensor spectral theory. Since no existing algorithms address these specific models, our experiments use numerical simulations with synthetic data to validate preliminary results. Details are provided in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide the code for our conceptually validation.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We show the experiment details in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The paper reports error bars and provides appropriate information about the statistical significance of the experiments, with results obtained from multiple repeated trials.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: For the experiment, the paper provides sufficient information on the computer resources needed to reproduce the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This paper conforms in every respect with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The paper provides a new tensor spectral perspective for the multi-output regression problem due to combinatorial distribution shift. It focuses solely on these technical aspects and does not have potential societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper does not describe safeguards for the responsible release of data or models because it focuses exclusively on the technical aspects of addressing combinatorial distribution shift in multi-output regression. Therefore, this question is not applicable (N/A).

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This paper does not use any external assets such as code, data, or models. Therefore, the question regarding the crediting of creators or original owners, as well as the respect for licenses and terms of use, is not applicable (N/A).

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer:[NA]

Justification: This paper does not introduce any new assets. Therefore, the question regarding the documentation of new assets is not applicable (N/A).

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing experiments or research with human subjects. Therefore, the question regarding instructions, screenshots, and compensation details is not applicable (N/A).

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve study participants or human subjects. Therefore, the question regarding potential risks, disclosures to subjects, and IRB approvals is not applicable (N/A).

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the problem of multi-output regression under combinatorial covariate shift, namely when the input domains in the testing set differ significantly from the training set. The authors view the functional mapping from the inputs to the vector-valued output, which is evaluated discretely in the input domain, as a tensor with missing tubes, and convert the problem of making test set predictions as a tensor completion problem. Utilizing the framework of low tubal-rank tensor decomposition, the paper generalizes this framework from the discrete case to the continuous case and proposes a functional t-SVD method that decomposes the functional mapping into a series of embedding functions with associated singular values. Then an empirical risk minimization (ERM) algorithm is proposed and evaluated with a toy example numerical experiment. The major contribution of the paper is a solid theoretical analysis of the approximability of the functional t-SVD as well as the excess risk of the ERM algorithm and the writing of the paper is clear.\n\nSTRENGTHS:\n* The paper bridges the low-rank tensor decomposition in the discrete domain to a functional approximation problem in the continuous domain, which is a rather novel approach when dealing with missing data issues for functional data.\n\n* The paper provides a series of solid theoretical analyses regarding the approximability of the vector-valued functions as well as the excess risk bound of the empirical risk minimizer.\n\nWEAKNESSES:\n1. The empirical analyses of the paper lack comparisons with some other benchmark approaches, making it hard to evaluate the effectiveness of the proposed methodology. It would be interesting and informative to consider other tensor decomposition approaches (CP, Tucker, Tensor-Train, Tubal) as benchmarks.\n\nQUESTIONS:\n1. The tubal rank framework introduced in this paper, to the best of my knowledge, works for 3-order tensors. Does that indicate that the functional t-SVD framework considered mainly works for inputs with dimension 2? Is it possible to generalize the current approach to higher-order tensors (and thus higher-dimensional inputs)?\n\n2. The functional t-SVD framework in Theorem 1 resembles the Mercer decomposition of kernels. What is the main advantage of the current approach over some existing alternatives such as the multi-output Gaussian Process regression?\n\n3. Maybe I missed it in the manuscript, but is there a systematic approach for choosing the rank $r$ when implementing the functional t-SVD? Is there any sense of optimal $r$ that might be derived by minimizing the excess risk bound w.r.t. $r$?\n\nLIMITATIONS:\n1. Additional numerical experiments that demonstrate the effectiveness of the proposed functional t-SVD over existing benchmark methods can be informative and helpful.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-17"}

comments: SUMMARY:
This paper proposes function t-SVD for multi-output regression under combinatorial shifts. Excess-risk bounds have been derived. Using simulation experiments risk bounds with combinatorial shifts has been compared with regular risk bounds.

STRENGTHS:
1) Proposal of functional t-SVD, which is a descent contribution.
2) Solid theoretical analysis

WEAKNESSES:
1) Lack of any real-data experiments. It is hard to understand the usefulness of the proposed method in real-world application though the theoretical contribution may be high. 
2) Lack of comparison to other related function tensor decomposition methods either by theory or experiments.

QUESTIONS:
Lack of any real-world application is limitation of the paper. Since the paper is proposing a functional tensor decomposition methods, it would be useful give at least one application to understand its learning capability. Can authors provide some experiments on real-data set?

Some references such as function tensor-train is missing [1]. Can authors provide more detailed related methods?

How does the proposed method compare with other functional tensor decomposition methods? How does the theoretical results (ERM bounds) compared with any existing results?  

1 Alex Gorodetsky, Sertac Karaman, Youssef Marzouk, A continuous analogue of the tensor-train decomposition, Computer Methods in Applied Mechanics and Engineering, Volume 347, 2019, Pages 59-84,

LIMITATIONS:
Limitations listed but insufficient in my opinion due to any experiments with real-data.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel approach to multi-output regression (MOR) under combinatorial distribution shifts (CDS) using a generalized tensor decomposition framework. The proposed Functional t-Singular Value Decomposition (Ft-SVD) extends classical tensor SVD to infinite and continuous feature domains, providing a new perspective on handling MOR tasks under CDS. The authors present a Double-Stage Empirical Risk Minimization (ERM-DS) algorithm designed to improve prediction accuracy in the presence of CDS. Through rigorous theoretical analysis, the paper establishes performance guarantees for the proposed algorithm and demonstrates its efficacy with synthetic data experiments.\n\nSTRENGTHS:\n1.\tClear and Reasonable Motivation: The paper presents a clear and reasonable motivation for using tensor completion to address multi-output regression under combinatorial distribution shifts. By framing the problem as a tensor completion task, the method leverages the strengths of tensor decomposition techniques to manage the challenge of unseen feature combinations in MOR, effectively improving generalization capabilities.\n\n2.\tSolid Theoretical Analysis: The detailed proofs and theoretical guarantees offered in the paper are impressive and contribute significantly to the understanding and validation of the proposed methods. This rigorous approach ensures that the findings are well-supported and credible\n\nWEAKNESSES:\n1.\tThe paper's definition, introduction, and highlight of Combinatorial Distribution Shift (CDS) are insufficient. It is not clear why CDS poses a significant challenge in MOR and why it is important.\n\n2.\tThe relevant definitions, assumptions, and proofs are presented with excessive detail, which makes it difficult for readers to grasp the core issues and contributions. A more concise and focused explanation would help in conveying the importance and difficulty of CDS in MOR.\n\n3. The paper does not adequately discuss the existing body of work on functional tensor completion, such as [Bigoni et al., 2016], [Luo et al., 2023], and [Fang et al., 2024]. Including a more comprehensive discussion on related works and how the proposed method compares to them would provide better context and highlight the contributions more effectively. \n\nRef:\n- Bigoni, Daniele, Allan P. Engsig-Karup, and Youssef M. Marzouk. \"Spectral tensor-train decomposition.\" SIAM Journal on Scientific Computing 38.4 (2016): A2405-A2439.\n\n- Luo, Yisi, et al. \"Low-rank tensor function representation for multi-dimensional data recovery.\" IEEE Transactions on Pattern Analysis and Machine Intelligence (2023).\n\n- Fang, Shikai, et al. \"Functional Bayesian Tucker Decomposition for Continuous-indexed Tensor Data.\" arXiv preprint arXiv:2311.04829 (2023).\n\nQUESTIONS:\nSee weakness\n\nLIMITATIONS:\nSee weakness", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new model for decomposing vector-valued functions of two vector arguments. The proposed model considers a SVD decomposition of these functions in Hilbert space, effectively extending t-SVD to Hilbert spaces. Here t-SVD, short for tensor-SVD, consists of applying SVD to each matrix slice of a tensor, after multiplication by a unitary (although authors restrict these to orthogonal) matrix $M$ in the third dimension. They propose using this approach for Multi-Output Regression, and claim that the proposed approach is especially suited for generalizing pairs of arguments that are not present in the training data, in a similar fashion as t-SVD may be used for tensor completion.\n\nSTRENGTHS:\nThe method proposed in the paper appears to be very promising. The overall approach seems to be highly generalizable and it may possibly be used together with several function approximation techniques.\n\nWEAKNESSES:\nThe biggest weakness is the lack of numerical experiments. The overall idea of the paper is interesting, but it is not ground-breaking. Without the multiplication by $M$, this is essentially a regression in each dimension. Here numerical experiments are needed as a way to investigate possible choices of $M$ in applications: should we choose well-known orthogonal basis given by DFT or DCT? Or should we do a data-driven approach?\n\nQUESTIONS:\nOverall, the statement of Theorem 4 is not very clear to me. On the other hand, it seems to be a generalization of reference [37] of the paper to multiple output regression. Can you explain what are the differences and similatities between both results? Which parts of the result in [37] generalize well to the new approach, and what were the novel analysis that needed to be developed?\n\nLIMITATIONS:\nI believe that the limitations are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-19"}