Title: Analysing Multi-Task Regression via Random Matrix Theory with Application to Time Series Forecasting
Venue: NeurIPS 2024
Accepted: yes
Keywords: Random Matrix Theory ; Optimization ; Regularization ; Multi-task regression ; Multi-task learning ; Multivariate Time Series Forecasting
Abstract: In this paper, we introduce a novel theoretical framework for multi-task regression, applying random matrix theory to provide precise performance estimations, under high-dimensional, non-Gaussian data distributions. We formulate a multi-task optimization problem as a regularization technique to enable single-task models to leverage multi-task learning information. We derive a closed-form solution for multi-task optimization in the context of linear models. Our analysis provides valuable insights by linking the multi-task learning performance to various model statistics such as raw data covariances, signal-generating hyperplanes, noise levels, as well as the size and number of datasets. We finally propose a consistent estimation of training and testing errors, thereby offering a robust foundation for hyperparameter optimization in multi-task regression scenarios. Experimental validations on both synthetic and real-world datasets in regression and multivariate time series forecasting demonstrate improvements on univariate models, incorporating our method into the training loss and thus leveraging multivariate information.

Analysing Multi-Task Regression via Random Matrix Theory with Application to Time Series Forecasting

Abstract

In this paper, we introduce a novel theoretical framework for multi-task regression, applying random matrix theory to provide precise performance estimations, under high-dimensional, non-Gaussian data distributions. We formulate a multi-task optimization problem as a regularization technique to enable single-task models to leverage multi-task learning information. We derive a closed-form solution for multi-task optimization in the context of linear models. Our analysis provides valuable insights by linking the multi-task learning performance to various model statistics such as raw data covariances, signal-generating hyperplanes, noise levels, as well as the size and number of datasets. We finally propose a consistent estimation of training and testing errors, thereby offering a robust foundation for hyperparameter optimization in multi-task regression scenarios. Experimental validations on both synthetic and real-world datasets in regression and multivariate time series forecasting demonstrate improvements on univariate models, incorporating our method into the training loss and thus leveraging multivariate information.

1 Introduction

The principle of multi-task learning, which encompasses concepts such as "learning to learn" and "knowledge transfer," offers an efficient paradigm that mirrors human intelligence. Unlike traditional machine learning, which tackles problems on a task-specific basis with tailored algorithms, multi-task learning leverages shared information across various tasks to enhance overall performance. This approach not only improves accuracy but also addresses challenges related to insufficient data and data cleaning. By analyzing diverse and multimodal data and structuring representations multi-task learning can significantly boost general intelligence, similar to how humans integrate skills. This concept is well-established [33] and has been successfully applied to a wide range of domains, such as computer vision [41], natural language processing [35, 37] and biology [17, 28, 42].

Our Method. Weconsider the multi-task regression framework [5] with T tasks with the input space X ( t ) ⊂ R d and the output space Y ( t ) ⊂ R q for t ∈ { 1 , . . . , T } . For each task t , we assume that we are given n t training examples organized into the feature matrix X ( t ) = [ x ( t ) 1 , . . . , x ( t ) n t ] ∈ R d × n t and the corresponding response matrix Y ( t ) = [ y ( t ) 1 , . . . , y ( t ) n t ] ∈ R q × n t , where x ( t ) i ∈ X ( t ) represents the i -th feature vector of the t -th task and y ( t ) i ∈ Y ( t ) is the associated response. In order to have a more tractable and insightful setup, we follow Romera-Paredes et al. [36] and consider the linear multi-task regression model. In particular, we study a straightforward linear signal-plus-noise model

that evaluates the response y ( t ) i for the i -th sample of the t -th task as follows:

where ε ( t ) ∈ R n t × q is a matrix of noise vectors with each ε ( t ) i ∼ N (0 , Σ N ) , Σ N ∈ R q × q denoting the covariance matrix. The matrix W t ∈ R d × q denotes the signal-generating hyperplane for task t . We denote the concatenation of all task-specific hyperplanes by W = [ W ⊤ 1 , . . . , W ⊤ T ] ⊤ ∈ R Td × q . We assume that W t can be decomposed as a sum of a common matrix W 0 ∈ R d × q , which captures the shared information across all the tasks, and a task-specific matrix V t ∈ R d × q , which captures deviations specific to the task t :

Given the multitask regression framework and the linear signal-plus-noise model, we now want to retrieve the common and specific hyperplanes, W 0 and V t , respectively. To achieve this, we study the following minimization problem governed by a parameter λ that controls the balance between the common and specific components of W :

where γ = [ γ 1 , . . . , γ T ] is a vector of task-specific regularization hyperparameters. Each γ t controls how much the model overfits (small γ t ) or underfits (large γ t ) on task t .

Contributions and Main Results. We seek to provide a rigorous theoretical study of (3) in high dimension, providing practical insights that make the theoretical application implementable in practice. Our contributions can be summarized in four major points:

We provide exact train and test risks for the solutions of (3) using random matrix theory. We then decompose the test risk into a signal and noise term responsible for the effectiveness of multi-task learning and the negative transfer, respectively.

We show how the signal and noise terms compete with each other depending on λ for which we obtain an optimal value optimizing the test risk.

In a particular case, we derive a closed-form data-dependent solution for the optimal λ ⋆ , involving raw data covariances, signal-generating hyperplane, noise levels, and the size and number of data sets.

We offer empirical estimates of these model variables to develop a ready-to-use method for hyperparameter optimization in multitask regression problems within our framework.

Applications. As an application, we view multivariate time series forecasting (MTSF) as a multitask problem and show how the proposed regularization method can be used to efficiently employ univariate models in the multivariate case. Firstly, we demonstrate that our method improves PatchTST [32] and DLinear [53] compared to the case when these approaches are applied independently to each variable. Secondly, we show that the univariate models enhanced by the proposed regularization achieve performance similar to the current state-of-the-art multivariate forecasting models such as SAMformer [19] and iTransformer [24].

2 Related Work

Usual Bounds in Multi-Task Learning. Since its introduction [1, 5, 26], multi-task learning (MTL) has demonstrated that transfer learning concepts can effectively leverage shared information to solve related tasks simultaneously. Recent research has shifted focus from specific algorithms or minimization problems to deriving the lowest achievable risk given an MTL data model. The benefits of transfer learning are highlighted through risk bounds, using concepts such as contrasts [23], which study high-dimensional linear regression with auxiliary samples characterized by the sparsity of their contrast vector or transfer distances [29], which provide lower bounds for target generalization

error based on the number of labeled source and target data and their similarities. Additionally, task-correlation matrices [30] have been used to capture the relationships inherent in the data model.

However, these studies remain largely theoretical, lacking practical algorithms to achieve the proposed bounds. They provide broad estimates and orders of magnitude that, while useful for gauging the feasibility of certain objectives, do not offer precise performance evaluations. Moreover, these estimates do not facilitate the optimal tuning of hyperparameters within the MTL framework, a critical aspect of our study. We demonstrate the importance of precise performance evaluations and effective hyperparameter tuning to enhance the practical application of MTL.

Random Matrix Theory in Multi-Task Learning. In the high-dimensional regime, Random Matrix Theory allows obtaining precise theoretical estimates on functionals (e.g., train or test risk) of data matrices with the number of samples comparable to data dimension [2, 13, 31, 46]. In the multi-task classification case, large-dimensional analysis has been performed for Least Squares SVM [48] and supervised PCA [47] showing counterproductivity of tackling tasks independently. We inspire our optimization strategy from [48] and conduct theoretical analysis for the multi-task regression, which introduces unique theoretical challenges and with a different intuition since we need to consider another data generation model and a regression learning objective. Our data modeling shares similarity with [52] which theoretically studied conditions of a negative transfer under covariate and model shifts. Our work focuses on the selection of hyperparameters within a general optimization framework, considering both a hyperparameter λ to relate all tasks and specific parameters γ t to balance overfitting and underfitting within each task, accommodating non-Gaussian distributions and employing recent developments such as deterministic equivalents and concentration inequalities.

High dimensional analysis for regression. Regression is an important problem that has been extensively explored in the context of single-task learning for high-dimensional data, employing tools such as Random Matrix Theory [12], physical statistics [9, 15], and the Convex Gaussian MinMax Theorem (CGMT) [43], among others. Typically, authors employ a linear signal plus noise model to calculate the asymptotic test risk as a function of the signal-generating parameter and the covariance of the noise. Our research builds upon these studies, extending them to a multi-task learning framework. In doing so, we derive several insights unique to multi-task learning. However, none of these previous studies offer a practical method for estimating the asymptotic test risk, which is dependent on the ground truth signal-generating parameter. Therefore, our work not only extends previous studies into the multi-task case within a generic data distribution (under the assumption of a concentrated random vector), but also provides a practical method for estimating these quantities.

3 Framework

Notation. Throughout this study, matrices are represented by bold uppercase letters (e.g., matrix A ), vectors by bold lowercase letters (e.g., vector v ), and scalars by regular, non-bold typeface (e.g., scalar a ). The notation A ⊗ B for matrices or vectors A , B is the Kronecker product. D x or Diag ( x ) stands for a diagonal matrix containing on its diagonal the elements of the vector x . The superscripts t and i are used to denote the task and the sample number, respectively, e.g., x ( t ) i writes the i -th sample of the t -th task. The canonical vector of R T is denoted by e [ T ] t with [ e [ T ] t ] i = δ ti . Given a matrix M ∈ R p × n , the Frobenius norm of M is denoted ∥ M ∥ F ≡ √ tr( M ⊤ M ) . For our theoretical analysis, we introduce the following notation of training data:

where n = ∑ T t =1 n t is the total number of samples in all the tasks.

3.1 Regression Model

We define V = [ V ⊤ 1 , . . . , V ⊤ T ] ⊤ ∈ R dT × q and propose to solve the linear multi-task problem by finding ˆ W = [ ˆ W ⊤ 1 , . . . , ˆ W ⊤ T ] ⊤ ∈ R dT × q which solves the following optimization problem under the assumption of relatedness between the tasks, i.e., W t = W 0 + V t for all tasks t :

where

The objective function consists of three components: a regularization term for W 0 to mitigate overfitting, a task-specific regularization term that controls deviations V t from the shared weight matrix W 0 , and a loss term quantifying the error between the predicted outputs and the actual responses for each task.

The optimization problem is convex for any positive values of λ, γ 1 , . . . , γ T , and it possesses a unique solution. The details of the calculus are given in Appendix B. The formula for ˆ W t is given as follows:

where with Q = ( Z ⊤ AZ Td + I n ) -1 ∈ R n × n , and A = ( D γ + λ 1 T 1 ⊤ T ) ⊗ I d ∈ R Td × Td .

3.2 Assumptions

In order to use Random Matrix Theory (RMT) tools, we make two assumptions on the data distribution and the asymptotic regime. Following [50], we adopt a concentration hypothesis on the feature vectors x ( t ) i , which was shown to be highly effective for analyzing machine learning problems [11, 14]. The justification and implications of these assumptions can be found in appendix A.2 and A.3.

Assumption 1 (Concentrated Random Vector) . We assume that there exists two constants C, c > 0 (independent of dimension d ) such that, for any task t , for any 1 -Lipschitz function f : R d → R , any feature vector x ( t ) ∈ X ( t ) verifies:

In particular, we distinguish the following scenarios: x ( t ) i ∈ R d are concentrated when they are (i) independent Gaussian random vectors with covariance of bounded norm, (ii) independent random vectors uniformly distributed on the R d sphere of radius √ d , and most importantly (iii) any Lipschitz transformation ϕ ( x ( t ) i ) of the above two cases, with bounded Lipschitz norm. Scenario (iii) is especially pertinent for modeling data in realistic settings. Recent research [39] has demonstrated that images produced by generative adversarial networks (GANs) are inherently qualified as concentrated random vectors.

Next, we present a classical RMT assumption that establishes a commensurable relationship between the number of samples and dimension.

Assumption 2 (High-dimensional asymptotics) . As d → ∞ , n t = O ( d ) and T = O (1) . More specifically, we assume that n/d a.s. - - → c 0 < ∞ with n = ∑ T t =1 n t .

Although different from classical asymptotic where the number of samples is implicitly assumed to be exponentially larger than the dimension, the high-dimensional asymptotic finds many applications including telecommunications [10], finance [34] and machine learning [11, 14, 48].

4 Main Theoretical Results

4.1 Estimation of the Performances

Given the training dataset X ∈ R n × d with the corresponding response variable Y ∈ R n × q and for any test dataset x ( t ) ∈ R d and the corresponding response variable y ( t ) ∈ R q , we aim to derive the theoretical training and test risks defined as follows:

The output score g ( x ( t ) ) ∈ R q for data x ( t ) ∈ R d of task t is given by:

In particular, the output for the training score is given by:

To understand the statistical properties of R ∞ train and R ∞ test for the linear generative model, we study the statistical behavior of the resolvent matrix Q defined in (6). The notion of a deterministic equivalent from random matrix theory [16] is particularly useful here as it allows us to design a deterministic matrix equivalent to Q in the sense of linear forms. Specifically, a deterministic equivalent ¯ M of a random matrix M is a deterministic matrix that satisfies u ( M -¯ M ) → 0 almost surely for any bounded linear form u : R d × d → R . We denote this as M ↔ ¯ M . This concept is crucial to our analysis because we need to estimate quantities such as 1 d tr ( AQ ) or a ⊤ Qb , which are bounded linear forms of Q with A , a and b all having bounded norms. For convenience, we further express some contributions of the performances with the so-called 'coresolvent' defined as ˜ Q ≡ ( A 1 2 ZZ T A 1 2 + I Td ) -1 .

Using Lemma 1 provided in the Appendix C.1, whose proofs are included in Appendices C.2, C.3 and C.4, we establish the deterministic equivalents that allow us to introduce our Theorems 5 and 1, characterizing the asymptotic behavior of both training and testing risks.

Theorem 1 (Asymptotic train and test risk) . Assuming that the training data vectors x ( t ) i and the test data vectors x ( t ) are concentrated random vectors, and given the growth rate assumption (Assumption 2), it follows that:

In addition, the asymptotic risk on the training data is given by

where ¯ ˜ Q , ¯ ˜ Q 2 ( I Td ) and ¯ Q 2 are respectively deterministic equivalents for ˜ Q , ˜ Q 2 and Q 2 .

We defer the full proof of this theorem to Appendix D and provide an outline of the test risk proof below. Note that derivations of the asymptotic risk for the training data are more complex and involve computing deterministic equivalents (Lemma 1 of Appendix C.1), which is a powerful tool commonly used in Random Matrix Theory. In the derived theorem, we achieve a stronger convergence result compared to the almost surely convergence. Specifically, we prove a concentration result for the training and test risk with an exponential convergence rate for sufficiently large values of n and d .

4.2 Error Contribution Analysis

To gain theoretical insights, we analyze (ATR) consisting of the signal and the noise components.

̸

Signal Term. The signal term can be further approximated, up to some constants as tr( W ⊤ ( AΣ + I ) -2 W ) with Σ = ∑ T t =1 n t d Σ ( t ) . The matrix ( AΣ + I ) -2 plays a crucial role in amplifying the signal term tr( W ⊤ W ) , which in turn allows the test risk to decrease. The off-diagonal elements of AΣ + I ) -2 amplify the cross terms ( tr( W ⊤ v W t ) for t = v ), enhancing the multi-task aspect, while the diagonal elements amplify the independent terms ( ∥ W t ∥ 2 2 ). This structure is significant in determining the effectiveness of multi-task learning.

Furthermore, both terms decrease with an increasing number of samples n t , smaller values of γ t , and a larger value of λ . The cross term, which is crucial for multi-task learning, depends on the matrix Σ -1 t Σ v . This matrix represents the shift in covariates between tasks. When the features are aligned (i.e., Σ -1 t Σ v = I d ), the cross term is maximized, enhancing multi-task learning. However, a larger Fisher distance between the covariances of the tasks results in less favorable correlations for multi-task learning.

Noise term. Similar to the signal term, the noise term can be approximated, up to some constants, as tr( Σ N ( A -1 + Σ ) -1 ) . However, there is a major difference between the way both terms are expressed in the test risk. The noise term does not include any cross terms because the noise across different tasks is independent. In this context, only the diagonal elements of the matrix are significant. This diagonal term increases with the sample size and the value of λ . It is responsible for what is known as negative transfer. As the diagonal term increases, it negatively affects the transfer of learning from one task to another. This is a critical aspect to consider in multi-task learning scenarios.

4.3 Simplified Model for Clear Insights

In this section, we specialize the theoretical analysis to the simple case of two tasks ( T = 2 ). We assume that the tasks share the same identity covariance and that γ 1 = γ 2 ≡ γ . Under these conditions, the test risk can be approximated, up to some constants, as

where the diagonal term (independent learning) D IL , the cross term (multi-task learning) C MTL , and the noise term (negative transfer) N NT have closed-form expressions depending on γ and λ :

(a)

n

= 0

.

5

(b)

n

= 1

.

5

(c)

n

= 2

.

5

Signal Term

Signal Term

Signal Term

Cross-Term

Test Risk

Cross-Term

Test Risk

Cross-Term

Test Risk

0 . 4

Cest

1.5

2.0

est

0.3

入*=0.60

S

0.15

0.10

入*=1.67

the

0.2

to

1.0入*=0.01

0.1

0.05

Contribution

0.5

uo

uo

0.0

0.00

0.0

Cont

H

0 . 1

Contri

0 . 05

10 -2

10-1

Lambda

100

101

102

10-2

10-1

Lambda

100

101

1 02

1 02

10-1

Lambda

100

101

1 02

d

d

d

We recall that c 0 has been defined in the Assumption 2. As previously mentioned, the test risk is primarily composed of two terms: the signal term and the noise term, which are in competition with each other. The more similar the tasks are, the stronger the signal term becomes. In the following plot, we illustrate how this competition can influence the risk. Depending on the value of the parameter λ and the sample sizes, the risk can either decrease monotonically, increase monotonically, or exhibit a convex behavior. This competition can lead to an optimal value for λ , which interestingly has a simple closed-form expression that can be obtained by deriving the R ∞ test w.r.t. λ as follows (see details in Appendix E.4):

4.4 Comparison between Empirical and Theoretical Predictions

In this section, we compare the theoretical predictions with the empirical results. Our experiment is based on a two-task setting ( T = 2 ) defined as W 1 ∼ N (0 , I p ) with W 2 = α W 1 + √ 1 -α 2 W ⊥ 1 . W ⊥ 1 represents any vector orthogonal to W 1 and α ∈ [0 , 1] . This setting allows us to adjust the similarity between tasks through α .

Figure 2 shows a comparison of the theoretical and empirical classification errors for different values of λ , highlighting the error-minimizing value of λ . Despite the relatively small values of n and p , there is a very precise match between the asymptotic theory and the practical experiment. This is particularly evident in the accurate estimation of the optimal value for λ .

α=0 . 0

α=0.0

α=0.125

Q=0.125

α=0.25

α=0.25

10 1

α=0.5

α=0.375

Q=0.375

α=0.5

10 0

Train

102

Test

10-3

10 -2

101

10°

101

102

10 3

10 ~2

101

10°

101

102

103

Lambda

Lambda

(a) Training MSE

(b) Testing MSE

Theoretical

Empirical

18

1 7

Training

1 6

15

1 4

103

1 02

1 01

100

101

Lambda

Theoretical

Empirical

12 .1

12 . 0

esting

11 . 9

11.8

11 . 7

1 03

102

1 01

100

101

Lambda

5 Hyperparameter Optimization in Practice

5.1 Empirical Estimation of Task-wise Signal, Cross Signal, and Noise

All the quantities defined in Theorem 1 are known except for the bilinear expressions 1 Td tr( W ⊤ MW ) and 1 Td tr ( Σ N M ) . These quantities can be consistently estimated under Assumptions 2 as follows (see details in Section F of the Appendix) :

We define the estimate of the noise as ˆ σ = lim λ → 0 γ t →∞ R ∞ train and the function κ -1 is the functional inverse of the mapping κ : R Td × Td → R q × q defined as follows

5.2 Application to Multi-task Regression

In our study, we apply the theoretical framework presented in our paper to a real-world regression problem, specifically, the Appliance Energy dataset which aims to predict the total usage of a house. This dataset is a multivariate regression dataset containing 138 time series each of dimension 24 . We select two of these features as 2 tasks to cast the problem as a multi-task learning regression problem.

Figure 3 presents a comparison between the theoretical predictions and empirical outcomes of our proposed linear framework. Despite the assumptions made in the main body of the paper, the theoretical predictions closely align with the experimental results. This demonstrates that our estimates are effective in practice and provide a reliable approximation of the optimal regularization.

In essence, our theoretical framework, when applied to real-world multi-task learning regression problems, yields practical and accurate estimates, thereby validating its effectiveness and applicability.

5.3 Relevance of the theoretical insights beyond the case of linear models

While non-linear models are widely used, establishing their theoretical foundations is challenging. Therefore, we focused on linear models, which, despite their simplicity, provide valuable insights into more complex models.

Our results show that test risk curves for non-linear models follow patterns predicted by our theory. This is expected because non-linear models in time series forecasting typically use a linear output layer for prediction. Thus, we can apply our theory to the inputs of this final linear layer. This approach is valid due to data concentration and the Lipschitz nature of neural networks, ensuring outputs of the non-linear part don't deviate significantly from the inputs.

Moreover, multivariate time series models often treat channels separately using univariate methods, missing cross-channel information. Our results in Section 5.4 show that our method surpasses univariate baselines by optimally regularizing with λ and γ , supporting our theory's applicability to non-linear models as the final linear layer effectively leverages concentrated inputs.

Finally, our regularization approach differs from traditional cross-task regularizations that use one task per dataset. We consider each prediction as a task and introduce γ t parameters alongside λ . These parameters enforce multivariate regularization and control underfitting or overfitting per task. This method is tractable since it's applied at the model's final layer.

The similarity between curves for non-linear and linear models indicates our findings are robust; nonlinear models also exhibit optimal regularization parameters, enhancing performance in multivariate forecasting.

5.4 Application to Multivariate Time Series Forecasting

Our theoretical framework is applied in the context of Multivariate Time Series Forecasting, with related work detailed in Appendix G.1. We previously applied this framework in a linear setting, and now aim to evaluate its empirical validity in the non-linear setting of neural networks. The results presented in this section represent the best test MSE, assuming the ability to find the optimal lambda value, which can be considered as an oracle scenario. A study of these limitations can be found in Appendix H.

Motivation. Our approach is applied to the MTSF setting for several reasons. Firstly, many models currently used are essentially univariate, where predictions for individual series are simply concatenated without exploiting the multivariate information inherent in traditional benchmarks. Given that these benchmarks are designed for multivariate forecasting, leveraging multivariate information should yield better results. Secondly, our theoretical framework can benefit this domain, as most predictive models use a linear layer on top of the model to project historical data of length d for predicting the output of length q . This characteristic aligns well with our method, making it a promising fit for enhancing forecasting accuracy.

Our approach. We propose a novel method for MTSF by modifying the loss function to incorporate both individual feature transformations f t and a shared transformation f 0 . Each univariate-specific transformation f t is designed to capture the unique dynamics of its respective feature, while f 0 serves as a common transformation applied across all features to capture underlying patterns shared among them. We consider a neural network f with inputs X = [ X (1) , X (2) , . . . , X ( T ) ] , where T is the number of channels and X ( t ) ∈ R n × d . For a univariate model without MTL regularization, we predict Y = [ Y (1) , Y (2) , . . . , Y ( T ) ] = [ f 1 ( X (1) ) , . . . , f T ( X ( T ) )] and Y ( t ) ∈ R n × q . We compare these models with their corresponding versions that include MTL regularization, formulated as: f t MTL ( X ( t ) ) = f t ( X ( t ) ) + f 0 ( Y ) with f t : R n × d → R n × q and f 0 : R n × qT → R n × q . We define our regularized loss as follows:

where Y ( t ) are the true predictions, f t represents the univariate model for each channel t , and λ is our regularization parameter, for which we have established a closed form in the case of linear f t . f 0 serves a role equivalent to W 0 , which was defined in our theoretical study and allows for the regularization of the common part. This component can be added at the top of a univariate model. The parameters γ t enable the regularization of the specialized parts f t .

In our setup, f t is computed in a similar way as in the model without regularization and f 0 is computed by first flattening the concatenation of the predictions of X ( t ) , then applying a linear projection leveraging common multivariate information before reshaping. The loss function is specifically designed to balance fitting the multivariate series using f 0 and the specific channels using f t . This

approach enhances the model's generalization across various forecasting horizons and datasets. More details on our regularized loss function can be found in Appendix G.2

Results. We present experimental results on different forecasting horizons, using common benchmark MTSF datasets, the characteristics of which are outlined in the Appendix G.3. Our models include PatchTST [32], known to be on par with state-of-the-art in MTSF while being a univariate model, a univariate DLinear version called DLinearU compared to its multivariate counterpart DLinearM [53], and a univariate Transformer [19] with temporal-wise attention compared to the multivariate state-of-the-art models SAMformer [19] and iTransformer [24] . Table 1 provides a detailed comparison of the test mean squared errors (MSE) for different MTSF models, emphasizing the impact of MTL regularization. Models with MTL regularization are compared to their versions without regularization, as well as SAMformer and iTransformer . All the experiments can be found in Appendix G.4.

Adding MTL regularization improves the performance of PatchTST , DLinearU , and Transformer in most cases. When compared to state-of-the-art multivariate models, the MTL-regularized models are often competitive. SAMformer is outperformed by at least one MTL-regularized method per horizon and dataset, except for ETTh1 with horizons of 336 and 720. iTransformer is consistently outperformed by at least one MTL-regularized methods regardless of the dataset and horizon.

The best performing methods are PatchTST and DLinearU with MTL regularization. These models not only outperform their non-regularized counterparts, often significantly as shown by Student's t-tests with a p-value of 0.05, but also surpass state-of-the-art multivariate models like SAMformer and Transformer . This superior performance is indicated by the bold values in the table.

Finally, MTL regularization enhances the performance of univariate models, making them often competitive with state-of-the-art multivariate methods like SAMformer and iTransformer . This approach seems to better captures shared dynamics among tasks, leading to more accurate forecasts.

6 Conclusions and Future Works

In this article, we have explored linear multi-task learning by deriving a closed-form solution for an optimization problem that capitalizes on the wealth of information available across multiple tasks. Leveraging Random Matrix Theory, we have been able to obtain the asymptotic training and testing risks, and have proposed several insights into high-dimensional multi-task learning regression. Our theoretical analysis, though based on a simplified model, has been effectively applied to multi-task regression and multivariate forecasting, using both synthetic and real-world datasets. We believe that our work lays a solid foundation for future research, paving the way for using random matrix theory with more complex models, such as deep neural networks, within the multi-task learning framework.

References

Ando, R. K., Zhang, T., and Bartlett, P. (2005). A framework for learning predictive structures from multiple tasks and unlabeled data. Journal of machine learning research , 6(11).

Bai, Z. and Silverstein, J. W. (2010). Spectral Analysis of Large Dimensional Random Matrices . Springer Series in Statistics. Springer New York, NY.

Box, G. E. P. and Jenkins, G. (1990). Time Series Analysis, Forecasting and Control . Holden-Day, Inc., USA.

Box, G. E. P., Jenkins, G. M., and MacGregor, J. F. (1974). Some Recent Advances in Forecasting and Control. Journal of the Royal Statistical Society Series C , 23(2):158-179.

Caruana, R. (1997). Multitask learning. Machine learning , 28:41-75.

ˇ Cepulionis, P. and Lukoševiˇ ci¯ ut˙ e, K. (2016). Electrocardiogram time series forecasting and optimization using ant colony optimization algorithm. Mathematical Models in Engineering , 2(1):69-77.

Chen, R. and Tao, M. (2021). Data-driven prediction of general hamiltonian dynamics via learning exactly-symplectic maps. In Meila, M. and Zhang, T., editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 1717-1727. PMLR.

Chen, S.-A., Li, C.-L., Arik, S. O., Yoder, N. C., and Pfister, T. (2023). TSMixer: An all-MLP architecture for time series forecasting. Transactions on Machine Learning Research .

Clarté, L., Loureiro, B., Krzakala, F., and Zdeborová, L. (2023). Theoretical characterization of uncertainty in high-dimensional linear classification. Machine Learning: Science and Technology , 4(2):025029.

Couillet, R. and Debbah, M. (2011). Random matrix methods for wireless communications . Cambridge University Press.

Couillet, R. and Liao, Z. (2022). Random matrix methods for machine learning . Cambridge University Press.

Dobriban, E. and Wager, S. (2018). High-dimensional asymptotics of prediction: Ridge regression and classification. The Annals of Statistics , 46(1):247-279.

Erd˝ os, L. and Yau, H.-T. (2017). A Dynamical Approach to Random Matrix Theory , volume 28 of Courant Lecture Notes . American Mathematical Society.

Feofanov, V., Tiomoko, M., and Virmaux, A. (2023). Random matrix analysis to balance between supervised and unsupervised learning under the low density separation assumption. In Proceedings of the 40th International Conference on Machine Learning , pages 10008-10033.

Gerbelot, C., Abbara, A., and Krzakala, F. (2022). Asymptotic errors for teacher-student convex generalized linear models (or: How to prove kabashima's replica formula). IEEE Transactions on Information Theory , 69(3):1824-1852.

Hachem, W., Loubaton, P., and Najim, J. (2007). Deterministic equivalents for certain functionals of large random matrices.

Hu, Y., Li, M., Lu, Q., Weng, H., Wang, J., Zekavat, S. M., Yu, Z., Li, B., Gu, J., Muchnik, S., et al. (2019). A statistical framework for cross-tissue transcriptome-wide association analysis. Nature genetics , 51(3):568-576.

Ilbert, R., Hoang, T. V ., and Zhang, Z. (2024a). Data augmentation for multivariate time series classification: An experimental study. In 2024 IEEE 40th International Conference on Data Engineering Workshops (ICDEW) , pages 128-139.

Ilbert, R., Odonnat, A., Feofanov, V., Virmaux, A., Paolo, G., Palpanas, T., and Redko, I. (2024b). Unlocking the potential of transformers in time series forecasting with sharpness-aware minimization and channel-wise attention.

Ilbert, R., Tiomoko, M., Louart, C., Feofanov, V ., Palpanas, T., and Redko, I. (2024c). Enhancing multivariate time series forecasting via multi-task learning and random matrix theory. In NeurIPS Workshop on Time Series in the Age of Large Models .

Kim, T., Kim, J., Tae, Y., Park, C., Choi, J.-H., and Choo, J. (2021). Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations .

Kingma, D. and Ba, J. (2015). Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR) , San Diega, CA, USA.

Li, S., Cai, T. T., and Li, H. (2022). Transfer learning for high-dimensional linear regression: Prediction, estimation and minimax optimality. Journal of the Royal Statistical Society Series B: Statistical Methodology , 84(1):149-173.

Liu, Y., Hu, T., Zhang, H., Wu, H., Wang, S., Ma, L., and Long, M. (2024). itransformer: Inverted transformers are effective for time series forecasting. In The Twelfth International Conference on Learning Representations .

Louart, C. and Couillet, R. (2021). Spectral properties of sample covariance matrices arising from random matrices with independent non identically distributed columns. arXiv preprint arXiv:2109.02644 .

Lounici, K., Pontil, M., Tsybakov, A. B., and Van De Geer, S. (2009). Taking advantage of sparsity in multi-task learning. arXiv preprint arXiv:0903.1468 .

Max Planck Institute (n.d.). Weather dataset.

Mei, S., Fei, W., and Zhou, S. (2011). Gene ontology based transfer learning for protein subcellular localization. BMC bioinformatics , 12:1-12.

Mousavi Kalan, M., Fabian, Z., Avestimehr, S., and Soltanolkotabi, M. (2020). Minimax lower bounds for transfer learning with linear and one-hidden layer neural networks. Advances in Neural Information Processing Systems , 33:1959-1969.

Nguyen, M.-T. and Couillet, R. (2023). Asymptotic bayes risk of semi-supervised multitask learning on gaussian mixture. In International Conference on Artificial Intelligence and Statistics , pages 5063-5078. PMLR.

Nica, A. and Speicher, R. (2006). Lectures on the Combinatorics of Free Probability . London Mathematical Society Lecture Note Series. Cambridge University Press.

Nie, Y., Nguyen, N. H., Sinthong, P., and Kalagnanam, J. (2023). A time series is worth 64 words: Long-term forecasting with transformers. In The Eleventh International Conference on Learning Representations .

Niu, S., Liu, Y., Wang, J., and Song, H. (2020). A decade survey of transfer learning (20102020). IEEE Transactions on Artificial Intelligence , 1(2):151-166.

Potters, M., Bouchaud, J.-P., and Laloux, L. (2005). Financial applications of random matrix theory: Old laces and new pieces. Acta Physica Polonica B , 36(9):2767.

Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 21:1-67.

Romera-Paredes, B., Aung, H., Bianchi-Berthouze, N., and Pontil, M. (2013). Multilinear multitask learning. In International Conference on Machine Learning , pages 1444-1452. PMLR.

Ruder, S., Peters, M. E., Swayamdipta, S., and Wolf, T. (2019). Evolution of transfer learning in natural language processing. arXiv preprint arXiv:1910.07370 .

Salinas, D., Flunkert, V., Gasthaus, J., and Januschowski, T. (2020). Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting , 36(3):1181-1191.

Seddik, M. E. A., Louart, C., Tamaazousti, M., and Couillet, R. (2020). Random matrix theory proves that deep learning representations of gan-data behave as gaussian mixtures. In International Conference on Machine Learning .

Sen, R., Yu, H.-F., and Dhillon, I. (2019). Think globally, act locally: a deep neural network approach to high-dimensional time series forecasting. In Proceedings of the 33rd International Conference on Neural Information Processing Systems , Red Hook, NY, USA. Curran Associates Inc.

Shao, L. (2015). Transfer learning for visual categorization: a survey. IEEE Transactions on Neural Networks and Learning Systems , 26(5):1019-1034.

Shin, H.-C., Roth, H. R., Gao, M., Lu, L., Xu, Z., Nogues, I., Yao, J., Mollura, D., and Summers, R. M. (2016). Deep convolutional neural networks for computer-aided detection: Cnn architectures, dataset characteristics and transfer learning. IEEE transactions on medical imaging , 35(5):1285-1298.

Sifaou, H., Kammoun, A., and Alouini, M.-S. (2021). A precise performance analysis of support vector regression. In International Conference on Machine Learning , pages 9671-9680. PMLR.

Sonkavde, G., Dharrao, D. S., Bongale, A. M., Deokate, S. T., Doreswamy, D., and Bhat, S. K. (2023). Forecasting stock market prices using machine learning and deep learning models: A systematic review, performance analysis and discussion of implications. International Journal of Financial Studies , 11(3).

Sorjamaa, A., Hao, J., Reyhani, N., Ji, Y ., and Lendasse, A. (2007). Methodology for long-term prediction of time series. Neurocomputing , 70(16):2861-2869. Neural Network Applications in Electrical Engineering Selected papers from the 3rd International Work-Conference on Artificial Neural Networks (IWANN 2005).

Tao, T. (2012). Topics in Random Matrix Theory , volume 132 of Graduate Studies in Mathematics . American Mathematical Society.

Tiomoko, M., Couillet, R., and Pascal, F. (2023). PCA-based multi-task learning: a random matrix approach. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J., editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 34280-34300. PMLR.

Tiomoko, M., Couillet, R., and Tiomoko, H. (2020). Large dimensional analysis and improvement of multi task learning.

UCI (n.d.). Electricity dataset.

Wainwright, M. J. (2019). High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge university press.

Wu, H., Xu, J., Wang, J., and Long, M. (2021). Autoformer: Decomposition transformers with Auto-Correlation for long-term series forecasting. In Advances in Neural Information Processing Systems .

Yang, F., Zhang, H. R., Wu, S., Ré, C., and Su, W. J. (2023). Precise high-dimensional asymptotics for quantifying heterogeneous transfers.

Zeng, A., Chen, M., Zhang, L., and Xu, Q. (2023). Are transformers effective for time series forecasting? In Proceedings of the AAAI Conference on Artificial Intelligence .

Zhang, H., Zheng, Y., and Qi, D. (2019). Multi-task learning for time series forecasting using neural networks. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 1305-1313. ACM.

Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., and Zhang, W. (2021). Informer: Beyond efficient transformer for long sequence time-series forecasting. In The Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Virtual Conference , volume 35, pages 11106-11115. AAAI Press.

Zhou, T., Ma, Z., Wen, Q., Wang, X., Sun, L., and Jin, R. (2022). FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In Proc. 39th International Conference on Machine Learning (ICML 2022) .

Appendix

Roadmap. This appendix provides the technical details omitted in the main paper. It starts with an overview of the setup considered in the paper in Section A . Section B offers a detail computation for ˆ W t and ˆ W 0 . Section C contains a proof of Lemma 1. Section D explains the theoretical steps for deriving the training and test risks, as well as the deterministic equivalents. Section E discusses the technical tools used to derive the main intuitions presented by the theory. Section F focuses on the derivation of the estimations of the main quantities involved in the training and test risks. Section G complements the experimental study by presenting additional experiments. Finally, Section H deals with the limitations of our approach in a non-linear setting.

Table of Contents

A Multi-task Learning setup

A.1 On the zero-mean assumption

We would like to note that we are assuming that both the noise ε and the feature x ( t ) i have zero mean. This is a common assumption in many statistical models and it simplifies the analysis. However, this assumption is not restrictive. In practice, if the data or the response variable are not centered, we can always preprocess the data by subtracting the mean. This preprocessing step brings us back to the zero-mean setting that we consider in our theoretical analysis.

A.2 On the Assumption 1

Data are concentrated random vectors, meaning high-dimensional data maintain stable under complex (Lipschitz) transformations. The strong performance of neural networks on tasks like image recognition and NLP suggests that these models produce stable predictions. As Lipschitz transformations, they maintain controlled distances between inputs, ensuring stability. Recent studies have demonstrated and experimentally confirmed that both real-world data and synthetically generated data using GANs exhibit concentration properties, supporting this assumption. This makes our assumption more realistic than traditional Gaussian assumptions, as it does not rely on specific hypotheses about the shape of the data distribution, but rather on the stability of statistical properties after transformation. Consequently, analyzing a framework of concentrated random vectors is more theoretically challenging than using Gaussian assumptions and represents a key novelty of our theory.

A.3 On the Assumption 2

The dimension d is of the same order of magnitude as the sample size n . This joint growth captures data complexity better than assuming a fixed feature size with increasing samples, which can oversimplify models. Our theory works for fixed d and n , unbiased by specific parameter choices. The accuracy of empirical predictions depends on both d and n , with variance scaling as O ( 1 √ dn ) . Larger d and n reduce variance, making empirical results more reliable and closer to theoretical values. Conversely, smaller d and n increase variance, affecting single predictions. However, this scaling is still better than n growing indefinitely with fixed d , where variance scales as O ( 1 √ n ) , leading to increased bias. Implication of the assumptions. In Section 5.2, Figure 3, the Appliance Energy dataset illustrates our assumptions' realism. Despite the moderate sample size of 42 samples with 142 dimensions and non-synthetic data, the theoretical curve fits the empirical predictions well. Additionally, with synthetic data ( d = 100 , n = 100 ), the theory matches the empirical curve across various hyperparameters (Figure 2), showcasing the predictive power of Random Matrix Theory.

B Minimization Problem

B.1 Computation of ˆ W t and ˆ W 0

The proposed multi task regression finds ˆ W = [ ˆ W ⊤ 1 , . . . , ˆ W ⊤ k ] ⊤ ∈ R dT × q which solves the following optimization problem using the additional assumption of relatedness between the tasks ( W t = W 0 + V t for all tasks t ):

where

The Lagrangian introducing the lagrangian parameters for each task t , α t ∈ R n t × q reads as

Differentiating with respect to the unknown variables ˆ W 0 , ˆ V t , ξ t , α t and b t , we get the following system of equation

Plugging the expression of ˆ W 0 , ˆ V t and ξ t into the expression of Y ( t ) gives

which can be rewritten as

̸

With Y = [ Y (1) ⊤ , . . . , Y ( T ) ⊤ ] ⊤ ∈ R n × q , α = [ α ⊤ 1 , . . . , α ⊤ k ] ⊤ ∈ R n × q , Z = ∑ T t =1 e [ T ] t e [ T ] t ⊤ ⊗ X ( t ) ∈ R Td × n , this system of equations can be written under the following compact matrix form:

Solving for α then gives:

Moreover, using ˆ W t = ˆ W 0 + ˆ V t , the expression of W t becomes:

C Lemma 1 and proof with Random Matrix Theory

C.1 Lemma 1

Lemma 1 (Deterministic equivalents for ˜ Q , ˜ QM ˜ Q and Q 2 for any M ∈ R n × n ) . Under the concentrated random vector assumption for each feature vector x ( t ) i and under the growth rate assumption (Assumption 2), for any deterministic M ∈ R n × n , we have the following convergence:

where ¯ ˜ Q 2 , ¯ ˜ Q and ¯ Q 2 are defined as follows

where

C.2 Deterministic equivalent of the resolvent ˜ Q

̸

The evaluation of the expectation of linear forms on ˜ Q and ˜ Q 2 can be found in the literature. To find a result that meets exactly our setting, we will cite [25] that is a bit more general since it treats cases where E [ x ( t ) i ] = 0 for t ∈ [ T ] and i ∈ [ n t ] . Unlike the main paper, and to be more general, the study presented below is 'quasi asymptotic' meaning that the results are true for finite value of d, n . Let us first rewrite the general required hypotheses, adapting them to our setting. For that purpose, we consider in the rest of this paper a certain asymptotic I ⊂ { ( d, n ) , d ∈ N , n ∈ N } = N 2 satisfying:

such that n and d can tend to ∞ but with some constraint that is given in the first item of Assumption 3 below. Given two sequences ( a d,n ) d,n ∈ I , ( b d,n ) d,n ∈ I > 0 , the notation a d,n ≤ O ( b d,n ) (or a ≤ O ( b ) ) means that there exists a constant C > 0 such that for all ( d, n ) ∈ I , a d,n ≤ Cb d,n .

Assumption 3. There exists some constants C, c > 0 independent such that:

n ≤ O ( d )

Z = ( z 1 , . . . , z n ) ∈ R Td × n has independent columns

for any ( d, n ) ∈ I , and any f : R Td × n → R 1 -Lipschitz for the euclidean norm:

∀ i ∈ { n, ∃ d ∈ N , ( d, n ) ∈ I } : ∥ E [ z i ] ∥ ≤ O (1) .

Theorem 2 ([25], Theorem 0.9.) . Given T ∈ N , Z ∈ R Td × n and two deterministic A ∈ R Td × Td , we note ˜ Q ≡ ( 1 Td A 1 2 ZZ ⊤ A 1 2 + I Td ) -1 . If Z satisfies Assumption 3 and M ∈ R Td × Td is a deterministic matrix satisfying ∥ M ∥ F ≤ 1 , one has the concentration:

where S = ( S 1 , . . . , S n ) = ( E [ z 1 z ⊤ 1 ] , . . . , E [ z n z ⊤ n ]) , for δ ∈ R n , ¯ ˜ Q δ is defined as:

and δ ( S ) is the unique solution to the system of equations:

We end this subsection with some results that will be useful for next subsection on the estimation of bilinear forms on ¯ ˜ Q .

Lemma 2 ([25], Lemmas 4.2, 4.6) . Under the setting of Theorem 2, given a deterministic vector u ∈ R Td such that ∥ u ∥ ≤ O (1) and two deterministic matrices U , V such that ∥ U ∥ , ∥ V ∥ ≤ O (1) and a power r > 0 , r ≤ O (1) :

C.3 Deterministic equivalent of bilinear forms of the resolvent

To simplify the expression of the following theorem, we take A = I Td . One can replace Z with A 1 2 Z to retrieve the result necessary for the main paper.

Theorem 3. Under the setting of Theorem 2, with A = I Td , one can estimate for any deterministic matrices U , V ∈ R Td such that ∥ U ∥ , ∥ V ∥ ≤ O (1) and any deterministic vector u , v ∈ R Td such that ∥ u ∥ , ∥ v ∥ ≤ 1 , if one notes B = 1 Td V or B = uv ⊤ , one can estimate:

where we noted:

If there exist T < n dinstinct matrices C 1 , . . . , C T such that:

and if we denote ∀ t ∈ [ T ] n t = # { i ∈ [ n ] | S i = C t } and:

the result of Theorem 3 rewrites:

Proof. Given i ∈ [ n ] , let us note Z -i = ( z 1 , . . . , z i -1 , 0 , z i +1 , . . . , z n ) and ˜ Q -i = ( 1 Td Z -i Z ⊤ -i + I Td ) -1 , then we have the identity:

Given u , v ∈ R Td , such that ∥ u ∥ , ∥ v ∥ ≤ 1 , let us express:

(12)

First, given i ∈ [ n ] , let us estimate thanks to (10):

Hölder inequality combined with Lemma 2 allows us to bound:

Second, one can also estimate thanks to Lemma 10:

again thanks to Hölder inequality combined with Lemma 2 that allow us to bound:

The independence between z i and ˜ Q -i (and ¯ ˜ Q ) then allow us to deduce (again with formula (10)):

Let us inject (13) and (14) in (11) to obtain (again with an application of Hölder inequality and Lemma 2 that we do not detail this time):

Putting all the estimations together, one finally obtains:

One then see that if we introduce for any V ∈ R n × n the block matrices:

then, if ∥ V ∥ ≤ O (1) , multiplying (15) with V and taking the trace leads to:

Now, taking U = S 1 1+ δ 1 , . . . , S n 1+ δ n , one gets the vectorial equation:

When ( I Td -1 Td Ψ) is invertible, one gets θ ( V ) = ( I Td -1 Td Ψ) -1 Ψ( V ) + O ( 1 √ d ) , and combining with (16), one finally obtains:

C.4 Estimation of the deterministic equivalent of Q 2

Theorem 4. Under the setting of Theorem 3, one can estimate:

Proof. The justifications are generally the same as in the proof of Theorem 3, we will thus allow ourselves to be quicker in this proof.

Using the definition of Q = ( Z ⊤ AZ Td + I n ) -1 , we have that

and one can then let appear ˜ Q thanks to the relation:

with, ∀ i ∈ [ n ] :

that finally gives us:

One can then express:

̸

Given i, j ∈ [ n ] , i = j , let us first estimate (thanks to Hölder inequality and Lemma 2):

since E [ z i ] = E [ z j ] = 0 . Now, we consider the case j = i to get:

̸

As before, we know that 1 Td E [ z ⊤ i ˜ Qz j ] ≤ O ( 1 √ d ) if i = j . Considering i ∈ [ n ] , we thus are left to estimate:

D Risk Estimation (Proof of Theorem 1)

D.1 Test Risk

The expected value of the MSE of the test data x ∈ R T × Td concatenating the feature vector of all the tasks with the corresponding response variable y ∈ R T × Tq reads as

D.2 Train Risk

In this section, we derive the asymptotic risk for the training data.

Theorem 5 (Asymptotic training risk) . Assuming that the training data vectors x ( t ) i and the test data vectors x ( t ) are concentrated random vectors, and given the growth rate assumption (Assumption 2), it follows that:

Proof. We aim in this setting of regression, to compute the asymptotic theoretical training risk given by:

Using the definition of Q = ( Z ⊤ AZ Td + I Td ) -1 , we have that

Plugging back into the expression of the training risk then leads to

Using the definition of the linear generative model and in particular Y = Z ⊤ W √ Td + ε , we get

To simplify this expression, we will introduced the so-called 'coresolvent' defined as:

Employing the elementary relation A 1 2 ZQ = ˜ QA 1 2 Z , one obtains:

Therefore we further get

Using deterministic equivalents in Lemma 1, the training risk then leads to

E Interpretation and insights of the theoretical analysis

E.1 Analysis of the test risk

We recall the test risk as

The test risk is composed of a signal term of a signal term S = W ⊤ A -1 2 ¯ ˜ Q 2 ( A ) A -1 2 W Td and a noise term N = tr ( Σ N ¯ Q 2 ) Td .

E.2 Interpretation of the signal term

Let's denote by ¯ Σ = ∑ T t =1 n t d t Td (1+ δ t ) 2 Σ ( t ) and ˜ Σ = ∑ T t =1 c 0 1+ δ t Σ ( t ) . The signal term reads as

Using the following identity,

This finally leads to

̸

The matrix H = ( A ˜ Σ + I ) -1 ( I + ¯ Σ ) ( A ˜ Σ + I ) -1 is responsible to amplifying the signal W ⊤ W in order to let the test risk to decrease more or less. It is is decreasing as function of the number of samples in the tasks n t . Furthermore it is composed of two terms (from the independent training W ⊤ t W ) and the cross term W ⊤ t W v for t = v . Both terms decreases as function of the number of samples n t , smaller values of γ t and increasing value of λ . The cross term depends on the matrix Σ -1 t Σ v which materializes the covariate shift between the tasks. More specifically, if the features are aligned Σ -1 t Σ v = I and the cross term is maximal while for bigger Fisher distance between the covariance of the tasks, the correlation is not favorable for multi task learning. To be more specific the off-diagonal term of H are responsible for the cross term therefore for the multi tasks and the diagonal elements are responsible for the independent terms.

To analyze more the element of H , let's consider the case where Σ ( t ) = I and γ t = γ . In this case the diagonal and non diagonal elements D IL and C MTL are respectively given by

Both function are decreasing function of λ , 1 /γ and c 0 .

E.3 Interpretation and insights of the noise terms

We recall the definition of the noise term N as

Now at the difference of the signal term there are no cross terms due to the independence between the noise of the different tasks. In this case on the diagonal elements of ( A -1 + Σ ) -1 matters. This diagonal term is increasing for an increasing value of the sample size, the value of λ . Therefore this term is responsible for the negative transfer. In the specific case where Σ ( t ) = I d and γ t = γ for all task t , the diagonal terms read as

E.4 Optimal Lambda

The test risk in the particular of identity covariance matrix can be rewritten as

Deriving R ∞ test with respect to λ leads after some algebraic calculus to

where the signal noise ratio is composed of the independent signal to noise ratio and the cross signal to noise ratio SNR = ∥ W 1 ∥ 2 2 + W 2 ∥ 2 2 tr Σ n + W ⊤ 1 W 2 tr Σ n

F Theoretical Estimations

F.1 Estimation of the training and test risk

The different theorems depends on the ground truth W that needs to be estimated through ˆ W .

To estimate the test risk, one needs to estimate functionals of the form W ⊤ M ˆ W and ε ⊤ M ε for any matrix M . Using the expression of W = AZQY , we start computing ˆ W ⊤ M ˆ W

Using the generative model for Y = Z ⊤ W √ Td + ε , we obtain

Employing the elementary relation A 1 2 ZQ = ˜ QA 1 2 Z , one obtains:

Using the deterministic equivalent of Lemma 1, we obtain

where We define the mapping κ : R Td × Td → R q × q as follows

F.2 Estimation of the noise covariance

The estimation of the noise covariance remains a technical challenge in this process. However, when the noise covariance is isotropic, it is sufficient to estimate only the noise variance. By observing that

we can estimate the noise level from the training risk evaluated at large γ and λ = 0 .

G Multivariate Time Series Forecasting

A version of this work with a stronger focus on time series applications can be found at [20].

G.1 Related Work

Multivariate Time Series Forecasting. Previous work has explored multivariate time series methods to enrich task representations and enhance diversity across time series applications [18].

Despite these advancements, random matrix theory, widely applied in multi-task learning, remains underutilized in multivariate time series contexts, particularly for multivariate time series forecasting (MTSF) [52, 54]. MTSF is common in applications like medical data [6], electricity consumption [49], temperatures [27], and stock prices [44]. Various methods, from classical tools [7, 45] and statistical approaches like ARIMA [3, 4] to deep learning techniques [8, 19, 32, 38, 40, 51, 53, 55, 56], have been developed for this task. Some studies prefer univariate models for multivariate forecasting [32], while others introduce channel-wise attention mechanisms [19]. We aim to enhance a univariate model by integrating a regularization technique from random matrix theory and multi-task learning.

G.2 Architecture and Training Parameters

Architectures without MTL regularization. We follow Chen et al. [8], Nie et al. [32], and to ensure a fair comparison of baselines, we apply the reversible instance normalization ( RevIN ) of Kim et al. [21]. All the baselines presented here are univariate i.e. no operation is performed by the network along the channel dimension. For the PatchTST baseline [32], we used the official implementation than can be found on Github. The network used in Transformer follows the one in [19], using a single layer Transformer with one head of attention, while RevIN normalization and denormalization are applied respectively before and after the neural network function. The dimension of the model is d m = 16 and remains the same in all our experiments. DLinearU is a single linear layer applied for each channel to directly project the subsequence of historical length into the forecasted subsequence of prediction length. It is the univariate extension of the multivariate DLinear, DLinearM , used in Zeng et al. [53]. The implementation of SAMformer can be found here, and for the iTransformer architecture here. These two multivariate models serve as baseline comparisons. We reported the results found in [19] and [24]. For all of our experiments, we train our baselines PatchTST , DLinearU and Transformer with the Adam optimizer [22], a batch size of 32 for the ETT datasets and 256 for the Weather dataset , and the learning rates summarized in Table 2.

Architectures with MTL Regularization. We implemented the univariate PatchTST , DLinearU , and Transformer baselines with MTL regularization. Initially, we scale the inputs twice using RevIN normalization. The first scaling is applied to the univariate components, and the second scaling is applied to the multivariate components. For each channel, we then apply our model without MTL regularization. The outputs are concatenated along the channel dimension, and this concatenation is flattened to form a matrix of shape (batch size, q × T ), where q is the prediction horizon and T is the number of channels. We then learn a square matrix W of shape ( q × T ) × ( q × T ) for projection and reshape the result to obtain an output of shape (batch size, q , T ). This method can be applied on top of any univariate model. Our regularized loss has been introduced in 5.4.

Training parameters. The training/validation/test split is 12 / 4 / 4 months on the ETT datasets and 70% / 20% / 10% on the Weather dataset. We use a look-back window d = 336 for PatchTST and d = 512 for DLinearU and Transformer , using a sliding window with stride 1 to create the sequences. The training loss is the MSE. Training is performed during 100 epochs and we use early stopping with a patience of 5 epochs. For each dataset, baselines, and prediction horizon H ∈ { 96 , 192 , 336 , 720 } , each experiment is run 3 times with different seeds, and we display the average of the test MSE over the 3 trials in Table 1.

G.3 Datasets

We conduct our experiments on 3 publicly available datasets of real-world time series, widely used for multivariate long-term forecasting [8, 32, 51]. The 2 Electricity Transformer Temperature datasets ETTh1, and ETTh2 [55] contain the time series collected by electricity transformers from July 2016 to July 2018. Whenever possible, we refer to this set of 2 datasets as ETT. Weather [27] contains the time series of meteorological information recorded by 21 weather indicators in 2020. It should be noted Weather is large-scale datasets. The ETT datasets can be downloaded here while the Weather

dataset can be downloaded here. Table 3 sums up the characteristics of the datasets used in our experiments.

G.4 Additional Experiments.

10 5

10 4

10 3

10 2

10 1

10 0

Lambda

0.380

0.400

0.420

0.440

0.460

0.480

0.500

0.520

0.540

Average MSE

= 0.0

= 0.001

= 0.01

= 0.1

= 0.15

(a) Dataset ETTh1, Horizon 96

10 5

10 4

10 3

10 2

10 1

10 0

Lambda

0.290

0.295

0.300

0.305

0.310

0.315

0.320

0.325

Average MSE

= 0.0

= 0.001

= 0.01

= 0.1

= 0.15

(b) Dataset ETTh2, Horizon 96

10 5

10 4

10 3

10 2

10 1

10 0

Lambda

0.150

0.160

0.170

0.180

0.190

0.200

0.210

Average MSE

= 0.0

= 0.001

= 0.01

= 0.1

= 0.15

(c) Dataset Weather, Horizon 96

10 5

10 4

10 3

10 2

10 1

10 0

Lambda

0.420

0.440

0.460

0.480

0.500

0.520

0.540

0.560

Average MSE

(d) Dataset ETTh1, Horizon 192

10 5

10 4

10 3

10 2

10 1

10 0

Lambda

0.345

0.350

0.355

0.360

0.365

0.370

Average MSE

(e) Dataset ETTh2, Horizon 192

10 5

10 4

10 3

10 2

10 1

10 0

Lambda

0.190

0.200

0.210

0.220

0.230

0.240

Average MSE

(f) Dataset Weather, Horizon 192

10 5

10 4

10 3

10 2

10 1

10 0

Lambda

0.440

0.460

0.480

0.500

0.520

0.540

Average MSE

(g) Dataset ETTh1, Horizon 336

10 5

10 4

10 3

10 2

10 1

10 0

Lambda

0.333

0.335

0.338

0.340

0.343

0.345

0.348

0.350

Average MSE

(h) Dataset ETTh2, Horizon 336

10 5

10 4

10 3

10 2

10 1

10 0

Lambda

0.245

0.250

0.255

0.260

0.265

0.270

0.275

0.280

Average MSE

(i) Dataset Weather, Horizon 336

10 5

10 4

10 3

10 2

10 1

10 0

Lambda

0.440

0.460

0.480

0.500

0.520

0.540

0.560

0.580

0.600

Average MSE

10 5

10 4

10 3

10 2

10 1

10 0

Lambda

0.385

0.390

0.395

0.400

0.405

0.410

0.415

Average MSE

10 5

10 4

10 3

10 2

10 1

10 0

Lambda

0.315

0.320

0.325

0.330

0.335

0.340

0.345

0.350

Average MSE

(j) Dataset ETTh1, Horizon 720

(k) Dataset ETTh2, Horizon 720

(l) Dataset Weather, Horizon 720

10 5

10 4

10 3

10 2

10 1

10 0

Lambda

0.370

0.375

0.380

0.385

0.390

0.395

0.400

0.405

Average MSE

= 0.0

= 0.001

= 0.01

= 0.1

= 0.15

DLinearU Baseline

(a) Dataset ETTh1, Horizon 96

10 5

10 4

10 3

10 2

10 1

10 0

Lambda

0.270

0.280

0.290

0.300

0.310

0.320

0.330

Average MSE

= 0.0

= 0.001

= 0.01

= 0.1

= 0.15

DLinearU Baseline

(b) Dataset ETTh2, Horizon 96

10 5

10 4

10 3

10 2

10 1

10 0

Lambda

0.150

0.175

0.200

0.225

0.250

0.275

0.300

0.325

Average MSE

= 0.0

= 0.001

= 0.01

= 0.1

= 0.15

DLinearU Baseline

(c) Dataset Weather, Horizon 96

10 5

10 4

10 3

10 2

10 1

10 0

Lambda

0.410

0.420

0.430

0.440

0.450

0.460

0.470

Average MSE

(d) Dataset ETTh1, Horizon 192

10 5

10 4

10 3

10 2

10 1

10 0

Lambda

0.300

0.320

0.340

0.360

0.380

0.400

Average MSE

(e) Dataset ETTh2, Horizon 192

10 5

10 4

10 3

10 2

10 1

10 0

Lambda

0.220

0.240

0.260

0.280

Average MSE

(f) Dataset Weather, Horizon 192

10 5

10 4

10 3

10 2

10 1

10 0

Lambda

0.430

0.440

0.450

0.460

0.470

0.480

0.490

0.500

Average MSE

(g) Dataset ETTh1, Horizon 336

10 5

10 4

10 3

10 2

10 1

10 0

Lambda

0.360

0.380

0.400

0.420

0.440

0.460

Average MSE

(h) Dataset ETTh2, Horizon 336

10 5

10 4

10 3

10 2

10 1

10 0

Lambda

0.260

0.280

0.300

0.320

0.340

Average MSE

(i) Dataset Weather, Horizon 336

10 5

10 4

10 3

10 2

10 1

10 0

Lambda

0.440

0.460

0.480

0.500

0.520

0.540

0.560

0.580

Average MSE

10 5

10 4

10 3

10 2

10 1

10 0

Lambda

0.400

0.425

0.450

0.475

0.500

0.525

0.550

0.575

Average MSE

10 5

10 4

10 3

10 2

10 1

10 0

Lambda

0.330

0.340

0.350

0.360

0.370

0.380

Average MSE

(j) Dataset ETTh1, Horizon 720

(k) Dataset ETTh2, Horizon 720

(l) Dataset Weather, Horizon 720

10 5

10 4

10 3

10 2

10 1

10 0

Lambda

0.380

0.400

0.420

0.440

0.460

0.480

0.500

0.520

Average MSE

= 0.0

= 0.001

= 0.01

= 0.1

= 0.15

Transformer Baseline

(a) Dataset ETTh1, Horizon 96

10 5

10 4

10 3

10 2

10 1

10 0

Lambda

0.280

0.300

0.320

0.340

Average MSE

= 0.0

= 0.001

= 0.01

= 0.1

= 0.15

Transformer Baseline

(b) Dataset ETTh2, Horizon 96

10 5

10 4

10 3

10 2

10 1

10 0

Lambda

0.160

0.180

0.200

0.220

0.240

0.260

0.280

0.300

Average MSE

= 0.0

= 0.001

= 0.01

= 0.1

= 0.15

Transformer Baseline

(c) Dataset Weather, Horizon 96

10 5

10 4

10 3

10 2

10 1

10 0

Lambda

0.410

0.420

0.430

0.440

0.450

0.460

0.470

0.480

Average MSE

(d) Dataset ETTh1, Horizon 192

10 5

10 4

10 3

10 2

10 1

10 0

Lambda

0.340

0.360

0.380

0.400

0.420

0.440

Average MSE

(e) Dataset ETTh2, Horizon 192

10 5

10 4

10 3

10 2

10 1

10 0

Lambda

0.200

0.220

0.240

0.260

0.280

0.300

0.320

0.340

Average MSE

(f) Dataset Weather, Horizon 192

10 5

10 4

10 3

10 2

10 1

10 0

Lambda

0.430

0.440

0.450

0.460

0.470

0.480

0.490

0.500

0.510

Average MSE

(g) Dataset ETTh1, Horizon 336

10 5

10 4

10 3

10 2

10 1

10 0

Lambda

0.370

0.380

0.390

0.400

0.410

0.420

Average MSE

(h) Dataset ETTh2, Horizon 336

10 5

10 4

10 3

10 2

10 1

10 0

Lambda

0.260

0.280

0.300

0.320

0.340

Average MSE

(i) Dataset Weather, Horizon 336

10 5

10 4

10 3

10 2

10 1

10 0

Lambda

0.460

0.480

0.500

0.520

0.540

0.560

Average MSE

10 5

10 4

10 3

10 2

10 1

10 0

Lambda

0.400

0.420

0.440

0.460

0.480

0.500

0.520

0.540

0.560

Average MSE

10 5

10 4

10 3

10 2

10 1

10 0

Lambda

0.330

0.340

0.350

0.360

0.370

0.380

0.390

Average MSE

(j) Dataset ETTh1, Horizon 720

(k) Dataset ETTh2, Horizon 720

(l) Dataset Weather, Horizon 720

H Limitations

While the study provides valuable insights through its theoretical analysis within a linear framework, it is important to acknowledge its limitations. The linear approach serves as a solid foundation for understanding more complex models, but its practical applications may be constrained. Linear models, though mathematically tractable and often easier to interpret, might not fully capture the intricacies and nonlinear relationships present in real-world data, especially in the context of multivariate time series forecasting.

To address this limitation, we decided to extend our algorithm's application to more complex models, specifically within the nonlinear setting of neural networks. This transition aims to evaluate whether the theoretical insights derived from the linear framework hold true empirically when applied to neural networks. As part of this endeavor, an optimal parameter lambda was selected by an oracle, leading to promising results, as detailed in Section 5.4. This oracle-based selection underscores the potential efficacy of our approach when appropriately tuned, even in more complex, nonlinear contexts.

It is important to note that the limitations are not related to the real-world data itself, as our setting performs well in the context of multi-task regression for real-world data, as shown in Section 5.2. The difficulty arises from transitioning from a linear to a nonlinear model. The results in Section 5.4 are particularly encouraging, demonstrating that our method can improve upon univariate baselines by regularizing with an optimal lambda, as indicated by our oracle. While the oracle provides an upper bound on performance, actual implementation would require robust methods for hyperparameter optimization in non-linear scenarios, which remains an open area for further research.

By expanding the scope of our theoretical framework to encompass nonlinear models, we pave the way for future work that could focus on the theoretical analysis of increasingly complex architectures

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The datasets used are open-source and all the implementation details are given to reproduce the experimental results. The code will be made available in case of acceptance.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: All details to reproduce experiments are given.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper analyzes a linear multi-task regression framework. It derives formulas for asymptotic train and test risk. The formulas provide insights in how the raw data covariances, singal-generating hyperplanes, noise levels, and size of data sets affect the risk. Motivated by the analysis on the linear framework, experiments on multivariate time series forecasting are performed to investigate the effectiveness of MTL regularization.

STRENGTHS:
- The paper is clearly written. It outlines main points and is easy to follow. 
- Throughout analyses, which use random matrix theory, on the multi-task linear regression framework are performed. Specifically, the asymptotic train and test risks are derived for analyzing the behavior of the framework.
- The analyses provide useful insights into the behavior of models.

WEAKNESSES:
- The analyses are performed on a simple linear model. They do not apply to general nonlinear models, which are much more common in real practice.
- There seems to be a disconnection between the theoretical analysis and the experimental results. The experiments seem to only highlight the effectiveness of MTL regularization and have no connection with the theoretical analysis.

QUESTIONS:
- Is there any connections between the theoretical analysis and the experiments? E.g. making use insights from the analysis to improve experimental results.
- Is splitting the linear operator into a local and global term a novelty of this paper?
- Similarly, is this paper the first work to apply this MTL framework (decompose prediction as a sum of global and local term) to nonlinear models?
- How are the regularization hyperparameter chosen in the experiments?
- In some of the results, incorporating MTL regularization worsen the performance. Why is that?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors analyse the problem of multi-task regression (for a linear model) under the assumption of concentrated random vectors.  The results obtained provide a comprehensive description of the performance of the model (including critically its generalisation performance).  The authors use this result for hyperparameter optimisation and show on real-world datasets that using the insights from their theoretical model they obtain state-of-the-art performance on multivariate time series forecasting.\n\nSTRENGTHS:\nI feel that this is a really nice achievement.  Computing generalisation results are notoriously difficult.  Such calculations are usually confined to rather simplistic models.  The model chosen is rather simplistic in being linear and make, what seems to me, a very strong assumption about the data.  Nevertheless, by choosing the appropriate setting (i.e. a multi-task scenario) and it appears that the concentration assumption seems to innocuous they come up with a useful result.  Given how hard it is to do meaningful theory in machine learning I certainly believe this paper deserves to be accepted.\n\nIt should also be noted, that the paper is technically accomplished.\n\nWEAKNESSES:\nThe paper will not be everyones taste.  The mathematics is not easy to follow, but that is often the nature of making a non-trivial technical contribution.\n\nIn the equation for $\\lambda^*$ immediately preceding section 4.4 there is a missing norm symbol.\n\nQUESTIONS:\nIs there an intuition on why the concentrated random vector assumption justified?  Is it that real data satisfies this assumption accurate, or is that the deviations from this assumption don't seem to be relevant?\n\nYou are tackling real world data using a relatively simple linear model and beating what I assume are sophisticated non-linear models.  I find this surprising.  Is there an explanation of why you do so well?  (Are the datasets you are using very simple--linear?,  are the number of examples small compared to the number of features so that more complex models will overfif?).\n\nLIMITATIONS:\nThis is fine.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors characterise the train and test risk of multi-task regression using random matrix theory. Assessment via Figures 2 and 3 shows a good match between theory and empirical. This motivates a regularised objective for learning mutivariate time-series.\n\nSTRENGTHS:\nThe theory contribution in section 4 is significant and original, and the assessment in sections 5.1 and 5.2 are significant.\n\nWEAKNESSES:\n*Three major weaknesses*\n\nA. There are at least two existing works on the theoretical error bounds for multi-task learning [Chai, NeurIPS 2009; Aston & Sollich, NeurIPS 2012]. While these are using Gaussian processes as a basis, comparison or remarks on the general effect of multi-task learning should be discussed with reference to two existing works. This is especially when all these are essentially linear-in-regressors models.\n\nB. The proof or at least outline of the proof of Theorem 1 *should* be in the main paper, given that this is *the* major contribution of the paper.\n\nC. It is totally unclear the relevance and contribution of the theory itself to section 5.3. Adding a cross-task regularization is already known to help (mostly).\n\n*Minor, but affects clarity*\n- The introduction to the paper is on multi-task multi-response/regression. The multi-response/regression part is not made clear.\n- Why is there a need to divide by $\\sqrt{Td}$ in (1)? Is it to make analysis easier.\n- There are two different uses of the letter $t$ in Assumption 1.\n- The setup in section 5.3 needs to relate back to the model in section 3.1 to make clearer their connections.\n\nQUESTIONS:\n1. The analysis of the noise term in section 4.2 says that increase in noise negatively affects transfer. However, it is precisely in the presence of  noise that borrowing statistical strength from other tasks is suppose to help. How do we reconcile this?\n\n2. In section 4.3, is it clear that $C_{MTL}$ is always positive, negative or no such conclusion can be made?\n\nLIMITATIONS:\nAuthors should include some technical limitations of their current work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
The authors derived theoretical insights into the train and test risks of the multi task regression loss using Random Matrix Theory.

STRENGTHS:
The closed form solution of the optimization parameter using RMT is novel. The authors set a trend in MTR to gain analytical expressions for optimization parameter and error risks.

WEAKNESSES:
1. The paper lacks clarity for instance on line 89, I am not sure what is meant by general optimization and other mathematical tools?

QUESTIONS:
1. What is $\gamma$ in (3)? 
2. Have you studied the implications of the assumptions made to use random matrix theory to derive the expressions of asymptotic train and test errors?
3. Shouldn't it be $\mathbf{Y}\in \mathbb{R}^{Tq\times n}$ in the equations after line 110?

LIMITATIONS:
Limitations are not discussed. My guess is that the assumptions made to be able to employ Random Matrix Theory need to be discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-15