Analysing Multi-Task Regression via Random Matrix Theory with Application to Time Series Forecasting
Authors: Romain Ilbert
Malik Tiomoko
Cosme Louart
Ambroise Odonnat
Vasilii Feofanov
Themis Palpanas
Ievgen Redko
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Random Matrix Theory ; Optimization ; Regularization ; Multi-task regression ; Multi-task learning ; Multivariate Time Series Forecasting
Primary area: learning_theory
TL;DR: This paper introduces a new multi-task regression framework using random matrix theory for improved performance estimation and presents a regularization-based optimization with empirical methods, enhancing univariate models.

Abstract: In this paper, we introduce a novel theoretical framework for multi-task regression, applying random matrix theory to provide precise performance estimations, under high-dimensional, non-Gaussian data distributions. We formulate a multi-task optimization problem as a regularization technique to enable single-task models to leverage multi-task learning information. We derive a closed-form solution for multi-task optimization in the context of linear models. Our analysis provides valuable insights by linking the multi-task learning performance to various model statistics such as raw data covariances, signal-generating hyperplanes, noise levels, as well as the size and number of datasets. We finally propose a consistent estimation of training and testing errors, thereby offering a robust foundation for hyperparameter optimization in multi-task regression scenarios. Experimental validations on both synthetic and real-world datasets in regression and multivariate time series forecasting demonstrate improvements on univariate models, incorporating our method into the training loss and thus leveraging multivariate information.

Analysing Multi-Task Regression via Random Matrix Theory with Application to Time Series Forecasting
Romain 1lb Malik Tiomoko' Cosme Louart
Ambroise Odonnat !:4
Vasilii Feofanov Themis Palpanas levgen Redko²
Huawei Noah's 's Ark Lab, Paris, France LIPADE, Paris Descartes University, Paris, France School of Data Science, The Chinese University of Hong Kong, Shenzhen, China Inria, Univ. Rennes 2, CNRS, IRISA
Abstract
In this paper, introduce novel theoretical framework for multi-task regression, applying random matrix theory to provide precise performance estimations, under nigh-dimensional, non-Gaussian data distributions. We formulate multi-task optimization problem as regularization technique to enable single-task models leverage multi-task learning information. We derive a closed-form solution for multi-task optimization in the context of linear models. Our analysis provides valuable insights by linking the multi-task learning performance various model statistics such as raw data covariances, els, as well as the size and number datasets. We finally propose a consistent estimation of training and testing errors, thereby offering robust foundation for hyperparameter optimization in multi-task regression scenarios. Experimental vali- dations on both synthetic and weal-world datasets in regression and multivariate time series forecasting demonstrate improvements on univariate models, incorporating our method into the training loss and thus leveraging multivariate information.
hyperplanes, noise lev-
Introduction
The principle of multi-task learning, which encompasses concepts such as "learning to learn" and "knowledge transfer," offers an efficient paradigm that mirrors human intelligence. Unlike traditional machine learning, which tackles problems on a task-specific basis with tailored algorithms, multi-task learning leverages shared information across various tasks to enhance overall performance, This approach not only improves accuracy but also addresses challenges related to insufficient data and data cleaning. By analyzing diverse and multimodal data and structuring representations multi-task learning can significantly boost general intelligence, similar to how humans integrate skills. This concept well-established [33] and has been successfully applied to a wide range of domains, such as computer vision [41], natural language processing [35, 37] and biology [17, 28, 42).
Our Method. We consider the multi-task regression framework [5] with T tasks with the input space ( Rd and the output space y)(t) C IR9 for t {l,..., T}. For each task t, we assume that we given n, training examples organized into the feature matrix (+) [x(") (t)
................................ xx/] E ddxx., and _!! Rox*, where x( E (t) represents the corresponding response matrix (t) = [y .....
the i-th feature vector of the t-th task and y E y+()) is the associated response. In order to have a more tractable and insightful setup, we follow Romera-Parede et al. [36] and consider the linear multi-task regression model. In particular, we study a straightforward linear signal-plus-noise model
*Correspondence to: romain.ilbes @ fotmail.fr.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

that evaluates the response (0) for the i-th sample of the 1-th task as follows:
Y X
Wr t((t) VTd
Vt fI,...,,T;
(I)
where e(1) E [R*+x4 is matrix of noise vectors with each e, (0, EN), EN Rxx4 denoting the covariance matrix. The matrix W, Rdxa denotes the We denote the concatenation of all task-specific hyperplanes by W [W].. ..... w RTáxa, We assume that W, can be decomposed as a sum of a common matrix Wo E Rdxg, which captures the shared information across all the tasks, and a task-specific matrix V, E Rdxq, which captures deviations specific to the task t:
hyperplane for task t.
W. = W V.
Given the multitask regression framework and the linear signal-plus-noise model, we now want to retrieve the common and specific hyperplanes, Wo and V,, respectively. To achieve this, we study the following minimization problem governed by a parameter À that controls the balance between the common and specific components of W:
Wo, (Vi) X argmin
1 2A
Woll
1
V.I.
1 Σ 2
Y
X W:
(3)
Y
VTd
where 7 = l...... TT) a vector of task-specific regularization hyperparameters. Each y controls how much the model overfits (small 71)) or underfits (large 71)) on task L.
Contributions and Main Results. We seek to provide a rigorous theoretical study of (3) in high dimension, providing practical insights that make the theoretical application implementable in practice. Our contributions can be summarized in four major points:
1. We provide exact train and test risks for the solutions of (3) using random matrix theory. We then decompose the test risk into a signal and noise term responsible for the effectiveness of multi-task learning and the negative transfer, respectively.
2. We show how the signal and noise terms compete with each other depending on for which we obtain an optimal value optimizing the test risk.
3. In particular case, we derive a closed-form data-dependent solution for the optimal A+, involving raw data covariances, signal-generating hyperplane, noise levels, and the size and number of data sets.
4. We offer empirical estimates of these model variables to develop a ready-to-use method for hyperparameter optimization in multitask regression problems within our framework.
Applications. As an application, we view multivariate time series forecasting (MTSF) as multi- task problem and show how the proposed regularization method can be used efficiently employ univariate models in the multivariate case. Firstly, we demonstrate that our method improves PatchTST [32] and DLinear [53] compared to the case when these approaches are applied indepen- dently to each variable. Secondly, we show that the univariate models enhanced by the proposed regularization achieve performance similar to the current state-of-the art multivariate forecasting models such SAMformer [19] and iTransformen [24].
2 Related Work
Usual Bounds in Multi-Task Learning. Since its introduction [1, 5, 26], multi-task learning (MTL) has demonstrated that transfer learning concepts can effectively leverage shared information to solve related tasks simultaneously. Recent research has shifted focus from specific algorithms or minimization problems to deriving the lowest achievable risk given an MTL data model. The benefits of transfer learning are highlighted through risk bounds, using concepts such contrasts [23]. which study nigh-dimensional linear regression with auxiliary samples characterized by the sparsity of their contrast vector transfer distances [29], which provide lower bounds for target generalization

---

error based on the number of labeled source and target data and their similarities. Additionally, lusk-correlttion matrices [30] have been used to capture the relationships inherent in the data model.
However, these studies remain largely theoretical, lacking practical algorithms to achieve the proposed bounds. They provide broad estimates and orders of magnitude that, while useful for gauging the feasibility of certain objectives, do not offer precise performance evaluations. Moreover, these estimates do not facilitate the optimal tuning of hyperparameters within the MTL framework, critical aspect of our study. We demonstrate the importance of precise performance evaluations and effective hyperparameter tuning enhance the practical application of MTL.
Random Matrix Theory in Multi-Task Learning. In the nigh-dimensional regime, Random Matrix Theory allows obtaining precise theoretical estimates on functionals (c.g., train or test risk) of data matrices with the number of samples comparable to data dimension [2. 13, 31, 46). In the multi-task classification case, large-dimensional analysis has been performed for Least Squares SVM [48] and supervised PCA [47] showing counterproductivtty of tackling tasks independently. We inspire our optimization strategy from [48] and conduct theoretical analysis for the multi-task regression, which introduces unique theoretical challenges and with a different intuition since we need to consider another data generation model and a regression learning objective. Our data modeling shares similarity with [52] which theoretically studied conditions of a negative transfer under covariate and model shifts. Our work focuses on the selection of hyperparameters within a general optimization framework, considering both a hyperparameter À to relate all tasks and specific parameters to balance overfitting and underfitting within each task, accommodating non-Gaussian distributions and employing recent developments such as deterministic equivalents and concentration inequalities.
High dimensional analysis for regression. Regression is important problem that has been extensively explored in the context of single-task learning for high-dimensional data, employing tools such as Random Matrix Theory [12]. physical statistics [9, 15], and the Convex Gaussian MinMax Theorem (CGMT) [43], among others. Typically, authors employ a linear signal plus noise model to calculate the asymptotic test risk function of the signal-generating parameter and the covariance of the noise. Our research builds upon these studies, extending them to a multi-task learning framework. In doing so, we derive several insights unique to multi-task learning. However, none of these previous studies offer a practical method for estimating the asymptotic test risk, which dependent on the ground truth previous studies into the multi-task case within generic data distribution (under the assumption of concentrated random vector), but also provides practical method for estimating these quantities.
parameter. Therefore, our work not only extends
3 Framework
Notation.
Throughout this study, matrices are represented by bold uppercase letters (c.g., matrix A), vectors by bold lowercase letters (e.g., vector v), and scalars by regular, non-bold typeface (e.g., scalar a). The notation A @ B for matrices or vectors A, B is the Kronecker product. Dx or Diag(x) stands for diagonal matrix containing its diagonal the elements of the vector x. The superscripts
t and i are used to denote the task and the sample number, respectively, c.g., xx writes the i-th sample of the t-th task. The canonical vector of RT is denoted by e!"T with [e(")], du. Given
matrix M E Rex=, the Frobenius norm M is denoted ||M|| / tr M). For our theoretical analysis, we introduce the following notation of training data:
Y = ( Y(T E R
Z =
e, e;
[T [L]
& x() E Rdxx
1=1
where n Er ne the total number of samples in all the tasks.
3.1 Regression Model
We define V = VT V T ReT'y and propose to solve the linear multi-task problem by finding W = W 1 WJ E pad'xg which solves the following optimization problem under the assumption of relatedness between the tasks, i.e., W, Wo + V, for all tasks
min (Wo. (
T JWW. V)
(4)
3

---

where
J(Wo V) 2X
1 Wol
1 2
T
V P 1 2
Σ
Y
X
W
vTd
The objective function consists of three components: a regularization term for Wo mitigate overfitting, task-specific regularization term that controls deviations V, from the shared weight matrix Wo, and a loss term quantifying the error between the predicted outputs and the actual responses for each task.
The optimization problem is convex for any positive values of X, ",,.... Yr, and it possesses unique solution. The details of the calculus are given in Appendix B. The formula for W, is given follows:
Wr
et
[T]
Id
AZQY
Wo (17 AL))
ZQY
TTd
TTd
where with Q
Td
AZ TT
+
In
-1
Ruxn
and A = (Dy + +11717) @ I RTxxa
3.2 Assumptions
In order to use Random Matrix Theory (RMT) tools, we make two assumptions on the data distribution and the asymptotic regime. Following [50]. we adopt a concentration hypothesis on the feature vectors (t) which was shown to be highly effective for analyzing machine learning problems [11, 14]. The justification and implications of these assumptions can be found in appendix A.2 and A.3.
Assumption (Concentrated Random Vector). We assume that there exists two constants C, c > 0 (independent of dimension d) such that, for any task t. for any (LLnschi f Rd any feature vector x E (t) verifies:
Vt 0: ((I(((((()) E[f(( 2 t) s Ce-(//)
E [[x(1== = and Cov(x(0) E(
In particular, we distinguish the following scenarios: x, (t) Rd are concentrated when they are (i) independent Gaussian random vectors with covariance of bounded norm, (ii) independent random vectors uniformly distributed on the Rd sphere of radius vd, and most importantly (iii) any Lipschitz transformation oxx(") of the above two cases, with bounded Lipschitz norm. Scenario (iii) is especially pertinent for modeling data in realistic settings. Recent research [39] has demonstrated that images produced by generative adversarial networks (GANs) are inherently qualified as concentrated random vectors.
Next, we present a classical RMT assumption that establishes a commensurable relationship between the number of samples and dimension.
Assumption 2 (High-dimenssoonal asymptotics). As d + co, O(d) and T = O(). More specifically, we assume that n/d a... cu < oo with ΣT Rt.
Although different from classical asymptotic where the number of samples is implicitly assumed to be exponentially larger than the dimension, the high dith-iinnniionll asymptotic finds many applications including celcoommuiccatios [10], finance [34] and machine learning [11, 14, 48].
4
Main Theoretical Results
4.1 Estimation of the Performances
Given the training dataset X Rexd with the corresponding response variable Y Raxy and for any test dataset (() e Rd and the corresponding response variable (()) e IR9, we aim to derive the theoretical training and test risks defined as follows:
T
1 E E Tn
x(X)],)
Rocst
T Ellly (xx())
Σ t=1

---

The output score 8(x(1) E R9 for data x(1) E Rd of task is given by:
s(x(1)
(e) m x T
1 W, Td
(ei") exx11) AZQY
(5)
VTd
In particular, the output for the training score is given by:
2 AZ Td
9 (X)
Z AZQY, Td
Q
ITd
(6)
To understand the statistical properties of Rrain and Riest for the linear generative model, study the statistical behavior of the resolvent matrix Q defined in (6). The notion of a deterministic equivalent from random matrix theory [16] is particularly useful here as allows us design deterministic matrix equivalent to Q in the sense of linear forms. Specifically, deterministic equivalent M of a random matrix M deterministic matrix that satisfies u(M M) + almost surely for any bounded linear form Raxd
We denote this as M ++ M. This concept is
crucial to our analysis because we need to estimate quantities such as Žtr (AQ) or a Qb, which bounded linear forms of Q with A, a and all having bounded norms. For convenience, further express some contributions of the performances with the so-called "coresolvent" defined as "zzz"A
Q
A 2 T'd
Ira)=
Using Lemma provided in the Appendix C.I, whose proofs are included in Appendices C.2, C.3 and C.4, we establish the deterministic equivalents that allow us to introduce our Theorems and 1, characterizing the asymptotic behavior of both training and testing risks.
Theorem 1 (Asymptotic train and test risk). Assuming that the training data vectors x(" and the test data vectors (8) are concentrated random vectors, and given the growth rate assumption (Assumption 2), follows that:
Lr
W T
A
(A)A
tr En Q2) Td
Rist
tr (En)
(ATR)
Td
signal term
noise terma
In addition, the asymptotic risk on the training data is given by
ir W A AWTA-//QAA-//2 ir ( -1/2
tr E, Q2) Tn
R
Tn
Tn
where Q (Ira) and Q2 respectively deterministic equivalents for Q, Q and Q?
We defer the full proof of this theorem to Appendix D and provide an outline of the test risk proof below. Note that derivations of the asymptotic risk for the training data are more complex and involve computing deterministic equivalents (Lemma of Appendix C.I), which a powerful tool commonly used in Random Matrix Theory. In the derived theorem, we achieve stronger convergence result compared the almost surely convergence. Specifically, we prove a concentration result for the training and test risk with exponential convergence rate for sufficiently large values of and d.
5

---

F Reest
Ily [(x)]
T
X
W
x AZQZ T W
x AZQE
T
VTd
TdvTd
Td
tr W
EW)
2tr (W EAZQZ W)
tr (W ZQZ AEAZOZ W) (Td)s
7
+ tr (5Te)
Td
(Td)²
tr (E
QZ AΣAZQE)
(Td)a
Ir ( ΣW) tr
EA (Ta Q)A :
+ tr E) + Ir (WT EW)
Td
Td
Td
tr ( (WTEAAQA--Y W A (Q)AA)
- 2
+
tr tr(En +0
Td
Td
Td
vd
4.2 Error Contribution Analysis
To gain theoretical insights, we analyze (ATR) consisting of the signal and the noise components. Signal Term. The signal term can be further approximated, up to some constants as tr ( (AE I)-2W) with Σ Σt nu (1),, The matrix (AE + I)- plays a erucial role in amplifying the signal term tr( W), which in turn allows the test risk to decrease. The off-diagonal elements of AA + I) -2 amplify the cross terms (tr(W, We) for * v), enhancing the multi-task aspect, while the diagonal elements amplify the independent terms (1))). This structure is significant in determining the effectiveness of multi-task learning.
Furthermore, both terms decrease with an increasing number of samples vi,, smaller values of "t and larger value of X. The cross term, which is crucial for multi-task learning, depends the matrix E, E.. This matrix represents the shift in covariates between tasks. When the features are aligned (i.e., Σ, E. La), the cross term is maximized, enhancing multi-task learning. However, larger Fisher distance between the covariances of the tasks results in less favorable correlations for multi-task learning.
Noise term. Similar the signal term, the noise term can be approximated, up to some constants, as tr(Ex(1 (A A- 1)) However, there is a major difference between the way both terms are expressed in the test risk. The noise term does not include any cross terms because the noise across different tasks is independent. In this context, only the diagonal elements of the matrix are significant. This diagonal term increases with the sample size and the value of X. It is responsible for what is known negative transfer. As the diagonal term increases, it negatively affects the transfer of learning from one task to another. This a critical aspect to consider multi-task learning scenarios.
4.3 Simplified Model for Clear Insights
In this section, we specialize the theoretical analysis to the simple case of two tasks (T= assume that the tasks share the same identity covariance and that ? 72 7. Under these conditions, the test risk can be approximated, up to some constants, as
= 2). We
Ric DIs (I (1 1 + (2 CATZL W W₂ + NNrtrE,
where the diagonal term (independent learning) DIL, the cross term (multi- task learning) CMTL. and the noise term (negative transfer) N.T have closed-form expressions depending on y and X:
D (co(A 7) + 1)2 cg
-20g + +7) 11)
CMTL (co(X+2) +))
(co(A + 7) -
(co(A + +A+ -cuA)))))
NTT
((co() 1)2 c(A2)²

---

Eignal
Polae
Sigeel Teen Cecan Becn
Soiee
Signal
Holas Teat Peak
Ccoan decs
Zear 2438
20a5 2648
Cooa8-
Trass 300000
2000.33
derttin 00
0:20
Lantida
(a) 5 0.5
(h)
1.5
(c) 5 = 2.5
Figure 1: Test loss contributions DIL, CASTL, NAT across three sample size regimes. Test risk exhibits decreasing, increasing, convex shapes based on the regime. X from theory are marked.
We recall that co has been defined in the Assumption 2. primarily composed of two terms: the signal term and the noise term, which each other. The more similar the tasks are, the stronger the signal term becomes. In the following plot, we illustrate how this competition can influence the risk. Depending on the value of the parameter and the sample sizes, the risk can either decrease monotonically, increase monotonically, or exhibit a convex behavior. This competition can lead to an optimal value for X, which interestingly has simple closed-form expression that can be obtained by deriving the test w.r.t. À as follows (see details in Appendix E.4):
As previously mentioned, the test risk is
competition with
71 SNR
7 with SNR d 2'
| W
W W2
X
+
trΣN
trEN
4.4 Comparison between Empirical and Theoretical Predictions
In this section, we compare the theoretical predictions with the empirical results. Our experiment is based on two-task setting (T = 2) defined as W, NN Ip) with W, a aW + v1 a W W represents any vector orthogonal to W1 and a E [0, 1]. This setting allows us to adjust the similarity between tasks through a.
Figure 2 shows a comparison of the theoretical and empirical classification errors for different values of X, highlighting the emror-minimiiing value of X. Despite the relatively small values of and p, there is very precise match between the asymptotic theory and the practical experiment. This is particularly evident the accurate estimation of the optimal value for X.
Train Beror
Post Rrror
bew
10.9
Lambda
Lambda
Figure 2: Empirical and theoretical train and test MSE as functions the parameter À for different values of a. The smooth curves represent the theoretical predictions, while the corresponding curves with the same color show the empirical results, highlighting that the empirical observations indeed match the theoretical predictions.

---

Toor iepirioni
Sepizioel
12.1
7ra3nicg Expor
19.0
11.4
0.4
01:-3
11.2
14
20
1:04
129
124
18%
155
Lembda
Lanbda
(a) Training MSE
(b) Testing MSE
Figure 3: Theoretical vs Empirical MSE as function of regularization parameter A. Close fit between theoretical and the empirical predictions which underscores the robustness of the theory light of varying assumptions well as the accuracy of the suggested estimates. We consider the first two channels as the the two tasks and d 144. 95 samples are used for the training and 42 samples are used for the test.
5 Hyperparameter Optimization in Practice
5.1 Empirical Estimation of Task-wise Signal, Cross Signal, and Noise
All the quantities defined in Theorem are known except for the bilinear expressions T7 tr( W MW) and tr tr (E, M). These quantities can be consistently estimated under Assumptions follows (see details in Section F of the Appendix):
Td tr (W MW) ((M) as 0,
Td tr (ExM) - 2trM 25
where ((M) 7 tr WWTTWWTT (M)M))
Ta trQ (A ( (M)AE)
We define the estimate of the noise as ő = lim R and the function x-1 is the functional
To
inverse of the mapping & RTdxTd + IRex4 defined as follows
x(M) = M - 2A Q A (A
n (Td)2
A EA (A
5.2 Application to Multi-task Regression
In our study, we apply the theoretical framework presented our paper to a real-world regression problem, specifically, the Appliance Energy dataset which aims to predict the total usage of a house. This dataset a multivariate regression dataset containing 138 time series each of dimension 24. We select two of these features 2 tasks cast the problem multi-task learning regression problem.
Figure 3 presents a comparison between the theoretical predictions and empirical outcomes of our proposed linear framework. Despite the assumptions made in the main body of the paper, the theoretical predictions closely align with the experimental results. This demonstrates that our estimates are effective in practice and provide reliable approximation of the optimal regularization.
In essence, our theoretical framework, when applied to real-world multi-task learning regression problems, yields practical and accurate estimates, thereby validating its effectiveness and applicability.
5.3 Relevance of the theoretical insights beyond the case linear models
While non-linear models are widely used, establishing their theoretical foundations is challenging. Therefore, we focused on linear models, which, despite their simplicity, provide valuable insights into more complex models.

---

Our results show that test risk curves for non- non-linear models follow patterns predicted by our theory. This is expected because non- non-linear models time series forecasting typically use a linear output layer for prediction. Thus, we can apply our theory to the inputs of this final linear layer. This approach valid due to data concentration and the Lipschitz nature of neural networks, ensuring outputs of the non-linear part don't 't deviate significantly from the inputs.
Moreover, multivariate time series models often treat channels separately using univariate methods, missing cross-channel information. Our results in Section 5,4 show that our method surpasses univariate baselines by optimally regularizing with and y. supporting our theory's 's applicability to non-linear models as the final linear layer effectively leverages concentrated inputs.
Finally, regularization approach differs from traditional cross-task regularizations that use one task per dataset. We consider each prediction as task and introduce 7/x parameters alongside X. These parameters enforce multivariate regularization and control underfitting overfitting per task. This method is tractable since it's applied at the model's final layer.
The similarity between curves for non-linear and linear models indicates our findings are robust; non- linear models also exhibit optimal regularization parameters, enhancing performance in multivariate forecasting,
5.4 Application to Multivariate Time Series Forecasting
Our theoretical framework is applied in the context of Multivariate Time Series Forecasting, with related work detailed in Appendix G.I. We previously applied this framework a linear setting, and now aim to evaluate its empirical validity in the -inear setting of neural networks. The results presented in this section represent the best test MSE, assuming the ability to find the optimal lambda value, which can be considered an oracle scenario. A study of these limitations can be found in Appendix H.
Motivation. Our approach is applied to the MTSF setting for several reasons. Firstly, many models currently used are essentially univariate, where predictions for individual series are simply concatenated without exploiting the multivariate information inherent in traditional benchmarks. Given that these benchmarks are designed for multivariate forecasting, leveraging multivariate information should yield better results. Secondly, our theoretical framework can benefit this domain, as most predictive models use a linear layer on top of the model to project historical data of length for predicting the output of length q. This characteristic aligns well with our method, making it promising fit for enhancing forecasting accuracy.
Our approach. We propose novel method for MTSF by modifying the loss function to incorporate both individual feature transformations ft and shared transformation fo- Each anivariate-speciffe transformation f. is designed to capture the unique dynamics of its respective feature, while fo serves as common transformation applied across all features to capture underlying patterns shared among them. We consider a neural network f with inputs X [X("), x2). ......... X(T),, where T is the number of channels and (() Raxd For a univariate model without MTL regularization, we predict Y [1) (() ..... Y (X()), )..., (x(()) and Y E Raaxq We compare these models with their corresponding versions that include MTL regularization, formulated as: ft (x(1)(()) = fc(x()) + fo(Y) with f: Raxd + Raxq and fo RuxxTT + Ruxx, We define regularized loss as follows:
T
C(X, Y) = Σ 10r00 -OY00-MMTIN (XT +X]60)11 + D 11 (1, .T...
where (()) are the true predictions, ft represents the univariate model for each channel I, and is our regularization parameter, for which we have established closed form in the case of linear fi- fo serves a role equivalent to Wo, which was defined in our theoretical study and allows for the regularization of the common part. This component can be added the top of univariate model. The parameters enable the regularization of the specialized parts ft-
In our setup. fi is computed in similar way in the model without regularization and fo is computed by first flattening the concatenation of the predictions of ((), then applying a linear projection leveraging common multivariate information before reshaping. The loss function specifically designed to balance fitting the multivariate series using fo and the specific channels using ft. This

---

Table I: MTL regularization results. Algorithms marked with t are state-of-the-ar multivariate models and serve as baseline comparisons. All others are univariate. We compared the models with MTL regularization to their corresponding versions without regularization. Each MSE value is derived from 3 different random seeds. MSE values marked with * indicate that the model with MTL regularization performed significantly better than its version without regularization, according to a Student's 's t-test with a p-value of 0,05. MSE values are in bold when they are the best in their row, indicating the top-performing models.
with MTL regultrization
without MTL regulasizaionn
Dataset
PatchTSTT
DLiseard
Transfoxner
FarchTBT
DLisearM DLinearU
Texnor
BAXforzez
TTrans
96
0.385
0,367'
0.368
0.387
0.397
0.386
0.370
0.381
0.386
192
0.405'
0.607
0:124
0.437
0.111
336
01,2113
0.431
0.433
0.442
0.431
0.481
0.437
0,423
0.457
720
0.430"
0.454
0,455*
0:451
0.519
0.470
96
01.91
1,267'
0.270
0.295
0.294
0.333
0.273
0.255
0.297
E7
192
0.346'
0.331'
0.337
0.351
0477
0.339
0.540
0.380
336
0.332'
0.367
0.596"
0,342
0.361
0.594
0.369
0.350
0,428
720
0.384'
0.412
0.405
0.393
0.395
0.831
0.428
0.391
0.427
96
0.148
0.140"
0.154
0.149
0.196
0.196
0.170
0.197
0.174
192
0.190
0.206'
0.196*
0.193
0.243
0.237
0.214
0.235
0.221
336
0.349"
0.258
0.246
0.283
0.260
0.276
730
0.316'
0.326*
0.331
0,322
0,359
0.345
0.326
0.334
0.358
approach enhances the model's generalization across various forecasting horizons and datasets. More details on our regularized loss function can be found in Appendix G.2
We present experimental results on different forecasting horizons, using common bench- mark MTSF datasets, the characteristics of which are outlined in the Appendix G.3, Our models include PatchTsT [32], known to be par with state-of-the-art in MTSF while being a univariate model, a univariate DLinear version called DLinearU compared to its multivariate counterpart DLinearM [53]. and univariate Transf ormer [19] with temporal-wise attention compared to the multivariate state-of-the-an art models SAMF ormer [19] and i TTransformer [24]. Table provides a detailed comparison of the test mean squared errors (MSE) for different MTSF models, emphasizing the impact of MTL regularization. Models with MTL regularization are compared to their versions
Results.
without regularization, as well as SAMf ormer and i Trrnssoommr. All the experiments can be found in Appendix G.4.
Adding MTL regularization improves the performance of PatchTST, DLinearU, and Transformer in most cases. When compared to state-of-the-ar multivariate models, the MTL-regularized models are often competitive. SAMf ormer is outperformed by at least one MTL-regularized method per horizon and dataset, except for ETTh1 with horizons of 336 and 720. ATransforms is consistently outperformed by at least one MTL-regularized methods regardless of the dataset and horizon.
The best performing methods are PatchTST and DLinearU with MTL regularization. These models not only outperform their non-regularized counterparts, often significantly as shown by Student's t-tests with p-value of 0.05, but also surpass state-of-the-art multivariate models like SAMformer and Trans fomer. This superior performance is indicated by the bold values in the table.
Finally, MTL regularization enhances the performance of univariate models, making them often competitive with state-of-the-arr multivariate methods like SAMformer and iTransf approach seems to better captures shared dynamics among tasks, leading to more accurate forecasts.
This
6 Conclusions and Future Works
In this article, we have explored linear multi-task learning by deriving a closed-form solution for an optimization problem that capitalizes on the wealth of information available across multiple tasks. Leveraging Random Matrix Theory, we have been able to obtain the asymptotic training and testing risks, and have proposed several insights into nigh-dimensional multi-task learning regression. Our theoretical analysis, though based on a simplified model, has been effectively applied to multi-task regression and multivariate forecasting, using both synthetic and real-world datasets. We believe that our work lays solid foundation for future research, paving the way for using random matrix theory with more complex models, such deep neural networks, within the multi-task learning framework.
10

---

Acknowledgements
Work partially funded by EU project A14Europe (101070000).
References
[1] Ando, R. K., Zhang, T., and Bartlett, P. (2005). A framework for learning predictive structures from multiple tasks and unlabeled data. Journal of machine learning research, 6(11).
[2] Bai, Z. and Silverstein, J. W. (2010). Spectral Analysis of Large Dimensional Random Matrices. Springer Series in Statistics. Springer New York, NY.
[3] Box, G. E. P. and Jenkins, G. (1990). Time Series Analysis, Forecasting and Control. Holden-Day Inc., USA.
[4] Box, G. E. P., Jenkins, G. M., and MacGregor, F. (1974). Some Recent Advances in Forecasting and Control. Journal of the Royal Statistical Society Series C, 23(2):158-179,
[5] Caruana, R. (1997). Multitask learning. Machine learning, 28:41-75.
[6] Čepulionis, P. and Lukoševičiūté, K. (2016). Electrocardiogram time series forecasting and optimization using ant colony optimization algorithm. Mathematical Models Engineering. 2(1):69-77.
[7] Chen, R. and Tao, M. (2021). Data-driven prediction of general hamiltonian dynamics via learning exactly-symplectic maps. In Meila, M. and Zhang, T., editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 1717-1727. PMLR.
[8] Chen, S.-A., Li, C.-L,, Arik, O., Yoder, N. C., and Plister, T. (2023). TSMixer: An all-MLP architecture for time series forecasting. Transactions on Machine Learning Research.
[9] Clarté, L., Loureiro, B., Krzakala, F., and Zdeborová, L. (2023). Theoretical characterization of uncertainty in high-dimensional linear classification. Machine Learning: Science and Technology, 4(2):025029.
[10] Couillet, R. and Debbah, M. (2011). Random matrix methods for wireless communications. Cambridge University Press.
[11] Couillet, R. and Liao, Z. (2022). Random matrix methods for machine learning. Cambridge University Press.
[12] Dobriban, and Wager, S. (2018). High-dimensional asymptotics of prediction: Ridge regression and classification. The Annals of Statistics, 46(1):247-279
[13] Erdós, L. and Yau, H.-T. (2017). Dynamical Approach Random Matrix Theory, volume 28 of Courant Lecture Notes. American Mathematical Society.
[14] Feofanov, V., Tiomoko, M., and Virmaux, A. (2023). Random matrix analysis to balance between supervised and unsupervised learning under the low density separation assumption. In Proceedings of the 40th International Conference on Machine Learning, pages 10008-10033.
[15] Gerbelot, C., Abbara, A., and Krzakala, F. (2022). Asymptotic errors for teacher-student convex generalized linear models (or: How prove kabashima's 's replica formula). IEEE Transactions Information Theory, 182441882411822
[16] Hachem, W., Loubaton, P., and Najim, J. (2007). Deterministic equivalents for certain function- als of large random matrices.
[17] Hu, Y., Li, M., Lu, Q. Weng, H., Wang, J., Zekavat, S. M., Yu, Z., Li, B., Gu, J., Muchnik, S., et al. (2019). A statistical framework for pross-tissue transcriptome-wide association analysis. Nature genetics, 51(3):568-57.

---

[18] Ilbert, R., Hoang, T. V., and Zhang, Z. (2024a). Data augmentation for multivariate time series classificatio:: An experimental study. In 2024 IEEE 40th International Conference on Data Engineering Workshops (ICDEW), pages 128-139.
[19] Ilbert, R., Odonnat, A., Feofanov, V., Virmaux, A., Paolo, G., Palpanas, T., and Redko, I. (2024b). Unlocking the potential of transformers in time series forecasting with sharpness-aware minimization and channel-wise attention.
[20] Ilbert, R., Tiomoko, M., Louart, C., Feofanov, V., Palpanas, T., and Redko, I. (2024c). Enhancing multivariate time series forecasting via multi-task learning and random matrix theory. In NeurIPS Workshop on Time Series the Age of Large Models.
[21] Kim, T., Kim, J., Tae, Y., Park, C., Choi, J.-H., and Choo. J. (2021). Reversible instance normalization for accurate time-series forecasting against distribution shift. International Conference on Learning Representations.
[22] Kingma, D. and Ba, J. (2015). Adam: method for stochastic optimization. In International Conference on Learning Representations (ICLR), San Diega, CA, USA.
[23] S.. S., Cai, T., and Li, H. (2022). Transfer learning for high high-ddmeeniionl linear regression: Prediction, estimation and minimax optimality. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(1):149-173.
[24] Liu, Y., Hu, T., Zhang, H., Wu, H., Wang. S., Ma, L., and Long, M. (2024). itransformer: Inverted transformers are effective for time series forecasting. In The Twelfth International Conference on Learning Representations.
[25] Louart, C. and Couillet, R. (2021). Spectral properties of sample covariance matrices arising from random matrices with independent non identically distributed columns. arXiv preprint arXiv: 109.02649.
[26] Lounici, K., Pontil, M., Tsybakov, A. B., and Van De Geer, S. (2009). Taking advantage of sparsity in multi-task learning, arXiv preprint arXiv:0903.1468.
[27] Max Planck Institute (n.d.). Weather dataset.
[28] Mei, S., Fei, W., and Zhou, S. (2011). Gene ontology based transfer learning for protein subcellular localization. BMC bioinformatics, 12:1-12.
[29] Mousavi Kalan, M., Fabian, Z., Avestimehr, S., and Soltanolkotabi, M. (2020). Minimax lower bounds for transfer learning with linear and one-hidden layer neural networks. Advances in Neural Information Processing Systems, 33:1959-19966 1969.
[30] Nguyen, M.-T. and Couillet, R. (2023). Asymptotic bayes risk of semi-supervised multitask learning on gaussian mixture. In International Conference Artificial Intelligence and Statistics, pages 5063-5078, PMLR.
[31] Nica, A. and Speicher, R. (2006). Lectures the Combinatorics of Free Probability. London Mathematical Society Lecture Note Series, Cambridge University Press
[32] Nie, Y., Nguyen, N. H., Sinthong, P., and Kalagnanam, J. (2023). A time series is worth 64 words: Long-term forecasting with transformers. In The Eleventh International Conference on Learning Representations.
[33] Niu, S., Liu, Y., Wang, J., and Song, H. (2020). A decade survey of transfer learning (2010- 2020). IEEE Transactions on Artificial Intelligence, 1(2):151-166.
[34] Potters, M., Bouchaud, J.-P., and Laloux, L. (2005). Financial applications of random matrix theory: Old laces and new pieces. Acta Physica Polonica B, 36(9):7767.
[35] Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, J. (2020). Exploring the limits of transfer learning with unified text-to-text transformer.
Journal of Machine Learning Research, 21:1-67.
12

---

[36] Romera-Paredes, B.. Aung, H., Bianchi- Berthh-e N., and Pontil, M. (2013). Multilinear multitask learning. In International Conference on Machine Learning, pages 1444-1452. PMLR.
[37] Ruder, S., Peters, M. E., Swayamdipta. S., and Wolf, T. (2019). Evolution of transfer learning in natural language processing. arXiv preprint 190007700
[38] Salinas, D., Flunkert, V., Gasthaus, J., and Januschowski, T. (2020). Deepar: Probabilis- tie forecasting with autoregressive recurrent networks. International Journal of Forecasting. 1613)11811-111.
[39] Seddik, M. E. A., Louart, C., Tamaazousti, M., and Couillet, R. (2020). Random matrix theory proves that deep learning representations of gan-data behave as gaussian mixtures. In International Conference on Machine Learning.
[40] Sen, R., Yu, H.-F., and Dhillon, I. (2019). Think globally, act locally: a deep neural network approach to high- ditheemnnsional time series forecasting. In Proceedings the 33rd International Conference on Neural Information Processing Systems, Red Hook, NY, USA. Curran Associates Inc.
[41] Shao, L. (2015). Transfer learning for visual categorization: survey. IEEE Transactions on Neural Networks and Learning Systems, 26(5):1019 1019 1034.
[42] Shin, H.-C., Roth, H. R., Gao, M., Lu, L., Xu, Z., Nogues, I., Yao, J., Mollura, D., and Summers, R. M. (2016). Deep convolutional neural networks for computer-aided detection: Can architectures, dataset characteristics and transfer learning. IEEE transactions on medical imaging, 35(5):1285-1298.
[43] Sifaou, H., Kammoun, A., and Alouini, M.-S. (2021). A precise performance analysis of support vector regression. In International Conference on Machine Learning, pages 9671-9680. PMLR.
[44] Sonkavde, G., Dharrao, D. S., Bongale, A. M., Deokate, S. T., Doreswamy, D., and Bhat, S. K. (2023). Forecasting stock market prices using machine learning and deep learning models; A systematic review, performance analysis and discussion of implications. International Journal of Financial Studies, 11(3).
[45] Sorjamaa, A.,
Hao, J., Reyhani, N., Ji, Y., and Lendasse, A. (2007). Methodology for long-term prediction time series. Neurocomputing, 70(16):881-22699 Neural Network Applications in Electrical Engineering Selected papers from the 3rd International Work-Conference on Artificial Neural Networks (IWANN 2005).
[46] Tao, T. (2012). Topics Random Matrix Theory, volume 132 of Graduate Studies Mathe- matics. American Mathematical Society.
[47] Tiomoko, M.. Couillet, R., and Pascal, F. (2023). PCA- multi-task learning: a random matrix approach. In Krause, A.. Brunskill, E., Cho, K.,, Engelhardt, B., Sabato. S., and Scarlett, J.. editors, Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages $4280-34300 PMLR.
[48] Tiomoko, M., Couillet, R., and Tiomoko, H. (2020). Large dimensional analysis and improve- ment of multi task learning.
[49] UCI (n.d.), Electricity dataset.
[50] Wainwright, J. (2019). High-dimensional statistics: A non-asymptotic viewpoint, volume 48. Cambridge university press.
[51] Wu, H., Xu, J.. Wang, J., and Long, M. (2021). Autoformer:
Decomposition transformers with for long-term series forecasting. In Advances in Neural Information Processing
Systems.
[52] Yang, F., Zhang, H. R., Wu, S., Ré, C., and Su, W. J. (2023). Precise high-dimensional asymptotics for quantifying heterogeneous transfers.
[53] Zeng, A., Chen, M., Zhang, I.., and Xu, Q. (2023). Are transformers effective for time series forecasting? In Proceedings of the AAAI Conference Artificial Intelligence.
13

---

[54] Zhang, H., Zheng, Y., and Qi, D. (2019). Multi-task learning for time series forecasting using neural networks. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1305-1313. ACM.
[55] Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, Xiong, H., and Zhang, W. (2021). Informer: Beyond efficient transformer for long sequence time-series forecasting. In The Thirty-Fifth AAAI Conference Artificial Intelligence, AAAI 2021, Virtual Conference, volume 35, pages 11106-11115. AAAI Press.
[56] Zhou, T., Ma, Z., Wen, Q. Wang. X., Sun, L., and Jin, R. (2022). FBDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In Proc. 39th International Conference on Machine Learning (ICML 2022).
14

---

Appendix
Roadmap. This appendix provides the technical details omitted in the main paper. starts with an overview of the setup considered in the paper in Section A. Section B offers detail computation for W, and Wo. Section C contains proof of Lemma 1. Section D explains the theoretical steps for deriving the training and test risks, as well as the deterministic equivalents. Section E discusses technical tools used to derive the main intuitions presented by the theory. Section F focuses on the derivation estimations of the main quantities involved in the training and test risks. Section G complements the experimental study by presenting additional experiments. Finally, Section H deals with the limitations our approach a non non-linear setting.
Table of Contents
Multi-task Learning setup
16
A.1 On the zero-mean assumption
16
A.2 On the Assumption
16
A.3
the Assumption
16
Minimization Problem
16
B.1 Computation of We and Wo
16
Lemma and proof with Random Matrix Theory
18
C.1 Lemma
18
C.2 Deterministic equivalent of the resolvent Q
18
C.3 Deterministic equivalent of bilinear forms of the resolvent
19
C,4 Estimation of the deterministic equivalent of Q?
Risk Estimation (Proof of Theorem
22
D.1 Test Risk
22
D.2 Train Risk
23
Interpretation and insights of the theoretical analysis
23
E.1 Analysis of the test risk
23
E.2 Interpretation of the signal term
24
E.3 Interpretation and insights of the noise terms
24
E,4
Optimal Lambda
24
Theoretical Estimations
25
F.1
Estimation of the training and test risk
25
F.2 Estimation of the noise covariance
25
G Multivariate Time Series Forecasting
25
G.1 Related Work
25
G.2 Architecture and Training Parameters
26
G.3 Datasets
26
15

---

G.4 Additional Experiments.
28
H Limitations
31
A Multi-task Learning setup
A.1 On the zero-mean assumption
We would like note that assuming that both the noise and the feature x; have mean. This is common assumption in many statistical models and simplifies the analysis. However, this assumption is not restrictive. In practice, if the data the response variable not centered, always preprocess the data by subtracting the mean. This preprocessing step brings us back to the zero-mean setting that we consider in our theoretical analysis.
(1)
A.2
On the Assumption
Data are concentrated random vectors, meaning sigh-dimensional data maintain stable under com- plex (Lipschitz) transformations. The strong performance of neural networks tasks like image recognition and NI.P suggests that these models produce stable predictions. As Lipschitz transfor- mations, they maintain controlled distances between inputs, ensuring stability. Recent studies have demonstrated and experimentally confirmed that both real-world data and synthetically generated data using GANs exhibit concentration properties, supporting this assumption. This makes our assumption more realistic than traditional Gaussian assumptions, as does not rely on specific hypotheses about the shape of the data distribution, but rather on the stability of statistical properties after transformation. Consequently, analyzing a framework of concentrated random vectors is more theoretically challenging than using Gaussian assumptions and represents key novelty of our theory.
A.3
On the Assumption 2
The dimension d is of the same order of magnitude as the sample size n. This joint growth captures data complexity better than assuming fixed feature size with increasing samples, which can oversimplify models. Our theory works for fixed d and n, unbiased by specific parameter choices.
The accuracy of empirical predictions depends on both d and with variance scaling as O Vdn
Larger d and reduce variance, making empirical results more reliable and closer to theoretical values. Conversely, smaller dl and increase variance, affecting single predictions. However, this
scaling is still better than growing indefinitely with fixed d, where variance scales as O leading to increased bias. Implication of the assumptions. In Section 5.2, Figure 3, the Appliance Energy dataset illustrates assumptions' realism. Despite the moderate sample size of 42 samples with 142 dimensions and non-synthetic data, the theoretical curve fits the empirical predictions well. Additionally, with synthetic data (d 100, 100), the theory matches the empirical curve across various hyperparameters (Figure 2), showcasing the predictive power Random Matrix Theory.
B Minimization Problem
B.1
Computation of W, and Wo
The proposed multi task regression finds w [T..... WIJ pdTxx which solves the following optimization problem using the additional assumption of relatedness between the tasks
(W, W. + V, for all tasks 0):
min
[(W.. V)
(7)
(Wo. X x
16

---

where
1 J(Wo. V) 2x (Wo Wo)
tr
(V Vi)
Σ (5)
Ex)
7t
( We
E. = Y
WE (1.
VTd
The Lagrangian introducing the lagrangian parameters for each task t, ar E IR X9 reads as
CWW.. (4, = - tr (Wo Wo) +
1
Σ
tr (El E()
2X
Tt
2
1=1
X())
(Wo + V.)
Σ
tr a;
Y
E.
VTd
1=1
Differentiating with respect to the unknown variables Wo. Vo, E.. ar and b,, we get the following system equation
T
a
W
A
VTd
t=1
1
X X
V
0
76
VTd
6.
0
(()) (()
W
( T V
&: 0
VTd
VTd
Plugging the expression of Wo. V, and & into the expression of ( gives
Y(U (() (1)
( (
+
+ az
Td
Td
which can be rewritten as
X(1) (
X (())
YO = (A+))
a
Qr + ac
Td
Td
wisit
With Y
LYO ..... Y(T) T
Raxq,
a a]
E
R=x4,
Z Σ o !T
(()) E Tuxn, this system of equations can be written under the following [TmmT
compact matrix form:
Q a
with Q
AZ
+ In
E Raxn and A = (Dy + xl117) @ la e RTaTTa
T'
Solving for a then gives:
QY
Moreover, using W,- Wo + V., the expression of W, becomes:
[T] T
AZa
W,
e 3 I
TTd
Zo
Wr = (1T 8 AL)
TTd
17

---

C
Lemma and proof with Random Matrix Theory
C.1
Lemma
Lemma 1 (Deterministic equivalents for Q QMQ and Q² for any M E R Under the concentrated random vector assumption for each feature vector x!" and under the growth rate assumption (Assumption for any deterministic M E Ruxn, we have the following convergence:
+
Q
QMQ + Q2(M).
Q² + Q2
where Q2. Q and Q2 are defined as follows
T
Co (t)
Q
+ ITd
d, Td
tr (2
c0 = A (e)")
m @ x
A
1 1 8;
Q2 (M) QMQ+ +
1
dj
QC
d=
17
1
1/
((M) E RRT
Td
1 +
T'd
1 tr ((QQ((1))
1 tr(C(9
Q)
Q2 = L Diagie (Ust).),
0
Td (1 + 8,)²
Td (1+
where
tr (C")
n tr (CW)C
V(M)=
n1
ERT
V
E RT
Td
1 8c
Td
(1 + 8t) + 6e.)
(E
(
C.2 Deterministic equivalent of the resolvent Q
The evaluation of the expectation of linear forms on Q and Q² can be found in the literature. To find result that meets exactly our setting, we will cite [25] that is bit more general since it treats cases where F[x:"1 * 0 for E [T] and i E [m+]. Unlike the main paper, and to be more general, the study presented below is "quasi asymptotic" meaning that the results are true for finite value of d, Let us first rewrite the general required hypotheses, adapting them to our setting. For that purpose, we consider in the rest of this paper certain asymptotic I C {(d, n), d N. n E N] = 2² satisfying:
{d. 3 n E N: (d, I N
and
{n. (n.3deN: (d,n) E
N.
such that and d can tend to but with some constraint that is given in the first item of Assumption below. Given two sequences (ad,a) dd.e..
> 0, the notation ad, S Obb,,)) (or S (b)) means that there exists constant C > 0 such that for all (d, n) E I, ad,n I Cba,n-
Assumption 3. There exists some constants C, 0 independent such that:
n (dd)
Z -Zi,..., Zn) E RTdxx has independent columns for any (d, n) I, and any f RTdxn - 1-Lipscchite for the euclidean norm:
IP (If E[f(Z)]I 2 6) S -dt
Vi E (n, 3d E N, (d, E T): ||E[z:]| S 0(1)
Theorem 2 ([25), Theorem 0.9.). Given T E N, Z RTdxn and two deterministic A RTxxTd note Q ( A ZZT A Ira)-1. If Z satisfies Assumption 3 and M E RdxxT is deterministic matrix satisfying |M][ S 1, one has the concentration:
P (((((M@) ( Lr(M@s()) (S)) 21) Cc-aa
18

---

where S = (Si,.... S,) (E[zzz] 1, ..... E[znz, for å R", Qs is defined as:
1 A
(s(S)= =
+ Ird
Td
1+ d.
VE[N]
and 8(S) is the unique solution to the system of equations:
-t (A)S.A-QOss)
Vi E [m]:
8(S);
n
We end this subsection with some results that will be useful for next subsection on the estimation of bilinear forms on Q.
Lemma 2 ([25), Lemmas 4.2, 4.6), Under the setting of Theorem 2, given deterministic vector RTd such that ||u|| < (()) and two deterministic matrices U, V such that [U]], S 0(1) and a power r > 0, r S O(1):
E I UUQ Vz: |7] 0())
. T Vz. - Tatr (ΣU ]1" 0 (+))
C.3 Deterministic equivalent of bilinear forms of the resolvent
To simplify the expression of the following theorem, take A Ira- One replace Z with A 3 Z retrieve the result necessary for the main paper.
Theorem 3. Under the setting of Theorem 2, with A = ITa, one can estimate for any deterministic matrices U, V RId such that ||U|, : VI| O(1) and any deterministic vector u. E RTd such
that ||]]] || one notes B V B uv one can estimate:
E [T((Q V(U
Td ((U)
In
1 Td
V(B
00
(8)
V7
where we noted:
Qs(), 8 8(S).
( (1+05)1+0,)
V tे
R
(je((]
tr
tr(UūS;Â)
VU E Raxn; V(U) = T7
E R
1+6.
iE(ni
VU, Rax=; (U, V) 7rttr (UÂVā) e R
If there exist T < n dinstinct matrices Cı,..., Cr such that: (S S.) (CI...., Cr), and if we denote Vt [T] n4 = # E [m] S; C.} and:
P I - nile tr (s,
E RT:
(Td)2
6,)
[.E(TT]
T
tr(S, QUQ) ỘS
VU E TdxTd
(U)= = QuỘ+ + (Td)2
1
(1+ (1++)(1+6,)
t,2=l
the result of Theorem 3 rewrites:
[E QUÂ - و 5o 10(2) (1)
(9)
19

---

Proof. Given E [n], let us note Z-; (Z1,.... 2;-1.0, Z++1
Zn) and Q-; -
Ira)- then we have the identity:
Q-iZi
Q Q
Quizy Q-
and
Q
(10)
Td
1 1 T22; Q-iZi
Given u, RTd, such that ||u||, |lv| I 1, let us express:
E
Td (Q-
UQv
2 E
S:
2 QuQr
(1I)
1+0: Ó,
(12)
First, given E [n]] let us estimate thanks to (10);
E u Quı
[ "_QQQ.QQQuı
1 [ Q,,
Td
Hölder incquality combined with Lemma 2 allows to bound:
E 2 [ /] = (1).
1
E u "2.zTā_.S,QUqq $ Td
1
E u Qz
Td
one can thus deduce:
+0( =E [ +0 +0
J QS,QuuQ) =E -E TE[TQ__S.QUU
(13)
Second, one can also estimate thanks to Lemma 10:
T Q z QUQv
Q z QUQv
E [u QU,zZ QUQQ] =
-E
1 T2z Q-
+ 6
again thanks to Hölder inequality combined with Lemma that allow bound:
8
T7 2 izi
E
u Q-,z,z, [Qzz,,,QQQv)
(1+
8
+
T2
Q Q-izi)
2
1 Td
F [ Q 12,2, [2, QUQve]+ o
S
6
2 22
Vd
The independence between z; and Q (and Q) then allow to deduce (again with formula (10)):
Q2,,, QuQv =E 11 Q S.QQQ .v
1
1 Q 2222 QUQZZ Q-iv
iv
1
+ Td
E
+0
8,
1+ 8
(14)
Let us inject (13) and (14) in (11) to obtain (again with an application of Hölder inequality and Lemma 2 that we do not detail this time):
( S:
1
u Q 22,2, QQQ2ZZZ0 TQ
(1))
E u T
2,2,
F
+
1+
Td
(1+) 1 Q :2
1 E u _
(S,QU~) +0
Td (1+
Vd
Putting all the estimations together, one finally obtains:
tr (QQ
E [ua] - [Quğ] (Td)² 1m1 (1+
E Ô Ō
<
1
(15)
Vd
One then see that if we introduce for any V [Rxx the block matrices:
20

---

0 = 1 L[ Q") (1+8)(1455)
)i.E(()) E ER
Elu(VQQ:
((V) = Ft(
WEENN E ER"
1+..
0(0, V) E (rVVV' ER,
then, if [ & 0(1), multiplying (15) with V and taking the trace leads to:
OUU, V) = VU V) + (U) + +0
(16)
Vd
Now, taking U
S I+5."" one gets the vectorial equation:
1+81
1
1
((V) = (V) + Td
Vo(V) +
Vd
When (T 1 v) is invertible, one gets ((V) = (Tro
1
VV- + (+)) and combining
Ta with (16), one finally obtains:
1
V) (U. + Td
1 (U) (Tra
( +
Td
v7
C.4 Estimation of the deterministic equivalent of Q?
Theorem 4. Under the setting of Theorem 3, one can estimate:
| [q]] I, + De 0
(17)
Vd
with, Vi [n]]
1 tr (s,))
tr (2,Q2(I)))
+ Td (1 6.)
Td (1 8.)2
Proof. The justifications are generally the same as in the proof of Theorem 3, we will thus allow ourselves be quicker in this proof.
Using the definition of Q
AZ Td
+ In)
we have that
Z Z
Z Z
Z Z
Q
I,, I,
+1
I,
Q
(18)
Td
Td
Td
and one then let appear Q thanks the relation:
ZQ QZ,
(19)
that finally gives us:
Q = I, Td
ZT ZQ
I,
z QZ
Td
One can then express:
2 Z QZ
1
Q² I,
Z QZZ Óz
Td
(Td)²
= I
Z
OZ
Z 2
Td
Td
21

---

Given i,j E [n)] i + j. let us first estimate (thanks to Hölder inequality and Lemma 2):
E 2 Q (5
Td L 2 QZ,}
0
Va
Td T 6:) 8,)
since
E[2;]
E|2y]
Now, we consider the case i get:
1 E 7 Q2
tr S
2
Ozi
+0
= Td (1 0.)2 +
Td
Td
(1+
As before, we know that estimate:
Ta
2
Qz)}
0
if i j. Considering i E [n]. thus are left to
Vd
S; (L,))(
1
tr
2
T
Q'z)
+0
Td
Td
(1+
Vd
D
Risk Estimation (Proof of Theorem 1)
D.1 Test Risk
The expected value of the MSE of the test data x E RTxTd concatenating the feature vector of all the tasks with the corresponding response variable y RTx7' reads as
Rinst
E fly g(x))121
xT AZQY Td
[
X
+e-
Td
W
T AZAQQ
W
E)
F
+
VTa
113
T
VTd
Td
X
W
x AZOZ T W
x AZQE
E
+E
T
VTd
TdvTd
Td
tr
EW)
2tr (W EAZOZ
T W)
tr (OTTZOO zZzz AEAZOZ W) (Td)3
E
+ tr (sTe) +
T
Td
(Td)²
ir (E
OZ AΣAZQE)
(Td)²
tr W ΣW)
2tr
(WTEA EA' (Ir ( - :w)
E
T
Td
Td
tr ZZZ AEAZQZ W) tr (E (TTQZ FZTZAAAT AΣAZQE
tr (E E)
(Td)3
(Td)2
tr TTEAA EA'
(IT -)A-àW) 3w)
tr (W ΣW)
tr (W ΣW)
+ +tr E) +
Td
Td
Td
tr
W EA W
w A (A)AA -:
1
tr(EN + O
Td
Td
Td
Vd
The test risk can be further simplified as
W A
QA)A A
ir (EN
1
R = tr (EN)+
+ 0
Td
Td
Vd
22

---

D.2 Train Risk
In this section, we derive the asymptotic risk for the training data. Theorem 5 (Asymptotic training risk), Assuming that the training data vectors x and the test data vectors x(t) are concentrated random vectors, and given the growth rate assumption (Assumption 2),
it follows that:
Rrain 7 tr (W A000A-1--0 -/- 1/2w)
Tn tr (WTA
A 1/2 (I + Tn tr (Ei Q2)
T'
Proof.
We aim in this setting of regression, compute the asymptotic theoretical training risk given
by:
1
Z AZ
R
F
Y
QY
Tn
Td
Using the definition of Q
AZ T4
+ITA
.we have that
ZTAZ AZ Z AZ
Z AZ
-1
T'd Q
=
+ IIr - ITa
+ITa
+
= IIa Q
Td
Td
Plugging back into the expression of the training risk then leads
- E
train
Tn
Using the definition of the linear generative model and in particular
2 W Vid
+ E, we get
1 I w +e)
R
F tr
z w
+e
Q²
Tn
Td
VTd
Tn Td E [ir
w)] + L (Iree
Tn
To simplify this expression, we will introduced the so-called "coresolvent" defined as:
A T A
Q =
Ira
Td
Employing the elementary relation A A QA Z. one obtains:
Td A QAiZQZ = A - zz A
Td ZQZZT
A - A-'Q²A-1,
Td
Therefore further get
Ri Tn
1
E tr W A - -1/ [/]
Tn E tr
W A ----
E E[tr (e Q'e)]
Tn
Using deterministic equivalents in Lemma 1, the training risk then leads to
Riaain Tn
tr [ A -1/2AA --122QA-1/2W) Tn tr (WTA A 1/2 -1/2 (Ir.) + Tn
tr (EN Q2) +
Vd
E Interpretation and insights of the theoretical analysis
E.1 Analysis of the test risk
We recall the test risk as
Rie (En) W
A T
- 1W (A)A
tr ΣN Q2)
1
Td
Td
Vd
The risk composed of a signal term of signal term S W (A and noise term N Q²) T'd
23

---

E.2 Interpretation of the signal term
Let's denote by E = Et= 1= +
nede
E(0 and = Ei= I7d.
E(0)
The signal term reads as
S W A Q2(AAA W.
Using the following identity,
A (A)A A (I+ Σ) A QAQA+
= =(E++I) (I+ Σ) (AΣ+I) +1)
This finally leads
s= WT (AE+1) -1 (I+ (AE+1) -1 w
The matrix H = (A +1) (I+ (ΑE+I) is responsible to amplifying the signal W W in order to the test risk to decrease more or less. It is is decreasing as function of the number of samples in the tasks ng. Furthermore it is composed of two terms (from the independent training W; W) and the cross term W, W. for * v. Both terms decreases as function of the number of samples ny, smaller values of "/ and increasing value of X. The cross term depends the matrix E, E, which materializes the covariate shift between the tasks. More specifically, if the features are aligned Σ, Σ, = I and the cross term is maximal while for bigger Fisher distance between the covariance of the tasks, the correlation not favorable for multi task learning. To be more specific the off-diagonal term of H are responsible for the cross term therefore for the multi tasks and the diagonal elements are responsible for the independent terms.
To analyze more the element of H. let's consider the case where ( = I and & -y In this case the diagonal and non diagonal elements DIL and CMTL are respectively given by
(co + +2) +) +)) caA²
-2o0
DIL (co(A +)) + 1)2 x2
CMTL (co(A 1)=
Both function are decreasing function of X, 1/7 and cp-
E,3 Interpretation and insights of the noise terms
We recall the definition of the noise term A as
N - tr EN EN (A-1
Now at the difference of the signal term there are no cross terms due to the independence between the noise of the different tasks, In this case on the diagonal elements of (A -1 Σ) -1 matters. This diagonal term is increasing for an increasing value of the sample size, the value of A. Therefore this term is responsible for the negative transfer. In the specific case where (() Ia and 7 = y for all task& the diagonal terms read
NAT (co(A+ 7)2 + (A + - co)2)2 + X2
((co(A + + 1)2 -2x22)²
E.4 Optimal Lambda
The test risk in the particular of identity covariance matrix can be rewritten as
Rics DIr. (II 1 + ||>1l + CarTI W W2 NN
Deriving Re with respect to leads after some algebraic calculus to
A n SNR
d 2
where the signal noise ratio is composed of the independent signal to noise ratio and the cross signal to noise ratio SNR [[
W, W
tr
24

---

Theoretical Estimations
F.1 Estimation of the training and test risk
The different theorems depends the ground truth W that needs be estimated through W.
To estimate the test risk, needs to estimate functionals of the form W MWW and e 'Ms for any matrix M. Using the expression of W AZQY, start computing W MW
W MW Y T QZ AMAZQY
Using the generative model for Y
Z VTa
e, obtain
Z W
T
Z Z VTd +&)
E W
MW
=F
+e
QZ AMAZQ
T
W Td
ZQZ AMAZOZ W +E QZ AMAZQE
Employing the elementary relation A 3 zQ = QA Z, one obtains:
Mww]
W
1
W
A Q ZA A MA ZZ W + E [= Oz AMAZQE
Td
E
W WTAAT
A
I Q
A (I- W +E QZ AMAZQE]
E [ MW 2E W MA AA- t
+E W W AWTA-'QAAMAiQA -
+ IF QZ AMAZQE
Using the deterministic equivalent of Lemma 1, we obtain
WWW + W MW 2W A A- QA MW - M(A, W (AA
++ T
M 2A QA 1 M A (A W (A
++ W KT (M + IrE, (A
where We define the mapping RTxxTd + Roxa as follows
K (M) = M 2A 1 - (AAM
F.2 Estimation of the noise covariance
The estimation of the noise covariance remains technical challenge in this process. However, when the noise covariance is isotropic, is sufficient to estimate only the noise variance. By observing that
σ tr Qu
lim
R
train
km
we can estimate the noise level from the training risk evaluated at large y and = 0.
G Multivariate Time Series Forecasting
A version of this work with stronger focus time series applications can be found [20].
G.1 Related Work
Multivariate Time Series Forecasting. Previous work has explored multivariate time series
methods to enrich task representations and enhance diversity across time series applications [18].
25

---

Despite these advancements, random matrix theory, widely applied in multi-tast learning, remains underutilized in multivariate time series contexts, particularly for multivariate time series forecasting (MTSF) [52, 54]. MTSF is common in applications like medical data [6], electricity consumption [49], temperatures [27]. and stock prices [44]. Various methods, from classical tools [7. 45] and statistical approaches like ARIMA [3, 4] to deep learning techniques [8, 19, 32, 38, 40, 51, 53, 55, 56], have been developed for this task. Some studies prefer univariate models for multivariate forecasting [32], while others introduce channel-wise attention mechanisms [19]. We aim to enhance univariate model by integrating a regularization technique from random matrix theory and multi-task learning.
G.2 Architecture and Training Parameters
Architectures without MTL regularization. We follow Chen et al. [8], Nie et al. [32], and to ensure a fair comparison of baselines, we apply the reversible instance normalization (RevIN) of Kim et al. [21]. All the baselines presented here are univariate i.e. no operation is performed by the network along the channel dimension. For the PatchTST baseline [32], we used the official implementation than can be found on Github. The network used in Transf ormer follows the one in [19], using a single layer Transformer with one head of attention, while RevIN normalization and denormalization are applied respectively before and after the neural network function. The dimension of the model is dm = 16 and remains the same all our experiments. DLinearU is a single linear layer applied for each channel to directly project the subsequence of historical length into the forecasted subsequence of prediction length. It is the univariate extension of the multivariate DLinear, DLinearM, used in Zeng et al. [53]. The implementation of SAMf can be found here, and for the i iTransf ormer architecture here. These two multivariate models serve as baseline comparisons. We reported the results found in [19] and [24]. For all of our experiments, we train baselines PatchTST, DLinearU and Trans pransformer former with the Adam optimizer [22], a batch size of 32 for the ETT datasets and 256 for the Weather dataset, and the learning rates summarized in Table 2.
Architectures with MTL Regularization. We implemented the univariate PatchTST, DLinearU, and Transformer baselines with MTL regularization. Initially, we scale the inputs twice using RevIN normalization. The first scaling is applied to the univariate components, and the second scaling is applied to the multivariate components. For each channel, regularization. The outputs are concatenated along the channel dimension, and this concatenation is flattened to form matrix of shape (batch size, q x T), where q is the prediction horizon and T is the number of channels. We then learn a square matrix W of shape (q x T) x (q x T) for projection and reshape the result to obtain an output of shape (batch size, q, T). This method can be applied on top of any univariate model. Our regularized loss has been introduced in 5.4.
then apply model without MTL
Training parameters. The training/validatioonnest split is 12/4/4 months on the ETT datasets and 20%/2%% and d = 512 for DLinearU and Transformer, using a sliding window with stride sequences. The training loss is the MSE. Training is performed during 100 epochs and we use early stopping with a patience of 5 epochs. For each dataset, baselines, and prediction horizon H E (96, 192, 336, 720), each experiment is run 3 times with different seeds, and we display the average of the test MSE over the 3 trials in Table
on the Weather dataset. We use a look-back window = 336 for PatchTST
create the
Table 2: Learning rates used in our experiments.
Dataset
ETTh//ETT2
Weather
Learning rate
0.001
0.0001
G.3 Datasets
We conduct our experiments on 3 publicly available datasets of real-world time series, widely used for multivariate long-term forecasting [8, 32, 51]. The 2 Electricity Transformer Temperature datasets ETTh1, and ETTh2 [55] contain the time series collected by electricity transformers from July 2016 to July 2018 Whenever possible, we refer to this set of 2 datasets as ETT. Weather [27] contains the time series of meteorological information recorded by 21 weather indicators 2020. It should be noted Weather is large-scale datascts. The ETT datasets can be downloaded here while the Weather
26

---

dataset can be downloaded here. Table 3 sums up the characteristics of the datasets used in our experiments.
Table 3: Characteristics of the multivariate time series datasets used in our experiments.
Dataset
ETTh//ETT12
Weather
features
7
21
time steps
17420
52096
Gramularity
hour
10 minutes
27

---

G.4 Additional Experiments.
0.1
G-021
5.001
3:15
0.15
D.00
PatctTor Danellne
1.01
DutesTor Zeaeline
PutchTor Emaellne
0.240
00.920
30.460
00.420
0.210
0.355-
50.150
0.420
0.160
0.420
Lanbda
(a) Dataset ETTh1, Horizon 96
(b) Dataset ET Horizon 96
(c) Dataset Weather, Horizon 96
0.540
rage NSE.
0. 0.370
0.240
0.320
0.2210
10.520
30.260
0.460
0.325
0.230
0.460
0.220
10
200
0.569
10
207
20
Lanbda
Lamboa
Lambaa
Dataset ETThI, Horizon
(e) Dataset ETTh2, Horizon 192
(1) Dataset Weather, Horizon 192
b.250
0:215
20.210
keetage
0.300
00:265
0.480
20.300
20.200
0..33
00223
2.225
0.333
Lanbda
Lanbda
Lambda
(g) Dataset ETThl, Horizon 336
(h) Dataset ETTh2, Horizon 336
(i) Dataset Weather, Horizon 336
0.400
0.250
0.554
6.41.
40
g,0.220
0..540
Peragg M3R
2.403
0.400
80:030
0.340
0.440
0..30
0.440
0.440
0.320
0.325
0.318
Lambda
Lambda
Lanbda
(j) Dataset ETTh1, Horizon 720 (k) Dataset ETTh2, Horizon 720
()) Dataset Weather, Horizon 720
Figure 4: Results of optimization method different datasets and horizons averaged across different seeds for each gamma and lambda values for the PatchTST baseline
28

---

0.0
0.425
0-230
00:200
0.259
0.360
0.210
Averege
0.385
0.380
0.290
0.200
0.315
0.133
0.310
0.150
Lambda
Lanada
Lambds
(a) Dataset ETTh1, Horizon 96
(b) Dataset ETTh2, Horizon
(c) Dataset Weather, Horizon 96
0.490
0.280
0.450
0.260
30.460
0.430
2.340
0.520
00.220
0.410
107
307
10%
(d) Dataset ETTh1, Horizon 192
(c) Dataset ETTh2, Horizon 192
(f) Dataset Weather, Horizon 192
0.500
0.450
5.060
Average MSE
0.460
6.400
seergge
0.400
2.400
0.280
0.440
0.280
0.260
1014
103
1.0
107
124
105
102
Lamoda
Lanoda
Lanida
(g) Dataset ETThI, Horizon
(h) Dataset ETTh2, Horizon 336
(i) Dataset Weather, Horizon 336
0.540
0.540
g
0.523
0..00
6.590
80.260
0.480
6.450
0.350
0.000
0.425
0.340
0.330
12.4
107
202
12:4
102
207
10
10:3
102
10%
305
Lambda
Lanada
Lambda
Dataset ETTh1, Horizon 720 (k) Dataset ETTh2, Horizon 720
(1) Dataset Weather, Horizon 720
Figure 5: Results of our optimization method on different datasets and horizons averaged across different seeds for each gamma and lambda values for the DLinearU baseline
29

---

0.520
0.000
0.350
0.440
segee MSC
0.344
0.260
8..40
0.322
0.2240
0.420
00390
0.450
0-240
0.230
0.180
3022
100
107
Lembod
Lamodd
Lambods
(a) Dataset ETTh1, Horizon 96
Dataset ETTh2, Horizon 96
(c) Dataset Weather, Horizon 96
0.640
00.40
MSE
0.520
0.000
0.440
2.600
0.240
0.440
Duzed
0.430
0..40
00.238
0.410
0.250
1000
10
10
10
124
10
1.04
307
10
1040
1.0%
105
Lambda
Dataset ETThI, Horizon 192
(c) Dataset ETTh2, Horizon 192
(f) Dataset Weather, Horizon 192
0r 0.050
E 2.4460
0.520
0.110
5.420
Average MB3
0.200
0.340
0.440
26,000
00.470
0.460
0.300
0.350
0.430
0.280
0.289
0.440
0.200
Lambdd
.anbda
Lambda
(g) Datasct ETTh1, Horizon 336
(h) Dataset ETTh2, Horizon 336
(i) Dataset Weather, Horizon 336
0.200
0.250
00.000
0.310
0.520
br.140
ho.3e0
0.008
00.50
0.450
CL640
0.460
0.338
0.490
1014
1002
10.5
10%
307
12:1
10
10%
207
307
Lambda
Landda
Lambda
Dataset ETTh1, Horizon 720
(k) Dataset ETTh2, Horizon 720
(1) Dataset Weather, Horizon 720
Figure 6: Results of our optimization method on different datasets and horizons averaged across different seeds for each gamma and lambda values for the Transf ormer baseline
30

---

H
Limitations
While the study provides valuable insights through its theoretical analysis within linear framework, it is important to acknowledge its limitations. The linear approach serves as a solid foundation for understanding more complex models, but its practical applications may constrained. Linear models, though mathematically tractable and often easier to interpret, might not fully capture the intricacies and nonlinear relationships present real-world data, especially in the context of multivariate time series forecasting.
To address this limitation, we decided extend algorithm's 's application to complex models, specifically within the nonlinear setting of neural networks. This transition aims to evaluate whether the theoretical insights derived from the linear framework hold true empirically when applied to neural networks. As part of this endeavor, an optimal parameter lambda was selected by an oracle, leading to promising results, as detailed in Section 5.4. This oracle-basee selection underscores the potential efficacy of our approach when appropriately tuned, even in more complex, nonlinear contexts.
is important to note that the limitations are not related to the real-world data itself, as our setting performs well in the context of multi-task regression for real-world data, shown in Section 5.2. The difficulty arises from transitioning from a linear to a nonlinear model. The results in Section 5.4 are particularly encouraging, demonstrating that our method can improve upon univariate baselines by regularizing with an optimal lambda, as indicated by our oracle. While the oracle provides an upper bound on performance, actual implementation would require robust methods for hyperarrameter optimization in non-linear scenarios, which remains an open area for further research.
By expanding the scope of our theoretical framework to encompass nonlinear models, we pave the way for future work that could focus on the theoretical analysis increasingly complex architectures
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
32

---

Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
33

---

Answer: [Yes]
Justification: The datasets used are open-source and all the implementation details are given to reproduce the experimental results. The code will be made available in case of acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. cc/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. cc/ccc/p/lcc sec/public//ae Seriiddes/ooeSuuu bilbmmsssionii sssissssisiycoooooooo for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
34

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification:
All details to reproduce experiments are given.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https http///eeurrppppeeeee httpp/////pccttcicccccieeieeeeee
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of work, none which we feel must be specifically highlighted here.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
35

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
36

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
37


Reviews:
[{"review_id": "m4mx4qDx0N", "reviewer": "Reviewer_7acX", "summary": "The authors derived theoretical insights into the train and test risks of the multi task regression loss using Random Matrix Theory.", "questions": "1. What is $\\gamma$ in (3)? \n2. Have you studied the implications of the assumptions made to use random matrix theory to derive the expressions of asymptotic train and test errors?\n3. Shouldn't it be $\\mathbf{Y}\\in \\mathbb{R}^{Tq\\times n}$ in the equations after line 110?", "limitations": "Limitations are not discussed. My guess is that the assumptions made to be able to employ Random Matrix Theory need to be discussed.", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 4, "strengths": "The closed form solution of the optimization parameter using RMT is novel. The authors set a trend in MTR to gain analytical expressions for optimization parameter and error risks.", "weaknesses": "1. The paper lacks clarity for instance on line 89, I am not sure what is meant by general optimization and other mathematical tools?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer 7acX for their valuable feedback and for recognizing our work's novelty.\n\nWe address the reviewer's concerns point by point below.\n\n>1. The paper lacks clarity for instance on line 89, I am not sure what is meant by general optimization and other mathematical tools?\n\nWe are happy to address the reviewer’s question concerning clarity.\n\nRegarding the optimization problem, we emphasize that our approach is more general than previously proposed methods that focus on two tasks and use only a hyperparameter $\\lambda$ under Gaussian assumptions [1]. Our method considers both a hyperparameter $\\lambda$ to relate all $T$ tasks and specific parameters $\\gamma_t$ to balance overfitting and underfitting within each task, accommodating non-Gaussian distributions.\n\nIn Sections 4.2 and 4.3, we highlight the benefits of our theoretical results, particularly the influence of these parameters on performance, including their significance in the decomposed terms of theoretical risk (signal and noise terms). To our knowledge, this is the first work to develop such a theory and provide direct insights into the hyperparameters’ impact on performance. Furthermore, our theory supports model selection for these hyperparameters.\n\nOur framework assumes a concentrated random vector for the data and operates in a big data context (where both sample size and feature size are large). Consequently, we employ various tools, relying on recent developments in Random Matrix Theory, such as deterministic equivalents and concentration inequalities, as detailed in the appendix. \n\nWe propose changing the original sentence for greater clarity to the following: \"*Our work focuses on the selection of hyperparameters within a general optimization framework, considering both a hyperparameter $\\lambda$ to relate all tasks and specific parameters $\\gamma_t$ to balance overfitting and underfitting within each task, accommodating non-Gaussian distributions and employing recent developments such as deterministic equivalents and concentration inequalities from Random Matrix Theory.*\" \n\nDoes this seem clearer and more accurate to the reviewer?\n\n\n[1] Precise High-Dimensional Asymptotics for Quantifying Heterogeneous Transfers. Fan Yang, Hongyang R. Zhang, Sen Wu, Christopher Ré and Weijie J. Su. (2023). \n\n\n>2. What is $\\gamma$ in (3)?\n\n$\\gamma$ is a vector whose dimensions correspond to the number of tasks, encompassing all the hyperparameters $\\gamma_1, \\ldots, \\gamma_T$. Therefore, $\\gamma = [\\gamma_1, \\ldots, \\gamma_T]$. \n\nThis hyperparameter serves as a regularization for each task, indicating how the model should overfit (small values of $\\gamma_t$) or underfit (high values of $\\gamma_t$) on the specific task. The reviewer can refer to Sections 4.2 and 4.3 for a discussion on its influence, where its impact on the two components of the test error (signal and noise terms) is highlighted. Specifically, smaller values of $\\gamma_t$ are favorable for increasing the signal term and decreasing the noise term.\n\nThis definition of $\\gamma$ will be added to the paper just before (3) for enhanced clarity.\n\n>3. Have you studied the implications of the assumptions made to use random matrix theory to derive the expressions of asymptotic train and test errors?\n\nWe kindly ask the reviewer to refer to the general comment for a detailed discussion on our assumptions. Additionally, recent studies have also utilized the concentration assumption [2, 3, 4].\n\n[2] Random Matrix Theory Proves that Deep Learning Representations of GAN-data Behave as Gaussian Mixtures. Mohamed El Amine Seddik, Cosme Louart, Mohamed Tamaazousti and Romain Couillet. (2020).\n\n[3] Word Representations Concentrate and This is Good News. Romain Couillet, Yagmur Gizem Cinar, Eric Gaussier and Muhammad Imr an. Proceedings of the 24th Conference on Computational Natural Language Learning. (2020)\n\n[4] Deciphering Lasso-based Classification Through a Large Dimensional Analysis of the Iterative Soft-Thresholding Algorithm. Malik Tiomoko, Ekkehard Schnoor, Mohamed El Amine Seddik, Igor Colin and Aladin Virmaux. ICML (2022).\n\n>4. Shouldn't it be $\\mathbf{Y} \\in \\mathbb{R}^{T q \\times n}$ in the equations after line 110?\n\nEach response variable $Y^{(t)}$ for task $t$ has dimensions $q \\times n_t$ , where $q$ is the prediction horizon length and $n_t$ is the size of the dataset for task $t$ . In other words, $Y^{(t)}$ aggregates all $n_t$ predictions for task $t$ , with each prediction outputting a $q$-dimensional vector. The vector $Y$ concatenates the predictions of all tasks along the sample dimension, meaning it aggregates all $n_t$ samples, with $n = \\sum_{t=1}^T n_t$. This justifies the size $q \\times n$.\n\n>5. Discussion about the assumptions made to be able to employ Random Matrix Theory\n\nAs detailed in our response to Question 3, we emphasize that our two main assumptions are more realistic for real-world machine learning and big data scenarios, where both sample size and feature dimension are large and of the same order of magnitude.\n\nAddressing such complex data distributions is theoretically challenging and requires new tools and concentration inequalities, which we believe adds valuable theoretical insights to our paper. For instance, the derivation of deterministic equivalents under this concentration framework is a notable novelty. Moreover, the close fit between our theoretical predictions and empirical results on real-world datasets (e.g., Appliance Energy) in Section 5.2 demonstrates the realism of our data assumptions.\n\nIf this explanation satisfies the reviewer's concerns, we can incorporate these clarifications about the assumptions underlying our use of Random Matrix Theory to enhance overall clarity and understanding.\n\nWe hope that the reviewer's concerns and questions have been addressed and we remain open to future discussions. Given our explanations, we would be grateful if the reviewer could reconsider the evaluation of our work accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3Re4fxpWHH", "reviewer": "Reviewer_8z1C", "summary": "The authors characterise the train and test risk of multi-task regression using random matrix theory. Assessment via Figures 2 and 3 shows a good match between theory and empirical. This motivates a regularised objective for learning mutivariate time-series.", "questions": "1. The analysis of the noise term in section 4.2 says that increase in noise negatively affects transfer. However, it is precisely in the presence of  noise that borrowing statistical strength from other tasks is suppose to help. How do we reconcile this?\n\n2. In section 4.3, is it clear that $C_{MTL}$ is always positive, negative or no such conclusion can be made?", "limitations": "Authors should include some technical limitations of their current work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The theory contribution in section 4 is significant and original, and the assessment in sections 5.1 and 5.2 are significant.", "weaknesses": "*Three major weaknesses*\n\nA. There are at least two existing works on the theoretical error bounds for multi-task learning [Chai, NeurIPS 2009; Aston & Sollich, NeurIPS 2012]. While these are using Gaussian processes as a basis, comparison or remarks on the general effect of multi-task learning should be discussed with reference to two existing works. This is especially when all these are essentially linear-in-regressors models.\n\nB. The proof or at least outline of the proof of Theorem 1 *should* be in the main paper, given that this is *the* major contribution of the paper.\n\nC. It is totally unclear the relevance and contribution of the theory itself to section 5.3. Adding a cross-task regularization is already known to help (mostly).\n\n*Minor, but affects clarity*\n- The introduction to the paper is on multi-task multi-response/regression. The multi-response/regression part is not made clear.\n- Why is there a need to divide by $\\sqrt{Td}$ in (1)? Is it to make analysis easier.\n- There are two different uses of the letter $t$ in Assumption 1.\n- The setup in section 5.3 needs to relate back to the model in section 3.1 to make clearer their connections.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer 8z1C for their thoughtful feedback. We are happy to read that the reviewer found the theory contribution significant and original. \n\nWe address the reviewer's concerns point by point below.\n\n> A. There are at least two existing works on the theoretical error bounds for multi-task learning [Chai, NeurIPS 2009; Aston & Sollich, NeurIPS 2012]. While these are using Gaussian processes as a basis, comparison or remarks on the general effect of multi-task learning should be discussed with reference to two existing works. This is especially when all these are essentially linear-in-regressors models.\n\nWe appreciate the reviewer’s recommendation.\n\nThe assumptions between our work and the two mentioned works differ. We assume that the vectors we work with are concentrated, while the mentioned works consider Gaussian data, which may not capture the full complexity of real-world data. Additionally, we assume that the dimension $d$ and the sample size $n$ grow together, such that $n = O(d)$, which is a more realistic assumption. As more samples are collected, a greater diversity of features can emerge. In contrast, the mentioned works assume a fixed feature size as $n$ increases.\n\nTherefore, the tools we use are different, even though they are complementary: the mentioned works derive theoretical bounds, whereas we derive exact performance metrics.\n\nFinally, we propose a model selection method derived from theory and validated experimentally. We apply this approach for the first time to multivariate forecasting, which presents a significant challenge.\n\nIf this response satisfies the reviewer, we can incorporate these comparison elements into our paper.\n\n\n> B. The proof or at least outline of the proof of Theorem 1 should be in the main paper, given that this is the major contribution of the paper.\n\nWe thank the reviewer for this valuable suggestion. We agree that Theorem 1 is central to our contribution, and we will ensure that the proof of Theorem 1 is included in the main paper to enhance its clarity and impact.\n\n> C. It is totally unclear the relevance and contribution of the theory itself to section 5.3. Adding a cross-task regularization is already known to help (mostly).\n\nWe appreciate the reviewer’s insightful comment and kindly ask the reviewer to refer to the general comment for a detailed discussion on the relevance and contribution of the theory to section 5.3\n\nWe hope this explanation clarifies the relevance of our experiments and their connection to our theoretical framework. If this satisfies the reviewer’s concerns, we can incorporate these clarifications into the paper to enhance clarity and understanding.\n\n> D. About the minor changes\n\nWe thank the reviewer for their comments. We have clarified the multi-response/regression part in the introduction. We divide by $\\sqrt{Td}$ as a scaling parameter to facilitate subsequent calculations. We appreciate the typo notice and have adjusted the use of $t$ for clarity. For Section 5.3, we will incorporate these connections into the paper to smooth the transition.\n\n> 1. The analysis of the noise term in section 4.2 says that increase in noise negatively affects transfer. However, it is precisely in the presence of noise that borrowing statistical strength from other tasks is suppose to help. How do we reconcile this?\n\nIndeed, as the noise increases, a growing $\\lambda$ will penalize multi-task regularization more, focusing on independent tasks. Conversely, $C_{MTL}$ is always negative and becomes more negative as $\\lambda$ increases. These two terms are thus in competition: excessive MTL regularization will lead to an increase in noise, despite the tendency of $C_{MTL}$ to decrease. It is essential to find a compromise, which is the goal of our Figure 1, where $\\lambda^*$ represents the best possible trade-off between $C_{MTL}$ and the noise term.\n\nWe hope this clarifies our approach and addresses the reviewer's concerns.\n\n> 2. In section 4.3, is it clear that $C_{MTL}$  is always positive, negative or no such conclusion can be made?\n\n$c_0$ is the limit of $n/d$ and we assume that $n = O(d)$. Therefore, the limit $c_0$ is strictly positive, making $C_{MTL}$ always strictly negative i.e.- the more aligned the different tasks are, the more this cross term tends to reduce the test risk.\n\n> 3. About the technical limitations of the current work.\n\nA limitations section is present in our Appendix H, focusing on the tractability of our linear approach compared to nonlinear models. Our experiments in multivariate time series forecasting aim to show that the theoretical insights presented in Figure 3 for linear models also hold for nonlinear models. This serves as a preliminary step towards future work on applying our theory to nonlinear models.\n\nWe can also discuss the formulated assumptions for using Random Matrix Theory in this section. These assumptions are more general compared to previous studies, especially since we do not assume Gaussian data. The two major assumptions are: first, that the data are concentrated random vectors, which is more realistic for modeling real-world data, and second, that the dimension $d$ is of the same order of magnitude as the sample size $n$, which is also a realistic assumption differing from classical asymptotic regimes where the sample size $n$ tends to infinity while the feature size remains fixed.\n\nWe propose adding these discussions to our limitations section, in addition to the existing discussion on the tractability of extending our study from linear to nonlinear models. Does this answer satisfy the reviewer's concerns about the limitations of our work?\n\nWe hope that we have adequately addressed the reviewer's concerns and questions and remain open to further discussion. We sincerely appreciate the reviewer’s reconsideration of our work based on these explanations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zKWmhyukzG", "reviewer": "Reviewer_Zuy6", "summary": "The authors analyse the problem of multi-task regression (for a linear model) under the assumption of concentrated random vectors.  The results obtained provide a comprehensive description of the performance of the model (including critically its generalisation performance).  The authors use this result for hyperparameter optimisation and show on real-world datasets that using the insights from their theoretical model they obtain state-of-the-art performance on multivariate time series forecasting.", "questions": "Is there an intuition on why the concentrated random vector assumption justified?  Is it that real data satisfies this assumption accurate, or is that the deviations from this assumption don't seem to be relevant?\n\nYou are tackling real world data using a relatively simple linear model and beating what I assume are sophisticated non-linear models.  I find this surprising.  Is there an explanation of why you do so well?  (Are the datasets you are using very simple--linear?,  are the number of examples small compared to the number of features so that more complex models will overfif?).", "limitations": "This is fine.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "I feel that this is a really nice achievement.  Computing generalisation results are notoriously difficult.  Such calculations are usually confined to rather simplistic models.  The model chosen is rather simplistic in being linear and make, what seems to me, a very strong assumption about the data.  Nevertheless, by choosing the appropriate setting (i.e. a multi-task scenario) and it appears that the concentration assumption seems to innocuous they come up with a useful result.  Given how hard it is to do meaningful theory in machine learning I certainly believe this paper deserves to be accepted.\n\nIt should also be noted, that the paper is technically accomplished.", "weaknesses": "The paper will not be everyones taste.  The mathematics is not easy to follow, but that is often the nature of making a non-trivial technical contribution.\n\nIn the equation for $\\lambda^*$ immediately preceding section 4.4 there is a missing norm symbol.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the rewiever Zuy6 for their positive feedback and for recognizing that our work is a nice achievement and technically accomplished. This type of feedback is very encouraging. \n\nWe answer below the reviewer's questions.\n\n> 1. In the equation for  immediately preceding section 4.4 there is a missing $\\lambda^*$ norm symbol.\n\nWe thank the reviewer for pointing out this typo. We have corrected the missing $\\lambda^*$ norm symbol in the equation immediately preceding Section 4.4.\n\n> 2. Is there an intuition on why the concentrated random vector assumption justified? Is it that real data satisfies this assumption accurate, or is that the deviations from this assumption don't seem to be relevant?\n\n\nWe appreciate the reviewer's question regarding the intuition behind the concentrated random vector assumption. This assumption is justified for several reasons:\n\nFirstly, the strong performance of neural networks on tasks like image recognition and NLP suggests that these models produce stable predictions. As Lipschitz transformations, neural networks maintain controlled distances between inputs, ensuring stability.\n\nSupporting this assumption, recent studies have rigorously demonstrated that several real-world datasets, as well as those synthetically generated with GANs, exhibit concentration properties [1, 2, 3]. This makes our assumption more realistic than traditional Gaussian assumptions, which often fail to capture the complex dependencies and structures present in high-dimensional, real-world datasets. In contrast, our approach focuses on the stability of statistical properties after transformation rather than the specific shape of the data distribution, providing a more flexible and realistic framework for analyzing complex data structures. \n\nWe hope this response adequately addresses the reviewer’s questions regarding the concentrated random vector assumption and its implications.\n\n\n[1] Random Matrix Theory Proves that Deep Learning Representations of GAN-data Behave as Gaussian Mixtures. Mohamed El Amine Seddik, Cosme Louart, Mohamed Tamaazousti and Romain Couillet. (2020).\n\n[2] Word Representations Concentrate and This is Good News. Romain Couillet, Yagmur Gizem Cinar, Eric Gaussier and Muhammad Imr an. Proceedings of the 24th Conference on Computational Natural Language Learning. (2020)\n\n[3] Deciphering Lasso-based Classification Through a Large Dimensional Analysis of the Iterative Soft-Thresholding Algorithm. Malik Tiomoko, Ekkehard Schnoor, Mohamed El Amine Seddik, Igor Colin and Aladin Virmaux. ICML (2022).\n\n> 3. Discussion about the good performance of our regularized linear models\n\nWe observe that linear models are highly effective and serve as a strong baseline in time series forecasting. This fundamental observation, coupled with the fact that many approaches use independent univariate predictions, where each feature contributes only to its own prediction, explains the strong baseline performance of our univariate DLinear model. We then apply our regularization method to this model, achieving state-of-the-art results. Several factors contribute to the exceptional performance of this regularized linear model:\n\n1. **Overfitting in Non-linear Models**: Many non-linear forecasting models, especially those based on transformers, tend to overfit [4], even when the number of features is much smaller than the number of samples. For instance, in our datasets, the number of features is 7 for ETTh1/ETTh2 and 21 for Weather, while the number of samples is in the tens of thousands for each dataset. Linear models are less prone to overfitting in these scenarios. However, our gains are evident not only in simple linear models but also in more complex transformer models such as *Transformer* and *PatchTST*, indicating the robustness of our regularization method.\n\n2. **Nature of the Data**: While it is possible that some relationships between features in the multivariate benchmarks we use may be linear, as in the Weather dataset that may exhibit some features with linear physical relationships, it is not straightforward to conclude that a purely linear model would perform well on these benchmarks. However, it is well-known that linear models can perform exceptionally well in forecasting tasks.\n\n3. **Our regularization method**: It is common for multivariate models in forecasting to aggregate univariate predictions made channel by channel. In our regularization method, we introduce a parameter $\\lambda$ to control the multivariate component and parameters $\\gamma_t$ for each task to determine the extent of overfitting we allow for a specific task. This dual approach enables us to effectively manage both the local (task-specific) and global (multivariate) aspects of each prediction, providing a balanced and flexible regularization framework.\n\nIn summary, the good performance of our regularized linear models can be attributed to a combination of these factors. \n\n[4] SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention. R. Ilbert, A. Odonnat and al. ICML 2024.\n\nWe hope that our comments have provided more insights in line with the reviewer’s feedback. We remain open to any further questions or suggestions upon the reviewer’s request.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NF4LMjBVqr", "reviewer": "Reviewer_EGU1", "summary": "This paper analyzes a linear multi-task regression framework. It derives formulas for asymptotic train and test risk. The formulas provide insights in how the raw data covariances, singal-generating hyperplanes, noise levels, and size of data sets affect the risk. Motivated by the analysis on the linear framework, experiments on multivariate time series forecasting are performed to investigate the effectiveness of MTL regularization.", "questions": "- Is there any connections between the theoretical analysis and the experiments? E.g. making use insights from the analysis to improve experimental results.\n- Is splitting the linear operator into a local and global term a novelty of this paper?\n- Similarly, is this paper the first work to apply this MTL framework (decompose prediction as a sum of global and local term) to nonlinear models?\n- How are the regularization hyperparameter chosen in the experiments?\n- In some of the results, incorporating MTL regularization worsen the performance. Why is that?", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is clearly written. It outlines main points and is easy to follow. \n- Throughout analyses, which use random matrix theory, on the multi-task linear regression framework are performed. Specifically, the asymptotic train and test risks are derived for analyzing the behavior of the framework.\n- The analyses provide useful insights into the behavior of models.", "weaknesses": "- The analyses are performed on a simple linear model. They do not apply to general nonlinear models, which are much more common in real practice.\n- There seems to be a disconnection between the theoretical analysis and the experimental results. The experiments seem to only highlight the effectiveness of MTL regularization and have no connection with the theoretical analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer EGU1 for their detailed feedback and for recognizing that our paper is clearly written while providing useful insights.\n\nWe address the reviewer's concerns point by point below.\n\n> 1. About the generalization to nonlinear models and the connection between the theoretical analysis and the experimental results\n\nWhile non-linear models are widely used, establishing their theoretical foundations is challenging. Linear models in multivariate TS forecasting are highly competitive and often approach state-of-the-art performance, which is why we focused on them for our theoretical study. They provide robust insights into the behavior of more complex non-linear models, which we explore experimentally. We show that for linear models, we can theoretically derive train and test risks by identifying components of signal, cross-term, and noise, helping us find an optimal regularization parameter $\\lambda$ to minimize test risk.\n\nExperimentally, we find that test risk curves for non-linear models follow similar patterns to our theory, as non-linear models often use a linear output layer in time series forecasting. Under the assumption of data concentration, the Lipschitz nature of neural networks ensures that the outputs of the non-linear part do not deviate significantly from the inputs.\n\nFor a detailed discussion on the connection between our theory and Section 5.3, we kindly ask the reviewer to refer to our general comment.\n\nDoes this response make the connection between our theoretical study and the experimental results, as well as the generalization to non-linear models, clearer for the reviewer?\n\n> 2. Is splitting the linear operator into a local and global term a novelty of this paper?\n\nWe thank the reviewer for this question. Splitting the linear operator into local and global terms is indeed a novelty of our paper. \n\nMost previous works focus on global regularization [1], managing multivariate information use. Our approach controls both global and task-specific information, allowing for more refined and flexible regularization, enhancing the model’s balance of general and task-specific information in regression and forecasting.\n\n[1] Precise High-Dimensional Asymptotics for Quantifying Heterogeneous Transfers. Fan Yang, Hongyang R. Zhang, Sen Wu, Christopher Ré and Weijie J. Su. (2023).\n\n> 3. Similarly, is this paper the first work to apply this MTL framework (decompose prediction as a sum of global and local term) to nonlinear models?\n\nTo the best of our knowledge, our work is the first to apply this MTL framework, which decomposes prediction into a sum of global and local terms, to nonlinear models such as transformers. This approach allows us to leverage the strengths of both local task-specific adjustments and global shared information, particularly in complex scenarios like multivariate time series forecasting.\n\n> 4. How are the regularization hyperparameter chosen in the experiments?\n\nWe thank the reviewer for this question. In our experiments, we selected hyperparameters based on the test risk. However, our primary goal was to demonstrate our framework's ability to leverage multi-task information and showcase its potential for future work with non-linear models. We chose pairs of regularization hyperparameters $(\\lambda, \\gamma)$ and plot the test risk curves of our non-linear models to observe if they match the expected behavior from our linear model theory. When we selected the best hyperparameter pair on the test set, we achieved significant gains. \n\nThe main aim is to compare linear and non-linear models. We found that test risk curves for non-linear models in time series forecasting resemble those of linear models, indicating consistency with our theoretical findings. This suggests our framework could extend to non-linear models, optimizing regularization parameters similarly.\n\nDoes this explanation help clarify our approach and findings for the reviewer?\n\n> 5. In some of the results, incorporating MTL regularization worsen the performance. Why is that?\n\nWe appreciate the reviewer’s insightful question on why results sometimes worsen with multi-task learning (MTL) regularization, even when deriving the optimal $\\lambda$ to minimize test risk. This concern is valid, as one would expect that using test data to find the optimal $\\lambda$ should consistently yield comparable or better performance.\n\nThe key reason is the discretization of $\\lambda$ values. In our experiments, we discretize $\\lambda$ over a small range rather than exploring all possible values continuously. This may miss the precise optimal value, especially if the discretization steps are too coarse, leading to suboptimal performance. Our goal was to show that even with coarse discretization, our method yields robust results, not to fine-tune $\\lambda$ for the best performance. However, when tasks are highly independent, performance may degrade if the chosen $\\lambda$ does not perfectly align with the optimal value.\n\nThis situation occurs in only a few horizons and datasets. For most datasets, our approach shows substantial gains, demonstrating its overall effectiveness. These observations support our goal of motivating further analysis of non-linear methods, suggesting that exploring non-linear extensions could capture similar performance benefits as shown in Table 1. The current experiments highlight the challenge of developing theoretical analyses for non-linear methods that match or exceed our results, opening promising research avenues.\n\nIn summary, the occasional worse performance is due to $\\lambda$ discretization. Our method generally shows strong results, and finer discretization can mitigate these issues. Exploring theoretical approaches for non-linear methods is the next exciting step.\n\nWe hope our responses have clarified the reviewer’s comments. We are happy to provide further details if needed and invite the reviewer to reconsider their evaluation if we have addressed their questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# **General Comment**

We thank all the reviewers for thoroughly and carefully reading our paper. We are deeply grateful for their recognition of the **novelty** (Reviewer 7acX), **originality and significance** (Reviewer 8z1C) of our contribution, and for acknowledging it as a **really nice achievement** (Reviewer Zuy6), noting that **computing generalization results are notoriously difficult** (Reviewer Zuy6) and that our analysis **provides useful insights into the behavior of models** (Reviewer EGU1).

We remain open to continuing this constructive discussion for the length of the rebuttal period and strongly believe that the paper benefited from the reviews.

To clear out any possible misunderstanding regarding the assumptions of our work and the connection with Section 5.3, we provide clarifications below.

### **About our assumptions**:
We believe these assumptions are more realistic for analyzing real-world machine learning algorithms compared to traditional ones like Gaussian data and fixed feature size with infinite sample size, which don't capture the full data complexity.

**Assumption 1**. Data are concentrated random vectors. Intuitively, this means that high-dimensional data points have stable properties when transformed by complex functions (more formally Lipschitz functions). 

The strong performance of neural networks on tasks like image recognition and NLP suggests that these models produce stable predictions. As Lipschitz transformations, they maintain controlled distances between inputs, ensuring stability. 

Recent studies have demonstrated and experimentally confirmed that both real-world data and synthetically generated data using GANs exhibit concentration properties, supporting this assumption. This makes our assumption more realistic than traditional Gaussian assumptions, as it does not rely on specific hypotheses about the shape of the data distribution, but rather on the stability of statistical properties after transformation. Consequently, analyzing a framework of concentrated random vectors is more theoretically challenging than using Gaussian assumptions and represents a key novelty of our theory.

**Assumption 2**. The dimension $d$ is of the same order of magnitude as the sample size $n$. This joint growth captures data complexity better than assuming a fixed feature size with increasing samples, which can oversimplify models.

Our theory works for fixed $d$ and $n$, unbiased by specific parameter choices. The accuracy of empirical predictions depends on both $d$ and $n$, with variance scaling as $O\left(\frac{1}{\sqrt{dn}}\right)$. Larger $d$ and $n$ reduce variance, making empirical results more reliable and closer to theoretical values.

Conversely, smaller $d$ and $n$ increase variance, affecting single predictions. However, this scaling is still better than $n$ growing indefinitely with fixed $d$, where variance scales as $O\left(\frac{1}{\sqrt{n}}\right)$, leading to increased bias.

**Implication of the assumptions**. In Section 5.2, Figure 3, the Appliance Energy dataset illustrates our assumptions' realism. Despite the moderate sample size of 42 samples with 142 dimensions and non-synthetic data, the theoretical curve fits the empirical predictions well. Additionally, with synthetic data ($d=100$, $n=100$), the theory matches the empirical curve across various hyperparameters (Figure 2), showcasing the predictive power of Random Matrix Theory.

### **About the connection with Section 5.3**: 

While non-linear models are widely used, establishing theoretical foundations for these models is challenging. Therefore, we focused on linear models, which, despite being simpler to study, can provide valuable insights into the behavior of more complex models.

Our study demonstrates that we can calculate the train and test risks for linear models by identifying components of signal, cross-term, and noise. These components help us find an optimal regularization parameter, $\lambda$, to minimize the test risk.

Experimentally, we aimed to observe if our theoretical insights apply to real-world data when using non-linear models. Our results show that the test risk curves for non-linear models follow similar patterns to those predicted by our theory. This is expected because, in time series forecasting, non-linear models typically utilize a linear output layer to project onto the prediction dimension. Thus, we can apply our theory to the inputs of the final linear layer of the model. This approach is valid because, under the assumption of data concentration, the outputs of the non-linear part of the model should not deviate significantly from the inputs due to the Lipschitz nature of a non-linear neural network.

Moreover, multivariate time series prediction models often treat each channel separately using univariate methods. These models could greatly benefit from incorporating information across multiple channels. The results in Section 5.3 show that our method can surpass univariate baselines by regularizing with optimal $\lambda$ and $\gamma$. This supports the applicability of our theory to non-linear models, as the final linear layer can leverage the concentrated inputs effectively.

Finally, our regularization approach differs from traditional cross-task regularizations that typically use one task per dataset. We consider each prediction within a dataset as a task and introduce $\gamma_t$ parameters alongside $\lambda$. These parameters not only enforce multivariate regularization but also indicate the degree to which we want to underfit or overfit a particular task. Moreover, this method is tractable as it is applied on top of the model at the final layer.

The fact that the curves for non-linear models closely resemble those for linear models indicates that our findings are robust and that non-linear models also exhibit optimal regularization parameters, enhancing performance in multivariate forecasting.

Meta-review: The paper presents a framework for multitask regression and precisely characterizes train and test risk incurred by the resulting estimator, leveraging random matrix theory. The approach is demonstrated on both multitask regression and multivariate time series forecasting. 

The AC and majority of reviewers find the proposed framework and analysis to be significant and exciting. 

We strongly encourage the authors to incorporate the discussion materials in their manuscript, and in particular their excellent points to justify their assumptions. We also urge them to add a small paragraph at the beginning of section 5.3, to reflect their rebuttal comments on the relevance of their theoretical insights beyond the case of linear models.
Decision: Accept (spotlight)