Attention boosted Individualized Regression
Authors: Guang Yang
Yuan Cao
Long Feng
Venue: NeurIPS 2024 poster (2024)
Keywords: Individualized regression
Vector correlation
Brain imaging data
Self-attention mechanism
Primary area: machine_learning_for_healthcare

Abstract: Different from classical one-model-fits-all strategy, individualized models allow parameters to vary across samples and are gaining popularity in various fields, particularly in personalized medicine. Motivated by medical imaging analysis, this paper introduces a novel individualized modeling framework for matrix-valued data that does not require additional information on sample similarity for the individualized coefficients. Under our framework, the model individualization stems from an optimal internal relation map within the samples themselves. We refer to the proposed method as Attention boosted Individualized Regression, due to its close connections with the self-attention mechanism. Therefore, our approach provides a new interpretation for  attention from the perspective of individualized modeling. Comprehensive numerical experiments and real brain MRI analysis using an ADNI dataset demonstrated the superior performance of our model.

Attention boosted Individualized Regression
Guang Yang
Yuan Cao
Department of Data Science College of Computing City University of Hong Kong guang yang@mynggay cityu. edu. hk
Department of Statistics and Actuarial Science School of Computing and Data Science The University of Hong Kong yuancao@hku. hk
Long Feng"
Department of Statistics and Actuarial Science School of Computing and Data Science The University of Hong Kong 1fengühku. hk
Abstract
Different from classical one-model-fits-al strategy, individualized models allow parameters to vary across samples and are gaining popularity in various fields, particularly in personalized medicine. Motivated by medical imaging analysis, this paper introduces novel individualized modeling framework for matrix-valued data that does not require additional information on sample similarity for the individualized coefficients. Under our framework, the model individualization stems from an optimal internal relation map within the samples themselves. We refer to the proposed method Attention boosted Individualized Regression, due to its close connections with the self-attention mechanism. Therefore, our approach provides new interpretation for attention from the perspective of individualized modeling. Comprehensive numerical experiments and real brain MRI analysis using an ADNI dataset demonstrated the superior performance of our model.
Introduction
Model-based machine learning methods have advanced significantly and become essential in modern data analysis. From linear regression to deep neural networks, most approaches fundamentally follow an one-model-fitsse not change for different samples. However, fields like medical diagnosis and treatment design, is important to explore and upply individualized models with parameters tailored to each sample, adapting their unique features. Due the heterogeneity among instances, individualized models are expected to provide more accurate predictions and personalized interpretations, which are their main advantages.
strategy, meaning that parameters of a well-trained model are fixed and do
Individualized modeling has been extensively investigated in research, with the earliest example possibly being the varying coefficient models [TO 6 in statistics community. A varying coefficient model usually includes an additional variable and represents the varying coefficient as function of this extra variable. These models have been applied and adapted in various contexts. For instance. [ explored spatial modeling using a spatially varying coefficient process. In a similar vein, [26] considered varying coefficient models in image response regression and proposed using deep neural networks to estimate the varying coefficients.
'Long Feng is the corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Beyond varying coefficient models, recent studies have also incorporated prior knowledge of sample similarity to regulate sample-specific coefficients. The fundamental assumption is that the similarity among coefficients for different samples relies on the sample similarity, meaning that the more similar the samples, the closer their coefficients. For instance, [24 tackled personalized medical models using a multi-task learning approach called FORMULA, assuming that models for similar patients are close and achieving this through Laplacian regularization. [25] developed the localized Lasso, which assumes known weighted network over samples that reflects the distance in parameter space. [IS] loosened the aforementioned prior assumption by considering additional covariates and assuming the existence of some measurement of similarity corresponding to similarity in parameter space. Moreover, they constrained the matrix of personalized parameters to be low-rank, so closeness in loadings implies closeness in parameters. While effective in various contexts, these methods heavily rely on the prior knowledge about parameter similarity, which might not be readily available in numerous real-world applications.
This paper aims to develop a novel individualized modeling framework for matrix-valued data, without the need for additional information on sample similarities. In our framework, model individualization is derived from the heterogeneity inherent in the samples themselves. Specifically, we seek to find an optimal sample-specific internal relation map to enhance model fitting and interpretation. The sample-specifii relation map allows us to capture the local dependence between patches within each matrix input, thereby enhancing prediction performance and model interpretability.
It is worth noting that the proposed individualized modeling framework with sample-specific internal relation map is highly connected to the self-attention mechanism [21], which has demonstrated its exceptional performance various field, including natural language processing, computer vision, and more. Due to such connection, we named the proposed framework Attention boosted Individualized Regression (AIR). Therefore, our approach could also provide a new interpretation for attention from the perspective of individualized modeling.
We should emphasize that our proposed approach is particularly well-suited for applications in personalized medicine and brain connectomics analysis, which initially motivated us to study individualized modeling.
recent years, the field of brain connectomics has experienced rapid growth due to advancements in medical imaging technology. This area of study focuses on examining comprehensive maps of connections within the human brain, playing a vital role in cognitive neuroscience, clinical diagnosis, and more. Brain networks can be represented by relation matrices, often established based on connections among regions of interest (ROIs). Besides sample features, the internal relationships within each sample may also influence relevant responses. This consideration has been addressed in the literature, such as [18 J. In this context, differentiated internal relations can emphasize heterogeneity among subjects, providing individual-level information about brain connectivity. This effect supports the use of internal relations in individualized models and further contributes to personalized medicine.
2 Attention boosted individualized regression
Given any matrix M, we first introduce matrix reshaping operator that allows us explore the internal relations within M. Let M have dimensions D1 x D2 and let d1, d2 be factors of Dj. D2. Define (p1, P2) (Di/d1 D2/d2). We can now define the operator (.) RD DixDa ((di)s))((dda)) as a mapping from M to
Red = vec (MTr) ....., vec (MAL))) P1.12
j.k represents the (j. k-)tth block of M with size dj x dz. The operator Rea, () essentially vectorizes each of the Pi P2 block (of size d, x d2) and stacks the vectorized blocks together. Denote the inverse operation of R.(d, as R Rd, (di,da) (:)- A special case occurs when (d1, d2) D2). in which case we have R(I,D.) (M)- R,,,D, (M) M. It is worth noting that this reshaping operation has also been applied in attention mechanisms, allowing us examine the relations or correlations among the P1 x P2 patches.
where d
Suppose we observe n samples with scalar outcomes yi E and matrix inputs X e Drxx> for i I, n. Given block size (d1, d2), we first reshape the original images to obtain
2

---

x; = (x:" E Rexd where PiP2 and d = d1d2. Then we consider the following individualized linear regression model with coefficient matrices varying across samples
1/ (X; Ei, i 1 n,
where C₁ E Rexd the unknown individualized coefficient matrix for i-th sample and E4 is the noise term. Note that the reshaping operation dz) (-) is one-to-one. Thus, model (2) is equivalent to + E, with (² = R (d) da) (C.). As previously mentioned, model (2) type of individualized regression has been studied under various constraints on the individualized coefficients, such as [10 25
= (x ((rr)))
In this paper, we propose to model C; with two components: a homogeneous coefficient C reflecting common effects and heterogeneous coefficient D, containing individualized information. Specifically.
C; = C Di, i= 1......,
(3)
For the heterogeneous coefficients, we further assume that they share an unknown common factor D across samples,
D. = A.
(4)
Here, A; represents unknown sample-specific factors serving as re- re-weighting matrix to aggregate the coefficients D, where the transpose is better connect with self-attention mechanism later. base coefficient matrix for the heterogeneous effects.
The matrix D E Roxd can be viewed Clearly, additional constraints on the individual factor A, are necessary to ensure the identifiability of the model. The choice of factor A; may vary depending the purpose. In this paper, we propose an ineeral-relation-boosstt A, E RRxx of the form
individualized factor for matrix-valued inputs. Specifically, we consider
A gX,
(5)
where W Rdxd is an unknown matrix to be learned, while g(-) RPXP - RPXP is known function that preserves dimension, of which different forms to be discussed. is worth recalling that X; = d2) (X.") E Roxd is the reshaped matrix. The reshaping operation (1 enables us to calculate the "generalized correlation" between patches through (5). When fixing W I and
setting g()) as the identity function, A, = X, x,X, reduces to standard covariance matrix of patches within i-th sample when X, is properly centered.
In the formulation (5), the individualized matrix A, is designed to capture the internal relationships among the p rows of reshaped matrix (or p patches original matrix) for each sample. Relations between two vectors can be measured in different ways, such correlation, similarity, distance, etc. Our formulation of (5) is motivated by the rotation correlation introduced by 20). For any two vectors and v, the rotation correlation is defined as
max u HV. H
where the matrix H is usually required to be orthogonal. This rotational correlation aims to find the maximized correlation between u and v with the best possible rotation. When is the identity matrix and ||u|||2 2 ||0||2 = 1, the rotation correlation reduces standard Pearson correlation. We note that
the (j. -th element of the sample-specific factor can be written as {A+}a = {X W ( where ( (X;), and {X,} are the j-th and k-tth rows of X;, respectively. In other words, {At)
related to the rotation correlation between [x; d. and (X; ).. However, our goal is not to maximize
the correlation between { {Xi),. and {X; h/ but find the optimal rotation that achieves the best fitting for the responses.
Combining (5), we obtain individualized model in the following form
1/4
(x +
Kig( X T D)
+
(6)
homagenecuus
heterageneous
Here, C' Rpxd D Rexd and W E ddd are the coefficient matrices that need be learned.
The decomposition of 6 allows us to understand and assess the individuation degree of each sample
3

---

and the entire model. At the sample level, a larger magnitude of the heterogeneous part indicates that the sample more distinctive, affected by its internal relations. At the model level, the larger the magnitude of the homogeneous part, the closer the model is to an ordinary linear model, and vice versa. Naturally, achieving proper balance between the two parts contributes to a better model fit.
We shall note that model (6) could be easily extended to a generalized linear model (GLM) setting to accommodate other types of outcomes. For example, by allowing certain link function f(-), may consider a GLM of the form f(E(y)) = (X;, Ct). Then, the coefficients C; could still be modeled as in to (5)_
To learn the coefficients C, D and W, we propose the following penalized minimization problem
min C.D.W
y3 x, C()) + A ICIF 22 DI
i=1
s.t. C C (X;
|W|| 1,
where || || is the Frobenius norm and Àι and A2 are regularization parameters to balance the homogeneous and heterogeneous effects. Besides, a norm constraint for W is also required due to identifiability consideration. We defer to Section for the computation of
3 Individualized regression and attention
We refer to our individualized modeling as Attention boosted Individualized Regression due to its connections with the self-attention mechanism. The self-attention mechanism was proposed in the seminal work (21), and the Transformer model based on it has demonstrated exceptional performance in natural language processing, computer vision, and other fields. In this section, establish the connection between the proposed model (6) and the self-attention mechanism.
Given the input X E Raxd the Scaled Dot-Product Attention mechanism computes the output using Q E Rexd K E Raxde, and V E Rand, representing query, key, and value, respectively. The three essential components are linearly transformed from X by
Q XW q, = X V = XWv
with corresponding weight matrices W Q. W K, and Wy. Incorporating softmax function for normalization, the Scaled Dot-Product Attention is defined as
QK"
f(X)= softmax
V.
(8)
ddz
In the attention mechanism, the first part softmax
QK
essentially computes the pairwise similarity
between queries and keys, normalized by a combination of scaling and row-wise softmax. With the resulting attention map. the output is obtained by reweighing the pairs' values. The attention map is provides an individualized map that captures information
the core of the attention mechanism, on pairwise similarity within each sample.
Moreover, the attention mechanism (8) could also be expressed in row-wise form. Let O Attention Q, K, V) be the output of the attention function. Further let o,, q; k; and v; be the i-th row of O, Q, K, and V. respectively. Then, (8) is equivalent
ΣS exp (q. k2/ d
Σ exp (q k./Vdz)
This form clearly highlights that the basis of the weights in the attention map is formed by vector correlation. fact, the dot-product-based pairwise similarity is derived from a nonlinear transformation of the correlation between pairs. Beyond softmax function, normalization in attention could also be accomplished using general function g(-). As a result, we obtain the following generalized attention
f(x) (QK
(10)

---

The attention mechanism in the form of (10) with nonlinear function g(-) can face computational challenges, as the direct computation of attention maps requires significant resources handle matrices. To address the computation issues, several recent works have emerged, such as sparse transformers 44, efficient transformers [13,, and more. Linear attention mechanisms have been studied as subcategory, which can dramatically decrease complexity from quadratic to linear. [T6] proposed a linear attention boosted on the first-order Taylor expansion of the exponential part in the softmax function, i.e., exp(q k) & 1+ 1 k. [12] presented the linearized attention using kernel functions, which measure the similarity between q and k through ((q) (kk). In this case, Φ(-) represents a specific kernel function. [22] introduced Linformer, which leverages the low-rankness of the attention map to reduce complexity to linear. Notably, [9 considered linear p(Y) = Y /n as scaling normalization, consequently,
f(x) -
(11)
Linear attention mechanisms are efficient because they bypass the need to compute n X n matrices by using associative multiplication, reducing complexity from O(1²) to (n)). While on the other hand, experiments show that Linear attentions does not result in significant compromise in performance.
Now we demonstrate the connections between our individualized regression model ( and the self-attention mechanism. We let the homogeneous coefficient C' = 0 and focus on the model
V1 Proposition 3.1. Suppose the model decomposed below
4, g X (
D) + E4-
(12)
12 holds and matrices W and D in model 12 could be
k (I) W WQ W T for two matrices W Q, WK E Raxds with dk S d,
(II) D BW v for two matrices B, W v E Bdddd with de S d.
Then, the following equation holds for each sample X;
Xi ((X,WX]) -0100, B).
(13)
where
= - =;= K W.WK K: V,=
Remark 3.2. Proposition establishhe the connection between our individualized regression model (12) and the self-attention mechanism (1O). We shall note that the product of the query and key Q.K. X, X.W., essentially acts internal relation map, capturing the inter-dependence between local patches. By applying an appropriate function g(),, we can obtain the normalized sample- specific internal relation map. Furthermore, the value matrix Vi be enhanced by multiplying with such relation map. The final outcome is obtained as the inner product of the aggregated features and the coefficient matrix.
It is important to note that the conditions in the proposition mild, they correspond the low-rank assumptions: (I) rank (W) I dg and (II) rank(D) S dg.. In particular, (I) is consistent with Theorem in (22), which demonstrated that the self-attention mechanism, i.e., the attention matrix, is low-rank. Moreover, if assumption (II) is not considered, (13) becomes equivalent to the simplified Vision Transformer [T] without considering the value.
Conversely, the equivalence (33) also helps understand our model from the perspective of the self-attention mechanism. Since the tuple (Wa Qu WK. K. Wv) represents embedding projections in self-attention, W W @ k W is equivalent to composite embedding that adaptive and determines the attention map. Meanwhile, D = BW v represents a projected regression coefficient that can be learned as a whole. The heterogeneous coefficients D; = XXx TT D can be considered as an aggregation of base coefficients through the attention map, contributing model interpretation. If we set g(-) as the identity function, (13) simplifies to línear attention, thus enjoying the computational
advantages of linear attention.
We should also note that with the growing popularity of transformers in natural language processing, iiatteentionbbseee architectures have begun to be introduced in computer vision, encompassing
5

---

various visual tasks such as detection, segmentation, and generation. However, we mainly discuss their initial involvement in regression and classification tasks [23 [5. In particular, [5] directly applied pure transformer to address the image classification problem and proposed Vision Transformer (ViT).
ViT treats images as sequences by dividing them into fixed-size patches and processes them using transformer architecture. ViT comprises two main components: the Encoder and the Classifier. In the transformer encoder, each attention map is computed for each image based on patch-wise similarity. The embedded patches are then followed by multilayer perceptron head that serves as regressoo//lassiffer. Although some details are not discussed, this simplification helps to understand the connection with model. More recently, [I]] proposed simplifying transformer blocks. By removing skip connections, value parameters, sequential sub-blocks and normalization layers, the simplified transformer has the potential to achieve fewer parameters and faster training.
4 Computation
In this section, we demonstrate the computation of the penalized minimization problem 7 From now we shall focus on the special case where g(z) = x is the identity function, which corresponds to linear attention. Namely, the model is
= (X, X x D) +
(14)
In this context, we develop an alternating minimization algorithm and highlight its benefits compared to gradient-based ones. First, we observe that the heterogeneous part in model (1) satisfies
X W.W x D x DX x. W (x, X W X, D)
(15)
Moreover, let w vec( and d vec( D) be the vectorization of W and D. It holds that
DX X, w) Z,, wdd where Z, x x,
X
(16)
and @ denotes the Kronecker product. Clearly, (16) displays bilinear form. We start our algorithm by initializing w as the top left singular vector Y4 Z, Formally,
= SVD,
Σ y: Z
(17)
where SVD represents the top left singular vector of a matrix.
Now we introduce our alternating minimization algorithm. Denote 2((1)
((')) and W as the
(4)
iterates in 1-th loop. According to (15). we alternatively update (0')) D and W as below.
Given W (1-1) denote (( x,
X X;, Then (0(0) ..0)) can be updated by
O00 ("."")) arggiin C.D
Σ yi - - ([x U(-") c, D))) + +1I11C| X2
n
(18)
Clearly, (18) can be seen as ridge-like regression with levels of penalization on distinct coefficients, which has an explicit solution shown in Section Alin the appendix.
Given (0(0) D then W
be updated by
W (4)
1 Σ y x,, -x0x X. w))²
argmin
(19)
W
1r.
(20)
implies that W could be obtained easily through ordinary least squares followed by normalization.
We summarize the alternating minimization algorithm in Algorithm in Section A in the appendix.
In practice, the regularization level (Ai. A2) are treated as hyperparameters and we can use cross-
validation to search for the optimal combination.

---

5 Theoretical analysis
In this section, provide theoretical guarantees for Attention boosted Individualized Regression. Specifically, we show that W and ( obtained by alternating minimization algorithm converge to the true counterparts at geometric rate. To simplify analysis, we focus on the heterogeneous part of model (T4), although our results can be extended to more general cases. Suppose that
y X x, W x D) +
(21)
Let w = veo(W) and d = vee( D), the optimization problem could be written as
Σ - xx 8 X ..wddd)
wdd)) }} + Xu /2]ld
min d, n 1m1
(22)
which is non-convex on w and d. For the rearranged images X; for i = I,..., n., we define
Z = (vec { (xTxx & x
vec (x @ X xx)
(23)
Here each row of Z represents a transformed sample. For the new feature matrix Z, we suppose the following RIP condition.
Condition 5.1. (Restricted Isometry Property) For each integer 1, 1,2,.... matrix P R is said to satisfy the r-RIP condition with constant 8, E (0, 1), if for all M E Rux of rank at most it holds that
(1 & - (/cM) S (1
M
(24)
The Restricted Isometry Property (RIP) was initially introduced by 2 for sparse vector recovery and subsequently extended by (17) for low-rank matrices, as in Condition Many random matrices with an adequately large number of independent observations, such as Gaussian sub-Gaussiag matrices, satisfy the RIP condition [17]. In our analysis, we require that Z defined in (23 satisfies the 2-RIP condition with constant 82-
To evaluate the estimation error of parameters, we consider an angle- based distance between two matrices. Formally, for any two matrices U and V with the same dimension, we define the distance This distance metric corresponds to the squared
as dist(U, V)= = /1 (U, /CIUU P. VVP).. sine value after vectorization, that is, dist(U, V) = sin,u, v), where vec(U) and v = vec(V). Now we are ready to present our main theorem.
Theorem 5.2. Suppose model (27) holds and solved by alternating minimization algorithm. Assume
(0)
that Z satisfies 2-RIP Condition 5.1 with constant 52. Denote pi ddist
W
W
as the initial
distance. Let K1 po/2+ 352/ (1 382) and pao22 + (35g + >22) 35 + X2) and assume
K1, R22 < 1. And Tı, 72 are noise related terms. Suppose K1/00 + TI S 140 and K2/40 + 72 S pg. Then,
after f. iterations we have
(C)
N1 + Ti
dist
W
W
< (NIK2)
uo +
(25)
1- K1
D (4)
K2T1 +
dist
D
k2p0
(26)
K1
Theorem 5.2 suggests that the estimation errors of (()) and D (t) converge at a geometric rate, with the contraction parameter being K2. On the right-hand-side of (5) and (26). the first term represents the optimization error, while the second term represents the statistical error. It becomes evident that the optimization error decays geometrically with each iteration t.
Theorem 5.3. Suppose model (27) holds and solved by alternating minimization algorithm. Assume
that Z satisfies 2-RIP Condition 5.1 with constant d2. Denote ph = W| (0)
WW]F as the initialization error. Let V1 = 2410 + 3020/11 - -322 and v2 = 2po + (362 + 22) 36, + A2), and assume vi, 92 <. 1. And TI, 72 are noise related terms. Suppose v1 + S pn and U2110 + T2 S nr-- Then, after iterations we have
18 (8) Yl1a S 333DD (w11z))
( +
T1 71
(27)
1- U142
7

---

Theorem 5.3 suggests that the prediction error decreases in a similar manner as the estimation errors in Theorem 5.2 It is important note that the error bounds in both theorems are dependent on suitable initialization. We employ spectral initialization as shown in (77. which has been proven to have an error closely approximating the true value.
6 Simulation
We conduct extensive simulation studies to evaluate the performance of our Attention boosted Individualized Regression compared related methods in this section. Besides, ablation studies deferred to Section B. in the appendix to show the advantage of combining the homogeneous and heterogeneous parts. Throughout the simulation, we assume that the data generated according to the model (6). The size of the images is set to 28 x 28, with a sample size of 4000 for training and 1000 for testing. The noise e, follows an i.i.d. No, 1). The coefficient matrices C and D are generated as two circles depicted in Figure [ For the images X;;", we assume that internal relations exist among blocks of size 4 within each image, where two blocks at random locations are correlated. The entries in X follow i..d (0, 1), while the correlated blocks are generated using the two methods below.
Case 1: With specific W, the internal relations are subject to (5). Consider low-rank W 2-u; 1. + ++-11uu u2vy where , and v1, 02 are random vectors with entries subject to i.i.d. (0, 1). Then, the correlated blocks are generated u, plus noise vectors with i.i.d. entries from (0 0.25).
Without specific W, the internal relations are Pearson correlation coefficients. Given a random vector u base with i.d. entries from ((0, 1), the correlated blocks are also generated as u plus noise vectors with i.i.d entries from (0, 0.25). Then A, is taken as the correlation matrix where (j, k)-th element of A, is the Pearson correlation coefficient between j-th and k-th blocks within X
Case 2:
Furthermore, we consider different levels of model individualization and investigate the effects on model performance. To this end, we define the degree of individuation (DI) of model (6) by the relative total magnitude of the heterogeneous part and homogeneous part. Specifically,
DI =I=v2 -i=vEii X, DX C)Z.
The performance of AIR is compared with four competing methods, including, low-rank matrix regression [LRMR,2] 27), tensor regression with lasso penalty [TRLaaso. (28), Deep Kronecker Network [DKN, 77, and Vision Transformer [ViT, [5,, respectively. Implementation details are provided in Section B.2 the appendix. Of note is that we cannot implement several individualized regression
methods 25 14 as they require additional information of unknown variables. We evaluate prediction performance of different methods, measured by the root mean squared error (RMSE) on test set:
/ (1/Thea) V7e - yz, The average and standard error of 100 repetitions are reported in Table T and the estimated coefficients of different methods are illustrated in Figure I [nd 4
Table 1: Prediction errors of different methods.
Methods TRLasso
DI
AIR
LRMR
DKN
VIT
0.5
4.422 (3.130)
6,616 (0.020)
8.215 (0.021)
4,886 (0.018)
18.429 (0.049)
Case 1
1.0
8.102 (0.325)
13.101 (0.040)
14.655 (0.044)
7.028 (0.032)
18.351 (0.047)
2.0
10.599 (0.816)
26.239 (0.081)
27.007 (0.085)
11.741 (0.043)
24.098 (0.069)
0.5
3,590 (0.046)
6,766 (0.018)
8,337 (0.021)
8,269 (0.018)
24.492 (0.063)
Case 2
1.0
6.632 (0.022)
13,408 (0.037)
14.739 (0.039)
14,964 (0.034)
29.939 (0.084)
2.0
13.002 (0.044)
26.864 (0.074)
27.484 (0.073)
28.686 (0.060)
44.036 (0.111)
The numerical results indicate that AIR outperforms all other methods, with the advantage increasing as the degree of individuation becomes greater. Figure | aand@demonsttaal that AIR, when solved by our algorithm, can accurately recover the shape of the true parameters. It is worth noting that in Case 2, even though our model is mis-specified with no explicit W exists, AIR still performs well. Common-mood
methods such as LRMR, TRLasso, and DKN tend to estimate the sum of the true coefficients for both parts. On the other hand, ViT typically requires a large number of samples and is thus not as effective due to the limited sample size.
8

---

True C
True D
Truc W
C from LRMR
C from TRLasso
Con
Co0
Cor
Figure I: Case simulation results with DI 1.0. The first three columns show true parameters and estimations from AIR. The last two columns show estimations from other methods except VIT, has no explicit coefficient matrix. An additional OLS estimation is added for reference.
7 Real data analysis
In this section, we analyze the relationship between cognitive assessment scores and brain MRI data from the Alzheimer's Disease Neuroimaging Initiative (ADNI). The ADNI is study Alzheimer's disease (AD) that includes clinical, genetic, and imaging data, covering AD patients, individuals with mild cognitive impairment (MCI), and healthy controls. We collected a total of 1059 subjects from ADNI and GO/2 phases with
State Examination (MMSE) score and brain MRI. The MMSE score measures patient's s cognitive impairment which can assist in the diagnosis of AD. Brain MRI were carefully preprocessed following a standard pipeline involving denoising, registration, skull-striping and so and were resized to tensors of size 48 x 60 x 48 for computation efficiency. Then we extracted middle coronal slices for each subject, resulting in images of size 48 48. Two samples are shown in the first column in Figure
We compare the methods described in the simulation section by 5-fold cross-validation in test RMSE, of which average and standard error are presented in Table AIR exhibits the best prediction performance among all methods, of which the significance can be shown by paired t-test. Furthermore, Figure compares estimations of different methods while illustrates the individualized estimations
from AIR for two different subjects, including the heterogeneous effect D ori and significant internal relations. To screen significant internal relations for each subject, we summarize relations of each node in the internal relation matrix A; and select top 5 as significant nodes. Subsequently, we mark these nodes corresponding locations in the original sample by red boxes and show their relations by a chord diagram. For example, the block (4, 5) in sample has the strongest relations, and is related to both (6, 4) and (6, 5), indicating the important relations between corpus callosum and
hippocampus. We also note that after separating heterogeneous effect, the homogeneous effect 8 highlights regions the hippocampus, which have been acknowledged in medical literature as crucial substructure associated with Alzheimer's disease . By this means, we can find important regions and relations among them for each subject, which is potential to help personalized treatment. In contrast, other methods do not reveal clear shapes and fail offer valuable interpretations.
Table 2: Prediction errors of different methods.
AIR
LRMR
TRLasso
DKN
VIT
3.145 (0.019) 3.715 (0.008) 3.292 (0.023) 3.261 (0.017) 3.282 (0.025)

---

Siptificant smuple
Sumple
2 for comple
2 from AIR
2 from LRMR
2.
SiquiEeat wicrasi sclatious
Sumple
5 for comple
2 from (MLasse
2 fion D3K
4.5
Figure 2: Results ADNI dataset. (I) Column shows two original samples. Column shows heterogeneous coefficients estimated by AIR. Column presents chord diagrams that illustrate the significant internal relations estimated by AIR. Each coordinate in the chord diagram corresponds to red box marked in the sample. (II) Columns 4 and 5 compare the homogeneous coefficients estimated by AIR with the coefficients, obtained from other methods.
8 Discussion
In this paper, we present an Attention boosted Individualized Regression model that emphasizes internal relationships within samples and is based the concept of rotation vector correlation. Our method is specifically tailored for data with heterogeneous internal relationships. By concentrating on the internal relations within samples, heterogencous nature of data, making it highly beneficial for various fields, particularly, brain imaging analysis and personalized medicine. On the other hand, realize that the AIR framework also has limitations. First, its capability to handle general data is more or less restricted. When there are minimal heterogeneous effects, its performance will be similar to ordinary linear model. Second, as discussed earlier, our framework could be viewed as simplified version of the Vision Transformer; however, such simplifications may also reduce its approximation power for Furthermore, this paper primarily investigates the linear form of AIR. Although the linear form performs well in the cases of interest, remains worthwhile to explore the generalization of the model in future work.
approach effectively addresses the complex and
complex scenarios.
Acknowledgments and Disclosure of Funding
We thank the anonymous reviewers for their helpful comments. Yuan Cao is supported by NSFC 12301657 and Hong Kong RGC grant ECS 27308624. Long Feng is supported by Hong Kong RGC grant GRF 17301 123 and ECS 21313922.
10

---

References
[1] M. Ball, V. Hachinski, A. Fox, A. Kirshen, M. Fisman, W. Blume, Kral, H. Fox, and H. Merskey. A new definition of alzheimer's disease: hippocampal dementia. The Lancet, 325(8419):14-16, 1985.
[2] E. Candes and T. Tao. Decoding by linear programming, IEEE transactions on information theory, 51((2):4203-4225, 2005.
[3] M. Chen, A. Radford, R. Child, J. Wu, H. Jun, D. Luan, and I. Sutskever. Generative pretraining from pixels. In International conference on machine learning, pages 1691-1703. PMLR, 2020.
[4] R. Child, S. Gray, A. Radford, and I. Sutskever. Generating long sequences with sparse transformers, arXiv preprint arXiv:1904.105000 2019.
[5] A. Dosovitskiy, L. Beyer, Kolesmikov, D. Weissenbomn Zhai, Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 1 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
[6] J. Fan, Q. Yao, and Z. Cai. Adaptive varying-coefficiet linear models. Journal of the Royal Statistical Society Series B: Statistical Methodology, 65(1):57-80, 2003.
[7] L. Feng and G. Yang. Deep kronecker network. arXiv preprint arXiv:2210.13327, 2022.
[8] A. E. Gelfand, H.JJ Kim, C. Sirmans, and S. Banerjee. Spatial modeling with spatially varying coefficient processes. Journal of the American Statistical Association, 98(462):387-36, 2003,
[9] S. Guha and A. Rodriguez. Bayesian regression with undirected network predictors with application to brain connectome data. Journal of the American Statistical Association, 1 116(534):5815599 2021.
[10] T. Hastie and R. Tibshirani. Varying-coefficient models. Journal of the Royal Statistical Society: Series B (Methodological), 55(4):757-779, 1993.
[11] B. He and T. Hofmann. Simplifying transformer blocks. arXiv preprint urXiv:2311.01900 2023.
[12] A. Katharopoulos, A. Vyas, N. Pappas, and F. Flcuret. Transformers are mns:: Fast autoregressive transformers with linear attention. In International conference machine learning, pages 5156-5165. PMLR, 2020.
[13] N. Kitaev, L. Kaiser, and A. Levskaya. Reformer: The efficient transformer. arXiv preprint arX1v:2001.04451, 2020.
[14] B. Lengerich, B. Aragam, and E. P. Xing. Learning sample-specific models with low-rank personalized regression. Advances in Neural Information Processing Systems, 32, 2019.
[15] B. J. Lengerich, B. Aragam, and E. P. Xing. Personalized regression enables sample-specific pan-cancer analysis. Bioinformatics, 34(13):i178-i186 2018.
[16] R. Li, J. Su, C. Duan, and S. Zheng. Linear attention mechanism: An efficient attention for semantic segmentation. arXiv preprint 2rXiv:2007.14900 2020.
[17] B. Recht, M. Fazel, and P. A. Parrilo. Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM review, 52(3):471-50,, 2010.
[18] J. D. A. Relión, D. Kessler, E. Levina, and S. F. Taylor. Network classification with applications brain connectomies. The annals of applied statistics, 13(3):1648, 2019.
[19] Z. Shen, M. Zhang, H. Zhao, S. Yi, and H. Li. Efficient attention: Attention with linear complexities. In Proceedings of the IEEF/CVF winter conference on applications of computer vision, pages 3531-3539, 2021.
[20] M. Stephens. Vector correlation. Biometrika, 66(1):41-48, 1979.

---

[21] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and
I. Polosukhin. Attention is all you need. Advances neural information processing systems, 30, 2017.
[22] S. Wang, B. Z. Li, M. Khabsa, H. Fang. and H. Ma. Linformer: Self-attention with linear complexity. arXiv preprint 000::00006.4478. 2020.
[23] X. Wang. R. Girshick, A. Gupta, and K. He. Non-local neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 7794-7803, 2018.
[24] Xu, J. Zhou, and P.-N. Tan. Formula: Factorized multi-task learning for task discovery in personalized medical models. In Proceedings of the 2015 SIAM International Conference on Data Mining, pages 496-504. SIAM, 2015.
[25] M. Yamada, Koh, T. Iwata, J. Shawe-Tay Taylor, and S. Kaski. Localized lasso for high- dimensional regression. In Artificial Intelligence and Statistics, pages 325-333. PMLR, 2017.
[26] D. Zhang, L. Li, C. Sripada, and J. Kang. Image response regression via deep neural networks. arXiv preprint arXiv:2006.09911, 2020.
[27] H. Zhou and L. Li. Regularized matrix regression. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 76(2):463-483 2014.
[28] H. Zhou, L. Li, and Zhu. Tensor regression with applications in neuroimaging data analysis. Journal of the American Statistical Association, 1085502) 2013.
12

---

Appendix
A Computation
The pseudocode of the alternating minimization algorithm is summarized in Algorithm As T is a ridge-like regression problem and updating
mentioned in Section updating
W ordinary least squares problem, both of which have explicit solutions. For the former, we consider the vectorzied version of the problem (32) With bold lowercase letters being the vectorization of corresponding matrices, we have
1 - Σ (e) u.,)
(
a
argmin
+
(28)
c,d
V1
==
argmin
NB 3 A
(29)
A1
where 3 stores all coefficients, N is the new design matrix within this step and A includes different intensities of penalization. Therefore, (29) has the following solution
0
A21
3 (N N Y
(30)
Similarly, the vectorization of 34 implies its OLS solution as below
(M M) M
w argmin
Mw
(31)
where Y is the response minus homogeneous part and M is the new design matrix within this step.
Algorithm 1 Alternating minimization algorithm
Input: X;, yt 1.....n. n.
Initialize w (0) = SVD, (E 2. repeat
100 000
Σ i=1
y/ - ([x, U(- [c. D)))² + + + D
argmin
C.D
n1
(32)
(0)
X (x
W
argmin
Vi
(33)
W
W
||-
(34)
until Converges or reaches maximal iterations.
Output: (T) D (T) (7)
B
Experimental extras
B.1
Ablation studies
We conduct ablation studies in this section to investigate the effects of homogeneous part and heterogeneous part. Specifically, we compare
(1) AIR: 34 (X,, C) + X D.) e. subject to D. X.WX, D. D.
13

---

(2) Hetero: yi (.) tE subject to D, X,WX; D.
(3) Homo: y: = (X,,)) +
Hetero refers to the AIR with only heterogeneous part, which is solved by alternately updating D and W. Homo refers to the AIR with only homogeneous part which is actually a linear regression model and can be solved by OLS directly. For comparison among these three models, we follow Case I and Case 2 in the simulation part, i.e. with and without explicit W when generating true internal relation matrices. We extend the degree of individuation (DI) {1/4, 1/2, 2, 4}, indicating the true model becomes more and more individualized. We plot the average prediction errors, i.e. RMSE on test set, based 100 repetitions, against DI in Figure Both subplots show that the Homo is better than Hetero when DI is small while get worse when DI increases. However, the AIR is always the best all over different DI. It demonstrates the advantage of combining the homogeneous and heterogeneous parts, which adapts the model to more scenarios.
Case 1
Case 2
40
50
30
30
RMSE
RMSE
20
10
10
1/4
Degree of individuation
Degree of individuation
AIR
Hetero
Homo
Figure 3: Results of ablation studies. Incorporating homogeneous part and heterogeneous part makes the AIR more robust, especially better than the one with only heterogeneous part.
B.2 Simulation
Codes of our approach are available httts:://github. com/YLKnight/AIR Implementation details of different methods are explained here. The AIR is implemented in Python with hyperparameters A and \ selected by 5-fold cross-validation, of which the candidate sets are both from to 10. LRMR and TRLasso are implemented by their Matlab code, with hyperparameters selected by BIC in default setting. DKN implemented by its Python code. The blocksizes are set as 2 x 2, 2 x 2 and 7 X 7, resulting in 3 layers while the rank is by default selected by BIC from I to 5. The VIT is trained by Adam optimizer in Pytorch. Followed by an MLP for regression, the transformer model includes 4 transformer blocks with 8 heads in each Multi-head Attention layer, and the patch size is set as
In all experiments, the CPUs used are Intel Xeon Gold 5218R and GPUs used are NVIDIA GeForce RTX 3090. Figure 4 below shows simulation results under Case 2.
14

---

True C
True D
C from LRMR
t from TRLasso
No explicit W
Con
Co0
Cor
Figure 4: Simulation results under Case 2 with = 1.0. There does not exist am explicit true W while the internal relation matrix A, is computed directly by patchwise Pearson correlation coefficients. Because such A, is close diagonal matrix, it is rational that W from AIR is close a diagonal matrix.
C Proofs
C.1 Useful lemmas
Lemma C.I. Define the distance of two vectors u. v E TRP as
(u, x)²
dist(u, v)
V
(35)
2
For any vectors u, IRP where ||u||2 L, it holds that
dist (u, v) I 1 -0/12 2112
(36)
Lemma C.2. For any vectors u, v E IRP, holds that
Iu - 2112 2 (1|u1) 12 + (|0|12)
(37)
|
[
Lemma C.3. Define
vec ((x[x)) @ (x.) veo ((xxx,) @ xxx1)
(38)
Z' (vec ((x1x,) 8 x11) ,..., ver ((x,x,,) X,,
(39)
If satisfies the 2-RIP condition with constant 82, Z' also satisfies the 2-RIP condition with constant 82.
15

---

Proof.
|| (e')ec(M)) 1 (vec(M _o z' Z've
Σ {vec(M)) vec (xxxx:) @ x))
{rea ((x:X:) 8 ex:)3 vec(M)
Σ M. ( [x] x:) 0 xxx) )
(M (x7 x:) @ x! 2
Σ vvec (MT)) vec (xx7x x.) s0 xx.]) {vec ((x]x.) xx7)) vec (MT)
Zvec M ( 11
According to RIP condition on Z,
S Z (M [M² F
|M | = =1+ |
(1 - 52) || = (1 52)
M
- (1 (1+ 62)
follows that
(1 (1- S n
| vcc( 11 S (1+ 52) M
which indicates that Z satisfies the same 2-RIP condition as Z. Lemma C.4. Suppose Z satisfies the 2-RIP condition with constant 82. For two matrices M1 and M we have
(Zvec(M Zvec(M2)) (M M2) S 362 | M
(40)
Proof. Due to RIP condition, we directly have / (M, + M2 S (1 + 5 + M2 which can be expanded as
Vee M )l + Zvec M2) + (
M 1), Zvec(M2))
< (1+ 82) M + || + 2(M,,
Again due to RIP condition, we also have
(1-62)]]MM 112- < Zv and 12- > 22
Consequently, it holds that
(1 52) |1 + |M + 12) 2v))
< (1 + 52)
M, M,
1/ + M P + 2 M2))
Namely,
(Zvec(M1). 1), (vec(M2) - (M, M2 < 82 M1 + | + (M))
Furthermore, note that the last inequality still holds if we replace M1 by AM, and M2 by
1/AM2. Optimizing the RHS with X, we get
(Zvec(Mi), Zvec - (M1, S 33 |
Proving the other side of the inequality is similar.
Lemma C.5. Let Z = X x;
& X
With || = ||d" l| = 1, denote E and E respectively
as
n
$ =
Z,dd
Z dd
zT.
Σ= Σ z (d') z,T
Then we have
$)11 < 1 - 36g
35,
dist d d'
(41)
16

---

Proof. First consider the minimal eigenvalue of Σ
Xmin
min
u T Eu
u| ==1
min
u Z Zidd Z u
I| 2=1
min
Σ u T z,d]
u Z.d)
||u
( Zi, ud T 2
min ||
Zvec und T |12
min
||u ==1
2 302
The inequality holds due Lemma C.4
Further consider
[(
Σ
u T (d,d');--)) -E)
=
max
2
|
1
(d, uu' 2idd 2Tu u u z.d(d)) z,,)
max
|
Zi ud ") (Z, (d, - ((2 ud ) (2))(())))
max
||u||=||||| 1 |||||=1
((dd
Zi, ud"
max
| 1= 1=1
Zvec v ((d.dr)d-d)) d') dd)))
Zvec ud
S 352 (d. )d d' + ud - (a)) )
35,dist(d, )
The inequality holds due to Lemma d'.
C.4 where the inner product equals to 0 because à 1 (d, d
C.2 Proof of Theorem 5.2
For case display, we present a prerequisite theorem before proof of Theorem 5.2 The following
theorem provides error bounds within each iteration, which is the key to prove Theorem 5.2
Theorem C.6. Suppose model (21) holds and solved by alternating minimization algorithm. Assume Condition 5.1 with a small constant 82. Let K1 po + 332// (l (1- 362) and K2 = po + +38g + A2)
382 + 22). Then we have
dist (1) w) S kn kn (ā .a)
+
(42)
dist ((0)) d < Kydist (tt1-1) w + T2
(43)
Proof. The procedures of proofs for 42 and 43 are the same, with some differences in details.
Let us focus on (42) first. Then the model 12 can be rewritten in matrix form below
Y = Mw +
17

---

with w vec(We W) and M defined as follows
M = veec DX] ...... vec X DX x x [X,
(44)
Suppose u and (t) are normalized, so we have || || (1) ||2 = for any t 2 in the following.
Let d =d") (0)
/I|d 2a2 and d" = dd/ldd be their unit vectors. Define two matrices in t-th iterations
x0
z,d ( (d'0) 27.
(0)
z (d') z!.
1=1
as 444 with D replaced by D (4)
Define M
Then, it holds that
||a 1
|B50 and M (M)) M M
M
M
=
-
Given D
have
-1
(1)
M
M
(1) (0)
MM)
M
Y
(t)
(1)
M ()
M
M
(Mw +
|dl2
(0)
Σ
(1)
+
(500)
M (1 E
||ā
(0)
ll2
Without loss of generality, suppose (d (0) d) similar way. Consider the f2-norm distance
2 0. The case that (a dd < 0 can be proved in
11 (1)
|d|| 2
w
13(00
(5())=
-1
w
M E
j 1221d
=(d")
d' r w - (5)) -1 (a .d') -
$ (1) -1
(2)
T
w +
M
E
ll |2lld||2
(d d + (10) ((d") (t)
d') ) - ->00
(1)
<
M
2
|2||d||22
42
43
(45)
Note that when (a") (1)
d) 2 0. we have < (d ,d') S Thus for A1,
(d dd) = dist? (a .d) S podist (a()
1
For A2, it holds that according to Lemma C.5
38g
dist (1) d
A2 <
38
18

---

For A3, first note that
1/2
-362.
M (0)
x (2) x! x,
Vee
n
Σ Ejvec
(x, x x3
sup
To
As result, A3 can be bounded by
To
A3 S
71
(1 - 352) || 11 12||d|12
One the other hand, according Lemma C.I we have for any
>> that
dist @ (8) w dist (c (t)
(c)(1) www) S
ce (1)
Therefore,
dist (wo w)
362
dist (a() .d'dd d) + TI
44
(46)
362
To prove (4). first we need to rewrite the model
12 in another matrix form below.
Y Nd
with d = vec(D) and N defined as follows
N (vec [ (x, ,..., vec (x,, (x,,X x
(47)
Note that |l||2 = t0 (1-1) 112 ||2 = 1. Define two matrices in /-th iterations
D (t-1)
Z (/ (1-1)
(1)---1) (
D (1-1)
Σ 7 (11) w (Z))
- ((-1)
with W replaced by W (1-1)
Define N
as 47
Then, it holds that
N (
N (111))
5 (t-1) and ( (x((-11) N = ((-111
Given W (1-1)
we have
(00) - ( (111-)) + x2t
(Ni--1) (Nd+e)
=($"1-1) + )ee (1(-1)) d (5((-11) + )22) (N
( ppm (1 + x22
( w) (3 +x21) - d
+ -(111 + do1)
(N
19

---

Then f--morm error of d example.
can be also bounded in a similar way. Take the case (ix w)) 2 0 for
(1)
dll2
|d||
< 1
(w0 (f 1)
w + (1-1)
+ )21) -1
((wolt-)), w) ( + x2I) E 2
B1
B2
1
$ (1-1-
+ )21)
-1
(v
N
8
(48)
2 2
B3
Resembling A1, A2 and A3, we have
B1
(20 (t-11, u0) S undist
(0 20
(49)
382 + A2 dist @w0 (t-1) 20
B2
(50)
1 302 + A2
B3 1 352 + A2
||e||2 = T2
(51)
Thus we have
dist a ad)d)
< 140 +
382 + A22
dist (100-1), +
(52)
1 302 + X2
(w (0)
Last we need to prove by induction that if dist
w
S no and po satisfies Kip0 + 7I S. po and
R2Ho + S Hu, then dist (2) w S lo and dist
( d
s no for any t 2 1.
When ==
dist
a d
< dist?
(w0 w +
352 + A2
dist (0) ( + T2
35g A2
382 + A2
dist (0)
(0) w +
44 +
1 362 + A2
R2/40 + 72
14
Furthermore,
dist (1) w)
< dist?
((2)) d
+
382
dist (1))
.))
+ T
352
362
dist (()) d +
< 140 T --32g
380
KiP00 + T2
< μo
This completes the proof of initial step 1.
20

---

When 2 suppose dist (io!-11) w S oo to prove the f-th case.
dist a )
(0(1-1) .w) + 3d2 + A2 dist (01111)
S dist
:0) + 72
1-302 + A2
362 x2
dist ( 200 +
x 140 +
1 - 3d2 + X2
< K2f00 + T2
140
Furthermore,
dist () w
S dist
(1)) dd
+
302
dist (1() .d) + aI
362
382
(
1 140 +
dist
+
1 --362/
K1100 + TI
S 140
This completes the induction. In words, the distances dist (i w) and dist a
d in all
iterations are guaranteed not exceed the initial distance Na, which is required when proving and (43) The proof is now complete.
Proof of Theorem
2
Theorem C.6 provides the error bounds within an iteration. Therefore, we have the following by recursion.
dist (1) w0) I Kidist (²") d +
s (Kit (0 20)
+ 1K172 + T
< (K1N2)
20
(KIK2) +
==0
s (K1N2)) no
K172 + 71
- Ki
On the other hand,
( .d)
S r2dist (0) 20)) +
dist
(11-1) .d)
s (Kini)dist
+ K2TT + 72
s (K1 dist ( d
1-2
(KIK2) + 12)
NoT1 + T2
< x n2p0
1 K1
The proof is completed.
C.3 Proof of Theorem
Theorem C.7. Suppose model (27) holds and solved by alternating minimization algorithm. Assume
Condition 5.1 with small constant dz. Denote (t)) ||d (())
||2/||d]22. Let 244 + 3822 382)
21

---

and 240 + (362 + 22) - 3d2 + A2). Then for any t 2 0 we have
- (1) 14 + 11 + T
(53)
1- U2
(
2
S x V2200 +
+
(54)
d||
1 1- V1
Proof. The proof of Theorem C.7 is also completed by induction, resembling that of Theorem 5.2 Thus we just note some key inequalities that are different in induction procedures. Suppose for any
t 2 that ||d (t)
d| I 140 and (c(") 2l12 S po.
According to 48 we have
Ii2"0 dll2
1 | |10(--1) -011
1
302 + A2
dist 1 (((11) 20 72
|||||2
1-36g \2
I 2|
1) (1-1) (t)
-wo13 + 1 3d- X2
352 + X2
||e(t-1) (1(-0))000) - wl12 +
302 + A2
s 2p0
||el1- 1) i0 (t-1) w
122 +
(55)
302 + x2
The second inequality holds because Lemma On the other hand, according to 45 we have
(()))
(2)
d"
112
+
382
dist
(7())
dz')
+ ti
382
(t)
d
38g
d 2
+ TI
|
38
|d||
352
||a
d|l
2p4 +
+ 71
(56)
1-
382
|||
Given (55) and 56), f2-norm errors can be also bounded in (-th iteration. The remaining proof can be completed in the same way as Theorem 5.2
Proof of Theorem 5.3
According to Condition firstly we have
| (1)
Y 12 = Zvec ()) (1())
wd
|
S / 8z (r) (2(0) (2())
T
wd
22

---

It follows that
(0)
w
(t)
wd
88 (t)
(0))
d
+
w
d
(1)
d|12
||d||
(1)
w
<|d||
|dll
d112
+
||d|
2
c(t)
2
+
(1) 2
(1)
w
<|d||
[1a
dll2
+2 +2|l000)
10
20
||d||
2
<|d||a
(11P2)² 122 Mo + 1172 T
+2 (1112) 1400000000-00000000 + M2T1 + 12
- 1/1122
1 1/1 1/22/
<3||d||2
(1122)² -1 Ho
71 72
1 1/122
Therefore have
1112 s /I (11122) 44 +
T1 12
F
1 v1 U2
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
In the discussion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]

---

Justification:
In the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question:
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Experimental details well-explained.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer:
[Yes]
Justification: In the appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cpippi.ccppulli//uuuia wwwiicccguie//eeeeeeeeeii CoodeSubeiiie ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental details are well-explained. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Numerical results reported are the mean and standard error of repetitions.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
26

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative emror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: In the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: ptttps://neuripst httpr/pp....ubblic//hhcccccctl.o
Answer: [Yes]
Justification: Conform.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In the discussion.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
27

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee whworee.wthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
28

---

Answer: [Yes]
Justification: Codes are submitted in the supplementary material.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The dataset used is public dataset and the study is non-clinical.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "EDtx5lkJ7F", "reviewer": "Reviewer_2raH", "summary": "The authors present an interesting approach to individualization of regression for heterogenous data. They first set up an individualized model with additive homogenous and heterogenous components containing matrix-valued coefficient matrices to be learned. They nicely establish the equivalence of their model under mild assumptions with scaled dot-product attention and linear attention approaches and provide a straightforward alternating minimization scheme. They provide both theoretical analysis (two theorems showing geometric decay of optimization error and prediction error under RIP) and experimental results (one synthetic and one real-world application) for their approach.", "questions": "####Major questions:\n\nWhy is there no anonymized link to code?\n\nWhy are the individualized coefficients so blocky for ADNI, and what could be done to mitigate this undesirable effect?", "limitations": "A more detailed limitations section should be added in the supplement.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper has a number of strengths. It is well written and relevant to current needs in the precision medicine. Although it is not clear to me that it is necessary, the relation with attention is interesting. The author theory holds under realistic assumptions, and they support their findings with a synthetic and a real world example.", "weaknesses": "#### Major Weaknesses\nI may have missed the anonymized link, but the authors have not provided any code for review, which I find problematic given the applied nature of the paper and the fact that implementation seems straightforward. \n.\nThe individualized coefficients for the ADNI data show are very rough due to blocking in D_i^ori. Is there some way to ameliorate this in future work?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review.\n\n>**Q1**: Why is there no anonymized link to code?\n\n**A1**: Codes were uploaded with submission to the “Supplementary Material” part as a zip file. Due to the request for anonymity, they are not public at the moment. We will provide the public link in the camera ready.\n\n>**Q2**: Why are the individualized coefficients so blocky for ADNI, and what could be done to mitigate this undesirable effect?\n\n **A2**: We would like to clarify that the blocky effect is essentially caused due to the reshaping operation (from $X_i^{\\text{ori}}$ to $X_i$). The reshaping operator $\\mathcal{R}(\\cdot)$ divides original images into blocks, vectorizes and stacks them for preprocessing. \nThe internal relation matrix $A_i$ aggregates blocks in $D^{\\text{ori}}$, then we obtain the individualized coefficients $D_i^{\\text{ori}}$. This is the reason that $D_i^{\\text{ori}}$ appears to be blocky. On the other hand, if we consider internal correlations among small blocks, the blocky effect can be mitigated significantly.\n\n>**Q3**: A more detailed limitations section.\n\n**A3**: Thanks for your suggestion. We will add a paragraph about limitations in the “Discussion” section, stating that:\n“On the other hand, we realize that the AIR framework also has limitations. First, the AIR model is designed for data with heterogeneous internal relationships, and its capability to handle more general data is more or less restricted. When there are minimal heterogeneous effects, its performance will be similar to an ordinary linear model. Second, as discussed earlier, our framework could be viewed as a simplified version of the Vision Transformer; however, such simplifications may also reduce its approximation power for more complex scenarios. Furthermore, this paper primarily investigates the linear form of AIR. Although the linear form performs well in the cases of interest, exploring the generalization of the model in future work is still worthwhile.”", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5z463ZFACV", "reviewer": "Reviewer_3WVs", "summary": "The paper proposes an approach for regression where common model coefficients can be modulated by sample-specific data. In particular, here the approach is applied to images (or matrices), where sample-specific data is derived from patch similarities (measured through rotation correlation), reflecting intra-image homogeneity. The model parameters (matrices of coefficients) are learned through penalised least-squares, with energy-minimising Frobenius norms on the unknown coefficient matrices to make the problem more well-posed. The authors then draw correspondances between their model and self-attention under some assumptions. They then show analytically convergence rates and error bounds for their model and the AD-style optimisation algorithm. Finally, simulation results and empirical results on brain imaging data show lower prediction errors compared to related methods.", "questions": "Is the performance of the method proposed significantly different from the other methods, both for the simulation and the real data case? (post-rebuttal: OK)\n\nHow is the lack of independence between slices from the same subject addressed in testing, in particular for cross-validation and error metric computation? (post-rebuttal: OK)", "limitations": "Limitations are not in a separate section and consist of one sentence. (post-rebuttal: OK)\n\nThere is no real negative social impact of this work.", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper provides an interesting connection between varying-coefficient models and self-attention under relatively mild conditions.\n\nThe ablation studies in appendix B.1 is interesting and helps highlight the contribution of individual coefficients in different cases.\n\nThe method proposed has applications for images and other matrix-valued data.\n\nThe figures are very helpful in conveying the aspects of the coefficient matrices.", "weaknesses": "Claims of superiority are not supported by hypothesis tests between the proposed method and the other 4 methods. (post-rebuttal: this is now OK)\n\nFor real data analysis, it is unclear how many subjects were selected, and with which diagnosis. Extracting 10 slice per subject is fine, but are they evaluated as independent or are results provided per-subject? (post-rebuttal: OK)\n\nIn addition, predicting cognitive scores from brain imaging is a very well studied task (in particular the MMSE-ADNI combination), not only cross-sectionally but also longitudinally. See e.g. 10.1016/j.neuroimage.2011.09.069, 10.1016/j.neuroimage.2014.03.036, 10.1109/PRNI.2015.28, or the review in 10.1016/j.jalz.2016.11.007. Here, the choice of the metric 'improvement compared to sd of scores in test set' obscures performance with respect to these and other previous work. I would suggest to provide MSE and MAE in the original MMSE scale for more clarity. (post-rebuttal: OK)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review.\n\n>**Q1**: Is the performance of the method proposed significantly different from the other methods, both for the simulation and the real data case? (Claims of superiority are not supported by hypothesis tests between the proposed method and the other 4 methods.)\n\n**A1**: Thanks for your question. For the simulation where the results are from 100 repetitions, we use z-test showing that the superiority of the proposed method to the other methods is significant.\nAs for real brain imaging analysis, we did not conduct significance tests in our original experiments because of the chronological division considered. But to have a rough understanding of the robustness of the proposed method, we retest the performances of these methods by 5-fold cross-validation. The result of 5 folds in real study is shown below. \n\n| Methods | AIR | LRMR | TRLasso | DKN | ViT |\n| :--------: | :--------: | :--------: | :--------: | :--------: | :--------: |\n| Fold 1 | 3.183 | 3.712 | 3.359 | 3.297 | 3.321 |\n| Fold 2 | 3.193 | 3.744 | 3.325 | 3.283 | 3.349 |\n| Fold 3 | 3.127 | 3.699 | 3.258 | 3.284 | 3.215 |\n| Fold 4 | 3.129 | 3.700 | 3.232 | 3.226 | 3.282 |\n| Fold 5 | 3.093 | 3.717 | 3.286 | 3.214 | 3.243 |\n| Mean (S.D.) | 3.145 (0.042) | 3.715 (0.018) | 3.292 (0.051) | 3.261 (0.038) | 3.282 (0.055) |\n\nBased on the mean and S.D. of the obtained RMSE, we may conduct a (rather rough) t-test and report the obtained p-value as below. \n\n| Tests | AIR vs LRMR | AIR vs TRLasso | AIR vs DKN | AIR vs ViT |\n| :--------: | :--------: | :--------: | :--------: | :--------: |\n| P-value | 0.000 | 0.001 | 0.002 | 0.002 |\n\nThese results suggest that the proposed method is significantly better than the others.\nWe will add the results of significance tests in the revised version.\n\n>**Q2**: The choice of the metric 'improvement compared to sd of scores in test set' obscures performance with respect to these and other previous work. I would suggest to provide MSE and MAE in the original MMSE scale for more clarity.\n\n**A2**: Thanks for your suggestion. We have reorganized the table with RMSE and add the results of hypothesis tests of 5-fold cross-validation for a reference of significance. Please see the table in A1.\n\n>**Q3**: For real data analysis, it is unclear how many subjects were selected, and with which diagnosis. How is the lack of independence between slices from the same subject addressed in testing, in particular for cross-validation and error metric computation?\n\n**A3**: Thanks for your questions. For the training set, the 7270 images are obtained from 727 subjects in the ADNI&GO phases where 229 are normal, 310 are with MCI (mild cognitive impairment) and 188 are with AD. For the test set, the 3320 images are obtained from 332 subjects in the ADNI2 phase where 140 are normal, 91 are with MCI (mild cognitive impairment) and 101 are with AD. We will explain this in the appendix.\n\nExtracting 10 slices can be viewed as a kind of augmentation of the dataset and we take the obtained images as independent samples, which is a common practice in data augmentation.\nDespite dependency, treating them as independent can be useful for training models, as it effectively increases the diversity and size of the dataset.\nAs for the dependency issue when testing, in this revision, we conduct another experiment with 1 middle slice per subject for validation and testing (total of 332 subjects), and 10 slices per subject for training. It shows that the result is very close to the previous one, which also demonstrates the robustness of our method.\n\n>**Q4**: Limitations are not in a separate section and consist of one sentence.\n\n**A4**: Thanks for your suggestion. We will add a paragraph about limitations stating that:\n“On the other hand, we realize that the AIR framework also has limitations. First, the AIR model is designed for data with heterogeneous internal relationships, and its capability to handle more general data is more or less restricted. When there are minimal heterogeneous effects, its performance will be similar to an ordinary linear model. Second, as discussed earlier, our framework could be viewed as a simplified version of the Vision Transformer; however, such simplifications may also reduce its approximation power for more complex scenarios. Furthermore, this paper primarily investigates the linear form of AIR. Although the linear form performs well in the cases of interest, exploring the generalization of the model in future work is still worthwhile.”", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WAMEYyMVlc", "reviewer": "Reviewer_EsRE", "summary": "This paper proposed an individualized regression method and applied it to medical image analysis. The method can handle matrix-valued data and does not require additional information on sample similarity. The authors also analyzed its relationship to the attention technique. Finally, the proposed method was evaluated on simulation and real data sets, and obtained improved performance.", "questions": "1. Section 2 should be rewritten to make it clearer since some technique details of the proposed method are hard to understand. \n2. Too many details of the method are unclear, such as the inverse operation of R, p1 *p2, and d1 * d2.\n3. How to determine the size of the blocks? How to determine the number of factors?\n4. Please double check Eq. (14).", "limitations": "The authors did not state the limitations of the proposed method. No conclusion of this paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A novel individualized regression method handling one-model-fit-all issue.\n2. The method does not require additional information on sample similarity.", "weaknesses": "1. The method can only work for matrix-valued data, such as image data.\n2. The real study is too simple and insufficient.\n3. Section 2 should be rewritten to make it clearer since some technique details of the proposed method are hard to understand.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review.\n\n>**Q1**: The method can only work for matrix-valued data, such as image data.\n\n**A1**: Thanks for your comment. Matrix-valued data, particularly images, are pervasive in many practical applications, and our method aims to provide a novel solution for these scenarios. While as the amount of tensor data increases, extending the method to tensor-valued data is a valuable direction to expand its applications. We will consider how to generalize our model in future research, including how to deal with higher dimensions, how to incorporate internal relations into the individualized coefficients, and more.\n\n>**Q2**: The real study is too simple and insufficient.\n\n**A2**: Thank you for your feedback. We have added a 5-fold cross-validation in real study to test the significance of the difference. It shows that the advantage of the proposed method is significant, of which results are in the following table.\n\n| Methods | AIR | LRMR | TRLasso | DKN | ViT |\n| :--------: | :--------: | :--------: | :--------: | :--------: | :--------: |\n| Fold 1 | 3.183 | 3.712 | 3.359 | 3.297 | 3.321 |\n| Fold 2 | 3.193 | 3.744 | 3.325 | 3.283 | 3.349 |\n| Fold 3 | 3.127 | 3.699 | 3.258 | 3.284 | 3.215 |\n| Fold 4 | 3.129 | 3.700 | 3.232 | 3.226 | 3.282 |\n| Fold 5 | 3.093 | 3.717 | 3.286 | 3.214 | 3.243 |\n| Mean (S.D.) | 3.145 (0.042) | 3.715 (0.018) | 3.292 (0.051) | 3.261 (0.038) | 3.282 (0.055) |\n| P-value | - | 0.000 | 0.001 | 0.002 | 0.002 |\n\nIn the coming days, we will attempt to apply our method to new real data to further validate its performance.\n\n>**Q3**: Section 2 should be rewritten to make it clearer since some technique details of the proposed method are hard to understand.\n\n**A3**: Thank you for your feedback. We will revise Section 2 to enhance clarity, particularly providing more technique details of the proposed method to make it clearer.\n\n>**Q4**: Too many details of the method are unclear, such as the inverse operation of R, p1 *p2, and d1 * d2.\n\n**A4**: Thanks for pointing it out. The inverse operation of $\\mathcal{R}$ is used to recover the reshaped images and corresponding coefficients to their original reshape. Besides, $p_1\\times p_2$ are the number of blocks and $d_1\\times d_2$ are the size of blocks. We have scrutinized the details and will make them clearer in the revised version.\n\n>**Q5**: How to determine the size of the blocks? How to determine the number of factors?\n\n**A5**: Thanks for your question. The division of images before implementation of our model is similar to that of Vision Transformer, where $16\\times 16$ is a common size of patches. In our paper, due to relevantly small size of images, the size of blocks should be smaller. We note that our method performs robust to moderately small size of blocks, so we determine the size of blocks by cross-validation among $4\\times 4$, $6\\times 6$ and $8\\times 8$.\nBesides, when the row-wise internal relations are of interest, such as EEG in which each row represents a channel, it will not involve division and the method can be directly applied.\n\n>**Q6**: Please double check Eq. (14).\n\n**A6**: Thanks for pointing it out. There is a typo that the first $W$ should be transposed. It is tantamount that we assume $W^T$ in the model, which has no effect on the existing results. Thank you again.\n\n> **Q7**: The authors did not state the limitations of the proposed method. No conclusion of this paper.\n\n**A7**: Thanks for your comments. We conclude in the “Discussion” section and will add a paragraph about limitations there, stating that:\n“On the other hand, we realize that the AIR framework also has limitations. First, the AIR model is designed for data with heterogeneous internal relationships, and its capability to handle more general data is more or less restricted. When there are minimal heterogeneous effects, its performance will be similar to an ordinary linear model. Second, as discussed earlier, our framework could be viewed as a simplified version of the Vision Transformer; however, such simplifications may also reduce its approximation power for more complex scenarios. Furthermore, this paper primarily investigates the linear form of AIR. Although the linear form performs well in the cases of interest, exploring the generalization of the model in future work is still worthwhile.”", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E6I2xvm3IQ", "reviewer": "Reviewer_hyyZ", "summary": "The paper introduces a method for self-attention-based individualized regression and derives it's relation to transformers. The method is evaluated in a simulation setting and on an Alzheimer Brain MRI dataset.", "questions": "- It would be great if the authors could comment on the scalability of the method\n- Is individualized regression applicable to multiple instance learning problems?", "limitations": "See questions", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Interesting theoretical treatment of individualized regression and its connection to transformers\n- Well written", "weaknesses": "- Only applied to tiny datasets with image size 48 x 48\n- Only two experiments", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review.\n\n>**Q1**: Only applied to tiny datasets with image size 48 x 48 and only two experiments.\n\n**A1**: Thank you for your feedback. The MRI scans are preprocessed to be of size $113\\times 137\\times 113$ and we further resize the extracted slices to the size $48\\times 48$ for computational efficiency.\nBesides, we have added a 5-fold cross-validation in real study to test the significance of the difference, which shows that the advantage of the proposed method is significant, with results shown in the following table.\n\n| Methods | AIR | LRMR | TRLasso | DKN | ViT |\n| :--------: | :--------: | :--------: | :--------: | :--------: | :--------: |\n| Fold 1 | 3.183 | 3.712 | 3.359 | 3.297 | 3.321 |\n| Fold 2 | 3.193 | 3.744 | 3.325 | 3.283 | 3.349 |\n| Fold 3 | 3.127 | 3.699 | 3.258 | 3.284 | 3.215 |\n| Fold 4 | 3.129 | 3.700 | 3.232 | 3.226 | 3.282 |\n| Fold 5 | 3.093 | 3.717 | 3.286 | 3.214 | 3.243 |\n| Mean (S.D.) | 3.145 (0.042) | 3.715 (0.018) | 3.292 (0.051) | 3.261 (0.038) | 3.282 (0.055) |\n| P-value | - | 0.000 | 0.001 | 0.002 | 0.002 |\n\nIn the coming days, we will attempt to apply our method to new real data to further validate its performance.\n\n>**Q2**: It would be great if the authors could comment on the scalability of the method.\n\n**A2**: Thanks for your suggestion. Two aspects of scalability are usually concerned.\n* Computational efficiency: As the algorithm actually alternatively solves linear models, the complexity is $O(D_1^3D_2^3)$ where $D_1$, $D_2$ are the size of images. As for the convergence rate, if we further suppose noise $\\epsilon_i$ is sub-gaussian, it can be proved that after $t\\ge t_0 + \\log(\\log(n)/n)/(2\\log(\\kappa))$ iterations, the distance errors can be bounded by $\\mathcal{O}(\\sqrt{\\log(n) / n})$ with high probability, where $t_0$ is a constant and $\\kappa$ is the contraction parameter. In summary, as the size of the dataset increases, larger images size brings more computational burden, which can be relieved by parallel processing and more in practice. On the other hand, larger sample size leads to fewer iterations and smaller errors.\n* Generalization: First we propose to combine the homogeneous and heterogeneous parts to make our model adaptive to more types of data. Second, we propose to model the internal relation matrix by a function $g$, which can introduce nonlinearity to make the model more flexible. However, as mentioned in the paper, the ability of the model to handle general data is limited, depending on the gap between the model and real cases, which is a common issue of model-based methods.\nThanks for your suggestion again and we will add the discussion in the revised version.\n\n>**Q3**: Is individualized regression applicable to multiple instance learning problems?\n\n**A3**: Thanks for the interesting question. In multiple instance learning problems, the training data is organized into bags, where each bag contains multiple instances. In the context of MIL, individualized regression can be applied by considering each bag as an individual data point and tailoring regression models to these bags. Our model can be also applied to multiple instance learning problems, in the sense that patches of an image are instances in a bag and internal relations determine bag-specific coefficients.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iOvWP2Xscc", "reviewer": "Reviewer_gg3f", "summary": "The authors propose an individualised matrix regression method, where the individualised part is shown to be related to self-attention. The method is presented nicely, with theoretical and empirical results that indicate the usefulness of the method.", "questions": "- When directly referring to references, use the Firstauthorlastname et al.~\\cite{ref} format. Should be possible to use \\citet{ref} for this.\n - Make equations part of sentences instead of something particular presented following a colon. Also, when equations are at the end of a sentence, end it with a full stop.\n - Equation 1: The d_1,d_2 subscript is missing from R.\n - Line 161: The functio \\rho is not explained properly/clearly. Is this a notational error?\n - It appears you are using the Frobenius matrix inner product, but it is never defined. Same with the vector inner product, seems like you are using the Euclidean inner product (dot product), but this is never defined.\n - Line 206: What do you mean by: \"... has the potential to achieve fewer parameters and faster training.\"?\n - In the theorems: Define W in the initial distance.\n - Explain what you mean by \"the truth\" on line 228.\n - Some spelling errors and typos, e.g. lines 239, 269, 498, and 535.\n - References: RNNs should be upper case in Ref 12, \"Lasso\" in Ref 25, and reference 24 has some problems with the spacing (copy from PDF?).\n - Label the first axis in Figure 3.\n - Lines 424-437: Should be \"Python\", \"Matlab\", and \"Pytorch\", since they are names.", "limitations": "- See my previous comment about personalised interpretation.\n - The discussion section is _very_ short. I understand the problem with space constratins, but do elaborate properly on strengths, weaknesses, limitations, and future work.\n - You say that Crowdsourcing data from human subjects and IRB approval is not applicable, but these are images from human subjects collected collaboratively in the ADNI project. You would actually likely need an IRB approval to perform this research, since you do research on data from humans, but it seems that doing research on public medical data is a gray area.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The paper is well-written and easy to follow. The method is clear, the connection to attention is clear, the theoretical results appear correct and the empirical results (on both simulated and real data) and convincing.", "weaknesses": "- The introduction mentions personalised interpretations, but that is never illustrated in the paper (though seems to be alluded to in the appendix?)\n - Proposition 1 seems to assume that the function g is an element-wise function (or otherwise a particular function, such that it can be transposed), but it is mentioned as being any general function.\n - The subscripts K and Q appear to have been swapped in the definition (I) of W. See line 176 also.\n - The convergence is geometric, but while one term disappears, there appears to still be a constant term left. So the question would be how large that constant is? How tight is the achieved bound in the limit as t -> \\infty?\n - It would be better to report mean and _standard error_ (standard deviation of the mean) instead of mean and standard deviation of the 100 repetitions. This makes it easier to compare the results between methods.\n - Equation 18 is presented without any constraints on W, but the pseudo-code normalises the weight matrix (Equation 31), and the proofs assume unit vectors. Are these the same? If not, dot he convergence proofs still hold? This should be clarified.\n - I'm missing a discussion on the computational complexity and run-time (and compared to other methods).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review.\n\n# Main questions\n\n>**Q1**: Personalised interpretations\n\n**A1**: Thanks for your feedback. For real data, we show in Figure 2 the individualized coefficients of different samples and their significant internal relations and frame out corresponding regions in original samples. Take sample 1 for example. The block (4, 5) has the strongest relations, and is related to both (6, 4) and (6, 5), indicating the important relations between corpus callosum and hippocampus. We also find that after separating heterogeneous effects, the homogeneous effects highlight the hippocampus region, which is widely known to be associated with Alzheimer's disease. We will add the discussion in the revised version.\n \n>**Q2**: The function $g$ in Proposition 1.\n\n**A2**: Thank you for question. Sorry for the confusion in Proposition 1. Yes, we are mostly focusing on element-wise $g$ function, including commonly used activation functions in attention mechanisms, such as row-wise softmax function, scaling function and more. We will clarify it in the revised version.\n\n\n>**Q3**: The convergence is geometric, but while one term disappears, there appears to still be a constant term left. So the question would be how large that constant is? How tight is the achieved bound in the limit as t -> \\infty?\n\n**A3**: Many thanks for the good question. When the error term is sub-Gaussian, it can be proved that after $t\\ge t_0 + \\log(\\log(n)/n)/(2\\log(\\kappa))$ iterations, the distance errors can be bounded by $\\mathcal{O}(\\sqrt{\\log(n) / n})$ with high probability, where $t_0$ is a constant and $\\kappa$ is the contraction parameter. We will remark this specifically in the revised version.\n\n>**Q4**: Equation 18 is presented without any constraints on W, but the pseudo-code normalises the weight matrix (Equation 31), and the proofs assume unit vectors. Are these the same? If not, do the convergence proofs still hold? This should be clarified.\n\n**A4**: Many thanks for your question. We apologize for the oversight. The Equation 18 should also require a norm constraint for W due to identifiability consideration. Therefore, this is consistent with the proof (where an unit vector assumption is imposed). We will update Equation 18 in the revisions.\n\n>**Q5**: Computational complexity and run-time.\n\n**A5**: Thanks for your feedback. As the algorithm actually alternatively solves linear models, the complexity is $O(D_1^3D_2^3)$ where $D_1$, $D_2$ are the size of images. On the other  hand, the optimization problem is non-convex, but it is bi-convex. Theorem 5.2 suggests a linear convergence rate of the alternating minimization algorithm (AMA) although the problem is non-convex. In contrast, gradient descent (GD) algorithms could achieve a linear convergence rate only if the objective function is strongly convex. But for general non-convex functions, linear convergence cannot be achieved. This suggests the advantage of AMA over GD. We will add the discussion on the computational complexity and report the run-time compared to other methods in the revised version. Thanks again for your advice.\n\n>**Q6**: Line 161: The function \\rho is not explained properly/clearly. Is this a notational error?\n\n**A6**: Thanks for your question. Here $\\rho$ stands for the linear function considered in the reference. Specifically, for a matrix input $\\boldsymbol{Y}\\in \\mathbb{R}^{n\\times p}$, define $\\rho(\\cdot)$ as $\\rho(\\boldsymbol{Y}) = \\boldsymbol{Y} / n$. We will make it clear when revising.\n\n>**Q7**: Line 206: What do you mean by: \"... has the potential to achieve fewer parameters and faster training.\"?\n\n**A7**: This sentence is to introduce the above reference titled ``Simplifying transformer blocks’’, in which the proposed simplified transformers enjoy faster training speed using fewer parameters. This further suggests the advantage of simplified models.\n\n>**Q8**: Explain what you mean by \"the truth\" on line 228.\n\n**A8**: The truth means true counterparts of $\\boldsymbol{W}^{(t)}$ and $\\boldsymbol{D}^{(t)}$, i.e. $\\boldsymbol{W}$ and $\\boldsymbol{D}$ in the true model.\n\n>**Q9**: References: RNNs should be upper case in Ref 12, \"Lasso\" in Ref 25, and reference 24 has some problems with the spacing (copy from PDF?).\n\n**A9**: The information on references were downloaded from Google Scholar directly, where there may be some issues. Thank you for pointing them out and we will make adjustments in the update.\n\n## Minor revisions\n* The subscripts K and Q appear to have been swapped in the definition (I) of W. See line 176 also.\n* It would be better to report mean and standard error (standard deviation of the mean) instead of mean and standard deviation of the 100 repetitions.\n* When directly referring to references, use the Firstauthorlastname et al.~\\cite{ref} format. Should be possible to use \\citet{ref} for this.\n* Make equations part of sentences instead of something particular presented following a colon. Also, when equations are at the end of a sentence, end it with a full stop.\n* Equation 1: The d_1,d_2 subscript is missing from R.\n* Define Frobenius inner product and Euclidean inner product used.\n* In the theorems: Define W in the initial distance.\n* Some spelling errors and typos, e.g. lines 239, 269, 498, and 535.\n* Label the first axis in Figure 3.\n* Lines 424-437: Should be \"Python\", \"Matlab\", and \"Pytorch\", since they are names.\n* Crowdsourcing data from human subjects and IRB approval.\n\nMany thanks to your carefulness. We will make adjustments according to the above suggestions and update in the revised version. Thank you again.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers appreciated that contribution sheds an interesting theoretical light, relevant for individualized medicine among other things, with fair empirical evaluation, though on somewhat toy data.
Decision: Accept (poster)