Breaking the Correlation Plateau: On the Optimization and Capacity Limits of Attention-Based Regressors
Authors: Jingquan Yan
Yuwei Miao
Peiran Yu
Junzhou Huang
Venue: ICLR 2026 Poster (2026)
Keywords: Attention Mechanisms
Joint Optimization
Pearson Correlation
Data Homogeneity
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We theoretically study why Pearson correlation plateaus in attention-based regression and introduce ECA framework to sharpen and extrapolate attention, consistently boosting correlation without compromising MSE, even on homogeneous datasets.

Abstract: Attention-based regression models are often trained by jointly optimizing Mean Squared Error (MSE) loss and Pearson correlation coefficient (PCC) loss, emphasizing the magnitude of errors and the order or shape of targets, respectively. A common but poorly understood phenomenon during training is the *PCC plateau*: PCC stops improving early in training, even as MSE continues to decrease. We provide the first rigorous theoretical analysis of this behavior, revealing fundamental limitations in both optimization dynamics and model capacity. First, in regard to the flattened PCC curve, we uncover a critical conflict where lowering MSE (magnitude matching) can *paradoxically* suppress the PCC gradient (shape matching). This issue is exacerbated by the softmax attention mechanism, particularly when the data to be aggregated is highly homogeneous. Second, we identify a limitation in the model capacity: we derived a PCC improvement limit for *any* convex aggregator (including the softmax attention), showing that the convex hull of the inputs strictly bounds the achievable PCC gain. We demonstrate that data homogeneity intensifies both limitations. Motivated by these insights, we propose the Extrapolative Correlation Attention (ECA), which incorporates novel, theoretically-motivated mechanisms to improve the PCC optimization and extrapolate beyond the convex hull. Across diverse benchmarks, including challenging homogeneous data setting, ECA consistently breaks the PCC plateau, achieving significant improvements in correlation without compromising MSE performance.

Published as a conference paper at ICLR 2026
BREAKING THE CORRELATION PLATEAU: ON THE OP- TIMIZATION AND CAPACITY LIMITS OF ATTENTION- BASED REGRESSORS
Jingquan Yan', Yuwei Miao", Peiran Yu, Junzhou Huang Computer Science and Engineering Department University of Texas at Arlington (jingquan, yan, yuwei .ia.,, no, pei ran yu, juhuanggeutaa.dde "Equal contribution.
ABSTRACT
Attention-based regression models are often trained by jointly optimizing Mean Squared Error (MSE) loss and Pearson correlation coefficient (PCC) loss, empha- sizing the magnitude of errors and the order or shape of targets, respectively. A common but poorly understood phenomenon during training is the PCC plateaw: PCC stops improving early in training, even as MSE continues to decrease. We provide the first rigorous theoretical analysis of this behavior, revealing funda- mental limitations in both optimization dynamics and model capacity. First, in regard the flattened PCC curve, we uncover critical conflict where lowering MSE (magnitude matching) can paradoxically suppress the PCC gradient (shape matching). This issue exacerbated by the softmax attention mechanism, particu- larly when the data to be aggregated is highly homogeneous. Second, we identify a limitation in the model capacity: we derived a PCC improvement limit for any convex aggregator (including the softmax attention), showing that the convex hull the inputs strictly bounds the achievable PCC gain. We demonstrate that data homogeneity intensifies both limitations. Motivated by these insights, pro- pose the Extrapolative Correlation Attention (ECA), which incorporates novel, inooreticlll-mmtivvtee mechanisms improve the PCC optimization and extrap- olate beyond the convex hull. Across diverse benchmarks, including challenging homogeneous data setting, ECA consistently breaks the PCC plateau, achieving significant improvements in correlation without compromising MSE performance.
INTRODUCTION
The attention mechanism is powerful tool for regression tasks where each input sample comprises multiple elements, such as tokens or image patches, and the elements' embeddings are aggregated using attention mechanism predict a continuous target (Lee et al. (019 Martins al. 2020 Born & Manica 2023). This paradigm is widely used in areas like digital pathology, time-series prediction and emotional analysis (Jiang et al. 2023 Ni al. 2023 Zhang et al. (023). For example, in video- based sentiment analysis, the attention mechanism is used to aggregate frame embeddings within video clip predict emotion variables, such sentiment intensity, arousal, and valence (Truong &
Lauw 2019
Xie et al. 2024).
Two characteristies are common in such applications. First, the in-sample data frequently exhibits higher homogeneity than cross- cross-sample data. For example, in pathology images, nearby regions tend to share more similar tissue or cell types than distant regions. Likewise, in video-based sentiment analysis, frames from the same clip are more similar than frames from different clips. Second, the regression performance is not simply evaluated by magnitude (e.g.. Mean Squared Error (MSE) but also by shape- the relative ordering of predictions, measured by the Pearson Correlation Coefficient (PCC). In many settings, capturing the correct correlations matters more than predicting the exact values (Pandit & Schuller 2019). For instance, in spatial transcriptomics, capturing the relative trend of gene expression (e.g., co-expression) is more informative than predicting exact magnitudes (Xiao) et al. 2024). To emphasize shape while retaining magnitude performance, models commonly

---

Published as a conference paper at ICLR 2026
Figure (a) Illustration of video-based sentiment analysis example. A sample is considered homogeneous when its within-sane sample dispersion ô is below dg- (b) A convex attention yields an aggregated embedding inside the convex hull of the sample's embeddings. (C) Our ECA extrapolates beyond the hull to amplify within-sample contrasts.
P5
Contes Aggrogatio
std(/4 Mol
RMS(
Frume Embeddings
Emotion Regression
Convex A Aggregator
Our BCA
(9)
(b)
(c)
trained with a joint loss function, such as Liotal MS + Apcc 1- p) where p is the PCC and ApcC
is a hyperparameter weight (Liu et al. 2022 Zhang et al. 2023 Zhu et al. 2025).
However, such joint training strategy applied to attention- regression model frequently exhibits puzzling PCC plateau, even with large PCC weight òpce. As shown in Figure 2 the PCC curve's 's slope flattens early in training, failing to improve further even while the MSE continues to decrease. This empirical phenomenon is partieularly severe with high in-sample homogeneity data. The un- derlying mechanisms driving this plateau remain unclear and it raises a critical question: why does the regression model fail optimize the correlation effectively?
In this work, we provide the first theoretical investigation into this question, analyzing the limitations of standard attention from two distinet perspectives: optimization dynamics and model capacity.
Limitation 1: Conflict in Optimization Dynam-
15
1.0
(A)
ies. Our analysis reveals a direct relation between the decrease of MSE and the flattening of the PCC curve the attention- based regression model. Us-
Val (A)
10
0.8
TTrann (CC
MSE
PCC
ing a decomposition of MSE (Proposition [. we
0.6
Train MSE
Val
show the following. Although training continues to reduce MSE by matching the predictions mean and standard deviation, the gradient the correla- tion term is paradoxically attenuated, flattening the PCC curve. This effect is amplified by in-sample homogeneity: when embeddings within sample are similar, the within-sample dispersion term in crease. The plateau becomes more noticeable as the PCC gradient shrinks and further suppresses the magnitude of the gradient.
0.4
Train PCC (B)
Val PCC (B)
250
Epoch
Figure 2: PCC plateau example: PCC flat- tens at early epoch while MSE continues to de-
within-sample homogeneity increases, specifi- cally, dataset more homogeneous than B.
Limitation 2: Model Capacity of Convex Aggregation. Notice that softmax attention is convex combination of element embeddings; we compare convex attention aggregation to pooning aggregation and derive an upper bound on the achievable PCC gain (Theorem 2.2 depending on the radius of convex hull formed by the in-sample embeddings. Greater in-sample homogeneity will shrink the convex hull and tightens this bound, yielding fundamental capacity limit for PCC gain.
Extrapolative Correlation Attention. Motivated by these theoretical insights, we propose Extrap- olative Correlation Attention (ECA), novel framework designed to overcome these limitations. To mitigate the optimization conflict in Limitation 1, ECA introduces a Diipersion-NNoraaiiee PCC loss to counteracts the attenuation effect, restoring the magnitude of correlation gradient. We also employ Dispersion-Aware Temperature Softmax to prevent the gradient collapse and near- uniform attention distribution under in-sample homogeneity. To address Limitation 2, ECA in- troduces Scaled Residual Aggregation, which allows the model to extrapolate beyond the convex hull, mitigating the PCC gain limitation induced by convex aggregation mechanism.
Contributions. We theoretically investigate the PCC plateau when training attention-based regres- sion model with a joint MSE+PCC loss. We identify two limitations in optimization dynamics and in model capacity.
2

---

Published as a conference paper at ICLR 2026
Optimization Dynamics. We identify conflict in optimization dynamics: MSE opti- mization increases og to match yy. the magnitude of the PCC loss gradient scales with 1/0g leading to flattened PCC curve during training. In addition, We show that within-sampl sample homogeneity further shrinks the PCC gradient through the Jacobian of softmax attention. Both parts contribute to the PCC plateau phenomenon.
Model Capacity. We prove that for any convex aggregator (e.g., softmax attention), the achievable PCC improvement over mean-pooling can bounded by the within-sample data homogeneity.
To address these limitations, we introduce ECA, a novel attention- baseed regression framework with three components:
Motivated by the optimization limitation, re-scale the PCC objective to counteract the 1/ 1ay factor, restoring correlation gradients magnitude while minimizing MSE.
Motivated by the convex aggregator limitation, we incorporate Scaled Residual Aggrega- tion that enables controlled extrapolation beyond the convex hull.
To handle within-sample homogeneity, we propose a Dispersion-Awee Temperature Soft- max to prevent attention collapse and amplify the informative contrasts within homoge- neous samples.
Across diverse regression benchmarks, ECA consistently overcomes the PCC plateau and achieves higher correlation gains while maintaining competitive MSE. Ablation studies confirm the contribu- tion of each component.
2 THEORETICAL ANALYSIS OF CORRELATION LEARNING IN ATTENTION-BASEE BASED AGGREGATION
We theoretically analyze why the PCC plateau occurs when training attention- based regression with a joint MSE+PCC loss. First, we relate MSE and PCC through decomposition that motivates joint training. Then prove that softmax attention, convex aggregator, limits both the optimization dynamics and the model's expressive capacity. These limits hinder correlation learning and explain the observed PCC plateau.
2.1 PROBLEM SETUP
Data Notation. We consider regression dataset with batch of S samples (Ie, V.). Each sample r, set of n, element with embeddings h, (h,) 1- where has Rd Let Vs 9. be the ground-truth target and model prediction for sample. The batch-level empirical means are My: Pi and standard deviations are oy: Tg. Define centered targets and predictions as a, = yy By and
b, : 0. plg.
Attention-basel Aggregation. The attention-based model processes each input sample to produce scalar prediction. An attention scoring function famn(.) (e.g.. KQV dot-product similarity gating mechanism) scores each embedding in h, and produces attention logits z, fau (fhs ) E R with entries Sst [z E R. Softmax converts these logits to positive attention weights a₂ = Softmax(2,, (2.) on the probability simplex with entries as = [a,]; E and Ei= asi 1. The sample-level embedding the convex combination V, En Ogi h E Rdd Finally, linear regression head with weights w E Rd and bias e E R produces the scalar prediction: ÿ. w Tv,+c. Note that this formulation is backbone-agnostic: {h.i) represents the features the final layer of any deep architecture (e.g., a multi-layer Transformer). Since these models typically derive the final prediction with a convex attention pooling (e.g. [CLS] token aggregation) followed by linear projection, our analysis of the attention- based aggregation and its interaction with joint MSE and PCC optimization applies regardless of the depth or complexity of the preceding backbone. We provide a detailed discussion on the applicability to deep architectures in Appendix
Learning Objective. We measure Pearson correlation between targets and predictions over the batch by = Cy) To optimize both magnitude and correlation of the prediction, we use the popular joint loss Liaal MSE(y, ))= + Apcc(1 p) where λpcC 2 0 balances the terms.
dyoy
3

---

Published as a conference paper at ICLR 2026
Homogeneity Measures. For each sample, let the in-sample mean be μ4, 1 -==1 haj- We quantify sathin-sample homogeneity by (1) the in-sample dispersion oN = Zj=1 haj - μ l
and (ii) the convex-hull radius R. = max; h u2ll2. These quantities are related by inequality
o, - R. I Vn, a with proof in Lemma D.3 Cross-sample homogeneity is summarized by the
root-mean-square (RMS) values Ø = (3 Σ s=1 09)
1/2 and R == (s E R2) 1/2
2.2 PRELIMINARIES: THE INTERPLAY BETWEEN MSE AND PCC
Proposition 2.1 (MSE Mean-std-coorelltioon Decomposition). The MSE between y and ğ can be decomposed as:
MSE(y, 9) (uy Myy
(yy -Oy)² +2 + 09 (1 (1-p).
(1)
maithhng
std matching
weighted correlation
Lemma 2.1 (Scaling- intiiiigiivviiinee of PCC). Let 0 and
R. PCC(y, mÿ + n) -PCC(V,
We defer the proof for Proposition 2.1 and Lemma 2.1 to Appendix A
Remark 2.1. Proposition 2.7 and lemma are model-independent and hold for any regressor.
Remark 2.2 (The PCC Plateau). By Proposition minimizing jointly targets (i) mean match- ing, (ii standard deviation matching, and (īii) weighted correlation. Because MSE is sensitive to affine transformations while PCC is invariant (Lemma(.1), optimization can reduce MSE by mainly adjusting mean and scale, with little improvement correlation. This explains why MSE may keep decreasing while PCC plateaus, and motivates adding the explicit correlation term Xpcc p) to the objective.
The discussion above provides intuitions for the PCC plateau phenomenon. We further empiri- cally validate this phenomenon on 8 UCI regression datasets using multi-layer Transformers (Ap- pendix D. Figure 7 shows consistent pattern across all tasks: PCC curve flattens before MSE convergence. This consistent pattern confirms that "PCC plateau" is a general correlation learning failure mode under stteetion-based joint optimization setting. We now provide a formal analysis by characterizing the PCC gradient attention regression models.
2.3 OPTIMIZATION DYNAMICS: THE CORRELATION GRADIENT BOTTLENECK
We study how the gradients propagate through the attention aggregator when optimizing the joint loss Liotal LMSE A p). We derive the gradients of both MSE and PCC with respect to the attention logits žsi to understand the optimization dynamics.
Lemma 2.2 (Softmax Aggregator Jacobian). The derivative of the aggregated embedding v, with respect to a pre-softmax logit z is 0v./Oz.: a.z -v,) and consequently, Oū./Oza, =
Q,. w (h,- Vs).
Theorem 2.1 (Gradient of PCC w.r.t. Attention Logits). The derivative of Pearson correlation p with respect to a pre-softmax logit 2si is
0p
0x
by a. (bai Vz).
(2)
Oz.1
Sog
og
To understand the interplay during joint optimization, we also examine the gradient of the MSE loss. Lemma 2.3 (Gradient of MSE w.r.t. Attention Logits). The derivative of MSE with respect pre-softmax logit Zei is
dLMSE
2 (Ú= ys) w..ww (h.i Vs).
Ø2,t
Gradient Decomposition and the Optimization Conflict. Comparing the PCC gradient (Equa- tion 2 and the MSE gradient (Equation (3)) we observe they share the same local structure factor Lgi ass w' (h.i ).). ). which governs attention adjustment within sample s. The difference lies entirely in the global scaling factors which depend on overall batch statisties:
4

---

Published as a conference paper at ICLR 2026
ÖL
5 9s
MSB
Loi where gA := 2(ūs -yo).
(4)
Ozgi
0p
5
POC
Lees
PCC = where g,
a 4y
b
(5)
0zgi
0
The relative impact of MSE versus PCC optimization is determined by the ratio of these global factors. We analyze this ratio using their Root Mean Square (RMS) values across the batch.
Corollary (PCCMSEE Gradient Ratio Decay). Assuming E [0, 11, the RMS ratio of the global scaling factors across the batch is bounded by:
RMS, (g) )C) I
1
1
Tglobal
(g
2 3/2
(6)
Corollary 2.1 identifies a gradient bottleneck where the PCC
signal attenuates relative MSE
of 011/0)/2) 3/2) This bound empirically validated in Figure on syntheric dataset
Gradient Ratio Magnitude
illustrating how magnitude matching dominates the optimiza- tion dynamics. We further analyze the magnitude of the PCC gradient alone, which reveals dependence on in-sample homo- geneity.
Gradient Ratio PCC/MSE
Theoretical Upper Bound
Prediction
Predictian d;
Valid Regian
Corollary 2.2 (PCC Gradient Magnitude Bound). The magni- tude of the PCC gradient in Theorem 2.7
can be bounded by
0p 02,5
1
4
n
I
|wllz
(7) Figure 3: Validation of gradient ra- tio (PCC/MSE) decay. The RMS ratio of PCC vs. MSE gradients (blue) is strictly constrained by the theoretical upper bound (red). The increase in prediction dispersion 0g (green) during training drives the attenuation the PCC gradient signal relative the MSE gradient.
0g
S
regression in in-sample weights dispersion
prediction deviation
batch seale
are provided in Appendix B.I Proofs for Lemma 2.3 Corollary 2.1 and
Detailed derivations for Lemma 2.2 Theorem 2.1
Corollary 2.2 are provided Appendix C
Remark 2.3 (The Two Bottlenecks of Softmax Attention for Correlation). The gradient analysis reveals two key bottlenecks in optimization dynamics that drive PCC plateaus:
1. Dominance of the MSE Gradient. Corollary 2.1 reveals a critical conflict the joint optimization: the ratio of the PCC gradient magnitude relative to the MSE gradient magnitude de- 3/2),
1.0-
cays rupidly a rate of (1) Training with the joint loss minimizes the MSE std-matching term in Equation ①. which drives 0g toward the target standard deviation a As 0g typ- ically increases during early training (see Figure F). the rela- tive contribution of the PCC gradient diminishes significantly. Consequently, the optimization becomes dominated by the MSE objective (magnitude matching), effectively downplaying PCC Figure 4: The standard deviation optimization (shape matching) and causing the plateau. even of predictions og increases during when the PCC loss weight XpeC is large. This motivates opti- training to match the standard de- mization strategies that counteract this rapid attenuation.
Baseline Training
0.5
ECA Training
Ground-truth
1000
2000
Epoch
viation of labels og under MSE loss.
2. Dependence on Within-sample Homogeneity. The gradient bound in Corollary 2.2 is proportional to the in-sample dispersion о.. When a sample's 's elements are homogeneous, provements to PCC via attention adjustment. Furthermore, since attention scoring functions are generally continuous, homogeneous inputs lead low-variance logits 2ei- Under fixe-temppeatuue softmax, this results in near- unirrrm weights ag R 1/ng, suppressing per-sample selectiviry. These effects motivate mechanisms that adapt attention sensitivity to the in-sample dispersion.
small and the PCC gradient magnitude reduces, effectively hindering im-
5

---

Published as a conference paper at ICLR 2026
2.4 MODEL CAPACITY: THE PCC CEILING OF CONVEX AGGREGATION
Beyond optimization dynamics, we study the limits of the aggregator's expressivity. Softmax atten- tion performs convex combinations, restricting aggregated embedding v, to the convex hull of the in-sample embeddings (h We study how much PCC improvement can be achieved by any con- vex aggregator over mean-pooling and show a capacity limit governed by in-sample homogeneity.
Prediction Decompositon. embedding plg. The prediction can be decomposed as: 9. = (w p + w (v. - Me). where
We decompose the sample embedding v, relative to the mean- meean-polingg
A
Y, is the prediction using mean- meca--ooling aggregation, and Aÿs is the attention citeentioniinduced perturbation. Theorem 2.2 (PCC Gain Bound for Convex Attention). Let := std, (T.) be the standard de- viation of the mean-pooling predictions; define baseline Po PCC(y, and p is the PCC after applying any convex attention mechanism. Provided ||w 2 > 0 and R oo/ Fo/l|w||2.
2 R
pol ou/ Su/l[w// 12 It
Detailed proof is provided in Appendix D
Remark 2.4. Theorem 2.2 shows that the PCC improvement bound of any convex attention de- pends only on the ratio between convex hull radius R (reflecting in-sample homogeneity) and the normalized standard deviation of mean-pooling baseline (ao/ [ool[w||2/ The bound is scale-invariant and independent of the regression head magnitude since To /o/l = std,(w' pt)/ = std, (ow.((ww)))) ( [l))).)).. When in-sample homogeneity is high (R is small), no convex attention can substantially increase correlation. The limitation arises because v. is confined to the convex hull; a small hull restricts the magnitude of adjustments to the aggregated embedding, thereby limiting the potential PCC improvement and motivating mechanisms that can extrapolate beyond it.
2.5 SUMMARY OF THEORETICAL INSIGHTS
Our analysis reveals that the difficulty of optimizing PCC using softmax attention stems from two aspects: optimization dynamics and model capacity. Remark (2.3 highlights the vanishing PCC gra- dients due cross-sample dispersion attenuation 1/0g and weak in-sample dispersion. Remark 2.4 further shows that any convex aggregator is restricted to the convex hull, which limits the possible PCC gain by ratio ft/ (aoo/ l122). Together, these effects explain the plateau and motivate a novel attention mechanism proposed the next section that addresses the identified bottlenecks accord- ingly.
3 EXTRAPOLATIVE CORRELATION ATTENTION (ECA)
Our analysis has identified fundamental limitations in softmax attention for optimizing joint MSE+PCC objective. To address these issues, we propose Extrapolative Correlation Attention (ECA), a novel drop-in attention module for regression that enhances both optimization and ex- pressivity. ECA incorporates three components: (i) Scaled Residual Aggregation break the con- vex hull constraint; (ii) Dispersion-Awee Aware Temperature Softmax, to avoid gradient collapse; and (iii) Dispersion-Normann Normalized PCC Loss, which compensates the 1/0, attenuation in correlation gradients.
3.1 BREAKING THE CONVEX HULL WITH SCALED RESIDUAL AGGREGATION (SRA)
Theorem 2.2 shows that any convex attention mechanism is sspacity-limiter in correlation improve- ment because the aggregated embedding V. lies inside the convex hull of the in-sample embeddings (ha)). This PCC gain bound is especially tighter when in-sample dispersion is low. To relax this limit, we introduce Scaled Residual Aggregation (SRA): instead of a strict convex aggregation, the model extrapolates along the residual (h,; M.). allowing v, to move beyond the convex hull.
Given the mean embedding μ, = 1/n, Σ. h we define the residual Av, as the attention- weighted deviation from the mean: Av. Σ, asi (h.i 1t). SRA scales this residual by

---

Published as a conference paper at ICLR 2026
learaalle factor %/8 1:
ECA
μs + 7/8 Av, ps + Yo assi 1t)).
We parameterize 7s using small, sample-specific MLP conditioned on the mean embedding Ms and use a shifted Softplus activation to ensure ") 2 1, that " Softplus(MLPs (1.)).
The factor 7 allows the model to amplify weak in-sample contrasts. When 1, SRA reduces to standard convex attention aggregation. For % > 1, the model extrapolates beyond the convex hull, and more importantly, it breaks the convexity constraint. In standard attention, the deviation ||vv,ll bounded by the radius of the convex hull R. SRA expands the reachable space by increas- ing the effective radius and fundamentally bypasses the capacity limit derived for convex aggregators in Theorem 2.2 In practice we optionally clip Y at maximum (e.g., Ymmax = 2) or add a regularizer
(7+ 1)2 to discourage excessive scaling.
3.2 DISPERSION- AWARE TEMPERATURE SOFTMAX (DATS)
While SRA enables extrapolation beyond the convex hull, the model still needs a informative direc- tion to extrapolate. In homogeneous samples, standard softmax produces flat attention Oggi = 1/n, (Remark 2.3). which pulls the aggregated embedding toward μt, and makes the residual Equa- tion 9 small (Av, z 0). With small residual, SRA provides little benefit. To address this, introduce Dispersion-Awant Temperature Softmax (DATS), which adapts the attention temperature the in-sample dispersion, sharpening attention when homogeneity is high:
We modify the softmax attention for Equation within-sample dispersion:
p with sample-specific temperature т. reflecting
Zsi
asi softmax
To Tmin +
hgg
1
(10)
V Th
Isisa,
Here Tiin lower-bounds the temperature for stability, and 2 0 is a hyperparameter that controls sensitivity. When embeddings within a sample are homogeneous, тs has lower value so small differences between logits žœ; become sharper after attention, yielding a meaningful deviation Av, that SRA can effectively amplify.
3.3 STABILIZING OPTIMIZATION: RISPERSION-NORMLIE
PCC Loss (DNPL)
Theorem 2.1 shows that the correlation gradient is attenuated by 1/ag. As MSE optimization im- proves standard deviation matching, ag increases, which shrinks the PCC gradient and contributes to PCC plateau. We counteract this attenuation effect with a Dispersion-Normalizee Normalized PCC Loss (DNPL), which rescales the PCC term by the current prediction standard deviation while blocking its gradient:
Âvcc StopGrad(ag) (1 p).
(11)
The StopGrad(() operation ensures we only adjust the gradient magnitude to counteract the attenu- ation, while leaving the learning objective's stationary points unchanged.
3.4 OVERALL OBJECTIVE
The complete ECA framework, including SRA and DATS, is fully differentiable and can be trained end-to-end. The overall learning objective combines the primary regression loss (MSE), the normal- ized PCC loss (DNPL), and the extrapolation regularizer.
Lioual LMSE + Apcc Σpcc + bey.
(12)
4 RELATED WORKS
Correlation Learning and Optimization. Correlation is a core metric in biological and medical areas (Langfelder & Horvath 2008 Lawrence & Lin 1989). In these domains, PCC is standard

---

Published as a conference paper at ICLR 2026
criterion for evaluating regression performance (Kudrat al., 2025 Long al. 2023 Because PCC is differentiable, it is often optimized directly as a loss in regression pipelines (Kudrat et al,
2025 Avants al. 2008). From multi-task perspective. many works combine PCC with MSE to
balence the prediction magnitude and shape (Yang et al. 2023 Liu et al. 2022 Balakrishnan et al.)
2019). However, the interaction dynamics between MSE and PCC under joint optimization remain
underexplored, motivating our analysis of gradient coupling and the observed PCC plateau.
Softmax Attention Aggregation. Softmax attention a comeestone component in the backbones of many representative regression models (Zhou et al. 2021 Gorishniy et al. 2021] Kim et al. 2019).
Despite its empirical success, recent theory has revealed expressivity limits of softmax mappings in related contexts (Yang et al., 2017 Kanai et al.] [2018]
Bhojanapalli et al.) 2020). However, to
our knowledge, no prior work analyzes the model capacity of softmax attention in terms of upper
bounds on achievable PCC improvements, especially with high in-sample homogeneity data.
5 EXPERIMENTS
We evaluate our ECA on four settings, including the challenging high in-sample homogeneity tasks. The evaluations consist of: (i) a synthetic regression dataset with controllable in-sample homogene- ity: (ii) three representative tabular regression benchmarks from the UCI ML Repository (Asuncion ctt al. 2007): (iii a clinical pathology dataset for spatial transcriptomic prediction, where nearby regions exhibit high homogeneity; and (iv) multimodal sentiment analysis (MSA) dataset, where consecutive video frames are highly homogeneous.
As ECA is drop-in replacement for softmax attention, integrate it into existing attention based regression models for each benchmark to measure the performance improvement. More details are provided in Appendix G
5.1 EXPERIMENTAL SETUP
We construct a synthetic dataset to validate our theory and proposed ECA method. We synthetic N samples, each with K element embeddings samples. In each sample, one key element carries signal along fixed unit direction w", while the remaining K-1 background elements cluster around shared sample mean. The label y the pro- jection of the sample mean onto w* with a term proportional to the key strength and small additive noise. We control within-sample homogeneity via 1) (larger v) means the key deviates further from the mean), yielding four homogeneity levels ō {0.10, 0.24, 0.42, 0.73) where lower ō indicates higher homogeneity. We compare regression model with one layer of ECA to one layer of standard softmax attention and report MSE and PCC.
Synthetic Dataset.
dimensions as input
UCI ML Repository Datasets. We evaluate on three representative tabular regression benchmarks: Appliance Energy Prediction (28 features, 1 target) (Candanedo) (017), Online News Popularity (58 features, 1 target) (Fernandes & Sernadela, 2015). and Superconductivity (81 features, target) (Hamidieh) 2018). We integrate ECA into the attention layer of the FT-Transformer (Gorishniy ct al. 2021 and report mean absolute error (MAE), MSE, and PCC.
Spatial Transcriptomic Dataset. We test spatial transcriptomics prediction from pathology images on the 10xProteomic dataset (IOx Genomics 2025 Yang et al., 2023), which contains 32, 032 slide- image patches paired with gene-expression measurements of breast-cancer slides. We follow the data processing and experimental settings of the EGN baseline (Yang et al., 2023), which jointly optimize MSE+PCC loss. We adopt ECA methods onto the EGN baseline and report MSE, PCC@F, F, PCC@S, and PCC@ M as evaluation metrics. The training set exhibits high in-sample embedding homogeneity with = 0.068 versus cross-sample oo = 0.164.
Multimodal Sentiment Analysis (MSA) Dataset. We use MOSI (Zadeh et al. 2016). standard MSA benchmark consists of 2, 199 monologue video clips with audio and visual inputs. As il- lustrated in Figure D consecutive frames within clip are more similar than frames across clips. Quantitatively, the video frames shows strong within- sample homogeneity with ā = 0.098 versus cross-sample du 0.170. We follow the commonly used MOSI processing protocol from THUIAR releases Yu et al. 2020 2021). Video frame embeddings are extracted with OpenFace (Amos et al.) 2016). Consistent with prior work, we report F1, PCC, and MAE evaluation metrics. We include
8

---

Published as a conference paper at ICLR 2026
10 representative baselines. ALMT (Zhang et al. 2023), the leading baseline optimizing with MSE loss, is selected to incorporate the ECA method.
Table 1: Results on three UCI tabular regression tasks. FT-Transformer baseline. Rows marked "w/o SRA/DATS/DNPL" are ablation studies that remove the corresponding ECA components, "w/o" "without" and bold indicating best result.
"+ECA" denotes adding ECA onto the
Appliance
Online News
Superconductivity
Method
MAE 4
MSEx 4 PCC
MAE 4
1 4
PCC
MAE +
MSEx
PCC
franmoerrrr
39.333
6.108
0.556
0.641
0.724
0,408
8.793
1.772
0.920
ECA (full)
38.665
5.790
0.598
0.631
0.712
0.420
7.976
1.582
0,930
ECA (w/o SRA)
39.208
5,994
0.575
0.637
0.725
0.410
8.377
1.695
0.920
ECA (w/o DATS)
38,906
6.037
0.561
0.645
0.740
0.418
8.630
1.709
0.927
ECA (W/ DNPL)
39.742
5.910
0.583
0.640
0.719
0.418
8,466
1.671
0.922
G = 0.10
= 10.24
00.4
G 0,73
Handard-Train MSE
Standard-Val MSE MSE
6-
ECA-Val MSE
4
Mean-pooling Train PCC
2
2
Standard-Train
Val PCC
250 SU0
750
ECA-Train PCC
Epoch
Epoch
Epoch
Epoch
PCA-SA
Figure 5: PCC and MSE curves on synthetic datasets with in-sample homogeneity 10.10, 0.24, 0.42, 0.73].
5.2 RESULTS AND ANALYSIS
Synthetic Dataset. Figure 5 shows case studies the training and validation curves under dif- ferent in-sample dispersion (with 0.10, 0.24, 0.42, and 0.73). The horizontal line indicates input embeddings. This result confirms theoretical study in three ways: 1) as the homogeneity intensifies, both the PCC of our ECA and baseline model decrease and the MSE get higher (task harder), and the achievable PCC improvement of convex (e.g., softmax) attention over mean-pooling decreases (Theorem 2.2 2) ECA consistently outperforms standard attention in both PCC and MSE across all four σs, achieving PCC gains of 4.80%, 5.76%, 4.68, 3.05% and MSE reductions of 20.3%, 40.8%, 54.0%, 66.7% (in order of in- creasing Ø), showing its ability to explore beyond the convex hull and improving the PCC without compromising the MSE. 3) The PCC curve of ECA keeps improving and converges later and at higher value than the standard attention baseline, indicating the effectiveness of our proposed DNPL. in mitigating the PCC attenuation effect identified in Remark 2.3
the PCC achieved by mean-pooling
Table 2: Results MOSI. from THUIAR's GitHub (Yu et al. 2020 2021): from (Hazarika et al., 2020): ** reproduced from
UCI ML Repository Datasets. Across three UCI tabular regres- sion tasks, adapting our ECA module into the FT-Transformer yields consistent improvements in both magnitude (MSE and MAE) and shape (PCC) met- rics. On Appliance, PCC in- creases by 0.042 and MSE de- creases by 0.318 x 103; on On- line News. PCC increases 0.012 from 0.408 while MSE decreases
public code with provided typer-parametert
Method
F1 f
MAE 4
PCC
TFNt 7adeh et al. 2017)
0.791
0.947
0.673
LMF
Liu et al. 2018)
0.824
0.917
0.695
EF-LSTM Willaams et
2018h
0.785
0.949
0.669
LF-DNNI Williams et al.
2018 8a
0.786
0.955
0.658
Graph-MFNT Zadeh et al.
0.784
0.956
0.649
MulT
Tsai et al. 2019
0.828
0.871
0.698
MISA
Hazarika al. 2020
0.836
0.777
0.778
0.012 from 0.724; on Supercon- ductivity, PCC - and MSE 1.772 x 102 + 1.582 x
ICCN*
Sun al. 2020
0,830
0,860
0.710
DLF**
Wang
2025
0.850
0.731
0.781
ALMT
Zhang
2023
0.851
0.721
0.783
10a. MAE also decreases across all datasets. These improvements provide strong evidence that ad- dressing our identified PCC gradient limitation and the convex attention capacity limit substantially
ALMT+LPCC
0.834
0.731
0.791
ALMT+Lroc + IEAA
0.859
0.695
0.806

---

Published as a conference paper at ICLR 2026
Table Three-fold regression PCC and MSE on 10xProteomic dataset.
Method
PCC@F1
PCC@S
PCC@M f
MSEJ
EGN
0.602 +: 0.160
0.647 + 0.164
0.629 +: 0.135
0.056 = 0.047 0.051 = 0.048
EGN+ECA
0.690 + 0.202
0.724 = 0.191
0.716 + 0.168
EGN Train
EGN Val
EGN+ECA Train
EGN+ECA Val
Foldo PCC&M
Foldi
Fold2 PCORM
Foldo
Foldı MSE
Fold2
1.0
1.0
1.0
0.10
0.10
0.10
0.5
0.5
0.05
0.05
0.05
0.0
0.0
0.00
0.00
Epoch
Epoch
Epoch
Epoch
Epoch
Epoch
Figure 6: PCC@M and MSE curves for EGN baseline and EGN+ECA on 10xProteomic dataset under each fold. Curves for PCC@S and @CC@F please refer to Figure 8 and Figure 9
mitigates the PCC plateau without compromising MSE. The ablations further support the contribu- tion of each components: removing DATS lowers Appliance PCC to 0.561, while removing SRA reduces Online News PCC to 0.418 and removing DNPL decreases Superconductivity PCC to 0.922.
Spatial Transcriptomic Dataset. We follow the EGN (Yang et al. 2023) setting and report three- fold results. Figure 6 plots PCC and MSE curves for training and validation. Across all folds, integrating ECA consistently improves both metrics. In fold 2, the EGN baseline's PCC flattens near epoch 4, but the MSE keeps decreasing, indicating clear PCC plateau. In contrast, PCC of EGN+ECA continues to increase, effectively breaking the PCC plateau and improving final valida- tion PCC by 16.51%. The same pattern holds in folds 0 and 1. Throughout training, EGN+ECA also achieves comparable or lower MSE than EGN alone, indicating that ECA successfully preserves magnitude information while achieving better PCC. Table 3 summarizes the overall performance, where EGN+ECA achieves +44644% for PCC@F, +11.9% for PCC@S, +13.81% for PCC@ M, and 9.83% reduction in MSE, showing that BCA effectively and robustly alleviates the PCC plateau without compromising the MSE.
Multimodal Sentiment Analysis (MSA) Dataset. Since ALMT optimizes MSE loss only, we test two settings: (i) ALMT with an additional PCC loss; and (ii) ALMT with ECA adapted into the video attention encoder and trained with the dispersion-noormaiieed PCC loss. Table 2 reports results of 10 baselines. Adding a PCC term yields small PCC gain but degrades F1 and MAE, reflecting the MSE-PCC conflict under strong in- iampee homogeneity. In contrast, adding ECA improves all metrics, achieving +2.3% PCC increase without sacrificing Fl or MAE.
6 CONCLUSION
This work presents the first theoretical investigation into the PCC plateau phenomenon observed when training attention-based regression models with a joint MSE+PCC loss, particularly under high data homogeneity. Our analysis identified two fundamental limitations in standard softmax at- tention: conflict in optimization dynamics that attenuate the correlation gradient, and an achievable PCC bound imposed by convex aggregation. To address these bottlenecks, we introduced ECA, novel plug-in framework incorporating mechanisms to stabilize optimization, adapt to homogene- ity, and extrapolate beyond the convex hull. Comprehensive experiments validate theoretical insights and demonstrate that ECA successfully breaks the PCC plateau, achieving significant cor- relation gains while maintaining competitive magnitude performance.
10

---

Published as a conference paper at ICLR 2026
7 ACKNOWLEDGEMENT
This work was partially supported by US National Science Foundation IIS-2412195, CCF-2400785, the Cancer Prevention and Research Institute of Texas (CPRIT) award (RP230363), the National Institutes of Health (NIH) RO1 award (IR01AT190103-01) and Microsoft Accelerate Foundation Models Research (2024).
8 REPRODUCIBILITY STATEMENT
We support reproducibility as follows.
Theoretical Study
The appendix has complete proofs for all theorems, lemmas, and corollaries in
the paper. See Appendix A results from Section 2.2 Appendix for Section 2.3 and Appendix [ for Section
Dataset Processing pendix G
Processing details for all datasets (synthetic and real-world) are listed in Ap-
Code Reproducibility We include an anonymous zip file with implementations for the synthetic and spatial transcriptomics datasets along with the hyperparameters we use for review. For the spatial transcriptomic dataset, we follow the EGN baseline preprocessing protocol. Due to the limited space, we did not include the dataset. The README.md explains how to get the data and run the code. The complete code can be found https: / /gi tuub com/jyan97 ECA hww.:ooott.veee-rrreeaattoootoee
REFERENCES
10x Genomics. 10x genomics datasets. htttp::/www.. datasets 2025. Dataset portal; accessed 2025-09-21.
Brandon Amos, Bartosz Ludwiczuk, and Mahadev Satyanarayanan. Openface: A general-purpose face recognition library with mobile applications. Technical report, CMU-CS-16-118, CMU School of Computer Science, 2016.
Reza Arefidamghani, Roger Behling, Alfredo N lusem, and Luiz-Rafael Santos. circumcentered- reflection method for finding common fixed points of firmly nonexpansive operators. arXiv preprint arXiv:2203.02411 2022.
Arthur Asuncion, David Newman, al. Uci machine learning repository, 2007.
Brian B Avants, Charles L Epstein, Murray Grossman, and James C Gee. Symmetric diffcomorphic image registration with cross-correlation: evaluating automated labeling of elderly and neurode- generative brain. Medical image analysis, 12(1):26-41 2008
Guha Balakrishnan, Amy Zhao, Mert R Sabuncu, John Guttag, and Adrian V Dalca. Voxelmorph: learning framework for deformable medical image registration. IEEE transactions medical imaging, 38(8):1788-18) 2019.
Srinadh Bhojanapalli, Chulhee Yun, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar. Low- rank bottleneck in multi-head attention models. In International conference on machine learning, pp. 864 873. PMLR, 2020.
Jannis Born and Matteo Manica. Regression transformer enables concurrent sequence regression and generation for molecular language modelling. Nature Machine Intelligence, 5(4):422444, 2023.
Luis Candanedo. Appliances Energy Prediction. UCI Machine Learning Repository. 2017. DOI: https:/doi..og/102244//////
11

---

Published as a conference paper at ICLR 2026
Vinagre Pedro Cortez Paulo Fernandes, Kelwin and Pedro Sernadela. Online News Popularity. UCI Machine Learning Repository, 2015. DOI: https//dbb.orr/12224355//3333
Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Advances in neural information processing systems, 34:18932-18943, 2021.
Kam Hamidieh. Superconductivty Data. https///bb..rg/112244425C54544
UCI Machine Learning Repository, 2018. DOI:
Devamanyu Hazarika, Roger Zimmermann, and Soujanya Poria. Misa: Modality-invarian and- specific representations for multimodal sentiment analysis. In Proceedings of the 28th ACM in- ternational conference on multimedia, pp. 1122-1131, 2020.
Shuai Jiang, Arief A Suriawinata, and Saeed Hassanpour. Mhattnsurv: Multi-head attention for survival prediction using whole-slide pathology images. Computers in biology and medicine, 158: 106883, 2023.
Sekitoshi Kanai, Yasuhiro Fujiwara, Yuki Yamanaka, and Shuichi Adachi. Sigsoftmax: Reanalysis of the softmax bottleneck. Advances in Neural Information Processing Systems, 31, 2018.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnel,, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. In International Conference Learning Representations, 2019.
Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6990 2014.
Dilfira Kudrat, Zongxia Xie, Yanru Sun, Tianyu Jia, and Qinghua Hu. Patch-wise structural loss for time series forecasting, In Fecty-second International Conference on Machine Learning, 2025.
Peter Langfelder and Steve Horvath. Wgena: an package for weighted correlation network analysis. BMC bioinformatics, (1):5599, 2008.
I Lawrence and Kuei Lin. A concordance correlation coefficient to evaluate reproducibility. Bio- metrics, pp. 255-268, 1989.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set ionnuuttaiioninvaaiin neural networks. In Interna-
transformer: A framework for attention-basee fional conference on machine learning, pp. 3744-3753. PMLR, 2019.
Miao Liu, Jing Wang, Liang Xu, Jianqian Zhang, Shicong Li, and Fei Xiang. Bit-mi deep learning- based model to non-intrusive speech quality assessment challenge in online conferencing appli- cations. In INTERSPEECH, pp. 3288 2888-3292, 2022.
Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, Amir Zadeh, and Louis-Philippe Moreney. Efficient low-rank multimodal fusion with modality-specific factors. arXiv preprint arXiv: 118860000044 2018.
Yahui Long, Kok Siong Ang. Mengwei Li, Kian Long Kelvin Chong, Raman Sethi, Chengwei Zhong, Hang Xu, Zhiwei Ong, Karishma Sachaphibulkij. Ao Chen, al. Spatially informed clustering, integration, and deconvolution of spatial transcriptomics with graphst. Nature Com- munications, 14(1):1155, 2023.
André Martins, António Farinhas, Marcos Treviso, Vlad Niculae, Pedro Aguiar, and Mario Figueiredo. Sparse and continuous attention mechanisms. Advances in Neural Information Pro- cessing Systems, 01-20999-21000 2020.
7elin Ni, Hang Yu, Shizhan Liu, Jianguo Li, and Weiyao Lin. Basisformer: Attention- time series forecasting with learnable and interpretable basis. Advances in Neural Information Pro- cessing Systems, 36:7 16.71222-71 2023.
Vedhas Pandit and Bjöm Schuller. The many-to many-to-man many mapping between the concordance correlation coefficient and the mean square error. arxiv e-prints, art. arXiv preprint arXiv: /90.00 2019.
12

---

Published as a conference paper at ICLR 2026
Zhongkai Sun, Prathusha Sarma, William Sethares, and Yingyu Liang. Learning relationships be- tween text, audio, and video via deep canonical correlation for multimodal language analysis. In Proceedings of the AAAI conference artificial intelligence, volume 34, pp. 8992-8999) 2020.
Quoe-Tuan Truong and Hady W Lauw. Vistanet: Visual aspect attention network for multimodal sentiment analysis. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 305-312, 2019.
Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the conference. Association for computational linguistics. Meeting, volume 2019, pp. 6558, 2019.
Meihong Wang, Fei Sha, and Michael Jordan. Unsupervised kernel dimension reduction. Advances in neural information processing systems, 23, 2010.
Pan Wang, Qiang Zhou, Yawen Wu, Tianlong Chen, and Jingtong Hu. DIf: dysentangled-languaaee focused multimodal sentiment analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 21180-21188, 2025.
Jennifer Williams, Ramona Comaneseu, Oana Radu, and Leimin Tian. Dnn multimodal fusion techniques for predicting video sentiment. In Proceedings of grand challenge and workshop on human multimodal language (Challenge-HMi) pp. 64-72, 2018a.
Jennifer Williams, Steven Kleinegesse, Ramona Comanescu, and Oana Radu. Recognizing emotions in video using multimodal dnn feature fusion. In Grand Challenge and Workshop on Human Multimodal Language, pp. 11-19. Association for Computational Linguistics, 2018b.
Xiao Xiao, Yan Kong, Ronghan Li, Zuoheng Wang, and Hui Lu. Transformer with convolution and graph-node co-embedding: An accurate and interpretable vision backbone for predicting gene expressions from local histopathological image. Medical Image Analysis, 91:103040, 2024.
Yifeng Xie. Zhihong Zhu, Xuan Lu, Zhiqi Huang, and Haoran Xiong. Infoenh: Towards multi- modal sentiment analysis via information bottleneck filter and optimal transport alignment. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation SAREC-COLING 2024), pp. 9073-9083, 2024.
Yan Yang, Md Zakir Hossain, Eric A Stone, and Shafin Rahman. Exemplar guided deep neural network for spatial transcriptomics analysis of gene expression prediction. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5039-5048, 2023.
Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W Cohen. Breaking the softmax 2017.
bottleneck: A high-rank rnn language model. arXiv preprint
Wenmeng Yu, Hua Xu, Fanyang Meng, Yilin Zhu, Yixiao Ma, Jiele Wu, Jiyun Zou, and Kaicheng Yang. Ch-sims: A chinese multimodal sentiment analysis dataset with fine-grained annotation of modality. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 3718 1118-3777222 2020.
Wenmeng Yu, Hua Xu, Ziqi Yuan, and Jiele Wu. Learning modality-speciff representations with self-supervised multi-task learning for multimodal sentiment analysis. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 10790-10797, 2021.
Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Moreney. Multimodal sentiment in- tensity analysis videos: Facial gestures and verbal messages. IEÉE Intelligent Systems, 31(6); 82 88, 2016.
Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis- Louis-Philipen Philippe Moreney. Tensor fusion network for multimodal sentiment analysis. arXiv preprint
2017.
AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis- Phillpph Morency. Multimodal language analysis in the wild: Cmu-mosel dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguis- fies (Volume 1: Long Popers), pp. 2236-2246, 2018
13

---

Published as a conference paper at ICLR 2026
Haoyu Zhang. Yu Wang, Guanghao Yin, Kejun Liu, Yuanyuan Liu, and Tianshu Yu. Learning tanguage-guiee adaptive hyper-modality representation for multimodal sentiment analysis. In The 2023 Conference Empirical Methods in Natural Language Processing. 2023.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 11106-11115, 2021.
Junchao Zhu, Ruining Deng, Tianyuan Yao, Juming Xiong, Chongyu Qu, Junlin Guo, Siqi Lu, Yucheng Tang. Daguang Xu, Mengmeng Yin, et al. Magnet: Multi-level attention graph network for predicting high-resolution spatial transcriptomics. arXiv preprint 1, 2025.
14

---

Published as a conference paper at ICLR 2026
Appendix
A PRELIMINARIES
A.1 PROOF FOR PROPOSITION 21
Proposition A.I (MSE Mean-std-correlationn Decomposition). Let y; U E be the ground truth and predictions across S samples. Let Mys Hg be the empirical means, and y og be the empirical standard deviations. The Mean Squared Error can be decomposed as:
MSF(y. D)
(J1g pyy) (ag -oy)² 200 0g (1 p)-
(13)
Proof. Write the error for sample i as
y/ D: [(y> (by) - (Us- [5)] + (uy 1k))
(14)
Squaring and averaging over 1..... S give us
N
MSE(U. 0)
Σ
(yr 0²)²
(15)
N
1=1
(ys (gy - -is (bg) ) + (Bty 12)
(16)
N
1=1
2(H4
Mg)
Σ
(yt Hyy (it m&)
(17)
i=1
The last vanishes since E,(r mby) Ev; ug) 0. Expanding the remaining square
and using the definitions of variances, Cov(y, 0) = Tg. we obtain
N
2
MSF(y, 0) = (ug M1) + o o
My)
-1g)
(18)
N
i=1
(Hg m4y) + o + of
20, Tgp.
(19)
Finally, rearranging of + oj -2 0g p (ag o,) 20, 0g (1 p) yields the claimed decom- position.
A.2 PROOF FOR LEMMA 21
Lemma A.1 (Scaling Invariance of PCC). Let E I \ {0} and n E R. For any sample {(y,, )
PCC (cC + n) = sign(m) PCC(y. 0)
(20)
Proof.
Using p(u,v) v) Cov(u.e) audi
and Umstn |mm| Tg.
PCC mÿ + n) = Cov(y, mÿ + +n) m Cov 9)
(21)
By Omitn 0g |m 0g
PCC(y, Ø) = sign(m) PCC(y.))
(22)
15

---

Published as a conference paper at ICLR 2026
B GRADIENT ANALYSIS OF CORRELATION
B.1 GRADIENT OF PEARSON CORRELATION W.R.T. ATTENTION LOGITS
Lemma B.1 (Softmax Aggregator Jacobian). The derivative of the aggregated embedding v, with respect to a pre-softmax logit žşi is 0v,/ 9/Ozzi a,i Vx) and consequently, 0ū./ (iz/ W. (h.i Vs).
Proof. Within fixed sample s, the aggregated embedding is V, ΣE= Baj. We first recall the derivative of the softmax function. The partial derivative of the j-th attention weight Ogj with respect to the i-th input logit 2 is given by:
daaj
(.j( asi);
(23)
Ozgi
where j.j is the Kronecker delta (ō.} if j. and 0 otherwise).
We can now compute the derivative of v, with respect to 2si using the chain rule:
Ov.
dasj haj
(24)
0zgt
Ozoi
j=1
n,
azj (gi
act)ba
(25)
The Σ aas j=1
Σ
(26)
/=1
The first term simplifies because dij is non-zero only when j =
C.y aggh
(27)
In the second term, Osi is independent of the summation index and can be factored out:
T2
R2
Σ
Cajanh, = aes
Σ
haj = V...
(28)
j=1
Combining these results, we obtain:
dv.
agihoi a,iV, (h.i v.).
(29)
Oz.
Consequently, since the prediction is U. = w V. + c, its derivative is:
Ôv
asi w wTww,hhh (hai V.).
(30)
Ozgi
Ōzsi
Theorem B.1 (Gradient of PCC w.r.t. Attention Logits). For any 8, the derivative of Pearson cor- relation with respect to pre- pre-softax logit Zei is
0p
1
0.
by p
a
(hai v.)-
(31)
Oz.:
Sog
og
Proof. We express the Pearson correlation p as the ratio p N/D, where N is the covariance and D is the product of standard deviations.
N = Cov 0) =
arbrs
D = -y0a
(32)
5
16

---

Published as a conference paper at ICLR 2026
Recall that d2 := yc Myy and b, ÿt My are the centered targets and predictions, respectively, satisfying E, a. = 0 and Σ, b = 0.
By the quotient rule, the derivative of is:
Op
ON D dz.i
N 0D D Ozgi
1 D
ON 0zgs
aD Ōzst
(33)
Zzgi
We compute the derivatives of N and D separately. Note that the logit 2ns only directly affects the prediction of sample s, i... 00. 02 = #
Step 1: Derivative of the Covariance (N).
ON
0b.
at
(34)
0zgi
8
0zgs
t=1
We expand the derivative of the centered prediction by Dt paji
bb.
Oi.
0pg
0ūt
1 8 Uh
Σ
(35)
az.:
Ozot
Dzgi
Dzs
5
w=1
Substituting this back into the expression for ON 0N/0z.i.
ON
Oit.
Diu
Σ
at
(36)
S
Øzsi
S
Zz
u=1
Σ
1
Σ
Σ
Ohn Zz.t
(37)
S
2zs
$2
Since the targets centered c, 0g 0), the second term vanishes:
ON
Σ at
(38)
Fzgs
az,s
Since 00./02.. = for t + s, the summation collapses to a single term:
ON
aj. 0z.i
S a
(39)
Step 2: Derivative of the Standard Deviation Product (D). Since og constant with respect to 2si. we have 0D /0z0 oy(0og/0zs). To find the derivative of dg, we first differentiate the variance of 8
1 Σ, b?.
0z.:
do 9
Ozsi
à
1 S
Σ
=
S
Σ
2b
Oz.s
bb.
(40)
Similar to Step 1, we substitute the expression for 0bx/0z, and use the fact that the predictions are centered (E, b. = 0):
00² 2
2
Oij
1
Jij
4
Σ6
b,
Σ
(41)
dz,1
S
Ozgs
5
dzst
2
die
2
0g Izos
Σ
b
Σ
by
Σ
(42)
Oz
S2
2
Σ
b
(43)
0z
17

---

Published as a conference paper at ICLR 2026
Again, since 09//02as 0 for t + s:
do 3
2 be S
09. Ozsi
9
(44)
Zzii
We the chain rule: (assuming og 0):
Br Bzsi
20 y Ø2.i
80g
Equating the two expressions and solving for
8og
8a.
8og 2
2
dog b. Aú.
(45)
8z.
S
0zet
Ozæt Sog
8zei
Therefore, the derivative of the denominator D is:
0D
Bog
oybe 0ū.
By Ozsi
(46)
Zzsi
Soy Ōzsi
Step 3: Combining the results. We substitute the derivatives of N (Equation 339 and (Equa-
tion (46) back into the quotient rule formula (Equation (33))
0p
ON
OD
0zgi
7
P 0zoi
(47)
0g dg.
Tybs
0g.
p
(48)
Oy
S Ozsi
Sog
Ozsi
bybs
02
p
(49)
So
σ4
dzsh
1
ag
P b 0g.
(50)
Sag
By
0g
azgi
Finally, we substitute the expression for 00z/Ozzi derived in Lemma B.1 (Equation 33 ):
Öp
02
b,
a.:W
(h.i V.).
(51)
azg1
Sog
oy σg
This concludes the proof.
C PROOFS FOR OPTIMIZATION DYNAMICS ANALYSIS
C.I PROOF OF LEMMA 2.3
(GRADIENT OF MSE W.R.T. ATTENTION LOGITS)
Proof. The Mean Squared Error (MSE) loss is defined
MSPF
Un
)z
(52)
1=1
We compute the derivative with respect to the attention logit asi using the chain rule:
0LMSE
ÖLMSE Fÿk
(53)
Dz.t R=1
Ozst
The derivative of the loss w.r.t. the prediction in is:
dLMSE
2(yn (n) (-1) 1-1)=3 S (ūr- yk))
(54)
Oij S
The derivative of the prediction ON w.r.t. the logit Zei is non-zero only if k = 8. From Lemma have:
2.2
Fÿ. a.; (h.i vz). Oz.1
(55)
Combining these results:
0LMSR Oz.i
3 (g. - V.) Thbs - v.).
(56)
18

---

Published as a conference paper at ICLR 2026
C.2 PROOF OF COROLLARY 2.1 (PCC/MSE GRADIENT RATIO DECAY)
Proof. We analyze the ratio of the global scaling factors Is
MSE and JPCC identified in Section 2.3
MSF
2(0s
Ms),
(57)
PCC
1
a2
by
(58)
og
We analyze their typical scale across samples using their Root Mean Square (RMS) values. We denote the empirical average over the batch as E,[-].
Step 1: RMS scale of the PCC global factor. We define normalized variables A, == a,jo,
and B. = b./ag- By construction, A, and BN have zero mean and unit variance (Var(Ag) Var B, 1), and their covariance is the PCC (Cov(A,, B,) p). The term inside the parenthesis of PCC is A, pBs. Its variance is:
War(A,-pl) -Var p² Var 2p Co, Bs)
(59)
=1 22- 2p² =1
(60)
Since the mean of A, pBB zero, the RMS magnitude of gf pPCC across samples is:
RMS.(g. PCC) - E. og
- - pBs) 0g
(61)
Step 2: RMS scale of the MSE global factor.
For the MSE global factor, we have:
RMS, (g. MSB)
[E, ya) 2/ E. (1. y
(62)
2v MSE((,)))
(63)
Step 3: Bounding the ratio of RMS global factors.
The ratio rglobal is defined as:
RMS.
V
Tglobal
RM
20 MSE
(64)
We use the MSE decomposition (Theorem 2.1
MSE(y, ý) = (jy - M2) + (ay - 0y) + 2 20 - p).
(65)
Since all terms are non-negative, we obtain lower bound for MSE:
MSE(y, 3) 2 2 o. 0g(1 p).
(66)
We assume E [0, 1], which is typical during training. We use the incquality:
1 1- = (1- p) < 2(1 p)
-I- S V2211-P
(67)
Plugging the bounds from equation 66 and equation 67 into the ratio definition equation 64
V2
Nglobal
(68)
20 v 20 (1
V2TT-p
(69)
20g v2. 12.000vv11-
(70)
20 0
2 i
3/2
This completes the proof.
17
19

---

Published as a conference paper at ICLR 2026
C.3 DERIVATION OF THE GRADIENT MAGNITUDE BOUND
Lemma C.I (Within-sample Dispersion Bound). Recall the definitions μ.s - + Σラー haj (within- sample mean) and o² Tu Lj=1 [laaj f4 1 (within-sample variance). Also recall V, Ej= hay where Caj 2 and Ej azj = 1. Then for every E (L....,ne).
has
voll - 2/T Sz.
Proof. Fix sample and index i. We use the triangle inequality by inserting the within-sample mean 14:
has voll = ||, ps + pha vall I ||hss - Mell + llv. - 1elll
(71)
We bound each term on the right-hand side separately.
Term 1: l psall- We first bound the deviation by the maximum deviation within the sample:
lig Mell max baj Mall-
(72)
The maximum of non-negative numbers is bounded by the square root of the sum of their squares (i.e., x S Ej 2² implies Ia S E ag):
max
[aaj Moll S
haj 1al12.
(73)
By the definition of o. the sum of squares is n. Thus,
h,i Mall < Vn,o² = Vn, 0s.
(74)
Term 2: ||v, - -4ull- We express the deviation of the aggregated embedding v, from the mean μ...
Since Σ, Osj 1, we have μu = Eg AsjHos.
p. = Σ asy haj - - agjiu =
aej (h.j - 14))
(75)
1=1
Using the convexity of the norm (Jensen's inequality):
||,
Ma
Σ
(o. Ms)
Ogjj |laj pall-
(76)
j=1
This weighted average is bounded by the maximum element:
Σ
aaj haj Mell
< max [
Mell-
(77)
As established for Term (Equation 74 ) the maximum deviation is bounded by nn, a.- There-
fore,
|v. M < dn
(78)
Conclusion.
Substituting the bounds from Equation 774) and Equation 78 into Equation
71
h, v.ll < Vn,o, + Vn,o, 2/n, o..
Lemma C.2 (Magnitude Bound of a Centered, Unit-variance Vector), Let II,... F. E R satisfy ΣS = 0 and - J² (equivalently Σ I = S). Then for every j E (1..... S), have |ay] v 1.
20

---

Published as a conference paper at ICLR 2026
Proof. Fix index j. Since the vector is centered (-=1 0), express 2j in terms of the other elements: zj Enzj zis- We analyze the squared magnitude æ using the Cauchy- Schwarz inequality and view the summation as dot product between vector ones R$-1 and the vector (15) E RS-1
Σ sgj
12 1
-S
Σ a
(79)
2911
We use the unit- varianari condition, into the incquality:
== x Σ
S. Therefore, Σnjj x² = S xz.
Substituting this
s x2)) = S(S - 1) (S 11)
(80)
Rearranging the terms isolate z²:
2 + (8 1)a < S(S 1)
(81)
S 5 イ < S(S - 1).
(82)
Dividing by S (which is positive) gives 23 < 8 - 1. Taking the square root yields the desired bound:
|zyl S 1.
Corollary C.1 (Gradient Magnitude Bound). The magnitude of the PCC gradient in Theorem 2.7 can be bounded by
0p
(
||
(83)
02.s
0g
S
Proof. We start from the expression for the PCC gradient derived in Theorem B.
0p 1 0.
p bg ag
a.sW
(h.: Va)-
(84)
θz Sag
By
We analyze the magnitude of this expression by applying the triangle inequality and the Cauchy-
Schwarz inequality (ww x
< || [|]
0p lasil
p b
[w (hoi
vo) || l vall-
(85)
0zei
Sog
oy
og
|ass
ag
by
+ lp
(86)
Sog
By
We now bound the individual components.
1. Attention weight: Since a, a probability from Softmax, 0 0 assi S
2. Correlation coefficient: By definition, -1 S - 1, so | <
3. Standardized scores: The terms ", and b, og are the standardized scores (z-scores) of the target and prediction for sample s. They form centered, unit-variance vectors across the S samples. By applying Lemma C.2 have:
0x
and
< <VS-L
Tg
4. Within-sample dispersion: The term | v.ll represents the deviation of the embedding
h from the aggregated embedding v.. By applying Lemma C.I have:
h væll 2/
21

---

Published as a conference paper at ICLR 2026
Substituting these bounds into the inequality:
0p
1
(VS-1+
11111-1111 ||w/l (2
(87)
Zzii
S
1
(2v5-1)
(2V
(88)
S
4 1)
||w|o,
(89)
Sog
Rearranging the terms to highlight the key factors identified in the main text:
Op
4v 1
(90)
Oz.i
0
S
regression in sample
prodiction deviation
batch
weights disporsion
This concludes the proof.
D ACHIEVABLE PCC BOUND OF CONVEX ATTENTION MODELS
This section provides the detailed analysis and proof of an intrinsic upper bound on the PCC gain that any convex attention mechanism achieve compared simple meea--oolingg baseline. This analysis formalizes the capacity limitation imposed by the convex hull constraint.
Setup and Decomposition. We consider dataset of samples. For sample 8, we have element embeddings h, (hai) where h,; E Idd Let μe, == Th, h be the mean-pooling embed- ding. A convex attention mechanism computes weights {a i=1 such that a 0 and Σx Ctas 1. The aggregated embedding is V. = Σ, ag The prediction is given by linear regression head (w, b): 0. w v, cc.
We decompose the prediction relative to the mean-pooling baseline:
Ú.
pb +c+
(v. 1ba))
(91)
5.
Here, De is the baseline prediction, and Ag. is the inrention-inducee perturbation.
Quantifying Dispersion and Variation. We introduce measures for within- saabin-sample dispersion (re- lated to in-sample homogeneity) and across-sample variation. Throughout this section, || 112 denotes the lg norm. We assume the empirical definition for standard deviation (normalized by 1/5).
Definition D.1 (Intrinsic Dispersion and Baseline Variation). For each sample s, define the maxi- mum within-sample deviation (the radius of the convex hull centered at the mean);
R. = max
hes
(92)
ISiSH,
Define the intrinsic within-sample dispersion R the root mean square (RMS) of these radii across the dataset:
S Σ
R
R2.
(93)
S
Let Go == std, (Te) denote the standard deviation of the baseline predictions across samples.
Remark D.I. Rt measures the intrinsic homogeneity the embeddings within samples, independent of the regression head w. do captures the variation of the mean embeddings projected onto the
regression space. By the scaling property of the standard deviation, de ||w]|2 std, (w TH.). where w w/|[w]/2-
22

---

Published as a conference paper at ICLR 2026
Centered Notation. We define the vectors of predictions across the S samples: y.y. E R From Equation (9 we have y = y + Ay.
We denote the centered versions of these vectors (by subtracting their respective means Pg: Hgg (aa)) as b, b, Ab. By linearity, the decomposition holds for centered vectors: b b + Ab. We denote the centered ground-truth targets as a, where a, = y/- Pay
The Pearson correlation coefficient (PCC) is the cosine similarity between centered vectors. Let = PCC(a, b) and Pa = PCC(a, b).
Bounding the Attention Perturbation. We first establish bounds on the magnitude of the pertur- bation Ay and its centered counterpart Áb.
Lemma D.1 (Bound Prediction Perturbation). For any convex attention weights {asih, the per- turbation for sample is bounded by:
Aual S |w|]2 R.
Consequently, the L2 norm of the centered perturbation vector is bounded by:
|bble I // Rt
Proof. We analyze the perturbation term. Since E; asi
Esisi
write μe, Es asilis.
Aú. = w (v, -4a) w Σ hai Σ anills
w
Os (hsi Ms).
1=1
Taking the absolute value:
A S |w/l2
Σ O'si Chsi mm)
CCauchy-Schwan
S |w/22 Σ
axillbai Mall
(Convexity of norm / Triangle inequality)
||w/22
Σ
asiRs
(Definition of R.)
|/ll2 Rg.
To bound the L2 norm of the uncentered perturbation vector Aý, we sum the squared bounds across samples:
A 112
Σ
<
w 22)
| 11 Σ
R
Using the definition of the RMS dispersion R22 Σ. R2g we get:
M < S /|/ R
| S /S 11/12
Finally, Ab is the centered vector of Ay. Centering vector (projecting onto the subspace thogonal to the constant vector (Arefiaamghani et al. 2022 Wang et al. 2010) is non-expansive operation on & space, hence || |bll2 < | < V
|w] R.
General Correlation Perturbation Lemma. We utilize general result bounding the change in the cosine similarity when vector is perturbed.
Lemma D.2 (Correlation perturbation). For any a, b, E S with ||01|2 < [[bl2.
(b + 5) |||||
b
2 ||8||2
[b 81|2
a b
|
|
Proof. Let å a/||a||2- Using triangle inequality, we have:
á (b + 5)
à
à 8
b + 8/2 b/2
=
b
812
[b/
b + 8)
(94)
23

---

Published as a conference paper at ICLR 2026
1
à 8
S (a. b)
+
(95)
+ 8|2 [b
|b/2
[b + 8
Ti
T2
We analyze each term. For T2, by Cauchy-Schiee (a 81
S ([012) and the reverse triangle
inequality (I|b + 8|12 2 ||b||2 [[0][2), we have T2
811a
Tb]
8
For Ti rewrite the expression: Ti a
| 1++ |
|b
Using Cauchy-Schwarz (/á b|
S |b (2), and the reverse triangle inequality - 1ot22-lbba
|bb v
[10l12) again, we get: Ti - ||bl2 [ 1661 proof.
||0|a
Summing 7i and T2 completes the
(612)
[
D.1 MAIN RESULT: ACHIEVABLE PCC GAIN
We combine these lemmas to prove the main theorem regarding the capacity ceiling of convex attention.
Theorem D.1 (Achievable PCC Gain Bound for Convex Attention). Po be the PCC achieved by the mean-pooling baseline y, and p the PCC achieved by any convex attention mechanism ý. Assume ||w]2 2| > O. If the intrinsic dispersion is small relative to the baseline variation such that R < op/ [| 21. then:
2 w 1lwl221 1p Pul x
2 R 0
00 - w//2 R
||||2 R
Proof. We apply Lemma D.2 using the centered vectors: m a (centered targets), b (centered baseline predictions), and ò Ab (centered attention perturbation).
First, we relate the L2 norms of the centered vectors to the defined dispersion measures. By the definition of the empirical standard deviation do. we have:
||b||2
(T. 2g)2 VSo0-
By Lemma D.I the perturbation is bounded by:
Abll2 I VS | R.
Next, we verify the condition required for Lemma D.2 ||ō||2 < ||n||2- The assumption R <
0o/ fo/l|w//a implies || R < Multiplying by vS, we get:
VS |wl] R v ou-
Therefore, ||bl|2 < |lb]/2, satisfying the prerequisite.
Now, applying Lemma D.2
2 A|bb||2
1 pol
[bll Abll
Since the function f(x) 2z/(C x) is monotonically increasing for C (where C' ||b|12), we substitute the upper bound for Ab 12
2 ( ||||2 Rt)
p pol <
VS oo - (VS || lt)
|| 1wl22 R
= 00 |w R
24

---

Published as a conference paper at ICLR 2026
Dividing the numerator and denominator by || (which is positive by assumption) yields the scale-invariant form:
2 2
1p Pal I
00/ |wl2
D.2 CONNECTION BETWEEN IN-SAMPLE DISPERSION AND RADIUS
Lemma D.3 (Dispersion- Radius Connection). For each sample 8, Ra/ / So os S R. Conse- quently, we denote ă = E. o?) 1/2
with mmmx max, Thy we have R/ /mmmxx S R.
E GENERALITY OF THEORETICAL ANALYSIS REGARDING ARCHITECTURE DEPTH
potential concern regarding our theoretical analysis whether the findings derived for attention aggregation apply to deep, multi-layer architectures. In this section, clarify that analysis fo- cuses on the aggregation mechanism at the readout stage, which dictates the final prediction behavior regardless of the depth of the preceding backbone.
Backbone-Agnostic Formulation. Our theoretical model assumes input embeddings h, [he In the context of deep learning. these not raw inputs but rather the latent repre- sentations produced by a backbone function fo)-) (comprising multiple Transformer blocks, FFNs, residual connections, and LayerNorms). The final prediction modeled ÿ, w v. where V. is convex combination of these final-layer representations. This formulation exactly matches the standard architectural paradigm used in modern Transformers:
[CLS] Token Aggregation: models like BERT ViT, the prediction is often derived from a specific [CLS] token. However, in standard Transformer layer, the output em- bedding of the [CLS] token is computed via the attention mechanism, which is convex combination (softmax) of the element embeddings from the previous layer. Thus, the read- out remains convex aggregation of the backbone's features, subject to the analysis we
present.
Global Average Pooling: Many regression heads utilize global average pooling over token embeddings, which is special case of convex attention where uniform weights are applied (a = 1/na).
The Readout Bottleneck. Our analysis (Theorem 2.2 establishes that the achievable PCC gain bounded by the convex hull of the input embeddings. Even with highly expressive deep backbone fg. if the final aggregation step is convex, the prediction g. is geometrically constrained to the interior of the simplex formed by the final-layer features {h Crucially, deep backbones do not inherently solve the homogeneity issue. In fact, models pre-trained with contrastive objectives often produce highly homogeneous embeddings for in-sample elements (semantically similar tokens or patches), which shrinks the convex hull radius R, and exacerbates the capacity limitation.
Empirical Validation Deep Architectures. Our experimental evaluation in Section 5.1 and Table 2 utilizes deep, multi-layer architectures, not shallow regressors.
PT-Transformer: A deep tabular Transformer with multiple attention layers.
ALMT: complex multimodal Transformer for sentiment analysis.
EGN: A multi-layer vision Transformer model for spatial transcriptomics.
In all these deep settings, we observed the PCC plateau phenomenon. The consistent improvement provided by ECA confirms that the convex readout mechanism acts bottleneck even in deep models, and that relaxing this constraint via extrapolative mechanisms is necessary for optimal cor- relation learning.
25

---

Published as a conference paper at ICLR 2026
F EXTENDED EMPIRICAL EVIDENCE OF THE PCC PLATEAU
To rigorously address the universality of the PCC plateau phenomenon and demonstrate that it is not an artifact of specific datasets or architectures, we conducted extensive additional experiments on diverse regression benchmarks from the UCI Machine Learning Repository (Asuncion al. 2007). We utilized multi-layer FT- "T-Transformer (Gorishniy et al. 2021 as the backbone to represent mod- ern, high-capacity attention-bain -based regression models.
Figure 7 illustrates the training (left) and validation (right) trajectories for both MSE and PCC. The curves marked with circles represent PCC metric, while the curves without circle represent MSE.
Figure Training and validation curves for multi-layer FT-Transformer on 8 UCI regression datasets. Left: Training set metrics. Right: Validation set metrics. The curves with circles in- dicate PCC (maximizing shape matching), while curves without circles indicate MSE (minimizing magnitude error). In all instances, PCC hits ceiling (plateau) significantly earlier than MSE, which continues to descend, highlighting the optimization gap.
ucl dataset automobile le STD best M0350. STD test PCC=00849)
Metrics
Validation Metrics
L.0
4
0.8
0.8
4
STD-Train MSE
3
0.6
0.6
MSR
MSR
STD Train PCC
2
0.4
2
0.4
STD Val PCC
1
0.2
1
0.2
0
50 Epoch
100
50 Epoch
100
(a) Automobile Dataset
dataset @attasp MSE-75.5108, STD PCC-0.7810
Train Metrics
Validation Metrics
600
0.8
600
0.8
MSE
400
0.6
MSF
400
0.6
U
STD-Val MSE
STD-Train PCC
0.4
0.4
200
200
STD-Val PCC
0.0
0.0
0
10
20
30
0
10
20
30
Epoch
Epoch
(b) Auto MPG Dataset
ucl bikesharinng PCC=0.9539
MSE=2805.660
Train Metrics
Validation Metrics
60000
60000
STD-Train MSE
MSE
0.6
40000
STD Val MSE
0.4
&
0.4
20000
20000
STD-Val PCC
0.2
100
50 Epoch
100
Epoch
(c) Bike Sharing Datasct
26

---

Published as a conference paper at ICLR 2026
uci dataset, productivity prediction of garment employees STD test MSE=0.0120, STD test PCC=0.8935
Train Metrics
Validation Metrics
1.0
1.0
0.20
0..000.0000000
0.4
0.15
STD-Train MSE
MSE
0.6
MSE
0.3
0.6
STD MSE
STD-Train PCC
0.4
0.4
PCC
0.05
0.2
0.1
0.2
0.0
0.0
0.0
20
40
20
Epoch
Epoch
(d) Garment Dataset
uci dataset real estate, valuation STD test MSE=54.6736, STD test PCC=0.8577
Train Metrics
Validation Metrics
1.0
2000
1.0
1000
1500
STD-Tain MSE
MSE
MSF
0.6
STD-Val MSE
750
0.4
1000
0.4
STD PCC
STD-Train
500
0.2
0.2
500
10
20
30
Epoch
Epoch
(c) Real Estate Dataset
uci dataset servo STD test MSE -0.577, STD test PCC-0.9079
Train Metrics
Validation Metrics
1.0
1.0
0.8
0.8
MSE
0.6
MSE
2
0.6
STD Val MSE
2
STD-Train PCC
STD-val PCC
1
0.2
0.2
0.0
0.0
Epoch
Epoch
(f) Servo Dataset
ucl dataset shares
STD best MSE=184637528 0000, STD test PCC=0.1289
1e8
Train Metrics
1e7 Validation Metrics
1.0
1.0
7
0.8
1.45
STD-Train MSE
MSE
0.6
0.6
STO-Val MSE
1.40
0.4
0.4
STD-Trann
STD PCC
0
0.2
5
0.2
1.35
100
50 Epoch
100
Epoch
(g) Shares Dataset
27

---

Published as a conference paper at ICLR 2026
dataset solar flare STD test MSE=0.6976, STD PCC=0.2090
Train Metrics
Validation Metrics
1.0
2.5
1.0
0.8
0.8
2.0
MSF
0.6
0.6
U
STD-Val MSE
1.5
STD-Train PCC
0.4
STD-Va PCC
0.6
0.2
1.0
0.2
0.0
0.0
20
40
20
40
Epoch
Epoch
(h) Solar Flare Dataset
Observations and Analysis:
Decoupling of MSE and PCC: Across all 8 datasets, we observe a distinct decoupling of the two metrics. the early stages of training, both MSE and PCC improve. However, distinct inflection point: the "PCC Plateau" consistently appears where the correlation gain flattens.
Persistent MSE Reduction: Crucially, after the PCC plateaus, the MSE continues to de- crease significantly. This confirms the optimization conflict identified in our theoretical analysis (Remark 2.3): the gradient dynamics of MSE minimization (specifically standard deviation matching) actively suppress the correlation gradient magnitude. The model con- tinues to learn magnitude information (lowering MSE) but loses the ability to optimize the shape/ordering of predictions (PCC).
Despite varying noise levels and feature dimensions across these diverse real-worke datasets, the optimization pattern correlation learning remains identical. This provides robust empirical evidence that the PCC plateau is fundamental failure mode in joint MSE-PCC training with attention-based models, which calls for specific architec- tural improvement as we proposed in Section 3
Universality:
G DETAILED EXPERIMENTAL SETTINGS
G.1 SYNTHETIC EXPERIMENT DETAILS
G.1.1 DETAILED DATA GENERATION PROCESS
We provide detailed description of the synthetic dataset generation process (DGP). The DGP is parameterized by the embedding dimension D, the number of elements per sample K, the signal contrast 17. the noise level v/ the ground truth extrapolation factor 1, the cross-sample variation scale d. and noise floor Fioor-
Defining Ground Truth Directions. We first establish the direction of the signal and the direc- tion of the noise.
1. Sample the ground truth regression vector (signal direction) w N(O,Ip)) Normalize | ||2 1.
2. Sample noise direction vector w/ We ensure orthogonality ((w*) w 0) using the Gram-Schmidt process and normalize || ||2
1.
Generating Sample Centers. For each sample s E (1...., 8), we sample the sample center (mean embedding) from an isotropic Gaussian distribution, controlling the cross-sample variation:
~ (0,
(96)
Generating Elements. We generate K elements for each sample s, distinguishing between K background samples and one key sample.
28

---

Published as a conference paper at ICLR 2026
Background Elements (i = 1,..., K - 1): These elements are generated by adding small isotropic noise (noise floor) to the sample center:
hgg Ms Ears
Esi NO, o B
(97)
Key Element K): The key element is generated by explicitly injecting signal contrast along w and uninformative noise along w-:
h,
+
+E,K.
(98)
Signal Contrast
Uninfonmative Naise
4. Defining the Target Variable. The target variable is defined based on an optimal representation that extrapolates the signal component by the factor y":
Ms + 7" (17 w").
(99)
The target label ys is generated by applying the ground truth regression vector w this optimal representation, with added label noise e ~ (O0 Iahaa::
(w") " v +
(100)
G.1.2 EXPERIMENTAL SETUP AND HYPERPARAMETERS
Model Architecture. We use simple gating attention architecture. The element embeddings h, are generated directly by the DGP (no pre-trained encoder). The attention logits are computed a linear layer parameterized by Wamm E Rx1, 2si h s1 Waun- The aggregated sample-level representation v, is computed using V. > Softmax ) The final prediction is computed via regression head W reg IR Dx1 and bias b: ú. N.ww +b
h
Optimization. We train the models using the Adam optimizer (inggma 2014 with standard pa- rameters (31 0.9, 82 0.999, 10-S). The objective function the joint loss Luxal LMSE AvooCece (or Lvce in Equation (1 for DNPL).
Hyperparameters. rized in Table
The default hyperparameters used in the synthetic experiments are summa-
Table 4: Hyperparameters for Synthetic Experiments
Parameter
Value
DGP Settings
D (Embedding dimension)
K (Elements per sample)
10
Nirin Nyal
2000/300
dB
1.0
Ottorr dlabll
0.01 / 0.01
Training Settings
Optimizer
Adam 0.001-0.01
Learning Rate
Epochs
1000
Avee (PCC loss weight)
0.3-0.8
ECA Settings
DATS Timia 1/
0-0.8 / 0.5-3
SRA Nry / Tmmxx
0.001 /
G.2 UCI ML REPOSITORY DATASETS
For each dataset, We report PCC, MSE and MAE between predictions ÿJ and ground -uunt targets y: over all samples in test set
cov(Ú.y)
PCC (g. y)
a())
29

---

Published as a conference paper at ICLR 2026
Mean squared error (MSE) and mean absolute error (MAE) computed sample-wise and averaged over all gene gene-windon window pairs:
N Σ 0 Yıl-
MSE (Ú.
Σ ( y:)
MAE (9.
N
1. Appliance Dataset The Appliances Energy Prediction dataset from the UCI Machine Learn- ing Repository contains experimental data collected over 4.5 months in a low-energy building. includes 19,735 instances with 28 real-valued features, including indoor temperature, humidity, and weather conditions. These attributes are recorded at 10-minute intervals. The goal to predict appliance energy usage integer value in watt-hours (Wh).
We replace the attention layer in the FT-Transformer (Gorishmly et al. 2021 with our ECA module and keep the remaining components unchanged. We use the following hyperparameters during train- rate=0.1", "embedding.dim--a" FT-Transformer baseline and the FT-T model. We use MSE and MAE evaluate magnitude matching and PCC to evaluate correlation matching.
ing: Shaach_size=128",
and "num _ for both the
2. Online News Dataset The Online News Popularity dataset contains 39,797 news articles from Mashable, each described by 58 numeric features (with the URL and timedelta features excluded). Its target the article's number of social-media shares (popularity) and we applied the log transfor- mation on the target to reduce the large range of data.
We replace the attention layer in the FT-Transformer (Gorishniy et al. 2021 with our ECA module and keep the remaining components unchanged. We use the following hyperparameters during train- ing: "batch sieiich_sze=128",.... "dropouttrate=O.1" "embedding dim=256" " and rate=0.1",
for both the FT-Transformer baseline and the FT-Transformer-ECA model. We use MSE and MAE to evaluate magnitude matching and PCC to evaluate correlation matching.
3. Superconductivity Dataset The UCI Superconductivity dataset contains 21,263 superconduc- tors with 81 real-valuee features, where the target is the critical temperature.
We replace the attention layer in the FT- Transffffrrr (Gorishmiy et al. (2021) with our ECA module and keep the remaining components unchanged. We use the following hyperparameters during train- ing: "batch size=128"; "dropout rate= "embedding dim=256" and "num heads=8" for both the F1-Transformer baseline and the FF-Trrnnfoomer+ECCA model. We use MSE and MAE to evaluate magnitude matching and PCC to evaluate correlation matching.
G.3 SPATIAL TRANSCRIPTOME DATASET
Data Processing. We follow the data processing as the EGN baseline (Yang et al. 2023). The 10xProteomic dataset contains 32.032 image- pangc-patch/exppession pairs curated from whole images (10x Genomics) 2025 Yang et al. 2023 Prediction targets are restricted to the 250 genes with the largest mean expression across the dataset (computed over all patches). For each target gene, we apply a log transform to expression values and then perform per-gene min-max normalization to map values into [0, 1]. We use the dataset-provides image patches without modifying tiling or slide-level grouping.
Evaluation Metric. We report PCC aggregated across genes and sample-wise metrics. For each gene g. PCC between predictions ūn and ground truths over all evaluation windows is
PCC(g) cov Vo)
o(gy) (vg)
We summarize (PCC(g)) across all target genes using: PCC@F (25th percentile), PCC@S (me- dian), and PCC0M (mean). Higher is better.
Mean squared error (MSE) and mean absolute error (MAB) are computed sample-wise and averaged over all gene-window pairs:
N
N MAE
MSE
Σ
(U
Σ 16
Vel
N
N
30

---

Published as a conference paper at ICLR 2026
EGN Train
EGN Val
EGN+ECA Train
EGN+ECA Val
Foldo PCC&F
Foldı PCC@F
Fold2 PCC
Foldo MSE
Foldı MSE
Fold2 MSE
1.0
0.10
0.10
0.10
0.5
0.5
0.5
0.05
0.0
0.0
0.00
0.00
0.00
50
0
50
50
50
50
0
50
Epoch
Epoch
Epoch
Epoch
Epoch
Epoch
Figure 8: Per-fold PCC @F comparison of EGN and EGN+ECA 10xProteomic dataset.
Train
EGN Val
EGN+ECA Irain
EGN+ECA Val
Foldo PCCES
Foldi PCC@S
Fold2 PCCES
Foldo MSE
Foldi MSE
Fold2 MSE
0.10
0.10
0.10
0.5
0.5
0.5
0.05
0.05
0.0
0.00
0.00
50
50
50
50
0
50
Epoch
Epoch
Epoch
Epoch
Epoch
Epoch
Figure 9: Per-fold PCC@S comparison of EGN and EGN+BCA 10xProteomic dataset.
The EGN and ECA improved version are implemented in PyTorch. The visual backbone is ViT with patch size 32, embedding dimension 1024, MLP dimension 4096, 16 atten- tion heads, and depth 8. We interleave the proposed Exemplar Block (EB) every two ViT blocks. Each EB uses 16 heads with head dimension 64, retrieving the k=9 nearest exemplars. Results are reported on a three-fold cross-validation with the same experimental setting in EGN, We train for 50 epochs with a batch size of 32. Experiments run on 2x A6000 GPUs. We apply the same pre- processing and gene-selection protocol throughout and keep dataset-provided tiles and slide/patient groupings intact for fair comparison.
Implementation.
Extra Results. Figure and Figure provide extra visualization results on PCC@S @CC@S and PCC@F metrics for each fold.
G.4 MULTIMODAL SENTIMENT ANALYSIS (MSA) DATASET
Data Processing We conduct multimodal sentiment analysis (MSA) experiments on the CMU- MOSI dataset (Zadeh et al.] 2016), standard benchmark comprising 2,199 opinion-centric video segments. Each segment pairs a short monologue clip with synchronized language, acoustic, and visual modalities and is annotated with continuous sentiment intensity score on a seven-point scale from -3 (strongly negative) to +3 (strongly positive). Following the widely adopted split, the data are partitioned into 1,284/ / 229/ 686 segments for train/validationtte
In line with community practice and THUIAR's public releases (Yu et al. 2020 2021). we use pre-extracted, unaligned features where acoustic streams are sampled at 12.5 Hz and visual streams at 15 Hz; text features are aligned to the same timeline.
Task formulation and metrics. We evaluate models in the regression setting (predicting the con- tinuous sentiment score) while also reporting the standard classification metric used in prior MSA work. Concretely, we report: (i) MAE: mean absolute error between predicted and ground-truth sentiment; (ii) PCC: Pearson's 's correlation coefficient between predictions and ground truth, mea- suring rank/linear consistency; and (iii) F1: the binary Fl score computed under the standard MOSI protocol from prior work and THUIAR's 's toolkit (Yu et al. 2020: 2021). This suite jointly captures absolute deviation (MAF), correlation structure (PCC), and discrete decision quality (F1) and is consistent with the prevailing MOSI evaluation practice.
To situate our method among representative approaches, we compare against widely cited MSA baselines that span tensor fusion, low-rank factorization, recurrent and deep feed-forward fusion, graph-basee based memory models, and cross-modal transformers. Below we list each baseline with its canonical citation (the superseripts indicate the source of numbers when report them in
Baselines.
31

---

Published as a conference paper at ICLR 2026
tables: from THUIAR's GitHub/toolkit, from the original paper, and reproduced by with released code/htyperparmmeters)
TEN: (Zadeh et al.
2017): Tensor Fusion Network introducing explicit multimodal tensor
interactions.
LMF* (Luu al. 2018): Low-Rank Multimodal Fusion redueing TFN's cubic complexity via factorization.
EF-LSTM (Williams et al.
2018b): Early-fusion recurrent model that concatenates
modalities prior to sequence modeling.
LF-DNNI (Williams et al. 2018a): Late-fusion deep network combining coodalit--specinfe
predictors at the decision level.
Graph-MFNi (Zadeh et al.
2018): Graph- sraph-structured Memory Fusion Network for cross-
view temporal reasoning.
MulT (Tai al. 2019): Cross-modal Transformer that performs directional attention
across modalities.
MISA Hazarika et al. 2020): Modaaity-InvarrantSppiiiii representations with con-
trastive objectives to disentangle factors.
ICCN' (Sun al., 2020); Cross-modal correlation networks leveraging canonical correla-
tion constraints.
DLF** (Wang et al. 2025 A recent strong multimodal baseline emphasizing deep latent fusion.
Attention- latent model trained with MSE; we use ALMT as the host architecture for our ECA augmentation.
ALMT (Zhang et al. 2023):
Implementation. Unless otherwise noted, we follow the standard MOSI preprocessing and the train/validationnfesst partition described above. For baselines, we adhere to the configuration choices recommended by the original authors or by THUTAR's toolkit (Yu et al. 2020 202T). Our primary report includes F1, PCC, and MAE on the held-out test set, with validation used solely for early stopping and hyperparameter selection. The model training with 2 A6000 GPUs
We adopt ALMT as the base architecture and keep the optimizer and data pipeline fixed across all runs to ensure a clean ablation. We train for 100 epochs with a batch size of 64, using Adam with 1r=le-4 and batching and loading
workers=8. Other hyperparameters are shown in Table
ALMT, in its original form, is trained with MSE loss only. To probe correlation-aware training and correlation-awas aggregation, evaluate controlled settings (all other hyperparameters are held constant):
1. ALMT PCC loss (no architectural change). We augment the training objective with differentiable PCC term, yielding a weighted sum
L = AMSE MSE + Arce LPcc:
where AMSE=1.0 and >pcc=1.0. This setting isolates the effect of explicitly encouraging high Pearson correlation during training while leaving the model architecture unchanged.
2. ALMT PCC loss + ECA (ours), In addition to the above loss, we adopt our proposed BCA method in the vision stream by modifying the ViT-style token aggregation step.
Extra Results Analysis. By comparing Setting to the MSE-only ALMT, we measure the gain from correlation-wware training alone. Comparing Setting to Setting isolates the benefit of ECA's vision-token aggregation. As summarized in Table 2 adding PCC improves correlation- oriented metrics without harming classification quality, and coupling PCC with our ECA aggregation further improves all three metrics (F1, PCC, MAE). These results indicate that ECA enhances the multimodal pipeline beyond loss-level changes, demonstrating tangible benefits for multimodality on MOSI.
32

---

Published as a conference paper at ICLR 2026
Table 5: Configuration for MOSI experiments (ALMT backbone).
Param
Value
Optimizer/ weight decay Ir
AdamW/1x 10
Batch size / epochs
64 100
Text / audio vision dims Projection dim (all modalities)
768 768/5/20
128
Token count token
Seq. lengths (Va/v)
50 375 / 500
Projection transformer (depth/heads/MLP)
1 / 8 128
Text encoder (heads/MLP)
8/ 128
Hyper-layer Nhepth/haadshhadddiinddppppuuu
3/8/ 16/0
Loss weights (MSE PCC)
1.0 1.0 v in Setting 2
ECA module
H
BATCH SIZE EFFECT ON PCC PLATEAU
potential concern regarding the joint optimization of MSE and PCC is whether the batch size S influences the optimization dynamics, specifically the onset of the PCC plateau. Theoretically, as discussed in Section 2.3 and the Equation (6) both the MSE and PCC gradient magnitudes scale proportionally with T/S. Consequently, the ratío between the two gradients that determines the effective optimization direction is invariant to the batch size. This suggests that the conflict causing the plateau is fundamental to the gradient properties (specifically the 1/0g attenuation) and the homogeneous PCC bound, rather than batch statistics.
To empirically validate this independence, dataset described Section[5.1] training the standard attention regression model with varying batch sizes S {32, 64, 128].
conducted an ablation study using the synthetic
Figure 10: MSE and PCC curve under various batch size settings.
Synthetic Dataset Eta 0.1, Gamma* 1.0, Lambda PCC=1.0, Batch size=32
1.6
1.0
1.4
Pearsen Correlation Coefficient (PCC)
Mean Squared Error (MSE)
1.0
Train MSE
Train PCC
0.6
IVal MSE
Val PCC
0.8
0.4
0.6
0.2
10
15
20
25
30
35
40
Epoch
33

---

Published as a conference paper at ICLR 2026
Synthetic Dataset Eta=0.1, Gamma*=1,, 1, Batch @atch_size=66
PCC=1.0,
1.8
1.0
1.6
0.8
Pearson Comrlaaioon Coefiicient (PCC)
1.4
Mean Squared Error (MSSE)
1.2
0.6
Trann MSE
Train PCC
1.0
MSE
Val PCC
0.8
Epoch
Synthetic Dataset Eta=0.1, Gamma*=1.0 Lambda_ PCC=1.0, Batch_ Batch_sizee1122 =izs==iieeee
1.0
1.8
1.6
0.8
Pearson Correlation Coefficient (PCC)
Mean Squared Error (MSE)
1.4
0.6
1.2
Train MSE
Train PCC
Val MSE
Val PCC
0.4
0.8
0.0
0
10
15
20
25
30
35
40
Epoch
Appendix presents the learning curves for these settings. analysis:
The results align with our theoretical
Consistent Plateau Behavior: Across all batch sizes, we observe the characteristic decou- pling of the metrics. The PCC curves (red) flatten and plateau significantly earlier than the MSE curves (blue), which continue to decrease steadily throughout training.
Generalization Gap: Both training (dashed) and validation (solid) curves exhibit similar trends, indicating that this phenomenon is optimization issue rather than generalization issue.
These results confirm that increasing decreasing the batch size does not resolve the PCC plateau, further validating the necessity of the architectural interventions proposed in our ECA to explicitly counteract gradient attenuation.
34

---

Published as a conference paper at ICLR 2026
1 DISCLOSURE OF LLM USAGE
large language model (LLM) was used only for language refinement (grammar, phrasing, and clarity) in the main paper and the appendix. The LLM did not contribute the research ideas, mathematical derivations, theorems, proofs, or experimental design. All technical content was inde- pendently produced and reviewed by the authors.
35


Reviews:
[{"review_id": "YPvwTrQo7B", "reviewer": "Reviewer_y2UY", "summary": "This paper studies the training behavior of attention-based regression models jointly optimized with mean squared error (MSE) and Pearson correlation coefficient (PCC) losses. The authors claim that such models commonly exhibit a “PCC plateau,” where correlation stops improving early in training while MSE continues to decrease. They analyze this through a theoretical decomposition linking MSE and PCC, derive a gradient bound suggesting that PCC gradients shrink with low within-sample dispersion and large prediction variance, and prove a convex-aggregation capacity limit. They propose Extrapolative Correlation Attention (ECA), combining (i) Scaled Residual Aggregation, (ii) Dispersion-Aware Temperature Softmax, and (iii) a Dispersion-Normalized PCC loss. Experiments on synthetic, tabular, biomedical, and multimodal sentiment datasets show small but consistent PCC improvements.", "questions": "1. Your argument that data homogeneity suppresses the PCC gradient and causes PCC plateau seems to conflate two different effects: homogeneity makes the pooled representation almost insensitive to attention weights, which would weaken gradients for any regression loss (including MSE), but you present it as if this is unique to PCC. At the same time, you also argue that PCC suffers an additional optimization-specific attenuation which would not affect MSE the same way. Can you clarify which of these you believe is actually responsible for the gradient bottleneck?\n\n2. The claim about the gradient bottleneck assumes that the prediction deviation increases early in training, but that quantity could also in principle decrease. Do you still observe a plateau if the initialization has a high output deviation?\n\n3. Why is the role of the batch size S not explored further? Did you try running the experiments with different batch sizes and see if it has an impact on the PCC plateau?\n\n4. Could you comment on the role of the direction of $w$ in theorem 2.2?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 1, "presentation": 3, "contribution": 1, "strengths": "1. The writing is well structured and easy to follow. The theoretical derivations are transparent and carefully presented.\n2. The work targets an interesting question: how attention-based regressors behave when trained with both magnitude and correlation-based losses. This is a topic that deserves rigorous study and of broad interest to the community.\n3. The authors validate their method across a diverse and well-chosen set of benchmarks, which strongly supports the generality and effectiveness of their approach.", "weaknesses": "1. The theoretical analysis is based on a simplified model: a single attention aggregation layer followed by a linear head. However, the models used in practice are significantly more complex. The paper does not discuss how other architectural elements might interact with or alleviate the identified problems.\n\n2. The experiments only validate the downstream effects rather than the mechanisms proposed by the theory. The paper would be more convincing with additional empirical evidence that isolate the role of each claim.\n\n3. PCC plateau is described as a widely relevant failure mode, but the authors do not provide any citations or external evidence of this claim. The only evidence they introduce in the beginning of the paper is figure 2 based on their own experiments. If this problem is well documented in the literature, the authors should provide citations. If external citations are truly unavailable, the authors must provide overwhelming evidence of the problem's existence themselves, or potentially reframe their contribution to be the identification and formalization of the problem itself.\n\n4. The condition is theorem 2.2 can make the result meaningless. The tightness of the bound is also not discussed (what if the denominator goes to zero?), and it does not explain or provide any intuition about the training dynamics, it is just about the best achievable PCC after training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DEMH0IrL2E", "reviewer": "Reviewer_4Xnu", "summary": "The paper studies a pervasive but underexplored failure mode in attention-based regression models trained with a joint Mean Squared Error (MSE) and Pearson Correlation Coefficient (PCC) objective: the “PCC plateau,” where PCC stops improving early in training even though MSE continues to fall. The authors trace this plateau to two coupled bottlenecks. First, an optimization bottleneck: minimizing MSE increases the variance of the model’s predictions, but the PCC gradient scales inversely with that variance, so the PCC term effectively “goes silent” as training progresses. This effect is exacerbated in settings where each sample’s elements are internally homogeneous, because the PCC gradient flowing through attention weights becomes extremely weak. Second, a capacity bottleneck: standard softmax attention is a convex aggregator, so its output is restricted to the convex hull of the input embeddings. The authors prove that in low-dispersion regimes this imposes a hard upper bound on how much PCC can improve beyond naive mean-pooling.\n\nGuided by this analysis, the paper introduces Extrapolative Correlation Attention (ECA), a drop-in alternative to softmax attention built to overcome both bottlenecks. ECA has three components: (i) Scaled Residual Aggregation (SRA), which explicitly extrapolates beyond the convex hull to increase representational contrast; (ii) Dispersion-Aware Temperature Softmax (DATS), which sharpens attention adaptively when a sample is internally homogeneous; and (iii) Dispersion-Normalized PCC Loss (DNPL), which rescales the PCC objective to keep its gradient active even late in training. Across synthetic, tabular, biomedical, and multimodal benchmarks, ECA prevents the PCC plateau: it continues improving correlation throughout training and delivers higher final PCC while matching or improving standard error metrics (MSE/MAE).", "questions": "In general, I like the idea of going beyond the convex hull of softmax attention. However, this limitation may not apply to the linear attention, or other variation of the attention that permits negative attention scores as discussed before. I believe further investigations on how ECA interacts with these non-convex attention would be beneficial.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, the paper is well written. Here are my understanding of the strength part:\n\n1. The paper identifies and analyzes an interesting “PCC plateau” phenomenon, an observed but under-theorized failure mode in attention-based regression models trained with joint MSE+PCC loss. In my perspective, because the phenomenon emerges in multiple architectures and datasets, the problem is likely to generalize and will be of interest to both theory-leaning and applied ML audiences. \n\n2. On the theory side, the paper decomposes why PCC plateaus: an optimization bottleneck, where minimizing MSE inflates prediction variance and in turn suppresses the effective PCC gradient (especially under homogeneous samples), and a capacity bottleneck, where softmax attention is inherently limited because it forms convex combinations and thus cannot escape a tight convex hull when all elements in a sample are very similar. Crucially, the proposed architectural solution — Extrapolative Correlation Attention (ECA) — is directly motivated by those two factors. This feels principled rather than heuristic. \n\n3. The empirical section is broad, well thought out, and aligned with the claims: The authors test on synthetic data with controlled homogeneity to stress-test exactly the regimes where theory predicts the plateau — and show that ECA keeps PCC improving instead of flattening. They evaluate on standard tabular regression benchmarks, showing consistent PCC gains and equal or better MSE/MAE after swapping in ECA, suggesting the method is not domain-specific. They move to challenging applied domains (spatial transcriptomics, multimodal sentiment) where correlation is a core metric and within-sample homogeneity is very strong. They also include ablations of ECA’s components, which helps establish that the gains are actually due to the proposed mechanisms, not just tuning or extra capacity.", "weaknesses": "While the paper provides a principled and well-motivated analysis, several aspects could be further clarified or extended:\n\n1. Architectural depth and generality. Most theoretical and synthetic experiments use a one-layer transformer. It remains unclear whether the proposed ECA mechanism yields similar improvements in deeper architectures, where subsequent layers (especially MLPs) could partially overcome the convex-hull limitation. Without results on multi-layer settings, the contribution may appear as a targeted fix for shallow regressors rather than a generally applicable improvement to attention.\n\n2. Effect of normalization layers. The current analysis does not account for the common use of pre- or post-LayerNorm in standard transformer blocks. Layer normalization can re-center and rescale token representations, which might already mitigate homogeneity and alleviate part of the optimization bottleneck. A more systematic investigation of how normalization interacts with the proposed mechanisms would strengthen the theoretical claims.\n\n3. Baseline coverage and fairness of comparison. The baselines focus mainly on vanilla softmax attention with a joint MSE + PCC objective. Missing comparisons include: (1) attention variants that allow negative or signed weights [1, 2], which can also move beyond convex combinations; and (2) feature-wise normalization or whitening methods that implicitly address variance inflation.\nIncluding these would clarify whether ECA’s gains stem from its theoretical design rather than simply greater expressiveness or modified normalization.\n\nReferences\n- [1] Zhang et al., Negative-Aware Attention Framework for Image-Text Matching, CVPR 2022.\n- [2] Lv et al., More Expressive Attention with Negative Weights, arXiv:2411.07176, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fhqt34K5qQ", "reviewer": "Reviewer_pBmc", "summary": "This paper studies the *PCC plateau* phenomenon where for attention-based regression models, PCC stops improving early in training even as MSE continues to decrease. One of the contribution of the paper lies in the theoretical characterization of the following conflict: lowering MSE  can paradoxically suppress the PCC gradient, particular when the data to be aggregated is highly homogeneous. Moreover, the authors derive a PCC improvement limit for any convex aggregator (including the softmax attention). Finally, to solve the aforementioned issues, the authors propose several tricks to rescale the PCC loss, adjust attention temperature. Empirical results prove their proposed methods work well in practice.", "questions": "**Question 1** Can you please explain how can we see the conclusion *PCC flattens at early epoch while MSE continues to decrease* in Figure 2? I feel the curves for training mse and training pcc seem to overlap.\n\n**Question 2** In theorem 2.2, because PCC is less than 1, if the RHS of the inequality os larger than 2, it provides meaningless bound. Therefore, the bound is only meaningful when the RHS is substantially less than 2. However, it is unclear whether this quantity satisfies this conditions.\n\n**Question 3** In the experimental sections, are the reposted MSEs training loss or test loss? In Table 1 and Table 2, it seems adding this additional regularization, the MSE also decreases. This is a little surprising to me. The reason is that when you add the additional regularization to improve PCC, essentially you're introducing the bias for the MSE loss, and resulting training loss should not be lower than the one without the regularization. Can you please provide some intuitions why this happens?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**Novel topics and clear interpretation:** I like the topic studied in this paper since learning PCC is an important problem, and there lacks theoretical understanding of why the PCC plateau happens. Moreover, the decomposition in Proposition 2.1 offers a clean interpretation of how the matching of mean, std, and correlation affect the MSE.\n\n**Clear theoretical analysis** In Section 2, the authors provide several propositions and theorems to argue why the PCC plateau happens by analyzing the gradients. Though the analysis has some weakness (see weakness section for details), these theoretical results provide a clean interpretations of how the homogeneity of the data, batch size, cross-sample deviation affect the overall gradients. Considering this work is the first to address this issue, I appreciate their contributions.\n\n**Strong empirical validation:** In the experiment section, the authors evaluate their proposed methods across four different datasets, and demonstrate sufficient improvement of the increase of PCC when their method has been applied. Moreover, the authors conduct ablation studies to identify the contributions of each component of their proposed algorithm.", "weaknesses": "**Missing comparison of gradient between MSE and PCC:** In Section 2.3, the author present gradient of PCC w.r.t. attention logits (see Theorem 2.1), then provide a bound for this gradient in Corollary 2.1, and finally discuss the implications of the bounds. It is clear how each term affect the gradient for PCC. However, it lacks the discussions of the bounds for MSE as well. Since this paper argues *PCC tends to flat earlier while MSE continues to decrease*, it is important to discuss the gradient of MSE, and compare them to support this claim.\n\n**Weak theoretical contribution:** The theoretical contributions of this work is mainly point-wise comparison (section 2.3 and 2.4). For example, in theorem 2.2, the authors state the bounds for $\\rho-\\rho_0$ for fixed $\\tilde R, \\sigma_0, w$. However, it lacks the understanding of how the quantities of interest evolve throughout the training, and what is the effect of them cumulatively on the PCC and MSE. \n\n**Limited discussions on the trade-offs:** This paper introduces a novel objective in equation 8 to improve the learning of PCC. However, adding regularization might affect the performance of MSE, training speed, or stability. However, none of this trade-offs has been discussed in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EWUxJXOVEE", "reviewer": "Reviewer_W6BH", "summary": "This paper investigates the \"PCC plateau\" phenomenon in attention-based regression models trained with a joint MSE and PCC loss, where PCC stops improving early even as MSE decreases. The authors provide a theoretical analysis identifying two main causes: 1) an optimization conflict where minimizing MSE paradoxically suppresses the PCC gradient, exacerbated by softmax attention, especially with homogeneous input elements within a sample; and 2) a model capacity limit where convex aggregators like softmax attention restrict PCC improvements based on the convex hull radius of the input embeddings. To overcome these limitations, the paper proposes Extrapolative Correlation Attention, incorporating a dispersion-normalized PCC loss, dispersion-aware temperature softmax, and scaled residual aggregation to stabilize PCC optimization, adapt to homogeneity, and allow extrapolation beyond the convex hull. Experiments across various benchmarks demonstrate that ECA successfully breaks the PCC plateau and improves correlation without degrading MSE performance.", "questions": "- While Corollary 2.1 provides a bound on the PCC gradient magnitude, it's not immediately clear if this bound alone sufficiently explains the relative slowdown of PCC compared to MSE during optimization. Could the authors provide a more direct comparison, perhaps by analyzing the relative magnitudes or scaling of the PCC gradient versus the MSE gradient components ?\n- For a more comprehensive understanding of the optimization dynamics, would analyzing the system using gradient flow provide further insights or more robust theoretical guarantees regarding the plateau phenomenon?\n- The central claim is that addressing the identified gradient attenuation and capacity bottlenecks improves PCC performance. Could the authors elaborate further on the core intuition for why mitigating these specific issues leads to better correlation learning?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper provides a thorough theoretical investigation into the PCC plateau phenomenon from two complementary perspectives: optimization dynamics and model capacity. This analysis successfully identifies key bottlenecks inherent in standard softmax attention for correlation learning, specifically the gradient attenuation conflict with MSE and the limitations imposed by convex aggregation.\n- The proposed Extrapolative Correlation Attention (ECA) method directly addresses the theoretically identified bottlenecks. Each component of ECA is well-motivated by the preceding analysis, offering targeted solutions to the optimization and capacity limitations.", "weaknesses": "- While the theoretical analysis provides valuable insights, certain aspects could benefit from further development or more rigorous treatment. Specific points regarding the theoretical sufficiency will be elaborated upon in the questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)