Neural Collapse in Multi-Task Learning
Authors: Youjun Wang
Boqi Li
Xin Zou
Weiwei Liu
Venue: ICLR 2026 Poster (2026)
Keywords: Neural Collapse
Primary area: optimization

Abstract: Neural collapse (NC) plays a key role in understanding deep neural networks. However, existing empirical and theoretical studies of NC focus on one single task. This paper studies neural collapse in multi-task learning. We consider two standard feature-based multi-task learning scenarios: Single-Source Multi-Task Classification (SSMTC) and Multi-Source Multi-Task Classification (MSMTC). Interestingly, we find that the task-specific linear classifier and features converge to the Simplex Equiangular Tight Frame (ETF) in the setting of MSMTC. In the setting of SSMTC, task-specific linear classifier converges to the task-specific ETF and these task-specific ETFs are mutually orthogonal. Moreover, the shared features across tasks converge to the scaled sum of the weight vectors associated with the task-specific labels in each task's classifier. We also provide the theoretical guarantee for our empirical findings. Through detailed analysis, we uncover the mechanism of MTL where each task learns task-specific latent features that together form the shared features. Moreover, we reveal an inductive bias in MTL that task correlation reconfigures the geometry of task-specific classifiers and promotes alignment among the features learned by each task.

Published as a conference paper at ICLR 2026
NEURAL COLLAPSE IN MULTI-TASK LEARNING
Youjun Wang', Boqi Li', Xin Zou', Weiwei Liu
'Dpaarment of Computer Science. Wuhan University, Wuhan, China
ABSTRACT
Neural collapse (NC) plays key role in understanding deep neural networks. However, existing empirical and theoretical studies of NC primarily focus on the single-task scenarios. This paper studies neural collapse in multi-task learning. We consider two standard feature-based multi-task learning scenarios: Single- Source Multi-Task Classification (SSMTC) and Multi-Source Multi-Task Classi- fication (MSMTC). Interestingly, we find that the task-specific linear classifier and features converge the Simplex Equiangular Tight Frame (ETF) in the setting of MSMTC. In the setting of SSMTC, task- susk-specific linear classifier converges to the task-specific ETF and these task-specific ETFs are mutually orthogonal. More- over, the shared features across tasks converge to the scaled sum of the weight vectors associated with the task-specific labels in each task's classifier. We also provide the theoretical guarantee for our empirical findings. Through detailed analysis, we uncover the mechanism of MTL where each task learns task-specific latent features that together form the shared features. Moreover, we reveal inductive bias in MTL that task correlation reconfigures the geometry of task- specific classifiers and promotes alignment among the features learned by each task.
INTRODUCTION
Deep neural networks have achieved impressive performance fields ranging from computer vision to natural language processing (Brown al.] 2020 Krizhevsky et al. 2012] [Simonyan & Zisser-
man 2015a Vaswani et al. 2017). Recent work (Papyan et al. 2020
has revealed an intriguing phenomenon deep neural networks, termed Neural Collapse (NC). When deep neural network
training enters the terminal phase of training (TPT) (Papyan et al. 2020) the stage where the train-
ing еттоr reaches 0 and the training loss continues decrease, the features and the last layer of classifiers exhibit common pattern across different network architectures and datasets as shown in Figure [a] and Figure I(d) This phenomenon can be summarized by the following four properties:
(NC1) Variability collapse. Each last-layer feature with the same class converges to its correspond- ing class mean.
(NC2) Convergence to Simplex ETF. The vectors of the class means (after centering by their global mean) converge to a Simplex ETF formally defined Definition 3 achieving equal lengths, equal pairwise angles, and maximal distance in the feature space.
(NC3) Convergence to self-duality. Up to rescaling, the class-means and last-layer classifiers converge to each other.
(NC4) Simplification to Nearest Class-Center The network classifier converges to selecting the class with the closest training class mean.
NC provides a mathematical characterization of features and last-layer classifier for deep neural network during TPT. NC has been observed and applied in various settings, such as imbalanced
learning (Fang et al. 2021b; Thrampoulidis et al. 2022) Xie et al. 2023 Yan et al. 2024 Yang ct al.
2022). transfer learning (Galanti al. [002]] Li et al. 2022 Munn et al. 2024), continual learning Yang et al. [023 Yu et al. 2023), learning for a large number of classes UJang et al.) 2024). multi-
label learning (Li et al. (2024), regression (Andriopoulos et al.) 2024), etc. More discussions about
NC are included in Appendix E. Following previous work, the penultimate layers of the network
"oorreponning author, Email: /uweeieei663@gmall

---

Published as a conference paper at ICLR 2026
test data
X
D
train data
x
H
2
(a) Single Task
(b) SSMTC
(c) MSMTC
(d) Neural Collapse
(c) SSMTC-NC
(f) MSMTC-NC
Figure 1: Illustration of single-task classification, single-source multi-task classification (SSMTC), and multi-source multi-task classification (MSMTC) settings (top row) and the corresponding neural collapse phenomenon (bottom row). The colored points represent features of samples from differ- ent classes. The arrows indicate the classifier weight vectors for each class. Figure (aa shows the single-task setting: feature extractor / and a classifier f. Figures [(b] and [(c) show the network structure in MTL, which consists of shared feature extractor h and multiple task-specific classi- fiers [ (+ E (1, Figure [(d) illustrates the geometric structure of the features and classifiers in the case of three-class classification.
MTL., we consider two binary classification tasks for clarity, As shown in Figure [(e] the features of training samples with label (0, 2) collapse to the scaled sum of wi and wf. Interestingly, the features test samples with single label align with ! and the features of test samples with single label 2 align with wf Figure ((f) shows that the
task-specific features collapse to the scaled task-specific classifier. In Figure I(e] and Figure [(f]
w and w both form the Simplex ETF where (1, Moreover, w. is orthogonal wn in SSMTC-NC as shown in Figure I(e)
are considered as the feature extractor and the last layer of the network is referred to as the classifier. However, most empirical and theoretical studies NC focus on the relationship between a single linear classifier and features in a single task, with limited attention given to multiple classifiers in multi-task learning (MTL). Our work investigates NC in the context of MTL, exploring the distinct latent features each task learns and how task correlation influences the latent features.
Related Work MTL. Feature-based MTL approach is one of the most popular MTL methods. which aims to learn the common representations for multiple tasks (Zhang & Yang 2022). Re- cent studies on feature-based MTL have predominantly focused on the biases introduced in the shared features learned across tasks (Collins et al., 2024:; Lippl & Lindsey 2024, Maurer et al.) 2016 [hheouda et al.] 2024 Wu et al.] (020). Our work focuses on the TPT, providing a geo- metric characterization of the feature space to offer a more nuanced understanding of MTL. More discussions about MTL are included in Appendix E.2 Following Rosenbaum et al. (2018); [Sener & Koltun (2018); Shenouda et al. (2024); Yu et al. (2020), we consider two common scenarios in MTL according to different datasets: Single-Soure SoueeSSuueeeeee Multi-Tast Classification (SSMTC) and Multi- Source Multi- Task Classification (MSMTC). SSMTC focuses classifying single label for sam- ples within each task. For example, each task classifies a label of an image in Multi-MNIST (LeCun)
et al. 2010). In contrast, MSMTC involves classifying data from distinct subsets of categories for
2

---

Published as a conference paper at ICLR 2026
each task, such in SIFARR00-Split-220 (Rosenbaum et al. 2018 where the dataset is divided into 20 tasks, with each task classifying 5 categories.
Contributions. As shown in Figure 1 study the geometric properties of the learned features and the tpsk-specific classifiers in MTL to provide nuanced understanding of MTL. Our contributions
can be summarized follows:
Multi-Tash Task Neural Collapse Phenomenon, As shown in Figure we study two standard
settings of MTL: SSMTC (Figure [(bb) and MSMTC (Figure I The two settings
hibit different Simplex ETF structures, termed SSMTC-NC (Figure [(e] and MSMTC-NC (Figure T(.). We find that each task-specific classifier converges to the task-specific Sim- plex ETF in both SSMTC and MSMTC settings, Interestingly, we also find that the shared features across tasks converge the scaled sum of weight vectors of the task-specific linear classifier for each task-specific label in SSMTC-NC, The task-specific last-layer features in MSMTC converge their respective task-specific classifiers.
Global Optimality of SSMTC-NC and MSMTC-NC. Theoretically, we prove that under the assumption of Unconstrained Feature Model (UFM) (Fang et al., 2021a: a, Mixon al. 2022), any global optimal solution satisfies the properties of SSMTC-NC and MSMTC- NC.
Insights on MTL. Through further detailed analysis, we find that shared features are formed by the features learned by each task in SSMTC. Moreover, reveal fundamental inductive bias in MTL that task correlation reshapes the space of task-specific classifiers, thereby promoting alignment among the features learned each task.
2 PROBLEM FORMULATION
2.1 MULTI-TASK NEURAL NETWORK
Our multi-task architecture employs hard parameter sharing (Rosenbaum al., 2018 Ruder 2017) as the multi-task neural network structure composed shared feature extraction layers 1: and task-specific linear classifiers {f*} (t (T]) where T is the number of tasks. Given sample x, the output g' for the t-th task is: g'(x)) = (1((hhaaa
where
f'(h) = W b'
he -WWoWW.....o (W + b.)... 1.....bu. b
W' represents the task-specific linear classifier of the last-layer and h (xx) is the shared feature of the input æ for all tasks. For a L-layer neural network, each layer consists of an affine transfor- mation followed by nonlinear activation function o--) (e.g..ReLU), We use 0sh denote shared parameters across all the tasks and 0 to denote task-specific parameters.
2.2 SINGLE-SOURCR MULTI-TASK CLASSIFICATION
We consider MTL problem over an input space and collection of task spaces (etr)) Each task space ) contains K, distinct classes. The MTL dataset is (æ,, vi ........... v/ JAe[N] that are independent and identically distributed. N is the number of samples, and of is the label of the t-th task for the i-th data point. We denote y; = (w!....,9/ ,y7") the label for the sample æ; and of the task-specific label in SSMTC. Consider a parametric hypothesis class per task as g (ar; 0sh 0*)
X' + yr and task-specific loss functions C((..)) y x y Rt, we minimize the following empirical risk:
min gth @1 ST.
Σ dl Le (0 0')
(1)
3

---

Published as a conference paper at ICLR 2026
where ct denotes task-specific weights and (' 0*) denotes empirical loss of the task t, defined as:
N C (g' (z;; 0th000)))) .()
L (0 (0th,0) -
N E=
2.3 MULTI-SOURCE MULTI-TASK CLASSIFICATION
The key difference between Single-Sourg Soulee Multi-Tass Classification and Multi-Source Multi-Task Classification is that the latter has multiple imput spaces {X* and corresponding task space (e"TT where T is the number of the tasks. Assume that each task has N data points {æt, y4 he([g that are independent and identically distributed. Similarly, denote the parametric hypothesis class g¹ (z; 0th 0*): X y² and the same task-specific loss functions (:,,)). Taking the task weights into account, we minimize the following empirical risk:
N
min gah
L (g (mf; 0t .0).))))
(2)
N
2.4 TRAINING OBJECTIVE FUNCTION UNDER UFM
Unconstrained Feature Model (UFM). The UFM (Mixon et al. 2022) and layer-peeled model (Fang et al. 2021a), in which the features of the last layer are treated as free optimization variables, commonly employed for the theoretical analysis of NC phenomena (Jiang et al., 2024 Sükenik iet al.] 20244] Xie et al. [023: Yang et al.] (022). The rationale behind this model is that modern deep networks are extremely ver-parameeterize and expressive such that their feature mapping can be adapted to any training data (Zang et al. 2021).
treat the feature (*) the free optimization variables. The training objective functions Eq. I and Eq (2) can be transformed into the following
Following Jiang et al. 2024 Yang al. 2022
E3.(3) and Eq. 4 based the UFM:
xd b' E le be the task-specific weights and biases of last layer linear classifier, H [hi; hgi..., hN] E IRdxN be the feature matrix and t be the one-hot encoding matrix for the t-th task. d is the dimension of the feature. We consider the following optimization problem:
Definition (SSMTC Nonconvex Training Loss under UFM). Let W E Rexx
T
T
T
min W.WW" WT H.b
Lc + AH H Aw Σ
& + A Σ
e' 1
6T
1=1
(3)
where AH xw and Xo are the positive regularization coefficients and control the strength of the weight decay which prevents the norm of W b' and from growing to infinity. this paper, || 1 denotes the Frobenius norm and 1|2 denotes the Euclidean norm of the vector.
Definition 2 (MSMTC Nonconvex Training Loss under UFM). Let W E K.xd d b' E R. be the task- susk-specific weights and biases of last layer linear classifier, H' e RdxNN be the task-specific feature matrix and Yt be the one-hot encoding matrix for the t-th task. We consider the following optimization problem:
T
min W1,W" W
Σ Lo + +b,
Σ
d (AH 11
Aw No 1b116"113)
WT H HT b b'.................. bT
1=1
(4)
Cross-entropy Loss. In this work, we adopt the cross-entropy (CE) loss Log.
Task Weights Following Sener & Koltun (2018); Yu al. (2020) Zhang & Yang 2022 we
use the uniform task weight in this paper. We show the experimental results in Appendix D33 of
SSMTC-NC and MSMTC-NC when using MGDA (Sener & Koltun 2018). Uncertainty Weight
(Kendall et al. 2018), PCgrad (Yu et al. (2020), DWA (Liu et al. 2019a), FAMO (Liu et al. 2023 FairGirad (Ban & Jil 2024) update the task weights.
4

---

Published as a conference paper at ICLR 2026
3
MAIN RESULTS
In this section, we present our empirical findings of SSMTC-NC and MSMTC-NC, and provide a theoretical guarantee.
3.1 SSMTC-NC
Following Fang et al. (202Ta): Mixon et al. (2022): Zhu et al. (2021) we assume that the training data balanced for each label.
N Tu-= n, vka [Ki], E [K2]...., ..... kr [Kr]
In this section, for simplicity, assume that the number of classes for each classification task is the same and use the uniform task weights (see Appendix [3. for generalizations):
Ki K2 c² = -...- = =1
KT K
(5)
(6)
We denote h k1. the j-th feature whose label is (k1, .h.....,Ki KT), fa kamink the mean of features with the corresponding label and NG the mean of all features. wi represents the clas- sification weight for class & in task t and प denotes the //f/l//f l2- h hhi.kar denotes the (h (h h2 - 1 (b A2.2..... - -All22 The Kronecker delta, denoted as dg is function used to indicate whether two variables x and y are equal. Ave represents the operation of taking the average. () denotes the dot product operation.
We use the objective function 3 to train the multi-tase task network. When we approach the TPT, we find the following experiment phenomena, which refer SSMTC-NC:
(NC1) Variability Collapse. Each feature with the same label converges to the mean.
Ave (hh (haaarr -1 kr (h, ky.... - )T 0
kr....r.d
(NC2) Task-specific Simplex ETF. The weights the classifier for each task form a Simplex ETF.
u llllll
1/ [22l - Vk, E [K], E (T)
K
(w w K
Su. K
yk, K' [
The spaces spanned by the weights of the classifiers for any two tasks are mutually orthogonal.
(NC3) Orthogonality.
(wtt w + O, Ak, W [K],t,t' [T] with t'
(NC4) Convergence to Sum of Task-specific Classifiers. The shared features among different tasks converge to the scaled sum of weight vectors of the task-specific linear classifier for each task-specific label.
h ke.....kr
(NC5) Simplification to Nearest Class-Center. The network's classification result converges to choose whichever class has the nearest training class-mean.
(argmax(w),, h) bl,,.... argmax(w h) + bZ argmin| h u kr....r ky ||2
ky
ky....ry
Theoretically, we rigorously analyze the global minimizers of Problem and get the form of minimizers to validate the experimental results.
5

---

Published as a conference paper at ICLR 2026
Theorem 3.1. Assume that the training data is balanced, the feature dimension is larger than the -N number classes, i.e. d 2 Σi= K; TT the regularization parameters satisfy λн ig and the Assumptions ( ( hold, then, any global minimizers of Problem (3) satisfy the following five properties, which correspond to the five phenomena of SSMTC-NC (SSMTC-NCI to SSMTC-NC5):
1. h ha...er
=1 -4 F1.
2. ||02 = 11002/112. (10), wbke = K-1 6 K-I R-,, Ak, E [K], E (T)
3. (ung, w O, Wk, [K],t, E [T] with t t
4. h kaunik - wl
11 whelle
5. (argmax(us) h) + s..
wggma h) + bly argmin| u kr. kr
k1,....yy
The detailed proof of Theorem 3.1 is deferred to Appendix Next, we delve into the implications of our findings from various perspectives.
Mutually Orthogonal Task-specific Classifiers. The orthogonality in NC3 ensures that the task- specific classifiers can be optimized independently.
Feature Learning in MTL. NC3 and NC4 suggest that the learned features of Problem (3) may be a combination of features from multiple orthogonal subspaces, which is further validated in the experimental section 5.1
General SSMTC-NC. Considering general training scenario without Assumptions and 6 similar SSMTC-NC phenomenon still exists. The detailed descriptions and theories are provided in Appendix B.2 and the experimental results are provided in Appendix D.2
3.2 MSMTC-NC
We assume that the number of training samples in each class is balanced for each task and the feature dimension is greater than the largest number of classes all tasks, i.e., d 2 max, K₃ 1. We use the objective function (4) to train the multi-task network and experimentally observe that the neural collapse phenomenon occurs for each task, referred to as MSMTC-NC.
(NCI) Within-class Variability Collapse. In each task, the features of the same class are clustered around the class mean.
Avc - (h) - PT) - 0, e [T)
kJ
(NC2) Convergence to Simplex ETF, The classifier and features of each task converge to Simplex ETF.
Ke 6k.
1
( w -
K. 1 Vt E [T', k, k' E [Ki]
K;
K₁
(pht pbae)) +
K, - Vi E [T], k k E [Kg]
K,
112
||00 11
+ 1, E [T) E [K.]
|
11
+ 0, [7, E [K.]
(NC3) Convergence to Self-duality.The linear classifiers and class-means of each task will con- verge align with each other.
क 12 0. E [K.]
(NC4) Simplification to Nearest Class-Center
argmax(w;j h)) + bf + argmin h wtll2, Vt [T]
b

---

Published as a conference paper at ICLR 2026
hi, denotes the feature of the j-th sample belonging class k in the t-th task and pa represents the mean, i.c., ps Σラ= / /ne (Nt denotes the number of samples per class in the t-th task). h denotes (14 - pc)/Ilp -all2
Theoretically, rigorously analyze the global minimizers of Problem and get the form of minimizers to validate the experimental results.
Theorem 3.2. Assume that the feature dimension is larger than the largest number of classes in all tasks, i.e., d 2 max, Kr - 1, the regularization parameters satisfy λ < =t and the number of training samples in each class is balanced for each task, any global minimizers of Problem 4 satisfy the following four properties, which correspond to the four phenomena of MSMTC-NC (MSMTC- NCI MSMTC-NC4):
1. h = p Vt [T],K E [Kg]]
2. | = 1| 112, (10) whte) K-1
K 8
1
4A
KT [Ki]
K-1
3. j4 / Vt E [T] k E [K,]
4. argmax h)) bb argmin||/ 22l1, Vt [T]
k
A detailed proof of Theorem 3.2 is provided in Appendix B.3
EXPERIMENT
4.1 DATASETS
The datasets used in this paper are Multi-MNIST (Sabour et al. 2017), Multi-CIFAR10, CIFAR100- Cross and CIFAR 100-Split (Rosenbaum al. 2018). The detailed description of the datasets provided in Appendix C Table 2 presents the datasets. It is worth mentioning that for SSMTC, CIFAR 100-Cross- 1 1x classification task can be viewed as multi-task learning with two tasks, each containing 10 classes. In contrast, for MSMTC, ZIFAR100-Split-5::
classification task involves multi-task learning with 20 tasks, each having classes. More experiments on CelebA (Liu et al. ImageNet-Ii (Deng et al. 2009) are shown in Appendix D.4
2013).
D.4
4.2 EXPERIMENTAL SETTINGS
This paper uses ResNet18, ResNet34 (He et al.] 2016). VGGI1 and VGG13 (Simonyan & Zis- serman 2015b) the shared feature extractor, and the uniform static weights. Throughout all the experiments, we use a SGD optimizer with fixed batch size 128, weight decay (Aиr Aw) = (5-10-4 5- 10 4) and momentum 0.9. In the experiment, the learning rate is initially set to 1-101 and gradually decays to 10 over 500 epochs using the ConcllAnna scheduler.
Additionally, employ different multi-task weighting strategies shown in Appendix D.3 in- cluding MGDA, Uncertainty Weight, PCgrad, DWA, FAMO, FairGrad. And we also consider the datasets with different number of classes for each task, as shown in Appendix D.2 Experiments different learning rate and (AH: Xw) are shown in Appendix D.5
4.3 EVALUATION METRICS
SSMTC-NC. Following Papyan et al. (2020); Sūkenik et al. (2024), we evaluate NC1 using within- class variance, denoted as SNCI. NC2 assesses the proximity of the classifier to the Simplex ETF using the angle and norm, denoted SNC2-1 and SNC2-2. NC3 is assessed by the maximum ab- solute value of the cosine between classifiers of different tasks, denoted as SNC3. NC4 is evaluated using the difference between the normalized feature mean and the scaled sum of task-specific clas- sifiers, denoted as SNC4, NC5 is evaluated using the error rate of classification based on Nearest Class-Center, denoted as SNC5.
7

---

Published as a conference paper at ICLR 2026
V166- Rartee34
VIAGE FasleeS
VIARE PeaflcSS
30-
150
199
159
NC1
VC2
VOGLL
VOGL Faskeet8
VOG1 fervel
185
A2-
Ld
85-
0.-
NC1
NC2
NC3
106
(a) NC on Multi-MNIST-10-100 (top) and Setti--FFAR1O-10-11 (bottom). The NC metrics for NCI, NC2, and NC3 are the means of the NC metrics on two tasks. The horizontal axis represents the epoch.
(b) Task-specific latent feature and shared fea- ture
Figure 2: Illustration of Feature Learning in SSMTC
MSMTC-NC. Following Papyan et al. (2020): Súkenik et al. (024). we use the mean of all tasks' metrics similar SSMTC-NC and name them MNCI, MNC2-1, MNC2-2, MNC3 and MNC4 respectively.
The specific metric form is included in the Appendix D.9
4.4 VERIFICATION OF SSMTC-NC AND MSMTC-NC
As shown in Figures 6 7 and Figures 8 in the Appendix, all the metrics approach zero during
the terminal phase of training. These results show that SSMTC-NC exists in different networks and datasets. Figures I0 II in the Appendix show that MSMTC-NC exists in different networks and datasets. Building on prior works (Li et al. (2244) Yang et al.) 2022). we conduct parameter- efficient training experiments leveraging NC properties in MTL as shown in Appendix D.7 Our results demonstrate the presence of both SSMTC-NC and MSMTC-NC phenomena, and verify the replaceability of final-layer parameters.
INSIGHTS FROM NC IN MTL
5.1 THE RELATIONSHIP BETWEEN SHARED FEATURES AND TASK-SPECIFE LATENT FEATURES
To further understand the shared features and what each task learns respectively in SSMTC, we pro- cess the training samples L.. from the Multi-MNIST-IOG 10 and Multi-CIFAR 11-1000--0000 10 datasets, retaining only the top-left and bottom-right features to obtain x² and æ respectively. We then feed test samples x", x into the neural network trained on x L. to obtain h² and h as shown in Figure 2(b) As shown in Figure 2(a) we perform NC analysis on h and the classifier W as well as on h" and the classifier W R We observe that both h and h" exhibit the NC properties, includ- ing within-class variability collapse (NC1), convergence to Simplex ETF (NC2), and convergence to self-duality (NC3). We also uncover properties corresponding to SSMTC-NC4:
(NC4) Convergence to Sum of Task-Specific Latent Features
L,R ki.ka
+0
11
Pkickz denotes the feature mean of sample a with label (k1, 12) after centering and normaliza- tion. ih and f denote the feature means of sample xh with class kj and sample R with class la after centering and normalization.
L.R
8

---

Published as a conference paper at ICLR 2026
2800
3.04
Emplr cal costa
Thecretccal codre
-155
2x6
(a) Label-balanced SSMTC-NC
(b) Task-balanced SSMTC-NC
(c) Cosine between w and w
Figure 3: Illustration of SSMTC-NC in the label-balanced case, SSMTC-NC in the task-balanced case, and cosine between wi and wp n denote the number of samples with label (k1, k2).
Task 1
Task 2
Task
Task 2
Figure 4: Grad-CAM++ visualizations for two tasks under independent and related settings. Each dashed block contains one example of each label. The first two blocks show Task and Task under independence (cos(w) (w), wf) 0), while the third and fourth blocks show Task and Task 2 when tasks are related (cos(w) w/) 0.5). Blue regions indicate the important regions for the task to predict the concept in the image.
NCI, NC2, and NC3 indicate that h² and ht are the features learned by the two tasks, which we refer as task-specific latent features. NC4 further explains that the shared features of samples in MTL are composed of task-specific latent features. The above findings dissect the specific structure of the shared feature, indicating that task specifc classifiers decompose the shared features into task-specific latent features.
5.2 IMPACT OF TASK CORRELATION ON TASK-SPECIHIC CLASSIFIERS
With the total sample size fixed N, we systematically vary the sampling proportions of each label pair (k1, k2) and investigate how task correlation (Li et al. 2025 Ma al. 2018)) impacts the task-specific classifiers. Through empirical experiments and theoretical analysis, we find that, when moving from the label-balanced to the task-balanced scenario, all other NC properties continue to hold and SSMTC-NC3 adapts to Correlated-NC:
(NC1) Variability collapse. Each feature with the same label converges the mean.
(NC2) Task-specific Simplex ETF. The weights the classifier for each task form a Simplex ETF. (Correlated-NC3) Alignment of task-specific classifiers. As hi. increases, cos(w), w increases.
(NC4) Convergence to weighted sum of task-specific classifiers.
(NC5) Simplification to Nearest Class- Ceass-Center.
Theorem 5.1. Consider two binary classification tasks, the training data is task-balanced, and the number of samples with label (ki, k2) is nh 82
&a (k1 E [2], k2 E [2]) Assume that the number of then any global minimizers of Problem satisfies the all samples N and dHdw < x K2 following conditions:

---

Published as a conference paper at ICLR 2026
1. ha 11
2. ||offlla 11 (1o ) = 201, - 1, E [2).4 29.4 [2]
3. As nhiida increases, cos( wk w increases.
4. h hi 1iw, + wwg
5. (argmax (10 h)) + Uby argmax h) + 122.)) argmin| wk k22 112 k1.
h1
The proof is shown in Appendix B.4 As shown in Figure ((a) and Figure 3(b] we set N = 8000. K and then adjust the number of samples belonging each label. As shown in Figure 3 when the number of samples with the label (0, 2) increases, the cosine between w and w/ increases. We explain the phenomenon theoretically and derive the closed-form expression for cos(w). (wi, wi) in Appendix B.4 which aligns with the experimental results as shown in Figure The additional experimental results, encompassing extended scenarios with K 2 2, are included in Appendix D.6
Through the detailed experiments and theoretical analysis, we find that task correlation reshapes the space of task-specific classifiers.
Implication of the alignment of task-specific classifiers. When tasks are correlated, the align- ment of task-specific classifiers implies that the features learned by the two tasks are aligned and closer to shared features. To verify the implication, we choose the Eyeglasses and Mouth $1 5lightly Open attributes on CelebA as Task 1 and Task 2, and boost task correlation. We use Grad-CAM++ (Chattopadhyay et al.) 2018) to visualize the important regions for the two tasks to predict the concept the image. As shown in Figure when tasks are related, their salient regions largely coincide, demonstrating that the two tasks attend to the same facial features and thus learn closely aligned representations.
6 CONCLUSION
In this work, we demonstrate neural collapse in MTL through SSMTC-NC and MSMTC-NC. We empirically validate both SSMTC-NC and MSMTC-NC across various network architectures and datasets. Theoretically, SSMTC-NC and MSMTC-NC. Through further experiments, we find that each task learns task- specific latent features that form shared features. Our empirical findings, supported by theoretical analysis, indicate that MTL is inherently biased toward leveraging task correlation to reconfi gure the geometry of task-specific classifiers and promote alignment among the features learned by each task. We believe our work offers valuable insights into the understanding of features and classifiers in MTL.
prove that any global optimal solution must satisfy the properties of
7 ACKNOWLEDGMENTS
This work is supported by the Key R&D Program of Hubei Province under Grant 2024BAB038, the National Key R&D Program China under Grant 2023YFC3604702.
REFERENCES
George Andriopoulos, Zixuan Dong, Li Guo, Zifan Zhao, and Keith W. Ross. The prevalence of neural collapse in neural multivariate regression. In NeurIPS, 2024.
Hao Ban and Kaiyi Ji. Fair resource allocation in multi-task learning. In ICML, 2024.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin,
10

---

Published as a conference paper at ICLR 2026
Scott Gray, Benjamin Chess, Jack Clark, Christopher Bernr,, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurlPS, 2020.
Aditya Chattopadhyay, Anirban Sarkar, Prantik Howlader, and Vineeth N. Balasubramanian. Grad- cam++: Generalized gradient-based visual explanations for deep convolutional networks. In
WACV,2015
Liam Collins, Hamed Hassani, Mahdi Soltanolkotabi, Aryan Mokhtari, and Sanjay Shakkottai. Provable multi-task representation learning by two-layer relu neural networks. In ICML, 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Cong Fang, Hangfeng He, Qi Long, and Weijie J Su. Exploring deep neural networks via layer- peeled model: Minority collapse imbalanced training. Proceedings of the National Academy of Sciences, 2021a.
Cong Fang, Hangfeng Hc, Qi Long. and Weijie J. Su. Layer-peeled model: Toward understanding well-trained deep neural networks. Proceedings of the National Academy of Sciences, 2021b.
Tomer Galanti, András György, and Marcus Hutter. On the role of neural collapse in transfer learn- ing. In ICLR, 2022.
Xiuwen Gong, Dong Yuan, Wei Bao, and Fulin Luo. A unifying probabilistic framework for partially labeled data learning. IEEE Trans. Pattern Anal. Mach. Intell., 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Jiachen Jiang, Jinxin Zhou, Peng Wang, Qing Qu, Dustin G. Mixon, Chong You, and Zhihui Zhu. Generalized neural collapse for large number of classes. In ICML, 2024.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In CVPR, 2018.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton, Imagenet classification with deep convo- lutional neural networks. In NeurlPS, 2012.
Yann LeCun, Corinna Cortes, Chris Burges, et Mnist handwritten digit datahase, 2010.
Hongkang Li, Yihua Zhang, Shuai Zhang, Pin- Yu Chen, Sijia Liu, and Meng Wang. When is task vector provably effective for model editing? generalization analysis of nonlinear transformers. In ICLR,, 2025.
Pengyu Li, Xiao Li, Yutong Wang, and Qing Qu. Neural collapse in multi-label learning with alck-all-label loss. In /CML, 2024.
Xiao Li, Sheng Liu, Jinxin Zhou, Xinyu Lu, Carlos Fernandez-Granda, Zhihui Zhu, and Qing Qu. Principled and efficient transfer learning of deep models via neural collapse, 2022.
Samuel Lippl and Jack W. Lindsey. Inductive biases of multi-task learning and finetuning: multiple regimes of feature reuse. In NeurlPS, 2024.
Bo Liu, Yihao Feng, Peter Stone, and Qiang Liu. FAMO: fast adaptive multitask optimization. In NeurlPS, 2023.
Shikun Liu, Edward Johns, and Andrew J. Davison. End-to-end multi-task learning with attention. In CVPR, 2019a.
Weiwei Liu and Ivor W. Tsang, On the optimality of classifier chain for multi- classification. In NeurlPS, 2015.
Weiwei Liu, Donna Xu, Ivor W. Tsang, and Wenjie Zhang. Metric learning for multi-output tasks. IEEE Trans. Pattern Anal. Mach. Intell. 2019b.

---

Published as a conference paper at ICLR 2026
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV, 2015.
Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H. Chi. Modeling task relation- ships in multi-task learning with multi-gate mixture-of-experts In KDD, 2018.
Yuren Mao, Weiwei Liu, and Xuemin Lin. Adaptive adversarial multi-task representation learning. In ICML, 2020a.
Yuren Mao, Shuang Yun, Weiwei Liu, and Bo Du. Tchebycheff procedure for multi-task text clas- sification. In ACL, 2020b.
Yuren Mao, Zekai Wang, Weiwei Liu, Xuemin Lin, and Wenbin Hu. Banditmtl: Bandit-based
multi-task learning for text classification. In ACI., 2021.
Yuren Mao, Zekai Wang, Weiwei Liu, Xuemin Lin, and Pengtao Xie. Metaweighting: Learning to weight tasks in multi-task learning. In ACL, 2022.
Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredes. The benefit of multitask representation learning. Journal of Machine Learning Research, 2016.
Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. Sampling Theory, Signal Processing, and Data Analysis, 2022.
Michael Munn, Benoit Dherin, and Javier Gonzalvo. The impact of geometric complexity on neural
collapse in transfer learning. In NeurlPS, 2024.
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 2020.
Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. In ICLR, 2018.
Sebastian Ruder. An overview of multi-task learning in deep neural networks, 2017.
Sara Sabour, Nicholas Frosst, and Geoffrey E. Hinton. Dynamic routing between capsules. In NeurlPS, 2017.
Ozan Sener and Vladlen Koltun. Multi-task learning multi-objective optimization. In NeurlPS, 2018.
Joseph Shenouda, Rahul Parhi, Kangwook Lee, and Robert D. Nowak. Variation spaces for multi- output neural networks: Insights multi-task learning and network compression. Journal of Machine Learning Research, 2024.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015a.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICI.R, 2015b.
Peter Súkeník, Marco Mondelli, and Christoph H Lampert. Deep neural collapse is provably optimal for the deep unconstrained features model. In NeurlPS, 2024.
Christos Thrampoulidis, Ganesh Ramachandra Kini, Vala Vakilian, and Tina Behnia. Imbalance trouble: Revisiting neural-collapse geometry. In NeurIPS, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
Sen Wu. Hongyang R. Zhang, and Christopher Ré. Understanding and improving information trans- fer in multi-task learning. In ICLR, 2020.
Liang Xie, Yibo Yang, Deng Cai, and Xiaofei He. Neural collapse inspired attraction-repulsion balanced loss for imbalanced learning. Neurocomputing, 2023.
12

---

Published as a conference paper at ICLR 2026
Hongren Yan, Yuhua Qian, Furong Peng, Jiachen Luo, zheqing zhu, and Feijiang Li. Neural collapse to multiple centers for imbalanced data. In NeurlPS, 2024.
Yibo Yang, Shixiang Chen, Xiangtai Li, Liang Xie, Zhouchen Lin, and Dacheng Tao. Inducing neural collapse in imbalanced learning: Do really need learnable classifier at the end of deep neural network? In NeurlPS, 2022.
Yibo Yang, Haobo Yuan, Xiangtai Li, Zhouchen Lin, Philip H. S. Torr, and Dacheng Tao. Neural collapse inspired feature-classffee alignment for few-shot class-incremental learning. In ICLR, 2023.
Longhui Yu, Tianyang Hu, Lanqing Hong, Zhen Liu, Adrian Weller, and Weiyang Liu. Continual learning by modeling intra-class variation. Transactions on Machine Learning Research, 2023.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn.
Gradient surgery for multi-task learning. In NeurIPS, 2020.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Commun. ACM, 2021.
Yu Zhang and Qiang Yang. A survey multi-task learning. IEEE Transactions Knowledge and Data Engineering, 2022.
Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A geometric analysis of neural collapse with unconstrained features. In NeurIPS, 2021.
13

---

Published as a conference paper at ICLR 2026
Appendix
APPENDIX ORGANIZATION
Table 1: Appendix Contents
Appendix
Basic Lemmas
Appendix B
Theoretical Proofs
Appendix
Dataset Illustration and Visualization
Appendix
D
Detailed Experimental Results
Appendix
Discussions about the Related Work
In Appendix A we provide the lemmas used in the proofs. Appendices B. and Appendix
B.3 present the proofs of Theorems [3.] and Theorems 3.2 respectively, and Appendix B8.2 offers the description and theoretical proof of the general SSMTC- In Appendix [B.4] we provide the proof for Theorem 5.1 [ropendxx/pprvvicie other metries. Appendix(D.2]contaiii supplementary experiments for general SSMTC and MSMTC, and Appendix providees empirical results SSMTC-NC and MSMTC-NC when we use multi- ple MTL methods to update the task weights. In Appendix D.4] we provide the experimental results on ImageNet, CelebA and Tiny- Imaeeema In Appendix [..] we run the experiments on different learning rates and (ΛH. Aw). In Appendix [.6] we show the experimental results when the train data is task-balanced. In Appendix [D.7] we provide the experimental results on efficient training utilizing NC properties in MTL. In Appendix I we discuss the difference between Li et al. (2024) and our work. Appendix E.2 MTL.
detailed description of the datasets. Appendix D.T presents
we discuss the connection between our work and related work in
A BASIC DEFINITION AND LEMMAS
Definition (Simplex ETF (Papynn et al. 2020 A collection of vectors m, E Rd 1,2 K, K,d 2 K is said to be a Simplex equiangular tight frame if:
M 1-VV
K UUK
1 kk1k)
(8)
where M = =mmm....... mx] E Badk U E RRxK allows rotation and sutisfies UTU-IA IK the identity matrix, and 1K is the all-ones vector. Specially, when = the angle between any two vectors in M is 180 degrees; When K 3, the angle between any two vectors is 120 degrees.
Lemma A.1. For a set of n vectors v, E R4,, the point x E Rd that minimizes the sum of squared Euclidean distances
1 x
is the mean of the vectors, i.e.,
n 8
Lemma A.2. Let f R2d + R be a strictly convex function. Then, for any æ1, Ø2, .., Bn Rdd the following inequality holds;
Σ (as) n
34
n
Equality holds if and only x 22
Fn
14

---

Published as a conference paper at ICLR 2026
Lemma A.3. Consider the function f(z) = olog(1 Bexp(-nx)) + n, where a, B, RR, and x E Rzo- The minimizer x" of (z) is determined as follows:
+ a,y,, then x² == 0.
Otherwise, x² 1 log (a,37 3).
B THEORETICAL PROOFS
Remark, In the SSMTC setting, all tasks share data drawn from the same source, and thus the cou- pling between tasks remains explicitly preserved through shared features. In contrast, in MSMTC, the optimization objective can indeed be decomposed into sum of single-task objectives. This decomposition reflects that the effect of parameter sharing becomes negligible, but it does not mean that the model becomes multiple independent models; the analysis is still conducted within a sin- gle multi-task model. Therefore, within the UFM framework, the influence parameter sharing inherently stronger in SSMTC than in MSMTC. The effect of parameter sharing disappears in MSMTC. The key reasons are twofold: (1) the multi-task model has high representation capacity (UFM model), and (2) the data are drawn from multiple distinct sources (the MSMTC setting). Under these two conditions, the shared features can freely adjust to each task independently, which allows the optimization problem decouple into task-specific subproblems. This decomposition holds in MSMTC but not in SSMTC. As a result, the theoretical results and experimental results observed in MSMTC closely resembles that in single task learning.
B. PROOF THEOREM 3.1
The key idea of the proof of Theorem 3.1 is to lower bound the entropy loss in Problem (3). The equality holds only if SSMTC-NC satisfied.
g @W!
W
W
H,b,,b..... (b.....,bbT)
H. b' b²
T
K
N
K Σ
ΣΣΣ
log
exp( hi,
with + + b()))
1=1,ii
T
AA
dw
W
2
Ao
16413
where hi represents the i-th sample of label l in the t-th task. First, we can find that the margin will not change if we subtract vector for all wo subtract scalar y for all of provided that f fixed. From Lemma A.T subtracting the mean minimizes the regularization loss. So, the minimizer must satisfy:
w 0, V1 [T]
(10)
K
b = 0, v
k=1
15

---

Published as a conference paper at ICLR 2026
Observe that
T K
K Σ
T
T
log(1+ +
exp(w/hk, - with + - b)))) + A H + Aw
+
Ab
18'12
1=1,I
1=1
@
T
K (whhik, bk])) + A H Aw Σ
T
log(1 (1 + (K 1)exp(
|
|| + A6
Σ
116112
K
(a)
(
T
K
K²
log(1 + (K 1)exp(
(K - 1)N (wFk
h + K
bk)))
T
An +
H
p + Xw
W
+ A Σ
|b'1l2
(b)
1=1
(
K
K
Nlog(1 + (K- - 11)exp(
(K
1)N
k
+ A
H
+
Aw
W
||
+
Σ
|
1=1
km
i=1
(c)
( Tllog(1 +K (K- 1)expp
K
Σ
T
w
AH
H
\w
Σ
|
(K 1)
kg
kp=1
5=1
(d)
(
T
TNlog(1 + (K 1)exp(
(A
H
Aw
(w))))
(K- 1JNT
4KAHdw
T w/
Aи H 2 + Aw
(c)
Invoking Lemma A.3 with TN, B = K - 1,1,7- y= [K (K-1)NT IKAHAW and x AH H 1 +
N
Aw Σ W F. we derive that if λy 2 4k. the minimizer is (W1 ........ WT H)
(O...., 0) and otherwise, the minimizer must obey x² > 0. Next, we examine the conditions required to reach the minimum value. In inequalities conditions under which equality holds are as follows:
app (b.. (ch and (d), we use Lemma A.2 The
11, E [K] \ [k), wi, hh, + bj, w/, h/e bl
N
V2 E |
w hi withi.
*
1ki, k2 [K], wt
h
h
+
N
ur
band
N +
1/2
(11)
K
K
K
V4,,22 E (T],
Σ
w
h4
h
Inequality (d) is also based on Ao
2 and the data is balanced for each label. The number of samples for each label is n. Inequality (d) holds with equality iff
ΣT
|6
Vt [T],k E [K].b] 0
(12)
Inequality @e) is based on
k1
by
wh < <1llh
w 1
(13)
1=1
16

---

Published as a conference paper at ICLR 2026
where À
NA and the condition for this equality to be true is: Ak
T KAw NAH Σ w
Vi E [n], h Kr
(14)
Due to the condition 14 remove bias in condition (IIp. By combining the condition (T0), and (1). we conclude
the optimizer must satisfy NC1, NC4. Because of the condition 12 we
that:
Aky, k2 E [K], t2 E [T], b1 * 12,
(15)
which satisfy NC3. Then we can infer that the optimizer must satisfy NC2, NCS. When one of the conditions is not met, the loss will be strictly greater than the minimum value. When of the conditions is not met, the loss will be strictly greater than the minimum value, so in summary, combining all the conditions for equality, we can conclude that the optimal solution must satisfy the SSMTC-NC.
B.2 GENERAL SSMTC-NC
When Assumption (5) and Assumption (6) are not satisfied, during the terminal phase of training using loss function in Problem (3), the features and classifiers of the network exhibit the following charactistics.
(NC1) Variability collapse. Each feature with the same label converges the mean.
(he (h k...kr (h (h - kr...rr 0
Ave kr.
(NC2) Task-speeific Simplex ETF. The weights the classifier for each task form Simplex ETF.
|| 1|2 - |1 12l - O. Vk, N' E [Ke), [T]
K₁ K,
(w) w
5k. K₁
Ak, k' [K],te [T)
(NC3) Orthogonality. mutually orthogonal.
The spaces spanned by the weights of the classifiers for any two tasks are
( T O, Ak [K,, M [Kel,t,t' [T] with +
(NC4) Convergence to weighted sum of task-specific classifiers. The shared features among different tasks converges the weighted of task- linear classifier weight vectors for each task-specific label.
Σt Vew 1E Velw 11212
ks kz
+
(NC5) Simplification Nearest Class-Center The network's classification result converges to choose whichever class has the nearest training class-mean.
(argmax(wk,, h) + +bb, b......, argmax (ammaxxwerrrr h2) + Drx hg
ky
argmin| kgo.kk
h Ai....y
17

---

Published as a conference paper at ICLR 2026
Similar to Appendix B.1 prove that the optimal solution of the loss function must satisfy the aforementioned constraints. Next, we analyze the loss function in Problem (3).
Kr Σ 111,
T
log(1 +
exp(wfth,
wih + b b()) + d H
7
+ AA
bb
b
t=
T
K.
Ke K,
llog(1 1 + (K₁ 1)exp(
(wft + b))))
T
AH
H
dw
Σ
c' [ + A Σ
c'
t=1
T
K
K?
R1
log 1 + (K₁ - 1)exp(
(w)
h
+ K2
b))
Kg
(K, 1).
1=1
kml
1=1
AA
+ Aw Σ d + d Σ H
ct bb11/
(g)
1=1
T
Ki
T
Nlog(1 + +(K, - 1)exp(
W
+
2 +
Aw
1
(K, - 1)N
1=1
k=1
1=1
(h)
x
( Nloog( (K₁ 1)exp(-
K,
K.
K2
w
kg
+
c K₁
W
h
(Ke 1)N
T
K,
AwHAwc Ke
Kr
Σ (e'llog(1 + (K, - 11)xp(
w
w
(K, 1)N
N
k=
Invoking Lemma A.3 when A < min, Ne'
IK. the minimizer must obey > are based on the Lemma A.2 and inequality (i) is based on:
Inequalities
K2
Ki
Kr
T
V c
K
W
h
of K.wy
ky
1=1
1=
ky==
i=1
#=1
K1
Kr
Σ
(All
11
ct K, (12)
1=
ky=
j=1
Kr
KT 113
cº Ke
Kelllu (12)
(16)
4X
ky=1
T
H
+
c' 1
4A
t=1
V
4A
N
H
Aw
Σ
d' ((
where X
NAn and the condition for equality is: DAw
V
h
k1 ET
1
Σ
Ve Kew
(17)
2X
==1
18

---

Published as a conference paper at ICLR 2026
When of the conditions is not met, the loss will be strictly greater than the minimum value. Combining all the conditions for equality, we can conclude that the optimal solution must satisfy the general SSMTC-NC.
B.3 PROOF OF THEOREM 3.2
The loss in Problem (4) can be viewed as weighted sum of the entropy losses of multiple indepen- dent variables losses and prove that for each loss, the minimizers (W HO) must satisfy Neural Collapse.
Thus, we decompose the loss into a weighted sum of multiple entropy
9 (W'
W H.
H², .b.....bb')
log(1
explwihk, withit + bi- bl))
(18)
1=1,
(AHll H1 + +w w/ +b.161113))
The specific form of the loss is shown in T8 which, as in equation specific loss g".
19
is decomposed into task-
g( WW
W H
HT (b.,...,'')) Σ
d' (²)
K2 T1
K
(19)
g'w, H²bb")
log(1 +
Σ
exp(wfhk. - w hi + +f - bl)))
1=1,1k
A H
Aw Xw
tbo110113
Similar to the proof of Theorem 3.1, since subtracting the mean from each w and 6% leaves the cross-entropy loss unchanged while reducing the regularization loss, the optimal solution must sat- isfy the following condition:
Kr
w = 0, Vt E [T]
k=
(20)
b
Vt [7]
Observe that:
19

---

Published as a conference paper at ICLR 2026
K2
log 1 +
Σ 1=1,1
exp(wfhif, - w + bj - -(]) + A 11 + Aw W + Ao 1010811
-=1
(
The
Kr hu + b())) A 1| Aw W
log(1 + (K₃ 1)xxp(
Ao 161161112
K,
(i)
(K)
K
Kg
ng + (K₁ 1)xxp(
ni(K, 1) ( Σ
h + n,b( - N H
Aw W || + >
(k)
K,n,log(1 + (K; 1)xxp(
w
h k + A H Aw
|
(K; k=1 i=
(I)
(m) (Lnzlog(1 + (Ke 1)exp(
1
14
(AH H Aw
A +x 1 I
Te ne(Ke, 1 V 4yHdW
(m)
Invoking Lemma A.3 with King, 3
K; 7 = m,(K. V DnnAw and
nr
AFIl H' 1 Aw 21=1 W' get that if > 2 2, the minimizer is (W H (0. 0)
and otherwise, the minimizer must obey x" > 0. In inequalities (i). (k. and (b. we use Lemma A.2 the conditions under which equality holds are as follows:
41,4 E [K.] \ {k}, wo, 1tk, + bf, = wf, tol i + bl2
Vi [ma]. th, w 12h.02
(21)
1ki, k2 [Kt), w
hhat + nelly = wi, Σ Σ
h nabiin
2=1
1=1
Vk [K] b = 0
Inequality (m) is based on:
n
w
h -
Al110
h
4A
mml
(22)
$ A1
where X
neW 4Ax
and the condition for equality to hold is:
Vi E [n.], wn V AH
(23)
Xw
Based on the above condition, we conclude that the classifier and features must satisfy the NC conditions within each task.
B.4 SSMTC-NC IN THE TASK-BALANCED CASE
Theorem B.1. Consider two binary classification tasks, the training data is task-balanced, and the number of samples with label (k1, k2) is n n (k1 [2], K2 [2]). Assume that the number of all samples is N, then any global minimizers of Problem 3 satisfies the following conditions: Consider two binary classification tasks, the training data is task aask-allanced and the number of samples with label (ki, k2) is n bi (k, E [2), h2 E [2)) Assume that the number of all samples is N and
Xh N 822
then any global minimizers Problem 3 satisfies the following conditions:
20

---

Published as a conference paper at ICLR 2026
1. ha fe 1
2. || l1affll2: (1ig (ige)) = 201, - 1, 1, E [2).4 e [2]
3. As rekk. increases, cos( w wxx) increases.
4. h hynoke=
why -w0
liwt twi l12
5. (argmax(wh,, h)) + b2g argmax h) + (22g) b1
argmin| h ki. Ao
kz
Proof. The key idea of the proof of Theorem 5.1 is to lower bound the entropy loss in Problem (3), and obtain the form of cos(w), wh, (k1 E [2]. K2 [2]). We denote the cross-entropy loss for Different from the theories in Appendix B.I we can only scale the cross-entropy loss for samples with the same labels.
samples with labels (ki, ,.,,.., KT) as u(ki, ki,,..,. , KT)).
u k1, 122)
2 n k2
Σ
log(1 Σ
exp(wth
h wh, h hi + bf b)r))
[=1
l=
2
Σ
log 1 + expl (xp(-22(wt,)) h k1 + bl2))
(24)
T
n 22:22
n M2. log(1 + (K - 1-exp(( ((
h Ki + 12.))
n A22
n A1
2 kz log + exp(- Σ (w),
h Ni Ao
+ b),))
n2 Ara
In 24 the second equality due the condition Combining the sum of all the cross-entropy loss and the regularization term, we derive the following composite loss function:
W! 2 H. b.bb... b²)
2
u(k1, k2) Ah H + dw
Σ
W
F
d
Σ
6 113
k- 1==1
Σ
(u(ki, 12) + HA
12 kg
Σ
kg
Aw
2
wk.
113)
2
(25)
k1=1 1==1
2
k Aa
(2n A2 log(1 expp
2Aw A
k1
n Akk
2
(AH
Aw
(2))
+ Ay
Σ
|
Aw
w
(12)
2
2
1=1
[=1
We can get the conditions that need to be satisfied as follows:
w/. h h wo, h k2 Vi,j E kna E [2]
(26)
w/, h Ki = w 1 No vki, A2 E [2], E [ag Ka
(27)
0, [2]
(28)
22
2
1 ks
Σ
wh, Vki, h2 E [2], i E kn.]
(29)
dw
21

---

Published as a conference paper at ICLR 2026
Table 2: Dataset illustration
Setting
Dataset
K
Multi-MNIST
2
Mulii-CIFARIO
2
2
SSMTC-NC
CIFAR 100-Cross
2
2 2
CelebA
2 2
2
MNIST-Split
2
2
2
MSMTC-NC
CIFAR-Split
2
CIFARIO0-Split
Split--x-5
3
8
Figure 5: The image on the left shows examples from Multi-MNIST with the label (3, 8), while the image on the right shows examples from MIIiI-CIFARIO with the label (truck, automobile).
Combining all the conditions metioned, can conclude that NCI, NC2, NC4, NC5 need to be satisfied. Moreover, we conclude that:
w 1| = || 112 = 1 1k k2 E [2]
(30)
w/, )² 21(w0 21120 21leel w
1=1
(31)
2 2
2nki,
log(1,
1)
dw
Let wh = -E w². And the cos(w), w w No 1a 1. From the analyt- ical expression of cos(ur), wo,)) we can conclude that cos(wn), wg,) increases with growing A1 when fixing the number of training samples N. So as pY,.s, (k1, x21 increases, cos(w)) w2 increases.
C DATASET ILLUSTRATION AND VISUALIZATION
C.I MULTI-MNIST AND MULTI-CIFARIO
MNIST and Multi-CIFARTO are used the case of two tasks in the SSMTC (Single-Source -usk Classification) scenario. As illustrated in Figure 3] we follow Sener & Koltun (2018): ) Li ct al. (024) to place the images from any two classes MNIST dataset in the top-left and bottom- right corners, respectively, with the empty areas filled using zero-padding. The same approach is applied to the CIFARIO dataset. We use Multi-MNIST-RN
to refer to the dataset constructed using the first P classes and the first Q classes of MNIST. Similarly, dataset constructed using the first classes and the first Q classes of CIFAR1O. By default, Multi- MNIST and Multi- CIFAR10 refer Multi-MNIST-1O-ID and Multi-CIFAR 10- respectively.
Serii-CIARRROPP.... denotes the
22

---

Published as a conference paper at ICLR 2026
C.2 CIFAR 1 1O-CCOSS AND CIFARI00- 100-SPLIT--5200
CIFAR 100-Cross- 10--Cross-10000 refers to assigning the label (i E (0.1, E {0, 2, .... 9}) to each class the CIFARI00 dataset. MAAAROO-Cros--I and each task classifies a task-specific label in the label (i.j). Specifically, the samples of class K' E {0. 2. ................................ .993 CIFARIO0 100 correspond to the samples with the label ([K/10J; K%10) in CIFARI00-Cross 10x 10. Similarly, CIFAR (00-Cross-4x4xx involves three tasks, utilizing the first 64 classes of CIFARI0O, and CIFARI00-CroSs CrO--Crros-333x3x3 involves four tasks, selecting the first 81 classes.
involves two tasks
For CIFAR 100-Spitt-5200, we follow Rosenbaum et al. (2018) to treat 20 coarse labels as distinct tasks and create multi-task dataset with 2500 train images and 500 test images in each task.
For the MNIST-Spllt-55: dataset CCFFARI-SSpli--555 the first five classes and the last five classes of MNIST (CIFARI0) are used as task-specific dataset respectively.
D DETAILED EXPERIMENTAL RESULTS
In Appendix D.I we provide experimental results the validation experiments of SSMTC-NC and MSMTC-NC. In Appendix D.2, we show the validation experimental results of SSMTC-NC and MSMTC-NC under different numbers of task classes and non-uniform task weights. In Appendix
we show that when using the MGDA, Uncertainty Weight, PCgrad, DWA, FAMO, FrirGrad algorithms to update task weights, NC still exists in SSMTC and MSMTC, In Appendix [.4 we conduct experiments on ImageNet, TinylmageNet and CelebA. In Appendix D.5 we the periments on different learning rates and (AH, Xw). In Appendix [.6] we show the experimental results when the train data task-balanced. In Appendix [D.7, we provide the experimental results on efficient training utilizing NC properties in MTL. We provide the specific experimental metrics
D.3
in the Appendix. We run experiments on a single NVIDIA RTX 4090 (24GB) GPU.
D.1 EXPERIMENTAL RESULTS ON SSMTC-NC AND MSMTC-NC UNDER LABEL-BALANCED CONDITIONS
Figures 6 and 7 depict the trends of the SNCI, SNC2-1, SNC2-2, SNC3, SNC4 and SNC5 met-
rics on the Multi-MNIST-1O-10 and Multi-CIFAR1O-10 10-100 datasets, all converging toward zero Figures 8 g
show the trends CIFAR100-Cres 100-Cross- 10x10, CIFAR 100-Cross-4x4x4 and CIFAR Cross-3x3x3x3 datasets. Figures 10 IT depict the trends of the MNC1, MNC2- 1, MNC2-2, MNC3, MNC4 metrics the MNIST-Split-5x2 CIFARIO- 10-Split-5x2 and CIFAR 100-Split-5x2) datasets.
The convergence of all metrics toward zero validates both the SSMTC-NC and MSMTC-NC phe-
nomena.
D.2 SUPPLEMENTARY EXPERIMENT FOR GENERAL SSMTC-NC AND MSMTC-NC
General SSMTC-NC. To verify the theory in Appendix B.3 empirically, we use Multi-MNIST-44 6, Multi-MNIST-3-9, Multi-CIFAR -----IIAARR---666 and Multi-CIFAR 10-3-9 to verify our results and set the task weight ratios be inversely proportional to 1.5 and 1 for Multi-MNIST-4-6, and set task weights to 3 and for Multi-MNIST-33 As shown in Figure 12 SSMTC-NC generalize to cases where the number classes and the task weights are different different tasks.
the class counts. Concretely, we set task weights
General MSMTC-NC. To verify the neural collapse across different task weights and different class number, use MNIST-Split-3-4 and MNIST-Split-3-7 datasets uniform and non-uniform task weights respectively. As shown in Figure [3] MSMTC-NC extends to cases where different tasks have varying numbers of classes and distinct task weights.
with
D.3 VALIDATION EXPERIMENTS OF NC UNDER MULTIPLE MTL LEARNING METHODS
We show the experimental results Multi- 10-10 and Multi-CIFARD CIFARIO- 10-00-11-0000000 datasets in the setting of SSMTC, and the experimental results on CIFAR 100- 101--0000-------- Split-5x20 in the MSMTC setting when using the MGDA algorithm. As shown in the Figure 14 and Figure 15 when the MGDA
23

---

Published as a conference paper at ICLR 2026
W663]
VO611
VG611
VG011
10
Woc11
Restee1l
VO011
ReeNetLE
VGGTT
ReeNerll
101
Redderl0
Vi011
Rechet34
12-2
SesNee34
ResNer33
Rediet34
30-1
19
10-1
13:3
10
19a4
13-1
200
400
203 SNC2-1 1
200
600
SNC1
SNC3
SNC4
VGG31
VGG11
VGG11
VGG11 VG013 3
100
VO013
V0613
VCG13
101
Restee11
ReNNetE
ReeNer1a
Reddee13
Restee3t
ReeNet3)
Reddeet3
10-2
20
10"
107
22-3
15a
20
12"4
13-1
13
200
400
203
200
400
SNC1
SNC2-1
SNC3
SNC4
Figure 6: Illustration of SSMTC-NC across different network architectures Multi-MNIST-10-100 (top) and Multi-CIFAR10- 10-100 (bottom). The horizontal axis represents the training epochs.
5:033
38"
SAC2-22
SMC
36
300
Figure 7: Illustration of SSMTC-NC across different network architectures Multi-MNIIST-101100 (top) and MitiiCIIFARROO000 10 (bottom) (Supplementary Experimental Metrics). The horizontal axis represents the training epochs.
1200
VO332
V9233
10-1
VOSIL
VO013
32 4
(O512
101
VODL1 VO013
Easlet34
10
220
402
SNC1
SNC2-1
SNC3
SNCA
19513
VOGLL Noveet AucVec2
(G513 Frshia3
VOG1L
SNC1
SNC2-1
SNC4
VOCIL Navvil
fastet23
22-1
101
10-1
200
202
402
200
400
202
SNC1
SNC2-1
SNC3
SNC4
Figure 8: Illustration of SSMTC-NC across different network architectures on CIFARI00-Cross 10x 10 (top), 10EAR100--ros-4xc- (middle) and zontal axis represents the training epochs.
(bottom). The hori-
24

---

Published as a conference paper at ICLR 2026
160
10'1
10a
[63
100
SNC2-2
SNCS
10
Vent sIntn
0.1
100
200
300
402
10
20
30
43
50
SNC2-2
SNCS
Figure 9: Illustration of SSMTC-NC across different network architectures on CIFARI00-Cross- 10x10 (top), CIFARI00-Cross-444, (middle) and MAAARIOO-Cros--300 (bottom). (Supple- mentary Experimental Metrics). The horizontal axis represents the training epochs.
VO213 VOGI
VOS12 VO033
101
10-1
TerleC4
0.1
Tachee24
0.0
0.00
203
400
50
100
130
30
100
150
10
20
30
MNC1
MNC2-1
MNC3
MNC4
VOS11
(0013
VOG11
0.6
VISSS
10--
02
0.2
0:2
0.0
MNC1
MNC2-1
MNC3
MNC4
VO013
V5513
1.5
V0033
0.75
05533
8.1
0.25
202
150
200
150
200
200
MNC2-1
Figure 10: Illustraiion of MSMTC-NC across different network architectures on MNIST-Split5x2 (top), CIFAR 10- Sppititt-52 (middle) and CIFAR PARRI000-Spit5x2222 (bottom). The horizontal axis repre- sents the training epochs.
6.30
40313
6375
6200
V0573
3.0
VOS
0.28
0.18
6622
6.38
8.00
320
200
320
235
102
150
230
MNC2-2
MNC2-2 [CIBAR10-SpII-554
MNCC22 SAAR100--plIt-52220
(MNIST-Split-S) 5x2)
Figure 11: Illustration of MSMTC-NC (metric MNC2) across different network architectures on MNIST-Splitxx2 (left), CIFAR 10-Sltt-2-- (middle) and CITARI00-Splii-xx00 (right). The hori- zontal axis represents the training epochs.
algorithm is used to update the task weights, SSMTC-NC and MSMTC-NC still exist though
25

---

Published as a conference paper at ICLR 2026
V8G13 ReshietB
VGG13
10-1
ResNer18
ResNett8
Restiet88
Reshierd
RecWee 94
ResNee34
Res
10
10P
10
10-
10-2
10t3
2000
400
0
200
200
400
SNC1
SNC2-1
SNC3
SNC4
VGG11
vG311
30a
VG611
Ve611
VG613
VG613
VSG13
Reshieti
10
ResNetla
10a
ResNo03
ResNot34
10-3
RosNet24
ResNa124
101
10-3
100
10:3
10-1_
10
10-3
10-1
200
400
SNC1
SNC2-1
SNC3
SNC4
10a
VGG13
VGG11
VOG15
VGG11
10
VGG13
VGG11
VSG11
VGG13 5
ResNett
ResWell8
ResNet28
Reshett8
ResNietd
BecNee3
ResNet34
ResNet34
10
10-2
107
10
10
200
400
2000
400
0
200
400
200
400
SNC1
SNC2-1
SNC3
SNC4
10a
VSG11
10
VSG13
Reshielia
10-1
ResNocls
VG613
BesNutio
10'
V8613
Bestetl8
Rashia34
RosNor34
10:1
ResNo124
10a
10a2
1.0
10a
10
10-3
10-)
10"
200
200
SNC1
SNC2-1
SNC3
SNC4
Figure 12: Illustaaioon of SSMTC-NC across different network architectures on Multi-MNIST-3-9 (first row), Multi-MNIST-466 (second row), Multi-CIFAN 10-3-9 (third row), and Multi-CIFARIO- 4-6 (fourth row). The horizontal axis represents the training epochs.
the task weights are constantly changing during training, indicating that SSMTC-NC and MSMTC- NC exhibit generalization across different task weights.
Similar to MGDA experiments, we show the experimental results using Uncertainty Weight, DWA, PCgrad, FAMO, FairGrad, As shown from Figure 16 to Figure 35, SSMTC-NC and MSMTC-NC still exist when using different multiple MTL learning methods.
D.4 EXPERIMENTS ON LARGE-SCALE DATASETS
We validate the SSMTC-NC and MSMTC-NC on ImageNet, TinyImageNet and CelebA. We utilize the first 512 classes ImageNet to construct 729 classes of ImageNet to construct the ImageNee-Cros--33333kta for SSMTC. We select the 1000 classes of ImageNet to construct the ImageNet-Split- 10x 100 for MSMTC. We select two groups of attributes on CelebA and keep the dataset balanced. As shown in Figures 36
utilize the first
and utilize the 1000 classes
of ImageNet to construct the
37 38
43
SSMTC-NC and MSMTC-NC still exist in ImageNet, TinyImageNet and CelebA.
26

---

Published as a conference paper at ICLR 2026
0.75
10
V6611) VGG11 3
VOKTI VIS113
V9611
VISI1
0.10
V5613
0.4
VIS113
Reck-rta
WGGT3
RoNr33
10"1
0.05
0.2
0.0
0.00
0.00
2000
400
100
100
200
300
100
20
40
50
MNC1
MNC2-1
MNC2-2
MNC3
1.0
VEGLL
VG013
0.15
V0015
V9231
0.075
0.050
V0033
VG631
Restects
VOOLU
VG211
0.75
0.50
VC013
V5G11
0.5-
10
0.05
0.025
0.25
0.000
0.00
200
400
100
100
200
300
100
200
20
40
60
MNC1
MNC2-1
MNC2-2
MNC3
MNC4
1.0
V6611
V6613
0.3
VG513
19611
0.10
V5613
V9611
1.5
VI511
VIS813
0.75
V5611
WGG
Reskvetth
0.2
Rethietl
RocWer1
10-
0.1
0.05
0.5
0.0
0.00
0.0
0.00
200
400
100
100
200
300
100
200
0
20
40
60
MNC2-1
MNC2-2
MNC3
MNC4
100
VGG12
V2631
VG011
VG211
V5G33
VGG13
0.2
Rochelt
V3013
0.10
VOG13
Aeclects
0.4
VG213
0.50
VGG13
10.
0.05
0.25
0.00
0.00
200
400
100
100
200
300
100
200
20
40
60
MNC1
MNC2-1
MNC2-2
MNC3
MNC4
Figure 13: Illustration of MSMTC-NC across different network architectures on MNIST-Split- 3-4 (first row) with uniform task weights, MNIST-Split-3-7 with uniform task weights (second row), MNIST-Split-3-4 with non-uniform task weights (third row), and MNIST-Split-3-7 with non- uniform task weights (fourth row).
6010
10
1001
VIZE1
V001
WOOD MANCEE
131
PAdRUCE
33.0
31-4
SNC1
SNC2-3
SNC2-2
SNCO
S3iCe
SNCS
137
V0113
10-1
vOT1 oereers
13'
02-
SNCC
SNC22
SNC2-2
SNC3
SNC4
SNCS
Figure 14: Illustration of SSMTC-NC across different network architectures on Multi-MNIST-IO- 10 (top) and 10- 10-10 (bottom) by using MGDA algorithm.
10f
VOG13
0.3
VG223
0.3
VGSL3
1.5
VG213
VG013
1ea
1.0
RaaMa1:
0.50
1ashu124
Iso Na.44
0.1
0.1
0.25
0.00
200
400
10C
200
360
100
200
40C
MNC1
MNC2-1
MNC2-2
MNC3
MNC4
Figure 15: Illustaation of MSMTC-NC on CIFAR SOARROO0-----t--- Split-5x20 by using MGDA algorithm.
27

---

Published as a conference paper at ICLR 2026
0.5
V8G13
Reofett38
0.4
Reshlett
0.5
ResNee18
Restiet88
10.
Resilee 34
Rasha
Recleed S4
0.3
100
0.3
10a
0.2
50
10"1
0.1
0.0
200
200
SNC1
SNC2-1
SNC3
SNC4
Figure 16: Experiment on Multi-MNIST using DWA method
100
0.5
150
R
RcsNe034
04
ResNet34
Reshett4
107
0.3
0.30
0.
50
0.05
0.1
0.00
200
400
200
400
200
400
0
200
400
SNC1
SNC2-1
SNC3
SNC4
Figure 17: Experiment on Multi-CIFAR10 using DWA method
D.5 EXPERIMENTS WITH DIFFERENT LEARNING RATES AND REGULARIZZTTIN COEFFICIENTS
We have supplemented the experimental results for learning rates of 10-2 and 10-3, as well as (AH, Aw) of 5. 10-3 and 5 10-5, As shown from Figure 44 to Figure 59 the experimental results further validate the generality of our findings.
D.6 EXPERIMENTS ON SSMTC-NC IN THE
CASE
10. By varying the num- ber of samples for each label (k1, K2) while keeping the train data task-balanced, we find that cos(w) w increases the the number of samples with label (k1, k2) increases as shown in Figures 60 61 62 The other NC phenomena still exist.
We experiments the Multi-MNIST with K = 2,
28

---

Published as a conference paper at ICLR 2026
10
VOG23
VSS61
2.6
VOG12
V8611
VG223
1.
VGGL3
0.75
VCGLS
Wid11
Nes
0.50
0.2
0.7
0.5
0.25
0.00
400
200
200
400
MNC1
MNC2-1
MNC2-2
MNC3
MNC4
Figure 18: Experiment MNIST- using DWA method
1.0
V06i1
VOCLS
0.6
0.4
V5611
VOG13
0.6
0.4
- Vidii
VOcL3
1.5
Voca3
vosil
0.75
VOGIS
0.50
10a
0.2
0.2
0.5
0.0
0.00
200
400
100
200
300
100
200
200
100
200
400
MNC1
MNC2-1
MNC2-2
MNC3
MNC4
Figure 19: Experiment on CIFARIO-Split-522 using DWA method
107
VSS13
VGGL1
0.3
VEGL1
VSGL3
VSG13
VGGI
125
VGG13 3
VIGEE
10.
Bas N4r
0.2
0.6
Nat
100
105
50-
0.1
10
0.0
6
400
400
SNC1
SNC2-1
SNC3
SNC4
Figure 20: Experiment on Multi-MNIST using Uncertainty Weight method
0.8
150
VGG11
VGG11
VG613
VGG13
B 4
0.20
ResNe118
0.6
ResNet28
Restietl
ResNa234
Res7ot14
RoshNet34
0.15
75
160
0.30
50-
0.2
0.05
25
0.00
0.0
200
400
200
400
200
400
200
400
SNC1
SNC2-1
SNC3
SNC4
Figure 21: Experiment on Multi-CIFAR10 using Uncertainty Weight method
1.0
V6611
TAs WATE
VIG13
0.2
0.2
W6611
1.0
V8611
VISS13
0.50
0.75
VGG11
0.1
0.3
1.0
0.0
0.0
200
100
200
300
100
200
MNC1
MNC2-1
MNC2-2
MNC3
MN
Figure 22: Experiment on MNIST-Split-5x2 using Uncertainty Weight method
V0011 A
VOC11 Rediver1 MaphN
0.2
VOGLL Restver
1.5
VOGLL Redtert
VOGAL
107
1.0
0.50
Rordent3
0.5
0.25
0.0
0.00
MNC1
MNC2-1
MNC2-2
MNC3
MNC4
Figure 23: Experiment on CIFARIO-Split-x22 using Uncertainty Weight method
29

---

Published as a conference paper at ICLR 2026
VG611
0.6-
VS611
VGa11
V6G11
V0011
ResMett2
VGG13
Reshett8
0.0
ResNet18
VGG13
VOGL3
ResNett8
10
ResNet34
0.4
Reshett4
ResMet34
Rethet34
10
0.4
0.2
10-1
0.1
50
0.0
0
200
400
200
400
200
400
0
200
400
SNC1
SNC2-1
SNC3
SNC4
Figure 24: Experiment on Multi-MNIST using PCgrad method
200
VGG11
VSG11
VGS11
VGG11
VGG13
VG613
0.8
VGG13
VoG13
Restett28
ReshettB
ResNet18
Reshett8
10
ResNet34
Reshettd
0.6
ResNer34
Reshiet4
1.00
109
0.2
50-
10-1
0.2
0.0
200
200
SNC1
SNC2-1
SNC3
SNC4
Figure 25: Experiment on Multi-CIFARIO using PCgrad method
105
VG611
VOGLS
VODS3
VG611
VISL1 VISIL
VOSIL
0.75
WGG11
VOCLE
0.50
103
0.1
0.1
0.0
200
600
200
400
200
200
MNC1
MNC2-1
MNC2-2
MNC3
MNC4
Figure 26: Experiment on MNIST-Split-5x2 using PCgrad method
40000
Bealut58
VG2L3
vest1
0.4
0.2
Vidi1
VGGL3
0.2
VCG33
V8G13
VCC13
Fas MuLz
0.75
3.50
VCC33
20000
0.0
200
MNC1
MNC2-1
MNC2-2 -2
MNC3
MNC4
Figure 27: Experiment on CIFARIO-Split-522 using PCgrad method
1.0
10a
VG911
VSGL1
VGS11
VSG11
VGG13
RecNet12
VGGL3
Reshiet18
0.8
VGG13
ResNet18
VSGL3
Reshittta
15
10
0.4
0.2
0.0
0.0
0
200
400
200
400
200
400
SNC1
SNC2-1
SNC3
SNC4
Figure 28: Experiment on Multi-MNIST using FAMO method
30

---

Published as a conference paper at ICLR 2026
10a
VGG11 VoG13
V6G11
VGG11
V6G11
ResNetta
ResNett8
VGG13
0.25
V8G13
ResNetta
60-
VOGL3
ResNett8
10r
Reshett4
ResNet34
ResNett4
ResNet34
40
0.15
10.
0.05
10
0.05
200
400
200
400
200
400
0
200
400
SNC1
SNC2-1
SNC3
SNC4
Figure 29: Experiment on Multi-CIFAR10 using FAMO method
10
V6611 VIGG13
WA611 W3613
V5611
VS613
1,0
VISLL
VIG133
0.75
WGG13
W661]
10-1
Restur2h
0.05
Redier54
Recke
0.0
0.00
0.0
0.00
200 MNC1
400
100
100
200
300
100
200
400
MNC2-1
MNC2-2
MNC3
MNC4
Figure 30: Experiment on MNIST-Split-5x2 using FAMO method
102
VGG32
0.4
VOG21
1.5
VGSA1
VCG13
VGG33
VCGLI
VOG13
VG2L3
VCG33
Beahet22
1.0
0.2
0.1
0.0
0.00
200
400
300
100
200
300
300
200
400
MNC2-1
MNC2-2
MNC4
Figure 31: Experiment on CIFAR10-Split-5x2 using FAMO method
100
125
38
0.5
Reshee18
BecNee11
Re
BestNeL 54
Beshwi34
RecNeet34 54
0.3
50-
0.2
10:
0.1
25
0.0
200
200
200
SNC1
SNC2-1
SNC3
SNC4
Figure 32: Experiment on Multi-MNIST using FairGrad method
10a
VS611
VSG11
VSG11
125
VSG11
V0013
VGG13
0.6
V3G13
V3G13
RecNette
Reshet18
Reshett8
1.00
Reshittta
150
ResNo134
0.10
0.4
75
10
0.05
3
0.00
0.0
200
400
200
400
200
400
SNC1
SNC2-1
SNC3
SNC4
Figure 33: Experiment Multi-CIFAR10 using FairGrad method
31

---

Published as a conference paper at ICLR 2026
10
VSS13
VSA11
0.4
ViS11 VI661 Redieeta
V0S11
VI6613
1.5
US61)
WG6
0.75
V5611
10-1
Redkeete
0.2
Rndheeta
1.0
Reck-r1
0:25
0.50
RecW-r1a
0.00
200
100
100
400
MNC1
MNC2-1
MNC2-2
MNC3
MNC4
Figure 34: Experiment MNIST-Split-5x2 using FairGrad method
1.00
0.06-
0.15
VOSLL
VOG23
0.75
VOGLS
VCGL3
0.50
101
0.25
0.05
0.02
0.25
0.00
200
400
300
100
200
300
100
200
400
MNC2-1
MNC2-2
MNC4
Figure 35: Experiment on
using FairGrad method
VS611
0.25
VSE11
VG611
VSG11
VOG13
3
ResNe118
0.20
ResNet18
Ne118
800
Reshati8
ResNet34
R ResN0034
Reshet34
0.15
600
0.10
400-
0.1
0.05
200
0.00
0.0
0-
200
SNC1
SNC2-1
SNC3
SNC4
Figure 36: Experiment on ImageNet-Cross 2xlnsx222xxxxxxxxx22x arggeNee-Cosss222x2222222222
VOGOLL
10623
100
0.25
3.50-
Rorkir1s
600
0.20
4.00
0.15
200
200
3.5
100
SNC1
SNC2-1
SNC3
SNC4
Figure 37: Experiment on ImageNet-Cross 3roos=-33x3x3x33x3
0.8
04
100
02
1.2
4.8
141
02:
61-
10
10
SMC1
SNC2-1
SNC3
SNCA
Figure 38: Experiment on Image Siiieet-Cross-ie 10x10x00
107
Valan Tanuforarr
C,2
Vilcn Taraformrr Rectent
Valan Thansforarr Reaveris Beahett4
Valan Frruforrrr
kecwert
Recteets
0.5
0.1
Sastett
0,0
200
200
400
200
200
MNC1
MNC2-1
MNC2-2
MNC3
MNC4
Figure 39: Experiment on ImageNet--Slit-1O000
32

---

Published as a conference paper at ICLR 2026
0.8
VGG13 Reshiet2
VGG13
ResNet18
ResNett8
300
Restiet88
ResNee34
RecWee 34
ResNet3d
200
0.2
0.0
0.9
200
400
200
400
200
400
200
400
SNC1
SNC2-1
SNC3
SNC4
Figure 40: Experiment on TinylmageNet-Cross 10x20
ViS11
VGSI1 VGS13
12-
VG613
VGS11
0.6
VGGEE
8
0.08
RocMelL
1.0 -
RoMMii2
0.5
0.06
0.6
0.3
0.4
0.
0.1
200
400
200
400
200
400
200
400
MNC1
MNC2-1
MNC3
MNC4
Figure 41: Experiment on TinylmageNet-Spltt--00022
10
3.0
VGS11
VSS11
WGG11
VOSL3
VOCLS
VOCA3
VCCLS
Restwr334
Resbar1
0.8
Racker34
RenNer28
0.3
RaoMer334
Redken:
12.5
Resteeta
13.0
3.6
0.2
04
02
0.0
200
400
200
400
200
400
0
200
400
SNC1
SNC2-1
SNC3
SNC4
Figure 42: Experiment on CelebA on three tasks (Smiling, High ehgghchhekbones, Mouth Slightly.open)
VS213
VGCL3
VG213
VGGL3
Renkat:
RetNer38
RetNe1::
Restleta
Nochv134
<dl NooMM:344
0.4
12.0
7.5
10a
0.2
2.5
10
0.0
200
400
200
400
200
400
200
400
SNC1
SNC2-1
SNC3
SNC4
Figure 43: Experiment on CelebA on three tasks Aarched_Eyebrows, Narrow_
Bags_ Under_Eyes.
33

---

Published as a conference paper at ICLR 2026
VGG1 VG613 ResNet1B Resherth
V661
VOG:1
V6G11
VGG13
VGGL3
V8G13
ResNet18
0.6
Restett1
Restette
Reshet31
Reshett3
ResNet34
0.2
0.0
200
400
200
400
200
400
200
600
SNC1
SNC2-1
SNC3
SNC4
Figure 44: Experiment on Multi-MNIST using regularization terms 5 10
VGG11
VGG11
VGG11
VDGE1
0.3
VGG13
Beshiet34
Resdial34
RenNar34
Resfee34
0.2
0.1
10f
0.0
0.0
SN C1
SNC2-1
SNC3
SNC4
Figure 45: Experiment on using regularization terms 10-5
VGGL1
VEGL1
VDG1
10a
ResNett
VGGL3
0.3
VOGL3
ReshettD
ResNet1
VOG1
100
Reshetto
VOG13
10.
Reshaette
Reshett3
Res
ResNet33
10a
0.2
0.2
60
10-1
0.1
20
10-2
200
400
200
400
200
400
200
400
SNC1
SNC2-1
SNC3
SNC4
Figure 46: Experiment on Multi Multi-CIFARIO using regularization terms 5 10-3
107
VGG11
VGG11
VGG11
100
VDG11
10a
VGG13
ResNot18
0.25
ResNet18
VGG13
0.15
VDG13
VDGL3
0.20
0.35
0.10
30
40
0.30
0.05
0.05
0.00
SNC1
SNC2-1
SNC3
SNC4
Figure 47: Experiment on Molti-CIFARRO using regularization terms 5 10-5
10P
vGG13
H5-13
VGG13
Redint53
0.02
Rodtera
Redien:3
0.50
Renterta
100
RooMe134
0.01
0.25
0.00
200
200
200
MNC1
MNC2-1
MNC2-2
MNC3
MNC4
Figure 48: Experiment MNIST-Split-5x2 using regularization terms 5 10-3
34

---

Published as a conference paper at ICLR 2026
0.15
10
96419
0.10
w6611
W6513
Redken
0.50
Reddent
109
0.025
0.25
0.00
0.00
200
200
MNC1
MNC2-1
MNC2-2
MNC3
MNC4
Figure 49: Experiment on MNIST-Split-5x2 using regularization terms 5- 10-5
VC2LL
0.75
(G611
0.50
Kedter13
0.50
Re
Roakee24
RuiNei34
0.25
400
200
MNC1
MNC2-1
MNC2-2
MNC3
MNC4
Figure 50: Experiment on CIFARIO-Split-522 using regularization terms 10-
107
VGGL3
VGGL2
0.4
V3013
VS011
0.15
VOG13
VOG11
VG211
VG213
0.75
VCGAL
VCG33
ReckoelR
Redken
0.00
200
400
200
400
200
MNC1
MNC2-1
MNC2-2
MNC3
MNC4
Figure 51: Experiment on CIFARIO-Split-522 using regularization terms 10
107
VOGLI
VGGL1
0.12
VGGL1
VDGL1
VG613
0.35
VG613
0.30
VGG13
100
VGG13
0.30
0.06
0.05
40
30a
0.02
0.00
0.00
SNC1
CN-22-
SNC3
SNC4
Figure 52: Experiment on Multi-MNIST using learning rate 1 10
30"
vaau
VGGL3
VGG11
VGG13
10-a
WGG11
V5G13
VGG11
VGG13
ResNer18
ResNet18
ResNett8
Reshiett
10a
10-3
10l
200
600
200
200
400
200
400
SNC1
SNC2-1
SNC3
SNC4
Figure 53: Experiment Multi-MNIST using learning rate 1 10
35

---

Published as a conference paper at ICLR 2026
VGGL1
0.20
VGGL1
VOGLI
120
VOGL1
101
VGEL3
VGE13
0.15
VOGL3
100
VOGL3
107
RcsNet18
0.15
ResNet18
Reshett8
80
Reshettl
0.10
0.30
101
101
0.05
9.65
0.00
0.00
200
400
200
400
200
400
200
400
SNC1
SNC2-1
SNC3
SNC4
Figure 54: Experiment on Multi-CIFARI0 using learning rate 1 10
VGG11
VGG11
0.20
V6611
VGG11
VGG11
VGG13
VEG13
VEGI3
0.150
ResNet34
0.15
ResNet34
0.125
100
107
0.10
107
0.075
0.05
104
400
SNC1
SN-221
SNC3
SNC4
Figure 55: Experiment on Multi-CIFARI0 using learning rate 1 10
10a
PG2L)
VG221
0.04
VGGAL VGGL3
VC-13
0.75
VCG11
VCG33
Reodertia
Redieria
0.02
10
0
0.00
20c
400
200
400
200
400
200
400
200
400
MNC1
MNC2-1
MNC2-2
MNC3
MNC4
Figure 56: Experiment on MNIST-Split-5x2 using learning rate 1 10
30a
VOGLL
0.2
VSCAL VSC13
VGSL1 VCGL3
VOG11 VOC13
0.75
VCG13
VCG1L
0.1
3.05
0.0
2.00
0,00
200
400
2ú0
40c
200
400
200
400
200
400
MNC1
MNC2-1
MNC2-2
MNC3
MNC4
Figure 57: Experiment on MNIST-Split-5x2 using learning rate 1 10
0.4
VOGIL Resket1
VGGL2 Rndiii16 Beahitt4
VGG11
0.75
VOGEL
Keclerta
0.2
Rahe224
0.1
keckwr1
0.50
Rerten1
0.0
200
400
MNC1
MNC2-1
MNC2-2
MNC3
MNC4
Figure 58: Experiment on CIFARIO-Split-522 using learning rate 1- 10-2
750
0.4
0.2
500
Rahez2 Borke33
Vu.1 bo
1.5
0.50
RaaMetaa
0.2
0.1
Rentert3
0.5
0.25
0.0
0.00
MNC1
MNC2-1
MNC2-2
MNC3
MN
Figure 59: Experiment on CIFARIO-Split-5x2 using learning rate 1- 10-3
36

---

Published as a conference paper at ICLR 2026
10-
10-3
0.25
0.00
10a
2x10
-0.50
107
-0.75
400
200
400
SNC1
SNC2-1
cos (W: w()
SNC4
Figure 60: Experiment on Multi-MNIST (K = 2, 8000).
0.75
10
0.50
10"
10-2
0.00
10a
-0.25
10-1
200
400
200
400
200
400
200
400
SNC2-1
cos (w)). w?)
Figure 61: Experiment on Multi-MNIST (K = 180000
100
10:1
10
80
10-2
10P
10-1
10-2
200
400
200
400
200
400
0
200
400
cos cos(w), (W), w?)
Figure 62: Experiment on Multi-MNIST (K = 10, N 50000).
D.7 EXPERIMENTS ON EFFICIENT TRAINING UTILIZING NEURAL COLLAPSE PROPERTIES
By leveraging the characteristics of neural collapse, modifications can be made to the multi-task network structure by reducing the dimension of the features the number of classes, achieving parameter savings without compromising performance. Given the properties of SSMTC-NC and MSMTC-NC, we initialize the task-specific classifiers orthogonal Simplex ETFs. We reduce the feature dimension to the sum of the number of classes for each task, and freeze the parameters of the last layer in the deep neural network during training.
As shown in Table the experimental results demonstrate that we achieve parameter savings without compromising model performance by utilizing the ETF classifier. When applied to CIFAR100-Spp Sp--St-i- 5x20, better performance can be achieved. For example, we achieve the 25.74% parameter saving when using the VGG11 network for classifying the CIFAR100-SS CIFAAR000SSlii-5x200 dataset.
Since the NC behavior in MSMTC closely resembles that in single task learning, it naturally suggests that methods developed for NC under imbalanced single-task learning could also be effective for imbalanced MSMTC. Following Yang et al. (2022), the final layer is replaced with Simplex ETF and dot-regression (DR) loss is adopted to improve accuracy under class imbalance. We evaluate the approach CIFAR-10 10 and CIFAR- 100 with imbalance ratio of 0.01. As shown in Table this strategy leads to consistent performance gains imbalanced MSMTC, indicating that insights derived from NC in single task learning could be transferable the MSMTC setting.
37

---

Published as a conference paper at ICLR 2026
D.8 EXPERIMENTS GENERAL MTL
When no assumptions are made about task distribution and class, we find that features of the same class collapse together illustrated in Figure 63
t-SNE visualization of features at epoch 500
75
50
25
t-SNE component 2
-25
-50
-75
-100
-100
-75
-50
-25
0
25
50
75
100
t-SNE component 1
Figure 63: Experiment on Multi-MNIST
We change the sample size for each class from 500 10 and found that neural collapse still exists, as illustrated in Figure (4 and Figure 65
cavplic
0.3
10 careli
10
180 orrgie
canoki
0.54
00
Figure 64: Experiment on Multi-MNIST
D.9 EXPERIMENTAL METRICS
Table 5 and Table show the metrics of verify SSMTC-NC and MSMTC-NC respectively.
38

---

Published as a conference paper at ICLR 2026
2.0
10
damplic camble
10 cinpees
pinc
1.5
10'
0.2
05
Figure 65: Experiment on MNIST-Split-5x2
Table 3: Performance and parameter saved using learned and ETF classifier
Dataset
ResNet18
ResNet34
VGG11
VGG13
Arch
Learned
ETF
Learned
ETF
Learned
ETF
Learned
ETF
Test Accuracy (%)
UIAARIO-Crros Crosss--s--0000
53.32
53.52
56.58
57.16
55.68
55.54
59.94
60.27
MNIST-Split SNIST-Split-5x2
99.90
99.83
99.81
99.78
99.68
99.78
99.85
99.75
90.44
91.22
92.04
91.82
90.61
91.43
93.38
93.46
CIFAR SHAAR100-Split5x200
62.45
65.20
60.03
61.90
68.82
69.76
69.07
70.53
Parameter Saved (%)
CIFARI00- MAARIOOO--OSS
21.26
11.18
0
25.74
25.24
CIFARIOG Split
MNIST-Split
0
20.71
0
10.87
0
25.13
0
24.63
CIFARIO-Spie
E DISCUSSION ABOUT THE RELATED WORK
E. RELATED WORK ABOUT NEURAL COLLAPSE
The difference between Li et al. (0024) and ours.
Analytical Focus. Li et al. (2024) primarily investigates neural collapse patterns under multi-label settings and implications on guiding multi-label training. Our work mainly focuses on the geometry of task-specific classifiers and features leamned by each task, and investigates how task correlation impacts features learned by each task.
Experiment Difference. Our experimental setup differs substantially from that of Li et al. (2024), encompassing the network architecture, loss function, and dataset construction. The network in Li et al. (2024) includes single classifier layer, while our work employs task-specific classifier layer for each task.
Theoretical Contribution. Whereas Li et al. (2024) establishes theoretical guarantees for NC in multi-label classification, our work provides the first formal characteriza- tion of NC phenomena in MTL. frameworks, incorporating heterogeneous task weights, class-imbalanced scenarios, and waaa---orraaatondiieeeee geometric transformations of task- specific classifier manifolds. Previous research never required consideration of alignment between task-specific classifiers and features. In their UFM proofs, h and w maintain one-to-one correspondence, expressed as h Aw, where the same for each class. In the SSMTC setting, we need to consider the weighting coefficients of features learned by different task-specific classifiers within the shared feature representation as shown in (32). These coefficients are influenced by the number of classes per task, the task weights assigned to each task, and the task correlation. Only with properly configured coefficients can the equality conditions of series of inequalities hold simultancously, thereby ensuring that the optimal solution is attainable.
h X Ar A2 + ATWT
(32)
39

---

Published as a conference paper at ICLR 2026
Table 4: Long-tailed classification aceuracy (%).
Datasct Arch
ResNet18
ResNet34
VGG11
VGG13
Learned
ETF+DR
Learned
ETF+DR
Learned
ETF+DR
Learned
ETF+DR
CIPARIO--ppit-5x2
73.47
77.21
74.92
78.29
73.32
76.62
74,81
79.12
CIFAR100-Spli--x200
53.24
56.61
54.13
57.21
53.76
55.87
54.48
56.13
Table 5: SSMTC-NC Metrics
SNC1
Ave [(h (hhtmee ((h (hhiere (l
kr
SNC2-1
Ave (lcos(wt, why) R- thet
SNC2-2
(ye
1 Aa//Ave(l will2,1llll wall2..... l[
SNC3
Ave
(llcos wx()11
A.k ie Set
SNC4
Ave k2
h d2
Σt who
E wi 12
SNC5
E (argmax h.) + bn,
argmax ( h.) + b argmin h: 2 " .1... [12]
kT
kg...Ky
E.2 RELATED WORK ABOUT MULTI-TASK LEARNING
In recent years, multi-task learning has witnessed substantial advancements (Liu et al., 2019b Liu
& Tsang 2015 Mao et al. 2022 2021 2020a b; Gong et al. 2023); however, research on its under-
lying mechanisms remains limited.
Shenouda et al. (2024) indicates that the norm associated with vector-valued variation spaces en- courages the learning of shared features that are useful for multiple tasks. Our framework provides mechanistic insights through two key advancements. (I) We explicitly characterize shared features as linear combination of task-specific classifiers. (2) Moreover, task-specific classifiers are aligned with the task task-specific latent features, which are the features learned by each task. Our work provides the granular characterization of how shared features are useful for each task in modern deep neural networks.
Collins et al. 2024) indicates that multi-task training aligns points with the same label across tasks. Consistent with the theoretical framework, experimental observations of NC1 further confirm that features sharing the same label collapse to their class mean.
40

---

Published as a conference paper at ICLR 2026
Table 6: MSMTC-NC Metrics
MNCI
A - (i) - pL)
MNC2-1
Ave ccos(wi, w + x-1
MNC2-2
(ee(td(([]ellll |/ lla,,.., w / llwill,,.... llwkll))))
MNC3
Ale =1
MNC4
Ave Arggaargmx k,4
hi.)) * argmin [nt ml(12]]
41


Reviews:
[{"review_id": "7r9lZmNR9j", "reviewer": "Reviewer_MDbD", "summary": "This paper investigates the phenomenon of neural collapse (NC) in multi-task learning (MTL), extending the existing understanding of NC beyond single-task settings. The authors propose two novel MTL-specific NC formulations—SSMTC-NC and MSMTC-NC—and provide both theoretical guarantees under the Unconstrained Feature Model (UFM) and extensive empirical validation across multiple datasets and architectures. They further explore how task correlation influences the geometry of task-specific classifiers and shared feature representations.", "questions": "1.How would the NC phenomena change in highly imbalanced MTL settings, both in terms of class and task distributions?\n\n2.Have the authors considered evaluating the proposed NC metrics in few-shot or cross-domain MTL scenarios to assess generalizability?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper is the first to systematically extend neural collapse to multi-task learning, introducing two well-defined geometric configurations (SSMTC-NC and MSMTC-NC).\n\n2.The authors provide rigorous proofs showing that both SSMTC-NC and MSMTC-NC emerge as global optima under the UFM setting.\n\n3.Experiments are conducted across a wide range of datasets (e.g., Multi-MNIST, CIFAR100-Split, CelebA) and architectures (ResNet, VGG), with additional studies on task weighting strategies and task correlation.", "weaknesses": "1. Most theoretical results assume balanced class and task distributions, which may not hold in practical MTL scenarios.\n\n2. Key figures (e.g., Figure 1, 2) do not include confidence intervals or measures of variability, which could mislead readers about the robustness of results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D3tTrrCmYq", "reviewer": "Reviewer_vAKg", "summary": "This work formalizes two settings, SSMTC and MSMTC, under the unconstrained feature model and prove that any global minimizer exhibits NC geometry extended to multi-task, introducing new NC measurement, SNC1–SNC5 for SSMTC and MNC1–MNC4 for MSMTC, to quantify convergence to the predicted NC geometry. Experiment is verified over multiple backbone and datasets. To my knowledge, the formulations should be new, and the contribution over decorrelation across tasks & correlation-aware alignment is also supposed to be new.", "questions": "- For thm3.1/3.2. All the NC results only sound given nontrivial $\\lambda_{H}\\lambda_{W}$ less than the threshold, but as you derived in line 764, when $\\lambda_{H}\\lambda_{W}$ is bigger than the threshold, the global minimizer is simply a trivial solution, making the NC properties no longer hold. How should I interpret this in terms of actual applicability? I suggest to add several real examples of $N$ and $K$ from your actual dataset to justify to what extent such condition would hold.\n\n- For the whole UFM setup, I am concerned that the actual network you trained is not optimizing the UFM objective you analyzed, which should be a deep shared feature extractor plus per-task heads with CE. The UFM analysis in multi-task form breaks the coupling between tasks, which seems like a toy model.\n\n\nPresentations:\n\n- Please properly use \\citet for better readability, e.g., line 181, line 182-183, and all the citations from Appendix E.\n\n- Please consider to add all eq ref in your proof. Some of them are not added, and it reads somehow inconvenient, e.g., line 954 ref to inequality m. \n\n- Please consider add a table to present all the metrics in section 4.3.\n\n- typo. Line 114, SSTMC-NC", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The overall work is well written. The setup and background sufficiently familiarize readers with the problem.\n- I briefly reviewed the appendix proofs and believe the main claims and high-level theoretical results are sound. Some clarity issues remain; please see my questions.\n- The experiments are extensive across different setups and backbones and are presented under multiple MOO weighting strategies. I believe most of the claims are verified.\n- As noted in the summary, the theoretical contribution is new and the derivations appear solid (I did not spot any major errors). However, the technical novelty is somewhat limited, as the work reads as an extension of single-task neural collapse rather than a new paradigm.\n\n---\n\nOverall, I believe this is a new contribution in theory towards multi-task NC, even though it might be somehow incremental to view from technical theoretical contribution. My current assessment is boardline; I will re-evaluate it based on the rebuttal. Still, please address the concerns in weakness section.", "weaknesses": "- The method assumes balanced labels. I suggest to add discussion of how label imbalance and skewness would affect your modules and final results, through some specific quantitative analysis.\n\n- Following from the previous point, although the balanced-label assumption aligns with prior work, the datasets used are relatively standard and small-to-medium for probing NC geometry. The effect on large, real-world datasets remains unclear and is therefore a limitation. Please consider adding at least one larger split to demonstrate robustness for your readers. For the medium-size experiment in Appendix D.4, report mean & std in the curve across multiple seeds. I also suggest moving the real-dataset experiments on ImageNet and CelebA from the appendix into the main paper, replacing some synthesized-data experiments.\n\n- Several constraints are hard to interpret in terms of realistic, practical impact, and I have concerns about the UFM setup. See Questions for details.\n\n- The clarity of writing and presentation should be improved. See Questions for details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qhbZUtfuIr", "reviewer": "Reviewer_Y9is", "summary": "This paper analyses Neural Collapse (NC) properties in Multi-Task Learning networks with single and\nmultiple sources (SSMTC and MSMTC). In the MSMTC scenario, the learned representations and task-specific classifier weight vectors converge to a unified simplex ETF geometry similar to STL. In the SSMTC scenario, each task's weights converges to a separate simplex ETF, and these are mutually orthogonal across tasks. They reveal some interesting properties concerning the\nlink between the task-specific classifiers as well as the relationship of the shared features and the task-\nspecific classifiers. Furthermore, they provide theoretical guarantees and extensive empirical validation across different network architectures, data sets and training conditions. Overall, this work extends NC analysis beyond the single-task case and offers valuable new, theoretically grounded insights for MTL.", "questions": "Do the observed NC structures also manifest on hold-out data? Can you say something about how NC relates to generalization in this context? In STL, NC has been linked to improved generalization. Any insight into how NC on training data translates to performance or structure on test data would be appreciated.\n\nIn SSMTC, the orthogonality between task-specific classifier subspaces seems elegant. In reality, deep features are typically shared across tasks. Do you view this orthogonality as a theoretical idealization, or do you think partial orthogonality might actually be beneficial in practice (e.g., for reducing task interference)?\n\nDo any of the differences observed between MTL and STL settings suggest concrete benefits of training tasks simultaneously (as done here) versus training the same tasks independently on separate models?\n\nThe theoretical proofs rely on the Unconstrained Feature Model (UFM) and the existence of a global optimum. In practice, deep networks rarely reach global optima. How sensitive are your predicted NC structures to this? For instance, does partial or approximate collapse occur under suboptimal convergence?\n\nYour linear combination view seems to suggest that the shared representation in MTL is fundamentally additive. Do you agree with this?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The manuscript provides a non-trivial theoretical extension of Neural Collapse to multi-task learning. It is proven that any global optimum in both the SSMTC and MSMTC scenarios must satisfy analogous NC properties.\n\nExtensive empirical evaluation is conducted, using multiple architectures, datasets, and multi-task loss weighting strategies. All of these choices are well-grounded in recent literature.\n\nThe findings reveal novel insights into geometric structures in how MTL networks organize their features. These insights deepen our understanding of how tasks interact in a shared model. This is important in a literature dominated by heuristic optimization (weighting) strategies without sound theoretical backdrop. \n\nThe paper shows that the degree of task relatedness can shape the learned feature geometry, which helps in treating tasks as heterogeneous and understanding when and why MTL works.", "weaknesses": "The discussion of practical implications is fairly limited. A better connection to practical MTL challenges could be made. For instance, it is unclear how these collaps properties might help improve training procedures. Linking the insights from this paper to issues such as task imbalance, negative task transfer and MTL architecture design would be very valuable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "myJ3KEfG89", "reviewer": "Reviewer_ge6o", "summary": "The manuscript discusses the phenomenon of Neural Collapse in the setting of multi-task learning. Specifically, it states that in two specific forms of multi-task learning with shared backbone and task-specific classification heads, adopting the unconstrained features model, neural collapse emerges. Besides theoretical results, also experiments on real datasets are provided.", "questions": "I have some question that I hope will help me to understand the work better, in particular its contribution. This list includes the relevant question from the \"weakness\" section above:\n\n1) For $T=1$, am I right that you recover the same NC properties as in single-task learning?\n\n2) For Theorem 3.1, is it possible that tasks are identical? If yes, why wouldn't the features and weight matrices become identical, violating NC3? \n\n3) Am I right that the UFM variant of MSMTC decouples the optimization tasks, so the \"parameter sharing\" aspect of MTL disappears? If yes, what insights into MTL do we still obtain?\n\n4) is there any other (hidden) conditions beyond the bounds on the regularization strength requires for Theorems 3.1 and Theorem 3.2 to hold?\n\n5) how is the setting of Theorem 5.1 not compatible with the assumptions of Theorem 3.1?\n\n6) given the many prior results on NC, how will the community benefit from your result that NC emerges (also) in MTL\n\n7) would you say that your proofs contain any steps of independent interest, i.e. besides their role in proving your theorems?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "+ theoretical insights into multi-task learning are a valuable addition to the neural collapse literature\n+ the extensive experiments do a good job of confirming that NC does emerge in practice", "weaknesses": "While generally of high quality, I find the work to have some shortcomings in the presentation of the work and the strength of the scientific contribution.\n\n* Presentation\n\nFor a predominantly theoretical paper, I found the presentation in several parts insufficiently precise or clear. For example, the main concepts SSMTC-NC (Section 3.1) and MSMTC-NC (Section 3.2) are introduces as the phenomenon that some quantities converge to some constants (often 0) in final phase of network training. However, the associated theorem state that any *global optimizer* of the training objectives *satisfies* SSMTC-NC (Theorem 3.1) and MSMTC-NC (Theorem 3.2), so there is no aspect of optimization or convergence. \nThe theorems are phrased without the constraints on the regularization strength that are usually required to prevent trivial solutions. The proofs (lines 764 and 943, respectively) correctly contain the statements that too strong regularization will lead to trivial instead of collapsed solutions.\n\nThe description of (theoretical) results would have benefited from giving more insights how the results are relevant and related, e.g. by discussing at least some border cases. \n1) For $T=1$, all results should reduce to the known ones for (balanced, ...) NC with cross-entropy loss. This seems to be the case, but it would be good to explicitly state it.  \n2) For SSMTC with $T>1$, nothing seems to prevent that multiple tasks are *identical*. Take, e.g., $y_i^1=y_i^2=\\cdots=y_i^T$ for all $i\\in[N]$. In that case, the multi-task problem would reduce to identical copies of a single-task problem. $W^1=W^2=\\cdots=W^T$ should be a globally optimal solution, and $H$ only needs to be of rank $K$ (which minimizes regularization cost). However this seems incompatible with NC3? The answer might be probably related to Theorem 5.1, where NC3 becomes Correlated-NC3, but the relation of assumptions is not clear to me. Potentially, more answers could be found in the proofs in the appendix, but given their repetitive nature, the manuscript's heavy notation and the too short time that ICLR provides for reviewing this year, I was not able to fully parse the proofs.\n3) For MSMTC with $T>1$, each tasks has its own data and only the backbone is shared. But in the UFM, each task has its own  unconstrained features, which remove the central element of \"hard parameter sharing\" MTL. With independent features, the problem seems to become completely decoupled into independent classification tasks. Since we know that individual UFMs exhibit NC, what is there left to prove? And why would the result still be interesting?\n\n\n* Scientific contribution\n\nThe submissions makes some conceptual, theoretical and experimental contributions. I do see that the conceptual contribution of establishing NC for MTL exist, but I do not find it particular strong or surprising, given the many settings in which final-layer NC has already been proven. The manuscript would benefit from a stronger discussion what benefit the community has from knowing that NC holds in the MTL-UFM setup. The theoretical contribution lies mainly in generalized definitions of NC, and the proofs of the main theorems. The latter mostly follow standard steps from single-task cross-entropy NC. There are new aspects, but e.g. I didn't notice any step in the proofs that I would expect to have broader impact outside of the scope of the work. The experiments are extensive and appear well done. I would encourage the authors to release their source code to allow for reproducibility. The results confirm collapse behavior for real data in many settings, but (as in previous studies) the difference between numeric training of real networks and properties of the global-optimizer in a UFM is quite big. Presumably, the solutions found numerically are not actual global minimizers, but how do they related? Some interpolating experiments could help to reduce this gap (e.g. numeric optimization of the UFM). \n\n\n* Minor comments:\n- reference (Yu et al, NeurIPS, 2020) is duplicate\n- notation such as \"5e-3\" should better be $5\\cdot 10^{-3}$\n- line 724: \"if we minus a vector/scalar\" should be \"if we subtract a vector/scalar\"\n- I found the line of argumentation in the proofs not always ideal to follow. You provide a number of inequalities, and extract conditions saying e.g. \"The conditions under which equality holds are as follows:\" (line 767). Usually, this would be a way to establish *sufficient* conditions, but for your results of the type \"all global minimizers have a certain property\", you need also *necessary* conditions. These follow e.g. from the \"only if\" part of Lemma A.2 (line 701), but such subtleties can easily get lost.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)