Some Neural Networks Inherently Preserve Subspace Clustering Structure
Authors: Karan Vikyath Veeranna Rupashree
Siddharth Baskar
Daniel L. Pimentel-Alarcón
Venue: ICLR 2026 Poster (2026)
Keywords: Clustering
Subspace
Neural Networks
Activation Functions
Preserving Structure
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: It has long been conjectured and empirically observed that neural networks tend to preserve clustering structure. This paper formalizes this conjecture. Specifically, we establish precise conditions for cluster structure preservation and derive bounds to quantify its extent. Through this analysis we are able to show that certain neural networks are learning parameters that preserve the clustering structure of the original data in their embeddings, without the need to impose mechanisms to promote this behavior. Extensive numerical analysis and experiments validate our results. Our findings offer deeper insight into neural network behavior, explaining why certain data types (such as images, audio, and text) benefit more from deep learning. Beyond theory, our findings guide better initialization, feature encoding, and regularization strategies.

Published as a conference paper at ICLR 2026
SOME NEURAL NETWORKS INHERENTLY PRESERVE SUBSPACE CLUSTERING STRUCTURE
Karan Vikyath Veeranna Rupashree", & Siddharth Baskar* Department of Electrical and Computer Engineering University of Sieconsin-Maddsse Madison, Wisconsin, USA (vee (veerannarupa, sbaskar2 @wi edu
Daniel L. Pimentel-Alarcón
Department of Biostatistics and Medical Informatics University of Wisconsin-Madison Madison, Wisconsin, United States
p pimentelen aarlwrriiss edu
ABSTRACT
It has long been conjectured and empirically observed that neural networks tend to preserve clustering structure. This paper formalizes this conjecture. Specifically, we establish precise conditions for cluster structure preservation and derive bounds quantify its extent. Through this analysis we are able show that certain neural networks are learning parameters that preserve the clustering structure of the original data in their embeddings, without the need to impose mechanisms promote this behavior. Extensive numerical analysis and experiments validate our results. Our findings offer deeper insight into neural network behavior, explaining why certain data types (such images, audio, and text) benefit more from deep learning. Beyond theory, our findings guide hetter initialization, feature encoding, and regularization strategies.
INTRODUCTION
At this point neural networks (NNs) need no introduction. They are the backbone modern artificial intelligence (AI), revolutionizing industries and reshaping the modern world. Their applications span diverse fields: healthcare, NNs enhance medical imaging, predictive analytics (Almarzouqi et al., 2022; Khang et al., 2024), and drug discovery, accelerating the identification of new therapeutics (Wong et al., 2024). In finance, they play a crucial role in fraud detection and algorithmic trading (Wang et al., 2021; Luo et al., 2024). Beyond these domains, NNs are driving scientific innovation, predicting protein structures (Zhou et al., 2024) and improving climate modeling (Ghimire et al., 2022). Convolutional neural networks (CNNs) revolutionized image processing (Chauhan al., 2018), recurrent neural networks (RNNs) reinvented sequential data processing (Al-Selwi et al., 2024), and attention mechanisms (Vaswani, 2017) transformed natural language processing, enabling advancements in machine translation (Wang et al., 2022), sentiment analysis (Sachin et al., 2020), and conversational AI (Saka et al., 2023). Meanwhile, generative adversarial networks (GANs) and diffusion models are redefining art and content creation (Goodfellow et al., 2020; Croitoru et al., 2023).
Despite the widespread use of neural networks, their inner workings, particularly their efficiency in clustering and predictive tasks, remain perplexing. Various studies have highlighted these challenges. For instance, it is unclear how neural networks can achieve high performance even when trained on randomized labels (Song et al., 2022), or how phenomena like double descent allows model performance improve with complexity beyond the point of overfitting (Belkin et al., 2019), or how weight initialization affects stochastic gradient descent (Narkhede et al., 2022; Bishop & Bishop, 2023).
Among these challenges, one major obstacle in deep learning theory is a detailed understanding of the complexities introduced by high-dimensional and non-linear transformations, which obscure how these models arrive at their decisions (Doshi-Velle & Kim, 2017: Lipton, 2018). These enigmatic challenges fundamentally rooted in activation functions like the sigmoid, the hyperbolic tangent, or the Rectified Linear Unit (ReLU) (Hara et al., 2015). These functions introduce non- non-lincarrty to NNs,
"Equal Contribution

---

Published as a conference paper at ICLR 2026
required for hierarchical feature learning and to represent highly complex predictive functions. The downside is that activation functions also impact loss functions, affecting optimization dynamics and complicating theoretical analysis. Recent studies have explored the expressivity and approximation properties of ReLUUs to provide new insights on their effect on neural network performance (Kou al., 2024). However, clear understanding of the fundamental role of activation functions and their effect on the behavior of NNs remains clusive.
This paper describes specific conditions under which certain activation functions preserve subspace clustering structure. More precisely, we characterize conditions to guarantee that if the input to a layer with a valid activation function has subspace cluster structure. the output will retain that same cluster structure in the output's ' embedding. One example of such activation function is the rectified linear unit (ReLU). This behavior has been long conjectured and empirically observed in many datasets (Papyan, 2020; Arora et al., 2018). Besides formalizing this conjecture, our analysis gives a deeper understanding of the underlying reasons for this behavior, providing insights into initialization parameters that promote cluster preservation. Our analysis also explains why NNs perform better for certain types of data, such as imagery, audio (Nanni et al., 2021), text data for natural language processing (Min et al., 2023), bioinformatics data (Karim et al., 2021), and financial and anomaly detection (Zamanzadeh Darban et al., 2024). It turns out that these types data inherently preserve subspace clustering structure under certain transformations.
The proof brings together ideas from statistical learning, principal component analysis (PCA), subspace clustering (SC), and perturbation theory, and it is divided in three main parts: (i) first show that the closed-form solution to a subspace clustering model (of which Euclidean clustering special case) can be accurately inferred from noisy data. (ii) Then we show that this solution is invariant to arbitrary linear transformations. (iii) Lastly, we show that under certain conditions on the network parameters and the activation function, such solution is robust to the corresponding transformation. We establish precise conditions for cluster structure preservation and derive bounds to quantify its extent, leveraging the Davis-Kahan (in(O) theorem. Numerical results confirm these bounds, and experiments further validate that ReLUS and several other related activation functions inherently preserve clustering structure. While our theoretical guarantees apply to single layer, they extend directly to multiple layers, as supported by our empirical findings.
Ultimately, this paper brings to the table new insights and a deeper understanding into the inner workings of neural networks and how they learn. Specifically, the main takeaway from findings is that neural networks that use ReLUs and similar activation functions seem to be learning to cluster in closed form.
2 RELATED WORK
The theoretical understanding of neural networks, due to their high-dimensional and non-line non-linear struc- tures, still remain an ongoing challenge. While the deep learning models achieve high performance, their interpretability and underlying mechanism are still not fully understood. Methods like feature attribution (Lundberg, 2017; Molnar, 2020) and mechanistic interpretability (Bereska & Gavves, 2024) have attempted to explain how neural networks process information. However, these methods often fail to capture the full structural properties.
The ability of deep networks achieve high performance even when trained on randomized labels (Asnicar et al., 2024) further complicates the understanding of learned representations. One con- tributing factor overparameterizztionn, where large networks manage to preserve feature structures despite having more parameters than necessary (Elhage et al., 2021). Additionally, the implicit biases introduced by optimizers such as stochastic gradient descent (SGD) have been linked to the preservation of clustering structures, suggesting that optimization dynamics play crucial role in shaping learned representations (Soudry et al., 2018).
Double descent is phenomenon (Belkin al., 2019; Nakkiran et al., 2021). in which performance initially deteriorates as the complexity of the model increases but improves again after exceeding the overfit threshold. Recent studies indicate that double descent may be linked to how activation functions structure the feature space, hence preserving the cluster boundaries (Advani et al., 2020; Zhang al., 2021).
2

---

Published as a conference paper at ICLR 2026
Additionally, network initialization and optimization strategies affect the extent to which clustering structures are preserved. Techniques such as Xavier and Kaiming initialization (Pan et al., 2022) help maintain stable gradients, indirectly influencing feature separability. Recent work on flat minima and generalization (Ding et al., 2024) has shown that flatter loss landscapes are associated with hetter-preserved clustering structures, property that may be influenced by activation functions like ReLU, Another work related to activation functiins highlights the importance of non-lincarities in preserving structures such as clusters. This also states that ReLU preserves topological features in latent spaces, enhancing the clustering of data points in learned representations (Xu, 2015).
3
MAIN RESULTS
Consider data matrix X* E R with columns given by
K
Σ
1l (iEn))
Un
k=1
where denotes the indicator function, {Rxlk= partition {l...., n3 indicating the clustering of the columns among K subspaces with bases U E Rmxx,, and v; E IR is the vector of coefficients of x with respect to the basis UK- These data have subspace cluster structure where each sample lies in one of K low-dimensional subspaces. This model is often known union of subspaces (UoS) model (Lipor & Balzano, 2017), generalized PCA (Vidal et al., 2005) or subspace clustering (Elhamifar & Vidal, 2013). Suppose our observed data is
= X +
where Z Rmxa can be interpreted as noise matrix, that the columns in X lie near the UoS, rather than exactly on Notice that Euclidean clustering is the special case (1) with 1-dimensional subspaces and constant coefficients (i.e., Uk R often denoted as f4, and v: Similarly, orthogonal nonnegative matrix completion (ONMF) (Ding et al., 2006), also used for its clustering capabilities (Pompili al., 2014), is the special case of (1) with K 1-dimensional orthogonal subspaces and U 2 0 for every k and v; 2 O for every i.
for every i).
Our main result, summarized in Theorem 3.1 below, specifies sufficient conditions under which certain layers preserve the subspace cluster structure described above. More formally, let a--) == max(0,, -) denote the layer's activation function, and let W denote the parameters of the layer, so that = o (X) is the output of the layer when X is fed (we are omitting the bias term b. which can be incorporated as a column of W by adding a constant row of ones in X). Given matrix, let P--) denote the projection operator onto its principal row-space of dimension := Σ 1 Ik. We will show that P encodes the subspace cluster structure above. The next theorem shows that under reasonable conditions, the projection matrices P P(X*) and P(Y) are close enough to one another and hence they encode the same clustering structure.
Theorem 3.1. Let 6(X", X) denote the gap between the th singular value of X* and the 11th singular value of X, and similarly for 5(WX*, WX) and 6(Y, WX). Suppose the smallest of these quantities, 8, satisfies:
128r
8
max Z[] [Z | -X =
(2)
Then
P( P(Y) < c/2.
Furthermore, x[,x E spann Ux) if and only if the (i,j)th th entry of is larger than e/2.
The proof of Theorem 3.1 is the next Section, and it follows by the Davis Davis-Kahan sin(@) Theorem (Davis & Kahan, 1970; Stewart & Sun, 1990). Intuitively, can be interpreted as the similarity between the principal subspaces of its arguments. Under this light, Theorem 3.1 requires (i) that X is
3

---

Published as a conference paper at ICLR 2026
PLY) P(Y)
Figure I: Projection matrices revealing the clustering structure of the data, which is preserved through every step of ReLU transformation. Lemma 4.1 shows that the projections of X and are close. Lemma 4.2 shows that the projections of X and WX are close. Lemma 4.3 shows that the projections of WX and Y are close when W satisfies certain conditions that the network encourages through the learning process. In contrast, random W's destroy the clustering structure.
close enough to X', which depends on the noise level in Z, (ii) that X and X* behave similarly under the linear transformation W, which depends on W and Z, and (iii) that the output Y of the activation function a is close enough to input WX, which depends a and W, which
turn determined by the learning process on the network. Notice that the condition 8 > O implicitly requires that n > r (otherwise there is not even an (I + 1)sh singular value). More than a requirement, this is fundamental identifiability condition, because it takes rk linearly independent vectors to identify Σ I' vectors identify union of K subspaces with dimensions .i..... IK- The spectral-gay gap assumption in 3.1 is introduced to establish sufficient conditions under which subspace clustering is guaranteed be preserved. While such gap may not hold uniformly across all real-world datasets, it provides a clean and interpretable theoretical mechanism linking perturbation stability to clustering structure.
an rx- rx-dimensional subspace, and it takes
There are four key players in Theorem 3.1: the data and the noise, which are beyond our control, the activation function a, which we are free to choose, and the parameters W, which are learned by the network. What makes this paper particularly interesting is that it is easy to construct matrices W that violate the conditions of Theorem 3.1 for any a. Trivial examples include matrices whose product with X yields many negative values be replaced with zeros in a ReLU transformation, destroying the subspace structure in the original data (see Figure 1). Naturally, one could use regularization techniques encourage weights W that satisfy the conditions of Theorem 3.1. This could involve, for example, for ReLU activations, adding a penalty term that favors nonnegative weights W or nonnegative products WX. The surprising part, as we will demonstrate below, is that neural networks tend to learn weights that inherently preserve the clustering structure even in the absence of explicit mechanisms enforcing this behavior.
4 PROOF
The proof of Theorem 3.1 is presented in three parts, showcased in Figure 1. First, we show that the clustering structure of X* can be estimated in closed-form. Then we establish that such estimator is invariant under arbitrary linear transformations. Finally, we demonstrate that for certain matrices W, the non-linear transformation induced by certain activation functions do not disrupt the clustering structure of our dlosed-fom estimator.
4.1 ESTIMATING THE GENERAL CLUSTERING STRUCTURE
The key observation that the features of X lie subspace whose projection operator encodes the clustering, which in turn will determine (Uk ji= and {vi)n (up to a basis rotation). To see this, define nę- := [&a as the number of columns in X' corresponding to the kt subspace, and let X; be the m X ng matrix containing such columns. For our analysis, assume without loss of generality that X* = [X
Xi] (otherwise simply multiply by permutation matrix on the right). Next let U == [U..... Uk] be the m x r matrix containing the concatenation of the bases { In addition, define Va the Ię nR matrix whose columns the coefficients of X with respect to Un, i.e. (vihien. Finally, let be the r x n Block-diagonal matrix whose diagonal blocks are ( Then V has the following group-sparse structure, where the white areas represent zeros:
4

---

Published as a conference paper at ICLR 2026
V.
V.
Vx
However, directly recovering V is challenging, since subspaces admit multiple equivalent bases. To obtain a unique representation of the clustering structure, we define the projection operator onto the space of the data
This way, we can rewrite (1) as X = UV + Z. Notice that encodes the clustering structure, the kth block, then x belongs the k cluster.
because if the ith column of V is Unfortunately, learning the specific basis V can be difficult, even if Z O, because subspaces have infinitely many bases, most of which do have the group-sparse structure of V. However, we can still estimate the clustering structure in V through the projection operator onto its row space. To see this, let (Vk/k= be orthonormal bases with the same spans {Vk] and use (V*)k= instead to construct V. That is, V is the
of [VVK [Vk]==] k=1
block-diagonal matrix whose diagonal blocks group-sparse structure V. Since the row-blocks of V are disjoint, follows by construction that V is orthonormal and spans the same subspace V. Recall that given a matrix, (-)) denotes the projection operator onto its principal r-dimensional row-space. Since projection operators are unique, long <
(Vx) k= This way, V has the
P(X) = P(V) V
In words, the condition requires that the sum of the dimensions of the K subspaces is smaller than the total number of samples, which is an information theooretc requirement for clustering (Pimentel-Alarcon & Nowak, 2016). From (3) we can see that the columns and rows of P(X*) have the exact same support as the rows in V (and V), and have the following general structure:
P(X)
We thus see that learning P(X") is just as effective as learning V or V in the sense that it encodes the clustering of X" because if the (i, entry of PXX") is nonzero, then the ith and jth columns of X* correspond to the same cluster. Therefore, we can recover the support of P(X*), we obtain the clustering of X as desired. Fortunately, since projection operators are unique, P(X*) ) can be directly estimated as P(X). The following lemma states that under general conditions on the noise Z, the difference between P P(X*) and P(X) is bounded.
Lemma 4.1. Let 8, > O be the gap between the th singular value of X* and the (r + 1)th singular value of X. Suppose
81 >
V 2Tr ZI/e =: m,
(4)
where denotes the smallest absolute value the support of P(X). Then
P P(X e/8.
(5)
Recall that 61 can be interpreted the similarity between the subspaces spanned by X and X. The condition (4) essentially requires that the noise Z is not too large relative to the variance X*, that the clusters are discernible from estimator P(X), and sample is misclustered.

---

Published as a conference paper at ICLR 2026
Proof. We will show that corresponding entries in P(X") and P(X) cannot differ by more than e/8. To see this, write
P(
[P(X) P( >
P( [ )1 2tr((())
2r 2tx(P(X) P(X') - 2(r P(()
=: 2(r - cos ( ) 2 sin v 2 Z /
where the last inequality follows directly by the Davis- sin( ()) Theorem (Davis & Kahan, 1970; Stewart & Sun, 1990). Then
Z
2r Z
P( - P( S
Substituting ó1 from (4), we see that
22r |2
22
P((X) P(x I
51
22'r||Z|
8
4.2 INVARIANCE TO LINEAR TRANSFORMATIONS
We will now show that linear transformations preserve clustering structure. More precisely, we will show that under reasonable conditions, the projection matrices of X and WX are sufficiently close to one another and so they share the same clustering structure. This is summarized in the following lemma:
Lemma 4.2. Suppose the gap 82 > 0 between the 1th singular value of WX* and the (r + 1)th singular value of W. satisfies ón /2/// Then [P(X) P(WX)] </4
Proof. Start with two triangle inequalities:
[P(X) P f PP PX [P(X))
P (IP(X))))((XX)) P(WX)) P P(X*) [| - P(WX)]|
(6)
where the last equality follows because as long as rank (W) r (implicitly required by the theorem), then P(X*) = P(WX*) = v Using the exact same arguments as in the proof of Lemma 4.1, can bound
2r
WZ
2r
WZ
P(WX*) - P(WX) <
82
WZ
8
Plugging this and (5) in (6) we obtain the lemma.
4.3 INVARIANCE TO CERTAIN ACTIVATION FUNCTIONS
Finally, we specify that under certain conditions on a and W, the clustering structure of WX the same as that of Y.
Lemma 4.3. Let 83 > 0 be the gap between the t singular value of Y and the (r + 1)th singular value of W.. Suppose dg / WX|]/e. Then [P((Y) P( & e/8.
The proof Lemma 4.3 follows by the same arguments in Lemma 4.1. The proof of Theorem 3.1 follows directly by Lemmas 4.1, 4.2, and 4.3, and three triangle inequalities.

---

Published as a conference paper at ICLR 2026
14
0.9
14
[IPX*) PMVII [IPX*)) PIWK)
12
03
Prodedoon Distacce
0.
[PUX] - PTN MPO
9.6
[POX] PYNJI-
24
0.4
00
10 20 30
30
10
30
40
Epach
Epoch
Figure 2: Left: Probability with which the conditions in Theorem 3.1 hold, guaranteeing that subspace structure is preserved through a ReLU transformation. Center: Projection distances as function of epochs with noise s² = 0.1, showing that the network is inherently learning weights that preserve clustering structure. Right: Projection distances for two different initializations. The initialization informed by our analysis (W >> 0) perfectly preserves clustering structure and is a local optima of the learning process (its gradient is numerically zero and its hessian determinant is positive).
5 INHERENT PRESERVATION OF SUBSPACE CLUSTERING STRUCTURE
According Theorem 3.1, several conditions must align that the cluster structure in X* is preserved in Y. First, the subspace signal in X must overcome the noise Z, as required by the bound on 6(X", X). Next, the transformation W must blow up the noise Z, required by the bound on 8(WX" WX). Finally, the activation function must not disrupt the cluster structure in WX, as required by the bound on 6(Y, WX). The first condition (on the data and the noise) entirely out of our control, and must be assumed. On the other hand, given the data, the second and third conditions depend entirely on the choice of a and W, which as Figure shows, may preserve the subspace structure in X destroy it entirely.
Characterizing when the conditions in Theorem 3.1 will hold can be difficult in practice, as X" is generally unknown, and W is the parameter to be learned by the training process (and our assumptions depend on both these quantities). As discussed earlier, we could regularize our objective function to encourage weights W that meet the conditions of Theorem 3.1. This could be achieved by penalizing negative entries in W or WX. The surprising part is that this seems to be unnecessary, as certain neural networks appear favor such solutions inherently.
To see this we present a numerical experiment to quantify the frequency with which our conditions hold synthetic data following the subspace clustering model. Specifically, we generated K rk- dimensional subspaces of IR"", each spanned by a matrix U2 E R with i.i.d. entries drawn from the standard gaussian distribution (O, 1), which we subsequently orthogonalized. We similarly populated Vx E Rrixx with i.i.d. N(0 1 //m) entries and constructed X VR + Zu and X X .......... X Then we populated Z Rmmx with (... s²) entries. Here s² represents the noise variance. Finally, we initialized W with i.i.d. uniform entries in the range (-m m²))). and proceeded to learn the parameters W of a single hidden feedforward layer autoencoder with 80 ReLU neurons using standard gradient descent with learning rate of 0.005. We trained this autoencoder using squared Frobenius reconstruction loss |X | For the dataset, we used n. = 4, m = 400, ngg 100, and K = 4, so that = 400. Then we proceeded to calculate 8 and function of the noise variance s², which is a proxy of |Z] in (2), and recorded the frequency with which our assumptions hold (i.c., when 5 > 17). All experiments were conducted on computer with an AMD Ryzen? 7 5800H CPU, 16 GB RAM, and an NVIDIA GTX 1660 Ti GPU (6 GB). Figure 2-Left summarizes the results of 100 independent trials for each value of s The results show that our bound degrades nicely with noise.
As Figure 1 shows, random weights will generally destroy the subspace clustering structure in Y. Figure 2-Center shows that the network is inherently learning weights W that satisfy the conditions of our theorem, thus preserving the cluster structure in Y. The surprising part is that it is doing so without any mechanism explicitly enforcing this behavior.
Practical insights. Beyond theory, our findings have practical gains. Notice that the conditions of Theorem 3.1 are generally met for ReL.S whenever WX > O, because then a (WX) -W. This
7

---

Published as a conference paper at ICLR 2026
Figure 3: Evolution of clustering structure over epochs in a 5-layer feedforward ReLU network. Left: Projection distance between the input X and linear transformation W at each layer. Center: Projection distance between the input X and the ReLU output Y at each layer. These distances can be seen loss of clustering structure. Their decrease over epochs indicate that the network is learning weights that preserve such structure. Right: Clustering accuracy at each layer over epochs, obtained from the projection operator as described above. Summary of 100 independent trials.
is trivially true whenever X > O and W > O. In many cases, X is nonnegative due to the nature of many modern datasets. Examples arise in network inference (Eriksson al., 2011), single-cell sequencing (Ding et al., 2006), drug discovery (Zhang et al., 2019), multi-omics (Chevrette & Currie, 2019), medical image processing (Riaz et al., 2020), and more (Huo et al., 2021). For example, hop counts in networked systems are nonnegative, pixel intensities are nonnegative, many biomedical features like age, heart rate, blood pressure, body temperature, glucose, cholcsterol, oxygen saturation, and enzime levels, white blood cell count, respiratory rate, etc., nonnegative. On the other hand, W may be initialized with nonnegative values ensuring that the clustering structure in the data is automatically preserved the beginning of the training process. This simple strategy leads to practical gains in terms of accuracy and length of the learning process. To see this we repeated the same experiments in Figure 2-Center, except that we increased the noise to 0.5, and initialized W with i.i.d. entries in the unit interval. The results are in Figure 2-Right, where we can see that this initialization perfectly preserves clustering structure and a local optima of the learning process that improves over other initializations.
6 BEYOND A SINGLE HIDDEN LAYER
Our analysis for single layer be directly extended multiple layers by applying a union bound on Theorem 3.1. Let X W and Y' denote the input, weights, and output of network at the eth layer. The subspace clustering structure will be preserved at the gth layer if P(Y²) close enough P(X"). This will be the case the network learns parameters W ..... w' that simultaneously satisfy the conditions of the mnion-bounded version of Theorem 3.1 (with factored by E). Most surprisingly, deep networks display the same behavior observed in their shallow counterparts, and inherently preserve the subspace clustering structure in the data through multiple layers, in the absence of explicit mechanisms enforcing this behavior.
To demonstrate this, replicate the exact same experiment as in Figure 2-Center, except that this time we used a network with ReI.U layers. Figure 3 summarizes the results of the training process, showing that the network starts with random weights that violate the assumptions of Theorem 3. destroying the clustering structure, As part of its training process learns parameters that preserve the clustering structure encoded in the projection matrices.
7 BEYOND RELU ACTIVATIONS
So far we have supported our conclusions with ReLU activations. However, there nothing specific in analysis that binds us to such activations. To demonstrate this present series of experiments with other activation functions and other architectures. Figure 4 shows the results for the same architecture in Figure except with for GELU (Hendrycks & Gimpel, 2023) and SiLU (Elfwing et al., 2017) activation functions, as well as a 3-layer LSTM architecture (Staudemeyer & Morris, 2019) with ReLU activations. In these experiments generated data with the same procedure
8

---

Published as a conference paper at ICLR 2026
PAU
140
17%
1.50
mg
13
005
Figure 4: Evolution of clustering structure over epochs in 5-layers feedforward networks using GELU (left) and SiLU (center) activation functions, and a 3-layer LSTM architecture with ReLU activations (right). The decrease in the projection distances over epochs indicates that the networks learning weights that preserve the clustering structure. This shows that clustering preservation is not an exclusive property of ReLUs or feed forward layers. Summary of 100 independent trials.
described in Figure 2-Center. This shows that clustering preservation is not an exclusive property of ReLUS or feedforward layers.
There are, of course, limitations to Theorem 1. It is easy to see that Theorem 3.1 cannot be easily extended to more sophisticated transformations that do not rely activation function. These complex transformations are more likely to disrupt the clustering structure of the data. Examples include the convolution operator (O' Shea & Nash, 2015) the Transformer (Vaswani et al., 2023). To demonstrate this we repeat the same experiments as in Figure 4, except using 3-layer CNN and 4-layer Transformer with different activation functions. The results are in Figure 5, showing that unlike linear operators, the convolution or attention transformations do not inherently preserve clustering structure.
linear transformation and an
Remark. We point that Transformers and CNNs are not failing at clustering. They still clustering accurately, same as all other networks. In fact, all models were intentionally selected because of their high clustering accuracy. We wanted models that cluster correctly because the goal of these experiments (and the paper) is not to analyze accuracy establish a new state-of-the--e Rather, we seek to investigate the behavior of deep networks, and understand the mechanisms they are using for clustering. We are focusing projection distances because, beyond accuracy, they reveal that ReL.Uttppe networks are clustering by learning the closed-for form- solution. The large projection distances exhibited by Transformers and CNNs show that these networks are clustering through some mechanism other than the closed-form solution, and such mechanism does not preserve the original clustering structure.
8 REAL DATA
We validate our conclusions across 12 diverse real-world datasets, including MNIST (Deng, 2012), CIFAR-10 (Krizhevsky, 2009), Extended MNIST (Cohen et al., 2017). Street View House Numbers (SVHN) (Netzer al., 201 1), KuzushijiMNNIST (Clanuwat et al., 2018), Fashion MNIST (Xiao et al., 2017), Flowers- 102 (Nilsback & Zisserman, 2008), Food-101 (Bossard et al., 2014), USPS Handwritten Digits (Hull, 1994). STL-10 (Coates al., 201 1). Oxford DTD dataset (Cimpoi et al., 2014), Oxford Pet dataset, and FuroSAT dataset (Helber et al., 2019) for the 5-layer network. The results and more details are presented in Figures projection distances reported throughout the paper not intended as clustering performance metrics. Rather, they measure the preservation of subspace structure during the learning process.
9 in the Appendix. We emphasize that the
In all experiments considered, the models achieve perfect or near-perfect clustering accuracy under standard metrics such as ACC, NMI, and ARI. These models were intentionally selected because goal is not compare clustering performance, but to analyze how successful models achieve clustering. Our findings suggest that ReLU-type networks cluster by implicitly learning the closed- form projection structure described in Section 4.

---

Published as a conference paper at ICLR 2026
CNN
Transformer
Figure 5: Evolution of projection distances epochs with different activation functions. Top: 3-layers CNNs. Bottom:
Transformers. These distances can be interpreted as a loss of clustering structure, showing that unlike linear operators, the convolution or Transformer do not inherently preserve clustering structure. Summary of 100 independent trials.
ACKNOWLEDGMENTS
This work was partially supported by NSF's CAREER Award #2239479.
REFERENCES
Madhu S Advani, Andrew M Saxe, and Haim Sompolinsky. High-dimensional dynamics of general- ization error in neural networks. Neural Networks, 132:428-4444 2020.
Safwan Mahmood Al-Selwi, Mohd Fadzil Hassan, Said Jadid Abdulkadir, Amgad Muneer, Ebrahim Hamid Sumiea, Alawi Alqushaibi, and Mohammed Gamal Ragab. Ron-Istm: From applications to modeling techniques and beyond -ystemmttcc review. Journal of King Saud Unniversity-Coppue and Information Sciences, pp. 102068, 2024.
Amina Almarzouqi. Ahmad Aburayya, and Said Salloum. Determinants predicting the electronic medical record adoption in healthcare: A sem-artinn artificial neural network approach. PloS one. 17(8): c0272735, 2022.
Raman Arora, Amitabh Basu, Poorya Mianjy, and Anirbit Mukherjee. Understanding deep neural networks with rectified linear units, 2018. URL htttss://arxtti wwt//org////// abss/bllll.. 01491.
Francesco Asnicar, Andrew Maltez Thomas, Andrea Passerini, Levi Waldron, and Nicola Segata. Machine learning for microbiologists. Nature Reviews Microbiology, 22(4):191-205, 2024.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- -lcaniing practice and the classical variance trade-off. Proceedings of the National Academy of Sciences, 116(32):15849-15554 2019.
Leonard Bereska and Efstratios Gavves. Mechanistic interpretability for ai safety review, arXiv preprint 1XXv:2444.1408/ 2024.
Christopher M Bishop and Hugh Bishop. Deep learning: Foundations and concepts. Springer Nature, 2023.
Lukas Bossard, Matthicu Guillaumin, and Luc Van Gool. Food-101 mining discriminative compo- nents with random forests. In European Conference on Computer Vision, 2014.
Rahul Chauhan, Kamal Kumar Ghanshala, and RC Joshi. Convolutional neural network (cnn) for image detection and recognition. 2018 first international conference on secure cyber computing and communication (ICSCCC), pp. 278-282 IEEE, 2018
10

---

Published as a conference paper at ICLR 2026
Mare G Chevrette and Cameron R Currie. Emerging evolutionary paradigms in antibiotic discovery. Journal of industrial microbiology & biotechnology, 46(3-4):257-271,22 2019.
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2014.
Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David
Ha. Deep learning for classical japanese literature, 2018.
Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Geoffrey Gordon, David Dunson, and Miroslav Dudík (eds.), Proceedings of the Fourteenth International Conference Artificial Intelligence and Statistics. volume 15 of Proceedings of Machine Learning Research, pp. 215-223, Fort Lauderdale, FL, USA, 11-13 Apr 2011. PMLR. URL https: /ttp::/pprooeeoo pttpppeieeeegg..epeeoepp.p ml press wow/toatessaisooooooaooaasaeeeaa 1a html.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. EMNIST: extension of MNIST to handwritten letters. CoRR, abs/1702.05333, 2017. http:/t/://axx.v
abs /100. 05373.
Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor lonescu, and Mubarak Shah. Diffusion models in vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9): 10850-10869, 2023.
Chandler Davis and William Morton Kahan. The rotation of eigenvectors by a perturbation. iii SIAM Journal on Numerical Analysis, 7(1):1-46, 1970.
Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6): 141-142, 2012.
Chris Ding, Tao Li, Wei Peng, and Haesun Park. Orthogonal nonnegative matrix t-factorizations for clustering. In Proceedings of the 12th ACM SIGKDD international conference Knowledge discovery and data mining, pp. 126-135, 2006.
Lijun Ding, Dmitriy Drusvyatskiy, Maryam Fazel, and Zaid Harchaoui. Flat minima generalize for low-rank matrix recovery. Information and Inference: Journal of the IMA, 13(2)iaae0009, 2024.
Finale Doshi Delei-Velez and Been Kim. Towards rigorous science of interpretable machine learning. 08600, 2017.
arXiv preprint
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weighte linear units for neural network function approximation in reinforcement learning, 2017. URL https://aaaaaa http:///wwww.oo///..////ooo abbs/ 1702.03118.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1):12, 2021.
Ehsan Elhamifar and Rene Vidal. Sparse subspace clustering: Algorithm, theory, and applications. IEEE transactions on pattern analysis and machine intelligence, 35(11):2765 2781, 2013.
Brian Eriksson, Paul Barford, Joel Sommers, and Robert Nowak. Domainimpute: Inferring unseen components in the internet. In INFOCOM, 201 Proceedings IEEE, pp. 171-175. IEEE, 2011.
Sujan Ghimire, Ravinesh C Deo, David Casillas-Pérez and Sancho Salcedo-Sama Boosting solar radiation predictions with global climate models, observational predictors and hybrid deep-machine learning algorithms. Applied Energy, 316:1 19063, 2022.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139-144 2020.
Kazuyuki Hara, Daisuke Saito, and Hayaru Shouno. Analysis of function of rectified linear unit used in deep learning. In 2015 international joint conference on neural networks (IJCN)) pp. 1-8. IEEE, 2015.
11

---

Published as a conference paper at ICLR 2026
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification, 2019. URL https:
//arxivv....///////// org/ httpa//g//////ooeeee./ee/eeaeeea abs/1705 1709. 00029.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus), 2023. URL https:
//arxixx org/ als /006. 08415.
J. Hull. database for handwritten text recognition research. IEEE Transactions on Putlern Analysis and Machine Intelligence, 16(5):550-555 5544, 1994. doi: 10.13,994.,21444.
Zepeng Huo, Lida Zhang, Rohan Khera, Shuai Huang, Xiaoning Qian, Zhangyang Wang, and Bobak J Mortazavi. Sparse gated mixture-of-expers to separate and interpret patient heterogeneity in ehr data. In 2021 IEEE EMBS International Conference Biomedical and Health Informatics (BIII), pp. 1-4. IEEE, 2021.
Md Rezaul Karim, Oya Beyan, Achille Zappa, Ivan G Costa, Dietrich Rebholz-Schuhman,, Michael Cochez, and Stefan Decker. Deep learning- -ased clustering approaches for bioinformatics. Brief- ings in bioinformatics, 22(1):393 415, 2021.
Alex Khang, Vugar Abdullayev, Eugenia Litvinova, Svetlana Chumachenko, Abuzarova Vusala Alyar, and PTN Anh. Application of computer vision (cv) in the healthcare ecosystem. In Computer Vision and Al-Integrated loT Technologies in the Medical Ecosystem, pp. 1-16. CRC Press, 2024.
Yiwen Kou, Zixiang Chen, and Quanquan Gu. Implicit bias of gradient descent for two-layer relu and leaky relu networks on nearly- orthoooll data. Advances in Neural Information Processing Systems, 36, 2024.
Alex Krizhevsky.
Learning multiple layers of features from tiny images. pp. 32-33, 2009. URL toronto. edw/~kriz/ www.kkreeeeeanngg--...ee features 1009-TR.pds.
httpps//wwet http://wwww.....oottttoooo
John Lipor and Laura Balzano. Leveraging union of subspace structure to improve constrained clustering. International Conference on Machine Learning, pp. 2130-2139. PMLR, 2017.
Zachary C Lipton. The mythos of model interpretability: In machine learning, the concept of interpretability both important and slippery. Quene, 16(3):31-57, 2018.
Scott Lundberg. unified approach to interpreting model predictions. arXiv preprint 17Xiv:1705.07874, 2017.
Luo, Wuyang Zhuo, and Binfei Xu. deep neural network-based assistive decision method for financial risk prediction in carbon trading market. Journal of Circuits, Systems & Computers, 33 (8), 2024.
Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heintz, and Dan Roth. Recent advances in natural language processing via large pre-trained language models: A survey. ACM Computing Surveys, 56(2):1-44 2023.
Christoph Molnar. Interpretable machine learning. Lulu, com, 2020.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021(12):120003 2021.
Loris Nanni, Gianluca Maguolo, Sheryl Brahnam, and Michelangelo Paci. An ensemble of convolu- tional neural networks for audio classification. Applied Sciences, 11(13):5796, 2021.
Meenal V Narkhede, Prashant P Bartakke, and Mukul S Sutaone. A review on weight initialization strategies for neural networks. Artificial intelligence review, 55(1):291-322 2022.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop Deep Learning and Unsupervised Feature Learning 2011, 201 URL http: /ufldl. stanford.edd edu/ Motsenuubers/niis/// how.ppseeessosssueeeeeererrrr
12

---

Published as a conference paper at ICLR 2026
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics Image Processing, pp. 722-729, 2008.
doi: 10. N..1109/CVGIP.2208..44
Keiron O' Shea and Ryan Nash. An introduction to convolutional neural networks, 2015. URL https:/ /tt////axivvvv org/ alss//as///// /bs//1111 08458.
Yu Pan. Zeyong Su, Ao Liu, Wang Jingquan, Nannan Li, and Zenglin Xu. A unified weight initialization paradigm for tensorial convolutional neural networks. In International Conference on Machine Learning, pp. 17238-17257. PMLR, 2022.
Vardan Papyan. Traces of class/cross-class structure pervade deep learning spectra, 2020. URL https://arxee org/ / 11865.
Daniel Pimentel-Alarcon and Robert Nowak. The information theoretic requirements of subspace clustering with missing data. In International Conference on Machine Learning, pp. 802-810, 2016.
thoooeation-theoreticc
Filippo Pompili, Nicolas Gillis, P-A Absil, and Franccois Glineur. Two algorithms for orthogonal nonnegative matrix factorization with application to clustering. Neurocomputing. 141:15 25, 2014.
Farhan Riaz, Saad Rehman, Muhammad Ajmal, Rehan Hañiz, Ali Hassan, Naif Radi Aljohani, Raheel Nawaz, Rupert Young, and Miguel Coimbra. Gaussian mixture model based probabilistic modeling of images for medical image segmentation. IEEE Access, 8:16846-16856 2020.
Sharat Sachin, Abha Tripathi, Navya Mahajan, Shivani Aggarwal, and Preeti Nagrath. Sentiment analysis using gated recurrent neural networks. SN Computer Science, 1:1-13, 2020.
Abdullahi Saka, Lukumon O Oyedele, Lukman A Akanbi, Sikiru A Ganiyu, Daniel WM Chan, and Sururah A Bello. Conversational artificial intelligence in the aec industry: A review of present status, challenges and opportunities. Advanced Engineering Informatics, 55:101869, 2023.
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE transactions on neural networks and learning systems, 34(1 (1 3:8135-8155 2022.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. Journal of Machine Learning Research, 19(70):1-57, 2018. URL http: fmlr. org/ hwtpg//apprs//eeee panpees/vl9// https//vl/iiii11 html.
Ralf C. Staudemeyer and Eric Rothstein Morris. Understanding Istm - tutorial into long short-term memory recurrent neural networks, 2019, URL htthh:tt///aaae //tp:v//aavvvvv///////////./.iia wwr.g
org/ has///b///i / 1909. 09586.
Gilbert W Stewart and Ji-guang Sun. Matrix perturbation theory. Academic press, 1990.
Vaswani. Attention all you need. Advances in Neural Information Processing Systems, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL https: //tp://..xvv....//// abs 1 63762.
Rene Vidal, Ma, and Shankar Sastry. Generalized principal component analysis (gpea). IEEE transactions on pattern analysis and machine intelligence, 27(12):195--199, 2005.
Haifeng Wang, Hua Wu, Zhongjun He, Liang Huang. and Kenneth Ward Church. Progress in machine translation. Engineering, 18: 143-153, 2022.
Jianian Wang, Sheng Zhang, Yanghua Xiao, and Rui Song. review on graph neural network methods in financial applications. arXiv preprint arXiv:2111.13367, 2021.
Felix Wong, Erica J Zheng, Jacqueline A Valeri, Nina M Donghia. Melis N Anahtar, Satotaka Omori, Alicia Li, Andres Cubillos- Ruiz, Aarti Krishnan, Wengong Jin, et al. Discovery of structural class of antibiotics with explainable deep learning. Nature, 626(7997) 2024.
2024.
13

---

Published as a conference paper at ICLR 2026
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnis, novel image dataset for benchmarking machine learning algorithms, 2017. URL https:://///..c htttps://aaaaa.. arwiv. http://axxvvv.aaaaaaaa/a wwtp:o/gg//oo/o.o.oo http:/bbb//110... 077477.
Bing Xu. Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv: 505.00500333 2015.
Zahra Zamanzadeh Darban, Geoffrey Webb, Shirui Pan, Charu Aggarwal, and Mahsa Salehi. Deep learning for time series anomaly detection: A survey. ACM Computing Surveys, 57(1):1-42, 2024.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.
Huikun Zhang, Spencer S Ericksen, Ching-pei Lee, Gene E Ananiev, Nathan Wlodarchak, Peng Yu, Julie C Mitchell, Anthony Gitter, Stephen J Wright, F Michael Hoffmann, et al. Predicting kinase inhibitors using bioactivity matrix derived informer sets, PLoS computational biology, 15 (8):e1006813 13, 2019.
Yanlin Zhou, Kai Tan, Xinyu Shen, Zheng He, and Haotian Zheng. A protein structure prediction approach leveraging transformer and enn integration. In 2024 7th International Conference on Advanced Algorithms and Control Engineering (ICAACE), pp. 749-753. IEEE, 2024.
A
APPENDIX A
The following section depicts experiments using 5 layer neural network for real data. The datasets included MNIST Deng (2012), CIFAR-10 Krizhevsky (2009), Extended MNIST Cohen et al. (2017), Street View House Numbers (SVHN) Netzer al. (2011), Kuzushiji-Min -NNIS Clanuwat et al. (2018), Fashion MNIST Xiao et al. (2017), Flowers-102 Nilsback & Zisserman (2008), Food-101 Bossard al. (2014), USPS Handwritten Digits Hull (1994). STL-10 Coates et al. (2011), Oxford DTD dataset Cimpoi et al. (2014), Oxford Pet dataset, and EuroSAT AT dataset Helber et al. (2019). Figure includes the projection distance vs epochs plots for MNIST, CIFAR- 10, and EXtended-MNIST. For the experiment, the model was run for 200 epochs with fixed learning rate for all the datasets included the experiments.
ReLU M84ST
ReLU CIFAR-10
RALL EWRAST.OG
[P0] my9g
(ho Wott
(PD) -W7l
30
75
Fpnch
100
200
Figure 6: Left: MNIST dataset for 100 trials. Center: CIFAR-10. Right: Extended-MNISs
Rell LI- Featiinndeuss
Fbd Frods-0r
D- KMNIST
4.000
+
9016
30
6401 mnn
PXC 1.9%)
E70 mrig
20
1.0
120
Figure 7: Left: Fashion-MNIST. Center: Food-101. Right: Kuzushiji-MNIST
14

---

Published as a conference paper at ICLR 2026
Rel. OslordiIT Pet
RoLU -TLL-10
RatU LSPS
1966
E70 NnG
J9S. KYng
E70 WnG
25
Prpeeton
Figure 8: Left: Oxford Pet Dataset. Center: STL10 Dataset. Right: USPS Dataset
Bel.U EurBAT
ToLU-DDD-Teeraee
Po. 1016
mns
Projeton
Ph
50008
Figure 9: Left: SVHN. Center: EuroSAT Dataset. Right: Oxford DTD Textures Dataset
B APPENDIX B
Multilayer Extension. The multi-layer bound mentioned in section 6 can be obtained with triangle inequality. Specifically,
-(Y²) ) < P - PPXX - -P(Y')) b
P -
And the spectral gap must be the maximum among the corresponding factors of all layers analogous to those equation 2.
15


Reviews:
[{"review_id": "DjcZcni1Jn", "reviewer": "Reviewer_QkDR", "summary": "The paper presents a theoretical analysis of when neural networks preserve clustering structure under a union of subspaces (UoS) model. It derives conditions ensuring that a network’s learned representations retain the subspace structure of clean data, even when inputs are noisy. Through a perturbation analysis, the authors show that the row space of the learned representation closely matches that of the original data. Experiments support the theory, revealing that certain networks naturally maintain clustering structure, offering insights into initialization, feature encoding, and regularization.", "questions": "- How is the final clustering performance measured w.r.t. to the ground truth (e.g., with ACC, NMI or ARI) of a network linked to its preservation of subspace clustering structure? I imagine that clustering performance of CNN's or Transformers would generally be higher than the performance of MLPs\n- In the paper you mention: \"The large projection distances exhibited by Transformers and CNNs show that these networks are clustering through some mechanism other than the closed-form solution, and such mechanism does not preserve the original clustering structure.\" Could you elaborate on this? This is a very interesting insight, and I would be curious to see a more in depth discussion in the paper on this point.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper fills a theoretical gap by analysing the clustering behaviour of neural networks using subspace clustering and perturbation theory, offering a proof for a long-standing empirical intuition.\n- Synthetic experiments demonstrate that gradient descent naturally enforces the theoretical condition without explicit regularisation.\n- Provides insights for neural network design decisions, like initialisation", "weaknesses": "- The UoS model and large spectral-gap condition may not hold in realistic settings\n- The paper relies only on projection-distance measures; evaluating standard clustering metrics (e.g., NMI, ARI) could show how clustering performance is linked to the network architecture and behaviour of projection-distances. \n- The paper only briefly discusses that breaking subspace clustering structure, as seen in CNNs and Transformers, may actually contribute to their strong empirical performance. I found this insight actually very interesting, as these architectures are more commonly used in practice than MLPs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5Je11Hlfie", "reviewer": "Reviewer_R2m9", "summary": "The paper attempts to develop theoretical understanding on the observation of some neural networks preserving clustering structure, in which precise conditions for clustering structure preserving is provided. Some theoretical results are developed to bound the gap beween the projected differences between the clean data, transformed data and the output data after one-layer transform. Moreover, numerical analysis and empirical evaluation on real world data are shown, with some explaination and discussions.", "questions": "Please read the weaknesses.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+ It is appealing to investigate the observation of clustering structure preserving with some theoretical justification.", "weaknesses": "1. The mathematical analysis in the paper is confusing. Symbols used in proofs are not clearly defined or without any interpretation. For example, the norm of the matrix, $\\| Z\\|$ and $\\| Z \\|_\\infty$ are not clearly defined, $\\sin (\\Theta)$ is not clearly defined. Even worse, the projection $P$ and $W$ are also not clear.   \n\n2. The derivations in the proofs are incorrect. \n- For example, in the proof of Lemma 4.1, the third equality seems incorrect. How to build the equality from a trace of the two projected matrix to the squared Frobenius norm of them? \n- In the proof of Lemma 4.2, the reviewer cannot see how \"plugging this and (5) in (6) we obtain the lemma.\" The reviewer cannot obtain the result in Lemma 4.2. The proof for Lemma 4.2 are wrong.  Moreover, for the matrix $W$, it is weight matrix, without any implicit assumption on the rank (if not specially claimed). \n\n3. Regarding to Theorem: It is not clear how the condition of $\\delta_1$ which is defined as the singular value gap of $X$ and $X^\\ast$ is actually used. \n\n4. The empirical evaluations are insufficient and incomplete. How the neural networks are trained actually? Which loss function is used? Are the empirical results obtained from the synthetic data generalizable in a broad sense, or it is just because of the special structure of the synthetic data? The way to generate the synthetic data is to form a set of nearly orthogonal subspaces. \n\n5. Is the initialization of $W$ reasonable? What happens if $W$ is initialized with an iid Gaussian of zero mean and variance of $1/m$? From the viewpoint of the reviewer, it is more natural and making sense, or even with nice property.  Why initialize $W$ with i.i.d. uniform entries in the range of $(- \\sqrt(m), \\sqrt(m))$? \n\n6. The worse results in Fig.2 (right panel) are not clearly interpreted.  In Fig. 5, why CNN and Transformers are failed to preserve the clustering structure? How the clustering accuracy is obtained in Fig.4?  \n\n7. Minor issues: \n- The format for the citation in some places seems not properly used.\n- The first three paragraphs of the introduction read like generated automatically via a LLM. Some contents are not factual.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zzDkGTReXD", "reviewer": "Reviewer_fY3w", "summary": "This paper theoretically investigates the ability of certain neural networks to preserve subspace clustering structures and formalizes this conjecture using empirical research.", "questions": "1. What specific training objectives did you use in the deep network, LSTM, and real-data experiments in Sections 6, 7, and 8? Was it autoencoders and mean squared error loss, or something else?\n2. For the multi-layer extension in Section 6, can it provide a rigorous mathematical formulation or derivation of how Theorem 3.1 applies to $L$ layers via the joint bound? What is the resulting total error bound, and what requirements does it impose on the error $\\epsilon_\\ell$ for each layer?\n3. The theory assumes additive noise Z. Has the conclusion been explored for more complex, non-additive, or data-dependent noise?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper offers a novel perspective, connecting the empirical success of neural networks to the classical theory of subspace clustering.\n2. The writing is generally clear. The proofs are well-structured and convey the core ideas.\n3. The experimental design is comprehensive, progressing from synthetic data to diverse real-world datasets, supporting the arguments.\n4. It provides a viable explanation for the effectiveness of deep learning on data with inherent clustering structure.", "weaknesses": "1. The theoretical extension from single-layer to multi-layer networks is overly simplified. Section 6 claims to extend the results to multi-layer networks, but this is not fully elaborated.\n2. The paper states that CNNs/Transformers achieve high accuracy without maintaining the original structure. It is worth exploring in depth the circumstances under which maintaining the original structure is preferable to reconstructing a new one.\n3. The paper does not clearly specify the specific loss function and architectural details used for training the neural network.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)