Multi-ReduNet: Interpretable Class-Wise Decomposition of ReduNet
Authors: Fengrong Li
Delin Chu
Venue: ICLR 2026 Poster (2026)
Keywords: interpretable machine learning
white-box neural networks
ReduNet
Multi-ReduNet
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose Multi-ReduNet, an interpretable white-box model that achieves efficient, class-wise feature learning with improved separability.

Abstract: ReduNet has emerged as a promising white-box neural architecture grounded in the principle of maximal coding rate reduction, offering interpretability in deep feature learning. However, its practical applicability is hindered by computational complexity and limited ability to exploit class-specific structures, especially in undersampled regimes. In this work, we propose Multi-ReduNet and its variant Multi-ReduNet-LastNorm, which decompose the global learning objective into class-wise subproblems. These extensions preserve the theoretical foundation of ReduNet while improving training efficiency by reducing matrix inversion costs and enhancing feature separability. We provide a concise theoretical justification for the class-wise decomposition and show through experiments on diverse datasets that our models retain interpretability while achieving superior efficiency and discriminative power under limited supervision. Our findings suggest that class-wise extensions of ReduNet broaden its applicability, bridging the gap between interpretability and practical scalability in deep learning.

Published as a conference paper at ICLR 2026
MULTI-REDUNET, INTERPRETABLE CLASS-WISE DE- COMPOSITION OF REDUNET
Fengrong Li National University of Singapore e05064 62@u. nus edu
Delin Chu
National University of Singapore ma matchud1@m Serhuud@@nuseeeee .ew...ddu..... 8g
ABSTRACT
ReduNet has emerged as a promising white-box neural architecture grounded in the principle of maximal coding rate reduction, offering interpretability in deep feature learning. However, its practical applicability is hindered by computational complexity and limited ability exploit class- saes-sp structures, especially undersampled regimes. In this work, we propose Multi-ReduNee and its variant Ne-.aaeee-essttorreee which decompose the global learning objective into class-wise subproblems. These extensions preserve the thcoretical foundation of ReduNet while improving training efficiency by reducing matrix inversion costs and enhancing feature separability. We provide concise theoretical justifica- tion for the class-wise decomposition and show through experiments on diverse datasets that our models retain interpretability while achieving superior efficiency and discriminative power under limited supervision. Our findings suggest that class-wise extensions of ReduNet broaden its applicability, bridging the gap be- tween interpretability and practical scalability in deep learning.
INTRODUCTION
High-dimensional data across finance, biomedicine, and social networks often exhibit undersam- pled regimes (feature dimension d >>>>> number of samples m)) due to limited samples, privacy strictions, and acquisition costs. In this settings, many distinct models can interpolate the training data equally well, which tends to exacerbate overfitting and unstable generalization (Hastic et al. Bühlmann & Van De Geer 2011), posing a fundamental challenge. While ReduNet (Chan 2022), white-box framework grounded in Maximal Coding Rate Reduction (MCR2) Yu 2020). provides interpretable feature learning with provable optimization, its global ((d))
2009
al.
et al.
complexity can be reduced via class-wise decomposition in undersampled, imbalanced regimes where class sizes vary significantly]
To overcome these limitations, introduce extensions: Multi-ReduNee and Multi-ReduNett LastNorm. By decomposing the global ReduNet objective into class-wiss wise subproblems, Multi- ReduNet improves computational efficiency and enhances representation separability in undersam- pled regimes. The LastNorm variant further refines this process by solely enforcing single normal- ization at the output, yielding consistent gains across multiple classifiers.
We contribute: (1) Theorem 2. proving MCR2 admits rigorous class-wiss decomposition via class- orthogonality (Theorem 1), enabling independent per-class optimization without loss of optimality; (2) Multi-ReduNet and
reducing computational complexity of each parameter from ((d3) to (mm3) via Woodbury identity while preserving interpretability; (3) exten- sive experiments six datasets (Reuters. MNIST. Mushion-MNISS Swarm, DrivFace, ARCENE) showing that, when averaging over four learning rates (0.5, 0.1, 0.05, 0.01) and three downstream classifiers (SVM, kNN, NSC), Multi-RRdduNe(( ) achieves 8.5-52.7 percentage points higher mean accuracy than ReduNet (e.g.. +30.7pp on Reuters, +52.7pp on DrivFace), while re- ducing wall-clock training time by about 2x on average (1.4-2.6x across datasets) and improving learning-rate robustness by roughly an order of magnitude (up to 9.8x smaller accuracy range across 17)- We used GPT-4 solely for language polishing. All technical content, analysis, and conclusions remain those of the authors.

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Learning from undersampled, high- high-dimensional data (m < d) arises across genomics (d > 20,000, m = 30-1000 (Nguyen & Rocke, 2002), mass spectrometry (Adam et al. 2002), and rare disease imaging (Litjens al. 2017p. The ARCENE dataset (Guyon et al. 2007) exemplifies this with 10,000 features but only m = 200 samples (m/d = 0.02), making representation learning in challenge.
the < d regime a widely-recognize
Dimensionality reduction methods such as PCA (Greenacre ct al. 2022) and LDA (Xing et al, 2001) alleviate sample scarcity by projecting data onto lower-dimensional subspaces. Generative approaches, including GAN-based data generation (Goodfellow et al. 2014), instead synthesize additional training examples to enrich the data distribution. However, these methods primarily oper- ate at the level of global statistical modeling and do not explicitly encode class-specific structure, by which we mean the cabel-conditioned geometry of per-class feature subspaces and their mutual rela- tions. This may limit their ability to learn structurally interpretable representations in undersampled regimes (m d).
Specialized deep learning models for data-scarce settings include few-shot learning frameworks (Prototypical Networks (Snell et al. 2017]. Matching Networks (Vinyals al., 2016)) and meta- learning (MAML (Finn et al. (017)) Siamese networks (Koch) 2015)). While effective, these oper- ate as black-box models with limited transparency.
Information-theoreti objectives (InfoMax (Hjelm et al. 2019), Information Bottleneck (Alemi al., aaaa.1220)) 2017), Rate-Distortion (Theis et al. 2017)) offer principled criteria for representation learning by maximizing mutual information trading off compression and prediction accuracy. However, these methods are typically implemented via deep neural encoders trained with variational bounds and stochastic gradient descent, which yields black-box feature maps without closed-form updates, class-specific structure, or transparent geometric interpretation.
ReduNet (Chan al. 2022) addresses this via white-box framework grounded in Maximal Cod- ing Rate Reduction (MCR2) (Yu et al., 2020), where each layer admits an analytic update and the resulting network provides interpretable, geometry-aware feature maps with provable optimization guarantees. However, ReduNet operates on global feature matrices with dense operators, leading to an (dd) per-parameter complexity in the feature dimension d, which quickly becomes prohibitive in nigh-dimensional, undersampled regimes. This motivates exploiting class-specific structure to decompose the optimization into smaller per-class problems, substantially reducing computational cost while preserving the MCR² objective.
Our work builds on this trajectory by extending ReduNet with class-wise decomposition. In contrast to black-box few-shot or generative models, our proposed Multi-ReduN and Multi-ReduNet- LastNorm retain interpretability while improving representation separability, computational effi ciency, and robustness to learning-rate choices in undersampled regimes.
PROPOSED METHODS
We now present our proposed extensions to ReduNet, designed to reduce computational complex- ity and improve hyperparameter robustness in undersampled regimes. We first review the Re- duNet framework and its connection to the Maximal Coding Rate Reduction (MCR?) principle (Section 3.1). We then introduce imp-ReduNet, which exploits the Woodbury identity reduce computational complexity of ReduNet (Section (3.2). and provide a theoretical justification showing that the ReduNet objective can be decomposed into independent class-wise optimization subprob- lems(Section 3.3 This motivates our proposed architectures, Multi-ReduNeet and Multi-ReduNet- LastNorm (Section 3.4,, which leverage class- wise decomposition to improve computational effi ciency and hyperparameter robustness while retaining interpretability.
3.1 REDUNET PRELIMINARIES
Drawing the principle of Maximal Coding Rate Reduction (MCR2) Yu et al. 2020), ReduNet (Chan et al. 2022) has been proposed as new class of boxte-box networks. It seeks to learn a feature representation Z Rdxm, where d is dimension of features and m is the number of samples, that
2

---

Published as a conference paper at ICLR 2026
maximizes the discrepancy between global and class-wise covariance complexities. The original objective (MCR²) with respect to the distortion e takes the following form:
1 log det (T aZ77
max e(Z,e) ( max 2
log det (T az (T
2
Global Coding Rate
Class-wise Coding Rate
s.t. 2 13 - 2TP my--
where II Ruxm denotes the membership matrix for class whose diagonal entries represent the probabilities of m samples in class j. my = tr(11) is the number of samples in class j. = d/(me²), aj = d/ (m,+²), and - =mj/m.
This objective maximizes the global coding rate (promoting inter-class diversity) while minimizing class-wise coding rates (enforcing intra-class compactness), yielding discriminative yet coherent representations.
During training, ReduNet,joinlly updates both representations and model parameters through layer- wise greedy optimization. At each layer I, a set of closed-form parameters {Ei, C/) are com- puted based on the current input features Zi where
+ (((+a21))--11 E,Z1; d log d1 + aZiZi aII+
d7
d log (T aj Z1
a, +a,21111121 C/ ZII.P
dz
These matrices govern the update of each training sample 2f E Rdd via:
2/4 = Psa-- + Engeet( +/(x/)
exp (-pp->lic (-XIC
Σø= exp (-X|IC) =fl))
where A, are hyperparameters, and Pd--- projects the update onto the unit sphere. Inference uses the same update rule as training, applying the learned E₁ and Of to test inputs.
Rationale for unit-sphere projection. The projection Psa-1(-) serves two critical purposes. First, we consider an MCR" objective optimized under the class-wise Frobenius-nome norm constraint | = my: projecting each column the unit sphere is simple sufficient way to enforce this bound consistently in both training and inference (it is not mathematically necessary, and any other hounded-norn parameterization satisfying || I|2 my would also be valid). Second, without norm control, the MCR objective could be trivially increased by multiplying Z by a large scalar, leading to degenerate solutions that exploit magnitude rather than learning meaningful discrimina- tive directions. The unit-sphere projection prevents this scaling degeneracy and forces the optimiza- tion to focus on finding discriminative subspaces in feature space instead of arbitrarily amplifying feature norms. We retain this projection in our Multi-ReduNee design for the same reasons.
3.2 IMPUREDNNE:: REDUCING COMPUTATIONAL COMPLEXITY
ReduNet requires d x d matrix inversions for E, and C₁. incurring O cost. When < d, we exploit the Woodbury identity to reduce this to ((m)): Lemma (Woodbury Identity). For any a E R and X E Rdxmm
(1 aXxTX-- = ax xxx
where the left side requires inverting a d x matrix, while the right side only requires inverting an matrix.
Applying Lemma to both E, and C/,, we reduce the per-parameter complexity in ReduNet from (d3) to O(m3), substantial gain when m & d. For example, on the ARCENE dataset (d = 10,000, Miain 159), this represents theoretical speedup factor of (10,000/ 159)3 ss 250,000x in the inversion step alone. We refer to this Lemma b-based implementation ReduNet as imp- ReduNet.
3

---

Published as a conference paper at ICLR 2026
While Lemma 1 addresses the dimensional bottleneck, it does not exploit the class structure of the data. When the total sample size m is itself large (e.g., m 1,000), the m x m inversion can still be expensive. This motivates further decomposition: mj x my problems, one per class? The theoretical justification for this strategy the subject of the next section.
break the x m problem into K smaller
For the complete derivation of Lemmal ! see Appendix
3.3 MULTI-REDUNET CLASS-WISE DECOMPOSITION
Having established that imp- ReduNet reduces the per-parameter cost from (dd)) to O(m²) (Sec- tion 3.2 we now address a complementary question: can we further exploit the class structure to decompose the global MCR objective into independent per-class subproblems?
Intuition. For K classes with sizes {m,}}ー m, = m), independent per-class optimization costs (m,)) < (m²)), especially when imbalanced. We show MCR² permits this decomposition without optimality loss. Crucially, class class-orthoge orthogonality emerges as a property of the optimal solution
of MCR (Theorem 1 rather than externall imposed constraint.
Theorem 1. Let = [z!...., E dddxm denote the feature matrix, and let {IP R xm km)kKKKK K be
1==1
diagonal membership matrices such that Ex= consider the MCR² objective
K II = Assume rank(ZII] S dj and Σ/=1 dj S
K
max Z
log det [ d
me²
2
Σ j=1
2m
my log det (I+ m3r² ZII )T)
(1)
subject to |1T1 | mj, where my = (r([IP). Then any optimal solution Z necessarily satisfies the class-ortho grthoogtttoyy property:
TZ for all i j,
where Z III denotes the class-j partition.
Notation. We follow the convention that columns of 'Z represent samples (i.e., Z E RRdx where rows are features and columns are samples). Thus, (2') 7 E m.m.xmy, the cross-class Gram matrix measuring inner products between class-i and class-j samples. The condition (Z') Z 0 expresses that the column spaces of Z' and Z are orthogonal.
Proof sketch. The proof proceeds by contradiction using determinant inequality for sums of posi- tive semi-definite matrices (Corollary in Appendix D.I
Assume the optimal Z' has (Z T 2 + for some classes f1 j2. By Corollary 1, the global coding rate det(I + Σ3 (Z*)) is strictly smaller than II, dct(I + (Z when classes overlap. We then construct an alternative solution Z' by re-orthogonalizing via SVD while preserving per-class singular values. This Z' achieves strictly higher objective value, contradicting optimality of Z*
The complete proof with detailed matrix algebra is in Appendix D.2
Theorem 2. Let Z* be any optimal solution to the global MCR² problem (1). By Theorem 1 Z satisfies class-orthon caas-orthogonality
can write Z+J z=' 0 for j Under this optimal class- orthogonal structure, suppose that rank( S d, for each class and Σø= dj d. Then the objective in (I) decomposes into K independent class-wise problems:
logdet(I+ + me² (z(Z))
my log dot(I
2' (2))
max Zi
m
m,e
subject to || = my.
Proof. Denote U1 as the optimal value the MCR?, U/ as sum of the optimal values of class wise problems (2), the proof follows by showing that (i) any class-wise feasible solution is also globally feasible (hence 1/2 < 81), and (ii) by Theorem the global optimum Z satisfies orthogoaltg,, making it feasible for the class-wis problems (hence U1 I 02). See Appendix D.3e for details. [1
4

---

Published as a conference paper at ICLR 2026
Theorem establishes that the global MCR² objective can be decomposed into K independent class- wise subproblems without loss of optimality. While Theorem I
was previously known (Chan et al. 2022). our proof via Corollary 1 is more direct and streamlined. Crucially, Theorem 2 enables the first practical algorithm for class-wise MCR² optimization.
Practical implications. Under undersampled scenarios < d, the data inherently fulfill the con- ditions of Theorems [ [ since the rank of each class-specific feature matrix is bounded by its sample size: j== " rank( S Σ m, m < d.
Classoorthogonality as an optimality condition. Theorem establishes that class-orthogonality necessary property of any global optimum of the MCR-, not a constraint we impose during optimization. Theorem 2 then shows that, under this optimal class-orthogonal structure, the global objective (1) equivalent a set of K independent class-wise problems (3). In practice, our iterative algorithm (Algorithm 1) optimizes these decomposed per-class objectives independently and does not enforce ( T Z 0 as a hard constraint; any approximate olasooorhog in the learned features arises from the optimization dynamics rather than from explicit regularization.
As with the original ReduNet implementation, numerical optimization on realistic datasets does not yield perfectly orthogonal class representations. Deviations from exact orthogonality arise from (i) convergence to local optima, (ii) finite optimization steps and numerical precision, and (iii) prop- erties of the input data X (e.g., limited class separability, noise, and model mismatch), which may prevent gradient- based methods from reaching the global optimum basin. Thus, the class-wise de- composition should be viewed theoretically justified reparameterization the level of global optima, while in practice it produces approximately disentangled class representations without re- quiring explicit orthogonality constraints.
Together, Theorems Danboreemm[addjjuufffify a class-wise decomposition strategy: instead of solving the global MCR objective, we can equivalently optimize K independent per-class subproblems:
max Ex
log det
[I me²
mi
log det [I m (()))
(
(3)
m
subject to || = my for each class j I.
K.
Gradient formulations. For optimization via gradient ascent, we compute the per-class gradients. Denote mez
d and aj mye
d The gradient of the first term (per-class expansion component) is:
azi log det (T=ZZ(ZZ'))) = 2a (I 20 (z=))) z
(4)
The gradient of the second term (per-class compression component) is:
azi log det (T+ (I+)) = 2a, (T (z,z z
(5)
Combining these, the gradient of the j-th class-wise objective (3) is:
VZ. = a aZ
aZ (Z Z my aj (I + (z+)) z'.
(6)
m
Iterative updates. In deep network with L layers, we apply gradient ascent each layer / I,...,L. L. Let zi e Rdxm, denote the class-j features at layer 1. We define the per-layer gradient matrices:
Ed a a1+
cf aa (I+ 0, 2i (2))
(8)
The gradient ascent update (before projection) is:
2/41 + Z/ +
E - mj Ci zz
(9)
where 17 is the learning rate. To enforce the norm constraint | / 1| = m, for each class j, we apply spherical projection:
Zi41 = Paa-- Z +n E
mi c zi ))
(10)
m
5

---

Published as a conference paper at ICLR 2026
where (-) normalizes each column to unit norm.
Note that Ey and Cf are both functions of the same class-wise covariance (2(), but they enter the update only through (Ej mm3/mmm/mm ) Zif: the expansion term (with coefficient a)) pushes features to spread out globally, while the compression term (with a,) pulls each class towards compact subspace. In Multi-ReduNee these operators are computed from class-wise covariances instead of the global covariance ZZ" making the optimization decoupled across classes.
The class-wise decomposition directly motivates the design of Multi-ReduNee. Importantly, our implementation directly optimizes the decomposed objectives (3). Building Theorem implement slass-wise decomposition via parallel per-class optimization. Training: Each class j updates independently using Z = Pss-i (Zi + ( of where E/. Cf are de- fined in equation and equation 8 and the computational complexity can be reduced by Lemma
Inference: Test samples use soft assignments #ं to aggregate class-specific updates: 21+1
Pad- (E)+ ( n z1 -,, #
3.4 MULT--REDUNET-LASTNOO
Mouti-ReduuNe-LLattornn is a variant of Multi-ReduNaa that shares the same class-wits wise MCR" decomposition in Theorem 2
The global feature matrix is partitioned Z = [Z], ..... ZK1, where each block Z E Rdxm, collects features from class j and is updated by its own operators Ej Cj- We use the term class-specifie structure to refer to this label-conditioned representation: each class associated with its own feature subspace spanned by Za, rather than being coupled through global covariance ZZ and different classes are encouraged to occupy (approximately) orthogonal or weakly overlapping subspaces. Multi-ReduNee and Multi-ReduNNt-LLsthhe serve and exploit this class-specific structure while providing white-box realization of class-wise MCR
are designed to pre-
Algorithm Training Algorithm of Multi-ReduNNe and Multi Multi-ReduNNe-LastNomm
Require: Input data X Rddmm class memberships (IP) 1
(IP) parameters e 0, X, learning rate
y).
1: Compute class sizes: my = tr([TT)) priors 7/ my
d 2: Set a mr?, and Oj m,c for j = I..... K
3: Initialize features: Z1 = X
4: for = 1 to L do
5: if = then
6: Extract class-wise inputs: {Z}- = k
7: end if
8: for - K do
Compute: Ej = (1+ -1, )-
9:
10:
Update features:
PS... + n (EE
(Multi-ReduNe
2/41 Zi + 2 7))
(Muti-ReduNee-LastNNonm)
11: end for
end for
13: if Multi-ReduNe-LLsttNor then
Apply Ps._. (.) all Z2 for 1,...,,
15: end if
16: return features (2))
( 1==11=1
K,L+1
Compared to Multi-ReduNet, Multi-ReduNet--astNNoo relaxes intermediate normalization by ap- plying the projection (.)-1(.) only at the final layer L (detailed analysis in Appendix E.2) This allows more flexible intermediate representations while maintaining comparability at the last layer, reducing projection overhead and improving hyperparameter robustness (Section 4.3). As in Multi- ReduNet, the Woodbury identity (Lemma reduces each class-specific inversion from (d3) to

---

Published as a conference paper at ICLR 2026
(m,)), yielding around 2 empirical speedups average across datasets. Complete training pro- cedures for Multi-ReduNet and Sulti-ReduNet-LastNomm are given in Algorithm
4 EXPERIMENTAL EVALUATION
4.1 EXPERIMENTAL SETUP
Setup. Experiments run on NVIDIA A100 GPUs with I= 5 layers (results for I. E (10, 15, 20, 25) in Appendix L.I. 2² 0.1, batch size 100. We evaluate on six undersampled datasets (Reuters, Swarm Behaviour, DrivFace, ARCENE) spanning text, flattened images, survey data, and medical diagnostics, plus three failure-mode datasets (Iris, Mice Protein, CIFAR- 10) detailed in Appendix [A] Final-layer features are classified using SVM (Cortes & Vapnik] [995), KNN (Cover & Hart 1967), and NSC (Chan et al. 2022). All results averaged over random seeds. summarizes dataset characteristics (Maain d ranges from 0.016 to 0.5).
MNIST, Fashion-MNNN
Table
Table 1: Dataset statistics for experimental evaluation
Dataset
d
Marin
Mees
K
Marain/ /d
Domain
Reuters
18,933
5,304
1,328
5
0.280
Text classification
MNIST
10,000
5,000
1,000
10
0.500
Flattened images
Fashion-MNIST
10,000
5,000
1,000
10
0.500
Flattened images
Swarm Behaviour
2,400
1,200
300
0.500
Survey data
DrivFace
4,096
484
122
0.118
Safety-critical CV
ARCENE
10,000
159
41
2
0.016
Medical diagnostics
4.2
MAIN RESULTS
Multi- Multi-ReduNNet reduces the per-parameter complexity from (dd) (ReduNet) to O(m)) via class- wise decomposition and the Woodbury identity (see Table 2 for detailed comparison). In ex- periments we compare Multi-ReduNNet and Multi-ReduNer--astteeee Woodbury-optimized imp-ReduNet. We also explored Random-Foress variants (ReduNet-RF, imp- ReduNet-RF) that replace the soft membership predictor (=) with Random Forest, but they neither improve accuracy nor efficiency and are therefore reported only in Appendix B
against ReduNet and the
Table 2: Theoretical Computational Complexity of ReduNet-Basse Based Models
Model
Computational Complexity
ReduNet
O (K + +1 dd) O(L (m3 + dm + dm + Ex= m + d. EØ= m3))
imp-ReduNet
Multi-ReduNet
OLL (dem + E my d. 2i mz))
Classification accuracy. We compare four ReduNet-based variants across six undersampled datasets. Due to the extreme undersampling regime, we avoid aggressive hyperparameter tuning and use a fixed learning rate Te 0.05 for all methods and datasets, unless stated otherwise. Run- ing on all six datasets showed that T/ yields stable training and competitive performance across models. Table reports test accuracy under this shared setting. Appendix F results over 77 {0.01, 0.05, 0, 0.5), which show consistent trends.
we provide the full
Bewee-eeduNee--LatNooooo yields the largest gains on the most severely undersam- pled and noisy datasets. On DrivFace and ARCENE, accuracy improves from 0.43-0.46 for Re- duNet to 0.73-1.00 across classifiers. On Reuters and Swarm Behaviour, we also observe sizable improvements (e.g., 0.802 + 0.985 SVM accuracy Reuters). In contrast, the subsampled MNIST and Mushion-MNISTT benchmarks, ReduNet already achieves strong performance and Multi- ReduNet(- Nosttoomm remains within few percentage points, indicating that the additional class-
Table 3 shows
7

---

Published as a conference paper at ICLR 2026
Beakes
HANNT
Foeiden SINIST
100
Lik Gobdle
40
Narberaflarennt
Nashir Sohir--/mmmm
Nomthrr of Laeemett
Namberaflaeemnt
Number Laleomn
Figure 1: Wall-clock training time (in seconds) with increasing network depth (L (5, 10, 15, 20, 25} layers) across six datasets. Multi-ReduNet (orange diamonds) and Multi- ReduNet-LastNNrm (green triangles) consistently achieve 1.4-2.6x speedup over ReduNet (red circles) due to class-wise decomposition reducing complexity from O(L (K + 1) d3) to OLL (dmm + Σ my + d. Σド= mž)).
Table Accuracy comparison of ReduNet variants undersampled regimes.
Reuters KNN
mnist
Model
SVM
NSC
SVM
KNN
NSC
ReduNet
0.802
0.670
0.922
0.906
0.930
0.903
imp-ReduNet
0.802
0.668
0.922
0.906
0.930
0.904
Multi-ReduNet
0.984
0.939
0.957
0.837
0.902
0.871
Multi-RcduNet-LastNNom
0.985
0.943
0.957
0.842
0.903
0.873
fashion-mnist
Swarm Behaviour
Model
SVM
KNN
NSC
SVM
KNN
NSC
ReduNet
0.858
0.826
0.836
0.802
1.000
0.996
imp-Redute ReduNet
0.858
0.825
0.836
0.802
1.000
0.996
Multi-ReduNet
0.798
0.790
0.800
1.000
1.000
0.929
Mult-RedduNe-LastNoom
0.801
0.802
0.803
1.000
1.000
0.927
DrivFace
ARCENE
Model
SVM
KNN
NSC
SVM
KNN
NSC
ReduNet
0.432
0.393
0.366
0.439
0.415
0.463
Rep-ReduNeet
0.432
0.393
0.366
0.439
0.415
0.463
Multi-ReduNet
1.000
0.951
0.995
0.829
0.732
0.805
Multi-ReduNNetLastNoom
1.000
0.978
0.995
0.829
0.732
0.805
wise flexibility is most beneficial in the more challenging, high-dimensional microarray and face datasets.
Comparison with classical dimensionality reduction baselines. To provide broader context, compare ReduNet, Multi- -etii-Rddeeee--sSNoooooo
against classical methods including PCA and LDA. Table 4 reports, for each method, the best test accuracy obtained over the shared learning- rate grid 77 E (0.01, 0.05, 0.1, 0.5) on all six datasets (for neural methods). For PCA, we tune the number of components Aeomp E (K, min(d, m)) and report the best accuracy.
8

---

Published as a conference paper at ICLR 2026
Table 4: Broader baseline comparison: best accuracy across classifiers (SVM, KNN, NSC)
Method
Reuters
MNIST
Fashion
Swarm
DrivFace
ARCENE
Global PCA + SVM
0.975
0.878
0.829
1.000
1.000
0.805
Class-wiss PCA + NSC
0.867
0.773
0.667
0.913
1.000
0.756
LDA
0.471
0.615
0.781
0.977
1.000
0.878
ReduNet
0.956
0,937
0.858
1.000
1.000
0.780
Multi-ReduNel
0.988
0.926
0,845
1.000
1.000
0.829
Multi-Ren Multi-ReduNNt-LN
0.988
0.926
0.858
1.000
1.000
0.829
(a) ReduNet
(b) RFF
(c) Multi Redliee
(d) Multi Multi-ReduNet-LN
Figure 2: t-SNE visualizations (van der Maaten & Hinton 2008) of learned test features. Rows (top to bottom): Reuters, MNIST, Fashion- Swarm Behaviour. Columns (L 5, 17 0.5, 2 0.1): ReduNet(1ss 1st column), ReduNet-RF(2nd column), Multi-ReduNet(3rd column), Multi- Leduneet column).
Dolti-ReduNNe-LastNomm excels on imbalanced text/sparse data (Reuters: 98.8% vs 97.5% PCA), where class-wise decomposition exploits per-class low-rank structure. However, LDA achieves higher accuracy ARCENE (87.8% vs 82.9%). indicating classical methods remain competitive on certain well-structured datasets.
Training efficiency. Across all six datasets, Multi- Seati-RRedNNe--Lsstteeee achieves between 1.4x and 2.6x faster training than ReduNet (Figure 1), with an average speedup about 2x. We discuss these efficiency gains in detail, including per-dataset breakdowns and depth dependence, in Section 4.4 and Appendix B
Feature visualization. Figure 2 shows Multi-Redi ReduNet variants (the third and forth columns) pro- duce more compact and well-separated clusters compared to ReduNet baselines, corroborating their enhanced class separability. Enlarged plots are in Appendix 1.2

---

Published as a conference paper at ICLR 2026
4.3 ROBUSTNESS ANALYSIS
Having demonstrated the superior accuracy and efficiency of Multi-ReduNet on multiple datasets, now investigate its robustness hyperparameter variations, which critical consideration for practical deployment.
Hyperparameter sensitivity & LastNorm ablation. Table reports (left) the performance range across learning rates E {0.01, 0.05, 0.1, 0.5} and (right) the ablation study comparing Multi- ReduNet vs. Mull-RecduNet-asslNooe
Table 5: Robustness and ablation analysis (left: hyperparameter range; right: LastNorm impact)
Range (pp) MR
Best Ace ( (%)
Dataset
(pp)
RN
MR-LN
MR
MR-LN
Reuters
67.5
3.3
3.2
98.8
98.8
+0.0
MNIST
86.3
27.1
20.6
92.6
92.6
+0.0
Fashion
71.7
10.7
8.1
84.5
85.8
+1.3
Swarm
32.1
1.0
1.0
100.0
100.0
+0.0
DrivFace
76.5
2.2
3.3
100.0
100.0
+0.0
ARCENE
41.4
9.7
2.4
82.9
82.9
+0.0
Average 62.6
9.0
6.4
93.1
93.3
+0.2
Left:
Performance range is the difference between the highest and lowest best accuracies (over SVM, KNN, and NSC) obtained across 17 {0.01, 0.05, 0.1, 0.5). is reported for ReduNet (RN), Multi- ReduNet (MR), and Multi-eduuNer-LsttNoor (MR-LN). Right: Best accuracy and improvement achieves consistent accuracy gains across all
(A) of MR-LN over MR. datasets.
The combined analysis shows Multi-ReduNet-LssNNNon achieves comparable accuracy to Multi- ReduNet (average +0.2 pp. with +1.3 pp on Mishion-MNISTT while exhibiting 9.8x better hyperpa- rameter robustness than ReduNet and improved stability compared to Multi-ReduNet (6.4 pp vs 9.0 pp average range). This demonstrates that relaxing intermediate normalization constraints allows more flexible representations while maintaining inter-class comparability.
4.4
COMPUTATIONAL EFFICIENCY
ReeduNet-LassNonn achieves an average 2.0x training speedup over ReduNet across all datasets (Table 7 in Appendix B. with the largest gain on ARCENE (2.6x) where extreme un- dersampling (m d = 0.016) maximally exploits the class-wise low-rank structure. Figure shows consistent efficiency gains across network depths: although the relative speedup stays in the 1.4- 2.6x range, the absolute wall-clock time gap grows with L, making the savings particularly signifi- cant for deep (L 20) and ligeesiiieniionll (d 10,000) models. Empirical speedups smaller than the theoretical O(dd/m) m)) gain for the inversion step alone, due to additional overheads (mem- ory traffic, interpreter costs), but still provide substantial practical benefits in the undersampled, high-dimensional settings we target.
CONCLUSION
We propose Multi-ReduNet and Multi-ReduNNe-LaatNNom. interpretable extensions of ReduNet tailored to undersampled regimes (m & d). By performing a class-wise decomposition of the MCR² objective, our approach improves computational efficiency and hyperparameter robustness, while achieving clear accuracy gains on severely undersampled, high-dimensional datasets.
Key contributions. We show that the global MCR objective decomposes into independent class- wise subproblems without loss of optimality (Theorem 2), by establishing class-orthogonaliity the global optimum. This yields the first practical class-wirs wise decomposition algorithm for MCR2. reduc- ing pey-layer computational cost and delivering empirical speedups undersampled benchmarks. Multi-Redd ReduNet--astNoonnee further enhances hyperparameter robustness by deferring normalization the final layer, while preserving the closed-form interpretability of ReduNet-style updates.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Bao-Ling Adam, Yinsheng Qu, John W Davis, Michael D Ward, Mary Ann Clements, Lisa H Cazares, O John Semmes, Paul Schellhammer, Yutaka Yasui, Ziding Feng, al. Serum pro- tein fingerprinting coupled with pattern-matching algorithm distinguishes prostate cancer from benign prostate hyperplasia and healthy men. Cancer research, 6211333309,3614 2002.
Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. In Proceedings of the International Conference on Learning Representations (ICLR), 2017. bttp::::::/
org 1bs21612.001110
Peter Bühlmann and Sara Van De Geer. Statistics for high-dimensional data: methods, theory and applications. Springer, 2011.
Kwan Ho Ryan Chan, Yaodong Yu, Chong You, Haozhi Qi, John Wright, and Yi Ma. Redunet: A white-box deep network from the principle of maximizing rate reduction. Journal of Machine Learning Research, 23(1), January 2022. ISSN 1532-4435.
Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine Learning, 20(3):273-297, 1995. doi: 10.1007/BF00999018..
Thomas M. Cover and Peter E. Hart. Nearest neighbor pattern classification. IEEE Transactions Information Theory, 13(1):21-21) 1967. doi: 10.1109/TT..996.1153399
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. In International conference on machine learning, pp. 1126-1135. JMLR.org, 2017.
lan J. Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warle-Farley, Sher- jil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. 27, 2014. URL httpsss//prroceed
http://pppeecoe.o........ee.esss File/ htlp:ff336c000022277777711111223
pdf
Michael Greenacre, Patrick J. F. Groenen, Trevor Hastie, Alfonso lodice D'Enza, Angelos Markos, and Elena Tuzhilina. Principal component analysis. Nature Reviews Methods Primers, 2:100,
2022. doi: https://doi.org/ 0ttp://///000000//45550000000010
Isabelle Guyon, Jiwen Li, Theodor Mader, Patrick Pletscher, Georg Schneider, and Markus Uhr. Competitive baseline methods set new standards for the nips 2003 feature selection benchmark. In Pattern Recognition Letters, volume 28, pp. 1438-144 1444. Elsevier Science Inc., 2007.
Trevor Hastic, Robert Tibshirani, and Jerome Friedman. The elements of statistical learning: data mining, inference, and prediction. Springer, 2nd edition, 2009.
R. Devon Hjelm, Alex Fedorov, Samuel Aavoic-Marchildon, Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https:
Soopenreviewnne////e Bxt.rrmmmmmddd110001ra
Gregory Koch. Siamese neural networks for one-shot image recognition. Proceedings of the ICML, 2015. /tthht/:::/// hwwp//wwwcccccooo cmu. wdw. htt.:///t//aaaaaaa/gggesss////ea
pdf
Geert Litjens, Thijs Kooi, Babak Ehteshami Bejnordi, Arnaud Arindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoorian, Jeroen AWM Van Der Laak, Bram Van Ginneken, and Clara I Sánchez. A survey on deep learning in medical image analysis. Medical image analysis, 42: 60-88, 2017.
Anay Majee, Suraj Kothawade, Krishnateja Killamsetty, and Rishabh Iyer. Score: submodular combinatorial representation learning. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMg.R 2024.

---

Published as a conference paper at ICLR 2026
Danh V Nguyen and David M Rocke. Tumor classification by partial least squares using microarray gene expression data. Bioinformatics, 18(1):39-50, 2002.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning- In Proceedings of the 31st International Conference Neural Information Processing Sys- tems, NIPS' 17, pp. 4080-4090, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Lucas Theis, Wenzhe Shi, Andrew Cunningham, and Ferenc Huszár. Lossy image compres- sion with compressive autoencoders. ArXiv, abs/1703.00395, 2017. URL https://api.
semanticscholase htt.://cchh.aaaaoooooooouuuaaeee www.aaarrrrrrcroooosasaaaa 8394199
Laurens der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Ma-
chine Learning Research, 9(86):2579-2655 2008. URL http: jmlr www.pperrg/papr///vv
wardermaateenDa..ba html
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. In Proceedings the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pp. 3637-3645, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819.
Eric P. Xing, Michael I. Jordan, and Richard M. Karp. Feature selection for high-dimensional genomic microarra data. In Proceedings of the Eighteenth International Conference Machine Learning, ICML '01, pp. 601-608, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1558607781.
Yaodong Yu, Kwan Ho Ryan Chan, Chong You, Chaobing Song, and Yi Ma. Learning diverse and discriminative representations via the principle of maximal coding rate reduction. In Proceedings of the 34th International Conference Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
A SCOPE AND LIMITATIONS
Our method is designed for specific regime and problem characteristics. Here we provide a compre- hensive discussion of where Multi Relti-RRedNNet excels and where it does not apply.
A.1
PERFORMANCE
Multi-ReduNet's effectiveness strongly dependent on the undersampling ratio m/d:
most beneficial in undersampled regimes with m/d < 1. Across all six datasets, its best accuracy over E {0.01, 0.05, 0.1, 0.5) matches or exceeds that of ReduNet on almost all settings, with Swarm Behaviour being the only case where the best accuracy is lower by 0.1 1.1. The advantage becomes more pronounced as under- sampling intensifies: when m/ < 0.5, of about 31 percentage points on Reuters (m/d- 0. 0.280), 53 points on DrivFace (m/d /d = 01118), and 31 points on ARCENE (m/d = 0.016), when averaged over the four learning rates. On Driv- Face, both ReduNet and Multi-ReduNet(( Norr
Where Multi-ReduNet excels: Multi-ReduNee-LastNoon
yields average accuracy gains
reach 100% accuracy at their best learn- ing rate, reflecting the dataset's 's simplicity after aggressive feature extraction, but Multi-ReduNNe- LastNorm substantially enlarges the range of learning rates that achieve high accuracy. Class- imbalanced datasets are also a natural fit: when class sizes mj vary significantly, the complexity K reduction from (dd)) to Of E m yields 2 empirical specdups imbalanced bench- marks such as Reuters and ARCENE. Moreover,
exhibits 31.1-73.29% better robustness across 77 than ReduNet (Table 5). making it preferable when hyperparameter tuning is costly unreliable. Finally, the white-box nature with closed-fome updates makes both variants attractive for interpretabillty-erttie applications such as medical diagnostics, scientific discovery, and regulatory settings where model transparency is mandatory.
12

---

Published as a conference paper at ICLR 2026
Where Multi-ReduNNt does not improve. Salti-ReduNet(Lasstooo offers limited benefits in well-sampled regimes (m/d 2 1). On two oversampled datasets, Iris (m/d = 26.3) and Mice Protein (m/d 9.8), its mean accuracy ReduNet: by about 0.1 percentage points on Iris and 5.7 points on Mice Protein, while training time is roughly 1.1 x that of ReduNet on average, yielding meaningful computational advantage. Multi-ReduNet is also unsuitable for complex natural images such as CIFAR-10, where spatial structure, color, and texture are crucial. All ReduNet variants, including ours, operate on vectorized inputs; applying them to CIFAR-10 requires grayscale conversion and flattening, which discard most spatial and color information and limit accuracy to around 26%, far below standard CNN baselines. CIFAR-10 10 instead requires convolutional architectures that exploit locality, which our framework does not provide. Finally, very deep networks are problematic: preliminary experiments (Tables [ 14 indicate optimization instabilities for depths I, > 20, likely due to vanishing gradients in the forward-only update scheme. This depth limitation is inherited from ReduNet and remains an open challenge for MCR MCR2-based white-box networks.
1 {0.01, 0.05, 0.1, 0.5) is slightly worse than
A.2 EMPIRICAL EVIDENCE FOR NON-IMPROVEMENT REGIMES
This subsection reports the concrete numbers (see Table underlying the regimes discussed in Section | "Where Multi-ReduNet does not improve," including oversampled tabular data (Iris, Mice Protein) and complex natural images (CIFAR-I0)
Datasets.
Tris is small classic tabular dataset with d 4 features and 150 samples from three classes. We randomly split the data into 80% training and 20% test, yielding well-sampled regime with m/d 26.3.
Mice Protein contains levels of 77 proteins measured in the cerebral cortex for 8 classes of control and Down syndrome mice exposed to contextual fear conditioning, with d and m 1080 samples. We again use an 80/20 random train-test split, giving m/d = 9.8.
CIFAR-10 consists of 32 32 RGB natural images from classes. To simulate an undersampled but structurally complex setting, we randomly select 800 training and 200 test images per class. Because all ReduNet variants operate on vector inputs, we convert each image to grayscale, resize to 100 100, and flatten it into d 10,000-dimensionall vector before training.
A.3 COMPARISON TO BLACK- -Box MODELS
We intentionally focus comparisons ReduNet and its variants (imp- Renp-RedNNet, RF-based variants) rather than black-box deep learning models (ResNets, Transformers) because:
Different design goals: Multi-ReduNet prioritizes interpretabilityy and theoretical grounding (MCR² principle) over raw accuracy. Black-box models sacrifice explainability for performance.
Computational regime mismatch: Black-box models require large datasets (m >> d) and GPUs. Our method targets tabular, undersampled regimes (m & d) where black-box models often over- fit.
Fair comparison: Comparing ReduNet isolates the contribution of class-wise decomposi- tion. Comparing to ResNets would conflate architectural differences (convolutional vs. fully- connected) with our theoretical contribution.
However, we acknowledge that for practitioners prioritizing accuracy over interpretability, black-box models may achieve higher performance on image datasets (MNIST,
CIFAR-10).
A.4 LIMITATIONS OF CURRENT THEERETICAL ANALYSIS
Class-otthogonalityy assumption. Theorem 2 uses class class-orthogonality as a property of the global optimum. While Theorem I shows that any global maximizer of the MCR² objective is class- orthogonal, our practical implementations (including the original ReduNet) optimize a parameter- ized, iterative approximation and never enforce (?')T Z
hard constraint. As result,
13

---

Published as a conference paper at ICLR 2026
Table 6: Quantitative results in non-beneficial regimes (Iris, Mice Protein, CIFAR-10).
= 0.5
17 = 0.1
Iris
SVM
KNN
NSC
SVM
KNN
NSC
ReduNet
0.978
1.000
0.844
0.911
1.000
0.822
0.911
1.000
0.889
0.911
1.000
0.867
Belti-ReduNet-LN
0.911
1.000
0.889
0.911
1.000
0.867
= 0.05
0.01
SVM
KNN
NSC
SVM
KNN
NSC
ReduNet
0.911
1.000
0.844
0.911
1.000
0.844
0.911
1.000
0.844
0.911
1.000
0.844
Multi-ReduNet-LN
0.911
1.000
0.844
0.911
1.000
0.844
global PCA+SVM
0.911
class-wise PCA+NSC
0.867
LDA
0.956
= 0.5
T) = 0.1
Mice
SVM
KNN
NSC
SVM
KNN
NSC
ReduNet
0.929
0.948
0.929
0.824
0.985
0,938
Multi-ReduNet
0.611
0.935
0.836
0.744
0.966
0.892
Multi-ReduNNtt-NN
0.605
0.938
0.833
0.744
0.966
0.892
" 0.05
T/ = 0.01
SVM
KNN
NSC
SVM
KNN
NSC
ReduNet
0.815
0.978
0.938
0.790
0.978
0.904
Multi-ReduNer
0.756
0.978
0.898
0.775
0.978
0.904
Multi-ReduNet-NN
0.756
0.978
0.898
0.775
0.978
0,904
global PCA+SVM
0.605
class-wise PCA+NSC
1.000
LDA
0.975
1 == 0.5
1) = 0.1
CIFAR-10
SVM
KNN
NSC
SVM
KNN
NSC
ReduNet
0.133
0.182
0.176
0.300
0.312
0.235
Multi-ReduNet
0.187
0.250
0.197
0.237
0.277
0.223
Relti-ReduNee-LNN
0.188
0.276
0.211
0.237
0.283
0.223
11 0.05
17 0.01
SVM
KNN
NSC
SVM
KNN
NSC
ReduNet
0,289
0.308
0.234
0.271
0.308
0.231
Multi-ReduNa
0.248
0.291
0.226
0.265
0.305
0.230
Multi-ReduNer
0.245
0.291
0.226
0.265
0.305
0.230
global PCA+SVM
0.276
wlass-wise PCA+NSC
0.265
LDA
0.125
the learned representations are best approximately orthogonal, with deviations that depend on the data and optimization dynamics, and providing convergence guarantees that relate these practical training procedures the ideal class-orthogonal solution remains open.
Frobenius norm constraint. The sphere projection Psa-1 enforces the strict constraint || m, at each normalized layer. This equality may be overly rigid in some regimes: empirically, the variant that relaxes intermediate normalization and only enforces it at the last layer Reduui-ReduNet- LastNorm) often matches or slightly improves the performance of Multi-ReduNet.. This suggests that softer layer-dependent normalization schemes is left for future work.
control could be beneficial, and systematic study of alternative
LastNorm variant: ReduNee--NNNNNteeeeee justification. We hypothesize that deferring normalization reduces gradient interference across lay- ers, but formal analysis is needed.
superior robustness (Table 5) lacks theoretical
14

---

Published as a conference paper at ICLR 2026
A.5
RECOMMENDATIONS FOR PRACTITIONERS
Based on our empirical findings, we recommend:
Use Multi-ReduNNet-LsttNooo when:
Data is undersampled (m/d d < 1)
Interpretability is required (medical, scientific, regulatory domains)
Hyperparameter tuning budget is limited (use default X, !))
Training time is bottleneck on undersampled datasets
Stick with ReduNet when:
Dataset is well-sampled (m//d 2 1)
Only interested in baseline MCR² performance
Avoid both methods when:
Working with natural images requiring convolutional structure (use CNNs)
Prioritizing accuracy over interpretability (use ensemble methods, deep learning)
Data has strong spatial/temporal correlations (use RNNs, GNNs)
A.6 FAILURE MODE ANALYSIS: WHEN CLASS- WISE DECOMPOSITION HURTS
The results Table o the intuition that class-wise decomposition is most beneficial when m < d. Once each class has many samples, estimating the global covariance is no longer ill- conditioned, and the advantages of the decomposition disappear can even hurt performance. This section provides theoretical intuition for these failure modes.
Theoretical intuition.
The MCR2 objective seeks to maximize:
K
1 log det(I + aZzz
1
R(Z, (2,
-
Σ
y log det(I a (112
2
In the undersampled regime (m < d):
The global covariance _ E Rdxd is rank-deficient (rank S. m < d), making the d'x d inversion (I + aZZ ) numerically unstable and computationally expensive (O(d))).
Class-wise covariances ZTP2 TT have even lower rank (S my < m), but the Woodbury identity allows us to invert smaller my X mj matrices instead, reducing complexity to OEE, mi) (dd))
When classes imbalanced, some my very small, making per-class optimization highly stable and fast.
In the oversampled regime (m d):
The global covariance ZZ well-conditioned and its inversion no longer a computational
bottleneck.
The Woodbury identity provides computational advantage: O(m²) Old)) when > d.
Class-wise decomposition introduces overhead: we now solve K separate subproblems instead of a single well-conditioned global problem, while their solutions remain implicitly coupled through the global representation geometry.
Loss of global structure: When my is large for all j. the global covariance ZZ captures rich inter-class relationships. Decomposing into K independent problems discards this information, leading to suboptimal feature learning.
Empirical validation. demontttessssssttes three failure modes where performs ReduNet and simple baselines.
under-
15

---

Published as a conference paper at ICLR 2026
Failure Mode 1: Oversampled, low-dimensional data (Iris, m/d 26.3). Why Multi- ReduNet fails:
With only d = 4 dimensions, the global 4 x 4 covariance matrix is trivial to invert (O = 64 ops).
Class-wise decomposition provides zero computational benefit: O = 3 35 128,625 > 64.
Global PCA SVM achieves identical performance (91.1%), confirming the problem is simple enough for linear methods.
Recommendation: For d < 10 and m/d > 1, use standard ReduNet or simple linear baselines +
Failure Mode 2: Oversampled, moderate-dimensionnal data (Mice Protein, m/d = 9.8). Why Multi-ReduNet fails (average 5.7pp))
Despite class imbalance, most classes have sufficient samples (my/d 1) that global covariance estimation is stable.
The 5.7pp degradation is significant, suggesting that decomposing the objective actively hurts feature learning when m/d is moderately large.
Loss of inter-class structure: Protein expression data has rich biological correlations across classes (e.g., proteins in related pathways). Class-wise decomposition discards these global de- pendencies.
Recommendation: For biological/medical data with m/d and rich global structure, use Re- duNet without decomposition or ensemble methods that preserve cross-class relationships as much as possible.
Failure Mode 3: Complex images with spatial structure (CIFAR-10, m/d 0.8). Why both ReduNet and Multi-ReduNet fail:
The best average accuracy of ReduNet achieves only 28.2%, Multi-ReduNer(LLN) 26.7%, both far below CNN baseline (41.2%, + 14.5pp gap).
The MCR² framework is fundamentally fully-connected, it treats all features as exchangeable,
ignoring 2D spatial locality.
Even though m/ d 0.8 is undersampled, the problem structure requires convolutional inductive bias, not low-rank decomposition.
Multi Mului-ReduNet's 1.5pp degradation over ReduNet suggests class wise decomposition provides no additional benefit when the fundamental architecture is mismatched.
Recommendation: For complex image data, always use convolutional architectures (CNNs, ViTs), ReduNet and Multi-ReduNet are more suitable for tabular data where features are semantically independent.
Summary of failure modes.
Oversampling (m/d > 1): Global covariance well-conditioned; class-wise decomposition adds overhead without benefit. Use standard ReduNet or linear baselines.
Spatial/structural data: Fully-connected architectures destroy spatial locality. Use CNNs for images, GNNs for graphs, RNNs for sequences, regardless of m/d ratio.
When Multi-ReduNet excels: m/d < 1, class imbalance (max, m3/ miny my > 3), tabu-
lar/sparse features, interpretabillty-critte domains.
These results validate our honest scope definition: Multi-ReduNee not a universal improvement, but targeted solution for undersampling.
16

---

Published as a conference paper at ICLR 2026
A.7 SUMMARY
Melti-ReduNNet does nor uniformly outperform ReduNet across all sampling regimes. Rather, it is specifically designed for undersampled settings, where computational efficiency and robustness to hyperparameter choices of primary importance. The regime-dependent behavior observed in (Table 3). together with the failure cases (Iris, Mice Protein, CIFAR-10), delineates the scope of applicability of the proposed class-wise decomposition and clarify the conditions under which its advantages diminish. Extending the present analysis to convolutional architectures for raster data and providing deeper theoretical characterization of the LastNorm variant's robustness remain interesting open directions.
B TRAINING TIME ANALYSIS
We empirically validate the theoretical computational complexity advantages of -ulti-ReduNNet(- LastNorm) by measuring wall-clock training time across all datasets. Table (7 reports the average time(in seconds) for (imp-)ReduNer (inp-)ReduNet-RF and Multi-ReduNNt( (aaeduuNe(LLLssNoom)) LastNorm) on 10, 15, 20, 25 layer networks.
Table 7: Empirical training time (s) comparison: (imp~)ReduNet vs (imp-)ReduNet-RF Multi- ReduNet(-L on different layer networks.
Dataset
Layer
RN
imp-RN
MR
MR-LN
RN-RE
imp-RN-RF
199.67
114.37
105.94
97.59
808.68
708.96
10
587.19
253.01
192.63
188.32
1810.85
1689.20
Reuters
15
636.02
368.33
308.30
304.35
2919.57
2622.49
20
910.04
512.90
386.63
377.94
3901.84
3434.01
25
1060.29
572.90
515.02
445.66
4396.90
4083.99
5
166.24
142.12
65.43
64.30
637.82
531.14
10
246.90
237.87
231.38
231.27
1223.27
1137.50
MNIST
15
289.82
279.97
280.56
276.14
1766.94
1577.26
20
364.51
344.81
336.09
320.04
2381.05
2174.87
25
605.86
496.89
395.99
348.25
3260.24
2856.73
5
101.34
76.68
37.71
28.09
593.73
574.40
10
234.04
177.46
141.76
140.85
1216.12
1136.85
Fashion-
15
311.42
193.51
185.41
178.37
1639.36
1568.50
MNIST
20
352.97
254.42
211.74
211.40
2203.54
2106.88
25
417.63
272.43
259.41
231.76
2975.36
2833.73
5
6.02
4.80
1.33
1.16
90.03
83.81
10
6.44
5.75
3.15
3.00
170.43
160.27
Swarm
15
8.66
7.25
5.80
4.00
248.69
231.81
Behaviour
20
13.02
11.44
10.57
7.67
334.50
316.63
25
19.02
18.80
15.35
13.28
451.37
431.56
7.70
7.04
7.02
6.44
40.99
35.32
10
11.53
9.61
9.08
8.61
80.27
76.58
DrivFace
15
14.19
12.58
11.79
11.48
122.89
111.11
20
20.26
14.09
13.47
12.34
149.13
144.15
25
28.02
24.39
16.75
15.05
215.26
185.21
17.99
8.70
8.35
7.94
30.10
20.29
10
27.29
21.41
12.37
12.06
50.70
46.73
ARCENE
15
38.54
25.55
15.57
14.33
82.46
50.05
20
50.58
30.03
21.15
16.91
108.04
68.31
25
62.30
47.89
28.08
24.23
128.10
83,91
Key observations:
Consistent speedup across all datasets: Multi-ReduNet- Lastoor achieves 1.4-2,6x speedup, with average of 2.0x across all six benchmark datasets.
17

---

Published as a conference paper at ICLR 2026
Highest gains in extreme undersampling: Reuters (m/d 0.28, speedup 2.2x) and ARCENE (m/d= 0.016, speedup 2.6x) exhibit the largest improvements, validating that class-wise decomposition maximally exploits low-rank structure when m < d.
Moderate gains balanced datasets: MNIST and -NIIIT (m/d = 0.5, speedups of 1.5x and 2.0x) show smaller but still substantial efficiency gains, indicating that the method remains beneficial even in moderately undersampled, balanced settings.
Scalability for deep networks. While the relative speedup remains in the 1.4-2.6x range, the absolute wall-clock time gap between ReduNet and with network depth L. Figure in the main text shows that this gap becomes especially large for L E (15, 20, 25).
grows
While the empirical speedup is far below the theoretical prediction, a 2.0 average improvement still translates to substantial wall-clock savings for practitioners training deep networks on undersampled data. For instance, a ReduNet model that requires 10 hours of training can be reduced to ~5 hours with feasible.
Practical implications:
Lorm,,,,,, making iterative experimentation more
C PROOF OF LEMMA (WOODBURY IDENTITY)
The Woodbury matrix identity is fundamental result in linear algebra that provides an efficient formula for computing the inverse of low-rank matrix updates.
Lemma (Woodbury Identity). For any a IR and X gdxm.
(1+ a = aX(1 + ax ax X)
where the left side requires inverting d matrix, while the right side only requires inverting an m x matrix.
Proof. The identity follows directly from the Shernaa--Morison-Woodbu formula. We verify the identity by multiplying both sides by (I + axXT): )
(1 (1+ (+xxx)) -X x
-
=I+aXXT -oX(++XXxxxxxx - X) + axx -X-XXXXX-XX
=I+aXX- - -I +ax aa x
=1+aXXT -
This confirms the identity. For computational applications, this reduces the inversion complexity from (dd) to (m3) when m & d.
D THEOREM PROOFS
D.1 PROOF OF COROLLARY (DETERMINANT INEQUALITY)
To prove Theorem 1, we rely on the following linear- algebbaic result:
Corollary (Determinant Inequality). Let (A₃ X,X } be collection of symmetric positive semi-definite matrices. Then:
K Aj
K det (T + Az),
det.
1+
<
1=1
1=1
with equality if and only if (X,) X X,
for all
Proof. Base case K =
18

---

Published as a conference paper at ICLR 2026
By det(I + AB) det(I + BA) for any matrices A, B Raxd
dot(I A1 + A2) dct(/ + X + x2xx))
del(/ + Xi x del( + (1 (xxx
(11)
dee(T + X1 X det X2 (T+ x
Since X1X))- / in the Loewner order (because 1 + X ž I), it follows that
x2 (1, 1 X2 X2:
and by monotonicity of det(( +.).
- det xx)
+
= det(l + X2x,))
(12)
By (11), (12),
det(1 + A1 + A2) de + del + A2).
(13)
Since / - X,[ (I - (1+ XIx) equality of (12)
holds if and only (XX]
By 1 -(--==
and (1 (1 - X1 = x
(1 xX, ( X1X,) "x, X₂
= = -xxxXx++++xxx=-- = =0
if and only if X X2 = 0.
follows that equality of (13) holds if and only if x X₂ = (or A142 0).
Thus the corollary holds for K = 2. Inductive step
Assume the statement holds for n 2 2. Set S₁ = + Σラ= Aj (> 0).
Apply Matrix Determinant Lemma,
n+1 Σ /=1
det(I +
A,) = det(S,, + An+1)
= det(S,, + Xn+1X
det(S,,) det(( + X S-1 Xn+1).
Because Sn 2 7, we have S 3 T hence
det + XT ( S det + xn+1) det(1 + Ant1).
Xn+1)=
(14)
Multiplying yields the desired inequality for n + 1.
Equality overall forces equality in both places:
1. From the monotonicity step: equality of (14) holds if and only if
xT x++ - 1 = 0.
By the "zero test", (I- S,,") 0, i.c S Xn,1 X
Multiplying by S, gives
(Sn, 11x =0 +
Az)X 0.
(15)
19

---

Published as a conference paper at ICLR 2026
Multiplying (15) by X,+1 gives
E 1 -.
Each summand is = hence each equals O; by the "zero test",
X An = x X Xa+1 = (X) Xa+1) (X] Xn+i) for all S n
i..
xj X for all j S n.
(16)
2. From the inductive hypothesis, equality of det(S.)) forces x X, for all <i<j n. Combining (16) with the inductive equality condition yields
x X5 0 for all si<jenti
Conversely, if X X, for all i + j. then
n+1
(I+AI) ... -1+ = I+
Σ
A
(since every mixed product A,, ... -A, with 2 2 vanishes),
j=1
so
#+1 II Ø=1
n+1
n+1
det + Aj) = det
II j=1
(T + (z)) det(T +
Σ Aj);
==1
and equality holds.
This completes the induction and the proof.
D.2 PROOF OF THEOREM 1 (CLAsS-ORTHOGONAAI))
Theorem 3 (Restatement of Theorem D.. At any local optimum 7" the MCR objective equation
under the constraints 1 I| my ,rank(2 ([') d for all j 1,..., K, and Ed, d3 S d, the class-wise representations satisfy
(Z")TZ" =0 for all + j.
where Z TTI denotes the feutures of class j.
Proof. We prove by contradiction using Corollary
Step 1: Assume non-orthogunality. Suppose for contradiction that Z is a local optimum with (Z^h)² na # 0 for some classes # J2. By Corollary
I. the global coding rate satisfies:
r+ (r+a
log det
log det aZ
(17)
with strict inequality due to the class overlap (Z*h) 3 + 0.
Step 2: Construct orthogonal alternative. We construct an alternative solution Z' by orthogonalizing the class partitions. For each Z = (V (SVD decomposition), con- struct orthogonal matrices { such that [U Ukk has orthogonal columns. This is possi-
ble since Σø= rank(2*) < Σx
1 dj < d. Define:
z = (V*).
By construction, (Z")TZZ) = for * j. and each ğ preserves the singular values of Zaj, hence satisfies all constraints.
20

---

Published as a conference paper at ICLR 2026
Step 3: Show strict improvement. Since Z has the same singular values as Z each per-class coding rate is preserved:
(I (I + (z²)) = log del (1 + (Z
However, by Corollary with equality condition, the orthogonality of } implies:
log det I ++a E
log det (Z))
Combining these, Z' achieves strictly higher objective value than Z", contradicting the optimality
of Z
Therefore, any local optimum must satisfy class-orthogonality.
D.3 PROOF OF THEOREM 2 (DECOMPOSITION EQUIVALENCE)
Theorem (Restatement of Theorem 2. Assume that the per-class representations Z E Rddm, are mutually orthogonal and ΣØ= my = m S d, the global MCR² objective is equivalent to the sum of K independent class-wise objectives: d
log det (1 (1+ me
((Z))T)
mi m
log det (1 (1 mye² 23 (2))T)]
d
max 2i
(18)
2
subject to || 1| = my.
Proof. Let
AR(Z) = 2 log det + azz")
1
Σ j=1
Yj log det (1+0)
where Z is the class partition of Z. And denote the global MCR objective in equation
( == log det (T (Z))
" log det 7 ('0)
denote the j-th class-wise objective in equation 18 and me? Ojj
2 79
my Let
m
K
U1 max AR(Z),
02
max (2)
2
be the optimal values of the global and class-wise problems, respectively.
Direction 1 (V2 S vi). Let [ be maximizers of the K class-wise objectives, and set Z' = [z" ..... k By the Frobenius constraints in the theorem statement, { class-wise problems and Z' is feasible for the global problem.
) is feasible for the
By the orthogonality assumption in the theorem statement, we can apply Corollary I in the equality case with Aj aZd (Z") that gives
K
log ( + + = = log det (r Σ (Z()T) =
Σ
log det + (zU))
j=1
Hence the global objective value Z' decomposes
1 Σ Y (T (''
AR(Z))= log det (T -
log det + +77" 7 log det (2)))
=
V2.
Since Z' is a feasible point for the global problem, obtain U2 AR(7') maxz AR(Z) f1-
21

---

Published as a conference paper at ICLR 2026
Direction 2 (U1 S V2). Let Z be global maximizer of AR. By Theorem || Z satisfies class- orthogonality, i.c. (2 Z = 0 for all + J. and each class-wise block z satisfies the Frobe-
nius constraint || mj. Thus { J== is feasible for the class-wise problems.
Applying Corollary aggain with A, aI 2 (Z and using the equality condition, we obtain
K
log det (1 (1 (1+aZ*2+T) = log det (1
Σ
brd (ri
Σ log det (1 (+z"
j=1
and therefore
AR(Z*) log dc [I + =z
Σ j=l
Y log + a2,
det([ (Z))) 7 log deg + (2)
K (R1(Z")
==1
K
$ max A AR, (Z') = 1/-
2
Since 01 (R",") this yields 11 S 12-
Combining the two directions gives 01 establishing the equivalence of the global and class- wise formulations.
D.4 PROOF COMPARISON: OUR APPROACH vs. PRIOR WORK
We now compare our proof strategy for Theorem (class-orthn catas--othooonaaltt)) with the approach used in prior work (Chan et al. 2022).
Proof strategy in Chan et al. (002)).
Chan et al. establish class-orthogonality through an analysis of the strict concavity of the log det (-) function. Their argument proceeds by deriving lower and upper bounds on the coding rate and the coding rate reduction, and then characterizing the equality conditions under which these bounds attained. Class-orthogonality follows from showing that any overlap between class-wise representa- tions leads to a strictly suboptimal value of the coding rate reduction. This approach is rigorous and general, but relies on a layered sequence of technical lemmas and bounding arguments.
Our proof strategy.
In contrast, our proof follows a purely algeraa route based on a determinant inequality (Corollary [ with explicit equality condition. The key observation is that for objectives of the form
1+ [+
log det
any non-orthogonal decomposition the positive semidefinite components {A,} necessarily yields strictly smaller value than orthogonal one. Leveraging this property, class-orthogonality is established via a short contradiction argument; assuming overlap between class representations, we construct alternative orthogonal solution that preserves all per-class constraints but achieves strictly larger objective value.
Discussion.
The determinant inequality in Corollary I issolatss the algebraic core underlying class- Class-orthogonality in MCR² Unlike concavity-band -based arguments, the present proof does not depend differential
22

---

Published as a conference paper at ICLR 2026
properties of the objective and makes the role of orthogonality explicit through a factorization dition of the determinant. This perspective also facilitates extensions to structured variants of MCR?, and normalized formulations considered in this work.
including the
More broadly, Corollary applies to any learning objective involving log det(1 + Σj Aj) with positive semidefinite components, such as SCoRe-LogDet (Majee et al., 2024), and may therefore be useful beyond the specific context of Nulti-ReduNNet
E GRADIENT DERIVATIONS AND ALGORITHM DETAILS
This appendix section provides step-by-step derivations of the gradient update equations used in Multi-ReduNet (Equations 4-5 in the main text), and eigenvalue spectrum analysis justifying the design.
E.1 STEP-BY- BY-SEP GRADIENT DERIVATIONS FOR MULTI-REDUNET UPDATES
We derive the closed-form gradient updates for the class-wise MCR objective used in Multi- ReduNet. Recall that for each class j E (I,..., K.). independently maximize:
R2(Z") 1 log det (1+ (2 (2Z))²
my log det (1+ (+==221111)
me²
subject to || = my, where Z E [Rdxm, contains features for class j samples.
Step 1: Gradient of the Global Coding Rate Term
The global coding rate contribution from class j is:
Rd global
1 log det (1 + (7))7,
d
2
mc²
Using the matrix calculus identity
a log [eee[++XX]] 2(1 + XX X, we obtain:
ox
0 0Rggotal 0Zi
1
20(T + (
2
= + at aZz
where F. aTT (Z))))- is the expansion operator for class j.
Step 2: Apply Woodbury Identity to Reduce Complexity
Direct computation of E requires inverting d matrix. By Lemma (Woodbury identity):
(I+ aZ =I azz (I a(Z)) (Z)
where the right-hand side only requires inverting an m, m, matrix (I + a(Z)) This reduces complexity from O(d)) (mm)).
Define preE/ = (I + az z -1 E <m Then:
El = a (I - az pref (z))T)
Step 3: Gradient of the Per-Class Coding Rate Term
The per-class coding rate term is:
RJ
mi 2m
log det (T (2))))
d
class
aj m
23

---

Published as a conference paper at ICLR 2026
Following the same matrix calculus rule:
òR Ozi
mj 2z
2m
m ag (1 + (1(+t,3(2))7)-1
m
= 7'
where "i = mi (class prior) and C = ag( + 02 (z)) is the compression operator for
class
Step 4: Apply Woodbury Identity to Compression Operator
Similarly, define preC/ = (1 + 0zg (2) 7/) E mR ; Then:
C = aj (I - af preC (2
Step 5: Combined Gradient Update
The gradient of the full objective R, = R Riglobal
Relsss is:
V vz,R = B zi 7/ C
Applying projected gradient ascent with learning rate 77 and projection onto the unit sphere Ps4- 1-):
44 Psa
(z) + -7,
7,04Z))
For Mulli-RedduNet-Lsstee the sphere projection is omitted at intermediate layers and applied only at the final layer L:
lcL. Psa-1 ( + - 1= L.
Z + -,
1/41
This completes the derivation of Equations 4-5 in the main text.
E.2 EIGENVALUUE, SPECTRUM ANALYSIS: JUSTIFYING MUITERRDUNET-LASNNOOR
We now provide numerical evidence for why per-layer normalization (Multi-ReduNee) may be overly restrictive compared to last-layer-only normalization Multi-RechuNet-Lastoonn))
Observation: In Multi-ReduNet, each layer / projects updated features onto the unit sphere: ||2/2,,ll2 1 for all samples i in class j. This enforces uniform feature norms across layers, which may conflict with the natural gradient dynamics.
Analysis: Consider the eigenvalue spectrum of the gradient update matrix G = E/ -, If the eigenvalues of G vary significantly across dimensions, forcing all features to have unit norm after each layer may distort the learned representations.
Interpretation:
The gradient matrix G/ has non-uniform eigenvalue spectrum, indicating that different feature dimensions evolve at different rates during optimization.
Per-layer normalization (P34-1 after every layer) forces all dimensions have unit magnitude, potentially suppressing the natural dynamics encoded in the eigenvalues.
Last-layer-only normalization allows intermediate representations evolve freely according to their natural gradient scales, only enforcing the Frobenius constraint | my at the final output layer.
Empirical Validation:
Table 5 shows that Multi ReduNet--LastNom achieves average 2.6% better hyperparameter robust- ness across datasets compared to Multi-ReduNet. supporting the hypothesis that relaxing interme- diate normalization improves optimization stability.
24

---

Published as a conference paper at ICLR 2026
F ACCURACY OF REDUNET VARIANTS ACROSS 7) AND DATASETS
In the main text, we report results either at a fixed learning rate = 0.05 (Table 3 or using the best test accuracy over small grid r E {0.01, 0.05, 0.1, 0.53 (Tables 4p. For completeness, the full accuracy results across all datasets and all four learning rates with a fixed depth I. = 5 are reported in T These results indicate that Multi-ReduNett ReeduNet(-LastNorm) exhibits more stable performance and achieves accuracy par with superior to ReduNet across 7/.
Table 8: Full test accuracy of ReduNet variants across all datasets and learning rates 1) {0.01, 0.05, 0.5..
0.5
17 = 0.1
Reuters
SVM
KNN
NSC
SVM
KNN
NSC
ReduNet 0.073
0.127
0.281
0.465
0.721
0.747
Multi-ReduNer
0.955
0.603
0.941
0.984
0.930
0.957
0.956
0.709
0.946
0.986
0.941
0.958
11 0.05
1) 0.01
SVM
KNN
NSC
SVM
KNN
NSC
ReduNet
0.802
0,670
0.922
0.956
0.878
0.949
Multi-ReduNet
0,984
0.939
0.957
0.988
0.949
0.957
Kulti-ReduNNttLN
0.985
0.943
0.957
0.988
0.950
0.957
global PCA+SVM
0.975
class-wise PCA+NSC
0.867
LDA
0.471
= 0.5
17 = 0.1
MNIST
SVM
KNN
NSC
SVM
KNN
NSC
ReduNet
0.074
0.019
0.020
0.901
0.913
0.908
Multi-ReduNet
0.414
0.655
0.361
0.797
0.878
0.869
Multi-ReduNet-LN
0.518
0.720
0.653
0.815
0.890
0.880
= 0.05
0.01
SVM
KNN
NSC
SVM
KNN
NSC
ReduNet
0.906
0.930
0.903
0.894
0.937
0.897
Melti-ReduNet
0.837
0.902
0.871
0.885
0.926
0.897
Multi-ReddNet-LN
0.842
0.903
0.873
0.905
0.926
0.909
global PCA+SVM
0.878
class-wise PCA+NSC
0.773
LDA
0.615
7) = 0.5
1) = 0.1
PaSHion-MNIST
SVM
KNN
NSC
SVM
KNN
NSC
ReduNet
0.137
0.141
0.073
0.824
0.812
0.841
Multi-ReduNe
0.369
0.738
0.402
0.744
0.762
0.778
Multi-ReduNNt-LN
0.371
0.776
0.584
0.749
0.773
0.798
1 0.05
1) = 0.01
SVM
KNN
NSC
SVM
KNN
NSC
ReduNet
0.858
0.826
0.836
0.852
0.826
0.836
Multi-ReduNer
0.798
0.790
0.800
0.845
0.813
0.831
Multi-RRedNNet-LN
0.801
0.802
0.803
0.858
0.828
0.835
global PCA+SVM
0.829
class-wise PCA+NSC
0.667
LDA
0.781
25

---

Published as a conference paper at ICLR 2026
Table continued
= 0.5
17 = 0.1
Swarm Behaviour
SVM
KNN
NSC
SVM
KNN
NSC
ReduNet
0.679
0.567
0.601
0.802
0.981
0.996
0.990
0.863
0.738
1.000
0.998
0.896
Retti-ReduNetLLN
0.990
0.867
0.765
1.000
0.998
0.896
= 0.05
0.01
SVM
KNN
NSC
SVM
KNN
NSC
ReduNet
0,802
1.000
0.996
1.000
1.000
0.979
1.000
1.000
0.929
1.000
1.000
0.956
Multi-ReduNet-LN
1.000
1.000
0.927
1.000
1.000
0.977
global PCA+SVM
1.000
class-wise PCA+NSC
0.913
LDA
0.977
= 0.5
T) = 0.1
DrivFace
SVM
KNN
NSC
SVM
KNN
NSC
ReduNet
0.295
0.098
0.104
0.219
0.235
0.169
Multi-ReduNet
0.820
0.978
0.852
1.000
0.918
0.984
Multi-ReduNet-LN
0.869
0.967
0.907
1.000
0.940
0.984
" = 0.05
T = 0.01
SVM
KNN
NSC
SVM
KNN
NSC
ReduNet
0.432
0.393
0.366
1.000
1.000
1.000
Multi-ReduNer
1.000
0.951
0.995
1.000
1.000
1.000
Mulli-ReduNet-LN
1.000
0.978
0.995
1.000
1.000
1.000
global PCA+SVM
1.000
class-wise PCA+NSC
1.000
LDA
1.000
") 0.5
1) 0.1
ARCENE
SVM
KNN
NSC
SVM
KNN
NSC
ReduNet
0.366
0.341
0.220
0.439
0.439
0.561
Multi-ReduNeet
0.561
0.659
0.732
0.683
0.683
0.829
Mulli-ReduNet-LN
0.561
0.683
0.829
0.805
0.732
0.805
11 0.05
17 0.01
SVM
KNN
NSC
SVM
KNN
NSC
ReduNet
0.439
0.415
0.463
0.341
0.707
0.780
Multi-ReduNe
0.829
0.732
0.805
0.829
0.780
0.829
Multi-RRedNNe-LNN
0.829
0.732
0.805
0.829
0.780
0.829
global PCA+SVM
0.805
class-wise PCA+NSC
0.756
LDA
0.878
G TRAINING AND EVALUATION PROCEDURES
G.1 REDUNET
Let (at y C Rd X [K] denote labeled training samples. For convenience, denote 11 (II' E Rmxm) K Jj=1 as set of diagonal matrices whose diagonal entries represent the membership of samples in K classes: (TT),, if y² j, and otherwise. Given the distortion e, ReduNet aims to learn interpretable features 2t via iterative gradient updates on maximizing a coding-rate based objective:
AR c) R(Z,e) (Z,
log det(( + d
27
tr(
log det(1 ir(li))22 2 (z)))))
d
me²
2m
26

---

Published as a conference paper at ICLR 2026
where Z ZII denotes features of class-j samples, AR, é) is the coding rate reduction. Features are updated layer-wise using closed-form statistics derived from previous representations Z₁. See Algorithm 2 for details. In essence, each layer computes per-class compression operators {C) j=1 and global expansion operator Ej from current features Z!:
dlogdet(I + aZZ
a aZiZZ -
dz
E.
tr(IP) dlogdet (1I + a (u,ZIlZ''
tr(IP) aj( + aj Z - -,TT',
2m
dz
Z1
m
and performs projected gradient update using their discrepancy. These updates are fully transparent and closed-form, making ReduNet interpretable by design.
Evaluation follows similar layer-wise procedure but omits gradient-based updates. At each layer, the learned compression operators Cf are used compute soft class attribution probabilities A²:
exp(-A1(7==1) # (2)) Σ exp((>IC)all)
where Cf approximates projection of 2 onto the orthogonal complement of class-j's subspace. This inference strategy is foundational to ReduNet and forms the basis for our class-wise infer- ence scheme in Multi-ReduNet (Section 3.3). The complete training and evaluation procedures summarized in Algorithm 2 and Algorithm 3, respectively.
We denote P (-) as the projection operator onto the d-dimensional unit sphere. It enforces that updated features reside on the sphere, which normalizes their magnitudes and enhances stability.
Algorithm 2 Forward Training Algorithm of ReduNet
Require: Input data = [x: xx .... _x E Raxmm X, 6, II, learning rate 7).
1: m {az- ,र( 1=1- {7g tr(I')
m j==1-
2: Initialize Z₁
3: for I = 1,,, I....IIa. do
4: Ej = ==a(I + =a[ 11 (0) =a 2,11 1
5: 1o,1 ,m do
6:
(*)) Σラ exp (-XICc 15=1
exp (-xl c/2:1)
7: 2f Psa- (af + 17 (Eiz) E
8: end for
9: end for
K,L return features ZL+1, the learned parameters { {C) (3)
Algorithm 3 Evaluation Algorithm of ReduNet
L.K Require: Input E Idd, network parameters {Eife,, (C
by À and learning rate
7).
1: Initialize
2: for 1,... do
exp (-11C//=111 1K
3: (#5
ΣK cxp =111)/=111) 1==1
4: Z1+1 = Pan-
21 +1 (Era - (2)))
5: end for
6: return feature 21+1
27

---

Published as a conference paper at ICLR 2026
G.2 MULTI-REDUNET AND MULTI-REDUNET-
Although ReduNet computes per-class compression terms e its optimization is global. This assumes shared structure across classes and prevents fine-grained control over class-specific representations. And in real-world high-dimensional data settings, particularly under sample scarcity, ReduNet's global training mechanism becomes inefficient. Each layer requires comput- ing class-wise compression matrices Cý E I and a global expansion matrix Ej E Rdxd leading to expensive matrix inversions when feature dimension d is large. This hinders deployment on resource-constrainee platforms.
To address this, we propose Multi-ReduNet, which decomposes the global ReduNet objective MCR² into K' class-wise subproblems:
Σ max
K
tr(IP) log det (r+ mc² ()))))
log det (1+ tr(Ti)) (
d
Z ER ==1
2
m
subject to norm constraints. Each subproblem independently updates Z using projected gradient ascent, where gradients are:
detg (1 + - (T
tI(P)
d log (e + "r(
2i (27))
Ei-1 x Zi + 1)
dzj
2m
dz/
d
d
2 (1)
zi
tr tr([I')
tr(l1))22 (1+ br(ll)22 2 (2)) 21).
d
d
17*
me²
me²
m
Note that Ey and Cf are both functions of the same class-wise covariance 2/ (2 T, but they arise from the expansion and compression log-det terms with different coefficients a and aj. The actual
update for class depends their difference (EV tr(1I)) C)) the two operators act as opposing forces (promoting global spread vs. within-class compactness) rather than as a simple rescaling. Crucially, in Multi-ReduNet these operators are computed from the class-wise covariance zi (Zt)² instead of the global covariance ZZT, which makes the optimization fully decoupled across classes.
m
Using Lemma 1, these matrix inverses can be computed efficiently via Woodbury identity. For training samples with known class membership, the features are updated by:
tr(IP) m
2 Psa (Z) t
During evaluation, since test labels are unknown, we compute soft membership scores by soft- max function #य. Then, unlike ReduNet, which only uses क to weigh compression terms, Multi- ReduNet performs full forward update within each class-specific subnetwork independently, ignoring A' during that step. These per-class updated features are finally aggregated using 4 as weights:
tr([T)) c22)) ++).
21+1 P
Σ 1.+n-(Efa
m
This forward scheme captures how confident the model is about test sample's class alignment, and allows each class branch to contribute accordingly.
To further reduce storage overhead, we adopt parameterized model storage: instead of saving all L. (2K) d x parameter matrices, we store only the learned features Z₁ per layer and reconstruct E/,C on-the-fly when needed.
Finally, introduce Malti-ReduNNe--Lstttoooo which differs by postponing unit-norm projection to the final layer. This provides more flexibility during intermediate optimization while ensuring fair comparison across classes at inference.
The complete training and evaluation procedures for Multi-ReduNer and Multi- LasNNet-- summarized in Algorithm& 4 and Algorithm respectively.
28

---

Published as a conference paper at ICLR 2026
Algorithm 4 Training Algorithm of Multi-ReduNer and Mulli-ReduNee-LLsttNoom
Require: Input data X Rddm, class memberships []=))=1>> parameters 0, X, learning rate
7)-
1: Compute class sizes: my = trll')),, priors my
2: Set a me. and 0j
d m,c" for j = 1, 1,... K
3: Initialize features: Z1 = X
4: for = 1 to L do
5: if - then
6: Extract class-wise inputs: {Z} = 2
7: end if
8: for j j to K do
9:
#Per-class forward update
Compute: preEf (1 a a(2i) 21)-1 Rm
11:
Compute: Fy = a(T - a7/ preFf. (
12:
Compute: preC = (I + (2))))) 2/) E RmyxXmy
Compute: Cj a.y (1 aj 2i preC/ (2))
14:
Update features:
2/41 (P. + -
(Multi-ReduNett
(Mult--eduNN::lastNNon))
15: end for
16: end for
17: if Multi-ReduNNe--lasttMee then
18: Apply PPa-- (.) all Z 1 for j 1, ..... K
19:
end if
K,++1 20: return features { 1,
priors {
H EXPERIMENTAL SETUP
H.1 EXPERIMENTS OF MULTI-REDUNET AND VARIANTS
We evaluate Melti-ReduNet and its LastNorm variant on six datasets spanning diverse modalities: Reuters (text), mnist (images), fashion-mmi (images), Swarm Behaviour (survey data), DrivFace (images), and ARCENE (medical diagnostics):
Reuters: the Reuters dataset is a commonly used text classification dataset and consists of a total of 135 document categories. For our experiments, we extracted data from the first five categories because these five categories have slightly more abundant sam- ples. The training set includes 5,304 samples, and the test set comprises 1,328 sam- ples. The dataset can be downloaded from http:/ /ttp://wwww.///... cad. 2 juu. edu. ww/hone//////eee dengcai/ wttpa////DDaaaeett.a..aa http//eeetaaaaaa....
MNIST: this is a widely used handwritten dataset in the field of machine learning, com- prising 70,000 grayscale images of size 28x28, representing the digits from 0 to 9. We randomly sampled 500 samples from each class of the data to form the training set and 100 samples from each class to form the test set. Additionally, each image sam- size of 100x100 and then flattened into a 10.000-dimensionall vec- tor. The dataset can be downloaded from https: www kaggle..g...... com/ datastss/ hojjatk/ hojjatk/mniiste /wt.//maaaataattttmeeeeeeeeeeeee
ple is reshaped
Fashion-MNIST: this dataset is dataset used for clothing image classification, containing 28x28 pixel images of clothing from 10 different categories. We randomly selected 500 samples from each class the data to form the training set and 100 samples from each class to form the test set. Each image is then rescaled to 100x 100 pixels and flattened into a 10.000-dimensional vector. The dataset can be downloaded from ht pettps: ///www. tensorf ow. www.oogggggaatteeeceoooooo log/ aash an_mniss
29

---

Published as a conference paper at ICLR 2026
Algorithm 5 Evaluation Algorithm of Multi-eeduNet and Multi- Mulli-ReduNet-LastNoon
Require: Input sample I R training features {2, = [Z,,..., 2,r1 E [dxm][+1 1++ hyperparam-
1: Compute 1-=1 K X, learning rate 7)
eters {7p
me² aj u(IP)e?
2: Set 21 = Ø
3: for = 1 to I do
-1 1+ a(2)((2)))) (1)
4: preE =
for = ,,...,K K
a) -1 for = ,,.... K
5: preC
E| = (I oZ) preff (21)
6:
7: CJ = æj a 2 preC/ (Z
8: Compute soft membership weights:
exp (-X1Ic/=11)
#/ (2))
[0,1]
Ei= K
exp (-1110/2ı11)
9: For each class j, compute tentative update:
2'4 = 21 + (F/ - 7/
10: Aggregate:
Psa-1 EE z1) 25 (z)))
if Multi-ReduNee
Σソ= 2'1t1 +5 (21),
if Mult--ReduNe-LastNorm
1I: end for Mult--RdduNet--assNNoo then
12: if
13: ZL+1 <- Ps--1 (21+1)
14: end if
15: return Final feature 2L+1
Swarm Behaviour: this dataset was obtained from an online survey run by UNSW, Aus- tralia. has 400-dimensional input features and 2 classes. We randomly extracted 1,200 samples for the training set and 300 samples for the test set. Detailed data information and download address are available at: hhttps://are
ics. uci edu/dataset/
524 wtwp///wwwwwbaii.....
DrivFace:
this dataset contains images sequences of subjects while driving in real sce- narios. It is composed of 606 samples acquired different days from 4 drivers with several facial features. We randomly extracted 484 samples for training set and the re- maining 122 samples for test set. Each sample is rescaled as 64 64 pixels and then flattened into 4096-dimensinal vector. The dataset can be download from https: //archive. ics. wttp://aatttee///////r.....
this dataset contains mass-spectromettri data from healthy individuals and can- cer patients. We split the data from both healthy individuals and cancer patients into training and sets with an 8:2 ratio, respectively. This dataset is one of 5 datasets of the NIPS 2003 feature selection challenge. The details and the download link are attps://archiver ics uei httpe/aaaaaaatttet/////////
ARCENE:
In each experiment, we benchmark six models: ReduNet, ReduNet-RF (replacing the internal mem- bership predictor with random forest classifier), imp-Redue Red-eet (using Lemma for parameter computation). Imp-ReduNet-RE (combining model and 3), Multi-ReddNet. and Multi ReduNet- LastNorm. The evaluation focuses on:
30

---

Published as a conference paper at ICLR 2026
Classification accuracy: Features from the final layer are evaluated using three down-
stream classifiers:
1. SVM: Support Vector Machine with RBF kernel.
2. KNN: kkk-nearest neighbors classifier with k = 5.
3. NSC (Nearest Subspace Classifier): for each class j. compute the mean by TR
of the learned features ZZ, and let U E [d be the top ry principal components of Z
Then, a feature is classified to class j' where j' arg mrggminje(..... lI U (=
M. We set 10 for all
Training efficiency: We compare total training time across models with different layer counts.
Computational complexity: Table presents theoretical parameter calculation costs in undersampled regimes.
Feature separability: We visualize test features learned by ReduNet, ReduNet-RF, Multi- ReduNet, and Multi-ReduNe Secduuee-LastNoome using t-SNE plots in Figure 2. These visualiza- tions are based on features extracted from models L = 5 layers. Since imp-ReduNet and imp-ReduNet-RF only optimize computation without modifying representations, their fea- tures are not visualized.
I EXTENDED EXPERIMENT RESULTS
1.1 CLASSIFICATINN ACCURACY OF REDUNET VARIANTS WITH VARYING LAYERS == 10, 15, 20, 25
The experimental results of ReduNet variants with 10, 15, 20, and 25 layers (with fixed 7 0.05, 2" = 0.1) are reported Tables 9-14. Specifically, Table 9 corresponds to Reuters, Table 10 to MNIST, Table 11 to Fashion-MNIST, Table 12 to Swarm Behaviour, Table 13 to DrivFace, and Table 14 to ARCENE. These results provide a comprehensive comparison across different depths and datasets.
Table 9: Accuracy comparison of ReduNet variants on Reuters
Layers=10
Layers=15 15
SVM
KNN
NSC
SVM
KNN
NSC
ReduNet
0.837
0.591
0,939
0.771
0.576
0.925
ReduNet-RF
0,441
0.562
0.623
0.441
0.562
0.552
imp-ReduNet
0.838
0.591
0.939
0.771
0.576
0.925
imp-ReduNet-RE
0.441
0.562
0.623
0.441
0.562
0.552
Multi-ReduNee
0.985
0.931
0.957
0.978
0.892
0.953
Mutt-ReeduNe:LLstNoom
0.986
0.944
0.957
0.981
0.929
0.956
Layers=20
Layers=
SVM
KNN
NSC
SVM
KNN
NSC
ReduNet
0.758
0.569
0.909
0.746
0.569
0.874
ReduNet-RF
0.440
0.562
0.826
0.439
0.561
0.468
imp-ReduNet
0.757
0.571
0.909
0.750
0.570
0.876
imp-ReduNet-RFF
0.451
0.572
0.846
0.441
0.561
0.488
Multi-ReduNet
0.977
0.830
0.950
0.971
0.732
0.953
Multi-ReduNet Siti-RRoduNet-LastNomm
0.977
0.907
0.951
0.974
0.879
0.950
I.2 ENLARGED T-SNE VISUALIZATIONS OF TEST FEATURES AND ANALYSIS
Figure 3 presents t-SNE visualizations of test features learned on the Reuters dataset across four ReduNet variants. The vanilla ReduNet (top-left) shows entangled feature clusters with significant overlaps between classes, indicating limited separability. ReduNet-RF (top-right) marginally im- proves class separation but still suffers from boundary ambiguity. In contrast, both Multi Multi-ReduNet (bottom-right) exhibit markedly improved clustering.
(bottom-left) and
31

---

Published as a conference paper at ICLR 2026
Table 10: Accuracy comparison of ReduNet variants on mnist
Layers=10
Layers=15
SVM
KNN
NSC
SVM
KNN
NSC
ReduNet
0.898
0.909
0.910
0.885
0.881
0.917
ReduNet-RF
0.354
0.468
0.672
0.261
0.250
0.278
imp-ReduNet
0.898
0.909
0.906
0.885
0.881
0.917
imp-ReduNet-RF
0.354
0.468
0.661
0.261
0.250
0.276
Multi-ReduNet
0.787
0.859
0.848
0.684
0.816
0.782
Multi-ReduNer-LastNNNee
0.788
0.868
0.858
0.728
0.848
0.838
Layers=20
Layers=25
SVM
KNN
NSC
SVM
KNN
NSC
ReduNet
0.880
0.875
0.910
0.876
0.874
0.906
ReduNet-RR
0.207
0.144
0.195
0.124
0.124
0.195
imp-ReduNet
0.880
0.875
0.908
0.876
0.874
0.906
imp-ReduNet-RF
0.208
0.153
0.195
0.124
0.124
0.195
Multi-ReduNee
0.595
0.748
0.725
0.534
0.736
0.668
Mult-RedduNe-LastNNors
0.664
0.805
0,800
0.628
0.805
0.779
Table 11: Accuracy comparison of ReduNet variants on fashion-mnit
Layers=10
Layers=15 15
SVM
KNN
NSC
SVM
KNN
NSC
ReduNet
0.822
0.799
0.842
0,806
0.788
0.837
ReduNet-RF
0.418
0.419
0.564
0.270
0.265
0.302
imp-ReduNet
0.822
0.799
0.842
0.806
0.788
0.832
imp-ReduNet-RF
0,418
0.419
0.565
0.270
0.265
0.304
Multi-ReduNet
0.707
0.746
0.764
0.604
0.745
0.723
Multi-ReduNNt--Lsttooo
0.720
0.763
0.788
0.618
0.772
0.755
Layers=20
Layers=25
SVM
KNN
NSC
SVM
KNN
NSC
ReduNet
0,801
0.793
0.830
0.799
0.799
0.818
ReduNet-RF
0.270
0.256
0.270
0.270
0.243
0.270
imp-ReduNet
0,801
0.793
0.830
0.794
0.799
0.813
Imp-ReduNet-RF
0.271
0.266
0.278
0.270
0.243
0.272
Multi-ReduNet
0.520
0.741
0.667
0.463
0.750
0.602
Multi-ReduNett-asttoon
0.529
0.765
0.723
0.470
0.774
0.700
with each class forming compact and well-separated regions. Notably, Multi ReduNett-astNeeree demonstrates the cleanest class delineation, suggesting that class-wise decomposition and final- layer normalization contribute synergistically to enhancing discriminative structure in the learned features.
Figure 4 shows the t-SNE projections of test-set features extracted by different ReduNet variants on mnist. ReduNet and ReduNet-RF (top row) exhibit limited class separation: while some clusters be- gin emerge (e.g., digits 0, 1, and 7), the overall feature distributions are entangled, with noticeable overlaps between semantically similar digits (c.g., 4,5). 5).
In contrast, mild boundary fuzziness. The clearest improvement appears in Multi- RestNoe- (bottom right), where all ten classes sharply delineated with minimal intra-class variance. The result- ing clusters are not only well-separated but also uniformly distributed, indicating improved feature compactness and discriminability.
(bottom left) yields significantly more structured clusters, albeit with
Figure presents the t-SNE visualizations of final-layer features learned by different ReduNet vari- ants on the fashion-mnin test set. Despite the increased complexity of this 10-class clothing dataset
32

---

Published as a conference paper at ICLR 2026
Table 12: Accuracy comparison of ReduNet variants on Swarm Behaviour
Layers=10
Layers=15
SVM
KNN
NSC
SVM
KNN
NSC
ReduNet
0.802
0.985
0.995
0.802
0.883
0.995
ReduNet-RF
0.920
0.911
0.925
0.869
0.861
0.887
imp-ReduNet
0.802
0.985
0.995
0.802
0.883
0.994
imp-ReduNet-RF
0.920
0.911
0.925
0.869
0.861
0.887
Multi-ReduNet
1.000
0.998
0.884
1.000
0.968
0.860
1.000
0.998
0.910
1.000
0.972
0.865
Layers=20
Layers=25
SVM
KNN
NSC
SVM
KNN
NSC
ReduNet
0.802
0.837
0.971
0.802
0.804
0.936
ReduNet-RR
0.869
0.817
0.825
0.869
0.773
0.800
imp-ReduNet
0.802
0.837
0.971
0.802
0.804
0.939
imp-ReduNet-RF
0.869
0.817
0.825
0.869
0.773
0.800
Multi-ReduNee
1.000
0.938
0.823
0.990
0.915
0.805
Mult-ReeduNet-astNNors
1.000
0.938
0.844
1.000
0.920
0.821
Table 13: Accuracy comparison of ReduNet variants on DrivFace
Layers=10 10
Layers= 5
SVM
KNN
NSC
SVM
KNN
NSC
ReduNet
0.322
0.202
0.169
0.279
0.164
0.240
ReduNet-RF
0.284
0.295
0.306
0.284
0.295
0.273
imp-ReduNet
0.326
0.202
0.169
0.279
0.164
0.240
imp-ReduNet-RF
0.284
0.295
0.306
0.284
0.295
0.303
Multi-ReduNet
0.995
0.934
0.984
0.973
0.907
0.967
Mult-RedduNe-LasttNono
1.000
0.951
0.989
0.990
0.956
0.984
Layers=20
Layers=25
SVM
KNN
NSC
SVM
KNN
NSC
ReduNet
0.317
0.131
0.218
0.256
0.131
0.191
ReduNet-RF
0.284
0.240
0.273
0.218
0.322
0.251
imp-ReduNet
0.317
0.131
0.218
0.256
0.131
0.191
RFF
0.284
0.240
0.273
0.218
0.322
0.249
Multi-ReduNet
0.919
0.929
0.945
0.891
0.939
0.923
Multi-ReduNer-Lasttooo
0.984
0.973
0.967
0.962
0.978
0.962
(relative to mnist), the separation and compactness of class-wise features vary substantially across models.
ReduNet (top-left) exhibits notable class entanglement, with overlapping clusters and unclear mar- gins between semantically distinct categories (e.g., classes 0, 3, and 7). The representation remains largely diffuse, reflecting its global coupling across classes and lack of explicit discriminability con- straints.
ReduNet-RF (top-right) shows similar structure ReduNet, indicating that global orthogonality constraints alone are insufficient for resolving subtle visual categories in fashion mishs.
Multi- Redti-RReuNet (bottom-left) introduces sharper decision boundaries and better class separation, thanks to its per-class decomposition strategy. Though some clusters still partially overlap, the overall layout is more glass-discriminative and geometrically organized.
Metti-ReddNNet-LLe LastNoom (bottom-right) achieves the most clearly separated and compact clusters, with minimal inter-class confusion and high intra-class cohesion. Notably, all 10 classes form dis- tinct, ovr-overlapping blobs, validating the effectiveness the final projection step in enforcing orthogonality and enhancing visual interpretability.
33

---

Published as a conference paper at ICLR 2026
Table 14: Accuracy comparison of ReduNet variants on ARCENE
Layers=10 10
Layers=15 15
SVM
KNN
NSC
SVM
KNN
NSC
ReduNet
0.536
0.512
0,488
0.439
0.414
0,439
ReduNet-RF
0.882
0.890
0.439
0.098
0.098
0.439
imp-ReduNet
0.536
0.512
0,488
0.439
0.418
0.439
imp-ReduNet-RF
0.882
0.890
0.439
0.098
0.098
0.439
Multi-ReduNet
0.756
0.707
0.805
0.634
0.708
0.805
Muli--ReduNee-LasttNNee
0.785
0.759
0.817
0.752
0.734
0.805
Layers=20
Layers=25
SVM
KNN
NSC
SVM
KNN
NSC
ReduNet
0.560
0.487
0.414
0.439
0.390
0.463
ReduNet-RR
0.901
0.890
0.437
0.098
0.098
0.439
imp-ReduNet
0.560
0.487
0.414
0.439
0.390
0.463
imp-ReduNet-RF
0.902
0.888
0.437
0.098
0.098
0.439
Multi-ReduNet
0.599
0.707
0.783
0.570
0.707
0.781
Mult-RedduNet-asttNors
0.712
0.708
0.805
0.651
0.685
0.786
Figure shows the t-SNE projections of test-time features the Swarm Behaviour dataset, binary classification task characterized by limited samples and subtle class variation.
ReduNet (top-left) fails to effectively separate the classes in the projected feature space. Most samples are scattered and interleaved, indicating weak class-discriminative structure. This reflects
U221 Vasaaloiic Tucaas Dazzbicc wob-fadddeeeattio
Figure 3: t-SNE visualizations of learned features Reuters
34

---

Published as a conference paper at ICLR 2026
L.SV Wer colon Packe Dacitaate
L9S vioud Laben Peutors srdieen RoOdREAT
14N6 Voualas in Faatr B- OIITER:
1206 Vaus latker Fattar Destaalen radl di Lisiiioii 00110
Figure 4: t-SNE visualizations of learned features on mnist
1 www.aaaliii bettur diarbaaai AUCUANI Milai
2.% Ditbl We desiiaatie for ansio Sware sobaaoo
-15
1.ME Visualatiin Festare Duirbstion Phdduted Dehanizz
123E Vasurization of IvtrreDstsubiinn
125
-55
Figure 6: t-SNE visualizations of learned features Swarm Behaviour
35

---

Published as a conference paper at ICLR 2026
USW Vouidiain Poazes Dusikdies leshoweeee
0256 Vise lupton Festore chelisaoon Sclech bet fatlicerii
626 cabor Fosto choloaie to EAE Lasthrreee falke-rrik
Figure 5: t-SNE visualizations of learned features on fashion-mnist
the challenge of capturing meaningful boundaries when using globally coupled updates without class-specifee refinement.
ReduNet-RF (top-right) shows slight improvement, with some local grouping of class-1 (orange) points, though the global overlap remains significant. The marginal gain suggests that random fea- ture projections alone are insufficient to resolve this low-data regime.
Multi-ReduNee (bottom-left) introduces clearer inter-class margins and tighter intra-class clusters. While the separation not perfect, distinct grouping patterns emerge, indicating that independent class-wise subspace optimization provides meaningful gains in geometric regularity and class align- ment.
Nulti-ReduNNt-IIaan LastNorm (bottom-right) delivers the clearest boundary among all variants. Class (blue) and class 1 (orange) form nearly disjoint clusters along a horizontal axis, with minimal cross- class confusion. The use of a final projection to enforce global separation yields feature space that is highly linearly separable, well-suited for downstream nonparametric classifiers like SVM or NSC.
36


Reviews:
[{"review_id": "faPC0OjmFf", "reviewer": "Reviewer_d7rQ", "summary": "The authors improve the scalability of the classifier ReduNet, which is specialized for solving tasks where there are a lot more features $d$ than samples $m$. Given $K$ classes and $m_j$ samples for class $j$, new algorithms scales as $\\mathcal{O}(\\sum_{j=1}^Km_j^3)$ compared to the $\\mathcal{O}(Kd^3)$ of the original algorithm. The scalability and performance is demonstrated on 4 real datasets.", "questions": "What is the $d_j$ in theorem 1? It is not introduced before being used.\n\nTheorem 1 defines class-orthogonality as $Z_j Z_i^T=0$. Shouldn't is be $Z_j^T Z_i=0$ instead following Corollary 1? \n\nWhy is the Frobenius norm $||Z_j||^2_F\\leq m_j$ bounded in theorem 1 but the optimization algorithm projects to the unit sphere? Projecting $m_j$ points on the unit sphere guarantees that $||Z_j||^2\\leq m_j$ , but the converse is not true. Perhaps a more in-depth introduction to ReduNet in the main manuscript would help.\n\nFrom my understanding, multi-redunet an improvement in terms of scalability : it solves the same problem as ReduNet but more efficiently. Then how can multi-redunet perform better than redunet in terms of accuracy? Is ReduNet reaching a different optimum?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The paper does a great job at explaining the ReduNet method to readers without prior knowledge on the field.\n\nThe theoretical contribution of the paper is significant since the new algorithm scales as $\\mathcal{O}(\\sum_{j=1}^Km_j^3)$ compared to the original $\\mathcal{O}(Kd^3)$.\n\nThe improvements in accuracy are also significant compared to prior work.", "weaknesses": "## Orthogonality Constraints\n\nAt line 270 it is stated that the optimization problem outlined at line 269 is solved \"subject to class-wise orthogonality and norm constraints\".\nI don't think the manuscript clarifies how these constraints are applied in practice. The proposed algorithm iteratively solves the equation at line 269 in order to infer the representation $Z_j$ of each class. The $Z_j$ are updated independently by maximizing a separate objective so it is not clear how orthogonality is enforced.\n\n## A smoother-introduction to Multi-ReduNet\n\nThe paper actually has two contributions : Imp-ReduNet which changes the inversion of a $d\\times d$ matrix to a $m\\times m$, and\nMulti-Redunet which furthers improves this to $K$ inversions of $m_j\\times m_j$ matrices. But the order in which these contributions are presented in confusing : multi-redunet is presented first and then imp-redunet. I think that describing imp-redunet first would help motivate the need to separate the objective into $K$ objectives.\n\nBy introducing imp-redunet first (via lemma 1) it is clear to the reader that the computational bottleneck is inversion of a $m\\times m$ matrix to compute the gradient of $\\text{log det}(I+\\alpha Z Z^T)$. If we were able to replace this with $K$ terms $\\text{log det}(I+\\alpha Z_j Z_j^T)$, we could inverse $K$ matrices of shape $m_j\\times m_j$ instead, which is a lot better. After this initial high-level motivation, and hinting that $Z Z^T=\\sum_{j=1}^K Z_j Z_j^T)$, the main theorems can be presented.\n\nThis is a subtle change, but I think it will improve the flow of the paper significantly.\n\n## Technical Overload\n\nOn a similar topic, the paper would benefit from moving some technical content to the appendix and focusing more on high-level ideas in the main manuscript. Notably, the proof of the Theorem could be moved in the appendix, and replaced with a high-level proof description.\nThis description would only need to accentuate the most crucial parts of the proof e.g. that $\\text{log det} (I+\\alpha Z Z^T)= \\text{log det} (I+\\alpha \\sum_{j=1}^K Z_j Z_j^T) =  \\sum_{j=1}^K \\text{log det} (I+\\alpha Z_j Z_j^T)$ whenever class representations are orthogonal.\nThis clarifies to the reader that class-orthogonality is the key assumption to separate the objective into $K$ sub-objectives.\n\nThe freed space in the main manuscript could be used to introduce intuition for ReduNet e.g. extended content from the Appendix B.\n\n## Figures 1\n\nThis is a very minor point, but I think that Figure 1 is hard to read. It is hard to see the color of the markers because it is very dark. Also, the marker color does not match with the line, so I constantly have to read the legend to be sure. Moreover, methods based on RF perform so bad that they hide the differences between ReduNet and Multi-Redunet (the main contributions of the paper). I would suggest removing the RF methods from the plot and simply indicate in the text that they take orders of magnitude more time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZtgbYlXI1q", "reviewer": "Reviewer_tTgf", "summary": "This paper proposes two class-wise extensions to ReduNet, namely, Multi-ReduNet and Multi-ReduNet-LastNor, to improve the performance under undersampled, high-dimensional conditions. The authors theoretically justify that the global MCR² objective can be decomposed into class-specific subproblems and leverage this result to design more efficient and class-discriminative models. Experiments on multiple datasets show consistent improvements in classification accuracy and training efficiency, while maintaining interpretability.", "questions": "Please refer to weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper gives a theoretical analysis showing that the MCR² objective under certain conditions can be equivalently decomposed into class-wise subproblems. \n2. The proposed models maintain the white-box property of ReduNet and preserve its forward-only optimization strategy. This makes the approach more transparent and easier to analyze compared to conventional backpropagation-based deep networks.\n3. The class-wise decomposition reduces the cost of matrix inversion from high-dimensional global matrices to smaller class-specific ones, which is computationally advantageous in settings with limited data and large feature dimensionality.", "weaknesses": "1. The method assumes that the class-wise decomposition is meaningful under undersampled regimes. However, the paper does not investigate how the approach behaves when this assumption is less valid, e.g., when the sample size is moderately large.\n2. The evaluation focuses on relatively simple or small-scale datasets (e.g., MNIST, Fashion-MNIST), which may limit the conclusions.\n3. The experimental comparisons are restricted to ReduNet and its variants. The paper could be strengthened by comparing against a broader set of interpretable or class-structured learning methods, to better situate the approach within the existing literature.\n4. Although Multi-ReduNet-LastNorm performs slightly better in most cases, the role of the final-layer-only normalization is not fully analyzed. It would be helpful to understand under what circumstances this variant is preferable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TLkaK61WoB", "reviewer": "Reviewer_KGJT", "summary": "This paper extends ReduNet — a theoretically grounded, interpretable architecture based on the principle of maximal coding rate reduction — to address its limitations in scalability and class-specific representation. The authors propose Multi-ReduNet and Multi-ReduNet-LastNorm, which decompose the global learning objective into class-wise subproblems. This decomposition maintains ReduNet’s theoretical interpretability while substantially improving computational efficiency by lowering matrix inversion costs. Moreover, it enhances feature separability, making the model more effective in undersampled  regimes. The paper provides theoretical justification for this class-wise formulation and demonstrates empirically, across multiple datasets, that the proposed models preserve interpretability and achieve better efficiency and discriminative power.", "questions": "See Weaknesses.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "1.\tThe empirical results align well with the theoretical analysis, and Multi-ReduNet demonstrates significant performance improvements in the undersampled regime.\n2.\tThe writing is clear, well-structured, and easy to follow.\n3.\tThe theoretical analysis seems rigorous, and I did not find errors in the proofs.", "weaknesses": "1.\tThe experiments in this paper are primarily conducted on toy datasets such as MNIST. I believe it is necessary to include experiments on more realistic datasets, such as CIFAR. In addition, I am curious about the performance of Multi-ReduNet in the oversampled regime — is it still competitive under such conditions?\n2.\tThe motivation behind Multi-ReduNet is clear and intuitive. However, the rationale for introducing Multi-ReduNet-LastNorm is somewhat unclear, as there is no theoretical comparison between the two. It would strengthen the paper to include a clearer discussion on the logical progression from Multi-ReduNet to Multi-ReduNet-LastNorm.\n3.\tI noticed that the main proofs are presented in the main text. This makes the paper rather math-heavy and potentially difficult to follow. It might be preferable to include simplified versions of the proofs in the main paper and move the detailed derivations to the appendix.\n4.\tIn Figure 2, the representations learned by ReduNet appear well-clustered. However, Table 2 reports relatively low classification accuracy for the same model. Did I miss something here? It would be helpful to clarify the reason behind this apparent discrepancy between the results.\n5.\tThe authors state that “Although the class-orthogonality property of MCR² optima has been established in prior work (Chan et al., 2021), our proof leverages a simpler and more streamlined argument.” I view this as one of the main theoretical contributions of the paper. However, it remains unclear in what sense the proof is simpler. A more detailed explanation or discussion would strengthen this claim.\n6.\tThe presentation quality of the paper could be improved. For example, the title in Figure 2 is too small and difficult to read. Moreover, if the detailed proofs are moved to the appendix, it would be beneficial to include more discussion or intuitive explanations in the main text to improve readability and accessibility.\n7.\tIn summary, this paper presents a theoretically grounded and interpretable extension of ReduNet with promising results. However, the work could be significantly strengthened through better presentation, clearer motivation for the proposed variants, and more comprehensive experiments on realistic datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h6hNbr9PDQ", "reviewer": "Reviewer_bnBb", "summary": "This paper improves the objective of ReduNet under the assumption of “undersampled regimes” and proposes two extensions, Multi-ReduNet and Multi-ReduNetLastNorm, for computational efficiency and representation separability", "questions": "1. I’m confused about the last two sentences from lines 32 to 35. Why scenarios where the number of features exceeds the number of samples will lead to overfitting and unstable generalization. Is there literature supporting this claim? The explanations of the background are missing.\n\n2. In Theorem 1, why does $Z^i( Z^j )^T= 0$ mean class-orthogonality? This seems to be misaligned with line 172. Should it be $( Z^i )^T Z^j= 0$?\n\n3. How does the last equality hold in Eq.(2) under the assumption $(Z^{*j_1})^TZ^{*j_2} \\neq 0$?\n\n4. The whole analysis in the paper assumes $m \\ll d$. Is this assumption even practical for most tasks? I won’t buy it if it’s just an idealistic setting, and rare in practical scenarios. \n\n5. Why compare to variants with a random forest classifier?\n\n6. How much does training time improve numerically? From figure 1, the proposed model is almost the same with ReduNet.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "1. The paper is well-structured.\n2. The mathematics is generally written in a professional way.", "weaknesses": "Major:\n\n1. The motivation is quite weak. In the introduction, the authors claim representation learning in the “undersampled regimes” is challenging, but without citing any literature or explaining whether this is a widely-accepted concern.\n\n2. Similarly, the questions to be tackled are also unclear. The authors do not define and thoroughly explain “class-specific structures” which they highlight as missing components in previous works.\n\n3. The contributions seem to be incremental and trivial. This paper only performs slight improvements on ReduNet under niche and small-scale settings. It does not bring in novel tools or appealing theoretical insights other than decomposing ReduNet’s objectives, either. In fact, the authors said in line 255:  “Although the class-orthogonality property of MCR2 optima has been established in prior work (Chan et al., 2021), our proof leverages a simpler and more streamlined argument.”, meaning the findings have already been established. These all make me question its significance.  \n\n4. The technical soundness is also questionable. The gradients derived in lines 274 and 277 are basically the same, subject to different scaling. I hardly believe they have a significant functional difference, which makes the decomposition in line 269 less compelling.\n\nMinor:\n\n1. No experimental details and discussion on limitations.\n\n2. Figures are quite hard to interpret immediately due to font size and colorization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)