Deep Hierarchical Learning with Nested Subspace Networks for Large Language Models
Authors: Paulius Rauba
Mihaela van der Schaar
Venue: ICLR 2026 Poster (2026)
Keywords: dynamic neural networks
efficient inference
adaptive computing
deep learning
low-rank adaptation
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We introduce Nested Subspace Networks (NSNs), a new way to build a single neural network that can instantly adjust its performance vs. computational cost at inference time.

Abstract: Large neural networks are typically trained for a fixed computational budget, creating a rigid trade-off between performance and efficiency that is ill-suited for deployment in resource-constrained or dynamic environments. Existing approaches to this problem present a difficult choice: training a discrete collection of specialist models is computationally prohibitive, while dynamic methods like slimmable networks often lack the flexibility to be applied to large, pre-trained foundation models. In this work, we propose *Nested Subspace Networks (NSNs)*, a novel architectural paradigm that enables a single model to be dynamically and granularly adjusted across a continuous spectrum of compute budgets at inference time. The core of our approach is to re-parameterize linear layers to satisfy a nested subspace property, such that the function computed at a given rank is a strict subspace of the function at any higher rank. We show that this entire hierarchy of models can be optimized jointly via an uncertainty-aware objective that learns to balance the contributions of different ranks based on their intrinsic difficulty. We demonstrate empirically that NSNs can be surgically applied to pre-trained LLMs and unlock a smooth and predictable compute-performance frontier. For example, a single NSN-adapted model can achieve a 50\% reduction in inference FLOPs with only a 5 percentage point loss in accuracy. Our findings establish NSNs as a powerful framework for creating the next generation of adaptive foundation models.

Published as a conference paper at ICLR 2026
DEEP HIERARCHICAL LEARNING WITH NESTED SUB- SPACE NETWORKS FOR LARGE LANGUAGE MODELS
Paulius Rauba
Mihaela van der Schaar University Cambridge
University of Cambridge
ABSTRACT
Large neural networks are typically trained for a fixed computational budget, creating a rigid trade-off between performance and efficiency that is ill-suited for deployment in sesource-constrinng or dynamic environments. Existing approaches to this problem present difficult choice: training a discrete collection of specialist models is computationally prohibitive, while dynamic methods like slimmable networks often lack the flexibility to be applied to large, pre-trained foundation models. In this work, we propose Nested Subspace Networks (NSNs), novel architectural paradigm that enables a single model to be dynamically and granularly adjusted across continuous spectrum of compute budgets inference time. The core of our approach is to re-parameterize linear layers to satisfy a nested subspace property, such that the function computed a given rank is strict subspace of the function at any higher rank. We show that this entire hierarchy of models can objective that learns to balance the contributions of different ranks based on their intrinsic difficulty. We demonstrate empirically that NSNs can be surgically applied pre-trained LLMs and unlock a smooth and predictable compute-performee NSN-adapted model can achieve 50% reduction in inference FLOPs with only 5 percentage point loss in accuracy. Our findings establish NSNs as powerful framework for creating the next generation of adaptive foundation models.
be optimized jointly via an uncertainty-awant
frontier. For example, single
INTRODUCTION
Motivation. When we deploy deep learning-based systems in practice, there a trade-off between two properties: how good the model is (performance) and how expensive it is to run (compute). Typically, the larger the model, the better the performance. When using such models at inference (deployment) time, we may want to choose, on-the-fly, how "expensive" vs "fast" a model should be. For instance, we may prefer (i) cheaper models for easier questions in language models; (ii) lower-compute models on phones when battery levels drop; or (iii) more expensive models for safety- critical requests such as medical diagnosis. In this paper, we consider exactly this problem-hov to build single network that can flexibly trade off performance and inference cost test time.
Current approaches. Most popular approaches fall into two main categories. On the one hand, conventional approaches operate by creating smaller, static artifacts from larger pre-trained model Cheng et al. 2017). using techniques like network pruning (Han et al. 2015b) Blalock et al.) (020) or knowledge distillation (Gou et al. (02T). More recently, parameter-efficiet fine-tuning methods like Low-Rank Adaptation (LoRA) (Hu et al. 2022) have gained popularity for adapting large models, but these also produce a static, low-rank adaptation for fixed budget. In theory, this approach yields highly optimized models for specific computational target. In practice, however, this strategy suffers from its static nature; creating model for a new budget requires repeating the entire, often costly, compression pipeline (Zhu & Gupta 2017). and it fails provide the granular, on-the-fly fly adaptability needed for dynamic environments.
On the other hand, recent methods using dynamic neural networks (Han et al. 202 operate by designing architectures that can be adjusted at inference time, such as slimmable networks that can drop channels (Yu et al.] 2018 Li et al. 1222) layers (Wu et al.] 2018). In theory, these approaches more readily take advantage of a single set of weights to serve multiple budgets. In practice, however, this strategy often comes at the price of much more challenging, specialized training schemes that

---

Published as a conference paper at ICLR 2026
LoRA Fine-tuning
Nested Subspace Networks
Vanilia MIP
(and variants)
Output
Trainable weights
Frozen weights
Trsinable adapter
Rank-trainable layers
Model
A Recda
Rrnde
Architecture
W gda
W E Rdes
X B, R
B Rdeax
Input
Trainable
inputioutput
Figure 1: Illustration of Nested Subspace Networks. NSNs convert linear layers into rank-trainable layers which enable dynamic control over the computational cost (FLOPs) of forward pass. Left: Standard MLP layers that are composed of trainable weights. Middle: LoRA fine-tuning which have frozen weights and trainable adapters. Right: Nested Subspace Networks replace each linear layer with single pair of shared factor matrices (A, B) defining a rank-trainable layer. The effective weight at rank r, W A, is obtained by using only the first r rows of and first columns of B. Different operating points (different ranks) therefore correspond to using different prefixes of the same (A, B) This allows for the construction of a compute-performant Pareto frontier at inference.
are typically applied from scratch (Cai el al., 2019). This makes them difficult to apply to the vast ecosystem of existing, pre-trained foundation models, which represent the vast majority of trained and used models today. Further, many of these techniques with some notable exceptions Yu & Huang 2019) offer only coarse, discrete set of operating points rather than smooth, continuous trade-off 'Teerapittayaoon et al.) 2016 Yu et al., 2018).
Three Desiderata Can we develop a better approach? Building on the discussion above, we contend that good solution to the dynamic inference problem should satisfy the following three desiderata.
Dí: Instant Adaptability. Instantly trade-off compute and performance at test-time without any additional overhead or expensive fine-tuning procedures in single neural architecture.
D2: Post-Hoc Applicability. Have the architectural generality to be applied to any pre-trained foundation model and be widely applicable for many classes of models.
D3: Granularity. It should provide a smooth, continuous spectrum of operating points along the prrputt-pprformance Pareto frontier, not just a few discrete, pre-determined choices.
In this work, we present an effective method that satisfies these criteria and introduces a new paradigm of flexible model deployment. Our contributions are three-fold.
Contributions. First, we introduce Nested Subspace Networks (NSNs), a novel architec- ture that represents continuous hierarchy of models within single set of weights, and we propose a practical uncertainty-aware training objective that makes this hierarchy learnable (Sec. 2 Second, we provide theoretical guarantees for granular budget control, showing that our method induces a smooth and predictable performance-compe tier, even for budgets not explicitly seen during training (Sec. 3). Third, we demonstrato the broad utility and effectiveness of NSNs through comprehensive experiments (Sec. + including the surgical adaplation large pre-trained language models, and show that a single adaptive network can match the performance of multiple specialist models.
fron-
2 NESTED SUBSPACE NETWORKS
Preliminaries.
Consider standard feed-forward neural network. A standard linear layer computes the affine transformation fxx) = Wx + b. where x Id the input vector, the weight matrix is W E Red dl,, and the bias b E RRd, The number of parameters in W scales with the product of the
2

---

Published as a conference paper at ICLR 2026
input and output dimensions which becomes large computational and memory bottleneck. This motivates the need for efficient parametrizations.
Low-rank factorization has become popular approach to mitigating the quadratic cost (Hu et al., 2022), where the full-rank matrix W is approximated with two smaller matrices W = BA, where E IR Rxda and B E Rd x Here, R < min (dm; doa)) is maximum rank. The transformation
becomes f(x) = (BA)x + b = B(Ax) + b.
2.1 THE NESTED SUBSPACE ARCHITECTURE
NSNs are a class of neural network architectures designed for parameter efficiency and dynamic, post-training adjustment of model capacity. The core principle is to re- re-parameterize a linear layer with a sequence of low low-rank approximations {W,),= that form a nested hierarchy, such that the image of each approximation is a subspace of the next. The architecture is built on the principle of low-rank factorization which we extend to overcome its static limitations and make it applicable to wide variety of network architectures. Concretely, unlike slimmable networks (Yu al., (2018). which vary channel width and therefore change intermediate tensor shapes, NSNs only vary the rank of a shared low-rank factorization, so all input-output dimensions of each layer remain fixed and the architecture can be inserted into pre-trained transformers and LLMs without modifying their interfaces or normalization layers.
The low-rank factorization reduces the model's active parameter count and, consequently, its required floating-point operations (FI.OPs). This yields a reduction when is below the break-even point: (rm + dema) < do.. which defines the break-even rank as di di +
Reducing FLOPs.
Definition 1 (Nested Subspace Network). A Nested Subspace Network (NSN) is a neural network architecture that incorporates one or more NSN layers. An NSN layer is linear transformation parameterized by a pair of factor matrices, A E Rxda and B & IR Red XR where R is fixed maximum rank. For rank
(1, (1...., R). the effective weight matrix W, R.du da is constructed from the submatrices A, (the first rows of A) and B, (the first columns of B). This is expressed as a sum of the rank-1 I outer products, i.c. W, := B, = Ei=1 b,a, where a, E
xxd is the i-th row of A and b, E Rd x is the i-th column of B.
Note that there is single pair of factor matrices (A, B) for NSN layer, and changing the rank only changes how many of their rows/columns are used to form Wr. not the underlying parameters. Appendix D.2 provides an intuitive, worked-out example with simple matrix. NSNs are flexible class of models that can operate on model architecture as long as it comprises linear layers. Therefore, it is applicable to models of different sizes, architectures, purposes, with varying inductive biases, etc. A central feature of NSNs is that it naturally gives rise to fundamental property that we exploit in this work: the nested subspace property.
Definition (Nested Subspace Property). The family of weight matrices (Wr)² generated by an NSN layer satisfies the nested subspace property if the image of the rank-r transformation a subspace of the image of the rank-(r + 1) transformation for all S r < R:
Im(,,) Im(W++1)
Vr E {1, R.I....R-11. 1}
A sufficient condition for this is that Ar+1 has full row rank for each r (Sec. D.I. This implies the existence of a filtration of vector spaces: Im W1) C Im(W)) -... C Im (RR)
What does this property mean in practice? NSN layers parametrize an entire hierarcky of models. In this hierarchy, due to the nested subspace property, the function class realized by a rank-r model is strict subset of the function class of rank-(r + 1) model. Therefore, with the right training scheme, we can choose "which network" we want to employ by deciding the rank.
However, this approach raises an immediate issue: how can we train a model that learns a hierarchy of nested sub-models? A naive approach would be to parametrize the model via low-rank factorization, select a highest rank, train it on the highest rank, and at test time truncate it to the desired rank r. While this technically satisfies the nested subspace property, there is no inductive bias to make the models operate along the found in Appendix B. An alternative approach could be to train simultaneously at different ranks
Pareto frontier (see Fig implementation details can be
3

---

Published as a conference paper at ICLR 2026
and sum cross entropies within each rank. However, such naive approach suffers from least three main issues: (i) does not take into account the intrinsic difficulty learning a lower rank model (harder) versus higher rank model (casier); (ii) It results in training instability resulting from large losses in low-rank models; (iii) It is computationally prohibitive to train at all possible ranks. In the eneertaittyawwrr training procedure that resolves these challenges.
next section, propose
Takeaway. NSNs create a hicrarchy of models within a single network using the nested subspace property. The key challenge is training one set of weights to be optimal across this entire hierarchy sinullaneously.
2.2 TRAINING WITH MULTI-RANE UNCERTAINTY
Our central goal is to learn a single parametriza- tion that simultaneously yields optimal perfor- mance for all sub-models. We posit that the failure of naive approaches stems from the differ- ences in intrinsic difficulty of learning different ranks (Sec. 2.1). Therefore, we treat the opti-
97.187.8
80.7
60.9
Accuracy (96
80
60-
28.7
2811
14.
mization of each sub-model at different ranks a multi-task learning problem with varying
Rank
Rank 4
Rank 8
Rank 16 Rank 32 Rank
Native rank training
Rank-64 training
difficulty levels.
What properties should this optimization ob- jective satisty? We seek a weighting mecha- nism that (i) automatically adapts the relative im- portance of ranks without per-rank hyperparame- ter tuning, (ii) is invariant to arbitrary rescalings of the factorization W BA, (iii) guarantees positive weights, and (iv) is cheap enough to apply inside every NSN layer and on every train- ing step. One way to reframe the "difficulty" of problem is by quantifying the aleatoric uncer- tainty of each task and weighting each task in proportion to that uncertainty (Kendall et al. 2018). We model the aleatoric uncertainty by introduc- ing learnable variance parameters of for each rank k. This variance is assumed to be heteroskedastic across ranks (i.e.. of * of for j) but homoskedastic within a rank (i.e., constant for all inputs).
Figure 2: Comparison of native-rank training and rank truncation for an ML. on CIFAR- 10. The plot compares the accuracy of individually training a model for each specific rank (Native rank training) versus training single model at a high rank (64) and truncating it to lower ranks at test time (Rank-64 64 training). The significant perfor- mance gap demonstrates that naively truncating a high-rank model results in poor performance.
Modeling assumption. We consider the classification case (the regression case is analogous). Following Kendall et al. (2018). we use the standard uncertainty- surrogate objective in which each rank's 's cross-entropy is weighted by learnable scale and regularized by corresponding log-term. Concretely, for rank i we use the per-rank contribution:
Lcs (k) + log ok
20
which serves purely as seighting-andrrgulaarn (i.e., it is not interpreted as a probabilistic likelihood over LCE).
surrogate for balancing tasks in classification
Formulating the uncertainty- rank R < the maximum rank used at training time-an variant rank R. Assuming independent uncertainty parameters across ranks, the total objective for a training step is the sum of the two surrogate terms:
training objective. During training, we sample an anchor
1 20 Ler CCE(R) R) + log oogan + ( fcE(F) + log log 20?
We reparameterize the variance by learning its logarithm 8s log(o?) and drop the constant factor Kendall al. 2018). This results in our final training objective, a function of the shared weights and B (which denne the model weights) and the learnable log- variances:
Luxal (A B, 8 8r) (exp(-<) (t)) LCE( (R) s1t (exp(-s) (() Lcn(r) 8r).
4

---

Published as a conference paper at ICLR 2026
Algorithm 1 Multi-rank uncertainty-weighheed training for NSNs (anchor at maximal rank)
Require: Dataset D, maximal rank R. (anchor), trainable ranks Æ S {l...., R). NSN fo(;;r), log-variances {8). = log o kex. cross-entropy loss CE, optimizer Opt
1: Initialize 0 and 8ą: f- 0 for all k E E
2: for each minibatch (X, Y) ~ do
3: Sample variant rank r Uniform \ {R))
4: Compute anchor logits Zn 4- fo(:; Rt,, anchor loss Ln + CE(ZR,
5: Compute variant logits Z, + fo(;;)), variant loss L, t (Z Y)
6:
Canchor <- cxp( (xp(-SR)LL + 8R, Lvariam & cxp(-s,) Lr Sr, Lioial + Lanchor + variiatt
7:
Opt.zero_gradg backpropagate Lical w.r.t. 0 and {5 Opt.step()
8:
end for
Why are the exponentials useful in this equation? It's useful to reason about this from three different perspectives. First, the exponentials useful because the reparameterization wi e ensures strictly positive weights and produces a strictly convex objective in Sg. This is useful, since it provides stable gradient updates, Second, this formulation yields closed-form optimum w = L/L-- This is useful because we (i) become scale invariant and (ii) have gradient balancing across ranks of different difficulty. Third, this is directly tied to building surrogates that are based on Gaussian regression likelihood for classification settings which are easy to optimize (Kendall et al. 2018). More details in Appendix D.7
Insights on the formulated objective. The uncertainty surrogate implicitly performs gradient balancing across ranks, since the effective contribution each term scales with exp(-8i) 82). This connects objective to established approaches for multi-task optimization that seek Pareto- stationary solutions by equilibrating task gradients (Sener & Koltun 2018). as well as to adaptive weighting schemes such as GradNorm (Chen al.) 2018) and Dynamic Weight Averaging (Liu et al., 2019). This satisfies the required optimization properties (see Appendix [ for more discussion) and promotes the well-behaved performance frontier we analyze in Sec. 4.3 The full algorithm is presented in Algorithm |
In practice, the inclusion of anchor rank significantly stabilizes training and helps to learn better final higher-rank model. Furthermore, we find that introducing a surriullu-lleaming- baaeed sampling strategy for the variant ranks substantially improves downstream results relative uniform sampling. Soorithhmssummatt
the full training procedure. Each iteration evaluates the model at the maximal anchor rank and at sampled variant rank, combines their losses through rank- specific uncertainty weights, and updates both the shared parameters and the log-variances. This mechanism jointly optimizes all submodels and stabilizes learning across heterogeneous ranks.
Takeaway. We formalize the joint optimization of differont ranks by learning rank-spocific homoskedastic variance via standard eeeertainty-eeighten surrogate.
2.3 INTERPRETING LOG VARIANCES AS A PROXY FOR RANK EXPRESSIVENER
What has this formulation achieved? Now, the learned log variances modulate the influence of each rank's loss contri- bution to the objective. High log-variance promotes gradi- ent attenuation by scaling the cross-entropy loss from high- variance ranks. Low log-variance amplifies the penalty loss for cross-entropy losses by increasing its magnitude. In fact, we can directly quantify the gradient contributions based on the loss as:
V VoLunal
Figure 3: Learned log-variances dur- ing training with the multi-rank uncer- fainty objective We train a single model with an anchor and variant ranks and find that higher ranks have lower task-dependent uncer- tainty during training.
Anchor Contribution
CIFAR- 10 dataset.
+ exp(-s, s, ( (T
Variant Contribution
5

---

Published as a conference paper at ICLR 2026
This mechanism allows the learned log variances to serve emergent proxy for the effective expressiveness of each rank-specific model. It follows that higher ranks, which possess greater representational capacity, should learn lower corresponding variances. As we demonstrate empirically, this is precisely the behavior we observe our experiments (Fig. 3).
3 PERFORMANCE INTERPOLATION BETWEEN RANKS
A central claim of our work is that NSNs provide granular control over the compute- compute-performance trade-off. This implies not only strong performance at a discrete set of trained ranks but also reliable behavior at interpolated ranks that were not explicitly part of the optimization objective. In the absence of theoretical guarantees, one might expect performance to be unpredictable or even collapse between these well-trained points. In this section, provide the formal underpinnings for the smooth and well- behaved nature of the performance-come compute frontier induced by NSNs. We begin by introducing mild assumption on the structure of the learned weights, from which derive formal bound on the interpolation error.
To formalize the notion of smooth frontier, we must first characterize the structure that multi-rank uncertainty training (Section (.2) imposes on the parameterization. We posit that the optimization encourages natural ordering of basis vectors by importance.
Assumption (Rank-l Component Energy Decay). The training procedure yields a parameteri- zation where the norms of the rank-1 component vectors are monotonically non-increasing with their index. For any indices i. such that 1 < < S R:
||yl|
< |aa]]
and
[by S b:|l
Our training objective motivates this assumption, since naturally encourages the model to allocate the most salient information to the lowest-indexed basis vectors, as they must be utilized by all nested sub-models. We confirm this assumption empirically (Appendix C.4) and show this is not satisfied in regular training regimes (Appendix C.5
By extending this result from the model's output to its expected loss, and assuming standard regularity condition on the loss function, we can establish a bound on the difference expected performance between any two ranks.
Proposition (Bound on Interpolation Error). Let the task loss function L(f(x; be L
Lipschitz continuous with respect to its first argument. Let F(r) Ex [ (Xx((x)))))))))(( r).y) be the expected error at rank r. For any ranks TT < Tin < R, the difference in expected error is bounded by:
E(ri)) - E(ri) SC C
Σ 1
|b, |a||
where C= LL [[][|[|]] is a task-dependent constant.
Proof in Appendix D.4 This result provides formal guarantee that the variation in model perfor- mance is controlled by the cumulative energy of the intermediate basis vectors. This is important because it justifies the use of NSNs for reliable control across continuous spectrum of computational budgets. We empirically evaluate this claim in Sec. 4.3
Takcaway. With mild decay assumption on the Iearned rank-1 components, we can bound the performance change between ranks which ensures that the com- pule performance trade-off remains smooth and predictable even for untrained ranks.
EXPERIMENTAL EVALUATION
Here, highlight key properties of method: why do use cross-entropy ranks, why our assumptions hold, how we can generalize to interpolated ranks and pre-trained LLMs.
'Tee results and code be found here: http://tthhhcooolllliaaaaaaaasse

---

Published as a conference paper at ICLR 2026
4.1 Is IT SUFFICIENT TO SIMPLY USE CROSS-ENTROPE ON TWO RANKS?
To better understand the mechanisms behind our proposed training strategy, we conduct a series of ablation studies. Our goal is to isolate the contribution of the core components of objective function and to validate our design choices against plausible alternatives. We compare several training variations, evaluating their impact not only on the highest-rank model but, more importantly, on the average performance of the resulting lower-rank and interpolated sub-models.
Setup. We conduct our evaluation on an image classification task using a standard multi-layer perceptron (MLP) architecture. The dataset is CIFAR- 10 throughout but we first map each image to a fixed feature representation using an prretraiid backbone and then train only small classifier on top of these frozen features (more details in Appendix [3.] We assess each method on three metrics: (i) the final test accuracy the highest-rank (anchor) model; (ii) the average accuracy across all in-distribution (ID) sub-ranks evaluated during training; and (iii) the average accuracy across out-o--distributiin (OOD) interpolated ranks not explicitly seen during training. All three variants-Login Regularization, Residual Orthogonality, and Hidden Regularization- implemented as independent ablations, each adding exactly one regularizer on top of the same anchor/variant "Two CEs" training setup, and they are never used concurrently. Details in Appendix B.2. The results are summarized in Table
Table Ablation study of different training objectives. Our core Two CEs formulation (highlighted) dramatically improves the performance of lower-rank (ID) and interpolated (OOD) sub-models. R denotes the anchor rank and r the variant rank. Performance is reported mean + std. dev.
Method
Key Formulation
Highest Test Ace
Avg. ID Acc
Avg. OOD Acc
Baselines
CE Only (Anchor)
(c)(
0.87 + 0.00
0.48 + 0.00
0.57 0.00
One CE Hard Ortho.
+ +IAAT I
0.87 1 0.00
0.42 + 0.00
0.50 0.00
Variations on Joint Training
Two CEs
CrF (12) + (cF(F)
0.88 1 0.00
0.79 1 0.00
0.81 0.00
Logits Regularization
+ logits( (2)
logits(r)
0.87 1 0.00
0.64 1 0.00
0.64 = 0.00
+ Residual Orthogonality
+ Ar A ATall 11
0.88 + 0.00
0.78 + 0.00
0.80 = 0.00
Hidden Regularization
+
hAa h
0.88 + 0.00
0.79 + 0.00
0.79 0.00
Results. Our analysis in Table T shows that the key to effective sub-model performance is the joint
optimization of anchor and variant rank ("Two CHs"). This simple objective acts an implicit regularizer. In contrast, we found that adding explicit regularization terms on top of our proposed objective was either redundant or detrimental. This supports our claim that the joint optimization of multiple ranks is sufficient mechanism for learning NSNs.
4.2 IS THE ENERGY DECAY ASSUMPTION SATISFIED?
Setup. We examine the energy-decay property on a Pythia- 2.8B model fine-tuned for sequence classification, com- paring two variants: a low-rank model whose MLP lay- ers are decomposed into rank- factors A and B via SVD, and standard dense model with unmodified lin- ear layers. For each of the 64 MI.P weight matrices (32 transformer blocks x 2 projections), we compute the dered norms ||a; 1|2. ||bil|a (low-rank) and the row/column norms ||W,,:12. [W.ill2 (dense), then report the mean one standard deviation across layers.
Rarts 25
210
Figure 4: Energy decay profiles. The energy decay assumption holds in low- rank linear layers trained with our cross- entropy objective but does not hold in a standard MLP setting.
Results. Results are summarized in Figure 4 We find that the basis vectors in low rank layers consistently show energy, unlike standard dense MLPss
7

---

Published as a conference paper at ICLR 2026
Rank 50
Rank 40
Rank 30
Rank 20
Rank 10
Rank
Rank
Validation
Accuracy
0.75
0.50
0.255
20000
20000
20000
20000
20000
20000
20000
Global Step
Global Step
Global Step
Global Step
Global Step
Global Step
Global Step
Only Anchor
Two CEs
Two CEs + Residual orthogomality
CE orthogonality
Logits Regularization
CEs Hidden regularization
Figure 5: Training Dynamics at Interpolated Ranks. Validation accuracy during training for various objective functions, evaluated at ranks not explicitly optimized for. Our proposed method maintains stable learning across all ranks, while simpler baselines exhibit instability and performance collapse.
4.3 DOES THIS EMPIRICALLY GENERALIZE TO INTERPOLATED RANKS?
We now empirically validate the theoretical guarantees for smooth interpolation presented in Sec. B We ask whether our method yields robust performance even ranks that were not explicitly part of the optimization process, thereby satisfying our desideratum for granular budget control (Sec [].
Setup. To investigate this, we analyze the performance of models trained with the various objective functions from our ablation study. We train single model using each objective, which involves sparse sampling of ranks. We then evaluate the resulting model not only on the anchor rank but also across a wide spectrum of intermediate, interpolated ranks to assess the stability and smoothness of the learned performance curve.
Results. Our findings show that the choice of training objective is important for achieving stable interpolation. As illustrated in Figure 5) baseline objectives that do not properly balance the learning dynamics across ranks often result in unstable collapsing performance these intermediate points. For example, training with a single cross-entropy objective leads to poor generalization at lower ranks. In contrast, proposed objective, which jointly trains anchor and variant rank, produces stable and monotonically improving accuracy curves across the entire hierarchy of ranks throughout training.
4.4 DOES THIS EMPIRICALLY GENERALIZE TO PRE-TRAINED LLMs?
We now evaluate the post-hoc applicability of NSNs to large, pre-trained language models (LLMs).
Before
Linar
MIP brock
Adapting Pre-trained Layers. Our procedure for adapting pre-trained LLM consists of surgically replacing the standard linear layers within its MLP blocks with NSN layers. A naive approach might be to randomly initialize the NSN factor matrices A and B. In practice, however, this method discards the information encoded in the pre-trained weights. To preserve such information in the pre-trained weights, we initialize the NSN factor matrices using Singular Value Decomposition (Appendix D.5).
Repcet
N times
Rank Lindar
Temporaal sine
Figure 6: Example Surgical Changes to lin- ear layers only on Gemma-2B The architec- ture of Gemma-2B. All MLP blocks contain three linear layers with GeLU activation function. We surgically replace all linear lay- ers with rank-adaptive linear layers, initial- ized W BA via SVD-decompositton
Setup. We apply this adaptation procedure to four publicly available L.MMs: Pythia-2.8hh 2.88, GPT-Neo- 2.7B, Gemma-2B, and Qwen2-0.58. After replacing and initializing the linear layers in their MLP blocks as described above, we fine-tune each model on downstream task. For our primary analysis with Pythia-2.8B. we use Natural Language Inference (NLI) benchmark which requires three-way classification to determine a premise entails, contradicts, or is unrelated to a given hypothesis. The fine-tuning for all models uses the uncertainty-aware objective described in Section 2.2
8

---

Published as a conference paper at ICLR 2026
FLOps Savings (%)
Savings
Savings
FLOps Savings 1%)
$8555555555
Accuracy (%)
Pythia-2.88
Gemma-28
0.8
0.5
Matrix Rank
Matrix Rank
Matrix Rank
Matrix Rank
Baseline
Figure 7: Accuracy vs rank trade-off for pre-trained LLMs with surgically adapted NSN layers. We can obtain large reduction in computational cost with minimal decreases in performance.
Results. Our results demonstrate that NSNs unlock a smooth and predictable compute-performant frontier for large, pre-trained models. For instance, Pythia-2.8B exhibits a monotonic degradation in accuracy the rank- and therefore the operational FLOPs is reduced (Fig. 77. This granular control allows for substantial efficiency gains with a modest performance trade-off; for instance, a 50% reduction in computational cost is achieved with only 5 percentage point drop in accuracy. We find this behavior is consistent across all four tested language models, where performance drops smoothly as the matrix rank is decreased. This establishes NSNs as an effective method for post-hoc adaptation of foundation models to dynamic inference scenarios.
Takeaway. NSNs can be surgically applied to large, pre-trained foundation models, which allows for smooth and predictable compute-performance trade-offs.
RELATED WORK
On the one hand, static compression methods aim to create smaller, more efficient, but ultimately fixed models from a larger pre-trained one. Techniques like network pruning (Han et al. [015b] Blalock et al. 2020) and knowledge distillation (Gou al.) 202T) excel at this, producing highly optimized artifacts for specific computational target. However, this approach is fundamentally statie; adapting the model to a new computational budget requires repeating the entire, often costly, compression pipeline, failing to provide the on-the-fi adaptability needed for dynamic environments.
On the other hand, dynamic neural networks (Han et al., 2021) are designed with inference- time adaptability in mind. Slimmable networks, for instance, allow for channels to be dropped dynamically to create sub-networks of varying widths (Yu et al.) 2018). While these methods offer the desired adaptability, they typically require specialized and complex training schemes that are applied from scratch (Cai et al., 2019). This makes them difficult to apply to the vast ecosystem of existing, pre-trained foundation models. Furthermore, many such techniques offer only coarse, discrete set of operating points rather than a smooth, continuous trade-off (Teerapittayano et al.) 2016).
More recently, paramette-efffcieett fine-tuning (PEFT) methods like Low-Rand Adaptation (LoRA) (Hu et al. 2022) have become a popular way to efficiently adapt large models. While LoRA also employs low-rank factorization, its goal to learn a single, static update for a fixed rank r. It is not designed to be dynamically adjusted at inference time; changing the computational budget would require training a new LoRA adapter with different rank. In contrast, NSNs leverage nested low-rank parameterization to enable a single model to be granularly and dynamically adjusted across an entire spectrum of ranks test time.
Beyond the themes discussed above, Appendix A proovides a broader survey situating NSNs within several additional research traditions. It outlines how classical flag-manifold methods study nested subspaces from geometric perspective, contrasting these representation--sace approaches with NSNs' parameter-space formulation. It also reviews dynamic-inferen infereece architectures such as Mat- Former, Flextron, and LLAMAFLEX, highlighting how these systems rely structural slicing or routing rather than the continuous, based hierarchy central to NSNs. Finally, the appendix connects NSNs to adjacent areas areas-including other low-rank adaptation, adaptive and robust MI.,
9

---

Published as a conference paper at ICLR 2026
Table A comparative analysis against our three desiderata for efficient, adaptable architec- tures. Recall the three desiderata where we seek a solution that: D1: learns a single, unified Trade-off Parametrization that allows for instant Test-tii time adaptabilityy)) D2: is broadly applicable through Post- Post-Training Re-puramelerrzztioon" and exhibits Architectural Agnosticism to modify existing pre-trained models; and D3: provides granular control by generating a Smooth trude-off Frontier across continuous spectrum of computational budgets. Our proposed method, Nested Subspace Networks (NSNs), is the first to satisfy all five criteria.
Method
Example
Trade-oft Paramitrization
Test time Ackittabiliy
Tiraning
Architectural Agrosficiim
Smooth Trade- Fiveriey
Re-pranzteriztiie"
Standand LoRA
Flu et al. 2023
DyLoRA
2012
LoK.N-Pruning
Universal Slimmable
Hunn
Base-for-AlllOPa
mr
Oiher
licrative Magnituee
Jan 3l 301.6
8
Movemeni Praning
Naeh et 27/
3
Response Bised K.
Falte 2013
/
Self Desill, (Early Exit)
Teerapillay
2016
Nested Subqpace Networks
and test-time adjustment sinneworks-clarifyyie complementarities and highlighting the distinctm echanism how NSNs induce order, controllable capacity and enable adaptation to foundation models.
DISCUSSION
The dominant paradigm deploying large models involves creating static artifacts, each trained for a fixed computational budget. This approach is ill-suited for dynamic environments where resource constraints can change on-the-fly. Contrary to the view that this requires training a discrete collection of specialist models- computationally prohibitive approach- make strong case that single, well-trained dynamic network can effectively and efficiently navigate this trade-off.
We introduced Nested Subspace Networks (NSNs), novel architectural paradigm that represents a continuous hierarchy of models within a single set of weights. We propose a structural design based on the nested subspace property that has practical, uncertainty- We show how an entire family of models can be optimized jointly. We further demonstrated that NSNs can be surgically applied post-hoc large, pre-trained foundation models, unlocking smooth and predictable campute-performen paper presents the first-ever approach dynamically convert any pre-trained foundation model to compute-adjustable model with minimal fine-tuning.
training objective.
frontier without requiring training from scratch. This
Why and how do NSNs work so well? We probe this question in our insights experiments (Sec. C Our analysis reveals that NSNs, regularized by their low-rank structure, converge to different local minima in the loss landscape compared to standard fine-tuning. This means that NSNs find distinct yet highly effective solutions in the loss landscape that allow the family of nested sub-models to work well (Sec. C.3p. We empirically verified the foundational nested subspace property, confirming that the vector spaces of lower-rank models indeed contained within those of higher-rank models, as intended by the architectural design (Sec. C.2). Furthermore, we attempt to understand what happens change the rank of the network. We find that the low-rank constraint
to the network structure acts as a bottleneck that encourages layers to learn redundant, globally useful functions (Sec. C.I). As capacity increases with higher ranks, layers diverge, adopting more specialized roles. Currently, NSNs shrink or augment all layers to the same rank; we think an interesting-ant nontrivial- future work is to develop layer-specific mechanisms for adaptive compute. This requires, however, solving the difficult problem of correlating problem-specine information with layer-specific representational capacity, a problem that has so far attracted little attention. Our findings and insights establish NSNs as powerful framework for creating the next generation of adaptive foundation models.
10

---

Published as a conference paper at ICLR 2026
Reproducibility statement github repository.
We confirm that all work and figures can be reproduced in the provided
Impact Statement This work advances the field of machine learning and has many downstream applications. While we primarily expect the use of nested subspace networks to be used for advancing positive goals, e-g. making existing systems compute-adapting and efficient, it plausible that such techniques could be misused for terminal goals which require the same neural network capabilities. The techniques provided in this paper are designed and tested for language models but can extend well beyond them to any neural network architecture.
Acknowledgements We would like to thank the reviewers for their helpful feedback. Claudio Fanconi provided very helpful feedback on an early version this paper. We thank Atlas Wang for his public engagement with our work which has resulted in us revising our related work. This work was supported by Azure sponsorship credits granted by Microsoft's Al for Good Research Lab.
REFERENCES
Supriya Agrahari and Anil Kumar Singh. Concept drift detection data stream mining: A literature review. Journal of King Saud University-Compee and Information Sciences, 34(10):9523 9540--900---00 2022.
Zhiwei Bai, Jiajie Zhao, and Yaoyu Zhang. Connectivity shapes implicit regularization in matrix factorization models for matrix completion. Advances in Neural Information Processing Systems, 37:45914-45955, 2024.
Firas Bayram, Bestoun S Ahmed, and Andreas Kassler. From concept drift to model degradation: An overview on performance-aware drift detectors. Knowledge-Baud Based Systems, 245: 2022.
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? Proceedings of machine learning and systems, 2: 29121464, 2020.
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. forallor-all: Train one network and specialize for efficient deployment. arXiv preprint arXiv: 09791, 2019.
Ruisi Cai, Saurav Muralidharan, Greg Heinrich, Hongxu Yin, Zhangyang Wang, Jan Kautz, and Pavlo Molchanov. Flextron: Many-in-one flexible large language model. arXiv preprint arXiv:2406. 10260, 2024.
Ruisi Cai, Saurav Muralidharan, Hongxu Yin, Zhangyang Wang, Jan Kautz, and Pavlo Molchanov. Llamaflex: Many-in-ou Ilms via generalized pruning and weight sharing. In The Thirteenth International Conference on Learning Representations, 2025.
Tianyi Chen, Tianyu Ding, Badal Yadav, Ilya Zharkov, and Luming Liang. Lorashear: Efficient large language model structured pruning and knowledge recovery. arXiv preprint arXiv:2310.18356. 2023.
Zhao Chen, Vijay Badrinarayanan, Chen-Ye Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balaneing in deep multitask networks. In International conference machine learning. pp. 794 803. PMLR, 2018.
Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. survey of model compression and acceleration for deep neural networks. arXiv preprint arXiv:1710.09282, 2017.
Fnu Devvrit, Sneha Kudugunta, Aditya Kusupati, Tim Dettmers, Kaifeng Chen, Inderjit Dhillon, Yulia Tsvetkov, Hanna Hajishirzi, Sham Kakade, Ali Farhadi, et al. Matformer: Nested transformer for elastic inference. Advances in Neural Information Processing Systems, 37: 37:140555-140564 2024.
Bruce Draper, Michael Kirby, Justin Marks, Tim Marrinan, and Chris Peterson. A flag representation for finite collections subspaces mixed dimensions. Linear Algebra and its Applications, 451: 15-32, 2014.
11

---

Published as a conference paper at ICLR 2026
Anton Dries and Ulrich Rückert. Adaptive concept drift detection. Statistical Analysis and Data Mining: The ASA Data Science Journal, 1):3712
Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. arXiv preprint 1rXiv::909.11555 2019.
Claudio Fanconi and Mihaela van der Schaar. Cascaded language models for cost-effective human-ai decision-making. arXiv preprint arXiv:2506. /887, 2025.
Dewan Md Farid, Li Zhang, Alamgir Hossain, Chowdhury Mofizur Rahman, Rebecca Strachan, Graham Sexton, and Keshav Dahal. An adaptive ensemble classifier for mining concept drifting data streams. Expert Systems with Applications, 40(155:5895-59066 2013.
Joao Gama, Pedro Medas, Gladys Castillo, and Pedro Rodrigues. Learning with drift detection. In Advances in Artificial Intelligence 2004: 17th Brazilian Symposium on Artificial Intelligence, Sao Luis, Maranhao, Brazil, September 29-Ocotber 1, 2004. Proceedings 17, pp. 186-2995. Springer, 2004.
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International journal of computer vision, 129(6)11799-188 1819, 2021.
Ben Halstead, Yun Sing Koh, Patricia Riddle, Russel Pears, Mykola Pechenizkiy, Albert Bifet, Gustavo Olivares, and Guy Coulson. Analyzing and repairing concept drift adaptation in data stream classification. Machine Learning, 111(100)3489-3222 2022.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint 1rXiv::510.000 2015a.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015b.
Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang. and Yulin Wang. Dynamic neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(11): 1433-7456,220 2021.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0255: 2015.
Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. Dynabert: Dynamic bert with adaptive width and depth. Advances in Neural Information Processing Systems, 33: 9782-9793, 2020.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Geoff Hulten, Laurie Spencer, and Pedro Domingos. Mining time-changing data streams. In Proceedings of the seventh ACM SIGKDD international conference Knowledge discovery and data mining, pp. 97-106, 2001.
Ajay Jaiswal, Yifan Wang, Lu Yin, Shiwei Liu, Runjin Chen, Jiawei Zhao, Ananth Grama, Yuandong Tian, and Zhangyang Wang. From low rank gradient subspace stabilization low-rank weights: Observations, theories, and applications. arXiv preprint arXiv: //339, 2024.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7482-7491, 2018
Changlin Li, Guangrun Wang, Bing Wang, Xiaodan Liang, Zhihui Li, and Xiaojun Chang. Dynamic slimmable network. Proceedings of the IEEF/CVF Conference on computer vision and pattern recognition, pp. 8607-8617, 2021.
Shikun Liu, Edward Johns. and Andrew J Davison. End-to-end multi-task learning with attention. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1871-1880, 2019.
12

---

Published as a conference paper at ICLR 2026
Jie Lu, Anjin Liu, Fan Dong, Feng Gu, Joao Gama, and Guangquan Zhang. Learning under concept drift: A review. IEEE transactions on knowledge and data engineering, 31 (1(12)2234-2366 2018.
Xiaofeng Ma, Michael Kirby, and Chris Peterson. The flag manifold tool for analyzing and comparing sets data sets. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4185-4194, 2021.
Nathan Mankovich, Emily J King, Chris Peterson, and Michael Kirby. The flag median and flagirls. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp- 10339-10347, 2022.
Nathan Mankovich, Gustau Camps- Valls, and Tolga Birdal. Fun with flags: Robust principal directions via flag manifolds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 330-340, 2024.
Nathan Mankovich, Ignacio Santamaria, Gustau Camps- and Tolga Birdal. A flag decompo- sition for hierarchical datasets. Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 18738-18748 18748, 2025.
Du Nguyen. Closed-form geodesics and optimization for riemannian logarithms of stiefel and flag manifolds. Journal of Optimization Theory and Applications, 194(1):142-166, 2022.
Xavier Pennec. Barycentric subspace analysis on manifolds. 2018.
Stephan Rabanser, Stephan Günnemann, and Zachary Lipton. Failing loudly: An empirical study of methods for detecting dataset shift. Advances in Neural Information Processing Systems, 32, 2019.
Paulius Rauba, Nabeel Seedat, Krzysztof Kacprzyk, and Mihaela van der Schaar. Self-healing machine learning: A framework for autonomous adaptation in real- world environments. Advances in Neural Information Processing Systems, 37:42225 42266,, 2024a.
Paulius Rauba, Nabeel Seedat, Max Ruiz Luyten, and Mihaela van der Schaar. Context-aware testing: A new paradigm for model testing with large language models. Advances in Neural Information Processing Systems, 37:1 STTE111555-112255
2024b.
Paulius Rauba, Qiyao Wei, and Mihaela van der Schaar. Statistical hypothesis testing for auditing robustness in language models. arXiv preprint NKKV:25006.09447 2025.
Haider Raza, Girijesh Prasad, and Yuhua Li. Adaptive learning with covariate shift-detection for non-stationary environments. In 2014 14th UK Workshop on Computational Intelligence (UKCI), pp. 1-8. IEEÉ, 2014.
Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning: Adaptive sparsity by fine-tuning. Advances in neural information processing systems, 33:20378 -20399, 2020.
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. Advances in neural information processing systems, 31, 2018.
Tom Szwagier and Xavier Pennec. Nested subspace learning with flags. arXiv preprint arXiv:2502.06022 2025.
Surat Teerapittayanon, Bradley McDanel, and Hsiang-Tsung Kung. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd international conference on pattern recognition (ICPR), pp. 2464-2469. IEEE, 2016.
Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. arXiv preprint arXivv2210.07558, 2022.
Mojtaba Valipour, Mehdi Rezagholizadeh. Hossein Rajabzadeh, Parsa Kavehzadeh, Marzich Tahaci, Boxing Chen, and Ali Ghodsi. Sortednet: A scalable and generalized framework for training modular deep neural networks. arXiv preprint arXiv:2309.M0255 2023.
13

---

Published as a conference paper at ICLR 2026
Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S Davis, Kristen Grauman, and Rogerio Feris. Blockdrop: Dynamic inference paths in residual networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8817-8826, 2018.
Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. Deebert: Dynamic early exiting for accelerating bert inference. arXiv preprint arXiv:2004.1290 2020.
Guangzhi Xiong, Eric Xie, Amir Hassan Shariatmadari, Sikun Guo, Stefan Bekiranov, and Aidong Zhang. Improving scientific hypothesis generation with knowledge grounded large language models. arXiv preprint 1XXi:22411.02388 2024.
Ke Ye and Lek-Heng Lim. Schubert varieties and distances between subspaces of different dimensions. arXiv preprint 1xXvv0400.00000 2014.
Ke Ye, Ken Sze-Wai Wong, and Lek-Heng Lim. Optimization on flag manifolds. Mathematical Programming, 194(1):621-60, 2022.
Jiahui Yu and Thomas Huang. Universally slimmable networks and improved training techniques.
In Proceedings of the IEEF/CVF international conference computer vision, pp. 1803-1811, 2019.
Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. arXiv preprint arXiv:1812.0892. 2018.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter-efficient fine- tuming. arXiv preprint arXiv:2303. 10512, 2023.
Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy pruning for model compression. arXiv preprint
2017.
Xiaojing Zhu and Chungen Shen. Practical gradient and conjugate gradient methods on flag manifolds. Computational Optimization and Applications, 88(2):491-52 2024.
14

---

Published as a conference paper at ICLR 2026
A EXTENDED RELATED WORK
In this section, we discuss related work which is not immediately related to our discussed problem setup (Sec. yet we find important to cover to better position this work.
Classical Bag-manifold literature The flag-maniton manifold literature studies nested subspaces to guaran- tee multi-resolution consistency representations. Classical foundations include Pennec's formula- tion of PCA and barycentric subspace analysis on flags, which motivates optimizing over sequences of projectors rather than a single Grassmann subspace (Pennec 2018]; recent geometric toolkits provide
algorithms and distances for optimization on flag manifolds (Ye et al. 2022 Nguyen 2022 Zhu &
Shen 2024 Ye & Lim 2014). Building on this, Szwagier & Pennec (2025) propose the "flag trick" which replaces single projector by average multilevel projector to enforce nestedness across dimensions. Parallel work develops flag-centric representations and statistics (Draper et al. 2014
Mankovich et al. 2022 Ma et al. 2021) and robust principal directions via "flagification" (Mankovich et al. 2024). Mankovich et al. (2025) introduce a flag decomposition that factorizes hierarchical datasets into presented in this paper is orthogonal to the flag-manifold literature and differs in mechanism, scope,
flat via block-modifies
algorithm. The work
and purpose.
In terms of mechanism; instead of optimizing data projections flag manifold, NSNs enforce a parameter-space filtration inside every linear layer, and couple ranks with an uncertainty-weigghee multi-rank objective which yields a smooth compute- performance frontier.
In terms of scope, flag-based approaches treat the nested structure as the object of opti- mization in representation space and are typically applied to moderate-dimensione linear features, whereas NSNs treat nestedness as an architectural prior that can be injected into arbitrary deep networks (c.g., transformers, CNNs).
In terms of purpose: the nature of NSN work is highly practical advance the dynamic inference paradigm by introducing an algorithm that obtains different properties from other dynamic inference algorithms. We see that the existing flag-based literature can be applied to enhance our proposed modeling paradigm, but our work does not rely on the explicit setups within the clssical flag- menifoll literature.
Slimmable and Universally Slimmable networks Slimmable Yuu et al. 2018) and universally- slimmable networks (Yu & Huang 2019) train single model that runs at multiple channel widths, enabled by "sandwich" "and in-place distillation rules (that this paper shows are unnecessary). In contrast slimmable networks, whose behavior between trained widths controlled only empirically through regularizers such as the sandwich rule, the nested subspace structure of NSNs lets us bound the change expected loss between any two ranks, yielding a theoretically controlled interpolation between compute budgets. Moreover, universally slimmable networks (Yu & Huang] 2019) require width-specific normalization statistics and repeated width sampling during training, which makes them costly and hard to use post-hoc adapters for large foundation models, whereas NSNs share single set of parameters and normalization across all ranks and can be added by a short SVD-based fine-tuning phase.
Further discussion on dynamic inference via nested/clastic architectures Once-for-all extends this idea by training a supernet whose sub-networks are specialized post-training (Cai et al. 2019).
For Transformers, MatFormer nests feed-forward blocks to slice width at inference, relies on supernet training and produces a finite sub-model choice set (Devvrit al. 2024). Flextron (Cai et al. 2024) converts a pretrained LLM into a nested elastic network and learns routers (static or input-adaptive) that select heads/neurons per budget, after continued training: provides many sub-models but through discrete head/width choices and routing rather than single operator with continuous capacity. LLAMAFLEX (Cai et al.] 2025) similarly starts from a pretrained LLM and trains a weight-shared, depthh/iitt--lliibble architecture and a Gumbel-Softmax router to "train once, deploy many", , interpolating between a set of anchor budgets. NSNs, compared to LLAMAFLEX. take a very different approach: we reparameterize linear layers into nested low-rank subspaces and train a single hierarchy of ranks, yielding smooth, theoretically bounded performance-coe compute curves even interpolated ranks. Therefore, NSNs are particularly attractive when architecture-agnosicc,
15

---

Published as a conference paper at ICLR 2026
traning-efffiient and granular control over compute budgets is required, rather than the router-driven depth/width slicing emphasized by LLAMAFLEX. Early-exit and dynamic-depth approaches like et al. 2016). LayerDrop (Fan al. 2019) or DeeBert (Xin et al. (2020)
cut computation by skipping layers or exiting early which changes where computation happens, not
how expressive each linear map is. They also yield discrete exits and require auxiliary heads or
vraining-time regularization. Another close in spirit approach is SortedNet |Valipour et al.) (002)) which enforces a generalized "sorted" (partly nested) parameter sharing scheme and trains many discrete sub-models via random sub-model sampling with gradient accumulation. In contrast to all of these, NSNs re-parameterize each linear layer as a single pair of factors whose first rank- components define an exact subspace of the r+1 model. We show jointly optimizing ranks with an wuvertainty-weiihheee two-rank objective gives smooth predictable interpolation across all ranks with theoretical guarantees, that we can employ SVD initialization to allow post-hoc surgical adaptation to pre-trained LLMs (without relying on knowledge-distillatdd distillall,,,,,,, neural architecture searches or other architecture-speee sperrfic work), and that this enables clear parameter sharing where the most important information is naturally ordered in the basis vectors based on the order of the ranks (which we show in Appendix C.4)
Low-rank adaptation literature A similar but functionally different literature is the low-rank adaptation and layer-adaptive rank selection literature. LoRA (Hu et al. (022) and its adaptive variants modify or fine-tune models in low rank, but they target fixed ranks per deployment; AdaLoRA (Zhang et al. 2023) reallocates rank across layers during fine-tuning yet still produces static configuration at interence. WeLore (Jaiswal et al. 2024) studies why low rank emerges in LLMs (via Maudient-Hessing subspace stabilization), then performs shot uniform rank projection and LRC-focused PEFT approach. While it offers strong compression and fine-tuning it chooses a fixed per-layer rank profile instead of training one model to operate continuously across many ranks at test time. DynaBERT (Hou et al. (020) provides dynamic width/depth BERT variants through distillation and importance re-writing-agee again discrete structural sub-networks rather than a single operator with nested rank. Complementary theoretical work analyzes the implicit regularization of overparameterized matrix factorization for matrix completion, showing that gradient flow traverses hierarchy of invariant manifolds and that the limiting solution transitions from minimum nuclear norm to minimum rank as the connectivity of the observed entries increases (Bai et al. 2024). This is related in spirit to NSNs- exploit low-rank factorizations trained by gradient methods but our approach explicitly parameterizes nested rank hierarchy to shape the cemputee-peformaane
frontier, rather than relying solely such connectivity-drive implicit biases. Relative to these lines, NSNs: (i) target rank the adaptation axis so the function class at rank r is a strict subset of rank r + 1; (ii) train all ranks jointly with
(iii) guarantees smooth performance-compur frontiers; (iv) is post-hoc applicable to pre-trained foundation models; and (v) is a standalone model that adapts all weights instead of relying on adapter top of frozen weights.
Adaptive and robust machine learning methods Our work can be also seen as being related to the adaptive and robust machine learning literature. Concretely, NSNs provide an adaptavle mechanism to control performance at test-time. Other such models exist within this area. Work on adapting machine
learning models at test time is rich, both in terms of looking at re-training them (Lu et al. 2018
Raza et al. 2014 Rabanser et af. 2019 Bayram et al., 2022). detecting errors (Agrahari & Singh 2022 Gama et al. 2004 Halstead et al. 2022), using adaptive algorithms (Farid et al.) 2013 Hulten
et al. 2001 Dries & Rückeri (009)) or more robust autonomous approaches for hypothesis-driven adaptation and testing (Rauba al. 2024a). We find many of these works complementary. whereby the adaptation approaches might benefit from Nested Subspace Network-type architectures. However, more research is needed to create mechanisms how to combine NSN-type models with adaptive ML. For instance, adaptive models could be deployed to select which sub-model to use at test-time. while hypothesis generation algorithms (Xiong et al., 2024) or context-awar testing (Rauba et al.. 2024b) can be used to operationally decide what is the minimally performant model required to test a particular set of procedures for cost-sensitive testing. One way to achieve this could be to select the nested subspace model that best satisfies the required criteria after performing model auditing across different sub-models (Rauba et al. (025) or selecting the cheapest sub-model that still matches the requirements of more expensive models via cascade frameworks (Fanconi & van der Schaar 2025). Therefore, our work can be easily extended to the adaptive ML literature but does not directly compete against it.
16

---

Published as a conference paper at ICLR 2026
B EXPERIMENTAL DETAILS
B. DETAILS ON NATIVE-RANK TRAINING VS RANK-TRUNCATION
The experiment presented Figure 2 aims to contrast two approaches for obtaining models at various computational budgets: (i) native-rank training and (ii) rank truncation. All experiments were conducted on the CIFAR-I0 dataset using ImageNet embeddings as input to a Multi-Layer Perceptron (MLP).
Native Rank. For the "Native rank training" bascline, we trained series independent specialist models. Each model corresponds to specific rank {1, 21,2.... 64). The linear layers of each MLP were parameterized using a low-rank factorization W = BA, where the inner dimension was fixed the target rank Every model was trained from scratch for 30 epochs using the same hyperparameters to represent the empirical Pareto frontier of performance for given rank.
Rank Truncation. For the "Rank-64 training" comparison, we trained single model with maximum rank of N = 64. At test time, to evaluate performance at a lower rank r < R, we simply truncated the factor matrices E RRxd and B E RdoueXR, Specifically, the truncated weight matrix Wr was constructed using only the first rows of A and the first r columns of B. This ensures the nested subspace property is structurally satisfied, but as Figure shows, this naive approach fails to train the shared parameters to be effective across the hierarchy of ranks.
B.2 IMPLEMENTATION DETAILS FOR THE ABLATIONS TABLE
For clarity, we summarize how the three ablation variants in Table Logits Regularization, Residual Orthogonality, and Hidden Regularization are implemented and how they relate to the core "Two CEs" objective.
Common setup: anchor / variant ranks and base objective. In all joint-training variants we select an anchor rank R and a strictly smaller variant rank r from the predefined rank set used throughout the paper. Evaluating the NSN at given rank k yields logits fa; k) and the corresponding cross-entropy loss
LCr (A:) =
The base objective (Table I row "Two CFs"))
LTwOCE = LCE(R) + LCE(r).
In the main experiments of Sec. 2.2 we use the uncertainty-weighteed surrogt
CI (cxp(- ix) (CE(R) sR) (exp(- (,) (CE(r) +.),
with learned log-variances a and 8,, All ablations keep this structure fixed and differ only by adding a single extra regularizer. Each row in Table activate multiple additional regularizers once.
[ corresponds to a separate training run; we never
Logits Regularization (*+ Logits Regularization"), This variant encourages the logits at ranks R and to match:
Dlogit
[|; R. f(;
implemented as MSE with the anchor logits treated as a fixed target (no gradient through fo; ft)).
Introducing a log-variance Slogit, the objective becomes
LTwoCE+Loitt =
cune TwOCE +
expp(Stogit Lloght + 2 Slogit
This matches the ce we_with_consisteeeeeeeeeeeeeeeee branch in the code.
Residual Orthogonality (*+ Residual Orthogonality"). Each NSN layer has basis matrix A iitx, where represents the i-th rank-1 direction. Given R. > decompose
A, A A
Ares A Ar41:h:-
17

---

Published as a conference paper at ICLR 2026
We penalize overlap between these subspaces via
Lortho
Σ
| 12
rea Fe
NSN layers
With oog-vvriance Sortho, the objective is
L ResOrco
TwOCE
Luee
exp( Sortho) Lortho +
Sortho
This corresponds to the ce re_orthogonality implementation.
The baseline "One CE Hard Ortho." in Table instead uses only (CE(ft) together with global
orthogonality penalty ||AA" -1 enforcing approximate row-orthonormalit (implemented in
De-orthooonnllity_hee
Hidden Regularization (*+ Hidden Regularization"). For each NSN layer we obtain pre- activation hidden representations at ranks A and r:
Bxd,
h(0 E BBxd
R E
We normalize each along the feature dimension,
normalize (2)
R normalize(H)) (
R
and define a consistency loss
Lient
Σ [ R.00 11
With oor-variance Steat, the full objective is
LTwoCB+Hidden
L
+
2 exp(p (tenn)teet)) Lieat + 2 Steat-
1
This matches the ce_with ceewith__eaturr__cmm steens branch in the code.
Summary. All rows in the "Variations on Joint Training" block share the same NSN architecture and the same anchor/variant rank selection. The base "Two CEs" loss uses cross-entropies at the two ranks; each ablation adds exactly one additional regularizer with its own uncertainty weight. Each variant is trained independently and evaluated separately.
B.3 DETAILS ON CIFAR-10 EMBEDDINGS AND THE MIP SETUP
The dataser is CIFAR-10 throughout but we first map each image to a fixed feature representation using preereee backbone and then train only small classifier top of these frozen features.
Backhone feature extractor. We use the torchvisi on implementation of ResNet-18 with ImageNet-IK pre-trained weights (ResNet18_we) 1__eeiggt_tss__ IMAGENETIK V1) fixed feature extractor. We remove the final classification layer and keep the network up to the global average pooling stage, yielding mapping
R2512 where ((x) is a 512-dimensional feature vector. CIFAR- 10 images are resized to 224 x 224 and normalized with standard ImageNet mean and variance before being passed through d. For each split (train/test), we precompute and store pairs (ob(;;;; y.). The ResNet backbone is kept in evaluation mode and never updated during any of the NSN experiments; gradients are disabled for all its parameters.
3x32x33
For the Base MLP baselines, both linear layers are standard dense linear transformations. For the NSN ("Low Rank Layer") models, both linear layers are implemented as NSN layers (Definition 1), i.e. they use the nested low-rank factorization with shared maximum rank f and can be evaluated at any active rank r < R. No other part of the network (in particular, the ResNet- 18 backbone) is re- parameterized or modified by NSNs. Thus, the CIFAR- 10 experiments should be understood ResNet- 18 - 512-D features, and (ii) all
as: (i) CIFAR- 10 images + frozen IaageNet-pretraiinn subsequent trainable layers in the MIP classifier are NSN (or dense) layers specified in Table
18

---

Published as a conference paper at ICLR 2026
C ADDITIONAL EXPERIMENTS
C.I INTER-LAYER WEIGHT SIMILARITY VS. MATRIX RANK
Objective. This experiment investigates the relationship between the representational capacity of layers and their functional roles within the network. The core question is whether increasing layer capacity (i.e., matrix rank) encourages layers learn more specialized, distinct functions or more similar, redundant ones. The guiding hypothesis is that low-rank constraint acts an informational bottleneck, forcing layers to learn redundant, globally useful functions, while increasing the rank enables and encourages functional specialization.
Methodology. For a trained Nested Subspace Network, we analyzed the weight matrices of the MLP layers various ranks, For each rank from to 1024, we reconstructed the effective weight matrix W. for each layer. We then computed the pairwise cosine similarity between the weight matrices of all layers the network. The average of these pairwise similarities was then plotted against the matrix rank to observe the overall trend.
Results and Interpretation. The results, shown in Figure 8 confirm the hypothesis. At very low ranks, the average inter-layer similarity is high, indicating that the network's s layers learn functionally similar and redundant representations. As the matrix rank and thus the layer capacity increase, the average cosine similarity between layers steadily decreases, approaching zero at the highest ranks. This suggests that with greater representational freedom, layers diverge to assume more specialized roles within the network. The low-rank constraint effectively regularizes the network, forcing layers to cooperate on learning general features, while higher ranks allow for more distributed and specialized division of labor.
Inter-Layer Weight Similarity vs. Matrix Rank
1.0
Average Similarity
Highest similanty 18 and 19) rank
0.8
Cosine Similarity
0.6
Lowest similarity (Layers and 201 at low rank
0.4
0.2
0.0
6
16
32
128
256
512
1024
Matrix Rank
Figure 8: Inter-layer weight similarity function of matrix rank. As layer capacity (rank) increases, the average cosine similarity between layers decreases. This suggests that layers transition from learning redundant, globally useful functions at low ranks to more specialized roles high ranks.
C.2 EMPIRICAL VERIFICATION OF THE NESTED SUBSPACE PROPERTY
Objective. The central design of Nested Subspace Networks relies the nested subspace property, where the function computed at a given rank a strict subspace of the function at any higher rank. This experiment designed to empirically verify if this theoretical property holds in practice after
19

---

Published as a conference paper at ICLR 2026
training. The key question is: does the vector space spanned by lower-rank weight matrix truly lie inside the vector space of a higher-rank matrix from the same trained layer?
Methodology. To quantify the degree of subspace containment, used three-step procedure for each trained NSN layer:
Reconstruct Weights: For pair of ranks, T'mall and F'arge we reconstructed their effective weight matrices, W. Wramn and W Paage"
2. Find Orthonormal Bases: We performed Singular Value Decomposition (SVD) on each weight matrix (W, = U, V²) to find orthonormal basis for its column space. The first r columns of the resulting U, matrix form this basis.
3. Calculate Containment Score:
We computed containment score to measure the extent which the smaller subspace is contained in the larger one. The score is defined as the normalized Frobenius norm of the projection of the smaller basis onto the larger basis:
score(r small:
Plarge
U U Urman
Tamall
A score of 1.0 indicates that the smaller subspace is perfectly contained within the larger one.
Results and Interpretation. The results are visualized in the heatmap in Figure The upper triangle of the matrix, where Targe 2 rsmall shows scores that are consistently 1.0 or very close This empirically confirms that the vector space of a lower-rank model is indeed a nested subspace of any higher-rank model after training. The lower triangle, where Tlarge < Tsmall shows scores significantly less than 1 1.O. This asymmetry is expected and acts as sanity check, confirming that higher-dimension space cannot be fully contained within diwer-dimensional one.
C.3 CONVERGENCE ANALYSIS OF LOW-RANK VS. STANDARD FINE-T TUNNTTNING
Objective. This experiment investigates whether model trained with Nested Subspace layers converges to the same solution the weight space as a model trained with standard fine-tuning. The hypothesis is that the two models will find different solutions, as the low-rank structure of NSNs acts as form of regularization that guides the optimization process toward a different local minimum.
Methodology. To compare the final learned weights, standard model was fine-tuned the task, and separate NSN-equipped model was trained using our proposed multi-rank objective. For the NSN model, the effective weight matrix for each layer was reconstructed at various ranks. We then computed the cosine similarity between the weight matrix of a layer from the standard fine-tuned model and the corresponding reconstructed matrix from the NSN model. This comparison was performed for all MLP layers, which were grouped into early (0-10), middle (11-20), and late (21-31) stages the network to observe depth-dependent trends.
Results and Interpretation. As shown in Figure 10 the weight matrices of the NSN model do converge to the same solution as the standard fine-tuned model. The cosine similarity increases with the rank, but even at the highest rank (1024), the similarity is only around 85
This result supports the hypothesis that the nested low-rank structure imposes a regularization effect. By constraining the possible solutions lie within pre-defined low-rank subspaces, the training process is guided to a different local minimum in the loss landscape than standard, unconstrained fine-tuning. This suggests that NSNs discover different, yet highly effective, set of parameters for solving the task.
C.4 VERIFYING ENERGY DECAY ASSUMPTION
To empirically evaluate Assumption 1, we perform an empirical investigation on chosen language model and multiple layers within this model. Specifically, directly inspect the learned basis vectors of every SyyaaiccLowRankLinea. layer in the NSN-adapted GPT-NeoX model (chosen for convenience).
20

---

Published as conference paper at ICLR 2026
How much is each weight Wea matrix contained in another W matrix?
1-
0.999
0.999
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
2
0.500
1.000
1.000
1.00
4
0.250
0.500
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
0.88
8-
0.125
0.250
0.500
0.999
1.000
1,000
1.000
1.000
1.000
1.000
1.000
Smaller Rank (fsmait)
Containment Score
16
0.062
1.000
0.6
32
0.031
0.062
0.125
0.250
0.5.0
1.000
1.000
1.000
1.000
1.000
1.000
64
0.016
0.031
0.063
0.1.5
0.250
0.500
0.999
1.000
1.000
1.000
1.000
0.008
0.016
0.033
0.062
0.125
0.250
0.500
1.000
1.000
1.000
1.000
0.2
256
0.004
0.008
0.016
0.031
0.062
0.125
0.250
0.500
0.999
1.000
1.000
512
0.002
0.004
0.008
0.016
0.031
0.063
0.125
0.250
0.500
1.000
1,000
1024
0.001
0.008
0.031
0.062
0.500
1.000
i
2
16
32
64
128
256
512
1024
Larger Rank (Maarge)
Note: Average score across all layers. score 1.0 means low-rank space fully contalind larger one.
Figure 9: Hcatmap of subspace containment scores between weight matrices of different ranks. The score measures the extent to which the column space of lower-rank matrix (Fsmali) contained within that of a higher-rank matrix (Mluge)). A score of 1.0 (dark blue) indicates full containment. The results empirically validate the foundational nested subspace property of the trained network.
Setup. Each such layer is parameterized as W, E.= b,a,, where the rows of provide the a components and the columns of B provide the b; components. For every layer, we compute the Euclidean norms || |all2 and |lb,|| across all rank- components i = 1...., R and test whether these sequences monotonically non-increasing. This monotonicity captures the "energy decay" structure posited by Assumption 1, which states that earlier basis components should contain more salient functional information than later ones. For each layer, we report: the maximum rank R, whether monotonicity holds (T/F), the number of violations, and the magnitude of the first and last component norms. This provides layer-by-layer diagnostic of how strongly the trained model conforms to the nested subspace ordering implied by our theoretical analysis.
Takeaway. The results reveal that most layers exhibit clear decaying trend in the norms of their rank-1 1 components, even when strict monotonicity is not perfectly satisfied. Violations are typically small and localized, while the overall decrease between the first and last components remains substantial. In total, they constitute 0.04% of all basis vector orderings which is extremely negligible; and we interpret this as noise the optimization process. This provides strong empirical support for Assumption 1: the optimization process tends to allocate high-energy, high-importance directions to early basis indices, enabling the smooth interpolation behavior and predictable compute-performet performance trade-offs that NSNs rely
To make the picture more precise, we also added a layer-by-laye table reporting how often the assumption is locally violated. These violations happen maasiiinallyttyiiallll one two indices within a layer. We estimate this amounts to only about one percent all basis vectors. Because they sparse and small, and given how consistently (about 99% of all basis vectors) this assumption
21

---

Published as a conference paper at ICLR 2026
Low-Rank Model Convergence to Standard Solution
1.0
0.8
Cosine Similarity to Standard Model
0.6
0.4
Early Layers (0-10)
0.2
Middle Layers (11-20)
Late Layers (21-31)
16
32
64
128
256
512
768
1024
Matrix Rank
Figure 10: Cosine similarity between weight matrices from standard fine-tuned model and Nested Subspace Network. The similarity increases with rank but never reaches 1.0, indicating that the low-rank constraint guides the NSN different, yet effective, local minimum in the loss landscape.
Encrgy vostra Dylunkkuneunnthhcr kyer
decay of bus vecters acosa SysemkLentu-kineer laysts
2.5
3.
2.0
Eneryy decay
vedtes scrss Droonkunetnttiier lyees
Energ decay of acess SysamiLawtt kiinea layers
25
24
230
522
1000
(a) Short-horizon energy decay. Mean and standard deviation of ||a]|2 and [bblla over the first 24 basis components, showing the local decay structure across layers.
(b) Full-range energy decay. Decay profile across entire rank spectrum, illustrating the global monotonic trend implied by Assumption 1.
Figure 11: Empirical energy decay patterns across NSN layers. Each plot aggregates the norms of rank-1 components b.) across WynamicLowRRnkLLing layers in the NSN-midde GPT-NeoX model. The short-range plot highlights early-index behavior, while the full-range plot captures the complete structural decay. Both views provide complementary evidence supporting the energy-ordering behaviour predicted by Assumption 1.
holds, we interpret them as noise in the optimization process. them as minor fluctuations introduced by the optimization dynamics (Table
22

---

Published as a conference paper at ICLR 2026
Layer
R
A-tton
A-Viol.
B-aton.
B-viol.
Adys
Absi
Birs
1024
3.141
1.405
5.141
LO 4b-+h
1024
2.578
1.188
2.562
1.297
L. 4h-+h
1024
2.797
1,195
2.781
L2 b-+4h
1024
O
F
3.828
1,336
3.828
L.2 4h-+h
1024
2,969
1.258
2.953
h-144
3,562
L3 4h-+h
1024
2.578
1.234
2.578
1.4 h-94t
1024
3.297
1,375
3.797
L4 4h
1.227
2.928
1.5 4h
1024
2.609
1,266
2,009
L6 b-
1024
F
3.625
1.352
3.625
L6 4b
2.531
1.281
2,511
h-14
L7 4h-th
1024
2.641
1,281
2.611
L8 h-14h
1024
3.641
1.344
3.641
b-
1.9 40 4h
1024
4.4
1,273
2.484
L10
1024
3.562
1.336
3,502
4h-+h
2.312
1,273
2.312
h-34
3-516
3.516
L11 4h-th
1024
2.312
1,281
2.312
1,12 h-th
1024
3.625
1.336
3,625
2.344
2.344
1.13 4h
1024
T
266
1.312
2.366
L14 h +4th
1024
3.656
1,344
3,666
2.266
1.336
2,266
3,656
L15 Ah
1024
2.422
1.344
2,422
L16 b 44
1024
3.672
1,352
3,672
44h
1.17 4h
1024
2.375
1.301
2.375
L15 h 4-4
1024
3.641
1.359
3.641
h-h4h
3.594
1,367
1.19 4h
1024
2.609
1,422
2,625
L20 b- -4t
1024
3.116
1,383
3,516
1,488
2,878
34h
3.438
1.21 4h
1024
T
1.984
1
2.984
1.22 h -4h
1024
3.422
1,405
3.422
3.047
3.375
1,422
3,375
1.23 -h
1024
T
2.734
1,500
2.750
1.24 h +4h
1024
3.344
1,430
3.344
2.484
34h
3.344
3.344
L25 4h++h
1024
T
O
2.328
1,523
2.338
1,26 h +4h
1024
F
3.312
1,488
3,312
2.234
1024
3
1,438
1.28 h-h4
1024
3.281
1
3.266
1.25 4h-sh
1024
2.391
1,555
2.901
4h
2,625
1,555
2,635
L30 bh
1024
T
0
T
3.297
1.438
3.297
1,30 th++h
1024
3,688
1,531
3,688
1.422
3.562
L31 4h
1024
3.688
1.445
3,688
Table Empirical evaluation of Assumption across Dynamic
layers. For each layer, we compute the norms of the rank-1 components (a:, b;). Columns indicate: maximum rank R. monotonicity flags for and B (A-mon., B-mon.), the number violations (A-viol, B-viol.), and the norms of the first and last components, which capture the magnitude of decay. This table quantifies how consistently the trained NSN architecture orders its basis directions by importance.
C.5 ENERGY PROFILES IN STANDARD DENSE MODELS
To complement the analysis in Appendix C.4 perform the same diagnostic procedure on a standard, unmodified GPT-NeoX model whose MLP blocks use conventional dense linear transformations. This comparison isolates whether the energy decay structure observed in NSNs also appears in ordinary architectures, or whether it instead a property induced by the NSN reparameterization and training objective.
23

---

Published as a conference paper at ICLR 2026
Danus mede: row wax energ? proffs horazn)
Dorae medet wo whe eneyy protil tlong fortear)
2Lind
269
660
1bkc
Danue modai: or wnee energy staflie Gherr
Derse mozei colammn energy prufie heriont
2.8
1.7
2.14
1.50
12
1.2
1200
(a) Short-horizon energy profile in dense model. Mean and standard deviation of row column norms over the first 24 indices for the dense GPT- NeoX MLP layers. The profiles are essentially flat, indicating no systematie dependence of weight energy on index; all weights have comparable energy.
(b) Full-range energy profile in a dense model. Row- and column-wise norms across the entire index range show no clear trend or decay, consistent with an ab- sence of any ordered structure in the weight energies. All indices exhibit similar magnitude, confirming that there effectively relationship between index and energy.
Figure 12: Lack of ordered energy decay in standard dense GPT-NeoX MLP layers. Unlike the NSN-adapted model, where rank-1 components exhibit a clear energy decay with basis index, the dense model's 's and column norms are nearly constant across indices. This indicates that the dense parameterization does not naturally impose an ordering of directions by energy: all weights have effectively the same energy, and there is meaningful relationship between index and importance.
Setup. For each dense MLP layer, we take the weight matrix W E IRdw and examine its rows and columns directly, without any factorization. We define
Wa.: b, = W.,
and compute the Euclidean norms ||a:||2 and ||b:l2 across all row and column indices. This is the direct analogue of the NSN analysis: if dense models naturally encode more important directions earlier in their parameterization, we would observe structured energy decay across indices. We evaluate monotonicity, quantify violations, and compute layer-averaged energy profiles exactly as in the NSN case. The resulting aggregated profiles are visualized in Figures 12a and 12b
Findings. Across all layers, the energy profiles are essentially flat: both row norms and column norms remain nearly constant a function of index. Unlike the NSN-adapted model, where low-index basis vectors consistently exhibit higher energy and a clear decay pattern, the dense model displays no meaningful ordering. Monotonicity is neither present nor expected; the norms fluctuate minimally and show no global trend. This indicates that in standard dense architectures, parameter indices do not correspond to any notion of directional importance, and no analogue of Assumption I emerges from training alone.
Takeaway. The absence of any structured energy decay in dense models highlights a key distinction between NSNs and conventional architectures. Whereas NSNs learn a highly organized hierarchy of basis directions- withh most of the functional energy concentrated in early rank components -denee models distribute energy uniformly with no discernible ordering. This comparison reinforces that the nested subspace structure arises from the NSN parameterization and training procedure rather than from generic properties of large neural networks. Figures 12a and T2b make this contrast explicit: NSNs exhibit sharp, consistent energy decay, whereas dense models show no relationship between index and energy all.
Violation Analysis Setup. To guantify how strongly the dense model violates the energy-ordering property, we compute violation rates in direct analogy to the NSN analysis. For each MLP layer, we examine the sequences of row norms and column norms,
W
b; W.
24

---

Published as a conference paper at ICLR 2026
Violation rate energy decay
(NSN vs dense GPT-NeoX,
50
46.95%
Vielatinn (% adjacent
30
10
0.00%
NSN
Dense
Figure 13: Aggregate violation rate of the energy-decay assumption in NSN and dense models. Bars show the percentage of adjacent index pairs that violate the monotonic decay condition, aggre- gated across all MLP layers. The NSN model exhibits extremely low violation rates (0.00% in this specific run), whereas the dense model shows violation rates that are similar to random orderings (which would be about 50%).
Energy-decay violations by depth
(NSN vs dense GPT-NeoX)
Model
NSN
Dense
Vislation (16 adjacent,
Transformer
Figure 14: Violation rate by transformer block. Violation rates are shown per transformer block for both NSN (circles) and dense (squares) models, with binomial standard errors. The NSN error rates are not visible because they are negligible (below 0.01% 101 on the graph). The dense model seems to have consistent pattern how often the energy decay assumption violated, yet this violation is consistently extremely high. This violation makes sense given that nearby transformer blocks should be correlated. The key takeaway is that regular training schemes do not induce sufficient ordering of basis vectors.
and record how often adjacent pairs fail the monotonic condition ||a: || lll2
|lb.||a >
||b,+ 1|2. Each adjacent index yields a binary event (violation or no violation), allowing us to compute per-layer and per-model violation rates. For fair comparison NSNs, we aggregate all adjacent comparisons across all layers and report both an overall violation rate and a depth-resolved profile with binomial standard errors.
The violation statistics provide complementary perspective to the energy profiles reported earlier. Across all layers and transformer blocks, NSNs demonstrate strikingly consistent adherence to the energy energy-decay structure, with only tiny fraction of adjacent pairs (typically well below 1 1%) violating monotonicity. Dense models, by contrast, display no such structure: violation rates are order of magnitude larger and show systematic dependence depth. Together, these
Takeaway.
25

---

Published as a conference paper at ICLR 2026
results reinforce that the nested subspace ordering is not an incidental artifact of large neural networks but a direct consequence of the NSN parameterization and training objective, which actively induce a stable and ordered hierarchy of basis directions.
26

---

Published as a conference paper at ICLR 2026
C.6 COMPUTATIONAL EFFICIENCY THROUGH SURGICAL REPLACEMENT
Objective. This analysis demonstrates the practical computational benefits of surgically re- placing standard linear layers with rank-adaptive linear layers in existing transformer architec- tures. The goal is to quantify the reduction in maating-point operations (FLOPs) achieved through low-rank decomposition while main- taining the nested subspace structure.
FLOps Savings (%)
68%
52%
45%
41%
39%
38%
37
37%
37%
37%
37%
0.9
Accuracy (%)
0.8
50% FLOps reduction for
5 p.p. accuracy loss
0.7
Methodology. We performed surgical modifi- cations to the Gemma-2B architecture by replac- ing all linear layers within the MLP blocks with rank-adaptive variants. Each original weight ma- trix W was decomposed using Singular Value Decomposition (SVD) to initialize the factorized form W B A, where B and A lower-rank matrices. The MLP blocks, which contain three linear layers with GeLJU activation functions, were systematically converted to support mul- tiple rank configurations while preserving the original model's functionality.
0.6
Baseline
1024
S12
256
128
2
Matrix Rank
Figure 15: Example Surgical Changes to linear layers only on Gemma-2B. The architecture of Gemma-2B. All MLP blocks contain three linear layers with a GeLU activation function. We surgi- cally replace all linear layers with rank-adaptive linear layers, initialized W ~ BA via SVD- decomposition
Results and Interpretation. The surgical re- placement approach enables significant compu- tational savings through reduced matrix oper- ations. By decomposing the original full-rank weight matrices into their low-rank approxima- tions, the number of parameters and corresponding FLOPs are substantially reduced. This modifica- tion allows for dynamic rank selection during inference, providing trade-off between computational efficiency and model capacity without requiring complete retraining of the base model.
27

---

Published as a conference paper at ICLR 2026
D ADDITIONAL THEORETICAL INSIGHTS ON NESTED SUBSPACE NETWORKS
D.1 CONDITION FOR NESTED SUBSPACE PROPERTY
Lemma (Sufficient condition for nested subspace property). rank(Ar) = r all R, then I,) = span(b],.... b,} and therefore Im(W, [ [m(WW+1)).
The SVD initialization as well our rank training objective make rank deficiency extremely rare in practice.
D.2 SIMPLE EXAMPLE OF AN NSN LAYER
Example (Toy NSN layer). To make this construction concrete, consider an NSN layer with
dnn dout = 2 and maximum rank R. = 2. We choose single pair of factor matrices
1 0
-
B B= 0 0
A
0
1
For rank r = 1, we use only the first row of A and the first column of B:
A
0].
Bi
0
W1 BiA
0
0
For rank r = 2, we use all rows/columns:
1 0
A = A,
B,
W2 = B2A2
0 1
Thus, the l and rank-2 effective weights Wi and W2 are both derived from the same factor matrices (A, B). Adjusting the rank simply changes how many basis vectors are active.
D.3 WIIY USE THE UNCERTAINTY-AWABE OBJECTIVE?
NSNs train hierarchy of fanktruncated submodels inside one set of weights by factorizing each linear layer W BA and enforcing rested-subspad structure across ranks. We view training across ranks as multi-task learning problem and propose use Kendall-style objective for this. Empirically, the learned log-variances decrease with rank which we interpret proxy for rank expresiveness. our ablations, we show that two cross-entropies deliver the required gains across ranks and adding additional regularization not productive.
What properties do we look for? We seek a weighting mechanism that (i) automatically adapts the relative importance of ranks without per-rank hyperparameter tuning, (ii) is invariant arbitrary rescalings of the factorization W = BA, (iii) guarantees positive weights, and (iv) is cheap enough to apply inside every NSN layer and every training step. The uncertainty-awaa objective surrogate satisfies these requirements: the reparameterization exp(- (p((sk) yields strictly positive, smoothly varying weights with strictly convex dependence on 8k, so optimization stable, and because operates on loss values rather than gradient norms it is insensitive to the scale ambiguity between A and B.
What are the benefits of using an uncertainty-aware objective? The uncertainty- objective directly addresses the heterogeneous difficulty of learning different ranks: lower ranks exhibit larger and noisier cross-entropy losses, which would otherwise dominate or destabilize the optimization if all ranks were weighted equally. Introducing rank rank-specific log- varianees sy yields effective weights exp(-8x) that adaptively attenuate gradients from high-uncertainty ranks while the additive 8s. term prevents trivial suppression of a task, so the hierarchy is trained jointly but stably in a single objective. This surrogate empirically leads well-behaved performance at low ranks and interpolated ranks that were never explicitly optimized. higher ranks consistently converge to lower log- variances than lower ranks (Fig. 3), providing quantitative measure rank expressiveness rather than treating the rank index as purely architectural hyperparameter.
addition, the learned sy form an interpretable diagnostic:
Would we use a different mechanism for weighting the ranks? In principle, any multi-task reweighting scheme could be applied across ranks, even methods such GradNorm (Chen al.
28

---

Published as a conference paper at ICLR 2026
2018), but these alternatives come with trade-offs that poorly matched to the NSN parameteriza- tion. For instance, gradient-norm. based methods require choosing a reference layer and repeatedly computing per-rank norms, and their behavior is sensitive to the arbitrary scaling between A and B in W BA, so the induced weights can drift for reasons unrelated to rank difficulty. Moreover, aggressively equalizing training rates across ranks can over-emphasize very low ranks early in training and degrade the anchor model, whereas our anchor-variant design intentionally biases optimization toward a strong high-rank solution while still improving smaller ranks. For these reasons we view the uncertainty- aware objective as the most practical default for NSNs, and leave more elaborate, possibly model-specific weighting schemes as future work rather than necessary components of the method.
D.4 PROOF OF PROPOSITION
We seek to demonstrate that the performance of the network at untrained. interpolated rank Tint remains close to the performance at an explicitly trained rank. This property relies on the structure induced by the training process. Let the shared, learned weight matrices be A e R and E Rdax R where b; R be the i-th column vector of B.
the maximum rank. Let a; E 1xd be the i-th row vector of and
Lemma 2 (Adjacent Rank Perturbation). Let r(x;;)==Ei b,a.)xx be the output of the linear layer for an input x at rank r. The perturbation to the output when moving from rank r to r + is bounded by:
[x(xr 1) [xx:r)]l < [b |ar+]] [|x]
Proof. The change in the weight matrix is W,+1 W, = by+1 The change in the output is thus (W,+1 W,)x. Applying the submultiplicatiee property of matrix and vector norms yields the result: ((b,+18,41)x S [b, a1
||x|| S ||b a- ||x]|
The total change the function output between rank Tı and mmm can be expressed as a telescoping sum. By the triangle inequality and Lemma
Proof of Proposition 7
(x; rima f(x;
Σ (f(x;)) [(x; 1))
Tin
I1xxx)) f(x; 1)ll
i=1+1
b
||:ll|
|||||
Due to the Lipschitz continuity of the loss L, the difference in expected error is bounded:
(r(im) E(ri) Le (x) f(x; r1)]]
Substituting the bound the function perturbation and defining C Le B[[[[]]] yields the final result.
D.5 SVD INITIALIZATION
We propose an initialization strategy based on Singular Value Decomposition that preserves the original model parameterization the outset of training. Formally, for given pre-trained weight matrix W, we compute its SVD, W = UEVT, and initialize the factor matrices E dou x and A E R Rxxda using the top ft singular components: Binit := UnV Ei and Aini := /E where UR and Vù contain the first columns of and V, and Σ is the diagonal matrix the top R singular values. This scheme ensures that at the maximum rank Ai, the NSN layer's 's effective weight matrix, W a B reconstructs the original pre-trained matrix W either exactly (if R = R) or with the smallest Frobenius norm (if N R).
29

---

Published as a conference paper at ICLR 2026
D.6 TRAINING COST OF NSN
Compared to a standard dense network with weight matrix W E Rdduxdu whose per-step training cost proportional din FLOPs, training Nested Subspace Network (NSN) layer with maximum training rank R and sampled variant rank R requires per-step FLOPS proportional to (R (din + dour). because each optimization step performs a forward-backward pass at the anchor rank R and another at the variant rank r. Using the break-even rank Rbe
dyder. for which a single dutt
low-rank pass matches the dense cost, the pessimistic case r = R Roe gives a total of about dou FLOPs per step, i.e., most roughly twice the cost of training one dense model with the same input and output dimensions. However, this single NSN training run yields a whole hierarchy of usable ranks test time, if K different computational budgets K' separate dense training runs (total cost z Kiinn domt) with one run whose cost only a constant factor above that single dense model, effectively amortizing the training cost over many operating points.
needed, the NSN still replaces
D.7 ON THE DERIVED FUNCTIONAL. FORM OF THE LOSS IN EQUATION
Why did we arrive the specific functional form in Equation 3 and, concretely, why are we using the exponential as the coefficient?
We start with our goal: We want positive weight for each rank-specific loss that adapts during training but does not collapse to zero or infinity. Let Lı denote the task loss at rank k and define positive weight wi > We can write out two equivalent parametrizations which are useful in our context:
1) Direct optimization over positive weights with log-barrier
min 2-0
Σ
WeLb
log wx
The term log uh prevents wa 0 and yields unique closed-fore form optimum wy for fixed model parameters. We can reparametrize this equation yield an equivalent re-parametrization found in the main body of the paper.
2) Reparameterizing wh - with 8% E ER
Ele Lb: Sk
This matches Eq. (up to a constant), with 8ą = log o
Why this particular form of the optimization? are few different ways to think about it.
(a) Positivity and simple optimization The mapping Wi = e + guarantees wi for all sk- Furthermore, the objective is convex in 5% for fixed Lu:
2²
LE + sk) = e Lx > O. e
If L3 > 0, then _- LL + S1) > for all s. This means the function stricly convex in s.
dsy
L₁ O, then the loss is Sy which is still convex. This convexity is useful property for gradient updates and helps to learn the different contributions effectively, as empirically shown in Fig. 3
(b) Closed-form optimal weights and scale invariance easy closed-form weights. For fixed model parameters,
This parametrization allows to obtain
(e
Li + sk) - - La 1.
Isk
Setting this to zero gives
1
- th
e
w Lx
Ltk
This gives useful properties:
30

---

Published as a conference paper at ICLR 2026
scss-scale invariance: If Lg cLa, then sg + 5 + log while wy 1 remains
unchanged.
Coarse gradient balancing: At the optimum, the contribution of rank
Wk V
VoL.,,
Ls
which prevents dominance by a loss with artificially large scale. This is particularly useful property since expect models with lower ranks to have higher loss due to their (definition- ally) lower expressivity. Recall that Eq. scales gradients by Volnoul = Σx e e Vo Lik- Therefore, jointly learning Sę allows the optimizer to attenuate gradients from noisier ranks (sy large) and amplify gradients from cleaner ranks (sy small), while the term +3g prevents collapse e th + O.
(c) Link to heteroskedastic uncertainty We can think of this loss as being directly tied to heteroskedsatic uncertainty the regression case. Concretely, for Gaussian regression noise, the negative log- ligeliill is
1
residuall| 2 + logo²
20
Setting Sę = log o² gives the structure E =') + sg. Classification lacks a Gaussian residual.
However, it is common in practice to this as surrogate objective. Equation acts surrogate for such Gaussian residual in the classification setting.
D.8 WHY VAR-NNAES ARE EMERGENT PROXIES FOR EXPRESIVENESS OF EACH RANK
Are the oog-aarianees free parameters? The log-variances are not free parameters, but they are trainable parameters. They are not free because in the objective
*"" (c + Sk:
each log-variance 8; is coupled to the rank-k loss. This coupling means their values depend directly on the loss within each model of given rank. However, still learn these values during training.
Why does log-variance serve as an emergent proxy? Short answer: This parameter tracks the residual loss for each rank. Higher residual loss (higher error for given rank) leads to higher learned uncertainty parameter. Thus, it emerges as proxy for expressiveness: higher residual loss indicates less expressive model, and the parameter tracks this loss.
Each rank-mode contributes differently to the training objective. For a rank model, the contribution is
Ls = e *h (kB(k) + 8k, where 8y is the log- log-variance and (k)( is the cross-entropy loss.
After training, at stationary point where the gradient is zero, we have
- 54 LCE(K) + +1=0 0,
0sk
which implies
e *1 Lcb((k) = 1,
and therefore
log LcF(k).
Thus, up to optimization noise and interactions with other parameters, the learned log-variances track the scale of the residual loss that rank.
More expressive vs. less expressive models. A more expressive model can reduce LCE(k) further, which forces 5k to be smaller. A less expressive model is stuck with a higher LCB (k) and therefore learns higher sg. Over training, this creates a relationship in which ranks that explain the data well end up with lower log- variances, while ranks that explain poorly end up with higher log- variances..
31

---

Published as a conference paper at ICLR 2026
Algorithm 2 Forward pass of a Nested Subspace Network (NSN) layer
Require: Input vector batch X RBBxd
Require:
Factor matrices A E RRRxd B E RR bias b Ruu
Require:
Active rank r E [1....,Rt) R}
Ensure: Output logits Y E Bxdu
1: A, < first r rows of
A, E R Rexda
2: B, <- first r columns of B
B, E R
3: H + X
D Project inputs to rank-r subspace, H E Bxr
4: Y < HB, + +1
Map back to output space
5: return Y
Algorithm 3 Forward pass of a Nested Subspace Network
Require: Input batch X
Require: NSN layers (Layer,) )= each with (A₁, Br, by) and shared max rank R
Require: Active rank {I....,,ff R}}
Ensure: Output logits Z
X
2: for = 1 to L do
3:
H <- NSNLayerForward(HH A,, Br, by.r)
Alg.
4:
if é < L then
5:
H < 6(H)
Apply nonlinearity, e.g. ReLU or GELU
6:
end if
7:
end for
8: 7 x- H
b Final logits
9: return
E PSEUDOCODE
32

---

Published as a conference paper at ICLR 2026
Algorithm 4 Multi-rank uncertainty-weiggtee. training for Nested Subspace Networks (anchor at maximal rank)
Require: Training dataset D
{(z,, y4)}
Require: Maximal rank R and set of trainable ranks K. C (1..... R} {A₁, Be, b1)/=1
Require: NSN model with parameters
Require: Rank-specific log-variances {sk REK with 8% log(o
Require: Optimizer Opt
Ensure: Trained NSN parameters and log- log-variances {
1: Initialize 0 and set sę <- 0 for all E K
2: for each training step do
3: Sample a minibatch (X,Y) ~
x-1 R 4: R
Anchor rank is always the maximal rank
5:
Kvar + t RJ
6:
r UnitormSample((C. (K
Variant rank is sampled from lower trainable ranks
7:
Set model rank factive f-
8:
Zn -NSNForward (active)
9:
CCE(R) t CrossEntropy (Z 1g:Y2 Y)
10: Set model rank Factive <-
Z, +- NSNForward (X, ractivo)
12:
(CE(r) < CrossEntropy (Z,, Y)
13:
$R + log-variance associated with rank R
14:
S, + log-variance associated with rank r
15:
Canchor + exp( -12) CCE 8/2
16:
Lvaariatt exp( (exp(-8)) 8,) (re(r) 8r
17:
Frotal + Lanchor + fvaaiint
Opt.zero_grade
19:
Backpropagate gradients of Crotal with respect to 0 and } Opt.step()
21:
end for
F ON LLM USAGE
The authors have used large language models for three purposes:
We have used LLMs to aid or polish our writing. This includes rephrasing text, shorterning, proof-reading for ambuigities or finding mistakes inconsistencies in notation
We used LLMs supplementary source of finding related work. While have primarily performed related work searches via google scholar, we have used the "Deep Research" functionality to find other related work that might have missed. This has resulted in us adding response- based KD and self- distiiilllation as related work to the paper.
We have used LLMs for research ideation early on the paper. This included brainstorming ways how to make efficient deep neural networks, what are the properties that such neural networks should have, among others.
Otherwise, all the ideas presented in the paper are our own. We take full responsibility for any errors found in the paper.
33


Reviews:
[{"review_id": "ehgzRutFj3", "reviewer": "Reviewer_Ji5G", "summary": "This paper presents a model architecture called Nested Subspace Networks (NSNs), which can be applied on top of modern neural network architectures both during pretraining and after pretraining. NSNs consist of a series of low rank subnetworks that approximate the original neural network, thereby reducing computational FLOPs during training, adaptation, and inference. A novel optimization procedure based on uncertainty estimation is also proposed to jointly optimize all subnetworks within the NSN architecture. Both theoretical analysis and experimental results are provided to demonstrate the effectiveness of the proposed NSNs and the optimization method.", "questions": "- The methods proposed in [2] seem to be very promising on top of slimmable networks. I wonder if they are also applicable to NSNs.\n\n\n[2] https://openaccess.thecvf.com/content_ICCV_2019/papers/Yu_Universally_Slimmable_Networks_and_Improved_Training_Techniques_ICCV_2019_paper.pdf", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "- The paper is overall well written and well motivated.\n- Improving the computational efficiency of large deep models and large language models is of practical importance.\n- The proposed NSN model architecture is clear and easy to follow.", "weaknesses": "- The NSN architecture does not appear to be highly novel. It essentially applies the slimmable neural network [1] on top of a low rank model architecture (across the rank dimensions).\n- The proposed \"training with multi rank uncertainty\" procedure is not clearly explained. I suggest adding an algorithm box to illustrate the detailed training steps.\n- Only FLOPs are reported; wall clock time would be a more informative metric to demonstrate both training and inference speedups.\n- It is not clear, during model training, what the overall cost of training an NSN is compared to the standard training of a dense model with a similar number of parameters.\n\n[1] https://arxiv.org/abs/1812.08928", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VzxIiOBfax", "reviewer": "Reviewer_bkcs", "summary": "This paper proposes Nested Subspace Networks (NSNs), an architectural  framework for neural networks that enables continuous and dynamic  adaptation to a range of computational budgets through a nested subspace reparameterization of linear layers. NSNs introduce a means to  dynamically select model \"rank\" at inference, with theoretical backing  for smooth interpolation across ranks and an uncertainty-aware  multi-rank training objective to promote optimality across the entire  rank spectrum. Empirical evaluations include MLP experiments on CIFAR-10, ablation studies, and applications to several large pre-trained language models (LLMs), demonstrating that NSNs achieve competitive accuracy with significantly reduced computational cost.", "questions": "1. Why choosing exponential in Equation 3 as the coefficient.\n\n2. Is there any empirical evidence about assumption 1.\n\n3. In Figure 7, it is shown that the model's score on the evaluation  benchmarks decreases continuously and monotonically as the rank is  reduced. How can we prove that this decline represents the desired  hierarchical degradation in reasoning ability, rather than simply a  score reduction caused by a decrease in the model's overall expressive  power? \n\nFor instance, is the score drop a result of the model becoming  inconsistently capable of solving tasks of the same difficulty level  (i.e., succeeding on some while failing on others)? Or is it due to the  model becoming uniformly incapable of solving a whole category of  problems at a certain difficulty level? Only the latter would  demonstrate an effective adjustment of reasoning and computational  effort according to task difficulty.\n\n4. A recent paper [1] on implicit regularization in matrix factorization (which is not cited in Related Works) also analyzes the hierarchical training dynamics that emerge when progressively increasing the matrix rank—an idea conceptually related to the nested structure proposed in this paper (see, for example, their Proposition 1). One of their key findings is that even without explicit rank constraints, the model naturally evolves from low-rank to high-rank representations during optimization. It would strengthen the theoretical positioning of this work if the authors could discuss or compare the necessity of explicitly enforcing rank constraints in NSNs.\n\n[1] Connectivity Shapes Implicit Regularization in Matrix Factorization Models for Matrix Completion, NeurIPS 2024.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper proposes a conceptually novel framework, Nested Subspace Networks, that unifies multiple model capacities within a single parameterization. \n- The approach is validated through controlled experiments, ablation analyses, and applications to pre-trained LLMs, supporting the claims of dynamic adaptability and compute–performance efficiency.\n- A key practical merit is the post-hoc applicability of NSNs: the framework can be retrofitted onto existing pre-trained models without retraining from scratch, addressing a critical deployment challenge for large-scale foundation models.", "weaknesses": "- Equation 2 needs clarification. Does the variant rank $r$ change in different epoch or remain unchanged during training. Is $\\mathcal{L}_{CE}(k)$ the cross entropy loss calculated when the linear weight using its first $k$ rows and columns?\n- Implementation details missing. The paper does not explain how Logits Regularization, Residual Orthogonality, and Hidden Regularization (as shown in Table 1) are implemented, nor whether these are used concurrently with the main “Two CEs” objective or as independent ablations.\n- Ambiguity in Section 4.1. The experiment uses CIFAR-10 but states that “inputs are ImageNet last-layer embeddings.”\nPlease clarify this setup: are ImageNet features used merely as a fixed embedding extractor, and are all MLP layers subsequently replaced with NSN layers?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gjteYn35Fo", "reviewer": "Reviewer_TDZV", "summary": "This work proposes a way to re-parametrize linear layers into low-rank factorization $W=A^{R \\times d_{in}} \\cdot B^{d_{out} \\times R}$ such that the compute budget can be dynamically adjusted without retraining by constructing the  effective weight matrix from the first r rows of A and first r columns of B. In other words, the effective layers forms a filter chain of images for the ranks r < R (a.k.a Nested Subspace Property / Layer). To train the layer effectively, the work leverages an uncertainty-weighted training objective by Kendall+2018 that balances the learning across ranks. The experiments show that the nested subspace layers can be introduced into pretrained LLMs to create a smooth rank-based accuracy-vs-compute trade-off that remains adjustable at test-time.", "questions": "- It is not clear how the anchor and variant rank are chosen. Section 2.2 says that the ranks are \"sampled\" but I am not sure why and how specifically. Is this applied consistently across training runs or is there a random element to it?\n- It is claimed that the log-variances \"serve as an emergent proxy for the effective expressiveness of each rank-specific model\" but it is not clear why that needs to be the case since they are as far as I can tell freely trainable parameters. The same goes for the \"uncertainty-weighted surrogate\" - how do we know that these parameters act as a meaningful surrogate for uncertainty?\n- Are you planning to release your source code and a drop-in layer implementation?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The approach is highly flexible and practical as it can be broadly applied to any pretrained models with linear layers through a straight forward Singular Value Decomposition. Rather than requiring the specification of a suitable rank up front that may not be known for a given problem and architecture the method allows to \"shut up and train\" to find an appropriate effective rank.\n- The resulting architecture is intepretable in the sense that it is straight-forward to inspect how much each ranks learns to represent both empirically and theorectically (the offered theoretical discussion provides a good intution of the approximation trade-offs involved).", "weaknesses": "- Section 4.1 investigates whether a single NSN, trained with the multi-rank objective, can match the performance of multiple, specialized model, but only does so for an somewhat artificial image classification setup where regular MLP baselines are trained from scratch. What is currently missing is a similiar quantitive comparison with LoRA baselines to establish whether an NSN evaluated for rank r matches a LoRA model that exclusively trained for rank r. This would make this work comparable with prior baselines and delinate the benifits of NSN learning vs predefined-rank models.\n- An explicit desideratum is granularity of adaption, but the trade-off here only works in discrete 'matrix rank' steps for each layer. Since different layers at different depths learn to represent different things it is not clear that a uniform rank pick across the entire model is ideal. Arguably, the most granular adaption would be picking the ideal rank size for each layer but it is not obvious how to do that efficiently. \n- The work mentions the connection to alternative weight-learning approaches such as GradNorm but there is no comparison or analysis on why the Kendall-loss formulation is the best choice.\n\n**Additional comments**\n\n- The core idea of the work could be introduced and visualized more clearly, e.g. by illustrating an example A and B and the resulting effective weight matrix. The right column of Figure 1 does not make it very clear that there is only 1 instance of A and B and not a duplication of many A and Bs.\n- L182 \"Following Kendall et al. (2018), we use the standard uncertainty-weighted surrogate objective\"; I recommend expanding on this to make the work more self-contained and accessible to readers who may not be familiar with  \n- \"a strict and dynamic computational budget\" - I found this confusing to read; it might be better to concretely lay out the use cases that this work has in mind.\n- L115 typo: that *form* a nested hierachy\n\nAside: The manuscript uses visually highlighted \"Takeaway\" messages that spell out what the reader may conclude from each section. I am not sure if this is helpful because it can feel repetitive, or, if it does not align with the reader's conclusion, somewhat presumptuous. For example, L276 \"smooth and predictable trade-off\" placed before any experimental results reads more like an unverifiable claim than a takeaway.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)