Adaptive Width Neural Networks
Authors: Federico Errica
Henrik Christiansen
Viktor Zaverkin
Mathias Niepert
Francesco Alesiani
Venue: ICLR 2026 Poster (2026)
Keywords: Neural Networks
Learning the Number of Neurons
Adaptive Width Learning
Dynamic Architectures
Information Compression
Variational Inference
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose a strategy to learn, during a single training, the width of neural network layers depending on the task and without imposing upper bounds.

Abstract: For almost 70 years, researchers have typically selected the width of neural networks’ layers either manually or through automated hyperparameter tuning methods such as grid search and, more recently, neural architecture search. This paper challenges the status quo by introducing an easy-to-use technique to learn an \textit{unbounded} width of a neural network's layer \textit{during training}. The method jointly optimizes the width and the parameters of each layer via standard backpropagation. We apply the technique to a broad range of data domains such as tables, images, text, sequences, and graphs, showing how the width adapts to the task's difficulty. A by product of our width learning approach is the easy truncation of the trained network at virtually zero cost, achieving a smooth trade-off between performance and compute resources. Alternatively, one can dynamically compress the network until performances do not degrade. 
In light of recent foundation models trained on large datasets, requiring billions of parameters and where hyper-parameter tuning is unfeasible due to huge training costs, our approach introduces a viable alternative for width learning.

Published as a conference paper at ICLR 2026
ADAPTIVE WIDTH NEURAL NETWORKS
Federico Errica
Henrik Christiansen NEC Laboratories Europe
Viktor Zaverkin NEC Laboratories Europe
NEC Laboratories Europe federico.errica @ neclab.eu
Mathias Niepert
Francesco Alesiani NEC Laboratories Europe
University of Stuttgart
NEC Laboratories Europe
ABSTRACT
For almost 70 years, researchers have typically selected the width of neural net- works' layers either manually through automated hyperparameter tuning meth- ods such as grid search and, more recently, neural architecture search. This paper challenges the status quo by introducing an easy-to-use technique learn un- bounded width of a neural network's 's layer during training. The method jointly optimizes the width and the parameters of each layer via standard backpropagation. We apply the technique to broad range of data domains such as tables, images, text, sequences, and graphs, showing how the width adapts to the task's difficulty. A by product of our width learning approach is the easy truncation of the trained network virtually zero cost, achieving smooth trade-off between performance and compute resources. Alternatively, one can dynamically compress the network until performances do not degrade. In light of recent foundation models trained on large datasets, requiring billions of parameters and where hyper-parameter tuning is unfcasible due to huge training costs, our approach introduces a viable alternative for width learning.
INTRODUCTION
Since the construction of the Mark Perceptron machine (Rosenblatt) 1958) the effective training of neural networks has remained an open research problem great academic and practical value. The Mark solved image recognition tasks with a layer of 512 fixed "association units" that in modern language are the hidden units of a Multi-Layer Perceptron (MLP). MLPs possess universal approximation capabilities when assuming arbitrary width (Cybenko 1989) and sigmoidal activations, and their convergence to good solutions was studied, for instance, in Rumelhart et al. (1986) where the backpropagation algorithm was introduced as "simple, easy implement on parallel hardware". and improvable by other techniques such as momentum that preserve locality of weight updates.
Yet, after almost 70 years of progress (LeCun et al. 2015). the vast majority of neural networks, be they shallow or deep, still rely on fixed choice of the number of neurons in their hidden layers. The width is typically treated as one of the many hyper-parameters that have be carefully tuned whenever we approach a new task (Wolpert) 1996). The tuning process has many names, such as model selection,
tuning, and cross- validaaai,,,,n,, and is associated with non non-negligible costs: Different architectural configurations are trained until one that performs best on a validation set selected (Mitchell 1997). The configurations' space grows exponentially the number of layers, so practitioners often resort to shortcuts such as picking specific number of hidden units for all layers and trying a few values, which greatly reduces the search space together with the chances of selecting better architecture for the task. Other techniques to tune hyper-parameters include constructive approaches (Fahlman & Lebiere 1989 Wu et al., 2020), which alternate parameter optimization and creation of new neurons, natural gradient- based heuristics that dynamically modify the network (Mitchell et al. 2023), bi-level optimization (Franceschi et al.) 2018). and neural architecture search (White et al. 2023), which often requires separate training for each configuration.

---

Published as a conference paper at ICLR 2026
space exploration problem is exacerbated by the steep increase in size of recent The architectures for language (Brown et al. 2020) and vision (Zhai et al.] (022). for example, neural where parameters are in the order of billions to accommodate for a huge dataset. Training these models requires an amount of time, compute power, and energy that currently makes it unfeasible for most institutions perform a thorough model selection and find good width parameters; the commonly accepted compromise is to stick to previously successful hyper-pane parameter choices. This may also explain why network pruning (Blalock al. 2020) Míshra et al. 202T), distillation (Zhang et al. 2019) and quantization (Mishra et al., (021) techniques have recently been in the spotlight, as they trade-off hardware requirements and performance.
This work introduces a simple and easy to use technique to learn the width of each neural network's s layer without imposing upper bounds (we refer to as unbounded width). The width of each layer is dynamically adjusted during backpropagation (Paszke et al. 2017). and it only requires slight modification to the neural activations that does not alter the ability to parallelize computation. The technical strategy is to impose a soft ordering of hidden units by exploiting any monotonically decreasing function with unbounded support on natural numbers. With this, we do not need to fix a maximum number of neurons, which is typical of orthogonal approaches like supemettorks (White) et al. 2023). As a by-product, we can achieve a straightforward trade-off between parametrization and performance by deleting the last rows/columns of weight matrices, i.e., removing the "least important" neurons from the computation. Finally, we break symmetries in the parametrization of neural networks: it is not possible anymore to obtain an equivalent neural network behavior by permuting the weight matrices, which reduces the "jostling" effect where symmetric parametrizations compete when training starts (Barber) 2012).
We test our method on MLPs for tabular data, a Convolutional Neural Network (CNN) LeCun al.] 1989) for images, a Transformer (Vaswani et al., (017) for text, Recurrent Neural Network (RNN) for sequences, and Deep Graph Network (DGN) (Micheli 2009; [Scarselli et al. 2009) for graphs, showcasing broad scope of applicability as MI.Ss are ubiquitous in modern architectures. Empirical results show, as may be intuitively expected, that the width adapts to the task is difficulty with performances comparable to fixed-width baseline. We also encourage compression of the network at training time while preserving accuracy, as well controlled trade-off at zero additional cost. Ablations suggest that the learned width not influenced by the starting width (under bounded activations) nor by the batch size, advocating for potentially large reduction of the preee-paraeeter configuration space.
post-hoc truncation inducing
2 RELATED WORK
Constructive methods dynamically learn the width of neural networks and are related in spirit to this work. The cascade correlation algorithm (Fahlman Lebiere 1989) alternates standard training with the creation of new hidden unit minimizing the neural network's residual error. Similarly, the firefly network descent (Wu et al. 2020) grows the width and depth of a network every N training epochs via gradient descent on dedicated loss. Yoon et al. (2018) propose an ad-hoe algorithm for lifelong learning that grows the network by splitting and duplicating units to learn new tasks. Wu et al. (2019) alternate training the network and then splitting existing neurons into offssring with equal weights. These works mostly focus on growing the neural network; Mitchell et al. (2023) propose natural gradient-based heuristics to grow/shrink layers and hidden units of MLPs and CNNs. The main difference from our work is that we grow and shrink the network by simply computing the gradient of the loss, without relying on human-defined heuristics. The unbounded depth network of Nazaret & Bleij (002)), from which we draw inspiration, learns the number of layers of neural networks. Compared to that work, we focus our attention to the number of neurons. modifying the internals of the architecture rather than instantiating a multi-ou one. In particular: i) learning the width requires a different formulation of the evidence lower-bound; and ii) the importance distribution needs to be monotonically decreasing, which does not encode the right inductive bias for the depth of a neural network. We also mention Bayesian nonparametric approaches (Orbanz & Teh 2010) that leam potentially infinite number of clusters in an unsupervised fashion, well as dynamic neural networks (Han et al. 2021) that condition the architecture on input properties. Finally, the work of Caron et al. (2025) investigates, in similar spirit to this work, how to introduce decreasing rescaling of the pre-activations of neurons in the context of very wide networks; the authors provide interesting global convergence results in the Neural Tangent Kernel (NTK) regime.
2

---

Published as a conference paper at ICLR 2026
Orthogonal Methods Neural Architecture Search (NAS) is automated process that designs
neural networks for a given task (Elsken et al.; [2019b] White et al. 2023) and has been applied to
different contexts (Zoph al., 2018 Liu et al. 2019: So et al. 2019). Typically, neural network
elements are added, removed, or modified based on validation performance (Hlken et al. 2019a
White al. 2021 2023). by means of reinforcement learning (Zoph & Le 2016), evolutionary
algorithms (Real et al. 2019). and gradient-based approaches (Liu et al., 2019). Typical NAS methods
require enormous computational resources, sometimes reaching thousands of GPU days (Zoph &
Le 2016), due to the retraining of each new configuration. While recent advances on one-shot NAS models (Brock et al., 2018: Pham et al., 2018: Bender et al.) 2018 Berman et al., 2020 [uu al.
b021b a) have drastically reduced the computational costs, they mostly focus on CNNs, assume a bounded search space, and do not learn the width. As such, NAS methods are complementary to our approach. Bi-level optimization algorithms have also been used for hyper-parameter tuning (Franceschi et al., 2018). where
are the variables of the outer objective and the model parameters those of the inner objective. The solution sets of the inner problem are usually not available in closed form, which has been partly addressed by repeated application of (stochastic) gradient descent (Domke [012 Maclaurin al. 2015) |Fanceschhi et al., (017). These methods are restricted to continuous hyper-parameters' optimization and cannot be applied to width optimization. Finally, pruning (Blalock et al. 2020) and distillation (Hinton et al., 2015) are two methods that reduce the size of neural networks by trading-off performances; the former deletes neural connections (Mishra al. 202T) or entire neurons (Valerio et al. [022 Dufort-Labbé smaller network (student) to mimic a larger one (teacher) (Gou et al. 2021). In particular, dynamic pruning techniques can compress the network at training time (Guo et 2016), by applying hard or soft masks (He et al.) (018):: for comprehensive survey pruning. please refer to (He & Xiao 2023). is worth mentioning the pruning strategy of Wolinski et al. (2020), which promotes learning in some neurons and penalizes others by appropriate rescaling of a potentially (infiniteddimensionaa) input; this work shares conceptual similarities with the our approach, though does not consider learning the rescaling factor a proxy for the width. Compared to most pruning approaches, work can delete connections and reduce the model's memory, but also grow it indefinitely; compared to distillation, do not necessarily need new training to compress the network. These techniques can be easily combined with our approach, whose main goal is not compression but rather the automatic adaptation of a neural network's width.
2024), the latter trains
ADAPTIVE WIDTH LEARNING
We now introduce Adaptive Width Neural Networks (AWN), a probabilistic framework that maxi- mizes simple variational objective via backpropagation over neural network's 's parameters.
We are given a dataset of N i..dd. samples, with input x E IR, F E N+ and target y whose domain depends on whether the task is regression or classification. For samples X E R RNx and targets Y, the learning objective is to maximize the tog-likelihood
log = log II p(ul2:)) Σ logp(M/(z))
(L)
with respect to the learnable parameters of p(y|xx)
We define P(ylar) according to the graphical model of Figure D (left), to learn neural network with unbounded width for each hidden layer L. To do so, we assume the existence of an infinite sequence of i.i.d. latent variables O₂ ] where Oe, is multivariate variable over the learnable weights of neuron n at layer f. However, since working with an infinite-width layer is not possible in practice, for layer f. That is, it truncates an infinite width to a finite value so that we can feasibly perform inference with the neural network. For a neural network of L layers, we define 0 = (0et/-n and A = (10k_ assuming independence across layers. Therefore, by marginalization one can write XY| = pY, X, 0 X )dAdd.
also introduce latent variable X that samples how many neurons to
3

---

Published as a conference paper at ICLR 2026
ft
dynamic truncationn level Dr
xfril)
(0)
x
Ae
E (1
< (1...,
neuron II)
D2
layer /
Figure 1: (Left) The graphical model of AWN, with dark observable random variables and white latent ones. (Middle) The distribution St over hidden units' importance at layer is parametrized by dy- The width of layer is chosen as the quantile function of the distribution fe evaluated at & and denoted by De. (Right) The hidden units' activations at layer É are rescaled by their importance.
We then decompose the joint distribution using the independence assumptions of the graphical model:
N
p(Y, 0|X) = II
plu, X, O1z;)
p(yi, A, 01zr)) - p(v: A, 0, x: x:)((A)p(()
(2)
==1
1
L p(0) p(Oen) =
I.
P(A) II p(de) II (((GGuZ t=1
II
N ( 0, diag(a?))
8=1
8=
t=
P(yil WWl,A,,,z;) 0,z;) = Neural Network as will be introduced in Section 3.1
where of: uuz. of. are syper-parameters For simplicity, we assumed a Gaussian prior for X, but please note that, when strict positivity is required, one should formally turn Folded Normal priors. The neural network is parametrized by realizations À ~ p(A). 0 ~ 0 - it relies on a finite number of neurons detailed later and in Section 3.1 and it outputs either class probabilities (classification) the mean of a Gaussian distribution (regression) to parametrize p(y: ( 0, ox;) depending on the task. Maximizing Equation [I. however, requires computing the evidence p(Y, X, dddd0, which is intractable. Therefore, we turn to mean-field variational inference (Jordan et al. 1999 Blei et al.)
2017) to maximize an expected lower bound (ELBO) instead. This requires to define a variational distribution over the latent variables q 0) and re-phrase the objective as:
log ( XX) E (A, log p(Y,,,0|X)
(5)
g g(>, 0)
where (A, 0) is parametrized by learnable variational parameters. Before continuing, we define the truncated width De, is the finite number of neurons layer É, the quantile function evaluated at k with k a prpperprrraeter.. of a distribution fe with infinite support over N+ and parametrized by de; Appendix providees desirable properties of fe in a similar vein Nazaret & Blei (0022). In other words, we find the integer such that the cumulative mass function of fe takes value k. and that integer Dj the truncated width layer We implement fe as discretized exponential distribution adhering Def. A.2, following the discretization strategy Roy (2003): For every x N+, the discretized distribution relies on the exponential's cumulative distribution function:
[x (1 e ---
(1-e-Ars)
We choose the exponential because it is a monotonically decreasing function and allows us impose an ordering of importance among neurons, detailed in Section 3.1
Then, we can factorize the variational distribution qA, A) into:
q(A, 0) ((A)
(AA) = II
((ee) II 1=1
(7)
8=1
L D (())))= II (0.)) ==
DO II
p(0w)))
g(0m)) Nonnn ptn,
(8)
D1=
GGeneral functions are allowed if threshold can be computed.
4

---

Published as a conference paper at ICLR 2026
Here, ve, Pên are learnable variational parameters and, as before, define pe {ptmIn=1 p {pe)z_ and v = (ve)e= Note that the set of variational parameters is finite as it depends on Dr.
By expanding Equation 5 using the above definitions and approximating the expectations at the first order, i.e. (g(A) [()] =
(w) and () [[(]]] = (() as Nazaret Blei (2022), we obtain the final form of the ELBO (the full derivation is in Appendix(C) C):
L Σ log plver2.o2)
L Dr
p(Peni of))
N
ΣΣ
log
log puu A=p,
a(Pen' PÊu)
n=1
where distributions' parameters made explicit to distinguish them. The first two terms in the ELBO regularize the width of the layers and the magnitude of the parameters when priors are informative, whereas the third term accounts for the predictive performance.
In practice, the finite variational parameters V, p are those used to parametrize the neural network rather than sampling X, O, which enables easy optimization via backpropagation. Maximizing Equation ( will update each variational parameter Mg, which in turn will change the value of D during training. If De increases we initialize new neurons and draw their weights from a standard normal distribution, otherwise we discard the weights of the extra neurons. When implementing mini-batch training, the predictive loss needs to be rescaled by N/M, where M the mini-bated size. From a Bayesian perspective, this is necessary as regularizers should weigh less if we have more data.
Compared fixed-wiie width network with weight decay, we need to choose the priors' values of u², az. as well as initialize the learnable vy. The latter can be initially set to same value across layers since they can freely adapt later, or can be sampled from the prior p(Ae). Therefore, we have two more hyper-parameters compared to the exed-width network, but we make some considerations: i) it is always possible to use an uninformative prior over λe. removing the extra letting the model freely adapt the width of each layer (as is typical of frequentist approaches); ii) the is known to be less stringent than that of do not need explore many values of μn² and o lii) our experiments suggest that AWN can converge to similar widths regardless of the starting point vg, so that we may just need perform model selection over one/two sensible initial values; iv) the more data, the less the priors will matter.
choice of higher level of
themselves (Goel & Degroot| 1981 Bernardo & Smith 2009),
3.1 IMPOSING A SOFT ORDERING ON NEURONS' IMPORTANCE
Now that the learning objective has been formalized, the missing ingredient is the definition of the neural network p(y] λ=v, 0=p, x.)) of Equation 4 as a modified MLP. Compared to standard MLP, we use the variational parameters v that affect the truncation width at each hidden layer, whereas P are the weights. We choose a monotonically decreasing function fú, thus when added its relative importance is low and will not drastically impact the network output and hidden activations. In other words, we impose soft ordering of importance among neurons. We simply modify the classical activation ht of a hidden neuron j at layer & as
neuron is
De-1 Σ
hラ o
W whah
felivv).
(10)
where Dz-1 is the truncated width of the previous layer, a is a non-linear activation function and wfe E Ptj. That is, we rescale the activation of each neuron & by its "importance" felj; vz). Note that the bias parameter is part of the weight vector as usual.
It easy to see that, in theory, the optimization algorithm could rescale the weights of the next layer by a factor ff( to compensate for the term vz) This could lead to degenerate situation where the activations of the first neurons are small relative to the others, thus breaking the soft-ordering and potentially wasting neurons. There are two strategies to address this undesirable effect. The first is to regularize the magnitude of the weights thanks to the prior p(0++1n), so that may be difficult to compensate for the least important neurons that have a high l/) The second and less obvious strategy is to prevent the units' activations of the current layer to compensate for high values by bounding their range, e.g., using ReLU6 or tanh activation (Sandler al. 2018)) We apply both strategies to our experiments, although we noted that they do not seem strictly necessary in practice.
5

---

Published as a conference paper at ICLR 2026
3.2 RESCALED WEIGHT INITIALIZATION FOR DEEP AWN
Rescaling the activations of hidden units using Equation 10 causes activations of deeper layers to quickly decay zero for AWN MLP with ReLU nonlinearity initialized using the well known Kaiming scheme (He et al. 2015). This affects convergence since gradients get close to zero and becomes slow to train deep AWN MLPs. We therefore derive rescaled Kaiming weight that guarantees that the variance of activation across layers is constant at initialization.
Theorem 3.1. Consider an MLP with activations as Equation 70 and ReLU nonlinearity. At ini- tialization, given af o Σ EE (/ Varlwff
Vaalaa] zs Varla,-
L1=1 f00
Proof. See Appendix E for an extended proof.
The extended proof shows there is connection between the variance of activations and the variance of gradients. In particular, the sufficient conditions over Var[wf.. are identical if one initializes ve in the same way for all layers. Therefore, we initialize weights from a Gaussian distribution with standard deviation
V/2
we guarantee that at initialization the variance of the deep network's i gradi-
f70)
ents will be constant. The effect of the new initialization (dubbed "Kaiming*") can be seen in Figure 7 where the distribution of activation values at initialization does not collapse in subsequent layers. This change drastically impacts overall convergence tion[4) 4). where it appears it would be otherwise hard to converge using a standard Kaiming initialization. A Algorithm [ summarizes the main changes to the training procedure, namely the new initialization and the update of the model's 's truncated width each training step.
the SpiralHard synthetic dataset (described in Sec-
Algorithm AWN Training Procedure
1: Input: Dataset D, initialized AWN model M (Section
2: Output: Trained AWN Model M
3: for each training epoch do
4: for batch in 2> do
3.3 FUTURE
5: update wiittt__iidt(MMM)
DIRECTIONS AND LIMITATIONS
6: I _M(htthh)
7: loss < ELBO(MM, batch, 9)
Eq.
MLPs are ubiquitous modern deep architectures. They the very end of CNNs, in each Transformer layer, and they process messages coming from neighbors in DGNs. Our experiments focus on MLPs to showcase AWN's broad ap- plicability, but there are many other scenarios where one can apply AWN's principles. For instance, one could impose soft ordering of importance on CNNs' filters at each layer, therefore learning the number of filters during training. However, doing so would require distinct formalization, which why this lies beyond the scope of the present work.
8: M + backprooaggttonn loss)
9: function update width(MM)
used at
10:
for layer é in M.hidden layers do
D; 5- quantile function se(:;v)) evaluated k // add/remove neurons
12: Use De to update pe, P
From a more theoretical perspective, we believe one could draw connections between our technique and the Information Bottleneck principle (Tishby et al. 2000), which seeks maximally representative (i.... performance) and compact representations (e.g., width). .. In addition, we could try to revisit the theoretical results of Caron et al. (2025) in the NTK regime to better understand convergence properties of adaptive-width neural networks. since both are based on asymmetric rescaling of activations.
4 EXPERIMENTS AND SETUP
The purpose of the empirical analysis is not to claim AWN is generally better than the fixed-width baseline. Rather, we demonstrate how AWN overcomes the problem fixing the number of neurons by learning it end-to-end, thus reducing the amount of hyper-parameter configurations to test. As such, due to the nature of this work and similarly to Mitchell et al. (2023). we use the remaining space thoroughly study the behavior of AWN, so that becomes clear how to use it in practice. We first quantitatively verify that AWN does not harm the performance compared to baseline models and compare the chosen width by means of grid-search model selection with the learned width of AWN.

---

Published as a conference paper at ICLR 2026
Second, we check that AWN chooses larger width for harder tasks, which can be seen increasing the hypotheses space until the neural network finds a good path to convergence. Third, we verify that convergence speed is not significantly altered by AWN, so that the main limitation lies in the extra overhead for adapting the network at each training step. As a sanity check, we study conditions under which AWN's learned width does not seem to depend on starting hypper-arammttes, so that their choice does not matter much. In addition, we analyze other practical advantages of training neural network under the AWN framework: the ability to compress information during training or post training, and the resulting trade-offs. Finally, we analyze the impact of different families of functions fe (e) compared the exponential mainly used in this work. Further analyses are in the Appendix.
We compare baselines that undergo proper typer-parameted tuning (called "Fixed") against its AWN version, where we replace any fixed MLP with an adaptive one. First, we train an MLP on 3 synthetic tabular tasks of increasing binary classification difficulty, namely a double moon, a spiral, and a double spiral that we call SpiralHard. A stratified hold-out split of 70% training/1095 10% validation/20% test for risk assessment is chosen at random for these datasets. Then, we consider larger real-world tabular tasks with higher feature dimensionality, namely pol, MiniBooNE, and credit card clients (Beyazit et al., 2023). Similarly, we consider a ResNet-20 (He et al.) 2016) trained on image classification tasks, namely MNIST (LeCun) 1998), CIFARIO, 10, and CIFARTOO (Krizhevsky) 2009), where data splits and preprocessing are taken from the original paper and AWN is applied to the downstream classifier. In the sequential domain, we implemented a basic adaptive Recurrent Neural Network (RNN) and evaluated on the PMNIST dataset (Zenke et al. (017) with the same data split as MNIST. In the graph domain, we train a Graph Isomorphism Network (Xu et al. 2019) on the NCII and REDDIT-B -B classification tasks, where topological information matters, using the same split and evaluation setup of Errica et al. (2020). Here, the first hidden layer MLP as well as the one used in each graph convolutional layer are replaced by adaptive AWN versions. On all these tasks, the metric of interest is the accuracy. Finally, for the textual domain we train a Transformer architecture (Vaswani et al. (017) on the Multi30k English- Geriin translation task (Elliott al. 2016). using pretrained GPT-2 Tokenizer, and we evaluate the cross-entropy loss over the translated words. On tabular, image, and text-based tasks, an internal validation set (10%) ) for model selection is extracted from the union of outer training and validation sets, and the best configuration chosen according to the internal validation set is retrained 10 times on the outer train/validation//estt splits, averaging test performances after an early stopping strategy on the validation set. Due to space reasons, we report datasets statistics and the hyper-parameter tried for the fixed and AWN versions in Appendix (F and 2 We ran the experiments on a server with 64 cores, 1.5TB of RAM, and 4 NVIDIA A40 with 48GB of memory.
G respectively?
5 RESULTS
We begin by discussing the quantitative results of our experiments: Table I reports means and standard deviations across the 10 final training runs. In terms of performance, we observe that AWN is more stable or accurate than a fixed MLP on tabular and sequential datasets; all other things being equal, it seems that using more neurons and their soft ordering are the main contributing factors to these improvements. On the image datasets, performances of AWN are comparable to those of the fixed baseline but for CIFAR 100, due an unlucky that did not converge. There, AWN learns smaller total width compared to grid search.
Results on graph datasets are interesting in two respects: First, the performance on REDDIT-B is significantly improved by AWN both in terms of average performance and stability of results; second, and akin to PMNIST, the total learned width is significantly higher than those tried in Xu et al. (2019); Errica et al. (2020). meaning a biased choice of possible widths had a profound influence on risk estimation for DGN models (i.e., GIN). This result makes it evident that it is important to let the network decide how many neurons are necessary to solve the task. Appendix H shows what happens when we retrain Fixed baselines using the total width as the width of each layer.
Finally, the results on the Multi30k show that the AWN Transformer learns to use 200x parameters less than the fixed Transformer for the feed-forwart networks, achieving a statistically comparable
?oode to reproduce results is available Adaptives wi 8th Neural -Neawore
7

---

Published as a conference paper at ICLR 2026
Table Performances and total width of MLP layers for the fixed and AWN versions of the various models used. The exact width chosen by model selection on the graph datasets is unknown since we report published results. "Linear" means the chosen downstream classifier is linear model.
Width (Fixed)
Width (AWN)
Fixed
AWN
Mean (Std)
Mean (Std)
Mean
(Std)
DoubleMoon
100.0 (0.0) 100.0 (0.0)
8.1
(2.8)
Spiral
99.5 (0.5) 99.8 (0.1)
65.9
(8.7)
SpiralHard
98.0 (2.0) 100.0 (0.0)
32
227.4
(32.4)
pol
99.3 (0.2) 99.2 (0.1)
48
84
(11.0)
MiniBooNE
92.9 (0.3) 93.2 (0.1)
32
53
(11.1)
credit card
81.6 (0.1) 81.8 (0.1)
51
(12.0)
PMNIST
91.1 (0.4) 95.7 (0.2)
24
806.3
(44.5)
MNIST
99.6 (0.1) 99.7 (0.0)
Linear 19.4
(4.8)
CIFARIO
91.4 (0.2) 91.4 (0.2)
Linear
80.1
(12.4)
CIFAR100
66.5 (0.4) 63.1 (4.0)
256
161.9
(57.8)
NCII
80.0 (1.4) 80.0 (L.1) (96-320)
731.3 (128.2)
REDDIT-B
87.0 (4.4) 90.2 (1.3) (96-320)
793.6 (574.0)
Multi3kk (J)
1.43 (0.4)
1,51 (0.2) 24576 123.2 (187.9)
test loss. This result appealing when read through the lenses modern deep learning, as the power required by some neural networks such as Large Language Models (Brown et al. (020)) is so high that cannot be afforded by most institutions, and it demands future investigations.
Adaptation to Task Difficulty and Convergence Intuitively, one would expect that AWN learned larger widths for more difficult tasks. This is indeed what happens on the tabular datasets (and image datasets, see Appendix [ where some tasks are clearly harder than others. Figure 2 (left) shows that, given the same starting width per layer, the learned number of neurons grows according to the task's difficulty. SpiralHard is significantly lower than that achieved by a single-layer MLP, which is consistent with the circuit complexity theory arguments put forward Bengio et al. (2006); Mhaskar et al. (2017). also appears that convergence not affected by the introduction of AWN, investigated in Figure 2 (right), which was not obvious considering the parametrization constraints encouraged by the rescaling of neurons' activations.
also interesting to observe that the total width for a multi-layer MLP on
100
Learoed Hidden Nourons
100
202
Accuracy
DostisMose
Danser
Soizaiaard (L=1)
101
Dodk
SpialHaad (L=2)
Spintllaad
AWN
10
Spinillind (1=2)
60
Foudl
10a
10A
10a
109
10a
10a
Epoch
Epoch
Figure 2: (Left) The learned width adapts to the increasing difficulty of the task, from the Double.oon to SpiralHard. (Right) AWN reaches perfect test accuracy with comparable amount of epochs DoubleMoon and Spiral, while it converges faster on SpiralHard.
Training Stability Analysis To support our argument that AWN can reduce the time spent per- forming hyper-parameter selection, we check whether AWN learns a consistent amount of neurons across different training rums and hyper-paramett choices. Figure ( reports the impact of the batch size and starting width averaged across the different configurations tried during model selection. Smaller batch sizes cause more instability, but in the long run observe convergence to similar
8

---

Published as a conference paper at ICLR 2026
width. Convergence with respect different rates holds, instead, for the bounded ReLU6 activation; Appendix showsss that unbounded activations may cause the network to converge more slowly to the same width, which is in accord with the considerations about counterbalancing the rescaling effect of Section 3.1 Therefore, whenever possible, we recommend using bounded activations.
309
10a
Batch Sizee
Rate
Learsed Nouroms
2014
Learned Hidden Nourons
10
128
0003
2041
10
1024
011
107
102
101
101
109
109
100
10a
20a
10a
10
10a
Epoch
Epoch
Figure 3: Training converges to similar widths on SpiralHard for different batch sizes (left) and starting rates v, but the latter seems to require bounded nonlinearity such as ReLU6 converge in a reasonable amount of epochs (right).
Online Network Compression via Regularization So far, have used uninformative prior p(A) over the neural networks' width. We demonstrate the effect of an informative prior by performing hidth-annealing experiment on the SpiralHard dataset. We set an uninformative p(0) and ReLU6 nonlinearity. At epoch 1000, we introduce p(Ae) = (Ne;; 0.05, 1), and gradually anneal the standard deviation up to 0.1 at epoch 2500. Figure 4 shows that the width the network reduces from approximately 800 neurons to 300 without any test performance degradation. We hypothesize that the least important neurons earry negligible information, therefore they can be safely removed without drastic changes in the output of the model. This technique might be particularly useful to compress large models with billions of parameters.
10
Sturt: Wilth Repularicotion
910
Accuracy (%6
Learned Hidden
80
10a
70
1065
1000
2000
5000
Epoch
102
1000
3000
4000
Epoch
Epoch
Figure 4: is possible to regularize the width at training time by increasing the magnitude of the loss term log p(P)
q(v)- The total width is reduced by more than 50% (left) while preserving accuracy (right). The inset plot refers to the loss term that AWN tries to maximize.
Post-hoe Truncation Achieves Trade-off between Performance and Compute Resources To further investigate the consequences of imposing soft ordering among neurons, we show that it possible perform straightforward post-training truncation while still controlling the trade-off between performance and network size. Figure shows an example for an MLP on the Spiral dataset, where the range of activation values (Equation 10 computed for all samples follows an exponential curve (right). Intuitively, removing the last neurons may have negligible performance impact at the beginning and a drastic one as few neurons remain. This is what happens, where we are able to cut MLP with hidden width 83 by 30% without loss of accuracy, after which smooth degradation happens. one accepts such a trade-off, this technique may be used to "distill" a trained neural network at virtually additional cost while reducing the memory requirements. Note that

---

Published as a conference paper at ICLR 2026
truncation heuristics that either random based the magnitude of neurons' activations (i.e., excluding the rescaling term) do not perform as well.
1.0
0.8
Activation Values
0.35
0.6
0.30
0.4
0.05
0.2
Tranation
Migasiale leaed Teanaaion
0.00
0..0
0.01
0.90
1.00
Neuron
Fraction of original (83)
Figure 5: (Left) Thanks the soft ordering imposed on the neurons, one can also truncate the neural network after training by simply removing the last neurons. (Right) The distribution of neurons' activations for all Spiral test samples follows an exponential-like curve.
Impact of Different Functions vz(;;v:) We conclude main analyses with ablation study on the use of different decreasing functions fr(i; considering the SpiralHard dataset, to un- derstand how different functions have different properties. We tested: a Power Law distribution feli; Pe, a) = Z(j + jsai)- with Z being normalizing term, initial E [2., and low degree saturation Jsat chosen between [O., 2.]: ii) sigmoid-like function fe ve. b) o(j vvjj-v), a is the sigmoidal activation, V/2 E [128, 256 is the transition point.
ve), where
The results are reported Table 2 When averaged over all possible ayper-paramettr configurations, there is no significant deviation in performance between distribution, and there is always at least one configuration attaining the best performance. However, the Power Law distribution introduces more neurons than the exponential due to its long-tail. Instead, the Sigmoidal function tends to allocates less neurons, but the drawback is the loss of the ability to truncate the network after training, since the importance of neurons is very close to before the transition point and the transition itself, for the chosen fixed parameters, is relatively sharp. The choice of the function family fe ve)) remains an interesting direction of further exploration, given that each family has different properties that might be more or less suitable depending on the use cases.
Table 2: Analysis of the impact of different familics of importance functions on SpiralHard validation performance and learned width, averaged across different hyper- hyper- parameter configurations.
feliive)
Mean Accuracy
Max Accuracy
Mean Total Width
Exponential
80.27 (19,9)
100.00
954.4 (1083.9)
Power Law
81.82 (16.6)
100.00
2952.4 (3371.6)
Sigmoidal
76.85 (18.3)
100.00
426.8 (268.4)
6 CONCLUSIONS
We introduced a new methodology to learn an unbounded width neural network layers within single training. by imposing a soft ordering of importance among neurons. Our approach requires very few changes to the architecture, adapts the width to the task's 's difficulty, and does not impact negatively convergence. We showed stability of convergence to similar widths under bounded activations for different hyper-parameters configurations, advocating for practical reduction of the width's search space. A by-product of neurons' ordering is the ability to easily compress the network during after training, which relevant in the context of foundational models trained on large data, which are believed to require billions of parameters. Finally, have tested AWN different models and data domains to prove its broad scope of applicability: a Transformer architecture achieved a similar loss with 200x less parameters.
10

---

Published as a conference paper at ICLR 2026
7 REPRODUCIBILITY STATEMENT
The code provided in the supplementary material relies on libraries for automatic experimentation. which enforce reproducibility of the results. These libraries require that the user specifies every detail, from data splitting strategies to hyper-parameter parameter selection and final evaluations, and subsequently the experiment is automatized and results are provided.
REFERENCES
David Barber. Bayesian reasoning and machine learning. Cambridge University Press, 2012.
Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoe Le. Understanding and simplifying one-shot architecture search. In Proceedings of the 35th International Conference on Machine Learning (ICML), 2018.
Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise training of deep networks. In Proceedings of the 20th Conference on Neural Information Processing Systems (NIPS), 2006.
Maxim Berman, Leonid Pishchulin, Ning Xu, Matthew Blaschko, and Gérard Medioni. Aows: Adaptive and optimal network width search with latency constraints. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2020.
José M Bernardo and Adrian FM Smith. Bayesian theory, volume 405. John Wiley & Sons, 2009.
Ege Beyazit, Jonathan Kozaczuk, Bo Li, Vanessa Wallace, and Bilal Fadlallah. inductive bias for tabular deep learning. In Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS), volume 36, 2023.
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? In Proceedings of machine learning and systems (MLSys), 2020.
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association. 112(518):55--87) 2017.
Andrew Brock, Theo Lim, J.M. Ritchie, and Nick Weston. SMASH: One-shot model architecture search through hypernetworks. In 6th International Conference on Learning Representations (ICLR), 2018.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv preprint, abs/2005. 2020.
Francois Caron, Fadhel Ayed, Paul Jung, Hoil Lee, Juho Lee, and Hongseok Yang. Over- parameterised shallow neural networks with asymmetrical node scaling: Global convergence guarantees and feature learning. Transactions Machine Learning Research, 2025. ISSN 2835-8856.
George Cybenko. Approximation by superpositions of sigmoidal function. Mathematics of control, signals and systems, 2(4):303-314, 1989.
Justin Domke. Generic methods for optimization- modeling. In Proceedings of the 15th International Conference on Artificial Intelligence and Statistics (AISTATS), 2012.
Simon Dufort-Labbé Pierluca D' Oro,o, Evgenii Nikishin, Razvan Pascanu, Pierre-Lue Bacon, and Aristide Baratin. Maxwell's demon at work: Efficient pruning by leveraging saturation of neurons. arXiv preprint, 2024.
11

---

Published as a conference paper at ICLR 2026
D. Elliott, S. Frank, K. Sima' an, and L. Specia. Multi30k: Multilingual english descriptions. In Proceedings of the 5th Workshop on Vision and Language. 2016.
Boglish-german german image
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Efficient multi-objective neural architecture search via lamarckian evolution. In 7th International Conference on Learning Representations (ICLR), 2019a.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. Journal of Machine Learning Research, 20:1-21, 2019b.
Federico Errica, Marco Podda, Davide Bacciu, and Alessio Micheli. A fair comparison of graph neural networks for graph classification. In 8th International Conference Learning Representations (ICLR), 2020.
Scott Fahlman and Christian Lebiere. The cascade-correlation learning architecture. In Proceedings of the 3rd Conference on Neural Information Processing Systems (NIPS), 1989.
Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. Forward and reverse gradient-based hyperparameter optimization. In Proceedings of the 34th International Conference on Machine Learning (ICML.,, 2017.
Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta- leatnle...nng. In Proceedings of the 35th International Conference on Machine Learning (ICML), 2018.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics (AISTATS), 2010.
Prem K. Goel and Morris H. Degroot. Information about hyperparamters in hierarchical models. Journal of the American Statistical Association, 76:1 10--147, 1981.
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A Journal
survey. International Journal of Computer Vision 129(6):1789-188) 2021.
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns, Advances in neural information processing systems, 29, 2016.
Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang. Dynamic neural networks: A survey, IEEE transactions on pattern analysis and machine intelligence, 44, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international Conference Computer Vision (ICCV), pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Yang He and Lingao Xiao. Structured pruning for deep convolutional neural networks: survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46, 2023.
Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. In Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI), 2018.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint, 2015.
Michael I Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. An introduction to variational methods for graphical models. Machine learning, 37:183-233, 1999.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Master's 's thesis, University of Toronto, 2009.
12

---

Published as a conference paper at ICLR 2026
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. con/exdb/nmist/, 1998.
Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, (()):541551) 1989.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436-444 2015.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In 7th International Conference Learning Representations (ICLR), 2019.
Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient- -based hyperparameter optimization through reversible learning. In Proceedings of the 32nd International Conference on Machine Learning (ICML), 2015.
Hrushikesh Mhaskar, Qianli Liao, and Tomaso Poggio. When and why are deep networks better than shallow ones? In Proceedings of the AAAI conference on Artificial Intelligence (AAAI), 2017.
Alessio Micheli. Neural network for graphs: A contextual constructive approach. IEEE Transactions on Neural Networks, 20(3):498 511, 2009.
Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. Accelerating sparse deep neural networks. arXiv preprint, 2021.
Rupert Mitchell, Martin Mundt, and Kristian Kersting. Self expanding neural networks. arXiv preprint, 2023.
Tom Mitchell. Machine learning. McGraw-her hhll,, 1997.
Achille Nazaret and David Blei. Variational inference for infinitely deep neural networks. In Proceedings of the 39th International Conference on Machine Learning (ICML), 2022.
Peter Orbanz and Yee Whye Teh. Bayesian nonparametric models. Encyclopedia of machine learning. 2010.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS), 2017.
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. Proceedings of the 35th International Conference on Machine Learning (ICML), 2018.
Lutz Prechelt. Early stopping-but when? In Neural Networks: Tricks of the trade, pp. 55-69, Springer, 1998.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence (AAAI), 2019.
Frank Rosenblatt. The perceptron: a probabilistic model for information storage and organization in the brain. Psychological review, 65(6):386, 1958.
Dilip Roy. The discrete normal distribution. Communications in Statistics Theory and Methods, 32: 1871-1883, 2003, doi: 10.108 1/sta- 120023256.
David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning representations by mack-propagating emrrr. Nature, 323:533-536. 1986.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018
13

---

Published as a conference paper at ICLR 2026
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions on Neural Networks, 20(1):61-80 2009.
David So, Chen Liang, and Quoe V Le. The evolved transformer. Proceedings of the 36th International Conference Machine Learning (ICML), 2019.
Xiu Su, Shan You, Tao Huang. Fei Wang, Chen Qian, Changshui Zhang, and Chang Xu. Locally free weight sharing for network width search. In Proceedings of the 9th International Conference on Learning Representations (ICLR), 2021a.
Xiu Su, Shan You, Fei Wang, Chen Qian, Changshui Zhang, and Chang Xu. Benet: Searching for network width with bilaterally coupled network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2175-218A 2021b.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint, 2000.
Lorenzo Valerio, Franco Maria Nardini, Andrea Passarella, and Raffaele Perego. Dynamic hard prun- ing of neural networks at the edge of the internet. Journal of Network and Computer Applications, 200: 103330, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS), 2017.
Aladin Virmaux and Kevin Scaman. Lipschitz regularity of deep neural networks: analysis and efficient estimation. Advances in Neural Information Processing Systems, 31, 2018.
Colin White, Sam Nolen, and Yash Savani. Local search is state of the art for nas benchmarks. In Proceedings of the 37th Uncertainty in Artificial Intelligence Conference (UAI), 2021.
Colin White, Mahmoud Safari, Rhea Sukthanker, Binxin Ru, Thomas Elsken, Arber Zela, Debadeepta Dey, and Frank Hutter. Neural architecture search: Insights from 1000 papers. arXiv preprint, 2023.
Pierre Wolinski, Guillaume Charpiat, and Yann Ollivier. Asymmetrical scaling layers for stable network pruning. Preprint, 2020.
David H Wolpert. The lack of a priori distinctions between learning algorithms. Neural computation, ((7):3341-199) 1996.
Lemeng Wu, Dilin Wang, and Qiang Liu. Splitting steepest descent for growing neural architectures. In Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS), 2019.
Lemeng Wu, Bo Liu, Peter Stone, and Qiang Liu. Firefly neural architecture descent: a general ap- proach for growing neural networks. In Proceedings the 34th Conference Neural Information Processing Systems (NeurIPS), volume 33, 2020.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In 7th International Conference Learning Representations (ICLR), 2019.
Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In 6th International Conference on Learning Representations (ICLR), 2018.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference Machine Learning (ICML), 2017.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV) 2019.
14

---

Published as a conference paper at ICLR 2026
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. In 4th International Conference on Learning Representations (ICLR), 2016.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V Le. Learning transferable architectures for scalable image recognition. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), 2018.
A TRUNCATED DISTRIBUTIONS' NOTIONS
Since we were mostly inspired by the work of Nazaret & Blei (022), we complement the main text with an introduction to truncated distributions. In particular, by truncating a distribution its quantile function evaluated at & (see Appendix [B] for visual explanation), the support of the distribution becomes finite and countable, hence we can compute expectations finite number of steps. truncated distribution should ideally satisfy some requirements defined in Nazaret & Blei (2022) and illustrated below.
Definition A.1 (Naaaret & Blei (2022) )
A variational family Q g(;; w) over N+ is unbounded
with connected and bounded members if
1. 4g E Q, support(()) is bounded
2. 7L E N 3g E Q such that L E argmax(q)
3. Each parameter in the set w a continuous variable.
The first condition allows us to compute the expectation over any q E Q in finite time, condition ensures there is parametrization that assigns enough probability mass to each point in the support of q. and condition 3 necessary for learning via backpropagation.
An important distinction with Nazaret & Blei 2022) is that we do not need want condition 2 to be satisfied. As a matter of fact, we want to enforce a strict ordering of neurons where the first is always the most important one. Condition 2 is useful, as done in Nazaret & Blei (2022). , to assign enough "importance" to specific layer of an adaptive-depth architecture, while previous layers are still functional to the final result. In the context of adaptive-width networks, however, this mechanism may leave a lot of neurons completely unutilized, letting the network grow indefinitely. That is why we require that each distribution fe is monotonically decreasing and all possible widths should have non-zero importance. We change the definition for the distributions we are interested in as follows:
Definition A.2. A family of distributions bounded, and decreasing members if
f(x; A) over N+ is unbounded with connected,
1. ff Q. support(f) is bounded
2. vʃ E Q, f is monotonically decreasing and Va E N+, >
3. Each parameter in the set À is continuous variable.
B How TO COMPUTE D₂ IN PRACTICE
To aid the understanding of how one computes De, we provide visualization of the exponential distribution, its discretized version (Equation ( and the quantile function evaluated at k = 0.9 in bigure
Note that typically, newly introduced neurons are multiplied by very small importance, and their random weights are sampled from a standard Gaussian. This makes their initial contribution very small, so it does not drastically alter the mes//gradients/oonergnnec From backpropagation viewpoint, the reason why new neurons are introduced that previous neurons need to become relatively more important. So, when the chosen importance distributions grow too slowly, as in the case of a Power law family distributions, this can slow down convergence because it takes a long important, and in addition large tail unimportant neurons is
time to make the last neurons added if the chosen quantile is too high. The exponential distribution, instead, worked pretty well without having to tune it extensively.
15

---

Published as a conference paper at ICLR 2026
Exponenial
Discretized Exponential
Probubiiity Density
0.8
Probabiiity Mass
0.8
0.6
0.6
0.4
quantii fuadice Am199
0.4
iusaeiil furcice A-0.00
0.2
0.9
0.0
Figure On the left, we provide visualization of the exponential density with 0.5, its cumulative density, and the quantile function evaluated at k = 0.9, which returns value close to 5. the right, we show the discretized exponential and its cumulative mass function; we can see that the quantile function of the continuous density provides an upper bound for the natural where the cumulative mass function is greater than 0.9. Therefore, one can safety use the ceiling of the output of the quantile function to determine the width De to use for given value of X.
C FULL ELBO DERIVATION
The full ELBO derivation as follows:
((A00) log
( 0)
=
(11)
E (A ( A) log pY, X, 0|X) log ())
(12)
( ( [og(a( 0, ()))(())((A)) log(g(A))((1))))
(13)
where factorized probabilities according to the independence assumptions of the graphical model. Then
AA) E o log (YJ 0, X) logp(A) + log p(0) log log (0]2)]
(14)
E flog ((A) log
p(0)
+ 1ogp(Y
(15)
(A) A
p(v)
p())
log q(v) (plvv)
log
(16)
where we use the first-order approximation in the last step. Note that when computing _2()) products from DE+1 infinity cancel Equation 13 follows by simply applying definitions, the iid assumption p(Y ||, p, X) = II plulk, p, x;), and by transforming products into summations using the logarithm.
q[A])] the
An important note about [f(A not differentiable with respect to Dr when there is jump in the quantile evaluated at k, so would seem we cannot apply it. This also happens in Nazaret & Blei (2022), Equation 11, when the function is a neural network containing non-difierentiable activations like ReLU, but we could not find discussion about it. Importantly, the function is not differentiable at countable set of points (one for each possible width), which has a measure null. Since the Lebesgue integral of the expectation is over reals, we should be able to remove this set from the integral while ensuring that the approximation still holds mathematically.
first-order (Taylor) approximation Ey(Avii xx)) (S(A, L/ 0)]
p(0) + log (Y))) A X) [f(,, p)] f(v p) is that the function f(>, 0) log v(A) + log q(
p(A)
D LIPSCHITZ CONTINUITY OF THE ELBO
Whenever the quantile of the finite distribution changes, we need to change the number of neurons of the neural network. Here, want to show that the change in the ELBO after change in the number
16

---

Published as a conference paper at ICLR 2026
of neurons is bounded. Therefore, provide theoretical result (theorem D.1 that shows that the ELBO satisfies the Lipschitz continuity.
Proposition D.I (AWN ELBO Lipschitz continuity). The ELBO loss of eq. 9 with respect the change in the depth Dr for the layer E, is Lipschitz continuous.
Proof. We focus on the term involving De of the ELBO, we write eq. 9
(
p(p)
log
(vV) + log
+ log
((vv))
where only the second and last terms depend on D (De) Let's first define
p(p) log
ΣΣ log p(pn)
En(DD))
{(pl))
f=n==
f=1
We have that fi(D)) is Lipschitz continuous, indeed, when De changes to Dy, we have
De
p(pn)
lfi(De) - f((D))=
Σ log
(17)
((pllv)
n=Dz
De
log p(pn)
<
(18)
{(pnlp)
== Dr
p(pn)
max
loog
1|D Del
(19)
q(pxlv))
Therefore
( S Dz Dr.
with M max, log p(p... (()..
We now look the last term,
fD( log p\ V, p.)
This function is a neural network followed by the squared norm. Therefore, (2(D) is continuous almost everywhere (Virmaux & Scaman| (0188). therefore Lipschitz both parameters and the input. When we change D; to De, we are adding network parameters, in particular, if we look at the two-layer network, have
Vo(W (e(W++bb) - V E ER W E Baxd
we then have new parameters, which can be shown to be
y' = Vo(Wx+ +
R pmxn' W' E Rn'xd,''e R",,'' é'eR"
new neurens
the second term also Lipschitz, therefore the function (D) also Lipschitz. Indeed the network is composed of linear operation and the glement-wise activation functions. If the activation functions are continuous and of bounded gradient, then the whole network is Lipschitz continuous.
E FULL DERIVATION OF THE RESCALED WEIGHT INITIALIZATION
This Section derives the formulas for the rescaled weight initialization both in the case of ReL.U and activations like tanh.
Background
We can rewrite Equation 10
h
pp
(20)
D
a o
Σ
w
sj
t-1
Pj
(21)
1=1
h
17

---

Published as a conference paper at ICLR 2026
where pf f(()).
As refresher, the chain rule of calculus states that, given two diffcrentiable functions RP - R and f = - RDD, their composition g fo IR is differentiable and
(g (t) = vo(f(t))
Vg(f(t)) = (t) ag(fb(t))
E EIID
0fi(t)
((()) - (E((). lb(1)) Rx
For reasons that will become clear later, we may want to compute the gradient of the loss function with respect the intermediate activations a at given layer, that is VL(a')
(aC(af) aC(a
0af
80
We focus on the i-th partial derivative aC(a.) where the only variable is af. Then, view the do;
computation of the loss function starting from af composition of function a
R = (ai-1(a))
x (a()) and another function (abusing the notation) C R that computes the loss value starting from a f+1
R
By the chain rule:
OC(a.)
(dL(att)
al(a
4+1 Ner
[aa (a daf
da Da ++1
(af)
(22)
da!
0asti
#+1 Da
N
dag
(10
Vo((())
(()
Theorem 3.1 Let us consider an MLP with activations as in Equation 27 Let us also assume that the inputs and the parameters have been sampled independently from a Gaussian distribution with zero mean and variance o² Var [ar[wfy] ji, f. At initialization, the variance of the responses of across layers is constant if, vl fl.....Ly
1
Varlw
for activation such that o'(0) ~
(23)
ΣS (Dtt2)
2
Varlw*] Σp (p, 12
for the ReLU activation.
(24)
In addition, we provide closed form formulas to preserve the variance of the gradient across layers.
Proof. Let start from the first case of a (0) ss 1. Using the Taylor expansions for the moments of functions random variables as Glorot & Bengio (2010)
Var[a
Var
w aj Pj
t-1 1-1--11 e-1
(25)
D
D
o'
E
Σ
w
a
1 e-1
Var
1 a
D3
(26)
Using the fact that pj is constant and that w and a are independent from each other
D
Dr
E
Σ
a
Σ
Ela R- '10 = 0.
(27)
Therefore, recalling that o'(0) (0)zz x
D
o'
FF
Σ
1-1
(28)
18

---

Published as a conference paper at ICLR 2026
As result, arrive
De-1
Var [a so Var
Σ
aijaj
2- pp
(29)
Because w and are independent, they are also uncorrelated and their variances sum. Also, using the fact that Var[aX] = Varar[> [a] for constant a,
De-
De-1 1 6- 1-1 = Σ
De-1 Var T 1 = Σ Var [wy 0 (o)-t)²
Var Σ
W
13
2-11
(30)
Finally, because the mean of the independent variables involved is zero by assumption, it holds that Varlul, f Varfua We can also abstract from the indexes, since the weight variables are i.idd and from that follows that Varla,] = Var[af] Vi,j. i.j E (1....,Drf obtaining
De
6-1 Var[a"] x Varlw"]Varja'
Σ
(1 10
(31)
noting again that the previous equation does not depend Var Var[o'-1), which can be achieved whenever
We want to impose Var[a"] x
Var[w")
(32)
Σj (0 (
Condition on the gradients From backpropagation perspective, a similar desideratum would be Tac(a( Var (ac(a
ensure that Var
da
0a
Using Equation 22 and considering in Glorot & Bengio 2010) that at initialization are in linear regime where o' o'(x) = 1,
Ta((a)) dag
Dett al(a 1)) day (a2tt((a (af)
Dett al(a,t)) (4) da (act)(af))
Var
Var
Var
(33)
Dagt da!
Da, P+1
da!
1=1
J=1
DELL ol(a 4+1)
De
Var
o'
Σ
s+1
kpkk
wost ps
(34)
8ast
jk
1
Using the same arguments as above one can write
al(aa) da
De+t
Dr-1
aa(aft1)
Var
Varlw
Var
da 41
p
Varlw*
(1pp)
Var
dagt 0a
f=1
(35)
'It is worth noting that, in standard MLPs p¹ = 1 so we recover the derivation of Glorot & Bengio 2010
since -D. (p5-1) would be equal to D, In Glorot & Bengio 2010 our D is denoted as "n." (see Equation 5).
"Note that what we impose is different from Glorot & Bengio (0100). where the specific position i is irrelevant. Here, we asking that the variance of the gradients for neurons in the same position i, but different layers, stays constant. Altematively, could impose equivalence for all
19

---

Published as a conference paper at ICLR 2026
Expanding, we get
De12
aL(a
Varl 10 ++
(p,))
Var
(36)
dag f+1
j=1
E42
Dr+t Σ
Di+a
al((at)))
Varlw N=++1
II
(p) 10tt
Var
(37)
day R42
f=1
constant wir..
++2
Deta
De1
= (x()
II (=t+1
V
Var
Σ 42
(38)
day
Therefore, we can recursively expand these terms and obtain
Var
[e((a))]
(pf)²
I
L
Varlur
Dr
Var
D1)
II
L-1 De
II Σ
dag
Di day
N==+1
b=
(39)
In this case, the variance depends on i just for the term (p!)², Finally, by imposing
[a((a()
0L(a
Var
Var
(40)
daf
0att1
and simplifying common terms we obtain
Dett
(pf) ( Var(w
Σ
(0 (0241/2)
(22
(41)
(pt+l)
Varlw 6+1 [+]
(42)
Deys (1/42)²
Both Equations and 42
verified when initialize the distributions fe in the way for all
layers, which implies (pt) 2 (p, 122 Note that without this last requirement, Equation 42 would
violate the i..dd assumption of the weights.
To show similar initialization for ReLU activations, need the following lemma.
Lemma E.1. Consider the ReLU activation y = max(0, x) and a symmetric distribution p(z) around Then Bsy²] 2 Var (r).
Proof.
+00
max((, p(( )x (0 z'p(x)do 1 d²p(z)dæ
Bly²1
(43)
2
Because p(x) is symmetric, B[x] 0. Then
x² p(r)dd 2 / (a - --EEEe -=xx=))p(xx) p((p))()) Var[z)]
1
(44)
2
2
Using Lemma E.I can rewrite Equation 26
D
1 ]
De-1
Varla Var
Σ
w
a
e-
1
Var
Σ 1
w
af
(45)
2
De-1
=2 Varlw= Varraa -11
1
(46)
j=1
20

---

Published as a conference paper at ICLR 2026
where approximation has been made. Similar to prior results, it follows that
Var[w*] ES=1" (D) 10
(47)
From backpropagation perspective, since (xx) (2013): we assume that the pre-activations a have zero mean, then the derivative of the ReL.U can have values with equal probability. One can show, by expanding the definition of variance, e+1) 1+1 Var(uust
use similar arguments in He al.
that Varlo'
k==
e-1 Wik
As a result, one can re-write Equation 26 as
DiIt ol(a)
D2
Var Σ
of dag a1
w
++1
f- Wie
Pi
(48)
Dr+t
aa(aatt dast!
Var
(49)
2
where we have made the assumption of independence between since the former term is likely constant. Following identical derivation above, obtain
24 (ER= jk o and wft f+1
2 12
Varlw
(50)
(11)²
De+s (D) 1)
which is almost identical to Equation 47 and not dependent on when we initialize fe in the same way for all layers.
Comparison of convergence for different initializations Below, we provide comparison of convergence between an AWN ReLU MLP initialized with the standard Kaiming scheme and one initialized according to theoretical results ("Kaiming*").
Laiming Layer
Laimings Layer
baiming Layer
kaimings Layer
kaining Lops
kiiningt Lager 3
900
10!
Laiming Layer
kaiminge Layor
Accunccy (%)
Donäty
Laiming layer
ksimings Laye
Laiminge
baining
70
10
60
10
300
400
Activation Values
Epoch
Figure 7: (Left) Effect of the rescaled initialization scheme ("Kaiming+") where neurons' activations are computed using Equation TO Compared to the standard initialization, the variance of activations agrees with the theoretical result. (Right) Without the rescaled initialization, convergence is hard to attain on the SpiralHard dataset (please refer to the next section for details about the dataset).
ReI.U-based MIP,
F DATASET INFO AND STATISTICS
The synthetic datasets DoubleMoon, Spiral, and SpiralHard are shown in Figure These binary classification datasets have been specifically created to analyze the behavior of AWN in controlled scenario where we are sure that the difficulty the task increases. We provide the code to generate these datasets in the supplementary material.
information on all datasets' statistics and the data splits created to carry out model selection (inner split) and risk assessment (outer split), respectively. Note that for Multi30k, the
21

---

Published as a conference paper at ICLR 2026
2
-20
Figure 8: The DoubleMoon, Spiral, and SpiralHard synthetic datasets are used to test AWN's inner workings, and ordered difficulty. Orange and blue colors denote different classes.
Table 3: Dataset statistics and number of samples in each split are shown.
# Samples
Features
# Classes
Outer Split (TR/VL/TE) Inner Split (TR/VL)
DoubteMoon
5000
2
3600/400/1000
3600/400
Spiral
5000
2
3600/400/1000
3600/400
SpiralHard
10000
2
7200/800/2000
7200/800
pol
15000
48
2
10800/1 I 200/3000
10800/1 200
MiniBooNE
130064
50
2
93645/10406
credit card clients
30000
23
2
21 2100/2400/6000
21600/2400
MNIST
70000
28x28
10
48610/9723/1 1667
48610/9723
CIFARIO
60000
32x32x3
10
41665/8334/10000
41665/8334
CIFARIO0
60000
32x32x3
100
41665/8334/10001
41665/8334
NCII
4110
37
2
33228/370412
3328/370
REDDIT-B
1620/180/200
1620/180
Multi30k
31000
128x50257
Multilabel
29046/968/1000
29046/968
maximum sequence length is 128 and the number of possible tokens is 50527. The classification loss computed by comparing the predicted token against the expected one, ignoring any token beyond the length of the actual target. For detailed description of the graph datasets, the reader can refer to Errica 2020).
G HYPER-PARAMETERS FOR FIXED AND AWN MODELS
For each data domain, we discuss the set of hyper-pppae parameters tried during model selection for the fixed baseline and for AWN. We try our best keep architectural choices identical with the exception of the width of the MLPs used by each model. In all experiments, we either run patience-based early stopping (Prechelt (998) or simply select the epoch with best validation score.
G.1 SYNTHETIC TABULAR DATASETS AND PERMUTED MNIST
We report the hyper-paramedt configurations tried for the fixed MLP/RNN and AWN in Table 4 In particular, the starting width of AWN under the quantile function evaluated at A: 0.9 is approximately 256 hidden neurons. We also noticed that learning rate value of 0.1 makes learning the width unstable, we did use in our experiments. The total number of configurations for the fixed models is 180, for AWN is 18. We also did not impose any prior on an expected size to see AWN recovers similar width compared the best fixed model (please refer Table [ starting from the largest configuration among the fixed models.
G.2 IMAGE CLASSIFICATION DATASETS
We train from scratch ResNet20 and focus architectural choices the MLP that performs classification using the flattened representation provided by the previous CNN layers. This ensures that any change performance only due the changes apply the ML.. The configurations
22

---

Published as a conference paper at ICLR 2026
Table 4: Hyper-parameter configurations for standard MLP/RNN and AWN versions on tabular datasets.
Shppe-Parameele
DoubleMoon
Spiral
Barch Size
32
[32, 128] (MIPIRNN), 128 (AWN)
[32, 128] (MIPPRNN) 128 (AWN)
Epochs
500
1000
5000
Hidden Layers
1
[1, 4]
Layer Width
16, 1228, 256]
Non linearity
[ReLU, LeakyRel.U, ReLU66
Optiiizer
Adam, learning rate E [0.1, 0.01] (MLPRNN), 0.01 (AWN)
AWN Specific
Quantile Fun. Threshold & Exponential Distribttions Rate of
0.9
0.01 [1.0, 10.0] Uninformative
Prior over A
shown in Table 5] note that width of 0 implies linear classifier instead of 1 hidden layer MLP. After previous results on tabular datasets showed that LeakyReLUU performs very well, we fixed in the rest of the experiments. In this case, we test 5 different configurations of the fixed baseline,
Table 5: Hyper-parameter configurations for standard MLP and AWN versions on image classification datasets.
Hyper-Paramene
MNIST CIFAR10 CIFAR100
Batch Size Epochs Layer Width (classification layer) Optimizer
128
200
0, 32, 128, 256, 512)
SGD, learning rate 0.1, weight decay 0.0001, momentum 0.9 Multiply learning rate by 0.1 at epochs 50, 100, 150
Scheduler
AWN Specific
Quantile Fun. Threshold
0.9
Exponential Distributions Rate
0.02
of
1.0
Prior
Uninformative
whereas we do not have to perform any model selection for AWN. The rate of the exponential distribution has been chosen have a starting width of approximately 128 neurons, which is the median value among the ones tried for the fixed model.
G.3 TEXT TRANSLATION TASKS
Due to the cost of training a Transformer architecture from scratch on medium size dataset like Multi30k, we use most hyper- parameters' configurations of the base Transformer in Vaswani et al.) (2017), with the difference that AWN is applied to the MLPs in each encoder and decoder layer. We train an architecture with encoder and decoder layers for 500 epochs, with a patience of 250 epochs applied to the validation loss. We use an Adam optimizer with learning rate 0.01, weight decay 5e-4, and epsilon 5e-9. We also introduce a scheduler that reduces the learning rate by factor 0.9 when plateau in the loss is reached. The number of attention heads is set to 8 and the dropout to 0.1. We try different widths for the MLPs, namely [128,256,51] 2,,551,,,024,244 The embedding size is fixed to 512 and the batch size is 128. The AWN version starts with an exponential distribution rate of 0.004, corresponding to approximately 512 neurons, and a =f=1.
G.4 GRAPH CLASSIFICATION TASKS
We train the AWN version of Graph Isomorphism Network, following the setup of Errica et al. (2020) and reusing the published results. We test 12 configurations shown in the table below. Please note that an extra MLP with I hidden layer is always placed before the sequence graph convolutional layers.
23

---

Published as a conference paper at ICLR 2026
Table 6: configurations for AWN versions on graph classification datasets.
Hyper-Paramettr
NCI1 REDDIT-B
Batch Size
[32,128]] 1000
Epochs
Graph Conv. Layers
[1 4]
Global Pooling
[sum, mean]
Optimizer
Adam, learning rate 0.01
AWN Specific
Patience
500 epochs 0.9
Quantile Fun. Threshold & Exponential Distributions Rate of
0.02
10.0
Prior over À
Uninformative
H RE-TRAINING A FIXED NETWORK WITH THE LEARNED WIDTH
While on DoubleMoon AWN learns similar number of neurons chosen by the model selection procedure and the convergence to the perfect solution over runs seem much more stable, this not the case on Spiral, SpiralHard and REDDIT-B, where AWN learns higher total width over layers. To investigate whether the width learned by AWN on these datasets is indeed the reason behind the better performances (cither more stable training better accuracy), on the baseline networks, this time by fixing the width of each layer to the average width over hidden We call this experiment "Fixed+".
again the experiments
layers learned by AWN and reported in Table T
Spiral
SpiralHard
REDDIT-B
Fixed
99.5(0.5)
98.0(2.0)
87.0(4.4)
AWN
99.8(0.1)
100.0(0.0)
90.2(1.3)
Fixed+
100.0(0.1)
83.2(17.0)
90.7(1.4)
The results suggest that by using more neurons than those selected by grid search possibly because the validation results were identical between different configurations it is actually possible to improve both accuracy and stability of results Spiral and REDDIT-B, while had convergence issue on SpiralHard. Notice that the number of neurons learned by AWN on REDDIT-B was well outside of the range investigated by Xu et al. (2019) and Errica al. (2020), demonstrating that the practitioner's bias can play an important role on the final performance.
aware of the best width achieved by the grid search approach of Errica al. (2020) on REDDIT-B, we ran an additional experiment on fixed models, testing smaller and larger widths compared to the one selected by AWN (average of 7933). What we discover the following.
Because
Table 7: We report REDDIT-B validation performances, averaged across the 10 outer folds, for fixed-width DGN. The other hyper- hyper- perretparseseers match those selected by AWN during model selection, as per this specific experiment.
Total Width
Mean
Std
50
89.28
3.50
250
92.89
1.45
500
92.67
2.03
793
92.83
3.55
1000
1.60
1250
92.78
2.33
1500
91.78
2.41
Though these results are statistically comparable, the average performance obtained using AWN's learned width seems to be close to a local optimum for the fixed-width model.
24

---

Published as a conference paper at ICLR 2026
I FURTHER QUALITATIVE RESULTS ON IMAGE CLASSIFICATION DATASETS
Because it well known that MNIST, CIFARIO, and CIFARIO0 are datasets of increasing difficulty, we attach figure depicting the learned number of neurons and the convergence of AWN on these datasets, following the practices highlighted in He et al. (0016) and described in Table 5 to bring the ResNet20 to convergence. Akin
tabular datasets, we see a clear pattern where the learned number of neurons, averaged over the 10 final training runs, increases together with the task difficulty. The AWN version the ResNet20 converges the same amount of epochs as the fixed one.
100
Dataset
Learned Hidden Neurons
CIPARAO
80
CHFARD00
Accuracy (5)
Dataset
20
MNIST
CIFARIO
CIFAR100
23
75
100
125
150
170
200
50
75
120
150
175
200
Epoch
Epoch
Figure 9: We present an analysis similar to Figure but for image classification datasets of increasing difficulty. One can observe the jumps caused by the learning rate's scheduling strategy.
COMPARISON WITH NEURAL ARCHITECTURE SEARCH (NAS)
some basic NAS methods whereas AWN's methodology does not. The truncation ability of AWN for pruning but rather confirmation of our claims. In fact, NAS and pruning can actually be combined with AWN and do not necessarily be seen as "comppting" strategies.
approach, thought instructive on the image classification datasets. Most NAS methods fix an upper bound on the maximum width,
Although complementary
intended as substitute
The results are shown in Table 8 That said, we compared the performance of AWN against simple NAS methods on the vision datasets. We tested grid search, random search (Following Wu al.) (2020), local search WWhite et al. (001)) and Bayesian Optimization (Barber 2012), fixing the same budget of for all methods and width values between (Linear classífier) and 512 neurons. Results comparable to those of AWN.
Table 8: Comparison between different NAS approaches and AWN. We remind the reader that NAS an orthogonal and complementary research direction to AWN.
Grid Search
Random Search
Local Search
Bayes. Optim.
AWN
MNIST
99.6(0.1)
99.6(0.0)
99.5
99.4
99.7(0.0)
CIFARIO
91.4(0.2)
90.1(
90.6
91.2
91.4(0.2)
CIFARION
66.5(0.4)
64.9(1.1)
64.9
65.9
63.1(4.0)
25

---

organized by type of Published as a conference paper at ICLR 2026
K NON-LINEARITY ABLATION STUDY
Below, we report an ablation study on the impact of non-linear activation functions. We analyze the convergence to the same width across different batch sizes and exponential starting rates on the SpiralHard dataset.
Leaky ReLU
RcLU
ReLU6
10a
Neurons
104
10c
10a
Batch Sizee
Batch Size
Batch Size
10a
10/
102
101
512
250
101
512
256
10
512
10P
107
107
10a
104
10a
10
102
10a
10a
109
10
102
10
10
Epoch
Epoch
Epoch
Leaky ReLU
ReL.U
ReLU6
100
10.
10
Rate x
Leamnd Hidden Neurons
Learned Hidden Neurous
Hidden Neurous
0.00
0.01
10a
10a
10a
Rate yr
Rate v
10a
10a
10a
0,035
10.
0:11
10a
0.0
10a
100
10a
100
10"
10
102
10a
104
10P
10a
102
10a
10a
109
10
102
10a
104
Epoch
Epoch
Epach
Leoky ReLU
R+LU
ReLU6
1.00
100
100
Rate A
Rate A
Rate A
0.02
0.05
0
Accuracy
0.1
Accuracc
0.1
Accuracy
0.1
70
70
60
60
101
10
10a 10
10a
Epoch
Epoch
Epoch
Figure 10: Impact of different batch sizes and starting exponential rates non-linear activation function.
These qualitative analyses support our claims that using a bounded activation is one way to encourage the network not to counterbalance the learned rescaling of each neuron. In fact, using a ReLU6 shows a distinctive convergence to the same amount of neurons among all configurations tried. The decreasing trend for LeakyReLU and ReLU activations may suggest that these configurations also converging to similar value than ReI.U6, but they are taking a much longer time.
26

---

Published as a conference paper at ICLR 2026
L LEARNED WIDTH FOR EACH LAYER ON GRAPH DATASETS
Split 1
2plit
Split 3
Split 4
Split 5
Leamed Hidden
101
10"
Split
Split
Split
Split
Split
Learned Hidden
10"
500
500
1000
500
1000
1000
1000
Epoch
Epoch
Epoch
Epoch
Epoch
Split
Split
Split
Split
Split
10a
Learned Hidden
10a
10a
10"
Split 6
Split 7
Split 8
Split 9
Split 10
Leared Hidden Nearoes
109
10a
10!
109
500
1000
600
1000
500
1000
500
1000
800
2000
Eppch
Epoch
Epoch
Epoch
Epoch
Figure 11: Learned neurons per layer (from to 5), averaged over 10 final runs for each of the 10 best configurations selected in the outer folds. The first and second rows refer NCII, the third and fourth refer to REDDIT-B. One can observe similar trends in most cases.
27

---

Published as a conference paper at ICLR 2026
M QUANTILE ABLATION STUDY
100
Quantle
Quanile
Lamed Hiddon Neurens
Accuncy (%)
107
10
107
100
10
10
10*
Dpoch
Epoch
100
Quantile
Quanilee
Leane Hodde Neurvn
Accuraty (%)
100
109
0
10
107
307
107
10P
107
16P
30*
Epoch
Epoch
105
100
Quantile
Quamilc
Letmel Hidken Neuros
04
Accuncc (R)
100
0
10
10a
107
103
10P
107
107
168
10
Dpoch
Epoch
100
Quantile
Quanile
Lornel Hilden Neucoss
I27
Accurcy (%
102
C0
101
104
10²
104
100
102
10
Egoch
100
Quinttlle
Quanile
Lormel Haldes Nammns
105
80
Ascurac (%)
103
60
10*
107
10"
10
Epoch
Epoch
Figure 12: We report learned width and validation accuracy trends averaged over 1-layer AWN configurations SpiralHard. We see how higher quantile i: (i.e., better ELBO approximation) grants more stable performances. All architectures keep adapting to the task.
28

---

Published as a conference paper at ICLR 2026
N REGULARIZATION ABLATION STUDY
In Section 3 we argued there are two ways we can maximize the importance rescaling effect, namely by using bounded activation functions and by regularizing the weights. In this section, we show with an ablation that these strategies are generally not required, in the sense that AWN is still able to solve the tasks, although might introduce more neurons than needed. The table below compares our original results on tabular data (Table D with those where we did not consider ReLU6 and where we imposed an almost null regularizer on the weights (o 100).
Table 9:
We compare performances and learned width of AWN, where we removed regularization and bounded activations from preerpparaeeters, with the results in the main paper.
Width (AWN)
Width
AWN
AWN
NoReg
(AWN NoReg)
Mean
(Std)
Mean
(Std)
Mean
(Std)
Mean
(Std)
DoubleMoon
100.0
(0.0)
100.0
(0.2)
8
(2.8)
17
(16.6)
Spiral
99.8
(0.1)
100.0
(0.0)
66
(8.7)
108
(26.8)
SpiralHard
100.0
(0.0)
100.0
(0.0)
227
(32.4)
539
(809.8)
pol
99.2
(0.1)
99.2
(0.1)
84
(11.0)
2335
(552.8)
MiniBooNE
93.2
(0.1)
93.8
(0.2)
53
(11.1)
4907
(1141)
credit card
81.8
(0.1)
81.8
(0.1)
51
(12.0)
53
(13)
We see that all cases, not using regularization nor bounded activations leads comparable performances but definitely higher widths. For some of the configurations tried, the width tends to grow lot, which is why we set maximum width of 5000 in the interest of time, although performances do not seem to decrease in spite of that. Further investigation into the results revealed that the weight regularization is what helps the most controlling stable behavior of the width, while the bounded activation did not have clear impact that. We therefore recommend some degree of regularization make sure that the effect of the importance rescaling properly taken into account by the optimization process.
29

---

Published as a conference paper at ICLR 2026
O WALL-CLOCK TIME COMPARISON
To further highlight the advantages brought by AWN, we report the average runtime (in seconds) required complete single model selection configuration for AWN and for the fixed model. In cases where model selection is not performed (e.g., AWN on vision tasks), we instead report the average runtime of final run. Results for NCI1 and REDDIT are excluded from this comparison, since they were taken directly from the literature. Based on these runtimes, we compute two measures: the ratio of average single-run times (Single-Run Ratio), and ii) the ratio of total time required by AWN and the fixed models to complete the entire model selection process (Model Selection Ratio).
Table 10: We report average wall-clock times for AWN and the fixed model, measured either over model selection configurations or final runs when no selection is performed. We also provide two ratios: i) single-run wall-clock times and fi) total model selection time (fixed models always test at least five widths).
Dataset
Avg Fixed (s)
Avg AWN (s)
Single-Rur Ratio (L)
Model Selection Ratio (L)
DoubleMoon
70
218
3.1 x
0.62 x
Spiral
120
141
1.1
0.24
SpiralHard
1982
7772
3.9x
0.78 x
pol
1251
1088
0,87
0.17 x
MiniBooNE
11794
4496
0.38x
0.07 x
creditcards
2104
1815
0,90
0.14x
PMNIST
842
2014
2.4x
0,48 x
MNIST
1838
3000
1.6x
0,33x
CIFARIO
1444
2880
2x
0,40
CIFARI00
1474
1740
1.2x
0.24
Multi30k
45100
19800
0.42x
0.09 x
The results confirm that AWN's overhead of adapting the network vanishes for larger datasets with higher feature dimensionality. Interestingly, on some of these datasets the average single-run wall time is even shorter than the fixed models. By being able reduce hyper-parameter tuning of the width while preserving most performances, the total model selection time ratios are consistently in favor of AWN, and in principle scale favorably with the number of widths tested. Note that memory consumption scales with the width of the layers, but our methodology does not introduce any noticeable extra memory requirements.
30

---

Published as a conference paper at ICLR 2026
P IMPACT OF REMOVING MOST IMPORTANT NEURONS
We complement the truncation analysis of Figure 5 by showing what happens when we remove individual neurons a group of them, starting from the most important ones. This analysis provides further insights about how neurons behave in AWN.
70
50
80
important least)
Segle Curdlaiee
Accuracy
0.6
0.2
Single Removed
Figure 13: We show the impact of removing the most important neurons, either in batch (top) or individually (bottom) on Spiral performances, using the same experiment as Figure 5
We observe that performance quickly drops, as expected, and we reach random performance after removing the first 440 neurons out of 83. The second analysis prunes individual neurons understand how much their absence influences performance which can have a different effect compared to removal of set of neurons to see if there exist two neighboring neurons that contribute similarly to the performance drop or do not contribute at all. Removing most of the first 25 neurons out of 83 contributes majorly performance degradation, but interestingly the removal of some neurons does not affect performances. Instead, removing any subsequent neuron has little impact (but for one) on performances, as one would expect. This suggests that we may be able to compress the neural network even more: we leave this investigation to future work.
31


Reviews:
[{"review_id": "mKhLN36txy", "reviewer": "Reviewer_DdyY", "summary": "This paper proposed a Bayesian method for setting the number of neurons per layer in a neural network. It starts with a (theoretically) infinite-width neural network and trains both the weights/biases and a width parameter for each layer. The chosen Bayesian method is Variational Inference.\n\nSummary of a **theoretical** training step:\n1. sample the width parameters $\\lambda_l \\sim \\mathcal{N}(\\nu_l, 1)$;\n2. use the $\\lambda_l$ to compute the neuron scalings $(f_l(i))_{i \\in \\mathbb{N}^+}$, where $f_l$ is a decreasing function parameterized by $\\lambda_l$ (the larger the $\\lambda_l$, the quicker $f_l$ decreases);\n3. given the $(f_l)_l$, compute the practical widths $(D_l)_l$: $D_l$ is the smallest integer such that $\\sum_{i=1}^{D_l} f_l(i) \\geq k = 0.9$;\n5. depending on the new widths $D_l$, prune or create neurons at each layer;\n6. provided that each neuron outputs $h_j^l = f_l(j) \\, \\sigma(\\sum_{i = 1}^{D_{l-1}} w_{ji}^l h_i^{l-1})$, that is, the output of the activation function $\\sigma$ scaled by a factor $f_l(j)$, perform a forward-backward and gradient step on the variational parameters $\\rho_{ji}^l$ of the weights $w_{ji}^l \\sim \\mathcal{N}(\\rho_{ji}^l, 1)$.\nIn practice, the variational parameters $(\\nu_l)_l, (\\rho_{ji}^l)_{lij}$ are trained directly, **without sampling** of the width parameters $\\lambda_l$ or the weights $w_{ji}^l$.\n\nThe neuron scalings $f_l(i)$ are a key component of the method, since they allow:\n* to practically deal with theoretical infinite-width layers;\n* to backpropagate the gradient of the loss up to the variational parameters $\\nu_l$: the loss does not depend continuously on the $D_l$, so $D_l$ cannot be used directly to optimize the loss. But $(f_l(i))_{li}$ works as a proxy for the \"layer width\" and: a) we have a continuous dependence of the loss on the $f_l(i)$; b) the $f_l(i)$ can be randomly generated with a continuous dependence on the variational parameters $\\nu_l$.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "# Clarity\nThe paper is well-written.\n\n# Significance\nThis paper proposes a Bayesian formulation of the problem of width selection when training a neural network, which is a well-grounded way of performing pruning and adding neurons (layer-wise). Additionally, the fact that, in theory, infinite-width neural networks are trainable (up to a decreasing scaling factor $f_l(i)$), makes this method fit several well-known theoretical frameworks (e.g., Neural Tangent Kernels).\n\n# Novelty\nThere are several Bayesian methods to select neurons or parameters, but, up to my knowledge, the method as a whole is new.\n\nPlease note that the idea of considering possibly infinite neural networks with decreasing scaling factors is not new. See for instance [1], which also cites [2] as a work introducing \"the idea of using asymmetrical scaling parameters\", which date respectively from 2023 (published in 2025) and 2020.\n\nHowever, the proposed method is a unified and original way to remove and add neurons, which is undeniably new. Additionally, the idea of **parameterized** asymmetrical scalings is new.\n\n[1] *Over-parameterised Shallow Neural Networks with Asymmetrical Node Scaling: Global Convergence Guarantees and Feature Learning*, F. Caron et al., Transactions on Machine Learning Research, 2025.\n\n[2] *Asymmetrical Scaling Layers for Stable Network Pruning*, P. Wolinski et al., [https://openreview.net/pdf?id=6GUIv9eYnD7](https://openreview.net/pdf?id=6GUIv9eYnD7), 2020.", "weaknesses": "I do not see major weaknesses in this paper. But, on the technical side, several aspects deserve a discussion :\n1. why choosing a discretized exponential distribution for $f_l$? Is there a theoretical/heuristic argument? One could choose, as in [2], slowly decreasing scalings, such as $x \\mapsto 1/x$ or $x \\mapsto 1/(\\sqrt{x} \\ln(x))$...\n2. according to Eqn. (7), $\\lambda_l$ could be either positive or negative, since it is a Gaussian random variable. This is not acceptable, provided Eqn. (6), where $\\lambda_l$ is a scaling factor in an exponential. Is this a problem in practice? Even if it is not, would it be possible to replace the Gaussian distribution by a distribution supported on $\\mathbb{R}^+$?\n3. p 4, line 179: \"Appendix A and B provide formal requirements about $f_l$\". It seems that these Appendices are not about such \"formal requirements\", they are about \"Background notions\" on variational inference (App. A) and \"How to compute $D_l$ in practice\" (App. B). I am very curious about these \"formal requirements\", would it be possible to provide some? Anyway, this reference to Appendices A and B is not consistent with their content.\n\nClarity: are some \"minus\" missing in the exponentials in Eqn. (6)? It would be nicer to write $e^{-\\lambda_l x}$ instead of $e^{\\lambda_l(x)}$ (removing the parentheses would also makes it easier to read).\n\nStyle (minor concern): In the abstract and in the introduction, the authors **seem** to claim that there is no research in automatic hyperparameter search (which includes hyperparameters related to neural network architecture). The authors write \"For almost 70 years, researchers have **primarily** relied on hyper-parameter tuning [...] This paper challenges the status quo [...]\", which is technically correct, but slightly misleading. It is more a matter of style, which may be seen as a little fancy, than a matter of knowledge of works on neural architecture search, which are cited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sBGHeHOUBR", "reviewer": "Reviewer_KoUG", "summary": "The paper investigates growing the width of the linear layers, by considering the width to be hypothetically infinite dimensional, they then estimate the number of neurons in a layer during each batch by a single learnt latent variable $v_l$ per layer, which shifts the importance distribution (they have a fixed monotonically decreasing function to compute the probability, that is the initial neurons are more likely to be important than the new ones), any new neurons within the cumulative probability threshold ( $k$ >0.9)  are added, and the neurons with the least index (importance) can be de. Since new neurons are expected to have low importance (as the function is monotonically decreasing) the addition is expected to be stable. The experimentation is performed on tablular, image and text data on MLP, CNN and Text respectively to show adaptability of the method.", "questions": "1. A neuron can be less important for a specific subset of data but important for another subset of data, limiting this spike by applying an expected importance of the neuron , can lead to multiple neurons computing the same features to circumvent the loss in spike, this phenomenon can increase the number of redundant parameters being learnt. If we randomly drop certain neurons with similar importance do we observe minimal drop in performance? \n2. Why did the paper not explore Adaptive Depth, is it because of stability issues (like in case of depth, the deeper the layer is its expected to be more important)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The main advantage of the paper is that the latent variable which controls the width is learnt, so the addition and deletion of neurons is fast and requires significantly less compute than existing hessian based methods.\n2. The training is stable at appropriate batch sizes, since the importance is low for large widths, and usually we start with a considerably large width for large models. \n3. The strict ordering based on index ensures that old neurons are preserved, which again improves the stability of the training, but at the cost of flexibility.\n4. The results show that usually, the training yields similar or better accuracy, with similar width and in some cases, the width is dramatically increased but the performance gains too.", "weaknesses": "1. Its harder for the neural network to change trajectory, since the importance is provided based on the index, if an old neuron has to be removed all newer neurons must be pruned as well, this limits the neural network from moving away from features it learnt.\n2. More redundant copies, the importance acts as a weight to the activation, therefore when the strength of an important neuron might not be enough, the neural network may choose to learn redundant copies to strengthen the importance of features. This may also lead to superposition of very important features on a small subset of the layer.\n3. If there are two features A and B being computed at index x and x+1, if B becomes more important than A as the training progresses, then either both A and B are dropped and re-learnt (which is hard if the index is small) or A transforms to B and B transforms to A over multiple iterations, which can slow convergence. The main drawback is we cannot effectively rearrange importance of neurons because of the hard index based importance.\n4. While appendix H shows that the learnt width gets similar performance when trained from scratch while fixed, is this the best width for the accuracy? could an ablation be done when training fixed neural networks with an increasing width in intervals of 100 until 1.5/2 x width.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QkbspGXkcM", "reviewer": "Reviewer_SBvZ", "summary": "This paper presents a framework that enables neural networks to dynamically learn and adjust their layer widths during training, removing the need for preset upper bounds. \nThe method leverages a probabilistic variational inference approach, adapted from “Variational Inference for Infinitely Deep Neural Networks” by Nazaret and Blei (2022), but focuses on varying layer widths rather than depths. \nBy automating the process of width selection, the approach streamlines model design and complements traditional hyperparameter search. \nExtensive experiments on both synthetic and real-world datasets demonstrate that Adaptive Width Neural Networks (AWNN) consistently match or outperform fixed-width baselines, allow efficient post-training compression via truncation, and remain robust across various hyperparameter choices and batch sizes.", "questions": "1. My understanding is that $D_l$ (the truncated width of layer l) can fluctuate, increasing or decreasing, during training. For exmaple, DoubleMoon in Figure 2. In such cases, are the weights for newly added neurons initialized from scratch each time, and are discarded weights permanently removed? Alternatively, if the width decreases and then increases again later, are previously discarded neurons and their weights reinstated, or are they reinitialized? \nI am concerned that frequent initialization and discarding of neuron weights could potentially slow down the optimization process.\n\n2. The training procedure described in Algorithm 1 appears computationally efficient. Given this, would it be practical to evaluate the method on larger datasets (e.g., ImageNet) and deeper models (e.g., ResNet)? What is the actual training cost of the proposed approach in terms of runtime and memory usage? Are there any hidden or unexpected costs that might limit scalability to large datasets and architectures?\n\n3. Regarding Table 1: does “Linear\" refer to a linear classifier with the width equal to the number of dataset classes (e.g., width 10 for MNIST and CIFAR-10)? If so, does AWNN tend to learn significantly larger widths than this baseline for these tasks?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Strong Motivation and Practical Impact:\n\nThe authors address the challenge of hyperparameter tuning, a particularly arduous and resource-intensive task for large-scale networks. Automating the search for the width hyperparameter is therefore a highly practical contribution, especially in modern architectures with billions of parameters.\n\n2. Efficient Post-Training Trade-Off:\n\nThe proposed method offers the compelling ability to control model complexity after training with zero additional cost. This property is particularly valuable for practitioners who need to deploy models with varying computational or memory constraints, as it facilitates rapid reduction in training costs across architectures of different depths and widths.\n\n3. Theoretical foundation: \n\nWhile I am not fully familiar with variational inference frameworks and cannot fully verify the technical details of the problem formulation in Section 3, the method closely follows the established framework of Nazaret and Beli (2022), and the overall design seems to have no evident problem, but I hope that other reviewers with stronger expertise in this area will thoroughly assess the technical soundness of the approach.", "weaknesses": "1. Limited Demonstration of Practicality and Scalability:\n\nAlthough the paper emphasizes practical implications, the experimental results do not fully demonstrate the method’s practical utility or scalability. Specifically, the approach is only applied to multi-layer perceptron (MLP) layers across various models; for convolutional neural networks (CNNs), it is restricted to the final MLP classifier layer. Given the paper’s claim of general applicability, it is unclear why the adaptive width mechanism has not been validated on other layer types, such as convolutional layers. Notably, there are several supernet approaches for adaptive width in CNNs such as [1][2], which would make for a valuable comparison to further highlight the novelty or limitations of this method.\n\n[1] Slimmable neural networks (ICLR 2019)\n\n[2] Alphanet: Improved training of supernets with alpha-divergence (ICML 2021)\n\n2. Scalability and Interdependence Across Layers:\n\nWhile the framework follows that of “unbounded depth neural networks” (UDN)\", the task of selecting layer widths in AWNN appears even more challenging, since truncation points must be chosen for every layer rather than just the overall depth. The paper assumes interdependence across layers but does not sufficiently address this complexity. In practice, changing the width of one layer can have significant effects on subsequent layers, and the impact of these dependencies deserves more thorough exploration and discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gq2s3v7buY", "reviewer": "Reviewer_A2U1", "summary": "The paper introduces Adaptive Width Neural Networks (AWNN), a probabilistic framework that maximizes a simple variational objective via backpropagation over a neural network’s parameters. Further a soft importance ordering is imposed on the neurons of the layer, whereby runtime dynamic truncation adaptively controls the width, and online compression can be realized naturally through regularization.", "questions": "no questions", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduces a new approach for learning netwroks of unbounded width. The paper provides a theoretical derivation for their model as well as expermients that support their claims.", "weaknesses": "The proposed method shares notable similarities with prior work on learning infinite depth, which may limit the perceived novelty. And there has already been some exploration in the direction of learning model's width.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)