Title: Adaptive Width Neural Networks
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Neural Networks, Learning the Number of Neurons, Adaptive Width Learning, Dynamic Architectures, Information Compression, Variational Inference

Abstract: For almost 70 years, researchers have primarily relied on hyper-parameter tuning to select the width of neural networks' layers. This paper challenges the status quo by introducing an easy-to-use technique to learn an unbounded width of a neural network's layer during training. The method jointly optimizes the width and the parameters of each layer via standard backpropagation. We apply the technique to a broad range of data domains such as tables, images, text, sequences, and graphs, showing how the width adapts to the task's difficulty. A by product of our width learning approach is the easy truncation of the trained network at virtually zero cost, achieving a smooth trade-off between performance and compute resources. Alternatively, one can dynamically compress the network until performances do not degrade. 
In light of recent foundation models trained on large datasets, requiring billions of parameters and where hyper-parameter tuning is unfeasible due to huge training costs, our approach introduces a viable alternative for width learning.

TL;DR: We propose a strategy to learn, during a single training, the width of neural network layers depending on the task and without imposing upper bounds.