Expressivity of Neural Networks with Random Weights and Learned Biases
Authors: Ezekiel Williams
Alexandre Payeur
Avery Hee-Woon Ryoo
Thomas Jiralerspong
Matthew G Perich
Luca Mazzucato
Guillaume Lajoie
Venue: ICLR 2025 Poster (2025)
Keywords: random neural networks
recurrent neural networks
plasticity
deep learning
neuroscience
multi-task learning
Primary area: applications to neuroscience & cognitive science

Abstract: Landmark universal function approximation results for neural networks with trained weights and biases provided the impetus for the ubiquitous use of neural networks as learning models in neuroscience and Artificial Intelligence (AI). Recent work has extended these results to networks in which a smaller subset of weights (e.g., output weights) are tuned, leaving other parameters random. However, it remains an open question whether universal approximation holds when only biases are learned, despite evidence from neuroscience and AI that biases significantly shape neural responses. The current paper answers this question. We provide theoretical and numerical evidence demonstrating that feedforward neural networks with fixed random weights can approximate any continuous function on compact sets. We further show an analogous result for the approximation of dynamical systems with recurrent neural networks. Our findings are relevant to neuroscience, where they demonstrate the potential for behaviourally relevant changes in dynamics without modifying synaptic weights, as well as for AI, where they shed light on recent fine-tuning methods for large language models, like bias and prefix-based approaches.

Published as a conference paper at ICLR 2025
EXPRESSIVITY OF NEURAL NETWORKS WITH RANDOM WEIGHTS AND LEARNED BIASES
Ezekiel Williams
Alexandre Payeur Mathematics and Statistics
Mathematics and Statistics
Université de Montréal ezekiel wi ams@mi a. quebec
Université de Montréal
a wexxandee.payeue.ee 1a. quebec
Avery Hee- Woon Ryoo
Thomas Jiralerspong
Computer Science Université de Montréal hee-woon. zyoo@mila, quebec
Computer Science
Université de Montréal thomas. jiralerspongemila.co quebec
Matthew Perich
Luca Mazzucato"
Neuroscience
Biology, Physics, and Mathematics
Université de Montréal
University of Oregon
matthew. cmmnntreaal
Lmazzuca@uozegon.. edu
Guillaume Lajoie
Mathematics and Statistics
Université de Montréal
guillaule lllume. la lajoieemille la. quebec
ABSTRACT
Landmark universal function approximation results for neural networks with trained weights and biases provided the impetus for the ubiquitous use of neu- ral networks learning models in neuroscience and Artificial Intelligence (AI). Recent work has extended these results to networks which a smaller subset weights (e.g., output weights) are tuned, leaving other parameters random. How- ever, remains an open question whether universal approximation holds when only biases are learned, despite evidence from neuroscience and Al that biases significantly shape neural responses. The current paper answers this question. We provide theoretical and numerical evidence demonstrating that feedforward neu- ral networks with fixed random weights can approximate any continuous func- tion compact sets. We further show analogous result for the approximation of dynamical systems with recurrent neural networks. Our findings are relevant neuroscience, where they demonstrate the potential for behaviourally relevant changes in dynamics without modifying synaptic weights, as well as for AI, where they shed light on recent fine-tuning methods for large language models, like bias and prefix baaedd approaches.
INTRODUCTION
The universal approximation theorems Hornik (1989); Funahashi (1989); Hornik (1991) of the late 1900s highlighted the expressivity of neural network models, i.c. their ability to approxi- mate or express broad class of functions through the tuning of weights and biases, heralding the central role that neural networks play in Machine Learning (ML) and neuroscience today. Since these Foundational studies, a rich literature has explored the limits of this expressivity by finding smaller parameter subsets that, when optimized, can still support the approximation of wide classes of functions or dynamics. Prior work has explored the approximation capabilities of Feedforward
co-senior authors

---

Published as a conference paper at ICLR 2025
Neural Networks (FNNs) and Recurrent Neural Networks (RNNs) where only the output weights
trained Rosenblatt et al. [962); Rahimi & Recht (2008): Ding et al. (2014); Neufeld & Schmocker
(2023 Jaeger & Haas (2004): Sussillo & Abbott 2009); Gonon al. (2023) Hart et al. (2021). and
deep FNNs where only subsets of parameters Rosenfeld & Tsotsos (2019). normalization parame-
ters BBurkhol
(2023); Giannou et al. (2023). binary masks-either over units or parameters-are
trained Malach Malach et al. (2200).
In this work, we study the expressivity of neural networks where only biases-which can be inter- preted as constant inputs to neural units-are learned. While this may seem like an odd pursuit, bias-related learning is central to several active areas of research. In AI, modern sequence models such as transformers can have their outputs reshaped based on examples or instructions presented in an unchanging prefix sequence. No changes of weights are performed in this case, only distinct inputs carrying information about the context are passed to pre-trained but fixed neural network components Li & Liang (202T): Marvin et al. (2023); Garg et al.) (2022); | Von Oswald et al. (2023). Even more closely related, training only biases is strategy that has been used recently Zaken et al. (20TT) for more efficient fine-tuning. In neuroscience, there is growing evidence that animals can mangre projections from higher cortical areas neuromodulatory nuclei- in order to rapidly and flexibly adapt network dynamics to multiple tasks Mazzucato et al. (2019); Ogawa et al. (2023); ; Perich et al. 2); Remington al. (20T8). These tonic inputs mediating the effects of projections from other brain areas to the given, local, circuit are typically modelled as biases (see Supplementary Table [T] Consequently, input-based learning is a conceptually crucial yet poorly understood component of both modern AI systems as well as of brain functions.
leverage inputs-via
Quantifying the expressivity of bias learning would show the degree to which the brain or neural networks can rely on the adaptation of bias-related parameters to structure their dynamics for new tasks, thus providing critical theoretical grounding to this growing literature. If tuning the biases of a neural network will only span a reduced set of functions, or output dynamics, then this would solidify the role of synaptic plasticity as the critical component in biological and artificial learning- Conversely, if one can express arbitrary dynamics solely by changing biases, this would motivate deeper investigation of when and how non-synaptic mechanisms might shoulder some of the effort of learning. In this paper, we take first step towards characterizing the expressivity of bias learning by studying the arguably worst-case scenario of a neural network with unstructured weights. In regime where all weight parameters are randomly initialized and frozen, and only hidden-layer biases are optimized-whie we term bias learning-wa we give theoretical guarantees demonstrating that:
1. bias-learning FNNs with wide hidden layers are universal function approximators with high probability;
2. bias-Icarning RNNs with wide hidden layers can arbitrarily approximate finite-time trajec- tories from smooth dynamical systems with high probability.
We further provide empirical support for, and a deeper interrogation of, these results with numerical experiments exploring multi-task learning, motor-control, and dynamical system forecasting.
1.1 RELATED WORKS
Machine Learning. Many efforts have explored neural networks that are trained to quickly meta- learn new tasks via dynamics in activation space alone, without any adaptation of weights (see Feldkamp et al. 1997); Klos et al. (2020); Cotter & Conwell ( [1990] 1991); Hochreiter et al. (2001); Subramoney et al. (2024). Like our work, this research proposes a mechanism by which network might "learn" any new task without changing weights. However, prior work differs from the current study in that it requires an initial meta-training of all parameters in a network, weights included, before operating in the "fast learning" regime where network variables maintain context information that allow the networks to rapidly adapt to new tasks. In some cases, these context variables can be thought of as biases Cotter& & Conwell (1991).
From a mathematical perspective, our work is closely related to masking, particularly the Strong Lottery Ticket Hypothesis (SLTH). This hypothesis conjectures that a desired network parameteri- network Ramanujan
zation could be found as a sub-network in a larger, appropriately-iittaliiin et al. 2020). SLTH is typically formulated with respect to weights, i.e., a subnetwork is defined
2

---

Published as a conference paper at ICLR 2025
by deleting weights from the full network. However, few studies have investigated SLTH where subnetworks are constructed by deleting units Malach et al. (2020). which we term SLTH over units. While study is different for its focus on function approximation via bias optimization, rather than finding "lottery ticket" subnetworks, key step in our analytic derivations relies on masking in fashion analogous proofs of SLTH over units. Thus, our work also provides two results that may be of interest to the SLTH theory: a novel proof of SLTH over units in single-layer FNNs, complementing the work of Malach et al. (2020) (see Connections with Malach et al. 2020 in BB.2 for details), and first proof of SLTH for RNNs (see Section $2 for more details). Flavours of the lottery ticket hypothesis for RNNs have been explored empirically Yu et al. (2019): Garcfa-Arias et al. 2 Schlake et al. (0222) but we have not encountered its proof, neither over weights nor units, the literature.
Neuroscience. Changes in input biases, which mediate a change in the input-output transfer func- tions of neurons, can explain the context-dependent effects of expectation Mazzucato et al.] (2019), movements Wyrick & Mazzucato 2021 and arousal Papadopools et al. 2024) on sensory process- ing across modalities and brain areas. Some of these effects occur by plasticity
changes in amygdalar projections to cortex, induced by associative learning Vincis & Fontanini (2016); Haley jet al.] (2020). Changes in neural firing threshold and network inputs, similar to bias modulations, were shown to shape network dynamics: threshold heterogeneity can improve network capacity Gast et al. (2024 2023) and reconfigure circuit dynamics on fast timescales |Perich et al. (2018); Reminnton et al. (2018). A recent study showed that, in RNNs trained perform neuroscience
tasks, learning biases via language model embedding leads to zero-shot generalization to new tasks Riveland & Pouget (2024). Within the reservoir computing approach modelling in neuroscience, where recurrent weights are random and fixed, bias modulations can toggle between multiple phases (including fixed point, chaos, and multistable regimes) and, strikingly, enable RNN multi multi-tasking in the absence of any parameter optimization Ogawa et al. (2023). While slightly different than bias parameters, a repertoire of dynamical motifs can also be generated in RNN reservoirs with dynamic feedback loops Logiaco et al. (2021) and by modulating inputs in pre- praitrained networks Driscoll et al. (2024).
THEORY RESULTS
2.1
FEEDFORWARD NEURAL NETWORKS
This section studies the single-layer FNN, whose output is given by:
(z, -)= = Σ A b,),
(1)
==1
with R E Raxd b RR, and a = Note that here, and throughout the paper, adopt the notation X,, and X., to denote the 7th row and jth column, respectively, of matrix X. We shall investigate the approximation properties of this neural network when all the weights in B and A are fixed and sampled uniformly from the n(I d)-dimensional centered hypercube, where the half-edges are of length+ y and only b is tuned. We begin by outlining the activation function assumptions necessary for our theoretical results.
Definition 1. The function & is a suitable activation if. when employed in the neural network of Eq. 1 it allows for universal approximation of the following kind: for any continuous h U RR
and any > 0, En E N and parameters 0 s.t. supzeu [[h(z) - (n, 0)ll I c, where U C Rdd is compact and ||
the 1-norm and will be throughout the paper.
From the universal approximation theorems of 1993 Leshno et al. (199)) Hornik 1993), a sufficient condition for é to be a suitable activation that it non-nonnoyyne polynomilll In this paper we conceptualize universal approximation as the approximation of continuous functions on compact sets with respect to an 0 functional norm, but we remark that the literature has also suudied other conditions on h (for example measurability) and other forms of convergence. For review of the literature, Pinkus [199)).
Definition 2. A suitable activation & is referred to parameter bounding activation if it allows for universal approximation even when each individual parameter, e.g. an element of a weight matrix or bias vector, is bounded by 7.
3

---

Published as a conference paper at ICLR 2025
Proposition 1. The ReLU and the Heaviside step function y-pparmetter bounding activations for any y > 0.
The proof is in Appendix $B.2 A key subtlety of parameter-bounding is that it is bound on individ- ual, scalar, parameters. Thus, as network grows in width the bias vector and weight matrix norms will still grow accordingly. This may be important, as research suggests band-limited parameters cannot universally approximate, at least for certain activation types Li et al. (2023). We leave it to future work to determine which other activations are parameter-bounding.
We make one final definition:
Definition 3. If 6 is "1-parameter bounding activation, is continuous, and 3r E R such that for x < ((x) = 0, then we say that & is a 7-biis--earniig activation.
Obviously, the ReLU bias-learning activation. We leave the study of discontinuous functions like the Heaviside to future work. We conclude with the main result of this section. Define PR to be a uniform distribution on [-R, + +A], where 7 < R oc.
Theorem 1. Assume that o is --bias-learning and, for compact U C Rd, h U - RK' is continuous. Then, for any degree of accuracy > O and probability of error 5 E (0, 1), there exists a hidden- layer width m N and bias vector b E Ih such that, with a probability of -8 a neural network given by Eq with each individual weight sampled from pn approximates h with error less than c.
Corollary 1. Assume that d = I, i.e. the input and output spaces of the network have the same dimension. Then the results of [ hold for hingle-hidden-llyer ResNets.
We provide intuition about the proof of Theorem 1 and its Corollary, whose details can be found in the Appendix (also see F. visual proof intuition). According to the universal approximation theorem, given a continuous function, we can find a one-hidden-llyer network, Ni, that is close to that function in the L norm on the (compact) space of its inputs. If Wi has been constructed using 7-parameter bounding activation functions, then we know that each parameter will be on the interval 7- Next, we construct a second network, .2, to approximate Ni by randomly sampling each its parameters, weight or bias, from pn- For N22 to approximate N₁. each parameter of N2 should fall within tiny window of an analogous parameter in Mi. This window must have half-length less than c to yield the desired error bound. Without loss of generality, we can assume < R- 7. Then, if we sample parameters uniformly on [- Rt.. there will be non-zero probability that a given parameter of N/2 will end up within the tiny e-window centered at corresponding parameter value in N1; because < R y we know that the e-window won't t stretch outside the distribution support. If we randomly sample a very large number of units to construct the hidden layer of N2 the probability of finding subnetwork of A2 corresponding Mi can be made arbitrarily close to 1. If the activation function is bias-learning we can use biases to pick out this subnetwork by setting them appropriately smaller than the threshold given in Definition 3 We remark that this proof estimates exceedingly massive hidden-layer widths that, based on our numerical results, over-estimate the scaling bias learning by orders of magnitude (see Remark 2 on Lemma We thus view this proof not as a statement about scaling but as statement of existence: for some sufficiently large but finite layer width, one can approximate the desired function with bias learning.
Proof Intuition:
2.2 RECURRENT NEURAL NETWORKS
Here, we study discrete-time RNN given by:
Tt = 00 + Bo(Wr,11 + Br,-1 + b). =Cr:
(2)
where re E RM for all 0 S T for some T E N, a and 3 control the time scale of the dynamics, W mmx C E Rixm and B and b are in the previous section. The parameters are now 0 = (W, B, b). The time-dependent input va belongs to a compact subset U. C Idd for all I. one gets the standard vanilla RNN formulation; alternatively, a and can be set to approximate continuous-ting dynamics using Euler's method.
Note that when a = 0,
We will approximate the following class of dynamical systems by learning only biases:
1 Fx), yt Qze,
zo Uz:
(3)
4

---

Published as a conference paper at ICLR 2025
where t and 2 as defined for the RNN, F U2 x U2= R is continuous, and Q R Because we build from the classic universal approximation results, we must be working with func- tions operating on compact sets. To guarantee that this will be the case must make several more assumptions about the dynamical system. First, Uz C RR is assumed to be a compact invariant set of the dynamical system: if the system is in U__ it remains there for all and for all inputs in Un. Second, we assume that the dynamical system is well-defined on slightly larger compact set U2 x Uz. where Ú= = [zo- + Co 50 E U2, |coo|| < c} for some c > O, with Ü2= 5 U2.
To generalize the approximation in 1 norm in our analysis of FNNS dynamical systems we consider an infinity norm over finite trajectories: SUPaoEUs, Σ (2,, de (ro, xx) where X = [zg..... . ro = ro(()) is a continuous map from U₂ into the hidden state space of the RNN, and Ux is the t-times product space of Un. Letting Pn be defined as in Section 2.1 the main result of this section is:
Theorem 2. Consider the RNN in 2 with & abbbsss-laamin activation, and imput, oulput, and recurrent weight parameters for each hidden unit sampled from pr. We can find a hidden-layer width, m. a bias vector, and a continuous hidden-state initial condition map To : U, + I such that, with a probability that is arbitrarily close to 1, the RNN approximates the dynamical system defined in 3 below any positive, non-zero, error in the inifinity norm over trajectories.
Proof Intuition: We provide high-level description here while the detailed proof and a schematic (Fig. E.1 can be found in the Appendix. As in Theorem the proof proceeds in two steps. First, the dynamical system is approximated by an RNN, R1, using universal approximation theory for RNNs (see e.g.. Schäfer & Zimmermann (2006)). R2 is then approximated by a much wider, random RNN, R2, with parameters sampled from pR. Analogous to Theorem []] we show that one can find a sub-network of hidden units in R2 that approximates R2 for very large hidden widths of R2..
3 NUMERICAL RESULTS
3.1 MULTI-TASK LEARNING WITH BIAS-LEARNED FNNS
We first validated the theory by checking whether a single-hidden-laye bias-learned FNN could perform classification on the Fashion MNIST dataset Deng (012)) increasingly well as its hidden layer was widened. We compared fully-trained networks, matching the number of trained parame- ters, with bias-learned networks where the frozen weights were sampled from a uniform distribution on [-0.1, 0.1] zero mean Gaussian with standard deviation 1 where d is the input dimension. The networks successfully learned the task and validation error decreased with the number of trained parameters (Fig. [T) which, for bias-learned networks, is equal to the number of hidden units. The largest gains in performance occurred before 5 x 10 units, with bias-learning achieving compara- ble, but slightly worse, performance compared to fully-trained models. We speculate that this gap in performance for large network sizes is due either to current deep learning training conventions being optimized for weight training, or to bias learning requiring even larger hidden layer sizes to mathh fully-traines performance. Interestingly, for very low parameter counts bias-lcarning out- performed fully trained networks (more visible with log-log axis scaling F.2 A). Because weights scale quadratically with layer width, a fully-trained network will have a smaller width than a bias- learned network with the same number of trained parameters (see E.2 B for the error scaling plot with layer width on the axiis).
va
Intuitively, bias learning should allow single random set of weights be used learn multi- ple tasks by simply optimizing task-specific bias vectors. We confirmed this by training a single- hidden-layer FNN with 3.2 x 10 hidden units on 7 different tasks: MNIST Deng (2012). KMM Clanwwl et al. (2018). Fashion MNIST Xiao et al. (217)). Ethiopic-MNIST, Vai- MNIST. and
Aomanya-MNIST from Afro-MNIST Wu et al. (2020). and Kannada-MNIST Prabhu (2019). All
tasks involved classifying 28 x 28 grayscale images into 10 classes. The random weights were fixed across tasks while different biases were learned. We compared bias learning against a fully-trained neural network with the same size and architecture (Fig. [B). We found that the bias-only network achieved similar performance to the fully- fully-trained network on most tasks (only significantly worse on KMNIST). An important difference here is that the networks had matched size and architecture, so that the number of trainable parameters in the bias-only network (3.2 x 10 parameters) was several orders of magnitude smaller than in the fully-trained case (< 2.5 x 107 parameters). Notably,
5

---

Published as a conference paper at ICLR 2025
Velietiio number of trwinable paramattes Incresses
Accuracy biaw-ril and fulystraieed networbs with the reservol
Uniteen
Faasdeett Accumny
Aul Thlnes
Rhal Na Mdxtion Enerr
4x12-1
siss CousSAin
4x
6-4-
2x422
9953
10000
19300
70000
10000
Oenarge Prieppi
VIAST Detaset
Derrade
ST
Teetine
kumber trainable
Units
Etriopic
Normallzed task varlance
Fashion
0..8
MNIST
0.4
Comanya
Vel
9
10
11
12
13
24
15:36
17
10
19
20
3
0.55
0.54
0.43
Correlationn Between
Figure 1: A. Validation accuracy on fashion MNIST vs. number of trained parameters for fully- trained (blue), bias-learned with uniformly distributed weights (light orange), and bias-leamedd with Gaussian weights (dark orange) networks. B. Validation accuracy tasks for learee (orange) and fully-trained (blue) networks. Errors for random seeds are barely visible the shaded regions in A, and omitted in B because the standard errors are of order 10 10-3 C. Top: K-mean clustering of Task Variance (TV) reveals task-selective clusters (see Fig E.3 for fully-trained network selectivity). Bottom: Spearman correlation between TV and bias
multiple image classification
vectors (mean across neurons in each cluster).
different set of biases was learned for each task. We conclude that bias-only learning in FNNs could be a viable avenue to perform multi-tasking with randomly initialized and fixed weights, but that it requires much wider hidden layer than fully trained networks. Lastly, we note that the networks of Fig. [ were trained with uniformly initialized weights, but that one can achieve similar, or even better performance with different weight initializations (see Fig. E.2C).
Next, we investigated the task-specific responses of hidden units by estimating single-unit Task Variances (TV) Yang et al. (2019), defined as the variance of a hidden unit activation across the test set for each task. The TV provides measure of the extent that given hidden unit contributes to the given task: a unit with high TV in one task and low TV for all others is seen as selective for its high-TV task. We clustered the hidden-unit TVs using K-means clustering (K chosen by cross- validation) on the vectors of TVs for each unit and found that distinct clusters of units emerged (Fig. [C) Some units reflected strong task selectivity (ex: cluster 3 for KMNIST and cluster 10 for Osmanya). a)). Others responded to many, or all, tasks (ex: clusters I and 8), although a smaller fraction of clusters exhibited such non-selective activation patterns. Overall, we conclude that multi-task bias learning leads to the emergence of task-specific organization. We note, however, that task selectivity does not necessarily mean task utility: for example, a neuron could have high variance for single task but that variance could be picking up noise and thus not functionally useful. We leave deeper investigation the functional significance of task-seleytivity to future work.
Finally, we explored the relationship between the bias of a hidden unit and its TV. If the neural networks are using biases to shut-off units, analogous to the intuition in our theory (Section 2I then the units that do not actively participate in task should be quiet due to low bias value learned during training on that particular task. In other words, this intuition would suggest that units should exhibit a correlation between bias and TV, especially in task-specific clusters. In our experiments, all clusters did exhibit the statistical trend of positive correlation between bias and TV, although to varying degree across clusters (see numbers at the bottom of Fig. TC..

---

Published as a conference paper at ICLR 2025
A
B
1:53
D:054
0.019
300
Learning
1255
Probabiitt of Matthh
0.9
2510
Mark Learning
Null Model
1.02-
08
Most Mask
Counts
200
0.75
Blas
Mask
0.50
0.8
102
0.5-
0.5
86-
10
16
20
25
0.
0.2
0.4
0.5
02
04
00
1.0
12
Epochs
Task Variance Thrwahold
Unit Variance
Figure 2: Comparing bias and mask learning on same weights. A. Learning curves for bias (orange) and mask (black) learning on MNIST. Inset: bias learning achieved roughly 1% higher test accuracy over mask learning (0.934 + 0.001SD bias vs. 0.919 + 0.002SD mask). B. Probability (g-axis) of the same unit being ON in both the bias-learning and mask-learning networks (orange line). A unit is 'ON' in mask learning if it is not masked out, and bias learning if it has task variance above given threshold (x-axis). Also shown is the probability of a unit being ON in two different training runs for mask meakkleanning (black dashed line), and a null model giving the expected overlap if the probability of a unit being ON in the bias-trained network is independent of whether it is ON in the mask network (see Appendix $C.2 for more details) C. Histograms of hidden unit variances, calculated over 10* test set MNIST samples, for bias trained (orange) and trainer (black). Unit variances below 0.1 are not shown. All curves, and histograms, are means, with shaded regions being 1SD over training runs
3.2 RELATIONSHIP BETWEEN BIAS LEARNING AND MASK LEARNING IN FNNS
As our theory shows that bias learning networks can universally approximate simply by turning units off, we wished to test whether bias learning performs similarly to learning masks, and to what extent solutions learned by these approaches are different from each other. We compared training mask to bias learning on networks with the same random input/output weight matrices. For mask- training, we approximated binary masks using 'soft' sigmoid masks with learnd gain parameters (see Methods). The approximation of a discontinuity with a differentiable function was done to allow optimization with gradient descent, a strategy with a history of use in ML ML, al. (2016) and computational neuroscience Zenke & Ganguli| (2018). The sigmoid was steepened over the course of training approximate the binary mask that was used at test time. We compared masks learned in this fashion with learned biases on singlehhideen-ee -ayer ReLU networks with 10 units. We observed a trend of bias-training slightly improving upon mask-training (Fig which was expected given that biases can be tuned over continuous range of values, including 0 and 1, while masks can only take or 1. Further research is needed determine if this trend is reliable across datasets and different network parametrizations, and whether there might be scenarios where one style of learning works better or worse.
Next, we compared the solutions found via bias and mask learning on the same set of randomly initialized weights. We calculated the variance of each hidden unit across 10 MNIST test images, in both the bias and mask-trained paradigms, tigate whether the same units were active during the task regardless of training style, we looked at which units were 'ON' in mask versus bias-trained networks. For mask learning unit was consid- ered ON if its mask was 1; for bias learning a unit was ON if its task variance was above a given threshold. We observed that neurons with higher task variances contributed more solving the task (Fig. E.44).. For a range of thresholds, we calculated the probability that a given unit was ON for both mask and bias learning (Fig We found that, for low thresholds, this match probabil- ity was intermediate between chance (grey line) and the high probability of match between two mask-learned training runs on the same set of weights (black dashed line). We further noted that, on average, mask learning used 4527 + 44 'ON' units (in this section all reported values are mSD over 5 samples) to solve the task. For bias learning, when the task variance threshold was moved from 0 to 0.1 test accuracy dropped about 1% and the number of ON units went from 10d (all units) to 6226+ 47. Thus mask learning. found a sparser solution than bias learning. To further investigate the differences and similarities between unmasked units, plotted the histograms of unit variances above the 0.1 threshold, and observed that bias learning used higher variances to accomplish the task (Fig 2C). Finally, we found moderate correlation between the task variances of units that were
measure of hidden layer representation. To inves-
7

---

Published as a conference paper at ICLR 2025
A
B
10
Recurrntt weights
Jacobian
Output
0.0
network
E
-1.0
Time
D
trained
Bias learning
Fully trained
Log training performance
0.5
Outputt
blss
Bas leamiing
-0.5
Serget
100
150
1.0 Gain recurreni inil.
0.9
0.8
1.0
Time
Gain recurrent inil.
Figure 3: Learning autonomous dynamical systems. A. Cosine generated by bias-learning RNN (dashed orange) and its target (solid black). B. Eigenvalue spectra for the recurrent weights (left) and the Jacobian at the start of training (right, grey squares) and mid-training (right, orange circles), when the network produced decaying oscillation with period 23.75, close to the target period of 25. Neural activity then approached a fixed point with respect to which the Jacobian was computed. C. Van der Pol oscillator (target in solid black) generated by the bias-learning RNN for recurrent gain of (dashed orange; see panel D) and gain of 0.9 (dashed dark orange). Output represents the oscillator's position, rescaled [-1, IJ. D. (Left) Sensitivity to distribution of recurrent weights. The fully-trained and bias-learning networks had the same number of learnable parameters. Initial recurrent weight matrix had elements sampled from (g/ Wm) 1). where g is the gain (Gain recurrent init.). Error bars denote SEM for 10. (Right) Schematics of the fully-trained (top) and bias-learning (bottom) autonomous RNNs. Colored links denote trained weights.
ON for both mask and bias learning (0.5030 + 0.0087) (Fig E.4C). C. In summary, we observe that, relative to mask learning, bias learning finds a different, but overlapping, solution MNIST.
3.3 BIAS LEARNING AUTONOMOUS DYNAMICAL SYSTEMS WITH RNNS
We studied the expressivity of bias learning RNNs trained to generate linear and nonlinear dynam- ical systems autonomously (i.e., with 0 in Eq. ( We found that RNNs with fixed and random Gaussian weights and trained biases were able to generate a simple cosine function (Fig. 3A).. We then elucidated the mechanism underlying RNN bias learning by comparing the Jacobian matrix after learning with the random recurrent weight matrix (which was held fixed during learning). We found that although the random weight matrix maintained a fixed and eircular eigenvalue distribution (Fig. BB, left), learning the biases shaped the Jacobian matrix to develop complex conjugate pairs of large eigenvalues underlying the oscillations (Fig. 3B, right). Therefore, bias learning strongly relies on the ability to shape the "effective connectivity matrix", i.e. the Jacobian, which involves the derivative of the activation and the recurrent weight matrix.
We next investigated whether bias learning relied on the statistics of the fixed recurrent weights. In light of Fig. 33, we thus hypothesized that bias learning would be affected by changes in the weight distribution, because bias learning can only control the derivative. We initialized an i.i.d. Gaussian distributed weight matrix Wiy ~ (O, 9 /N). where g is referred to as its 'gain'. We then trained bias-learning networks to generate van der Pol oscillator (Fig. 3C). We found that bias learning required a large enough gain (at least = 1) and failed for < (Fig BD). This was not purely due to a restricted dynamic range for the network activity since the network to reproduce the first peak of the oscillator and then flatlined (Fig. BC). In contrast, fully-trained networks with the same number of training parameters (Fig. 3D) not sensitive to the value of
able
8

---

Published as a conference paper at ICLR 2025
A 2.00
B
1.0
2-2-
m
595
Barooy
6913
Easkared outaieed 11 careaddd oet sarsaredd
294
14:1 mannns
Dully caardd 0905
systems. A. Validation R² vs. number of trainable Figure 4: Learning non-autonomous systmss. model parameters for fully-trained (blue) and bias lea-nee (orange) RNNs. Training RNNs with bias learning became unstable below a network width of 64. B. Predictions from the fully-trained and bias-learned networks (both with hidden layer width of 1024) trajectory of the Lorenz system unseen during training. Standard deviation error bars were computed over 5 seeds, but are not visible due their small magnitudes. C. Predictions of both the fully-trained and bias-learned networks diverge from the ground truth signal when one starts feeding back their own outputs as their inputs, in place of the ground-truth time-series sustanndt, starting from the grey line).
the gain at initialization. This result thus highlights that, when the hidden- layer size is fixed, the initial distribution of weights limits the capability of bias-learning networks.
3.4 BIAS LEARNING XON-AUTONOMOUS DYNAMICAL SYSTEMS WITH RNNS
To further test bias learning in RNNs, we trained RNN to predict future time-steps of a partially ob- served dynamical system, namely a single dimension of the Lorenz attractor (see Appendix $C.44or details). As in the autonomous DS case, only the biases of the input layer were trained and the weights were random and frozen. However, here the network received the observed dimension of the Lorenz system an input. Given the observed dimension at time-poine poee-point t, and its value at previ- ous time-steps encoded the RNN's hidden state, the objective the task was to predict the future value t + r, where r 27 chosen to be the half-width at the half-max of the auto-correlation of the observed dimension of the Lorenz system.
The performance of the bias-learned RNNs scaled, as function of trainable model parameters, in a qualitatively similar fashion to the bias-trained FNNs [Fig.444, see Fig. E.2D for scaling as func- tion of layer width). Both the fully-trained and sufficiently wide bias-learned networks accurately predicted future points of the Lorenz system, evidenced by a consistent R² metric of > 0.99 (n=5) achieved by networks with a hidden-layer width of 1024 on a window of the Lorenz time-series held out during training (Fig. 4B). However, when the networks were fed their own previous predictions as input, in place of the ground-truth time series, in an autoregressive, 'self-sustained' fashion, their prediction accuracy decreased, demonstrating the damaging effect of small compounding deviations propagated through time (Fig.
3.5 BIAS-LEARNED MOTOR CONTROL WITH RNNS
Finally, we tested whether bias learning could solve a center-out reaching task, paradigm routinely used to study motor control in human and non-human primates|Ashe & Georgopoulos 1994);: [Shad- mehr & Mussa-Ivaldi (194)). Starting from the center of the workspace, the subject must move the selected end effector (e.g., their right hand) to reach several peripheral targets placed equidistantly on circle. We modelled this task using an RNN with random weights and learned biases, where linear readouts controlled point-mass arm (a unit mass modeling the arm's 's behavior; see Methods). The task objective was to reach the peripheral targets in second, with near-zero velocity and force at the end of movement, which were imposed using regularization terms in the loss function. A 1,024-units network required approximately 10a training epochs-where one epoch involved pre- senting all targets once-to solve the task Frig.5AA (Note that a parameter-matched. fully-trained RNN can also solve this task, in ~ 10 epochs (Fig. E.5).) Importantly, a single set of biases was used reach all targets for each trained network. The resulting trajectories successfully reached the targets (Fig. 5B, dark curves and black targets), and exhibited the characteristic bell-shaped ve- locity profile Harris & Wolpert (1998) (Fig. 5C. top). Crucially, the trained network generalized

---

Published as a conference paper at ICLR 2025
A
C
Training
seed 0
10
seed
0.1
seed 2
Lose
13"
Speed (mis)
Generalization
10"
0.1
10
0.0
0
2 Epoch
0.0
0.5
1.0
x10'
Time (8)
Figure 5: Center-out reaching task. A. Training loss for 3 network initializations. B. Trajectories for the trained (black) and tested (grey) targets. C. Speeds ((i2 + 1/2)1/3 for the trained (top) and tested (bottom) targets (mean + SD across targets).
decently to new targets on the circle (Fig- BB light curves and grey targets; Fig. 5C. bottom). To achieve this, the network had to produce both acceleration and deceleration when given information about the Cartesian position of target never seen in the training period. These results highlight the flexibility of bias learning in generating diverse open-loop controls.
4 DISCUSSION
In this paper, we presented theoretical results demonstrating that FNNs and RNNs with fixed ran- dom weights but learnable biases can approximate arbitrary functions with high probability. We showcased the expressivity of bias-learned networks in multi-task, times-series, and motor control tasks, and interrogated their learned representations by analyzing task selectivity and comparing with mask-learning in FNNs, and by analyzing eigenvalues in RNNs.
We underline four limitations of our study that might inspire future research. First, the convergence results for dynamical systems were only for finite-time trajectories. One could overcome this limi- tation by studying convergence in stationary distribution. Second, a potential confounding factor in our comparison of bias and mask learning is that the latter approach used a learning schedule in the steepness parameter for the soft masks. It is possible that the altered learning dynamics due to this scheduling contributed to mask and bias learning finding different solutions. Addressing this con- found is an important direction for future work. Third, better grasp of bias learming scaling-how layer width or parameter count increases as function of performance-is needed. Our theory does not shed light here as it significantly overestimates how bias-learned network should scale (Re- mark 2 on Lemma 33 in Appendix). Some insight can be gleaned by noting that, with bias-learning activations, tuned biases can express any mask-learned solution. Thus, results showing that mask learning layer widths scale polynomially in the inverse error and the size of performance-matle matched random feature model (see Malach et al. (2020) Theorem 3.2) represent worst-case scaling for bias learning. Whether bias learning scales better than mask learning could be a good starting point for addressing scaling. Fourth, while our results appeared to generalize beyond lne-layer networks (Fig. E.2 E), a more detailed study of bias-learned deep FNNs is left to future work.
We lastly highlight the need for greater biological detail in future studies of bias learning. Past experimental Ferguson & Cardin (202)) and theoretical work Wyrick & Mazzucato (2021): Ogawa al. (2023) showed that neural mechanisms modulating biases, like firing threshold tonic inputs, may effect other neuronal properties, like neuron input-output gain. As our proofs rely on masking, they demonstrate universal approximation not just for bias learning but for any learned mechanism that can mask neurons. Exploring paradigms where gain (Stroud et al. (2118) and biases are learned in concert could be an interesting direction to exploit this. Finally, the observed distribution of synaptic weights in the brain is not uniform but for bias learning models with more structured weight initializations. We hypothesize that structure in the weights intermediate between fully random and fully learned might yield an optimal combination of performance and training efficiency. If such structure improves hidden-layer scaling, this could enable bias learning support temporal credit assignment algorithms that struggle in weight space due to the 22 scaling of synapses. A fascinating alternative is that the same number of parameters could achieve given task performance, regardless of whether they weights biases.
et al. 2005), highlighting the need
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
E.W. was supported by an NSERC CGS D scholarship, and wishes to thank the other members of the Lajoie lab for support and for helpful discussions. T.J. was supported by an NSERC CGS M scholarship. M.G.P. was supported by grant the Fonds de recherche du Québec Santé (chercheurs- boursiers en intelligence artificielle). L.M. was partially supported by National Institutes of Health grants ROINSI 18461. RO1MHI27375 and R01DA055439 and National Science Foundation CA- REER Award 2238247. GL acknowledges CIFAR and Canada chair program.
REFERENCES
James Ashe and Apostolos Georgopoulos. Movement Parameters and Neural Activity in Motor Cortex and Area 5. Cerebral Cortex, 466):390-600, November 1994, ISSN 1047-3211. doi: 10. 10.1093/cercor44..m 6.5.99. URL TRLLhttps: https///aaaammiccccco oup httpp.ccc//ee.cco/attcllla/////. 590/390434
Publisher: Oxford Academic.
Nicolas Brunel and Xiao-Jing Wang. Effects of neuromodulation in a cortical network model of ob- ject working memory dominated by recurrent inhibition. Journal of computational neuroxcience, 11:63-85, 2001.
Rebekka Burkholz. Batch normalization is sufficient for universal function approximation cnns. In The Twelfth International Conference on Learning Representations, 2023.
Tarin Clanuwat, Mikel Bober- Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv preprint arXiv:1812.01713: 2018.
N.E. Cotter and P.R. Conwell. wixghtweight networks can learn In 1990 IJNN International Joint Conference on Neural Networks, pp. 553-559 vol.3, June 1990. doi: 10.1109/UCNN..1990 137898.
Neil E Cotter and Peter Conwell. Learning algorithms and fixed dynamics. In IJCNN-91-Seatil International Joint Conference Neural Networks, volume 1, pp. 799-809 IEEE, 1991.
Li Deng. The mnist database of handwritten digit images for machine learning research. 1EEE Signal Processing Magazine, 29(6):141-142, 2012.
Shifei Ding, Xinzheng Xu, and Ru Nie. Extreme learning machine and its applications. Neural Computing and Applications, 25:549-5566 2014.
Laura N Driscoll, Krishna Shenoy, and David Sussillo. Flexible multitask computation in recurrent networks utilizes shared dynamical motifs. Nature Neuroscience, 27(7):1349-138 2024.
L. A. Feldkamp, Puskorius, and P. C. Moore. Adaptive behavior from fixed weight networks. Information Sciences, 98(1):217-235, May 1997. ISSN 0020-0255. doi: 10. 1016500220-0255599000112222 URL
articcleep11/500220555558
62
Katie A Ferguson and Jessica A Cardin. Mechanisms underlying gain modulation in the cortex. Nature Reviews Neuroscience, 21(2):80-92) 2020.
Barbara Feulner, Matthew G Perich, Raeed H Chowdhury, Lee E Miller, Juan A Gallego, and Clau- dia Clopath. Small, correlated changes in synaptic connectivity may facilitate rapid motor learn- ing. Nature communications, 13(1):5163, 2022.
Ken-Ichi Funahashi. On the approximate realization of continuous mappings by neural networks. Neural networks, 2(3):183-192) 192, 1989.
Ángel López García-Arias, Masanori Hashimoto, Masato Motomura, and Jaehoon Yu. Hidden- fold networks: Random recurrent residuals using sparse supermasks. arXiv preprint 111.v:2/11.12330. 2021.

---

Published as a conference paper at ICLR 2025
Luis Carlos Garcia del Molino, Guangyu Robert Yang, Jorge F Mejias, and Xiao- Kiao-Jing Wang. Para- doxical response reversal of top-down modulation in cortical circuits with three interneuron types. Elife, 6:c29742, 2017.
Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583-30598, 2022.
Richard Gast, Sara Solla, and Ann Kennedy. Macroscopic dynamics of neural networks with heterogeneous spiking thresholds. Physical Review E, 107(():024306, 2023.
Richard Gast, Sara A Solla, and Ann Kennedy. Neural heterogeneity controls computations in spiking neural networks. Proceedings of the National Academy of Sciences, 121(3):e231188512,,, 2024.
Angeliki Giannou, Shashank Rajput, and Dimitris Papaillopooulos. The expressive power of tuning only the normalization layers. arXiv preprint arXiv:2302.07937 2023..
Lukas Gonon, Lyudmila Grigoryeva, and Juan-Pablo Ortega. Approximation bounds for random neural networks and reservoir systems. The Annals of Applied Probability, 33(1):28-60) 69, 2023.
Melissa S Haley, Stephen Bruno, Alfredo Fontanini, and Arianna Maffei. Ltd at amygdalocortical synapses as a novel mechanism for hedonic learning. Elife, 9:e55175, 2020.
Christopher M. Harris and Daniel M. Wolpert. Signal-dependent noise determines motor plan-
ning. Nature, 394(6695):880-7744 August 1998. ISSN 1476-4687. doi: 10.1038/29528. URL https: h/t:::/wwwwwww// nwtura.euee....... wtt.:/eetaaaatteeeeeeeeee /etriiieeee//__) Number: 6695 Publisher: Nature Pub- lishing Group.
Allen G Hart, James L Hook, and Jonathan HP Dawes. Echo state networks trained by tikhonov least squares are 12 (μ) approximators of ergodic dynamical systems. Physica D: Nonlinear Phenomena, 421:13 2021.
Sepp Hochreiter, A Steven Younger, and Peter R Conwell. Learning to learn using gradient descent. In Artificial Neural 21-25, 2001 Proceedings 11, pp. 87-94. Springer, 2001.
2001: International Conference Vienna, Austria, August
Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural networks, (2):251-257, 1991.
Kurt Hornik. Some new results on neural network approximation. Neural networks, 6(8):1069- 1072, 1993.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are uni- versal approximators, Neural networks, 2(5):359-366, 1989.
Herbert Jaeger and Harald Haas. Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication. science, 304(5667):78-80, 2004.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparraaaaappaaameeieiiitiio with gumbel-softmax. arXiv preprint ar /61 /.0/ 144, 2016.
Christian Klos, Yaroslav Felipe Kalle Kossio, Sven Goedeke, Aditya Gilra, and Raoul-Martin Memmesheimer. Dynamical Learning of Dynamics. Physical Review Letters, 125(8), August 2020. ISSN 0031-9007, 1079-7114. doi: 10. 10.1100// 125.00008............ 125.0281......... URL https:
7/link aps. org 1wt.:01111oo/hh/eeeett.... 125. 088103
Moshe Leshno, Vladimir Ya Lin, Allan Pinkus, and Shimon Schocken. Multilayer feedforward net- works with nonpolynomial activation function can approximate any function. Neural networks, 6(6):861-867, 1993.
Ming Li, Sho Sonoda, Feilong Cao, Yu Guang Wang, and Jiye Liang. How powerful shallow neural networks with bandlimited random weights? In International Conference on Machine Learning, pp. 19960-19981. PMLR, 2023.
12

---

Published as a conference paper at ICLR 2025
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint 0rXiv:2101.001900 2021.
Laureline Logiaco, LF Abbott, and Sean Escola. Thalamic control of cortical dynamics in model of flexible motor sequencing. Cell reports, 35(9), 2021.
Eran Malach, Gilad Yehudai, Shai Shalev-Schwarte and Ohad Shamir. Proving the lottery ticket hypothesis: Pruning is all you need. In International Conference on Machine Learning, pp. 6682- 6691. PMLR, 2020.
Ggaliwango Marvin, Nakayiza Hellen, Daudi Jjingo, and Joyce Nakatumba-Nabende. Prompt engi- neering large language models. In International conference on data intelligence and cognitive informatics, pp. 387-402. Springer, 2023.
Luca Mazzucato. Neural mechanisms underlying the temporal organization of naturalistic animal behavior, Elife, 11:e76577, 2022.
Luca Mazzucato, Giancarlo La Camera, and Alfredo Fontamini. Expectation-induced modulation of metastable activity underlies faster coding of sensory stimuli. Nature neuroscience, 22(5): 787-796, 2019.
Ariel Neufeld and Philipp Schmocker. Universal approximation property of random neural net- works. arXiv preprint arXiv:23/2.0844 2023.
Shun Ogawa, Francesco Fumarola, and I. Luca Mazzucato. Multitasking via baseline control in recur- rent neural networks. Proceedings of the National Academy of Sciences, 120(33)):2304439410,, 2023.
Lia Papadopoulos, Suhyun Jo, Kevin Zumwalt, Michael Wehr, David A McCormick, and Luca Mazzucato. Modulation of metastable ensemble dynamics explains optimal coding moderate arousal in auditory cortex. arXiv preprint arXiv:2404.03902, 2024.
Matthew G. Perich, Juan A. Gallego, and Lee E. Miller. A neural population mechanism for rapid learning. Newron, 100(4):64 2018. ISSN 0896-6273, doi: https://di..rr//10 /0g 1016/j. couron.2018.09.00
Allan Pinkus. Approximation theory of the mlp model in neural networks. Acta numerica, 8:143- 195, 1999.
Vinay Uday Prabhu. Kannada-mnist: A new handwritten digits dataset for the kannada language. arXiv preprint arXiv:1908.01220 2019.
Ali Rahimi and Benjamin Recht. Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning. Advances in neural information processing systems, 21, 2008
Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Raste- gari. What's hidden in a randomly weighted neural network? In Proceedings of the IEEE/CVF conference on computer vision and pallern recognition, pp. 1 11893-19909 2020.
Stefano Recanatesi, Ulises Pereira-Obilinovic, Masayoshi Murakami. Zachary Mainen, and Luca Mazzucato. Metastable attractors explain the variable timing of stable behavioral action se- quences. Neuron, 119(1)133-1153, 2022.
Evan D. Remington, Devika Narain, Eghbal A. Hosseini, and Mehrdad Jazayeri. Flexible senso- rimotor computations through rapid reconfiguration of cortical dynamics. Neuron, 9805):1005- 1019.e5, 2018. ISSN 0896-6273. doi: https://doi.org/10.. www.mmmuro..01..0000
Reidar Riveland and Alexandre Pouget. Natural language instructions induce compositional gener- alization in networks of neurons. Nature Neuroscience, 27(5):988-99) 2024.
Frank Rosenblatt et al. Principles of neurodynamics: Perceptrons and the theory of brain mecha- nisms, volume 55. Spartan books Washington, DC, 1962.
13

---

Published as a conference paper at ICLR 2025
Amir Rosenfeld and John K Tsotsos. Intriguing properties of randomly weighted networks: Gen- eralizing while learning next to nothing. In 2019 16th conference on computer and robot vision (CRV), pp. 9-16. IEEE, 2019.
Anton Maximilian Schäfer and Hans Georg Zimmermann. Recurrent neural networks are univer- sal approximators. In Artificial Neural Networks-ICANN 2006: 16th International Conference, Athens. Greece, September 10-14, 2006. Proceedings. Part I 16. pp. 632-640 Springer, 2006.
Georg Stefan Schlake, Jan David Hüwel, Fabian Berns, and Christian Beecks. Evaluating the lottery ticket hypothesis to sparsify neural networks for time series classification. In 2022 IEEE 38th International Conference on Data Engineering Workshops (ICDEW), pp. 70-73. IEEE, 2022.
R. Shadmehr and F. A. Mussa-Ivaldi. Adaptive representation of dynamics during learning of a
motor task. Journal of Neuroscience, 14(5):3208 3224, May 1994. ISSN 0270-6474, 1529. 2401. doi:
14E.///NEEEUOSS.1------0---0 1994. URL https: //wp:::/wwww.. /wwww jneurosci. otg/coontent/11/5/3008 Publisher: Society for Neuroscience Section: Articles.
Sen Song, Per Jesper Sjöström, Markus Reigl, Sacha Nelson, and Dmitri B Chklovskii. Highly nonrandom features of synaptic connectivity in local cortical circuits. PLoS biology, 3(3):668, 2005.
Jake P Stroud, Mason A Porter, Guillaume Hennequin, and Tim Vogels. Motor primitives in space and time via targeted gain modulation in cortical networks. Nature neuroscience, 21(12): 1774-1783, 2018.
Anand Subramoney, Guillaume Bellec, Franz Scherr, Robert Legenstein, and Wolfgang Maass. Fast learning without synaptic plasticity in spiking neural networks. Scientific Reports, 14(1):8557, 2024.
David Sussillo and Larry F Abbott. Generating coherent patterns of activity from chaotic neural networks. Neuron, 63(4):544-557, 2009.
Roberto Vincis and Alfredo Fontanini. Associative learning changes cross- cross-modal representations in the gustatory cortex. Elife, 5:216420, 2016
Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordy- intsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. International Conference on Machine Learning, pp. 35151-35174, PMLR, 2023.
Daniel Wu, Andrew C Yang, and Vinay Prabhu. Afro-mnist: Synthetic generation of mnist-style
datasets for low-resource languages, 2020.
David Wyrick and Luca Mazzucato.
State-dependent regulation of cortical processing speed via gain modulation. Journal of Neuroscience, 41(8))33988 4005, 2021.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnise a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv: (xXiv:770.007744 077477 2017.
Guangyu Robert Yang, Madhura R Joglekar, Francis Song, William Newsome, and Xiao-Jing Wang. Task representations in neural networks trained to perform many cognitive tasks. Nature neuroscience, 22(2):997-306 2019.
Haonan Yu, Sergey Edunov, Yuandong Tian, and Ari S Morcos. Playing the lottery with rewards and multiple languages: lottery tickets in rl and nlp. arXiv preprint arXiv: /006.
arXiv:/906.022: 1906.02768, 2019.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg, Bitfit: Simple parameter--ffficien fine-tuning for transformer-based masked models. arXiv preprint arXiv:2106.10199 2021.
Friedemann Zenke and Surya Ganguli. Superspike: Supervised learning in multilayer spiking neural
networks. Neural computation, 301))1514-1141, 2018
14

---

Published as a conference paper at ICLR 2025
A
BIOLOGICAL MAS-RELATER MECHANISMS
System
Bias model
Mechanism
Effect
Refs
Enhance memory
Dopamine projections to cortical circuit
Bias of NMDA or GABA conductances
Working memory
signal-to-noise ratio and resistance to distractors
Brunel & Wang 2001
Paradoxical reversal of somatostatin cell responses to running
Behavioral modulation of visual responses
Thalamic projections visual cortex
to pyramidal and somatostatin cells
Garcia del Molino al. (017))
Inhibitory biases of thalamic neurons
Basal ganglia projections thalamus
Toggle on/off diffcrent motor primitives
Motor sequences
Logiaco et al. (2021)
Select initial conditions (anticipatory activity) for motor primitives
Secondary motor cortex projections primary motor cortex (M1)
Recanatesi
Motor sequences
M1 neurons
(2022):
Mazzucato
(0222)
Premotor cortex projections primary motor cortex (MI)
Shift pre- movement preparatory states
Short-term motor adaptation
Perich
Recruitment of MI neurons
(2018): Feulner
at al. 2022
Mazzucato (2019); Vincis
Amygdalar projections to gustatory cortex
Acceleration of taste coding via gain modulation
Expectation and taste processing
Biases of cortical neurons
& Fontanini
(016); Haley
2020
Acceleration of visual coding via gain modulation
Movements and visual processing
Thalamic projections to visual cortex
Wyrick
Biases cortical neurons
Mazzucato
(2021)
Optimal encoding of auditory stimuli at intermediate arousal
Neuromodula- tory projections to auditory cortex
Arousal and auditory processing
Biases cortical neurons
Papadopoulos et al. 2024)
Table 1: Potential biological mechanisms for bias-related plasticity in the brain.
B MATHEMATICAL PROOFS
Throughout the appendix the proofs are restated for ease of reference. We will always take || ||-|| be the 1-norm unless stated otherwise.
|/|to to
B. RANDOM NEURAL NETWORK FORMULATION
The proofs of this section revolve around masked, random, neural networks:
15

---

Published as a conference paper at ICLR 2025
FM aro M O Bo(Wro Bx + b), IM ArM
(4)
where a E R, 0 oc, m E. No ro. i'm R.T in E ER'M E {0, 10" and all other matrices and vectors have real elements with the dimensions required by the above definitions. We assume that & is 7-parameter bounding and that each individual (scalar) parameter, be it weight or bias, is sampled randomly-bed before masking from a uniform distribution on [-9,5] (note that here are using where used R in the main text). In this way the parameters with compact support. M = then we drop the superscript. To account for feed-forward neural networks we simply set W to the matrix and a, 0, (this is assumed in Section (8.2).
random variables
W.l.o.g. assume there are n non-zero elements in M.. We construct W-M E -Rex--the recurrent matrix restricted to participating ith row and 5th column of the matrix if M, 0. We construct B.M E Raxd A. E Rix=, and bM E Rn by deleting the jth row of B, A, and 7th element of b M; O.
non-maaked) hidden units-by beginning with W and deleting the
Consider the case where the jth element of ro is whenever M, = 0. Then, regardless of whether EqAArepresent feed-forward network or the transition function for RNN, the masked units will always be zero. We can thus simply track the n units that correspond with I's in M as the outputs, M will depend solely on these. We observe that the behaviour of these units can described by the following network:
M aro + Bo(W-M + BM + BM),,
5m A AMM...
(5)
is networks of the form of Eq that will be the primary subject of study in what follows. Note that the '~'" over the r, is dropped to denote the fact that is different vector on account of dropping the zero units. In the feed-forward case we use subscripts, as we have done above, to denote hidden layer width. Whenever we discuss RNNs or dynamical systems we will instead use the subscript to denote time.
B.2 PROOFS FROM SECTION
Proposition 1. The ReLU and the Heaviside step function y--parameter bounding activations for any 7 >
Proof. We prove this solely for the ReLU, the logic for the Heaviside is effectively the same. Let thus be a ReLU. First, observe the following useful property: for all >> 0 we have aω(x) ((rr). From this, consider the neural network of hidden layer width with ReLU activations, yn. (0). and observe:
A
Bi x
bi
0 (a))
Un (0)
a²
a a²yn
(6)
i=1
Moreover, a E N we have
a²n
Vn = Σ o B. bb) = yarn (a)n(())
8n so that each element is simply re-scaled and repeated version of the original parameters; we have a² repeats for each term replace the a² factor in the LHS of Eq. 6
where = 06_
01
Now, given arbitrary compact set U E Rd, continuous function h : Rd - R. and s > 0, by the universal approximator theory (see e.g. Leshno et al. [1993] ; Hornik (1993)) we can find n such that
sup [1(x)) - (x, 0)1 se zEU
(8)
16

---

Published as a conference paper at ICLR 2025
holds. Because n is finite ciently large M. Suppose we want the parameters to be bounded instead by y with M > Y > O. we select a E N s.t. a > M then we can find Youn (z, 0) such that yoan (2, 0) = yn (z, 0). Thus we
bound every individual (scalar) parameter by M, for some suffi-
have found a parameter-bounding ReLU neural network satisfying Eq g completing the proof.
Remark: The intuition behind this result, for the ReLU, is credited to a reply to the Universal Approximation Theorem with Bounded Parameters question Mathematics Stack Exchange.
The following lemma constitutes the core of Theorem It shows that one can achieve universal approximation, in the sense needed for the theorem, using masking. The theorem then follows by manipulating biases to achieve masking.
Lemma 1. Let U + R' be a continuous function on compact support U C Rd. Then for any > 0,8 10, WE E {0, 1 satisfying the following:
find layer width E N such that with probability at least 1
sup zEU
|1([x) ym (x)ll Se.
(9)
Proof. First, we find a neural network with parameters that approximate the desired function h. Given the assumptions 6, we use Proposition 2 to find and parameters 0r {A*, B b*) such that
sup [M(x) vn 0') S c 2EU'
(10)
because U is compact and h is continuous.
We make observations: first, all choices of x are from compact set, U, by assumption and the parameters of a given random or non-random neural net are also from a compact set-the n dimensional hyper-cube with edge length 25. Second, the function Yo. continuous function of and the parameters. By these two observations the function is continuous function on the compact product space of inputs and parameters, and thus admits Lipshitz constant, Kn. This will come in handy momentarily.
Next, we construct a masked random network that approximates yn with high probability. By Lemma 3 we can find a random feed-forward neural network of hidden layer width m such that a mask, M, exists satisfying 00;- BM ε for some arbitrarily & > 0. In particular, we can choose
10: B.
M <E=
(11)
2K nl
for all with probability at least - 6. If we are in the regime of probability 1 - where the mask satisfying the above error bound exists then we get
(x) vn(xx, < Kn (100M -0
|| + || xl1) S Kan(d (nnld+1+11e 1)e
(12)
where, in addition to Eq 11 we used the fact that (x) = yn (n, 0M),, the continuity and compact- ness mentioned above, and repeated upplication of the triangle inequality. Importantly, this bound holds for all æ E U. Because Φ is assumed continuous, the function f(r) = [|rr (z) yn (xr]) is
also continuous. By the extreme value theorem 3 x² U such that supreu f(x) = f(x*). Since r² E U the bound from Eq 12 applies and we have:
I1 (x) (n(()) = I1y. (x*) vo (xx )1 #EU
sup
6
(13)
17

---

Published as a conference paper at ICLR 2025
Using the triangle inequality, Eq 10 and Eq 13 gives supzed l1b(z) -1m)))ymmmm (a))] - with probability 1 6.
Connections with Malach et al. 2020: As mentioned in the main text, the previous lemma is closely related past results on the SLTH over units for MLPs with one hidden layer. In Theorem 3.2 of Malach et al. (2020), it is proven that one can match the performance of a random feature model (an MLP where only the output layer trained) by scaling the output of mask-learned network. This theorem differs from ours in three important ways. First, it proves a result for a potentially different class of activation functions, second, requires scaling of the output of the network-und -unike our proof which does not require and, third, it compares the performance of mask-learned networks to random feature models, rather than directly proving that mask-learned networks can approximate wide classes of functions. We believe that one should be able to achieve a result similar to ours by combining Malach et al.'s Theorem 3.2 with Theorem of Rahimi & Recht (2008) (or a similar result on learning with random networks), but we leave the details of this to future studies.
U + R' is continuous. Then, for any degree of accuracy > 0 and probability of error 8 E (0,1), there exists a hidden- layer width m E N and bias vector b E Thm such that, with a probability of 8, a neural network given by Eq 7 with each individual weight sumpled from pr approximales h with error less than e.
Theorem 1. Assume that @ is -ebii--eearnn and, for compact U C R.d,
Proof. Observe that, once we have choosen an m satisfying the desiderata of Lemma ! because e is assumed to be --bias-learning, of á are bounded, we can implement the mask by setting b; to be very negative for every i such that M, = O. For every b, such that M 1 we simply leave b, at its original randomly chosen value.
is some finite value and all variables that make up the input
Corollary 2. Assume d 1, that is, the output and input spaces are the same. Then the results of Lemma 7 and Theorem 0 also hold for res-nets; that is, networks whose output is of the form
x Vm (x).
Proof. This follows by observing that h(x) + x is also continuous function and then replacing h(z) with f(z) + x in Eq and rearranging.
Remark: While the error can be made arbitrarily small, the limit of zero error itself is undefined. This is because our proof relies on first approximating the given smooth function with neural network with all parameters tuned and then approximating this second network using bias-learning to pick-out a matching sub-network from a large random reservoir; the probability of perfectly matching the fully tuned network with the bias-learned network zero. This could be addressed by using an integral representation for continuous functions instead of directly using a unite-widtt neural network to approximate the given function (see e.g. |Rahmii & Recht (2008): et al. (2023). ). As one will see below, this remark also applies to the recurrent neural network result.
B.3 PROOF FROM SECTION 2.2
Analogous to the section containing the feed-forward proofs, we first state and prove lemma which comprises the core of the proof for recurrent neural networks. This lemma shows that one can achieve universal approximation in the _ norm over trajectories sense (see section $2.2) with high probability using masking in randomly initialized RNN, and in this way provides proof of the SLTH over units for RNNs. The proof of the main theorem in this section then follows quite straightforwardly.
Lemma 2. Consider a discrete time, partially observed dynamical system of the form of. and satis- fying the same conditions as, the one in Eq Let 0 < T < co, F, E Un Vt, c >> O and 8 E (0, 1). Then we can find an RNN with appropriately chosen initial conditions and a layer width m E N such that with probability at least 1 - 8 3M E {0, 13mh satisfying the following:
T
sup 20 Fo EUS t=1
Σ
llve - MM <E
(14)
18

---

Published as a conference paper at ICLR 2025
Proof. It is well known that arbitrarily approximate this dynamical system with an RNN Schäfer Zimmermann (2006); we provide a simple proof this in Proposition 3 In particular, for arbitrary e > 0 we can find an RNN of the form in Eq with hidden layer width E N and output y, satisfying:
sup
Σ ll EU2 t=1
vell 2
(15)
Consider an n-width RNN of the form of Eq with wwrameter-bounding activation functions and with initial conditions ro ra(=)) for zo Uz. determined by 3q This will result in //2 ye (e (o((t)))) 0) being continuous function in all of its arguments. Moreover, all of its argu- ments are from compact sets: Ug is compact, Ug compact, and 0 can be chosen be on the hypercube with half-edges of length 7. Thus admits a Lipschitz constant Kn. We make use of this below.
Let the parameters of the above-defined approximating RNN, from Eq 15 be given by 0" = {A*, W "0,,"),,, b"). Then by Lemma we can find a random RNN of hidden width m and with parameters 0 such that mask, MM, exists satisfying
107 OM <E=
(16)
2K, Tn,nn(n+++ d+)
for with probability at least 1 - 8. Then, by the Lipshitz property yt, we get
T
T
M
Dell -
Σ
Kn (1 - -0-1 || Io.(t-1)]l + 1|z0 zoll)
(17)
1=1
< KnTnn + +1++ +++++1 1)e.
(18)
This follows from yr" = (ro -0:(t- 1), BM) where we have defined ro to be equal to To; for all such that M; and for all other indices-an and the fact that both ro and ro are continuous functions of 2g. This bound is independent of both inputs and initial condition, and, for any param- eters within the hypercube defined by 7, holds for all 2o E U2 and inputs To:(T-1) E Uz. By the Σ.= 11y. rall is contin-
continuity assumptions on the activation function, (oo(t-) uous function of 2g and -0:(T-1) for any 0" and gMM meaning that, analogous to the feed-forward case studied previously, can use the extreme value theorem to find maximal values a E U,_ and (o:(t-1) E Uz such that SupzoEUs. Ey f(20, Toi(t-1)
f(2) (o)( It thus follows that
"0 EU2 Et. [ly Dell < $. by Eq's 16 and
18
The triangle inequality along with
Eq 15 completes the proof.
Theorem 2. Consider the RNN in 2 with & biias--eeaninn activation, and input, oulput, and recurrent weight parameters for each hidden unit sampled from pr. We can find a hidden-layer width, m. a bias vector. and a continuous hidden-state state initial condition map U, + Rh such that, with a probability that is arbitrarily close to the RNN approximates the dynamical system defined in to below any positive, non-zero, error in the inifinity norm over trajectories.
Proof. This follows directly from Lemma 2 by observing that replace the mask by simply setting biases to some sufficiently low value.
B.4 SUPPLEMENTARY LEMMAS
The following result is well known in the literature; e.g. Proposition of Leshno et al. (1993).
19

---

Published as a conference paper at ICLR 2025
Proposition 2. For any 0 3m N s.t.
sup
[1((x) Un (x, 0') S
(19)
zEi
Corollary 3. The above holds if we restrict the oulput weight matrix of the neural network to have rank equal to the oulput dimension.
Proof. This is because the set of full rank matrices is dense in mx for m, n E N.
Consider matrices W* E R,xx, B" E R A E Rixn and vector by E R We can vectorize and concatenate their elements into the single long vector R", where = n(n + ++++1) Assume that [8; T for all
Next, construct W E Rmxx B TERR BR , .A R and vector E R.. by sampling each element randomly from uniform distribution on [-5,5] where 7 = y + Ay for A > 0. We analogously group these into single vector, construct sub-matrices of W, B, A, and sub- vector of by deleting column and row pairs in W, rows in B. columns in A, and elements of whose indices correspond to E {l,..., m] such that M; O. For a given M, define 0. to be the vector constructed by flattening and concatenating these sub-matrices and vector. We then have the following lemma:
(=((m++++++)) Observe that for each M E {0. 1)m, we can
Lemma 3. For 8", defined above, and arbitrary & > 0, 5 E (0, 1), we can find m > n. such that with probability at least 1 - 5 3M {0, 1 with only n non-zero elements such that 16, 6.MI < f a blog will satisfy the result, where min(s, Ar). (+)]
all i {L....,T] J.. In particular, any 2
Proof. In what follows we set min(s, A7). This simplifies the below probability bound that we derive because it means the probability of falling within an e window of desired parameter will not change, even if the desired parameter is very close to its bound, Hy. We will refer to the event that the desiderata of the lemma are satisfied for e, rather than E, as A1; that is: WM E {0, 1)mh with only non-zero elements such that 10;- A.M| for all E (1.....m). The event that the desiderata are not satisfied Aç.
Assume that m = kn for some k E N+ Consider the 'block' mask M s.t. M = only for i {(k - 1)n + I.....,in] with 0 < k2 S k. Note that the n elements selected by these block masks are non-overlapping for two different k1. Let event A2 be the event that there is block mask that occurs satisfying the desiderata of the lemma with error e. Clearly A2 C A A C AS = P(Af) I P(Ag). Ag is the probability that there is no block mask satisfying the desiderata. Observe that
P(AS) = P
n fath block mask doesn't 't work] 11=1
= II
P( block mask doesn't 't work))
k =1
- P block mask works})
e
(20)
Ag
which follows from the fact that the elements of the matrices are independently sampled and the elements corresponding to sub-matrice selected by a given block mask are independent of those associated with another block mask. By making m very large we can make P(Aş) arbitrarily small. Because P(Af) < P(d))-and the desiderata of the lemma with error
not satisfied
solely on Ahe result follows by selecting m = m such that P(Ag) < 8. We thus see that
the probability of finding a sufficient mask occurs with probability at least 8. Lastly, because
we have found a mask that satisfies per- parameter error e, and because e S E, we have proved the lemma.
20

---

Published as a conference paper at ICLR 2025
Remark 1: We note that, in Eq 20 will likely also depend implicitly on y. If is very small then we will need to stack many ReLUs on top of each other to attain large enough dynamic range to approximate the desired function (see $2.T). leading to larger number of units. Conversely, if is very large we will need to sample large number of units before we get parameters appropriately close to the desired subnetwork configuration. This suggests the existence of some sweet spot in the value which we leave for future work to explore.
We observe that this bound appears to be very weak. For example, if one wished to use it to find masked network to match an MLP with input, hidden, and output dimensions of only 1, 3, and respectively, with pee--ppareeee per-parameter eтror of 0.05 an error probability of 0.1, and with 0.1, this bound would suggest we need hidden layer of m 2 8.34 x 10 neurons in the bias learning network. light the numerical experiments, it is clear that while the math here provides proofs of existence for bias learning it massively over-estimates the layer widths required in practice, and thus does not say anything useful about the hidden layer scaling required by bias-learning.
Remark 2:
For the following proposition we consider the discrete time dynamical system that we wish to ap- proximate to be as in 3
Proposition 3. For finite 0 < T < co, 0, and any |a| so, 3 we can find RNN the style of Eqq. of hidden width E N and a continuous mapping Tu U2 + IR" for the initial value of the RNN such that:
T Σ
sup
llue (ro(2o), (o.(T-))) - Fo(T--))
(21)
SaEU..P0.C EUS t=1
where de the output of the RNN and yt that of the dynamical system.
The main portion of this result is well known, see e.g. SSchäfer & Zimmermann (006)_ For com- pleteness, we provide an example proof below.
Proof. In what follows, W.I.O.G will assume that the error is smaller than We want to approximate the dynamical system:
E++1 = F(z,, y1 C.z, zo E Uz.
(22)
co : 20 E U2, ||co|| < cJ. where U2 is an invariant set defined, by assumption, on set Ü= = {=o + (see 2.2
We define the set:
U = {[2 + co x] U2, U,|||oll|
< <e.
(23)
Importantly, this set compact given the compactness assumptions on U and Uz. Also note that, since F is assumed continuous, it will be Kp-Lipschitz this compact set for some constant Ky. r- By the compactness just discussed and the continuity of F, we can use the corollary to Proposition to find a neural network of hidden dimension N that approximates F' with maximum-rank output matrix, A. We write this neural network:
2 = az + BAp(W2 Br b) = F22,z).
(24)
assuming E Ug and U. with E R*xn, W Raxx B Raxd and b R., In particular, we can find arbitrary with 0 < < c such that:
[| F(2,,))) <E=
(25)
T (a Er- K),
where RG = ||C|| Fix T 2 1. To prove that we can approximate the underlying dynamical system, use induction starting time /. 1. The base case will be
21

---

Published as a conference paper at ICLR 2025
|2 21l1 |10 vo) - F F xo)ll E
(26)
by our choice of and initial condition, and that [zu, zol E Us- Importantly, this implies also that || 21 || < E. Because & < c this means that [21 æ1] E Uz..
For = 1. t-1 KfE.. We thus make the induction hypothesis that || - < Ex =0= Ki
t-1
and that [2, xy] E U If 1 we are finished. If >> we assume 1 < t < T and use this hypothesis to prove the induction step:
|| 2411l (1t, (( xe)] + |1F((), x.) - F1),))))
(27)
E +
KFl
& zell = Σ K T
(28)
Because $ S 12+41 2141]T
Unn. Then
Σ
ll vell S RoTs Σ kp.ee. Se.
(29)
While we have approximated the dynamical system it not yet in the standard rate-style RNN form. However, can obtain the rate form by changing from tracking to a different dynamical variable, rz E IR", that satisfies & = Art. We will make a brief detour to characterize this variable.
Because A is full rank, col(A) IR we can find an index forms a basis for R'. we construct a matrix A, E Rax* whose columns are simply the basis vectors this matrix will have inverse A, We can then define the initial condition, To, element- wise as:
(I....,m} such that {A.
rox( = [A
20 iEv
(30)
10
i i v.
This function is clearly continuous and satisfies Aro ΣiEv Autos EiEv A4 2 A,A, 20 2g. If we then define Tt = ar:-1 d(W Ar + BI--1 + b) for all 1<tst we see that ri is simply the state variable for an RNN of the style of Eq/2] and that it satisfies &, Ar, for all <
It follows that f CAr, Vt. Thus, this RNN approximates the original partially observed dynamical system in the sense described in section 2.2
C METHODS FOR NUMERICAL RESULTS SECTION
C.1 METHODS FOR SECTION 3.1
In this section all networks were single hidden layer FNNs. For figure 1A, networks with widths of 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384, and 32768 were trained, with the width being indicated on the z-axis. For figures IB and IC, width of 3.2 x 104 was used.
For figure 1A, all networks were trained FashionMNIST, with 5 10a training samples and 1 test samples, using ADAM with a learning rate of 0.01. Training was run for 20 epochs with a batch size of 512.
Xavier uniform initialization with gain of 1.0 was used for the fully trained networks, while the frozen weights for the bias-only networks were sampled from either uniform distribution on [-0.1, 0.1] or from mero Gaussian with standard deviation 1 where d is the input dimen- Va
sion.
22

---

Published as a conference paper at ICLR 2025
For Fig E.3 network with 3.2 x 10* units was trained on x 10 random samples from a concate- nation of the 8 MNIST style datasets used in Figure ! Weights were initialized to Gaussians with mean 0 and SD
v biases were initialized uniformly on [-0.01, 0.01], and ADAM with learning
rate of x 10 10-6, and batch size 512 was used to optimized cross entropy loss for 15 epochs. The network reached test accuracies of 0.9924 (Ethiopic) 0.8624 (Fashion). 0.8883 (Kannada), 0.0000 (KMNIST)), 0.9354 (MNIST), 0.9995 (Nko), 0.9979 (Osmanya), and 0.9910 0 (Vai). Thus, it failed to learn KMNIST, which we hypothesize be due to interference with the other datasets. this figure and in Fig D we used the Kmeans from Scikit-Learn with 20 clusters. Before clustering/plotting normalized the variances by dividing each dimensional vector unit variances by the maximal value plus a small constant to avoid division by zero.
C.2 METHOOS FOR SECTION 3.2
In this section all networks were single hidden layer FNNs with 10 ReLU units. Weights were each initialized Gaussian with mean and standard deviation of 1/28-thhe the inverse square root of the input dimension-and were left unchanged during training.
Both bias and mask networks were trained on MNIST, with 10 training samples and 10 test samples, using ADAM with a learning rate of 0.01. Trained parameters were each initialized uniformly [-0.01, 0.01] (for the mask learned networks the bias vector was initialized in this way and left untrained), and training was run for 30 epochs with batch size of 512. For bias learning, the trained parameters were the 10 element bias vector, b; for mask learning, trained parameters were a 10 element vector of gains g.
The output of the 7th unit in the mask-trained network was ReLU x + b) Digmoidd (:) Here, was the input, flattened, MNIST image, W the random hidden weight matrix and 7 an annealing parameter. Starting from To I, at each epoch т was decreased so that at epoch k T CTk-1 with chosen such that at the final epoch T30 0.001. This scheme was chosen so that by the end of training the sigmoid functions would be almost step functions, well-approximating the desired binary mask. We found that this worked better than immediately setting 0.001. For testing, units were evaluated with the Sigmoid factor binarized (r = oo).
The null model used in 2ig was calculated by drawing 10 samples from two, independent, Bernoulli random variables whose probability of being ON (drawing a 1) were equal to the mean fraction of ON units in the bias and mask-traines networks respectively.
C.3 METHODS FOR SECTION 3.3
C.3.1 COSINE GENERATION (FIG. 3A-B)
We used N 200 hidden recurrent units with ReLU activations. Biases were initialized from 4400, 1). The recurrent weights were sampled from ((00, N-1) and the output weights from (O,,NN The target period of the cosine was 25 and the total duration to generate was 125. Learning rates were 0.1 for bias learning and 0.001 for the fully-trained network; other parameters for the Adam optimizer were left at their default values in Pytorch.
C.3.2 VAN DER POL OSCILLATOR (FIG. CDD)
We used 675 hidden ReLUs for the bias-learning network and 25 for the fully-trained network. The recurrent weights were sampled from [00 IN where is the "gain"; other weights were initialized in the previous subsection. The Van der Pol oscillator obeyed H p(1 2²) for = 2. Learning rates were 0.1 for bias learning and 10-4 for the fully-trained network and other optimization parameters were above.
23

---

Published as a conference paper at ICLR 2025
C.4 METHODS FOR SECTION 3.4
C.4.1 RNN ARCHITECTURE AND HYPERPARAMETERS
The architecttre is single-layer me-dimensional input/ouiput recurrent neural network with ReLU activations. The training optimizer was Adam with learning rate of 0.001 and weight decay value of 0.1.
C.4.2 LORENZ ATTRACTOR
The equations used to generate the partially-observed Lorenz attractor were the following:
dr
dy x(p - dt ry Bz
dz
dt o z),
dt
where the initial point (0, 1, 0) and a, p. and B are 10, 28, and j. respectively. The trajectory was generated using Euler's method with step size of 0.01.
C.5 METHIODS FOR SECTION 3.5
We used = 1,024 hidden ReLU units in a vanilla RNN (Eq. with a = 0 and / = 1). The input to the network was the 2D Cartesian position of the targets, which were 0.07 away from the center of the workspace. The point-mass arm obeyed a discrete-time version of the following dynamics dē
dr = =(t), dt
f(t) df _f_ u(t) - f(t)
dt
m dt
where x is the 2D arm position, & its velocity and the applied force. The initial conditions were initially rest. The force was obtained from
=0) =0) = f() O, i.c., the arm exponential filtering of the controls u(t) with time constant Tf = 0.04 s. The network generated the controls u(t) via linear readout of its activity:
Cr. The input (B), recurrent (W) and
output (C) matrices
initialized as: Bij U(-1/
Wij NO0 1.5625/N), and
Ci (O, 0.5NN). The loss function was
d%
12
(k) T
1
T
+ 70
7/
(31)
82
82
83
where dę Dicoos((Ttttt (2tk/ K), (2 KT] k= 0, ..... . 1, represent the position of the K ((k)
(k) (k)
peripheral targets distance D 0.07 from the center target. Here, 4T 2T and fr
the final position, velocity and force for the kth target (T = 1s 1s/0.01s = 100, where 0.01 s was our integration time step). Therefore, the objective term) with near-zero velocity (second term) and force (third term). Parameters 5, = 0.01, 8, 0.02 and 8y 0.08 were used rescale the position, velocity and acceleration terms. Hyperparameters 0.04 controlled the relative weight of the velocity and force costs with respect to the position loss. The learning rate of the standard Adam optimizer was set to 3 x 10-3 and training was stopped when loss of 5 x 10 was reached.
to reach the target the end of the trial (first
Tv = 0.2, and
D TRAINING TIME
We ran a brief experiment on training time, which we report as mean 1 1SD over 5 seeds. Training a fully-trained single-layer MLP with 10 hidden units on MNIST for epochs takes 69. (9.177 + 0.22 seconds. Training only biases for the same task and architecture takes 61.51 + 0.92 seconds. Thus, for matched layer widths training biases with standard Pytorch implementations provides slight advantage. Note that one can maintain performance and get faster training by simply reducing the width of the fully-trained network: 100 hidden unit fully trained net takes 37.77 + 0.04 on the same test.
E SUPPLEMENTARY FIGURES
24

---

Published as a conference paper at ICLR 2025
b
b
X
X
1.
2.
3.
4.
Figure E.1: Proof Intuition. Step 1. Consider smooth, partially observed Dynamical System (DS) on an invariant set, S (or function y(r) on a compact set Q2.. Step 2. Let 0 < T. Approximate DS on Q using RNN R, (or approximate y on Z using FNN Ni). Step Randomly initialize larger RNN, R2 (or FNN N2). Check for sub-network of units with parameters matching Ri (W1). Step 4. Adjust biases outside the sub-network be very negative-thus shutting off the units outside the sub-network-ann and those inside the sub-network to match those of R₁ (N1).
A
B
Bas leaming Unitorm
Bas leaming Usiform
Fraal Freal Valitation Prmr
6xx30-2
Fully buly-rrindd
Final Vaiidastion Ernor
Fuly-rained
feaming Goussion
4x
3xx30-1
3x
2x 1
2x30-
5003
10000
15000
Number of trainable paremeters
layer
C se
Uniform
Normal
Normal Mixture
D
3.00
045
vardanon 94
030
035
040
0:5
0.00
32
13B Wooth
756
532
1304 3048 1906
0.9
Accuracy
0.8
0.7-
fully trained
bias leamnigg
1-322 1-128
6+300
3-303
1-0
1-100
Dasibutton
Numbar of Hidden Layers
Figure E.2: Supplementary Experiments. A. As in F except with and y-axes log-scaled. B. As in A except a-aaxis is layer width. C. Performance on MNIST (y-axis) of bias learned MLPs as function of distribution from which each individual weight initialized (x-axis). Orange: uniform on [-7, 7]: Blue: 0 mean Gaussian with standard deviation Green: mixture of Gaussians centred at =u each with standard deviation 0.015. Layer width 10, 000 units. D. As in Fig 4 A except æ-axis is layer width. E. Performance (y-axis) of fully trained versus bias learned (uniform initialized weights) MLPs on fashion MNIST as a function depth (æ-axis). Layer width is 2048 units for both learning types. All plots: plotted means and +SD over 5 random seeds.
25

---

Published as a conference paper at ICLR 2025
athiopic
fashion_nee mnist
0.8
kannada
Datasee
kmnist
mnist
0.4
osmanya
0.0
5000
10000
15000
20000
25000
30000
Figure E.3: Supplementary for Figure Figure but for single fully-trained network to meta-dataset containing the MNIST style datasets used in Fig The bottom row is cluster index. We observe task selectivity that is qualitatively similar to bias-lcarning. See $C.I for methods.
C
1,0
H
Normelized Matchhs
Mask: Trained Variannee
0.8
1.0
Test
0.5
04-
Null Model
Thresholded Threshold
Bas vs. Mark
0.5
0.0
2000
4000
6000
8000
Unib
Unit
0.0
1.5 0 Trained Unit Variance
0.5
1.0
200
Number of ON Units
Bias
Figure E.4: Supplementary for Figure 2. A. Decrease bias learning test accuracy as the task variance threshold, above which units are ON, is increased. r-axis is number of ON units; y-axis is test accuracy on MNIST. Threshold values are those used Fig B. Plotted is mean and SD over 3 training runs. Probability that given unit is ON/OFF for versions of network with the same weights trained by bias versus mask learning. Left group of bars is probability of the unit being ON for bias learning given that ON for mask learning, middle is the same but for OFF instead of ON, right is the probability of either match occurring (OFF or ON in both training paradigms). Grey null model (see section C.2 for details); orange the bias-mask comparison; black is the same comparison but between two mask-learning training runs. C. Scatter plot of hidden unit variances taking only units that are ON for both mask and bias learning; bias-trained on x-axis and mask- trained on y-axis. Error bars are over 5 training runs. We observe mean correlation coefficient of 0.5030 + 0.0087. We note that the correlation between two mask-lcarning runs on the same random weights is 0.9999 + 0.0000. Correlation coefficients are over 5 samples; plot data is from one, representative, pair of bias/mask networks.
A
B
C
Training
seed
10
Loss
16"
Speed (mis)
Generalization
10"
10
0.0
0
3
3m
0.0
0.5
1.0
Epoch
Time (9)
Figure E.5: Supplementary for Figure 5 Same figure as Fig. 5 in the main text, but for fully- trained networks. The architecture and initial parameters (sceds) were the same as for the bias- learning networks (see Methods). The only change than that for bias learning. For these matched parameter initializations (in particular, given the high variance for the recurrent weights), the solutions found by the tulin-t network had more temporal variations compared to bias learning.
that the learning rate 10 times smaller
26


Reviews:
[{"review_id": "oiKQ0OIpjF", "reviewer": "Reviewer_1PGv", "summary": "Previous work has investigated the expressivity of feed-forward neural networks (FNNs) when only subsets of parameters are trained (ie. only the output layer, normalization parameters, etc…). In the same vein, the authors introduce a method of training feed-forward neural networks by randomly sampling fixed weights and subsequently learning only the biases, termed bias learning. They provide theoretical and empirical evidence that demonstrates that FNNs trained through bias learning can approximate any continuous function on compact sets - meaning that they are theoretically as expressive as fully-trained FNNs.\n\nThey start with a theoretical treatment of bias learning where they carefully define their terms and introduce their theorems. A simplified version of the rigorous proof is as follows: 1) Train a fully connected network (N1) where the weights are constrained to lie in some fixed range. 2) Create a new network (N2) by randomly sampling the hidden neuron weights from the fixed range in 1. 3) After sufficient sampling, there exists a subnetwork of neurons in N2 that is ‘identical’ to the neurons in N1. 4) By training the biases, the outputs of neurons outside of this subnetwork can be removed, leaving N1 from N2 bias training. They provide a similar proof for recurrent neural networks (RNNs).\nNext, the authors provide empirical evidence supporting their theory and explore the expressivity of bias-learned networks. They do this in multiple ways, including performing multi-task learning with bias learning in 7 tasks (MNIST, KMNIST, Fashion MNIST, etc.), comparing bias learning and mask learning in FNNs, and applying bias learning on an RNN trained on both an autonomous and non-autonomous dynamical system. The main takeaways are as follows: 1) multi-task bias learning leads to emergence of task-specific functional organization revealed by clusters of activation patterns measured by task variance, 2) compared to mask learning, bias learning had less sparse solutions and higher unit variance values, 3) bias learning in RNNs can succeed in time-series forecasting of non-linear dynamical systems with high enough gains.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- Overall, there is strength in its novelty of proving that bias learning in neural networks can have high expressivity that performs almost as well as a fully-trained network. This is significant because bias learning trains fewer parameters than a full network.\n- Nature of bias learning is more behaviorally relevant in the context of tonic inputs, intrinsic cell parameters, threshold adaptation, and intrinsic excitability\n- The theoretical proofs are very thorough, and backed up by numerical proofs.", "weaknesses": "- In response to bias learning having fewer parameters to learn, no data was shown on training time\n- Little background was given on mask learning (the mask learning section was also super short - felt less developed relative to other parts of the paper). This is important because two of their highlights in the results relate to mask learning.\n- Makes claims (i.e. lines 253 - 259, lines 418-420) that could have been easily backed up by data, but were not.\n- Figure 1 color scheme is weird\n- Practically, not sure how exciting this is (i.e. other models can do what this model does - it's just that they use a different approach)\n- The work seems highly related, in spirit, to neural tangent kernel approaches and other methods that consider wide NNs, but no references to that work were made.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LEsanvSsm5", "reviewer": "Reviewer_jeUd", "summary": "The authors show that both feedforward and recurrent neural networks can act as universal function approximators for functions and dynamical systems respectively, even when only biases are learned. They propose an alternative proof for the theorem that masking is all you need (Strong Lottery Ticket Hypothesis), and extend that and the bias result to RNNs approximating dynamical systems. They authors demonstrate their results using simple simulations, and discuss relevance to AI and neuroscience.", "questions": "In Section 3.3.2 - The authors write RNN, but then say that the recurrent state is given? Also what is provided as an external input? Is it the recurrent state? The difference between the 'standard' and the 'self-sustained' networks is not clear. To me the self-sustained way is the standard, and if somehow the recurrent state is provided (at each time step?), then the network is just acting as a feedforward network. Then in this case, I suspect that to actually use the RNN (usual self-sustained way) to learn the dynamics, the authors would need a lot more units.\n\nThe authors have not explained how (positive & negative) biases may arise in neuroscience if not by synaptic weights. As they mention threhsold changes etc. change the neural gain (and possible have a strongly non-linear effect). What about the role of inhibition and other brain areas switching parts of the network on and off?\n\nThe authors should bring out the differences between their proof and the Malach et al proof.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The authors connect well to the neuroscience and AI/ML literature and explain the proofs in an intuitive manner. \nThe extension to RNNs and dynamical systems is also commendable as these often receive reduced attention in the ML community.\nThe issue with the \"gain\" g in the weight distribution is well brought out.", "weaknesses": "The section 3.3.2 on the Lorenz system is not clearly written and the architecture and external input to the network are not clear. \n\nAt first glance, the result seems to be a simple extension of the masking theorem of Malach et al 2020.  The difference with that proof should be made clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EPSPliO5gr", "reviewer": "Reviewer_kfuh", "summary": "In this work, the authors prove a universal approximation result for the expressivity of neural networks with frozen weights but trainable biases. In particular, they build upon well-known universal approximation results of feedforward and recurrent architectures, and show via a simple mask learning-like argument that sufficiently large networks with randomly chosen weights can be constructed to approximate any function (feedforward) or finite-time trajectory (recurrent). They conduct experiments comparing fully trainable architectures to bias-learning variants, demonstrating that bias-only learning can achieve reasonable performance on some simple tasks.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The main expressivity results shown are well-explained and seem mathematically tight. Considering that these results made use of a reduction to mask learning problems, the authors also do a good job discussing the relationship between their findings and those of the mask learning literature.", "weaknesses": "A crucial aspect of this work with regards to its practical relevance is how large a bias-trained network needs to be to achieve similar performance to a fully trained network. Surely the scaling is better than the extreme network expansions constructed for the existence proofs, but how much better? The authors allude to performance as a function of trainable parameter count scaling similarly to fully trained networks, and thus only needing quadratic scaling in layer width, but they only evidence this explicitly with comparisons to mask learning networks, which in my view are also less expressive than standard networks (for fixed number of parameters). I would like to see a more detailed investigation of this question. For example, could the authors extrapolate from the MNIST experiments (Fig. 1a) whether the required scaling is indeed quadratic? I imagine this scaling would also depend significantly on the task difficulty and the frozen weight initialization. \n\nOverall, the tasks the authors used to demonstrate efficacy of bias-only learning seemed restrictively simple, by the standards of both the machine learning and computational neuroscience literature. In particular, for the RNN experiments, only simple 1D pattern generation tasks were considered. I would be interested in seeing how biased-trained RNNs perform on simple \"cognitive\" tasks often used to assess task-trained RNNs in the computational neuroscience literature (e.g., interval timing, delayed match-to-sample). For example, I imagine that any task that requires the construction of many stable fixed points could be quite difficult for bias-learning RNNs, and might require prohibitive scaling of network size compared to fully-trained counterparts (e.g., N-bit flip flop).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VKzpyUmyST", "reviewer": "Reviewer_WzyV", "summary": "Note: I’m not an expert in the field of pruning and universality.\nThe authors show that random neural networks with trained biases are universal approximators. This is shown both for feedforward and recurrent networks. The authors use an argument that is similar to masking, and use negative biases to implement the masking. Numerical simulations show that several benchmarks reach similar performance when training biases or training weights as well.", "questions": "1.\tDefinition 2: bounded by gamma?\n2.\tLine 186 large hidden layer width. Can you provide a rough estimate? What is the scaling? I assume it is roughly (R/eps)^n. In Malach et al, the size was polynomial. If I understand correctly, line 944 gives such scaling, and by approximating log(1+z)=z it is indeed (1/eps)^(n^2).\n3.\tFig E2. It is hard to compare what happens from 5000 parameters or so. Perhaps a logarithmic or ratio plot would help.\n4.\tFig E2 – fully trained was still better than bias-only, even when controlling for parameter number. Do you know why this is? Is this related to the main concern raised above?\n5.\tLine 304. Correlation between TV and bias. Was this computed for every unit, and then averaged within each cluster?\n6.\tLine 304 If the theory is aligned with training, then the number of units should be much higher than simply the square of fully trained network. If Figure E2 suggests otherwise, then why expect the mechanism of the proof to hold? Further – what are the values of biases? Are some of them extremely low – effectively shutting down neurons?\n7.\tCorrelation values between mask and bias – what is the correlation between different realizations of the training process?\n8.\tLine 461 – similar scaling in RNN and FNN. Figure E2C shows that the fully trained saturates at 64^2 parameters. The bias trained network is shown up to 64^2 parameters, so we can’t see whether fully-trained RNNs with less than 64^2 parameters behaves similarly to FNN.\n9.\tLine 462 stability in larger windows. Perhaps I didn’t understand this, but is this really stability or simply more test sets? Because the network is fed the true dynamics, is there a meaning to larger windows?\n10.\tFigure 4C – How does the fully trained network generalize in this scenario?\n11.\tLine 481 – I think some discussion of scaling should go into the main text, even if the proof is in the appendix.\n12.\tLine 483 – task-selective clusters similar to fully trained. This was not shown. Specifically, quantification of how task-selective are bias-trained vs. fully-trained networks.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The questions of pruning, masking and universality are all important questions for neuroscience and machine learning. In neuroscience, it is known that many cell-autonomous adaptation mechanisms exist (e.g., spike threshold adaptation), and the wide distributions of firing rates hint that these properties could be a form of long-term plasticity as well.\nProving universality results has opened the door for more application-oriented research in the past. The combination of mathematical proofs with systematic simulations and a wide literature review is a strength.", "weaknesses": "My main concern is the scaling of network size, which seems exponential and is also missing from the main text. The results of Malach et al 2020 suggest that masking is weaker than weight-pruning, unless the size of the network is exponential.\nIf I understand correctly, I expect a scaling of (R/eps)^n. Because there are n independent event with probability (R/eps). Line 944 (appendix) states a scaling, which by approximating log(1+z)=z is indeed (1/eps)^(n^2).\nGiven this large scaling, and the results of Malach et al that with polynomial scaling neuron-pruning is weak, it seems strange that bias learning is as strong as weight learning. Indeed – the actual numerical results do not show comparable performance. As the tasks become harder, the gap widens. Also when controlling for the number of learned parameters, bias learning is still weaker.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)