Pretraining with Random Noise for Fast and Robust Learning without Weight Transport
Authors: Jeonghwan Cheon
Sang Wan Lee
Se-Bum Paik
Venue: NeurIPS 2024 poster (2024)
Keywords: Random noise training
Network pretraining
Pre-regularization
Feedback alignment
Error backpropagation
Weight transport problem
Biologically-Plausible Algorithm
Primary area: neuroscience_and_cognitive_science
TL;DR: Pretraining with random noise using a feedback alignment algorithm allows fast learning and robust generalization without weight transport.

Abstract: The brain prepares for learning even before interacting with the environment, by refining and optimizing its structures through spontaneous neural activity that resembles random noise. However, the mechanism of such a process has yet to be understood, and it is unclear whether this process can benefit the algorithm of machine learning. Here, we study this issue using a neural network with a feedback alignment algorithm, demonstrating that pretraining neural networks with random noise increases the learning efficiency as well as generalization abilities without weight transport. First, we found that random noise training modifies forward weights to match backward synaptic feedback, which is necessary for teaching errors by feedback alignment. As a result, a network with pre-aligned weights learns notably faster and reaches higher accuracy than a network without random noise training, even comparable to the backpropagation algorithm. We also found that the effective dimensionality of weights decreases in a network pretrained with random noise. This pre-regularization allows the network to learn simple solutions of a low rank, reducing the generalization error during subsequent training. This also enables the network to robustly generalize a novel, out-of-distribution dataset. Lastly, we confirmed that random noise pretraining reduces the amount of meta-loss, enhancing the network ability to adapt to various tasks. Overall, our results suggest that random noise training with feedback alignment offers a straightforward yet effective method of pretraining that facilitates quick and reliable learning without weight transport.

Pretraining with Random Noise for Fast and Robust Learning without Weight Transport
Jeonghwan Cheon Sang Wan 1,23 Se-Bum Paik
DDpartmeet of Brain and Cognitive Sciences,
Graduate School of Data Science, *Kim Jaechul Graduate School of Al Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea (jeonghwan518, sangwan, sbpaik)0kaist.ac. ac.
Abstract
The brain prepares for learning even before interacting with the environment, by refining and optimizing its structures through spontaneous neural activity that resembles random noise. However, the mechanism of such a process has yet to be understood, and is unelear whether this process can benefit the algorithm of machine learning. Here, we study this issue using neural network with a feedback alignment algorithm, demonstrating that pretraining neural networks with random noise increases the learning efficiency as well as generalization abilities without weight transport. First, we found that random noise training modifies forward weights to match backward synaptic feedback, which is necessary for teaching errors by feedback alignment. As a result, a network with pre-aligned weights learns notably faster and reaches higher accuracy than a network without random noise training, even comparable to the backpropagation algorithm. We also found that the effective dimensionality of weights decreases in network pretrained with random noise. This pre-regularization allows the network to learn simple solutions of low rank, reducing the generalization error during subsequent training. This also enables the network to robustly generalize a novel, out-of- distribution dataset. Lastly, confirmed that random noise pretraining reduces the amount of meta-loss, enhancing the network ability to adapt to various tasks. Overall, results suggest that random noise training with feedback alignment offers a straightforward yet effective method of pretraining that facilitates quick and reliable learning without weight transport.
Introduction
The brain refines its network structure and synaptic connections even before birth, without exposure to sensory stimuli [14] In the early developmental stages, the spontaneous neuronal activity that appears in various brain regions is considered to play critical role during the development of neuronal circuits by pruning neural wiring and adjusting synaptic plasticity (58)). this activity is disrupted during the developmental stages, the outcome can be long-lasting neuronal deficits 1ЯI1. Computational studies suggest that such a refined network structure enables certain crucial functions of the brain, such as initializing function I22 [1] and structure [16] [7]. These experimental and theoretical studies commonly indicate that spontancous, random neuronal activity plays a critical role in the development of the biological neural network before data are encountered by the network. However, the detailed mechanism of how these prenatal processes contribute to learning after birth, i..,, with subsequent sensory stimuli, remains elusive.
At the synaptic level, learning can be defined as the process by which the brain adjusts the strength of synaptic connections between neurons optimize the network for specific task 20). The synaptic weights of each neuron can change to minimize the error between the expected and the
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

actual output of a task, often referred to as the credit assignment problem [3] However, in general, it is not well known how individual neurons modify these synaptic connections and thus achieve network goal under a condition in which numerous neurons are linked in multiple layers. In other words, how neurons can estimate errors to modify their synaptic connections during learning remains unknown.
In machine learning, backpropagation algorithms have successfully addressed this issue even in deep neural networks |2-266. Backpropagation can provide feedback with regard to forward errors through the symmetric copying of forward weights via a backward process. During this process, structural constraint, i.e., symmetric forward and backward weights, is necessary to assign proper error values to individual neurons [27-29]] However, this process appears to be biologically implausible due to the weight transport problem 23 30,32), in which individual neurons must somehow be aware of the exact synaptic connections of their downstream layers to update their weights, a state considered to be practically impossible in a biological brain.
An alternative algorithm, feedback alignment, achieves successful network training even without weight transport by employing fixed random feedback pathways (32). This study shows that a network can align its weights to synaptic feedback during data training, and this simple process enables error backpropagation. It has been shown that soft alignment between forward and backward weights, which can be achieved during learning data, is enough back- proppaaae errors. This finding may provide biologically plausible scenario in which the credit assignment problem can be resolved, yet there is an issue remaining the process requires massive data learning to develop the structural constraint, and it significantly underperforms compared to backpropagation on challenging tasks This cannot be addressed even with currently known advanced learning rules (35)
33 34
This situation is contradictory to the notion that the brain can learn even with very limited experience in the initial stages of life [41-44] Thus, the question arises as to how early brains can estimate and assign errors for learning with limited experience. To address this issue, here we focus on the role of spontaneous activity at the prenatal stage in the brain, showing that training random noise, which mimics spontaneous random activity in prenatal brains, is possible solution; random noise training aligns the forward weights to synaptic feedback. enabling precise credit assignment and fast learning. We also observed that random noise training can pre-regulate the weights and enable robust generalization. Our findings suggest that random noise training may be a core mechanism of prenatal learning in biological brains and that it may provide a simple algorithm for the preconditioning of artificial neural networks for fast and robust learning without the weight transport process.
2 Preliminaries
Biological and artificial neural networks have different structures and functionalitics, but they share certain factors in common, such that information is processed through hierarchical layers of neurons with nonlinear response function. In the current study, we consider a multi-layer feedforward neural network for pattern classification, fo I + Rd, parameterized by 0 = (W, b It takes input x e R" and outputs a vector y e Rd with L layers. Through a forward pass, the network computes hidden layer output by propagating the input through the network layers, follows:
01+1 = WWhh + b,, h ((0--))
where WT is the forward weights, by is the bias vector, and ó is the nonlinear activation function. In the first layer = O, h, x. We used rectified linear unit (ReLU) activation function. Φ(z)= = max r). In the last layer I = L- - 1, we used a softmax function, (x) = softmax(z) = (e /< e - Thus, the network outputs probability distribution over d classes After the forward pass, the amount of error is calculated by measuring the difference between the network output fo fx(
(x) and the target label y. We used the cross-entropy loss (5). which is defined as follows:
(()) = 1 d
yij log fx()))
where N the number of samples, is the number classes, and yıj is the target label for the i-th sample and the j-th class. The purpose of learning is to minimize the error L()). To achieve this, the network parameters 0 are adjusted by assigning credit to the weights that contribute to the error, which is known as the credit assignment problem.

---

2.1 Backpropagation and weight transport problem
To solve the credit assignment problem, backpropagation (BP) computes the gradient of errors with respect to the weights and uses it as teaching signal to modulate the aforementioned parameters. The gradient is calculated by the chain rule, with propagation from the output layer to the input layer, as follows:
ac = fo() - y. 51
ac = (W] ② (o1))
81
dog
do:
where dy is the enror signal at layer L, d' is the derivative of the activation function, and denotes the dlement-wiss product. The weight update rule given by
AW, =
where 1) is the learning rate. The backpropagation algorithm successfully solves the credit assignment problem, but it requires heavy computation the next layer to update the current weights. Notably, backpropagation is considered as biologically implausible, because it is impossible, in the brain, to transmit the synaptic weights from the next layer to the current layer. This is known as the weight transport problem 30 31
the complete information of the synaptic weights of
2.2 Feedback alignment
To address the weight transport problem, the idea of feedback alignment (FA) [32] was proposed as biologically plausible alternative to backpropagation. In feedback alignment, the backward synaptic feedback is replaced with random, fixed weight matrix B, in the feedback path, as follows:
dc
8o: (B184+1) o (':)..
(5)
The only difference between backpropagation and feedback alignment is the replacement of the transpose of the forward weights W with the fixed random feedback weights B, to calculate the error signal. The fact that the network can learn tasks from error teaching signals that calculated from random feedback is explained by the observation that the network modifies the forward weights W, to match the transpose of the feedback weights B, roughly during training. This makes the error teaching signal (5) similar to backpropagation 3 thus enabling the network to learn the task.
Random noise pretraining with feedback alignment
Algorithm 1 Random noise pretraining
1: procedure RANDOM NOISE PRETRAINING(netwurkk fo R - Rd)
2: for each epoch do for each batch do
3:
4:
x ~~ ((, o²), y ~ u((0 d 1)
> sampling random noise
5:
[()) = Loss (x),))
forward pass, equation (I. (2)
6:
8r ac
compute error
7:
for layer 1-1 to 0 do
8:
W, = Wt - 17 8141 h
>> update weights, equation compute error, equation
9:
8, B, 81+1) 3 (o))
During the developmental stage, spontaneous neural activity in the brain plays a critical role in shaping and refining neural circuits. Initially wired immature neural circuits undergo modifications of their connections through the processes of regulated cell formation, apoptosis, and synapse refinement through spontaneous neural activity 4 These pre-sensory activities and development processes are universal across sensory modalities, such as the visual [4 47 auditory [48, 49], and sensorimotor systems [50] [1]. We focus here few characteristics of spontaneous neural activity in the brain. Spontaneous neural activity not correlated to external stimuli but can refine and optimize neural circuits, before interaction with the external world can take place.
Here, we propose a type of random training that inspired by the spontaneous and prenatal neural activity in the brain pretrain the neural network (Algorithm I In every iteration, sampled

---

random noise inputs X from a Gaussian distribution (0, I) and random labels from discrete uniform distribution (0, Nexdout - 1), without any correlation. The network fo was initialized with random weights and trained with the feedback alignment algorithm. In this study, we examined the effects of random noise training on the subsequent learning processes in model neural networks to understand the potential benefits of pretraining with random noise in biological brains and whether this strategy is applicable to machine learning algorithms.
4 Results
4.1
Weight alignment to synaptic feedback during random noise training
b
Backoropagation (BP) and feedback alignmmet (FA)
Possible synaptic aircuit (Lilicrap al., 2016)
Training neural network with random input and label
Rondom label
Random inpurt
W.
Forward path
W.
W
W;
B
(Rumelhart el 1986)
0.6
W.
B,
Rato
0.4
0.2-
d
Backward path
0.2
6.1
B,4
(Lilicrap et al., 2016)
Intensity
Label
Loss across training random ingut and label
Alignment angle between weights and synaptic feedbacks
Averaged alignment angle with various input condition
Before
Individual neuron
After
3.5
Victor angle (deg W--)
100
102
160
Vedor angle (dlg WL-B)
120
100
Current study
Loos
90-
80
Gaussian
2.5
30-
30
30
60
Uniform
+10
101
20
40
80
0.09
10
90
40
0.09
0.5
1.5
No. training random imputs
Ratios
No. training random inputs
Ratics
Vartation of Inguts (o)
Figure 1: Weight alignment to randomly fixed synaptic feedback induced through random noise training. (a) Forward and backward pathways of backpropagation and feedback alignment. (b) Possible scenario of the feedback alignment algorithm biological synaptic circuit. Schematic of random training, where the input x and label are randomly sampled and paired in each iteration. (d) Cross-entropy loss during random training. (e) Alignment angle between forward weights and synaptic feedbacks in the last layer. (f) Alignment angle with various random input conditions.
To simulate neural network initially wired by random weights and fixed random synaptic feedback. we adopted network setting from the feedback alignment algorithm (Figure in which the weight transport problem can avoided through the use of fixed random synaptic feedback. Thus, unlike backpropagation, this process is considered possible to exist in biological neural networks with local synaptic connections (Figure [[)). We used a two-layer feedforward neural network with ReLU nonlinearity for classification, fo 784 + IR with 100 neurons in the hidden layer. By means of random noise training (Algorithm) T). we trained the neural network with random inputs sampled from Gaussian distribution (0, 7). with labels also randomly sampled independently (Figure Ipp.
We observed that the training loss decreased noticeably during random training, even in the absence of meaningful data and even when x and are randomly paired (Figure IT)). During the random training process, we focused on the alignment between the forward weights and the synaptic feedback. As described in the literature (32, the alignment of W, and B,, i.e., similarity between dpp and 8Fx, crucial for calculating the error teaching signal precisely. To evaluate the alignment, we used cosine similarity, which is widely used for measuring the distance between two vectors.

---

Definition, Given the forward weights W E Rmxn and backward weights B, E R=xm we measured alignment using cosine similarity. We define that W, and B, as aligned if the angle <(W)), )x, (Br); is significantly smaller than 90 degrees.
Notably, found that the weights of neurons are aligned the corresponding synaptic feedback weights during the random training process (Figure [p). We also observed that the angle between the forward weights and synaptic feedback of individual neurons in the hidden layer decreased asymptotically during random training. In a randomly initialized network, the alignment angle appeared be close 90º, demonstrating that the backward error signal is randomly distributed (Figure D left, alignment angle in an untrained network vs. 90", n 100, one-sample t-test, NS, P 0.492). However, after random training, the alignment angle decreased significantly, implying that the backward teaching signal becomes valid to back- propagaee errors (Figure I right, alignment angle in an untrained network vs. a randomly trained network, n = 100, two-sample t-test, P < 0.001). We confirmed that this is not simply due to input bias under a particular condition but reproduced robustly with various input conditions (Figure TT. These results suggest that neural networks can pre-learn how to back- paok-propagate errors through random noise training.
4.2 Pretraining random noise enables fast learning during subsequent data training
c
Effect of random Iraining on data learming
Accuracy during data training
Weight alignment
Random
(MNIST)
1.0
w/ (FA)
Vector angle (deg. Wr-B)
120
Random
Data
0.8-
w/o (FA)
100
pretraining
wi
3'3
eccurary
0.6
Convergence speed
1.0
60
0.4
0.9
0.2
0.8-
40
wlo random pretraining
3'3
0.7
10
20
30
40
50
60
100
150
200
Epochs
Epochs
Trajectory of weights in latent space
Distance between and latent space
Order dependence
Distance between W
random training
latent space
57
PC2 (25 (1%)
2.5
Data training
W
Random
training
Dalance (IV) at)
PC2 (23, 11
2.5
Data
W
Random
Distance (IWT-B1) BI)
Data
Random
NS
Data
Random
Random
.255
Data
-5
-5
255
2.5
20
40
60
80
100
-2.5
2.5
50
100
150
200
PC1 (72. 595)
Epochs
PC1 (72.59%)
Epoche
Figure 2: Effect of random noise pretraining on subsequent data training. (a) Design of the MNIST classification task to investigate the effect of random training. (b) Test accuracy during the training process, where the inset demonstrates the convergence speed of each training method, calculated by (c) Alignment angle between weights and synaptic feedback across random training and data training. (d) Trajectory of weights (W1) toward synaptic feedback (B,) in latent space obtained by PCA for random and data training. (e) Distance between the weights (W1) and the synaptic feedback (B1). (f) Order dependence of the trajectory of the weights (W1). (g) Distance between the weights (W,) and the synaptic feedback (B,) for different orders of random and data trainings.
the AUC of the test accuracy.
Next, we compared networks with and without random pretraining in terms of subsequent data training outcomes (Figure (h) We trained the networks using a subset of the MNIST dataset [52], a widely used benchmark for image classification. We found that a randomly pretrained network learns the data more quickly and achieves higher accuracy compared a network that is not randomly pretrained (Figure (2p). To quantify the speed of learning, we calculated the area under the curve (AUC) of the test accuracy and found that the convergence of the randomly pretrained network is significantly faster than that in the network without random pretraining (Figure Db inset, w/o vs. w/ random pretraining (FA), Nae = 10, t-test, *P < 0.001). Notably, the convergence speed of the randomly pretrained network appeared comparable to that of the network trained with backpropagation (Figure
5

---

2p. inset, w/ random pretraining (FA) vs. BP, fla 10, t-test, *P < 0.001). We also observed that the weight alignment gap between untrained and randomly pretrained networks is maintained during data training (Figure As a result, at the end of the data training step, the alignment angle of the randomly trained network was significantly smaller than that of the untrained network (Figure Z: w/o vs. w/ random pretraining, n 100, t-test, P < 0.001). This result suggests that a combination of random pretraining and subsequent data training can enhance the weight alignment, which leads to more precise error teaching.
To understand the weight update dynamies by random and data training, we visualized the trajectory of weights in latent space as obtained by principal component analysis (PCA) [53] (Figure 2d). We conducted PCA on the weights of the last layer (W1) for the random and data training conditions. First, we confirmed that in both random and data training, the weights become closer to synaptic feedback (Figure P. untrained vs. data training, net = 10, t-test, *P < 0.001; untrained vs. random training, nee TO, t-test, P < 0.001; random vs. data training, net = 10, t-test, *P < 0.001). Notably, we observed that the updated trajectory of weights by random training and data training have different directions in the principal component space and that the effects of random training depend on the order of the random and data training (Figure 2 g) - the enhancement of weight alignment was more significant when data training was performed after random training (Figure 2g random training vs. random and data training. Raat = 10, t-test, *P < 0.001) compared to when training is done in reversed order (Figure data training vs. data and random training, neet 10, t-test, NS, P= 0.999). Particularly, when we trained the network with data first, subsequent random training could not move the weights; thus, the weights did not become closer to synaptic feedback. This result suggests that weight alignment by random noise pretraining cannot be replaced by data training and that it is crucial to perform random training prior to data training.
b
d
MNIST
F-MNIST
CIFAR-10
CIFAR-100
STL-10
9
0.9
0.
03
0.5
BP
Dapth
Depth
Depth
Accuracy
Accurnty
Accurecy
Accurscy
0.2
Accurecy
C4
0.56
0.65
0.1
92
0.1
0.3
wi random (FA)
80
100
100
random (FA)
Epochs
Epoche
Epoche
0.92
0.8
0.2
5
Depth
Dopth
Depth
Dopth
Dapth
Figure 3: Comparison of model performance across different image datasets and network depths. (a-c) Final accuracy after convergence. Experiments were conducted with networks of varying depths on different tasks: (a) MNIST, (b) Fashion-MNIST, (c) CIFAR-10, (d) CIFAR-100, and (e) STL-10.
Table 1: Performance of the two-layer model for each dataset (MNIST. Fashion-MNIST. CIFAR- 10, CIFAR- 100, STL-10). Each performance value (%) is presented as the mean + standard deviation from three trials. Extended results with various model depths can be found in the Appendix.
MNIST
F-MNIST
CIFAR-10
CIFAR-100
STL-10
BP
97.82 + 0.03
88.87 = 0.03
54.01 + 0.20
24.55 + 0.10
42.72 + 0.20
w/o
97.26
0.07
87.47 + 0.25
50.54 1 0.22
20.17 1 0.30
36.21
0.91
FA
w/
97.76 1 0.07
88.26 1 0.07
53.58 1 0.12
24.45 1 0.10
41.01 1: 0.16
AACC
0.49 + 0.06
0..99 + 0.31
3.04 + 0.20 M.28 0.38
M.81 0.86
Next, we further investigated the model's classification performance across various image datasets and network depths (Figure 3 [.. In earlier experiments with two-layer networks and MNIST, we showed that random noise pretraining enhances both the learning speed and accuracy of networks to levels comparable with backpropagation. We extended these experiments to networks of varying depths (Figure 33) and confirmed that the benefits of random noise pretraining generalize to deeper networks. Additionally, we evaluated performance across various image datasets, including Fashion- MNIST [54] (Figure Bp), CIFAR-10 (55) (Figure 3b), CIFAR-100 (55) (Figure 3)), and STL-10

---

We found that random noise pretraining significantly narrows the performance gap between feedback alignment and backpropagation across different datasets and depths. These results suggest that pretraining with random noise can serve as general strategy for improving the performance of neural networks trained with feedback alignment algorithms, making it comparable to backpropagation.
56 (Figure 3)).
4.3 Pre-regularization by random noise training enables robust generalization
a
b
C
d
Untrained vs. random noise trained network
Readout
Singular value spectrum of weights
Dimensionality of weights across random training
probabiiity distribution
x10*
Untrained
Intialization
Denstly
10
Untrained
W2
Effective rank
to+
0
Random
0.5
60
100
-102
Normalized
Random trained
07
Danelty
10P
W2
Random trained
0.5 Probabillty
10
40
N-layer network
Soried Indax
No. training random inputs
g
Trsining vs. test loss
Training set size vs. generalization
Effective dimensionallit learned feature
Network depth Feature dimensionalitt
Generaltzatin
oriice
Girmm
Generoiization ertor
0.78
0.6
Effective rank
wil
Toxt loxs
0.5
1.5
w/o Wi
Effective
0.25-
random
0.5
0.75
100 200
400 800 1800
Feature
Training loss
Traning set size
Depth
Figure 4: Pre-regularization by random noise training enhances generalization (a) Untrained network and pre-regularized network through random noise training. (b) Distribution of the readout probability. (c) Singular value spectrum of the forward weights. (d) Effective rank of forward weights during random noise training. (e) Generalization error between the training error and test error (training set size: 1600, network depth: 3). (f) Generalization error for various training set sizes (network depth: 3). (g) Effective dimensionality of the Gram matrix, the cosine similarity of feature vectors across neurons (training set size: 1600, network depth: 3). (h) Effective dimensionality of the Gram matrix for various network depths (training set size: 1600).
Next, we compared the difference between an untrained network and a randomly trained network in terms of their activation and weight (Figure 4a). First, we found that the readout probability of the untrained network is distributed over a wide range (Figure top, untrained vs. chance level, 1000, Wilcoxon signed-rank test, P < 0.001), whereas that of the randomly trained network is well regularized, close to the chance level (Figure Db. bottom, random noise trained vs. chance level, 1000, Wilcoxon signed-rank test, NS, P 0.096). We also observed that the singular value spectrum of forward weights changes significantly by random training (Figure 4 such that a small portion of singular values become dominant in the randomly trained network. To measure the effective dimensionality of the weights quantitatively, we used the effective rank of the weights.
Definition. Given matrix A E mmx" is decomposed into A UEVT by singular value decompo-
sition (SVD), the singular values are { minn(m.n) sorted in descending order. The effective rank is defined as the Shannon entropy of the normalized singular values,
E; ã. log o, where 5. = - Σ, og- Without loss of generality, we used the effective rank as the exponential of [57].
We observed that the effective rank of forward weights decreased significantly during random training (Figure 4.), implying that random training regularizes the weights initially and predisposes the network to learn simple solutions of low rank. Given the notion that low-rank solutions show better generalization performance outcomes, we hypothesized that this preuragulai by random training enables robust generalization during subsequent data training by inducing low-rank solutions.
7

---

To test the generalization ability of the network, we measured the gap between the training error and the test error during subsequent data training. We observed that the generalization error was noticeably lower in randomly pretrained network compared to a network trained solely on the data (Figure F and that this tendency was maintained with variations of the training set size (Figure f w/o vs. w/ random pretraining, Pnet 10, t-test, P < 0.001). This result suggests that pre-regularizatio by random pretraining can enable robust generalization during subsequent data training.
Next, we compared the representation of learned features in a randomly pretrained network and network trained without random pretraining. We used the Gram matrix, defined as the cosine similarity of feature vectors across neurons. Notably, we found that the effective rank of the Gram matrix was significantly lower in a randomly pretrained network compared an untrained network after subsequent data training (Figure As. w/o vs. w/ random pretraining, net = 10, t-test, *P < 0.001) and that this tendency was maintained regardless of the network depth (Figure f 1 w/o vs. w/ random pretraining, nee = 10, t-test, P < 0.001). This finding suggests that peellaualaiination by random training can enable networks to learn simpler solutions, leading to better generalization performance during subsequent data training.
b
Training in-distribution data (MNIST)
Out-af-distiiutiion generalization lest transformed MNIST
OOD test benchmark
Translated
Scaled
Rotsted
USPS
NNIST
Translated Stated Rotsted
0.9
USPS
0.6
random
0.25
w/ rendom
Test accuracy
5
Test
wel
random
3
3
3
3
random
OOD
0.75
OOD lest
02
Network
training
w/o
V/
Figure 5: Robust generalization of "fittoofddistriuution"" tasks in randomly pretrained networks. (a) Training in-distribution data (MNIST) in untrained and randomly pretrained networks. (b) Out-of- distribution generalization tests transformed MNIST. (c) Out- -istritiinttion generalization tests on USPS dataset.
We also tested the generalization performance of the networks for "outtof-disttriutton" tasks by training the network with the MNIST dataset and testing it with various distrr--dittibuttion tasks (Figure 5p). First, we generated a MNIST dataset of translated, rotated, and scaled images and then used these images as out-of-distribution tasks (Figure left). We observed that randomly pretrained network showed significantly higher test accuracy on out-of-distribution tasks than a network trained without random pretraining (Figure 5p. right, w/o vs. w/ random pretraining, Uhat = 10, t-test, *P- < 0.001). We also observed that the randomly pretrained network showed higher test accuracy the USPS dataset [58), which is widely used benchmark dataset for out- of-distribution tasks (Figure 5t. w/o vs. w/ random pretraining, nne = 10, t-test, *P < 0.001). This result suggests that pre rreer-gularizatonn by random pretraining enables robust out-of generalization during subsequent data training.
4.4 Task-agnostic fast learning for various tasks by a network pretrained with random noise
Lastly, we examined whether random training is generally beneficial for various tasks. We compared the task adaptation capacity of an untrained network and a randomly pretrained network on three tasks: (1) MNIST classification (52). (2) Fashion-MNIST [54), (3) Kuzushij-MMIIST [59] (Figure 64, left). To measure the ability of fast adaptation to various tasks quantitatively, we computed the
lota-loss, suggested in previous study of meta-
Definition. Given the task distribution T (Tr) the meta-loss of network fo is defined as (f) = =ET Cr, (6)), where Cy, (8)) denotes the loss of the task Tr and 6 is the adapted parameter for T; [60].
We observed that the meta-loss decreased gradually during the random training process (Figure Gp. right). Considering that the training was solely performed with random inputs and labels on the three tasks to measure the meta-loss, this result suggests that networks can learn how to adapt to various tasks without any -pec-ic data. Next, we trained the untrained networks and randomly pretrained
8

---

b
Task distribution
Meta-loss during random training
Trajectory of weights in lalent during adaptation various tasks
Task-agnosiin fast adaptation
0.6
Limaaieed
Tank
case presoined
1,0
Teak
protooited
Tink (MhitST)
0.8
Took
Metoe oos
0.4
W
Accuracy
0.6
(F-MNIST)
Took MNSIT)
i>>
0.2
0.2
Trsc
disoss
-10*
PC2 (14.5%) 5%)
Untrained
50
PC1 (53.9%)
No. training rendom inpuls
Epodhs
Figure 6: Task-age fask-agnostic fast learning for various tasks in randomly pretrained networks. (a) Three tasks used to test the task-agnostic property of random training, showing the meta-loss during the random training process. The meta-loss is calculated from the sum of the losses measured during adaptation to each task. (b) Trajectory of weights in the latent space for adaptation each task of an untrained network and a randomly pretrained network. (c) Adaptation to each task of an untrained network and a randomly pretrained network.
networks on each task separately. We conducted PCA on the weights of the last layer (W2) to visualize the trajectory of weights in latent space during the adaptation to each task. We observed that the trajectory of weights during random noise training moves closer synaptic feedback (B2), which makes the adaptation to each task more efficient (Figure 6p). This suggests that random training is task-agnostic but provides efficient and fast learning subsequent learning. Lastly, we compared the adaptation to each task in an untrained network and randomly pretrained network. We observed that the randomly trained network showed remarkably fast adaptation each task compared the untrained network (Figure(6p, w/o vs. w/ random pretraining, fnet = 10, t-test, *P < 0,001). These results highlight the task aggnssicc property of random training, which enables networks to quickly adapt to various tasks.
5 Discussion
We showed that random noise pretraining enables neural networks to learn quickly and robustly without weight transport. This finding bridges the gap between biologically plausible learning mechanism and the conventional backpropagation algorithm, as the symmetry of forward and backward weights can easily be achieved by random noise pretraining. Moreover, the results here provide new insight into the advantage of random training as a means of preconditioning network for robust generalization.
peroo-backpropagation without weight transport. Early work in neuroscience identified basic learning rules, such as Hebbian learning [61] and ppike-timing-dependee Although these rules have been experimentally observed and are thus biologically plausible, they are not sufficient to explain the brain's 's remarkable learning ability thoroughly [65] 66]. On the other hand, while the backpropagation algorithms used in artificial neural networks have shown impressive learning capabilities, they are considered biologically implausible due to the weight transport problem (23. Our results provide a new perspective on this issue, bridging the gap between these training rules. We showed that symmetry among forward and backward weights. which is necessary to proo-pro errors, can readily developed by learning random noise, similar to that during the brain's 's prenatal stage. Our findings suggest probable scenario for significantly narrowing the performance gap between previously suggested biologically plausible learning rules and backpropagation.
plasticity (STDP) 64)
Recent studies explored noise as a biologically plausible mechanism to enhance learning efficiency without the need for weight transport. For instance, the weight mirror algorithm (36] uses noisy firing to align feedback weights with forward weights. Similarly, phaseless alignment learning [40] leverages layer-wiss wise noise as an additional information carrier to achieve weight alignment. While these approaches have been reported to outperform traditional feedback alignment algorithm, merely incorporating noise into existing feedback alignment algorithms yields no improvement in learning performance [40. contrast, our results indicate that exposure to random noise "before encountering real data" significantly enhances vanilla feedback alignment. Unlike previous approaches, our

---

strategy utilizes random noise for pre-condition the network, preparing it to learn more effectively. This aligns with biological observations that neural noise predominates in the early stage of brain development [67. even prior to exposure to external stimuli. It is important to note that our proposed method is not limited to the feedback alignment algorithm; pretraining with random noise could be beneficial for other algorithms, which we intend to explore in our follow-up studies.
Pre-regularization for robust generalization. We suggest a task- and model-agnostic pretraining strategy that involves simply training the network with random noise. Notably, our results here show that random noise training can enhance the learning efficiency and generalization ability of the network, for which various tricks and techniques have been proposed to improve. We found that pretraining on random noise reduces the effective dimensionality of the weights, facilitating the learning of low-rank solutions for various tasks. Previous studies on generalization have shown that the low-rank bias of neural networks plays a crucial role in their generalization ability, a finding we have confirmed in this study [6 Additionally, our results highlight that random noise pretraining functions as a form of meta-learning, enhancing the network's 's ability to adapt rapidly to different tasks. In contrast to previous approaches that utilized data from diverse task distributions 660) our method achieves similar effects by merely training with noise. is important to note that the straightforward strategy of random noise training can significantly influence the network's 's learning dynamics- - effects that previous machine learning techniques have sought to achieve. This approach may reflect potential strategy employed by the brain to attain notable generalization capabilities. Furthermore, it suggests a novel pretraining strategy for artificial neural networks.
Insights into developmental neuroscience. Unlike artificial neural networks, the brain is ready to learn before encountering data. the early developmental stage before eye-opening, spontaneous random activity emerges in the brain, which is considered essential for the normal development of early circuits [IH4]. However, the functional advantage of learning from random noise before external sensory inputs remains unclear. Our study provides plausible scenario that the brain utilizes spontaneous random activity to pre-align the synaptic weights for error learning and pre-regularization of synaptic connections for robust generalization. Specifically, we showed that random training reduces the effective dimensionality of the weights, which can be considered as a form of pruning, as previous neuroscience studies reported that the brain's 's synaptic connections are pruned substantially during development, particularly dependent on spontancous activity [48). Despite the fact that the present study is based model neural networks, the results here are consistent with range of experimental findings in developmental neuroscience.
6 Broader impacts and limitations
Broader impacts. Feedback alignment algorithm and its advanced modifications without weight transport are motivated by the need to suggest a learning method that is compatible with deep neural networks with biological plausibility. It can be useful particularly when implemented in physical circuits, as nowadays deep learning without weight transport is utilized in neuromorphic chip engineering. Given that backpropagation requires dynamic access to memory due to weight transport, it is not free from the issue of energy inefficiency. Our results are not solely limited to demonstrating the role of biological prenatal learning but can also be extended for more practical purposes; for instance, it is promising strategy for the preconditioning of neuromorphic chips.
Limitations. Although our study offers new perspective on pretraining neural networks with random noise, some limitations must be considered. The current study focuses on results using feedforward neural networks with feedback alignment algorithms. Regarding the scalability of the method, further investigation is needed for other architectures, such as convolutional neural networks. Notably, we achieved meaningful results showing that pretraining with random noise can also benefit standard backpropagation, which will be further explored in our follow-up studies, which will emphasize random noise pretraining as a general strategy for neural network training.
7 Code availability
Python 3.11 (Python software foundation) with PyTorch 2.1 was used to perform the simulation. The code used in this work is available at /attps://githus com/ hwtp//a//aaaoooooooooooooooooooo
10

---

Acknowledgements
This work was supported by the National Research Foundation of Korea WNRF-202RR/A2C300022 A2C3008991 to S.P.), the Singularity Professor Research Project of KAIST (to S.P.), the Institute of Information & communications Technology Planning & Evaluation (UIP)) (RS-2023-00233251,, RS-2019- to S. W.L.) and the Ministry of Science & ICT(RS-2024-00446666 to S. S.W.L.).
References
[1] Lucia Galli Lamberto Maffei. Spontaneous impulse activity rat retinal ganglion cells in prenatal life. Science, 242(48755;90-91, 1988.
[2] James B Ackman, Timothy Burbridge, and Michael C Crair. Retinal waves coordinate patterned activity throughout the developing visual system. Nature, 490(7419):219 19):9999291225 225, 2012.
[3] Noelia Leticia Pérez-Saiz, Henrik Gezelius, Anton Filipchuk, Miguel Valdeolmillos, and Guillermina I López- Bendito. Prenatal activity from thalamic neurons governs the emergence of functional cortical maps in mice, Science, 364(64444:988-9990, 2019.
Alejandro Cempere-Ferränder Teresa Guillamón- Vivancos, Francisco J Martini,
[4] Francisco J Martini, Teresa Guillamón-Vivvancos, Verónica Moreno-Juan, Miguel Valdeolmillos, and Guillermina López-Bendito. Spontaneous activity in developing thalamic and cortical sensory networks. Newron, 109(16):251-2554 2021.
[5] Oriane Blanquie, Jenq-Wei Wei Yang, Werner Kilb, Salim Sharopov, Anne Sinning, and Heiko Luhmann. Electrical activity controls area-specific expression of neuronal apoptosis in the mouse developing cerebral cortex. Elife, 6:e27696. 2017.
[6] Larry Katz and Carla Shatz. Synaptic activity and the construction of cortical circuits. Science, 274(5290): 1333-11131133-1188, 1996.
[7] Werner Kilb, Sergei Kirischuk, and Heiko J Luhmann. Electrical activity patterns and the functional maturation of the neocortex. European Journal of Neuroscience, 34(10):1677-166, 2011.
[8] Nobuhiko Yamamoto and Guillermina López-Bendito. Shaping brain connections through spontaneous neural activity. European Journal of Neuroscience, 35(10):595-1664, 2012.
Heiko J Luhmann, Anne Sinning, Jenq- Wei Yang. Vicente Reyes-Puerta, Maik C. Stüttgen, Sergei Kirischuk, and Werner Kilb. Spontaneous neuronal activity developing neocortical networks: from single cells to large-scale interactions. Frontiers in neural circuits, 10:40, 2016.
[10] Nicolas Heck, Antje Golbs, Therese Riedemann, Jyh-Jang Sun, Volkmar Lessmann, and Heiko J Luhmann. Activity-dependent regulation of neuronal apoptosis in neonatal mouse cerebral cortex. Cerebral cortex, 1eo:::133-1999 2008.
[11] Andrew D Huberman. Colenso M Speer, and Barbara Chapman. Spontaneous retinal activity mediates development of ocular dominance columns and binocular receptive fields in vl. Neuron, 52(2):247-254, 2006.
[12] Gwangsu Kim, Jaeson Jang. Seungdae Back, Min Song, and Se-Bum Bum Paik. Visual number sense in untrained deep neural networks. Science advances, 7()xcabd6127, 227, 2021.
[13] Seungdae Baek, Min Song, Jaeson Jang, Gwangsu Kim, and Se-Bum Paik. Face detection in untrained deep neural networks. Nature communications, 12(1):3328, 2021.
[14] Jeonghwan Cheon, Scungdac Back, and Se-Bum Paik. Invariance of object detection in untrained deep neural networks. Frontiers Computational Neuroscience, 16:1030707, 2022.
[15] Hyeonsu Lee, Woochul Choi, Dongil Lee, and Se-Bum Paik. Comparison visual quantities untrained neural networks. Cell Reports, 42(8), 2023.
[16] Jinwoo Kim, Min Song, Jacson Jang, and Se- Bum Paik. Spontancous retinal waves can generate long-range horizontal connectivity in visual cortex. Journal of Neuroscience, 40(34):6584 6599, 2020.
[17] Mark V Albert, Adam Schnabel, and David J Field. Innate visual learning through spontaneous activity patterns, PLoS Computational Biology, 1000110 2008.

---

[18] Wulfram Gerstner, Richard Kempter, Leo Van Hemmen, and Hermann Wagner. A neuronal learning rule for sub-millisecond temporal coding. Nature, 38366555)776-7,, 1996.
[19] Shigeru Kitazawa, Tatsuya Kimura, and Ping- Bo Yin. Cerebellar complex spikes encode both destinations and errors arm movements, Nature, 392(6675):494 497, 1998.
[20] Georg Keller and Richard HR Hahnloser. Neural processing of auditory feedback during vocal practice in songbird. Nature, (7226))) 2009.
[21] Marvin Minsky, Steps toward artificial intelligence. Proceedings of the IRE, 49(1):8 30, 1961.
[22] Blake A Richards and Timothy P Lillicrap. Dendritic solutions to the credit assignment problem. Current opinion in neurobiology, 54:28-36, 2019.
[23] Timothy P Lillicrap. Adam Santoro, Luke Marris, Colin J Akerman, and Geoffrey Hinton. Backpropagation and the brain. Nature Reviews Neuroscience, 21(6):335-33 336, 2020.
[24] David E. Rumelhart, Geoffrey F. Hinton, and Ronald J. Williams. Learning representations by back- propagating errors. Nature, 323:533-536, 10 1986.
[25] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[26] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521 521((553) 2015.
[27] Qianli Liao, Joel Leibo, and Tomaso Poggio. How important is weight symmetry in backpropagation? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
[28] Andrew M Saxe, James I. McClelland, and Surya Ganguli. Fxact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv: 131.6/231, 2013.
[29] Konrad P Körding and Peter König, Supervised and unsupervised learning with two sites of synaptic integration. Journal of computational neuroscience, 1 11:0072215, 2001.
[30] Stephen Grossberg. Competitive learning: From interactive activation to adaptive resonance. Cognitive science, 11(1):23-66 63, 1987.
[31] Francis Crick. The recent excitement about neural networks. Nuture, 337(6203) 1989.
[32] Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin Akerman. Random synaptic feedback weights support error backpropagation for deep leamnin.. Nature communications, 7(1):13276, 2016.
[33] Sergey Bartunov, Adam Santoro, Blake Richards, Luke Marris, Geoffrey E Hinton, and Timothy Lillicrap. Assessing the scalability of polooically-motivaate deep learning algorithms and architectures. Advances in neural information processing rystems, 31, 2018.
[34] Navid Shervani- Tabar and Robert Rosenbaum. Meta-leaming biologically plausible plasticity rules with random feedhack pathways. Nature Communicationss 14, 12 2023.
[35] John F Kolen and Jordan B Pollack. Backpropagation without weight transport. Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN 94), volume 3, pages 1375-1380, IEEE, 1994.
[36] Mohamed Akrout, Collin Wilson, Peter Humphreys, Timothy Lillicrap, and Douglas B Tweed. Deep learning without weight transport. Advances in newral information processing systems, 32, 2019.
[37] Arild Nøkland. Direct feedback alignment provides lcarning in deep neural networks. Advances in neural information processing systems, 29, 2016.
[38] Yali Amit. Deep learning with asymmetric connections and hebbian updates. Frontiers in computational neuroscience, 13:18, 2019.
[39] Pau Vilimelis Aceituno, Matilde Tristany Farinha, Reinhard Loidl, and Benjamin F Grewe. Learning cortical hierarchies with temporal hebbian updates. Frontiers in Computational Neuroscience, 13:1136010. 2023.
[40] Kevin Max, Laura Kriener, Garibaldi Pineda Garcfa, Thomas Nowotny, Ismael Jaras, Walter Senn, and Mihai A Petrovici. Learning efficient backprojections across cortical hierarchies in real time. Nature Machine Intelligence, pages 1-12, 2024.
12

---

[41] Hermann Bulf. Scott P Johnson, and Eloisa Valenza. Visual statistical learning in the newborn infant. Cognition, 121(1):127-122, 2011.
[42] Ingrid B Johanson and WG Hall. Appetitive learning in l-day-old rat pups. Science, 205(4404):419 421, 1979.
[43] Tiffany M Field, Debra Cohen, Robert Garcia, and Reena Greenberg. Mother-stranger face discrimination by the newborn Infant Behavior and development, 191):19-25, 1984.
[44] IWR Bushneil, F Sai, and Jim T Mullin. Neonatal recognition of the mother's face. British journal of developmental psychology, 7(1):3115, 1989.
[45] Irving John Good. Rational decisions. Journal of the Royal Statistical Society: Series B (Methodological), 14(1):107-11,, 1952.
[46] Richard H Masland, Maturation of function in the developing rabbit retina. Journal of Comparative Newrology, 175(3):275-28) 286, 1977.
[47] James B Ackman and Michael C Crair. Role of emergent neural activity in visual map development. Current opinion in neurobiology, 24: 14:166-175, 2014.
[48] Nicolas X Tritsch, Eunyoung Yi, Jonathan Gale, Elisabeth Glowatzki, and Dwight Bergles. The origin of spontaneous activity in the developing auditory system. Nature, 450(7166):50-55 55, 2007.
[49] Helen Kennedy. New developments understanding the mechanisms and function of spontaneous electrical activity in the developing mammalian auditory system. Journal of the Association for Research in Otolaryngology, 13:437-44 445, 2012.
[50] Viktor Hamburger, Eleanor Wenger, and Ronald Oppenheim. Motility the chick embryo in the absence of sensory input. Journal of Experimental Zoology, 162(2):133-15)) 159, 1966.
[51] Scott R Robinson, Mark $ Blumberg, Maura $ Lane, and Lisa Kreber. Spontaneous motor activity in fetal and infant rats is organized into discrete multilimb bouts. Behavioral neuroscience, 114(2):328, 2000.
[52] Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE signal processing magazine, 29(6): 141 142, 2012.
[53] Svante Wold, Kim Esbensen, and Paul Geladi. Principal component analysis. Chemometrics and Intelligent Laboratory Systems, 2:37-52, 8 1987.
[54] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for henchmarking machine learning algorithms. arXiv preprint /7X 2017.
[55] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[56] Adam Coates. Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature lcarning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215-223. JMLR Workshop and Conference Proceedings, 2011.
[57] Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In 2007 15th European signal processing conference, pages 606-610. IEFE, 2007.
[58] Jonathan J. Hull. database for handwritten text recognition research. IEEE Transactions on pattern analysis and machine intelligence, 16(5):550-554, 1994.
[59] Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese litcrature. arXiv preprint arXiv:/8/2.01718, 2018.
[60] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-lcarning for fast adaptation of deep networks. In International conference on machine learning, pages 1126-1135. PMI.R, 2017.
[61] Donald Olding Hebb. The organization of behavior: A neuropsychologicall theory. Psychology press. 1949.
[62] Robert C Froemke and Yang Dan. Spple-tining-deeeeddett synaptic modification induced by natural spike trains. Nature, 43366479) 2002.
[63] Yang Dan and Mu-ming Poo. Spike deiinnede plasticity of neural circuits. Neuron, 44(1):23-3) -300 2004.
13

---

[64] Natalia Caporale and Yang Dan. Spike demineede plasticity: a hebbian learning rule. Annu. Rev. Neurosci., 31:25-46, 2008.
[65] H Sehastian Seung. Learning in spiking neural networks by reinforcement of stochastic synaptic transmis- sion. Neuron, 40(6):1063-1072 2003.
[66] Yiting Dong, Dongcheng Zhao, Yang Li, and Yi Zeng. An unsupervised sidp-based spiking neural network inspired by biologically plausible learning rules and connections. Neural Networks, 165:799-808, 2023.
[67] Ann M Skoczenski and Anthony M Norcia. Neural noise limitations on infant visual sensitivity. Noture, 66666))))7777770 1998.
[68] Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems, 32, 2019.
[69] Minyoung Huh, Hosscin Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, and Phillip Isola. The low-rank simplicity bias in deep networks. arXiv preprint vrXiv22103.10422 2021.
[70] Aristide Baratin, Thomas George, César Laurent, R Devon Hjelm, Guillaume Lajoie, Pascal Vincent, and Simon Lacoste-Julien. Implicit regularization via neural feature alignment. In International Conference on Artificial Intelligence and Statistics, pages 2269-2277. PMLR, 2021.
[71] Dan Zhao Combining explicit and implicit regularization for efficient learning in deep networks. Advances in Neural Information Processing Systems, 35:3024-3038, 2022.
[72] Parth Natekar and Manik Sharma. Representation based complexity measures for predicting generalization in deep learning. arXiv preprint 2rXIv:2011202775.2000 2020.
14

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We clearly state the main claims and contributions in both the abstract and the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of work in the final section of the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
Our paper does not contain any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide all the necessary information to reproduce the main experimen- tal results. The experimental settings and parameters are specified in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provide the URL to the code repository for reproducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ipss.cc/pill wttpcccppppppppeeeeeeeeeeeeeeeee httpe//gggeeee/eeeeebbiisssssss /Cc////oddiiisssseeeeeee ssiiiiiisiiiiyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
The training and testing details are provided in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We reported error bars and specified the significance levels in the figures and their corresponding texts.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative еттоr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We indicated the specifications of the computing resources used in the experi- ments. Additionally, we want to note that our simulation does not require high computational resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't Itdn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have reviewed the NeurlPS Code of Ethics and ensured that our research conforms it in all respects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discussed the potential societal impacts of our work in the final section of the paper. However, since our research addresses fundamental questions in computational neuroscience, we not anticipate any negative societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Our paper presents no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We used publicly available datasets and properly cited the original sources.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, batasets.papersstt paperswi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide the URL to the code repository. which includes access to the code and models along with accompanying documentation.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve research on human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "UUNgjbLpX8", "reviewer": "Reviewer_kKCk", "summary": "The paper explores how pretraining neural networks with random noise can improve learning efficiency and generalization without relying on weight transport, inspired by spontaneous neural activity in developing biological brains.\n\nKey Findings:\n1. Random noise training aligns forward weights with synaptic feedback, enabling more precise error backpropagation without weight transport.\n2. Networks pretrained with random noise learn subsequent tasks faster and achieve performance comparable to backpropagation.\n3. Random noise pretraining acts as a form of regularization, reducing the effective dimensionality of weights and leading to more robust generalization.\n4. Pretrained networks show better performance on out-of-distribution tasks and adapt more quickly to various tasks.", "questions": "1) Please tune baselines. One can fool oneself very easily on the datasets and small networks you study. First, please grid search all hyperparameters in a reasonable range for the backdrop baseline and more crucially for the FA from scratch baseline. Try all common initialization schemes, and tune for them learning rate and batchsize. Please study carefully initialized random feedback weights. \n2) Please study the following crucial baseline: Initialize the feedback weights B with W transpose. Your claim is essentially that when you do this you train faster than with B random. I am highly doubtful of this, and therefore your claims. I agree that your pretraining might align the weights but I am quite confident that this will not lead to better optimization. In the beginning of training weights (so only W in your case) change quickly and therefore even initial perfectly aligned weights will not bring help much. \n3) How can you overcome the limitations of depth? What about layerwise local losses? If the result only hold for one-hidden layer networks, I am not confident to have this accepted at Neurips.", "limitations": "Please mention that your results only apply to 1-hidden layer neural networks.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper investigates, to the best of my knowledge, a novel idea for pretraining neural networks to improve downstream feedback alignment. The paper is well written and easy to follow, experiments are nicely presented.", "weaknesses": "Unfortunately I think the evidence the paper presents is not enough to back up their claims. The bulk of the experiments only studied one-hidden neural network trained on MNIST. Results and claims will most likely be overestimated and will not hold in general. The paper also does not provide any theoretical insights why this initial pre-training might help.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The bulk of the experiments only studied one-hidden neural network trained on MNIST\n\n**: Please find our response in Global Response, Issue 1**\n\n**We expanded our experiments by increasing the depth of the network and confirmed that even with deeper architectures, the accuracy of the randomly noise-pretrained network consistently surpasses that of the baseline FA** (Response Figs 1, 3 and Response Table 1).\n\nFurthermore, beyond MNIST, we assessed the impact of random noise training on fMNIST, CIFAR10, CIFAR100, and STL10, consistently achieving higher accuracy (see Response Figs 2, 3 and Response Table 1).\n\n> Please tune baselines.\n\n**We are confident that our results were not obtained from different baselines as you expressed concerns.** \n- Please note that our results are based on hyperparameters typically chosen for effective learning in both FA and BP. We conducted and compared baseline FA (FA w/o), FA with random noise pretraining (FA w/), and BP under carefully controlled conditions.\n- We controlled hyperparameters such as batch size and learning rate, as well as initial forward weights, to isolate differences attributable to different learning algorithms. Specifically, in a single trial, we duplicated one randomly initialized network into three, starting with identical weights. These networks were then trained using the same hyperparameters but different learning methods and algorithms (FA w/o, FA w/, BP).\n- Additionally, both baseline feedback alignment and backpropagation showed significant performance in benchmark tests. We have presented these results numerically (Response Table 1), which we believe further supports that our findings were obtained under appropriate training conditions.\n- Regarding weight initialization, we primarily utilized He initialization since we employed ReLU as the non-linear function in the network. As backpropagation utilizes W for backward computation, we ensured that B in feedback alignment followed the same statistical characteristics as W.\n\n**Through further experiments, we demonstrated that varying hyperparameters (batch size, learning rate) and initialization within a reasonable range consistently reproduced our results** (Response Figure 5). Due to the 1-page PDF limit for the global response, we were unable to include it, but we confirmed that our results are reproducible when using tanh as the non-linear function.\n\n> Please study the following crucial baseline: Initialize the feedback weights B with W transpose.\n\n**As you suggested, we initialized B to match the transpose of W and conducted learning** (Response Fig 6). This approach resulted in gradients and alignment fairly similar to backpropagation during training. Thus, **we confirmed that this setup does not hinder learning and instead demonstrates efficiency comparable to backpropagation**.\n\n- During the learning process (especially at the beginning), weight alignment may slightly loosen but remains mostly valid.\n- This baseline example illustrates that achieving initially aligned forward and backward weights allows for learning comparable to backpropagation, even without enforced synchronization of backward weights during training.\n- It's a specific instance highlighting the validity of random noise training in aligning forward and backward weights in a biologically plausible manner before data-driven learning.\n\n> How can you overcome the limitations of depth?\n\n**Our additional experiments confirmed that random noise pretraining scales effectively to deeper networks and more challenging tasks, despite the limitations of baseline feedback alignment.**\n\nIndeed, previous studies on biologically plausible backpropagation without weight transport often focus on simple network structures and easy datasets [Lillicrap et al., Nat. Comm., 2016; Dellaferrera and Kreiman, ICML, 2022; Toosi and Issa, NeurIPS, 2023], and this limitation results in lower learning capacity and presents challenges in scaling up due to biological constraints that preclude weight transport.\n\n**Through our additional experiments, we have shown that random noise pretraining consistently outperforms baseline feedback alignment, even in deeper networks.** However, we also observed that the performance gap with backpropagation widens as the number of layers increases.\n\n- This challenge partly arises from the difficulty of achieving precise weight alignment in early layers using feedback alignment, without weight transport.\n- The concept of \"layerwise local loss\" may provide ideas to address this issue, potentially enabling us to scale our results more effectively and achieve performance comparable to backpropagation in very deep networks. Specifically, this approach involves optimizing consecutive 2-layer blocks independently within a deep network. We are actively exploring several strategies to narrow the performance gap between feedback alignment and backpropagation using this approach, which we are preparing as part of our follow-up project.\n\nWe sincerely appreciate Reviewer kKCk's insightful comments. We acknowledge that in our initial work, we may not have sufficiently demonstrated the reproducibility and generalizability of our results, potentially leading to concerns about overestimation. Thanks to your feedback, we have conducted extensive additional experiments, which we believe have strengthened our arguments. We are grateful for your constructive criticism and we hope that our additional findings are convincing.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3bMbjBJDT1", "reviewer": "Reviewer_5zAn", "summary": "This paper explores the idea that the brain uses spontaneous prenatal activity to optimize its structure. This is done by showing the benefits of pre-training on random noise for artificial neural networks that are trained with the feedback alignment method - a training strategy that is more biologically-plausible than traditional backpropagation. Through experiments involving networks with and without pre-training, it is shown that pre-training enables these networks to reach convergence speeds comparable to backpropagation and enables them to generalize more effectively to various tasks.", "questions": "Figure 2.b demonstrates how a network that has undergone random pre-training converges more quickly than a network without pre-training. Do these plots account for the amount of “time” spent pre-training? If not, how much would considering the extent of pre-training close the gap in training speed between these two strategies?\nWhat is the overall contribution this paper wishes to make? I am not clear as to whether this paper wishes to a) validate established neuroscience findings using artificial neural networks or b) use insights from the brain to propose a way to improve current learning strategies in deep learning. If a), I have trouble understanding how recreating findings from experimental neuroscience in an artificial neural network trained with a more biologically-plausible algorithm constitutes a novel research contribution. If b), the empirical results here indicate that models trained with feedback alignment improve as the forward weights become more similar to the fixed backward weights - a property that is established in traditional artificial neural networks via the symmetry in their forward and backward weights. Has pre-training on random inputs shown any benefits for networks trained with traditional backpropagation (e.g., faster convergence, faster task generalization, increased accuracy, etc.)? Perhaps this would be an interesting investigation that establishes a benefit of biologically-inspired learning for traditional neural networks. \n\nAnswers to these questions might persuade me to move my score.", "limitations": "Limitations are somewhat stated and addressed. As mentioned, all of the experiments are run on a feedforward network. It could be worthwhile to verify to what extent these results are reproducible when using a different architecture, such as a recurrent or convolutional neural network.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The explanations of both the theory and empirical results are clear and well explained, for the most part. \nInteresting and thorough analyses were conducted to demonstrate that random pre-training of an artificial neural network trained with feedback alignment results in quicker convergence and more greater task generalization properties. I particularly appreciated the variety of studies investigating the trajectories of various metrics (e.g., accuracy, weight alignment, etc.) over time.", "weaknesses": "I am unclear as to what the Definition in lines 147-149 is trying to convey. Does “cosine angle” mean cosine similarity? If so, does \\mathbb{E} signify this chosen loss function, or is it an expectation over the similarities between the forward weight matrices and random feedback matrices across individual neurons? Why is \\angle only used in front of the forward weight matrices?\nThroughout Figure 2, the authors show numerous analyses (i.e., b, c, e, and g) of how certain metrics change during training. However, the label of the X-axis says “No. of training inputs.” However, given that the various experiments outlined throughout the Appendix list the number of training epochs as 100, I am inclined to believe that that is what the authors meant instead. Further clarity in this matter would be appreciated.\nA minor grievance, but referring to models without pre-training as “untrained” can be misleading.\nWhile not critical to the overall message of the paper, Appendix sections C.4 and C.5 could be further strengthened by adding the training trajectories of a network trained with back-propagation for comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> I am unclear as to what the Definition in lines 147-149 is trying to convey.\n\nWe acknowledge that the definition in Lines 147-149 regarding alignment between forward and backward weights is unclear.\n- \\mathbb{E} represents the expectation over the parallels between W and B across independent neurons.\n- \\angle refers to the angle between W and B, not each of W and B.\n\nWe will revise the definition of weight alignment as follows:\nGiven the forward weights W and backward weights B, we used cosine similarity as a measure of alignment for individual neurons. We assert that W and B are aligned if \\angle{W^T, B} is significantly smaller than 90 degrees.\n\n> However, the label of the X-axis says “No. of training inputs.”\n\nWe understand that the term \"No. of training inputs\" is not commonly used in machine learning and can be misleading. In the final version of the manuscript, we will correct it to \"epoch.\"\n- Initially, we used \"No. of training inputs\" due to the ambiguous definition of an epoch in random noise training.\n- Random noise training samples input x and label y in a random distribution. This is why we did not use \"epoch\" on the x-axis, which typically refers to traversing the entire dataset once.\n- To clarify, we will label the x-axis as \"epoch\" and explain in the text that in random noise training, samples in each epoch are newly sampled.\n\n> Referring to models without pre-training as “untrained” can be misleading.\n\nIn our paper, we distinguish four network states:\n1. Untrained (no training performed)\n2. Only random noise trained\n3. Data trained with an untrained network\n4. Data trained with a random noise trained network\n\n- Most results cover states (3) and (4). In Figure 3a, 3b, and 3c, \"untrained network\" refers to a randomly initialized, untrained network (1), and \"random trained\" refers to a network trained only with random noise (2).\n- We will organize the terms and colors to clearly distinguish these four network states in the final version of the manuscript.\n\n> Appendix sections C.4 and C.5 ...\n\nAs suggested, we will include results with backpropagation in Appendix C.4 and C.5.\n- We already have these results and have confirmed that the network with random noise pretraining learns much closer to backpropagation than to baseline feedback alignment.\n- This strengthens our argument that random noise training improves feedback alignment learning to levels comparable to backpropagation.\n\n> How much would considering the extent of pre-training close the gap in training speed between these two strategies?\n\n**: Please find our response in Global Response, Issue 2**\n\nAs you pointed out, comparison of convergence speed in Figure 2b does not account for the \"time\" spent on random noise training and only considers subsequent data training. \nWe agree that it is important to investigate whether random noise pretraining ensures fast convergence even when considering the time spent on pretraining. This factor can further emphasize the significance of our research.\n- We conducted additional analyses and measured the time to converge in real data learning after random noise pretraining with various lengths (Response Fig 4).\n- Our analyses showed that networks with random noise pretraining can learn data quickly, even when including the time spent on pretraining in most conditions. Additionally, this analysis allowed us to estimate the optimal pretraining length for resource-efficient learning.\n\n> What is the overall contribution this paper wishes to make?\n\nOur goal is to improve current strategies in deep learning by drawing insights from the brain.\n- There exists a significant accuracy gap between backpropagation and biologically plausible learning strategies without weight transport. While backpropagation is effective, it is computationally intensive, requiring dynamic memory access due to weight transport (access W in memory to compute backward updates).\n- As pointed out, the results of pretraining with random noise and aligning forward-backward weights resemble traditional backpropagation properties, but our approach achieves this without weight transport.\n- Therefore, our interest lies not in achieving comparable performance to backpropagation but in achieving it without weight transport. Hence, our results provide insights into achieving learning efficiency levels comparable to backpropagation without using weight transport.\n- It is noteworthy that feedback alignment (and random noise pretraining) does not require weight transport and relies solely on local information. In the context of energy-efficient neuromorphic chip engineering, feedback alignment is sometimes used for learning, albeit with some performance trade-offs. Our results demonstrate that such sacrifices are minimal in our model.\n\nWe briefly discussed these arguments in the broader impact section, but we realize that our intentions were not explicitly stated. This will be clearly articulated in the final manuscript's discussion.\n\nOn the other hand, we also discovered that pretraining with random noise benefits learning with backpropagation, although we did not include these results as they are not the main focus of the study.\n- For instance, results related to weight alignment (Figure 1) are specific to the feedback alignment algorithm. However, findings related to the low-rank bias of forward weights and its resulting better generalization (Figure 3, 4) are also observed in backpropagation.\n- Please note that we have identified additional benefits of random noise pretraining in backpropagation and are preparing this as part of a follow-up study.\n\nWe sincerely appreciate Reviewer 5zAn's careful consideration in improving the organization and clarity of our manuscript. We particularly value the opportunity to articulate how our research aims to contribute to the field and to present additional results that highlight the strengths of our work. We hope that our rebuttal and additional experimental findings will be convincing.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LfT92ewX5i", "reviewer": "Reviewer_o5xh", "summary": "The authors argue that networks trained with feedback alignment can be pre-trained with random input-output pairs. They demonstrate that this allows for faster learning (after the input-output pairs). They also show that the effective dimensionality of the network activity decreases if the pretraining is used, which could explain an improved generalization.", "questions": "I don't get what is the task agnostic section supposed to show. If the point is that the network with pre-training is better than the network without pre-training in general, then why introduce concepts such as meta-loss or task-agnostic. It seems to me that is equally valid (and maybe better) to say simply that the random pre-training is great in all tasks (and exemplify by a few).", "limitations": "Yes. But \n- 314-315: I don't get what the fact that the architecture that they mention does not use weight transport implies that more experimental studies are needed. to confirm their results. What does the lack of weight transport (which is probably true) have to do with more experiments? If they want to \"prove\" that weight transport is not present in cortex don't worry, no one thinks it's present. Anatomically the apical projections (top-down) are different than the basal ones anyway (bottom-up)", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper provides a simple way to accelerate training with feedback alignment. The results are convincing in terms of faster learning, and I haven't seen a paper using random pre-activations. I think it's a good insight, and it is well presented.", "weaknesses": "There is a critical missing point in my opinion.\nOne is that the authors do not train until convergence, neither for the random pre-training nor for the task related data. I could think that there could be some limitations if the networks are trained for a long time:\n- For example, it could be that having a too-low rank during pre-training is bad for learning later on; since the rank is related to the expresiveness of the network this would be plausible (although it might not be the case).\n- It could also be that at the end of training a network trained only on data performs better at the cost of more computation (I personally don't think so, but it should be verified). Although I think I can see a performance plateau in Fig. 2 b, it should be put in logscale to see if the performance really saturated or if it can keep increasing. If it is the second, then the authors should see when does it stop increasing and whether random+data really achieves better end-of-training results. \n- A related point: I haven't seen the performances in numbers. While I would not care too much for them on the main text, they should be added somewhere just to clarify that the training is done to a reasonable level; if the performance is only up to 90% or something that can be achieved with a logistic regression on MNIST then there would be a problem.\nNote that this is not a critical flaw; it's still a good idea to pre-train for faster learning, even if that could potentially lead to slightly lower performances later on (at least where biology is concerned, 92% accuracy or 93% is not really a big deal).\n\nThere are also minor issues to correct or comment:\n- Line 141: The loss decreased noticeably during random training. Should clarify that they mean the training loss (sorry if it is a bit pedantic)\n- Definition 1has a wording problem: the claim is that Wl and Bl are aligned if the expected angle decreases asymptotically during learning. Lets say that we have W^* after learning. If I just happen to have initialized the weights as W* by chance from the beginning, the statement in lines 148 and 149 would tell me that those weights are not aligned (even though they are the same). \n\nA few related works:\n-  Error backpropagation without weight transport: While it is true that the weight transport is thought to be impossible exaclty it could be achieved by some form of hebbian-like learning, see for example Amit, Y. \"Deep learning with asymmetric connections and Hebbian updates.\" Frontiers in computational neuroscience 13 (2019). \n- STDP and backpropagation: Although it is true that STDP is not enough to explain learning, there are simple architectures that can implement learning very similar to Backprop using a rate-based version of STDP (Aceituno et al. \"Learning cortical hierarchies with temporal Hebbian updates\"). Importantly, that not all the architectures require weight symmetry for training (the paper menitoned has a table listing them)\n\nFinally, I wanted to add a suggestion that I think would improve the paper. But it involves math and I don't know if the authors are up for that, so feel free to ignore it.\nI think it should be possible to show that the pretraining would reduce the effective rank. In a \"batch\" of pre-training with S samples and C classes, backprop would push the activity to become aligned with one of the C output directions. More analytically,:\n-  the weight updates are Dw = x_pre delta_post, and delta_post is always one of S random vectors. \n- But there are only C classes, and thus the deltas within a class should be of a low rank statistically. More specifically\n  + Compute the expectation and variance over the Jacobian,\n  + for ReLus is the product of feed-forward weights from the current layer onwards, with a probability of 0.5 that a given neuron in the path will drop out of the derivative ).\n  + The variance will decrease with increasing S, while the expectation will remain constant.\n- Once you have c delta_post, you can be reasonably sure that the result of learning will map the activity of the previous layer into something covering only the span of delta_post. \n  + This can be done by taking approximating the averaged BP rule as a delta rule.\nI hope is not too unclear", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> One is that the authors do not train until convergence, neither for the random pre-training nor for the task related data.\n\n**: Please find our response in Global Response, issue 1**\n\nWe acknowledge the suggestion that comparisons should be made after ensuring full convergence during network training.\n- We conducted additional experiments on various datasets beyond MNIST. In these experiments, we ensured that sufficient convergence was achieved before concluding training (Global Response, issue 1).\n- Specifically, we ensured training accuracy reached saturation close to 1, or that validation accuracy no longer increased for 10 epochs, indicating the completion of learning.\n- We will update all network training and analyses performed in the manuscript according to these criteria. Importantly, even after ensuring convergence, all trends and qualitative differences remained consistent.\n\n> It could be that having a too-low rank during pre-training is bad for learning later on\n\n**We found that even with extensive random noise pre-training, the rank reduction was not dramatic enough to impede the learning of rich representations.**\n- After random noise training, the effective rank decreased only slightly in most cases (Supple. C.3). This introduces a bias towards learning low-rank solutions in subsequent training but does not significantly hinder representation learning.\n- Depending on the network structure (width and depth), certain layers experienced a decrease in effective rank during random noise pre-training, which was subsequently recovered during data training. This suggests that while initial rank may affect information representation, it can be compensated for in later stages of learning.\n\nIn additional experiments, we proposed an effective pre-training length, also addressing concerns about excessively long pre-training periods (Please find our response in Global Response, issue 2).\n\n> It could also be that at the end of training a network trained only on data performs better at the cost of more computation\n\n**We quantitatively compared accuracy after achieving sufficient convergence and confirmed that random noise training generally results in higher accuracy regardless of network depth or dataset type** (Response Figs 1, 2, 3; Response Table 1).\n\n> I haven't seen the performances in numbers.\n\n**We provided numerical results for additional experiments** (Response Table 1). We will include these numerical results in a separate appendix in the final manuscript.\n\n> it's still a good idea to pre-train for faster learning, even if that could potentially lead to slightly lower performances later on.\n\nWe appreciate your comments emphasizing the benefits of pre-training for faster learning, especially in biological systems. **Our results indicate that our method achieves both faster learning and higher performance.** Thus, our approach demonstrates that learning speed and accuracy need not be mutually exclusive but can be concurrently enhanced through random noise pre-training.\n\n> Minor issues (description for loss reduction, definition of alignment)\n\nWe will clarify that the loss refers to \"training loss\" for random noise.\n\nWe acknowledge that our current definition of “alignment” may not hold in some cases. We will revise the definition as follows:\nGiven the forward weights W and backward weights B, we measured alignment using cosine similarity. We define W and B as aligned if \\angle{W^T, B} is significantly smaller than 90 degrees.\n\n> A few related works:\n\nWe will discuss the importance of weight symmetry for high-performance deep learning and also mention that there are learning architectures that do not require weight symmetry. Relevant papers will be cited in this discussion.\n\n> I think it should be possible to show that the pretraining would reduce the effective rank.\n\nWe appreciate your theoretical insights, which bolster our empirical findings.\n- Your detailed suggestions for theoretical proofs are insightful, particularly in understanding how random noise training reduces effective rank, especially in scenarios with limited readout capabilities.\n- During this rebuttal period, we focused on conducting additional experiments to validate our results in broader contexts. Due to time constraints, we could not fully develop and present the theoretical proofs, despite your helpful guidance.\n- We understand that including these proofs would significantly strengthen our findings, and we aim to incorporate them into the final manuscript.\n\n> I don't get what is the task agnostic section supposed to show.\n\nWe used \"task-agnostic\" to indicate that random noise training provides benefits in subsequent learning across various datasets. However, we acknowledge that this term might be unclear. We agree that \"generally useful for various tasks\" better conveys our intended meaning, as you suggested. We will replace the section title and descriptions accordingly in the manuscript.\n\n> I don't get what the fact that the architecture that they mention does not use weight transport implies that more experimental studies are needed.\n\nWe appreciate your perspective and detailed explanation regarding the biological implausibility of weight transport.\n- Describing specific biological circuits, as you suggested, would provide more compelling evidence than simply stating that weight transport is biologically implausible. We will include this in Section 2, Preliminaries, where we discuss the weight transport issue and mention that in the brain, apical (top-down) and basal (bottom-up) projections are anatomically distinct.\n- We will remove this description from the limitations section.\n\nWe extend our gratitude to Reviewer o5xh for recognizing the significance of our research. Your thoughtful review has encouraged us to conduct additional experiments during the rebuttal period, further validating the importance of our findings.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate the reviewers' thorough evaluation of our manuscript and their constructive feedback. We have revised our manuscript based on their comments, including additional analyses and simulations. We are confident that our revisions address all concerns raised, further validating our results. Below, we provide detailed responses to each question. Thank you for your consideration.

## Issue 1: Validation of Model Performance Across Various Image Datasets and Network Depths

**We have ensured that our model's performance remains consistent under more general conditions by testing it across five distinct types of datasets and varying the network depth from 2 to 5 layers** (Response Table 1).

### **Global Response Table 1.** Performance of each model with depth variation (2 – 5 layers) for five different datasets (MNIST, FMINST, CIFAR10, CIFAR100, and STL10)
\* Each performance value (%) is represented as the mean ± standard deviation from three trials.

||Model depth||2 Layer|3 Layer|4 Layer|5 Layer| 
|-|-|-|-|-|-|-|
|MNIST|BP||97.8±0.0|97.9±0.0|97.8±0.2|97.7±0.2|
||FA|w/o|97.3±0.1|97.0±0.1|96.8±0.2|95.6±0.2|
|||w/|97.8±0.1|97.6±0.1|97.3±0.2|97.0±0.0|
|||ΔACC|**▲0.5±0.1**|**▲0.6±0.1**|**▲0.5±0.3**|**▲1.5±0.2**|
|FMNIST|BP||88.9±0.0|88.8±0.1|88.6±0.0|88.5±0.1|
||FA|w/o|87.5±0.3|87.4±0.2|86.3±0.8|83.8±2.2|
|||w/|88.3±0.1|88.4±0.0|87.9±0.1|87.7±0.2|
|||ΔACC|**▲0.8±0.3**|**▲1.0±0.2**|**▲1.6±0.9**|**▲3.9±2.4**|
|CIFAR10|BP||54.0±0.2|53.8±0.2|53.2±0.2|52.3±0.0|
||FA|w/o|50.5±0.2|47.3±0.8|46.0±0.3|39.1±2.9|
|||w/|53.6±0.1|52.4±0.1|51.5±0.3|50.5±0.8|
|||ΔACC|**▲3.0±0.2**|**▲5.0±0.8**|**▲5.6±0.1**|**▲11.4±2.5**|
|CIFAR100|BP||24.6±0.1|24.7±0.0|24.5±0.1|24.3±0.1|
||FA|w/o|20.2±0.3|17.2±0.5|14.6±0.1|8.2±1.4|
|||w/|24.5±0.1|22.8±0.4|18.7±0.7|13.3±1.5|
|||ΔACC|**▲4.3±0.4**|**▲5.6±0.2**|**▲4.1±0.5**|**▲5.1±1.3**|
|STL10|BP||42.7±0.2|43.0±0.3|42.9±0.1|42.6±0.2|
||FA|w/o|36.2±0.9|35.0±0.9|36.7±0.2|31.5±5.1|
|||w/|41.0±0.2|41.5±0.2|41.4±0.1|39.0±0.6|
|||ΔACC|**▲4.8±0.9**|**▲6.5±1.0**|**▲4.6±0.4**|**▲7.5±5.4**|

We benchmarked the final accuracy of networks trained using baseline feedback alignment (FA, w/o), feedback alignment with random noise pretraining (FA, w/: our model), and backpropagation (BP) (Response Table 1). Specifically, **we evaluated these models at depths ranging from 2 to 5 layers (Response Fig. 1). Furthermore, we compared these models across several complex and large datasets, including MNIST, fashion-MNIST, CIFAR10, CIFAR100, and STL-10** (Response Figs. 2 and 3).

- **We observed that incorporating random noise training at various network depths and on diverse datasets consistently resulted in higher final accuracy, often comparable to that achieved with backpropagation** (Response Figs.1 - 3).
- As the network depth increased, the gap in final accuracy between models with and without random noise training widened (Response Figs. 1, 3).
- Importantly, the beneficial impact of random noise training on final accuracy became significantly more pronounced as task complexity increased (Response Figs. 2, 3). For instance, while random noise training improved accuracy by 0.49% in MNIST, it increased by 3.04% in CIFAR10, 4.28% in CIFAR100, and 4.81% in STL-10. This gap widens further as the network becomes deeper.

Reviewer o5xh expressed concerns regarding the convergence of learning in our analyses. To address this, **we extended our training duration until convergence was confirmed, with validation accuracy showing no further increase** (patience: 10 epochs) (Response Figs. 1, 2). Additionally, in response to suggestions from reviewers o5xh and kKCK, **we have included numerical presentations of final accuracy to provide a clearer assessment of model performance.**

## Issue 2: Computational Benefits of Random Noise Training Considering Total Training Epochs (Pretraining + Data Training)

**We confirmed that random noise pretraining accelerates convergence and reduces computational resources, even when considering the total training duration (pretraining + data training)** (Response Fig. 4). 

By varying the duration of noise training, we demonstrated that, despite the additional time required for noise training, the total training time remained significantly shorter than that of training with data alone in most conditions.

- We conducted subsequent data training on networks pretrained with random noise for 2, 5, 10, 20, and 50 epochs, measuring the epochs required for training to converge (validation accuracy no longer increasing, patience: 10 epochs). We maintained consistency in the number of samples used per epoch during both random noise training and subsequent data training, ensuring direct comparability of epoch times.
- We found that longer periods of random noise training resulted in shorter subsequent data training times to achieve convergence (Response Fig. 4a, Random + data vs. Data training only). As reported before, when comparing data training alone, the learning time consistently proved much shorter in networks pretrained with noise. Notably, **even when the time for noise training was added, the overall training duration for the noise-training algorithm remained substantially shorter than that for training with data alone in most conditions** (Response Fig. 4b).
- Based on this analysis, **we estimated an optimal duration for random noise pretraining that ensures the most efficient use of resources** (Response Fig. 4b, optimal). We also found that at this optimal duration, the improvement in accuracy remained significant.

These additional experimental results demonstrate that, considering the resources expended in random noise training, it represents a more efficient approach (reduced training time, overall computation) compared to training with data alone.

These results will also help address Reviewer o5xh's concern that random noise training may become too lengthy, by suggesting an effective duration for random noise training.

Meta-review: This submission introduces a novel pretraining approach inspired by spontaneous neural activity. Using random noise with a feedback alignment algorithm, the authors demonstrate improved learning efficiency and generalization without relying on weight transport. The method aligns forward weights with synaptic feedback, leading to faster convergence and better performance across various tasks, including out-of-distribution scenarios. The additional experiments and clarifications provided during the rebuttal period further validate the findings. All reviewers voted for acceptance.
Decision: Accept (poster)