Title: Pretraining with Random Noise for Fast and Robust Learning without Weight Transport
Venue: NeurIPS 2024
Accepted: yes
Keywords: Random noise training

Network pretraining

Pre-regularization

Feedback alignment

Error backpropagation

Weight transport problem

Biologically-Plausible Algorithm
Abstract: The brain prepares for learning even before interacting with the environment, by refining and optimizing its structures through spontaneous neural activity that resembles random noise. However, the mechanism of such a process has yet to be understood, and it is unclear whether this process can benefit the algorithm of machine learning. Here, we study this issue using a neural network with a feedback alignment algorithm, demonstrating that pretraining neural networks with random noise increases the learning efficiency as well as generalization abilities without weight transport. First, we found that random noise training modifies forward weights to match backward synaptic feedback, which is necessary for teaching errors by feedback alignment. As a result, a network with pre-aligned weights learns notably faster and reaches higher accuracy than a network without random noise training, even comparable to the backpropagation algorithm. We also found that the effective dimensionality of weights decreases in a network pretrained with random noise. This pre-regularization allows the network to learn simple solutions of a low rank, reducing the generalization error during subsequent training. This also enables the network to robustly generalize a novel, out-of-distribution dataset. Lastly, we confirmed that random noise pretraining reduces the amount of meta-loss, enhancing the network ability to adapt to various tasks. Overall, our results suggest that random noise training with feedback alignment offers a straightforward yet effective method of pretraining that facilitates quick and reliable learning without weight transport.

Pretraining with Random Noise for Fast and Robust Learning without Weight Transport

Abstract

The brain prepares for learning even before interacting with the environment, by refining and optimizing its structures through spontaneous neural activity that resembles random noise. However, the mechanism of such a process has yet to be understood, and it is unclear whether this process can benefit the algorithm of machine learning. Here, we study this issue using a neural network with a feedback alignment algorithm, demonstrating that pretraining neural networks with random noise increases the learning efficiency as well as generalization abilities without weight transport. First, we found that random noise training modifies forward weights to match backward synaptic feedback, which is necessary for teaching errors by feedback alignment. As a result, a network with pre-aligned weights learns notably faster and reaches higher accuracy than a network without random noise training, even comparable to the backpropagation algorithm. We also found that the effective dimensionality of weights decreases in a network pretrained with random noise. This pre-regularization allows the network to learn simple solutions of a low rank, reducing the generalization error during subsequent training. This also enables the network to robustly generalize a novel, out-ofdistribution dataset. Lastly, we confirmed that random noise pretraining reduces the amount of meta-loss, enhancing the network ability to adapt to various tasks. Overall, our results suggest that random noise training with feedback alignment offers a straightforward yet effective method of pretraining that facilitates quick and reliable learning without weight transport.

1 Introduction

The brain refines its network structure and synaptic connections even before birth, without exposure to sensory stimuli [1-4]. In the early developmental stages, the spontaneous neuronal activity that appears in various brain regions is considered to play a critical role during the development of neuronal circuits by pruning neural wiring and adjusting synaptic plasticity [5-8]. If this activity is disrupted during the developmental stages, the outcome can be long-lasting neuronal deficits [9-11]. Computational studies suggest that such a refined network structure enables certain crucial functions of the brain, such as initializing function [12-15] and structure [16, 17]. These experimental and theoretical studies commonly indicate that spontaneous, random neuronal activity plays a critical role in the development of the biological neural network before data are encountered by the network. However, the detailed mechanism of how these prenatal processes contribute to learning after birth, i.e., with subsequent sensory stimuli, remains elusive.

At the synaptic level, learning can be defined as the process by which the brain adjusts the strength of synaptic connections between neurons to optimize the network for a specific task [18-20]. The synaptic weights of each neuron can change to minimize the error between the expected and the

actual output of a task, often referred to as the credit assignment problem [21-23]. However, in general, it is not well known how individual neurons modify these synaptic connections and thus achieve a network goal under a condition in which numerous neurons are linked in multiple layers. In other words, how neurons can estimate errors to modify their synaptic connections during learning remains unknown.

In machine learning, backpropagation algorithms have successfully addressed this issue - even in deep neural networks [24-26]. Backpropagation can provide feedback with regard to forward errors through the symmetric copying of forward weights via a backward process. During this process, a structural constraint, i.e., symmetric forward and backward weights, is necessary to assign proper error values to individual neurons [27-29]. However, this process appears to be biologically implausible due to the weight transport problem [23, 30-32], in which individual neurons must somehow be aware of the exact synaptic connections of their downstream layers to update their weights, a state considered to be practically impossible in a biological brain.

An alternative algorithm, feedback alignment, achieves successful network training even without weight transport by employing fixed random feedback pathways [32]. This study shows that a network can align its weights to synaptic feedback during data training, and this simple process enables error backpropagation. It has been shown that soft alignment between forward and backward weights, which can be achieved during learning data, is enough to back-propagate errors. This finding may provide a biologically plausible scenario in which the credit assignment problem can be resolved, yet there is an issue remaining - the process requires massive data learning to develop the structural constraint, and it significantly underperforms compared to backpropagation on challenging tasks [33, 34]. This cannot be addressed even with currently known advanced learning rules [35-40].

This situation is contradictory to the notion that the brain can learn even with very limited experience in the initial stages of life [41-44]. Thus, the question arises as to how early brains can estimate and assign errors for learning with limited experience. To address this issue, here we focus on the role of spontaneous activity at the prenatal stage in the brain, showing that training random noise, which mimics spontaneous random activity in prenatal brains, is a possible solution; random noise training aligns the forward weights to synaptic feedback, enabling precise credit assignment and fast learning. We also observed that random noise training can pre-regulate the weights and enable robust generalization. Our findings suggest that random noise training may be a core mechanism of prenatal learning in biological brains and that it may provide a simple algorithm for the preconditioning of artificial neural networks for fast and robust learning without the weight transport process.

2 Preliminaries

Biological and artificial neural networks have different structures and functionalities, but they share certain factors in common, such that information is processed through hierarchical layers of neurons with a nonlinear response function. In the current study, we consider a multi-layer feedforward neural network for pattern classification, f θ : R m → R d , parameterized by θ = { W l , b l } L -1 l =0 . It takes input x ∈ R m and outputs a vector y ∈ R d with L layers. Through a forward pass, the network computes a hidden layer output by propagating the input through the network layers, as follows:

, where W l is the forward weights, b l is the bias vector, and ϕ is the nonlinear activation function. In the first layer l = 0 , h l = x . We used a rectified linear unit (ReLU) activation function, ϕ ( x ) = max(0 , x ) . In the last layer l = L -1 , we used a softmax function, ϕ y ( x ) = softmax ( x ) = { e x i / ∑ d j =1 e x j } d i =1 . Thus, the network outputs a probability distribution over d classes. After the forward pass, the amount of error is calculated by measuring the difference between the network output f θ ( x ) and the target label y . We used the cross-entropy loss [45], which is defined as follows:

, where N is the number of samples, d is the number of classes, and y ij is the target label for the i -th sample and the j -th class. The purpose of learning is to minimize the error L ( θ ) . To achieve this, the network parameters θ are adjusted by assigning credit to the weights that contribute to the error, which is known as the credit assignment problem.

2.1 Backpropagation and weight transport problem

To solve the credit assignment problem, backpropagation (BP) [24] computes the gradient of errors with respect to the weights and uses it as a teaching signal to modulate the aforementioned parameters. The gradient is calculated by the chain rule, with propagation from the output layer to the input layer, as follows:

, where δ l is the error signal at layer l , ϕ ′ is the derivative of the activation function, and ⊙ denotes the element-wise product. The weight update rule is given by

, where η is the learning rate. The backpropagation algorithm successfully solves the credit assignment problem, but it requires heavy computation to use the complete information of the synaptic weights of the next layer to update the current weights. Notably, backpropagation is considered as biologically implausible, because it is impossible, in the brain, to transmit the synaptic weights from the next layer to the current layer. This is known as the weight transport problem [30, 31].

2.2 Feedback alignment

To address the weight transport problem, the idea of feedback alignment (FA) [32] was proposed as a biologically plausible alternative to backpropagation. In feedback alignment, the backward synaptic feedback is replaced with a random, fixed weight matrix B l in the feedback path, as follows:

The only difference between backpropagation and feedback alignment is the replacement of the transpose of the forward weights W l with the fixed random feedback weights B l to calculate the error signal. The fact that the network can learn tasks from error teaching signals that are calculated from random feedback is explained by the observation that the network modifies the forward weights W l to match the transpose of the feedback weights B l roughly during training. This makes the error teaching signal (5) similar to backpropagation (3), thus enabling the network to learn the task.

3 Random noise pretraining with feedback alignment

During the developmental stage, spontaneous neural activity in the brain plays a critical role in shaping and refining neural circuits. Initially wired immature neural circuits undergo modifications of their connections through the processes of regulated cell formation, apoptosis, and synapse refinement through spontaneous neural activity [4-8]. These pre-sensory activities and development processes are universal across sensory modalities, such as the visual [46, 47], auditory [48, 49], and sensorimotor systems [50, 51]. We focus here on a few characteristics of spontaneous neural activity in the brain. Spontaneous neural activity is not correlated to external stimuli but can refine and optimize neural circuits, before interaction with the external world can take place.

Here, we propose a type of random training that is inspired by the spontaneous and prenatal neural activity in the brain to pretrain the neural network (Algorithm 1). In every iteration, we sampled

random noise inputs x from a Gaussian distribution N (0 , I ) and random labels y from a discrete uniform distribution U (0 , N readout -1) , without any correlation. The network f θ was initialized with random weights and trained with the feedback alignment algorithm. In this study, we examined the effects of random noise training on the subsequent learning processes in model neural networks to understand the potential benefits of pretraining with random noise in biological brains and whether this strategy is applicable to machine learning algorithms.

4 Results

4.1 Weight alignment to synaptic feedback during random noise training

a

b

C

Backpropagation(BP)

Possible synaptic circuit

Training neural networkwithrandom input and label

andfeedbackalignment(FA)

Random

Random

X

h

y

label

W

input

8

W。

Forward path

Wo

W

[o.]

W,T

W。"

B

0

(Rumelhartetal.,1986)

0.6

0.3

W

0.4

Ratio

0.2

W。

Backwardpath

Rat

0.2

0.1

B

0

0

B,ki

4

-2

0

0

Label classes

3

6

9

d

e

f

Loss across training

Alignmentanglebetweenweightsandsynapticfeedbacks

Averagedalignmentangle

random input andlabel

Before

Individual neuron

After

withvariousinput condition

3.5

150

150

150

120

Vector angle (deg, WT-B)

Vector angle (deg, WT-B)

120

120

120

3

Current

SS07

06

SN-

90

06

*

study

■

60

09

F09

2.5

30

30

F00

60

Uniform

2

×104

0+

×1040+

40

0

No. training random inputs

10

20

30

40

50

0

Ratios

0.03

0

No. training random inputs

10

20

30

40

50

0

Ratios

0.03

八

0.5

1

1.5

2

Variation of inputs (o)

To simulate a neural network initially wired by random weights and fixed random synaptic feedback, we adopted a network setting from the feedback alignment algorithm (Figure 1a) in which the weight transport problem can be avoided through the use of fixed random synaptic feedback. Thus, unlike backpropagation, this process is considered possible to exist in biological neural networks with local synaptic connections (Figure 1b). We used a two-layer feedforward neural network with ReLU nonlinearity for classification, f θ : R 784 → R 10 with 100 neurons in the hidden layer. By means of random noise training (Algorithm 1), we trained the neural network with random inputs sampled from a Gaussian distribution N (0 , I ) , with labels also randomly sampled independently (Figure 1c).

We observed that the training loss decreased noticeably during random training, even in the absence of meaningful data and even when x and y are randomly paired (Figure 1d). During the random training process, we focused on the alignment between the forward weights and the synaptic feedback. As described in the literature [32], the alignment of W l and B l , i.e., similarity between δ BP and δ FA, is crucial for calculating the error teaching signal precisely. To evaluate the alignment, we used cosine similarity, which is widely used for measuring the distance between two vectors.

Definition. Given the forward weights W l ∈ R m × n and backward weights B l ∈ R n × m , we measured alignment using cosine similarity. We define that W l and B l as aligned if the angle ∠ ( W T l ) i , ( B l ) i is significantly smaller than 90 degrees.

Notably, we found that the weights of neurons are aligned to the corresponding synaptic feedback weights during the random training process (Figure 1e). We also observed that the angle between the forward weights and synaptic feedback of individual neurons in the hidden layer decreased asymptotically during random training. In a randomly initialized network, the alignment angle appeared to be close to 90°, demonstrating that the backward error signal is randomly distributed (Figure 1e, left, alignment angle in an untrained network vs. 90°, n = 100 , one-sample t-test, NS, P = 0 . 492 ). However, after random training, the alignment angle decreased significantly, implying that the backward teaching signal becomes valid to back-propagate errors (Figure 1e, right, alignment angle in an untrained network vs. a randomly trained network, n = 100 , two-sample t-test, ∗ P < 0 . 001 ). We confirmed that this is not simply due to input bias under a particular condition but is reproduced robustly with various input conditions (Figure 1f). These results suggest that neural networks can pre-learn how to back-propagate errors through random noise training.

4.2 Pretraining random noise enables fast learning during subsequent data training

a

b

C

Effect of random training on data learning

Accuracy during data training

Weight alignment

Random

noise

(MNIST)

Data

Network

1.0

BP

W/ (FA)

Vector angle (deg, WT-B)

120

Random

Data

→

0.8

w/o (FA)

100

pretraining

w/random

Test accuracy

0.6

Convergence speed

1.01

*

80

w/o

0.4

60

F09

8:0

0.2

40

w/orandom

pretraining

0.7-

w/

0

20+

+

0

10

20

30

40

50

0

50

100

150

200

Epochs

Epochs

e

f

g

Trajectory of weights

DistancebetweenBandWT

Order dependenceof

Distance between B and WT

inlatentspace

inlatent space

random training

inlatentspace

5

5

②

PC2 (25.11%)

2.5

Data

WT

Random

training

Distance (IWT-BII)

6

PC2 (25.11%)

2.5

Data

WT

Random

Distance (IWT-BII)

Data

Random

training

?

NS

Data

②

3

Random

-2.5

4

Random

*

-2.5

B

·B

Data

-5

-5

-2.5

2.5

5

3+

0

20

40

60

80

100

-5+

-5

-2.5

0

2.5

5

1+

0

50

100

150

200

PC1 (72.59%)

Epochs

PC1 (72.59%)

Epochs

Next, we compared networks with and without random pretraining in terms of subsequent data training outcomes (Figure 2a). We trained the networks using a subset of the MNIST dataset [52], a widely used benchmark for image classification. We found that a randomly pretrained network learns the data more quickly and achieves higher accuracy compared to a network that is not randomly pretrained (Figure 2b). To quantify the speed of learning, we calculated the area under the curve (AUC) of the test accuracy and found that the convergence of the randomly pretrained network is significantly faster than that in the network without random pretraining (Figure 2b, inset, w/o vs. w/ random pretraining (FA), n net = 10 , t-test, ∗ P < 0 . 001 ). Notably, the convergence speed of the randomly pretrained network appeared comparable to that of the network trained with backpropagation (Figure

2b, inset, w/ random pretraining (FA) vs. BP, n net = 10 , t-test, ∗ P < 0 . 001 ). We also observed that the weight alignment gap between untrained and randomly pretrained networks is maintained during data training (Figure 2c). As a result, at the end of the data training step, the alignment angle of the randomly trained network was significantly smaller than that of the untrained network (Figure 2c, w/o vs. w/ random pretraining, n = 100 , t-test, ∗ P < 0 . 001 ). This result suggests that a combination of random pretraining and subsequent data training can enhance the weight alignment, which leads to more precise error teaching.

To understand the weight update dynamics by random and data training, we visualized the trajectory of weights in latent space as obtained by a principal component analysis (PCA) [53] (Figure 2d). We conducted PCA on the weights of the last layer ( W 1 ) for the random and data training conditions. First, we confirmed that in both random and data training, the weights become closer to synaptic feedback (Figure 2e, untrained vs. data training, n net = 10 , t-test, ∗ P < 0 . 001 ; untrained vs. random training, n net = 10 , t-test, ∗ P < 0 . 001 ; random vs. data training, n net = 10 , t-test, ∗ P < 0 . 001 ). Notably, we observed that the updated trajectory of weights by random training and data training have different directions in the principal component space and that the effects of random training depend on the order of the random and data training (Figure 2f, g) - the enhancement of weight alignment was more significant when data training was performed after random training (Figure 2g, random training vs. random and data training, n net = 10 , t-test, ∗ P < 0 . 001 ) compared to when training is done in a reversed order (Figure 2g, data training vs. data and random training, n net = 10 , t-test, NS, P = 0 . 999 ). Particularly, when we trained the network with data first, subsequent random training could not move the weights; thus, the weights did not become closer to synaptic feedback. This result suggests that weight alignment by random noise pretraining cannot be replaced by data training and that it is crucial to perform random training prior to data training.

a

b

d

MNIST

F-MNIST

CIFAR-10

CIFAR-100

STL-10

0.9

0.6

0.3

0.5

BP

Depth =2

Depth = 2

Depth=2

0.6

0.5

0.2

0.2

0.4

0.4

0.95

0.85

0.3

0.1

0.2

0+

0.4

0

0.1

0

0.3

w/ random (FA)

0

40

80

0

50

100

0

50100

w/o random (FA)

Epochs

Epochs

Epochs

0.92

0.8

0.3

0

0.2

4

5

2

3

4

5

2

3

4

5

2

3

4

5

2

3

4

5

Depth

Depth

Depth

Depth

Depth

Next, we further investigated the model's classification performance across various image datasets and network depths (Figure 3, Table 1). In earlier experiments with two-layer networks and MNIST, we showed that random noise pretraining enhances both the learning speed and accuracy of networks to levels comparable with backpropagation. We extended these experiments to networks of varying depths (Figure 3a) and confirmed that the benefits of random noise pretraining generalize to deeper networks. Additionally, we evaluated performance across various image datasets, including FashionMNIST [54] (Figure 3b), CIFAR-10 [55] (Figure 3c), CIFAR-100 [55] (Figure 3d), and STL-10

[56] (Figure 3e). We found that random noise pretraining significantly narrows the performance gap between feedback alignment and backpropagation across different datasets and depths. These results suggest that pretraining with random noise can serve as a general strategy for improving the performance of neural networks trained with feedback alignment algorithms, making it comparable to backpropagation.

4.3 Pre-regularization by random noise training enables robust generalization

*

a

b

C

d

Untrained vs.random

Readout

Singular value spectrum

Dimensionalityof weights

noisetrainednetwork

probability distribution

ofweights

across random training

Normalized singular value

x10-2

21

Untrained

L66

Initialization

Density

102

Untrained

1

W。

Effective rank

10-2

0

Random

0

0.5

0

50

100

noise

HHNS

Random trained

Density

102

1

W。

Random trained

10-1

0

96+

×104

0.5

1

0

50

100

0

10

20

30

40

N-layer network

Probability

Sorted index

No.training random inputs

e

f

g

h

Training vs. test loss

Training set size vs.

Effectivedimensionality

Networkdepthvs,

generalization error

oflearnedfeature

Featuredimensionality

2.5

*

Gram

error

matrix

*

0.75

0.5

2

*

Effective rank

6

Effectiverank

w/o

Test loss

Generalization

5

w/

FS0

0

1.5

w/o

5

w/

1

0.25

w/o random

0.5

*

4

2

w/random

o+

0

0

1

1

0.75

0.5

0.25

0

100200 400 800 1600

Feature

w/

w/o

3

4

5

6

Training loss

Traning set size

similarity

Depth

Next, we compared the difference between an untrained network and a randomly trained network in terms of their activation and weight (Figure 4a). First, we found that the readout probability of the untrained network is distributed over a wide range (Figure 4b, top, untrained vs. chance level, n = 1000 , Wilcoxon signed-rank test, ∗ P < 0 . 001 ), whereas that of the randomly trained network is well regularized, close to the chance level (Figure 4b, bottom, random noise trained vs. chance level, n = 1000 , Wilcoxon signed-rank test, NS, P = 0 . 096 ). We also observed that the singular value spectrum of forward weights changes significantly by random training (Figure 4c) such that a small portion of singular values become dominant in the randomly trained network. To measure the effective dimensionality of the weights quantitatively, we used the effective rank of the weights.

Definition. Given a matrix A ∈ R m × n is decomposed into A = U Σ V T by singular value decomposition (SVD), the singular values are { σ i } min ( m,n ) i =1 sorted in descending order. The effective rank ρ is defined as the Shannon entropy of the normalized singular values, ρ = -∑ i ¯ σ i log ¯ σ i , where ¯ σ i = σ i / ∑ i σ i . Without loss of generality, we used the effective rank as the exponential of ρ [57].

We observed that the effective rank of forward weights decreased significantly during random training (Figure 4d), implying that random training regularizes the weights initially and predisposes the network to learn simple solutions of a low rank. Given the notion that low-rank solutions show better generalization performance outcomes, we hypothesized that this pre-regularization by random training enables robust generalization during subsequent data training by inducing low-rank solutions.

To test the generalization ability of the network, we measured the gap between the training error and the test error during subsequent data training. We observed that the generalization error was noticeably lower in a randomly pretrained network compared to a network trained solely on the data (Figure 4e) and that this tendency was maintained with variations of the training set size (Figure 4f, w/o vs. w/ random pretraining, n net = 10 , t-test, ∗ P < 0 . 001 ). This result suggests that pre-regularization by random pretraining can enable robust generalization during subsequent data training.

Next, we compared the representation of learned features in a randomly pretrained network and a network trained without random pretraining. We used the Gram matrix, defined as the cosine similarity of feature vectors across neurons. Notably, we found that the effective rank of the Gram matrix was significantly lower in a randomly pretrained network compared to an untrained network after subsequent data training (Figure 4g, w/o vs. w/ random pretraining, n net = 10 , t-test, ∗ P < 0 . 001 ) and that this tendency was maintained regardless of the network depth (Figure 4h, w/o vs. w/ random pretraining, n net = 10 , t-test, ∗ P < 0 . 001 ). This finding suggests that pre-regularization by random training can enable networks to learn simpler solutions, leading to better generalization performance during subsequent data training.

a

b

Trainingin-distribution

Out-of-distributiongeneralizationtestontransformedMNiST

OoDtestonbenchmark

data (MNIST)

Translated

Scaled

Rotated

USPS

MNIST

TranslatedScaledRotated

0.9

USPS

0.5

w/

*

*

￥

*

random

Test accuracy

Testaccuracy

0.85

w/

0.4

random

0.8

w/o

0.3

random

w/o

random

ID

OOD

0.75

OOD

0.2

Networktraining

test

w/

w/o

w/

w/o

w/

w/o

w/

w/o

test

We also tested the generalization performance of the networks for "out-of-distribution" tasks by training the network with the MNIST dataset and testing it with various out-of-distribution tasks (Figure 5a). First, we generated a MNIST dataset of translated, rotated, and scaled images and then used these images as out-of-distribution tasks (Figure 5b, left). We observed that a randomly pretrained network showed significantly higher test accuracy on out-of-distribution tasks than a network trained without random pretraining (Figure 5b, right, w/o vs. w/ random pretraining, n net = 10 , t-test, ∗ P < 0 . 001 ). We also observed that the randomly pretrained network showed higher test accuracy on the USPS dataset [58], which is a widely used benchmark dataset for outof-distribution tasks (Figure 5c, w/o vs. w/ random pretraining, n net = 10 , t-test, ∗ P < 0 . 001 ). This result suggests that pre-regularization by random pretraining enables robust out-of-distribution generalization during subsequent data training.

4.4 Task-agnostic fast learning for various tasks by a network pretrained with random noise

Lastly, we examined whether random training is generally beneficial for various tasks. We compared the task adaptation capacity of an untrained network and a randomly pretrained network on three tasks: (1) MNIST classification [52], (2) Fashion-MNIST [54], (3) Kuzushiji-MNIST [59] (Figure 6a, left). To measure the ability of fast adaptation to various tasks quantitatively, we computed the meta-loss, as suggested in a previous study of meta-learning.

Definition. Given the task distribution T ∈ {T i } n i =1 , the meta-loss of network f θ is defined as L meta ( θ ) = ∑ T i ∈T L T i ( θ ′ i ) , where L T i ( θ ′ i ) denotes the loss of the task T i and θ ′ i is the adapted parameter for T i [60].

We observed that the meta-loss decreased gradually during the random training process (Figure 6a, right). Considering that the training was solely performed with random inputs and labels on the three tasks to measure the meta-loss, this result suggests that networks can learn how to adapt to various tasks without any task-specific data. Next, we trained the untrained networks and randomly pretrained

a

b

Task

Meta-lossduring

Trajectoryofweightsinlatentspace

Task-agnosticfast adaptation

distribution

random training

duringadaptationtovarioustasks

0.6

Random noise

1.0

1

Random noise

Untrained

Task 1

pretrained

Task 1

pretrained

Task 1

0.5

0.8

Task 2

(F-MNIST)

(MNIST)

Meta-loss

M

0.4

0.3

9.0%)

3

P

0

W

Random

W'

Task 1

Task3

B

0.4

0.6

Task3

*

*

*

(K-MNIST)

Task3

0.2

Task3

0.2

10 classes

0.1

0

25

50

×104

PC2 (14.5%)

1

0

-1

-2

0

0

1

2

T

3

T

Untrained

4

5

1

No.trainingrandominputs

2

PC1 (53.9%)

Epochs

networks on each task separately. We conducted PCA on the weights of the last layer ( W 2 ) to visualize the trajectory of weights in latent space during the adaptation to each task. We observed that the trajectory of weights during random noise training moves closer to synaptic feedback ( B 2 ), which makes the adaptation to each task more efficient (Figure 6b). This suggests that random training is task-agnostic but provides efficient and fast learning in subsequent learning. Lastly, we compared the adaptation to each task in an untrained network and a randomly pretrained network. We observed that the randomly trained network showed remarkably fast adaptation to each task compared to the untrained network (Figure 6c, w/o vs. w/ random pretraining, n net = 10 , t-test, ∗ P < 0 . 001 ). These results highlight the task-agnostic property of random training, which enables networks to quickly adapt to various tasks.

5 Discussion

We showed that random noise pretraining enables neural networks to learn quickly and robustly without weight transport. This finding bridges the gap between a biologically plausible learning mechanism and the conventional backpropagation algorithm, as the symmetry of forward and backward weights can easily be achieved by random noise pretraining. Moreover, the results here provide new insight into the advantage of random training as a means of preconditioning a network for robust generalization.

Error-backpropagation without weight transport. Early work in neuroscience identified basic learning rules, such as Hebbian learning [61] and spike-timing-dependent plasticity (STDP) [62-64]. Although these rules have been experimentally observed and are thus biologically plausible, they are not sufficient to explain the brain's remarkable learning ability thoroughly [65, 66]. On the other hand, while the backpropagation algorithms used in artificial neural networks have shown impressive learning capabilities, they are considered biologically implausible due to the weight transport problem [23]. Our results provide a new perspective on this issue, bridging the gap between these training rules. We showed that symmetry among forward and backward weights, which is necessary to back-propagate errors, can be readily developed by learning random noise, similar to that during the brain's prenatal stage. Our findings suggest a probable scenario for significantly narrowing the performance gap between previously suggested biologically plausible learning rules and backpropagation.

Recent studies explored noise as a biologically plausible mechanism to enhance learning efficiency without the need for weight transport. For instance, the weight mirror algorithm [36] uses noisy firing to align feedback weights with forward weights. Similarly, phaseless alignment learning [40] leverages layer-wise noise as an additional information carrier to achieve weight alignment. While these approaches have been reported to outperform traditional feedback alignment algorithm, merely incorporating noise into existing feedback alignment algorithms yields no improvement in learning performance [40]. In contrast, our results indicate that exposure to random noise "before encountering real data" significantly enhances vanilla feedback alignment. Unlike previous approaches, our

strategy utilizes random noise for pre-conditioning the network, preparing it to learn more effectively. This aligns with biological observations that neural noise predominates in the early stage of brain development [67], even prior to exposure to external stimuli. It is important to note that our proposed method is not limited to the feedback alignment algorithm; pretraining with random noise could be beneficial for other algorithms, which we intend to explore in our follow-up studies.

Pre-regularization for robust generalization. We suggest a task- and model-agnostic pretraining strategy that involves simply training the network with random noise. Notably, our results here show that random noise training can enhance the learning efficiency and generalization ability of the network, for which various tricks and techniques have been proposed to improve. We found that pretraining on random noise reduces the effective dimensionality of the weights, facilitating the learning of low-rank solutions for various tasks. Previous studies on generalization have shown that the low-rank bias of neural networks plays a crucial role in their generalization ability, a finding we have confirmed in this study [68-72]. Additionally, our results highlight that random noise pretraining functions as a form of meta-learning, enhancing the network's ability to adapt rapidly to different tasks. In contrast to previous approaches that utilized data from diverse task distributions [60], our method achieves similar effects by merely training with noise. It is important to note that the straightforward strategy of random noise training can significantly influence the network's learning dynamics - effects that previous machine learning techniques have sought to achieve. This approach may reflect a potential strategy employed by the brain to attain notable generalization capabilities. Furthermore, it suggests a novel pretraining strategy for artificial neural networks.

Insights into developmental neuroscience. Unlike artificial neural networks, the brain is ready to learn before encountering data. In the early developmental stage before eye-opening, spontaneous random activity emerges in the brain, which is considered essential for the normal development of early circuits [1-4]. However, the functional advantage of learning from random noise before external sensory inputs remains unclear. Our study provides a plausible scenario that the brain utilizes spontaneous random activity to pre-align the synaptic weights for error learning and pre-regularization of synaptic connections for robust generalization. Specifically, we showed that random training reduces the effective dimensionality of the weights, which can be considered as a form of pruning, as previous neuroscience studies reported that the brain's synaptic connections are pruned substantially during development, particularly dependent on spontaneous activity [4-8]. Despite the fact that the present study is based on model neural networks, the results here are consistent with a range of experimental findings in developmental neuroscience.

6 Broader impacts and limitations

Broader impacts. Feedback alignment algorithm and its advanced modifications without weight transport are motivated by the need to suggest a learning method that is compatible with deep neural networks with biological plausibility. It can be useful particularly when implemented in physical circuits, as nowadays deep learning without weight transport is utilized in neuromorphic chip engineering. Given that backpropagation requires dynamic access to memory due to weight transport, it is not free from the issue of energy inefficiency. Our results are not solely limited to demonstrating the role of biological prenatal learning but can also be extended for more practical purposes; for instance, it is a promising strategy for the preconditioning of neuromorphic chips.

Limitations. Although our study offers a new perspective on pretraining neural networks with random noise, some limitations must be considered. The current study focuses on results using feedforward neural networks with feedback alignment algorithms. Regarding the scalability of the method, further investigation is needed for other architectures, such as convolutional neural networks. Notably, we achieved meaningful results showing that pretraining with random noise can also benefit standard backpropagation, which will be further explored in our follow-up studies, which will emphasize random noise pretraining as a general strategy for neural network training.

7 Code availability

Python 3.11 (Python software foundation) with PyTorch 2.1 was used to perform the simulation. The code used in this work is available at https://github.com/cogilab/Random .

References

Lucia Galli and Lamberto Maffei. Spontaneous impulse activity of rat retinal ganglion cells in prenatal life. Science , 242(4875):90-91, 1988.

James B Ackman, Timothy J Burbridge, and Michael C Crair. Retinal waves coordinate patterned activity throughout the developing visual system. Nature , 490(7419):219-225, 2012.

Noelia Antón-Bolaños, Alejandro Sempere-Ferràndez, Teresa Guillamón-Vivancos, Francisco J Martini, Leticia Pérez-Saiz, Henrik Gezelius, Anton Filipchuk, Miguel Valdeolmillos, and Guillermina LópezBendito. Prenatal activity from thalamic neurons governs the emergence of functional cortical maps in mice. Science , 364(6444):987-990, 2019.

Francisco J Martini, Teresa Guillamón-Vivancos, Verónica Moreno-Juan, Miguel Valdeolmillos, and Guillermina López-Bendito. Spontaneous activity in developing thalamic and cortical sensory networks. Neuron , 109(16):2519-2534, 2021.

Oriane Blanquie, Jenq-Wei Yang, Werner Kilb, Salim Sharopov, Anne Sinning, and Heiko J Luhmann. Electrical activity controls area-specific expression of neuronal apoptosis in the mouse developing cerebral cortex. Elife , 6:e27696, 2017.

Larry C Katz and Carla J Shatz. Synaptic activity and the construction of cortical circuits. Science , 274(5290):1133-1138, 1996.

Werner Kilb, Sergei Kirischuk, and Heiko J Luhmann. Electrical activity patterns and the functional maturation of the neocortex. European Journal of Neuroscience , 34(10):1677-1686, 2011.

Nobuhiko Yamamoto and Guillermina López-Bendito. Shaping brain connections through spontaneous neural activity. European Journal of Neuroscience , 35(10):1595-1604, 2012.

Heiko J Luhmann, Anne Sinning, Jenq-Wei Yang, Vicente Reyes-Puerta, Maik C Stüttgen, Sergei Kirischuk, and Werner Kilb. Spontaneous neuronal activity in developing neocortical networks: from single cells to large-scale interactions. Frontiers in neural circuits , 10:40, 2016.

Nicolas Heck, Antje Golbs, Therese Riedemann, Jyh-Jang Sun, Volkmar Lessmann, and Heiko J Luhmann. Activity-dependent regulation of neuronal apoptosis in neonatal mouse cerebral cortex. Cerebral cortex , 18(6):1335-1349, 2008.

Andrew D Huberman, Colenso M Speer, and Barbara Chapman. Spontaneous retinal activity mediates development of ocular dominance columns and binocular receptive fields in v1. Neuron , 52(2):247-254, 2006.

Gwangsu Kim, Jaeson Jang, Seungdae Baek, Min Song, and Se-Bum Paik. Visual number sense in untrained deep neural networks. Science advances , 7(1):eabd6127, 2021.

Seungdae Baek, Min Song, Jaeson Jang, Gwangsu Kim, and Se-Bum Paik. Face detection in untrained deep neural networks. Nature communications , 12(1):7328, 2021.

Jeonghwan Cheon, Seungdae Baek, and Se-Bum Paik. Invariance of object detection in untrained deep neural networks. Frontiers in Computational Neuroscience , 16:1030707, 2022.

Hyeonsu Lee, Woochul Choi, Dongil Lee, and Se-Bum Paik. Comparison of visual quantities in untrained neural networks. Cell Reports , 42(8), 2023.

Jinwoo Kim, Min Song, Jaeson Jang, and Se-Bum Paik. Spontaneous retinal waves can generate long-range horizontal connectivity in visual cortex. Journal of Neuroscience , 40(34):6584-6599, 2020.

Mark V Albert, Adam Schnabel, and David J Field. Innate visual learning through spontaneous activity patterns. PLoS Computational Biology , 4(8):e1000137, 2008.

Wulfram Gerstner, Richard Kempter, J Leo Van Hemmen, and Hermann Wagner. A neuronal learning rule for sub-millisecond temporal coding. Nature , 383(6595):76-78, 1996.

Shigeru Kitazawa, Tatsuya Kimura, and Ping-Bo Yin. Cerebellar complex spikes encode both destinations and errors in arm movements. Nature , 392(6675):494-497, 1998.

Georg B Keller and Richard HR Hahnloser. Neural processing of auditory feedback during vocal practice in a songbird. Nature , 457(7226):187-190, 2009.

Marvin Minsky. Steps toward artificial intelligence. Proceedings of the IRE , 49(1):8-30, 1961.

Blake A Richards and Timothy P Lillicrap. Dendritic solutions to the credit assignment problem. Current opinion in neurobiology , 54:28-36, 2019.

Timothy P Lillicrap, Adam Santoro, Luke Marris, Colin J Akerman, and Geoffrey Hinton. Backpropagation and the brain. Nature Reviews Neuroscience , 21(6):335-346, 2020.

David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning representations by backpropagating errors. Nature , 323:533-536, 10 1986.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems , 25, 2012.

Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature , 521(7553):436-444, 2015.

Qianli Liao, Joel Leibo, and Tomaso Poggio. How important is weight symmetry in backpropagation? In Proceedings of the AAAI Conference on Artificial Intelligence , volume 30, 2016.

Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120 , 2013.

Konrad P Körding and Peter König. Supervised and unsupervised learning with two sites of synaptic integration. Journal of computational neuroscience , 11:207-215, 2001.

Stephen Grossberg. Competitive learning: From interactive activation to adaptive resonance. Cognitive science , 11(1):23-63, 1987.

Francis Crick. The recent excitement about neural networks. Nature , 337(6203):129-132, 1989.

Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin J Akerman. Random synaptic feedback weights support error backpropagation for deep learning. Nature communications , 7(1):13276, 2016.

Sergey Bartunov, Adam Santoro, Blake Richards, Luke Marris, Geoffrey E Hinton, and Timothy Lillicrap. Assessing the scalability of biologically-motivated deep learning algorithms and architectures. Advances in neural information processing systems , 31, 2018.

Navid Shervani-Tabar and Robert Rosenbaum. Meta-learning biologically plausible plasticity rules with random feedback pathways. Nature Communications , 14, 12 2023.

John F Kolen and Jordan B Pollack. Backpropagation without weight transport. In Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94) , volume 3, pages 1375-1380. IEEE, 1994.

Mohamed Akrout, Collin Wilson, Peter Humphreys, Timothy Lillicrap, and Douglas B Tweed. Deep learning without weight transport. Advances in neural information processing systems , 32, 2019.

Arild Nøkland. Direct feedback alignment provides learning in deep neural networks. Advances in neural information processing systems , 29, 2016.

Yali Amit. Deep learning with asymmetric connections and hebbian updates. Frontiers in computational neuroscience , 13:18, 2019.

Pau Vilimelis Aceituno, Matilde Tristany Farinha, Reinhard Loidl, and Benjamin F Grewe. Learning cortical hierarchies with temporal hebbian updates. Frontiers in Computational Neuroscience , 17:1136010, 2023.

Kevin Max, Laura Kriener, Garibaldi Pineda García, Thomas Nowotny, Ismael Jaras, Walter Senn, and Mihai A Petrovici. Learning efficient backprojections across cortical hierarchies in real time. Nature Machine Intelligence , pages 1-12, 2024.

Hermann Bulf, Scott P Johnson, and Eloisa Valenza. Visual statistical learning in the newborn infant. Cognition , 121(1):127-132, 2011.

Ingrid B Johanson and WG Hall. Appetitive learning in 1-day-old rat pups. Science , 205(4404):419-421, 1979.

Tiffany M Field, Debra Cohen, Robert Garcia, and Reena Greenberg. Mother-stranger face discrimination by the newborn. Infant Behavior and development , 7(1):19-25, 1984.

IWR Bushneil, F Sai, and Jim T Mullin. Neonatal recognition of the mother's face. British journal of developmental psychology , 7(1):3-15, 1989.

Irving John Good. Rational decisions. Journal of the Royal Statistical Society: Series B (Methodological) , 14(1):107-114, 1952.

Richard H Masland. Maturation of function in the developing rabbit retina. Journal of Comparative Neurology , 175(3):275-286, 1977.

James B Ackman and Michael C Crair. Role of emergent neural activity in visual map development. Current opinion in neurobiology , 24:166-175, 2014.

Nicolas X Tritsch, Eunyoung Yi, Jonathan E Gale, Elisabeth Glowatzki, and Dwight E Bergles. The origin of spontaneous activity in the developing auditory system. Nature , 450(7166):50-55, 2007.

Helen J Kennedy. New developments in understanding the mechanisms and function of spontaneous electrical activity in the developing mammalian auditory system. Journal of the Association for Research in Otolaryngology , 13:437-445, 2012.

Viktor Hamburger, Eleanor Wenger, and Ronald Oppenheim. Motility in the chick embryo in the absence of sensory input. Journal of Experimental Zoology , 162(2):133-159, 1966.

Scott R Robinson, Mark S Blumberg, Maura S Lane, and Lisa S Kreber. Spontaneous motor activity in fetal and infant rats is organized into discrete multilimb bouts. Behavioral neuroscience , 114(2):328, 2000.

Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE signal processing magazine , 29(6):141-142, 2012.

Svante Wold, Kim Esbensen, and Paul Geladi. Principal component analysis. Chemometrics and Intelligent Laboratory Systems , 2:37-52, 8 1987.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017.

Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.

Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages 215-223. JMLR Workshop and Conference Proceedings, 2011.

Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In 2007 15th European signal processing conference , pages 606-610. IEEE, 2007.

Jonathan J. Hull. A database for handwritten text recognition research. IEEE Transactions on pattern analysis and machine intelligence , 16(5):550-554, 1994.

Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv preprint arXiv:1812.01718 , 2018.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning , pages 1126-1135. PMLR, 2017.

Donald Olding Hebb. The organization of behavior: A neuropsychological theory . Psychology press, 1949.

Robert C Froemke and Yang Dan. Spike-timing-dependent synaptic modification induced by natural spike trains. Nature , 416(6879):433-438, 2002.

Yang Dan and Mu-ming Poo. Spike timing-dependent plasticity of neural circuits. Neuron , 44(1):23-30, 2004.

Natalia Caporale and Yang Dan. Spike timing-dependent plasticity: a hebbian learning rule. Annu. Rev. Neurosci. , 31:25-46, 2008.

H Sebastian Seung. Learning in spiking neural networks by reinforcement of stochastic synaptic transmission. Neuron , 40(6):1063-1073, 2003.

Yiting Dong, Dongcheng Zhao, Yang Li, and Yi Zeng. An unsupervised stdp-based spiking neural network inspired by biologically plausible learning rules and connections. Neural Networks , 165:799-808, 2023.

Ann M Skoczenski and Anthony M Norcia. Neural noise limitations on infant visual sensitivity. Nature , 391(6668):697-700, 1998.

Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems , 32, 2019.

Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, and Phillip Isola. The low-rank simplicity bias in deep networks. arXiv preprint arXiv:2103.10427 , 2021.

Aristide Baratin, Thomas George, César Laurent, R Devon Hjelm, Guillaume Lajoie, Pascal Vincent, and Simon Lacoste-Julien. Implicit regularization via neural feature alignment. In International Conference on Artificial Intelligence and Statistics , pages 2269-2277. PMLR, 2021.

Dan Zhao. Combining explicit and implicit regularization for efficient learning in deep networks. Advances in Neural Information Processing Systems , 35:3024-3038, 2022.

Parth Natekar and Manik Sharma. Representation based complexity measures for predicting generalization in deep learning. arXiv preprint arXiv:2012.02775 , 2020.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We clearly state the main claims and contributions in both the abstract and the introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of our work in the final section of the paper.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Our paper does not contain any theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide all the necessary information to reproduce the main experimental results. The experimental settings and parameters are specified in the supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide the URL to the code repository for reproducibility.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The training and testing details are provided in the supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We reported error bars and specified the significance levels in the figures and their corresponding texts.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We indicated the specifications of the computing resources used in the experiments. Additionally, we want to note that our simulation does not require high computational resources.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the NeurIPS Code of Ethics and ensured that our research conforms to it in all respects.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discussed the potential societal impacts of our work in the final section of the paper. However, since our research addresses fundamental questions in computational neuroscience, we do not anticipate any negative societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper presents no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We used publicly available datasets and properly cited the original sources.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide the URL to the code repository, which includes access to the code and models along with accompanying documentation.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing or research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve research on human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors argue that networks trained with feedback alignment can be pre-trained with random input-output pairs. They demonstrate that this allows for faster learning (after the input-output pairs). They also show that the effective dimensionality of the network activity decreases if the pretraining is used, which could explain an improved generalization.\n\nSTRENGTHS:\nThe paper provides a simple way to accelerate training with feedback alignment. The results are convincing in terms of faster learning, and I haven't seen a paper using random pre-activations. I think it's a good insight, and it is well presented.\n\nWEAKNESSES:\nThere is a critical missing point in my opinion.\nOne is that the authors do not train until convergence, neither for the random pre-training nor for the task related data. I could think that there could be some limitations if the networks are trained for a long time:\n- For example, it could be that having a too-low rank during pre-training is bad for learning later on; since the rank is related to the expresiveness of the network this would be plausible (although it might not be the case).\n- It could also be that at the end of training a network trained only on data performs better at the cost of more computation (I personally don't think so, but it should be verified). Although I think I can see a performance plateau in Fig. 2 b, it should be put in logscale to see if the performance really saturated or if it can keep increasing. If it is the second, then the authors should see when does it stop increasing and whether random+data really achieves better end-of-training results. \n- A related point: I haven't seen the performances in numbers. While I would not care too much for them on the main text, they should be added somewhere just to clarify that the training is done to a reasonable level; if the performance is only up to 90% or something that can be achieved with a logistic regression on MNIST then there would be a problem.\nNote that this is not a critical flaw; it's still a good idea to pre-train for faster learning, even if that could potentially lead to slightly lower performances later on (at least where biology is concerned, 92% accuracy or 93% is not really a big deal).\n\nThere are also minor issues to correct or comment:\n- Line 141: The loss decreased noticeably during random training. Should clarify that they mean the training loss (sorry if it is a bit pedantic)\n- Definition 1has a wording problem: the claim is that Wl and Bl are aligned if the expected angle decreases asymptotically during learning. Lets say that we have W^* after learning. If I just happen to have initialized the weights as W* by chance from the beginning, the statement in lines 148 and 149 would tell me that those weights are not aligned (even though they are the same). \n\nA few related works:\n-  Error backpropagation without weight transport: While it is true that the weight transport is thought to be impossible exaclty it could be achieved by some form of hebbian-like learning, see for example Amit, Y. \"Deep learning with asymmetric connections and Hebbian updates.\" Frontiers in computational neuroscience 13 (2019). \n- STDP and backpropagation: Although it is true that STDP is not enough to explain learning, there are simple architectures that can implement learning very similar to Backprop using a rate-based version of STDP (Aceituno et al. \"Learning cortical hierarchies with temporal Hebbian updates\"). Importantly, that not all the architectures require weight symmetry for training (the paper menitoned has a table listing them)\n\nFinally, I wanted to add a suggestion that I think would improve the paper. But it involves math and I don't know if the authors are up for that, so feel free to ignore it.\nI think it should be possible to show that the pretraining would reduce the effective rank. In a \"batch\" of pre-training with S samples and C classes, backprop would push the activity to become aligned with one of the C output directions. More analytically,:\n-  the weight updates are Dw = x_pre delta_post, and delta_post is always one of S random vectors. \n- But there are only C classes, and thus the deltas within a class should be of a low rank statistically. More specifically\n  + Compute the expectation and variance over the Jacobian,\n  + for ReLus is the product of feed-forward weights from the current layer onwards, with a probability of 0.5 that a given neuron in the path will drop out of the derivative ).\n  + The variance will decrease with increasing S, while the expectation will remain constant.\n- Once you have c delta_post, you can be reasonably sure that the result of learning will map the activity of the previous layer into something covering only the span of delta_post. \n  + This can be done by taking approximating the averaged BP rule as a delta rule.\nI hope is not too unclear\n\nQUESTIONS:\nI don't get what is the task agnostic section supposed to show. If the point is that the network with pre-training is better than the network without pre-training in general, then why introduce concepts such as meta-loss or task-agnostic. It seems to me that is equally valid (and maybe better) to say simply that the random pre-training is great in all tasks (and exemplify by a few).\n\nLIMITATIONS:\nYes. But \n- 314-315: I don't get what the fact that the architecture that they mention does not use weight transport implies that more experimental studies are needed. to confirm their results. What does the lack of weight transport (which is probably true) have to do with more experiments? If they want to \"prove\" that weight transport is not present in cortex don't worry, no one thinks it's present. Anatomically the apical projections (top-down) are different than the basal ones anyway (bottom-up)", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the idea that the brain uses spontaneous prenatal activity to optimize its structure. This is done by showing the benefits of pre-training on random noise for artificial neural networks that are trained with the feedback alignment method - a training strategy that is more biologically-plausible than traditional backpropagation. Through experiments involving networks with and without pre-training, it is shown that pre-training enables these networks to reach convergence speeds comparable to backpropagation and enables them to generalize more effectively to various tasks.\n\nSTRENGTHS:\nThe explanations of both the theory and empirical results are clear and well explained, for the most part. \nInteresting and thorough analyses were conducted to demonstrate that random pre-training of an artificial neural network trained with feedback alignment results in quicker convergence and more greater task generalization properties. I particularly appreciated the variety of studies investigating the trajectories of various metrics (e.g., accuracy, weight alignment, etc.) over time.\n\nWEAKNESSES:\nI am unclear as to what the Definition in lines 147-149 is trying to convey. Does “cosine angle” mean cosine similarity? If so, does \\mathbb{E} signify this chosen loss function, or is it an expectation over the similarities between the forward weight matrices and random feedback matrices across individual neurons? Why is \\angle only used in front of the forward weight matrices?\nThroughout Figure 2, the authors show numerous analyses (i.e., b, c, e, and g) of how certain metrics change during training. However, the label of the X-axis says “No. of training inputs.” However, given that the various experiments outlined throughout the Appendix list the number of training epochs as 100, I am inclined to believe that that is what the authors meant instead. Further clarity in this matter would be appreciated.\nA minor grievance, but referring to models without pre-training as “untrained” can be misleading.\nWhile not critical to the overall message of the paper, Appendix sections C.4 and C.5 could be further strengthened by adding the training trajectories of a network trained with back-propagation for comparison.\n\nQUESTIONS:\nFigure 2.b demonstrates how a network that has undergone random pre-training converges more quickly than a network without pre-training. Do these plots account for the amount of “time” spent pre-training? If not, how much would considering the extent of pre-training close the gap in training speed between these two strategies?\nWhat is the overall contribution this paper wishes to make? I am not clear as to whether this paper wishes to a) validate established neuroscience findings using artificial neural networks or b) use insights from the brain to propose a way to improve current learning strategies in deep learning. If a), I have trouble understanding how recreating findings from experimental neuroscience in an artificial neural network trained with a more biologically-plausible algorithm constitutes a novel research contribution. If b), the empirical results here indicate that models trained with feedback alignment improve as the forward weights become more similar to the fixed backward weights - a property that is established in traditional artificial neural networks via the symmetry in their forward and backward weights. Has pre-training on random inputs shown any benefits for networks trained with traditional backpropagation (e.g., faster convergence, faster task generalization, increased accuracy, etc.)? Perhaps this would be an interesting investigation that establishes a benefit of biologically-inspired learning for traditional neural networks. \n\nAnswers to these questions might persuade me to move my score.\n\nLIMITATIONS:\nLimitations are somewhat stated and addressed. As mentioned, all of the experiments are run on a feedforward network. It could be worthwhile to verify to what extent these results are reproducible when using a different architecture, such as a recurrent or convolutional neural network.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores how pretraining neural networks with random noise can improve learning efficiency and generalization without relying on weight transport, inspired by spontaneous neural activity in developing biological brains.\n\nKey Findings:\n1. Random noise training aligns forward weights with synaptic feedback, enabling more precise error backpropagation without weight transport.\n2. Networks pretrained with random noise learn subsequent tasks faster and achieve performance comparable to backpropagation.\n3. Random noise pretraining acts as a form of regularization, reducing the effective dimensionality of weights and leading to more robust generalization.\n4. Pretrained networks show better performance on out-of-distribution tasks and adapt more quickly to various tasks.\n\nSTRENGTHS:\nThe paper investigates, to the best of my knowledge, a novel idea for pretraining neural networks to improve downstream feedback alignment. The paper is well written and easy to follow, experiments are nicely presented.\n\nWEAKNESSES:\nUnfortunately I think the evidence the paper presents is not enough to back up their claims. The bulk of the experiments only studied one-hidden neural network trained on MNIST. Results and claims will most likely be overestimated and will not hold in general. The paper also does not provide any theoretical insights why this initial pre-training might help.\n\nQUESTIONS:\n1) Please tune baselines. One can fool oneself very easily on the datasets and small networks you study. First, please grid search all hyperparameters in a reasonable range for the backdrop baseline and more crucially for the FA from scratch baseline. Try all common initialization schemes, and tune for them learning rate and batchsize. Please study carefully initialized random feedback weights. \n2) Please study the following crucial baseline: Initialize the feedback weights B with W transpose. Your claim is essentially that when you do this you train faster than with B random. I am highly doubtful of this, and therefore your claims. I agree that your pretraining might align the weights but I am quite confident that this will not lead to better optimization. In the beginning of training weights (so only W in your case) change quickly and therefore even initial perfectly aligned weights will not bring help much. \n3) How can you overcome the limitations of depth? What about layerwise local losses? If the result only hold for one-hidden layer networks, I am not confident to have this accepted at Neurips.\n\nLIMITATIONS:\nPlease mention that your results only apply to 1-hidden layer neural networks.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}