Towards Biologically Plausible Convolutional Networks
Authors: Roman Pogodin
Yash Mehta
Timothy P Lillicrap
Peter E. Latham
Venue: NeurIPS 2021 Poster (2021)
Keywords: biologically plausible deep learning
convolutional networks
Hebbian plasticity

Abstract: Convolutional networks are ubiquitous in deep learning. They are particularly useful for images, as they reduce the number of parameters, reduce training time, and increase accuracy. However, as a model of the brain they are seriously problematic, since they require weight sharing - something real neurons simply cannot do. Consequently, while neurons in the brain can be locally connected (one of the features of convolutional networks), they cannot be convolutional. Locally connected but non-convolutional networks, however, significantly underperform convolutional ones. This is troublesome for studies that use convolutional networks to explain activity in the visual system. Here we study plausible alternatives to weight sharing that aim at the same regularization principle, which is to make each neuron within a pool react similarly to identical inputs. The most natural way to do that is by showing the network multiple translations of the same image, akin to saccades in animal vision. However, this approach requires many translations, and doesn't remove the performance gap. We propose instead to add lateral connectivity to a locally connected network, and allow learning via Hebbian plasticity. This requires the network to pause occasionally for a sleep-like phase of "weight sharing". This method enables locally connected networks to achieve nearly convolutional performance on ImageNet and improves their fit to the ventral stream data, thus supporting convolutional networks as a model of the visual stream.

Towards Biologically Plausible
Convolutional Networks
Roman Pogodin
Yash Mehta Gatsby Unit, UCL y mehta@ucl ac.
Gatsby Unit, UCL
roman. pogodin. 17@ucl... ac.
Timothy P. Lillicrap
Peter E. Latham Gatsby Unit, UCL pel@gataby.
DeepMind: CoMPLEX, UCL coumtzero@goglle.com
wucpsbbyyycl.ooo ucl. ac..bby...l...uu.u
Abstract
Convolutional networks are ubiquitous in deep learning. They are particularly useful for images, they reduce the number of parameters, reduce training time, and increase accuracy. However, as a model of the brain they are seriously prob- lematic, since they require weight sharing something real neurons simply cannot do. Consequently, while neurons in the brain can locally connected (one the features of convolutional networks), they cannot be convolutional. Locally connected but non-cone non-convolutional networks, however, significantly underperform convolutional ones. This is troublesome for studies that convolutional networks to explain activity in the visual system. Here we study plausible alternatives to weight sharing that aim the same regularization principle, which is make each neuron within a pool react similarly to identical inputs. The most natural way to do that by showing the network multiple translations the same image, akin saccades in animal vision. However, this approach requires many translations, and doesn't It remove the performance gap. We propose instead add lateral connectivity to a locally connected network, and allow learning via Hebbian plasticity. This re- quires the network pause occasionally for sleep-like phase of "weight sharing". This method enables locally connected networks to achieve nearly convolutional performance on ImageNet and improves their fit the ventral stream data, thus supporting convolutional networks as model of the visual stream.
Introduction
Convolutional networks cornerstone of modern deep learning: they're widely used in the visual
domain [1I 3
speech recognition 4 text classification [5). and time series classification [
They have also played an important role in enhancing our understanding of the visual stream 7 Indeed, simple and complex cells in the visual cortex [8] inspired convolutional and pooling layers in deep networks (with simple cells implemented with convolution and complex ones with pooling). Moreover, the representations found in convolutional networks are similar to those in the visual
stream [10 [3 (see for an in-depth review).
Despite the success of convolutional networks at reproducing activity in the visual system, as a model of the visual system they somewhat problematic. That's because convolutional networks share weights, something biological networks, for which weight updates must be local, can't do [14] Locally connected networks avoid this problem by using the same receptive fields convolutional networks (thus locally connected), but without weight sharing [5]. However, they pay a price for biological plausibility: locally connected networks known to perform worse than their
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

convolutional counterparts on hard image classification tasks 15 [6 There is, therefore, need for a mechanism bridge the gap between biologically plausible locally connected networks and implausible convolutional ones.
Here, we consider such mechanisms. One is use extensive data augmentation (primarily image translations); the other is to introduce an auxiliary objective that allows some form of weight sharing, which is implemented by lateral connections; we call this approach dynamic weight sharing.
The first approach, data augmentation. is simple, but we show that suffers from two problems: requires far more training data than is normally used, and even then it fails to close the performance gap between convolutional and locally connected networks. The second approach, dynamic weight sharing, implements sleep-like phase in which neural dynamics facilitate weight sharing. This is done through lateral connections in each layer, which allows subgroups of neurons to share their activity. Through this lateral connectivity, each subgroup can first equalize its weights via anti- Hebbian learning, and then generate an input pattern for the next layer that helps it to do the same thing. Dynamic weight sharing doesn't It achieve perfectly convolutional connectivity, because in each channel only subgroups of neurons share weights. However, it implements a similar inductive bias, and, we show in experiments, it performs almost as well as convolutional networks, and also achieves better fit to the ventral stream data, as measured by the Brain-Score 12
Our study suggests that convolutional networks may be biologically plausible, as they can be approximated in realistic networks simply by adding lateral connectivity and Hebbian learning. As convolutional networks and locally connected networks with dynamic weight sharing have similar performance, convolutional networks remain a good "model organism" for neuroscience. This is important, they consume much less memory than locally connected networks, and run much faster.
2
Related work
Studying systems neuroscience through the lens of deep learning is an active area research, especially when it comes to the visual system [18]. As mentioned above, convolutional networks in particular have been extensively studied as a model of the visual stream (and also inspired by it) [ and also mentioned above, because they require weight sharing they lack biological plausibility. They have also been widely used to evaluate the performance of different biologically plausible learning rules [15
19
20
21
25).
Several studies have tried to relax weight sharing in convolutions by introducing locally connected networks [15
26 also shows that local connectivity itself can be learned from fully connected network with proper weight regularization). Locally connected networks perform as well as convolutional ones in shallow architectures [15] 26]. However, they perform worse for large networks and hard tasks, unless they're re initialized from an already well-performing convolutional solution [[6] or have some degree of weight sharing 2 In this study, seek biologically plausible regularizations of locally connected network to improve their performance.
16 26
Convolutional networks are not the only deep learning architecture for vision: visual transformers 30, BI) and more recently, the transformer-like architectures without self-attention have shown competitive results. However, they still need weight sharing: at each block the input image reshaped into patches, and then the same weight used for all patches. Our Hebbian-basee approach to weight sharing fits this computation as well (see Appendix A.4
(e.g..
28
29
32
33
64
3
Regularization in locally connected networks
3.1 Convolutional versus locally connected networks
Convolutional networks are implemented by letting the weights depend on the difference in indices. Consider, for simplicity, one dimensional convolutions and a linear network. Letting the input and output of a one layer in a network be zÿ and 2,, respectively, the activity in a convolutional network is
N
2
W Øー
(I)
j=l

---

Fully connected
Locally connected
Convolutional
O
1
21
1
1000
0
000
O 24
O
24
O
24
WI # W
W # W4
W W4
Figure I: Comparison between layer architectures: fully connected (left), locally connected (middle) and convolutional (right). Locally connected layers have different weights for each neuron to 24 (indicated by different colors), but have the same connectivity as convolutional layers.
where N is the number of neurons; for definiteness, we'll assume N is the same in in each layer (right panel in Fig. D Although the index j ranges over all neurons, many, not most, of the weights are zero: W is nonzero only when j 51 k/2 < N for kernel size k.
For networks that aren't convolutional, the weight matrix w,-j is replaced by wig.
N
WiyJyj.
Again, the index j ranges over all N neurons. all the weights are nonzero, the network is fully connected (left panel in Fig. D. But, as in convolutional networks, we can restrict the connectivity range by letting Wi be nonzero only when non-convolutional, network (center panel in Fig. M
k/2 < N, resulting in locally connected, but
3.2 Developing convolutional weights: data augmentation versus dynamic weight sharing
Here we explore the question: is it possible for a locally connected network develop approximately convolutional weights? That is, after training, is it possible to have Wa A w;-s? There is one straightforward way do this: augment the data to provide multiple translations of the same image. so that each neuron within channel learns to react similarly (Fig. ( A potential problem is that a large number of translations will be needed. This makes training costly (see Section 5), and is unlikely to be consistent with animal learning, as animals see only handful of translations of any one image.
A
B
O
1OOD
2
11
22
O
8000
-000
23
000
24
24
W ~ W2
W ~ W4
W3 ~ W4
Figure 2: Two regularization strategies for locally connected networks. A. Data augmentation, where multiple translations of the same image presented simultancously. B. Dynamic weight sharing, where a subset neurons equalizes their weights through lateral connections and learning.
less obvious solution to modify the network so that during learning the weights become approximately convolutional. As we show in the next section, this can be done by adding lateral connections, and introducing sleep phase during training (Fig. 33). This solution doesn't 't need more data, but it does need an additional training step.

---

A Hebbian solution to dynamic weight sharing
If we were to train locally connected network without any weight sharing or data augmentation, the weights of different neurons would diverge (region marked "training" in Fig. BA). Our strategy to make them convolutional is to introduce an occasional sleep phase, during which the weights relax to their mean over output neurons (region marked "sleep" in Fig. BA). This will compensate weight divergence during learning by convergence during the sleep phase. If the latter is sufficiently strong. the weights will remain approximately convolutional.
A
X1
W
+ W
W=
z (11
W1
2
X2
z z(x (x2)
W4
X3
z ( (33
sleep
training
sleep
time
Figure 3: A. Dynamical weight sharing interrupts the main training loop, and equalizes the weights through internal dynamics. After that, the weights diverge again until the next weight sharing phase. B. A locally connected network, where both the input and the output neurons have lateral connections. The input layer uses lateral connections to generate repeated patterns for weight sharing in the output layer. For instance, the output neurons connected by the dark blue lateral connections (middle) can receive three different patterns: X1 (generated by the red input grid), X2 (dark blue) and X3 (green).
To implement this, we introduce lateral connectivity, chosen to equalize activity in both the input and output layer. That's shown in Fig. BB. where every third neuron in both the input (x) and output (z) layers are connected. Once the connected neurons in the input layer have equal activity, all output neurons receive identical input. Since the lateral output connections also equalize activity, all connection that correspond to a translation by three neurons see exactly the same pre and postsynaptic activity. A naive Hebbian learning rule (with weight decay) would, therefore, make the network convolutional. However, we have to take care that the initial weighs are not over- over-written during Hebbian learning. We now describe how that is done.
To ease notation, we'll let w; be a vector containing the incoming weights to neuron i: (w:), = Waj- Moreover, we'll Il let run from to k independent of i. With this convention, the response of neuron i, 2, to a i-dimenniionl input, x, is given by
2,
WizTz.
(3)
j=1
Assume that every neuron sees the same x, and consider the following update rule for the weights,
Aw;
2
1
2j
x 2 (w, -wini))
N
where whiit the weights at the beginning of the sleep phase (not the overall training).
This Hebbian update effectively implements SGD over the sum of (z, - =, plus regularizer (the second term) to keep the weights near winit If we present the network with M different input
vectors, Xm., and denote the covariance matrix No Σm X then, asymptotically, the weight dynamics in Eq. 4 converges to (see Appendix
w/ = (C+g1)-
w wiit + +2 wyiii init
(5)
j=1
where is the identity matrix.

---

A
-log SNR
B
-log SNR
01
k-18
0-
min SNR
2
-2-
1e-1
l=l
41
44
le2
-6
-8
y=1e-2
-6*
-10
-8-
y- le-3
-12
le-3
101
-14
0.5k
1.5k 2k iter
2k
4k
6k
iter
Figure 4: Negative logarithm of signal-to-noise ratio (mean weight squared over weight variance, Eq. (6)) for weight sharing objectives in layer with 100 neurons. Different curves have different kernel size, (meaning k- inputs), and regularization parameter, y. A. Weight updates given by
Eq. 4 Black dashed lines show the theoretical minimum. B. Weight updates given by Eq. 8 with
10. In each iteration, the input presented for 150 ms.
As long as C is full rank and y is small, we arrive at shared weights: W ~ N E init It might seem advantageous
7 O, as non-zero 7 only biases the equilibrium value of the weight. However, non-zero ensures that for noisy input, X; x + & (such that the inputs to different neurons the same only average, which is much more realistic), the weights still converge (at least approximately) to the mean of the initial weights (see Appendix A
In practice, the dynamics in Eq. converges quickly. We illustrate it in Fig. A by plotting log S.R,, over time, where SNR the signal to noise ratio of the weights, is defined as
1 Σ 1
(, E. ((wi)) - + (we)))
SNR
For all kernel sizes (we used 2d inputs, meaning k.2 inputs per neuron), the weights converge to a nearly convolutional solution within few hundred iterations (note the logarithmic scale of the y axis in Fig. 4A). See Appendix for simulation details. Thus, to run our experiments with deep networks in realistic time frame, perform weight sharing instantly (i.e., directly setting them to the mean value) during the sleep phase.
4.1 Dynamic weight sharing in multiple locally connected layers
As shown in Fig. BB the k-dimensional input, x, repeats every neurons. Consequently, during the sleep phase, the weights not set to the mean of their initial value averaged across all neurons; instead, they're 're set to the mean averaged across a set of neurons spaced by k. Thus, one dimension, the sleep phase equilibrates the weights in i different modules. In two dimensions (the realistic case), the sleep phase equilibrates the weights in 2:2 different modules.
We need this spacing to span the whole k-dimensional (or k2 for 2d) space of inputs. For instance, activating the grid on the left in Fig. 3 generates X1, covering one input direction for all output neurons (and within each module, every neuron receives the same input). Next, activating the blue grid generates X2 (a new direction), and so on.
In multiple layers, the sleep phase is implemented layer by layer. In layer I, lateral connectivity creates repeated input patterns and feeds them layer / + After weight sharing layer the new pattern from + 1 is fed to 1 + 2, and so on. Notably, there's ' no layer by layer plasticity schedule (i.e., deeper layers don't have wait for the earlier ones finish). as the weight decay term in Eq. ( ensures the final solution is the same regardless of intermediate weight updates. As long as a deeper layer starts receiving repeated patterns, it will eventually arrive at the correct solution. Our approach has two limitations. First, this pattern generation scheme needs layers to have filters of the same size. Second, we assume that the very first layer (e.g., V1) receives inputs from another area (c.g., LGN) that can generate repeated patterns, but doesn't 't need weight sharing.
5

---

4.2 A realistic model that implements the update rule
Our update rule, Eq. implies that there is linear neuron, denoted rs. whose activity depends on the upstream input, 2, w; X, via direct excitatory connection combined with lateral inhibition,
N
=
w
The resulting update rule is anti- anti-Hebbian, -7; x (see Eq. In a realistic circuit, this can be implemented with excitatory neurons r; and an inhibitory neuron Tinh, which obey the dynamics
Tř: = - + w x
a rinh + b
(8a)
Thin Finh
Σ
rg b,
(8b)
where b is the shared bias term that ensures non non-negativity of firing rates (assuming E; w; is positive, which would be the case for excitatory input neurons). The only fixed point of this equations is
= N Σ w x a
Σ w
b+ asl
Σ N
W
(9)
which is stable. As a result, for strong inhibition (a >> 1), Eq. can be implemented with an anti-Hebbian term - (r: (o--bbx b)x. Note that if w. x is zero on average, then is the mean firing rate over time. To show that Eq. (9) provides enough signal, we simulated training in a network of 100 neurons that receives new each 150 ms. For range of l and y, it converged to a nearly convolutions
solution within minutes (Fig. 3 each iteration is 150 ms). Having finite inhibition did lead to worse final agnaalto-noise ration (a 10 in Fig. 3), but the variance of the weights was still very small. Moreover, the nature of the a-induced bias suggests that stopping training before convergence leads better results (around 2k iterations in Fig. B). See Appendix Afror a discussion.
5 Experiments
We split our experiments into two parts: small-scale ones with CIFARIO, CIFAR100 35 and TinylmageNet (36), and Icage-scale ones with ImageNet (37. The former illustrates the effects of data augmentation and dynamic weight sharing on the performance of locally connected networks; the latter concentrates on dynamic weight sharing, as extensive data augmentations are too com- putationally expensive for large networks and datasets. We used the AdamW [88 optimizer in all runs. As our dynamic weight sharing procedure always converges to nearly convolutional solution (see Section 4. set the weights to the mean directly (within each grid) to speed up experiments. Our code is available at implementation).
bio- phi--plusible-comn (Py PoTch [39
Datasets. CIFARIO consists of 50k training and 10k test images of size 32 32, divided into 10 classes. CIFAR 100 has the same structure, but with 100 classes. For both, we tune hyperparameters with 45k/5k train/validation split, and train final networks on the full 50k training set. Imagent consists of 100k training and 10k validation images of size 64 x 64, divided into 200 classes. As the test labels are not publicly available, we divided the training set into 90k/10k train/validation split, and used the 10k official validation set as test data. ImageNet consists of 1.281 million training images and 50k test images of different sizes, reshaped to 256 pixels in the smallest dimension. As in the case for TinyImageNet, we used the train set for 1.271 million/ 10k train/ validdaiiition split, and 50k official validation set as test data.
Networks. For CIFAR 10/100 and TinylmageNet, we used CIFAR 10- adaateed ResNet20 from the original ResNet paper [ The network has three blocks, each consisting of layers, with 16/32/64 channels within the block. We chose this network due to good performance on CIFAR10, and the ability to the corresponding locally connected network into the 8G VRAM of the GPU for large batch sizes on all three datasets. For ImageNet, we took the half-width ResNet18 (meaning 32/64/1 28/256 block widths) to be able to fit a common architecture (albeit halved in width) in the locally connected regime into 16G of GPU VRAM. For both networks, all layers had 3 x 3 receptive

---

Table Performance of convolutional (conv) and locally connected (LC) networks for padding of in the input images (mean accuracy over 5 runs). For LC, two regularization strategies were applied: repeating the same image n times with different translations (n reps) or using dynamic weight sharing every n batches (ws(n)). LCC nets additionally show performance difference w.r.t. conv nets.
Regulurizer
Connectivity
CIFAR10
CIFAR100
Tiny ImageNet
Top-1 accuracy (%)
Diff
Top-1 accumacy
Diff
Top-5 accuracy (1)
Diff
Top-1 uccuracy (S)
Diff
Top-5 accuracy (8)
Diff
QOIV LC
88.3
59.2
84,9
38.6
65.1
80,9
-7.4
49.8
-9.4
75.5
-9.
29.6
-9.0
52.7
-124
4C-4 reps
82.9
-5.4
52.1
-7.1
76.4
-8.5
31,9
-6.7
54,9
-10.2
Data Trunslation
LCC 8 regs
83.8
.4.5
54.3
-5.0
77.9
-7.0
33,0
-5.6
55.6
-95
reps
85.0
-3.3
-6.1
34.0
-46
56.2
LC - w)
87.4
-0.8
58.7
-0.5
83.4
-1.6
41.6
3.0
66.1
1.1
Weight Sharing
LCC ws((()
85.1
-3.2
55.7
-3.6
80.9
-4.0
37.4
-1.2
61.8
-3.2
ws(100)
82.0
-48
37.1
-1.5
62.8
-2.
field (apart from few x 1 residual downsampling layers), meaning that weight sharing worked over 9 individual grids each layer.
Training details. We ran the experiments on our local laboratory cluster, which consists mostly of NVIDIA GTX1080 and RTX5000 GPUs. The small-scale experiments took from 1-2 hours per run up to 40 hours (for TinylmageNet with 16 repetitions). The large-scale experiments took from 3 to 6 days on RTX5000 (the longest run was the locally connected network with weight sharing happening after every minibatch update).
5.1 Data augmentations.
For CIFARIO/I0K we padded the images (padding size depended on the experiment) with mean values over the training set (such that after normalization the padded values were zero) and cropped to size 32 32. We did not use other augmentations to separate the influence of padding/random crops. For TinylmageNet, we first genter-cropped the original images to size (48 + 2 pad) x (48 + 2 pad) for the chosen padding size pad. The final images were then randomly cropped to 48 48. This was done to simulate the effect of padding on the number of available translations, and to compare performance across different padding values on the images of the same size (and therefore locally connected networks of the same size). After cropping, the images were normalized using ImageNet normalization values. For all three datasets, test data was sampled without padding. For ImageNet, we used the standard augmentations. Training data was resized to 256 (smallest dimension), random cropped to 224 x 224, flipped horizontally with 0.5 probability, and then normalized. Test data was resized to 256, center cropped to 224 and then normalized. In all cases, data repetitions included multiple samples of the same image within a batch, keeping the total number of images in a batch fixed (e.g. for batch size 256 and 16 repetitions, that would mean 16 original images)
5.2 CIFAR10/100 and TinyImageNet
To study the effect of both data augmentation and weight sharing on performance, we ran experiments with -on-anement images (padding 0) and with different amounts of augmentations. This included padding of 4 and 8, and repetitions networks performed much worse than convolutional, although weight sharing improved the result little bit (see Appendix B.. For padding of (mean accuracy over 5 runs Table T see Appendix B for max-min accuracy), increasing the number of repetitions increased the performance of locally connected networks. However, even for 16 repetitions, the improvements were small comparing to weight sharing (especially for top-5 accuracy TinyImageNet). For CIFAR 10, our results are consistent with an earlier study of data augmentations in locally connected networks [40]. For dynamic weight sharing, doing it moderately often every 10 iterations, meaning every 5120 images For TinyImageNet, sharing weights every 100
4, 8, and 16. Without augmentations, locally connected
did as well as 16 repetitions on CIFAR iterations (about every 50k images) performed much better than data augmentation.
Sharing weights after every batch performed almost as well as convolutions (and even bit better on TinyImageNet although the difference is small if we look at top-5 accuracy, which is less volatile metric for 200 classes), but is too frequent to be a plausible sleep phase. We include to show that best possible performance of partial weight sharing is comparable to actual convolutions.
7

---

Table Performance of convolutional (conv) and locally connected (LC) networks on ImageNet for 0.5x width R (1 run). For LC, we also used dynamic weight sharing every batches. LC nets additionally show performance difference w.r.t. the conv net.
Weight sharing
ImageNet
Connectivity
frequency
Top-1 accuracy (%)
Top-5 accuracy (%)
Diff
Diff
conv
63.5
84.7
LCC
46.7
-16.8
70.0
-14.7
I.C.
1
61.7
-1.8
83.1
-1.6
LC
10
59.3
-4.2
81.1
-3.6
L.C
100
54.5
-9.0
77.7
-7.0
For padding of 8, the performance did improve for all methods (including convolutions), but the relative differences had a similar trend as for a padding of (see Appendix [B.. We also trained locally connected networks with one repetition, but for longer and with much smaller learning rate to simulate the effect of data repetitions. Even for 4x-8x longer runs, the networks barely matched the performance of I-repetition network on standard speed (not shown).
5.3 ImageNet
On ImageNet, we did not test image repetitions due to the computational requirements (c.g., run- ning 16 repetitions with resources would take almost months). We used the standard data augmentation, meaning that all networks see different crops the same image throughout training.
Our results are shown in Table 2 Weight sharing every and 10 iterations (256/2560 images, respectively, for the batch size of 256) achieves nearly convolutional performance, although less frequent weight sharing results in more significant performance drop. In contrast, the purely locally connected network has a large performance gap with respect to the convolutional one. It is worth noting that the trade-off between weight sharing frequency and performance depends on the learning rate, as weights diverge less for smaller learning rates. It should possible to decrease the learning rate and increase the number of training epochs, and achieve comparable results with less frequent weight sharing.
5.4 Brain-Score ImageNet-trained networks
In addition to ImageNet performance, we evaluated how well representations built by our networks correspond to ventral stream data primates. For that we used the Brain-Score 112 [7], a set of metrics that evaluate deep networks' correspondence to neural recordings of cortical areas V1 V4 and inferior temporal (IT) cortex The advantage of Brain-Scoun is that it provides standardized benchmark that looks at the whole ventral stream, and not only its isolated properties like translation invariance (which many early models focused on 45 46 47
41
42
V2
primates, as well as behavioral
data
We do not directly check for translation invariance in models (only through V1/V2 data). However, as our approach achieves convolutional solutions (see above), trivially have translation equivariance after training: translating the input will translate the layer's s response (in our case, each k-th translation will produce the same translated response for kernel of size k). (In fact, it's pooling layers, not convolutions, that achieve some degree of invariance in convolutional networks - this is an architectural choice and is somewhat tangential to our problem.)
48
Our results are shown in Table 3 (higher better; see http:://wwwbbrai wttp:/wwwaii-scor..rrrgg scrrssocrrrrrr///ooo for the scores of other models). The well-performing locally connected networks (with weight sharing/slecp phase every or 10 iterations) show overall better fit compared to their fully convolutional counterpart meaning that plausible approximations to convolutions also form more realistic representations. Interestingly, the worst-performing purely locally connected network had the second best V1 fit, despite overall poor performance.
In general, Brain-Score correlates with ImageNet performance (Fig. 2 in 12] This means that worse Brain-Score performance of the standard locally connected network and the one with weight sharing every 100 iterations can be related to their poor ImageNet performance (Table 2 (It also means that our method can potentially increase Brain-Score performance of larger models.)

---

Table Brain-Scont of ImageNee-trained convolutional (conv) and locally connected (LC) networks on ImageNet for 0.5x width ResNet18 (higher is better). For LC, we also used dynamic weight sharing every n batches. "The models were evaluated on Brain-Score benchmarks available during submission. If new benchmarks are added and the models are re-evaluated on them, the final scores might change; the provided links contain the latest results.
Weight sharing
Brain-Score
Connectivity
frequency
average score
V1
V2
V4
IT
behavior
link
conv LC
.357
.493
.313
.459
.370
.148
brain-score score
.349
.542
.291
.448
.354
.108
brain-scor sounn-cooe.coo
L.C LC LC
.396
-512
.339
.468
.406
.255
brann-scmee.r///mod////
10
.385
.508
.322
.478
.399
.216
bran-score.
100
.351
.523
.293
.467
.370
.101
wrann-coore.or/mmon
6
Discussion
We presented two ways to circumvent the biological implausibility of weight sharing, a crucial component of convolutional networks. The first was through data augmentation via multiple image translations. The second was dynamic weight sharing via lateral connections, which allows neurons to share weight information during a sleep-like phase; weight updates are then done using Hebbian plasticity. Data augmentation requires a large number of repetitions in the data, and, consequently, longer training times, and yields only small improvements in performance. However, only a small number of repetitions can be naturally covered by saccades. Dynamic weight sharing needs separate sleep phase, rather than more data, and yields large performance gains. In fact, it achieves near convolutional performance even on hard tasks, such as ImageNet classification, making it much more likely candidate than data augmentation for the brain. In addition, well performrfnn locally connected networks trained with dynamic weight sharing achieve a better fit to the ventral stream data (measured by the Brain-Score 122 [7].. The sleep phase can occur during actual sleep, when the network (i.e., the visual system) stops receiving visual inputs, but maintains some internal activity. This is supported by plasticity studies during sleep (e.g.
There are several limitations to our implementation of dynamic weight sharing. First, it relies on precise lateral connectivity. This can be genetically encoded, or learned early on using correlations in the input data (if layer 1 can generate repeated patters, layer 1 + 1 can modify its lateral connectivity based on input correlations). Lateral connections do in fact exist in the visual stream, with neurons that have similar tuning curves showing strong lateral connections [5]]. Second, the sleep phase works iteratively over layers. This can be implemented with neuromodulation that enables plasticity one layer at a time. Alternatively, weight sharing could work simultaneously in the whole network due to weight regularization (as ensures that the final solution preserves the initial average weight), although this would require longer training due to additional noise in deep layers. Third, in our scheme the lateral connections are used only for dynamic weight sharing, and not for training or inference. As realistic model in Section 2 implements this connectivity via an inhibitory neuron, we can think of that neuron as being silent outside of the sleep phase. Finally, we trained networks using backpropagation, which is not biologically plausible [18]. However, weight sharing scheme is independent of the make-ppase training algorithm, and therefore can be applied along with any biologically plausible update rule.
Our approach to dynamic weight sharing is not relevant only to convolutions, First, is applicable to non-convolutional networks, and in particular visual transformers 28, 29 3C 31 (and more recent MLP-based architectures [32] [ BE.)). In such architectures, input images (and intermediate two- dimensional representations) are split into non-overlapping patches; each patch is then transformed with the same fully connected layer a computation that would require weight sharing in the brain. This can be done by connecting neurons across patches that have the same relative position, and applying our weight dynamics (see Appendix A.4). Second, (23 faced a problem similar to weight sharing- weight transport (i.e., neurons not knowing their output weights) when developing plausible implementation of backprop. Their weight mirror algorithms used an idea similar to ours: the value of one weight was sent another through correlations in activity.
Our study shows that both performance and the computation of convolutional networks can be reproduced in more realistic architectures. This supports convolutional networks model of the

---

visual stream, and also justifies them as "model organism" for studying learning in the visual stream (which is important partially due to their computational efficiency). While our study does not have immediate societal impacts (positive or negative), it further strengthens the role of artificial neural networks as a model of the brain. Such models can guide medical applications such as brain machine interfaces and neurological rehabilitation. However, that could also lead to the design of potentially harmful adversarial attacks on the brain.
Acknowledgments and Disclosure of Funding
The authors would like to thank Martin Schrimpf and Mike Ferguson for their help with the Brain- Score evaluation.
This work was supported by the Gatsby Charitable Foundation, the Wellcome Trust and DeepMind.
References
[1] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25: 1097 1105, 2012.
[2] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprini arXiv: 1556, 2014.
[3] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEÉ conference on computer vision and pattern recognition, pages 770-778, 2016.
[4] Ossama Abdel-Hamid, Abdel-rahman Mohamed, Hui Jiang, Li Deng, Gerald Penn, and Dong Yu. Convolutional neural networks for speech recognition. IEEE/ACM Transactions on audio, speech, and language processing, 22(10):1533-155 153-- 2014.
[5] Alexis Conneau, Holger Schwenk, Loic Barrault, and Yann Lecun. Very deep convolutional networks for text classification. arXiv preprint arXiv:
2016.
[6] Fazle Karim, Somshubra Majumdar, Houshang Darabi, and Shun Chen. Lstm fully convolutional networks for time series classification. IEEE access, 6:1662-16666 1669, 2017.
[7] Grace W. Lindsay. Convolutional Neural Networks as Model of the Visual System: Past, Present, and Future. Journal of Cognitive Neuroscience, pages 1-15, 02 2020.
[8] David H Hubel and Torsten N Wiesel. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. The Journal of physiology, 160(1):106-154, 1962.
[9] Kunihiko Fukushima and Sei Miyake. Neocognitron: self-organizing neural network model for mechanism of visual pattern recognition. In Competition and cooperation in neural nets, pages 267-285. Springer, 1982.
[10] Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James J DiCarlo. Performance-optinn
hicrarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, 1 (11()))8619-862) 2014.
[11] Seyed-Mahdi Chaligh-Razav and Nikolaus Kriegeskorte. Deep supervised, but not un- supervised, models may explain it cortical representation. PLoS computational biology, 10(11)xe003311 2014.
[12] Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J. Majaj. Rishi Rajalingham, Elias B. Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Franziska Geiger, Kailyn Schmidt, Daniel
L. K. Yamins, and James J. DiCarlo. Brain-score: Which artificial neural network for object recognition is most bike?-like? bioRxiv preprint, 2018.
[13] Santiago A Cadena, George Denfield, Edgar Y Walker, Leon A Gatys, Andreas S Tolias. Matthias Bethge, and Alexander S Ecker. Deep convolutional models improve predictions of macaque responses to natural images. PL computational biology,
2019.
[14) Stephen Grossberg. Competitive learning: From interactive activation to adaptive resonance. Cognitive science, 11(1):23-63, 1987.
10

---

[15] Sergey Bartunov, Adam Santoro, Blake Richards, Luke Marris, Geoffrey E Hinton, and Timothy Lillicrap. Assessing the scalability of biologically
deep learning algorithms and architectures. In Advances in Neural Information Processing Systems, pages 9368-99 (968-9378, 2018. [16] Stéphane d' Ascoli, Levent Sagun, Joan Bruna, and Giulio Biroli. Finding the needle in the haystack with convolutions: on the benefits of architectural bias. arXiv preprint arXiv: 190.00666.. 2019.
[17] Martin Schrimpf, Jonas Kubilius, Michael J Lee, N Apurva Ratan Murty, Robert Ajemian, and James J DiCarlo. Integrative benchmarking to advance neurally mechanistic models of human intelligence. Neuron, 2020.
[18] Blake A Richards, Timothy P Lillicrap. Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, et al. A deep learning framework for neuroscience. Nature neuroscience, 22(1 22(11):1711-177) 2019.
[19] Arild Nøkland. Direct feedback alignment provides learning in deep neural networks. In Advances in neural information processing systems, pages 1037-1045, 2016.
[20] Theodore H Moskovitz, Ashok Litwin- Kumar, and LF Abbott. Feedback alignment in deep convolutional networks. arXiv preprint arXiv: 8/2044888/.006488, 2018.
[21] Hesham Mostafa, Vishwajith Ramesh, and Gert Cauwenberghs. Deep supervised learning using local errors. Frontiers in neuroscience, 12:608, 2018.
[22] Arild Nøkland and Lars Hiller Eidnes. Training neural networks with local error signals. arXiv 2019.
preprint 100XXv::9011000000
[23] Mohamed Akrout, Collin Wilson, Peter Humphreys, Timothy Lillicrap, and Douglas B Tweed.
Deep learning without weight transport. In Advances Neural Information Processing Systems, pages 974-982, 2019.
[24] Axel Laboricux, Maxence Ernoult, Benjamin Scellier, Yoshua Bengio, Julie Grollier, and Damien Querlioz. Scaling equilibrium propagation to deep convnets by drastically reducing its gradient estimator bias. arXiv preprint 2xXiv:0006.03224 2020.
[25] Roman Pogodin and Peter E. Latham. Kernelized information bottleneck leads to biologically plausible 3-factor hebbian learning in deep networks. arXiv preprint arXiv:2006.07123, 2020.
[26] Behnam Neyshabur. Towards learning convolutions from scratch. arXiv preprint
2PXVv:2007.33577 2020.
[27] Gamaleldin Elsayed, Prajit Ramachandran, Jonathon Shlens, and Simon Korbllihh. Revis- iting spatial invariance with low-rank local connectivity. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning. volume 119 of Proceedings of Machine Learning Research, pages 2868-288888 2879. PMLR, 13-18 Jul 2020.
[28) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Mathias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:20/0./ 1929, 2020.
[29] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pages 213-229. Springer, 2020.
[30] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint arXiv:2012.12877 2020.
[31] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04155 2020.
[32] Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai Thomas Unterthiner, Jessica Yung, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy, Mlp-mixer: An all-mlp architecture for vision, 2021.
[33] Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, and Hervé Jégou. Resmlp: Feedforward networks for image classification with data-efficient training. arXiv preprint 0rXiv:2105.03404 2021.
11

---

[34] Hanxiao Liu, Zihang Dai, David R So, and Quoc V Le. Pay attention to mlps. arXiv preprint arXiv:2105.08000, 2021.
[35] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[36] Ya Le and Xuan Yang, Tiny imagenet visual recognition challenge. CS 231N, 7:7, 2015.
[37] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255 leee, 2009.
[38] Iyya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:/711.051000 2017.
[39]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf. Edward Yang, Zachary De Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019.
[40] Jordan Ott, Erik J. Linstead, Nicholas LaHaye, and Pierre Baldi. Learning in the machine: To share or not to share? Neural networks the official journal of the International Neural Network Society, 126:235-2444 2292 2020.
[41] Jeremy Freeman, Corey M. Ziemba, David Heeger, Eero Simoncelli, and J. Anthony Movshon. A functional and perceptual signature of the second visual area in primates. Nature Neuroscience, 16(7):974-981, Jul 2013.
[42] Tiago Marques, Martin Schrimpf, and James J DiCarlo. Multi-scale hierarchical neural network models that bridge from single neurons in the primate primary visual cortex to object recognition behavior. bioRxiv, 2021.
[43] Najib J. Majaj. Ha Hong, Ethan A. Solomon, and James J. DiCarlo. Simple learned weighted sums of inferior temporal neuronal firing rates accurately predict human core object recognition performance. Journal of Neuroscience, 35439)11402-1144 2015.
[44] Rishi Rajalingham, Elias B. Issa, Pouya Bashivan, Kohitij Kar, Kailyn Schmidt, and James J. DiCarlo. Large-scale, high-resolution comparison of the core visual object recognition behavior of humans, monkeys, and fate-of-the-as deep artificial neural networks. bioRxiv, 2018.
[45] Maximilian Riesenhuber and Tomaso A. Poggio. Hicrarchical models of object recognition in cortex. Nature Neuroscience, 2:1019-1025, 1999.
[46] Peter Földiák. Learning invariance from transformation sequences. Neural computation, 3(2):194-200, 1991.
[47] Guy Wallis, Edmund Rolls, and Peter Foldiak. Learning invariant responses to the natural transformations of objects. In Proceedings of 1993 International Conference on Neural Networks (JCCNN-93-Nagoyu, Japan), volume 2, pages 1087-1090. IEEE, 1993.
[48] Laurenz Wiskott and Terrence Sejnowski. Slow feature analysis: Unsupervised learning of invariances. Neural computation, 14(4):715-77) 2002.
[49] Sushil K Jha, Brian E. Jones, Tammi Coleman, Nick Steinmetz, Chi-Tat Law, Gerald D. Griffin, Joshua D. Hawk, Nooreen Dabbish, Valery A. Kalatsky, and Mareos G. Frank. dlee-deeenndent
plasticity requires cortical activity. The Journal of Neuroscience, 25:9266- - 9274, 2005.
[50] Carlos Puettes-Mestril and Sara J Aton. Linking network activity to synaptic plasticity during sleep: hypotheses and recent data. Frontiers in neural circuits, 1 11:61, 2017.
[51] William H. Bosking, Ying Zhang, Brett Schofield, and David Fitzpatrick. Orientation selectivity and the arrangement of horizontal connections in tree shrew striate cortex. J. Neurosci., 15, 1997.
[52] Robert Mansel Gower, Nicolas Loizou, Xun Qian, Alibek Sailanbayev, Egor Shulgin, and Peter Richtárik. Sgd: General analysis and improved rates. In International Conference on Machine Learning, pages 5200-5209. PMLR, 2019.
[53] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. Proceedings of the IEEE international conference computer vision, pages 1026-1034, 2015.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section C
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See the supplementary material.
(b) Did you include complete proofs of all theoretical results? [Yes] See the supplementary material.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material URL)? [Yes] See http://www...ooooooaaoo/o//////a http://dd///owwaadbbb----aaaaaae bio-plausible-com and the supplementary material.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See the supplementary material.
(c) Did you report error bars (e.g., with respect to the random seed after running ex- periments multiple times)? [Yes] See the supplementary material for scall-scale experiments. [No] The large-scale experiments were run I time due to resource con- straints.
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes] See Section S
4. If you using existing assets (e.g., code, data, models) curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No] We used publicly available datasets and open-source repositories.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] Implementation:
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No] We used publicly available datasets.
(c) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [No] We used publicly available datasets.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "WOopp5wkZdc", "reviewer": "Reviewer_t9AE", "summary": "The paper proposes a biologically plausible substitute for convolutional networks, which uses locally connected networks with data augmentation and a weight-sharing procedure. The experiments on some benchmark datasets show that the proposed algorithm achieves good results close to convolutional networks.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n\nThe paper proposes the weight sharing procedure that is the key point of the algorithm, which enforces the weights of each neuron to be the same and close to their average. This idea is interesting and novel. It shows from simulations the weights could converge with a relatively small number of iteration, and also gives a realistic neural dynamic model to justify the weight sharing update. The experiments on CIFAR and ImageNet data show adding weight sharing can improve the LC network performance. \n\nWeakness\n\n1. The weight sharing update rule is given in (4), which is also Hebbian's learning rule and effectively drives all the weights to the mean. Here z_i should be taken as the input signal from previous layer x, and (z_i - 1/N\\sum z_j) should be the activation of the neuron (fire rate). It is a little bit strange that in the lateral connections, the neuron j sends its input signal z_j directly to the neuron i, instead of the activations. Maybe there are some steps omitted, but I think the authors should make the dynamics clearer.\n2. I'm not sure that in the real data experiments, whether the authors implement the update rule (4) or simply compute the final solution (5), or even just take the mean of the weights. Also, if using (4) or (5), does the covariance matrix C computed from current batch or all previous batch, I was not able to find these details in the paper.\n3. It can be observed from table 3 in Appendix B that the results are much worse when removing the padding, and also increase the frequency of weight sharing operation is not very helpful in improving accuracy. It shows padding plays a more important role in the model and improvements from the weight sharing procedure are relatively marginal.\n\n\n--- after rebuttal ---\n\nI appreciate the author's feedback. After reading the response and other reviews, I would like to keep my original rating for this paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "HgVj4iJxzNJ", "reviewer": "Reviewer_M4Vu", "summary": "The paper presents a way to increase biological plausibility in convolutional neural networks. In particular, this paper focuses on two alternatives to weight sharing that aim at the same regularization principle. In particular, the authors propose data augmentation via multiple image translations and, much more relevant, dynamic weight sharing via lateral connections, which allows neurons to share weight information during a sleep-like phase. The paper is clear, well-structured, and I think it has value and interest for the community. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper presents a way to increase biological plausibility in convolutional neural networks. In particular, this paper focuses on two alternatives to weight sharing that aim at the same regularization principle. In particular, the authors propose data augmentation via multiple image translations and, much more relevant, dynamic weight sharing via lateral connections, which allows neurons to share weight information during a sleep-like phase. The paper is clear, well-structured, and I think it has value and interest for the community. \n\nMy main concern, or question/comment, is related with the main motivation of this paper: What is the problem with the fact that ConvNets are biologically implausible or that could be incompatible with current understandings in neuro-biology? I consider that all proposals to improve deep nets training or design are welcome, but I do not see the need of explicitly searching per se for biologically realistic algorithms. As Yann LeCun would say (https://cs.nyu.edu/~yann/talks/lecun-ranzato-icml2013.pdf): \"Let's be inspired by nature, but not too much\". It is indeed nice to be able to emulate/imitate nature, but we mainly need to understand what is actually relevant for our practical purposes: \"How do we know which details are important? How do we know which details are merely the result of evolution, and the constraints of biochemistry? For airplanes, we developed aerodynamics and compressible fluid dynamics, and we figured out that feathers and wing flapping were not crucial\". From this point of view, I think that any research endeavor is worth of trying, but I would like to see a better contextualization and/or justification of why we need more biologically plausible models. For instances, the authors also remark that \"we trained networks using backpropagation, which is not biologically plausible\". Why is that problematic? In other words, from my point of view, being inspired by nature is fine, but that inspiration should be translated into meaningful and justified design of the method in question, which not only should make sense, but also be qualitatively novel and quantitatively advantageous with regard to previous approaches. \n\nRelated to the aforementioned comment, there are some statements in the paper that remain a bit unclear to me:\n- \"Such models can guide medical applications such as brain machine interface and neurological rehabilitation. However, that could also lead to the design of potentially harmful adversarial attacks on the brain\". How the introduced approach could be used in neurological rehabilitation? What is an adversarial attack on the brain?  \n- \"it achieves near convolutional performance even on hard tasks, such as ImageNet classification, making it a much more likely candidate than data augmentation for the brain\". What does it mean to be a \"more likely candidate [...] for the brain\"?\n\nFrom the methodological point of view, I have several comments/questions that, in my humble opinion, if taken into account would make the paper stronger: \n- In the context of ConvNets for image classification problems, locally connected networks, if I'm not mistaken, according to Figure 1, imply the use of different filters for different regions of the image. Is that correct?\n- Does this dynamic weight sharing via lateral connections slow down the training procedure? By how much?\n- \"The lateral connections are used only for dynamic weight sharing, and not for actual computation\". What does this mean exactly?\n- What is the actual difference in terms of parameters to train between the ConvNets and the LC networks employed in the experiments?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, they do, at the end of Section 4.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "8IZhkmkTVJ", "reviewer": "Reviewer_eCob", "summary": "The main point of this paper is that we can rest easy about the biological plausibility of convolutional networks, because a simple hebbian mechanism - along with a “sleep phase” - provides  a method for making initially random weights match up, i.e., makes a locally connected network convolutional. The hebbian learning mechanism moves weights at every kth interval to the same value, while keeping them from moving very far from their initial value. \n\nThrough experiments on several small datasets, the authors show that this mechanism works better than simply using lots of translations of the image to make the weights similar. In addition, the performance of the hebbian-regularized model is not far from a standard convnet, and on Tiny ImageNet, it actually performs a bit better than a standard convnet. \n\nOn ImageNet, the mechanism results in performance similar to a regular convolutional network.\n\nI have read the authors' response, and I am going to stick with my rating. If the paper does not get into NeurIPS, I encourage the authors to continue their work and submit it somewhere else, e.g., WACV or Vision Research.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: As far as I know, this is original.\n\nQuality: The work appears sound. There is a proof of the dynamics of the hebbian part in the appendix that I didn’t check. The claims in the abstract are well supported by the results.\n\n Clarity: Is the submission clearly written? Yes: the paper is quite clear.\n\nSignificance: Are the results important? To the extent that they give a biologically-plausible mechanism for generating nearly-convolutional networks, yes. One has to care about such things; I do. The work provides a justification of layered convolutional networks as models of the visual system. \n\nMinor comments:\nline 227: it is to frequent -> it is too frequent.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors are clear about the limitations; the model works best when the sleep phase is implausibly frequent. The model requires fairly precise connectivity, and there is a somewhat implausible sequential application of the hebbian part, where layer l is regularized before layer l+1. While the authors suggest that this could be done simultaneously, but would take longer, this is not demonstrated. \n\nIn terms of societal impact, I don’t see any way that this would have societal impact, and neither do the authors.\n", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "4qJqJFC9m9-", "reviewer": "Reviewer_uAw7", "summary": "This paper designed a convolutional neural network that relaxed the weight-sharing constraint by adding local lateral connections to implement dynamic weight sharing as a \"sleep phase\". The idea was inspired by the biological neural network, where a strict weight-sharing mechanism does not exist. They further used a Hebbian learning rule for weight update during the sleep phase. Their results suggest such a design may have comparable performance as the traditional CNN on image recognition tasks, and it is more computationally efficient than using data augmentation (e.g., translation) to force weight-sharing.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Questions and Comments:\n1. Is each dynamic weight sharing update (the \"sleep phase\") an iterative process? Or, more specifically, is it a one-step update as described in Eq. (4), or does it converge to Eq. (5) during one intermediate \"sleep phase\" before starting the next \"training phase\"?\n2. Fig. 3A illustrates the assumption that the model design will reconcile weight divergence during separate learning and weight convergence during the sleep phase. Has the author verified whether the proposed model actually exhibits this learning behavior during training?\n3. The result in Table 1 suggests that more frequent weight sharing (e.g., once every batch) shows better visual task performance. Is this because \"equalizing\" the weights every batch may force the learning process to avoid large divergence that cannot be well-compensated by the \"sleep phase\"?\n4. In this work, the authors remove the weight-sharing constraint in the CNN but introduce a \"sleep phase\" as a weight regularization process. As the original motivation is to make the model more \"brain-like\", what is the analogy of this \"sleep phase\" in the biological neural network? Is there any inspiration about how and when this dynamic weight-sharing process should happen during the learning process?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received 3 marginal accepts and 1 accept. The reviewers stated that their lack of enthusiasm had to do with the somewhat niche aspect of the work (not because of technical or experimental limitations). Many NeurIPS attendees do care about biology and the work should be of interest to this community as noted by one of the reviewers. The AC thus recommends acceptance.

Decision: Accept (Poster)