Architecture Agnostic Neural Networks
Authors: Sabera J Talukder
Guruprasad Raghavan
Yisong Yue
Venue:  (2021)
Keywords: Architecture Agnostic
Sparse
Binary
Stochastic
Pruning
Biologically Inspired

Abstract: In this paper, we explore an alternate method for synthesizing neural network architectures, inspired by the brain's stochastic synaptic pruning. During a person’s lifetime, numerous distinct neuronal architectures are responsible for performing the same tasks. This indicates that biological neural networks are, to some degree, architecture agnostic. However, artificial networks rely on their fine-tuned weights and hand-crafted architectures for their remarkable performance. This contrast begs the question: Can we build artificial architecture agnostic neural networks? To ground this study we utilize sparse, binary neural networks that parallel the brain’s circuits. Within this sparse, binary paradigm we sample many binary architectures to create families of architecture agnostic neural networks not trained via backpropagation. These high-performing network families share the same sparsity, distribution of binary weights, and succeed in both static and dynamic tasks. In summation, we create an architecture manifold search procedure to discover families of architecture agnostic neural networks.

Under review as a conference paper at ICLR 2021
ARCHITECTURE AGNOSTIC NEURAL NETWORKS
Anonymous authors
Paper under double-blind review
ABSTRACT
In this paper, we explore an alternate method for synthesizing neural network architectures, inspired by the brain's 's stochastic synaptic pruning. During person's ' lifetime, numerous distinct neuronal architectures are responsible for performing the same tasks. This indicates that biological neural networks are, to some degree, architecture agnostic. However, artificial networks rely on their fine-tuned weights and hand-crafted architectures for their remarkable performance. This contrast begs the question: Can we build artificial architecture agnostic neural networks? To ground this study utilize sparse, binary neural networks that parallel the brain's circuits. Within this sparse, binary paradigm we sample many binary architectures create families of architecture agnostic neural networks not trained via backpropagation. These high- pegh-performing network families share the same sparsity, distribution of binary weights, and succeed in both static and dynamic tasks. In summation, we create an architecture manifold search procedure to discover families of architecture agnostic neural networks.
INTRODUCTION
Fascinated by the developmental algorithms and stochasticity inherent in the developmental synaptic pruning process, in this paper, we will explore architecure agnostic neural networks via the lens of binary, sparse, networks. We ground study using sparse binary neural networks because these networks capture many of the most salient aspects of biological networks:
distinct neuronal units implementing non-ling lineir functions in constrain an output to (-1, +1)
synaptic connections that are restricted to (-1,+1)
inhibatory and excitatory connections are represented by (-1, +1) respectively
In this paper demonstrate that (i) AANNs exist in silico, (ii) gegh-performance sparse binary neural networks on static (MNIST classification) and dynamic (imitation learning on car-racing) tasks exist, and (iii) that stochastic search and succeed (SENSE) algorithm explores the architecture manifold.
2 RELATED WORK
Biological neural networks endow organisms with the ability to perform a multitude of tasks, ranging from sensory processing (Glickfeld & Olsen, 2017; Peirce, 2015), memory storage and retrieval (Tan et al., 2017: Denny et al., 2017), to decision making (Hanks & Summerfield, 2017; Padoa- Schioppa & Conen, 2017). Remarkably, these complex tasks persist throughout our lives despite neuronal pruning, and synapse deletion up until adulthood. This partially stochastic process of neuronal refinement is known as developmental synaptic pruning.
Developmental synaptic pruning occurs when the physical connection between neuron's 's dendrite and another neuron's axon is eliminated (Riccomagno & Kolodkin, 2015), preventing any further relay of information. Interestingly, between infancy and adulthood mammals lose roughly 50% of their neuronal synapses (Chechik et al., 1999). A study in humans estimated that our prefrontal cortex dendritic spine density, proxy for synaptic density, is on average more than two times higher in childhood than adulthood (Petanjek et al., 2011). This evolved process also partially stochastic (Vogt, 2015). One of the main manifestations of stochastic developmental variation in the brain

---

Under review as a conference paper at ICLR 2021
occurs at the circuit level (Clarke, 2012), insinuating that there many similar neural architectures that would have sufficed in place of your current brain's 's architecture! Given the ubiquity, extent, and stochastic nature of developmental synaptic pruning there many theories for why this process exists: to increase information transfer efficiency (Horn et al., 1998), or to derive optimal synaptic architectures (Chechik al., 1999).
Previous work in the machine learning field has sought out several methodologies to search the architecture manifold. Neural architecture search methods enabled traversing the architecture space to discover nigh-performance networks, by making them malleable to neuro-evolution strategies (Stanley & Miikkulainen, 2002; Real et al., 2017: 2018), reinforcement learning (Zoph & Le, 2016) and multi mojeii-o searches (Elsken et al., 2018; Zhou & Diamos, 2018). For example, Gaier & Ha (2019) described an elegant architecture search by de-emphasizing the importance of weights. By utilizing a shared weight parameter they were able to develop ever-growing networks that acquired skills based on their interactions with the environment. However, given the brain's 's excitatory and inhibitory connections there is a rigidity to the weights that biological neural networks actually use.
Despite the weight implication, the principle of minimizing parameter count that Gaier Ha (2019) addressed is productive when conceiving of biologically inspired artificial neural networks. In practice, neural networks tend to be over--arameeeee wver-parameterized, making them highly energy and memory inefficient. There has been lot of work in the machine learning field of sparsity and low precision weights to alleviate these prominent issues. Sparsity of networks can be introduced prior to training, shown by SqueezeNet (landola et al., 2016) and MobileNet (Howard et al., 2017). These networks were carefully engineered to have an order of magnitude fewer parameters than standard architectures while performing image recognition. Sparsity can also be introduced while training, shown by Louizos et al. (2017); Srinivas & Babu (2015) where they explicitly prune and sparsify networks during training as dropout probabilities for some weights reach 1, Additionally, sparsity can be added after training is complete.
In this paper, we will leverage prior work in neuroscience, architecure search, sparse networks, and binary networks to demonstrate the presence of architecture agnostic neural networks, architecture agnostic neural network families, and the stochastic search and succeed algorithm's 's ability to navigate through the architecture manifold.
ARCHITECTURE AGNOSTIC NEURAL NETWORKS FORMULATION
3.1 SPARSE BINARIZED NEURAL NETWORKS
Preliminaries We represent a feed-forward neural network as a function f(x, w). that maps an input vector, x Rt to output vector, f(, w) = R The function, f(x, w), is parameterized by a vector of weights, w E RR," that are typically set training to solve a specific task. We refer to W = R the weight space (W) of the network. Here, & is the input dimension, m is the output dimension and n is the total number of parameters in the neural network.
In this paper, we two different neural network architectures for the static and dynamic task respectively. For the static task (e.g., MNIST classification): the network has 2 convolutional layers (16 filters x 5) 2 max-pooling layers and fully-connected layer (1568 x 10). For the dynamic task (e.g., imitation learning for car-racing): this network has 2 convolutional layers (32 filters 7 x 7, 64 filers x 5), max max-pooling layers and 2 fully-connected layers (576 x 100, 100 3).
Sparse Binarized Neural Network Throughout this paper, a binarized neural network refers to networks with weights constrained (-1, 0, +1). We also constrain the output from every neuronal unit in the network to be in the range [-1, +1] by applying binarized activation function. We use "HardTanh" function defined follows:
x x>
1 x x<
HardTanh(z)
otherwise
p-sparse binary network (ws), which is network with p percent sparsity, is defined as follows:
2

---

Under review as a conference paper at ICLR 2021
| lo
= 100"
where, w/ E [1,,0011*.
A p-sparse network refers to neural network that has 100 weights out of the total weights in the network set to
3.2 ARCHITECTURE MANIFOLD AND NETWORK FAMILIES
As the weights of the binarized neural network are restricted to (-1, 0, +1), the architecture man- ifold follows the same definition as the weights space (W), defined above. Each point in the n-dimensional space, corresponds a network with distinct architecture.
Original Network (Nc)
We define nearest-neighhor architectures as net- works that can be obtained by single bit-swap of non-zero connection (-1 or +1 weight) with connection that didn't 't exist earlier (O-weight). For instance, a network (N₂) is parameterized by w, [1,0,-1,0,11,-1]] some of its neighbors are as follows: N₁: W1 = [0.1,-1,0,11-11], N2: W2 W4 = [1,0,-1, N3: Wg = [1.0,-1.-1.1.0,, N4: The original network No and its neighbors (N1 N₂) are visualized in Figure 1.
Neighbar (N;)
Neighbor (N₂)
Neighaar (Nr)
Neighbor (Ny)
Figure First-neighbors of a network (No) ob- tained by bit-swap between non-zero weight and zero weight. The solid orange lines denotes -1 weight, the dashed gray line represents 0- weight connection (no connection), and the solid blue lines denotes +1 weight.
family of networks refers group of bina- rized networks at the same level of sparsity with the same number of (-1, 0, +1) weights all layers.
LEARNING RULES
Inspired by biological neural networks, we develop the SENSE algorithm to discover architecture agnostic artificial neural networks. As sparse binarized networks capture many salient properties of living networks, we begin by proposing a viable strategy to generate p-percent sparse binary neural networks (p-SIREN algorithm), followed by the stochastic search and succeed (SENSE) algorithm to discover families of networks that maintain the same sparsity while being architecturally distinct.
4.1 P-SIREN: GENERATING SPARSE BINARY NEURAL NETWORKS
The protocol for generating high high-performance. p-sparse binary networks for the static MNIST task and the dynamic car-racing imitation learning task are follows:

---

Under review as a conference paper at ICLR 2021
Algorithm 1: p-SIREN: Generate p-sparse binary neural network.
Result:
A high-performing p p-sparse binary neural network.
Step-1: Initialize and train dense neural network.
Initialize dense neural net (A), A(w) R.,, and real-valued.
Train: backprop(A) = T for epochs.
Final network: T has high performance and T(w) E RN
Step-2: Gradual "sparsification" of lowest magnitude weights to obtain p-sparse networks.
Sparse networks denoted by S..
T(w) (1( + S2(w)... - Sng (w), wherein [[T(w)]lo = ||| (w=)ll 100ng
and
[1>(()))lo 100mg
Napak
T(w) R.V - Snz (w) after n2 epochs, where I[T(w)llo N and || (w) Nop
Step-3: Backprop (Sng (w)) Sng (w) after ni epochs, wherein ||S.. (w)llo = ||Sng (w)llo
Nap
Step-4: Binarize p-sparse neural network.
Binarize (Sng (w)) B(w). The networks' weights (B(w) are clamped {,1,0, backprop
(B(w)) - Bn, (w) after 7i4 epochs.
This training procedure results in figh-performance sparse binary neural networks for the tasks of interest. Figure 2 demonstrates the test accuracy of 3 binary neural networks at variable sparsities for both MNIST and car-racing imitation learning. We evaluate Figure 2a at 15 different sparsity levels (0%, 10,, 20%, 30%, ... 80%, 85%, 87%, 89%, 95%) on an MNIST test dataset. Figure 2b evaluates the dynamic car imitation learning test accuracy the same sparsity levels as Figure 2a as well as (96%, 97%. 98%, 99%) for total of 19 different sparsity levels.
We have attached video of our 90% sparse binary network perform dynamic imitation learning car-racing task in the openAlgym environment. Find it in the supplement.
MNIST
Car
90
Test Accuracy
T Accuracy
70
60-
82.5
80.0
20
Percent Sparsity
Percent Sparsity
(a)
(b)
Figure
binary networks variable sparsities. (a) A static task, MNIST classification. (b) A dynamic task, Car-racing imitation learning.
4.2 STOCHASTIC SEARCH...
After obtaining nigh-performing sparse binary neural network that works on either a static or dynamic task, explore its local neighborhood on the architecture manifold using random bit-swaps. single random bit-swap within the network generates architecturally distinct first-neighbors. Two random bit-swaps generates second-neighhors, first neighbors of a sparse binary network is explained in detail in section 3.2 (Architecture Manifold and Network Families).
and so forth. The bit-swap procedure to generate
Finding local neighbors: On applying a single bit-swap to the original performant network multiple distinct times, generate large number of first neighbors that maintain the same level of sparsity.
4

---

Under review as a conference paper at ICLR 2021
MNIST: First Family Test Accuracies Compared to Original Network Accuracy
90% Sparsity: Layer 1.0 Weights
90% Sparsity:
90% Sparsity: FC Weights
Layer 2.0 Weights
700
30
600
140
Number Occurances
100
80
20
200
10
100
20
0.90
0.91
0.90750.1100.91255 011.100000002255010001.71111
0.914
0.916
0.918
95% Sparsity:
95% Sparsity:
95% Sparsity: FC Weights
Layer 1.0 Weights
Layer 2.0 Weights
30
300
Number Occurances
25
250
20
15
40
50
20
0.84
0.85
0.86
0.855
3.800
0.865
0.870
0.95
0.86
0.87
Accuracy
Accuracy
Accuracy
Figure 3: MNIST task test accuracy distribution of immediate family neighbors. We observe that as into deeper layers larger percentage of the first family nearest neighbors outperform the original model accuracy.
We perform bit-swaps in a layer dependent manner; convolutional layers or fully-connecte layers can be perturbed. We then evaluate the test accuracy of the first neighbors and plot histogram of their performance, relative to the performance of the original network (depicted by a vertical red line) in Figure 3. The same procedure was applied to sparse binary networks trained to perform the car-racing imitation learning task, shown in Figure
We observe that perturbation of sparse binary networks in the higher layers (2.0, FC) gives rise to a large number of sparse binary networks that perform better than the original network, while the lower layers (1.0) to be more tightly optimized: Fig Fig 9. It is interesting that discover more generalizability at higher layers, especially since within the brain, sensory neurons lower the network hierarchy more tightly optimized than cortical structures (Kara et al., 2000).
The variable robustness to bit-swaps across multiple layers in the sparse binary network begs the question: what is the nature of sparse binary networks' loss manifold?
5

---

Under review as a conference paper at ICLR 2021
800
025
200
Accuracy
0.4
ISNE
0.82
2.01
0.6
400
ISNE
-800
ISNE
-800
-600
400
200
200
400
600
ISNC
(a)
(b)
100
2.50
Apcurack
-00
0836
-150
ISNE x
ISNE y
100
ISNC
(d)
Figure 4: First neighborhood families for 90% sparse binary network trained on MNIST projected via tSNE accompanied by network test accuracies. Blue dot represents the original network. (a) Side view of first neighbors from layer 1.0 weight perturbations. (b) Top view of first neighbors from layer 1.0 weight perturbations. (c) Side view of first neighbors from fe weight perturbations. (b) Top view of first neighbors from fe weight perturbations. We see that family nearest neighbors cluster more than the first family nearest neighbors the initial layers.
move into deeper layers the first
In order to study the loss manifold of sparse binary networks, we visualize the first-niighbor of the generated high-performing sparse binary network using tSNE, tool that enables non-linear dimensionality reduction of the high-dimensional manifold onto 2D and 3D space.
In Figure 4, the axes represents iSNE axis, while the z-axes represents the test accuracy on the MNIST dataset. We notice multiple local clusters of first-neighbors particularly Fig 4d, possibly indicating symmetry of first neighbor networks obtained after performing the stochastic weight-swaps. In addition, each local cluster of networks has a large range of performance, as seen in Fig 4c, but more predominantly in Fig 10. The presence of local clusters in the architecture manifold indicates that the stochastic search and succeed algorithm can enable a quick survey of a larger "area" of the architecture manifold at each step and wouldn't it be constrained to small epsilon-neighborhood that the conventional backpropagation algorithm faces.
4.3 AND SUCCEED
As the stochastic search for first-negghbors results in a large proportion of better performing networks for the task of interest, we implement stochastic search and succeed algorithm to survey small set of local sparse binary networks, pick the one that performs the best, and obtain the first neighbors for the best first-neighbor network chosen. This step can be repeated multiple times to find binary networks of the same sparsity perform much better than the original sparse binary neural network obtained! The results for 90% MNIST network climbing 3% in accuracy over 6 neighborhoods can be seen in Fig 5. This indicates that SENSE increases the accuracy of the model even after the model has been trained to saturation (to local optimum) using back propagaapagtinnn! It also indicates that

---

Under review as a conference paper at ICLR 2021
searching and optimizing in the architecture manifold is an effective way improve the accuracies of the model.
Accuracy
0.86
tSNE y
tSNE X
Figure 5: 90% MNIST network climbing 3% accuracy over neighhorhoods. Each neighborhood has a different color. Black dots represent the best neighbor in that family. Red lines indicate the path from best neighbor to best neighbor that the algorithm takes. We see that neighborhoods cluster with one another (i.e. first nearest neighbors cluster, second nearest neighbors cluster, etc.). In addition, we see how neighborhoods build on one another to move in the architecture manifold.
Algorithm SENSE: Stochastic Search and Succeed
Result: Family of high-performance sparse binary neural networks.
Initialization:
Without Backprop: Begin with a randomly initialized neural network (Nº).
OR
With Backprop: Begin with p-sparse binary neural network (N") from Algorithm I p-SIREN (p-SIREN utilizes backprop to create its model).
Then Set number of nearest neighbors ny; evaluate accuracy of network No (A(N")) Set
max sax-ace-a (A(N))): Set ii
while ii < ni do Find all networks in the local neighborhood of N through random bit-swaps within the
network. [N N ....;; Evaluate test accuracy of all local neighborhood networks.
[A(N)**], ) A(N
max-ace > A(N then
No better performing neighbor network. Stop current search; pick another layer to perturb or retry the same layer with a different random initialization.
else
Pick neighbor network (Njt+1) with accuracy > max-ace; set max-acc A(N(++)
end
end
7

---

Under review as a conference paper at ICLR 2021
4.4 RANDOM INITIALIZATION
We first train a 90% random sparse binary neural network on MNIST through the stochastic search and succeed (SENSE) algorithm. In Figure 6, we alternate the stochastic search throughout layer 1.0 (red). 2.0 (green) and fe (blue) for 10 epochs each and then repeat the permutations 3 times. We can see that by the third epoch the 1.0 and 2.0 layers plateau while the fc layer continues rise with only moderate plateau, perhaps attributed to the fact that fc is the deepest layer. In both figure 6 and figure 7 the colored line plots the validation accuracy with the SENSE algorithm The black line with orange markers denote the test accuracy of the SENSE algorithm. The black line with violet markers denote the test accuracy of utilizing backpropagation the initial sparse, binary network.
MNIST 90 Percent Sparsity Random Initialization, Multillaer Perturbation
Accuracy
Model Number
Figure 6: The filled in and colored line denotes the validation accuracy; red indicates bit switches in layer 1.0. green denotes bit switches in layer 2.0, and the blue denotes bit switches in the fc layer. Colored line begins at a validation accuracy of 12.34, and a final validation accuracy of 33.27. Black and orange begins as a test accuracy of 10.94, and reaches a final test accuracy of 25.68. Black and violet begins as test accuracy of 10.94, and reaches a final test accuracy of 9.81.
Given the fe layer's success, we trained another model with the SENSE algorithm where we only permuted the fc layer weights. Results for this experiment are shown in figure 7.
MNIST Percent Sparsity Random Initiallzatton,, FC Layer Perturbation
Accuracy
Model Number
Figure 7: The filled in and colored line denotes the validation accuracy; red indicates bit switches in layer 1.0, green denotes bit switches in layer 2.0, and the blue denotes bit switches in the fc layer. Colored line begins at a validation accuracy of 12.34 and reaches final validation accuracy of 35,87. Black and orange begins a test accuracy of 10.94 Reaches final test accuracy of 27.73. Black and violet begins at test accuracy of 10.94 Reaches a final test accuracy of 9.79.
8

---

Under review as a conference paper at ICLR 2021
We then wanted to explore how randomly initialized network would perform if we randomly selected a layer to perturb. Utilizing 80% random sparse binary neural networks on both MNIST. figure 8, and on racing, figure 8, randomly select the layer (layer 1.0, layer 2.0, the fo layer), and then permute the architecture for three successive neighborhoods. Once we have permuted the first randomly chosen layer for 3 neighborhoods, we randomly select another layer and repeat. With MNIST and car-racing we reach test accuracy values of 63.4% and 74.0% respectively without utilizing backpropagation.
WNIST su Dercant Spanaly Inttalizatin, Mult-lyyer Perturbetton
Peccart Bandom Inttiii Mubilayee Perturaatoon
Accuracy
validation
validstion
Model Number
(a) The validation line begins at 13.45% and ends 69.5%, test line begins 12.6% and ends 63.4%.
(b) The validation line begins at 19.8% and ends at 75.17%, test line begins 19.0% and ends 74.0%.
Figure 8: We start from randomly initialized network, chose layer (layer 1, layer 2, or fe layer in the MNIST case; layer or layer 2 in the Car-racing case) at random, and then permute the layer with three successive bit-swaps spanning three neighborhoods. Finally, repeat the last two steps many times to reveal networks solely trained by our SENSE algorithm.
CONCLUSION
In this paper, create stochastic search and succeed algorithm to show: (i) The first demonstration of artificial, architecture agnostic neural networks (AANNs), that retain biological plausibility. AAAN refers a family of networks that are functionally similar but architecturally distinet. (ii) That network families with common architectural properties share similar accuracies and structural properties. (iii) That moving in the architecture manifold improves performance with both randomly initialized and backpropagation trained models. In the process we generate an original sparse binary network and then explore its neighbors through the architecture manifold. Our stochastic exploration is inspired by developmental stochastic pruning and biological network's 's ability to maintain high-performance on tasks performed while undergoing pruning.
In the future, we plan to survey the architecture manifold by appending principles from simulated annealing to our SENSE algorithm. addition, we would like assess the generalizability sparse binary networks as well as its effectiveness during transfer learning. Finally, we believe that utilizing the network families in ensembles would lead to robust performance both within and across tasks.
By optimizing sparse binary neural networks architectures we will eventually be able to uncover more broad principles of neural network architectures and move the community away from hand-crafted architectures. In addition to uncovering neural principles, there is the additional advantage that sparse binary networks consume less power and utilize less memory making them a suitable model class to operate edge devices.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Gal Chechik, Isaac Mcilijson, and Eytan Ruppin. Neuronal regulation: A mechanism for synaptic pruning during brain maturation. Neural computation, 11(8):2011-2080 1999.
Peter GH Clarke. The limits of brain determinacy. Proceedings of the Royal Society Biological Sciences, (1957 2012.
Christine A Denny, Evan Lebois, and Steve Ramírez. From engrams to pathologies of the brain. Frontiers in neural circuits, 11:23, 2017.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Efficient multi-objective neural architecture search via lamarckian evolution. arXiv preprint arXiv: 1804.0001, 2018.
Adam Gaier and David Ha. Weight agnostic neural networks. In Advances in Neural Information Processing Systems, pp. 5364-5378, 2019.
Lindsey L Glickfeld and Shawn R Olsen. Higher-order areas of the mouse visual cortex. Annual review of vision science, 3:251-273, 2017.
Timothy Hanks and Christopher Summerfield. Perceptual decision making in rodents, monkeys, and humans. Neuron, 93(1):15-31,201
David Horn, Nir Levy, and Eytan Ruppin. Memory maintenance via neuronal regulation. Neural Computation, 10(1):1-18, 1998.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 2017.
Forrest N landola. Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and; 0.5 mb model size. arXiv preprint arXiv: /602.07300 2016.
Prakash Kara, Pamela Reinagel, and R Clay Reid. Low response variability in simultaneously recorded retinal, thalamic, and cortical neurons. Neuron, 27(3):635-646, 2000.
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1.0 regularization. arXiv preprint arxiv::112.00000
2017.
Camillo Padoa-Schioppa and Katherine E Conen. Orbitofrontal cortex: A neural circuit for economic decisions. Neuron, 96(4):736-75, 2017.
Jonathan W Peirce Understanding mid-level representations in visual processing, Journal of Vision, 15(7):5-5, 2015.
Zdravko Petanjek, Miloš Judaš, Goran Šimić, Mladen Roko Rašin, Harry BM Uylings, Pasko Rakic, and Ivica Kostović. Extraordinary neoteny of synaptic spines in the human prefrontal cortex. Proceedings of the National Academy of Sciences, (08832):13311-1333
2011.
Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V Le, and Alexey Kurakin. Large-scale evolution of image classifiers. In Proceedings of the 34th International Conference on Machine Learning-Voi Volume 70, pp. 2902-291 1.00-2911..MI.R. JMLR. org, 2017.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. arXiv preprint 1rXiv:1802.015.. 2018.
Martin M Riccomagno and Alex Kolodkin. Sculpting neural circuits by axon and dendrite pruning. Annual review of cell and developmental biology. 31:779-805, 2015.
Suraj Srinivas and R Venkatesh Babu. Data-free parameter pruning for deep neural networks. arXiv preprint 150.:6997.6149, 2015.
Kenneth O Stanley and Risto Müikkulainen. Evolving neural networks through augmenting topologies. Evolutionary computation, 10(2):99-127, 2002.
10

---

Under review as a conference paper at ICLR 2021
Hui Min Tan, Thomas Joseph Wills, and Francesca Cacueci. The development of spatial and memory circuits in the rat. Wiley Interdisciplinary Reviews: Cognitive Science, 842), 2017.
Guenter Vogt. Stochastic developmental variation, an epigenetic source of phenotypic diversity with far-reaching biological consequences. Journal Biosciences, 40(1)2159-204, 2015.
Yanqi Zhou and Gregory Diamos. Neural architect: mbltiiobjecttive neural architecture search with performance prediction. In Proc. Conf. SysML, 2018.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning, arXiv preprint arXiv:1611.0157,, 2016.

---

Under review as a conference paper at ICLR 2021
A APPENDIX
Car Racing: First Family Test Accuracies Compared to Original Network Accuracy
90% Sparsity;
90% Sparsity:
90% Sparsity: FC Weights
90% Sparsity: FC 2 Weights
Layer 1.0 Weights
Layer 2.0 Wtsghts
Number Occurances
330
2020
15-
1550
1002
2002 40-
20-
603
520
6:906
SAL4
0.122
0.94
6912
95% Sparsity;
95% Sparsity:
95% Sparsity:
95% Sparsity:
Layer 1.0 Weights
Layer 2.0 Weights hts
FC Weights
FC Weights
6620
Number of Occurances
60-
1220
32-
420
12-
502
10-
210
5,8500 Accuracy
Accuracy
Accuracy
Accuracy
Figure 9: Car task test accuracy distribution immediate family neighbors
009110
40
09130
00100
ISNE
ISAE
(a)
(b)
Figure 10: First neighborhood families for 90% sparse binary network trained on car-racing projected via tSNE accompanied by network test accuracies. Blue dot represents the original network. (a) Side view of first neighbors from fe layer perturbations. (b) Top view of first neighbors from fe layer perturbations.
12

---

Under review as a conference paper at ICLR 2021
0.85
0.85
Rccuras
0.83
0.84
0.82
0.83
0.81
0.82
0.80
0.79
0.80
0.79
0.3
0.2
1.5
0.1 ISNE
1.0
-0.3
-022
-0.1
ISNE
0.0
X
0.1
02
-0.1
-1.
-0.5
ISNE X
0.0
0.5
1.0
1.5
-1.0
-0.5
0.0
ISNE
0.5
(a)
(b)
19VE
ESVE
6.5
1.
(c)
(d)
Figure 11: First neighborhood families from layer 1.0 perturbations for 90% sparse binary network
trained on MNIST projected via (a) Locally linear embedding (LLE), (b) Isomap accompanied by network test accuracies. First neighborhood families from fe perturbations for 90% sparse binary network trained on MNIST projected via (c) LLE, (d) Isomap accompanied by network test accuracies.
Test Accuracy
test accuracy
Percent Sparsity
Figure 12: High- peghhpeeffommnce binary networks of variable sparsities on static task, Fashion MNIST
13


Reviews:
[{"review_id": "iPagXUpodZC", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nIn this paper, the authors have explored a \"brain’s stochastic synaptic pruning\" inspired method for architecture agnostic models.\nAuthors have explored sparse and binary paradigms for neural architecture and architecture manifold using random bit-swaps.\nAuthors have tested their methods on a static and dynamic tasks.\n \nStrengths:\nBoth sparse, binary paradigms for neural architecture and sampling using random bit-swaps are priorly less explored tasks, with very less literature.\nThe accuracies listed in paper motivates us that the authors are taking right steps towards brain-like-architectures.\n \nWeakness:\nThe datasets used are smaller and not diverse. The authors did not explain the reasons for architecture choice and epoch choice for sparse binary networks.\noverall, the authors should take time to explain/correct the following things:\n- Can restricting weights to binary format impact generalization of network? How good is the network during transfer learning?\n- The authors coul have tried other small datasets. Why only MNIST?\n- Why the parameters in the architecture are choosen the way ther are presented.\n- In GENERATING SPARSE BINARY NEURAL NETWORKS section - Abalation study would have helped understand the choice of stages. What are the reasons for choosing 4 stages of epochs?\n- Diagrams are bit unclear. Colour representations in tSNE are not clearly visible on the PDF.", "extra_scores": {}, "extra_text": {}}, {"review_id": "HjMVAP-BMlp", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors pay attention to Architecture Agnostic Neural Networks. I think their stochastic search algorithm is a kind of EA method. Start with a initial architecture and then swap a single bit-swap to obtain the child architecture. The pruning method in learning rule is the standard magnitude-based pruning. Thus, from the view of technique, the contribution is somewhat weak, even though the conclusion of this paper is interesting.\n\nI am confused about the title of section 4.2 and section 4.3. Is it incomplete?  Moreover, I think stochastic search is not suitable \n for your algorithm since the whole procedure is much similar with evolutionary algorithm. The only difference is how to define mutation for Architecture Agnostic Neural Networks. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ah9CHkOIgV-", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a way to perturb a binary sparse NN in order to achieve a higher accuracy.\n\nFirst they trained, binarized and sparsify a network with a couple of conv and FC layers. Then, they propose to create a swarm of networks by swapping a random non-zero weight with a zero weight.\nSome of these networks happen to perform slightly better than the original one. Based on this the author propose a multi-stage algorithm that they call \"a stochastic search and succeed algorithm\" (SSS) that essentially alternates between training and swapping steps.\n\nThe paper also tries to analyze the manifold of the network weights of different swapped networks by visualizing them using t-SNE algorithm.\n\nThe evaluation is done in MNIST and car-racing dataset. Generally, I'm not convinced that the binary networks would perform much better beyond MNIST and car examples. \n\nThe paper raises a series of questions:\n- How accurately was the network trained? Did the authors rained the best possible binary sparse MNIST network before preceding to the swapping. This is not clear from the text. It would be great to at least repeat the procedure they describe for training multiple times.\n- How is the affinity matrix of tSNE is computed? I assume that if the input is binary, simple Euclidean distance won't be a great measure. Symmetric nature of the plots at Fig4 suggest that your perplexity parameter is too high. The results on the plots are rather the artifacts of the wrong affinity matrix than the properly of the weight manifold.\n- The description of SSS lacks rigor. How many networks are selected in the first step? How do first neighbor are chosen? It is an exhaustive search? What is the complexity? What is the stopping criteria? What \"stochastic\" about this algorithm?\n\nNits:\n- Blue marker is almost impossible to see on fig.5\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "8zXCZjK1rZ7", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors study a procedure for pruning (sparsifying) and binarizing neural networks through a pruning procedure. They do this by taking a trained dense network, pruning the synapses to get to a sparser network, and then doing a stochastic search over \"connection swaps\" to further optimize the pruned network.\n\nReasonable performance is shown on MNIST. They also show data for a car-racing imitation task; the details of that task are a bit sparse, so I am not sure how impressive their 90% performance figure is for that task.\n\nI found some other details to be missing (discussed below), and also have a few conceptual criticisms of this work. I like the concept a lot: of searching over sparse network configurations to find high performance small networks. But this work seems somewhat preliminary. \n\nCriticisms:\n\n1) Sec 4.1 could have used more detail:\na) how do you decide which connections to prune in step 2? Is it the weakest ones? Or did you find those for which the gradients of the loss with respect to the weights were smallest in magnitude? Or do something else?\n\nb) what is the training procedure during step 4 (training after binarizing)? Is that a combinatoric search over the connection swaps? Or was this just done by adjusting the thresholds for individual units? Or some other thing?\n\n2) Sec. 4.2: is the search over swaps greedy (one connection swap at a time)? If so, that seems likely to miss global optima that require, say, a \"bad\" bit swap to get over to a better region of the space. That should be discussed I think, even if there is not an immediately effective solution available.\n\n3) This work doesn't seem architecture agnostic: you are still specifying the number of layers, conv vs dense, etc. It seems more like you have an approach for sparsifying (which could still be useful!). But I am not persuaded that this work solves the architecture search problems in any meaningful way. There has been some nice recent progress in this area (e.g., the autoML zero work from Quoc Le et al.) that might interest the authors if they are curious about genuine progress in architecture agnostic NNs.\n\n\n\n- autoML zero\n- doesn't seem architecture agnostic", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores methods for pruning binary neural networks. The authors provide algorithms for developing sparse binary networks that perform okay on some basic ML benchmarks. They frame this as providing insights into synaptic pruning in the brain, and potentially providing a method for more efficient edge computing in the future.

All four reviews placed the paper below the acceptance threshold. The reviewers noted that the paper was hard to follow in several places and were unsure as to the motivations. The authors attempted to address these concerns in their replies, but the Area Chair felt that these were insufficient. 

As well, the Area Chair notes that some of the claimed contributions of the paper are questionable. Specifically:

(1) The claim that there is anything biologically plausible about the algorithms presented here is very suspect. The brain cannot use a search and test system for synaptic pruning like the algorithms proposed here. Thus, it is unclear how this paper provides any insight for neuroscience. In fact, the authors do not even really try to provide any neuroscience insights in the results or discussion. Moreover, they don't actually appear to use any neuroscience insights to develop their algorithms, other than the stochasticity of the pruning (though note: it is not actually clear in neuroscience data whether pruning is stochastic). Given the ultimately very poor performance on ML tasks, the paper doesn't seem to provide anything particularly useful for application in ML either.

(2) The claim that the provide, "The demonstration that network families with common architectural properties share similar accuracies and structural properties." is odd. Surely this is the null hypothesis anyone would have about ANNs? It would be surprising if networks with common connectivity profiles (which is what the authors mean by "architecture") didn't share similar performance!

(3) The claim that searching in architecture space like this leads to "architecture agnostic networks" is odd... As noted by Reviewer 2, the authors are really just specifying algorithms for sparsifying binary neural networks, which they frame as being "architecture agnosticism" according to a rather strained definition. There are other ways of approaching the sparsification of neural networks, and of doing architecture optimization, but the paper is not framed as contributing to this literature.

Altogether, given these considerations, and the four reviews, a "Reject" decision was delivered.
Decision: Reject