Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: In this paper, the authors study a procedure for pruning (sparsifying) and binarizing neural networks through a pruning procedure. They do this by taking a trained dense network, pruning the synapses to get to a sparser network, and then doing a stochastic search over "connection swaps" to further optimize the pruned network.

Reasonable performance is shown on MNIST. They also show data for a car-racing imitation task; the details of that task are a bit sparse, so I am not sure how impressive their 90% performance figure is for that task.

I found some other details to be missing (discussed below), and also have a few conceptual criticisms of this work. I like the concept a lot: of searching over sparse network configurations to find high performance small networks. But this work seems somewhat preliminary. 

Criticisms:

1) Sec 4.1 could have used more detail:
a) how do you decide which connections to prune in step 2? Is it the weakest ones? Or did you find those for which the gradients of the loss with respect to the weights were smallest in magnitude? Or do something else?

b) what is the training procedure during step 4 (training after binarizing)? Is that a combinatoric search over the connection swaps? Or was this just done by adjusting the thresholds for individual units? Or some other thing?

2) Sec. 4.2: is the search over swaps greedy (one connection swap at a time)? If so, that seems likely to miss global optima that require, say, a "bad" bit swap to get over to a better region of the space. That should be discussed I think, even if there is not an immediately effective solution available.

3) This work doesn't seem architecture agnostic: you are still specifying the number of layers, conv vs dense, etc. It seems more like you have an approach for sparsifying (which could still be useful!). But I am not persuaded that this work solves the architecture search problems in any meaningful way. There has been some nice recent progress in this area (e.g., the autoML zero work from Quoc Le et al.) that might interest the authors if they are curious about genuine progress in architecture agnostic NNs.



- autoML zero
- doesn't seem architecture agnostic
review_title: Pruning and binarizing neural networks

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors pay attention to Architecture Agnostic Neural Networks. I think their stochastic search algorithm is a kind of EA method. Start with a initial architecture and then swap a single bit-swap to obtain the child architecture. The pruning method in learning rule is the standard magnitude-based pruning. Thus, from the view of technique, the contribution is somewhat weak, even though the conclusion of this paper is interesting.

I am confused about the title of section 4.2 and section 4.3. Is it incomplete?  Moreover, I think stochastic search is not suitable 
 for your algorithm since the whole procedure is much similar with evolutionary algorithm. The only difference is how to define mutation for Architecture Agnostic Neural Networks. 
review_title: Interesting work

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes a way to perturb a binary sparse NN in order to achieve a higher accuracy.

First they trained, binarized and sparsify a network with a couple of conv and FC layers. Then, they propose to create a swarm of networks by swapping a random non-zero weight with a zero weight.
Some of these networks happen to perform slightly better than the original one. Based on this the author propose a multi-stage algorithm that they call "a stochastic search and succeed algorithm" (SSS) that essentially alternates between training and swapping steps.

The paper also tries to analyze the manifold of the network weights of different swapped networks by visualizing them using t-SNE algorithm.

The evaluation is done in MNIST and car-racing dataset. Generally, I'm not convinced that the binary networks would perform much better beyond MNIST and car examples. 

The paper raises a series of questions:
- How accurately was the network trained? Did the authors rained the best possible binary sparse MNIST network before preceding to the swapping. This is not clear from the text. It would be great to at least repeat the procedure they describe for training multiple times.
- How is the affinity matrix of tSNE is computed? I assume that if the input is binary, simple Euclidean distance won't be a great measure. Symmetric nature of the plots at Fig4 suggest that your perplexity parameter is too high. The results on the plots are rather the artifacts of the wrong affinity matrix than the properly of the weight manifold.
- The description of SSS lacks rigor. How many networks are selected in the first step? How do first neighbor are chosen? It is an exhaustive search? What is the complexity? What is the stopping criteria? What "stochastic" about this algorithm?

Nits:
- Blue marker is almost impossible to see on fig.5



review_title: An interesting idea for random architecture search for binary sparse networks

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:
In this paper, the authors have explored a "brain’s stochastic synaptic pruning" inspired method for architecture agnostic models.
Authors have explored sparse and binary paradigms for neural architecture and architecture manifold using random bit-swaps.
Authors have tested their methods on a static and dynamic tasks.
 
Strengths:
Both sparse, binary paradigms for neural architecture and sampling using random bit-swaps are priorly less explored tasks, with very less literature.
The accuracies listed in paper motivates us that the authors are taking right steps towards brain-like-architectures.
 
Weakness:
The datasets used are smaller and not diverse. The authors did not explain the reasons for architecture choice and epoch choice for sparse binary networks.
overall, the authors should take time to explain/correct the following things:
- Can restricting weights to binary format impact generalization of network? How good is the network during transfer learning?
- The authors coul have tried other small datasets. Why only MNIST?
- Why the parameters in the architecture are choosen the way ther are presented.
- In GENERATING SPARSE BINARY NEURAL NETWORKS section - Abalation study would have helped understand the choice of stages. What are the reasons for choosing 4 stages of epochs?
- Diagrams are bit unclear. Colour representations in tSNE are not clearly visible on the PDF.
review_title: Official blind review 3

Meta-review: This paper explores methods for pruning binary neural networks. The authors provide algorithms for developing sparse binary networks that perform okay on some basic ML benchmarks. They frame this as providing insights into synaptic pruning in the brain, and potentially providing a method for more efficient edge computing in the future.

All four reviews placed the paper below the acceptance threshold. The reviewers noted that the paper was hard to follow in several places and were unsure as to the motivations. The authors attempted to address these concerns in their replies, but the Area Chair felt that these were insufficient. 

As well, the Area Chair notes that some of the claimed contributions of the paper are questionable. Specifically:

(1) The claim that there is anything biologically plausible about the algorithms presented here is very suspect. The brain cannot use a search and test system for synaptic pruning like the algorithms proposed here. Thus, it is unclear how this paper provides any insight for neuroscience. In fact, the authors do not even really try to provide any neuroscience insights in the results or discussion. Moreover, they don't actually appear to use any neuroscience insights to develop their algorithms, other than the stochasticity of the pruning (though note: it is not actually clear in neuroscience data whether pruning is stochastic). Given the ultimately very poor performance on ML tasks, the paper doesn't seem to provide anything particularly useful for application in ML either.

(2) The claim that the provide, "The demonstration that network families with common architectural properties share similar accuracies and structural properties." is odd. Surely this is the null hypothesis anyone would have about ANNs? It would be surprising if networks with common connectivity profiles (which is what the authors mean by "architecture") didn't share similar performance!

(3) The claim that searching in architecture space like this leads to "architecture agnostic networks" is odd... As noted by Reviewer 2, the authors are really just specifying algorithms for sparsifying binary neural networks, which they frame as being "architecture agnosticism" according to a rather strained definition. There are other ways of approaching the sparsification of neural networks, and of doing architecture optimization, but the paper is not framed as contributing to this literature.

Altogether, given these considerations, and the four reviews, a "Reject" decision was delivered.
Decision: Reject