Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: The work extends an existing algorithm to train a convolutional neural network by selecting a subset of fixed random weights by (1) using multiple random values per weight, (2) using a sampling procedure to select the particular random value for each weight. If these networks are finetuned after random-value selection is performed, they perform close to networks purely trained with SGD on CIFAR-10 and MNIST.

Strong points:
- Interesting results that extend results on the lottery ticket hypothesis and random weight learning.

Weak points:
- The paper builds heavily on an existing algorithm, but it does not cite it in the method section.
- The paper makes many claims and fails to provide evidence for these claims.
- The work is not clearly motivated. It is unclear why this problem is interesting or the results insightful.

Recommendation (short):
While the results are interesting, the paper is poorly motivated and does not conform to standards of scientific work. I recommend rejecting this work.

Recommendation (long):
The method section appears to be independent work, but it is the same as Ramanujan et al., 2019 extended with multinomial sampling. While Ramanujan et al., 2019 are cited, it is not cited in this section. Furthermore, the paper claims that random initialized networks and trained networks perform the same, but it does not lay out evidence or an argument for this. Otherwise, it is unclear why this method is interesting. I recommend rejecting this work.

Comments for authors:

I think these are some good initial results that you have, and you can work with that, but right now, the paper has many flaws that need to be ironed out before you make another attempt. I do not think you can get this work accepted in this round and instead should try to learn as much as possible from the discussion for a resubmission.

I think fixing the claim, references, and so forth will be easy. The hard question is, why is your work interesting. Using multiple random values per weight can be seen as some form of quantization
of weights. Why is your way of doing something similar to quantization more interesting than other forms of quantization? 

Please also consider if it is true that selecting fixed random weights is training or not. Clearly, you are optimizing weights. It does not matter if you do it with SGD, an evolutionary strategy, or your algorithm; in the end, you optimize weights to have particular values. I would say that it should be considered training. But if your method is considered just a different optimization process compared to SGD, why is it interesting? Finding subnetworks, like done in Ramanujan et al., 2019, is interesting because you have smaller trained networks, but you do not have subnetworks.

It could be interesting if you do (1) a thorough analysis that yields some insights and make this an analysis paper, or (2) try to get better performance by doing both optimizations of weights and selection of weights (but this is very similar to Wortsman et al. 2019).

Some minor things: 
- Equation 3 has an additional W (h(*) already contains the W)
- Figure 6 has an annotation error; I believe the upper line is supposed to be the PS method


Ramanujan et al., 2019: What’s Hidden in a Randomly Weighted Neural Network?, https://arxiv.org/abs/1911.13299
Wortsman et al., 2019: Discovering Neural Wirings, https://arxiv.org/abs/1906.00586

review_title: Results interesting but not insightful. Motivation unclear. Some issue with scientific method.

final_score: rating: 5: Marginally below acceptance threshold
review_content: ### Summary

The paper investigates a type of neural network in which one of K possible fixed weights is chosen in each neuronal connection. The weights themselves are fixed and random, but the scores that determine which of the weights is chosen are updated through back-propagation using a straight-through estimator. The accuracy and behavior of such networks is studied for small networks on small datasets (MNIST, CIFAR).

### Quality, clarity, originality and significance

The paper is well-written and easy to follow.

The main idea seems interesting at first sight and it is well-motivated, but after some consideration of related effects in neural networks, the results do not seem very surprising (see below). I think a deeper exploration of the connection to other phenomena would be necessary to make this paper relevant to the conference, e.g. to weight quantization (to few bits per weight) or to (variational) dropout. The paper seems to not go beyond ad-hoc conclusions of the form that these (peculiar) networks "perform competitively on challenging datasets" (which seems to be a bit of an overstatement to me). The authors also claim that the trained networks might be useful for initialization, but to really make this point strongly, a comparison to other practical methods of data-driven initialization on larger datasets with larger architectures might be needed to convince the reader.

Why do the results not seem surprising to me: It is possible that I misunderstood the algorithm (which we could clarify in the rebuttal period, of course), but in my understanding of the described approach, the straight-through estimation of the scores will lead to preferring the selection of larger (or smaller) weights where standard gradient descent training would lead to larger (or smaller) weights. This is consistent with the distribution of the selected weights as shown in several figures, with the observation that uniform initialization works better than Normal initialization, and with the observation that "both GS and PS tend to prefer weights having large magnitudes as learning progresses". It would also account for the observation of the similarity in error rates when the network has a sufficient number of weights to choose from.

* Pros: interesting idea, the paper is well-written
* Cons: of limited interest to the conference audience I believe; not clear if there is practical relevance or potential to improve scientific understanding

### Detailed/minor comments
* The main bullet points at the end of the introduction were not fully substantiated in the paper in my opinion: (1) I was not fully convinced of "a performance equivalence between random initialization and training", because the slot machines are effectively *trained*. (2)"demonstrates that current networks can model challenging non-linear mappings extremely well even with random weights" is not 100% clear because the weights are a *choice* among a set of *initially* random weights, but that choice is a result of training. (3)"connects to recent observations". This seems to happen mostly in the two sentences before 4.3 and there connection *and* statement are not entirely convincing to me "We find a random 6 layer network that performs as well as a 6 layer trained network without any form of pruning." Here it seems to me that the slot machine after training is in fact not random, because it was trained, and that training exploited weight-correlations that potentially span multiple layers. The argument could be extended to regular training in the sense that regular training just picks out the random weights among all the random floating point numbers. (This is a bit exaggerated, but I think it shows why the term "random 6 layer network" may be an overstatement after training.) 
* Why is K chosen from the set {2, 8, 64, 128} - it seems that some natural values in this sequence are missing or that a more log-uniform spread would be more "natural". Maybe there is a specific reason that is not obvious, then it could be mentioned.

### Update after author replies and discussion
I have updated the review score after reading the authors' reply and revision of the paper.
review_title: Instead of updating weights, we can train a network by picking from a set of (random) weights.

final_score: rating: 4: Ok but not good enough - rejection
review_content: ##########################################################################

Summary:

This paper proposes a method to train a neural network by selecting a weight from a set of $K$ randomly generated weights for each edge in the network. Each edge has a different set of random weights. Quality score is assinged to each of $K$ random weights, which determines the weight used in the forward calculation. Instead of optimizing weights directly, the proposed method optimizes the quality scores with the straight-through gradient estimator. Experimental results show that the neural network trained by the proposed method achieves high accuracy compared to random initialization even when $K=2$.

##########################################################################

Reasons for score: 

Overall, I vote for rejecting. The authors say in Sec. 3 that the goal is to construct non-sparse neural networks with completely random weights that achieve high accuracy. However, the model obtained by the proposed method is no longer a network with completely random weights, because the authors optimize quality scores instead of original weights. It is empirically shown that a neural network can achieve a high accuracy by properly selecting weights from a set of random weights prepared in advance. However, such a result is not so surprising from the viewpoint that the quality scores are optimized.
Also, this paper has few practical implications. I would like to see if the network can still achieve a high accuracy when every edge has a common set of $K$ random weights. If this is the case, the proposed method may lead to a network that is efficiently compressed.


##########################################################################

Minor concerns:

(p.1) a fixed a set of random weights -> a fixed set of random weights


--

Regarding the author responses, I have updated my rating.
review_title: Weights obtained by the propoesd method are no longer completely random weights

final_score: rating: 7: Good paper, accept
review_content: **Update after authors' response**
I want to thank the authors for their response, and I am happy to see additional results on shared sets of weight values (which allows to easily relate the work to methods for training low-bit-width networks). To further increase impact and significance of the work it would be necessary to really flesh out the advantage of the proposed method over other, similar methods ("it is not too surprising that the method works, but why would I prefer it over other methods?"). Nonetheless, the paper presents novel empirical analysis that adds to the body of work on non-standard training of neural networks. To make a clear stance for the reviewer discussion I have therefore increased my score to 7, though I would rate the paper at the lower end of score-7 papers. 
---

**Summary**
The paper proposes a novel scheme to obtain well performing neural networks without classical adaptation of weights. Instead, each connection can have one out of K randomly drawn values, and “training” consists of a backpropagation-based procedure to find which value out of the K possible values to select for each weight. The method can be interpreted as finding a high-performing sub-network within a larger network of random weights. However, in contrast to previous methods that literally implement the latter, the proposed method is computationally more efficient. Experiments are performed on a number of neural network architectures on MNIST and CIFAR-10.

---
**Contributions, Novelty, Impact**

1) Proposal of a novel scheme for finding well-performing networks without explicit training of weights. This is interesting and adds to a growing body of recent work on alternatives to classical training of neural nets (which is insightful for both, developing better training algorithms but also understanding the nature of neural network training). My concern is that the proposed method is conceptually very similar to previously known approaches (pruning a larger network which is also discussed in the paper, but also some methods for training low-bit-weight networks such as [1] and [2]). While the proposed method is an interesting alternative implementation, the advantages compared to the other approaches are fairly limited. Accordingly the potential impact of the work might be somewhat limited as well, I’m afraid.

2) A nice and extensive set of ablations and control experiments, as well as repetitions of experiments to establish statistical significance of the results. The paper, in particular the experimental part is well executed, and the ablations and controls allow for being optimistic about the generality of the findings, which has a positive influence on the potential impact of the work.

3) The paper shows that networks obtained with the proposed scheme can also act as a good initialization for further fine-tuning, leading to very well performing classifiers. This process is also analyzed in terms of overall computational budget (FLOPS) and in 2 of 3 cases shown compares favourably against standard neural network training. In terms of impact this is another nice result to add, but probably not strong enough to replace standard initialization anytime soon.


[1] Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1, Courbariaux et al. 2016

[2] XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks, Rastegari et al. 2016

---
**Score and reasons for score**
I’m quite torn with this paper - on one hand the method works well, is thoroughly analysed and the paper is very well written and polished (in many respects I would even say this is an exemplary paper). On the other hand the paper suffers from only adding a quite simple variation on existing work. Particularly the work on training low-bit-width networks ([1] and [2] above, as well as later extensions to more than single-bit weights) is conceptually very similar - the forward pass uses constrained weight values but gradient information is accumulated in a real-valued variable. As the appendix notes, the main idea could also be implemented simply via pruning in a larger network (though less computationally efficient). 

To further strengthen the paper it would be good if the paper could answer one (or both) of the following questions: (i) how do the results contribute to understanding weight initialization and the training process, what can the reader learn that wasn’t known already, (ii) what are the concrete advantages of the proposed method over previously proposed alternatives, what does it do better, what shortcomings does it address (is it faster, is training more stable, …)?

I would love to give this paper a very high score because of the great execution and presentation, but the lack of novel insights, or clear methodological advantages makes this hard. I am currently voting for a weak accept (because the paper is very well written and experiments are thorough, if the scoring was based on novelty alone I’m not sure that the paper would clear the bar for a top-tier conference). I am of course happy to reconsider my final verdict in light of the other reviews and authors’ response.

---
**Strengths**
1) Great presentation of the work, a very well written paper, and good main results.

2) Experiments are well executed - multiple repetitions, many controls and ablations that one wants to see to improve confidence in the generality of the findings.


---
**Weaknesses**
1) Little novelty. The proposed algorithm is a nice idea, but it’s not too surprising that it operates essentially on par with a previously proposed pruning method since the algorithm can even be conceptually recast as pruning in a larger network.

2) The writing puts a lot of emphasis and focus on the distinctive features of the method (understandably so given how close it is compared to other methods). But I think it’s also fine to not start off in an almost defensive fashion and simply state that this is another possibility of implementing the same idea but with the following advantages (and disadvantages).

---
**Correctness**
Reasoning in the paper is sound, experiments are well executed and many control-experiments and ablations are shown.

---
**Clarity**
The paper is very well written, results are nicely presented and related literature is discussed in a useful fashion.

---
**Improvements (that would make me raise my score) / major issues**

1) The main issue is a lack of novelty. Addressing either (i) how the paper adds new knowledge (in light of the current body of literature) or (ii) stating the precise advantages of the proposed method over alternatives would be crucial. I don’t see an obvious way for (i), but for (ii) a starting point could be to do more detailed comparison against other methods, in particular Ramanujan and see whether the proposed method compares favorably e.g. in terms of training stability of robustness w.r.t. hyper-parameters.

2) Another possibility to add novelty to the paper would be to focus on low-bit-width training, where instead of drawing K values for each weight separately, values are re-used per layer or even across the whole network (typically first and last layers need to be treated differently, i.e. they require more bit-width). Reliable and robust methods to obtain e.g. 2-, 4- or 8-bit networks is a timely and important topic, and the proposed method has potential to contribute to that body of work as well (though it is admittedly a bit of a deviation from the current story and main focus of the paper). I want to add this as a suggestion to the authors (it could work, but it could also severely reduce focus and clarity of the paper), not necessarily as an improvement that I’d expect to see.

---
**Minor comments**

A) The ICLR header seems to be missing in the PDF file.

B) In the probabilistic version of the method it might be worth experimenting with some “annealing” schedule where the randomness of the method is gradually reduced over training (e.g. via reducing a Gumbel Softmax temperature). Making the magnitude of the scores very large has essentially the same effect but is less controllable since it has to be indirectly influenced via the learning rate. I would expect convergence/test-time performance of the method to benefit from such a schedule and perhaps even help close the gap to “greedy selection”.

C) Is the supplementary material supposed to go into a separate document (did not double-check the submission instructions)?

D) A bit of a nitpick, the phrase that “weights are never updated” might suggest some miraculous phenomenon - I’d rather say there’s a set of weight values, from which one value is probabilistically selected. So if one considers the weight of a particular connection as a random variable (across different forward passes) then the expected value over that random variable changes (smoothly) as training progresses - resembling a standard weight update process quite closely.
review_title: Simple idea, well executed, well presented, and with good results - but lacks a bit in novelty

Meta-review: The idea behind this paper is to develop a training algorithm that chooses among a fixed set of weights for each true weight in a neural network.  The results are reasonable -- though difficult to quantify as either good or surprising -- performance from the algorithm. A perhaps interesting point is that additional fine-tuning from these found networks can, in some cases, best the accuracy of the original network.

The pros of this paper are that it is a neat original idea. With the exception of the limited scale of the benchmarks (i.e., the selected architectures), the paper is largely well-executed.  The primary shortcoming of the paper, as discussed by the reviewers, is the lack of clarity in its implications. Specifically, it is difficult to position the result as contributing to a practical aim or leading to additional future work.  

Based on the reviews and discussion, my recommendation is Reject. In particular, this paper would be significantly improved by bringing in a strong motivational context and, therefore, additional comparisons.

For example, the context for the work of Ramanujan et al. (2019) is that, perhaps, it is possible to find subnetworks of large initialized networks that will permit more efficient training. In Appendix A, this paper proposes that the technique here could be cast as pruning within a much larger network. Following results from Zhu and Gupta [1] and also Ramanujan et al. (2019), finding a sparse network within a larger network can produce a more accurate network than training a network of equivalent size to the sparse. Therefore, these results could, potentially, be cast and as a more efficient way to perform the techniques of Ramanujan et al. (2019). 

Alternatively, the results that demonstrate that fine-tuning the identified networks improves performance over the standard network could be more robustly evaluated and perhaps cast as either an alternative training technique or leveraged as a technique like warm starting [2].

This is a very interesting and promising direction. It appears that the paper just needs a bit more distillation.

[1] To prune, or not to prune: Exploring the efficacy of pruning for model compression. Michael Zhu and Suyog Gupta. In International Conference on Learning Representations Workshop Track, 2018.

[2] On Warm-Starting Neural Network Training. Jordan T. Ash, Ryan P. Adams. NeurIPS 2020
 

Decision: Reject