Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This work investigates compression-aware training and introduces a new hyper-parameter called "Non-regularization period". Basically, the paper proposes to apply weight regularization and compression less frequently so that they can use stronger regularization/compression, hence achieving high compression ratio and model accuracy.

Overall, the idea of asynchronous regularization and compression is interesting and worth further investigation. However, I vote for a reject for now because
(1) The paper is hard to follow and the writing can be significantly improved especially for the abstract and introduction. I felt rather confused when I first read the abstract and introduction. The authors frequently use the word "weight update", however it is unclear whether it refers to gradient update or the update caused by regularization/compression. Besides, the title is also a little confusing. There's little discussion about the choice of batch size and I don't really understand how weight regularization is decoupled from batch size after reading the paper. I suggest the authors to think more about the title.
(2) The reason why such a training scheme improves model compression is unclear and needs further investigations. In the paper, the authors first interpret model compression as a way of inducing weight decay (and random weight noise). Particularly, the model accuracy is roughly constant over different value of NR period (as shown in Figure 3 and Figure 4) when weight decay is used. However, the results of model compression are quite different (according to Figure 5). The best performance is only achieved with very large NR period in the case of SVD and pruning. I think the authors should give some explanations for that difference. Also, I notice that quantization behaves more similarly to weight decay and it requires further discussion.

To sum up, I like the idea of asynchronous regularization/compression, but I'm not quite satisfied with current version of paper. I encourage the authors to improve their writing and add more discussions (see my point (2)) to the paper. I'm willing to increase my score if the authors can address my concerns.
review_title: Official Blind Review #4

final_score: rating: 8: Accept
review_content: I think that the paper is very well written, I like it. The authors localized a phenomenon  and demonstrated how to exploit it. I trust the results because I performed exactly the same experiments for CIFAR-10 with longer non-regularization periods and found that there is no effect (this is also that the authors show in the paper)  but I didn't test on other datasets and obviously didn't think about potential benefits for compression. 
Since model compression is not my field, I would just make a general suggestion to include and compare to state-of-the-art techniques. 

---
Update: 
The authors updated the paper and replied to all reviewers in detail. I still think that the paper should be accepted. 
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper studies a model compression algorithm, where training consists in alternatingly updating a full precision weight vector using SGD for a number of steps N, and then compressing the weight vector to a quantized or low-rank representation. The authors present an empirical study of how the properties of this algorithm change as they change the number N.

The contribution is marginal, and the paper is hard to read.
Periodically compressing networks weights, rather than at every iteration, is not new. The paper is missing a "related work" section that embeds the proposed algorithm in the literature.
The experiments are small scale and not exhaustive and there is very little comparison to previous work and alternative methods.

Questions/comments for the authors:
- When you say "regularization" you seem to usually mean "compression"? This was confusing to me.
- What do you mean by "asynchronous regularization"? You seem to mean periodic compression?
review_title: Official Blind Review #2

Meta-review: This paper proposes to apply regularizers such as weight decay or weight noise only periodically, rather than every epoch. It investigates how the "non-regularization period", or period between regularization steps, interacts with other hyperparameters. 

Overall, the writing feels somewhat scattered, and it is hard to identify a clear argument for why the NRP should help. Certainly one could save computation this way, but regularizers like weight decay or weight noise incur only a small computational cost anyway. One explicit claim from the paper is that a higher NRP allows larger regularization. There's a sense in which this is demonstrated, though not a very interesting sense: Figure 4 shows that the weight decay strength should be adjusted proportionally to the NRP. But varying the parameters in this way simply results in an unbiased (but noisier) estimate of gradients of exactly the same regularization penalty, so I don't think there's much surprising here.

Similarly, Section 3 argues that a higher NRP allows for larger stochastic perturbations, which makes it easier to escape local optima. But this isn't demonstrated experimentally, nor does it seem obvious that stochasticity will help find a better local optimum.

Overall, I think this paper needs substantial cleanup before it's ready to be published at a venue such as ICLR.

Decision: Reject