Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: I am the emergency reviewer. Sorry for the late.

This paper studies a very interesting topic: eliminating small magnitude components of weight and activation vector instead of eliminating small magnitude components of gradients. A clear interpretation and definition towards the forward and backward propagation is presented. The difference between meProp versus SWAT is also plain. Based on some experiment results shown in Figure2, authors announced that accuracy is extremely sensitive to sparsification of output gradients. Thus algorithms SWAT and SAW are proposed to prune the model, which are respectively training with sparse weights and activations, and SWAT only sparsifies the backward pass. Top-K selection is implemented to select which components are set to zero during sparsification.

Strengths:
1. The writing logic ascends step by step.
2. Authors showed the harmfulness of the sparsity of gradients by experiment results. Also the comparison between the sparsity of weights and activations are meaningful.
3. Sufficient experiments are done to generalize SWAT to different models, and the results are fascinating on ImageNet.

Weaknesses: 
It's a borderline paper. 
1. lack of novelty. The paper has shown a lot experiment results on basic models, but the raising of Top-K algorithm is not novel. Why it is Top-K but not other metrics for selecting zero components? In this view, the paper is likely to be a project summary.
2. Less comparison to other basic pruning models. More experiments should be done to compare SWAT with other sota pruning models. Then the results will be convincing.

review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This paper proposes SWAT as a training algorithm for sparse networks on different architectures. The paper claims being able to reach a level of sparsity with no drop in accuracy.  The goal is to minimize the computations during training time. To this end, SWAT sets to zero the vectors where necessary. Different from other approaches, SWAT uses sparse computation in the forward and backward passes. The intuition behind is that eliminating small components does not have an impact on the training process but can be used to minimize the computation required. 

Some Comments:

- The paper is a bit on the empirical side with a decent number of experiments to demonstrate the effectiveness of the proposal. I am on the border between accepting and rejecting. 


- The top-K implementation is interesting. Page 7 suggests the top-K do not change during training which is reasonable as the update is limited to those components. Would it be possible to avoid completely that compute and quickly select K early in the training process? I would find that an interesting future direction. 

- In the experimental section, I missed actual numbers. At the moment, if I understand correctly, the paper is based on theoretical compute savings. How feasible is this considering the sparsity of the operation (assuming unstructured sparsity)? 

- In the case of structured sparsity, how this differs from the early pruning process of regularization based pruning algorithms? For instance, in the first reference (compression-aware training), the authors claim the model can be compressed in the early training. If that is the case, how different is SWAT from those type of methods? In those related works, the accuracy does not drop. Implementation wise, those algorithms do make the backward pass also sparse (setting to 0 the gradients). 

- At the moment, the algorithm is using a magnitude-based sorting. Would it be possible to have other sorting approaches?


Minor things:

- for clarity, I would summarize the algorithm in section 2.2 rather than in the appendix. 

- I am surprised by the imagenet training setting. Why only training for 50epochs? The standard training process is 90epochs changing the learning rate in the 30th and 60th. 

- I guess the S% sparsity contribution can be improved (rephrased). If the training algorithm sets to zero N parameters seems to me obvious that there will be no drop in accuracy compared to that training process. What is the drop in accuracy referring to?

- check the references. While the list is quite comprehensive, some of them are not referred in the text. Please, add comments where appropriate.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper studies training neural networks with sparse weights and sparse activations (SWAT training). By using sparse weights in forward passes as well as sparse weights and activations in backward passes, SWAT can reduce the computation overhead and also reduce the training memory footprint. The primary contributions of the paper are in three folds: 1) The authors empirically compare the impact of (activation) gradient sparsity and weight + activation sparsity on the model performance---the comparison shows that the weight + activation sparsity has less influence on the model accuracy; 2) Across different models on CIFAR and ImageNet dataset, SWAP can reduce the training flops by 50% to 80% while using roughly 2 to 3x less training memory footprint saving (weight + activation); 3) The authors empirically study on why training using top-K based sparsification can attain strong model accuracy---the magnitude-based top-K approach can roughly preserve the directions of the vectors.

I think the claimed contributions are well-validated in general. The design decisions of the approach are well supported by empirical observations and the components of the approach (different top-K methods) are studied properly. Additionally, I like the authors' synthetic-data studies to shed light on why top-K based sparsity can work well. Given the above reason, I give week accept and I am willing to raise the score if the following questions / concerns can be resolved in the rebuttal / future draft:

1. In results such as in figure 4, we observe that using intermediate levels of sparsity can actually demonstrate better generalization performance than the dense baseline training approach. I was wondering if this is because the default hyperparameter produces better training loss in sparse training than in dense training, and consequently the sparse training test performance is also improved over dense training. Without showing this, it is not fully convincing that intermediate sparsity helps prevent overfitting and generalizes better (as the authors discussed in the text).

2. For "Impact on Convergence" in section 3.2, it is not clear to me what the authors are using as a metric for the degree of convergence. Thus I can not evaluate the claims here.

3. For "Efficient Top K implementation" in section 3.2, the authors suggest  only computing the K-th largest elements periodically to further improve efficiency. However the empirical evidence of whether this approach will significantly degrade the model performance at the end of training is not provided.

4. For the GFLOPS comparison in Figure 7, could the authors elaborate what operations are included into the count? As the sparse operations requires additional indexing operations for computation, I was wondering whether the GFLOPS can realistically reflect the real latency / energy efficiency of the SWAT approach.

5. How the memory access count calculated at the end of page 7? Is it counting the number of float point values (activations, activation gradients, weights) that needs to be fetched for forward and backward pass?

6. At the first paragraph in page 8 (last paragraph above section 4), do the authors imply that the activations of BN layers is not sparsified? Could the authors provide a bit more evidence on how (any why) sparsification of BN activation impacts the model performance.


review_title: Official Blind Review #3

Meta-review: The paper is proposed a rejection based on majority reviews.
Decision: Reject