Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper presents a sparse neural network training algorithm. The technical highlights include:

(1) The proposed algorithm adopts a scheme similar to Reptile, which sequentially trains several sparse sub-nets. 

(2) To handle the conflicting direction of weight updates among sub-nets. Gradient correction is conducted to alleviate the negative gradient inner product.

(3)As for signal sparse subset training, two observations are reported that enabling the freedom of exploring unique architectures of each sub-net elevates the learning capacity of AST over the S-Net, and intermittent sparsity increment among sub-nets stabilizes AST process.
strengths_and_weaknesses: Strengths: 
1. The idea that learning sparse sub-nets sequentially and recursively is interesting. 
2. The sparse network learned by the proposed algorithm has better performance than previous algorithms.
3. The paper writing is good, with sufficient related work introduction.

Weaknesses:
My major concerns are:
1. Experiments are kind of weak.
2. Some of the proposed technical points lack sufficient experiment verification.
3. The algorithm novelty.
questions: 1. If we view the training of each sub-network as a learning task. Is the overall framework a special case of Reptile?

2. The effect of gradient correction is not clear. What will the “negative gradient” and “conflicting direction of weight updates among sub-nets ” bring about to the learning process? I haven’t seen sufficient reason that it is necessary to deal with the issue. The experiment comparison is only conducted on two Cifar datasets.  
 
3. The two observations in Sect. 3.3 are not sufficiently justified in the experiment.

4. Some critical technical details are missing. For example, is the final sparse network an ensemble of all trained sub-nets or the last sparse sub-net?
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper combines two recently emerging methods, sparse training and dynamic inference, to obtain multiple subnetworks with different sparsity levels. Compared with traditional sparse training methods, the proposed method (AST) is able to obtain various sparse NNs with one training run. Compared with dynamic inference,  AST does not need to perform forward and backward for every sub-net. 

I believe this paper, albeit short on technical novelty, does provide a sufficiently novel and important empirical evaluation and details analysis of sparse training and dynamic inference, and could be accepted - assuming the authors can address the issues I have detailed in my review.


strengths_and_weaknesses: ## Strengths

(1) To the best of my knowledge, the combination of sparse training and dynamic inference is novel, which can further improve the efficiency of sparse training. 

(2) The paper is relatively clear and I can obtain the general idea easily after reading the introduction. 

(3) The experimental setting is clear and fair, demonstrating the effectiveness of the proposed methods.

(4) The empirical study in this paper is extensive (e.g., Completely subset (CS) vs Non-disjoint (ND) ), providing insights for researchers who want to follow this topic. 

## Weakness

(1) Several naive baselines are missing. This paper only compares with several sparse training baselines and Jointly-Trained, while some naive baselines are missing. On the top of my head, for instance,  (1) train a dense model, and prune to various sparsity with a short time of fine-tuning. (2) train a sparse model with sparse training methods (e.g., GraNet), and prune to various sparsity with a short time of fine-tuning. While this procedure involves fine-tuning, we can still keep the overall training time as same as AST by decreasing the pretraining time. 

(2) The accuracy of AST are relatively lower compared with the backbone GraNet on ImageNet. I can imagine that the performance of AST is lower than GraNet, since training one network is intuitively easier than training multiple networks. Is the performance degradation caused by the insufficient updates of each subnet? Since the author keeps the overall training time the same as GraNet, the overall training time of each is 1/N (N is the number of subnets) of the GraNet? 

(3) A more elaborate description between the backbone technique GraNet and AST is required to help readers understand the proposed method more clearly.

(3) There are quite many typos and inconsistencies in the paper. A significant polishment is required to be published. i> ''s+r'' in Eq.2 should be  capital. ii> RigL and Rigl should be consistent. iii>  line 129, sub-net 1 < sub-net 2 < sub-net 3: the meaning of this comparison is vague. Does this mean sparsity or model size? iv> what is the difference between inner group and inter group? It would be better to add the inter group in Figure3 as well. v> figure2: I assume the sparsity of the subnet with different colors is different from each other, but green and blue have exactly the same number of connections. vi> the readability of the Tables in the paper is low. I suggest arranging different baselines according to their accuracy, e.g., in ascending order. 
questions: Please refer to the weakness below. 
limitations: yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper introduces an algorithm for obtaining multiple sparse neural networks with different sparsity levels in a single training run. The authors also propose a gradient correction strategy that contains projecting the gradient of a sub-net to the plane of the prior sub-net; the authors explain that this makes the direction of the gradient updates among various models consistent.
strengths_and_weaknesses: Strengths:

- The proposed method is efficient in terms of deriving multiple sparse networks during a single round of training. 
- Figures 2 and 3 describe the proposed methods clearly.
- While the writing and the descriptions can be improved, the paper is well-organized.

Weaknesses:

- One of the main concerns is that while the method derives multiple sub-networks, the performance of each sub-network falls behind the competitors in several cases. The reported accuracies are mostly compared to a single method [23]. I think the results should be compared with more methods given similar training costs to evaluate the quality of the sub-network. E.g., authors can consider deriving subnetworks with similar sparsity levels during a single run of GraNet and compare the accuracies.
- The training costs are given in terms of training rounds. However, as different methods can have different training costs, the costs should be reported using another metric, e.g., FLOPs. Memory costs of keeping the masks for each sub-network need to be discussed more clearly.
- AST-GC requires extra computation at Equation 5, which needs to be considered in the discussions and FLOPs computation.  
- Some points are referenced in the Appendix; however, they are missing. E.g., Line 147 regarding discussion on Equation 4. Line 388 Limitations of the work 
While the general idea is interesting, I believe the paper is not ready for publication yet. 
questions: Most of the concerns/suggestions have been discussed in the previous section. I add some minor comments here:
- In some places, descriptions of methods/experiments are not clear. in Section 3 (or 2), I suggest authors add a formal definition of a sub-net. At first read, it is not clear how W_i are related, and they are coming from the same model. Adding a pseudo-code can help to better clarify this.
- Typos: Lines 139 (,),233 (remains), 290 (iteratively) 
- Line 260, The results from the Appendix are better to be included in the paper. 
- Figure 8. It’s better to include the sparsity levels for each sub-network in the figure.
- How many runs are performed for each experiment?
- I believe [1] is a related work in terms of deriving multiple sparse networks at a single run.

[1] Liu, S., Chen, T., Atashgahi, Z., Chen, X., Sokar, G., Mocanu, E., ... & Mocanu, D. C. (2021). Deep ensembling with no overhead for either training or testing: The all-round blessings of dynamic sparsity. arXiv preprint arXiv:2106.14568.

limitations: Limitations are not discussed in the paper.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes to alternatively train multiple sparse sub-nets to obtain a supernet that supports dynamic inference with varying sparse rates, termed AST. Experiments on ImageNet and CIFAR-10/100 validated the efficacy of the proposed AST, which is also characterized by efficient training compared with traditional supernet methods.

strengths_and_weaknesses: Strengths:
1. The proposed grad correction mechanism is well-motivated.
2. The manuscript is well-organized and the proposed AST is easy to follow.

Weaknesses:
1. The practical value of AST is highly limited. 1) It is a consensus that unstructured pruning methods hardly gain acceleration on general hardware. In contrast, the advantage of unstructured pruning falls into significant parameter saving (storage reduction) without performance compromise. However, AST trains multiple subnets towards "dynamic inference" for sparse networks, which is unreasonable since dynamically running sparse subnets can not save inference costs as structured methods do. Moreover, the resulted supernets even bring more parameter burden than classic sparse training methods. (Although the authors have provided N:M fine-grained experiments, the patterns are selected with 2:4, 3:4, 7:8, 15:16. Here 3:4, 7:8, 15:16 patterns can not be applied on real hardware as far as I can know. Also, the experiments are only conducted on CIFAR-10. Overall, it is not convincing to me.)
2. Lack novelty. The key component of AST is combining prune-and-regrow and supernet-training. Despite the originality of gradient correction, the limited practical value also degrades the overall contributions of this paper.
3. The performance of AST is poor, compared with SOTA sparse training methods like GraNet. 

questions: 1. For the gradient correction part, does Eq.5 work if the overlap of two subnets is very small?
2. I recommend the authors provide experiments of structured sparsity to demonstrate the efficiency of AST against S-Net, US-Net, OFA.
limitations: The authors do not discuss the limitations of their work as well as the negative societal impact.
review_title: null

Meta-review: metareview: The authors presented an alternating sparse training method for building dynamic inference networks. The method comes with an extensive empirical study over multiple benchmark datasets showcasing its advantages over prior dynamic inference methods. The paper is generally well organized and clearly presented. The technical novelty, however, seems to be somewhat limited given that the inspiration of method and theoretical interpretations are drawn largely from the findings in Repitle for gradient based meta-learning. The reviewers also raised several other concerns regarding the connection to meta-learning, additional baselines and experiment details. The authors managed to address some of these in their responses. 

After discussions, the reviewers reached a majority consensus that this is an interesting and technically sound paper where strengths outweigh weaknesses. Therefore, I recommend that paper can be accepted if room available and the promised revision is made to address the issues raised. 

confidence: Less certain
Decision: Accept