Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposes an approach to learning sparse deep neural networks (neural networks where most of the weights are set to zero) using annealing: starting by training a large overparameterized neural network to fit the training data perfectly, then adding weight penalties according to a prespecified schedule. A theoretical argument is provided to argue why this is reasonable, with two new theory proofs provided which extend previous work presented in Sun et al. 2021. Experiments on synthetic data show that the proposed algorithms perform feature selection better than other existing algorithms. Experiments on CIFAR10 show the algorithms are competitive in terms of the accuracy vs. pruning tradeoff.
main_review: This paper is mostly clear and well-written, aside from some typos. However, its contribution is unclear to me. I'll address each contribution in turn:

Theory: Proofs are provided for the asymptotic normality of the connection weights and asymptotic normality of the prediction. These appear to be novel contributions that build upon the previous work of Sun et al. 2021. I did not check the proofs carefully, but they appear to be non-trivial and novel contributions. However, I was unable to see their significance; the implications for the algorithms were unclear. 

Algorithms: Two algorithms for training sparse DNNs are proposed. These algorithms seem reasonable, but I was left wondering about the theoretical and practical value. Line 230 says, “the MCMC algorithm is likely to hit the region around the optimum of the target distribution,” but if nothing concrete can be stated what is the advantage of starting from the global optimum and then annealing? Practically, this annealing procedure is more computationally intensive than training with standard regularization methods, and we are left with an empirical question of whether the annealing is more computationally-efficient than other annealing methods.

Experiments: the experiments show that the annealing approach results in sparse networks that have competitive accuracy and calibration to other methods. However, I have concerns about the fairness of these experiments. As mentioned in Guo 2017, miscalibration is a result of overfitting and can largely be avoided by early stopping on the validation set NLL (not 01-loss). However, early stopping was not mentioned, and in the supplementary materials the authors say the benchmark comparison models were trained for a fixed number of epochs. In addition, the hyperparameters (lambda, sigma_0^end) for the annealing algorithms appear to be chosen in order to give good results, reducing my confidence in the fairness of these comparisons. Overall, I'm not convinced that this is a promising approach for training sparse DNNs in practice, if that was the goal. 


limitations_and_societal_impact: The authors adequately address this.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: Overall, the work presents an interesting direction into NN pruning. However, the paper is quite hard to follow and overly complex in notation, with little effort to assist the reader. Further, the paper severely lacks useful devices, such as figures or in-depth description, to explain the methods to readers, making the work challenging to follow.  Last, some of the experimental evaluation is somewhat questionable, as specified in the details comments.  Though the direction seems useful, additional work is needed in communicating the results and in experiments in order to warrant acceptance.

main_review: Per Section Comments
———————————

Section 2.1:
- The paper is reasonable to follow up until the statement of assumptions (around line 106). However, at this point, why are these assumptions reasonable? It’s tricky to see what some of these assumptions mean (i.e., A.2.1 in particular) and why it might be reasonable to make other assumptions (i.e., A.1, this is rarely the case in practice, A.2.2 why we expect the polynomial bound to hold).  Even if it is the case that some of these assumptions may only be plausible, for the sake of the analysis, more context at this point would be highly useful for the reader.

Section 3.1
- Gradually decreasing the temperature: How is this specified?
- It would be highly useful to provide some more intuition around algorithm 1.  Where does the constraint in (iii) come from, for example. Is \sigma_0^init define anywhere?
- A graphical figure here to explain the algorithm in more detail would be highly useful.

Experiments:
- Investigating the results provided in section 4.2, it seems like the proposed methods yield slightly test accuracy. Yet, the pruning ratio is slightly lower as well for these networks. Because the difference in accuracy is so slight and the pruning ratio changes slightly as well, can you account for these changes in evaluation? Looking over the results, it seems hard to justify that the method is better than baselines, given this dynamic, as the results are currently written.
- Further, the proposed method (if I’m not mistaken) uses additional fine-tuning after the variable selection is applied (i.e., IV of alg. 1).  Do the other variable selection methods do additional fine-tuning? Some discussion about this would be good, because it seems difficult to compare this method (which allows further fine-tuning) with methods that do not.

limitations_and_societal_impact: NA
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The submission addresses the issue of sparsifying trained deep neural networks in an optimal way, preserving performance and improving calibration. It builds on the formalism and results of Sun et al. (2021), proves additional results and proposes two prior-annealing algorithms, which they demonstrate on synthetic and real data, comparing favorably to existing sparsifying algorithms. 
main_review: Originality
----------------
This submission relies heavily on work by Sun et al. (2021), which is properly acknowledged, and expands on that work. The two prior-annealing algorithms seem novel, and the real-data, larger-scale evaluation (ResNets on CIFAR-10, following the set up of Lin et al., 2020 for meaningful comparison) make that work more accessible to current deep learning practitioners. 

Quality
----------
To the best of my understanding, the theory is sound, and the theoretical part of the paper feels complete. The claims of performance and calibration against other sparsity-inducing methods are well supported by experiments, and conclusions drawn by the authors are faithful to the experimental results.

Some of the broader claims may be over-stated, though ("how to make statistical inference with deep neural networks", "tamed the powerful deep neural networks", l.56-57), especially the ones that are not restricted to _sparse_ deep networks.
In order to support those claims, it would be better to have, for instance:
- comparisons of test accuracy against non-pruned network (in Table 2)
- comparisons of the measures of Table 3 (NLL, JS-Distance, ECE) against values obtained by algorithms aimed at improving calibration (e.g., temperature scaling, mixup training, focal loss training...) for regular, dense DNNs, rather than only considering sparsifying algorithms.

Clarity
----------
Considering the complexity of the theoretical work, the article makes a good job at explaining the issues and results informally, giving a good intuition. The overall organization makes sense, and I appreciate the division between main paper and appendix.

I believe a reader already well familiar with Sun et al. (2021) could implement the additional procedures described in this submission and reproduce the results.

Significance
-----------------
 The perspective of taking a trained, well-performing DNN and turning it into a sparse, well-calibrated model, in a principled way, and with minimal loss of performance, is quite appealing, and I expect practitioners will want to use this method.
This submission adds additional theoretical guarantees compared to earlier work, making this method more solid.


limitations_and_societal_impact: The authors did mostly address the limitations of their work in the text.
Potential societal impact were not discussed, but this is mostly theoretical work, so it seems fine to me.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: 1. Theoretical contributions: the authors extend the theory established by Sun et al. [2021] and study the asymptotic behaviors of sparse deep learning.
2. Methodological contributions: the authors propose prior annealing algorithms, one from the frequentist perspective and one from the Bayesian perspective, to implement the corresponding sparse neural nets. They show the performance of the prior annealing algorithms on one synthetic example and the CIFAR10 datasets.
main_review: The authors inherit the sparse DNN framework used in Sun et al. [2021], and they investigate the asymptotic normality of connection weights and predictions.  The theory established in the paper are around the true DNN model $\mu^*(\beta^*, \gamma^*, x)$, which is a data-dependent $L^2$ projection of $\mu^*(x)$ onto a pre-specified neural network space $\mathcal{G}_n$. My intuition is that one would need prior knowledge on what an adequate network structure looks like, and do the pruning based on that, like the CIFAR10 example. It appears more as a refinement to already mature neural net models.
And it is not clear how one can relate the theory to asymptotic behavior around $\mu^*(x)$ since that will depend on the approximation error. Maybe the authors can elaborate more on that. In addition, how the noise $\sigma^2$ could impact the model performance, is it assumed known?

For the two prior annealing algorithms, the results from the experiments look decent. But it is somewhat disconnected from the asymptotic theory and it is not obvious how to assess the uncertainty in prediction. It is not clear how the prediction intervals in Figure 1 are constructed, it seems like a variant of bootstrap. 

Some minor comments:
1. The notation for log-likelihood function $l_n(\beta)$ sometimes appear as $nl_n(\beta)$. Line 126-127, the expression is not properly placed. Overall, the notations in the paper are really heavy and section 2.1 is directly borrowed from Sun et al. [2021].

2. For the references, some conferences as NeurIPS and ICML have inconsistent formats.

limitations_and_societal_impact: I didn’t spot a discussion on limitations. Maybe the authors can discuss based on the reviews and provide the future direction of their work.
review_title: null

Meta-review: This paper proposed a framework for sparse deep learning, with the primary goals as addressing mis-calibration problems. The paper contains both theoretical analysis as well as some experiments to demonstrate the theoretical claims on Bayesian ResNets. 

This paper sits at the intersection of network pruning and uncertainty calibration so it might benefit both communities. However, the paper heavily assumes prior knowledge presented in Sun et al. (2021) which is a very recent JASA paper. I would expect this paper to be very difficult to read for deep learning practitioners interested in network pruning.

The paper stated that "This work, together with Sun et al. (2021), has built a solid theoretical foundation for sparse deep learning". Therefore in my view, acceptance decision critically relies on how significant the contribution of this paper is on top of Sun et al. (2021). 
- From my understanding after a brief read, the key theoretical result in this paper is the asymptotic consistency/normally estimate for the defined "ground truth" parameters. 
- While Sun et al. (2021) established posterior consistency and structure identification consistency, that is regarding the estimation of the data distribution, rather than the underlying parameters. 
- Therefore I think the two paper's theoretical results are different aspects of the same framework. This makes me think that in some sense this is a project split into two publications? 

Experimentally, the proposed approach and the approach of Sun et al. (2021) perform quite close in many of the experiments. 
- This also raises a practical question in that, to what extent it is important to ensure "consistency in parameter estimation"? 
- It is even more confusing that the "ground truth" parameters are defined using "universal approximation of neural network", so I would think "consistency in function estimation" might be more important here, which is already discussed in Sun et al. (2021) to certain extent.
Decision: Accept (Poster)