Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes soft pruning and soft l0 regularization. 

The soft pruning learns a binarization function based on a sigmoid. I am curious to understand how the algorithm can let some low magnitude weights not be pruned and some large magnitude weights be pruned in the end. The paper claims the key is updating all the weights while maintaining the masks for inference. That seems to me ineffective compared to L1 or similar methods. Also, even in the case the weights are not masked, eventually, those gradients should become zero (as they are masked in the forward pass) and therefore preventing them from having large values, isn't it? That needs some clarification. 

As the paper then focuses on L0 regularization, I missed comparisons to related work on that regard. This is not the first paper aiming at using L0 and proposing a differentiable approach. How this compares to others?  For instance, a quick search led to LEARNING SPARSE NEURAL NETWORKS THROUGH L0 REGULARIZATION in ICLR 2018. 



I am confused with one of the contributions is to "provides a trace of checkpoints with varying pruning ratios and
accuracies. Because of this, the user can choose any desired checkpoint based on the sparsity
and performance requirements for the desired application"
Why is this different from any other approach? As soon as the code saves the checkpoint (which most do) then, the user has access to the same flexibility, right?


The section about the hyperparameters is confusing. How are the hyperparameters determined? Each architecture is using a different hyperparameter, how a user could set these?

The experimental results do not seem to support the novelty and the text is kind of misleading. LTP is below Renda and Kusupati. Text suggests Kusupati uses a better baseline, would it be possible to show results of LTP on that baseline? if not, why not?
For Renda, seems like the key difference is the training process. Would be good to see the benefits of a longer training process for LTP. It is not clear to me that LTP can get better results even training for longer. 

Results on more compact networks compared to the self-implementation of Global pruning seem promising for larger compression rates. 

The paper also claims the benefit of learning the threshold per layer, however, provides no result on the distribution of those parameters. Would be interesting to see how these values are distributed in each architecture to reinforce the value of not using a single value for all layers and architectures. 






review_title: A parameter based pruning method based on a relaxation of L0 regularization and a novel per layer threshold learning

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposed a new method to prune a neural network. The method is interesting, innovative and effective. It makes it possible to learn tunning parameter via back propagation, hence learn together with network's weights. 
The work is well motivated. 
The paper is well structured, the writing is clear and easy to follow.
The conducted experiments are thorough and clearly show the efficiency of the proposed method. The paper contains enough information to replicate the experiments.

The work would be beneficial for others if the code is published open.

A question for clarification: When hard prunning the network (section 4.1), we just replace sigmoid(x) by step(x)?

Post-discussion update: I have read the updated paper and other reviews, especially from reviewer #2. While I am still positive about the approach/methodology, I am not confident about the technical details of the experiments, without which, it's very hard to justify the effectiveness of the method. I share other reviewers' views regarding inconsistencies, e.g. Tables 2-5, that have not been fixed in the updated paper.
review_title: Review of the paper Learned Threshold Pruning

final_score: rating: 4: Ok but not good enough - rejection
review_content: ## Summary

The paper introduces a new type of soft threshold operator in conjunction with appropriate weight regularization that can be used in the context of neural network pruning to obtain sparse, performant networks from pre-trained, dense networks. The main idea is to replace the Heaviside step function that occurs in "hard threshold" pruning, which is non-differentiable, by a sigmoid function that can be differentiated and thus enables the efficient training/optimization of relevant pruning parameters. Pruning is hereby performed on a per-layer basis by training a regularized per-layer threshold. 

## Score

I am quite intrigued by the method and I think it has potential. It seems easy enough to implement while providing decent improvement over competing methods, at least judging from the sparse experimental results that were presented. This brings me to the big weakness and the reason for my score. The experiment section does not provide enough evidence in my opinion to justify acceptance since I cannot say with full confidence that the method reliably performs well. And even when the approach does not outperform existing methods in all aspects, at least I would like to be able to judge what are the scenarios where the method performs well. More details are provided below. 

## Ways to Improve My Score

Mainly, please address the points I mentioned in the "Weaknesses". Some of them can be addressed by updating the writing. However, the major concern of mine is the experiment section. I think it requires a full overhaul including more comparison methods, standardized experiment settings, better organized presentation of the results, and clear description of the hyperparameter choices. 

## Strengths

* The concept of the introduced soft threshold operator is easy to follow and intuitive. I appreciate the detailed description of the resulting derivatives in Section 3 and the provided intuition. As such the method is well-described and the benefits are clear. 

* The various aspects of their ablation studies are interesting, c.f. Figure 1, Figure 2. It helps better understanding some of their design choices. 

* The presented experimental evidence seems to hint at very decent performance, especially at the ImageNet scale. This is encouraging to see and underscores the intuition behind the method. 

## Weaknesses

* To me, the biggest weakness is the presented experimental evidence. It seems scattered, the presentation is confusing, and most of all it makes it extremely difficult to assess the performance gains of the presented method in comparison to existing methods. Some points that I would like to list specifically are below: 
  1. There is no single figure that allows me to assess the prune-accuracy trade-off across a large range of prune ratios and for various comparison methods. Something like Figure 4 in the paper by Kusupati et al. 2020 (https://arxiv.org/pdf/2002.03231.pdf) is, in my opinion, necessary in order to more reliably assess the resulting performance. 
  2. The authors only present a selected set of prune ratios and resulting accuracies in their Tables 2-5. Also the results are presented inconsistently. Table 3 only reports compression rate. Table 4 reports Top-5 accuracy and compression rate but not Top-1 accuracy. Table 2, 5 present Top-1 and Top-5 accuracy but only a few selected comparison methods that differ from the one presented in Table 4. 
  3. More comparison methods: Unstructured pruning has seen quite a few advancements over the last couple of years and as such I believe it is crucial to compare to many more pruning methods. This is particularly important since standardized comparisons are missing and so simply presenting some results gathered from other papers is not enough in my opinion. 
  4. Were experiments repeated multiple times? I can see that Figure 3 was based on 10 repeated runs. What do the error bars represented? Also, what about the other experiments? Could the authors clarify how the other numbers were generated and also report mean and standard deviation. The experiments should be repeated at least a couple of times. 

* A more thorough comparison to STR (Kusupati et al. 2020, https://arxiv.org/pdf/2002.03231.pdf). STR shares a lot of similarities with this work in the sense that STR also introduces a per-layer threshold for pruning that can be efficiently optimized using a differentiable soft threshold operator. There is only one comparison point in Table 2, which however seems to be based on a different implementation thus resulting in a different baseline accuracy. There is also no discussion how the soft threshold operators of STR and LTP differ and what makes one better than the other.  

* The experimental hyperparameters are not fully listed and the ones that are listed are scattered throughout the paper. Also, the authors did not provide code; so I couldn't check their implementation either. In particular:  
  1. The contributions list in the introduction mentions the number of pruning and fine-tuning epochs for some experiments but the experiment section doesn't provide a full overview of pruning+fine-tuning epochs. Table 3 provides some of these numbers but not all details. Also what are the particular reasons for the choices? ResNet50 seems to require 30 total pruning epochs, while MobileNetV2 requires 101 epochs. Why? 
  2. What does $\sigma^2_{|w_{kl}|}$ in equation 15 refer to? 
  3. What about training parameters? Are those the same as in the original paper? What about an overview table with all training parameters? 
  4. How were the comparison methods implemented? Were they even implemented or were the results taken from the respective papers? 

* I am not convinced that $L_1$ and $L_2$ regularization don't work for pruning in general as the authors claim in Sections 2 and 3.2. I understand their point and in their case their approximation to $L_0$-regularization indeed seems to play a central role but I wonder how true this statement is in general. In particular, while batch normalization (BN) may allow for arbitrary re-scaling of layers, the _relative magnitude_ of weights may still be impacted by $L_1$-regularization. Moreover, after all they don't use $L_0$-regularization either, just another differentiable approximation to $L_0$-regularization (just like $L_1$ is a differentiable approximation to $L_0$). 

## Other Minor Feedback

* I find the introduction and related work interesting and it serves as an appropriate motivation for the work. However, I find it somewhat disingenuous and/or misleading not to mention global magnitude-based pruning in the first sections. The authors cite a lot of related work that requires manual and/or more complicated approaches to identifying per-layer sparsity patterns when the most obvious solution is to perform global magnitude pruning (GMP). Since this usually works quite well as baseline and the authors also compare to GMP in their experiment section, I believe this merits a longer discussion where the authors compare to GMP. 

* Section 3.3 could be simplified. I was pretty confused since a lot of prior quations are cited in the explanation between newly introduced equations. That makes it pretty hard to read and I believe the introduced concepts and resulting equations could be streamlined. E.g. you could introduce all relevant equations in a continuous paragraph instead of jumping between equations, which results in the heavy use of equations citations. 
review_title: An exciting new approach to pruning but the execution is flawed

final_score: rating: 6: Marginally above acceptance threshold
review_content: My overall feeling about this paper is that I really liked it. I felt it was clearly written, nice pacing, didn't bombard us with things we already know, nor did it skip over things we don't know about. I like the approach of using the sigmoids to be able to learn the thresholds and the weight pruning. I really like this approach. In practice, there are still several hyper-parameters to tune, which are apparently model-specific (table 1), and it seems unclear how to directly set the sparsity (we can only influence it using lambda, not set it directly, as far as I can see?). Nevertheless I do really like this approach and direction, ie of learning things as much as possible. The approach seems to me analagous to using eg Adam instead of SGD.

I feel the experiments section could be written a bit more clearly so as to make the assertions in the conclusion and introduction stand out as really obvious. For example, the assertion that LTP can be used in the presence of batch normalization was not made apparent in the experiments section I felt, nor was it shown as a benefit compared to other baselines, I felt.



Details:

'computationally extensive' => 'computationally expensive'

figure 1 really far from where it is first referenced. had to hunt for it....

writing really clear. Good exposition of related papers. Pacing very nice. Very easy to understand.

Good choice of which information to present. (cf papers that present lots of well-known knowledge, or skip over some key advanced concepts).

Kind of a detail, but I like the use of $sigm$ to denote sigmoid, (cf many papers will write the sigmoid out in full, which makes the equations much harder to read)

Personally, I would prefer that $\sigma_T(w_{kl}) = \frac{\partial sigm((w^2_{kl}-\tau_l)/T)}{\partial w_{kl}}$ is defined before equation 3. Otherwise I first read equation 3, wondering what is $\sigma_T$, and then realize it is defined underneath.

To be honest I'm not a fan of the notation $\sigma_T(\cdot)$, since when I first read it I parsed it as $sigm_T(\cdot)$. I would prefer either using some other symbol, or perhaps writing out the full partial derivative, though that seems probably long, without some other symbol, or perhaps adding a $'$, like $\sigma_T'(\cdot)$.

I'd also prefer that the definition of $\sigma_T(w_{kl})$ includes the derivative in this definition, ie:

$$
\sigma_T'(w_{kl}) = \frac{\partial}{\partial w_{kl}} sigm\left(\frac{w^2_{kl} - \tau_l}{T}\right) = ... etc ....
$$

I like the paragraph that describes the behavior of $\sigma_T'$

I like the exposition of the various regularization methods. Concise and yet easy to understand.

$\eta$, in equation 8 is not obviously defined anywhere. I went hunting for it, but couldn't find it. Please define it before using it.

Equation 8 comes out of nowhere, with no explanation of what it means, or why it is. The rest of the paper explains things very well, but equation 8, I'd have to think a lot, wasn't immediately obvious to me where it comes from, what it means.

I'm not sure what the syntax $\sim T$ means here. It normally means 'is distributed as', but that meaning doesn't seem to make sense here? It looks like it's being used to mean $\approx$?.

Ok, I had to go all the way back to equation 2 and the paragraph after it. Looks like $\sim$ is being used here to mean $\approx$. I think that using $\approx$ would be more standard, and easier to understand? Ok, I googled $\sim$, and it turns out that it can often be used to mean 'is approximately the same order of magnitude as', eg https://math.stackexchange.com/a/2177014/45703  But I personally found it confusing because it is very often used to mean 'is distributed as', so personally I would prefer to have a short explanation like 'where $\sim$ means "is approximately the same order of magnitude as"'

I guess the other reason I find it confusing though is this equation implies to me that $|w^2_{kl} - \tau_l| = 0$ would not be in the region, but in fact the region is I feel something like:

$$
|w^2_{kl} - \tau_l| \lesssim T
$$

Personally I think I would prefer the conditions for the transitional region written in this way; would be less confusing for me; I think.

Similarly equation 5 would be:

$$
\sigma_T(w_{kl}) \approx \frac{1}{T}, \text{ for } |w^2_{kl}-\tau_l| \lesssim T
$$

and then equation 8 becomes:

$$
\eta \cdot \left| \frac{\partial \mathcal{L}^T}{\partial w_{kl}}\right| \ll T , \text{ for }  |w^2_{kl}-\tau_l| \lesssim T
$$

I'm not sure though why this derivative is in this constraint? Isnt the constraint simply that

$$
\sum_{l=1}^L \sum_{k=1}^K I\left[ |w^2_{kl} - \tau_l| \right] > m
$$
where $I[\cdot]$ is an indicator function, and $m$ is some positive integer?

ie, at least some points need to be in the transitional region. I'm not sure I follow why we need a derivative in the constraint. Please can you add some description around equation 8 so I can follow what is going on :)

And also trying to work through equation 8, it seems like it is saying that we want to make the derivatives as close to zero as possible, relative to T. But isnt this the opposite of what we want? Dont we want to have a reasonable number of derivatives that are not near zero?

Ok, after equatino 9, we get some explanation for equation 8 :) But I think the explanation could be moved forward somewhat :)))

Ok, based on this explanation, ie the one after equation 9, $\eta$ is probably learnin rate. But please define $\eta$, near equation 8 :)

From the explanation, I'm not sure that equation 8 is a condition that actually *prevents* premature pruning, so much as a heuristic to minimize weights moving too quickly out of the transitional region. Preference to be clearer about this, since it would certainly have helped me to understand equation 8 more easily and quickly :)

equation 11, the brackets could be nicer looking if use "\left(" and "\right)", I feel (so they are as large vertically as the derivative fractions they contain)

$\lambda$ in equation 11 is not defined. From section 4.1, we can see it is a hyper-parameter to be tuned, but that is not stated in equation 11. Preference to state at equation 11 that $\lambda$ is a hyper-parameter, and $\eta_{\tau_l}$ is the learning rate for the threshold of layer l. Hmmm, does this mean that each layer has its own learning rate to tune for its threshold? If there is one single learning rate for the thresholds, I think it might be clearer to represent it as $\eta_\tau$? If there are per-layer learning rates, then this seems to be to contradict the implied promises in the introduction that we don't have per-layer hyper-parameters to set?

Ok, looks like $\lambda$ was first used in equation 9. But still wasnt defined there I think?

I'd also prefer that equation 9 was defined before presenting equation 8, on the whole. This way I can read sequentially, not have to skip forwards and backwards.

The sentence just before and after equation 12 is very complex and hard to take in. Please consider breaking into smaller simpler sentences. eg 

"$\partial L_T/\partial w_{kl}$is given by 9, 10, 3 and 7 as:

(equation goes here)

This includes $\sigma_T(w_{kl})$, which from equation (5) $\approx 1/T$, and will become large for $T \ll 1$. This means that the gradient will become large, and constraint (8) will be violated.
"

Figure 1 I feel needs a lot more explanation.
- why are they so symmetrical about the y=x line? doesnt this imply that the number of weights less than theshold before pruning and the number of weights less than theshold after pruning is similar?
- why is the y-axis labeled 'w'? I thought the pruned weights are 'v'?
- why make the plot? What is the motivation of this plot?
- why is the scale of the axes radically different between left and right (0-16 vs 0-3) ?
- why is the left hand plot preferable to the right hand plot?
- why is the gap around threshold in the right hand plot a bad thing?
- why is the proportion of weights below threshold similar in both left and right?


4. experiments

I like the table of hyper-parameters in table 1. (cf many papers skip over which hyper-parameter settings were used, making reproducing the work challenging)

Appreciate the explanation of the significance of $\lambda$ as the primary hyper-paramter determining the sparsity levels. I feel that this explanation could be moved back to equation 9.

Appreciate the observations on how to set $\lambda$, and $\eta_{\tau_l}$.

I wouldnt really call figure 2 an 'ablation study'. It's more like a comparison study of various baselines and approaches I feel? An ablation study I feel would be more like:

- no regularization
- no drop second term in 12 (and simply not use any clamping etc in its place)

I think the ablation study should go after the imagenet pruning results. (I mean, I think it's traditional to put ablation studies after the section of results vs other baselines/sota models)

Table 2 is very unclear to me
- why is your method not at the bottom of the table
- I think your own method should be in bold and say "(ours)" after the name
- looking at the table, it's not very clear why we should choose LTP?
   - the highest rate and top1 looks to be Kusupati et al?
- in the text description, it says that Kusupati uses a stronger baseline, ie STR
   - why don't you use STR too?
- in the text it says that Renda et al needs more training
   - why not put the amount of training required as an additional column in the table?

Jumping to table 3 mid-paragraph is I think jarring. I think first talk about table 2 in one paragrpah, then talk about table 3 in the next. Or at least don't mix and match across tables, at least without having first presented each table on its own first. I feel. Like the sentence 'In fact as table 3 shows' cannot I feel precede the clause 'Finally, figure 3 provides ...', which presents what is table 3. Oh, that's figure 3, not table 3. Anyway, I think table 3 needs some introduction, please.

Yes, so, it seems to me that the resnet50 results from table 3 could be added to table 2 perhaps?

Figure 3 should be in a separate paragraph, since it is not a comparison with baselines/other models. It's just an obseration about LTP itself. And really, without any comparison with how other models/pruning strategies are, I'm not sure it is very meaningful to me? Like, for all I know other models have smaller error bars?

I think table 4 presentation should follow immedaitely table 2 presentation.

Then table 3 presentation.

Figure 3 might be best in a separate 'appendix'-y sub-section at the end of section 4, I feel. Since it's not comparing to other models, like table 2, 4 nad 3 are.

Table 4. If torchvision gives worse results than caffenet, then why not use caffenet instead, or re-implement the caffenet version of alexnet in torch? Otherwise, we can see taht LTP results in table 4 dont match the baselines, and we cannot tell if this is because the torchvision baseline is weaker, and LTP is strong, or whether LTP is weaker than eg Ye et al.

In fact, Ye et al only drops 0.1% top-5 accuracy compared to original, whereas LTP drops 0.4%, compared to torchnet original, so I feel that justifying the lower top-5 error rate on the weaker baseline model is not entirely sufficient?

Table 5 looks like the strongest table to me. Might be worth putting it first? I feel that it could be useful to highlight the top results in each column in each scenario in bold? I think that ideally each column should be a single scenario (whereas here each column is multiple scenarios), then it is easy to highlight the top in each column. For example you could put the different rates as different columns, and use eg top-5 accuracy throughout the table. (or put top 1 and top 5 accuracies as tuples perhaps?)

I kind of think that table 3 should be folded into the other tables.

I think it's not clear from these tables why we should use eg LTP instead of Kusupati et al, or Ye et al. I think that either make it clear in the table somehow, or perhaps put in the text. Like eg "Our method needs considerably less hyper-parameter tuning than existing SoTA methods, whilst achieving nearly the same accuracies for similar levels of compression."

In the conclusion, you mention batch normalization, but this was not brought to the fore in the experiments section. Like, I would expect to see some models that can only be pruned using LTP, and other approaches fail to prune, but I don't remember this being shown clearly in the experiment section?

Basically, I think the assertions in the conclusion are exciting, but aren't made clearly obvious in the experiments section. I think for each assertion in the conclusion there should be a single table or graph that shows this assertion very clearly, in comparison to other possible baselines.

[post discussion edit]

After discussion, I lowered my  score to 'marginally above acceptance threshold':
- the theory section of the paper looks very interesting to me
- I find it hard to see clearly from the experimental section the extent to which the method beats existing methods
- I feel that the experiments could be made more rigorous to clearly show the benefit compared to other techniques
- concretely, I feel taht the tables could be structured in such a way that one can glance at each single table, and see clearly in what way LTP is better than the baselines. Concretely, for the results tables:

- table 2: LTP gives worse accuracy than Renda, and worse compression. The text mentions training epochs are fewer for LTP, but the table doesn't show this benefit (there is no column with number of training epochs)
- table 3: this table is a little apples and oranges I feel. it shows that the number of training epochs is less for LTP than Renda, but the compression ratio is slightly less. I feel that you could have compressed a little more, to make the compression ratios comparable. In addition, I feel it is important to include the accuracy in the table. Without accuracy, then I feel it is not possible to compare.
- table 4: I feel you could do whatever is needed to do to ensure that the baseline model you are using matches the baseline that other teams are using. This could mean porting NTP to caffe, or porting caffe network into torch. Currently, the LTP pruning is on a worse 'parent' model, and performs worse than the other baselines in terms of accuracy. I'm not sure it's sufficient to hand-wavingly just add/subtract the delta in performance between the baselines to the LTP results (which is not explicitly being done, but if one doesn't do that, then one would have to assume that LTP performs worse, I feel)
- which only leaves table 5 that plausibly provides an apples-for-apples comparison, but only for a single baseline
review_title: Really like the approach, and the exposition; experiments section could be made clearer potentially

Meta-review: This paper proposes a method for differentiable pruning that replaces the hard thresholding of standard pruning, with a soft version that permits taking the gradients of the pruning threshold. The proposed benefits are an accuracy that is better or competitive with alternative methods as well. Moreover, the paper suggests the technique to be efficient.

The pros of this paper are that it is working in an interesting setting of differentiable pruning, with the hope of -- in some sense -- simplifying the pruning process or at least unifying the process with standard training.  The technique is plausibly justified in its technical development. The paper also follows with a significant number of experiments. 

The cons of this paper are that the conceptual framework -- beyond the initial idea -- is not fully clear. In particular, this paper does not elucidate a clear set of claims and hence, results in the difficulty on the Reviewers part in detangling the claims and identifying the appropriate comparisons.

For example, the paper doesn't take up a simple claim that it is state-of-the-art in accuracy vs parameter measures (and would seem not to given the results of Renda et al. (2020)).  It need not necessarily make this claim, but there are suggestions to such a claim early in the paper. If this is not an intended claim, then the paper can remove any suggestions to such (i.e., the claims around new SoTA for networks not evaluated in prior work). 

The paper has a somewhat tentative claim that it is more efficient (in the total number of epochs of training) versus other techniques (Table 3).  However, the presented results are only at a single-point versus other methods.  Renda et al. (2020) directly consider accuracy versus retraining cost trade-offs. Appendix E of that paper provides one-shot pruning results for ResNet-50 showing accuracy on par with that presented here.  The number of retraining epochs is also similar to here. This paper, however, only compares against the most expensive iterative pruning data point in the other paper.

In sum, my recommendation is Reject. This is promising work that needs only (1) to include a few testable claims and (2) to re-organize the results (and perhaps run a limited set of new results) to thoroughly explore those claims. For example, if the most important claim is accuracy vs retraining cost, then it needs to show a more complete trade-off curve of the two results.  Of course, this, in principle, opens the door to comparisons to many other techniques in the literature.

Decision: Reject