Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a method to reduce the number of operations in a CNN by reducing the input to subsequent layers via the clustering of (intermediate) input features/pixels. The authors claim to incorporate the method to be more inherently combined with network architecture coining it, transient redundancy convolution. In particular, the method clusters the rows (convolutions) and replaces the original input with clustered centroids. The paper addresses the issues of how to perform backpropagation with the proposed formulation and provides guarantees for robustness and convergence.  

strengths_and_weaknesses: The idea of clustering the inputs on the fly is interesting compared to the reducing the size of the input to NN only, or compressing the network and fixing its parameters. In TREC for every input data, we perform individual clustering at every stage of the forward pass to reduce further convolution operations. As a result, the redundancy is computed at every stage of the inference which makes it more flexible than reducing the size of the input to NN or decreasing the size of the network. 

The analysis of complexity, convergence, and robustness is giving better insight into the theoretical guarantees of the algorithm. 

The paper seems to heavily rely on the DeepReuse. It's following the same ideas, for example, using the same method for “clustering”, LSH hashing, same parameters L and H. Please state clearly the differences between the two approaches, and what elements of your method cause the improvements in TREC as compared to DeepReuse (also, I hardly see the randomness as the big disadvantage of DeepReuse).

questions: In my understanding by clustering, we reduce the number of convolution operations (or scalar products given the GEMM). Theoretically, it sounds promising but in practice, can you actually obtain any speed-ups? Do you also decrease the number of channels of your model (maybe if you remove all the convolutions for a given channel)? Are you planning to release the code?

I'm not sure where the terms come from. For example, in pruning literature, I never heard of the term “lasting redundancy”. Also, the phrases “eliminate 96% of the transient redundancy” can be puzzling without the proper definition in terms of FLOPs, for example.  I'd be cautious with using new terms, which may create confusion.

limitations: Little is said about the work’s limitations.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes, implements and tests a method to remove transient (data-point dependent) redundancy in the prpagation and backpropagation of convolutional neural networks.
strengths_and_weaknesses: Originality: medium
This paper proposes the use LSH for removing (transient) data-point dependent computation cost of neural networks. While the idea has been used several times earlier (eg. in the Reformer [https://arxiv.org/abs/2001.04451]), its application for convolutional networks poses several technical challenges. While it is not too hard to see the solution to those challenges, it is still important and useful for giving a clear recipe for using them in practice and measuring their effectiveness in a real world scenario. In that sense, I think this paper makes a real contribution. Also the paper gives some theoretical guarantees in the limit which are new.

Quality: medium
This work proposes an LHS-based method to remove redundant computations in convolutional network. The method needs to do online clustering on the edge device, which results in a data-dependent reduction of computations. The paper reports an "acceleration" of the  layers applying this method by up to a 18X speedup. However this is not backed by a clear table of actual numbers, nor its dependence on batch size, neither ablation analysis or even mentioning the device on which this acceleration was reached. More thorough numbers are reported on the inference of the whole network for micro-controllers and the paper reports a total of 3.61X speedup on microcontrollers for the CIFAR-10X network, which would be impressive on its own and a good start.
However, the lack of evidence on data sets with larger than 32x32 images is disappointing and may question the relevance of the approach for real world use cases.
The theoretical analysis is nice and seems sound to me, but it works only in the limit and does not come with actual bounds, so its relevance for practical application seems questionable. Also it is unclear how far the theoretical bounds are from the real situation and whether the practical situations are within the range of the theoretical results being of any relevance whatsoever.

Clarity: medium
There are several unclear points in this paper:
- The motivation for implementing backpropagation at all is unclear: Do we want to train on the edge device? Or do we want to fine-tune the model for the edge device such that it performs better after optimization than the original network? I assume the latter, but this questions are left for the reader, while the paper puts a lot of emphasis on making backpropagation work. In the latter case (the network finetuned for higher quality evaluation), the paper lacks clear evidence and ablation analysis for the need/efficacy of that appraoch.

Significance: medium
If the method works it could give a clear recipe to improve the latency of CNN inference on certain compute-constrained edge devices. However, it is likely that with inroads of deep learning, specialized hardware will be used in most use cases, which might remove the need for such methods in the not too far future.

Generally, I think that the paper gives a valuable baseline and recipe for reducing the latency of inference of CNN on microcontrollers, but the experimental evidence given in this paper is not very conclusive, neither is the relevance of the theoretical evidence. I would raise my score in the presence of stronger practical evidence (evaluation on a dataset with larger images) and better motivation(/ablation analysis, if applies) for the backpropagation-related sections.
questions: - Do we want to train on the edge device? Or do we want to fine-tune the model for the edge device such that it performs better after optimization than the original network?
- Any empirical evidence on data-sets besides CIFAR-10?
- Do the theoretical evidence give rise to concrete, quantitative bounds?
limitations: Yes the paper addresses the limitations of the methods. I can't see any particular negative societal impact specific to this line of work besides generic concerns that would apply to any methods that improve the efficiency of neural network evaluation in general.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a principled way to detect and avoid transient redundancy for CNN inference. By introducing a new CNN operator named TREC, it fuses the detection and avoidance of transient redundancy into CNN, making them part of its inherent architecture. TREC comprises three components in series on top of Im2col+GEMM based convolution: LSH-based clustering that reduces the size of the input to the conversion, matrix multiplication of weights and the compressed input, and a recovery step that restores results to the original GEMM output sizes.
strengths_and_weaknesses: Strength
- By introducing a new CNN operator named TREC, it fuses the detection and avoidance of transient 40 redundancy into CNN, making them part of its inherent architecture
- TREC is compatible with both forward and backward propagations, enabling a plug-and-play replacement for convolutional layers in mainstream CNNs without additional effort.
- Theoretical analysis on the convergence is also provided.

Weakness
- The introduction of transient redundancy is not quite clear, it would be great to have some intuitive illustrations or numbers to demonstrate the redundancy. Why TREC makes the design and how it makes inference faster in sec 3.1 is also not quite clear.
- The evaluated CNN architecture seems not representative for common usage e.g., resnet.
- It is not clear whether architectures with BN layers can use TREC conv layers and how BN layers to be changed with the changed input feature maps.
questions: - “Experiments show that by removing 96.22% transient redundancy, TREC achieves an average of 4.28x speedup compared to the conventional convolution operator.” What is the maximal possible speedup if 100% transient redundancy is removed?

- How would the BN layers (if the architecture has BN) be accommodated with such reduced input feature maps?
limitations: No limitation is discussed.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a convolution operation, dubbed TREC, with transient redundancy that is exploited during inference to reduce computation. Specifically, the authors use LSH to cluster layer inputs in the forward pass and use the cluster centers in the weight multiplication to compute the outputs. The authors propose a differentiable implementation for the LSH-based clustering which allows for training the TREC, thereby improving the accuracy compared to prior methods. In addition to empirical evaluations, the authors provide a theoretical analysis of the convergence and robustness of TREC.
strengths_and_weaknesses: Strengths:
- The paper is well written and the ideas are clearly explained.
- The combination of theoretical evidence with empirical results helps strengthen the claims.
- The ideas for making the clustering differentiable are interesting.

Weaknesses:
- The LSH-based clustering is previously proposed in the Deep Reuse paper. Therefore, the main contribution here is enabling backward propagation with a differentiable implementation for the clustering. The authors should explicitly clarify this in the paper and give proper reference to the prior art.
- The method is only evaluated on small-scale benchmarks (CIFAR-10) while prior work includes results on ImageNet. As such, a question rises regarding the applicability of the proposed method when evaluating larger scale datasets. 
questions: Please see the weaknesses above. 

Questions and limitations:
- The reported accuracies for the Deep reuse paper are quite different from what is reported in the original paper (they claim no accuracy loss). Can the authors please clarify on the main differences in the evaluation setup that causes this difference?

-  Since the prior work Deep reuse, which is using a quite similar method, has evaluations on ImageNet, the authors are encouraged to add these evaluations to show that the scope of the proposed method is not limited to small datasets and the accuracy gains are persistent in more challenging benchmarks.
limitations: The authors have not discussed the limitations and potential negative social impact of their work
review_title: null

Meta-review: metareview: This work proposes a redundancy pruning mechanism for convolutional neural networks to optimize their performance for extremely resource-constrained edge devices, such as microcontrollers.

The work is based on a novel gradient-optimized locality sensitive hashing approach that removes a lot of the indeterminacy of previous approaches. This method can achieve reliable high performance while improving the network inference latency significantly (eg 4x in some circumstance)

I think that the alone the idea of tuning LSH via SGD directly for clustering purposes is a good one and is of interest to the wider community.
confidence: Less certain
Decision: Accept