Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces learnable compressible subspaces, which attempts to learn a set of models that can be switched at inference time to adapt to different resource requirements. This work is motivated by previous work in neural subspaces and slimmable networks. It is evaluated on CIFAR10 and ImageNet and compared against other recent works for adaptable inference models. These results show under certain conditions LCS can maintain higher accuracies at larger sparsities compared to other works.
main_review: === Strengths ===

Table 1 is useful in summarizing the related works and the claimed advantages for LCS.

The experiment section listed a significant amount of detail on hyperparams and methods.

The small discussion and figures on batch norm stats shifts were interesting and helpful for motivating group and instance norm for this application.

This area of adaptive inference is becoming more and more important with larger models and specialized big-little architectures.

=== Weaknesses ===

It is unclear how these networks switch between models at inference time, which of course should depend on the type of compression used.  For sparsity, this seems like it would require dynamically pruning the model at inference time which seems very dangerous. For unstructured sparsity, this may require special hardware for taking advantage of that unstructured sparsity. For quantization, this may require hardware can support fine-grained switching of the quantization bitwidth.

My understanding is that batch norm stats have to be recomputed for NS and US in a post-training way but not necessarily at inference time. It doesn't seem fair to avoid this step since it can be done before model deployment and takes a fraction of the training time. Leaving it out in the evaluation also nullifies the comparison if this is not fair.

The claim that other methods need additional batch norm params and cannot support fine-grained compression level is mostly correct, but the importance of this seems overstated. In practice, it seems more reasonable to chose a smaller subset of model configurations that can be fully tested before deployment, and the batch norm params should be nearly negligible compared to the weights. Also, quantization LCS does of course limit the number of compression levels, and structured pruning LCS limits the number of compression levels to the number of channels (which is similar to US).

The existence of gamma and alpha together is confusing to follow. Since the parameterization in linear, it seems like only one of these should be necessary.

There should be other works included for building robust compressible models, e.g. Robust Quantization (Neurips20).

The writing is clear but repetitive in some areas. For example, I believe there are 5 sentences talking about being inspired by Wortsman in the first few pages.

=== Questions ===
In the Related Works, the neural subspace method is described as operating on simplices, but the description in Section 3.1 seems to be on lines. Is this deliberate?

Please correct me if I'm wrong but isn't the unstructured compressible point method Dropout? There might be differently weighted probabilities and dynamic dropout probabilities, but they seem fundamentally the same. 

For quantization, what hardware supports dynamic fine-grained switching from 3-8 bits? How are the pruned channels or pruned individual weights chosen at runtime in an adaptive way? 
summary_of_the_review: This paper is an interesting proposal that attempts to apply the ideas of neural subspaces to produce a set of compressed models at varying points on the accuracy / efficiency curve. Yet, these methods in the end seem more about learning robust compressible models and stray far from the original neural subspace idea, especially with compressible points. In my current understanding, these networks seem to have no demonstrated advantage to universally slimmable networks, which have a simpler validation process, runtime switching method, and more intuitive training procedure. The comparison against these networks and others needs to be better justified since I currently do not understand why fine-tuning is not allowed. If I misunderstood the method significantly, I would be willing to increase my score, but currently I suggest rejecting the paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper present a method for learning a compressible subspace of neural networks that contains a fine-grained spectrum of models that range from highly efficient to highly accurate. The proposed method allows choosing the proper point of trade-off between accuracy and efficiency at inference time, according to the available resources. There are also efforts to reduce the runtime tweaking overhead like replacing BatchNorm with GroupNorm.
main_review: Strengths: the paper is well motivated, as adaption to runtime available resource is important. 

Weaknesses:
* The method is a direct extension of the learning subspace method.
* There are important details missing from the paper. E.g., there is no info on how to generate a alpha via the state of hardware in runtime. This creates severe difficulty in understanding and reproducing the method.
* There is no material to measure hardware performance. For example, only the accuracy of the classification models are given, but memory bandwidth, latency or FPS are not available to quantitatively measure the advantage. 
* The paper is not properly peer-compared. For example, the work is not well compared with pruning and quantification methods.

Questions for the Author(s):
* please elaborate on the definition of the compression function f and the intuition behind?
* how to choose the hyper-parameter alpha in a hardware run-time?
* What will happen to the arch of a model, if pruning is also performed? 
* If using this method in a hardware, how to change the quantization meta-parameters(scale and zero-point) accordingly?

    
summary_of_the_review: The paper proposes a method that reasonably extends the learning subspace method to allow performing accuracy-efficiency tradeoff according to runtime available resource. The method has been evaluated on several classification tasks and find to be useful.

However, the paper does not clearly explain how the compression is performed, with important details like choice of alpha missing. The measurement of speedup is not that quantitative, lacking realworld test stats. It is very difficult to evaluate the contribution of this paper under these conditions.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes to balance the inference accuracy and efficiency by training a subspace of neural networks and then adapting the network within the subspace at inference time.
main_review: The novelty of this paper looks limited. It is an extension of recent work on learning network subspaces. Meanwhile, compared to existing works with adaptive networks, the advantages of this work are the finer-grained compress level and no need to recalibrate BN. The benefit of such improvements looks trivial.

Many vital details are missing in this paper. The detailed form of the compression level function $\gamma(\alpha)$ and the compression function $f(\omega, \gamma)$ are not provided. These are the core of the algorithm and the authors need to show them. 

Also, the authors do not explain what determines the dimension n of the stochastic function $\alpha$. If $\alpha$ controls the sparsity at the level of each weight, n will be extremely large, and the training overhead of the proposed algorithm is extremely large because they need to do n forward passes and backward passes for each batch. Even $\alpha$ controls the sparsity at the layer level, the training overhead will still be formidable. I cannot find much information about n in this paper. The authors need to provide more details about the dimension n of $\alpha$.
summary_of_the_review: Many important details are missing in this paper. For example, the finer-grained compression level is a major selling point of this paper, but the authors did not even provide the compression level function $\gamma(\alpha)$.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes to learn compressive subspaces which can adaptively compress the network during inference. It constructs either linear subspace or a single endpoint for compression. It replaces BN with GroupNorm to avoid re-calibrating during inference or after adjustment. Their method is evaluated in three different scenarios: structured sparsity, unstructured sparsity, and quantization.
main_review: The paper is well written and easy to follow. The ideas of constructing a linear subspace and using the function $f(w(\alpha), \gamma (\alpha))$ to perform compression during inference are novel. The analysis of BN parameters in adjustment provides quantitative analysis in this area. The experiments are exhaustive and can well support their ideas.

However, as the paper claims, they bias the subspace to contain high-accuracy solutions at one end and high-efficiency solutions at the other end. In my understanding, two endpoints are using the same network architecture. How to train a network to obtain $w_1$ and $w_2$ in this case?
summary_of_the_review: I think the paper is well written; the method is novel and interesting; the experiment can well support the claims. Just need to clarify some details
review_title: null

Meta-review: This paper proposed a method for adaptive network compression at inference time. However, the paper contains various issues raised by the reviewers that needs to be addressed.
Decision: Reject