Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper introduces an end-to-end method for the neural network compression, which is based on compressing reparametrized forms of model parameters (weights and biases). Shannon-style entropy coder is applied to the reparametrizations.

Reparameterized forms are tensors stored in the compressed format. During inference, they are uncompressed into integer tensors and transformed via parameter decoders into weights/biases of convolutional and dense layers.

During training, model parameters are manually partitioned into groups, and within a group they are considered as samples from the same learned distribution. Similarly, parameter sharing is introduced among the corresponding parameter decoders.
Loss function, which is minimized during training is a sum of rate loss (self-information of all reparametrizations) and cross-entropy classification loss under reparametrization. A trade-off between compressed model size and model accuracy can be explored by varying a constant before the rate loss.

During optimization, several tricks are applied. “Straight-through” gradient estimator is used to optimize the loss function over discrete-valued reparametrizations by means of stochastic gradient descent. Relaxation is used to obtain good estimates for both the rate term and its gradient during training.

The proposed idea is well-founded and intuitive. The proposed method is extensively evaluated on different classification network architectures and datasets. It provides good compression while retaining a significant portion of accuracy.

In the experiments, It'd be interesting to see a comparison on more efficient networks like MobileNets, ShuffleNets on ImageNet dataset. Also, I wonder whether under the same compression rate the proposed method outperforms DeepCompression (Han et al., 2015) in terms of accuracy? (for example, for LeNets and VGG-16)

review_title: Official Blind Review #4

final_score: rating: 8: Accept
review_content: The paper presents a compression algorithm for neural networks. It uses a linear projection to map the weights and biases to a latent space where they are quantized using a learnt coding distribution. The method is simple yet it shows strong performance of a variety of compression benchmarks.

Originality: The core idea is related to (Balle et al. 2018), but the paper puts its own twist to it with the projections and applies it to model compression. It is certainly an interesting direction to explore.

Presentation: The paper is well written. It is easy to read and understand. It properly cites prior works and contains all the technical details. I appreciate that the authors fit the paper into the recommended 8 pages.

Impact: The method shows very strong performance in terms of compression ratio, but its unclear whether the compressed model can be used to speed up inference. Currently the main use case would be saving storage/bandwidth.

Questions:
- In section 2.2, the paper talks about the form of the coding distribution. It has d components and l dimensions. How is d determined?
- Section 2.1, How are the weights grouped for coding? is every filter its own group? If the experiments use different groups for different models, how is it decided which model uses which approach?
- \phi was fixed for the CIFAR10/ImageNet experiments. Could you provide further insight into why this choice was made?

Assessment:
While the idea is not groundbreaking, it is very well presented and evaluated and shows strong performance.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The authors propose a novel, information theoretic approach to learning compressed neural networks, whereby model parameters $\theta$ are expressed in an (approximately discrete) latent space as $\theta = f_{\psi}(\phi)$. During training, "reparameterizations" $\Phi$ are treated as draws from a generative distribution $q(\phi)$. As per the source coding theorem, penalizing the entropy of $q$ (more accurately, of $\hat{q}$) directly promotes compressible choices of $\Phi$ and, therefore, of $\Theta$. After training, arithmetic coding of $q$ is used to compress $\Phi$. Empirical results on a suite of common benchmarks demonstrates the proposed algorithm gracefully trades off between model performance and size.


Frequency Domain Interpretation:
  Here, I am operating off of the assumptions that, in Sect 3.2 paragraph 3, $\phi_{w}$ should be $\psi_{w}$. If so, can "Our Method (DFT)" also be interpreted as frequency domain SQ (of convolutional filters)? When trained on natural images, CNN filters are typically "smooth" (see any visualization thereof) and this smoothness translates as a prior of sorts on $\Phi_{w}$. This insight was previously explored in [1, 2] for purposes of compressing CNNs. Does evidence of this manifest in your experiments? For example, do the empirical distributions of high- and low-frequency components of $\phi_{w}$ differ?


Feedback:
  The authors propose a general framework where in (seemingly) arbitrary decoders $f$ can be used for purposes of "reparameterization". It is therefore somewhat disappointing that, in practice, $f$ is restricted to affine transformations and, in some cases, even fixed. Memory issues notwithstanding, it is unclear how well complex decoders $f$ could be jointly learned with (surrogate) encodings $\tilde{\Phi}$ and density models $\tilde{q}$.

  The lack of comparison between the proposed method's test-time throughput and that of its baselines leaves open questions regarding potential real-world use cases. Much of the time, methods for compressing neural networks divide their attention between both memory footprint and prediction speed. Outright ignoring this aspect of the problem seems odd. As a reviewer, I would much rather see the authors take a proactive approach in addressing this issue.


Questions:
  - What do results look like if convolutional decoders $f_{\text{conv}}$ are jointly learned during training?
  - How sensitive is the algorithm to different groups assignments? Can assignments be reliably made by simply looking at the architecture?


Minor Comments:
  - The general notation of the paper is cumbersome at times, can $\phi$ to be changed to, e.g., $\tilde{\theta}$? Similarly, can notation such as $\theta_{k, W}$ be simplified as $W^{k}$?

  - Section 2 jumps around a bit. Consider trying to order things "chronologically" such that training comes before compression? By the same token, details regarding "Model Partitioning" could be move to the end of the section such that the preceding material just refers to $\theta$ more abstractly?

  - Some additional details regarding training (esp. discretization and model distributions $\tilde{q}$) would be appreciated.


[1] "Compressing convolutional neural networks in the frequency domain", Chen et al. 2016
[2] "CNNpack: Packing convolutional neural networks in the frequency domain", Wang et al. 2016
review_title: Official Blind Review #2

Meta-review: The paper describes a simple method for neural network compression by applying Shannon-type encoding. This is a fresh and nice idea, as noted by reviewers. A disadvantage is that the architectures on ImageNet are not the most efficient ones. Also, the review misses several important works on low-rank factorization of weights for the compression (Lebedev et. al, Novikov et. al).   But overall, a good paper.
Decision: Accept (Poster)