Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper aims to improve the runtime of NN inference via a novel DNN compression idea. Specifically, the authors propose an approach that allows fast queries at inference directly on the compressed model without decompressing the model back. They show comparisons to prior work on AlexNet andVGG16 architectures.
main_review: The idea of enabling inference directly on the compressed model without decompression is quite interesting. However, I could not verify if the proposed method indeed shows any improvement in runtime speedup or compression rate, or accuracy by looking at the experimental results. It is also not very straightforward to me that there must be a speedup without any empirical support because all the weights need to be queried for inference and I do not see how this is different from decompression. Here are my main concerns and questions:  

- The paper is not very easy to follow and needs significant improvement in writing. There are repeating sentences, many typos, and the general flow is not very fluent. I especially recommend making Section 3 more clear since it is the most important section of the paper and some concepts need to be better explained by providing some background information.

- I believe the related work section misses many references, especially in the model encoding part, which is the most related direction to this paper. For example, [1], [2], [3], and [4] would be good candidates for the related work section since they also propose new methods to encode neural network models. 

- The authors compare their work with Huffman coding in Deep Compression paper [5]. However, there are more recent methods such as [1], [2], [3], and [4] that are directly comparable. A comparison with these papers would strengthen the paper.

- The authors claim that their method is near-optimal but they do not provide any reason as to why this is the case. In Section 4.1, they mention that the space consumption is $n \log(\sigma) + O(n)$ bits and claim that this is near-optimal to the information-theoretic lower bound. A $n \log(\sigma)$ space consumption is actually an upper bound, which is exact only when the weights are uniformly distributed. There are several empirical results that show that weights are not uniformly distributed, e.g., [2] shows that trained weights look Laplacian. Therefore, the proposed method has a space consumption close to the worst-case estimate of the entropy, which is not close to the optimal model compression since we know that weights are not uniform. 

- $\sigma$ is used for both the alphabet and the alphabet size in different places. It would be nice to fix this to avoid confusion.

- There are some missing citations. For example Huffman Coding(?) in page 5. 

- Maybe most importantly, the authors claim in Section 6.1 that the proposed method has the best compression rate of $19.8$% for AlexNet and $18.72$% for VGG-16.  However, by looking at Tables 1 and 2, the only baseline (Huffman Coding) has a better compression rate of $14.65$% and $11.94$%, respectively. It means that the text and the results in the tables are not consistent, and the proposed method underperforms Huffman Coding. 

- I find the following sentence from the paper very confusing: "For Pruned-Quantized AlexNet/VGG-16, our method
achieves the compression rate with only $1 \%$ difference, compared with the optimal solution.". Can the authors explain what the optimal solution here is? 


[1] Wiedemann, Simon, et al. "Deepcabac: Context-adaptive binary arithmetic coding for deep neural network compression." arXiv preprint arXiv:1905.08318 (2019).

[2] **v1 of** Isik, Berivan, Albert No, and Tsachy Weissman. "Rate-Distortion Theoretic Model Compression: Successive Refinement for Pruning." arXiv preprint arXiv:2102.08329 (2021).

[3] Havasi, Marton, Robert Peharz, and José Miguel Hernández-Lobato. "Minimal random code learning: Getting bits back from compressed model parameters." arXiv preprint arXiv:1810.00440 (2018).

[4] Oktay, Deniz, et al. "Scalable model compression by entropy penalized reparameterization." arXiv preprint arXiv:1906.06624 (2019).

[5] Han, Song, Huizi Mao, and William J. Dally. "Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding." arXiv preprint arXiv:1510.00149 (2015).
summary_of_the_review: Overall, I think the paper is not ready for publication due to the reasons I listed in the previous section.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper introduces a method to compress DNN networks with succinct data structures for efficient inference. The paper claims that DNN inference suffers from a storage bottleneck and proposes a lossless compression scheme using wavelet trees to reduce the size of DNN models. The paper performs an evaluation of the proposed method on AlexNet/VGG-16 and shows that the method outperforms existing methods for compressing models using Huffman Coding. 
main_review: Strengths:

- The paper explores an interesting idea to perform lossless compression for efficient DNN inference.
- The proposed method seems to be able to achieve a better compression rate than Huffman Coding, a compression technique used in DeepCompression from Han et al. 2015.

Weaknesses:
- The missing discussion of related work makes the significance of the contribution questionable. 
- The comparison with existing work is weak.
- The paper lacks details on how speedups are measured, which is important for model inferencing. 
- Writing needs to be improved. 

summary_of_the_review: The paper explores an interesting idea for compressing DNN networks, e.g., through lossless compression. This could be potentially quite impactable but the current state of the paper raises several concerns:

1. There have been significant missing discussions related to model compression. In particular, if we look that the related work section, it is very weird that most of the cited papers in Section 2 are work done prior to 2017, whereas there have been vast advancements for model compression in the past several years that the paper seems to choose to ignore, such as [1-3]. Of course, many existing compression techniques are not lossless in theory, but they often achieve comparable performance as the original model, e.g., quantization, that cannot be dismissed completely. 

2. The evaluation section is weak. The comparison is made primarily with the Huffman coding method used in Deep Compression [4], which was done six years ago and no longer represents the state-of-the-art. Furthermore, the evaluation was also done on AlexNet and VGG, both of which have been heavily studied. To be more convincing, the paper should compare at least with some lossy compression techniques such as integer quantization and one more advanced model architecture such as ResNet and Transformers. 

3. The paper lacks descriptions of implementations and how speedups are calculated. Is the speedup measured through proxies or measured execution time? Is Succinct Data structure GPU hardware friendly? How does it perform on GPU in comparison with models served via cuDNN?  What are the implementations used for the baseline? 

4. The paper would benefit from thorough proofreading.

Typos:
1. Missing references, e.g. Huffman Coding (?), in the first paragraph of Section 6. 

[1] Dong et al. HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precision, https://arxiv.org/abs/1905.03696, 2019

[2] Choudhary et al. A comprehensive survey on model compression and acceleration, 2020

[3] Deng et al. A comprehensive survey on model compression and acceleration, 2020

[4] Cho et al. On the Efficacy of Knowledge Distillation, 2019

[5] Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding, Han et al., 2015, https://arxiv.org/abs/1510.00149
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: It is presented an approach to store a compressed deep neural network in a form that allows fast inference. The deep neural network is defined as an element- and block-wise Runtime Accessible Sequence (RAS). Moreover, the RAS elementary operands are defined as tuples to account for parameter quantization and pruning. Given the neural network RAS formulation, succinct data structures, i.e. wavelet tree, are used to store the network during inference.  The approach is evaluated on AlexNet and VGG-16 architectures which are quantized and pruned. The results are promising compared to the Huffman encoding. 
main_review: Paper strengths:

+ The idea of proposing a different data structure to store a neural network is novel for accelerating inference.

+ The paper shows good performance in the experimental part compared to the Huffman encoding.


Paper weaknesses: 

- Claim on decompress compressed model requirement: There is a plethora of prior work that does not require the model to be decompressed during inference. For instance, distillation approaches, as well as factorization and pruning methods. The paper is motivated by a claim, which does not hold based on the prior work. This is also reflected in the experimental section which prior works from the aforementioned categories are completely ignored in the comparisons with the prior work. The claims of the work are a major limitation.

- The proposed data structure for the neural network is method agnostic. It would help the paper to perform experiments based on several quantizations and pruning approaches. This experiment would add a significant value to the work.

- The main contribution is how to formulate the compressed neural network (in terms of data structure) to accelerate the inference. This idea is not quite clear after reading the paper. Overall, the writing part needs some improvement to improve the clarity of the work.

- Related work on lossless compression: It is important to extensively discuss the related work on lossless compression and memory-efficient deployment. A few useful references:
	- Lossless Compression of Structured Convolutional Models via Lifting (2021).
	- Lossless Compression of Deep Neural Networks (2020).
	- Universal deep neural network compression (2020).
	- Hardware-Based Real-Time Deep Neural Network Lossless Weights Compression (2020)
	- Compact and Computationally Efficient Representation of Deep Neural Networks (2019).

- Prior work comparisons: There is only one comparison with the Huffman encoding. It would be helpful to include more approaches (see above). 

- It would be helpful to include a ResNet-like architecture since it is the standard to be used nowadays. 


Improvements:

- The claims in the introduction could be supported by references.

- "As revealed by prior works, the impacts of these compression schemes.." references are missing.

- The quality of the figures, e.g. Fig 1/2/3, is poor. Rendering on a better resolution is recommended. 
summary_of_the_review: Overall, the paper presents an interesting idea. It is relatively easy to follow it, but the clarity of the paper needs improvement. The experimental part needs additional work, as already discussed. There are major claims which might need to be reformulated. There are also missing references at different points. Overall, the work is not ready for publication yet.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper explores the compression of weights in a DNN. The compression technique presented permits the uncompressed values to be easily recovered without first decompressing the data. The approach appears to take advantage of Wavelet Trees after the data has been structured in some way. 

Results are presented showing compression rates (vs. Huffman encoding) and speedups on a CPU. 
main_review: The paper discusses an interesting and important problem and proposes what appears to be a novel approach. There are two main issues for me at the moment 1) it is difficult to understand the precise scheme that has been proposed, and 2) the evaluation seems very limited and somewhat unclear.

1. To confirm, pruning and quantization was performed but only to the point that the model accuracy did not change? 
2. What were the results of quantization? 
3. When pruning/quantization is applied, wouldn't it be clearer to use 15.64MB as the new baseline and report CR compared to this. Claiming "only 1% difference" hides the fact that the HF model size = 6.8 and with WT-B it is 9.063. Obviously, the overall CR is low if you compare to the original model size of 232MB.
4. Again, results for the performance of the quantized and quantized/pruned models would be of interest, but I don't think are presented? (i.e. without your compression scheme). There are many simple data formats suitable for stored sparse data of course (e.g. using sparsity maps or Z-RLC, CSC/CSR perhaps). Would these be faster but use slightly larger models?
5. You don't compare to other simple compression schemes, e.g. Dynamic Prediction Reduction (DPRed) perhaps? 
6. Are there variations of the Wavelet-tree scheme that could have been evaluated?
7. I don't think you can say Huffman encoding is strictly optimal? 
8. Speedups are achieved through the reduction of memory traffic I assume, what is the overhead of accessing these compressed formats? e.g. if applied to a small model that fitted in the cache? I assume the architecture of the target processor has a big impact here too and the overhead will perhaps be smallest on the desktop style processor you target. How would the results differ on embedded cores for instance?

Low-rank approximation would be another technique to discuss or compare to perhaps.

I found some parts of the paper to be difficult to follow unfortunately, e.g. page 3. I suspect this is quite simple, but I struggled. Perhaps better diagrams or an example would help.

The bitmap Figures are also very difficult to see (e.g. Figure 1 and Figure 3 - the y-axis could perhaps cover a smaller range, avoding bitmaps would help too.

I understand the technique is lossless, but it would perhaps still be useful to confirm the model accuracy. 

minor:
* "aimming"
* missing reference for Huffman "(?)"

summary_of_the_review: I feel that the description of the work needs to be improved. The evaluation also needs improvement. For these reasons I am recommending the paper is rejected.
review_title: null

Meta-review: ### Summary

The paper proposes a technique that enables inference directly on a compressed model without decompressing the model.

### Discussion

- Strengths
  - An important problem as well as a compelling direction, namely inference without decompression.
 
- Weaknesses: 
  - The reviewers provided a number of both broad and specific criticisms of the work. 
  The most salient point is the lack of comparison to modern baselines.  Notably, the primary comparison is to a 2015 technique that, while seminal, has since been followed by significant related work (e.g, that identified by Reviewer eHWE, R8Un, and G6tm). In concert, the evaluation should consider at least one more contemporary network in the domain, such as a ResNet.

### Recommendation

I recommend Reject.  At current, this work is the first step in a strong, compelling direction. However, the work needs to be contextualized within a more modern context of contemporary results
Decision: Reject