Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes an interesting approach to compress the weights of a network for storage or transmission purposes. My understanding is, at inference, the network is 'recovered' therefore there is no difference in processing time (slight differences in accuracy due to the approximation in recovering the weights).

- The idea is nice although it's applicability is limited as it is only for distribution of the model and storing (is storage really a problem?). 

Method:
- the idea of using the Bloomier filter is new to me. However, the paper is miss-leading as the filtering is a minor part of the complete process. The paper introduces a complete pipeline including quantization, and pruning to maximize the benefits of the filter and an additional (optional) step to achieve further compression. 

- The method / idea seems simply and easy to reproduce (except the subsequent steps that are not clearly detailed).

Clarity

- The paper could improve its clarity. At the moment, the Bloomier is the core but needs many other components to make it effective. Those components are not detailed to the level of being reproducible.
- One interesting point is the self-implementation of the Deep compression algorithm. The paper claims this is a competitive representation as it achieves better compression than the original one. However, those numbers are not clear in tables (only in table 3 numbers seem to be equivalent to the ones in the text). This needs clarification, CSR achieves 81.8% according to Table 2 and 119 according to the text.

Results:
- Current results are interesting. However I have several concerns:
1) it is not clear to me why assuming similar performance. While Bloomier is weightless the complete process involves many retraining steps involving performance loss. Analysis on this would be nice to see (I doubt it ends exactly at the same number). Section 3 explicitly suggest there is the need of retraining to mitigate the effect of false positives which is then increased with pruning and quantization. Therefore, would be nice to see the impact in accuracy (even it is not the main focus of the work). 

2) Resutls are focused on fully connected layers which carry (for the given models) the larger number of weights (and therefore it is easy to get large compression numbers). What would happen in newer models where the fully connected layer is minimal compared to conv. layers? What about the accuracy impact there? Let's say in a Resnet-34.
3) I would like to see further analysis on why Bloomier filter encoding improves accuracy (or is a typo and meant to be error?) by 2%. This is a large improvement without training from scractch.
4) It is interesting to me how the retraining process is 'hidden' all over the paper. At the beginning it is claimed that it takes about one hour for VGG-16 to compute the Bloomier filters. Howerver, that is only a minimal portion of the entire pipeline. Later in the experimental section it is mentioned that 'tens of epochs' are needed for retraining (assuming to compensate for errors) after retraining for compensating l1 pruning?.... tens of epochs is a significant portion of the entire training process assuming VGG is trained for 90epochs max.

5) Interestingly, as mentioned in the paper, this is 'static compression'. That is, the model needs to be completely 'restored' before inference. This is miss-leading as an embedded device will need the same requirements as any other at inferece time(or maybe I am missing something). That is, the benefit is mainly for storing and transmission.

6) I would like to see the sensibility analysis with respect to t and the number of clusters. 

7) As mentioned before, LeNet is great but would be nice to see more complicated models (even resnet on CIFAR). These models are not only large in terms of parameters but also quite sensitive to modifications in the weight structure.

8) Results are focused on a single layer. What happens if all the layers are considered at the same time? Here I am also concerned about the retraining process (fixing one layer and retraining the deeper ones). How is this done using only fully connected layers? What is the impact of doing it all over the network (let's say VGG-16 from the first convolutional layer to the very last).

Summary:

All in all, the idea has potential but there are many missing details. I would like to see clearer and more comprehensive results in terms of modern models and in the complete model, not only in the FC layer, including accuracy impact.  
review_title: Interesting idea, limited applicability (storing and transmission of models), very limited results and missing details. I would like to see clearer and more comprehensive results in terms of modern models and in the complete model, not only in the FC layer, including accuracy impact.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The problem of lossy compression of neural networks is essentially important and relevant. The paper proposes an interesting usage of Bloomier filters in lossy compression of neural net weights. The Bloomier filter is proposed by others. It is a data structure that maps from sparse indices to their corresponding values with chances that returns incorrect values for non-existing indices. The paper compares its method with two baseline methods (Magnitude and Dynamic network surgery DNS) to demonstrates its performance.

I find the paper fairly interesting but still have some concerns in the technical part and experiments.

Pros:
1. The paper seems the first to introduce Bloomier filter into the network compression problem. I think its contribution is novel and original. The paper may interest those who work in the network compression domain.
2. The method works well in the demonstrated experimental cases.

Cons:
1. The technical part is partially clear. It might be worthwhile to briefly describe the encoding/construction algorithm used in the paper. It is recommended to describe a bit more details about how such encoding/decoding methods are applied in reducing neural net weights.
2. One drawback of the proposed method is that it has to work with sparse weights. That requires the method to be used together with network pruning methods, which seems limiting its applicability. I believe the paper can be further improved by including a study of the compression results without a pruning method (e.g., comparing with Huffman in table 3). 
3. What is the reason there is no DNS results reported for VGG-16? Is it because the network is deeper?
4. The experimental part can be improved by reporting the compression results for the whole network instead of a single layer.
5. It seems the construction of Bloomier filter is costly and the proposed method has to construct Bloomier filters for all layers. What is the total time cost in terms of encoding and decoding those networks (LeNet and VGG)? It would be nice to have a separate comparison on the time consumption of  different methods.
6. Figure 4 seems a bit misleading. The comparison should be conducted on the same accuracy level instead of the ratio of nonzero weights. I recommend producing another new figure of doing such comparison.
7. The proposed idea seems somewhat related to using low rank factorization of weight matrices for compression. It might be worthwhile to compare the two approaches in experiments.
8. I am specifically interested in discussions about the possibility of encoding the whole network instead of layer-by-layer retraining.
review_title: Lacking evaluations with few technical concerns

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: The paper addresses the actual problem of compression of deep neural networks. Authors propose to use another technique for sparse matrix storage. Namely, authors propose to use Bloomier filter for more efficient storage of sparse matrices obtained from Dynamic Network Surgery (DNS) method. Moreover, authors propose elegant and efficient trick for mitigating errors of Bloomier filter. Overall, the paper present clear and completed research of the proposed technique.
Clarity and Quality: The paper is well structured and easy to follow. Authors provide a reader with a large amount of technical details, which helps to reproduce their method. The paper contains detailed investigation of every step and aspect in the proposed pipeline, which made this research well-organized and complete.
Though the presentation of some results can be improved. Namely, core values for compression and improvement are presented only for two biggest layers in networks, but more important values are compression and improvement for whole networks.
Originality and Significance: The main contribution of the paper is the adaptation of Bloomier filter for sparse network obtained from almost any procedure of networks sparsification. However, this adaptation is almost straightforward, except the proposed trick of network fine-tuning for compensating false positive values of Bloomier filter. Significance of the results is hard to estimate because of several reasons:
Values of compression and improvement are presented only for two layers, not for the whole network.
According to Fig. 4, encoding of sparse matrices via Bloomier filter is efficient (compared to CSR) only for matrices with nonzero ratio greater than 0.04. So this method can’t be applied to all layers in network, that can significantly influence overall compression.

Other Comments:
The procedure of network evaluation is totally omitted in the paper. So a model supposed to be “unpacked” (to the dense or CSR format) before evaluation. Considering this, comparison with CSR could be made only for sending the model over a network. Since, CSR format can be efficiently used during evaluation.
Minor Comments:
(Page 5) “Because we encode $k$ clusters, $t$ must be greater than $\lceil \log_2 k\rceil$”. Perhaps, “...$r$ must be greater than $\lceil \log_2 k\rceil$” would be better for understanding.
(Page 7) “Bloomier filter encoding increased the top-1 accuracy by 2.0 percentage points”. Perhaps, authors have meant top-1 error.
review_title: ICLR 2018 review. Weightless

Meta-review: Pros:
-- Use of Bloomier filters for lossy compression of nets is novel and well motivated, with interesting compression performance.
Cons:
-- Does lossy compression for transmission, doesn’t address FLOPS required for runtime execution. A lot of times, client devices do not have enough cpu to run large networks (title should be udpated to mean compression and transmission)
-- Missing results for full network, larger deeper network.

Overall, the content is novel and interesting, so I would encourage the authors to submit to the workshop track.

Decision: Invite to Workshop Track