Venue: ICLR 2017 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper describes an extension of the HasheNets work, with several novel twists. Instead of using a single hash function, the proposed HFH approach uses multiple hash function to associate each "virtual" (to-be-synthesized) weight location to several components of an underlying parameter vector (shared across all layers). These components are then passed through a small MLP to synthesize the final weight.

This is an interesting and novel idea, and the experiments demonstrate that it improves substantially over HashedNets. However, HashedNets is not a particularly compelling technique for neural network model compression, especially when compared with more recent work on pruning- and quantization-based approaches. The experiments in this paper demonstrate that the proposed approach yields worse accuracy at the same compression ratios as pruning-based approaches, while providing no runtime speedup benefits. While the authors mention the technique is only 20% slower (which I am pleasantly surprised by), I don't understand why this technique should ever be used over competing approaches for the kinds of networks the authors present experimental results on. The authors suggest that the technique could be combined with pruning based approaches... this may be true, but no experiments to this effect are provided. The paper also suggests that ease of setting the compression ratio is a benefit of HFH, but I don't think that's a sufficient win to justify the numerous other downsides (in accuracy and speed). 

In response to a question, the authors point out that the technique works very well for compressing embeddings, and for this setting the technique does appear like a genuinely useful contribution, given the marginal overhead and substantial train-time benefits. If the paper focused on this setting and showed experimental results on e.g. language modeling tasks or other scenarios with high-dimensional sparse/one-hot inputs require large embedding layers, I could enthusiastically recommend acceptance. However for the CNN and MLP networks which are the main focus of the experiments, I don't think the technique is suitable, as much as I like the basic idea.
review_title: Neat idea applied to the wrong settings.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper presents a method to reduce the memory footprint of a neural network at some increase in the computation cost. This paper is a generalization of HashedNets by Chen et al. (ICML'15) where parameters of a neural network are mapped into smaller memory arrays using some hash functions with possible collisions. Instead of training the original parameters, given a hash function, the elements of the compressed memory arrays are trained using back-propagation. In this paper, some new tricks are proposed including: (1) the compression space is shared among the layers of the neural network (2) multiple hash functions are used to reduce the effects of collisions (3) a small network is used to combine the elements retrieved from multiple hash tables into a single parameter. Fig 1 of the paper describes the gist of the approach vs. HashedNets.

On the positive side,
+ The proposed ideas are novel and seem useful.
+ Some theoretical justification is presented to describe why using multiple hash functions is a good idea.
+ All of the experiments suggest that the proposed MFH approach outperforms HashedNets.
On the negative side,
- The computation cost seems worse than HashedNets and is not discussed.
- Immediate practical implication of the paper is not clear given that alternative pruning strategies perform better and should be faster at inference.

That said, I believe this paper benefits the deep learning community as it sheds light into ways to share parameters across layers of a neural network potentially leading to more interesting follow-ups. I recommend accept, while asking the authors to address the comments below.

More comments:
- Please discuss the computation cost for both HashedNets and MFH for both fully connected and convolutional layers.
- Are the experiments only run once for each configuration? Please run multiple times and report average / standard error.
- For completeness, please add U1 results to Table 1.
- In Table 1, U4-G3 is listed twice with two different numbers.
- Some sentences are not grammatically correct. Please improve the writing.

review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposed a very complex compression and reconstruction method (with additional parameters) for reducing the memory footprint of deep networks.

The authors show that this complex proposal is better than simple hashed net proposal. One question: Are you also counting the extra parameters for reconstruction network for the memory comparison? Otherwise, the experiments are unfair.  

Since hashing and reconstruction cost will dominate the feed-forward and back-propagation updates, it is imperative to compare the two methods on running time. For hashed net, this is quite simple, yet it created an additional bottleneck.  Please also show the impact on running time. Small improvements for a big loss in computational cost may not be acceptable. I am not convinced that this method will be lightweight. If we are allowed complicated compression and reconstruction then we can use any off-shelf methods, but the cost will be huge


review_title: Hashing and Reconstruction Cost will make networks slow

final_score: rating: null
review_content: 1. Can you provide any concrete numbers as to how much slower this approach is during inference? It seems like it would be quite substantial, since the weight values must be computed on the fly on each pass for memory savings to be realized.
2. There are many other techniques in the literature without such speed tradeoffs, including several mentioned in the paper such as the pruning work of Han et al. HFH seems to provide worse accuracy at the same compression ratio than these methods, while also having significant performance implications. I think the fact that HFH improves on HashedNets is an interesting finding, but are there circumstances where using it would be a good idea in practice?
review_title: Questions

Meta-review: This paper presents some interesting and potentially useful ideas, but multiple reviewers point out that the main appeal of the paper's contributions would be in potential follow-up work and that the paper as-is does not present a compelling use case for the novel ideas. For that reason, the recommendation is to reject the paper. I would encourage the authors to reframe and improve the paper and extend it to cover the more interesting possible empirical investigations brought up in the discussion. Unfortunately, the paper is not sufficiently ground breaking to be a good fit for the workshop track.
Decision: Reject