Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper focuses on the problem of  neural network compression, and proposes a new scheme, the neural epitome search. It learns to find compact yet expressive epitomes for weight parameters of a specified network architecture. The learned weight tensors are independent of the architecture design.  It can be encapsulated as a drop in replacement to the current
convolutional operator. It can incur less performance drop. Experiments are conducted to show the effectiveness of the proposed method. However, there are some concerns to be addressed.
-It is not too clear how to learn the epitomes and transformation functions.
-Authors stated that the proposed method is independent of the architecture design. From the current statements, it is not explained clearly.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: In this work, the authors describe a technique for compressing neural networks by learning a so-called Epitome (E), and a transformation function (\theta) such that the weights for each layer can be constructed using \theta(E). The epitome and the transformation function can be learnt jointly while optimizing the network for the task specific loss. 

The main idea of this paper is really interesting, and the experimental results which compare against other recent techniques also validate the proposed technique. However, while I think the main idea is relatively clear, I personally found the description of the proposed techniques -- particularly 3.2 and 3.4 -- to be somewhat hard to follow, particularly given that some details only appear in the Appendix. I would suggest that the authors try to revise this section by trying to move Figures 4 and 5 from the appendix into the main text. Some additional suggestions also appear below.

Overall, I would while I like the ideas in this paper, based on the current presentation I am inclined to rate the paper as a “weak reject”, though I would raise my rating if the paper was revised to improve the presentation.

Main comments:
1. The authors mention that “During inference, (the) routing map enables the model to reuse computations when the expanded weight tensors are formed based on the same set of elements in the epitomes and therefore effectively reduces the computation cost.” It would be nice to include some results which indicate what the savings are with the proposed routing map.

2. Section 3.2: There were a few aspects of section 3.2 that I think could be improved for clarity.
A.) Personally, I found Figure 2 somewhat tricky to follow. I would suggest removing the 3x2 “Epitome” and “Generated Kernel” figures on the extreme right of the image, since I’m assuming the only goal of these is to indicate that “orange” and “blue” correspond to “Epitome” and “Kernel” respectively. I would also suggest mentioning the correspondence of the colors as the first sentence in the caption. Finally, if possible, I would suggest adding a small description alongside the (a), (b), (c) subcaptions: e.g., (a) straightforward but non-differentiable mapping, … , (c) Generated Weigh Kernel.

B.) I believe that the authors use a separate E for each layer, and that Epitomes are not shared across layers. I may have missed this in the text, but it would be useful to clarify this explicitly again in the section.

C.) The “parameterized transformation layer” is mentioned before Equation 3. I think it would be useful to mention that this is implemented using neural networks in your work for clarity. E.g.: “To handle the above two obstacles, ... three
parts: (1) a parameterized transformation learner η (implemented using a neural network in this work) used to learn a set of starting indices for patches in epitome (i.e. all elements in the same epitome patch share identical starting indices); … and an interpolation based generator (Eqn. 3).”
or
“To handle the above two obstacles, ... three
parts: (1) a parameterized transformation learner η (See Section 3.3) used to learn a set of starting indices for patches in epitome (i.e. all elements in the same epitome patch share identical starting indices); … and an interpolation based generator (Eqn. 3).”

D.) The exact structure of the “parameterized transformation layer” wasn’t exactly clear to me. In 3.3, the authors mention that it consists of “of two convolutional layer, followed by a sigmoid function … takes the feature map of the convolutional layer as input”. Please clarify exactly what is fed in as the input to this network e.g., (input feature map: F, and the indices i,j).

3. I personally also found Section 3.4 which discusses the Computation reduction was also somewhat hard to follow. Some clarification questions: Is the memory/computation cost of the storing/creating the routing map included in the compression calculations? I think it is important to factor these costs when computing the savings achieved by the model. Also, I was unclear on what R_{cin} and R_{cout} are, and why they appear in Equation 5. Could the authors please clarify. 

Minor Comments:
1. Abstract: “Traditional compression methods … all assume that network architectures
and parameters should be hardwired.” What does it mean for them to be “hardwired” in this context?
2. Abstract: “Experiments demonstrate that, … with 25% MAdd reduction and AutoML for Model Compression (AMC) by 2.5% with nearly the same compression ratio.” --> “Experiments demonstrate that, … with 25% MAdd reduction, and a 2.5% Madd reduction for AutoML for Model Compression (AMC) with nearly the same compression ratio.”

------- Update after Author Response --------
I would like to thank the authors for their responses and for the updates which strengthen the paper in my view. I have updated my score accordingly.

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: In this paper, the authors learn epitomes, which are small weight tensors which can be used with a learnt transform to produce tensors of an appropriate size (e.g. the sizes used in MobileNet v2).  This gives a reduction in the number of parameters required, and the number of MAdds in theory.

This paper is badly written, and could do with a rewrite:

- Citations are used incorrectly (\cite should be used when the citation is meant to be read as part of the sentence). 
- "less elements" --> "fewer elements"
- "misuse the notion" --> "abuse the notation"? 
- "for fair comparisons" --> "for a fair comparison"

The method is poorly explained; I have read Section 3.2 several times, and I'm still not entirely certain of what's going on. Figure 1 is helpful, but Figure 2 is not, and could be redesigned. 2(b) makes it look like you are going from a 3x3 epitome to a 2x2 kernel, which is clearly not what is happening. I think it would be helpful to give a detailed pictoral example of an epitome mapping to a weight tensor, with arrows between relevant indices changing. 

On initial reading, I thought the method allowed dynamic allocation of your epitomes to lots of different tensor sizes. From what I can tell, the network has to be trained from scratch for each possible size, so it isn't flexible in that respect.  From what I can gather, the paper is presenting an alternate approach to *downscale* networks, as opposed to say, reducing width or depth. The comparisons to different widths of MobileNet v2 make more sense under this scenario.

The main sell of the methods appears to be on the basis of MAdd reduction. This makes me nervous, as it doesn't necessarily correspond to actual speed-up or a reduction in energy (see https://arxiv.org/abs/1801.04326). EfficientNet was mainly about Madds too, but they provided some inference times. Would it be possible to add these? The method used with the integral image sounds expensive.

The results look good, but error bars,on the CIFAR experiments at the very least, would be appreciated.

Pros:
-------
- Good results
- Method appears largely novel (although bears some resemblance to https://arxiv.org/abs/1906.04309)

Cons:
--------
- Badly written
- The method is poorly explained
- Uncertainty re: MAdds as a primary comparator

I propose a weak reject for this paper for two primary reasons:
1) The standard of writing, and the explanation of the all-important method are not up to scratch for a top tier conference
2) I have concerns regarding the MAdd calculations. Perhaps you could provide some pseudo-code in the author response?

I am happy to upgrade my score if the authors deal with these issues sufficiently.
review_title: Official Blind Review #1

Meta-review: The paper proposed a novel way to compress arbitrary networks by learning epitiomes and corresponding transformations of them to reconstruct the original weight tensors. The idea is very interesting and the paper presented good experimental validations of the proposed method on state-of-the-art models and showed good MAdd reduction. The authors also put a lot of efforts addressing the concerns of all the reviewers by improving the presentation of the paper, which although can still be further improved, and adding more explanations and validations on the proposed method. Although there's still concerns on whether the reduction of MAdd really transforms to computation reduction, all the reviewers agreed the paper is interesting and useful and further development of such work would be useful too. 
Decision: Accept (Poster)