Title: No title found

Abstract: No abstract found

:



Figures and Tables:

Peer review (iclr2020):
## Summary The paper proposes a method to compress CNNs inspired by the information bottleneck. More specifically, the authors propose to prune the less informative filters of the CNN according the an IB related objective. To be able to do that, the original IB objective is modified by substituting the MI regularization term (between input and compressed representation) by the total number of filters in the network. To be able to maximize this new NP hard objective, the authors proposed a greedy approach to select the filters to prune. This is done sequentially for each layer from high to low level representations. The experimental section shows a qualitative and quantitative analysis of the method, but not comparison to any compression method proposed in the literature at all. Unfortunately, the paper is very difficult to read. The presentation is unclear and not well-structured. Several details of the final algorithm are missing and the numerous typos and grammatical errors make the reading even harder. ## Comments The structure of the paper should be revisit. For example, the introduction is an extended related work, which is actually more complete than the actual related work section. There are several grammatical errors and typos making real hard to understand some central parts of the paper. I encourage the authors to do a major rewriting of the manuscript to be sure that contributions and the novelty of their paper can be fairly evaluated. Regarding the method, while the method that the authors propose is derived with the IB functional in mind, the final objective is only loosely couple to the IB functional. In the end, the objective maximizes the mutual information between the output and the input subject to an L_0 regularization in the number of filters of the CNN, which is an approach that does not seem particularly novel. Since the problem is NP hard, the authors propose a greedy approach. They further schedule the pruning by taking into consideration one layer at a time starting with the high level layers. This seems reasonable as it used general well-known insights about the structure of CNNs, but again does not seem particularly novel or technically difficult. Finally, I may have missed some important details since it was particularly difficult to understand the writing at the end of section 3. Specifically, at the end of page 3, the authors seem to suggest that selection is based on AUC and saliency is based on *relevance to the input and output variables*. However, there is no mention to the AUC in equation (4) and equation (5) comes out of the blue. In (6) they seem to suggest that they look at the AUC rather than then mutual information but I have to admit I did not understand this bit. In the experimental section, the authors explore the proposal but they do not provide a single comparison to the numerous compression methods that exist in the literature. This makes it extremely hard to evaluate the impact of the contribution. Moreover, several details about the implementation are missing, making it again difficult to understand what the final algorithm is. * Several grammatical and notation errors (not an exhaustive list): Introduction *compression methods that developed with iteratively* => Review grammar *They argue that both the optimal* => *They argue that the optimal* *the fitting phase and the compression phase, they also* => *the fitting phase and the compression phase. They also* *to architectural simplicity, whether* => *to architectural simplicity. Whether* *piecewise constant function of parameters* => *piecewise constant function of the parameters* *DNNs, these* => *DNNs. These* *ranked by informative scores preserved on the input and output labels* => Review grammar *backpropagately* (not sure it is used correctly in this context) *regards model compression* => *regarding model compression* Related work *On the one hand, in paper Shwartz-Ziv & Tisbhy (2017) they suggested* => *On one hand, Shwartz-Ziv & Tisbhy (2017) suggested* *to open the box of neural networks with information* => vague statement, not clear the meaning of this. *with information, whereas* => *with information. Whereas* *cannot be attributed to the use of IB functional but should be considered as an outcome of making DNNs stochastic* => *cannot be attributed to the use of the IB functional but the added stochasticity in the DNNs* *such as sufficient, maximally compressed, admitting a simple decision function and robust, which presented that the goal formulated* => *such as being sufficient, maximally compressed, admitting a simple decision function and robustness, claiming that the goal formulated* *we go deeper to prune those both highly informative* => *we go deeper by pruning those that are both highly informative* *To the end,* => *Finally,* Section 3 *to utilize the facility of transfer learning* => vague, imprecise *Suppose there are high dimensional filters* => *Suppose there is a set of high dimensional filters* C is used both for the dimensionality of the filters and the number of filters T_S and S are used interchangeable *in one intermediate convolutional layer that each characterizing some properties of the input, in* => *in one intermediate convolutional layer, each of them representing some properties of the input. In* *,as suggested* => *. As suggested* *discarding variable T_C* => *discarding the variable T_C* k is used both for the dimensionality of the filters and the number of filters (The end of this section is particularly difficult to understand and I could not follow the details)