Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper presents a method to disentangle intermediate features between two different deep neural networks. More specifically, given two networks, the proposed approach aims to find consistent and inconsistent feature components for a certain layer in each network. If one network is more powerful to the other (e.g., ResNet and AlexNet), the method can figure out which components are weak or strong (i.e., helpful to the performance) for the given task. The authors design a simple yet effective algorithm for extracting knowledge consistency. In addition, they provide a variety of practical experiments including network diagnosis, feature refinement, and network compression. Most of the experimental results support that the proposed method can practically extract consistent feature components with promising improvements.

Main concerns:

1. In section 4.1, the authors provide both the unreliable features and blind spots with their visualization (Figure 3). It is expected that blind spots are more informative than the unreliable features since they are part of a more powerful DNN. However, in Figure 3, the Blind spots do not seem to be important features. This makes the lack of the motivation of visualizing the effectiveness of features. Could you provide more promising visualizing results?

2. In equation (1), it seems that the scaling factor p^(k+1) can be unnecessary because ReLU is scale-invariant and \Sigma_(k+1) can scale the feature h^(k+1) with p_(k+1). Does p_(k+1) significantly improve the performance or negligible?

3. In Table 3, using x^{(0)} to the refined feature shows the best result under VGG-16. Are the result in Table 4 improved when changing the refined features? E.g., only x^{(0)} or x^{(0)} + x^{(1)}?

In short, the proposed disentangle method is simple, novel and very useful in many practical applications as provided in the experimental section. But, some experiments and intuitions of network construction are not fully promising. Thus, I vote for weak acceptance.

Minor concerns:

1. Does the proposed method show meaningful results when the tasks are different but the datasets are the same? Or the same task but different datasets.

2. The disentanglement network can be constructed with other operations, e.g., h^{(k)} = W^{(k)} concat(x , x^{higher}). Did the author try to design the network with other operations?

3. In the fourth paragraph on the first page, the reference of “Wolchover (2017)” need to change “(Wolchover, 2017)”.

4. The proposed method can be applied in other tasks, not with image datasets. Is the algorithm promising for other tasks?
review_title: Official Blind Review #4

final_score: rating: 8: Accept
review_content: The submission proposes a method for extracting "knowledge consistency" in neural networks and using that toward analyzing different aspects of them, eg understanding the representations, explaining knowledge distillation, and analyzing network compression. What's defined as consistent knowledge is essentially the stable parts of representations of different networks trained for the same task (stable-->consistent). I found the submission insightful, well executed, and backed up by many experimental results. 

Strengths: 
A) the proposed concept, extracting the *consistency* among representation of different networks, is interested and using that towards understanding what's going on under-the-hood of neural networks makes sense. I'm not aware of a similar existing method and didn't see one among the citations, so my current assumption is that this proposal is pretty novel. 

B) the extracted representational consistency quantity appears to be meaningful and strong, as authors were able to use it toward explaining several existing phenomena (e.g. knowledge distillation, network compression, etc). 

C) Authors perform extensive experimental studies on various aspects of neural networks in relation to the proposed representational consistency quantity. I applaud the efforts made by authors.  

Improvements/questions:
D) The submission uses a few different phrases in close connection or interchangeably, eg "fuzziness", "order", "linear/nonlinear transformation", "easy/hard to guess/estimation" (see the last 2 paragraphs of page 2). While I understand what the authors are trying to convey, those concepts are not in principle necessarily the same, so some clarification/unification would make the presentation more solid. For instance: nonlinear<--> higher order<-->fuzzy<-->hard to guess. Also guess<-->estimate. 

E) Sec 3 is the most important part of the paper and the technical meat. However, I found it harder to follow compared to the rest of the paper. It probably deserves more than 1 page. Authors could consider smoothing the presentation and adding details even at the cost of slightly extending the length. 

F) I think section 4.2 could walk the reader through more details to make sure the basics are understood, as this is the first time results of the method are being presented. E.g. fig 3 could be analyzed further and the color maps should be defined. 

G) I found the post hoc explanation in the last paragraph of page 6 somewhat dubious. 

H) In the feature refinement experiment (paragraph 2 of sec 4.3), is the process done in a recursive manner (first layer 1, then layer 2, and so on)? Or as a one time process? The former seems stronger. 

I) The consistency quantity is based on comparing different networks trained for the *same task*. Have you considered doing the same among different networks trained for *different tasks*? Would that say something about similarities among tasks and multi-task learning in a fashion similar to the analysis of Taskonomy 2018? 

J) I did not understand "and Beyond" in the title. I'd consider a more directly descriptive title. 

K) Seems like authors define "knowledge" and "visual concept" to be the invariant part of a feature (see paragraph 3 of intro). I don't see a particular problem with that, but a direct statement like 'invariant features' would have resonated with me just fine, while whether we can call that "knowledge" is a matter of (unnecessary) semantics in my opinion. 


=====
Post rebuttal comments: 

Thanks to authors for the detailed response and additions. I read through the comments and skimmed the revised PDF. Overall I preserve my positive rating toward this paper. The updates did improve the paper, but I would recommend the authors to use the camera ready opportunity to further address the original comments especially with respect to the delivery. 

I find the added experiments in multi task learning interesting especially given the time constraints of rebuttal. However the adopted setting is not the strongest case as what is defined to be "two tasks" is basically a superclass classification of a fine grained classifier. More concrete datasets and tasks specifically targeting challenging multi task learning are available now (see the full discussion in the original review). I would recommend taking those into consideration either for more experimentation or calibrating the conclusion made out of the current experiment. 

I recommend acceptance.  
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The goal of this paper is to analyze knowledge consistency between pretrained deep neural nets. In order to do so the paper trains neural networks to predict a hidden layer of one DNN using a hidden layer of another DNN. The model is interesting in that it is multi layer but it also allows decomposing its prediction as the sum of outputs of neural nets with different numbers of hidden layers. The prediction is dubbed "consistent features", which are further decomposed in consistent features of different complexity levels, while the error is dubbed "inconsistent features".

The paper then attempts to use this decomposition in "consistent" and "inconsistent" features in a number of ways. Assuming hidden layers of DNN A are being used to predict layer of DNN B, if A is stronger than B then the inconsistent features of B are claimed to be "unreliable features". If A is weaker, then the inconsistent features of B are claimed to be "blind spots". A figure is given to support this intuition, but I think some real evidence would have to be collected to support a claim like this.

Similarly the authors analyze model compression and distillation with their technique. They are able to show a decrease in the variance of inconsistent features as a models get progressively compressed or as a models get better during generations of distillation. Again the evidence that of the usefulness of this analysis seems very limited. The visualizations of consistent and inconsistent components don't seem to give any clear evidence.

The most interesting part of the paper shows experiments from boosting performance of a model by finetuning on features consistent with another model. None of the improvements seem close to state of the art though, and might just be some byproduct of model ensembling.

Overall the paper is very interesting, but more convincing experimental results would have to be collected to prove that the method is actually useful.
review_title: Official Blind Review #2

Meta-review: This paper presents a method for extracting "knowledge consistency" between neural networks and  understanding their representations. 

Reviewers and AC are positive on the paper, in terms of insightful findings and practical usages, and also gave constructive suggestions to improve the paper. In particular, I think the paper can gain much attention for ICLR audience.  

Hence, I recommend acceptance.
Decision: Accept (Poster)