Title: Understanding intermediate layers using linear classifier probes
Authors: Guillaume Alain
Yoshua Bengio
Venue: ICLR 2017

Abstract: Neural network models have a reputation for being black boxes. We propose a new method to better understand the roles and dynamics of the intermediate layers. This has direct consequences on the design of such models and it enables the expert to be able to justify certain heuristics (such as adding auxiliary losses in middle layers). Our method uses linear classifiers, referred to as ``probes'', where a probe can only use the hidden units of a given intermediate layer as discriminating features. Moreover, these probes cannot affect the training phase of a model, and they are generally added after training. They allow the user to visualize the state of the model at multiple steps of training. We demonstrate how this can be used to develop a better intuition about models and to diagnose potential problems.

TL;DR: New useful concept of information to understand deep learning.

Review discussion:
The experimental results depicted in Figure 8 are very intriguing. Have you tried using addition instead of concatenation and running the same experiment? It would be very interesting to compare the two scenarios, especially since the concatenation operation seems to have made the first layers of the network seem "not very useful, increasingly so as approaching the concatenation point".
Regarding the Inception experiments, have you considered adding an additional trainable transform layer before the softmax? This could reduce the feature dimensionality and make the problem more feasible (of course at the cost of not being able to use just a linear classifier on the original features). One additional question regarding the Inception case is whether Pooling operations should be used before feeding the features to the linear classifier, as an alternative way to look at the problem.
First of all, thanks for taking the time to read our paper, for asking questions and commenting.
About Figure 8, we have not tried to use addition instead of concatenation. A lot of people suggested this approach because this is how it's done in some other models with skip connections. The reason why we used concatenation was to make sure that the combination of the two layers was "equally or strictly more informative". If the signal from either layer was unusable, then it would not get any kind of feedback. We agree that it's a good idea to use addition instead, or that it would have been interesting to run another experiment with addition. Part of the reason why we included this model in the paper was to demonstrate a pathological behavior that could be debugged properly by using linear classification probes. One could say that the model was meant to be broken in some way. It's not about the models; it's about how you can now see inside the models.
Concerning the Inception experiment, this was one of the suggestions that we received. We can either project to a lower dimension by taking a linear transformation, or by taking a random subset of the features. We have to note that we don't want to have "second-order probes" where there is a non-linearity inside the probes. This would ruin the convexity of the probes (which is nice to have).
However, a trained linear projection to a lower subspace could still be used. Let's say that we have 1.5 million features and 1000 classes. We want to avoid 1.5x10^6 times 1000 classes. If we project to a lower-dimension space this means that it has to be of dimension less than 1000.
Otherwise it would be pointless since we would still have the same huge matrices. Can be project down to 100 ? We still have a big-but-manageable matrix in that case. How about down to 10 dimensions ? This is getting ridiculous at this point. One solution would be to have some kind of sparsity so not all inputs would be used. In that case, the projection would be a feasible idea.
We could project groups of features together, and I believe that this is why you're mentioning pooling. It seems like one way to do it. We would still need to pick some kind of schema to bunch features together into a kind of multi-stage linear operation. In a way, this structure enforces sparsity. I like your suggestion.
This paper is essentially to test the classification rate based on the features from different layers of the neural network, which has been explored in many early works, e.g., DECAFF.  What  technical difference does this paper make?
The experimental results in this paper show that 1) (Fig 5a) too many random layers are harmful. 2) (Fig 5b) training is helpful. 3) (Fig 7) lower layers converge faster than higher layer. 4) (Fig 8) too deep network is hard to train, and skip link can remedy this problem. They are all well-known conclusions, and more comprehensive discussions have been made in previous work. Can the linear classifier probe provide some new explanations for these phenomena? Based on the paper's observations, can we get some new clues to design better neural network architectures?
The analysis in this paper is all for classification network. Can the linear classifier probe be useful for non-classification model, e.g., image generation networks, regression-based localization networks?
I got a feeling that this paper somewhat wants to overwrite the definition of information, as a particular technical term, using the linear classifier.
However, the linear classifier is a particular discriminative formulation driven by particular loss function and particular data. Is it general enough as a measurement for information? And is it possible to provide some theoretical justification.
In general, the intro arguments of this paper is reasonable to me, but I have the above questions about the technical soundness and significance.
Thank you for your comments. These are good observations. I hope that I can clarify some things about our paper and our motivations.
You are certainly correct in saying that this paper overlaps with the DeCAF paper. I will admit that I missed that paper and I will absolutely make sure that I refer to it in ours. From my reading of the DeCAF paper, it is more about transferability of features from one dataset to another. They compare their features with those of SURF.
In our particular case, we are not seeking to argue that the intermediate features are useful. That has been demonstrated often, as you point out. We are interested in looking at those features at all layers, and at multiple times during training. The DeCAF paper is more interested in finding out which layer has the best features, and it is not concerned the temporal aspect of training.
You list a collection of other well-known conclusions, and again we are not claiming that those are novel. The contribution of our paper comes from suggesting a tool to observe these behaviours in better light. Visualizing something more clearly should come as pre-requisite for better explanations. The animations showing the “domino effect” in probe performance are something that we have not seen anywhere else, and we would argue that they help with our intuition about the models.
Many models picked in our paper were pathological by design. Our point is that we can immediately see the pathology by using linear classifier probes. We are not arguing in favour or against skip connections. We are not making suggestions for model design in general. This is not in the scope of the paper. The reason why we use classification instead of other tasks is because it is a good starting point, and it can be entirely sufficient to illustrate the concept. We certainly have ideas about how to apply this to other settings (ex : denoising auto-encoders).
We are suggesting the word “information” for the discussion in the context of this paper.
But we would certainly not try to overwrite the definition of information as Shannon entropy. We were looking for a useful concept to help with our intuitive understanding of the internal dynamics of deep models, and we though that linear classifier probes were a good way to achieve that. They are certainly not universal / general enough to the point where we could provide a strong theoretical justification to argue that they are “absolute”. To us, the question was : looking at the plots and the animations, did we feel that we had a better understanding of the model by plotting the probe performances ?
The authors propose a method to investigate the predictiveness of intermediate layer activations. To do so, they propose training linear classifiers and evaluate the error on the test set.
The paper is well motivated and aims to shed some light onto the progress of model training and hopes to provide insights into deep learning architecture design.
The two main reasons for why the authors decided to use linear probes seem to be:
- convexity
- The last layer in the network is (usually) linear
In the second to last paragraph of page 4 the authors point out that it could happen that the intermediate features are useless for a linear classifier. This is correct and what I consider the main flaw of the paper. I am missing any motivation as to the usefulness of the suggested analysis to architecture design.
In fact, the example with the skip connection (Figure 8) seems to suggest that skip connections shouldn't be used. Doesn't that contradict the recent successes of ResNet?
While the results are interesting, they aren't particularly surprising and I am failing to see direct applicability to understanding deep models as the authors suggest.
We are indeed saying that it could be possible for intermediate features to be utterly useless to a linear classifier.
However, this extreme situation should be very rare, and in practice (so far) we have never encountered a situation where the intermediate layers were useless but the final layer was suddenly very good. It seems to follow a smoother transition instead.
In fact, one should also look at this the other way around. Let's say that we had a binary classification task, and we conjectured that some intermediate layer (or branch) was useless, that it contained no information. If we measured that we could get 80% correct classification with a linear classifier probe on its features, then we should quickly revise our opinion about that layer. There is basically no way to say that features contain no information when a linear classifier probe gets 80% correct on a (balanced) binary classification task.
Regardless of what constitutes "information", we claim that it can at least be detected in that way.
Now, we are not claiming that our measurement of "information" is absolute in any way. We are trying to illustrate that it's a useful concept to analyze a model and understand what's happening.
In the case of the skip connection (Figure 8), the model was selected to be pathological. It was a case where a model was designed to be too deep, and the skip connection seemed to fix the problem at first, but we can reveal the fact that there is still a problem with the model. Half of the model is dead, and that's not a good use of skip connections.
We are not trying to undermine skip connections, and certainly not suggest that Resnet is bad. Our point is that *sometimes* skip connections are bad, and then we can diagnose that bad behavior by using linear classifier probes.
This is a subject in which we want to strive to make the paper clearer. We are indeed showing certain model based on the fact that they are bad, because that's one of the situations where linear classifier probes will shine. We want to talk about linear classifier probes, and all the models shown in the paper are just case studies to illustrate the role of probes.
This paper proposes to use a linear classifier as the probe for the informativeness of the hidden activations from different neural network layers. The training of the linear classifier does not affect the training of the neural network.
The paper is well motivated for investigating how much useful information (or how good the representations are) for each layer. The observations in this paper agrees with existing insights, such as, 1) (Fig 5a) too many random layers are harmful. 2) (Fig 5b) training is helpful. 3) (Fig 7) lower layers converge faster than higher layer. 4) (Fig 8) too deep network is hard to train, and skip link can remedy this problem.
However, this paper has following problems:
1. It is not sufficiently justified why the linear classifier is a good probe. It is not crystal clear why good intermediate features need to show high linear classification accuracy. More theoretical analysis and/or intuition will be helpful.
2. This paper does not provide much insight on how to design better networks based on the observations. Designing a better network is also the best way to justify the usefulness of the analysis.
Overall, this paper is tackling an interesting problem, but the technique (the linear classifier as the probe) is not novel and more importantly need to be better justified.
Moreover, it is important to show how to design better neural networks using the observations in this paper.
I feel that the reviewer has understood our paper and that they raise good points.
We would like to address what the problem 1. by saying that we are convinced that there will always be an arbitrary aspect to whatever we select as probe. We picked a linear classifier so that it would be sufficently "elementary" to convince people that it was not "doing too much work" in stead of the model. And at the same time it would have to be just flexible enough to do a meaningful task that would represent an interesting measure of quality.
By asking for something absolute, or waiting for a "crystal clear" concept to show up, we might be leaving aside a powerful tool. We need more ways to understand deep models, even when they are not perfect. More analysis/intuition will come by using the tool (i.e. linear classifier probes).
As for problem 2. we agree that it would be great to have a case of "practical use" where a new counter-intuitive model was discovered based on an analysis with linear classifier probes. Does this mean that everything has to come all at once ? In a way, if we invented a new kind of Resnet based on the insights from linear classifier probes, then the model itself would probably be the main object of the paper, and not really the linear classifier probes.
This paper proposes a method that attempts to "understand" what is happening within a neural network by using linear classifier probes which are inserted at various levels of the network.
I think the idea is nice overall because it allows network designers to better understand the representational power of each layer in the network, but at the same time, this works feels a bit rushed.
In particular, the fact that the authors did not provide any results in "real" networks, which are used to win competitions makes the results less strong, since researchers who want to created competitive network architectures don't have enough evidence from this work to decides whether they should use it or not.
Ideally, I would encourage the authors to consider continuing this line of research and show how to use the information given by these linear classifiers to construct better network architectures.
Unfortunately, as is, I don't think we have enough novelty to justify accepting this work in the conference.

Meta-review:
role: PC
data: comment: The reviewers generally agreed that the research direction pursued in the paper is a valuable one, but all reviewers expressed strong reservations about the value of a linear probe on intermediate features. The lack of experiments on more complex state-of-the-art networks is also potentially problematic. In the end, it seems that there is insufficient evidence that the proposed approach is actually a useful tool in its present state, though the authors should be encouraged to pursue this line of research further.

Decision: Reject