Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: After reading author responses:

Thank you to the authors for your detailed responses. With regard to the highlighted implication that "the harder feature can be obscured completely by a spurious one; i.e., there are settings in which the model just won't adopt the harder feature at all" --  to clarify, while my phrasing may not have made this apparent, I was assuming this implication in my interpretation of the results. So my impression of the finding is not changed substantially by the author response. However, I do want to give appropriate acknowledgment of the value of explicitly testing/confirming intuitive explanations of model behaviors, and it is clear that other reviewers find value in the contribution, so I am bumping my score up a bit.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This paper addresses a seeming contradiction between findings that indicate encoding of linguistic information in models' internal representations, and findings that show models not to use more sophisticated linguistic information during fine-tuning on downstream tasks. The paper hypothesizes that a model's use of a given feature can be explained as a function of extractability of the feature in combination with the amount of evidence for that feature's predictive reliability. The authors test on toy, non-language data as well as natural language data, and find support for their hypothesis. 

All in all I think this is a reasonably clear and well-written paper, with a concrete and intuitive hypothesis. My main concern is that the motivating issue is a bit of a strawman, in that the posited explanation was fairly obvious as a means of reconciling the "contradiction" raised at the start of the paper. I can't speak for the rest of the community, and it may be that this is something that people have found puzzling -- but speaking for myself I can say that I haven't at any point considered the highlighted "contradiction" to be a contradiction, having simply assumed something like the explanation hypothesized in this paper. Now, there is of course value in providing concrete evidence supporting intuitive assumptions made by the community. However, as the authors point out,  related intuitions have already been supported by, e.g., evidence that models will more readily pick up on "easy" examples over "difficult" examples. So it's not clear to me that the paper is making a sufficiently novel, surprising contribution at present.

I think one way in which these findings would be more compelling would be if the measure of extractability were defined independently of empirical classifier sensitivity. As it is, the experiments are seemingly demonstrating that the more readily a classifier is able to pick up on a given feature, the more readily another classifier will use that feature during learning.  I have to assume that this will strike most readers as obvious. However, if extractability/MDL were measured independently of classification performance, then we would presumably learn some interesting and valuable things about what determines extractability for these models.

Smaller notes:

The two sets of experiments are described as "synthetic" versus "natural language" -- but if I'm understanding correctly, the natural language examples are generated synthetically. If this is correct, then the current framing of the distinction is misleading.

Figure 2 is difficult to interpret, and the placement of the legend is odd-looking and confusing. Fig 3 is also pretty difficult to extract information from -- generally presentation of information could be made clearer for the reader.

The wording on p3 can be taken to imply that MDL was introduced by Voita & Titov (2020). I would recommend rephrasing and/or also citing earlier MDL references.
review_title: Clearly defined hypothesis, but limited contribution; finding seems to support existing knowledge/assumptions

final_score: rating: 7: Good paper, accept
review_content: The paper aims to bridge the gap between model interpretation using probing and model's use of spurious features. They show that the findings of MDL with respect to a feature correlate with the extractability of the feature, given the evidence of representing the feature is available in the training data. The results are presented using both synthetic and natural language data.

I really like the premise of the paper, which is connecting the research on the linguistic learning of a model with the presence of important and spurious features in the data.

One issue I have with the work is the simplistic assumptions that are likely to be different in the real-world data. Real-world data may have various spurious features and it is possible that not one feature alone is playing a role in pushing the model to rely on spurious features. It can be a combination of spurious features plus the relative presence of important features. It is hard to imagine how this method will scale to real-world datasets. I would like the authors to comment on it.

Moreover, the findings are quite expected. In general, the probing methods including MDL were mainly aimed at analyzing the linguistic learning of the representations. In that case, MDL is scoring the representations with respect to various linguistic properties. Here, the authors are using MDL to look at how input features are represented in the model. Statistically, MDL is likely to look at the same things as the model is looking at since both of them are based on the same training data and input features. Please comment on this, in case I misunderstood the point.


Minor comments:
- what is the reason for low performance when using Glove?
review_title: The paper aims to bridge the gap between model interpretation using probing and model's use of spurious features. They show that the findings of MDL with respect to a feature correlate with the extractability of the feature, given the evidence of representing the feature is available in the training data. The results are presented using both synthetic and natural language data.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: 
# Summary
This paper studies the relationship between extractability of features from pre-trained representations and how much a fine-tuned model uses that feature. The extractability of features is measured by the minimum description length of a probing classifier trained to detect the feature from the pre-trained representations (using the online code version of Voita and Titov). The degree to which a fine-tuned model uses the feature is measured by the amount of evidence required for a model to tease apart spurious from non-spurious features (called "target" features). Evidence here means examples where a spurious feature occurs but a non-spurious feature does not occur. When there are many such examples (high spurious-only rate), it is easier for a model to reject the spurious feature and learn to rely on the target feature. The "degree to which a fine-tuned model uses a feature" is defined as the minimal spurious-only rate at which the model can accomplish the task. 

The paper has two kinds of experiments, on synthetic and more natural data. The synthetic data are sequences of symbols where the task is to identify simple properties like occurrence or repetition of symbols. The experiments are set up such that varying rates of spurious-only examples are presented during training, providing increasing amounts of evidence against the spurious feature (presence of the symbol 2) and in favor of the target feature. The target feature is identical to the label, that is, it is 1 when the example corresponds to the label and 0 otherwise. The paper reports extractability of the spurious and target features via the MDL of a probing classifier. The metric of interest is the relative MDL, where higher means the feature is more extractable. When the features are more extractable, less evidence is required for the model to reject spurious features. With less extractable features, more evidence is required. 

The natural language examples are made with acceptability judgements of examples generated by grammars for three linguistic phenomena (subject-verb agreement, negative polarity items, and filler gap dependencies). Here again the setting is similar, modulu a tweak on how to calculate extractability. The main result here is high (negative) correlation between extractability and evidence required for rejecting the spurious feature. 

# Main comments
1. This paper fills an important gap in the NLP interpretability literature that has recently been a cause of concern in the community. On the one hand, probing classifiers tell us something about the existence (and more recently the extractability) of properties in pre-trained models' representations. But they do not tell us whether a model uses those properties. One the other hand, many challenge sets and test suites tell us whether a model can successfully perform a task requiring some linguistic property. The paper aims to connect these two aspects, and it does so quite convincingly, although I have some reservations below. 
2. The experimental setup is well designed. The use of synthetic data allows a fairly clean setup where spurious and non-spurious features are distinct and simple. The experiments of training with increasing amount of spurious-only examples are instructive. 
3. The natural language examples are important as they go beyond synthetic data and closer to a naturalistic scenario. However, these are still templatic sentences and synthetic in a sense. I wouldn't call these naturalistic examples. Ideally, experiments on naturally occurring data would be more convincing. Or, at the very least a discussion of this issue should be made. 
4. The paper makes use of recent advances in interpretability work, including information-theoretic probing, and draws connections to a broad range of related work. 
5. The assumption that the extent to which a model uses a feature can be measured by the spurious-only error rate (at some spurious-only occurrence rate) is questionable in my opinion. In a very clean setting like the synthetic data, I could maybe accept it. But, "using" is in fact a causal concept, while a causal mechanism has not been demonstrated. The paper alludes to this point in the discussion, but I think the discussion around this point should be expanded, and the strong claims should be rephrased or modulated. 
 

# Questions and other comments
1. The paper makes the assumption that the target feature t and the label are the same. I am not convinced about the "without loss of generality" claim. In practice, it is not easy to isolate a feature t that is identical with the label. How would this assumption affect the generalization of the approach to more realistic scenarios? 
2. The task is a binary classification task. The features holding is also binary, that is either a feature holds (1) or not (0). But, suppose the label is 0, then the t feature is also 0, meaning it does not hold. This seems contrary to what is meant. This could be a confusion on my part. 

## Synthetic data
3. Why is MDL computed by training a classifier to distinguish s-only from neither, and not from some other part of S? 
4. Footnote 3 is concerning - Aren't MDLs higher than a uniform code meaningless? 
5. The classifier is not so simple (LSTM + 1-layer MLP). Why is that? How does the identity of the classifier affects the results? 
6. The both-error subplot in figure 2 shows a slight increase in error rate with large s-only rate. Does this mean that the model has (falsely) learned to reject the example when s is in it? That is, it has learned another spurious feature, just in the other direction, instead of learning to rely on t. 
7. A similar pattern is found in the t-only error subplot. There, even with high s-only rate, the models don't classify t-only examples correctly. I wonder why this plot is different from the s-only error plot, as this shows a directional behavior. Some discussion of this would be useful. 
8. There seems to be a stark contrast between the contains-1 feature and the other three, both in terms of MDL and in figure 2. Is it possible to show a more gradual behavior between the two extremes? 

## Natural language examples
9. Why are the training sets so small? How does this affect MDL numbers and their validity? Apropos footnote 1. 
10. Why exactly is it hard to generate t-only examples? The appendix is indeed helpful in making sure the MDL(t) calculation method is legit, but more clarification around this issue would be good.  
11. Here, s-only error is used as "the use of the spurious feature", but this is only one aspect in which a model may make use of s. It may be that a model makes more complicated use of s, when s is found in combination with t. The discussion touches upon this point by acknowledging that the work does not establish a causal relationship between extractability and feature use. I'd go even further and say that "feature use" should be defined in causal terms. 
12. What is the performance (F-score) for determining s-rate*? is that the performance on s-only examples? on other examples? Why the shift to F-score now?
13. Discuss y-axis differences in figure 3c. BERT needs much less evidence than (some cases of) GloVe and T5. How does that impact the analysis?
14. The term "learning curves" for figure 4 is confusing: those aren't results during training, right? They are results after training, each time with a different rate of s-only examples. 


review_title: Important work filling a gap in current NLP interpretability literature

Meta-review: The paper studies the features extracted by the pre-trained language model and how fine-tuning makes use of these features. The paper is well-motivated by two lines of research in the NLP area -- 1) probing approaches for understanding the features extracted in the pre-training model, 2) model behavior analysis that shows models take shortcuts for making predictions. The paper provides a comprehensive study to bridge the gaps between these two lines of discussion. 

All the reviewers agree the paper has strong merits and concerns have been addressed.  

Decision: Accept (Poster)