Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper describes two issues with existing probing methods within the NLP interpretability space and proposes an alternative approach which does not exhibit the same flaws.
main_review: Strengths:
 - The paper describes flaws in multiple recent high-profile NLP interpretability papers
 - The paper proposes an alternative mechanism (PROBELESS) which behaves differently from existing methods
 - Models studied are cross-lingual, unlike most probing work which has focused on English
 - Ranking and intervention experiments are generally convincing and well-executed 

Weaknesses:
 - Generally I found this paper was difficult to read. For example, the conflation between probe quality and ranking quality was not clear to me in the abstract/introduction, and perhaps an illustrative example or figure would help clarify this point. Another option would be to clarify >>early on<< that rankings are non-optimal due to e.g., greedy inference procedures which do not search over the whole space of rankings. Similarly, a figure representing the intervention procedure and CLWV metric might be useful.
 - Pitfall II is not especially novel, since it is well-known that most probing methods in NLP are fundamentally correlational in nature. Despite this, findings about the PROBELESS model and interventions are a useful contribution to the literature and fall in line with other work (e.g., ICLR paper from Lovering, et al. 2021) showing that extractability of information correlates with its usefulness in model predictions. 

Questions:
 - Why is consistency across languages a desirable attribute here? (End of Section 2.2)
 - Can you re-explain what I'm supposed to be getting out of Figure 3a? Must be missing something.
 - In 3.2.2 it seems that "Gaussian is memorizing" is posed as a negative, but this is not evident to me. Why would memorization in this case (of a small set of function words) be considered a bad thing?

Writing Comments/Nits: 
 - In the introduction: "We see this framework as <exhibiting> Pitfall I"
 - The following sentences from Section 2.1 are difficult to interpret without reading the referenced paper: "However, we observed that the original algorithm distributes the neurons equally among labels, meaning that each label would contribute the same number of neurons at each portion of the ranking, regardless of the amount of neurons that are actually important for this label" 
 - "PROBELESS prefers neurons that separate different labels over neurons that have a similar value across labels" could be rewritten as something like "PROBELESS assigns high values to neurons which are most sensitive to a given attribute"  
 - Appendix A.2: doesn't this assume that all rankings of PROBELESS are equally likely? This might not be true if some neurons vary more than others, for example. [Please correct me if I am wrong.] 
summary_of_the_review: Although the paper is somewhat difficult to read (see suggestions), it provides a useful new approach to single-neuron probing and summarizes some weaknesses in recent existing work.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper identifies two pitfalls of existing methods for ranking individual neuron contributions to certain linguistic attributes, and shows empirically that these pitfalls indeed exist. Additionally, it proposes a new ranking method free of these pitfalls, and shows its effectiveness.

main_review: 
# Strengths

This paper identifies issues with existing methods for ranking neurons containing linguistic attributes and presents a detailed experiment showing it empirically. To address these issues, a new ranking method is proposed and it is effective.

# Weaknesses

n/a
This paper identifies issues with existing methods for ranking neurons containing linguistic attributes and proposes an effective solution.

summary_of_the_review: This paper identifies issues with existing methods for ranking neurons containing linguistic attributes and proposes an effective solution.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper responds to several recent works focused on identifying important individual neurons for particular classifying tasks. They consider 2 existing methods which rely on an external probe to rank the neurons in a network. They also introduce a method that does not rely on a probe, instead ranking neurons according to the difference between their values across labels.

The primary focus in this paper is on two claimed flaws of the existing neuron ranking methods. The first is that the authors feel evaluation of the neuron rankings is unfair because a high quality probe can have higher performance on a worse ranking, simply because it is better able to take advantage of the data that is offered. The second is that not all rankings actually point to neurons that are specialized for a particular task or label, instead just indicating highly informative neurons.

In analyzing these two flaws, the authors develop evaluation metrics for the rankings of these neurons. They consider a ranking better if the neurons encode information that is specific to the label while maintaining the particular lemma being encoded. They also consider the ranking to be better if removing the features indicated damages the performance of the language model. They test two ways of modifying the network to remove important neurons, first by ablating the neurons and then by learning a geometric translation that moves a word towards a different attribute label.

They also consider the ranking better if a ranking from top to bottom outperforms a random ranking, which outperforms a reversed ranking. This seems like a fairly weak standard to hold a ranking to, but some rankings fail to adhere to it.
main_review: 
Strengths:
- PROBELESS seems to be genuinely an extremely good neuron ranking method and is simple and intuitive.
-  The new methods for evaluating neuron ranking could be very useful, especially if the experimenter is interested in rankings that specialize in a particular task. 
- I was very interested in the finding that gaussian models memorize the data, likely because they rely on a smaller number of potential features.
-  The analysis is extremely in depth. It provides both conjectures and well justified explanations for why certain rankings outperform others.
-  I particularly like the analysis of the translation method of removing neurons, because it uses the saturation point where the lemma starts to disappear, rather than a single arbitrary point or some kind of AUC.
-  The paper was generally clear to me.

Weaknesses:
- It’s not clear to me that the linear and gaussian methods specifically aim to find neurons that specialize (instead possibly targeting much less specialized but significant neurons), whereas their probeless method clearly targets neurons specializing in a particular task label. However, their metric is based on the neuron’s significance for a particular label, while controlling for preserving lemma information, which is not clearly the goal of the other methods. The major problem in this paper is that the decisions made in evaluation and ranking are not entirely well justified.  
-  I’m not convinced that it’s fair to evaluate rankings based on whether the rankings assigned by one probe are useful to a different probe.
- I would like to see some discussion of why single neuron analysis actually is preferred over analyzing distributed dimensions. In particular, the fact that translation outperforms the ablation method does indicate that the attributes in question could be encoded in a weighted distribution over neurons rather than directly into single neurons.
- On page five, the authors hypothesize that the linear  probe method is more table because only a few dimensions are informative and gaussian distributed. I would consider this a conjecture, but I would like to see experiments that might support this hypothesis.

Minor:
- I would have liked to see an explanation of the tasks that were used in section 2.2.

QUESTIONS:
- How are A, L (the attribute and lemma labeler) trained?

summary_of_the_review: This paper is an in-depth analysis with new perspectives on how we should think about neuron ranking. It does not discuss much the difference between their philosophy towards the purpose of neuron ranking, compared to the implicit philosophy in the prior works which they compare. Overall it seems valuable but may not be entirely fair to the existing literature.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper revisits two methods, Linear and Gaussian, as described in the literature, to probe language models based on individual neurons. The paper suggests that these methods contain two limitations - 1. by ranking the neurons on a linguistic task, the methods conflate between ranking quality and the probe's classification quality; 2. the probing methods do not take into account whether the individual neurons are at all used by the model in the downstream linguistic tasks. Finally, the paper presents a new probing method, which does not rely on training a classifier, and shows that this simple method is able to discern among the neurons better than the preceding two methods in the literature.

main_review: The paper begins by first explaining the background of the two methods in the literature, Linear (Dalvi et al 2019) and Gaussian (Torroba Hennigen et al 2020).
Then, the authors describe their proposed "Probeless" method, and then dive deep into the results and how the above two methods suffer in view of the pitfalls.

## The good

One great thing about this paper is how methodical the authors are in their experimental setup. I liked the idea of disentangling the rankings from the classifiers and using them together to see whether an expected ranking (top-to-bottom) is actually better than a worse ranking (random or bottom-to-top). The choice of the metrics also make sense to me. The intervention experiments (Pitfall II) are really interesting, as I think the proposed method could be useful for other related works too. It is indeed very crucial to explore whether the "top-ranked" neurons are actually contributing to the language model output. The results are very interesting - it seems ablation is not a good intervention approach, whereas Translation is a much better strategy. Similar methods have been used in representational probing literature (Amnesic Probing, https://arxiv.org/abs/2006.00995), it would be great to add a note on how they compare.

## The bad

The paper can perhaps present the results/discussion in a better way, as several aspects are not clear to me.

### Section 2.1

- The authors mention they do not exactly use the method shown by Dalvi et al 2019 for Linear, and propose an alternate way to use Linear by computing the mean absolute value of the weights associated with it. Does this enhancement mean it is different from the original Linear model? It would be useful to extrapolate this difference in the paper.
- In the definition of "Probeless", it is not clear what is \hat{z}. Perhaps the authors indicate it is the mean of the words that does not possess the attribute/label information? Maybe I'm missing something obvious.

### Section 2.3

- "We expect to see overlap in the selected neurons" - it is not clear why we should expect to see an overlap among multiple languages. Any explanation/citation to back this claim up?

### Section 3.2

- The results section is a bit hard to parse given the introduction of two new concepts, G>L and L>G. Perhaps the authors should clearly state what these are earlier (it becomes slightly clear after reading Section 3.2.2 that G>L corresponds to Gaussian being better than Linear when it is paired with worse rankings than optimal, and vice-versa))
- Possibly the ordering of the results is incorrect - the t-SNE plot explanation could come after explaining 3.2.1 and 3.2.2

### Section 3.2.2

- Gaussian memorizing in parts-of-speech attribute is expected, as many POS tags can be simply predicted by learning the type of words (POS tagging can be solved by non-neural CRF taggers, and neural probes always gets very high scores in this task anyway). I don't know how this makes Gaussian a "bad" probe. In fact, the inability of learning the basic POS tag associations by Linear probe suggests its weakness, not strength. This suggests a capacity issue.
- Thus, G>L and L>G scenarios are highly task dependent. This calls for their study relative to the task and not in aggregate.

### Section 3.2.3

- The authors say "Probeless provides decent performance (and is inherently consistent)" - however I could not find any results of Probeless probe in Figure 3.

### Section 4.4.3

- I am skeptical of the claims in this section as it seems the numbers are cherry-picked. From Table 1, the Linear probe seems to be better than Probless in English tense. In Appendix Table 3, however, authors show more scenarios where Probeless tops other methods. This table should be moved to the main page.
- However, the claims do not hold well in terms of XLM-R model, as Linear is comparable to the Probless method (gets 3 tasks vs 4 in the latter).

### General comments

- The paper lacks a thorough comparison with recent work. For example, I totally agree in the conclusion that high probing accuracy does not necessarily entail the information is actually useful for the model. Similar investigations have been made in the representational probing literature (Pareto Probing: Trading-Off Accuracy and Complexity, EMNLP 2020)

### Experiment Suggestion

- The authors investigated the results primarily with M-BERT, and they also investigate with XLM-R but the results are not so clear in the latter model. The authors can perhaps also investigate one more similar model, say M2M, to be a tie-breaker for a better conclusion.

summary_of_the_review: Overall I found the paper to be quite illuminating on highlighting the different aspects of probing with individual neurons. However, the paper requires more polish in presenting the results and discussions of their work. I believe the finding is important for the community, especially the intervention analysis. However, I'm a bit skeptical of the claims (as highlighted in my review). Improving the writing of the paper and performing an additional model experiment could be useful to make the paper strong for acceptance.

Edit: Following the rebuttal, I'm increasing my original score of 5 to 8
review_title: null

Meta-review: This paper analyses interpretation methods that use probes to evaluate the information in individual neurons of a deep network and shows that it confounds probe quality and ranking quality, and encoded information and used information. The paper proposes a new method which does not suffer from the same drawbacks. The reviewers were positive about this paper, and the discussion between the reviewers and authors resulted in the authors adding multiple clarifications. I ask the authors to try to optimize the paper for clarity further. I recommend acceptance.
Decision: Accept (Poster)