Is Probing All You Need? Indicator Tasks as an Alternative to Probing Embedding Spaces
Authors: Tal Levy
Omer Goldman
Reut Tsarfaty
Venue: EMNLP 2023 Findings (2023)
Keywords: Probing
Probe
Indicator
Word Representations
Embedding Space
Interpretability
Context
Social Bias
Morphology
Semantics
Gender
Concept Erasure
TL;DR: Proposing indicators (probes that do not involve training an auxiliary model) to elicit information from embedding spaces, aside from classifier-based probes, due to their inherent problems

Abstract: The ability to identify and control different kinds of linguistic information encoded in vector representations of words has many use cases, especially for explainability and bias removal. This is usually done via a set of simple classification tasks, termed \textit{probes}, to evaluate the information encoded in the embedding space. However, the involvement of a trainable classifier leads to entanglement between the probe’s results and the classifier’s nature. As a result, contemporary works on probing include tasks that do not involve training of auxiliary models. In this work we introduce the term \textit{indicator tasks} for non-trainable tasks which are used to query embedding spaces for the existence of certain properties, and claim that this kind of tasks may point to a direction opposite to probes, and that this contradiction complicates the decision on whether a property exists in an embedding space. We demonstrate our claims with two test cases, one dealing with gender debiasing and another with the erasure of morphological information from embedding spaces. We show that the application of a suitable indicator provides a more accurate picture of the information captured and removed compared to probes. We thus conclude that indicator tasks should be implemented and taken into consideration when eliciting information from embedded representations.

Is Probing All You Need?
Indicator Tasks as an Alternative to Probing Embedding Spaces
Tal Levy
Omer Goldman
Reut Tsarfaty Bar-Ilan University
Bar-Ilan University
Bar-Ilan University
talevy1 7emryy11@mmaa...m.mm com
omer gmer.golooe dman@gma com
reut. reut.tsarfatyee www.fatyebiuuaaccc.. i1
Abstract
Probing tasks (henceforth probes) are designed to evaluate the information existing in represen- tations by training a simple classification model (Conneau et al., 2018; Niven and Kao, 2019). The existence of a probing model that can classify the queried linguistic property is assumed to imply that the respective linguistic information exists in the representations (Alain and Bengio, 2017; Belinkov and Glass, 2019).
The ability to identify and control different kinds of linguistic information encoded vec- tor representations of words has many use cases, especially for explainability and bias re- moval. This usually done via of simple classification tasks, termed probes, to evalu- ate the information encoded in the embedding space. However, the involvement of trainable classifier leads to entanglement between the probe's results and classifier's 's nature. result, contemporary works on probing include tasks that do not involve training of auxiliary models. In this work we introduce the term indicator tasks for non-trainable tasks which are used to query embedding spaces for the ex- istence certain properties, and claim that this kind of tasks may point direction opposite to probes, and that this contradiction compli- cates the decision whether a property exists in an embedding space. We demonstrate our claims with two test cases, one dealing with gender debiasing and another with the erasure of morphological information from embedding spaces. We show that the application of a suit- able indicator provides a accurate picture of the information captured and removed com- pared to probes. We thus conclude that indica- tor tasks should be implemented and taken into consideration when eliciting information from embedded representations.
However, probes have several, frequently con- templated, drawbacks. Generally, probes are meant to 'un-black-box' word representations, that is, act as lenses into what information exists in the representations. Yet the actual relation between the probe's results and the representations thereof remains unclear. This has led some researchers to claim that the probing classifiers should be kept simple (Conneau et al., 2018), others are in favor of complex ones (Pimentel et al., 2020a), and few advocate for dispensing with probing classifiers altogether (Immer al., 2022).
A solution frequent in the literature (Wu et al., 2020; Zhou and Srikumar, 2022; Ravfogel et al., 2022, inter alia) is to accompany the results of probes that involve training an auxiliary classifica- tion model, with results over substantially different kind of tasks that are designed to query linguis- tic information directly from the representations, and do not require training another model. In this work we propose to collectively refer to such tasks by the name indicator tasks, as they provide di- rect indication of the stored information. We claim that indicator tasks are generally superior to train- able probing models, and provide more faithful information regarding the complex structure of the embedding spaces.
1 Introduction
Pre-trained vector representations of words have in- troduced a stark improvement the performances of any NLP task they have been applied to, and proved to be effective in classification and predic- tion of real world tasks (Peters et al., 2018; Devlin et al., 2018; Liu et al., 2019b; Zeng et al., 2021). This has led researchers grow interest in ana- lyzing the encoded linguistic features in the rep- resentations. To this end, probing tasks, initially proposed by Köhn (2016) and Adi et al. (2016), are frequently used.
We substantiate our claim with test cases that involves concept erasure: gender debiasing (Bolukbasi al., 2016) and morphological prop- erty removal. We show that in both cases the evalu- ation of erasures with indicators provides a fuller

---

picture of the information erased and retained. Our two test cases are dissimilar in many ways. Gen- der debaising involves a single, binary, unwanted, social bias that models learn implicitly from the distribution of words in the corpus. Morphological features, on the other hand, are series of multi- valued grammatical properties, expressed explicitly in the text, and are not undesired per se, yet their erasure may benefit processing tasks in morpholog- ically rich languages.
as the primary method for querying linguistic in- formation in dense vectors. A probe is classifier designed to elicit property of interest directly from the representation (Adi et al., 2016; Alain and Bengio, 2017). Their simplicity and speed are ap- pealing as opposed to the alternative of comparing performance over downstream tasks. However, re- searchers are yet to reach consensus regarding several problems with probes (Immer et al., 2022).
Many issues with probes are derived from their usage of linear or neural models. Most prominently are the two interleaved problems of classifier and task selection. It is well received that probing classifier should be simple (Liu et al., 2019a), since as Hewitt and Liang (2019) stressed, with enough training data, sufficiently expressive probe could solve any task regardless of the representations. On the other hand, Pimentel et al. (2020a) have shown that simple tasks are not informative enough when applied to contextual representations. Thus a probing task should not be simplistic. However, researchers found that complex tasks often require complex classifier (Belinkov and Glass, 2019), which are not recommended, as mentioned above.
We apply two different concept erasure meth- ods to them, respectively: RLACE (Ravfogel et al.. 2022) for gender debiasiin, and INLP (Ravfogel et al., 2020) expanded to multiple features for mor- phological features removal. We evaluate the suc- cess of these methods with standard linear probes, as well as with suitable indicator tasks: Word Em- bedding Association Test (WEAT, Caliskan et al., 2017) and KNN-bias correlation (Gonen and Gold- berg, 2019) for gender debiasing, and for morpho- logical information removal we present a novel Double Edged Outlier Detection (DEOD) that indi- cates both morphological and lexical semantic information. All representations are obtained from contextualized BERT models (Devlin al., 2018).
We are thus led to a three-way trade off between the complexity of the task, the complexity of the classifier, and the efficacy of the probe. Given that simple models do not fare well on complex tasks, we cannot prefer a simple probing model and a complex task simultaneously. However, opt- ing for either more complex model simpler task would limit the reliability and quality of in- formation given by the probe on the existence of property in the embedded space.
The results in both test cases, despite their dis- similarities, point to the same direction. They show that the conclusions pointed to by the indicators are contradictory to those obtained from the probes, When removing gender bias from the representa- tions we found probes to report complete erasure of the bias while our indicators still manage to pick it up from the vectors. In the second test case, we compared two possible extensions of INLP and found that probes point to one extension possibil- ity as superior, while the indicator points to the other, not only by indicating the removal of the morphological properties, but also by examining the overlapping dimension of semantics.
Previous works suggested various solutions to mend these problems. Hewitt and Liang (2019) suggested to examine probes control tasks, in which labels are distributed randomly, in addition to testing the probes on the desired task. Probes are then considered successful only if they perform well on the target task while failing on the control task. Alternatively, Elazar et al. (2021) suggested amnesic probing, in which a probe is considered successful only if it is possible to erase the informa- tion from the representations and make the same probe fail. The problem with these methods is that they take a process with complex problems and only complicate the process further, making it even harder to determine what in the results of a given probe is ascribable to representations, to the probe, or to the suggested new mechanism.
We therefore conclude that when eliciting lin- guistic information from embedded spaces, results from indicator tasks should be presented alongside standard probes, and results from probes should not be overstated.
2 Background: The Probing Conundrum
The question of evaluating linguistic properties in dense word representations has been long stand- ing one, entering alongside the introduction of static word embeddings (Mikolov et al., 2013). Nowadays, probing tasks are generally accepted

---

Thus, in order formulate the indicators in our case studies we draw inspiration from such tasks. pervolume (Pimentel et al., 2020a) to provide an indication on the ease of extraction of properties rather than on their existence. These methods, while useful, embrace the aforementioned trade off and relinquish the attempt of solving it. We, on the other hand, advocate for completely different approach that bypasses the entire debate.
Another line of works suggested amending the oulput of the probing process itself by adding de- scription length (Voita and Titov, 2020), mutual information (Pimentel et al., 2020b) or Pareto hy-
The advantages of indicators have led to their use in various works, usually to complement the picture painted by standard probes. For example, Zhou and Srikumar (2022) used "DirectProbe" (Zhou and Srikumar, 2021) as an indicator in addition a 2- 2-layered non- non-linear classifier reinforee and ex- plain their conclusion that fine-tuning BERT does not always improve performance. Similarly, Mer- chant et al. (2020) used an MLP-based probe in addition to an indicator named "structural probe" (Hewitt and Manning, 2019) to show that after fine-tuning, word representations keep the linguis- tic structure they learn during pre-training. They completed the picture by showing with the indi- cator task of representational similarity analysis (Kriegeskorte et al., 2008) that fine-tuning affects the top layers more than the bottom layers. Lastly, the work presenting the concept erasure method RLACE Ravfogel et al. (2022), used in our first test case below, demonstrated its effectiveness by utilizing both probes and indicators.
In general, the probing conundrum seems to stem from the presence of a train set which makes it un- clear whether the probe employs artifacts in the training data to solve the task or evaluates the en- coded information itself. Therefore an alternative that does not require a training set and a trainable model is warranted.
3 Introducing: Indicators
As a result of these various issues with probing, previous works had suggested to elicit linguistic properties from embedding spaces without the in- volvement of trainable auxiliary model. These studies mostly relied on the similarity between vec- tors, either between the representations themselves (Wu et al., 2020) or between attention weights (Htut et al., 2019). In addition, clustering was also taken as possible task derived from vectors alone (Zhou and Srikumar, 2021). Here we propose to refer to the different kinds of such tasks with the umbrella term indicator tasks, as they indicate the presence absence of a property in the representation with- out any intervention or entanglement of the task itself with the vectors is supposed to study.
Here we demonstrate that despite having a shared goal, i.e., querying information from dense representations, probes and indicators may result in contradicting conclusions. The implication of such disharmonious results is that if we want fuller picture of the information existing in the represen- tations, probing results must be accompanied with those obtained by indicators.
As will be demonstrated in the test cases below, the specific modus operandi of trainable probes makes them less likely to detect signals that exist in the embedding space but deviate from the pattern they were made to spot. We will demonstrate that by pitting linear probes, that were selected due their simplicity in accordance with the issues laid out in the previous section, against simple similarly- based indicators. We will show that this difference is significant enough to make indicator tasks detect gender bias and morphological information that are undetectable by a probe.
By virtue of not involving a trainable classifier, indicators cancel out many of the arguments against probes. The tasks can be complex but still fast to devise and cheap to implement and run, without a risk of over-fitting the training data- as there is none. Most importantly, there no other model that interjects, ascribing the performance directly to the embedded representations, where the task is essentially performed in a zero-shot setting. All it takes is to design or select the indicator that suits your needs. In sense, indicators can be perceived resurgence of intrinsic evaluation methods that were popular with static representations. These methods were typically formulated as word simi- larity tasks, either explicitly (Hill et al., 2015, in- ter alia) or implicitly (Levy and Goldberg, 2014).
4 Concept Erasure
Concept erasure methods are aimed at remov- ing information from word representations. For- mally, their objective is, given set of word representations X = (a1,,..Tr E IRd) and a corresponding property distributed over words P {P1,..Pa)) to learn a function r(-) such that

---

r(X) (r(m)) ... (xn)) are not predicative of P. but preserve as much information as possible from X (Ravfogel et al., 2022).
Examples for male biased words successful, general, carned, president, politician, engineer. captain, fatal Examples for female biased words aesthetic, looks, singer, lipstick, pink, smiled, skirts, pretty
However, there is no consensus on how to evalu- ate the success of the erasure procedure. The evalu- ation methods presented in previous works usually rely specifically on the nature of the crasure method and the nature of the target property. In addition, timely field tendencies or "trends" are also influen- tial, with earlier works devising intrinsic evaluation methods (Bolukbasi et al., 2016; Zhao et al., 2018; Gonen and Goldherg, 2019) and more recent works utilizing probes (Ravfogel et al., 2020, 2022).
Figure 1: Examples from the 100 most biased words in the unperturbed embedding space induced by BERT.
5.2 Experiments
We assess the success of RLACE in removing gen- der bias from representations obtained from BERT (Devlin et al., 2018). We use a standard probe to classify words as masculine or feminine, and compare it to indicators from Gonen and Goldberg (2019) that are used to identify bias, and that were originally executed over non-contextualized em- beddings.
Our case studies, presented in detail in the fol- lowing sections, systematically compare both eval- uation methodologies, with an added effort of ad- justing the former to contextualized embeddings. Each of these test cases involves different erasure method, a different property to be erased, and, to evaluate the success of the erasure method, differ- ent indicator(s) in addition to probes. Common to both cases is the contradicting conclusions drawn from evaluating the erasure method with probes compared to its evaluation with indicators.
Data To create the contextualized vocabulary we used GUM (Zeldes, 2017), an English universal dependencies dataset (Nivre et al., 2020). Note that every word has several representations, each for every context. As is customary in gender bias works, we excluded from the vocabulary words that explicitly gendered, such as man, queen and the like, in order to examine only unjustified gender bias.?
5 Case Study 1: Gender Debiasing
Of all unwanted social biases in word represen- tations (e.g, Nadeem et al., 2021), gender bias is the most researched both as an end goal in and of itself, and as a test case for information removal. We adopt gender debiasing as simple test case of removing binary unwanted property from word representations, and we examine the efficacy of recent state-of-the-art concept erasure method contextualized embeddings.
Tagging biased words order to apply the in- dicator tasks, presented below, we first need to tag set of words that are gender biased in the unper- turbed space. Following Bolukbasi et al. (2016), tag words gender biased if their vectors are similar to those of explicitly gendered words. In practice, this is done by projecting the vectors of all non-explicitly-gendeee words onto a "gender direction". ". calculated by a ID-PCA on a sub space spanned by the difference vectors of several gen- dered word pairs, e.g., she - he, woman măn, etc. The words corresponding to the 10,000 vec- tors with the highest positive projection are tagged feminine while those with the highest negative pro- jection are tagged as masculine. Figure I exempli- fies some of the words that appear be with the strongest bias.
5.1 RLACE
Relaxed Linear Adversarial Concept Erasure (RLACE, Ravfogel et al., 2022) is a post hoc adver- sarial method for removing a single binary property from an embedding space. Unlike other adversar- ial methods that incorporate changes to the model during training. this method operates on the embed- ded vectors after the training is complete. It aims identify a linear subspace from the representa- tion and remove it, by solving a linear minimax game and projecting the vectors using the resulted matrix.
'ee used HuggingFace implemented bert-base-uncee uncased (Wolf al., 2020).
"In addition, we excluded sentences in which at least one word is explicitly gendered, since, in the contextualized em- bedding case, other words in a sentence could legitimately inherit gender-related properties from these gendered neigh- horing words.

---

The list of biased words is fixed throughout the experiments. That is to say, both probes and indi- cators were tasked with discerning between biased and unbiased words that were tagged only based on the original BERT representations.
Method
Original
RLACE
Probe (Accuracy)
99.95
48.7
WEAT (d)
1.17
1.13
KNN- Bias (Pearson)
0.638
0.601
Measuring gender bias in BERT's embedding space before and after RLACE projection. Both the probes and the indicators should not detect bias after the application of RLACE, so lower is better. While probes point to no bias at all, both indicators point only a small drop.
Table I:
Probes We train and test linear classifier (Per- ceptron; Pedregosa et al., 201 to predict mascu- line and feminine of the words in the list.
Indicators Our indicators for this tasks are the ones used by Gonen and Goldherg (2019), extended here to a contextualized setting.
results still detect bias, as 0.601 is by means negligible correlation and d score that is above 0.8 considered high (Caliskan al., 2017). Both results are significant with p-values of O.
One indicator is the Word Embedding Associ- ation Test (WEAT), introduced by Caliskan et al. (2017). It fixes a set of words related to stereotyp- ically gendered occupations and traits, and tests whether group of words suspected as biased has a tendency for uneven association with one of the fixed word sets. We experimented with the sets related to career (stereotypically male) and fam- ily (female). The metric reported is the standard WEAT's ss d, see Caliskan et al. (2017) for details. Another indicator, devised by Gonen and Gold- berg (2019), is meant to detect implicit bias, i.e., the situation in which words are not more simi- lar to explicitly gendered words, but they may be similar among themselves. For example, an era- sure method may move the word "nurse" further from "woman", but its neighbors may still include high abundance of stereotypically gendered words like "receptionist". Therefore, the KNN-bias cor- relation indicator checks the Pearson correlation between the percentage of biased words in the 100 nearest neighbors of a target word and the bias in the target word itself, i.e., its projection onto the gender direction.
When judging the success of RLACE, it is clear that the probe and the indicators lead to markedly different conclusions. While the linear probe indi- cates the successful removal of the bias, it is only able to tell whether there is linear separation be- tween the potentially biased words. The indicators, on the other hand, are able detect biases that are expressed by means that are not necessarily linear.
5.4 Discussion
All in all, this test case demonstrates that indicators were able to expose information that linear probes could not. This should not surprise us as the prob- ing model is quite simple with the ability to detect only a very specific kind of separation.
Most probably RLACE was able erase linear gender bias since it was designed primarily for linear concept erasure. However, as mentioned in Section 2, using a more complex probe that could detect non-linear separation would have come at a price of opacity. The indicators, on the other hand, were able to surface biases that were expressed in non-linear terms while still being simple and transparent due to the lack an auxiliary model. In the next section present more compli- cated test case that involves the erasure of multiple properties, and therefore includes multiple probes whose results are less easily interpretable.
5.3 Results
Table I includes the results for both probes and in- dicators before and after the application of RLACE. It shows that the probe's classification accuracy drops from almost perfect accuracy before the erasure to to 48.7% after it, faring bit worse than a random guess. On the other hand, the indicators' drop in performance after the RLACE application is far less impressive, with minor drops of 0.04 in WEAT's d and 0.037 drop in the KNN-bias' Pear- son correlation. In absolute terms, the indicators
Case Study 2: Morphological Properties
In our second test case, we aim for representations that are ignorant of morphological features, such as tense, person etc. Using such representations can make the treatment of morphologically rich languages with many inflections per lemma simpli- fied to great extent, with models for downstream tasks not needing to worry about unseen inflections
*Caliskan et al. (2017) have a couple of more fixed sets: art related words for female bias and mathematics und science for male. However, there was no hias detected with respect to these sets even before dehiasing.

---

(Gong et al., 2018: Czarnowska al., 2019). As success criteria for removing morphological knowl- edge from word representations, we utilize probes and a novel indicator.
'hhe representations could encode properties using overlapping entries, which upon removal in prior iterations, cannot be relied on. This fact also leads this variant to be order dependant - it is unclear how would the removal of property P1 affect the removal of property P2 and vise-versa. The non- regressive variant, on the other hand, is indifferent to order, but multiplying the projection matrices one after another could be numerically unstable.
To this end, we extend an existing concept era- sure method, Iterative Nullspace Projection (INLP, Ravfogel et al., 2020), to accommodate the erasure of multiple properties. We propose two possible, and novel, extensions and test these alternatives for their efficacy using standard probes, and using a novel indicator task that we design, which indi- cates on the presence of morphological and lexical semantic information simultaneously.
In order to decide between the variants we test both of them using both a probe and an indicator task we designed, presented in the next section.
6.2 The Morpho-Semantic Indicator
6.1 Iterative Nullspace Projection
Consider the quadruple (walked, hiked, strolling, bumped). Semantically, the words (walked, hiked, strolling) are related and (bumped) is the seman- tic outlier. Morphologically, the words (walked, hiked, bumped) are related as they share the eed inflectional form, and (strolling) gerund is the mor- phological outlier. We follow this general scheme of quadruples with both a morphological and a semantic outliers to create a novel indicator task: double edged outlier detection (DEOD).
INLP is a linear concept crasure method, with the objective that no linear classifier could do better than guess. The method constructs a subspace ig- norant of property P (e.g tense) by iteratively training linear classifiers predicting p P (e.g past) and projecting the representations unto their nullspace. The process sets to eliminate the entries used to predict the features by various classifiers. For more details see (Ravfogel et al., 2020). While INLP is designed for a single property, here we would like to use for a complete feature bundle.
Outlier detection is an intrinsic method proposed by Hemacho-Colladee and Navigli (2016) with the objective of identifying single outlier in a set of words. In our case the set contains two outliers, semantic and the other morphological. By predict- ing single outlier from each set, models take part in a zero sum game, and we get to evaluate which of the two dimensions of meaning is prioritized in the representations.
Multi-Class Setting INLP is intended to remove a single property P which can correspond to any morphological property we choose. However, would like to erase as many properties possi- ble. We therefore extend INLP by iterating over a set of morphological properties by applying INLP for each property in a procedure we term Iterative Iterative Nullspace Projection or NN...
In the case of morphological erasure, this in- dicator task is also used to verify that the repre- sentations keep much of the lexical semantic information as possible. It important note that attempting a similar approach with probes, i.e., eliciting information of the interplay between two partially overlapping dimensions of word-level in- formation, is tough to envision.
Although intuitive, extending INLP in this man- ner could damage the embeddings beyond usability, each removed property reduces the dimensional- ity rank of the vectors. Moreover, repeated multipli- cations could be numerically unstable. Therefore we explore two possible variants of I'NLP.
In the first variant, Regressive, the representa- tions are projected after each property removal, before moving on to remove the next property. In the Non-Regressive variant, the representations are only projected after a projection matrix is calcu- lated for all properties, at the end of the procedure.
Task Definition Given a quadruple Q {wi, W2, W3, W4), the task is to identify (one of) the outliers in a procedure based on cosine simi- larity. Our working hypothesis is that embedding spaces are capable to encode both lexical semantics and morphology to some extent and the interplay between them is affecting the decisions taken by the model when picking an outlier.
Choosing Variant At first glance, the differ- ence between the variants may seem small, but manipulating the input representations after each iteration, as done in the regressive variant, could af- fect the removal of the next properties significantly.
We use the two metrics termed hard and soft out- lier detection, proposed by Camacho Collados
Camacho-Collados and

---

Navigli (2016). Hard outlier detection is measured with accuracy. It is the percentage of quadruples in which the semantic (morphological) outlier was chosen. Soft outlier detection, measured with Out- lier Position Percentage (OPP), is based on the position of the true outlier in the ranked quadru- ple and not only reflecting whether it was ranked first. See Camacho-Colladte and Navigli (2016) for further details about these metrics.
was used both for determining what morpholog- ical properties are to be erased with I-NLP and for training the probes over the final projected repre- sentations. The calculation of projection matrices as part of N'LLP is done only using the train and dev sets, and the reported scores of the final probes are calculated over test set of sentences that were not seen during training.
The linear probing model for each morpholog- ical category is multi-class version of SGD linear classifier (Pedregosa et al., 2011).
Data Generation In order to generate data for the DEOD task we used UniMorph (Batsuren et al., 2022) to determine what words are of the same morphological category, and FastText embeddings (Bojanowski et al., 2017), assess semantic simi- larity between words.
Language Models
For the contextualized repre- sentations used the following models:
Hebrew: AlephBERTGimme (Guetta et al., 2022)
Specifically, from UniMorph we sampled word with a certain morphological characterization T. e.g., walked which is characterized as a past tense form. Then, using the word embeddings, we add the most similar words with the same morphologi- cal characterization, for example hiked and strolled, and one dissimilar word as the semantic outlier, like bumped. The morphological outlier then created by inflecting one of the semantically similar words to another morphological category, for example we could replace strolled with strolling. The result is the quadruple (walked, hiked, strolling, bumped). This procedure is repeated until enough quadru- ples are sampled. Since this method is language agnostic, it can be applied for different languages.
English: bert-base-uncased (HuggingFace im- plementation; Wolf et al., 2020)
Spanish: Beto (Cañete et al., 2020)
Since the indicator requires words to be represented out of context, a single representation per word was obtained by averaging the contextualized represen- tations of all its occurrences in the UD sentences.
Metrics For evaluating the PNLP variants using probes we report the classifier Fl score averaged over all morphological properties. We expect classi- fiers to perform poorly on representations that were obtained by good 12NLP variant. The evaluation of the I'NLP variants using the DEOD indicator task is done using the task's own metrics: accuracy for hard outlier detection and OPP for soft outlier detection. We expect a successful I2NLPP to pre- vent the detection of the morphological outlier. On the other hand, we expect higher performance in detecting the semantic outlier.
All in all, we created 2,000 quadruples for each language. Since both UniMorph and FastText cover many languages, the data generation method is easily applicable to many languages.
6.3 Experiments
The Languages We experimented with three lan- guages: English, Spanish and Hebrew, these lan- guages are typologically different both in the num- ber of morphological features they express and in the means they employ to express them; with one isolating, one suffixing and one templatic language.
Baseline and Skyline For both the probe and the indicator we provide as a baseline the respective metric assessed over the original representation. without application of any I2NLPP variant.
As a skyline we evaluate the respective metric as it would be in an ideal situation, where all mor- phological information is completely erased from the representations. For the probe we provide the averaged F1 expected from classifier that ran- domly guesses the morphological label. For the indicator we replaced all word vectors with those of the respective lemmas to ensure erasure of all morphological information.
Probes In order to ascribe morphological infor- mation to words in their context we used the mor- phological layer of Universal Dependencies (UD; Nivre et al., 2020) as our data Set. This data
"Static representations are used instead of contextualized ones since we need for the DEOD task a representations complete words out context.
PI cases where UD hreaks white-spaced words to multiple units we merged the morphological tags of all subwords.

---

Lang
Before
Reg
Non-Reg
Skyline
outlier detection. This the opposite of what is expected of good morphological erasure method. This picture is reinforced by the results for the semantic outlier detection in Table 3b, showing that the non-regressive variant is also better at pre- serving the semantic affinities of the space. Its results are impressively similar those of the sky- line, while the regressive variant hinders the per- formance even comparing to the baseline. This is in stark contradiction to the results of the probes on the same projected spaces, that pointed to the regressive 12NLP variant as the superior one.
Heb
94.52
61.89
79.62
40.28
Spa
92.99
50.32
76.88
33.89
Eng
92.80
46.39
64.24
37.5
Table 2:
Before and after projection in average F1 scores on classifying Tense, Gender, Number, Person, Case and VerbForm. The better N-NLPP variant prevents classification so lower is better. The regressive variant appears as the better method.
Morphological Non-Reg
Lang
Before
Reg
Skyline
Hard
Outlier Detection
Heb
8.91
15.22
3.49
0.36
Spa
14.37
17.64
4.72
0.36
Eng
9.40
20.20
3.60
0.16
6.5 Discussion
Soft Outlier Detection
Heb
56.74
56.20
49.73
26.97
As with the previous test case, here as well probes and indicators point in polar opposite directions, with probes giving the advantage the regressive 12 while the DLOD indicator showing that the non-regressive is the superior variant. Also simi- to the previous test case the probes' lack of ability to detect non-linearly expressed information. However, unlike the gender bias case, the probes are still doing better than guessing. since even the results for the regressive variant show worse era- sure than the skyline. That is to say that the probes were still able to detect residual linearly expressed morphological information.
Spa
61.26
56.95
52.44
27.71
Eng
58.59
59.50
50.00
26.77
(a) Morphological outlier detection
Semantic
Lang
Before
Reg
Non-Reg
Skyline
Hard Outlier Detection
Heb
74.41
49.56
84.11
85.34
Spa
70.85
44.82
86.58
86.28
Eng
77.45
41.10
88.70
87.76
Soft Outlier Detection
Heh
91.45
78.43
95.13
96.06
Spa
90.13
75.99
95.73
96.53
Eng
92.27
74.36
96.3
96.94
(b) Semantic outlier detection
The indicator task, on the other hand, is much more complex than the indicators used for gender bias detection. It is not restricted to linear separa- tion and it indicates simultaneously on two types of information in the representations: morpholog- ical and lexical semantics. The indicator allows us to verify that the non-regressive I'NLP crases better the information is supposed to erase, while preserving better the information it is supposed to preserve. The indicator's double purpose allows evaluation of more aspects of the space and exam- ine the I'LLP variants not only from the morpho- logical angle, and for this reason we argue in favor of preference to the non non-regressive variant.
Table 3: Before and after projection indicator results. While Regressive distorts the embedding space com- pletely, Regressres achieves our goal reducing the morphological affinity the space while not harm- ing the lexical semantics encoded.
6.4 Results
Table 2 shows the average F1 scores of probes in predicting morphological features for both 12NLP variants. It is apparent that the regressive N2P is more effective than the non-regressive in removing morphological properties.
Table shows the results for the indicator task. The morphological outlier detection results in Ta- ble 3a show that the non-regressive I'NLP variant is better in removing morphological properties as the performance after the projection is worse both in term of hard and soft detection. Note that when compared to the performance over the original vec- tors, applying the regressive I'NLP makes it easier to spot the morphological outlier, in terms of hard
7 Conclusions
In this work we compare probes that do and do not require training an auxiliary classification model, offering indicator tasks as a term that covers the latter kind. We show, in different test cases, that although it is tempting to think of probes and indicators as complementary in understanding the inner structure of embedding spaces, sometimes the methods result in contradictory conclusions.
"For soft outlier detection, the average OPP of guess is 50% so the table points to an almost complete erasure.

---

We claim that the inherent problems with probes and the flexibility of indicators should lead to pri- oritizing the latter in interpreting the results of property removal procedures. Nevertheless, probes are of course still useful in inspecting embedding spaces, but their results should not be overstated and they should be used only when interested in the specific insights they hold, e.g., the existence of linear separation in case of linear probes.
larity not in absolute terms but in relation to sim- ilarities between other pairs of words (Camacho- Collados and Navigli, 2016). Another problem of intrinsic evaluation is the lack of guaranteed cor- relation with performance on downstream tasks (Chiu et al., 2016). However, this problem exists for probes as well (Conneau et al., 2018).
Lastly, it is unelear whether indicator tasks in general, and our DEOD task in particular, can de- tect linear separability in the representations, which is the most common use of probes. For this reason we do not advocate for the abandonment of probes, but rather for limiting their usage to specific goals, like detection of linear separation.
Although the design of indicators requires more creativity and result in tasks that are mostly tailored for specific kinds of questions under investigation, we assert that the advantage they give in better understanding the information, that does and does not exist in word representations, is worth the effort. We therefore conclude that indicators should at the very least be presented alongside probes when examining embedding spaces.
Acknowledgements
We thank Shauli Ravfogel and an anonymous re- viewer (VEsr in openreview) for insightful com- ments and fruitful discussions. This research was funded by the European Research Council (ERC- StG grant number 677352), the Israeli Ministry of Science and Technology (MOST grant number 3- 17992), and the Israeli Innovation Authority (IIA KAMIN grant), for which we are grateful.
Limitations
The use of indicator tasks, as presented in this pa- per, solves many problems existing in the current use of probes to query linguistic information from embedded representations. However, there are of course issues that remain unsolved and may require further research.
References
For example, the interpretability of the absolute results is still unclear with indicators as with probes. Suppose we ran a probing experiment or an indicator task and got 87%. Is it good? How could we tell? Could we safely declare the information exists in the representation? (Belinkov, 2022)
Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg, 2016. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks.
Guillaume Alain and Yoshua Bengio. 2017. Under- standing intermediate layers using linear classifier prohes.
Another issue that may arise with the usage of indicator tasks is the lack of a simple recipe for the creation of such tasks. Despite their disadvantages, the design of probing tasks is simple and merely requires picking a target property and a classifier. On the other hand, the design of indicator tasks remains a tougher mission. In this paper we pointed to similarities between vectors as the main source of information for indicator tasks, and indeed from it can be derived many tasks such as clustering and comparison to other sources of similarity. However, the details may require some maieor-made decisions.
Khuyagbaatar Batsuren, Omer Goldman, Salam Khal- ifa, Nizar Habash, Witold Kieraš, Gábor Bella, Brian Leonard, Garrett Nicolai, Kyle Gorman, Yusti- nus Ghanggo Ate, Maria Ryskina, Sabrina J. Mielke, Elena Budianskaya, Charbel El- EhaKsiissi, Tiago mentel, Michael Gasser. William Lane. Mohit Raj. Matt Coler, Jaime Rafael Montoya Samame, De- lio Siticonatzi Camaiteri. Benoît Sagot, Esaú Zu- macta Rojas, Didier López Francis, Arturo Once- vay, Juan López Bautista, Gema Celeste Silva Vil- legas, Lucas Torroba Hennigen, Adam Ek, David Guriel, Peter Dirix, Jean-Philippe Bernardy, An- drey Scherbakov, Aziyana Bayyr-ool, Antonios Anastasopoulos, Roberto Zariquiey, Karina Sheifer. Sofya Ganieva, Hilaria Cruz, Ritván Karahóga, Stella Markantonatou, George Pavlidis, Matvey Plu- garyov, Elena Klyachko, Ali Salehi, Candy An- gulo, Jatayu Baxi, Andrew Krizhanovsky, Natalia Krizhanovskaya, Elizabeth Salesky, Clara Vania, Sar- dana Ivanova, Jennifer White, Rowan Hall Maud- slay, Josef Valvoda, Ran Zmigrod, Paula Czarnowska,
In addition, as conceptual continuation of in- trinsic evaluation methods, the indicator DEOD task may suffer from the same problems that haunted intrinsic tasks, mostly with regards of the definition of similarity (Faruqui et al., 2016). How- ever, our indicator is based on the outlier detection task that tended to these issues by examining simi-

---

Irene Nikkarinen, Aelita Salchak, Brijesh Bhatt, Christopher Straughn, Zoey Liu, Jonathan North Washington, Yuval Pinter: Duygu Ataman, Marcin Wolinski, Totok Suhardijanto, Anna Yablonskaya, Niklas Stoehr, Hossep Dolatian, Zahroh Nuriah, Shyam Ratan, Francis M. Tyers, Edoardo M. Ponti, Grant Aiton, Aryaman Arora, Richard J. Hatcher, Ritesh Kumar, Jeremiah Young. Daria Rodionova, Anastasia Yemelina, Taras Andrushko, Igor Marchenko, Polina Mashkovtseva, Alexandra Serova, Emily Prud hommeaux, Maria Nepomni- ashchaya, Fausto Giunchiglia, Fleanor Chodroff, Mans Hulden, Miikka Silfverberg, Arya D. Me- Carthy, David Yarowsky, Ryan Cotterell, Reut Tsar- faty, and Ekaterina Vylomova. 2022. Unimorph 4.0: Universal morphology-
Long Papers), pages 2126-2136, Melbourne, Aus- tralia. Association for Computational Linguistics.
Paula Czamowska, Sebastian Ruder, Edouard Grave, Ryan Cotterell, and Ann Copestake. 2019. Don't forget the long tail! a comprehensive analysis of morphological generalization in bilingual lexicon in- duction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IICNLL) pages 974-983, Hong Kong, China, Association for Com- putational Linguistics.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lec, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language under- standing. CoRR, abs/1810.04805.
Yonatan Belinkov. 2022. Probing classifiers: Promises, shortcomings, and advances. Computational Linguis- tics, 48( 48(1)207-2
Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. 2021. Amnesic probing: Behavioral expla- nation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 9:160- 175.
Yonatan Belinkov and James Glass. 2019. Analysis
Methods in Neural Language Processing: A Survey.
Transactions of the Association for Computational Linguistics, 7:49-72.
Manaal Faruqui, Yulia Tsvetkov, Pushpendre Rastogi. and Chris Dyer. 2016. Problems with evaluation of word embeddings using word similarity tasks. In Pro- ceedings of the 1st Workshop on Evaluating Vector- Space Representations for NLP, pages 30-35, Berlin, Germany, Association for Computational Linguistics.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information.
Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. 2016. Man to computer programmer as woman is to homemaker? debiasing word embeddings.
Hila Gonen and Yoav Goldberg. 2019. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word emheddings but do not remove them. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Humun Language Technologies, Volume 1 (Long and Short Papers), pages 609-614 Minneapolis, Minnesota. Association for Computa- tional Linguistics.
Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-lik biases. Science, 356(6334)
José Camacho-Collados
Collados and Roberto Navigli. 2016. Find the word that does not belong: A framework for an intrinsic evaluation word vector representations. In Proceedings of the 1st Workshop on Evaluating Vector- Saator-Spaceeaa Representations for NLP, pages 43-50, Berlin, Germany. Association for Computational Lin- guistics.
Chengyue Gong, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie- Yan Liu. 2018. Frage: Frequency-agnostic word representation. Advances neural information processing systems, 31.
José Cañete, Gabriel Chaperon, Rodrigo Fuentes, Jou- Hui Ho, Hojin Kang, and Jorge Pérez. 2020. Span- ish pre-trained bert model and evaluation data, In PML4DC at ICLR 2020.
Eylon Guetta, Avi Shmidman, Shaltiel Shmidman. Cheyn Shmuel Shmidman, Joshua Guedalia, Moshe Koppel, Dan Bareket, Amit Seker, and Reut Tsar- faty. 2022. Large pre-trained models with extra-large vocabularies: A contrastive analysis of hebrew bert models and a new one to outperform them all.
Billy Chiu, Anna Korhonen, and Sampo Pyysalo. 2016. Intrinsic evaluation of word vectors fails predict ex- trinsic performance. In Proceedingy of the 1st Work- shop on Evaluating Vector-Space Representations for NLP. pages 1-6, Berlin, Germany. Association for Computational Linguistics.
John Hewitt and Percy Liang. 2019. Designing and in- terpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Nut- ural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNIP", pages 2733-2743, Hong Kong, China. Association for Computational Linguistics.
Alexis Conneau, German Kruszewski, Guillaume Lam- ple, Loic Barrault, and Marco Baroni. 2018. What you can cram into a single $&1#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the As- sociation for Computational Linguistics ( (Volume 1:
John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word represen- tations. In Proceedings of the 2019 Conference of

---

the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4129-4138, Minneapolis, Minnesota. Association for Computational Linguistics.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013, Efficient estimation of word representa- tions in vector space.
Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. StercoSet: Mcasuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pupers), pages 5356-5371, 1, Online. Association for Computational Linguistics.
Felix Hill, Roi Reichart, and Anna Korhonen. 2015. SimLex-9999 Evaluating semantic models with (gen- uine) similarity estimation. Computational Linguis- tics, 4 (14):6665-695
Phu Mon Htut, Jason Phang, Shikha Bordia, and Samuel R. Bowman. 2019. Do attention heads in bert track syntactic dependencies?
Timothy Niven and Hung-Yo Yu Kao. 2019, Probing neu- ral network comprehension of natural language argu- ments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pages 4658 1668,4664, Florence, Italy. Association for Compu- tational Linguistics.
Alexander Immer, Lucas Torroba Hennigen, Vincent Fortuin, and Ryan Cotterell. 2022. Probing as quanti- fying inductive bias. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 1839- 1851, Dublin, Ireland. Association for Computational Linguistics.
Joakim Nivre, Marie-Catherine de Marneffe, Filip Gin- ter, Jan Hajië, Christopher D. Manning. Sampo Pyysalo, Sehastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal dependencies v2: An evergrowing multilingual treebank collection.
Are Köhn. 2016. Evaluating embeddings using syntax- based classification tasks proxy for parser per- formance. In Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NIP,, pages 67-71, Berlin, Germany. Association for Com- putational Linguisties.
Pedregosa, G. Varoquaux, A. Gramfort, Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duch- esnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830.
Nikolaus Kriegeskorte, Marieke Mur, and Peter A Ban- dettini. 2008 Representational similarity analysis- connecting the branches of systems neuroscience. Frontiers in systems neuroxcience, page 4.
Oner Levy and Yoav Goldberg. 2014. Linguistic regu- larities in sparse and explicit word representations. In Proceedings of the Eighteenth Conference on Compu- tational Natural Language Learning, pages 171-180, Ann Arbor, Michigan. Association for Computational Linguistics.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018 Deep contextualized word repre- sentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computa- tional I. Linguistics.
Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019a. Lin- guistic knowledge and transfcrability of contextual representations. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, Volume I (Long and Short Pa- pers). pages 1073-1094, Minneapolis, Minnesota. Association for Computational Linguisties.
Tiago Pimentel, Naomi Saphra, Adina Williams, and Ryan Cotterell. 2020a. Pareto probing: Trading off accuracy for complexity. In Proceedings of the 2020 Conference Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 3138-3153, On- line. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining ap- proach. ArXiv, abs/1907.11692.
Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. 2020b. cetemattoonthheorete probing for linguistic structure. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 4609 4622, Online. Association for Computa- tional I Linguistics.
Amil Merchant, Elahe Rahimtoroghi, Ellie Pavlick, and Ian Tenney. 2020. What happens to BERT embed- dings during fine-tuning? In Proceedings of the Third BlackboxNLP Workshop on Analyzing and In- terpreting Neural Networks for NLP, pages 33-44, Online. Association for Computational Linguistics.
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg, 2020. Null out: Guard- ing protected attributes by iterative nullspace projec- tion.

---

Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan Cotterell. 2022. Linear adversarial concept erasure.
Elena Voita and Ivan Titov. 2020. Information-theoretic probing with minimum description length. In Pro- ceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP), pages 183- 196, Online. Association for Computa- tional Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Seao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: state-of-thh-arr natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Zhiyong Wu, Chen, Ben Kao, and Qun Liu. 2020. Perturbed masking: Parameter-free probing for ana- lyzing and interpreting bert.
Amir Zeldes. 2017. The GUM corpus: Creating mul- tilayer resources the classroom. Language Re- sources and Evaluation, 5163)581-61..
Kun Zeng, Yibin Xu, Ge Lin, Likeng Liang, and Tiany- ong Hao. 2021. Automated classification of clinical trial eligibility criteria text hased on ensemble learn- ing and metric learning. BMC Medical Informatics and Decision Making. 21.
Jieyu Zhao, Yichao Zhou, Zeyu Li. Wei Wang, and Kai- Wei Chang. 2018. Learning gender-neutral word embeddings.
Yichu Zhou and Vivek Srikumar. 2021. DirectProbe: Studying representations without classifiers. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5070-5083, Online. Association for Computational Linguistics.
Yichu Zhou and Vivek Srikumar. 2022. A closer look how fine-tuning changes BERT, In Proceedings gs of the 60th Annual Meeting of the Association for Computaaiooal Linguistics (Volume 1: Long Papers), pages 1046-1061 Dublin, Ireland. Association for Computational Linguistics.


Reviews:
[{"review_id": "l5MJOPJcWM", "reviewer": "Reviewer_VEsr", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your review, your expertise in this area is apparent. \nWe were especially happy to read that you found our analytical approach clever and “an enlightening guide for selecting representation analysis methods”.\n\n\nRejection reasons #1 and #3 are complementary and advocate for the examination of complex probing models together with sophisticated probing methods. However, as you correctly noticed, and as we explained for control tasks and amnesic probing (l156-l173), these complex methods do not solve the trade offs of probing – they only mitigate them.\n\n\nFor this reason, our paper suggests getting rid of the trade offs in probing altogether, by using indicators instead of probes. This is not to say that complex probing methods do not hold value, and future research should certainly pit them against indicators. But our work was aimed at presenting a simple method that solves a complex problem, and we think that its simplicity makes it superior over complex methods that only mitigate the pitfalls of probing.\n\n\nRegarding the move of focus on ease of extraction, we consciously focused on checking the existence of a property in the representations or a lack thereof — rather than assessing the ease of its extraction. This is especially warranted in cases of completely unwanted properties, such as the case of gender bias.\n\n\nAll in all, we find this discussion enlightening and we believe that others in the community will benefit from it as well. We therefore hope that you could see this discussion as a strength rather than a weakness of our work. We will of course  elaborate on the points made here in this discussion, together with the mentioned methods, in the camera ready version, and we’d be happy to continue the discussion either anonymously here or acknowledgedly if the paper gets accepted for publication.\n\n\nRegarding the exclusion of experiments over downstream tasks (rejection reason #2), although this is a good idea in general, it was left out of scope of this work since we had no space for it and we found it less important for our main point. The main reason for that was that we were concerned mostly with the existence of the properties in the representations and not their effect on applications. This is also the reason we experimented with erasure methods as test cases.\n\n\nRe: “It would be interesting to observe the impact of I2NLP on each morphological feature, even if presented in the appendix.” \nThis is a great idea and we will provide such breakdown in an appendix in the camera-ready.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors introduce a compelling topic. If complemented with more recent probing experiments, this paper could become an enlightening guide for selecting representation analysis methods.\n2. Utilizing a concept erasure technique on a linguistic feature and subsequently comparing the two analytical paths is a clever approach to unveil the contrasting results.", "reasons_to_reject": "1. This paper seems to face either an issue of overclaiming or lack of experiments. The concern arises from the fact that all case studies in this paper solely compare a simple linear classifier probe with task-tailored indicator tasks. To address this, the paper should either incorporate more complex and recent probing methods that consider their complexity in the results (e.g., Pareto probing or MDL) or temper its claims by focusing solely on examining general probes with **linear** auxiliary classifiers.\n2. This paper lacks a comparison between the two analysis methods concerning the model's final output. While it reveals contradictory results between the classifier-based probing task and the indicator task, the key question is how these results relate to the model's practical use. For example, in the gender debiasing section, we observed a major drop in linear probe accuracy when using RLACE, while other indicators showed only a minor decline. However, determining which approach holds greater significance in the output, such as when filling a masked token, remains uncertain.\n3. The Probing Conundrum appears to be resolved or at least mitigated by examining the extent to which linguistic features are well-encoded in the representations. Classifier-based probing methods become informative when incorporating model complexity measures (Considering the probing result + the probe's complexity). As highlighted by Pimentel et al. (cf. Missing references 1), the focus should be on the ease of extracting linguistic features from the representations, as seen in certain approaches like Pareto probing or MDL (cf. Missing references 2).", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper centers on contrasting probes that necessitate training a classifier with those that solely analyze the properties of the representations without any training. Based on the perplexity of classifier-based probes, which might possess their own inherent blackbox characteristics, the authors assert that such probes do not distinctly elucidate representational features. As an alternative, they introduce the term \"indicator tasks\" to describe analyses that reveal a property without necessitating model training through the probing task itself. Using concept erasure, the authors present two case studies in which a probe with a linear auxiliary classifier yields conflicting outcomes when compared to one or more indicator tasks. Ultimately, they advocate prioritizing indicator tasks when examining linguistic properties in representations to draw more plausible conclusions.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- It would be interesting to observe the impact of I2NLP on each morphological feature, even if presented in the appendix.", "Missing_References": "1. Information-Theoretic Probing for Linguistic Structure, Pimentel et al,. ACL 2020\n2. Information-Theoretic Probing with Minimum Description Length, Voita and Titov, EMNLP 2020"}, "extra_text": {}}, {"review_id": "AcfN4AU12E", "reviewer": "Reviewer_p7Cp", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your thorough review, typo fixes and rephrasing suggestions. We are glad you saw the added value of our second test case, we agree that examining morphology is an interesting subject in and of itself and is of value to the community.\n\n\nWe were also happy to read that most points mentioned under “reasons to reject” are fairly minute weaknesses or rephrasing suggestions that will be addressed in the camera-ready. Below are specific answers to your most significant points:\n\n\nRe: “l194: I don't find the explanation of why they're called indicator tasks convincing... - I'd rather extend the term 'probing task' so that it's more specific” \nIndicators are just an umbrella term for probing methods that do not involve a trainable classifier. It’s mainly used to avoid calling them “probes that do not involve a trainable classifier” as the term “probe” per se is too broad and overloaded with the assumption of a trainable classifier.\nRe: “I'd like to see examples here: one case where the bias remains after RLACE (and is detected by the indicators and ignored by the probe) and one where the bias is removed (and either detected/ignored by one of the evaluation methods).”\nThanks for this suggestion. It will indeed be very helpful to demonstrate the details of the differences between the probes and the indicators. we will provide such analysis in the camera-ready version.\n\n\nRe: “l288: discuss early in the paper why you picked those two use cases”\nThis is a valuable suggestion and we will follow up on it in the camera ready. In brief, we chose the test cases that are the most dissimilar: one deals with a binary property that is of social importance that was heavily discussed in the literature, and the other examines multiple non-binary properties that are linguistic in nature and that the application of erasure methods to them requires innovation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I particularly like the second use case where the extent of morphological information in word embeddings is investigated. I think it's of value for the EMNLP audience. ", "reasons_to_reject": "l33f: \"stark improvement in the performances of any NLP task they have been applied to\" --> that statement is wrong and far too general\n\nl72: describe here why you call them indicator tasks early on\n\nl194: I don't find the explanation of why they're called indicator tasks convincing...  - I'd rather extend the term 'probing task' so that it's more specific\n\nl288: discuss early in the paper why you picked those two use cases\n\nl196: \"intervention or entanglement of the task itself with the vectors it is supposed to study\" -- unpack, not clear how that statement captures what you do\n\nl254: integrate Section 4 in Section 5\n\nSection 5.3: \" All in all, this test case demonstrates that indicators were able to expose information that linear probes could not.\" --> I'd like to see examples here: one case where the bias remains after RLACE (and is detected by the indicators and ignored by the probe) and one where the bias is removed (and either detected/ignored by one of the evaluation methods). \n\nl462: even though I can see where the authors are coming from, 'Iterative Iterative Nullspace Projection' as a term is weird...\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes the notion of 'indicator tasks' as a contrasting paradigm to 'probing tasks', subsuming all non-trainable tasks for querying embedding spaces. ", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "See above. Plus typos here and there, e.g.,\n\nl17: this kind of tasks --> these kinds of tasks\n\nl303: methods, that incorporate --> remove comma\n"}, "extra_text": {}}, {"review_id": "WIiV0AB5mI", "reviewer": "Reviewer_8oKZ", "summary": "", "questions": "A) Would it be possible to present some qualitative analysis on the results to understand better why probe is worse than random and the indicators are not that bad in the results of Case Study 1? \n\nB) Could that be done for Case Study 2 as well?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your review and for finding this work interesting for the NLP community.\n\n\nRe: “Sometimes it is difficult to follow what has been done Maybe the analysis of the results could benefit with something more visual, such as the illustration of the errors to understand in which cases probes fail to capture gender biases, specially after using RLACE”\n\n\nWe will restructure the experimental setup of both test cases to clarify them.\nWe will also expand on why we chose these test cases specifically, as well as the choice of languages and analyze results in light of the different languages and the morphological properties separately.\nWe will also provide a visualization in an appendix for the gender bias test case. It will be a visualization of gender-associated professions on a graph ranging from male to female biased, before and after applying RLACE.\n\n\nQuestions A & B:\nA qualitative analysis is a good idea. We will include an error analysis in the camera ready. We will also look for explicit examples where bias remains in the representations and probes fail to capture it, cases in which these examples will have shared characteristics will be reported in the camera ready.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Evaluation of embeddings, specially towards understading bias, is of interest of the NLP community\nThey proposed an alternative to probes, which presented better results in the paper", "reasons_to_reject": "The paper could be clearer. Sometimes it is difficult to follow what has been done\nMaybe the analysis of the results could benefit with something more visual, such as the illustration of the errors to understand in which cases probes fail to capture gender biases, specially after using RLACE", "questions_to_authors": "A) Would it be possible to present some qualitative analysis on the results to understand better why probe is worse than random and the indicators are not that bad in the results of Case Study 1? \n\nB) Could that be done for Case Study 2 as well?", "paper_topic_and_main_contributions": "The authors present the 'indicator tasks' as an alternative to probes, to evaluate embedding spaces. They show that although indicator tasks can be more laborious to be created, they can more properly capture problems such as biases in embeddings spaces, specially after so linear transformation has been already applied on the embeddings.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper makes an argument for eschewing probing tasks in favor of using behavioral evaluations (“indicator tasks”) without a trained probe component for understanding the linguistic structure of neural models, largely upon the rationale that linear probes show random chance after linear concept erasure, while behavioral evaluations demonstrate that the model still behaves as if it has some knowledge of the linguistic property of interest even after linear concept erasure. The overall claim, as I understand it from the paper and author discussion, is that the simplicity of behavioral evaluations is preferable over changing the probing methodology to add complexity through controls or sample efficiency arguments.

This paper warranted discussion among the reviewers and AC. I think that the methodological problems of understanding unsupervised learned structure are nuanced, and the reviewers seem to believe this work contributes to the discussion thereof. However, I agree with reviewer VEsr that considerable research has been done in this discussion that this work does not engage with properly. The issue seems to be in whether the experiments in this paper demonstrate an issue not already handled by other advancements in analysis methodology.

From my own perspective, I’d ask the authors to consider discussing whether it really is the case that one of probing or indicator tasks must take precedence over the other; they seem just to be testing rather different hypotheses about the model – one about linear extractivity of interesting properties, the other about behavior, so the fact that they lead to differing results isn’t necessarily a huge methodological problem.
Decision: Accept-Findings