Can LLMs Facilitate Interpretation of Pre-trained Language Models?
Authors: Basel Mousi
Nadir Durrani
Fahim Dalvi
Venue: EMNLP 2023 Main (2023)
Keywords: Interpretation
explainability
Large Language Models
Neuron Analysis
TL;DR: Large Language Models can facilitate the interpretation of deep NLP models

Abstract: Work done to uncover the knowledge encoded within pre-trained language models rely on annotated corpora or human-in-the-loop methods. However, these approaches are limited in terms of scalability and the scope of interpretation. We propose using a large language model, ChatGPT, as an annotator to enable fine-grained interpretation analysis of pre-trained language models. We discover latent concepts within pre-trained language models by applying agglomerative hierarchical clustering over contextualized representations and then annotate these concepts using ChatGPT. Our findings demonstrate that ChatGPT produces accurate and semantically richer annotations compared to human-annotated concepts. Additionally, we showcase how GPT-based annotations empower interpretation analysis methodologies of which we demonstrate two: probing frameworks and neuron interpretation. To facilitate further exploration and experimentation in the field, we make available a substantial ConceptNet dataset (TCN) comprising 39,000 annotated concepts.

Can LLMs Facilitate Interpretation of Pre-trained Language Models?
Basel Mousi
Nadir Durrani Fahim Dalvi
Qatar Computing Research Institute, HBKU, Doha, Qatar (bmousi, ndurrani,
, J edu. qa
Abstract
2019a; Belinkov et al., 2020), involve human-in- the-loop (Karpathy al., 2015; Kádár et al., 2017; Geva et al., 2021; Dalvi et al., 2022) to facilitate such an analysis. The use pre-defined linguistic concepts restricts the scope of interpretation to only very general linguistic concepts, while human-in- the-loop methods are not scalable. We circumvent this bottleneck by using large language model, ChatGPT, as an annotator enable fin-inained interpretation analysis.
Work done uncover the knowledge encoded within pre-trained language models rely on an- notated corpora or human-in-the- in-thh--ooop methods. However, these approaches are limited in terms of scalability and the scope of interpretation. We propose using large language model, Chat- GPT, an annotator to enable fine- fine-graine grained in- terpretation analysis of pre-trained language models. We discover latent concepts within pre-trained language models by applying ag- glomerative hierarchical clustering over con- textualized representations and then annotate these concepts using ChatGPT. Our findings demonstrate that ChaiGPT produces accurate and semantically richer annotations compared to human-annoten annotated concepts. Additionally, showcase how GPT-based annotations power interpretation analysis methodologies which we demonstrate two: probing frame- works and neuron interpretation. To facilitate further exploration and experimentation the field, we make available substantial Concept- Net dataset (TCN) comprising 39,000 anno- tated concepts."
Generative Pre-trained Transformers (GPT) have been trained on an unprecedented amount of tex- tual data, enabling them to develop a substantial understanding of natural language. As their capa- bilities continue to improve, researchers are finding creative ways to leverage their assistance for var- ious applications, such as question- prestion-anssering in financial and medical domains (Guo 2023), simplifying medical reports (Jeblick al., 2022). and detecting stance (Zhang et al., 2023). We carry out an investigation of whether GPT models, specif- ically ChatGPT, can aid the interpretation pre-trained language models (pLMs).
1 Introduction
A fascinating characteristic of neural language models is that words sharing any linguistic relation- ship cluster together in high-dimensional spaces (Mikolov et al., 2013). Recent research (Michael et al., 2020; Fu and Lapata, 2022; Dalvi et al., 2022) has built upon this idea by exploring representation analysis through latent spaces pre-trained mod- els. Building on the work of Dalvi al. (2022) we aim to identify encoded concepts within pre-trained models using agglomerative hierarchical clustering (Gowda and Krishna, 1978) on contextualized rep- resentations. The underlying hypothesis is that these clusters represent latent concepts, capturing the language knowledge acquired by the model. Unlike previous approaches that rely on predefined concepts (Michael al., 2020; Durrani et al., 2022) or human annotation (Alam et al., 2023) to label these concepts, we leverage the ChatGPT model.
A large body of work done on interpreting pre- trained language models answers the question: What knowledge learned within these models? Researchers have investigated the concepts en- coded in pre-trained language models by probing them against various linguistic properties, such as morphological (Vylomova et al., 2017; Belinkov et al., 2017a), syntactic (Linzen et al., 2016; Con- neau al., 2018; Durrani et al., 2019), and seman- tic (Qian al., 2016; Belinkov al., 2017b) tasks, among others. Much of the methodology used in these analyses heavily rely on either having access to an annotated corpus that pertains to the linguistic concept of interest (Tenney et al., 2019; Liu et al.,
'https: /ttp///eeorxxeeeeeee// qcri.ao.oooooooooooooo worg/projcctss// www.sfforrrr--cccept--oe

---

Concept Discovery
Concept Labeling
Sementic
Morphological
LLM
Latent Concept Analysis
Lexical Syntactic
Media
Cities
Uniterpretable
Medical Terms
Positive Words
Model
&
Emntions
Names
Human Annotation
Word Position
Cricket
Sports
Numbers
Scores
Data
Hypher ated
Enviromental Terms
Countries
Adjectives
Films
Adverbs
Verbs
Applications
Nouns
Title case
Gendier Pronours
Adverbs Timee
Representation
Analysis
Random
Animals
Neuron Analysis
Basketball Teams
Middle
Eastern
Pre-existing
Conflict
ontology taggers
Model Control
Figure ChatGPT as an annotator: Human annotation or taggers trained on pre-defined concepts, cover only a fraction of model's 's concept space. ChatGPT enables scaling up annotation to include nearly all concepts, including the concepts that may not have been manually annotated before.
Our findings indicate that the annotations pro- duced by ChatGPT are semantically richer and ac- curate compared to the human-annotated concepts (for instance BERT Concept NET). Notably, Chat- GPT correctly labeled the majority of concepts deemed uninterpretable by human annotators. Us- ing an LLM like ChatGPT improves scalability and accuracy. For instance. the work in Dalvi et al. (2022) was limited to 269 concepts in the final layer of the BERT-base-casee (Devlin et al., 2019) model, while human annotations in Geva et al. (2021) were confined to 100 keys per layer. Using ChatGPT, the exploration can be scaled to the entire latent space of the models and many more architectures. We used GPT to annotate 39K concepts across pre-trained language models. Building upon this finding, we further demonstrate that GPT-based annotations empowers methodologies in interpre- tation analysis of which we show two: i) probing framework (Belinkov et al., 2017a), ii) neuron anal- ysis (Antverg and Belinkov, 2022).
entities (e.g. NE:PER), investigate whether model distinguishes between male and female names or probing for "Cities in the southeastern United States" instead of LO:
Neuron Analysis Another line of work that we illustrate to benefit from GPT-annotated latent con- cepts the neuron analysis i.c. discovering neu- rons that capture a linguistic phenomenon. In con- trast to the holistic view offered by representation analysis, neuron analysis highlights the role of in- dividual neurons (or groups of them) within a neu- ral network (SSajaad et al., 2022). We obtain neu- ron rankings for GPT-annotated latent concepts using a neuron ranking method called Probeless (Antverg and Belinkov, 2022). Such fine-grained interpertation analyses of latent spaces enable us to see how neurons distribute in hierarchical ontolo- gies. For instance, instead of simply identifying neurons associated with the POS: Adverbs, we can uncover how neurons distributed across sub-concepts such adverbs of time (e.g.. "tomor- row"") and adverbs of frequency (e.g., "daily"). Or instead of discovering neurons for named entities (c.g. :E:PER), we can discover neurons that capture "Muslim Names" versus "Hindu Names".
Probing Framework We train probes from GPT- annotated concept representations to explore con- cepts that go beyond conventional linguistic cate- gories. For instance, instead of probing for named

---

eighth
second-bigge
second
Lorgest
biund-biggest
ggeett
Hotest-roted
underto took
Turched
Monama
Hamad
Ade!
dh posc Mansour
Tourth-Largest
Rekated ted weathered
Stufio Shaheen
second- Longest
secordd
ceded
ond reused ahaacched eaturing
ascending
Treqanted
de
ALwal
eede
Samir
Bandaauud
second-Leading
iing
Leading
ascaadd-to- toad--t--lest
third-most
corpoi an wutie
enr ched
ScaveEsed
Haf Baftarr Nahyon Lyad oosan 150
Digget-evee
second-mose
firth most
second-hi test
antting
higghstt Level
best hrpper
second-Seaditen est
roknar relaeee
conmuered red
superseded
Potateds
practised
crattedeeviee ced
Nat Ta Jume ira Abedi
Joberal
Muarman Sauaaad
Ri yad A
Than
Robbo
Foygad
Tharg
Saroa
most watched
lighst proftle
samp
al Spced Asstit
(a) Hyphenated Superlatives
(h) Verhs
(c) Arab Names
Figure 2: Illustrative Examples of Concept Learned in BERT: word groups organized based on (a) Lexical, (b) Parts
Speech, and (c) Semantic property
To summarize, we make the following contribu- tions in this work:
chical clustering is employed to cluster the words. Initially, each word forms its own cluster. Clusters are then merged iteratively based Ward's mini- mum variance criterion, using intra-cluster variance as dissimilarity measure. The squared Euclidean distance evaluates the similarity between vector representations. The algorithm stops when K clus- ters (encoded concepts) are formed, with K being
Our demonstration reveals that ChatGPT of- fers comprehensive and precise labels for la- tent concepts acquired within pLMs.
We showcased the GPT-based annotations of latent concepts empower methods interpre- tation analysis by showing two applications: Probing Classifiers and Neuron Analysis.
2.2 Concept Annotation
We release Transformers Concept-Net, ex- tensive dataset containing 39K annotated con- cepts to facilitate the interpretation of pLMs.
Encoded concepts capture latent relationships among words within a cluster, encompassing var- ious forms of similarity such as lexical, syntactic, semantic, or specific patterns relevant to the task or data. Figure 2 provides illustrative examples of concepts encoded in the BERTTbaase-assed model.
2 Methodology
We discover latent concepts by upplying clustering on feature vectors ($2.1). They are then labeled using ChatGPT ($2.2) and used for fine-grained interpretation analysis ($2.3 and 2.4). A visual representation of this process is shown in Figure 1.
This work leverages the recent advancements in prompt-based approaches, which are enabled by large language models such as GPT-3 (Brown et al., 2020). Specifically, we utilize zero-shot learning strategy, where the model is solely provided with a natural language instruction that describes the task of labeling the concept. We used ChatGPT with zero-shot prompt to anno- tate the latent concepts with the following followingsettinngss settings:
2.1 Concept Discovery
Contextualized word representations learned in pre- trained language models, can identify meaningful groupings based on various linguistic phenomenon. These groups represent concepts encoded within pLMs. Our investigation expands upon the work done in discovering latent ontologies in contextu- alized representations (Michael et al.. 2020; Dalvi al., 2022). At a high level, feature vectors (con- textualized representations) are first generated by performing a forward pass on the model. These representations are then clustered to discover the encoded concepts. Consider pre-trained model M with L layers: 1,, 11,12,..,,1, Using dataset D W1, W2, .... WN, we generate feature vectors D M z 2/....,2, 2.2 Agglomerative hierar-
Assistant is large language model trained by OpenAI
Instructions: Give a short and concise label that best describes the following list of words: ["word 1", "word 2", .... "word N"]
2.3 Concept Probing
Our large scale annotations of the concepts pLMs enable training probes towards fine-grained
'We experimented with several prompts, see Appendix A. for details.
22 denotes the contextualized representation for word w;

---

concepts that lack pre-defined annotations. For example we can use probing assess whether model has learned concepts that involve biases re- lated to gender, race, or religion. By tracing the input sentences that correspond to an encoded con- cept C' in a pre-trained model, we create annota- tions for particular concept. We perform fine- grained concept probing by extracting feature vec- tors from annotated data through forward pass on the model of interest. Then, we train a binary classifier to predict the concept and use the probe accuracy as a qualitative measure of how well the
3 Experimental Setup
Latent Concept Data We used a subset of the WMT News 2018 datasct, containing 250K ran- domly chosen sentences (R:M tokens). We set a word occurrence threshold of 10 and restricted each word type to a maximum of 10 occurrences. This selection was made to reduce computational and memory requirements when clustering high- dimensional vectors. We preserved the original em- bedding space to avoid information loss through di- mensionality reduction techniques like PCA. Con- sequently, our final dataset consisted of 25,000 word types, each represented by 10 contexts.
model represents the concept. Formally, given a set of tokens W = [w1, w2.... w E C, we generate feature vectors, sequence of latent rep- resentations: W- M, z' = . 2 for each word to; by doing a forward pass over 8j. We then train a binary classifier over the representations to predict the concept C' minimizing the cross-entropy loss:
Concept Discovery We apply agglomerative hi- erarchical clustering on contextualized feature vec- tors acquired through a forward pass pLM for the given data. The resulting representations in each layer are then clustered into 600 groups.*
Concept Annotation We used ChatGPT avail- able through Azure OpenAl service to carryout the annotations. We used a lemperature of O and a top p value of 0.95. Setting the temperature to 0 controls the randomness in the output and produces deterministic responses.
[(0) =
Σ log Po(c./w.)
exp is the probability that word x; is assigned concept c. We learn the weights / Dx using gradient descent. Here D is the dimensionality the latent representations zy and L is the size of the concept set which for a binary classifier.
where Po(cilz;)
exp(0.
Pre-trained Models Our study involved several 12-layered transformer models, including BERT- cased (Devlin et al., 2019), RoBERTa (Liu et al., 2019b), XLNet (Yang et al., 2019), and ALBERT (Lan et al., 2019) and XLM-RoBERTa (XLM-R) (Conneau et al., 2020).
2.4 Concept Neurons
An alternative area of research in interpreting NLP models involves conducting representation analysis at a more fine-grained level, specifically focusing on individual neurons. Our demonstration show- cases how the extensive annotations of latent con- cepts enhance the analysis of neurons towards more intricate concepts. We show this by using neu- ron ranking method called Probeless (Antverg and Belinkov, 2022) over our concept representations. The method obtains neuron rankings using an accu- mulative strategy, where the score a given neuron n towards a concept C' is defined as follows:
Probing and Neuron Analysis For each anno- tated concept, we extract feature vectors using the relevant data. We then train linear classifiers with a categorical cross-entropy loss function, optimized using Adam (Kingma and Ba, 2014). The training process involved shuffled mini-batches of size 512 and was concluded after 10 epochs. We used a data split of 60-20-20 for train, dev, test when train- ing classifiers. We use the same representations to obtain neuron rankings. We use NeuroX toolkit (Dalvi al., 2023a) to train our probes and neuron analysis.
R(n,C) pa()) -u(C)
*Dalvi et al. (2022) discovered that selecting K within the range 600 1000 struck satisfactory balance between the pitfalls of excessive clustering and insufficient clustering. Their exploration of other methods ELbow and Silhouette did not yield reliable results.
where p(C) is the average of all activations =(n, w). w C, and (() is the average of activations over the random concept set. Note that the ranking for each neuron n is computed independently.
htt / //wpraaaaaeaaaaeeeeeeeeeeeeaaeea mwcrosoftt. wwt.cocmm//neeeeeeee http///nus//rruucss/// rognitive-servicess//eeeeee seervices/oopaaaai -ttp://eraaaissiiceeeeee service

---

Q1
Acceptable
Unacceptable
Annotation SEM LEX Morph SYN Unint.
Majority Fliss Kappa
244
25
ChatGPT
85.5
1.1
3.0
X
3.3
0.71 ("Substantial agreement")
BCN
68.4
16.7
3.0
2.2
9.7
Q2
Precise
Imprecise
Table 3: Distribution (percentages) of concept types in ChatGPT Labels vs. Human Annotations: Semantic, Lexical, Morphological, Syntactic, Uninterpretable
Majority
181
60
Fliess Kappa
0.34 ("Fair agreement")
Table I:
Inter-annotator agreement with 3 annotators. Q1: Whether the label is acceptable or unacceptable? Q2: Of the acceptable annotations how many precise versus imprecise?
be 0.71 which is considered substantial according to Landis and Koch (1977). However, for Q2, the agreement was 0.34 (indicating fair level of agree- ment among annotators). This was expected due the complexity and subjectivity of the task Q2 for example annotators' knowledge and perspective on precise and imprecise labels.
Q3
1
Equal
BCN f
No Majority
Majority
82
121
58
Fliess Kappa
0.56 ("Moderate agreement")
Table 2:
Annotation for Q3 with 3 choices: GPT is better, labels are equivalent, human annotation better.
ChatGPT Labels versus Human Annotations Next we compare the quality of ChatGPT la- bels to the human annotations using BERT Con- cept Net, human annotated collection of la- tent concepts learned within the representations of BERT. BCN, however, annotated in the form of Concept Type:Concept Sub Type (e.g.. weeeeetttiinnnnn sport: ice_hockey) unlike GPT-based annotations that are natural language descriptions (e.g. Terms related to ice hockey). De- spite their lack of natural language, these reference annotations prove valuable for drawing compara- tive analysis between humans and ChatGPT. For Q3, presented humans with word cloud and three options to choose from: whether the LLM annotations are better, equalivalent, or worse than the BCN annotations. We found that ChatGPT out- performed or achieved equal performance to BCN annotations 75.5% cases, as shown in Table 2. The inter-annotator agreement for Q3 was found to be 0.56 which is considered moderate.
4 Evaluation and Analysis
4.1 Results
To validate ChatGPT's effectiveness annota- tor, we conducted human evaluation. Evaluators were shown a concept through word cloud, along with sample sentences representing the concept and the corresponding GPT annotation. They were then asked the following questions:
Q1: Is the label produced by CharGPT Ac- ceptable or Unacceptable? Unacceptable an- notations include incorrect labels or those that ChatGPT was unable to annotate.
Q2: Ifa label is Acceptable, is Precise or Imprecise? While a label may be deemed acceptable, it may not convey the relationship between the underlying words in the concept accurately. This question aims measure the precision of the label itself.
4.2 Error Analysis
Q3: Is the ChatGPT label Superior or Infe- rior to human annotation? BCN labels pro- vided by Dalvi et al. (2022) are used human annotations for this question.
The annotators identified 58 concepts where human annotated BCN labels were deemed superior. We have conducted an error analysis of these instances and will now delve into the cases where GPT did not perform well.
In the first half of Table 1, the results indicate that 90.7% of the ChatGPT labels were considered Acceptable. Within the acceptable labels, 75.1% were deemed Precise, while 21.9% were found be Imprecise (indicated by Q2 in Table 1). We also computed Fleiss' Kappa (Fleiss al., 2013) to measure agreement among the 3 annotators. For Q1, the inter-annotator agreement was found to
Sensitive Content Models In 10 cases, the API calls triggered one of the content policy models and failed provide label. The content policy models aim prevent the dissemination of harm- ful, abusive, or offensive content, including hate speech, misinformation, and illegal activities. Fig- ure 3a shows an example of a sensitive concept that

---

gunpoin
celf
E EXES
Kills
teases
Averged
Roses
Radi ohead
you
gassing
recovers
Rufus
Monkeys
decapitated
eooele
Reddi
ng
obduct
beheading
nanhand
led caned
iossened
mai
mai
tasered crsfied
abduct
semen
Lambosts nav wwnateuunddrgggo
Acarmee owe snou deru
Dumps age File
net
expands
moC OUI
Les
Booee
negins
ua
Cobai n
R
Kinks Clapton
ingo Blup Muse srram Stavis0Lis
eeppe Lin
obaing
Ga giggbbuug
dg
Banonarana
Coldplay
Coltrane
Enery
wi
strtstraa strangulation mee
stabs drowns Telts audiirnn Injures
pranks
Jimi
mugged
wwweedeeeeeooooo ectrocution
Henderxx
electrecatedmassen
clans
Pistols
seuefflld
(a) Crime and Assault
(b) 3rd Person Singular Present-tense
(c) Rock Bands and Artists in the US
205
171
171
19
366
488
A total of 285 could prove to be rather impressive on pitch that was turning almost from the first ball But there were runs at Cheltenham where Gloucestershire finished on 315 for seven with the only century of the entire Championship day going to Ryan Higgins out for 161-ball 105 Back Nottingham it was the
259
31:51
3M
238
3
$9 -972855
15
TM
190
243481144
Australia 's bowlers eviscerated this time Eoin Morgan "s team rewrote the record books once again by making phenomenal 481 for six in this third one-day international.
132 235.2
2399 272
(d) Cricket Scores
Figure 3: Failed cases for ChatGPT labeling: a) Non-labeled concepts due to LLM content policy, b) Failing to identify correct linguistic relation, e) Imprecise labeling d) Imprecise labels despite providing context
includes words related to crime and assault. This problem can be mitigated by using version of LLM where content policy models are not enabled.
Insufficient Context Sometimes context contex- tual information is important correctly label concept. While human annotators (of the BCN cor- pus) were provided with the sentences in which the underlying words appeared, we did not pro- vide the same to ChatGPT to keep the prompt cost- effective. However, providing context sentences in the prompt along with the concept to label re- sulted in improved labels for 11 of the remaining 40 error cases. Figure 3d shows ple where providing contextual information made ChatGPT to correctly label the concept as Cricket Scores as opposed to Numerical Data the label that it gives without seeing contextual information. However, providing context information didn't con- con- sistently prove helpful. Figure 3c shows con- cept, where providing contextual information did not result in the accurate label: Rock Bands and Artists in the US, identified by the humans.
Linguistic Ontologies of the concepts, hu- man annotations (BCN) were better because the concepts were composed of words that were re- lated through a lexical, morphological, or syn- tactic relationship. The default prompt we used label the concept tends to find semantic simi- larity between the words, which did not exist in these concepts. For example, Figure 3b shows a concept composed of 3rd person singular present-tent verbs, but ChatGPT incorrectly labels as Actions/Events in News Articles. However, humans are robust and can fall back to consider various linguistic ontologies.
such exam-
The BCN concepts are categorized into semantic, syntactic, morphological, and lexical groups (See Table 3). As observed, both humans and ChatGPT found semantic meaning to the concept in majority of the cases. However, humans were also able to identify other linguistic relations such as lexical (e.g. grouped by lexical property like abbrevia- tions), morphological (c.g. grouped by the same parts-of-speech), or syntactic (e.g. grouped by po- sition in the sentence). Note however, that prompts can be modified to capture specific linguistic prop- crty. We encourage interested readers to see our experiments on this in Appendix A.2-A.3.
Uninterpretable Concepts Conversely, we also annotated concepts that were considered uninter- pretable or non-meaningful by the human anno- tators in the BCN corpus and in 21 out 26 cases, ChatGPT accurately assigned labels these con- cepts. The proficiency of ChatGPT in processing extensive textual data enables to provide accurate labels for these concepts.
We gave 10 context sentences to ChatGPT.

---

Label
ALBERT XLNet
fine-grained concepts are encoded within the latent space of a model enable applications beyond inter- pretation analysis. For example has direct appli-
c301
Gender-related Nouns and pronouns
0,95
0.86
c533
LGBTQ+
0.97
0.97
c439 Sports commentary terms
0.91
0,81
c173 Football team names and stadiums
0.96
0.94
cation in model editing (Meng et al., 2023) which
c348
Female names and titles
0.98
0.94
first trace where the model store any concept and
c149 Teanis playess' names
0.95
0.92
Spunish Male Names
0.96
then change the relevant parameters to modify its behavior. Moreover, identifying concepts that are associated with gender (e.g.. Female names and ti titles), religion (e.g. Islamic Terminology),
Cities and Universities in southeastem US
0.97
c263 Locations in New York City
0.95
0.92
c247 Scandinaviaa/Nrdicc names and places
0,98
0.95
c438 Verbs for various actions and outcomes
0.94
0.87
c44 Southeast Asian Politics and Ethnic Conflict
0.97
0.94
and ethnicity (e.g., Nordic names) can aid in eluci- dating the biases present in these models.
c421 Names of people and places in the middle east
0.94
0.95
c245 Middle East comflict
1,00
0.93
c553 Islamic terminology
0.96
Criminal activities
c128 Medical and Healtheare cerminology
0.98
5.2
Neuron Analysis
Table 4: Using latent concepts make cross-model comparison using Probing Classifiers
Neuron analysis examines the individual neurons or groups of neurons within neural NLP models to gain insights into how the model represents lin- guistic knowledge. However, similar to general interpretabiliit, previous studies in neuron analysis are also constrained by numan-in-the-loop (Karpa- thy et al., 2015; Kádár al., 2017) or pre-defined linguistic knowledge (Lakretz et al., 2019; Dalvi et al., 2019; 99; Hennigen et al., 2020). Consequently, the resulting neuron explanations are subject to the same limitations we address in this study.
5 Concept-based Interpretation Analysis
Now that we have established the capability of large language models like ChatGPT in provid- ing rich semantic annotations, we will showcase how these annotations can facilitate extensive fine- grained analysis on large scale.
5.1 Probing Classifiers
Our work demonstrates that annotating the latent space enables neuron analysis of intricate linguis- tic hierarchies learned within these models. For example. Dalvi et al. (2019) and Hennigen et al. (2020) only carried out analysis using very coarse morphological categories (e-g. adverbs, nouns etc.) in parts-of-speech tags. We now showcase how our discovery and annotations of fine-grained la- tent concepts leads to a deeper neuron analysis of these models. In our analysis BERT-based part- of-speech tagging model, we discovered 17 fine- grained concepts of adverb (in the final layer). It is evident that BERT learns highly detailed semantic hierarchy, maintains separate concepts for the adverbs of frequency (e.g., "rarely. sometimes") versus adverbs of manner (e.g., "quickly, softly"). We employed the Probeless method (Antverg and Belinkov, 2022) to search for neurons associated with specific kinds of adverbs. We also create a super adverb concept encompassing all types of adverbs, serving as the overarching and generic representation for this linguistic category and ob- tain neurons associated with the concept. We then compare the neuron ranking obtained from the su- per concept to the individual rankings from sub concepts. Interestingly, our findings revealed that the top-ranking neurons responsible for learning
Probing classifiers is among the earlier techniques used for interpretability, aimed examining the knowledge encapsulated in learned representations. However, their application is constrained by the availability of supervised annotations, which often focus on conventional linguistic knowledge and are subject to inherent limitations (Hewitt and Liang, 2019). We demonstrate that using GPT-based an- notation of latent concepts learned within these models enables direct application towards fine- grained probing analysis. By annotating the latent space of five renowned pre-trained language mod- els (pLMs): BERT, ALBERT, XLM-R, XLNet, and RoBERTa- - we developed a comprehensive Trans- formers Concept Net. This net encompasses 39,000 labeled concepts, facilitating cross- cros-archittectral comparisons among the models. Table 4 show- cases subset" of results comparing ALBERT and XLNet through probing classifiers.
We can see that the model learns concepts that may not directly align with the pre-defined hu- man onotology. For example, it learns a con- cept based on Spanish Male Names or Football team names and stadiums. Identifying how
'For a larger sample of concepts and additional models, please refer to Appendix B.

---

Super Concept
Sub Concepts
Alignment
Super Concept
98
289
657
537
261
315
408
569
607
Adverbs
17
0.36
c155: Frequency and manner
0.30
c155
27
439
315
145
105
699
709
132
405
c136: Degree/Intensity
0.30
+ c057: Frequeney
0.40
c136
557
145
26
422
623
252
439
160
315
Nouns
0.28
c66
557
439
483
27
412
105
50
149
315
673
+ c231: Activities and Objects
0.60
+ c279: Industries/Sectors
0.60
657
98
439
289
145
261
663
735
569
172
746
+ c440: Professions
0.10
Adjectives
17 0.21
c332
98
439
289
145
663
735
408
172
604
+ c299: Product Attributes
0.30
+ c053: Comparative Adjectives
0.30
c570
98
439
289
65
145
261
172
663
408
735
+ c128: Quality/Appoppriteeness
0.40
Numbers
17
0.23
c444
308
341
417
607
138
429
+ c549: Prices
0.50
c222
98
289
439
569
145
735
261
746
408
155
c080: Quantities
0.10
c593: Monetary Values
0.10
Figure 4: Neuron overlap between an Adverb Super Concept and sub concepts. Sub concepts shown are Adverbs fre- quency and manner (c155), Adverbs of degree/intensity (c136), Adverbs of Probability and Certainty (c265), Adverbs of Fre- queney (c57), Adverbs manner and opinion (c332), Adverbs of preference/choice (c570), Adverbs indicating degree tent (c244), Adverbs of Time (c222).
Table 5: Neuron Analysis on Super Concepts extracted from BERT-base-cased-PO. model. The alignment column shows the intersection between the top 10 neurons in the Super con- cept and the Sub concepts. For detailed results please check Appendix C (See Table 11)
the super concept often distributed among the top neurons associated with specialized concepts, as shown in Figure for adverbial concepts. The results, presented in Table 5, include the number of discovered sub concepts the column labeled # Sub Concepts and the Alignment column indi- cates the percentage of overlap in the top 10 neu- rons between the super and sub concepts for each specific adverb concept. The average alignment across all sub concepts is indicated next to the su- per concept. This observation held consistently across various properties (e.g. Nouns, Adjectives and Numbers) as shown in Table 5. For further details please refer to Appendix C).
6 Related Work
With the ever-evolving capabilities of the LLMs, researchers are actively exploring innovative ways to hamness their assistance. Prompt engineering, the process of crafting instructions to guide the behav- ior and extract relevant knowledge from these ora- cles, has emerged as new area of research (Lester et al., 2021; Liu et al., 2021; Kojima et al., 2023; Abdelali et al., 2023; Dalvi et al., 2023b). Recent work has established LLMs highly proficient annotators. Ding et al. (2022) carried out evalua- tion of GPT-3's performance as a data annotator for text classification and named entity recognition tasks, employing three primary methodologies to assess its effectiveness. Wang et al. (2021) showed that GPT-3 as an annotator can reduce cost from 50-96% compared human annotations 9 NLP tasks. They also showed that models trained using GPT-3 labeled data outperformed the GPT-3 few- shot learner. Similarly, Gilardi et al. (2023) showed that ChatGPT achieves higher zero-shot accuracy compared to crowd-source workers in various anno- tation tasks, encompassing relevance, stance, top- ics, and frames detection. Our work is different from previous work done using GPT as annotator. We annotate the latent concepts encoded within the embedding space of pre-trained language models. We demonstrate how such large scale annotation enriches representation analysis via application in probing classifiers and neuron analysis.
Note that previously, we couldn't 't identify neu- rons with such specific explanations, like distin- guishing neurons for numbers related to eurrency values from those for years of birth or neurons differentiating between cricket and hockey-related terms. Our large scale concept annotation enables locating neurons that capture the fine-grained as- pects of a concept. This enables applications such manipulating network's 's behavior in relation to that concept. For instance, Bau et al. (2019) identi- fied "tense" neurons within Neural Machine Trans- lation (NMT) models and successfully changed the output from past to present tense by modifying the activation of these specific neurons. However, their study was restricted to very few coarse concepts for which annotations were available.

---

7 Conclusion
racial slurs in the hate speech detection task. This restricts our concept annotation approach to only tasks that are not sensitive to the content policy.
The scope of previous studies in interpreting neural language models is limited to general ontologies or small-scale manually labeled concepts. In our research, we showcase the effectiveness of Large Language Models, specifically ChatGPT, as a valu- able tool for annotating latent spaces in pre-trained language models. This large-scale annotation of latent concepts broadens the scope of interpreta- tion from human-defined ontologies to encompass all concepts learned within the model, and elimi- nates the comman-in-the-owee effort for annotating these concepts. We release a comprehensive GPT- annotated Transformers Concept Net (TCN) con- sisting of 39,000 concepts, extracted from wide range of transformer language models. TCN em- powers the researchers carry out large-scale inter- pretation studies of these models. To demonstrate this, we employ two widely used techniques in the field of interpretability:: probing classifiers and neuron analysis. This novel dimension of analysis, previously absent in earlier studies, sheds light on intricate aspects of these models. By showcasing the superiority, adaptability, and diverse upplica- tions of ChatGPT annotations, we lay the ground- work for more comprehensive understanding of NLP models.
The information in the world is evolving, and LLMs will require continuous updates to reflect the accurate state of the world. This may pose a challenge for some problems (c.g. news summa- rization task) where the model needs to reflect updated state of the world.
References
Ahmed Abdelali, Hamdy Mubarak, Shammur Absar Chowdhury, Maram Hasanain, Basel Mousi, Sabri Boughorbel, Yassine EI Kheir, Daniel Izham, Fahim Dalvi, Majd Hawasly, Nizi Nazar, Yousseif Elsha- hawy, Ahmed Ali, Nadir Durrani, Natasa Milic- Frayling, and Firoj Alam. 2023. Benchmarking ara- bic ai with large language models.
Firoj Alam, Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Khan, Abdul Rafae, and Xu. 2023. Conceptx: A framework for latent concept analysis. In Pro- ceedings of the Thirty-Sevntth AAAI Conference Artificial Intelligncee (AAAT, Poster presentation), pages 16395-16397.
Omer Antverg and Yonatan Belinkov. 2022. On the pitfalls of analyzing individual neurons in language models. In International Conference on Learning Representations.
Limitations
Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2019. Iden- tifying and controlling important neurons in neural machine translation. In Proceedings of the Seventh International Conference on Learning Representa- tions, ICLR 119, New Orleans, USA.
We list below limitations of our work:
While it has been demonstrated that LLMs signif- icantly reduce the cost of annotations, the com- putational requirements and response latency can still become a significant challenge when deal- ing with extensive or high-throughput annotation pipeline like ours. In some cases is important to provide contextual information along with the concept to obtain an accurate annotation, caus- ing the cost go up. Nevertheless, this is one time cost for any specific model, and there is optimism that future LLMs will become more cost-effective run.
Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass, 2017a. What do neural machine translation models learn about morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL '17, pages 861-872, Vancouver, Canada. Association for Computational Linguistics.
Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2020. On the linguistic representational power of neural machine translation models. Computational Linguistics, 45(1):1-57.
Existing LLMs are deployed with content pol- icy filters aimed at preventing the dissemination of harmful, abusive, or offensive content. How- ever, this limitation prevents the models from effectively labeling concepts that reveal sensitive information, such as cultural and racial biases Iearned within the model to be interpreted. For example, we were unable to extract a label for
Yonatan Belinkov, Lluís Mårquez, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2017b. Eval- vating layers of representation in neural machine translation on part-of speehh and semantic tagging tasks. In Proceedings of the Fighth International Joint Conference on Natural Language Processing (Volume Long Papers), pages 1-10, Taipei, Taiwan. Asian Federation of Natural Language Processing.

---

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler. Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Firie Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. CoRR, abs/2005.14165
Bosheng Ding, Chengwei Qin, Linlin Liu, Lidong Bing. Shafıq Joty, and Boyang Li. 2022. Is gpt-3 a good data annotator?
Nadir Durrani, Fahim Dalvi, Hassan Sajjad, Yonatan Be- linkov, and Preslav Nakov. 2019. One size does not fit all: Comparing NMT representations of different granularities. In Proceedings of the 2019 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Tech- nologies, ACL '19, pages 1504-1516, Minncapolis, Minnesota, USA. Association for Computational Lin- guistics.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Out Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Associa- tion for Computational Linguistics, pages 8440-843 8451. Association for Computational Linguistics.
Nadir Durrani, Hassan Sajjad, Fahim Dalvi, and Firoj Alam. 2022. On the transformation of latent space in fine-tuned nip models. Proceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 1495-1516, Abu Dhabi, UAE, Association for Computational Linguis- tics.
Alexis Conneau, German Kruszewski, Guillaume Lam- ple, Loic Barrault, and Marco Baroni. 2018. What you can cram into a single &&!#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the As- sociation for Computational Linguistics, ACL 18, pages 2126-2136, Melbourne, Australia. Association for Computational Linguistics.
Joseph Fleiss, Bruce Levin, and Myunghee Cho Paik. 2013. Statistical methods for rates and proportions.
Yao Fu and Mirella Lapata. 2022. Latent topology induction for understanding contextualized represen- tations.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer foed-forward layers are key- value memories. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 5484-5495, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Fahim Dalvi, Nadir Durrani, and Hassan Sajjad. 2023a. Neurox library for neuron analysis of deep nip mod- els. In Proceedingy of the 61 Annual Meeting of the Association for Computational Linguistics: Sys- tem Demonstrations, pages 15-83, Toronto, Canada. Association for Computational Linguistics.
Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, D. Anthony Bau, and James Glass. 2019. What is one grain of sand in the desert? analyzing in- dividual neurons in deep nlp models. In Proceedings of the Thirty-Thind AAAI Conference on Artificial Intelligence (AAAI, Oral presentation).
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms crowd-workers for text- annotation tasks.
K Chidananda Gowda and G Krishna. 1978. Agglomer- ative clustering using the concept of mutual nearest neighbourhood. Patlern recognition, 10(2):005-112
Fahim Dalvi, Maram Hasanain, Sabri Boughorbel, Basel Mousi, Samir Abdaljalil, Nizi Nazar, Ahmed Abdelali, Shammur Absar Chowdhury, Hamdy Mubarak, Ahmed Ali, Majd Hawasly. Nadir Dur- rani, and Firoj Alam. 2023b. Llmebench: flexible framework for accelerating lims benchmarking.
Biyang Guo. Xin Zhang, Ziyuan Wang, Minqi Jiang. Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023. How close is chatgpt to human experts? comparison corpus, evaluation, and detection.
Fahim Dalvi, Abdul Rafae Khan, Firoj Alam, Nadir Dur- rani, Jia Xu, and Hassan Sajjad. 2022. Discovering latent concepts learned in BERT. In International Conference on Learning Representations.
Lucas Torroba Hennigen, Adina Williams, and Ryan Cotterell. 2020. Intrinsic probing through dimension selection. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 197-216, Online. Association for Computational Linguistics.
Jacob Devlin, Ming-Wai Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. NAACL-HLT '19 pages 4171-4186. Minneapolis, Minnesota, USA. Association for Com- putational Linguistics.
John Hewitt and Percy Liang. 2019. Designing and in- terpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Nat- ural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2733-2743, Hong Kong. China.

---

Katharina Jeblick, Balthasar Schachtner. Jakob Dexl, Andreas Mittermeier, Anna Theresa Stüber, Johanna Topalis, Tobias Weber, Philipp Wesp, Bastian Sabel, Jens Ricke, and Michael Ingrisch. 2022. Chatgpt makes medicine easy to swallow: An exploratory case study on simplified radiology reports.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing. CoRR, abs/2107.1358k
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. RoBERTa: A robustly optimized BERT pretraining approach. ArXiv: 1907. 1662..
Akos Kádár, Grzegorz Chrupata, and Afra Alishahi. 2017. Representation of linguistic form and func- tion in recurrent neural networks. Computational Linguistics, 43(4):761-780.
Andrej Karpathy, Justin Johnson, and Li Fei-Fei. 2015, Visualizing and understanding recurrent networks. arXiv preprint 0rXiv:1506.02078
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2023. Locating and editing factual associa- tions in gpt.
Diederik Kingma and Jimmy Ba. 2014. Adam: Method for Stochastic Optimization. arXiv preprint arXiv:1412.6980.
Julian Michael, Jan A. Botha, and Ian Tenney, 2020. Asking without telling: Exploring latent ontologies in contextual representations. In Proceedings of the 2020 Conference on Empirical Methods in Natu- ral Language Processing, EMNLP '20, pages 6792- 6812, Online. Association for Computational Lin- guistics.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2023. Large lan- guage models are zero-shot reasoners.
Yair Lakretz, German Kruszewski, Theo Desbordes, Dieuwke Hupkes. Stanislas Dehaene, and Marco Ba- roni. 2019. The emergence of number and syntax units in LSTM language models. In Proceedings of the 2019 Conference of the North American Chap- ler of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers). pages 11-20, Minneapolis, Minnesota. Association for Computational Linguistics.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word represen- tations in vector space. In Proceedings of the ICLR Workshop, Scottsdale, AZ, USA.
Peng Qian, Xipeng Qiu, and Xuanjing Huang. 2016. Investigating Language Universal and Specific Prop- erties in Word Embeddings. In Proceedings of the 54th Annual Meeting of the Association for Com- putational Linguistics, ACL '16, pages 1478-1488, Berlin, Germany, Association for Computational Lin- guistics.
Zhenzhong Lan, Mingda Chen, Sebastian Good- man, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. ALBERT: lite BERT for self- supervised learning of language representations. ArXiv:1999.11442
Hassan Sajjad, Nadir Durrani, and Fahim Dalvi. 2022.
Neuron-level interpretation of deep NLP models: A survey. Transactions of the Association for Compu- tational Linguistics,
Richard Landis and Gary G Koch. 1977. The mea- surement of observer agreement for categorical data. biometrics, pages 174.
Tan Tenney, Dipanjan Das, and Ellie Pavlick. 2019.
BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4593- 4601, Florence, Italy, Association for Computational Linguistics.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for aarameter-efficiee prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pages 3045-3059, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Ekaterina Vylomova, Trevor Cohn, Xuanli He, and Gho- lamreza Haffari. 2017. Word representation models for morphologically rich languages in neural machine translation. pages 103-108.
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the Ability LSTMs to Learn Syntax-Sensitive Dependencies. Transactions of the Association for Computational Linguistics, 4:521- 535.
Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021. Want to reduce label- ing cost? gpt-3 can help.
Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019a. Lin- guistic knowledge and transfcrability of contextual representations. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies. NAACL '1,, pages 1073-1094, Minneapolis, Minnesota, USA. Association for Com- putational Linguistics.
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Car- bonell, Russ R Salakhutdinov, and Quoc V Le. 2019, Xinet: Generalized autoregressive pretraining for lan- guage understanding. Advances in neural informa- tion processing systems. 32.
Bowen Zhang, Daijun Ding, and Liwen Jing. 2023. How would stance detection techniques evolve af- ter the launch of chatgpt?

---

Appendix
Using this improved were able to correct 100% of the labeling errors in the concepts having lexical coherence. See Figure 7a for example. With the default prompt was labelled as Superlative and ordinal adjectives and with the modified prompt, it was labeled as Hyphenated, cased & -based suffix.
A Prompts
A.1 Optimal Prompt
Initially, we used a simple prompt to ask the model to provide labels for a list of words keeping the
system description unchanged:
A.3 Prompts for POS Concepts
Assistant is large language model trained by OpenAI
able to modify the prompt correctly lahel concepts that were made from words having common parts-of-speech. From the prompts we tested, the best performing one is below:
Similarly
Prompt Body: Give the following list of words a short label: ["word 1", "word 2", "word N
Give a short and concise label describing the common part of speech tag between the words of the given list Note: The part of speech tag should be chosen from the Penn Treebank. If there's common part of speech tag reply with 'NA'
The output format from the first prompt was unclear as included illustrations, which was not our intention. After multiple design iterations, we developed prompt that returned the labels in desired format. In this revised prompt, we modified the system description as follows:
Assistant is large language model trained by OpenAI.
In Figure 7b, we present an example of concept labeled as Surnames with 'Mc' prefix. How- ever, it is important to note that not all the names in this concept actually begin with the "Mc" prefix. The appropriate label for this concept would be NNP: Proper Nouns SEM: Irish Names. With the POS- based prompt, are able to achieve the former.
Instructions: When asked for labels, only the labels and nothing else should be returned.
We also modified the prompt body to: Give a short and concise label that best describes the following list of words: ["word 1", "word 2", .... "word N"]
A.4 Providing Context
Our analysis revealed that including contextual in- formation crucial for accurately labeling con- cepts in certain cases. As shown in Figure 8, con- cepts were incorrectly labeled as Numerical Data despite representing different entities. Incorporat- ing context enables us to obtain more specific la- bels. However, we face limitations in the number of input tokens impacts the quality of the labels. Using context of 10 sentences we were able to correct of the 38 erroncous labels.
Figure shows some sample concepts learned in the last layer of BERT- basecc along with their labels.
A.2 Prompts For Lexical Concepts
provide the model, which
During the error analysis (Section 4.2), we dis- covered that GPT struggled to accurately label con- cepts composed of words sharing lexical property, such as common suffix. However, we were able to devise solution to address this issue by curat- ing the prompt effectively label such concepts. We modified the prompt to identify concepts that contain common n-grams.
A.5 Other Details
Tokens Versus Types We observed that the qual- ity of labels is influenced by the word frequency in the given list. Using tokens instead of types leads to more meaningful labels. However, when the la- tent concept includes hate speech words, passing token list results in failed requests due content policy violations. In such cases, we opted to pass the list of types instead. Although this mitigates the issue to a certain extent, it does not completely
Give a short and concise label describing the common ngrams between the words of the given list Note: Only one common ngram should be returned. If there is no common ngram reply with 'NA'

---

Por
Lakes
Detroit Vegas
utah as
solor
AvivSt.r
Ralestingan
Levi
The
Harrey,
Isrgeli
Jewish
Jen
Sinai
gays
transgender
L
GBT
Queer
Venerosoual
chi goa
Bet
TTra Trans
WLscons
Judatsm
Jew
Gaza
serveot
OP
trat
t
crans
Minnesota Salt Arizznn Indiana
M Ma Lokethicage
Palestinians
ser
Rabbi
day
gay
hasosesual
bissexa
lesbian
Kansas
Israel
Jews
LGBTQ
queer
Netroyyah
harasexuality
(a) Geographic Locations in the
(h) Middle East Conflict
(c) LGBTQ+
Lo Melisso
Helen Elzzo Mi chelll L:Le
Lita Dakoto wes.
Hector
JectorJoro
Joine
Jorge
Santiago ter
Manuel
Men, women
men S
sateed www To
Luvy
Amanda
U Erika
Laara Ms
Nicole
Sofia.
c..
Jane
Sara
Andres
Pedro
Gustavo
Ju
1
Juan
Corlos Jubsen
Al Alejandro
restoi
nol
gender
woman
g
with
male
Lady
lomen
Hee
Cauren
Jose JoseJavierJoss
girl
solas
She
she.
Ling Jenni fer Natalie
Cocori
Dold L
Kate
tris
Luis Ricardo Andrés Fernando
Pablo RuberRaul
Pablop
Lenin
femi nimm
housewi
nen
f Temale
wire
latios
(d) Female Names and titles
(c) Spanish Male Names
Gender related and pronouns
Taggen
Cambodic Son
Aung
Vielnuncse
Al aabomo
Clemson Vaidrerr
Mushville
Jose Ventura
Jose
Palm
WIZLES
Callfornian
Oakland
lemws
the
Thi
Thai
and
Kentucky
Louisville
Santa
Monica
Lering
O.SSan
with
Hills
Kyi
That
eetam
Bgg
Nom
Roy
At
Lant
Utottssion
a
Californn
Coust
in
Valleey
Sacramenta
Myanmar Rohingya
r
Chi
Memphis
Every Mistissippi
Ohlo
Tennessee
Angeles
Orange
Suu
Columbus Georgia Lexington
Francisco Diego
Seeethe
Serveley
Hun
Burma
0aks
(h) SE Asian Polities and Ethnic Con- flict
(i) List of cities and universities in
Geographic Locations in Califomia
southeastern US
Figure Sample Concepts Learned the last layer BERT
Ku
11 -anda
HolocaustPKK
Welccn
condicted
crlal bai rlabbai L
Hezbol 1 Lah
Jh
ew
neo -Naii
Mari Treasure
T
referendum
ury
Mint
stdlcted
joiled
sentenced
t
conanbee
Boda
-nti-Semiie tittis
sm
Smoc
miint
Constitution
Sup
Ie
eme
coup
gutlty
prosecuted
p
pa
pri
parole ee
20r1
Nazis
Klux Nazz
uith
Seritic
ews
Amnesty
Capi
Pentogon
prosecutio cor
desivtionn
Muscrwitz
Klan
Smi thsonitan
ussuss inssssssiiaiion
recip
Irial
Inp sooee
STusoner
(a) Ho
(c)
Ku
Jewish
Qanda
Holocauste PKK
neo
Nazi
RDP
A woreeeeo
resorted ed
olidd
SPPCRE ad
killsdeath ddes
sul
Care
Lae
turders
rego
Hezbol
Lah
hasctat
booked
de
Fed
Sked
Dead dying
Ki
erriiia
Diee
derided
Willer
an
ti
Smoc
ditched
emai
Led
ramped
shied
mur
murdered
antisemity Sm
tweeted
tester
Buock
murdere
di
ces
Nazis
Nazz
uith
Senttic
ews
puid
poamed
Killdi die
ied
Klux
Klan
gushed Flocked wowed.
(d)
(e)
Figure 6: Failed Requests in Albert

---

Tag
Human Label
Label Response
3- 3-Keyword Response
c533
IXXhyphenated
Superlative and ordinal adjectives. Accomplishments and Awards
['second', "highest, "biggest"]
c84 LEXchybenatted
I' Awarddwinnnig'' "Nominated", 'Multi-ime")
c783
LEX.hypenateed
Sports scores and point differentials.
I' poonnts' "wins", 'scores']
LEX.hyphenated
Describing people's relationships family status. Tennis Scores.
[family. relationships, parenthood!
c869
["Tennis', 'Scores', "Games']
c833 LEXchyhhemaeed
Logation based adjectives
[based, area. listed]
c588 LEXhhyphenated
US political party affiliations by state and district.
[Republican, Democrat, State Abbreviations ('Scores', 'Football', "Winning']
0639 LE LEX:byybenstee
Football scores.
6934 LEX:bybennteed
Sports scores.
I'scores', 'victories', 'defeats"]
c850 EExccss:tiitle case
Philippine Places and Names
['Philippines' , "Tourism', "Volcano"] [Last names, English, List] ('Fostball', 'Sports', "Legends']]
LEX:case:tille cease
List surnames.
IEX:case:tille cuse
Sports-rellted terms.
c231 postiioiioo word
Sports Teminology
['Fostballers', , 'Tries, 'Subatitutes'
c784 poNiiositinnn:iiss word word Numerical data.
[Numbers, Decimals, List]
c728 posiiioitio word
Actioa-oricated verbs and adjectives,
[Iuproving, Ensuring. Capturing]
c672 SY N-postiion:fisi word
Verhs descrihing actions and states.
[Fluemt Struggling, Showcasing]
c886 LEX.casettile_.am case
Describing communication actions. Basehall player names.
[Referring, Recalling, Revealing)
c865 1 EX cuse
[Bregmun, Scherzer, Puig]
c734 LEXreasettitle... case
Island nanass.
I'Islands'. 'Caribbean', 'Indian Occan')
c818 LEX:casctttle_.cc
Echnicities Cities Balkans
['Bosnian', . 'Albanian', 'Yugoslavia'
Table 6: Prompting ChatGPT to label concept with keywords instead of one label
trittain-bssed
McIver
McCar rick
Vanoo G Glasgow- Dal
L as based
Edinburgh-baser
Aik
Linkng
Aiken- Gough
M
Mckaton
As
Rut
ta mamaayy hames Chic Chip-bikes bakes
McAdams
Mcllow
Geneve beeed Ange Les red
Boston-based
Chi Chicago catooa--cssedlaan Landdryyaaaddd Land bosed
Lagiiiin Pupok
Mc Givern McGovern
McElure McGurk
McGreavy
Pitastmons
Burteligh
hnstinghan busod
Sydney-basee based--aadddd.... Kong bagg-baaedd
Maccs
Bradford- based
McEntee McKinnon
Lusker"
(a) Hyphenated,cassed & -based suftix
(b) NNP: Proper Nouns
Figure 7: Ilustrattnng lexical and POS concepts: (a) A concept that exhibits multiple lexical properties, such as being hyphenated and cased. ChatGPT assigns a label based on the shared "-based" ngram found among most words the cluster. (b) ChatGPT labeled this concept NNP (proper noun)
12,217:55
15.51 3.
3 84.
25,
77:66
32.90
$1.9 99 2927
R2S 12.9
777. 77.7777 " 411 5m H
9:2.13,
Nrzzich 27.5
13.99m... 9
299g 688.
10-8 0-50-000
15-8
35-83-1811 1
for 4
00
102
32
for
36-6
29.9
7.87,15 8.:3 53.5 26.44.
2114.3m011
2 9
78.
9
64 4 13 9. 9.6
2
2.67 6 21 88 79 41.220.12111.... 220.17
14.9
17-18 3.
6-8
18-1
0
8-11-5
N
1.89
85-771-3
23.5 +
32.7 9.61 11
9t 813. 813.
47.44 :4:00
7.2
17.1
59
65550
13
$3.5 "
22 27 17.9
26 No 5
81 5.9
2 8 5a
5ud.
1.95
8-47-2 18-7
6.20
8
T
9-6
42
P.o 0%)
4-7716
7-3
14-7
2-6
9-11
$8
5 19.4 1..4666
6
12
07
8
3
11 41
6.75
for-3
7..4..3.7-mtt
1.78
14, 4.4
(8)
(b)
(c)
Figure& 8: Highlighting the Significance of Context: (a) Money Figures (b) Percentages (c) Baseball Scores.
All of these concepts were mislabeled as Numerical values by ChatGPT. Providing context sentences we are able to obtain the correct label

---

resolve it. Refer Figure 6 for examples of failed requests with Albert.
adverbs, and numerical data. To assess the align- ment between the sub concepts and the super con- cept, we calculated the occurrence percentage of the top 10 neurons from the sub concept within the top 10 neurons of the super concept. The outcomes of this analysis can be found in table I I, illustrating the average alignment between the sub concepts and the super concepts.
Keyword prompts We also explored prompts to return keywords that describe the concept instead of returning concise label in an effort to produce multiple labels like BCN.
Instructions: When asked for keywords, only the keywords and nothing else should be returned.
Neurons Associated with the Names concepts We replicated the experiment using named entity concepts derived from the final layer of bert-base- cased. The findings are presented in table 12.
If asked for 3 keywords, the keywords should be returned in the form of [keyword_1, keyword_2, keyword_3]
To ensure compliance with our desired output format, we introduced a second instruction since the model was not following the first instruction as intended. We also modified the prompt body to:
Give keywords that best describe the following list of words
Unfortunately, this prompt did not provide accu- rate labels, illustrated in Table 6.
B Probing Classifiers
B.1 Running Probes At Scale
Probing For Fine-grained Semantic Concepts
We used the NeuroX toolkit train a linear probe for several concepts chosen from layers 3, 9 and 12 BERT-base-cased. We used a train/val/test splits 0.6, 0.2, 0.2 respectively. Tables and 9 show the data statistics and the probe results respectively, Table 10 shows results of probes trained on con- cepts chosen from multiple layers of ALBERT. In Table 7 we carried out cross architectural compar- ison across the models by training probes towards the same of concepts.
Neuron Analysis Results
Neurons Associated with POS concepts We per- formed an annotation process on the final layer of fine-tuned version of BERT-base-cased specifi- cally focusing on the task of ofrts-of-speech tag- ging. Once obtained the labels, organized them into super concepts based on shared char- acteristic among smaller concepts. For instance, we grouped together various concepts labeled as nouns, as well as concepts representing adjectives,

---

Label
BERT
Sel ALBERT
Sel XLNNt
Sel XLM-R
Sel RoBERTa Sel
c301
Gender-related Nouns and
0,98 0.16
0.95 0.14
0.86 0.24
0.94 0.23
0.95 0.26
c533
LGBTQ*
1 0.18
0.97 0.33
0.97 0.43
1 0.25
0.14
c4399
Sports commentary terms
0.94 0.2
0.91 0.18
0.81 0.05
0.87 0.11
0.86 0.09
c173 Football team names and stadiums
0.94 0.2
0.96 0.27
0.94 0.24
0.95 0.2
0.97 0.34
c348 Female names titles
0,98 0.29
0.98 0.29
0.94 0.21
0.96 0.16
0.97 0.24
c149 Tennis players' names
0.98 0.27
0.95 0.25
0.92 0.19
0.92 0.17
0.92 0.19
Spanish
c564 Cities and Universities southeastern US
0.97 0.12
0.97 0.11
0.9
0.97 0.29
0.96 0.22
c263 Locations New York City
0.95 0.25
0.95 0.22
0.92 0.26
0.95 0.26
0.95 0.17
c247 Scandinavian/Norda names and places
0.97 0.22
0.98 0.27
0.95 0.29
0.96 0.21
0,98 0.29
c438 Verbs for various actions and outzomes
0.97 0.12
0.94 0.09
0.87 0.23
0.92 0.11
0,92 0.14
c44 Southeast Asian Politics and Ethnic Conflict
0.97 0.17
0.97 0.19
0.94 0.25
0.93 0.09
0,95 0.16
c421 Names of people and places middle east
0.97 0.06
0.94 0.28
0.95 0.22
0.93 0.31
0.92 0.12
c245 Middle East conflict
0.98 0.26
0.25
0.93 0.29
0.93 0.25
0.95 0.22
c553 terminology
0.89
0.95
c365 Criminal activities
c128 Medical and Healtheare terminology
0.98
0.98 0.21
0.94 0.24
0.95 0.27
Table 7: Training Probes towards latent concepts discovered in various Models. Reporting classifter accuracy
test-set along with respective selectivity numbers
Loden
McCain
Ranale
Bush
Reagan
Thathheree
umo
Obbaa
Barack
Socdan
Lineat
Gerald
Aug
November JulyOct
June
March
Mar
August
Cept netenber
becchree
Fuel
sooerss
petrol
ectni
powe
Energy
superpowe
Power
GO
D
May
January
Sept February
fu
1E
Electric
TRUMP
uther
Woodras
illrory
Tneodore
Nov
April
totere
engines
prweered
elecriic
direpiand
Berm Bernir
Romney
Lunden
rrum
Jan
bun
tnes
Clinorr
Hussetn
bush
Decembe Feb
energy
Nasser
Dec
Englin Englnan diesel
Elactricity
curbine
engine
(a) Political Figures
(b) List of Months
(c) Finergy Related terms
unvei Lnng
wrangLing ting
si sizzling crippling
Label L ing
nature
Scope W WLath Feee
e Sprods
IS
had i S
were
Seerehaver
have
having
Has
totalling
grappline
monttng
auttin
backling
ang
proteiing Talkes
durati
angles
ang
Setio
Lengths
radius
speed
grovity
iodtiming
tony
tining
size
sizes
distonce
sagen Iagen
Hos
Had
Have
Nere
be
behas
been
was
n
are
Having
Be
distance
chickters
Being being
Rowl howlingoo dwindling
ailing
angle
Is Are
unsilliy
(d) Verbs Ending ing
(e) Measurement Units
Verbs Various Tense Forms
monorchrulers duke
queen
Duchess
time e S hapless topless
shirtles
heartlessscorelees
1bn 20bn m1 mil11 1illion min billion
Jei 1111 mut m11-mm11 Eiton
crown
ruler
duchess
s
Prin
point ess
aoallecs
Majestyl
King
prince
selfless
menta
teari base less
princes
Queen
Crown
sleeveless
t rillion
mrinces
Duke
Royal
throe
Kings
king
Teperiol
lifeless
winless joressooo Lose
job
10bn Million
Princess
(g) Royalty and Monarchy
(h) Adjectives with less suffix
Monetary Values
Figure 9: Sample Concepts learned in the ALBERT Model

---

Layer
Tag
Label
Tokens
Types
Sents
Truin
Val
Test
c90
Financial
220
22
214
285
95
96
c336
Pholography-relateer terms.
290
29
273
388
130
130
c112
Middle Eastern Conflict
620
62
523
992
331
331
3
c506
Diversity and Ethnicity.
240
24
225
331
111
112
3
c390
List of surnames.
4298
430
4049
5530
1844
1844
c331
Emotions/Feelings.
400
396
484
162
162
Animal names.
220
90
c25 Keywords
discrimination inequality.
147
List of female names.
2913
292
2735
3867
1289
1290
c414
Healtheare
510
51
475
752
251
251
c31
List of male first names.
1130
113
1078
1422
474
474
cl73 Animals
760
76
704
994
332
332
3
<72 Natural Disasters and Weather Events
701
71
635
1022
341
341
3
c514 English counties
297
30
286
373
124
125
c178 Body Parts
430
405
588
196
196
c340 Media and Joumaliis.
379
518
173
Verbs
1028
103
1018
1243
414
c408 -Verhs ending in "-ing"
510
51
504
615
205
206
c479 City names
130
127
159
53
54
3
c343 Sumames
490
49
464
613
204
205
3
c577 Disability terms.
140
14
133
172
58
58
c26 Negative sentiment.
798
118
782
1036
346
346
c122 Security Measures
457
584
195
Label:
Extremism/Termriin
Middle Eastern and North Affican countrics and cities
531
282
<192
Diversity and Identity
314
506
169
169
c468 Russian male names.
125
123
153
51
52
<588 Gender-related terms.
161
19
146
236
79
79
c74
Financial terms
672
96
607
1118
373
373
c503
Middle East Conflict.
230
27
185
404
135
135
c325
Violent Crimes
292
287
386
129
Reseurch.
of
458
c507
Positive Adjectives
389
505
168
169
c345 List of Chinese surnames.
407
65
378
567
189
190
c259 List of names
223
174
221
273
91
92
12
c62 Adverbs
1221
351
1133
3769
1256
1257
12
c128 Medical and Healtheare Terminology.
395
70
369
662
221
221
c301
Gender-related nouns and promoums,
418
377
883
294
295
637
of male names.
872
372
1460
487
Adverbs
928
264
393
List of surnames.
832
3652
6378
2126
c432
List of Male Numes
279
1.59
227
474
158
158
c439 Sports commentary
250
181
189
687
229
230
12
cl73 List of football team names and stadiums.
373
81
287
849
283
284
12
c348 List of female names and titles.
575
301
571
774
258
258
12
c142 Conflict and War
407
106
385
582
194
194
c245 Middle East Conflict
249
196
453
151
of cities universities
United
c533
LGBTQ+
131
118
188
63
c19
Complex sclationships and interactions between family members and partners.
3416
56
333
546
182
182
<263
Locations in New York City
205
48
186
386
129
129
12
c487
List of Spanish male names.
184
63
174
242
81
81
12
c247
Scandinvvian/ooriic names and places.
334
305
502
168
168
c44
Southeust Asian Politics and Fahnic Conflict
210
33
149
332
111
Verbs for various actions and outcomes.
896
377
1600
534
Names people places the
Islamic Temnnology..
164
85
c149 List of temnis players'
238
82
183
394
132
132
c365
Criminal activities
365
88
337
496
166
166
Table 8: Statistics for concepts extracted from Bert-base-cased and the training, dev, test splits used to train the classifier

---

Layer
Tag
Label
val
val
ace sel val test
test ace sel test
c90 Financial terms.
0.98
0.65
0.33
0.98
0.78
0.20
c336 Photography-relane related temns.
0.99
0.74
0.25
0.76
0.24
c112
Middle Eastern Conflict
0.99
0.89
0.10
0.86
0.14
c506
Diversity and Ethricity.
1
0.78
0.22
0.98
0.75
0,23
c390
of surnames.
0.97
0.82
0.15
0.97
0.82
Emotiins/Feelings
0.82
0.16
0.99
0.78
Animal
1
c25
Keywords related discrimination and inequality.
0,99
0.81
0.18
0.98
0.77
0.21
c500 List of female
0.98
0.82
0,16
0.99
0.83
0,16
c414
Healthcare
1 0.77
0.23
0,79
0.21
c31
List of male first names.
0.99
0.85
0.14
0.83
0.17
c173
Animals
0.99
0.78
0,21
0.99
0.75
0.24
c72 Natural Disasters and Weather Events
0.99
0.80
0.19
0.99
0.78
c514 English counties
0.26
0.76
Rody
0.15
0.98
0.89
c340 Media and Journalism.
0.76
0.22
0.78
0.22
c432 Power and Status.
0.99
0.79
0.20
0.78
0.22
Verbs
0.99
0.88
0,11
0.99
0.89
0.10
c408 -Verhs ending in "-ing"
1
0.68
0.32
0,73
0.27
c479 City names
0.98
0.68
0,30
0.83
0.17
c343 Surnames
0.74
0.26
0.98
0.74
0.24
c577 Disabllity-relteed terms,
1 0.82
0.18
0.78
c26 Negative sentiment.
0.79
0.19
0.99
0.19
Security
c423 Lahel: Islamic ExtrenismmTemmrrisn
1 0.77
0.23
0.85
0.15
c299 Middlle Fastern and North African countries and cities
0.99
0.79
0.2
0.99
0.78
0.21
c192 Diversity and Identity
0.99
0.88
0,11
0.98
0.88
0.1
c468 Russian male names.
1
0.63
0.37
0.61
0.39
c588 Geader-related terms.
1 0.69
0,31
0.99
0.76
0.23
<7 Financial terms
0.99
0.86
0.13
0.97
0.83
0.14
c503 Middle East Conflict.
0.75
0.24
0.99
0.71
Violent
0.98
c535 Academic Research.
1 0.88
0.12
0.99
0.84
0.15
c256 List of names
0.98
0.76
0,22
0.98
0.74
0.24
c507 Positive Adjectives
0.98
0.78
0,2
0.98
0.79
0.19
c345 List of Chinese sumames.
0.99
0.86
0.13
0.87
0.13
12
c259 List of names
0.98
0.89
0.09
0.99
0.89
0.1
c62 Adverbs
0.97
0.82
0.15
0.96
0.81
0.15
c128 Medical Healtheare Terminology.
0.99
0.19
0.98
0.82
0.18
Gender-related mouns and promouns.
c281 Adverbs
0,99
0.19
0.99
0.78
0.21
c220 List of surnames,
0.97
0.86
0.11
0.96
0.85
0.11
c432 List of Male Names
1
0.71
0.29
0.97
0.73
0.24
12
c439 Sports commentary terms.
0.9
0.82
0.08
0.94
0.74
0.20
12
c173 List of football team names and stadiums.
0.99
0.82
0.17
0.99
0.87
0.12
c348 List of female names and titles.
0.99
0.75
0.24
0.98
0.7
0.28
c142 Conflict
0.86
0.11
0.96
0.86
0.1
Middle
0.98
0.72
c210 of male names
0.71
0.26
0.97
0.74
0.23
c564 List of cities and universities in the southeaster United States.
0.99
0.76
0.23
0.97
0.85
0.12
c533
LGBTQ+
1
0.71
0.29
0.82
0.18
12
c19 Complex relationships and interactions between fumily members and partners, 0.98
0.79
0.19
0.98
0,81
0.17
12
c263 Locations in New York City
0.95
0.67
0.28
0.95
0.7
0.25
12
c487 List of Spanish male names.
0.98
0.84
0.14
0.95
0.69
0.26
c247 Scandiavisa/Noedic names and places.
0.98
0.77
0.21
0.97
0.75
0.22
for Politics Fihnic Conflict
and outcomes. c421 Names of people and places the Middle East
0.98
0.91
0.07
0.97
0.07
c553 Islamic Terminology.
1
0.7
0.3
0.85
0.15
12
c149 List of teonis players' names.
0.95
0.73
0.22
0.98
0.72
0.26
12
c365 Criminal activities
0.95
0.77
0.18
0.97
0,82
0.15
Table 9: Training Probing Classifiers for the concepts shown in Table 8

---

Layer Cluster Tlg
nee
acc
acc
acc
12
c189
Superlatives
0.98
0.17
c248
Substance abuse.
0.97
0.6
c361
LGBTQ+ Gender-related Terms
0.9
3
c756
Gender and Sex Labels
0.87
0.72
0.15
1
0.8
0.2
0
c720
Gender and Sex Labels
0.74
0.26
1
0.55
0,45
0
c402
List of female names.
0.97
0.72
0.25
0.98
0.82
0.16
12
c127
Geopolitical entities and affiliations.
0.97
0.68
0.29
0.98
0.57
0.41
0
c707
Names of US Presidents and Politicians
0.55
0.45
1
0.55
0.45
c101
Speech verbs.
0.98
0.84
1
0.7
Negative adjectives.
Food items.
0.92
c149
Fruit and plant-reluted words.
0.7
0.3
0.14
3
c705
Tourism-relats' terms
0.95
0.67
0.28
0.91
0.83
0.08
12 c196
Verbs of Authority and Request
0.95
0.68
0.27
0.98
0.89
0.09
12
c398
Energy sources.
0.67
0.33
1
0.69
0.31
c185
Gender-related terms
0.98
0.64
0.34
0.96
0.68
0.28
3
<213
Finance and Taxation.
0.97
0.81
0.16
0.98
0.65
0.33
Descriptors geographic regions and types organizations.
Locations United States
0673
List Italian first
0.8
c67
List of male names.
0.99
0.81
0.18
0.99
0.81
0.18
c883
Nouns
0.97
0.83
0.14
0.99
0.81
0.18
c898
TV Networks
0.68
0.32
1
0.55
0.45
12
c653
List of years.
0.9
0.1
1
0.91
0.09
c697
Military Terminology
0.62
0.38
1
0.62
0.38
c560
Political idcologies and systems.
0.58
0.42
0.94
0.75
0.19
Table 10: Probe Results for some concepts chosen from several layers ALBERT
Blaney-
Magrussen
lerting
chLone
Bowyen...
Lavendish
ALaMOKy
A
grader Integrated preeee ww-.eexttgcurtti cul Nontessari
schoolyard
Five-place
Sauber
Supercars
Mans
Hul
kerbbrgg
potorazerta
FIAtop
Tenn iN
Mi
co Kan
accclet batly Schol in
all girls r' choo
zed
Massa
Ayr
Ayrton Eeshovee prorio UF
OHNe
school er dders
vis
after= schoo
Tres Harvick ok
Gurney
Truux
mo
Sagan
Senna
grandh ghank Ferraris
Speedway
start Penske
Neurrey
Buemi
fürst
le
IL
Minn
O wi
Ga e
Mich
Wis
A-leve S scho hhooooobys
Sats School ogg Preporattory
headmi stress
possial
Sreeeect Seafeeetchhhoc chooleeeee Lerss
csnaadte
nuserracy
Maths
(a) Motorsport Terminology
(b) US State Abbreviations
(c) Education-related terms
Figure 10: Example of concepts that were deemed uninterpretable the BCN but were correctly labeled by
ChaiGPT

---

cluster
score
Nouns
Nouns
c268
Nouns
0.4
c405
Nouns
0.0
c315
Nouns
0.3
c231
Nouns related to various activities and objects
0.6
0468
Nouns
0.2
c524
Nouns
0.2
Nouns
Nouns related
industries
o440
Nouns related various professions and groups
0.1
c202
Noums
0.3
c237
Adjectives with clear category theme
0.2
c299
Adjectives describing attributes of products services
0.3
c96
Adjectives describing owaership. operation or support of various eatities and technologics
0,3
c95
Adjectives describing various types of related events or phenomena
0.1
c198
Adjectives with no clear label
0.2
c53
Compurative Adjectives
0.3
c335
Comparative Adjoctives
Comparative Adjectives
Descriptive/Ajeetivee Labels
c505
Location hased Adjectives
0.2
cll
Adjectives describing various types of entities
0.0
c466
Adjectives describing ownership. operation, support of various entities and technologies.
0.1
c419
Adjectives describing negative or challenging situations.
0.6
c128
Adjectives descrihing the quility or appropriateness of something.
0.4
c458
Adjectives
0.0
Compurative Adjectives
Time-related frequency adjectives
c155
Adverbs frequency
c136
Adverbs of degreefintensity.
c58
Adverbs of time and transition.
0,5
c41
Adverhs of degree and frequency.
0.5
c589
Adverb intensity/degree
0.2
c265
Adverbs of Probability and Certainty
0.3
c251
Adverbs frequency manner.
0.2
c57
Adverbs Frequency
0.4
Temporal Adverbs.
<302
Frequency Adverbs
0.2
<332
Adverbs of manner and opinion.
0.4
c546
Adverbs of degree/intensity.
0.3
c570
Adverbs of preference/cboice.
0.5
c244
Adverbs indicating degree or extent.
0.3
c222
Adverbs Time
0.5
Adverbs describing degree intensity.
c487
List numerical values.
0.2
Numerical Data.
Numericul
List numbers
c287
Numeric Data.
0.0
c101
List of numerical values.
0,5
c494
List of numerical values.
0.5
c579
Numerical data.
0.2
<537
List numerical values.
0.2
0435
Numerical data.
0.3
List numerical values.
List prices.
c398
Numerical Data.
c359
List of numerical values.
0,1
c477
List of monetary values.
0,1
c593
List of monetary values.
0.1
c80 Numeric quamtities.
0.1
Table 11: Neuron Analysis Results on Super Concepts extracted from BERT-base waaeT-bae--ccssse model. The alignment column shows the intersection between the top 10 neurons in the Super concept and the Sub concepts.

---

cluster label
score
c259
List of names
0.4
of male names.
c328
politicians, public figures, and athletes.
c220
List of sumames.
0.4
e433
List of names.
0.5
c262
List of sumames
0.4
c210
List of male first names.
0.1
c231
List of female names.
0.4
c383
List of names
0.2
c280
List of names.
0.2
surnumess.
c344
Irish sumames
0.3
c6
Sumames
0.4
c75
List of female names.
0.7
c269
List of celebrity names
0.4
c578
List of sumnames.
0.2
c535
List of names
0.2
c487
List of Spunish male names.
0.2
c340
Last names.
List of
0.1
c353
List of names in the entertainment industry.
0.2
c568
List of names.
0.4
c378
iist of sumuumes.
0.1
c375
Sumaaees
0.4
c149
List of tennis players' names.
0.4
6325
List of names.
0.2
sports players' names
c594
List of sumames.
0.6
Table 12: Name clusters extracted from the last layer of BERT-base-cased


Reviews:
[{"review_id": "8xxhkxiTIb", "reviewer": "Reviewer_VLTK", "summary": "", "questions": "1. It is mentioned that previous interpretation analysis techniques rely on having access to annotated corpora or human involvement. Could you elaborate more on the limitations this imposes? For example, does it restrict the scope or scale of analysis that can be performed?\n\n2. In the error analysis, several factors are identified affecting ChatGPT's performance like sensitive content, lexical properties, insufficient context etc. Do you have any ideas on how to further improve the prompts or methodology to address these?\n\n3. One concern could be the computational expense of querying ChatGPT extensively. Do you have any ideas to make this more efficient or cost-effective at scale?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**R: The main contribution and novelty are not clearly articulated. The authors claim they are using ChatGPT to annotate latent concepts in pretrained language models, but do not sufficiently justify why this is an important advancement compared to prior work using human annotations or predefined ontologies. The potential benefits of ChatGPT annotations are hinted at but not convincingly showcased.\nR:  It is mentioned that previous interpretation analysis techniques rely on having access to annotated corpora or human involvement. Could you elaborate more on the limitations this imposes? For example, does it restrict the scope or scale of analysis that can be performed?**\n\nA: The use of predefined linguistic concepts and human-in-the-loop methods are restrictive in different ways:\n\n- Pre-defined ontologies result in a narrow view of the model's knowledge. \n- Human-in-the-loop are both costly and subjective impeding scalability\n\nDalvi et al., (2022) showed that the pre-trained language models learn novel concepts which _do not strictly adhere to any predefined linguistic groups_. However, manual annotations of these concepts are costly, making the human-in-the-loop infeasible. For an accurate understanding of decision making in these models, we must understand how knowledge is preserved within the latent space using large scale annotations of the encoded concepts. Our research showcases that using an LLM overcomes this hurdle, offering comprehensive and highly accurate linguistic annotations for latent concepts acquired within deep NLP models. While our demonstration applies to enhancing Probing and Neuron Analysis interpretation frameworks, we firmly believe that our findings hold potential to advance the broader landscape of NLP model explainability, for instance Kim et al, 2018, Geva et al., 2021.\n\n**R: The methodology lacks key details. The concept discovery process using clustering is described briefly but specifics like the dataset, clustering algorithm parameters, number of clusters etc are missing. The process of prompting ChatGPT for annotations is also unclear - what was the exact prompt, how were hyperparameters like temperature set, how many examples were annotated.**\n\nA:  We had to condense the details due to space limitations. Nevertheless, specific details can be found in the paper:\n\n**Data details** can be found in **Section 3: Latent Concept Data (First paragraph: l234).**\n\n**Details about the clustering** algorithm parameters are elaborated upon in **Section 3: Concept Discovery (Second paragraph: l246)**. Specifically, we employed an agglomerative hierarchical clustering approach with 600 clusters. We apply Ward’s minimum variance criterion that minimizes the total within-cluster variance.\n\n**The exact prompt used:** please refer to **Section 2.2 Concept Annotation (l180).** The exact prompt is provided at the conclusion of this section, while additional prompts tested during the study are listed in Appendix A.1.\n\n**ChatGPT hyperparameters: Section 3 Concept Annotation (Third paragraph: l251)** contains the relevant information. We used a temperature of 0 to control output randomness and generate deterministic responses. We used the default value (0.95) for the top p.\n\n**The number of annotated examples:** We employed GPT to annotate 39K concepts across 5 pre-trained language models **(Section 5.1, last paragraph: l418).** Additionally, to perform a qualitative comparison of ChatGPT's annotations to human annotations of latent concepts, we annotated 269 concepts within the final layer of BERT-base-cased. We will mention this more clearly in the paper.\n\n**R: In the error analysis, several factors are identified affecting ChatGPT's performance like sensitive content, lexical properties, insufficient context etc. Do you have any ideas on how to further improve the prompts or methodology to address these?**\n\nA: Sensitive Content: OpenAI has introduced a [moderation endpoint tool](https://platform.openai.com/docs/guides/moderation) that, based on the input, identifies which content policy the input violates. We can use the classification label from the tool as a coarse label for the encoded concept. Alternatively, we could explore a version of LLM where content policy models are deactivated.\n\nLexical Properties: To tackle this issue, we propose creating multiple diverse prompts that capture specific linguistic properties, in addition to the generic prompt. This approach enables us to offer multiple explanations for a latent concept. In Appendix A2-A3, we demonstrated how prompts can be adjusted to encompass specific properties like common ngrams or prevalent parts of speech tags between words.\n\nProviding Context: Our findings indicate that utilizing contextual information (up to 10 sentences) with LLM enhances annotation accuracy. Refer to Figure 3c for an example. For larger concepts with numerous words, a cost-effective strategy could involve offering neighboring 5-grams for each word rather than complete sentences, or subsampling the words for which context is provided.\n\n**R: One concern could be the computational expense of querying ChatGPT extensively. Do you have any ideas to make this more efficient or cost-effective at scale?**\n\nA: An idea might involve limiting the number of words in an underlying concept for making queries. We haven't yet investigated the alterations in annotation accuracy while using a smaller concept subset for querying. Another complementary approach could involve implementing caching and maintaining a concept database along with labels. If a significant overlap is found between the concept to be queried and concepts already existing in the database, it may be feasible to leverage the existing concept's label or opt for a significantly smaller subset of the concept during the querying process.\n\n### References:\n\nKim, B.; Wattenberg, M.; Gilmer, J.; Cai, C.; Wexler, J.; Viegas, F.; et al. 2018. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In Proceedings of the International Conference on Machine Learning, 2668–2677. JMLR.\n\nMor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484–5495, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper tackles an important problem in NLP research - interpreting and analyzing pre-trained language models. Understanding what knowledge these powerful models encode is crucial.\n* The key idea of using LLMs like ChatGPT as annotators to label latent concepts is novel and has not been explored before. This enables scaling up interpretation analysis.\n* Thorough experiments compare ChatGPT annotations to human annotations, demonstrating its accuracy and richer semantic labeling.\n* The paper clearly shows the benefits of using ChatGPT annotations through two applications - probing and neuron analysis. This provides a proof of concept.\n* Releasing a large dataset of 39K annotated concepts will facilitate further research in this exciting direction.\n* The methodology is sound, combining concept discovery through clustering with prompt engineering for annotations. Evaluation is rigorous.\n* The writing is clear and easy to follow. The introduction provides sufficient background. Experiments and results are well explained.\n* Limitations regarding computational expense, content filtering, and concept drift in LLMs are properly acknowledged.\n", "reasons_to_reject": "* The main contribution and novelty are not clearly articulated. The authors claim they are using ChatGPT to annotate latent concepts in pretrained language models, but do not sufficiently justify why this is an important advancement compared to prior work using human annotations or predefined ontologies. The potential benefits of ChatGPT annotations are hinted at but not convincingly showcased.\n\n* The methodology lacks key details. The concept discovery process using clustering is described briefly but specifics like the dataset, clustering algorithm parameters, number of clusters etc are missing. The process of prompting ChatGPT for annotations is also unclear - what was the exact prompt, how were hyperparameters like temperature set, how many examples were annotated.\n", "questions_to_authors": "1. It is mentioned that previous interpretation analysis techniques rely on having access to annotated corpora or human involvement. Could you elaborate more on the limitations this imposes? For example, does it restrict the scope or scale of analysis that can be performed?\n\n2. In the error analysis, several factors are identified affecting ChatGPT's performance like sensitive content, lexical properties, insufficient context etc. Do you have any ideas on how to further improve the prompts or methodology to address these?\n\n3. One concern could be the computational expense of querying ChatGPT extensively. Do you have any ideas to make this more efficient or cost-effective at scale?\n", "paper_topic_and_main_contributions": "**Topics**\n* Interpretation and analysis of pre-trained language models (LMs)\n* Using large language models (LLMs) like ChatGPT as annotators for latent concepts\n* Enabling fine-grained probing and neuron analysis with LLM annotations\n\n**Main Contributions**\n* Show that ChatGPT provides accurate and semantically rich annotations for latent concepts learned by LMs\n* Demonstrate that GPT annotations empower fine-grained probing for concepts beyond human-defined categories\n* Illustrate how GPT annotations enhance neuron analysis towards more intricate concepts\n* Contribute a framework for using LLMs to facilitate interpretation and analysis of LMs\n\nIn sum, the paper explores using ChatGPT to annotate latent concepts discovered through clustering of LM representations. It then leverages these annotations to enable more detailed probing and neuron analysis. The main contribution is illustrating how LLMs can be utilized to interpret and analyze pre-trained LMs at a finer granularity than enabled by human annotations.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "No", "Typos_Grammar_Style_and_Presentation_Improvements": "No"}, "extra_text": {}}, {"review_id": "XhjeYacOVW", "reviewer": "Reviewer_zd3p", "summary": "", "questions": "* Your definition of feature vectors is somewhat unclear. Which layer does the upper (l) notation for z denote?\n* What parameters were used for the agglomerative clustering?\n* Why do you use a number of clusters as stopping criteria instead of a similarity threshold, which should capture better the dynamic nature of human linguistic concepts?\n* Line 204: What is S_i? Did you mean D_i?\n* Line 138: What do you mean by \"word type\"?\n* Table 1: 0.34 is not a good agreement (even for hard tasks), it's fair at most. Personally, I wouldn't trust annotations with this score and continue to formalize the subjective aspects of the annotation task (as much as possible, I don't expect perfect agreement).\n* The neuron analysis is hard to understand. What are the scores in Table 4?\n\n", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**R: It was hard for me to understand the neuron analysis. Rewriting it in a form that will allow readers who are not familiar with that type of analysis to understand is required. Moreover, I cannot trust my review regarding this part, as it is hard to criticize something I do not fully understand. If the authors would clear this part in the rebuttal I could modify my review accordingly.**\n\nA: We had to trim down the details due to space limitations. Hopefully, with the additional page provided for the final version, we will be able to provide a more comprehensive explanation to enhance readers' understanding. Below, we attempt to further explain this section of the paper:\nIn contrast to the holistic view offered by representation analysis, neuron analysis highlights the **role of individual neurons** (or groups of them) within a neural network. The methodologies employed in neuron analysis (please refer to Sajjad et al., 2022) typically require human-in-the-loop intervention to comprehend a neuron's function (e.g., Karpathy et al., 2015, Kadar et al., 2017, etc.), making their analysis infeasible to scale. Alternatively, they rely on linguistic annotations of predefined concepts (e.g., Lakretz et al., 2019, Hennigen et al., 2020, Mu and Andreas, 2020). Consequently, the resulting neuron explanations are subject to the same limitations we address in this study.\n\nOur research demonstrates that annotating the latent space using LLM (Large Language Model) overcomes these constraints. We conducted neuron analysis on the latent space of the BERT model using the Probeless method (Antverg and Belinkov, 2022). This analysis unveiled neurons that capture intricate semantic hierarchies — such as those distinguishing adverbs of time from adverbs of frequency, those encoding price-related numbers, and those capturing monetary value indicators (Please refer to Figure 4). This method can be seamlessly applied to any encoded concept by leveraging its GPT label, facilitating the discovery of neurons associated with that concept. In summary, our approach obviates the need for human intervention or annotated data to make such discoveries, addressing a limitation that hampers other interpretation methods.\n\n**R: The contribution of the analysis, while interesting, is rather limited (two very specific domains).**\n\nA: While our demonstration applies to enhancing Probing and Neuron Analysis interpretation frameworks, we firmly believe that our findings hold potential to advance the broader landscape of NLP model explainability, for instance:\n\n- It is possible to extend the TCAV framework (Kim et al., 2018) beyond the confines of predefined concepts to the entire latent space of the model.\n- Corpus-based methods explain neurons by leveraging the concept of shared ngrams, often using linguistic concepts (Na et al. 2019) or manual annotations (Kadar et al. 2017). Our approach can be seamlessly applied to these existing methodologies.\n- The ongoing research in latent ontology induction (Michael et al., 2020, and Fu and Lapata, 2022) can take advantage from the incorporation of LLM annotation into their methodology.\n\n\n**R: Your definition of feature vectors is somewhat unclear. Which layer does the upper (l) notation for z denote?**\n\nA: It doesn't refer to a particular layer, but rather to any layer \"l\" within the network. Each layer encodes a distinct array of concepts.\n\n**R: What parameters were used for the agglomerative clustering?\nR: Why do you use a number of clusters as stopping criteria instead of a similarity threshold, which should capture better the dynamic nature of human linguistic concepts?**\n\nA: Our aim in this work was to evaluate the efficacy of LLMs as annotators of latent spaces, and hence we relied on the settings prescribed  in Dalvi et al., (2022). We used 600 clusters applying Ward’s minimum variance criterion that minimizes the total within-cluster variance. In their exploration Dalvi et al., found alternative methods such as ELbow and Silhouette to not yield reliable results.\n\n**R: Line 204: What is S_i? Did you mean D_i?**\n\nA: S_i denotes the ith sentence within dataset D. Due to limitations on sentence length during inference, we perform a forward pass over individual sentences to acquire the corresponding contextual feature vectors\n\n**R: Line 238: What do you mean by \"word type\"?**\n\nA: We are referring to a unique variation of a word, essentially an element of the vocabulary of the dataset. We consider ten distinct contexts per word type.\n\n**R: Table 1: 0.34 is not a good agreement (even for hard tasks), it's fair at most. Personally, I wouldn't trust annotations with this score and continue to formalize the subjective aspects of the annotation task (as much as possible, I don't expect perfect agreement).**\n\nA: We apologize for the typographical error on line 305. Our intention was to denote 'Fair' (as also evident in Table 1) and not 'Fair to good.' We will rectify the error. Additionally, we intend to elaborate on the complexity and subjectivity associated with annotating Q2. This complexity varies depending on the annotator's knowledge and their subjective perspective regarding the boundary between precise and imprecise labels.\n\n**R: The neuron analysis is hard to understand. What are the scores in Table 4?**\n\nA: We have tried to explain the neuron analysis section in the first comment. Did you mean Table 4 (Probing Analysis Results) or Table 5 (Neuron Analysis)?\n\nTable 4 is showing accuracy numbers for probing classifiers (representation analysis) when they are trained towards predicting different encoded concepts. Classifier accuracy is used as a measure to demonstrate how strongly a concept is represented within the latent space of a model. We use this to carry out a comparative analysis among ALBERT and XLNet models in Table 4 and with 3 other models (BERT, XLM-R and RoBERTa) in Table 7 (Appendix B).\n\nTable 5 along with its corresponding Figure 4 present the outcomes of our neuron analysis. In this analysis, we showcase how neurons within a super concept (such as all adverbs in POS tagging) are distributed among sub-concepts (various forms of adverbs). This distribution is quantified by the percentage of overlap between neurons from the sub-concepts with those from the super-concept. Our LLM annotation reveals the sub-concepts inherent to a super-concept within the latent space. This, in turn, facilitates an in-depth neuron analysis, offering more precise insights into the underlying neurons that constitute these concepts.\n\nThank you for your feedback regarding the typos and unclear sections in the paper. We are committed to addressing these concerns and improving clarity in the camera-ready version.\n\n### References:\n\nHassan Sajjad, Nadir Durrani, and Fahim Dalvi. 2022. Neuron-level Interpretation of Deep NLP Models: A Survey. Transactions of the Association for Computational Linguistics.\n\nAndrej Karpathy, Justin Johnson, and Li Fei-Fei. 2015. Visualizing and understanding recurrent networks. arXiv preprint arXiv:1506.02078.\nAkos K´ad´ar, Grzegorz Chrupała, and Afra ´Alishahi. 2017. Representation of linguistic form and function in recurrent neural networks. Computational Linguistics, 43(4):761–780.\n\nYair Lakretz, German Kruszewski, Theo Desbordes, Dieuwke Hupkes, Stanislas Dehaene, and Marco Baroni. 2019. The emergence of number and syntax units in LSTM language models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 11–20, Minneapolis, Minnesota. Association for Computational Linguistics.\n\nLucas Torroba Hennigen, Adina Williams, and Ryan Cotterell. 2020. Intrinsic probing through dimension selection. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 197–216, Online. Association for Computational Linguistics. \n\nJesse Mu and Jacob Andreas. 2020. Compositional explanations of neurons. NeurIPS 2020\nOmer Antverg and Yonatan Belinkov. 2021. On the pitfalls of analyzing individual neurons in language models. In International Conference on Learning Representations.\n\nKim, B.; Wattenberg, M.; Gilmer, J.; Cai, C.; Wexler, J.; Viegas, F.; et al. 2018. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In Proceedings of the International Conference on Machine Learning, 2668–2677. JMLR.\n\nSeil Na, Yo Joong Choe, Dong-Hyun Lee, and Gunhee Kim. 2019. Discovery of natural language concepts in individual units of CNNs, ICLR 2019\n\nJulian Michael, Jan A. Botha, and Ian Tenney. 2020. Asking without telling: Exploring latent ontologies\nin contextual representations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP ’20, pages 6792–6812, Online. Association for Computational Linguistics.\n\nYao Fu and Mirella Lapata. 2022. Latent topology induction for understanding contextualized representations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This is an interesting approach to shifting the annotation load from humans to LLMs and it can be useful in other domains as well.\n* Our current understanding of the knowledge encoded by LLMs is lacking. Thus, this is a relevant research direction. \n* The usage of the classifier on top of the feature vectors is a nice idea. ", "reasons_to_reject": "* It was hard for me to understand the neuron analysis. Rewriting it in a form that will allow readers who are not familiar with that type of analysis to understand is required. Moreover, I cannot trust my review regarding this part, as it hard to criticize something I do not fully understand. If the authors would clear this part in the rebuttal I could modify my review accordingly.\n*  The contribution of the analysis, while interesting, is rather limited (two very specific domains). ", "questions_to_authors": "* Your definition of feature vectors is somewhat unclear. Which layer does the upper (l) notation for z denote?\n* What parameters were used for the agglomerative clustering?\n* Why do you use a number of clusters as stopping criteria instead of a similarity threshold, which should capture better the dynamic nature of human linguistic concepts?\n* Line 204: What is S_i? Did you mean D_i?\n* Line 138: What do you mean by \"word type\"?\n* Table 1: 0.34 is not a good agreement (even for hard tasks), it's fair at most. Personally, I wouldn't trust annotations with this score and continue to formalize the subjective aspects of the annotation task (as much as possible, I don't expect perfect agreement).\n* The neuron analysis is hard to understand. What are the scores in Table 4?\n\n", "paper_topic_and_main_contributions": "This paper explored to what extent LLMs capture linguistic concepts. They focus on two unexplored commonsense domains -- taste and physical properties. The authors used ChatGPT to scale up the annotation process of semantic concepts and show that it is comparable to human quality. They compare different LLMs in terms of concept understanding in the two mentioned domains and find that BERT is surprisingly good compared to the much larger GPT-based models. They will also release their dataset.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* I'd probably make a section for each type of analysis. Switching from concept vectors to neurons is a bit confusing. Having one section for each flows better.\n* The explanations of both \"Probing Framework\" and \"Neuron Analysis\" are not very clear to a reader who is unfamiliar with these terms.\n* Line 140: use the pLMs acronym.\n* The paragraph referring to the results in Table 1 does not states that the precision is 67.6%. For the sake of completeness please write it.\n* Please note you use the acronym BCN before defining it."}}, {"review_id": "NDGWUAxEtP", "reviewer": "Reviewer_ZW2Y", "summary": "", "questions": "A: Have you investigated the differences across layers? Did you observe any variations in concepts from different levels?\n\nB: How does the concept clustering process, including the choice of clustering algorithms and hyperparameters, impact the overall probing process?\n\nC: Could you provide more clarity on the training process of the binary classifier? The section on Concept Probing (sec 2.3) is challenging to comprehend, particularly regarding the relationships between word clusters, annotated concepts, sentences containing the word clusters, and how they are utilized to extract features. It would be helpful to clarify how contextualized feature vectors are obtained. ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**R: The comparison of ChatGPT annotations with human-annotated concepts is based on a small sample size of 269 concepts, which might not encompass all possible scenarios or complexities.**\n\nA: We agree that our evaluation of 269 concepts may not cover all scenarios, but our aim was to compare LLM annotations to gold standard human annotations. Creating a human-annotated concept database like BCN is costly and time-consuming. Our study shows LLM annotations as a viable, cost-effective, and scalable alternative.\n\n**R: One significant aspect of latent concepts, their hierarchy or levels of abstractness, isn't discussed in this paper.**\n\nA: We touched upon this briefly while presenting our results in the Neuron Analysis (Section 5.2).  We showcased how our LLM annotation unveils the underlying sub-concepts of a super-concept within the latent space. Consequently, this enables a comprehensive examination of neurons, providing more refined insights into the foundational constituents of these concepts. Exploring this avenue further could yield intriguing possibilities, perhaps even modifying the prompt to the LLM to encode each concept in broader and finer categories leading to a form of hierarchy.\n\n**R: The findings from this paper are based on the use of ChatGPT as an annotator. The degree to which these findings can be generalized to other language models remains uncertain. However, given the focus on ChatGPT, this could be an area for future exploration.**\n\nA: While the extent to which these findings can be extended to other language models presents uncertainty, the trajectory of advancements in LLMs suggests that future models will likely be even more potent. We remain hopeful that these improvements will enable us to attain even more refined annotations of latent concepts. What would be interesting to see is whether less powerful models can achieve comparable effectiveness in annotating latent spaces, especially if our aim is to reduce inference costs. This avenue of exploration holds promising potential for further research.\n\n**R: Question A: Have you investigated the differences across layers? Did you observe any variations in concepts from different levels?**\n\nA: In our layer-wise analysis of concepts, we discovered that the concepts in initial layers predominantly focussed on word morphology and reconstructing word structure that is disintegrated due to sub-word segmentation. Consequently many concepts composed of words grouped based on common ngrams are found. But deeper within the layers, we found a mixture of concepts based on morphological, semantic and sometimes syntactic similarities. In the final layers we note that the concepts became increasingly aligned with the downstream tasks for which the model was trained. For instance, we began to observe concepts encompassing positively and negatively polarized words, indicative of sentiment classification tasks. Similarly, clusters formed in accordance with POS tags within models tailored for predicting such linguistic features. We excluded this analysis from the paper due to space limitations and its limited coherence with the overall content.\n\n**R:  Question B: How does the concept clustering process, including the choice of clustering algorithms and hyperparameters, impact the overall probing process?**\n\nA: We have explored agglomerative clustering with varying cluster sizes, ranging from 600 to 1000. Additionally, we've been investigating alternative clustering algorithms like K-means, aiming to replace the resource-intensive agglomerative clustering approach. In our results, we have found the resulting concepts from different parameters and algorithms to be similarly aligned with human-defined concepts, and have not seen these to have any impact on the overall probing process.\n\n**R:  Question C: Could you provide more clarity on the training process of the binary classifier? The section on Concept Probing (sec 2.3) is challenging to comprehend, particularly regarding the relationships between word clusters, annotated concepts, sentences containing the word clusters, and how they are utilized to extract features. It would be helpful to clarify how contextualized feature vectors are obtained.**\n\nA: After identifying an encoded concept through the process of Concept Discovery (Section 2.1) and assigning it a label C via Concept Annotation (Section 2.2), we assemble data points in the format <z_i^l, C>, where z_i^l signifies a contextualized feature of word w_i ∈ C in layer l. This feature captures the embedding of the word w_i within a sentence, derived by conducting a forward pass through the network up to layer l. To construct a training dataset for the binary classifier, we replicate the positive samples data size for the negative class. This is accomplished by extracting feature vectors using words that are not part of the concept C, denoted as ∉C. By combining these two sets of data, we amass the necessary information to effectively train a binary probe. Both probing classifiers and neuron analysis is carried out using the same data. We will rewrite and expand on details in the paper. \n\n**R: The clarity between lines 146-153 needs further improvement. Could you provide more information on the input and output? Are the representation for 5M tokens from 12 layers being clustered?**\n\nA: We run clustering for each layer independently. This enables us to analyze the types of concepts learned within various layers of the network.\n\n**R:  The use of \"ConceptNet\" in the abstract (line 022) is confusing, considering the widespread use of the knowledge graph ConceptNet.**\n\nA: We intend to use _Transformers Concept Net_ for our repository of concepts, indicating that this is a collection of concepts specifically associated with a class of models (i.e. transformers).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The interpretability of language models is a crucial research area, and this paper offers a unique approach by utilizing large-language models as annotators to enhance the interpretability of pre-trained language models. \n\n- The use of large-language models as annotators enables the generation of precise labels for concepts within pre-trained language models. While this idea is not entirely new, the validation and analysis presented in this paper are intriguing.\n- Additionally, the release of the resource Transformers Concept-Net, containing 39K annotated concepts from five pre-trained language models, will facilitate future research in the field of interpretability.\n- This approach can potentially be extended to probe concepts in other open-source large language models, thus aiding the community in understanding the differences across various language models.", "reasons_to_reject": "I wouldn't recommend rejecting the paper if the author addresses all the raised questions and improves the paper's readability. However, there are a few potential minor weaknesses to consider:\n\n- The comparison of ChatGPT annotations with human-annotated concepts is based on a small sample size of 269 concepts, which might not encompass all possible scenarios or complexities. One significant aspect of latent concepts, their hierarchy or levels of abstractness, isn't discussed in this paper.\n- The findings from this paper are based on the use of ChatGPT as an annotator. The degree to which these findings can be generalized to other language models remains uncertain. However, given the focus on ChatGPT, this could be an area for future exploration.", "questions_to_authors": "A: Have you investigated the differences across layers? Did you observe any variations in concepts from different levels?\n\nB: How does the concept clustering process, including the choice of clustering algorithms and hyperparameters, impact the overall probing process?\n\nC: Could you provide more clarity on the training process of the binary classifier? The section on Concept Probing (sec 2.3) is challenging to comprehend, particularly regarding the relationships between word clusters, annotated concepts, sentences containing the word clusters, and how they are utilized to extract features. It would be helpful to clarify how contextualized feature vectors are obtained. ", "paper_topic_and_main_contributions": "This paper addresses the challenge of interpreting the latent concepts learned by pre-trained language models (pLMs). The authors specifically focus on the task of latent concept learning, which involves labeling high-level concepts to describe clusters of words derived from pre-trained language models. They input sentences into pre-trained language models and cluster the embeddings from each layer to identify latent concepts.\n\nLarge language models, specifically ChatGPT, are utilized to generate concept labels, which are then validated for their acceptability and precision. The primary contribution of this paper is the scaling up of concept labels, which facilitates the analysis of concepts across multiple pre-trained language models and the exploration of the relationship between neuron activation and fine-grained linguistic phenomena. This paper contributes to the field of computationally-aided linguistic analysis by demonstrating the potential of large-language models, such as ChatGPT, in interpreting smaller pre-trained language models.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. The clarity between lines 146-153 needs further improvement. Could you provide more information on the input and output? Are the representation for 5M tokens from 12 layers being clustered? \n2. The use of \"ConceptNet\" in the abstract (line 022) is confusing, considering the widespread use of the knowledge graph ConceptNet."}}]

Author rebuttal:

Meta-review: Quality, clarity, originality and significance

TLDR: The paper proposes using LLMs, in particular, ChatGPT, to discover/label latent concepts in pre-trained language models. The LLM helps to find semantically richer annotations compared to human annotations. The resulting resource with annotated concepts is successfully used to perform interpretation analysis with both probing and neuron interpretation. The reviewers appreciate the importance and novelty of the studied problem and find the resulting resources and analysis framework facilitating future work in the area. The cons are about improving the clarity of the writing and the limitations of their work, which the authors have already readily acknowledged.


Pros:
1. Significance 1 - all reviewers agree that the paper addresses an important problem - interpreting pre-trained language models.
2. Originality - all reviewers appreciate the novelty of the contributions made in the paper -- using LLMs as annotators to label latent concepts and find the work can be used in other domains/pre-trained models/LLMs as well.
3. Significance 2 - the released resource of annotated concepts will facilitate further research in this direction (VLTK, ZW2Y).
4. Quality - VLTK finds the methodology to be  sound, combined with a rigorous evaluation.
5. Clarity - VLTK also finds the writing to be clear and easy to follow.

Cons:
1. Clarity - all reviewers find the neuron analysis hard to understand, but the authors already give more details in the rebuttal and indicate they'll include them in the final version as well.
2. Clarity/quality - some reviewers identified certain limitations in the analysis performed in the paper, which could be addressed in the limitations section as well, and the authors acknowledge these already in their rebuttal -- using two very specific domains (zd3p), comparison of annotations with human-annotated concepts is based on a small sample size set (ZW2Y), the findings are based only on the use of ChatGPT as an annotator (ZW2Y).
Decision: Accept-Main