Interpreting Embedding Spaces by Conceptualization
Authors: Adi Simhi
Shaul Markovitch
Venue: EMNLP 2023 Main (2023)
Keywords: Interpretability
knowledge tracing
TL;DR: This paper introduces a novel method that transforms embedding spaces into a conceptual space, allowing for better understanding, debugging, and semantic representation evaluation of large language models.

Abstract: One of the main methods for computational interpretation of a text is mapping it into a vector in some embedding space. Such vectors can then be used for a variety of textual processing tasks.  Recently, most embedding spaces are a product of training large language models (LLMs).  One major drawback of this type of representation is their incomprehensibility to humans.  Understanding the embedding space is crucial for several important needs, including the need to debug the embedding method and compare it to alternatives, and the need to detect biases hidden in the model.
In this paper, we present a novel method of understanding embeddings by transforming a latent embedding space into a comprehensible 
conceptual space.  We present an algorithm for deriving a conceptual space with dynamic on-demand granularity.  We devise a new evaluation method, using either human rater or LLM-based raters, to show that the conceptualized vectors indeed represent the semantics of the original latent ones.  We show the use of our method for various tasks, including comparing the semantics of alternative models and tracing the layers of the LLM. The code is available online https://github.com/adiSimhi/Interpreting-Embedding-Spaces-by-Conceptualization.

Interpreting Embedding Spaces by Conceptualization
Adi Simhi, Shaul Markovitch The Henry and Marilyn Taub Faculty of Computer Science
Technion Israel Institute of Technology (adi. simhi, shaulm}@cs. technion. ac. il
Abstract
cess. This limitation becomes crucial in several im- portant scenarios, including the need explain the decisions made by a system that employs the model, the necessity to debug the model and compare it with alternatives, and the requirement to identify any hidden biases within the model (Burkart and Huber, 2021; Ribeiro et al., 2016b; Madsen et al., 2022).
One the main methods for computational in- terpretation of text is mapping it into vector in some embedding space. Such vectors can then be used for variety of textual process- ing tasks. Recently, most embedding spaces are a product of training large language models (LLMs). One major drawback of this type of representation is their incomprehensibility to humans. Understanding the embedding space is crucial for several important needs, includ- ing the need to dehug the embedding method and compare to alternatives, and need detect biases hidden in the model. In this paper, we present a novel method of under- standing embeddings by transforming a latent embedding space into a comprehensible con- ceptual space. We present an algorithm for deriving a conceptual space with dynamic demand granularity. We devise new evalua- tion method, using either human rater LLM based raters, to show that the conceptualized vectors indeed represent the semantics of the original latent ones. We show the use of our method for various tasks, including comparing the semantics alternative models and tracing the layers of the LLM. The code is available online!.
Current LLMs process text by projecting it into an internal embedding space. By understanding this space, we can therefore gain an understanding of the model. Such understanding, however, is challenging the dimensions of the embedding space are usually not human-understanddable.
The importance of interpretability has been rec- ognized by many researchers. Several works present methods for explaining the decision of a system that uses the embedding (mainly classifiers) (e.g. Ribeiro et al., 2016a; Lundberg and Lee. 2017a). Some works (Senel al., 2022; Faruqui et al.. 2015) perform training or retraining for gen- erating a new model that is interpretable, thus de- touring the problem of understanding the original one. Another line of work assumes the availability of an embedding matrix and uses it to find orthogo- nal transformations, such that the new dimensions will be more understandable (Dufter and Schütze, 2019; Park al., 2017). Probing methods utilize classification techniques to identify the meaning as- sociated with individual dimensions of the original embedding space (Clark et al., 2019; Dalvi et al., 2019).
1 Introduction
Recently, there has been significant progress in Natural Language Processing thanks the devel- opment of Large Language Models (LLMss). These models are based on deep neural networks and are trained on extensive volumes of textual data (Devlin et al., 2019; Raffel et al., 2020; Liu et al., 2019).
In this work, present a novel methodology for interpretability LLMS by conceptualizing the original embedding space. Our algorithm enables the mapping of any vector in the original latent space to a vector in a human-undeerstaddable con- ceptual space. Importantly, our approach does not assume that latent dimension corresponds to an explicit and easily-interpretabbl concept.
While these powerful models show excellent per- formance variety of tasks, they suffer from significant drawback. Their complex structure hin- ders our ability to understand their reasoning pro-
http://hhhhh..ooom/////////eeenp hann:a/hi/i/necccccennnnndniggg Spacesshh-CConeepuualiiiioo
Our method can be used in various ways:

---

1. Given an input text and its latent vector, our algorithm allows understanding of the seman- tics of it according to the model.
vector & (that represents the concept o:) using any given similarity measure sim. The algorithm then outputs a vector in the conceptual space, using the similarities as the dimensions.
2. It can help us to gain an understanding of the model, including its strengths and weaknesses, by probing it with texts in subjects that are of interest to us. This understanding can be used for debugging given model or for comparing alternative models.
We have thus defined a meta-algorithm CES (Conceptualizing Embedding Spaces) that, for any given embedding method f. a set of concepts C and mapping function from concepts to text, takes vector in the latent space I. and returns vector in the conceptual space
3. Given decision system based on the LLM,, our algorithm can help to understand the de- cision and to explain it using the conceptual representation. This can also be useful in de- tecting biased decisions.
CES (1) = (sim(1, E.)...., sim(l, Gx))"
A graphical representation of the process is de- picted in Figure 1.
If we use cosine similarity as sim, and use a normalised f function, we can implement CES as matrix multiplication, which can accelerate our computation. First, observe that, under these re- strictions, cosine similarity is equivalent to the dot product between vectors. Let U uI..... us be the standard basis in dimensions a base of L. We can look at the projection of U in the space, by using function such that ((u)) = (o(u)),... ,("?)) where p(u²) cosinclu;, Gj) u4 Tz. We can now create a k matrix M = (d(u)),.... ((ug)) Using this matrix, we get CESACT M.
Our contributions are:
1. We present model-agnostic method for in- terpreting embeddings, which works with any model without the need for additional training. Our approach only requires black box that takes a text fragment as input and produces a vector as output.
2.
We present a novel algorithm that, given an on- tology, can generate a conceptual embedding space for any desired size and can be selec- tively refined to specialize in specific subjects.
2.1 Generating Conceptual Spaces
To allow conceptual representation in various lev- els of abstraction, have devised a method that, given hierarchical ontology, generates concep- tual space of desired granularity.
3. We introduce a new method for evaluating algorithms for embedding interpretation using cither a human or an LLM-based evaluator.
2 The Conceptualization Algorithm
For the experiments described in this paper, we chose Wikipedia category-directed graph as our ontology, as it provides a constantly- updated, wide and deep coverage of our knowledge, but any other knowledge graph can be used instead. Since the edges in the Wikipedia graph are not labeled, we performed an additional step of assigning score to each edge, based on its similarity to its siblings, which we named siblings score (see Appendix A).
Let be a space of textual objects (sentences, for example). Let L = L1 x x... x Ls be a latent embedding space of k dimensions. Let f T + L be function that maps text object to vector in the latent space. Typically, ff will be an LLM or LLM-based.
Our method requires two components: A set of concepts = :t,....,C, On defining a conceptual space = C1 x ... x On. and a mapping function T T:C. C' + T that returns a textual representation for each concept in C. In the pre-processing stage, we map each con- cept E C' to vector in L by applying of on r(c), the textual representation of We thus define vectors L, CI...., Cn2 such that a = f(r(c:)..
A major strength of the hierarchical representa- tion of concepts is its multiple levels of abstrac- tion. For our purpose, that means that we can re- quest concept space with a given level of gran- ularity. Given concept graph G, we define d(c), the depth of each concept (node) as the length of the shortest path from the root. We designate by C' (c E Cld(c) = i) as the set of all concepts with a depth of exactly i. For example, MATHE- MATICS and HEALTH are concepts from C²,, and
Given a vector / E L (that typically represents some input text), we measure its similarity to each

---

Conceptual Space
Latent Space
CES(L
f(r(ou))
simo
t= (())
sim(1,)) 61
G [((aa))
Figure 1: An outline of methodology.
MATHEMATICAL TOOLS and PUBLIC HEALTH are their direct children and are concepts from C².
Algorithm Selective Refinement
Input:T", size, remove Output:C*
2.2 Selectively Refined Conceptual Spaces
C C C
One problem with fix-depth conceptual spaces is the large growth in the number of nodes with the increase in depth. For example, in our implemen- tation, |C'| 37, |C²| 706 and |C&| 3467. Another problem arises in domain-specific tasks, where high-granularity concepts are needed in spe- cific subjects but not others. Lastly, is often difficult to know ahead of time what is the required granularity for the given task.
while [C] < size do
emb +- AVG (CESFCCTF()))
è concept in C with max weight in emb best + +p% of children(() with highest sib-
lings score
C+ CU best
remove True then C+C
end if end while return C
We have therefore developed algorithm that, given contextual text T' C T of input texts and the desired concept-space size, generates a con- cept space of that size with granularity tailored to T". The main idea is to refine categorics that are strongly associated with T', thus enlarging the dis- tances between the textual objects, allowing for more refined reasoning. We use the symbol C* to indicate a concept space that is created this way.
utilize the labels of the training examples alongside their text. We assign each concept the examples for which it is the top concept. The entropy of this then combined linearly with the text-based weight described above to de- termine its final value. As before, the node with the maximal value is chosen for expansion. The underlying intuition is that concepts representing texts from different classes require refinement allow better separation.
The algorithm (1) starts with C² as its initial con- cept space. It then iterates until the desired size is achieved. At each iteration, the contextual text 7" is embedded into the current space using CES. The concept with the largest weight after the projection to CES is then selected for expansion. The intu- ition is that this concept represents main topic of the text, and will therefore benefit the most from more refined representation. The algorithm selects its best p% children for some p. judged by their siblings score, and adds them to the current con- ceptual space. In addition, the algorithm utilizes a flag removeP to decide whether to remove the expanded concept. We observed that retaining the parent often improve the quality of the model interpretation.
2.3 Mapping Concepts to Text
The function 7 maps concepts to text. When the concepts the ontology have meaningful names, such as in the case of Wikipedia categories, we can just use that maps into these names. We have also devised a more complex function, f, that maps concept
concatenation of the concept name with the names of its children?. Given a concept e with name t, and children names top and Log, f(c) = "Le such as le and leg". This
the embedding is used for classification task,
"We take the two best children (the highest siblings score)

---

approach has two advantages: It exploits the elab- orated knowledge embedded in the ontology for potentially more accurate mapping, and it produces full sentences, which may be a better fit for f that was trained on sentences.
same examples, the difference in the representation of two additional models (SBERT and ST5).
3.2 Evaluation on Classification Tasks
To show that our representation matches the origi- nal one generated by the LLM, we first show that learning using the original embedding dimensions as features and learning using the conceptual fea- tures yield similar classifiers. Most works try to show such similarity by comparing accuracy re- sults. This method, however, is prone to errors. Two classifiers might give while agreeing only on 60% of the cases. Instead, we use method that is used for rater agreement, reporting two numbers: the raw agreement and Cohen's ' kappa coefficient (Cohen, 1960).
3 Empirical Evaluation
It is not easy to evaluate an algorithm whose task is to create understandable representation that matches the original incomprehensible embedding. We performed a series of experiments, including a human study, that show that our method in- deed achieved its desired goal. For all the exper- iments, we have used RoBERTa sentence embed- ding model (Reimers and Gurevych, 2019; Liu et al., 2019) as our f. unless otherwise specified. All models used in this work were applied with their default parameters. Whenever the concept space C* was used, we set size 768 to match the size used by SRoBERTa, but we observed that using much smaller values yielded almost as good results. The default value for removeP is false. For r, the function that maps concepts to text, we have just used the text of the concept name (with a length of 4.25 words on average in G). 4
accuracy of 80%,
We use the following datasets (all in English): AG News 5, Ohsumed and R8 6, Yahoo (Zhang et al., 2015), BBC News (Greene and Cunning- ham, 2006), DBpedia 14 (Zhang al., 2015) and 20Newsgroup 7. We use only topical classifica- tion datasets, as the concept space we use does not include the necessary concepts needed for tasks like sentiment analysis. If a dataset has more than 10,000 examples, we randomly sample 10,000. The results are averaged over 10 folds. We use ran- dom forest (RF) learning algorithm with 100 trees and a maximum depth of 5. The conceptual space used by CES is C", using the training set as the contextual text T".
3.1 Qualitative Evaluation
We first show several examples of conceptual repre- sentations created by CES to get some insight into the way that our method works. We have applied SRoBERTa to 3 sentences from 3 different recent CNN articles to get latent embedding vectors. We have used the first 10 sentences of each article as the contextual text T' for generating C*.
Table 2 shows the agreement between a random forest classifier trained on the LLM embedding and classifier trained on the conceptual embedding generated by CES. For reference, we also show the agreement between the LLM-based classifier and a random classifier. We report raw agreement and kappa coefficient (with standard deviations). We can see that all the values are relatively high, indicating high agreement between the LLM em- bedding and CES's embedding. Note that Kappa can range from -1 to +1 with 0 indicating random chance. For the sake of completeness, we also re- port the accuracies of the two classifiers which are proved to be quite similar.
Table shows the conceptual embeddings gen- erated by CES. We show only the 3 top concepts with their associated depth. Observe that the con- ceptual vectors are understandable and intuitively capture the semantics of the input texts. Note that the representations shown are not based on some new embedding method, but reflect SRoBERTa's understanding of the input text. In Appendix E, we study, using the same examples, the effect of the concept-space granularity on the conceptual repre- sentation, using a fixed-depth concept space instead of C Lastly, in Appendix F, we study, using the
We repeated the experiment using a KNN clas- sifier (n=5) with cosine similarity. The results are shown in Table 3. We can see much higher agree-
'Model dildiiiirrobetta-vv from Hugging Face. sim- plicity we refer it as SRoBERTa
'Available hutpinetttpp://rrouss..o http::/ouuuu..i.iiiiuiuuuuggguuu guppi.i/ cor pus of newnneesattee articles.html "Available
'The total runtime for the experiments described here was 24 hours on 8 cores of Intel Xcon Gold 5220R CPU 2,20GHz. The graph creation from the full Wikipedia dump of 2020 tonk several days with a maximal memory allocation of 100GB.
http://wwwagll.coo/eeeee
ohr8r52 used for Ohusmed and R8 datasets
"taken from sklearn datascts python library

---

sentence
C2
This is now very contagious virus
VIRUSES (3)
DISEASE OUTBREAKS
VIRUS TAXONOMY (4)
The search for life on Mars and ocean worlds in our solar system
LIFE IN SPACE (2)
HYPOTHETICAL LIFE FORMS (2)
DISCOVERIES BY
ASTRONOMER (3)
The bias in these Al systems presents a serious issue
ARTIFICIAL INTELLIGENCE (3)
MACHINE LEARNING
COMPUTING AND SOCIETY (3)
(3)
Table 1: Example of the model outputs on the sentences. The number in parenthesis is the depth of the concept.
ment between the LLM--ased and CES-based clas- sifiers.
2. A human rater that does not have access to the training set and does not have access to the test text. The only data presented to the human is the top 3 concepts of the CES representation of the LLM embedding. 3 graduate students were used for rating.
We tested the sensitivity of our algorithm to the values of the removel = True and f parameters. The results are shown in Appendices B and C. We see that both parameters have little effect performance.
We claim that if there is high agreement between the two, then the conceptual representation indeed reflects the meaning of the LLM embedding.
Appendix D includes additional positive results on the triplets dataset (Ein-Dor et al., 2018).
To allow classification by the human raters, out of the 7 datasets described in the previous subsec- tion, we chose the 4 that have meaningful names for the classes. To make the classification task less complex for the raters, randomly sampled two classes from each dataset, thus creating binary classification problem. For each binary dataset, we set aside 20% of the examples for training classi- fier based on the LLM embedding, using the same method and parameters as in the previous subsec- tion. The resulting classifier was then applied to the remaining 80% of the dataset.
3.3 Evaluating Understandability
While these results look promising, they may not be sufficient to indicate that CES indeed reflects the semantics of the text according to the LLM. Con- sider the following hypothetical algorithm. Let D be the size of the LLM embedding space. The algo- rithm selects D random English words and assigns each to an arbitrary dimension. This hypotheti- cal algorithm satisfies two requirements: Using for the classification tasks will always be in 100% agreement with the original (as we merely renamed the features), and its generated representation will be understandable by humans, we use words in natural language. However, it is clear that it does not convey humans any knowledge regarding the LLM representation. In the next subsections, we describe a novel experimental design with hu- mans and with other models. This design aims to validate our assertion that CES produces compre- hensible representations that genuinely capture the semantics of the LLM embedding.
Out of this test set, we sample 10 examples on which the LLM-based classifier was right and 10 on which it was wrong". This is the test set that is presented to the human raters. Each test case is represented by the 3 top concepts of the CES em- bedding, after applying feature selection on the full embedding to choose the top 20% concepts. As be- fore, the conceptual space is C* with size = 768 and with the training set used as contextual text T'. The instruction to the human raters was: "A document belongs to one of two classes. The doc- ument is described by the following 3 key phrases (topics): 1, 2, and 3. To which of the two classes do you think the document belongs to?". The final human classification of a test example was com- puted by the majority voting of 3 raters. For the LLM- classification, we used two learning al- gorithms. The first is Random Forest (RF) with the 1. A classifier that was trained on a training set using the LLM embeddings.
3.3.1 Evaluation By Humans
We have designed human experiment with the goal of testing the human understandability of the latent representation by observing only its concep- tual mapping. The experiment tests the agreement, given a set of test examples, between two raters:
"except for the Ohsumed dataset where only 7 wrong an- swers were found

---

dataset
Rand/LLM raw agreement
CES/LLM raw agreement
CES/LLM kappa coefficient
CES accuracy
LLM accuracy
accuracy diff
20 Newsgroup
0.05
0.61
0.02
0.58 + 0.02
56.4
68.0
11.6
AG News
0.25
0.87
0.01
0.83 + 0.02
84.9
85.7
0.8
DBpedia 14
0.07
0.01
0.84 +: 0.01
87.0
88.4
Ohsumed
0.10
0.69
0.01
0.58 1: 0.01
40.4
41.5
1.1
Yahno
0.10
0.63
0.02
0.59 + 0.02
57.9
61.5
3.6
R8
0.23
0.87 = 0.01
0.76 + 0.02
79.9
79.8
-0.1
BBC News
0.19
0.96 0.01
0.95 + 0.02
95.8
97.0
Table 2: Agreement (raw and kappa) between LLM- and CES-based RF classifiers.
dataset
Rand/LLM raw agreement
CES/LLM
CES/LLM
CES accuracy
LLM accuracy
accuracy diff
raw agreement
kappa coefficient
20 Newsgroup
0.05
0.82 = 0.02
0.81 + 0.02
78.9
86.9
8.0
AG News
0.25
0.92 = 0.01
0.90 + 0.01
87.9
89.7
1.8
DBpedia 14
0.07
0.93 0.01
0.92 + 0.01
94.1
94.0
-0.1
Obsumed
0.10
0.75
0.02
0.73 + 0.02
65.2
72.4
7.2
Yahoo
0.10
0.75 = 0.01
0.73 + 0.02
65.2
69.2
4.0
R8
0.23
0.92
0.01
0.87 1: 0.02
89.4
93.6
4.2
BBC News
0.97 + 0.01
97.1
Table 3: Agreement (raw and kappa) between LLM- and CES-based KNN classifiers.
dataset
raw agreement with RF
raw agreement with NC
3.3.2 Evaluation by Other Models
We repeated the experiments the last subsection, with the same test sets, but instead of using human raters, we used a LLM rater. The LLM rater re- ceives the top 3 concepts, just like the human raters, and makes a decision by computing cosine similar- ity between its embedding of each class name its embedding of the textual representation of the 3 concepts. The 3 LLMs used for rating are SBERT (Reimers and Gurevych, 2019) 9, ST5 (Ni et al., 2022) and SRoBERTa. Note that the two uses of SRoBERTa are quite different. The one used for the original classification based on a training set and a learning algorithm, while the model used for rating just computes similarity between the class name and the 3 concepts.
AG News
0.85
0.80
BBC News
0.80
0.75
Ohsumed
0.65
0.82
Yahoo 0.65
0.75
Table 4: Human- -FF and human-NC raw agreement.
same parameters as Section 3.2. The second is Nearest-Centee Classifier (NC) which computes the centroid of each class and returns the one clos- est to the test case.
Table shows the raw agreement between the LLM-based and the human classification, for the two learning algorithms. Kappa coefficient was not computed as the test set is too small. The results are encouraging as they show quite a high agree- ment. Note that the learning algorithm had access the full training set, while the human could see the conceptual representation of only the test case. Indeed, we can see that the agreement with the less sophisticated NC classifier is higher on average than the agreement with the RF classifier.
An alternative approach to ours is to assign a meaning to each dimension the latent space. We denote this approach by Dimension Meaning As- signment (DMA). We have designed two competi- tors that represent the DMA approach.
The first one, termed DMA words is based on
2 "Model bert-buse-nli-ne mai--men-ttoenss from Hugging Face Model sentence-t5-large from Hugging Face

---

vocabulary of 10K frequent words We represent each word by our LLM, yielding 10K vectors of size 768. We now map each dimension to the word with the highest weight for it. We make sure that the mapping is unique. The second one, which we call DMA concepta is built in the same way, us- ing, instead of words, the concepts in C Lastly, DMAC C* is added as an ablation experiment where the transformation part of method is turned off. Table 5 shows the results expressed in raw agree- ment. We can see that CES method performs better than the alternatives (except for two test cases).
4.2 Using CES for LLM Tracing
Another application of CES is analyzing the layers of the LLM, in a similar way to the Logit lens method (nostalgebraist, 2020). This be very useful for debugging the model. We show here an
example of tracing the changes of the embedding through the layers of BERT and GPT2!2
We create a representation for each layer by calculating the average of the token embeddings within that layer! We then use CES map these vectors to the conceptual space. We then trace the relative weight of each concept throughout the layers to gain an understanding of the modeling process.
The previous two subsections (3.3.1 and 3.3.2) have presented evidence supporting our fundamen- tal claim that the conceptual representation gen- erated by CES accurately captures the semantic content of the input text based on the LLM model.
In this case study, we analyze the text "Govern- ment" using C conceptual space. We follow 6 concepts: the 3 top ones for the initial layer and the top ones for the final layer. Figure 2 shows the changes in the weights of the concepts through- out the modeling process. The y axis shows the ranking of each concept.
4 CES Application
4.1
Using CES for Comparing Models
One major feature of our methodology is that it allows us to gain an understanding of the semantics of trained models. This allows when considering alternative models, compare their semantics, to understand the differences between their views of the world, and compare their potential knowledge gaps. We demonstrate this by comparing the views of three LLMs, SBERT, ST5, and SRoBERTa on two example texts, by observing their conceptual representations in ( generated by CES.
The figure offers a clear visualization of the changes in the relative weights of these concepts across the different layers. Notably, in Figure 2a, the concepts TRANSPORT, MEDICINE, and COR- RUPTION, which had low rankings in the initial layer, have significantly ascended to become the top concepts in the final layer. similar transition using different concepts is found in Figure 2b.
Related Work
Table 6 shows the top 3 concepts of the vector generated by CES for the LLMs given the text "FC Barcelona". We can see that while SRoBERTa and ST5 give high weight to the sport aspect of the input text, SBERT does not.
The problem of interpretability has received sig- nificant attention recent years. A large body of research (Ribeiro et al., 6a; Lundberg and Lee, 2017b; Yeh et al., 2020; Rajani et al., 2020; Ribeiro et al., 2018; Ebrahimi et al., 2018; Ross et al., 2021; Wu et al., 2021) is devoted to generate explana- tion for the decision of the model (mostly classi- fication). Many methods utilize nearby examples or counterfactuals provide users with reasoning behind the decision.
To validate this observation, we compare, for each of the 3 models, the cosine similarity in the latent space between "FC Barcelona" and the sport- related phrase "Miami Dolphin", to its similarity to the city-related phrase "Politics in Spain".
The results support our observation. SBERT embedding is more similar to the city aspect em- bedding while the two others more similar to the sports text embedding.
Several works set a goal, like ours, of under- standing the model itself, rather than its decisions. Most of these works attempt to assign some mean- ing to the dimensions, either of the original latent space or of different space that the original one is transformed to.
In Table 7, for the input "Manhattan Project", we can see that ST5 gives high weight to the military project while SBERT gives high weight to concepts related to New York and to theater. SRoBERTa recognizes both aspects.
12 Model pert-base-uncasee and gpt2 from Hugging Face. including the input initial embedding. "tther methods, such as using the last token, can he casily incorporated.
"httss//www.it..ce http::/wwww....ccccccccco//aiiaa ccprice/wordlist.. 10000

---

Evaluation Model
Method
Yahoo
BBC
AG News
Ohsumed
SBERT
DMA words
0.60 0.60
0.55 0.80
0.55
/ 0.47
DMA concepta
0.65 0.55
0,55 0.70
0.50 0.35
0.65 / 0.44
DMA C
0.60/ 0.50
0.60 0.75
0.65 0.70
0.59 0.42
CES
0.80/ 0.90
0.70 0.85
0.75 0.60
0.71 0.53
ST5
DMA
0.65/0065
0.35 0.60
0.45 0.50
0.65 0.47
0.70 0.70
0.45 0.30
0.60
0.53 0.35
DMA C*
0.60/ 0.60 0.60
0.65 0.40
0.55 0.60
0.53 / 0.35
CES
0.80 / 0.90
0.80 0.75
0.60 0.55
0.76 0.82
SRoBERTa
DMA words
0.50 / 0.70
0.35 0.40
0.55 / 0.60
0.59 0.41
DMA CRAnCepE
0.60 0.40
0.35 0.50
0.60 0.45
0.71 0.53
DMA C
0.40 / 0.60
0.55 0.40
0.35 0.40
0.71 / 0.53
CES
0.85 / 0.85
0.75 0.80
0.70/ 0.65
0.82 / 0.76
Table 5: Evaluation by SRoBERTa-RF SROBERTa-NC.
Model
d(t. Miiami Dolphins")
d(t,"politics Spain")
SBERT
GOVERNMENT OF SPAIN
SPANISH PEOPLE
CATALAN CULTURE
0.42
0.57
SRoBERTa
TEAMS
SPORT BY CITY
SAINTS
0.40
0.30
STS
TEAM SPORTS
SPORTS TEAMS
PEOPLE ORGANIZATION
0.79
0.75
Table 6: t= tFFC Barcelona", FC Bancelona top concepts using CES and validation by the LLM.
Model
CI
d(t. "Nuclear bomb")
d(t. "New York")
SBERT
CITY-STATES
NEW YORK CITY NIGHTLIEE
THEATRE BY CITY
0.49
0.74
SRoBERTa
MILITARY PROJECTS
NEW YORK CITY NIGHTLIFE
SPACE PROGRAMS
0.36
0.34
STS
NUCLEAR TECHNOLOGY
NUCLEAR POWER
NUCLEAR ENERGY
0.84
0.79
Table 1="Manhattan Project", Manhattan Project top 3 concepts using CES and validation by the LIM..
One relatively early approach tries to find or- thogonal or close to orthogonal transformations of the original embedding matrix (Dufter and Schütze, 2019; Park et al., 2017; Rothe and Schütze, 2016) such that set of words with high weight in given dimension related and thus hopefully represent some significant concept. The advantage of these orthogonal methods is that they do not lose infor- mation due to the orthogonality. Several of these works (Arora et al., 2018; Murphy et al., 2012;
each dimension. Note that our work is different as does not assume that each latent dimension corresponds to human-understanduble concept.
Recent methods (Dar al., 2022; nostalgebraist. 2020) assume access the model's internals, par- ticularly the un-embedding matrix, map a latent vector to the token space.
Other works (Yun et al., 2021; Molino et al., 2019) created new tools to help interpret the model. Yun et al. (2021) uses dictionary learning to view the transformer model as linear superposition of transformer factors. Molino et al. (2019) introduces tool for doing simple operations such as PCA and t-SNE on embedding.
Subramanian et al., 2018; Ficsor and Berend, 2021; Berend, 2020) transform the original embedding to sparse one to improve the interpretability of each dimension. One limitation of these methods is their reliance embedding/diccionnryy matrix.
Probing methods try to interpret the model by studying its internal components. Vig et al. (2020)
Senel et al. (2018) assigns a specific concept to

---

109
100
101
200
10
30
10
10
Lxyers
Layers
(a) BERT
(b) GPT2
Figure 2: BERT/GPT2 layers for "Government" text.
make changes the input find what parts of the model (specific attention heads) a bias comes from. Tenney et al. (2019) use probing on BERT model to find the role of each layer in the text in- terpretation process. Bau et al. (2019) and Dalvi et al. (2019) show how linguistic properties are dis- tributed in the model and in specific neurons. Clark
Limitations
There are several limitations to the work presented here:
1. For the tracing application (Section 4.2), we used rather limited (but common) approach of averaging the embedding vectors of cach token.
al. (2019) create attention based probing clas- sifier to find out what information captured by each attention head of BERT. Lastly, Sommerauer and Fokkens (2018) use supervised classifiers to extract semantic features.
2. Most of our experiments performed us- ing only the SRoBERTa model.
3. We did not include experiments using CES for explanation and for debugging. Such applica- tion will be performed in future work
Some works (Mathew et al., 2020; An et al., 2022: Bouraoui et al., 2022; Faruqui et al., 2015; Senel et al., 2022; Şenel et al., 2021) tackle the problem by training retraining to create interpretable model. Unlike those methods, our approach focuses on understanding the original models while preserving their performance, rather than using interpretable models as substitutes.
Our evaluation was done using only Wikipedia category graph as an ontology. Using alternative knowledge graphs can be interest.
8 Ethics Statement
6 Conclusion
The primary objective of our method is to facil- itate a deeper comprehension of the embedding space. Our model serves as tool to enhance un- derstanding of the underlying model. By utilizing the model, offensive mappings in the concept space of CES can be revealed. However, it is important to note that our model is strictly intended for the pur- pose of assisting in understanding and debugging problems in LLMs.
In this work, introduce novel approach to LLM interpretation that maps the latent embed- ding space into space of concepts that are well- understood by humans and provide good coverage the human knowledge. We also present a method for generating such a conceptual space with an on- demand level of granularity.
We evaluate our method by an extensive set of ex- periments including novel method for evaluating the correspondence of the conceptual embedding the meaning of the original embedding both by humans and by other models. Finally, we showed applications our method for comparing models, analyzing the layers of the model, and debugging.
References
Ning An, Meng Chen, Li Lian, Peng Li, Kai Zhang, Xiaohui Yu, Yilong Yin. 2022. Enabling the interpretability of pretrained venue representations

---

using semantic categories. Knowl. Based Syst., 235:107623.
deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. 2018. Linear algehraic structure of word senses, with applications to polysemy. Trans. Assoc. Comput. Linguistics, 6:483-495.
Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James R. Glass. 2019, Identifying and controlling important neurons in neu- ral machine translation. In 7th International Confer- ence on Learning Representations ICIR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Philipp Dufter and Hinrich Schütze. 2019. Analytical methods for interpretable ultradense word embed- dings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Nat- ural Language Processing, SMNLL-IJCNLL 2019, Hong Kong, Chima, November 3-7, 2019, pages I 1185- 1191. Association for Computational Linguisties.
Gábor Berend. 2020. Sparsity makes sense: Word sense disambiguation using sparse contextualized word rep- resentations, In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 8498-8508.
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. Hotflip: White-box, adversarial examples for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers, pages 31-36. Association for Computational Linguisties.
Zied Bouraoui, Víctor Gutiérrez-Basulto, and Steven Schockaert. 2022. Integrating ontologies and vector space embeddings using conceptual spaces (invited paper). In International Research School in Artificial intelligence in Bergen, AIB 2022, Jume 7-11, 2022, University of Bergen, Norway, volume 99 of OASIes, pages 3:1 3:1.3:30. Schloss Dagstuhl- - Leibnniz-Zentrum für Informatik.
Liat Ein-Dor, Yosi Mass, Alon Halfon, Elad Venezian. Ilya Shnayderman, Ranit Aharonov, and Noam Slonim. 2018. Learning thematic similarity metric from article sections using triplet networks. In Pro- ceedings of the 56th Annual Meeting of the Associa- tion for Computational Linguistics, ACL 2018, Mel- bourne, Australia, July 15-20, 2018, Volume 2: Short Papers, pages 49 49-54.4405544 Association for Computational Linguistics.
Nadia Burkart and Marco F. Huber. 2021. A survey on the explainability of supervised machine learning. J. Artif. Intell. Res., 70:245-317.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of bert's attention. In Proceed- ings of the 2019 ACL Workshop BlackboxNLP: An- alyzing and Interpreting Neural Networks for NLP, BlackboxNI.PG. @wt.hhxxN..@@AA........ ACI. 2019, Florence, Italy, August 1, 2019, pages 276-286. Association for Computational Linguistics.
Manaal Faruqui, Jesse Dodge, Sujay Kumar Jauhar, Chris Dyer, Eduard H. Hovy, and Noah A. Smith. 2015. Retrofitting word vectors to semantic lexi- cons. In NAACL HLT 2015, The 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- mologies, Denver, Colorado, USA, May 31 June 5, 2015, pages 1606-1615. The Association for Com- putational Linguistics.
Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological mea- surement, 20(1):37-46.
Tamás Ficsor and Gábor Berend. 2021. Changing the basis of contextual representations with explicit mantics. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing: Student Research Workshop. pages 235 247.
Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, Anthony Bau, and James R. Glass. 2019. What is one grain of sand the desert? analyz- ing individual neurons in deep NLP models. In The Mhirty-Third AAAI Conference on Artificial Intelli- gence, AAAI 2019, The Thirty-Firss Innovative Ap- plications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, FAAI 2019, Hon- olulu, Hawait, USA, January 27 February 1, 2019, pages 6309-633113 AAAI Press
Derek Greene and Padraig Cunningham. 2006. Practi- cal solutions to the problem of diagonal dominance in kernel document clustering. In Machine Learn- ing, Proceedings of the Twenty-Third International Conference (ICMI. 2006), Pittsburgh, Pennsylvania, USA, June 25-29, 2006, volume 148 of ACM Interna- tional Conference Proceeding Series, pages 377-384. ACM.
Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. 2022. Analyzing transformers in embedding space. arXiv preprint arxiv:2209002335.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis,

---

Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907. 1692.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res., 21:140:1-140:61
Scott M. Lundberg and Su-In Lee. 2017a. A unified approach to interpreting model predictions. In Ad- vances in Neural Information Processing Systems 30: Annual Conference on Neural Information Process- ing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 4765-477 -4774.
Nazneen Fatema Rajani, Ben Krause, Wengpeng Yin, Tong Niu, Richard Socher, and Caiming Xiong. 2020. Explaining and improving model behavior with k nearest neighbor representations. arXiv preprint 20Xiv:2010.09030.
Scott M. Lundberg and Su-In Lee. 2017b. A unified approach to interpreting model predictions. In Ad- vances in Neural Information Processing Systems 30: Annual Conference on Neural Information Process- ing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 4765-4774.
Nils Reimers and Iryna Gurevych. 2019, Sentence-bert: Sentence embeddings using siamese nett-ns.works. In Proceedings of the 2019 Conference on Empiri- cal Methods in Natural Language Processing and the 9th International Joint Conference on Natural Longuage Processing, EMNLP-IJCNV 2019. Hong Kong, China, November 3-7, 2019, pages 3980-3990. Association for Computational Linguistics.
Andreas Madsen, Siva Reddy, and Sarath Chandar. 2022. Post-hoe interpretability for neural nip: A survey. ACM Computing Surveys, 1-4.)
Binny Mathew, Sandipan Sikdar, Florian Lemmerich, and Markus Strohmaier. 2020. The POLAR frame- work: Polar opposites enable interpretability of pre- trained word embeddings. In WWW '20: The Web Conference 2020, Taipei, Taiwan, April 20-24, 2020, pages 1548 1558, ACM/ / IW3C2.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 201 Model-agnostic interpretability of machine learning, arXiv preprint arXiv: 1606.05386
Marco Túlio Ribeiro, Sameer Singh, and Carlos
Guestrin. 2016b. "why should I trust you?": Ex- plaining the predictions of any classifier. In Pro- ceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Min- ing, San Francisco, CA, USA, August 1 13-17, 2016, pages 1135-1144. ACM.
Piero Molino, Yang Wang, and Jiawei Zhang. 2019. Parallax: Visualizing and understanding the seman- tics of embedding spaces via algebraic formulae. In Proceedings of the 57th Anmual Meeting of the Associ- ation for Computational Linguisties: System Demon- strations, pages 165-180.
Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018 Anchors: High-precision model- agnostic explanations. In Proceedings of the Thirty- Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Arti- ficial Intelligence (IAAI-18), and the Sth AAAI Sym- posium on Educational Advances in Artificial Intel- ligence (EAAI-18), New Orleans, Louisiana. USA, February 2-7, 2018, pages 1527-1535. AAAI Press.
Brian Murphy, Partha Pratim Talukdar, and Tom M. Mitchell. 2012. Learning effective and interpretable semantic models using non-negative sparse embed- ding. In COLING 2012, 24th International Confer- ence on Computational Linguistics, Proceedings of the Conference: Technical Papers, 8-15 December 2012, Mumbai, India, pages 1933-1950. Indian Insti- tute of Technology Bombay.
Alexis Ross, Ana Marasovic, and Matthew E. Peters. 2021. Explaining NLP models via minimal con- trastive editing (mice). In Findings of the Associ- ation Computational Linguistics: ACI/UJNLPP 2021, Online Event, August 1-6, 2021, volume ACL/JUNLP 2021 of Findings of ACL, pages 3840- 3852. Association for Computational I Linguistics.
Jinno Ni. Gustavo Hernandez Ábrego, Noah Constant, Ji Ma, Keith B. Hall, Damiel Cer, and Vinfei Yang. 2022. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. In Findings of the As- sociation for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 1864-1874. Association for Computational Linguistics.
nostalgebraist. 2020.
Interpreting gpt: the logit lens. LessWrong, Available at https://ww/ lesswrong. wttccoottssseeeeeeeeoteeeeeesset
Sascha Rothe and Hinrich Schütze. 2016. Word embed- ding calculus in meaningful ultradense subspaces. In Proceedings of the 54th Annual Meeting of the As- sociation for Computational Linguistics. ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 2: Short Papers. The Association for Computer Linguisties.
interpreting gpt- the- logit-
Sungjoon Park, Jin Bak, and Alice Oh. 2017 Rotated word vector representations and their inter- pretability. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Process- ing, EMNLP 2017, Copenhagen, Denmark, Septem- ber 9-11, 2017, pages 401-411. Association for Com- putational Linguistics.
Lüth Kerem Senel, Furkan Sahinuç, Veysel Yúcesoo, Hinrich Schütze, Tolga Çukur, and Aykut Kog. 2022. Learning interpretable word embeddings via bidirec- tional alignment of dimensions with semantic con- cepts. Inf. Process. Manag,, 59(3).02255.

---

Lütf Kerem Senel, Ihsan Utlu, Furkan Şahinuç, Hal- dun M Ozaktas, and Aykut Koç. 2021. Imparting interpretability to word embeddings while preserving semantic structure. Natural Language Engineering, 27(6):721-746.
learning: contextualized embedding as a linear su- perposition of transformer factors. In Proceedings of Deep Learning Inside Out (DeelJO); The 2nd Work- shop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 1-10.
Lutfi Kerem Senel, Ihsan Utlu. Veysel Yücesoy, Aykut Kog, and Tolga Çukur. 2018. Semantic structure and interpretability of word embeddings. IEEE ACM Trans. Audio Speech Lang. Process. 26(10):1769 1779.
Xiang Zhang. Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text clas- sification. In Advances in Neural Information Pro- cessing Systems 28: Annual Conference on Neural In- formation Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 649-657.
Pia Sommerauer and Antske Fokkens. 2018, Firearms and tigers are dangerous, kitchen knives and zebras are not: Testing whether word embeddings can tell. Proceedings of the 2018 EMNLP Workshop Black- boxNLP: Analyzing and Interpreting Neural Net- works for NLP. pages 276-286.
A Siblings score
Let G = (V. be a knowledge graph, where V is set of concepts and E " V x V is set of links between concepts. Let Obj(c) be the set of objects belonging to concept c. We say that eı is-a c2 if Obj(c) S Obj(c2)- We define parents(c) {c' E
Anant Subramanian, Danish Pruthi, Harsh Jhamtani, Taylor Berg-Kirrpatten Kirkpatrick, and Eduard H. Hovy. 2018. SPINE: sparse interpretable neural embeddings. In Proceedings of the Thirty-Seconel AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th inno- vative Applications of Artificial Intelligence (IAAI- 18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (FAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 4921 4928. AAAI Press.
{d E cl(d,c) E} and children(c) Vl(e, E E}. Given a node e and a parent node P. we define siblings(c, p) = children(p) - {c}. The main idea behind our method of detecting is-a links is that set of siblings connected to a specific parent through is-a links should be similar, We estimate the similarity between a node and its siblings by the similarity between their set of par- ents. Instead of using a binary decision, we chose to assign a continuous value in [o, 1] that will be used by our algorithms for generating conceptual spaces.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NIP pipeline. In Proceedings of the 57th Conference of the Associa- tion for Computational Linguistics, ACL 2019, Flo- rence, Italy, July 28- August 2, 2019, Volume I: Long Papers, pages 4593-4601. Association for Computa- tional Linguistics.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stu- art M. Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. In Advances in Neural Information Processing Sys- tems 33: Annual Conference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, viriual.
We can now define the siblings score of an edge (p, c) as:
AVERAGGFessttnnnn [parents(c) (erennttte)ppprettale parents(c))
We remove from each node X% (35% in our experiments) of its parent links with the lowest siblings score.
T Wu, MM Tulio Ribeiro, J Heer, and Weld. 2021, Polyjuice: Generating counterfactuals for explaining, evaluating, and improving models. In Joint Confer- ence of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Interna- tional Joint Conference on Natural Language Pro- cessing (ACL-IJCNLP 2021).
B Testing the effect of the removeP parameter
Our algorithm for generating on-demand concep- tual spaces 2.2 retains a parent after expanding it and adding its children. This has several advan- tages, but we commonly prefer embedding spaces that are orthogonal. In this section, we test the performance of our method if we delete the par- ent after expansion (controlled by the remove? parameter). We ran the classification task de- scribed in Section 3.2 with the only difference that the removeP parameter is set to True. The results are shown in Table 8. We can see that the differ- ences are insignificant.
Chih-Kuan Yeh, Been Kim, Sercan Ömer Arik, Chun Liang Li, Tomas Pfister, and Pradeep Ravikumar. 2020. On completeness-wware concept-based expla- nations in deep neural networks. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Sys- tems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Zeyu Yun, Yubei Chen, Bruno Olshausen, and Yann Le- cun. 2021. Transformer visualization via dictionary

---

dataset
Raw agrec- ment
kappa coef
raw
kappa coef
models
raw agree- ment
kappa
agree- ment
coef
removeP
remove
True
True labels and LLM labels True labels C labels
0.726
0.452
True
0,692
0.384
20 News-
0.61
+
0.58
+
0.62
0.59
+
LLM labels and C labels
0.820
0.640
group
0.02
0.02
0.02
0.02
AG
0.87
+
0.83
+
0.87
0.83
+
Table 10: Raw agreement and kappa coefficient between SRoBERTa LLM labels, True labels and CES using C labels on Wikipedia triplet dataset.
News
0.01
0.02
0.01
0.02
DBpedia
0.85
0.84
+
0.87
0.86
+
14
0.01
0.01
0.01
0.01
Ohsumed
0.69
T
0.71
0.59
D Evaluation on Similarity Task
0.01
0.01
0.02
0.02
Yahoo
0.63
0.59
+
0.64
0.60
+
In this section, we use the conceptual representa- tion in the context of algorithm that estimates semantic similarity between sentences by measur- ing cosine similarity between their embeddings, Specifically, we evaluate the agreement between us- ing the latent embedding generated by SRoBERTa and using the conceptual embedding generated by CES with C (We cannot use C* since we do not have any contextual text to be used as T').
0.02
0.02
0.01
0.01
R8
0.87
+
0.76
+
0.88
0.76
+
0.01
0.02
0.01
0.02
BBC
0.96
+
0.95
+
0.96
0.95
+
News
0.01
0.02
0.02
0.02
Table 8: LLM- and CES-based classifiers' agreement. Using removeP as True.
The dataset used is the triplet test that was gener- ated from Wikipedia articles (Ein-Dor al., 2018). Each test consists of three sentences, all from the same Wikipedia article. Two sentences are from the same section and the third is from a different section. A sentence is labeled as more similar to the one from the same section than to the one from the other section. We used subset of 1000 triplets randomly sampled from the full dataset.
dataset
kappa coef
raw agree- ment
kappa coel
agree- ment
20 News-
0.61
+
0.58
+
0.62
+
0.59
1
group
0.02
0.02
0.01
0.02
AG
0.87
0.83
+
0,87
0.83
+
News
0.01
0.02
0.01
0.01
DBpedia
0.85
0.84
+
0,84
0.83
+
0.01
0.01
0.01
0.02
Ohsumed
0.69
0.58
+
0.69
0.58
+
The results are shown in Table 10. We can see that CES embedding and SRoBERTa embedding have a high raw agreement and Kappa coefficient, larger than their agreement with the true label.
0.01
0.01
0.01
0.02
Yahoo
0.63
0.59
+
0.64
0.59
+
0.02
0.02
0.01
0.02
R8
0.87
+
0.76
+
0.88
0.77
+
0.01
0.02
0.01
0.02
E A Qualitative Evaluation using Fixed-Depth Concept Spaces
BBC
0.96
0.95
+
0.96
0.95
+
News
0.01
0.02
0.01
0.02
We ran the same qualitative evaluation, as shown in Section 3.1, the sentences taken from CNN. Instead of using the concept space C*, fixed-depth spaces, C²,, C² and C Our goal is to study the effect of the granularity of the concept space on the way the latent vectors are represented.
Table 9: LLM- and CES-based classifiers agreement.
Using ? function.
used
Testing the effect of the r function
One of the major components of our method is the T function that maps a concept into text object that is then converted to latent vector. For the experiments described in this work, have used that just outputs the concept names. In this section, we repeat the classification tests with 7 (see Section 2.3). Table 9 shows the results. We can see that the differences are insignificant.
The top five concepts of each input sentence for each concept space are presented in Tables 11, 12 and 13. For comparison, concepts of the C* concept space. We can see the refinement of the top concepts as the depth grows. Using C' the conceptual representation gives a very general and non non-specific account of the text's meaning. Using the more refined C² and C33
also include the top

---

concept spaces, we can gain deeper understanding of the input text, We can also notice that C* has an advantage over the fixed-depth alternatives as can use more refined concepts when needed without compromising the size.
F
A Qualitative Evaluation using Different Models
We ran the same qualitative evaluation, as shown Section 3.1, 1, on the sentences taken from CNN on all three models: SBERT, ST3, and SRoBERTa. Our goal is to study the difference between the models in qualitative test.
The top five concepts of each input sentence for each model presented in Tables 14, 15 and 16. It seems that all of the models "understood" the texts similarly. Table 16 we can see difference between SBERT and the other models. It seems that SBERT gave more weight to the word bias while the other models gave more weight to the word A/ from the input sentence.

---

C
C
C
C
MASS MEDIA
ORGANIZATIONS ASSOCIATED WITH THE COVID- 19 PANDEMIC
VIRUSES
VIRUSES (3)
C2
PEOPLE
GLOBAL HEALTH
INFECTIOUS DISEASES
DISEASE OUTBREAKS
(3)
HEALTH
HEALTH DISASTERS
DISEASE OUTBREAKS
VIRUS TAXONOMY (4)
CULTURE
REPRODUCTION
VACCINATION
COVID-19 PANDEMIC IN EUROPE (5)
WORLD
EVOLUTION
VIRAL MARKETING
COVID- 19 PANDEMIC IN ASIA (5)
Table 11: An example of the top concepts of the model's output for the input "This is now very contagious virus", taken from CNN.
C²
C
C
C1
LIFE
LIFE SPACE
EXTRATERRESTRIAL
LIFE IN SPACE (2)
LIFE
WORLD
HYPOTHETICAL LIFE FORMS
MESOZOIC LIFE
HYPOTHETICAL
FORMS (2)
SCIENCE AND
ORIGIN OF LIFE
PALEOZOIC LIFE
DISCOVERIES BY
TECHNOLOGY
ASTRONOMER (3)
GEOGRAPHY
COSMOLOGY
EXPLORERS
ARTIFICIAL LIFE (2)
C5
HUMANITIES
FICTIONAL LIFE FORMS
POLAR EXPLORATION
ASTRONOMICAL
CATALOGUES (2)
Table 12: An example of the top concepts of the model's output for the input "The search for life on Mars and ocean worlds in our solar system", taken from CNN.
Ca
R
CONCEPTS
INTELLECTUAL COMPETITIONS
ARTIFICIAL INTELLIGENCE
ARTIFICIAL INTELLIGENCE (3)
POLICY
LEARNING
COLLECTIVE INTELLIGENCE
MACHINE LEARNING (3)
Ca
ETHICS
ISSUES IN ETHICS
COMPUTER ETHICS
COMPUTING AND
SOCIETY (3)
C4
POLITICS
SOCIAL SYSTEMS
MACHINE LEARNING
INTELLECTUAL COMPETITIONS (2)
SCIENCE AND TECHNOLOGY
CONCEPTUAL SYSTEMS
CLASSIFICATION SYSTEMS
INFORMATION SYSTEMS
(3)
Table 13: An example the top concepts of the model's output for the input "The bias these Al systems presents serious issue", taken from CNN.

---

SBERT
STS
SRoBERTa
DISEASE OUTBREAKS (3)
VIRUSES (3)
VIRUSES (3)
DISASTERS (2)
COVID- 19 PANDEMIC IN EUROPE (5)
DISEASE OUTBREAKS
(3)
DOOMSDAY SCENARIOS
COVID-19 PANDEMIC ASIA (5)
VIRUS TAXONOMY
(3)
HAZARDS (3)
DISEASE OUTBREAKS (3)
COVID- 19 PANDEMIC
IN EUROPE (5)
CRIMINAL PROCEDURE (4)
PUBLIC HEALTH EMERGENCY OF INTERNATIONAL CONCERN (3)
COVID-19 PANDEMIC IN ASIA (5)
Table 14: An example of the top concepts of the model's output for the input "This now very contagious virus", taken from CNN. The number in parenthesis is the depth of the concept in the concept graph.
SBERT
STS
SRoBERTa
C1
ATMOSPHERE EARTH (3)
LIFE IN SPACE (2)
LIFE SPACE (2)
OUTER SPACE (3)
DISCOVERIES BY
HYPOTHETICAL LIFE
ASTRONOMER (3)
FORMS (2)
C3
SOLAR SYSTEM IN
HUMAN SPACEFLIGHT (3)
DISCOVERIES BY
FICTION (4)
ASTRONOMER (3)
C4
DISCOVERIES BY
ASTROBIOLOGY (3)
ARTIFICIAL LIFE (2)
ASTRONOMER (3)
C5
ASTRONOMICAL.
ASTRONOMICAL
ASTRONOMICAL CATALOGUES (4)
LOCATIONS FICTION (4)
OBJECTS
Table 15: An example of the top concepts of the model's output for the input "The search for life on Mars and ocean worlds in our solar system", taken from CNN. The number in parenthesis is the depth of the concept in the concept graph.
SBERT
STS
SRoBERTa
CONFLICTS (2)
ARTIFICIAL NEURAL NETWORKS (4)
ARTIFICIAL
INTRI LIGENE (3)
SEXUALITY AND HUNDER-RELATER
MACHINE LEARNING (3)
MACHINE LEARNING (3)
PREJUDICES (3)
Cs
GLOBAL CONFLICTS (3)
ARTIFICIAL
COMPUTING AND
INTELLIGENCE (3)
SOCIETY (3)
C4
POLITICAL CORRUPTION (2)
SOCIAL SYSTEMS (2)
INTELLECTUAL
COMPETITIONS (2)
C5
ANTI-ISLAM
ARTIFICALL LIFE (2)
INFORMATION SYSTEMS
SENTIMENT (4)
(3)
Table 16: An example the top concepts of the model's output for the input "The bias these Al systems presents serious issue", taken from CNN. The number in parenthesis is the depth of the concept in the concept graph.


Reviews:
[{"review_id": "T0h4ObI82F", "reviewer": "Reviewer_iTU3", "summary": "", "questions": "General:\n- Why do we need the conceptual space in the first place? Is it not \"enough\" or maybe even equivalent to having close (i.e. highly similar) concepts in L, which describe the text vector t in L?\n\nAbstract:\n- The first sentence in the abstract is confusing to me: One of the \"main methods\" for interpreting a text is to embed it? Not really, is it? It's one of the main methods to process natural language using computers etc., but for interpreting a text, humans do not map it to vectors -- probably to concepts, though!?\n\nSection 1:\n- The embeddings of small LMs like skip-gram or GloVe were also not interpretable -- did you consider these as well? (referring to ll. 36)\n- ll. 49: by \"understanding\" the embedding space, we might get some insights on those layers, but not into the rest of the model.\n\n\n2.2:\n- l. 181: |C¹| = 37: if I understand correctly, this means that in the first level of the hierarchy, there are only 37 concepts in total? Or am I missing something -- it seems a rather small number for top-level concepts in Wikipedia to me.\n- I am not 100% certain I understand the \"selective refinement\". What does \"the concept with the largest weight\" mean exactly?\n- How is \"removeP\" determined? Is it simply a parameter you set at the beginning?\n\n2.1:\n- Can you please provide a URL to the \"Wikipedia category directed graph\"\n\n\n3.:\n- For the function tau, you use the string representation of the concept, correct? Might this induce some confusion for the embedding model, since there is no context for these concepts?\n\n3.1:\n- The 10 sentences you use as context, are they preceding the sentence under consideration? How are they connected to the sentence except that they are from the same article?\n\n\n3.2:\n- The Kappa coefficient is \"relatively high\" on some of the datasets, but also quite low on others, e.g., 20 news group, Ohsumed. Also, the difference in accuracy on 20 news group is pretty large, especially when compared to the other datasets. Do you have an explanation for that?\n- Maybe I oversaw it -- please mention that all datasets are in English.\n\n3.3.2:\n- Please add to the table captions that the scores represent \"raw agreement\".\n\n4.2:\n- The plots in Figure 2 are really interesting! I just wonder what they actually mean -- the concept of \"weapons\" with input \"government\" in GPT-2 gets stronger with higher layers -- why? What was the context for \"government\"?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the feedback on our work as well as the points on the related work and the typos.\n\nFollowing the questions in the reviews, we performed the experiments of Section 3.2 with a simpler classifier, KNN (K=5, DIstance function is cosine).  The results, which are much better than the RF ones, are shown here.  We will add this table to the paper.\n\n| Dataset | raw agreement| kappa coef| CES acc| LLM acc|\n|---------|--------------|-----------|--------|--------|\n| 20 news | 0.82         | 0.81      | 79     | 87     |\n| ag_news | 0.92         | 0.90      | 88     | 90     |\n| DBpedia | 0.93         | 0.92      | 94     | 94     |\n| Ohsumed | 0.75         | 0.73      | 65     | 72     |\n| Yahoo   | 0.75         | 0.73      | 65     | 69     |\n| R8      | 0.92         | 0.87      | 90     | 94     |\n| BBC     | 0.98         | 0.97      | 97     | 98     |\n\nAbout the questions you have regarding the paper:\n\n- Why do we need the conceptual space in the first place? Is it not \"enough\" or maybe even equivalent to having close (i.e. highly similar) concepts in L, which describe the text vector t in L?\n  - The dimensions in L were created by the training mechanism of the LLM and are not necessarily associated with human-recognizable concepts.  One can view them as a private language of the LLM.  The idea of our work is to map these implicit vectors (rather than individual dimensions) to a space that is recognizable by humans, thus helping us to understand this private language of the machine.\n- One of the \"main methods\" for interpreting a text is to embed it? Not really, is it? It's one of the main methods to process natural language using computers etc., but for interpreting a text, humans do not map it to vectors -- probably to concepts, though!?\n  - We agree and we will modify this sentence to: “One of the main methods for processing natural language by a computer is through embedding…”\n- The embeddings of small LMs like skip-gram or GloVe were also not interpretable -- did you consider these as well? (referring to ll. 36)\n  - Testing our method on older types of  LMs should be interesting and can be potentially added in the future.  In this work, however, we concentrated on transformer-based approaches.  \n- 49: by \"understanding\" the embedding space, we might get some insights on those layers, but not into the rest of the model.\n  - Our main focus is not on understanding the complete model but on understanding the semantics of the input text according to the model.  \n- 181: |C¹| = 37: if I understand correctly, this means that in the first level of the hierarchy, there are only 37 concepts in total? Or am I missing something -- it seems a rather small number for top-level concepts in Wikipedia to me.\n  - These are only the most abstract concepts such as Mathematics, People, Music, Ethics, Philosophy, etc. The current concepts in Wikipedia at that level can be found on this Wikipedia page https://en.wikipedia.org/wiki/Category:Main_topic_articles\n- I am not 100% certain I understand the \"selective refinement\". What does \"the concept with the largest weight\" mean exactly?\n  - To clarify this step we will add the following: “The concept with the largest weight after the projection to CES is then selected for expansion.  The intuition is that this concept represents a main topic of the text, and will therefore benefit the most with more refined representation.”\n- How is \"removeP\" determined? Is it simply a parameter you set at the beginning?  \n  - This is indeed a system parameter.  \n- Can you please provide a URL to the \"Wikipedia category directed graph\"\n  - The Wikipedia category-directed graph is accessible by starting with the root URL: https://en.wikipedia.org/wiki/Category:Main_topic_articles. We included the modified category graph we have extracted in the files we submitted with this submission.\n- For the function tau, you use the string representation of the concept, correct? Might this induce some confusion for the embedding model, since there is no context for these concepts?\n  - The average length of a concept name is 4.25 which is sufficient in most cases.  However, we included an additional type of \\tau, called \\widehat\\tau (section 2.3) that adds children's concept names as a context. An experiment comparing the two can be found in Appendix D. In the future, we consider adding some text from the Wikipedia articles belonging to the concept as context.\n- The 10 sentences you use as context, are they preceding the sentence under consideration? How are they connected to the sentence except that they are from the same article?\n  - We used the first 10 sentences of the article to generate C*. The sentence we tested on was found elsewhere in the text. Since we assume that a news article is focused on a specific topic, we expect the first 10 sentences to provide an appropriate context.\n- The Kappa coefficient is \"relatively high\" on some of the datasets, but also quite low on others, e.g., 20 news group, Ohsumed. Also, the difference in accuracy on 20 news group is pretty large, especially when compared to the other datasets. Do you have an explanation for that?\n  - 20 Newsgroup and Ohsumed have more labels than the others, thus increasing the likelihood of disagreement. Recall that Kappa is between -1 and 1, and for example, Kappa=0.6 is considered (in general) as a good agreement.  We are not sure about the reason for the larger difference in accuracy for the 20-newgroup dataset.  However, in a new experiment included here, using KNN classifier, we observe this phenomenon less strongly.\n- Maybe I oversaw it -- please mention that all datasets are in English.\n  - We will\n- 3.3.2: Please add to the table captions that the scores represent \"raw agreement\"\n  -  We will\n- 4.2:The plots in Figure 2 are really interesting! I just wonder what they actually mean -- the concept of \"weapons\" with input \"government\" in GPT-2 gets stronger with higher layers -- why? What was the context for \"government\"?\n  - We used the word \"Government\" without context as if it was the start of the sentence. We believe that it could be that in the initial layers, the embedding connects to closely related concepts. But in the final layers, it extends to a broader understanding of the context and to words that should appear next to it ( it could be that GPT2 expects that next to “Government” we should see a context related to the concept WEAPON).\n- humans vs. LLM ratings, the test set is very small\n  - Indeed we would have preferred a larger test set.  There were, however, two reasons for this.  The first was limited access to human taggers.  The second is our requirement for a balanced test set - i.e., one with an equal number of correct LLM-based classifications and incorrect ones.  Since two of the 4 datasets had a small number of classification errors, we had to limit the test set size.\n- a citation for DMA?\n  - This is a term that we defined in Section 3.3.2.  It is an acronym for Dimension Meaning Assignment.\n- Koc et al., 2018: Imparting Interpretability to Word Embeddings while Preserving Semantic Structure\n  - will be added in the last paragraph of the related work as it is a work that changes the objective function of GloVe to make it more interpretable, similar to the paper “ Learning interpretable word embeddings via bidirectional alignment of dimensions with semantic concepts” that is cited in this paragraph.\n- Sommerauer & Fokkens, 2018: Firearms and Tigers are Dangerous, Kitchen Knives and Zebras are Not: Testing whether Word Embeddings Can Tell\n  - It is an interesting paper that talks about semantic features that can be abstracted from the embedding using a supervised classifier. We will add it to the related work in the second to last paragraph that discusses probing methods.\n- Molino et al., 2019: Parallax: Visualizing and Understanding the Semantics of Embedding Spaces via Algebraic Formulae\n  - It introduces a tool for doing simple operations such as PCA and t-SNE on any embedding. We will cite it in the related work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The work is well motivated and set into context with related approaches.\n- It is definitely relevant for research in embedding spaces (wrt biases, model decisions, etc.)\n- I like the idea of having different granularity for the concept spaces.\n- It seems that the proposed method works well and it is certainly an interesting take on embedding space interpretation, but I am not 100% certain I understood everything (see questions).", "reasons_to_reject": "- The method needs some clarification -- see questions.\n- The evaluation is interesting, but at least the results using a classifier trained on the concept space are not super convincing, since the agreement scores are not very high. When comparing humans vs. LLM ratings, the test set is very small.", "questions_to_authors": "General:\n- Why do we need the conceptual space in the first place? Is it not \"enough\" or maybe even equivalent to having close (i.e. highly similar) concepts in L, which describe the text vector t in L?\n\nAbstract:\n- The first sentence in the abstract is confusing to me: One of the \"main methods\" for interpreting a text is to embed it? Not really, is it? It's one of the main methods to process natural language using computers etc., but for interpreting a text, humans do not map it to vectors -- probably to concepts, though!?\n\nSection 1:\n- The embeddings of small LMs like skip-gram or GloVe were also not interpretable -- did you consider these as well? (referring to ll. 36)\n- ll. 49: by \"understanding\" the embedding space, we might get some insights on those layers, but not into the rest of the model.\n\n\n2.2:\n- l. 181: |C¹| = 37: if I understand correctly, this means that in the first level of the hierarchy, there are only 37 concepts in total? Or am I missing something -- it seems a rather small number for top-level concepts in Wikipedia to me.\n- I am not 100% certain I understand the \"selective refinement\". What does \"the concept with the largest weight\" mean exactly?\n- How is \"removeP\" determined? Is it simply a parameter you set at the beginning?\n\n2.1:\n- Can you please provide a URL to the \"Wikipedia category directed graph\"\n\n\n3.:\n- For the function tau, you use the string representation of the concept, correct? Might this induce some confusion for the embedding model, since there is no context for these concepts?\n\n3.1:\n- The 10 sentences you use as context, are they preceding the sentence under consideration? How are they connected to the sentence except that they are from the same article?\n\n\n3.2:\n- The Kappa coefficient is \"relatively high\" on some of the datasets, but also quite low on others, e.g., 20 news group, Ohsumed. Also, the difference in accuracy on 20 news group is pretty large, especially when compared to the other datasets. Do you have an explanation for that?\n- Maybe I oversaw it -- please mention that all datasets are in English.\n\n3.3.2:\n- Please add to the table captions that the scores represent \"raw agreement\".\n\n4.2:\n- The plots in Figure 2 are really interesting! I just wonder what they actually mean -- the concept of \"weapons\" with input \"government\" in GPT-2 gets stronger with higher layers -- why? What was the context for \"government\"?", "paper_topic_and_main_contributions": "The authors propose to make embedding space more interpretable with respect to the model that produced them by conceptualizing the embedding space. The concept(ual) space is supposed to be more comprehensible for humans.\nThey motivate their work by the need for debugging embedding models (in this case LLMs) and for detecting biases, as well as for explaining decision made by systems incorporating LMs.\nThey propose an algorithm for conceptualization as well as a new evaluation technique to test the conceptualization.\nAs concept database, they use a Wikipedia graph. For labeling edges between concepts, they calculate sibling scores based on the similarity of the sibling concepts to detect \"is-a\" relations.\nSince concepts often are hierarchical and not all down-stream tasks might need all concepts, the conceptual space can have different granularity, depending on how \"deep\" a concept can branch out (or how many children concepts are available) and what is needed.\nThe authors accomplish this by adapting the concept space iteratively to the input text.\n\nFor evaluating their method, they use both human and LLM judgements.\n\nThe authors' main contribution is a new variant of how to make embedding spaces more interpretable, together with an interesting evaluation.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- a citation for DMA?\n\nThese references are not necessarily missing, but maybe they could give some further ideas since they are based on similar idea(s), i.e., \"conceptualization of embeddings\":\n\n- Koc et al., 2018: Imparting Interpretability to Word Embeddings while Preserving Semantic Structure\n- Sommerauer & Fokkens, 2018: Firearms and Tigers are Dangerous, Kitchen Knives and Zebras are Not: Testing whether Word Embeddings Can Tell\n- Molino et al., 2019: Parallax: Visualizing and Understanding the Semantics of Embedding Spaces via Algebraic Formulae", "Typos_Grammar_Style_and_Presentation_Improvements": "The paper is well structured and written.\n\nTypos:\n\n- l. 76: \"that latent dimensionS correspond ..\" / \"that A latent dimension corresponds ..\"\n\n\nStyle:\n\n- The conceptual space C and the set of concepts C are hard to differentiate, maybe use CS for conceptual space?"}}, {"review_id": "I9abwbfJmA", "reviewer": "Reviewer_EU1c", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the feedback as well as the points on other related work.\n\nRegarding the question you have:\n\n- One limitation of the work is that it only works for document classification.\n  - It is important to understand that the classification experiments described in this paper have one goal: to show that the human-understandable conceptual representation and the original latent representation are strongly correlated.  We do not propose to use the conceptual features for classification - instead, we propose to use the conceptual features for explanation, debugging, and debiasing as shown in Section 4.\n- As the goal of the transformation was to create interpretable representations, using a more transluent classifier (other than a random forest with 100 classifiers) would be more adequate.\n  - Please note that the goal of the classification experiment was to show agreement between classifiers that use the latent representation and ones that use the conceptual representation in order to show that our explicit vectors indeed reflect the semantics of the latent vectors.  Nevertheless, we accepted the recommendation of the reviewer and performed the experiments of Section 3.2 with a simpler classifier, KNN (K=5, DIstance function is cosine).  The results, which are much better than the RF ones, are shown here.  We will add this table to the paper.\n\n    | Dataset | raw agreement| kappa coef| CES acc| LLM acc|\n    |---------|--------------|-----------|--------|--------|\n    | 20 news | 0.82         | 0.81      | 79     | 87     |\n    | ag_news | 0.92         | 0.90      | 88     | 90     |\n    | DBpedia | 0.93         | 0.92      | 94     | 94     |\n    | Ohsumed | 0.75         | 0.73      | 65     | 72     |\n    | Yahoo   | 0.75         | 0.73      | 65     | 69     |\n    | R8      | 0.92         | 0.87      | 90     | 94     |\n    | BBC     | 0.98         | 0.97      | 97     | 98     |\n\n- Besides the agreement rate of the classifiers relying on the original and the interpretable features, their accuracy is equally important, which results are delegated to the appendix.\n  - Please note that in Section 3.2 we give a detailed explanation of why the agreement results are more important.  However, we will move the accuracy results to the main paper.\n- It seems that the use of the transformed representations degrade classification performance noticeably. Some amount of performance drop is fine in exchange for more interpretable representations, but this kind of tradeoff should be made explicit and clearly articulated.\n  - We are not suggesting using the interpretable representation for performing textual tasks (such as classification) but rather using it for explanation and debugging the results of using the LLM.  To show that our explicit representation is strongly correlated with the implicit one, we show that classifiers using the two types of representation are highly correlated. The accuracy when using the two representations is also highly correlated except in one case (20 newsgroups).  However, our new KNN results show closer results.\n- the alternative baseline approaches that assign meaning to each dimension of the original latent space are too naive, i.e., the proposed CES approach has access to C*, the concept inventory tailored for the given task, whereas the alternative approach uses either a vocabulary of the most frequent words or a concept inventory that does not enjoy the benefit of being adjusted to the domain in question.\n  - The goal of DMAwords and DMAconcepts is to represent a class of methods that try to assign a meaning per latent dimension.  It would be indeed interesting to try DMAc*, which is a sort of ablation experiment where the transformation part of our method is turned off.\n- Since the automated assessment is not constrained by human labor, this kind of comparison might have been performed over more documents.\n  - Indeed we would have preferred a larger test set.  There was, however, a reason for this.  We had a requirement for a balance test set - i.e., one with equal numbers of correct LLM-based classifications and incorrect ones.  Since two of the 4 datasets had a small number of classification errors, we had to limit the test set size.\n- the applied baseline is too simple and the comparison with other existing approaches is lacking.\n  - Many of the existing approaches that create new representation attempt to transform the LLM representation to a more interpretable one and focus on showing that the new representation leads to a good or better performance.  The goal of our work is completely different.  Our goal is to convey to a human user the semantics of any given LLM representation (rather than altering it).  In order to compare to other approaches with respect to this goal, we would need to use each of the methods with a black-box LLM, generate some human-understandable semantic representation of each input sentence, test the correlation between this representation and the original latent representation, and test the comprehensibility to humans.  We believe that the work presented here is one of the first to provide such a comprehensive evaluation.  We found it extremely difficult to compare CES to alternatives using such a methodology but we hope that others working towards similar goals will build and expand the methodology introduced here.\n- We will add the first and third papers ([1],[3]) in the related work in the part on sparse transformation (3rd paragraph). Additionally, we will change “One limitation of these methods is their reliance on an embedding matrix, limiting their applicability to static models.” to “One limitation of these methods is their reliance on an embedding/dictionary matrix”.\n- The second paper Yun, Zeyu, et al. \"Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors.\" is very interesting. It uses dictionary learning to see the transformer model as a linear superposition of transformer factors. We will cite it in the related work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Developing interpretable (document) representations is important topic, for which a simple approach is provided.", "reasons_to_reject": "The proposed approach only works for document classification, the applied baseline is too simple and the comparison with other existing approaches is lacking.  \nThe classification performance drops quite noticeably compared to the use of the original (yet uninterpretable) representations.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes an approach for transforming the otherwise opaque dimensions of contextualized sentence embeddings into directly interpretable categories of an ontology and evaluate them in various topic classification tasks.  \nThe proposed approach (dubbed CES) turns some concept inventory into text, transforms them into the representation space of a sentence encoder, finally constructs a transformation matrix of these vectorized concept vectors that maps the latent vectorial representations of the encoder into such a space in which the individual dimensions are directly tied to one of the concepts from the concept inventory.  \nThe idea is simple and sound, even though the concepts seem to be converted into the representation space somewhat naively.  \n\nThe representations with interpretable dimensions are evaluated in 7 different document classification tasks.  \nOne limitation of the work is that it only works for document classification.  \n\nThe quality of the transformed representations is assessed is via measuring the prediction similarity of such random forest classifiers that use either the transformed or the original representations as document features.  \nAs the goal of the transformation was to create interpretable representations, using a more transluent classifier (other than a random forest with 100 classifiers) would be more adequate.  \nCould the prediction similarity be due to the large number of classifiers used in the ensemble of classifiers?  \n\nBesides the agreement rate of the classifiers relying on the original and the interpretable features, their accuracy is equally important, which results are delegated to the appendix.  \nIt seems that the use of the transformed representations degrade classification performance noticeably.  \nSome amount of performance drop is fine in exchange for more interpretable representations, but this kind of tradeoff should be made explicit and clearly articulated.  \n\nThe understandability of the representations from the perspective of document classification is assessed by humans as well as in an automated way.  \nRelated to the automated evaluation of the understandability of the transformed representations, the alternative baseline approaches that assign meaning to each dimension of the original latent space are too naive, i.e., the proposed CES approach has access to C*, the concept inventory tailored for the given task, whereas the alternative approach uses either a vocabulary of the most frequent words or a concept inventory that does not enjoy the benefit of being adjusted to the domain in question.  \nThis way, it remains unknown to what extent CES performs better due to the more useful vocabulary or the actual approach itself.  \nWhen performing the automated undestandability assessment, based on the figures in Table 4, only 20 documents per dataset was used.  \nSince the automated assessment is not constrained by human labor, this kind of comparison might have been performed over more documents.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The statement in the related work section that sparsification only works for static embeddings is not true, see e.g. [1,2,3].  \n[3], being an extension of (Senel et al., 2018) for the contextual case could serve as a stronger baseline as it is also capable of assigning human interpretable labels to the dimensions of transformed latent representations.\n\n[1] Berend, Gábor. [\"Sparsity makes sense: Word sense disambiguation using sparse contextualized word representations.\"](https://aclanthology.org/2020.emnlp-main.683/)  \n[2] Yun, Zeyu, et al. [\"Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors.\"](https://aclanthology.org/2021.deelio-1.1/)  \n[3] Ficsor, Tamás, and Gábor Berend. [\"Changing the Basis of Contextual Representations with Explicit Semantics.\" ](https://aclanthology.org/2021.acl-srw.25/)"}}, {"review_id": "1Sjfzf8mvP", "reviewer": "Reviewer_nPyK", "summary": "", "questions": "Concepts representations are not contextualized. What about ambiguous context? The method proposed in 2.3 might circumvent this issue, but it should be properly tested. \nThis approach is related with topic modeling, in the specific case of fixed topic. How this approach competes against these supervised topic modeling approaches ?\nIf there exist some correlation in the concept space (concepts are not orthogonal enough in term of semantic, e.g. in the case of colinear concept's representations) then a lot of the initial information will be lost (contrary to what is stated in 3.3). The approach seems not to lose that much on classification task, but how the chosen concepts impact the performance of the obtained representation should be carefully evaluated. \n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We first want to thank you for the review.\n\nFollowing the questions in the reviews, we performed the experiments of Section 3.2 with a simpler classifier, KNN (K=5, DIstance function is cosine).  The results, which are much better than the RF ones, are shown here.  We will add this table to the paper.\n\n| Dataset | raw agreement| kappa coef| CES acc| LLM acc|\n|---------|--------------|-----------|--------|--------|\n| 20 news | 0.82         | 0.81      | 79     | 87     |\n| ag_news | 0.92         | 0.90      | 88     | 90     |\n| DBpedia | 0.93         | 0.92      | 94     | 94     |\n| Ohsumed | 0.75         | 0.73      | 65     | 72     |\n| Yahoo   | 0.75         | 0.73      | 65     | 69     |\n| R8      | 0.92         | 0.87      | 90     | 94     |\n| BBC     | 0.98         | 0.97      | 97     | 98     |\n\nRegarding the questions you have:\n\n- The article lacks a comparison with competitors\n  - Many of the existing approaches that create new representation attempt to transform the LLM representation to a more interpretable one and focus on showing that the new representation leads to a good or better performance.  The goal of our work is completely different.  Our goal is to convey to a human user the semantics of any given LLM representation (rather than altering it).  In order to compare to other approaches with respect to this goal, we would need to use each of the methods with a black-box LLM, generate some human-understandable semantic representation of each input sentence, test the correlation between this representation and the original latent representation, and test the comprehensibility to humans.  We believe that the work presented here is one of the first to provide such a comprehensive evaluation.  We found it extremely difficult to compare CES to alternatives using such a methodology but we hope that others working towards similar goals will build and expand the methodology introduced here.\n- The article lacks an analysis of the impact of the chosen concepts (even on the granularity)\n  - In Appendix F we show a qualitative evaluation of different granularity of fixed and flexible depth of conceptual space on the embedding. In those results, we can see the advantage of the flexible-based conceptual space over the fixed one.\n\n- Concepts representations are not contextualized. What about ambiguous context? The method proposed in 2.3 might circumvent this issue, but it should be properly tested.\n  - The average length of a concept name is 4.25 which is sufficient in most cases.  However, we included an additional type of \\tau, called \\widehat\\tau (section 2.3) that adds children's concept names as a context. An experiment comparing the two can be found in Appendix D. In the future, we consider adding some text from the Wikipedia articles belonging to the concept as context.\n- This approach is related with topic modeling, in the specific case of fixed topic. How this approach competes against these supervised topic modeling approaches ?\n  - As we understand, topic modeling uses unsupervised ML to identify clusters or groups of similar words within a text to create a topic. Our work has one step of generating the conceptual space given a context and another step of projecting the embedding space into the conceptual one. Our work can be used for explaining LLM-based topic modeling.\n- If there exist some correlation in the concept space (concepts are not orthogonal enough in term of semantic, e.g. in the case of colinear concept's representations) then a lot of the initial information will be lost (contrary to what is stated in 3.3)\n  - We believe that orthogonality is less important for generating explanation, however,  to make the concept space “more orthogonal”,  we have a parameter removeP, If this parameter is True the algorithm in 2.2 removes the parent node of a node that is added to the concept space, this modification should help with the orthogonality. We ran a test with this parameter in Appendix C.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Simple yet effective method\nThe approach is carefully evaluated, both with human in the loop and automatic/semi-automatic protocols. \nThe method can be adapted to many application domains, as one can tailor the concept space to the dataset at hand. \n", "reasons_to_reject": "The article lacks a comparison with competitors, that are well presented on the related work section. \nThe article lacks an analysis of the impact of the chosen concepts (even on the granularity). \n", "questions_to_authors": "Concepts representations are not contextualized. What about ambiguous context? The method proposed in 2.3 might circumvent this issue, but it should be properly tested. \nThis approach is related with topic modeling, in the specific case of fixed topic. How this approach competes against these supervised topic modeling approaches ?\nIf there exist some correlation in the concept space (concepts are not orthogonal enough in term of semantic, e.g. in the case of colinear concept's representations) then a lot of the initial information will be lost (contrary to what is stated in 3.3). The approach seems not to lose that much on classification task, but how the chosen concepts impact the performance of the obtained representation should be carefully evaluated. \n", "paper_topic_and_main_contributions": "In this article, authors propose to project the representation of a piece of text obtained with a deep language model into a “concept” space (more precisely, computing cosine similarity between the piece of text representation and each concept representation) for improving LLM interpretability. In the experiments, these concepts are built using Wikipedia categories, for which selection depends on the required level of granularity. The method is evaluated on  7 common datasets with both human and automatic evaluations. ", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission presents a method for mapping from embedding space into a space defined by a set of human-interpretable concepts, using textual descriptions of each concept. Evaluation is performed via text classification. The initial reviews flagged some weaknesses in evaluation, which were addressed by further evaluations during the discussion period. In the end, all reviewers rated the paper as "strong" on excitement and either "good" or "strong" on soundness.
Decision: Accept-Main