Latent Topology Induction for Understanding Contextualized Representations
Authors: Yao Fu
Mirella Lapata
Venue: Submitted to ICLR 2023 (2023)

Abstract: Recently, there has been considerable interests in understanding pretrained language models. This work studies the hidden geometry of the representation space of language models from a unique topological perspective. We hypothesize that there exist a network of latent anchor states summarizing the topology (neighbors and connectivity) of the representation space. we infer this latent network in a fully unsupervised way using a structured variational autoencoder. We show that such network exists in pretrained representations, but not in baseline random or positional embeddings. We connect the discovered topological structure to their linguistic interpretations. In this latent network, leave nodes can be grounded to word surface forms, anchor states can be grounded to linguistic categories, and connections between nodes and states can be grounded to phrase constructions and syntactic templates. We further show how such network evolves as the embeddings become more contextualized, with observational and statistical evidence demonstrating how contextualization helps words “receive meaning” from their topological neighbors via the anchor states. We demonstrate these insights with extensive experiments and visualizations.

Under review as a conference paper at ICLR 2023
LATENT TOPOLOGY INDUCTION FOR UNDERSTANDING CONTEXTUALIZED REPRESENTATIONS
Anonymous authors
Paper under double-blind review
ABSTRACT
Recently, there has been considerable interests in understanding pretrained lan- guage models. This work studies the hidden geometry of the representation space of language models from unique topological perspective. We hypothesize that there exist network of latent anchor states summarizing the topology (neighbors and connectivity) of the representation space, we infer this latent network in a fully unsupervised way using structured variational autoencoder. We show that such network exists in pretrained representations, but not baseline random or positional embeddings. We connect the discovered topological structure to their linguistic interpretations. In this latent network, leave nodes can be grounded to word surface forms, anchor states can be grounded linguistic categories, and connections between nodes and states can be grounded to phrase constructions and syntactic templates. We further show how such network evolves as the em- beddings become more contextualized, with observational and statistical evidence demonstrating how contextualization helps words "receive meaning" from their topological neighbors via the anchor states. We demonstrate these insights with extensive experiments and visualizations.
INTRODUCTION
Recently, there has been large interests in analyzing pretrained language models (PLMs) (Rogers al., 2020; (ewiit & Manning 2019 Hewitt & Liang, 2019 Chen et al. 2021 Chi al. 2020 Liu et al. 2019) due to their huge success. This work aims to investigate the topological properties, i.e., neighbors and connections of embeddings, of contextualized representations. Informally, we ask what does the "shape" of the representation manifold "look like", and what do they mean from a linguistic perspective. Formally, we hypothesize that there exists a spectrum of latent anchor embeddings serve as local topological centers within the manifold. As quick first impression, Fig. [ states that we will discover in the following sections. Since such structure cannot be straightforwardly observed,
shows the latent
unsupervised methods to infer the topology latent variables.
Our unique topological perspective, combined with unsupervised latent variable induction technique, offers systematically different methodology than the mainstream probing work. Most existing approaches usually define a supervised linear classifier as the probe (Hewitt & Manning 2019
Hewitt & Liang, 2019 Hewitt al.) 2021 Liu et al. 22019), targeting for pre-defined properties using pre-annotated data. Such a priori approaches make maximal pre-assumptions and consequently, it would be hard make new discoveries other than those already assumed from the very beginning, Our work takes an a posteriori approach, which makes mininal rreaassuptionss without using any annotation for supervision. Consequently, we achieve systematically different (yet complementary) results to the results from supervised probing literature. For example, while arguments make by supervised probing are strictly aspect-specific (e.g., how specific properties like syntax can be extracted out from other properties), we visualize all local topological centers as latent states, ground their meaning to lexical, syntactical, and semantic interpretations, and show how these properties mixed with each other).
discoveries are more holistic and integrated (c.g., in Fig [
We use structured variational autoencoder (VAE) (Diederik P. Kingma 2013) to infer the latent topology, VAEs common and intuitive models for learning latent variables. We focus the manifold where contextualized embeddings lay in (c.g.. the last layer outputs of a fixed, not fine-tuned. BERT BERTDevvin et al. 2019). We hypothesize there exists wide spectrum of static latent states within

---

Under review as a conference paper at ICLR 2023
A. Ancher States within Representation Space
B. States Words
Comessondence
1424
b2o
242
290
426
1074
Cantent States
Function States
Frequent words
Infrequent words
State
Interpretation
Corresponding
Occurrecce
1424
"Gett"
get 482 let 168 got 121 getting 79 geta 38 gotten 20
1746
"Use"
use using used 164 uses 53 usage
426
Past tense
#Wed 46 #eted 21 #Pd 18 built 13 caused 1.1.11 | Istarfed 10 made 10 formed
MAS
964
Present continuous tenee
#Wing 168 #Allig 22 running 22 poeting 17 meving 16 driving 15 eeting 15
Iu)-suffix
+ély 137
actually
exactly
completely 28 usually
1984
[s] suffix Plural
people 620 ##ts 130 those police 30 users 27 folks 28 citizens 25
820
J-suffi negation
don 468 doeen 173 dddh 119 isn 85 won 62 aren
pitt barkeley 24 cleveland chicago 19 printer 18 rachester toranta
1D16
Rellgion
god Jesus bible Israel church christian 64 christ 63 armenian 60| larael 54 turkish 54 jewishh 51 arab 48 christian americann 26
WES
242
Country
1328
Tech
system 214 software 111 car 100 eystems 61 hardware 52 technology 30 driver 28 game 101 team 76 children 67 hockey 55 games 50 baseball 32 players 31
1887
Sports
Figure 1: A: There exist spectrum of latent anchor states spread over the representation space serving as local topological centers. B and lower table: example linguistic interpretations and corresponding word distributions of latent states. Latent states encode rich mixture of lexical, morphological, syntactic and semantic constructions.
this manifold and assume two basic and minimal generative properties of the states: (1). state should summarize the meaning of corresponding words and contexts; (2). transitions between different states should be able to reconstruct sentence structures. We model these two properties as emission and transition potentials of a CRF (Sutton al. 2012) inference network. Since a VAE is generative model trained by reconstructing sentences, essentially we infer states that are informative enough to generate/ reconstruct words and sentences.
In our experiments, first show that the hypothesized topology does exist in embeddings produced by pretrained language models, but not in baseline (random and positional) embeddings. We further ground the discovered topology, i.e., the local state-word emission and the global state-state transition, to their linguistic interpretations. For local state-word emission, we show that states summarize rich types word surface forms ranging from lexicon, morphology, syntax to semantics variations. For global state-state transition, we differentiate two types of states within the space: states encoding function words and states encoding content words. We identify function states that serve as "hubs" in state transitions and attract content words of similar meanings to be close. Consequently, rich set of phrase construction phenomena can be reconstructed from state transitions.
Furthermore, we highlight one important finding about how contextualization directly changes the underlying topological structure. We provide statistical and observational evidence. Statistically, we show that before contextualization. most function words are concentrated around a few head states and are isolated from content words; after contextualization, these function words spread over full state distribution and are better mixed with content words. Observationally. structures change with contextualization. For example, before contextualization, the neighbors of suffix #ed, fing just random tokens; after contextualization, the neighbors of #ed become past tensed words like had, did and used, and the neighbors of #ing become present continuous tensed words like running, processing and writing.
word neighbor
2

---

Under review as a conference paper at ICLR 2023
Putting everything together, our results provide holistic and analytical view of the representation space topology. In our discovered latent network, leave nodes can be grounded to word surface forms, anchor states can be grounded to linguistic categories, and connections between nodes and states can be grounded to phrase constructions and syntactic templates. We hope our work can inspire new insights and deeper understandings of pretrained language models.
2 RELATED WORK
Geometric Properties of Language Models We are interested in the geometrics properties of contextualized embeddings (e.g.. last layer output of BERT), which roughly asks what is the "shape" of the space and what they represent. Generally, accurate and comprehensive geometric description of the manifold is challenging, due to its high-dimentional and non-linear nature. Previous work focuses on different geometrie aspects, such as distance metrics (Wang & Ponce 2021 Reif et al.) 2 linear subspaces (Hewitt & Manning, 2019 Cai et al., (021)) and hyperbolic geometry (Chen et al. 202T). We focus on the topological structure, which studies how embeddings and states are connected to each other thus forming meaningful latent networks.
Relation to Supervised Prohing Collectively known as "Bertology" (Hewitt Manning, 2019 Rogers et al. 2020), the goal of probing is to discover what meaningful information are encoded within large language models. The mainstream approach is mostly a priori, which starts from a target linguistic property like part-of-speech (Liu et al. 2019; Hewitt et al. 2021). dependency edge (Tenney
ct al. 2019b ab, syntax trees (Hewitt & Manning 2019) Hewilt & Liang 2019, Reif et al. 2019
Chi al. 2020), sentiments (Chen et al., 2021 Wu et al. 2020), then fine-tune supervised weak classifier (a.k.a. a probe). The higher the classification score, the more information contained. Our unsupervised approach takes systematically different (and complementary) methodology from three perspectives: (I). our method is posteriori (v.s. existing work is a priorí), and we hold minimal assumptions as we do not require supervision data, v.s., existing work holds maximal pre- assumption by targeting some property at the very beginning; (2). from an information theoretical perspective, existing work studies how to extract information from a channel (the probe) with limited capacity (Pimentel et al., 2020 Voita & Titov) (020), while we directly study the original representation, without extracting information through a channel; (3). consequently, we show how different information is mixed with each other in the raw representation space, while existing work studies how one type of information can be extracted from the rest.
Other Unsupervised Methods tic (Kim et al., 2020). geometric (Cai et al. 2021), cluster-based (Dalvi et al. 2022), and other information from PLMs (Wu al., 2020, Michael al., 2020). Our focus here is the network topology within the representation space, which is not yet thoroughly studied. Amongst the large volume of Bertology research, the closest unsupervised work to ours are: Dalvi et al. (2022) who use clustering to discover latent concepts within BERT, and we will later use their results as a comparison our discoveries: Michael al. (2020) who discovers latent ontology in an unsupervised way: Cai et al. (202T) who study the geometric properties of BERT with a focus on isotropy. There is also supervised method for extracting static embeddings from contextualized embeddings (Gupta & Jaggi 2021p. These work more or less involve cluster structures within BERT. Our major difference is that we take an important further step from word clusters to state-state transitions and show how traversal over states leads to sentence constructions. In the latent variable literature, our inference model uses a classical CRE-VAE formulation (Fu et al., 22200 Mensch & Blondel, 2018). Existing work uses this formation for other tasks like structured prediction (Ammar et al. 2014) or sentence generation (Li & Rush 2020) while we discover latent structures within PLMs.
There are also unsupervised works proposes to extract syntac-
METHOD
Latent States within Representation Space Given a sentence x = [x,,..., 2T), we denote its contextualized representations as [r,, rT] PLM(x) where PLM(-) denotes pretrained encoder (here we use BERT and our method is applicable to any PLM). Representations r for all sentences lie in one manifold M, namely the representation space of the language model. We hypothesize there exists a set of N static latent states 8), & that function as anchors and outline the space topology (recall Fig. I]. We emphasize that all parameters of the PL. are fixed (i.e., fine-tuning), so all
3

---

Under review as a conference paper at ICLR 2023
learned states are intrinsically within M. We focus on two topological relations: (I). state-word relations, which represent how word embeddings may be summarized by their states and how states can be explained by their corresponding words; (2). state-state relations, which capture how states interact with each other and how their transitions denote meaningful word combinations. Taken together, these two relations form latent network within M (Fig. [and later Fig. 3
Modeling For state-word relations, we associate each word embedding T2 with a latent state indexed by 2t E Nl, We an emission potential o(I,, =x) model how 2, likely to summarize Tr. The corresponding embedding of 2 is then 8 For state-state relations, we assume a transition matrix ((2-11 2t) modeling the affinity about how state 2t-1 are likely transit state 25. Together p(2, 21) and (22-11 2x) form the potentials of a linear-chain CRF:
log ( 2t) r! 8z,
log + (2t-1, 2x) _
where the vector dot product follows the common practice of fine-tuning contextualized representa- tions. The probability of state sequence given a sentence is:
( m)) = II o (2)--1, 2
where Z is the partition function. Note that only embeddings of states 1 [s]:.... 8N] ] are learnable
parameters of the inference model Go. To infer we use common CRE-VAE architecture shared by
previous work (Ammar et al. 2014 Li & Rush 2020 top of the encoder:
Fu et al. 2022) and add a generative model on
po( 2) = II p(rrlana)) (1=-1)
h, = Dec(8 hz-1)
(3)
p(21/z1:t-1 softmax(FF((Is.) hel))
p(ae(21:)) softmax (FF(h)))
where 0 denotes the decoder parameters, Dec(-) denotes the decoder (we use an LSTM), h, denotes decoder states, and FF(.) denotes a ford-forward network. We optimize the 3-ELBO objective:
LELO = Egg (=|=> (=|=z) [log (x, 2)] (H((2(2(x))
(5)
We further note that the decoder's 's goal for help inducing the latent states, rather than being powerful sentence generator. After training, we simply drop the decoder and only look at the inferred states. Maximizing p(xe(= encourages 21.t. (thus their embeddings 82) to reconstruct the sentence and (1--1) encourages previous 21:11 to be predictive to the next (so we can learn transitions). Essentially, this formulation trys to find "generative" states s within the representation space M that are able to predict sentences and their next states.
4 EXPERIMENTAL SETTING
Dataset, Model Parameters, and Learning We perform experiments on the 20News dataset (Kus ner et al. 2015). common dataset for latent variable modeling (initially for topic modeling) as testbed. We primarily focus the last layer output of BERT wwt---baseuucceee (BERTLAST), yet our method is applicable to any larger size, GPTT-styled, or mncoder-decoderrsttlleee models. In terms of model parameters, the dimension of the states is the same as contextualized embeddings, which is 768. We use a light parameterization of the decoder and set its hidden state dimension to 200. Again, the purpose of the decoder is to help induce the states, rather than being powerful sentence generator. We set the number of latent states N to 2000. Recall that the vocabulary size of uncased BERT is 30522, which means that if uniform each state approximately corresponds to 15 words, serving as a type of "mmta" word. We further note that setting N = 2000 is somehow a sweet point according to our initial experiments: larger N (say 10K) is too fine-grained and under-clusters (words of similar linguistic roles are divided into different states) while smaller N (say 100) over-clusters (words of different roles are gathered into the same cluster). Gradient-based learning of CRFs inference model is challenging due to the intermediate discrete structures. So we use approximate gradient and entropy from Fu et al. (2022) which enables memory- efficieen differentiable training of our model. During training, we tune the 3 parameter in Eq. 5 to prevent posterior collapse, which is a standard training technique for VAEs. All experiments are performed on Nvidia 2080Ti GPUs.
model
4

---

be inferred Under review as a conference paper at ICLR 2023
Table Grounding inferred states existing linguistic categories. POS and ENT
from the word directly while DEP, CCG and BCN require more contxxt. Meaningful states emerge in BERTZERO and BERTLAST, but not in baseline POSITIONAL and RANDEMB.
POS
ENT
DEP
CCG
BCN
POSITIONAL
RANDEMB
392
21
366
241
334
BERTZERO
673
37
450
275
BERTLAST
804
51
740
583
628
Table 2: Human evaluation (averaged over annotators) of states not grounded to existing annotations. Although some states cannot be aligned with existing linguistic annotations, most of them are still meaningful to the annotators and only a small portion them are truly uninterpretable.
LEX
SYN
SEM
Not Interpretable
BERTZERO
83
54
135
39
BERTLAST
40
56
Baseline Embeddings We compare against: (1). POSITIONAL states induced from positional embeddings. As there is no content information, we expect the induced structures to be very poor. (2). RANDEMB, fixed random state embeddings sampled from Gaussian distribution sharing the same mean and variance with BERTLAST embeddings. We also expect no meaningful topology from this embedding. (3). BERTZERO, static word embeddings from the zeroth layer of BERT. We expect linguistically meaningful topology emerge within BERTZERO and BERTLAST. Furthermore, we are particularly interested in the comparison between BERTZERO and BERTLAST, the differences can shed light what happens after contextualization.
DISCOVERIES
The first part of our experiments studies on local topology of the representation space, focusing on how states serve topological centers of their corresponding words. first show that states from language models can either be grounded to existing annotations like POS tags, or can be recognized by human annotators as lexical, syntactic, semantic category, while states from baseline embeddings cannot be grounded. Then show how contextualization shapes the space topology by comparing BERTZERO and BERTLAST, highlighting how state-word connections change with contextualizaiton.
The second part of our experiments focus on global topology of the representation space, highlighting how transitions between states form the backbone the overall space structure. We identify important function states that serve as hubs of the latent network, attracting content words of similar meaning to be topologically close to each other. We also show how contextualization shapes the global transitions. Finally, we put everything together and give holistic statement of the space structure and their linguistic interpretations.
5.1 LOCAL STATE-WORD TOPOLOGY
To see whether the inferred states are mean- ingful, we ground them to common linguistic categories. Specifically, we consider: (I). POS, part of speech tags; (2). ENT, named entities; (3). DEP, labels of dependency edges linking word to its dependency head; (4). CCG, CCG supertags which contain rich syntax tree information and are usually referred "almost parsing" (Liu et al., 2019); (5). BCN, BERT Concept Net from Dalvi et al.) (2022), a hierarchy of concepts induced from BERT, mostly about semantics and similar to a topic model. We obtain the POS, ENT, DEP annotations our 20News dataset using pretrained parser
Emergence of Linguistically Meaningful States

---

Under review as a conference paper at ICLR 2023
No: Altered Nlgned
Function wordss Content words
Function words Content words
States Inferred BERT Zero
States Inferred from BERT Last
Figure 2: Contextualization changes space topology. Each bar represents latent state with height equals to log frequency. Orange/ blue represents the portion of function/ content word corresponding state. In BERTZERO, most function words are concentrated around head states. After contextual- ization (BERTLAST), function words mix with content words and spread over all states.
with the spacy library. We obtain the CCG and BCN annotations from their own websites. After training, we use Viterbi decoding to decode states, and each state may correspond to multiple copies of the same word from different contexts. We say latent state aligns with a predefined tag 90% of the word occurrence corresponds to this state also corresponds to tag. For example, suppose state-0 occurs 100 times the full validation set, 90 times of which correspond to either "happy" or "sad" and 10 times correspond to other random words. In this case, "happy" and "sad" takes the dominant portion of state-O (90 out of 100), and both are adjectives, so we say state-O aligns with the POS tag adjective. We set the threshold to 90% because it is intuitively high enough. We select the model that has the largest number of aligned tags to the union of the five types of annotations during validation.
Table I shows the grounding results. We see there are significantly less number of states in Po- SITIONAL and RANDEMB that can be grounded to existing categories, and states inferred from BERTZERO and BERTLAST can be better grounded (e.g, 804 states in BERTLAST can be grounded to POS tags, while this number in RANDEMB is only 392). These results mean that meaningful states emerges in embeddings from language models, but not in baseline embeddings. Note that although there are still certain numbers of states from RANDEMB that can be grounded, it can be explained by the fact that the training process still tries to place states that are useful for sentence resonstruction, even these states are placed between random embeddings. We also notice that states from BERTLAST aligns better with categories requiring more context (DEP, CCG and BCN), this also expected as BERTLAST is contextualized while BERTZERO is not.
For states that cannot be grounded to existing categories, we show that they are still linguistically meaningful by human annotation. Specifically, we ask human annotators (three graduate students with 100+ TOEEL test scores) to annotate the corresponding words of state are: (I), LEX: words that are textually similar. (2). SYN: words share similar morphological-syntee
rules. (3). SEM:
words with related meaning. (4). N.I.: Not Interpretable. The results are show in Table 2 Generally, most of the states still contain meaningful linguistic information from the annotators perspective and only a small portion of them are truly uninterpretable.
Contextualization Shapes State Topology space topology by taking a closer look on what is encoded in BERTLAST and changes in BERTZERO. To this end, we differentiate two types of words: (1). function words (e.g.. preposition, conjunction, determiner, punctuation .etc) whose main role is to help sentence construction but do not have concrete meanings on their own; (2). content words (e.g., nouns, adjectives, verbs, adverbs) who have concrete meaning. It turns out that contextualization results in very different behavior about the encoding of these two types.
Now we show how contextualization shapes the
Figure 2 shows how function/ content words are encoded before/ after contextualization. We see two effects of contextualization: (1) before contextualization, most function words are concentrated around few head states; after contextualization, these function words spread over the full distribution, not just head states. This shows that the meaning of function words is distributed from head states to all states according to their context. (2). before contextualization, most states are either function-only or content-only (as most bars are either orange-only or blue-only); after contextualization, most states contain both function and content states (as most bars have both blue and orange portions). This shows that the meaning of function words is entangled together with their neighbor content words. Intuitively, contextualization helps function words "receive" meaning from their context.

---

Under review as a conference paper at ICLR 2023
Table 3: Contextualization changes local neighhor structures. Try comparing red tokens and see their neighbor words before/ after contextualization Tokens previously opaque (as their corresponding latent state are not meaningful) gain linguistic clarity (as their corresponding states encode meaningful linguistic constructions).
Before Contextualization (BERTZERO)
After Contextualization (BERTLAST)
Symbol
S2851 sizes typea numberrsg numberrs
S2i8
money76
costs1
paysı love worthat
@ 13522
samegro ordinaryys average;
@1184
comm14
orgasz
addressg |listss
Prefix
resas
presse I miS4si co2ss piz:
oldss7
a
recently42 yearss pres6
un1922 persn1 I diaes multigg #cooss
un1524 I iisse imm75 mis162 coniss lum
Suffix
#ingisos #tingios #leyse Hlightss
#ingi563
running226 processingıs writingss
#lymm I dearso | thyss #morejs #rouse
#ly9e3 actuallys4s I exactlyses simply222
#egans #edaes #verso #izers figis
#d1o12 hadsap I #edais didsm I used
#ssus flloz I #los ftpss #mgg #h62
#51839 filesgs bookS machines
#saos S333 #t13 #pi20 #u iiss
peopleusi #isee2 thosee61 users21o folks190
Lexicon
decided2so I decisionay decide determine102
decision22o I positionzos chooseiss command
determinedgy decisions
actions decideşı
be7125 been) beingas7 gone
ber192 are am I is becomes2
We now revisit Fig. T that we briefly mentioned in $ Figure is produced by t-SNE (Van der Maaten & Hinton 2008) jointly over the states and embeddings from BERTLAST and illustrates the local topology (because t-SNE preserves more local information) of the representation space. Blue/ white circles in Fig. correspond to blue/ orange bars in Fig. 2 directly shows how states spread over and "receive" meaning from their neighbor word embeddings and encode to different types of word clusters.
We further highlight certain topological changes before/ after contextualization in Table Before contextualizatton, we see (1). the symbol $ does not have meaningful neighbors; (2). the suffix #ing and #ed are just ordinary subwords; (3). the word be's neighbor is its morphological variants. After contextualization, we see (1). the symbol $ encodes money; (2). the suffix fing and #ed encode tense; (3). be's neighbor becomes linking verbs. Contextualization makes these tokens "receive" meaning from their contexts.
5.2 GLOBAL STATE TRANSITION TOPOLOGY
Now we move our focus from local topology (how states form local centers of their neighbor words) to global topology (how all states are inter-connected to each other). We first visualize the transition network to see how backbone of the space topology follows a clear long-tail structure. We show state transitions encode meaningful phrase construction and contextualization changes the transition topology. Finally, we put everything together and give a holistic statement of the space topology.
Overall State Transition Topology We visualize the induced state-state network in Fig. 3 using t-SNE again (this time without word embeddings). Blue circles represent states with more content words while white circles represent states with more function words. Circle size represents state frequency. To see how states transit to each other, we compute the state transition statistics from the state sequences decoded from the validation dataset. The transition histogram also shown in Fig. 4 We use blue edges to denote frequent (stronger) transitions and yellow edges to denote less frequent (weaker) transitions. Table showws example transitions and their corresponding word bigram occurrences. In Fig. 3A we see: (1). both nodes and edges follow a long-tai -til distribution: there are few frequent nodes/ edges taking the head portion of the distribution, and many infrequent nodes/ edges taking the tail portion of the distribution. Note that the yellow background in Fig. A consists of many weak edges. (2). frequent states are more inter-connected and tail states are more spread. Fig. 3 zooms in the top states, and we see function states usually the hub of the edges. This is also evidenced in Table44 as we can see many different content words transit to the function word to (e.g.. free-to, willing-to, go-to, went-to), and the function word "to" can transit to other content words (c.g., to-buy, to-sell, buildd. Here the state encoding to is a hub connecting other states and words.
7

---

Under review as a conference paper at ICLR 2023
A. All 20K states
B. Top 500 states
355
States more content words & more frequent
States content words & less frequent
303
States more function words more frequent
States more function
frequent
Stronger transition
8561
C. Example Transition
Weaker transition
Figure 3: A: State transitions, as the backbone the representation space, show a clear long-tail structure. Tail states more spread and transitions tail states mostly from head states (yellow edges). Head states are more enter-connected (blue edges). Blue edges denote more frequent (stronger) transitions and yellow edges denote less frequent (weaker) transitions. B: transitions between top states, function states usually (white circles) serve as hubs of transitions (many white circles at the center). C. Example transitions, Table below for their interpretations.
Table 4:
Example state transitions. Function words (like "to") serve as hubs that attract content words of similar meanings be within the same latent state. Subscript numbers denote occurrence.
Transition
Example states from BERTLAST
Explanation
Function word + content word
886-989
to-buysi to-sella to-builds to-purchases to-create4 co-produces free-to:4 willing-to, hard-to; easy-to. happy-to: safe-to;
to do sth. adjective to movement to move direction passive voice passive voice
1671-1441
785-7
go-tom9 going-tow went-tos trip-to, moved-to; come-to:
785-565
come-out, come-ups went-upa go-down; went-outs I bent-backi caused-by, #ed-bys #ttd-bys produced-by: I made-bya driven-by,
426-198
303-426
is-made: bbe-converted) been-formedy are-formeds been-developedi
Content word content word
1537-1537
ms- ms-windowsjj source-code; windows-programm sverating-systente
computers time viing as noun
355- 1201
three-years; five-years; 24-hours; ten-years; 221 21-days;
593-964
mnage-processing: meter-reading; missile-spotting; speed- scanning:
Figure shows transition distribution. The bars here correspond edges in Fig. Color denote the
portion of function/ content words (node color in Fig. 3) and height correspond to edge color in Fig. 3 We observe: (1). transitions usually mixtures of function and content states. (2). contextualization makes function words less concentrated around head transitions (in Fig. left, BERTLAST has less orange portion than BERTZERO), and more spread within the distribution (in Fig. right, BERTLAST
has longer tail of orange bars than BERTZERO).
5.3
A HOLISTIC STATEMENT OF LATENT SPACE TOPOLOGY
Now we can finally put everything together and reach a holistic and bottom-up description about how words, phrases, and sentences constructed within the space topology. This mechanism consists of
8

---

Under review as a conference paper at ICLR 2023
Recttin
Bertl sat
Fantloo
BeriZar
Contant
Content
Top
State bigrams top frequency
Figure State transition distribution. Again, function words exist more at top transitions in BERTZERO. After contextualization (BERTLAST), they become more spread within the distribution.
1. Function states
2. Content states
3. Phrase construction
go
buy
to
the
went
to buy
car
Sentence Construction
O
trip
purchase produce
move
Same
should
anything 128
be
worse
1984
999
be
prosecuted
How
can
1529
1209
than
that
503 been
1355
457
= 174
717
1874
fixed
can
1999
even
1011
1719
This
has
we
define
term
4A.
Some should be prosecuted
4B.
can anything be worse How can we even define that term
This
fixed
Even
him
The 681 -
line
broader 1969
1473
670
would
be
pleased
320
than
you
think
1122 I 1529
1380
1154
717 = 1405 1209
= 295
1440
appreciated
868 It
than
you
think
Any
info
greatly
actually worse
Even him would pleased
The broader you think It actually worse than you think
Any info will greatly appreciated
Figure 5: Four steps illustration of how words, phrases, and sentences are constructed as nodes, connections between nodes, and traversals over the latent topology. Numbers mean latent state index. Structurally similar sentences share overlapped paths of latent states.
four steps, and is illustrated in Fig Step 1: there exist function states that correspond to specific function words (as evidenced in Fig. Step 2: there exist content states that correspond to content words with similar lexical/ syntactic/ semantic meaning (as is evidenced in Table | and Fig. 1 Step transitions between function and content states correspond meaningful phrase constructions (as evidenced in Fig. Big.Bandd and Table 4). Step 4: traversal of states encodes sentence within the space (this is a corollary combining step 1-3). Fig. (.s.55sshn Jsshws sentences sharing overlapped traversal. State transition chains encode the sentence templates and state-word emissions fill in the content. When the transition chains of two sentences overlap, the two sentences tend to be syntactically similar. Overall, in this latent network, leave nodes correspond to specific words, anchor states correspond to linguistic categories, and connections between nodes and states correspond to phrase constructions and syntactic templates.
CONCLUSIONS
In this work, we study the topological structure the representation space contextualized rep- resentations. Our analytics starts from the hypothesis that there exists latent network of states that summarize the representation space topology. We verify such states exist in embeddings from language models, but not in basedline embeddings. We further study how state transitions mark the backbone of the representation space and encode meaningful phrase constructions. Contextualization shapes both local and global topology of the representation space. We hope this work deepens the understanding of language models and inspires new modeling techniques based on the topology of the representations.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Waleed Ammar, Chris Dyer, and Noah A. Smith. Conditional random field autoencoders for unsupervised structured prediction. Proceedings of the 27th International Conference on Neural Information Processing Systems Volume 2, NIPS 14, pp. 3311-3319, Cambridge, MA, USA, 2014. MIT Press.
Xingyu Cai, Jiaji Huang, Yuchen Bian, and Kenneth Church. Isotropy in the contextual embedding space: Clusters and manifolds. In International Conference Learning Representations, 2021.
URL htt.hhttps:://peneeeee
net forum? for 6OWDA
Boli Chen, Yao Fu, Guangwei Xu, Pengjun Xie, Chuanqi Tan, Mosha Chen, and Liping Jing. Probing (bert) in hyperbolic spaces. In International Conference Learning Representations, 2021. URL
htttps://ppenreve/
net. forum? 11=11d111 17VWWXY2yh
Ethan A. Chi, John Hewitt, and Christopher D. Manning. Finding universal grammatical rela- tions in multilingual BERT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5564-5577, Online, July 2020. Association for Computational Linguistics. doi: 10. 1/252// 1120.acl-main 493.
URL https: //ac htttpss//acllattioooo...oom org/
Fahim Dalvi, Abdul Rafae Khan, Firoj Alam, Nadir Durrani, Jia Xu, and Hassan Sajjad. Discovering latent concepts learned in BERT. In International Conference on Learning Representations, 2022.
URL
net forumm POTMOOT
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 00.18653/w1/119-44444 URL https: //aclantho 1 wwa.y.orrgNN 1423
Max Welling Diederik P. Kingma. Auto-encoding variational bayes. In International Confer- ence on Learning Representations, 2013. URL htttps://operrevileee net.siewwwweee..eee...... htt.://wwwwweeeoommmm forum?id
33x91d2-9Fy22
Yao Fu, Chuanqi Tan, Bin Bi, Mosha Chen, Yansong Feng, and Alexander M. Rush. Latent template induction with gumbel-crf. In NeurlPS, 2020.
Yao Fu, John P. Cunningham, and Mirella Lapata. Scaling structured inference with randomization. In International Conference on Machine Learning, 2022.
Prakhar Gupta and Martin Jaggi. Obtaining better static word embeddings using contextual embedding models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguis- tics and the I 1thh International Joint Conference on Natural Language Processing (Volume Long Papers), pp. 5241-5253, Online, August 2021. Association for Computational Linguistics. doi: 10. N0.18653/v1/2022 hat.:///00..aaloooaaaa lol-longg.40. URL
org /2tgg/2221...2 aww.::21.1cll.. a0l
408
John Hewitt and Percy Liang. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLA) pp. 2733- 2743, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19 1275. htRl:tttpps:/alaantooll. wtwp://9gg.or//11111177000
John Hewitt and Christopher D. Manning, A structural probe for finding syntax in word representa- tions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4129 4138, Minncapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/1//19 URL URI h ogy. ogw.org/........ 9-19
10

---

Under review as a conference paper at ICLR 2023
John Hewitt, Kawin Ethayarajh, Percy Liang, and Christopher Manning. Conditional probing: mea- suring usable information beyond a baseline. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pp. 1626-1166600 1639, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/200 emolp-main.13 122. URL URL http::/ttt/cccc.t http:////ccaaaatooogooooooooooo /tp/g//222/2 emn p-ma 122
Taeuk Kim, Jihun Choi, Daniel Edmiston, and Sang goo Lee. Are pre-trained language models aware of phrases? simple but strong baselines for grammar induction. In International Confer- ence on Learning Representations, 2020. URL https:://ppenreiieett net.e/ewweeeeeeeeeeeeeeeeeeee H H1xPR3NCPB
net
Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. From word embeddings to document distances. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 957-966, Lille, France, 07-09 Jul 2015, PMLR. URL httpps::/procceeeiiinnt ml www.s/ppees/v377 kusnerb15. html
Xiang Lisa Li and Alexander Rush. Posterior control of blackbox generation. In Proceedings of the 58th Annual Meeting the Association for Computational Linguistics, pp. 2731-2743, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v12200..cc mli.mmm22242.3 URL
https: //acl//cccccccc antho 1 wog..orgg 2020
in. 243
Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volune (Long and Short Papers), pp. 1073-1094, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/1//19-1111 URL https
//aclanthology.oo www.ogg.oorg/g...
Arthur Mensch and Mathieu Blondel. Differentiable dynamic programming for structured prediction and attention. In International Conference on Machine Learning. pp. 3462-3471. PMLR, 2018.
Julian Michael, Jan A. Botha, and Ian Tenney. Asking without telling: Exploring latent on- tologies contextual representations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6792-6812. 121,. Online, November 2020. Association for Computational Linguisties. doi: 10. 100.18653/112000 wttp:///......mmmai.....
URL
wwt.oorgg....... emw.:mmmmmlpmmmmmmmmmm map 552
Tagoo Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. intormation-theorett probing for linguistic structure. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4609-4622, Online, July 2020. Association for Computational Linguistics. doi: 10.18653//120200.cc- main.mammmmaa...... URL
main. 42C
Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. Visualizing and measuring the geometry of bert. In H. Wallach, H. Larochelle, A. Beygelz- imer, F. d'Alché-Buc, Fox, and Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. U
neurips
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguisties,,, 8:842-866, 2020. doi: 10. 10.1162/tac_a_0034 URL / http://eeecaaaaoooooooor////////
Charles Sutton, Andrew McCallum, et al. An introduction to conditional random fields. Foundations and Trends@ in Machine Learning, 4(4):267-31) 2012.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4609, Florence, Italy, July 2019a. Association for Computational Linguistics. doi: 10. 18618/53/ 1//119-152. htRpshtss:: a aaaaccanttttn www.chhoool............ 9-142

---

Under review as a conference paper at ICLR 2023
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. What do you learn from context? probing for sentence structure in contextualized word representations. In International Conference on Learning Representations, 2019b. URL htttps://openreviees
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Elena Voita and Ivan Titov. Information-theoreiicc probing with minimum description length. In Pro- ceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 183-196, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/ v1/2020.cmmlp-maa watpmnlllmmii.ieemm main. 14. URL http::/ss//.aaaahhoooo.oooo org/ 2020. emnlp- metp-mmain... 14
Binxu Wang and Carlos R Ponce. The geometry of deep generative image models and its applications. arXiv preprini arXiv:2101.006 0.00066, 2021.
Zhiyong Wu, Yun Chen, Ben Kao, and Qun Liu. Perturbed masking: Parameter- free probing for analyzing and interpreting BERT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4166-4176, Online, July 2020. Association for Computational Linguistics. doi: 18.18653/v www.0220...llai.... 2020.acl-miin.
URL https: // acc Lanthologyy org/
A
APPENDIX
You may include other additional sections here.
12


Reviews:
[{"review_id": "wHtyKLiwSy", "reviewer": "Reviewer_QWYM", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors study the topology of representations in language models including how contextualization affects neighbors and connectivity, relationships with linguistic interpretations, and evolution of embeddings through model layers. First the authors consider state-word emissions and global state-state transitions and show that states can summarize linguistic information such as lexicon, morphology, syntax and semantics. Next the authors highlight the effect of contextualization on the topological structure of representations.  In particular, they find that word neighbor structure changes with contextualization.", "summary_of_the_review": "The authors present interesting ideas for understanding latent topology of the contextual representations. This is an important and challenging problem but the method and empirical evaluations are limited, hard to decipher, and not based on large scale evaluations. Many anecdotes are provided in the paper but it is difficult to objectively make the conclusions the authors arrive at in this work. ", "strength_and_weaknesses": "Strengths:\n\n- The authors provide nice illustrative figures for their methods\n\nWeaknesses:\n\n- To me it is unclear what types of claims can be made rigorously in this work and it seems like the authors have presented some interesting anecdotes but the evaluation methodology is not made explicit. Many of the claims including those relating to traversal uncovering phrases and sentences along with the study on states encoding linguistic information seem unfounded based on the limited experimentation in the paper. \n- When reading the paper, I am unable to reconcile the bold claims made in the introduction with the presented empirical studies.\n- Small scale human studies are shown with three individuals with similar background (e.g. graduate students) performing annotators. This seems like an especially small sample size with a lot of bias and the authors should recognize that a study run in this manner should not be used to bolster claims.\n- It is unclear how selection of states to provide annotation for was proposed and the experiments, in my opinion, are not very conclusive due to the experimentation setting. \n- The authors claim that N=2000 is a sweet point but provide no clear evidence for this other than some further anecdotes.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "la0cvzPX-4", "reviewer": "Reviewer_mZMM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Summary\n\nRepresentations learned within deep transformer models learn rich linguistic knowledge. Most of the prior work uses a probing framework which is limited to pre-defined aspects of language. Recent methods are using unsupervised methods to study ontology in pLMs. This paper aims at discovering latent states that can provide a generative explanation of the representations. The authors train a sequence-level CRF on top of contextualized vectors, which are then stacked with an LSTM and trained in an autoencoder fashion. The authors then align stats of the CRF with linguistic annotations.\n\nThe authors found that: i) the states don’t necessarily align with linguistic concepts, ii) contextualization causes function words to acquire meaning from the content words, iii) lower layers encode shallow lexical concepts while the higher layers account for contextual information. Finally the authors showed that the states of CRF can be traversed to discover meaningful phrasal constructions. \n\n\n", "summary_of_the_review": "I believe this is a good paper under progress. The method is novel but not executed well. More exploration across different dimensions such as including more pLMs possibly including LSTMs and static models, exploring the whole network and not just 2 layers, including probing as baseline would make the paper stronger. This will hopefully bring more interesting findings and also put the authors on a better footings for the claims they are making.  ", "strength_and_weaknesses": "Strengths:\n\nUnsupervised methods are an exciting alternative to understanding pLMs as shown by some recent work (Micheal et al 2020, Dalvi et al 2022). The authors have presented a novel method along this line of work. \n\nI liked the visualizations in the paper.\n\nThe idea to align CRF states (as a proxy to pLMs) with the linguistic concepts is interesting.\n\nWeaknesses:\n\nI would have liked the authors to study models other than BERT to strengthen their analysis and show if their conclusions are general across different models. They also only explored the initial and final layers of BERT. Prior analysis has shown that much of the linguistic knowledge is present in the middle layers. Initial layers are mostly reconstruction word structure from subword segmentation and final layers are mostly geared towards the task at hand. \n\nWhile the method is exciting, there are not enough crisp findings in the paper. Results such as lower layers capture lexical concepts and higher layers learning contextual information is not a novel discovery. \n\nBERT and likes are trained to predict the masked word. They are not known to learn sentence representation. It is therefore quite bold to assume that states can be traversed over the discovery typology to induce phrases and sentences. This requires further investigation, but the authors just flashed this towards the end and did not spend more time and experimentation towards this.\n\nIt would have been great to somehow use any probing method as a baseline to match and contrast the findings we obtain in that framework. Commenting on the strengths and weaknesses of both with empirical evident would make the paper much stronger. \n\nI think the paper is not written well. It’s hard for a novice reader to understand.  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ds4vD9XuqM", "reviewer": "Reviewer_5qaG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper exposes the hidden topology behind a pretrained BERT language model by training a CRF-VAE, which describes latent states and their relationships. It shows that many of the discovered states are linguistically meaningful. State transitions are also interpretable and can be followed to construct sentences.", "summary_of_the_review": "This paper introduces an insightful approach to infer the topology of a pretrained language model. However, whether this structure (especially state transitions) is intrinsic to the underlying language model is somewhat ambiguous.", "strength_and_weaknesses": "Strengths\n\nThe analysis is quite detailed, describing both interactions between states and tokens, as well as transitions across states to generate phrases.\n\nThe paper compares both local (token embeddings) and contextualized (last layer) representations, showing how function words become more spread out across states in the latter case.\n\nWeaknesses\n\nState transitions appear to be an artifact of the CRF-VAE, rather than an intrinsic property of the BERT representations.\n\nThere are quite a few linguistically grounded states for random embeddings (Table 1), although less than for BERT representations. While the paper briefly addresses this phenomena, it should be described more clearly. How could we know from the results only that there is no meaningful underlying topology?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2ncWBhYRM8", "reviewer": "Reviewer_ZfD1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a novel method for inferring structure present in the latent embedding layers of large language models. The proposed method predicts word embeddings from latent states and state transitions in the form of a linear CRF. These states conceptually represent \"meta\" words, and the authors use a CRF-VAE architecture to learn the latent states. They analyze the resulting latent state embedding by attempting to identify how often a given latent state will produce a word in a consistent \"category\", where the categories include things such as part of speech and dependency edges. They compare this analysis on both random and pretrained BERT models and observe that the latent states trained on the pretrained BERT model have a higher tendency to capture meaningful linguistic categories. They further include analysis on how this changes between the initial embeddings and contextualized embeddings (last layer) of BERT, concluding that there are \"anchor states\" which correspond to linguistic categories, a long-tail of \"leaf nodes\" which correspond to specific words, and that the state transitions represent meaningful phrases.", "summary_of_the_review": "This work has a novel approach to interpreting the structure present in large language models. While the current presentation is a bit rough, I did not find any glaring conceptual errors, and I am sure that the authors will be able to improve things for a potential camera-ready version. I think the work is potentially worthy of acceptance, pending answers to some of the questions I posed above in the \"weaknesses\" section.", "strength_and_weaknesses": "### Strengths\n1. The provided approach is straightforward and follows a generally useful high-level principle in ML - if the model you are using does not have a certain property (in this case, interpretability), simply fit a model which *does* have this property on top of your model.\n2. The qualitative results shown are compelling, and seem to provide insight into otherwise opaque language models.\n\n### Weaknesses\n1. The biggest concern for this approach is whether the analysis is providing insight into the *embedding geometry* of BERT, or simply the statistics of the data itself, for which the embeddings of BERT are a very strong proxy. For example, it seems like it would be interesting to compare a CRF model trained directly on the surface tokens with the BERTZero analysis. I wonder if the authors had any thoughts on this.\n2. Table 2: As explained, this evaluation seems highly susceptible to confirmation bias. To avoid this, it seems pertinent to (at least) include states from RandEmb and Positional versions as well, and perhaps some other external sources also (eg. synsets from WordNet, ConceptNet, alternative linguistic similarity datasets) to provide externally-validated \"interpretable\" and \"non-interpretable\" sets. Such an evaluation would be far more robust, though the same annotators who performed the current annotations in Table 2 would not be able to take part in this annotation, as they have already observed the clusters from BERTZero and BERTLast.\n3. I was surprised that there was not a large appendix with an extensive collection of qualitative examples of the groundings for various latent states, as well as a critical analysis of when things went wrong. Even better would be an interactive online demo (a la exBERT) where one can observe the latent clusters and transitions for a given sentence. The absence of an extensive compilation of analyses is somewhat surprising for a paper of this style. \n4. (Minor) The writing needs some work, there are a large number of grammatical and typographical errors (see the Clarity section).\n\n### Questions\n\n1. p.2 “For example, before contextualization, the neighbors of suffix #ed, #ing are just random tokens; after contextualization, the neighbors of #ed become past tensed words like had, did and used, and the neighbors of #ing become present continuous tensed words like running, processing and writing”  It is unsurprising that the word neighbor structures change with contextualization, however this particular example seems a bit surprising, because it would seem (a priori) that the suffixes #ed and #ing will always be surrounded by past and current-tense words, regardless of context. Is my intuition incorrect here?\n2. p.4 “Again, the purpose of the decoder is to help induce the states, rather than being a powerful sentence generator.”  While this is true, it also seems like it might be useful to report some measure of quality of the decoder. There are various degenerate settings in which the results presented in Table 1 (or, indeed, even better) could be obtained but which do not actually capture anything about the topology. For example, if 1999 of the latent states were all used to represent a particular word (say \"ball\"), and the last latent state was used for every other word. Obviously this would be a terrible sentence predictor, but it would do very well on the evaluation in table 2. To some extent, the additional analysis (eg. Figure 2) provides reasonable evidence that this is not occurring. Just to clarify - the model evaluated in all later sections is the same as that which yielded the results presented in Table 1, correct?\n3. Table 1:  Could you show how many elements were not match to any category? Based on Table 2, it looks like 311 for BERTZero and 144 for BERTLast, which implies that 218 states in the first case and  950 states in the second were mapped to more than one category. It would be interesting to show some analysis or examples of states which were classified in multiple categories.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject