Disentangling Transformer Language Models as Superposed Topic Models
Authors: Jia Peng Lim
Hady W. Lauw
Venue: EMNLP 2023 Main (2023)
Keywords: Topic Modelling
Mechanistic Interpretability
Pre-trained Language Models
Transformers
TL;DR: We interpret Transformer Language Models as Topic Models with topics in superposition and propose a novel approach to disentangle superposed topics.

Abstract: Topic Modelling is an established research area where the quality of a given topic is measured using coherence metrics. Often, we infer topics from Neural Topic Models (NTM) by interpreting their decoder weights, consisting of top-activated words projected from individual neurons. Transformer-based Language Models (TLM) similarly consist of decoder weights. However, due to its hypothesised superposition properties, the final logits originating from the residual path are considered uninterpretable. Therefore, we posit that we can interpret TLM as superposed NTM by proposing a novel weight-based, model-agnostic and corpus-agnostic approach to search and disentangle decoder-only TLM, potentially mapping individual neurons to multiple coherent topics. Our results show that it is empirically feasible to disentangle coherent topics from GPT-2 models using the Wikipedia corpus. We validate this approach for GPT-2 models using Zero-Shot Topic Modelling. Finally, we extend the proposed approach to disentangle and analyse LLaMA models.

Disentangling Transformer Language Models as Superposed Topic Models
Jia Peng Lim
Hady W. Lauw
Singapore Management University jiapeng. 1im. @021.smu. edu.sg
Singapore Management University hadywlauwesmu. edu. sg
Abstract
neuron as topic. is thus natural treat the in- terpretation of TLM as topic model with topics in superposition. Arora et al. (2018) show that polyse- mous words, i.e., multiple meanings, are in linear superposition within word embeddings. Similarly, we posit to linearly disentangle the logit interpreta- tion of neurons in superposition and examine the neurons in the residual stream (see Section 4). We treat neurons from MLP as belonging the higher- dimensional residual stream.
Topie Modelling is an established research area where the quality of given topic is mea- sured using coherence metrics. Often, we in- fer topics from Neural Topic Models (NTM) by interpreting their decoder weights, consist- ing of top-activated words projected from indi- vidual neurons. Transformer-baat based Language Models (TLM) similarly consist of decoder weights. However, due to its hypothesised su- perposition properties, the final logits originat- ing from the residual path are considered un- interpretable. Therefore, we posit that we can interpret TLM superposed NTM by propos- ing novel weight-based, model-agnostic and corpus-agnostic approach search and disen- tangle decoder-only TLM, potentially mapping individual neurons to multiple coherent topics. Our results show that empirically feasi- ble to disentangle coherent topics from GPT-2 models using the Wikipedia corpus. We val- idate this approach for GPT-2 models using Zero-Shot Topic Modelling. Finally, we ex- tend the proposed approach disentangle and analyse LLaMA models.
Our second insight acknowledges the decade- long interpretability research stemming from topic models. Since Blei (2012), there has been a push to reconcile the evaluation of topie models with the human notion of coherence (Mimno et al., 201 I; Lau et al., 2014; Röder et al., 2015), and still de- bated (Doogan and Buntine, 2021; Hoyle et al.. 2021). We show that their findings might help ad- vance the interpretation of TLM (see Section 3.1).
Our third insight contributes towards tackling the disentanglement problem from novel angle. We map the disentanglement problem to an NP-Hard classical graph problem and propose computa- tionally feasible approach to solve it via heuristics and exact solving (see Section 5). Wary of mirages (Schaeffer al., 2023), we propose a non-trivial baseline and empirically show that the disentangled superposition interpretations are indeed meaningful and not due to chance (see Section 6).
1 Introduction
The term 'superposition' in machine learning typi- cally refers to overlapping concepts. Elhage et al. (2021) allude to the property of superposition in transformers when diseussing the difficulty of inter- preting its multilayer perceptrons' (MLP) neurons in Transformer Language Models (TLM). Black et al. (2022) also find neurons transformers hav- ing polysemantic behaviour. We believe is possi- ble to interpret neurons in superposition, motivated by three key insights.
Contributions, This work aims deploy topic modelling methodologies, specifically in the area of interpretability, to automate the search and eval- uation of concepts within TLMs. We propose a novel neights-based corpus-agnostic approach to disentangle topics from decoder-only TLM with automated evaluation. These extracted topics constrained to the themes in the chosen corpus. In static setting, when considering each GPT-2's neurons (residual, MI.) individually, we empiri- cally show that many are in superposition and may embody multiple distinct or related concepts.
Our first insight lies in the similarity between TLM and Neural Topic Models (NTM) (see Sec- tion 2). Many analyses of transformers examine some variation of neuron activations paths (Shazeer et al., 2020; Dong et al., 2021). Again, NTM also uses paths (albeit usually across one layer) and in- terprets the highest activated logits from a given

---

TLMs are pre-trained on large number of to- kens across a plethora of themes. One possible application is to mine the TLM to identify and ex- plain the neurons' role(s) concerning these learnt themes. To further validate our results dynamically, we employ topic modelling task on GPT-2 in zero-shot manner (ZSTM, see Section 7). Given corpus imput, we register top activating neu- rons and investigate the projected topics. Finally, we extend our proposed approach' on LLaMA (see Section 8), detailing our intuition and approach overcoming the barrier to interpretability intro- duced by its choice of tokenization.
Elimination
Promolion
Logit aotissstttiaai activation Frenggo
X Hi-C
Hi-1
Lo-C
Lo-1
Tokens sorted by logits
Figure The logit distribution of different fn in GPT- 2-XL with three observable areas: 1) Elimination the left tail, 2) Mixture in the middle, and 3) Promotion at the right tail. Vertical lines are 7 thresholds to shortlist token pools. Different paths have similar logit distribu- tion.
2 Related Work
The area closest to our work is mechanistic interpretability (Olah et al., 2020), a white-box approach to explain models via model weights. Many works focus on explaining attention (Voita et al.. 2019; Clark et al., 2019; Hao et al.. 2021). Geva al. (2022) show that sub-updates in feed-forward network layers are interpretable. Mil- lidge and Black (2022) finds interpretable singular value decompositions Transformer weights. Dar et al. (2023) analyse transformers in the embed- ding space. Elhage et al. (2022) and Henighan et al. (2023) analyse properties of superposition in toy transformers. More recently, Cunningham et al. (2023) and Bricken et al. (2023) use sparse autoencoders extract interpretable features from neurons. Without introducing additional neural net- works, our work contributes towards the automated discovery and evaluation superpositions in neu- rons from TLMs.
Alignment Research.
lingual S-BERT document embeddings (Reimers and Gurevych, 2019) for use a zero-shot cross- lingual setting, Wang et al. (2023) propose a TLM- agnostic method to add additional concept tokens, where from its representation, inferring topics after fine-tuning to any specific supervised task. For our zero-shot topic modelling task, we do not employ additional training or modifications GPT-2.
Disentangling Topics in Superposition
For any given neuron in language model M, dif- ferent inputs to M may activate the same neuron to express different concepts. We consider these concepts be in superposition within n, and de- scribable in human-interpretable manner using groups of words - topics. If we can obtain its final logit representation fn E R we posit that it is empirically possible to linearly disentangle fn into multiple interpretable vector representations i in vocabulary space V with left- over abstract vector representation On (Equation 1),
Topic Models. While carlier works utilised prob- abilistic graphical models (Blei et al., 2003), recent works focus neural-based approaches. There many NTMs (Miao al., 2016; Srivastava and Sutton, 2017; Zhang and Lauw, 2020; Dieng al., 2020; Zhang al., 2022) that incorporate an autoencoder framework (Kingma and Welling, 2014), while others (Yang et al., 2020; Pham and Le, 2021; Zhang et al., 2023) utilizes graph neu- ral networks (Kipf and Welling, 2017; Liu et al., 2019). TLMs have considerable influence such neural topic models. Bianchi et al. (2021a), Grootendorst (2022), Han al. (2023) use TLM embeddings in their topic modelling approaches. Bianchi et al. (2021b) propose an NTM that under- goes self-supervised training on pre- pre-trained multi-
(S) on token-space S,
K
fn = On
ik Vn E M
(1)
k=
Similar to the behaviours observed in Geva et al. (2022), we observed that fn. is dividable into three distinct behaviours: promotion, elimination, and mixture (Figure 1). If i, exists, we expect in promote or climinate specific themes. For uninter- pretable mixture behaviour, we liken it to an.

---

Projection
Shortlisting
Exuct-solving
Token
Word pool
Word sets
Logit distribution/ neuron-path
Topics
nome tragan windom spear myth goddess serpent slain odin [0.178]
Logits activation strength
pos
pession bumning flame eches candle verggammce piercing killing explosive flames (0.100)
corpus
cyber hackers hacking
neg
cybersecurity securtty technology encryption software ethical (0 (0119)
career sociology advertisers
scholarship reporters newspaper
neg
Token soried by
pos
Project token pools corpus' vocahulary word-statistic odgo-weighted graph
Use heuristics to break graph into smaller subgruphs
economies arts journalism (0.103)
pos and neg token pools from logit distribution
Extract coherent topics from subgraphs
Figure 2: Illustration of three-step approach. Topic examples are from random flai in GPT-2-XL. The word graphs shown are for illustration purposes. Values in parenthesis are the respective topics' NPMIw.
Origin
D₁ MLP (Hi)
Dj Residual (Lo)
My
W/ 7.
X1
cHi-C fja
Lo--C fjd
Ti
Complete (C)
T2
772
Hi-C
T
Lo-C
W3 T,(.)
X1
Distance
OO
Ta
A
T1
Ta
W/2 -g
X
Immodiate (I)
f
La-1
M
My
M₁
C
Hi-l
OO
Lo-I
T'
Da residual
Dj residual stream
Ta
Figure 3: Illustration of high-level abstraction of the decoder-only transformer model in D₁ and lower-dimensional D; residual stream.
Figure 4: Visualisation of neuron paths S Paths (Origin-Distance) projected from individual neurons.
3.1 Three-Step Disentanglement Approach
P Plw,, I+e log P(w:) P(ws)]
Figure 2 provides an overview of our framework, and we outline the three steps below.
upmilw,; uy) - Mog(P((w)) wy)
(2)
Projection. We begin by projecting fn onto cor- pus statistics. Intuitively, knowing what to look for helps the search. We use Wikipedia as our reference corpus and Normalised Pointwise Mu- tual Information (NPMI) as our corpus statistic. Introduced in Bouma (2009), the NPMI score of given topic is the mean of NPMT between each possible word-pairs in the group (Equation 2). Lau al. (2014) and Röder al. (2015) found correla- tions between NPMI scores and human ratings on a subset of Wikipedia corpus. We calculate NPMT on Wikipedia's 's statistics (NPMIw) and ZSTM's selected corpus statistics (NPMIc).
We use Wikipedia-EN corpus statistics prepared in Lim and Lauw (2023) (Wikipedia-V400) V40K3 that contains its top 40K most frequent vocabulary ex- cluding stop-words. For NPMI calculated from this corpus statistics, Lim and Lauw (2023) bench- marked its correlation to human judgement = 0.66, with 40 unique study participants split across eight different study groups. We reiterate that it is possible to use other corpora for this pro- jection step (see Section 7).
"gi thub. cow/Preferreddt///eeee wtwpeedA//toiiiicooeeeeee metriccs with arti- ele bodies processed using Attardi (2015).

---

Shortlittngg Since our interests lie in in. the tails of fn. we start by shortlisting the top-7 and bottom-7 tokens, respectively termed pos and neg token pool. Producing word pools from these to- ken pools, we subsequently individually segment into word sets via heuristics using corpus statistics. For some this step may produce zero word sets, suggesting the concepts within do not lie within the corpus domain.
Different Paths. There are many possible paths from any n E M to the final logit layer. We con- strain our search to origin-distance pairs with origin (higher (Hi) or lower (Lo) residual dimension) and distance (immediate (I) or complete (C)). Figure 4 visualises the different activation paths for the four modes. Let neuron path 15 represent the final logits of neurons (Equation 7), at index d in layer j. Its D-dimension boolean representation (Equa- tion 8), with Dj and Dn representing lower and higher dimensions, as the initial input and injected into starting layer j (Equations 9,10,11,12).
Exact Solving. Finally, within each word set, we locally optimize it against NPMTw to get a b- sized coherent topic. A standard fixed topic size allows parity in comparison across differently-sizee word sets and ease of human identification. We validate our approach in Section 6, where we show empirically that the distribution of tails in differs from non-trivial random distribution.
Syd = Mu ,,d,L)
(7)
nd = (bi, bD): bu = b, =0
(8)
For Paths Hi-C and Hi-l, we examine MLP neu- rons in Т3 computing all subsequent layers after layer j (Equuation 9) with limits [1, Lj. Hi-C obtains complete neuron path şHi-C (Equation 7). However, j.d Hi-l only computes its transformation to the Dr- dimension (Eiquation 10). Starting from Dn-MLP is equivalent to evaluating path Lo on layer j on compositional input.
4 Path Definitions
Transformer Architecture. From Elhage et al. (2021), we adopt a similar notation style. However, we further differentiate the residual stream into higher and lower dimensions, respectively D2 and Dt, as illustrated in Figure 3. We analyse the acti- vation paths of decoder-only transformers. Given a pre-trained language model M with layers, with tokens from token-space S. we get its initial embeddings 26 via embedding layer ME (Equa- tion 3). At each layer 1, 24 E ! increases its dimensions via W and is passed into Transformer block Tj. The -utputt of Ti is then transformed back to D; space via W' (Equation 3). Final logits f IR S1 obtained using unembedding layer My on the Di-residual layer zy (Equation 4).
05 Dr
Hi-C a '.d.
Tid W WI) -W!
[=j
(9)
T
jels
Hi-l J,d,L
W 0 d Dr
(10)
For Paths Lo-C and Lo-1, we examine neurons in Dy-residual layer j before Ty. Path Lo-C com- putes complete path (Equation 1 I1) and Lo-I only computes layer j (Equation 12).
37 = (x((t) ( w/) wó 0<ISL
1= 0
(3)
Lo-C j.d.
nd Ti(afar w2) WW)
0dd<D:
(=j-1
(11)
jSlSL
(c(L)
(4)
Lo-I "jd,L = Tymna- Wf) Wy. 0 d < Dj (12)
Presenting higher-level abstraction, we omit details that differ between different models, such as positional embeddings and normalisation. Usually, in each layer I, attention is applicd via the various attention heads H, (Equation 5) (Vaswani et al., 2017). Attention layer A is connected to MLP mi in residual manner (Equation 6).
From each ppath we shortlist top-7 and bottom-T tokens, forming pos and neg token pools represent- ing in. After projecting it onto a corpus, we obtain word pools that we represent as word graphs
Methodology
In a word pool G, each word V is connected via word-pairs (v, 0j) E weighted by corpus statistic Wey,y. We can formulate extracting b-sized cliques from as disentangled topics into the fol-
Ar(x) Σ h(z)
(5)
KEH,
Ti(x) = x + (x) + (x + A())))
(6) lowing graph problem.

---

Algorithm 1: Star Heuristic
Function Star_Heuriss Hleuristic(graph G, edge limits 0,
size limits
max
Eig Wij
(13a)
1
Delete
ijEE
(1,, E G if Wu < 0: Mu > Ou:
Delete e Gf [ee 8)) Gl < KG
s.t
3i {0.1), ViEV
(13b)
Stars + [Create_ Star(n, GJ G];
Eijj {0,1), Vi
(13c)
Order star E Stars descending
V
starmeaa score;
Chosen f- 0:
Σ Iv =
(13d)
Sets <-
for star Stars do stancentre E Chosen then continue;
Σ Cij = b(b 1)
(13e)
2
end
iJEK
g + {u E star.neighbours/o & Chosen);
Ci S Tz Vi,ji E
(13f)
g 2 Ku then
g (v top Ku largest
Ci
(13g)
Weareemeee : v stannniihhoouss::
end
< gu star.centre;
Automated Quantitative Evaluation
[g] 2 Ky then Scts + Sets Ug:
GPT-2 (Radford et al., 2019) is widely used in mechanistic interpretability research. We use model sizes ranging from S (124M) to XI. (1558M) (See Table 16 in Appendix B for details). Token space SGPT22 contains 50,257 tokens encoded using the byte-pair encoding (BPE) algorithm (Sennrich et al., 2016). We map 31K of GPT-2's tokens di- rectly to 19K words E Wikipedia-V400
end
for É de
Chosen + Chosen U U
end
end
return Sets Function Create_Star G):
return stancentre +- V.
star.neighbours < [uyl(e, 25) E G).
2,, Emacorpphaar Weyay
VWKi. No further processing is required, as we recover a large portion of Vwiki-
starmean _scoeess_eeeeee
Baseline. We use random distribution of log- its to derive non-trivial set of coherence topics that occur due to chance. In random word pool, the number of possible b-sized topics scales expo- nentially with т. resulting in favourable odds of producing topics that create a false illusion of in- terpretability. The random baseline uses the same sampling process and aaperrparameters of GPT- 2 to generate 50,000 random token pools. Sampling Parameters. To shortlist the impor- tant tails of J. we tune hyper-parameters using the first 200 neurons from the first layer of different paths (See Appendix A.1). We locally optimize each word set on NPMIw to get a topic of size 10. Sampling Observations. Comparing GPT- to its random baseline, differences in word sets/neuron are significant (see Table 1). We ob- serve that word pools contain hundreds of words, with pos pools double the size of neg pools. Addi- tionally, pos pools have more word sets per neuron than neg pools, although the mean size of word sets remains similar. The random baseline pro- duces more word sets per neuron than neg pools but produces fewer word sets than pos pools while sampling a significantly larger number of words.
Maximum Edge- Weighted b-Clique (MEWC). Given graph G, with vertices V and edges E, find max-war weighted b-sized clique in G. This prob- lem is NP-hard (Dijkhuizen and Faigle, 1993), as it reduces to the maximum clique problem (Karp, 1972). As G has a non-trivial amount of vertices, finding a top arbitrary number of non von-overlapping MEWC in G is computationally challenging. To enable feasible computation to search for highly- design star heuristic (AI- gorithm 1), inspired by Park et al. (1996)'s facet- defining star inequalities, breaking into smaller sub-graphs for exact solving.
weighted b-cliques,
Exact Solving Subgraphs. Gouveia and Mar- tins (2015) explored the feasibility of solving differ- ent convex formulations of MEKC on G of varying degrees and sizes. We adapt Park et al. (1996)'s for- mulation to fit our use case. We arrive at a Mixed Integer Program (Eq. 13) that maximises the weight of selected edges (Eq. 13a), selecting edges (i, f) and vertex i by boolean variables eú, (Liq. 13f) and Ty (Eq. 13g) respectively. We constrain clique size to (Eq. 13d, 13c) and selection of edges within selected clique (Eq. 13f, 13g).

---

Words/Word Pool
Word Sets/Word Pool
Words/Word Set
modes
pos
neg
pos
neg
pos
neg
S-Hi-C
388.1 + 122.1
216.680 + 87.9
6,90 = 5,43
2,07 + 1.89
15.19 + 4.84
12.82 + 3,30
S-Hi-l
480.0 104.9
229.119 = 87.0
9.86 5.79
2.19 2.22
14.37 + 4.26
12.65 = 3.21
S-Lo-C
326.4 125.2
228.968 + 86.1
5.92 5,00
1.70 1.11
15.10 4.76
12.49 2.91
S-Lo-I
437.3 + 113.1
336.300 = 114.1
8.11
3.76
14.47 + 4.38
12.74 3.14
XL-Hi-C
473.0 = 129.8
250. + 83.9
3.08 4.09
14.88 + 4.55
13.66 + 4.00
XI-Hi-I
478.2 96.0
244.775 = 106.4
10.02 = 3.73
3.84
14.60 1.600
14.00 = 4.17
XL-Lo-C
466.4 102.6
244.433 # 79.7
10.02 5.06
2.48 3.64
14.78 4.51
13.44 3.96
XL-Lo-1
450.4 + 89.2
351.506 + 106.3
6.88 + 4.10
3.99 3.55
13.76 + 3.94
13.01 + 3.42
Random
559.2 = 14.2
5.77 + 1.87
12.04 # 2.37
Table 1: Sampling statistics of GPTT-2 Full results in Table 11 in App. B. Using the Mann-Whitney differences in distributions between GPT-22 and random baseline on number of Word Sets/Word Pool are significant at p < 0.001. For GPT-2, there are no empty word pools, i.e. each word pool contains some vocabulary from Vwai-
Whitney U test.
Mean num. Topics
Top-1 Topic (Topical Pool) NPMly score
Mean NPMh score per Topical Pool
Total of topics
%
Abstract Pool
modes
pos
neg
pos
neg
pos
neg
pos
neg
pos
neg
both
S-Hi-C
6.76
2.08
0.202 a: 0.06
0.216 +: 0.09
0.130
0.164
211,612
22,106
7.5
35.6
6.0
S-Hi-I
9.76
2.19
0,218 1 0.06
0.166 1 0.08
0.119
0.132
351,968
24,634
1.1
34.8
0.6
S-Lo-C
5.68
1.70
0.215 + 0.07
0.199 0.09
0.142
0.162
40,865
4,725
34.9
7.4
S-Lo-1
8.10
3.56
0.227 + 0.07
0.191 + 0.08
0.122
0.117
70,502
21,913
2.8
16.6
1.2
XL-Hi-C
10.88
2.98
0,232 + 0.07
0.160 0.07
0.127
0.126
3,135,250
291,738
3.1
34.0
0.6
XI-Hi-I
9.12
3.78
0.199
0.160 + 0.06
0.117
0.118
2,665,849
535,733
2.4
26.9
0.0
XL-Lo-C
9.62
2.47
0.227
0.07
0,155 à: 0.07
0.128
0.126
712,337
59,817
1.8
34,2
0.9
XLALo-
6.88
4.00
0,212 & 0.06
0,182 & 0.07
0.111
0.113
508,179
217,321
1.9
14.6
0.3
Rundom
5.77
0.145 0.03
0.064
288,091
0.54
Table Aggregated results across all layers. Using the Mann-Whitney test, differences in distributions top-1 NPMIw scores between GPT-2-samples and random baseline are significant at p < 0.001. Topical Pools and Abstract Pools are pools that respectively produce some topics and no topics. We consider a word frequently occurring (FOW) when it in more than 2.5% of the topics and remove "stop topies" with more than five FOW.
Disentangled Topics The results Table show significant differences between random base- line and GPT-2's pos pools in 1op-1 and mean NPMIw score. Although there is close differ- ence between random baseline and neg pools in top-1 NPMIw score, there is still sizable differ- ence between their mean NPMIw score. Across different model sizes, larger models seem to gen- erate more topics per frath_ Additionally, while Top-1 NPM[w for pos pools remains consistent, we see slight decline of scores in NPMIw in Top-1 NPMIw for neg pools and mean NPMIw. Compared to pos pools, neg pools have more ab- stract fpath producing zero word sets. From these results, we empirically show that it is possible to disentangle multiple superposed topies.
Zero-Shot Topic Modelling (ZSTM)
In a traditional topic modelling task, we seek derive K number of topics of size for corpus anal- ysis. In this ZSTM task, since the pre-training of the TLM is not specific to the selected corpora. the corpus becomes the analysis tool for TLM instead. If our interpretation of its neurons is valid, given corpus, expect neurons with similar concepts to be activated frequently. Otherwise, there will be contradiction. Obtaining a competitive topic set will validate our previous findings. We do not re- quire additional tokens, prompting, or fine-tuning tie-tunnng. Approach. We implement a voting scheme giv- ing each token some votes for each layer, examin- ing neurons that exceed the vote threshold (See Ap- pendix A.2). For GPT-2, we extract disentangled topics from Paths Lo-C and Lo-I of the selected neurons. To select the final K topics, we use a greedy algorithm (Lim and Lauw, 2022) optimiz- ing on corpus NPMI; with diversity constraints.
Case Study. We showcase an example from random frath in GPT-2-S (Table 3). The extracted topics have similar and varied themes: finance (#1), visuals (#2), places (#3, 4), and sports (#5, 6).

---

Word pool (alphubeticallly-sotee)
Word Sets
Extracted Topics (NPMIw>
ubility adult arch avenue avenues away ball bar bure bay black cup ceiling chance clock commer- cial competition
domestic wholesale fair sell price product fi- nancial markets commercial value demand con- sumers trade free market
wholesale price demand sell product value market markets domestic con- sumers (0.197)
dark green white vert waved finely thin black spoi wing cup long upper edge wide bare lightly surface adult ground shape distribution lower pale
long pale black white thin green upper wide dark finely (0.187)
ner crossed dark demand distri- bution domestic draw edge est fair financial finely floor grah green ground hall hand hands held holding ind kneeling level lightly line lines long lower lying market markets minute minutes open pale platform platforms price product resting round run sell shape sit sit- ting slip spot stage straight street surface thin trade trophy upper value vert waved white wholesale wide wing wood yard (456 ommitted words)
arch crossed lying wood clock knceling resting bar hall bay sit sitting level cciling corner floor
hall lying resting corner sitting ceil- ing arch floor kneeling (0.106)
avenue avenues yard street est line lines plat- forms platform ind
est line lines ind yard platform platforms avenue avenues street (0.103)
trophy open stage away competition round held minute minutes draw
stage open round draw away tro- phy minute held minutes competition (0.102)
holding ability grab straight hand hands slip run chance ball
slip straight hands hand ability hold- ing chance ball grab (0.069)
Table 3: GPT-2-S, Path Hi-C, Layer 11 Neuron 245, pos pool. More examples in Appendix C.
Model
NPMIc
NPMBw
TU
A (%)
Model
NPMIc
NPMIw
TU
A (8)
ProdLDA
-0.059
0.045
0.903
ProdIDA
-0.028
0.007
0.741
ProdLDA-8
0.085
0.075
0,911
ProdLDA-9
-0.023
0,014
0.830
CTM
-0.060
0.052
0.916
CTM
0.027
0.050
0.895
CTM-8 8
0.016
0.070
0.945
CTM-8
0.067
0.057
0.835
BERTopic
-0.002
0.069
0.887
BERTopic
-0.049
0,023
0.906
GPT-2-S-Lo-C
0.164
0.123
0.955
20.57
GPT-2-S-Lo-C
0.135
0.021
0.850
32.31
GPT-2-S-Lo-I
0.169
0.115
0.950
20.57
GPT-2-S-Lo-I
0.143
0.043
0.885
32.31
GPT-2-M-Lo-C
0.177
0.117
0.965
23.73
GPT-2-M-Lo-CC
0.156
0.021
0.865
46.52
GPT-2-M-Lo-I
0.186
0.120
0.920
23.73
GPT-2-M-Lo-1
0.148
0.042
0.865
46.52
GPT-2-L-LLo-C
0.197
0.115
0.945
14.99
GPT-2-L-La--C
0.164
0.028
0.855
38.20
GPT-2-L-Lo-I
0.170
0.1188
0.965
14.99
GPT-2-L-Lo-I
0.143
0,033
0.855
38.20
GPT-2-XL-LO-C
0.184
0.112
0.940
15.37
GPT-2-XL-LO-C
0,180
0.023
0.845
43.39
GPT.2.XT..Io.I
0.179
0.140
0.955
15.37
GPT-2-XI.LL.-
0.159
0.045
0.850
43.39
(a) 20NewsGroup. K = 20. Min. word-pair count = 50.
(b) MI0, K = 20. Min. word-pair count = O.
Table 4: A subset of results for topic set sizes K 20. The best baseline scores for the metric underlined. Bolded PLM scores are better than the best baseline. The complete results are Appendix B. Full results for topic set sizes K 20 in Table 14, K 50 is in Table 13. Baseline scores are the mean of 10 independent runs with eгror bars in Table 15. A is the percentage of neuron paths frulh examined, determined by token votes.
Corpus. We generate sets of 20 topics for 20NewsGroup', BBC-News (Lim and Buntine, 2014), M10 (Greene and Cunningham, 2006), and DBLP (Tang et al., 2008; Pan et al., 2016) pro- cessed from OCTIS (Terragni et al., 2021), with additional sets of 50 topics for 20NewsGroup and BBC-News. Corpus details in Appendix A.2.
Baselines. We use three baselines. ProdLDA (Srivastava and Sutton, 2017) and CTM (Bianchi et al., 2021a) are both popular and competitive autoencoder-ben based NTM trained on the pre-defined train and validation sets in OCTIS. We derive addi- tional sets of results by selecting the best compos- ite topies (suffixed with -ß) using the same greedy algorithm. Lastly, we include BERTopic (Groo- tendorst, 2022) which uses class-based TF-IDF on S-BERT document embeddings*. For parity rea- sons, topics comprise the best ten words (b = 10) appearing in Hikipedia-V40K.
Metrics. To account for rare occurrences, we set a minimum count of word pairs for NPMIc caleu- lation, measuring relevance to the corpus. NPMIw evaluates the generality and coherence of the topic. Desiring uniqueness amongst topics, Topic Unique- ness (TU) (Dieng et al., 2020), measuring the pro- portion of unique words in K topics. To obtain competitive sets of results, we must overcome three-way trade-off between the metrics.
Evaluation. Overall, the topics reclaimed in GPT-2 models are comparable stronger baselines, suggesting the static interpretations of individual neurons are likely to be meaningful.
'http://people, www.ssaaalli........... www.miiteedu miittedu/jrenniee 2@Newsgroups/
"all-mpnet-hase-v2 S-BERT model.

---

Words/Word Pool
Word Sets/Word Pool
Words/Word Set
Empty Pools
modes
pos
neg
pos
neg
pos
neg
pos
neg
7B-Hi-C
609.3 + 323.2
39.6 = 52.9
7.22
3.07 # 3.18
13.59 = 4.29
14.10 + 4,21
14.0%
85.8%
7B-Hi-I
143.5 +: 64.2
143.5 + 64.4
3.60 a: 2.25
3.63 a: 2.29
13.82 4.26
13.68 3.75
15.4%
15.4%
7B-Lo-C
646.4 314.0
33.1 43.6
7,51 4.990
2.66 +: 2.76
13.57 4.26
13.81 # 4.01
11.6%
89.3%
7B-Lo-I
145.1 + 85.4
142.9 76.5
2.70
2.58
13.86 = 4.29
13.77 3.79
19.6%
23.0%
13B-Hi-C
310.8 + 121.7
100.9
7.35
2.18 = 1.93
13.04 14,21
1.2%
48.0%
13B-Hi-I
162.3 + 38.3
162.4
3.97 +: 1.85
4,00 + 1.88
13.61 +
13.51 + 3.59
2.4%
2.4%
13B-Lo-C
312.9 + 123.4
32.2
7.39 + 3.08
2.10 1.84
14.25 = 4.69
12.96 2
1.1%
48.7%
13B-Lo-I
166.1 + 50.0
158.2 41.5
4.13 d: 2.12
4.05 + 2.04
13.72 + 4.13
13.52 d: 3.59
2.0%
4.8%
Random
173.7 # 13.2
4.88 1.30
13.65 3.63
Table Sampling statistics on random LLaMA layers, Before the removal of stop topics. Using the Mann- Whitney U test, differences in distributions the number of Word Sets/Word Pool are significant with p < 0.001.
Mean num. Topics per neuron
Top-1 Topic (Topical Pool)
Mean NPMI score
Total num. of Topics
% of abstract neurons
NPMI score
per topical neurons
modes
pos
neg
pos
neg
pos
nng
pos
neg
pos
neg
both
7B-Hi-C
6.98
3.07
0.128 + 0.08
0.102 # 0.06
0.031
0.060
2,094,472
153,571
7.4
42.9
4.7
7B-Hi-I
3.60
3.63
0.103 + 0.05
0,103 = 0.05
0.050
0.050
1,071.527
1,082,455
7.7
7.7
0.5
7B-Lo-C
7.24
2.66
0.130 0.08
0.098
0.06
0.030
0.060
831,424
37,233
6.2
44.7
4.1
7B-Lo-1
3,80
4,10
0,101 + 0.05
0,105 + 0.05
0.049
0.048
400,292
413,202
9.8
11,5
0.1
13B-Hi-C
7.35
2.18
0.122 0.05
0.054:0006
0.039
0.033
4,016,410
627,028
0.6
24.0
0.0
13B-Hi-1
3,97
4,00
0.094 + 0.05
0.094 0.05
0.037
0.038
2,143.710
2,159,556
1.2
1.2
0.1
13B-Lo-C
7.39
2.10
0.122 + 0.05
0.052 = 0.06
0.039
0.032
1,496.621
220.585
0.5
24.3
0.0
13B-Lo-1
4.13
4.05
0.095
0.04
0.091 0.04
0.038
0.037
829,311
788.671
1.0
2.4
0.0
Random
0.03
0.038
244,145
Table Aggregated results of all LLaMA layers, stop topies excluded. Using the Mann Hann-Whitney test, differences in the distribution of Top-1 NPMI scores are significant with p 0.001.
The top K extracted topics from GPT-2 models performed the best on 20NewsGroup (Table 4a). At K = 20, BBC-News have similar results to 20NewsGroup, and DBLP has similar results to MI0 (Table 4b). Evaluating 20NewsGroup and BBC-News at K 50, performance for top K extracted topics is also competitive.
is possible to recover 36K vocabulary words in Vwii from SLLAMA. Using the previous analysis on GPT-2, we examine all layers of LLaMA models sized 7B and 13B.
Sampling Observations. Only pos pools from complete paths (Hi- (Hi-C/Lo-C) show a sizeable dif- ference compared the random baseline. For the various pool from other modes, while the differ- ences are significant, their values are similar or weaker than the random baseline, which suggest they might not be as meaningful.
8 Extending to LLaMA
LLaMA (Touvron et al., 2023) is another pop- ular decoder-only TLM. Its 32K tokens SLLAMA encoded using SentencePiece's BPE (Kudo and Richardson, 2018). Cross-referencing SLLAMA with Vwiii produces few common words, as its tokens are sub-words which are uninterpretable as-is, ne- cessitating additional processing of token pools. Observing the possibility of recovering coherent topics directly from GPT-2's token space and con- sidering word as topic of sub-words, we posit that it is possible to recover vocabulary from SILaMA- We decode each lower-cased v E Vwai via SentencePiece (excluding start token 
) into its constituent tokens. Projecting the tokens onto VwKi. we consider a word recovered when all of its constituent tokens are present in the token pool.
Disentangled Topics. Similar to the observa- tions in sampling, when compared to the random baselines, pos pools from complete paths produce more topics and are better in Top-1 NPMIw. How- ever, pools from the other modes exhibit similar or weaker characteristies, implying that these dis- tributions are indistinguishable. In GPT-2, when compared to their bascline, pools from all modes appear to be meaningful, exhibiting different dis- tributions across metrics. However, results from LLaMA suggest we can only infer and extract top- ics from pos pools produced by complete paths.
*We exclude multi-word entities representation in Visss, as they are joint together using underscores.

---

Word pool (alphaheticall sorttl) acid ais bar baritone bases bass bird blog boer bone broad cap egi cis closely command commanding composer containing contains con- tributions dating dean der des die dna dusk editing extra famous Ida lisher Mag flat floor follows funce tion functionality functionally func- tioning functions generates goom- eiry germany ghost gis guesi gull' hand handing hang host http hlips integration intentionally jerry lang legs lets like log long longer marker meaning modification mortally nat- urally nosa upr ones perry pho- tos plan platform platforms possess
Word sels
Extracted Topics (NPMly)
8
acid bases cap closely containing contains dna editing follows function functionally function- ing generates host long marker modilication nat- urally possess stem ra
dna ma bases modification acid stem liunction naturally containing contains (0.115)
bird bone broad dusk extra flat floor hand legs like longer meaning
weight toes hand flat broad legs touch louching bome soft (0.111)
2
sand soft standing
touch touching weight toes
blog egi lda hitp hilps lang log nona non upr photos platform sport upload www
cgi www sport photos hitp hiips log up- load platform blog (0.081)
3
baritone command composer contributions des die famous geumetry germany guest slogan van
contributions die command famous ger- many van des composer guest der (0.048)
ais har hass functionality functions integra- tion plan platforms unit gis
integration plan bar functionality gis functions ais platforms cis unit (0.001)
5
post pull realize ma sally sand sas sent slogan soft sport standing stem stratogic ted toes touch touching unit upload van weight www (336 cmmmtted words)
dating dean ghost hang intentionally jerry pull realize sally led lets
dean pull lets sally hang ghost realize 6
jerry ted dating (-0:053)
bocr fisher flag gulf' handing mortally perry post sus sent strutegic commanding
post mortally gulf perry flag command- 7 ing sus sent handing boer (-0.059)
Table 7: LLaMA-13B, Path Hi-C, Layer Neuron 206, pos pool. More examples in Appendix C.
Case Study. From LLaMA-7B (Table 7), topics consists various themes: cell biology ($1), descrip- tors ($2), website ($3), mixed ($4), technical (85). However, some topics seem incoherent, with our best guesses: horror trope (86) and British military ($7). The size of word sets might hint at some aspect of topic quality.
Tokenization. With LLaMA organizing its knowledge at the token level, reconstructing vo- cabulary from SILaMA introduces another layer of abstraction. This abstraction may result in some in- formational gap between the word and token level that reduces our ability to infer from the neuron directly.
We believe these are the key challenges over- come when pursuing prompt-less explanations of TLMs.
9 Discussions
Aside from comparing across different sizes within the model family, we can also compare across model families. There are differences between the results between GPT-2 and LLaMA. In this section, we discuss the possible cause and effect.
10 Conclusion
We demonstrate that our approach from novel an- gle, disentangling superposed topics from TLM via graph-based formulation optimizing topic mod- elling metric NPMIW, revealing and analysing su- perposed topics on GPT-2, evaluated on ZSTM, and extendable to LLaMA. Additionally, we show that TLM is quantifiable by topic coherence and the number of superposed topics in an automated manner. Comparison using a random baseline gives us confidence when validating interpretations to be meaningful and not due to chance. These metrics might advance our understanding of architectural design decisions and warrant future investigation on other open-sourced TLM.
Topical Sparsity. The larger sizes of TLMs might induce greater topical sparsity, with larger models able to spread their learnt concepts over a larger number of neurons. With greater topical spar- sity, there is greater difficulty in inferring concepts directly from the ncuron as its characteristics may be similar to the random baseline. With millions of extracted pos pools topics from complete paths of both models, the topics from LLaMA are fewer per neuron and less coherent when compared GPT-2, Hence, it is likely that the learnt concepts, with similar themes to Wikipedia, are less concentrated within LLaMA.
A Acknowledgments
Number of Training Tokens. With LLaMA training on larger and more diverse corpus, it likely learn more concepts absent from Wikipedia. These out-of-domain concepts are un- detectable when projected onto Vwiki causing the neuron to appear less coherent.
This research/project is supported by the National Research Foundation, Singapore under its AI Sin- gapore Programme (AISG Award No: AISG3-PhD- 2023-08-055T). We thank our reviewers for their kind feedback.

---

Limitations
David M. Blei, Andrew Y. Ng, and Michael Jordan. 2003. Latent dirichlet allocation. J. Mach. Learn. Res., 3:923-1022.
Language. Corpora used primarily consist of English vocabulary. The TLMs used are also mostly trained on tokens from English-based cor- pora. Since our approach is corpus-agnosiie ex- tracting multi multi-lingual information may be feasible with an appropriate corpus.
Gerlof Bouma. 2009. Normalized (pointwise) mutual information in collocation extraction. Proceedings of the Biennial GSCL Conference 2009.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly. Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby. Yifan Wu, Shauna Kravec. Nicholas Schiefer. Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Bray- den McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. 2023, Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Cir- cuits Thread.
Exact-Solving Time Limit. Required due to large numbers of sub-graphs. Some sub-graphs fail to solve within the given time limit of 2 or minutes. Nevertheless, we include the sub-optimal topic as the representative topic of the sub-graph. Topic Quality. While NPMIw is likely to give a good indicator towards the quality of the topic. There is possibility that some sets of words appear coherent to group of experts while appearing incomprehensible to others.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT's attention. Pro- ceedings of the 2019 ACL Workshop BlackboxNLP. Analyzing and Interpreting Neural Networks for NLP, pages 276-286, Florence, Italy, Association for Com- putational Linguistics.
Ethics Statement
We do not foresee any undesired implications stem- ming from our work. Conversely, we hope that our work can advance AI Ethics research.
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. 2023. Sparse autoencoders find highly interpretable features language models.
References
Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. 2023. Analyzing transformers in embedding space.
Sanjecv Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. 2018. Linear algebraic struc- ture of word senses, with applications to polysemy. Transactions of the Association for Computational Linguistics, 6:483-49
Adji B. Dieng, Francisco J. R. Ruiz, and David M. Blei. 2020. Topic modeling in embedding spaces. Truns- actions of the Association for Computational Linguis- tics,
Giusepppe Attardi. 2015, Wikiextractor. https:// github. cwm/ http///tttt//iiittaattttttc
G. Dijkhuizen and U. Faigle. 1993. A cutting-plame approach to the edge-weighted maximal clique prob- lem. Eumpean Journal of Operational Research, 69(1):121-130)
Federico Bianchi, Silvia Terragni. and Dirk Hovy 202 Pre-training is a hot topic: Contextualized document embeddings improve topic coherence. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguisties and the 11th International Joint Conference on Natural Language Processing Volume 2: Short Papers), pages 359-766, Online. Association for Computational Linguistics.
Yihe Dong, Baan-Bapistte Cordonnier, and Andreas Loukas. 2021. Attention is not all you need: pure attention loses rank doubly exponentially with depth. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 2793 2803. PMLR.
Federico Bianchi, Silvia Terragni, Dirk Hovy, Debora Nozza, and Elisabetta Fersimi. 2021b. Cross-lingur contextualized topic models with zero-shot learning. Proceedings of the 16th Conference of the Euro- pean Chapter of the Association for Computational Linguistics: Main Volume, pages 1676-1688, Online. Association for Computational Linguisties.
Caitlin Doogan and Wray Buntine. 2021. Topic model or topic twaddle? re-evaluating semantic inter- pretability measures. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages Association for Computational Linguisties.
Online.
Sid Black, Lee Sharkey, Connor Leahy, Beren Millidge, CRG, merizian, Eric Winsor, and Dan Braun. 2022. Interpreting neural networks through the polytope lens.
Nelson Elhage. Tristan Hume, Catherine Olsson. Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds. Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and
David M. Blei. 2012. Probabilistic topic models. Com- mun. ACM, 55(4):77-84.

---

Christopher Olah. 2022. Toy models of superpo- sition. Transformer Circuits Thread.
Diederik Kingma and Max Welling, 2014. Auto- Encoding Variational Bayes. In 2nd International Conference Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Conference Truck Proceedings.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli. Zac Slatieed-Doddde Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Ka- plan, Sam McCandlish, and Chris Olah. 2021. A mathematical framework for transformer circuits. Transformer Circuits Thread.
Thomas N. Kipf and Max Welling, 2017. Semi- supervised classification with graph convolutional networks. In 5th International Conference on Learn- ing Representations, ICLR 2017. Toulon, France, April 24-26, 2017, Conference Truck Proceedings. OpenReview.net.
Taku Kudo and John Richardson. 2018. SentencePicce: A simple and language independent subword tok- enizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguisties.
Mor Geva, Avi Caciularu, Kevin Wang. and Yoav Gold- berg. 2022. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing. pages 30-45, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Luis Gouveia and Pedro Martins. 2015. Solving the maximum edge-weight clique problem in sparse graphs with compact formulations. EURO Journal on Computctionall Optimization, 3-1):
Jey Han Lau, David Newman, and Timothy Baldwin. 2014. Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality. In Proceedings of the 14th Conference of the Euro- pean Chapter of the Association for Computational Linguistics, pages 530-539, Gothenburg, Sweden. Association for Computational Linguisties.
Derek Greene and Padraig Cunningham. 2006. Practical solutions to the problem of diagonal dominance in kernel document clustering. In ICML, volume 148 of ACM International Conference Proceeding Series.
Jia Peng Lim and Hady Lauw. 2022. Towards reinter- preting neural topic models via composite activations. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 3688-3703, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
pages 377-384.
Maarten Grootendorst. 2022.
Bertopic: Neural topic modeling with class-based tf-idf procedure. arXiv preprint 1rXiv:2203.0579/
Sungwon Han, Mingi Shin, Sungkyu Park, Changwook Jung, and Meeyoung Cha. 2023. Unified neural topic model via contrastive learning and term weighting. In Proceedings of the 17th Conference of the Euro- pean Chapter of the Association for Computational Linguistics, pages 1802-1817, Dubrovnik, Croatia. Association for Computational Linguisties.
Jia Peng Lim and Hady Lauw, 2023, Large-scale corre- lation analysis of automated metrics for topic models. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 1877--1898,, Toronto, Canada. Association for Computational Linguistics.
Yaru Hao, Li Dong, Furu Wei, and Ke Xu. 2021. Self- attention attribution: Interpreting information interac- tions inside transformer. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pages 12963-12971.
Kar Wai Lim and Wray L. Buntine. 2014. Bibliographic analysis with the citation network topic model. In ACML, volume 39 of JMLR Workshop and Confer- ence Proceedings.
Qi Liu, Maximilian Nickel, and Douwe Kiela. 2019. Hyperbolic graph neural networks. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Tom Henighan, Shan Carter, Tristan Hume, Nelson Elhage, Robert Lasenby, Stanislav Fort, Nicholas Schiefer, and Christopher Olah. 2023. Superposition, memorization, and double descent. Transformer Cir- cuits Thread.
Yishu Miao, Lei Yu, and Phil Blunsom. 2016. Neu- ral variational inference for text processing. In Pro- ceedings of The 33rd International Conference Machine Learning, volume 48 of Proceedings of Ma- chine Learning Research, pages 1727-1736, New York, New York, USA.
Alexander Hoyle, Pranay Goel, Denis Peskov, An- drew Hian-Cheong Jordan Boyd-Graber, and Philip Resnik. 2021. Is automated topic model evaluation broken?: The incoherence of coherence. In Newral Information Processing Systems.
Richard M. Karp. 1972. Reducibility among Combina- torial Problems. pages 85-103. Springer US, Boston, MA.
Beren Millidge and Sid Black. 2022. The singular value decompositions transformer weight matrices are highly interpretable.

---

David Mimno, Hanna M. Wallach, Edmund Talley, Miriam Leenders, and Andrew McCallum. 2011. Op- timizing semantic coherence in topic models. In Proceedings of the Conference on Empirical Meth- ods in Natural Language Processing, EMNLP 1I, page 262-272, USA. Association for Computational Linguisties.
Silvia Terragni, Elisabetta Fersini, Bruno Giovanni Galuzzi, Pietro Tropeano, and Antonio Candelieri. 2021. OCTIS: Comparing and optimizing topic mod- els is simple! In Proceedings of the 16th Confer- ence of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 263-270, Online. Association for Computa- tional Linguistics.
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. 2020. Zoom in: An introduction to circuits. Distill. Https:/distill https///issillllub2222iiiii pub/2022/errreuis//oommiii
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LJama: Open and efficient foundation language models.
Shirui Pan, Jia Wu, Xingquan Zhu, Chengqi Zhang, and Yang Wang. 2016. Tri-party deep network represen- tation. In IJCAI, pages 1895-1901.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L. ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Kyungchul Park, Kyungsik Lee, and Sungsoo Park. 1996. An extended formulation approach to the edge- weighted maximal clique problem. European Jour- nal of Operational Research, 95(3):671-682.
Dang Van Pham and Tuan M.
V. Le. 2021. Neural topic models for hierarchical topic detection and visualiza- tion. In ECML/KKDD.
Elena Voita, David Talbot, Fedor Moiseev, Rico Sen- nrich, and Ivan Titov, 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lift- ing, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguisties, pages 5797-5808, Florence, Italy. Association for Computational Linguistics.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natu- ral Language Processing (EMNLP-JJCNLP), pages 3982-3992, Hong Kong, China. Association for Com- putational Linguisties.
Xinyi Wang. Wanrong Zhu, and William Yang Wang. 2023. Large language models are implicitly topic models: Explaining and finding good demon- strations for in-context learning. arXiv preprint arXiv.230..11916
Liang Yang. Fan Wu, Junhua Gu. Chuan Wang. Xi- aochun Cao, Di Jin, and Yuanfang Guo. 2020. Graph attention topic modeling network. In Proceedings of The Web Conference 2020, wWW '20, page 144-154, New York, NY, USA, Association for Com- puting Machinery.
Michael Röder, Andreas Both, and Alexander Hinneb- urg. 2015. Exploring the space of topic coherence measures. In WSDM, pages 399-408.
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. 2023. Are emergent abilities of large language mod- els a mirage?
Ce Zhang and Hady W. Lauw. 2020. Topic modeling document networks with adjacent-encoder. Proceed- ings of the AAAI Conference on Artificial Intelligence, 34(04):6737-677) 67745.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 1725. Berlin, Germany. Association for Computational Lin- guistics.
Delvin Ce Zhang, Rex Ying, and Hady W. Lauw. 2023. Hyperbolic graph topie modeling network with con- tinuously updated topic tree. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Dis- covery and Data Mining, KDD '23, page 3206-3216, New York, NY, USA. Association for Computing Machinery.
Noam Shazeer, Zhenzhong Lan, Youlong Cheng. Nan Ding, and Le Hou. 2020. Talking- heads attention.
Linhai Zhang, Xuemeng Hu, Boyu Wang, Deyu Zhou, Qian- Wen Zhang, and Yunbo Cao. 2022. Pre-training and fine-tuning neural topic model: A simple yet ef- fective approach to incorporating external knowledge. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume I: Long Papers), pages 5980-5989, Dublin, Ireland. As- sociation for Computational Linguistics.
Akash Srivastava and Charles Sutton. 2017. Autoencod- ing variational inference for topic models. In ICLR (Poster).
Jie Tang, Jing Zhang, Limin Yao, Juanzi Li, Li Zhang, and Zhong Su. 2008 Ametminer: extraction and mining of academic social networks. In KDD, pages 990-998.

---

A Supplementary Information
Corpus
edgee
Sub-graph size
Algorithm Words. We consider each vertex and its neighbours as a star. Since G is a complete graph, each star will be similar to other stars. To in- crease variation among stars, we retain edges with Wo,,y [Ot, Ou), deleting all other edges. Next, Wy, greedily select stars with unchosen centres and their remain- ing unchosen vertices with largest Wa fulfilling set size constraints [KV, Ka]- After reducing to smaller sub-gr graphs of size 24, now computa- tionally feasible to solve these sub-graphs exactly.
20NewsGroou
[-0.6, 1.0]
BBC-News
1-0.6, 1.01
DBLP
[-0.1. 0.3]
32
M10
[-0.1, 0.31
Table 9: Hyper-parameters used in heuristic pruning of NPMI graph for ZSTM.
prioritising stars with larger
Voting Scheme. We give each token ten votes per layer, awarded to the highest activated neuron. Neurons that exceed the vote count threshold will be examined, with the neuron acting the origin. The vote count threshold is set relative corpus size, at 25% for 20NewsGroup and BBC-News, and 1 % for M10 and DBLP.
A.1 Sapee-Parameters for GPT-2
For shortlisting of token pools, we set token pool size-parameter т 900 for fruth obtained from GPT-2. In our heuristic step, we keep edges with W,,,, that lies between f [0.1, Word-set size constraints [10, 24]. GPT-2's tokens can be mapped directly onto Wikipedia's V.
Greedy Algorithm. From the extracted topics, we choose K best topics greedily using heuristics prioritising for NPMIc and modulated by c, which represents the maximum similar words between each topic. We search 1 S S
3 and select the
A.2 Zero-Shot Topic Modelling Details
most competitive.
Corpus size. Table details the corpora statistics. These corpora may be in the large datasets, such as CommonCrawl, used in the pre-training of PI.Ms. However, selecting these small corpora would, at best, constitute a tiny percentage of the total tokens used to train these PLMs. Furthermore, some of our neural topic model baselines will also utilise PLM embeddings. When possible, we try to mitigate the influence of rare word pairs, based on token count and corpus size, by setting the minimum count of word pairs for NPMIC calculation for M10, DBLP, 20NuusGroup and BBC-News to O,
Corpus
K
ProdLDA-0
CTM-5
GPT-2
20NewsCroup
20
2
BBC-News
20
2
2
DBLP
20
2
3
2
MI0
20
3
2
20NewsCroou
3
BBC-News
3
2
Table 10: Hyper-paramest for greedy-algo.
A.3 Additional I.LMMA Details
Hy pypeerparamettrr. To shortlist token pools, we set size-parameter T = 900 for frath obtained from LLaMA. In our heuristic step, we keep edges with Weey; that lies between 0 [0.025, 1]. Word-set size constrained ripos [10, 32]. Kneg [10, 24].
5, 50, 50 respectively.
Corpus
Does]
Tokensi
[V]
eWiki (%)
20NewsCrroup
16,309
783,151
1,612
1,582 (98.1)
BBC-News
2,225
267,259
2,949
2,892 198.1)
MI0
8,355
49,385
1,696
1,613 (95.1)
A larger upper-limit reduces number of word-sets generated.
DBLP
54,595
294,757
1,513
1,394 (92.1)
Word Explosion. LLaMA's pool can recon- struct a larger number of words from a smaller set of tokens. To prevent this scenario, we automat- ically reduce т, 50 at time, until the word pools generated have less than 900 words.
Table 8: Numerical descriptions of corpora.
Hyper-Parameters. For GPT-2, we set 7 900. Table 9 contains the 0 paper-parrmeter for edge-weights. 6/ have to be reduced below 0 to produce stars sub-graphs. For M10 and DBLP, their small document size and rare word pairs strongly influence their NPMIc. To mitigate this, we reduce fg. We set minimum word-pair count that will retain a majority of words such that they contain at least a word pair.

---

B Supplementary Tables
Words/Word Pool
Word Sets/Word Pool
Words/Wont Set
modes
pos
neg
pos
neg
pos
neg
S-Hi-C
388.1 122.1
216.680
87.9
6.90 + 5.43
2.07 1.89
15.19 + 4.84
12.82
3.30
S-Hi-l
480.0 = 104.9
229.119 87.0
9.86 5.79
#.22
14.37 + 4.26
12.65 # 3.21
S-Lo-C
326.4 125,2
228.968 86.1
5.92 8.11
1.70
15.10
12.49 + 2,91
S-Lo-I
437.3 113.1
336.300 114.1
3.56
14.47 4.37
12.74 3.14
M-Hi-C
357.2 = 98.4
262.420 = 94.0
2.69
14.72 - 4.66
12.74 + 3,22
M-Hi-I
510.5 = 103.2
338.652
93.3
5..23
3.87
14.41 4.25
12.93 # 3.30
M-Lo-C
346.0 + 113.5
268.337
101.8
5.61
4.44
2.01
14.65 4.55
12.27 + 2.73
M-Lo-I
454.9 = 95.9
383.143 99.5
6.79 4.43
4.87
13.81 3.98
12.96 + 3.29
L-Hi-C
409.9 114.6
267.437
82.1
8.72
3.08
14.43
13.76 + 4.09
L-Hi-I
473.7 = 88.8
211.120 # 110.7
9.27
5.37
14,40 4.50
14.10 + 4.19
L-Lo-C
408.4 94.7
248.464
75.9
7.79
2.31
14.43
13.14
L-Lo-I
449.7 = 92.2
340.385
96.9
6.997
3..18
13.94
12.89 + 3.27
XL-Hi-C
473.0 129.8
250.104 = 83.9
5.229
3.08 4.09
14.88 4.55
13.66 + 4.00
XL-Hi-I
478.2 = 96.0
244.775 106.4
3.773
3.110
14.60 4.58
14.00 + 4.17
XL-Lo-C
466.4 = 102.6
244.433 79.7
10.02 5.06
2,48 3364
14.78 4.51
13.44 + 3.96
XL-Lo-I
450.4 89.2
351.506 = 106.3
6.88 + 4.10
3,99 3.55
13.76 + 3.94
13.01 = 3.42
Random
559.2 = 14.2
5.77 + 1.87
12.04 + 2.37
Table II: Sampling statistics from various paths of GPT-2. All neurons of respective modes produced non-empty word pools. We conduct the Mann- Hann-WWitney Whitney test between the random baseline and non-random methods on the number of word sets/neuron, and their differences significant with p 0.001.
Mean num. Topics
Top-1 Topic (Topicul Pool) NPMIw score
Mean NPMIw score
Total num. of topics
& of Abstract Pool
per Topical Pool
modes
pos
neg
pos
neg
pos
neg
pos
neg
pos
neg
both
S-Hi-C
6.76
2.08
0,202 +: 0.06
0,216
0.09
0.130
0.164
211,612
22, 106
7.5
35.6
6.0
S-Hi-I
9.76
2.19
0,218 = 0.06
0.166 0.08
0.119
0.132
351,968
24.634
1.1
34.8
0.6
S-Lo-C
5.68
0.215 0.07
0.199 0.09
0.142
0.162
40,865
4,725
11.0
34,9
7.4
S-Lo-I
8.10
3.56
0.227 0.07
0.191 0.08
0.122
0.117
70,502
21,913
2.8
16.6
1.2
M-Hi-C
4.89
2,70
0.192 + 0.06
0.184 0.08
0.128
0.133
405,662
88,914
7.8
33.2
4.2
M-Hi-I
10.84
3.80
0.232 + 0.06
0.162 0.06
0.119
0.108
1,048,59,
270,000
0.8
13.8
0.4
M-Lo-C
2.12
0.209
0.169
0.134
0.122
108,242
16,967
9.6
33.7
6.0
M-Lo-I
6.80
4.87
0.216 0.07
0.190
0.114
0.111
158,952
97,378
2.4
9.3
0.3
L-Hi-C
8.26
3.04
0,221 = 0.06
0.165 + 0.08
0.130
0.126
1405,522
214,319
3.9
30.9
1.3
L-Hi-I
8.37
4.69
0.188 0.05
0.180 0.06
0.111
0.128
1,469,191
241,072
2.4
36.1
0.2
L-Lo-C
7.76
2.41
0,226 0.06
0.164
0.136
0.126
338,917
39,909
2.6
32.1
1.7
LALo-
6.66
3.96
0.208 + 0.07
0.186
0.111
0.114
291,240
133,769
2.6
0.4
XL-Hi-C
10.88
2.98
0.232 0.07
0.160
0.127
0.126
3,135,250
291,738
3.1
34.0
0.6
XL-Hi-I
9.12
3.78
0.199 + 0.05
0.160 0.06
0.117
0.118
2,665,849
535,733
2.4
26.9
0.0
XL-Lo-C
2.47
0,227 = 0.07
00.07
0.128
0.126
712,337
59,817
1.8
34.2
0.9
XL-Lo-
0.212 0.06
0.182 0.07
0.111
0.113
508,179
217,321
1.9
0.3
Random
5.77
0.097 + 0.03
0.029
261.949
Table 12: Exploratory results from mining topics in GPT-2 with "stop topics" removed. We conduct the Mann-
Whitney U test between random baseline and non-random methods on top-1 NPMIw scores. Differences between GPT-2-sampled and randomly-sampled for both metrics are significant with p 0.001. Abstract neurons are neurons of respective paths that have no projected topics.

---

Model
NPMle
NPMIw
TU
(%)
Model
NPMIc
NPMIw
TU
(%)
ProdLDA
-0.135
0.021
0.750
ProdLDA
-0.131
0.026
0.667
ProdLDA-8
0.018
0.061
0.796
ProdLDA-8
-0.108
0.044
0.768
CTM
-0,100
0.044
0.825
CTM
-0.118
0.048
0.701
CTM-8
0.048
0.075
0.809
CTM-3
-0.088
0.056
0.767
BERTopic
-0.045
0.064
0.799
BERTopic
GPT-2-S-Lo-C
0.134
0.074
0.870
20.57
GPT-2-S-Lo-C
0.100
0.056
0.852
GPT-2-S-Lo-I
0.151
0.066
0.866
20.57
GPT-2-S-Lo-I
0.111
0.063
0.838
39.56
GPT-2-M-Lo-C
0.156
0.075
0.864
23.73
GPT-2-M-Lo-C
0.111
0.051
0.856
44.71
GPT-2-M-Lo-I
0.155
0.082
0.858
23.73
GPT-2-M-LL-I
0.113
0.057
0.870
44.71
GPT-2-L-Lo-C
0.164
0.077
0.862
14.99
GPT-2-L-Lo-C
0.118
0.062
0.840
28.96
GPT-2-L-La-I
0.155
0.079
0.864
14.99
GPT-2-L-LO-I
0.115
0.056
0.848
28.96
GPT-2-XL-LO-C
0.171
0.076
0.860
15.37
GPT-2-XL-LO-C
0.123
0.058
0.858
31.74
GPT-2-XL-LO-1
0.171
0.090
0.866
15.37
GPT-2-XL-LO-1
0.122
0.071
0.836
31.74
(a) 20NewsCroup.
50
(b) BBC-News, 50. topics all runs.
BERTopic unable to produce
Table 13: Results for topic set sizes K = 50. The best baseline scores for the metric are underlined. Bolded PLM
scores are better than the best baseline. Baseline scores are mean of 10 independent runs with error bars Table 15 in Appendix B. The minimum word co-occurrence count is 50 for NPMIc calculation.
Model
NPMIc
NPMIw
TU
A (%)
Model
NPMIc
NPMIw
TU
A (%)
ProdLDA
-0.059
0.045
0.903
ProdLDA
-0.085
0.027
0.850
ProdLDA-3
0.085
0.075
0.911
ProdLDA-8
-0.069
0.059
0.888
CTM
-0.60
0.052
0.916
CTM
-0.052
0.053
0.828
CTM-8
0.016
0.070
0.945
CTM-8
0.009
0.072
0.818
BERTopic
-0.022
0.069
0.887
BERTopic
-0.036
0.077
0.862
GPT-2-S-Lo-C
0.164
0.123
0,955
20.57
GPT-2-S-Lo-C
0.139
0.119
0.920
39.56
GPT-2-S-Lo-I
0.169
0.115
0.950
20.57
GPT-2-S-Lo-I
0.135
0.108
0.900
39.56
GPT-2-M-Lo--
0.177
0.117
0,965
23.73
GPT-2-M-Lo-C
0.147
0.106
0,915
44.71
GPT-2-M-Lo-I
0.186
0.120
0,920
23.73
GPT-2-M-Lo-1
0.141
0.123
0.920
44.71
GPT-2-L-L--C
0.197
0.115
0.945
14.99
GPT-2-L-LO-C
0.139
0.116
0.920
28.96
GPT-2-L-Lo-I
0.170
0.118
0.965
14.99
GPT-2-L-LO-I
0.147
0.114
0.920
28.96
GPT-2-XL-LL-C
0.184
0.112
0.940
15.37
GPT-2-XL-Lo-C
0.148
0.112
0.900
31.74
GPT-2-XL-LO-I
0.179
0.140
0.955
15.37
GPT-2-XL-L--I 0.143
0.122
0.920
31.74
(a) 20NewsGroup. K 20. Minimum word co-occurence
(b) BBC-News, K = 20. Minimum word co-occurence count
count is 50 for NPM1e calculation.
50 for NPMIe calculation.
Model
NPMIc
NPMIw
TU
A (%)
Model
NPMIc
NPMIw
TU
(%)
ProdIDD
-0.041
0.024
0.897
ProdIDA
-0.028
0.007
0.741
ProdLDA-8
-0.061
0.025
0.886
ProdLDA-8
-0.023
0.014
0.830
CTM
-0.035
0.034
0.863
CTM
0.027
0.050
0.895
CTM-3
-0.001
0.050
0.812
CTM-8
0.067
0.057
0.835
BERTopic
-0.212
-0.024
0.791
BERTopic
-0.049
0.023
0.906
GPT-2-S-Lo-C
0.082
0.7
GPT-2-S-Lo-C
0.135
0.021
32.31
GPT-2-S-Lo-I
0.075
0.038
0.780
28.63
GPT-2-S-Lo-I
0.143
0.043
0.885
32.31
GPT-2-M-Lo-C
0.087
0.035
0.800
43.12
GPT-2-M-LO--
0.156
0.021
0.865
46.52
GPT---M-Lo-I
0.081
0.040
0.800
43.12
GPT-2-M-LO-I
0.148
0.042
0.865
46.52
GPT-2-L-Lo-C
0.091
0.042
0.800
34.18
GPT-2-L-L--C
0.164
0.028
0.855
38.20
GPT-2-L-Lo-I
0.085
0.039
0.845
34.18
GPT-2-L-Lo-I
0.143
0.033
0.855
38.20
GPT-2-XL-LO-CC
0.103
0.033
0.850
38,98
GPT-2-XL-LL-CC
0.180
0.023
0.845
43.39
GPT-2-XL-LO-1
0.088
0.057
0.845
38.98
GPT-2-XI-LL--I
0.159
0.045
0.850
43.39
(c) DBLP, K = 20. Minimum word co-occurence count for NPMIc calculation.
(d) M10, 20. Minimum word co-occurence count is 0 for NPMIc calculation due to low token and document count.
Table 14: Results for topic set sizes K = 20. The best baseline scores for the metric underlined. Bolded PLM scores are better than the best baseline. The mean of independent runs with error bars in Table in Appendix

---

NPM1e
NPMlw
TU
ProdLDA
00.399 0.014
0.045 +: 0.007
0.903 + 0.021
ProdLDA-0
0.085 + 0.021
0.075 + 0.007
0.911 00010
CTM
00.600 a 0.024
0.052 a 0.008
0.916=000 0.012
CTM-3
0.016 0.019
0.070 0.007
0.945 0.007
BERTopic
0.002 0.023
0.069 0.007
0.887 0.012
(a) 20NewsGroup,
20
Neurons/layer
Layers
Total Neurons
NPMIc
NPMIw
TU
GPT-2
ProdLDA
-0.085 = 0.025
0.027 = 0.010
0.850 = 0.035
S-Hi-C
3,072
12
36,864
ProdIDA-1
-0.069 0.018
0.059 +: 0.011
0.888 = 0.016
CTM
-0.052 + 0.014
0.053 = 0.013
0.828 0.041
S-Hi-I
3,072
12
36,864
CTM-8
0.009 + 0.019
0.072 0.006
0818 = 0.015
S-Lo-C
768
12
9,216
BERTopic
0.036
0.077 0.007
0.862 = 0.008
S-Lo-I
768
12
9,216
M-Hi-C
4,096
24
98,304
(b) BBC-News, 20
M-Hi-I
4,096
24
98,304
NPMIc
NPMIw
TU
M-Lo-C
1,024
24,576 24
ProdLDA
-0.01 0.015
0.024 0.004
0.897 0.013
M-Lo-1
1,024
24,576 24
Pr-
0.886 0.009
L-Hi-C
5,120
36
184,320
CTM
0.013
0.034 0.009
0.863 0.019
L-Hi-I
5.120
36
184,320
CTM-8
.0.01 0.011
0.050 0.007
0.812 0.020
BERTopic
-0,212 0.016
-0.024 0.004
0.791 + 0.0188
L-Lo-C
1,280
46,080 36
LAaT
1,280
36
46,080
(c) DBLP, K 20
XL-Hi-C
6,400
48
307,200
NPMIg
NPMIw
TU
XL-Hi-I
6,400
48
307,200
ProdLDA
-0,028 0.024
0.007 0.010
0.741 = 0.027
XL-Lo-C
1,600
48
76,800
Pr
-0.023 0.020
0.014:0.00
0,830 = 0.021
XL-Lo-I
1,600
76,800 48
CTM
0.027 0.013
0.009
0.026
CTM-3
0.067
0.057 0.009
0.015
LLaMA
BERTopic
-0.049 = 0.018
0.023 = 0.012
0.906 = 0.011
7B-Hi-C
11,008
32
352,256
(d) M10, K 20
7B-Hi-I
11.008
32
352,256
7B-Lo-C
4,096
131,072 32
NPMLa
NPMIw
TU
7B-Lo-1
4,096
131,072 32
ProdLDA
00.135 + 0.012
0.021 = 0.005
0.750 = 0.024
13B-Hi-C
13,824
40
552,960
ProdLDA-9
0.018 0.012
0.061 = 0.003
0.796 = 0.010
13B-Hi-I
13,824
40
552,960
CTM
-0,100 0.014
0.044 0.006
0.825 0.022
CTM-3
0.048 0.006
0.075 0.004
0.809 0.007
13B-Lo-C
5.120
40
204,800
BERTopic
0.045
0.064 0.002
0.012
13B-Lo-I
5,120
40
204,800
(e) 20NewsGroup. K 50.
Table 16: Model parameters statistics. For Paths Hi-C and Hi-I, the neurons refer those in the transformer block's s MLP, usually containing more neurons than Paths Lo-C and Lo-I, located the residual stream.
NPMIc
NPMIw
TU
ProdLDA
-0.131 + 0.016
0.026 = 0.006
0.667 = 0.033
ProdLDA-9
.0.108 0.016
0.044 + 0.011
0.768 + 0.009
CTM
0..118 0.025
0.048 0.006
0.701 = 0.051
CTM-3
.0088 0.010
0.056 0.005
0.767 0.008
(0 BBC-News, K = 50. BERTopic could not produce enough topics for all 10 independent run and is excluded.
Table 15: Baseline results with error bars. Underline scores are best within metric.

---

C
Examples
Words pools are alphabeticalll-ssotedd
Word pool
Word sets
Extracted Topics (NPM)w)
urchbishop bishop
canterbury
lutheran oslo dunish stockholm viking norse norwegian sweden swedish helsinki copen- hagen scandinavian
copenhagen swedish oslo norwegiun helsinki stockholm viking danish scan- dinavian sweden (0.237)
cathodral conversion copenhagen
danish doctrine earthquake
helsinki leicester lutheran mercy norse norwegian norwich orthodox oslo pilgrim pope pre- miership repentance resurrection sacrament salvation scandinavian sins souls stockholm sweden swedish viking worcester (332 ommitted words)
salvation souls doctrine orthodox resurrection mency sins conversion sacrament repentance
salvation souls doctrine repentance sins mercy conversion sacrament resurrcc- tion orthodox (0.179)
archbishop pope worcester premiership earth- quake leicester norwich cathedral pilgrim hishop canterbury
archbishop bishop pope carthquake pilgrim canterbury norwich worcester cathedral leicester (0.098)
Table 17: GPT-2-XL, Path Hi-C, Layer 26 Neuron 5201, neg pool.
Vikings (#1) and Religion (f2, 3).
Word pool
Word sets
Extracted Topics (NPM)w)
answer arry bat beat bet bet- ter hoot certainly client come commund control currectly crowd daily day decided don drive ex- actly february feel feeling flip folks friday fun function fune- tions getting going good got guys install installed know let like little module modules monday month morning need night ought pass people pretty saturday smart sort sunday tell throw throws thurs- day tie tonight toss tossed track- ing mesday understand wednes- day
wednesday saturday tuesday monday month moming fehruary good daily day night thurs- day sunday tonight friday
sunday morning monday wednesday night friday saturday thursday day mues- day (0.331)
don let guys exactly going tell feel ought derstand need answer getting folks better come pretty like certainly people sont feeling fun lit- tee know
let pretty tell going like fun don feel know guys (0.179)
modules boot array client command install in- stalled tracking control smart function func- tions module
module modules client smart control functions install tracking installed array (0.1
correctly crowd drive got decided pass flip hat throws tie beat lossed toss
bat throws drive tossed got crowd pass tie toss (hrow (0.109)
(358 ommitted words)
Table 18: GPT-2-XL, Path Hi-C, Layer 26 Neuron 5201, pos pool. Stop topic (#I), relationshhpp-reltted (#2), software (#3), and baseball/cricket (f4)..
Word pool
Word sets
Extracted Topics (NPM)y)
aluminum chemist Foods mag- nesium mineral oxide phosphate phosphorus salts soda thermal ura- nium vapor (251 ommitted words)
uranium magnesium chemist oxide thermal phosphate phosphorus vapor foods mineral alu- minum soda salts
aluminum mineral magnesium soda phosphate phosphorus uranium salts ox- ide thermal (0.183)
Table GPT-2-XI., Path Lo-C, Layer 2 Neuron 613, neg pool. Some pools only have one topic. Chemistry (#1).

---

Word pool
Word sets
Extracted Topícs (NPM)w)
uccess udam udvertising ulex andy answer app apr ass author ben blog book chapter check com con- tent contents dear dec don down- load editor email facehook fea- tures file google got help hey hit http hitps install jan jason jon josh kill let like love mar me- dia michael net news notes nov online page phil photo picture post posted preview print puh- lished quote read reddit rick said sam sep server source story sub- scribe summary led text thank tom trump twitter update updated user version video watch web wow written www yes (213 ommitted words)
dec com web nov jan www sep mur hilps hilp apr
www apt jun dec sep hitps nov http mar com (0.317)
page twitter google posted reddit app media post email blog user photo online update con-
twitter page content reddit online lace-
book posted user app google (0.246)
tent video news advertising watch download preview updated trump facebook
tom ben phil andy adam michael alex ted jason rick help kill jon josh
phil michael sam andy kill alex josh ben jason tom (0.199)
chapter read story contents quote notes written author picture published text summary editor hook
contents chapter read written published author story summary text book (0.196)
hit let wow got answer love like yes ass suid hey dear thank don
like let dear got thank answer said don love yes (0.131)
uccess install check source net features sub- scribe print version file server
nel access source version check install file features server print (0.070)
Table 20: GPT-2-XI,, Path Lo-C, Layer 2 Neuron 613, pos pool. Stop topic (#1), due to "months", ", social media (#2), conversational (#3), names (#4), story (#5), and software (#6).
Word pool
Word sets
Extracted Topics (NPMIw)
ubsolutely access allowing amaz-
don mayhe know lot definitely said pretty bit icel going hopefully kind absolutely sort sure wonderful fun good awesome certainly yes got like think
lot bit definitely think hopefully mayhe fun don feel know (0.218)
ing ancient app available awe-
some basic beat beautiful best better bit blood casting certainly character common convenient crazy critical definitely difficult digital direct don dream eas- ily easy effective experience ex- treme extremely feel feeling fi- nal free friends fun going good got happy heart highly hit hope- fully hot incredible information intriguing jump kind know like
hit pop crazy spending extremely easy stuff version single number digital sweet reach hot
easy pop sweet stuff hot single version crazy hit number (0.122)
character truly look dream incredible stunning thought perfect happy casting totally friends sad amazing
perfect truly totally sad character amaz- ing happy thought look incredible (0.105)
team lost jump second easily final beat times super best won
beat jump won super best lost times team second final (0.096)
little look lost lot maybe be memory negative numher pain perfect per- formance pop positive precious pretty quick rare reach resource resources rich role sad safe said second single sort spending stuff stunning success super sure sweet team thing things think thinking thought times totally touch trea- sure truly useful usually valuable version virtual won wonderful writing yes
available basic allowing information memory virtual quick better difficult free convenient direct safe app access
app access basic available allowing free direct information memory virtual (0.093)
little blood resource resources valuable thing things rich beautiful ancient rate treasure precious
ancient valuable treasure little rare pre- cious rich beautiful thing things (0.090)
highly pusitive writing negative thinking suc- cess performance rolc intriguing critical
writing performance positive negutive intriguing critical role highly success thinking (0.08
experience extreme feeling effective useful usu- ally common touch heart pain
heart experience useful effective com- mon touch extreme pain feeling usually (0.053)
(423 ommitted words)
Table 21: GPT-2-XL, Path Hi-l, Layer 26 Neuron 5201, pos pool. Conversational (#1), pop music (#2), story-related (#3),, sports (#4), software-related (#5), exploration (#6), art-related (#7), and medical-related (#8).
Word pool
Word sets
Extracted Topics (NPMIw)
air alert black broad carbon ey- cle dam dark deck deep electric electrical energy gas global heat height high higher horizontal iron light long low narrow negative oxygen painted power range red reduced steel steep structural sup- ply upe thick tight water white wide zero
gas low dam electric electrical energy supply heat water power
supply electric electrical energy heat dam water gas low power (6.1
high long broad steep narrow deck thick tight height range wide
narmow range high thick deck steep long wide broad height (0.146)
higher reduced glohal oxygen cycle steel zero air negative structural iron carbon
oxygen higher structural iron carbon steel negative cycle reduced zero (0.091)
tape white horizontal painted dark deep alert light black red
black white red light tape alert horizon- tal deep dark painted (0.080)
(322 ommitted words)
Table 22: GPT-2-XI., Path Lo-I, Layer 38 Neuron 556, pos pool. Power generation (#1), Visuals (#2, 4), and chemistry (#3).

---

Word pool
Word sets
Extracted Topics (NPM)))
angel birds breeding canary cats champugne coastline crop dums deities demon demons descend devils dogs dragons drought eggs farmers flora glaciers grapes growers guide horses hunt lake lakes land landscapes lion lions maize mill peaks plateau pond ponds reptless river rivers roses south southern spirits strawher- ries tails tasmania trout valleys vampires waters watershed wind wolves
waters watershed tasmania trout pond ponds drought lake valleys lakes river rivers mill dams
lakes river dams pond ponds rivers wa-
ters watershed lake trout (0.321)
south southern wind coastline land flora land- scapes peaks plateau glaciers
southern flora coastline wind landscupes plateau land peaks south glaciers (0. (0.109)
crop breeding farmers maize grapes eggs grow- ers roses champagne strawberries
breeding eggs straw berries champugne roses maize grapes crop farmers growers (0.065)
spirits dragons hunt vampires deities demon devils angel demons
devils cats hunt spirits dragons vampires deities angel demon demons (0.044)
tails horses wolves reptiles guide canary de- scend birds lion lions dogs
tails guide lion lions horses reptiles birds dogs descend wolves (0.021)
(310 ommitted words)
Table 23: GPT-2-XL, Path Lo-I, Layer 13 Neuron 1126, neg pool. Nature (#1, 2), agriculture (#3), magical beasts (#4), and animals (45).
Word pool
Word sets
Extracted Topics (NPM)y)
acceleration actors alive allow al- lows angry arrow atoms break breath broke calculations camera case com combine computation connection connections contin- ued convenient cop crash crashed crowd deliver denotes depen- dence depth detail details distance editing entropy estimation explain explained explanation facing fast figure flash formula function functions glass ground hulf heavy hooked http hitps hundreds icons identify including inserting large lead leads leave libraries like like- wise manipulate masses mechan- ical media met miles minus net object objected objective official pad pages particular pusses pass- pause periods play played pos- sible pressed privileges programs promises provides recursive sav-
com connection connections detail details http libraries media official pages today upload user users vidcos watch https
pages com www user users http watch https upload videos (0.230)
allow allows case computation depth timation formula function functions leads like object particular possible programs pro- vides showed shown starting step words zero recursive
recursive formula estimation computa- object allows function step func- tions zero (0.146)
broke crashed crowd distance facing ground half hooked hundreds including lead miles mi- nus passes passing play played shut thirty yards
distance ground lead crowd broke miles half passes passing yards (0.087)
arrow atoms calculations denotes dependence explain explained explanation glass iden- tify large mechanical typical understanding entrupy
entropy mechanical typical atoms iden- tify calculations explain explained expla- nation understanding (0.074)
hreak breath continued fast leave objective peri- ods pressed skip slight stating temporury pause
periods temporary pause pressed slight stating fast leave break continued (0.022)
ulive cop met objected promises talk vietim violent wanting angry
met talk vietim violent promises wanting alive objected cop angry (0.010)
ing scripted showed shown shut skip slight starting stating step talk temporary terrain thirty today typical understanding upload user victim videos violent want- ing watch words www yards zero (111 ommitted words)
crash deliver fall figure flush heavy likewise masses pad terrain acceleration
terrain heavy deliver crash likewise fig- ure flash 0.033)
fall acceleration (-
actors camera combine convenient icons in- serting manipulate privileges suving scripted editing
convenient actors camera scripted insert- ing combine manipulate saving icons editing (-0.141)
Table 24: LLaMA-13B 13B, Path Hi-C, Layer 3 Neuron 224, pos pool. Websites (#1), mathematics (#2, 3), movement (#5), police-interaction (#6), physics-related (#7) and acting-related (48).

---

Word pool
Word
Extracted Topics (NPM)w)
#
bed chemist class dsc education eng geology level life management mas- ter mining msc physicist program psychology school schooling sci- ences social sport student students subsequently training bsc
sciences program education bsc psy- chology student students msc school master (0.176)
access action umity upplication av- ery background bail bed belly block bone break brig bsc capped cause chi charge chase chemist citi class code color con connected connec- tion conning context control custom digital distinct diving dsc dust edit- ing education end eng eps extension fashion fhi fins force forensic four- teenth frames fuse geary gen gen- eral genes geology gold hand hand- haryana havana hex high inning ips issued jail james jane kane kits lane lens lets level lihrary licutenant life like line lobe lobed location lone Ips macy main management mary master mastered mastering middle midline mining model mold molded molding mounted mse music neu- ral occupied offense office offline outing parking physicist pits play power practice pre primary probe program protection psychology pub- licity quality rare realistic recording refit resin ret retail retailers room ro- lation runs scar school schooling sciences scratch scratched scratches screen security sent server service shape size small social space special sport sporting start starts store string structure student students style sub- sequently suffered support surprise system target targeted targeting test tls toes tooth toothed tops training versus video videos visited walk wall web wine witness witnesses wounded
chase fashion location occupied parking retail retailers space sport- ing store larget macy
retail retailer location space occu- pied macy fashion parking store get (0.129)
2
color diving fins like lobe middle shape size small structure style toes toothed lobed
like toothed size shape lobe color lobed fins middle small (0.120)
3
access application extension library security server support system web tss
library access extension support ap- plication tls security system web server (0.1
4
action background bail charge con- nection fbi forensic haryana high ips issued jail jane office probe scam sent special witness witnesses cbi
scum buil charge jail special connec- tion probe fbi sent cbi (0.108)
5
avery break capped citi end four- teenth frames lone offense outing play recording rotation start starts versus walk inning
play inning start starts lone walk recording break end offense (0.063)
6
block custom gold kits lens model mold molding resin molded
molded lens molding gold kits block custom model resin mold (0.062)
7
context handful Ips music power rare string video videos eps
music lps string rare context eps handful video videos power (0.034)
cause digital dust hand pits protec- tion quality scratch scratched screen suffered scratches
quality hand scratch scratches dust cause screen digital protection suf- fered (0.033)
belly bone connected distinct fuse neural runs tooth wall midline
fuse bone tooth belly midline neural runs connected wall distinct (0.026)
10
con control main mounted prac- tice primary refit room service tops wounded conning
comning mounted wounded tops con- trol room refit primary main service (0.016)
11
code force gen general genes lieu- tenant line pre test ret
pre ret lieutenant test code force gen genes line general (-0.006)
12
(486 ommitted words)
amity geary havana james kane lane mary surprise visited wine brig
visited james mary brig surprise kane lane wine amity havana (- 0.098)
hex lets mastered mastering offline publicity realistic targeted targeting editing
targeted targeting editing lets hex publicity realistic offline mastered mastering (-0.197)
14
Table 25: LLaMA-13B, Path Lo-C, Layer 36 Neuron 592, pos pool. Education (#1), retail (#2), body- parts-related (#3, 6, 10), software (44), law & order (#5), cricket (#6), sculpting (47), music videos (#8), surface-damage (49), and mixed (#11, 12, 13, 14).

---

Word pool
Word sets
Extracted Topics (NPM)r>
#
concurrent devices displays dual gates generates google internally launched micro mouse platforms power refresh rohot sap smart systems usage virtual processor
devices systems usage robot pro- cessor displays virtual smart google platforms (0.114)
annotations audi authority avoided balls beat book books catch city comments concurrent devices displays dropped dual eventually fill filled finish finished log functions gates gene generates goals gold google institutions internally jurisdiction launched majority micro mouse multiple municipal municipali- ties orange outer overall overlap patch plants platforms population power pre- sented processor purposes refresh robot rust sant sap second slope smart span species starting surprising systems thick tool transparent trup usage variance virtual walls works
audi avoided balls beat catch dropped eventually finish goals gold overall sec- ond starting suppising finished
beat goals overall finish finished balls dropped starting sccond catch (0.083)
2
authority city functions institutions ju- risdiction majority municipal population purpuses sant municipalities
jurisdiction purposes city insti- tutions functions municipal mu- nicipalities authority majority population (0.081)
3
fill filled log hid orange outer lup patch plants slope span species transparent trap walls thick
fill filled plants species walls transparent thick outer patch ange (0.072)
4
(91 ommitted words)
book books comments gene multiple presented tool uses variance works annotations
tool books annotations uses works comments presented gene multiple bnok (0.031)
54
Table 26: LLaMA-13B, Path Lo-I, Layer 30 Neuron 704, pos pool. Internet-of-things (#1), competition (#2), government (#3), nature (f4),, and literature (#f 5).
Word pool
Word sets
Extracted Topics (NPM)w)
active aspect dual literally marker stutic suffix syntax term type word verb
syntax suffix aspect term type marker verb literally word dual (0.099)
active anonymous aspect catalog com- ment conservation description dual funds header landscape launch listing lit- crally marker mole muscum names pet photos practical profile profiles recom- mend recommended reliable repository situ society static suffix syntax technical term type verh wehsite websites word (90 ommitted words)
anonymous catalog comment header launch listing names photos profile pro- files recommend reliable repository tech- nical website websites
profile profiles photos anony- mous website websites technical comment names launch (0.076)
2
description funds landscape male mu- seum pet practical recommended situ ciety conservation
funds society description pet landscape conservation situ practical recommended (-0.039)
3
Table 27: LLaMA-7B, Path Lo-C, Layer 30 Neuron 460, pos pool. Linguistics (#1) and archival (#2, 3).
Word pool
Word sets
Extracted Topics (NPMIw)
black blackish embedded forth fresh jet modified paired portion short small tail tall fins
black paired tall portion short hlackish modificd fins tail small (0.089)
1
acid announced basically better black blackish blog cease clean comment com- pact controls cut embedded fins flows forth fraction fresh furnace hours jet kitchen laid load love model modified paired pipe portion posted press product publication random randomly reuse said select short shortly small smart subset tail tall think tweet waste (102 ommitted words)
announced blog comment hours love posted press publication shortly think tweet
publication tweet comment an- nounced press love blog think posted shortly (0.069)
2
acid compact controls fraction model product random randomly said select sum subset
product fraction said model se- lect compact random sum subset randomly (0.046)
3
basically better cease clean flows fur- nace kitchen laid load pipe reuse smart waste
load kitchen waste better pipe clean cut furnace flows (0.025)
Table 28: LLaMA-7B, Path Lo-C, Layer 2 Neuron 460, pos pool. Fish-related (#1), social-media (#2), computation (#3), and plumbing (#4).


Reviews:
[{"review_id": "g5wm145BUx", "reviewer": "Reviewer_Y4Bt", "summary": "", "questions": "A. The zero shot topic modeling method proposed in the paper is attractive. But how does the projection mechanism work when some words in the corpus cannot match the tokens of the large language model?\nB. What are the main advantages of this method compared with traditional neural topic models which leverage the embedding of large language models for topic modeling,", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "R1: We will use the extra page to improve on this.\n\nQA: In Section 8, we demonstrate this use case. Roughly 90% of LLaMA's 32K tokens are sub-words, and many words from our Wiki corpus do not exist in LLaMA's token space. We empirically demonstrate that we can treat words as a \"topic\" representation of tokens (sub-words), forming word pools to extract coherent topics.\n\nQB: Traditional neural topic models (with LLM embeddings) \"explain\" corpus, whereas our approach seeks to interpret and examine LLMs in a white-box setting. We are unaware of any neural topic models for extremely large corpus since the resources required are logically better used on general-purpose LLMs. Our proposed approach examines LLM's knowledge, which is pre-trained on these billions (or trillion) token corpus, and presents its analysis in a human-coherent manner. In doing so, we are applying topic modeling evaluation methodology in the field of mechanistic interpretability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The method of deriving topics looks novel to me.\n2. The experimental section of this paper is remarkably comprehensive.", "reasons_to_reject": "1. This paper may be hard to understand by some readers who are not familiar with disentangling superpositions. More preliminary introductions will make the paper easier to follow.", "questions_to_authors": "A. The zero shot topic modeling method proposed in the paper is attractive. But how does the projection mechanism work when some words in the corpus cannot match the tokens of the large language model?\nB. What are the main advantages of this method compared with traditional neural topic models which leverage the embedding of large language models for topic modeling,", "paper_topic_and_main_contributions": "This paper offers a novel perspective to derive meaningful topics from transformer language models.\nThis paper proposes a weight-based, model-agnostic and corpus-agnostic approach to search and disentangle decoder-only TLM, mapping individual neurons to multiple coherent topics.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "W4ML1QWuFj", "reviewer": "Reviewer_AWCb", "summary": "", "questions": "Question A: Is your method available for large language models such as GPT3 ?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "R1: In Sections 6 and 8, our Wiki corpus has a billion tokens (5M documents, 40K vocabulary). Our Python code produces the word graphs for our Wiki corpus in 6 hours using CPU AMD EPYC 7502 @ 2.50GHz utilizing 40 cores. A similar test on a larger corpus with 1.5 billion tokens (1M documents, 40K vocabulary) produces its word graph in 7 hours. The theoretical complexity is O(Num. docs x (vocab. in doc.)$^2$). We will release our implementation code. \n\nWiki is the largest corpus that have been used for coherence measure. No larger corpus has been benchmarked and shown better correlation to human judgment.\n\nIn Section 7, the purpose of the zero-shot topic modeling task is to verify whether we can obtain a set of relevant coherent topics from TLM's top activated neurons when we use the corpus as the input. This further validates the link between the neuron and its extracted superposed topics. We opted for smaller datasets for this validation, as the corpora are well understood. In practice, since this is a validation step, we do not need to pass the corpus into the TLM to identify the top activated neurons for topic extraction.\n\nQA: Our approach should work on any TLM, provided the user can access the model's weights. Producing the logit distribution of each neuron has a similar complexity to next token generation. We expect GPT3 to have many more neurons, requiring more resources to evaluate the interpretability of the model. GPT3 is still closed-source at the time of writing, if it is publicly available, a similar analysis could be conducted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The interpretation of TLM as superposed NTM is novel.\n* The three step disentanglement process is clear: the project step is used to get corpus statistics, the shortlisting step used some heuristics to create smaller graphs and the exact-solving step locate the subgraphs and return coherent topics.\n* Experiments using widely used language models such as GPT2 and Llama are impressive. ", "reasons_to_reject": "* Not sure whether this method is scalable, as I see the old small data sets for topic modelling (e.g. 20 Newsgroup), it would be better to analyse the computation complexity when the word graph is huge. ", "questions_to_authors": "Question A: Is your method available for large language models such as GPT3 ?", "paper_topic_and_main_contributions": "This paper proposed a weight based, model agnostic and corpus-agnostic method to search and disentangle decoder based Transformer Language Models, the method convert the topic disentanglement problem into a graph optimisation task. Experiments on GPT2 and Llama shows the effectiveness of this method. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "o59MIdeWxw", "reviewer": "Reviewer_K1Jc", "summary": "", "questions": "- How does the model compare to the other benchmark models in terms of embedding coherence (see e.g. Octis)\n- How does it compare to the other models in terms of topic diversity\n\n- See comment from above", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the feedback. We will work to improve our communication of the problem and our approach.\n\nWe would like to put in a clarification that in the field of topic modeling, the subject of interest is the corpus, models/approaches are developed to analyze the corpus. Whereas in the **mechanistic interpretability** field, the subject of interest is the model (LLM/TLM), and we develop approaches to analyze LLMs to increase our white-box understanding of the inner workings of LLMs. The objective of our work is to interpret and investigate TLMs at the neuron level for explanability. We use the corpus as a tool to constrain our scope of investigation within the TLM. Essentially, our work applies topic modeling evaluation methodology to the mechanistic interpretability field.\n\nThere are many challenges in the LLM mechanistic interpretability field, and our work attempts to reduce the barriers (some listed here for convenience):\n\n(1) Current neuron-level interpretation of LLMs is limited. Usually, one neuron is mapped to a concept (represented as a topic) on the token space, and interpreting each neuron for human understanding in an LLM is challenging. We show an empirical approach to map one neuron to many human-coherent concepts.\n\n(2) Most LLM interpretability work is done on GPT2, due to the challenges of working with LLaMA's tokens. We propose a viable option on how to advance the field on LLaMA's token space.\n\n(3) Many existing interpretation studies are not quantifiable at a large scale and require similarly large user studies, which only well-funded institutes can conduct at scale. We apply existing topic modeling evaluation methodology to show that automation to measure interpretability can be achieved, lowering barriers to entry.\n\n**On the involvement of LLM:**\nWe have to involve LLM as we are trying to investigate LLM.\n\n**On creating word pools:**\nAs our subject of interest is the LLM, the word pools are created from the logit distribution of the individual neurons. The corpus is a tool to restrict the scope of the investigation. By projecting to the corpus, we are analysing if the behaviour of the neuron of the LLM emulate similar information found in the corpus.\n\n**On hierarchical clustering:**\nWe are aware of the methods. We considered it and decided against it due to three reasons:\n\ni. It adds another layer of complexity to the approach, with additional consideration to which algorithm, distance function, and features to use.\n\nii. The size of word pools will vary as LLaMA's word pools do not have a fixed size.\n\niii. The size and number of word sets will also vary between word pools. The coherent topics extracted might not reflect the word pool due to the presence of outliers in the clustering process.\n\n**On topic diversity:**\nFor section 7, we evaluated Topic Diversity as TU, the proportion of unique words in the set of Topics, and it is in the paper. Our results show topic sets extracted from LLM are comparable or better.\n\nFor sections 6 and 8, since the number of topics generated far exceeds the vocabulary size of Wiki corpus, evaluating TU is not necessary. With regards to possible duplicate topics, we remove topics with words that occur very frequently across the entire set of topics. Otherwise, it is acceptable when different neurons in the LLM express similar topics.\n\n**On the fairness of using NPMI:**\nWe are evaluating NPMI as the tool to extract human-coherent interpretations from LLMs instead of using NPMI to evaluate our approach.\n\nOur reasons for using NPMI is rooted in Wiki's NPMI$_W$'s correlation with human judgment (line 140-150), and we wish to analyze the LLM in a human-coherent manner. For sections 6 and 8, we use NPMI$_W$ to extract human-coherent topic representations from LLMs, which we compare to extracted topic representations from a random distribution as the baseline. Our results show that the topics from LLMs are better, suggesting that the interpretation approach is meaningful and not due to chance. \n\nFor section 7, NPMI$_C$ differs from NPMI$_W$, as NPMI$_C$'s correlation with human judgment is unknown. ProdLDA and CTM are augmented with an NPMI$_C$ optimizing variant.\n\n**Zero-shot Topic Modeling (Section 7):**\nZero-shot Topic Modeling is not the goal of this paper. We use this task to evaluate whether the superposed topics derived from the neurons are reflective of their extracted superposed topics, and thus meaningful. Given a corpus, we expect it to activate neurons with similar extracted topics. To validate this hypothesis, we must derive an equivalent or better set of coherent topic representations from the activated LLM neurons. Otherwise, there will be a contradiction if the top activated neurons are unable to produce a set of topics reflective of the corpus.\n\n**On embedding coherence:**\nIt is beyond the scope of our work. Our work focuses on interpreting LLMs at the neuron level and their corresponding final logit distribution. As mentioned in our related works section, there are other research works on LLMs interpretation in the embedding space.\n\n\nOur goal is to extend the currently accepted methodology of evaluating the interpretability of Topic Models to evaluate the interpretability of LLM at the neuron level, lowering barriers to the field of mechanistic interpretability. We hope our clarification and answers have shed additional light and show that our work is not just a zero-shot topic modeling paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Novel idea\n- Generally well written paper\n- Leveraging pre-trained LLMs could highly benefit the area of topic models\n- good experimental results", "reasons_to_reject": "I must admit that I am facing some difficulty in comprehending the concept of \"projecting to a corpus\" despite carefully reviewing the presented materials. It is possible that my understanding of this particular aspect might be influenced by my own perspective or prior knowledge. \n\nRegarding the methodology and interpretation of the results, it seems that the primary contribution of the LLMs in your algorithm lies in the creation of the word pool. However, creating great NPMI-topics can already be accomplished by solely utilizing word-co-occurrence statistics from the entire corpus and subsequently applying hierarchical clustering to these word-word statistics. I wonder whether the involvement of LLMs may be considered somewhat redundant in this context. I would appreciate it if you could kindly shed some light on whether it is necessary to create the word pools initially or if the graph construction could be achieved without this step. \n\n\n-Using NPMI as a metric used in creating your topics as well as using it to evaluate your models seems unfair.\n-The evaluation is fairly limited. Other metrics should be considered.\n-Document-topic matrix not analyzed/result of algorithm.", "questions_to_authors": "- How does the model compare to the other benchmark models in terms of embedding coherence (see e.g. Octis)\n- How does it compare to the other models in terms of topic diversity\n\n- See comment from above", "paper_topic_and_main_contributions": "-Disentangling Decoder-Only Transformer-Based Language Models (TLM) to extract coherent topics\n- Basically Zero-Shot Topic Modeling on trained large language models", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "should be \"BERTopic\"", "Typos_Grammar_Style_and_Presentation_Improvements": "-Sentence starting in line 052 is not a complete sentence."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a novel approach to topic modeling using decoder-only Transformer-Based Language Models (TLMs). The proposed approach is clear and easy to follow. All reviewers agree on the novelty of the paper's approach which executes the idea well and evaluates it thoroughly, including analysis of quantitative evaluation and extending to LLaMA. Experiments using widely used language models such as GPT2 and Llama are impressive and remarkably comprehensive. From the experimental results, we can find that leveraging pre-trained TLMs could highly benefit the area of topic models.
Decision: Accept-Main