Intepreting & Improving Pretrained Language Models: A Probabilistic Conceptual Approach
Authors: Hengyi Wang
Zhiqing Hong
Desheng Zhang
Hao Wang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Pretrained Lauguage Models
Generative Models
Probabilistic Graphical Models

Abstract: Pretrained Language Models (PLMs) such as BERT and its variants have achieved remarkable success in natural language processing. To date, the interpretability of PLMs has primarily relied on the attention weights in their self-attention layers. However, these attention weights only provide word-level interpretations, failing to capture higher-level structures, and are therefore lacking in readability and intuitiveness. In this paper, we propose a hierarchical Bayesian deep learning model, dubbed continuous latent Dirichlet allocation (CLDA), to go beyond word-level interpretations and provide concept-level interpretations. Our CLDA is compatible with any attention-based PLMs and can work as either (1) an interpreter which interprets model predictions at the concept level without any performance sacrifice or (2) a regulator which is jointly trained with PLMs during finetuning to further improve performance. Experimental results on various benchmark datasets show that our approach can successfully provide conceptual interpretation and performance improvement for state-of-the-art PLMs. 

Under review as a conference paper at ICLR 2023
INTEPRETING & IMPROVING PRETRAINED LANGUAGE MODELS: A PROBABILISTIC CONCEPTUAL APPROACH
Anonymous authors Paper under double-blind review
ABSTRACT
Pretrained Language Models (PLMs) such as BERT and its variants have achieved remarkable success in natural language processing. To date, the interpretability of PLMs has primarily relied on the attention weights in their self-attention layers. However, these attention weights only provide word-level interpretations, failing capture higher-level structures, and are therefore lacking readability and intuitiveness. In this paper, we propose hierarchical Bayesian deep learning model, dubbed continuous latent Dirichlet allocation (CLDA), go beyond word- level interpretations and provide concept-level interpretations. Our CLDA is compatible with any attention-base PLMs and can work as either (1) an interpreter which interprets model predictions at the concept level without any performance sacrifice or (2) a regulator which is jointly trained with PLMs during finetuning to further improve performance. Experimental results various benchmark datasets show that our approach can successfully provide conceptual interpretation and performance improvement for tate-of-thee-rr PLMs.
INTRODUCTION
Pretrained language models (PLMs) such as BERT Devlin et al. (2018) and its variants Lan al. (2019): Liu et al. (2019); He et al. (202T) have achieved remarkable success in natural language processing. These PLMs are usually large attention- based neural networks that follow a pretrain- finetune paradigm, where models are first pretrained on large datasets and then finetuned for specific task. As with any machine learning models, interpretability in PL.Ms has always been a desideratum, especially in decision-critice applications (c.g., healthcare).
To date, the interpretability of PLMS has primarily relied on the attention weights in their self- attention layers. However, these attention weights only provide raw word-level importance scores as interpretations. Such low-level interpretations fail to capture higher-level semantic structures, and are therefore lacking in readability, intuitiveness and stability. For example, low-level interpretations often fail to capture influence of similar words to predictions, leading to unstable or even unreasonable explanations (see Sec. 4.2 for details).
In this paper, we make an attempt to go beyond word-level attention and interpret PLM predictions at the concept (topic) level. Such higher-level semantic interpretations complementary to word-level importance scores and tend to more readable and intuitive. The core of our idea is to treat a PLM's contextual word embeddings (and their corresponding attention weights) observed variables and build probabilistic generative model to automatically infer the higher-level semantic structures (e.g.. concepts or topics) from these embeddings and attention weights, thereby interpreting the PLM's predictions at the concept level.
Specifically, we propose class of hierarchical Bayesian deep learning models, dubbed continuous latent Dirichlet allocation (CLDA), to (1) discover concepts (topics) from contextual word embeddings and attention weights PLMs and (2) interpret individual model predictions using these concepts. is worth noting that CLDA is 'continuous" because it treats attention weights continuous-valee word counts and models contextual word embeddings with continuous-ve
entries; this is in stark contrast to typical latent Dirichlet allocation Blei al. (2003) that can only handle bag-of words (both words and word counts are discrete values). Our CLDA is compatible with any attention based PLMs and can work either interpreter, which interprets model predictions the concept level without

---

Under review as a conference paper at ICLR 2023
any performance sacrifice, regulator, which is jointly trained with PLMs during finetuning to further improve performance. Our contributions are as follows:
We propose novel class of models, CLDA, to go beyond word-level interpretations and in- terpret PLM predictions at the concept level, thereby improving readability and intuitiveness. Our CLDA is compatible with any attention-based PLMs and can work either inter- preter, which interprets model predictions without performance sacrifice, or a regulator, which is jointly trained with PLMs during finetuning to further improve performance. We provide empirical results across various benchmark datasets which show that CLDA can successfully interpret predictions from various PLM variants at the concept level and improve PLMs' performance when working as a regulator.
2 RELATED WORK
Pretrained Language Models. Pretrained language models are large attention -baeed neural networks that follow a ceetrain-finetun paradigm. Usually they are first pretrained on large datasets in self- supervised manner and then finetuned for a specific downstream task. BERT Devlin et al. (2018) is pioneering PLM that has shown impressive performance across multifple downstream tasks. Following BERT, there have been variants, such as Albert LLbert|Lan Lan et al. (2019). DistiBBERT |aahh et al. (2019). and TinybertJiaa parameters. Other variants such as RoBERTa Liu et al. (2019) and BART et al. (2119) improve the performance using more sophisticated training schemes for the masked language modeling learning objective. More recently, there have also been BERT variants that design different self- supervised learning objectives to achieve better performance; examples include DeBERTa He et al. (2021), Electra Clark et al. (2020), and XLNet Yang et al. (2019). While these PLMS naturally provide attention weights for each word to intepret model predictions, such low-level interpretations fail to capture higher-level semantic structures, and are therefore lacking in readability and intuitiveness. In contrast, our CLDA goes beyond word-level attention and interpret PLM predictions at the concept (topie) level. These higher-level semantic interpretations are complementary to word-leved importance scores and tend to more readable and intuitive.
al. 2019), that achieve performance comparable BERT with fewer
Topic Models. Our work is also related to topic models Blei 2012);; Blei et al. (2003), which typically build upon latent Dirichlet allocation (LDA) Blei et al. (2003 Topic models takes the (discrete) bag-of-words representations of the documents (i.e., vocahulary-length vectors that count word occurrences) as input, discover hidden topics from them during training. and infer the topic proportion vector for each document during inference Blei al. (2003); Blei & Lafferty (2006); Wang et al. :012):Chang & Blei (2009). Besides these 'shallow' topic models, there has been recent work that employs 'deep' neural networks to learn topic models more efficiently CCrd et al. (2017); Xing et al. (2017); (Penell et al. (2020), using techniques such as amortized variational inference. There also work that improves upon traditional topic models by either leveraging word similarity as regularizer for topic-word distributions Das et al. (2015); (Batmangellich et al. (2016) or including word embeddings into the generative process Hu et al. Dieng et al. (2020); Bunk & Krestel (2018). Here note several key differences between
CLDA and the methods above. (1) These methods focus on learning topic models from scratch given a collection of raw documents, while our CLDA learns topic models directly from the latent representations inside PLMs. (2) They assume word representations are static (i.e., the representation of a word remains the same across different documents), while PLMs' word representations are contextual (i.e., the representation of a word varies across different documents according to context). In contrast, our CLDA does not have such an assumption. (3) They assume word counts are discrete numbers, which not applicable PLMs where each word has a continuous-vall valued (or real- real-valued) word count (i.e., its attention weight). In contrast, our CLDA naturally handles continuous-vale -alued word counts from PL.Ms in a differentiable manner to enable end-to-end training. Therefore these prior methods are not applicable to PL.s.
3 METHODS
In this section, we formalize the problem of conceptual interpretation of PLMs, and describe our methods for addressing this problem.
2

---

Under review as a conference paper at ICLR 2023
Problem Setting and Notation. We consider corpus of M documents, where the m 'ht document contains Lm words, and a PLM f(Dm), which takes as input the document m (denoted as mm) with L. words and outputs (1) a CLS embedding Cm E R (2) L.n contextual word embeddings em A [em=ly=m and (3) the attention weights a/!i) - [a ( between each word and the last-layer CLS token, where h denotes the lith attention head. We denote the average attention weight over
H heads as amj = π - Omj and correspondingly am _ [Qmm]]== (see the PLM at the bottom of Fig. D. In PLMs, these last-layer CLS embeddings are used as document- representations for downstream tasks (e.g., document classification). Furthermore, our CLDA assumes K concepts (topics) for the corpus. For document m, our CLDA interpreter tries to infer a concept distribution vector 6m. RRK (also known as the topic proportion in topic models) for the whole document and concept distribution vector Φmj = [0mji] E TRxK for word j document m. our continuous embedding space, the i' t' concept is represented by Gaussian distribution..N(4 E,). of contextual word embeddings. The goal is to interpret PLMs predictions at the concept level using the inferred document-leve concept vector 0m. word-level concept vector Omny and the learned embedding distributions {N E for each concept (see Sec. 4.2 for more detailed descriptions and visualizations).
(h)
3.1 CONTINUOUS LATENT DIRICHLET ALLOCATION
Method Overview. Different from static word embeddings Mikolov et al. 2013) and topic models, PLMs produce contextual word embed- dings with continuous-value entries [emj]/=] 1 and more importantly, associate each word embedding with a continuous-value attention weight [mmjlj= challenges.
CLDA
attentions
Predictor
1mm therefore this brings unique
Ci
Cwt
Car
C
Cave
To effectively discover latent concept structures learned by PLMs at the dataset level and inter- pret PLM predictions at the data-instance level, our CLDA treats both the contextual word em- beddings and their associated attention weights as observations to learn a probabilistic gener- ative model of these observations, as shown in Fig- [1 The key idea to use the attention weights from PLMs to compute a virtual continuous count for each word, and model the contextual word embedding distributions with Gaussian mixtures. The generative process of CLDA is as follows (we mark key differences from LDA in blue and show the corresponding graphical model in Fig. 2
BERT
[CLS]
zioe
[SEP)
He
/uice
Figure 1: Overview of our CLDA framework.
1. For each document m, m S M,
(a) Draw the document-level concept distribution vector o., Dirichlet(a)
(b) For each word j. < < L...
i. Draw the word-level concept index 2mj CCategorical(O.))
ii With a continuous word count Wmjj E R from the PLM's attention weights,
A. Draw the contextual word embedding of the PLM from the corresponding Gaussian component Cmj ~ "N(Hu Ezmn)
Given the generative process above, discovery of latent concept structures in PLMs at the dataset level boils down to learning the parameters {n4, E for the concepts. Intuitively the global parameters {1, E, J are shared across different documents, and they define a mixture of K Gaussian distributions. Each Gaussian distribution describes 'cluster' of words and their contextual word embeddings.
Similarly interpretations of PLM predictions at the iata-instanne level is equivalent to inferring the latent variables, i.c., document-level concept distribution vectors o,, and word-level concept indices Zmj- Below highlight several important aspects of our CLDAA designs.
3

---

Under review as a conference paper at ICLR 2023
Attention Weights as Continuous Word Counts. Different from typical topic models Blei et al. (2003); Blei (2012) and word embeddings Mikolov et al. (2013) that can only handle discrete word counts, our CLDA can handle continuous (virtual) word counts; this better aligns with continuous attention weights in PLMMs. Specifically, we denote as Wmjj the continuous word count for the j'th word in document We explore three schemes of computing Wmj-
Identical Weights: Use identical weights for different words, i.e, Wmj = 1, j. This is equivalent typical discrete word counts.
Attention-Based Weights with Fixed Length: Use Wmjj = L' L'amj, where L' is a fixed
sequence length shared across all documents.
Orrentin-BBsed Weights with Variable Length: Use Wmj Lem Ammamsf Ebm ama, where
Lm true sequence length without padding. Note that in practice, Eim amk o due to padding tokens in PLMs.
Theoretical Analysis Comparing Different Schemes. We provide theoretical analysis in the Appendix show- ing the advantages of attention-based schemes against the identical scheme.
BERT
j
Differential Word Counts as Variables, As discussed above, CLDA use attention weights (amzlf=i to compute
α
Om
2m
virtual word counts [mmil in our contextual topic mod- els, making them continuous and differentiable. Therefore, the gradients of CLDA's learning objectives (as discussed in Sec. 3.22 w.r.t. the attention weights can backprop through the large neural PI.M during joint training. This makes it possible for CLDA to work PLM regulator (or regularizer) to further improving the PLM's performance via such joint training. Alternatively, we could also choose not to backprop the CLDA's learning objective through the PLM; this way, CLDA can work as a PLM interpreter without affecting the PLM's accuracy (more details in Sec. 3.3
Figure Graphical model of our CLDA. The striped circle represents continuous word counts.
Contextual Continuous Word Representations. Note that different from topic models Blei et al. (2003) and typical word embeddings Mikolov et al.] (201)); Dieng et al. (2020) where word represen- tations are static, word representations PL.Ms are contextual; specifically, the same word can have different embeddings in different documents (contexts). For example, the word 'soft' can appear as the Ji 'th word in document m, and as the ja 'th word in document mg, and therefore have two different embeddings (i.c., emph * emztz).
Correspondingly, our CLDA, we do not constrain the same word to have static embedding; instead we assume that a word embedding is drawn from Gaussian distribution corresponding to its latent topic. It is also worth noting that word representations in CLDA is continuous, which is different from typical topic models Blei et al. (2003) based (discrete) bag-of-wor wordords representations.
3.2 INFERENCE AND LEARNING
Below we discuss the inference and learning procedure for CLDA. We start by introducing the inference of document-level and word-level concepts (i.e., 2mg and mm)) given the global concept parameters (i.c., [H,, K,)]K, and then introduce the learning of these global concept parameters.
3.2.1 INFERENCE
Inferring Document-Level and Word-Level Concepts. We formulate the problem of interpret- ing PLM predictions at the concept level inferring document-level and word-level concepts. Specifically, given global concept parameters {M,,, k the contextual word embeddings Amm. and the associated attention weights a., [amj]f=n PLM produces for each A [em document m, our CLDA infers the posterior distribution of the document-level concept vector 0m. i... p(0mm em, ams (H4 E.))K and the posterior distribution of the word-level concept index Zmzj i.e., pl p(2ng) Cm, am, {14, Σ. }))
Variational Distributions. These posterior distributions are intractable: we therefore resort to variational inference Jordan et al. (1998; Blei et al. (2003) and use variational distributions (0 7m)
4

---

Under review as a conference paper at ICLR 2023
and g(2mj (Pmj) to approximate them. Here Ym parameters to be estimated during inference. This leads to the following joint variational distribution:
R and Pnf - [0mjill R are variational
q0 (zm3) )= Im : 1 (mm))fm) g00ml II, g(zmj) Pmy))
Evidence Lower Bound. For each document finding the optimal variational distributions is then equivalent to maximizing the following evidence lower bound (ELBO):
Σ Lm
C( (0 Hmm D/m; ; {(u,, ).) Byg (0m a)]
Eg log 2mg 0mm]
j=1
Lm log p(em (mgs Mamy E Emmy) )] Eyllog g(0m)l
Σ Lm
By a( (2m))]
(2)
where the expectation is taken over the joint variational distribution in Eqn. 1
Likelihood with Continuous Word Counts. One key difference between CLDA and typical topic models Blei et al. (2003); [Blii [Bei) (2012) is the virtual continuous word counts (discussed in Sec. 33.1
Specifically, we define the likelihood in the third term of Eqn. 2an.2as.. as:
p(emy Rmjj Mamyi Ezmy) = [M(emj' Mmj+ "mj)
(3)
Note that Eqn. the likelihood of Wmj (virtual) words, where Wmy can be continuous value derived from the PLM's attention weights (details in Sec. 3.1.
Correspondingly, in the third item of Eqn. 2 we have:
Ey Elo p(em)) Zmnj Mams" Ezmy) E
Em.ia OmjiWmm log (ems p4: Σs)
Σ m.j.s Wmf { [ems - 11)²' (ems m1) - log](22))) |2,1//13
(4)
Update Rules. Taking the derivative of the ELBO in Eqn. w.r.t. Ømji (see the Appendix for details) and setting it to 0 yields the update rule for Omzti
Onji
expl V (7mm) V Ymi') $ (emj ",, (emj H1)
(5)
with the normalization constraint Ein1 Omnji = 1.
Yim ai + Σ, PmjjWmjj Wmj+
(6)
where a è |a,],< inference algorithm will alternate between updating Omfi for all (m,j. i) tuples and updating Ymi for all (m, 2) tuples.
K is the hyperparameter for the Dirichlet prior distribution 0 In summary, the
3.2.2 LEARNING
Learning Dataset-Level Concept Parameters. The inference algorithm in Sec. 3.2.1 assumes availability the dataset-level (global) concept parameters {,,,,,,),,K { To learn such these parameters, one needs to iterate between (1) inferring document-level variational parameters Y as well as word-level variational parameters 4mg in Sec. 3.2. and (2) learning dataset-level concept parameters {M, K.))KK
Update Rules. Similar to Sec. 3.2.1 we expand the ELBO Eqn. (see the Appendix for details), take its derivative w.r.t. u, set it to yielding the update rule for learning nnj
Em.y Penge Wmg 14: Em. Pengetimm
Similarly, setting the derivatives w.r.t. Σ to O we have
Σ, Em.; Prajs PenjeWyy(et (M, (4,) -2,)
(8)
Em. Omijemmn
Effect of Attention Weights. From Eqn. 7 and Eqn. g we can observe that the attention weight of the j'th word in document m, i.e., amjj affects the virtual continuous word count Wnj (see Sec. 3.1 thereby affecting the update of the detaset-level concept center and covariance μ. and Σ,
5

---

Under review as a conference paper at ICLR 2023
Specifically, if we attention- weights with fixed length or variable length in Sec. 3.1 the continuous word count Wmj will be proportional to the attention weight amjj- Therefore, when updating the concept center l4, as weighted average of different word embeddings emy, CLDA naturally places more focus on words with higher attention weights amj from PLMs, consequently making the interpretations sharper and improving performance (see Sec. 4.2 for detailed results and the Appendix for theoretical analysis).
Ho
A
NO
BERT
Ko
X
Om
mm
M
Interestingly, we also observe that PLMs attention weights on stop words such as 'the' and 'a' tend to be much lower; therefore CLDA can naturally ignore these concept-irrelevant stop words when learning and inferring concepts (topics). This is in contrast to typical topic models Blei al. (2003 ; Blei 2012) that require preprocessing to remove stop words.
Figure 3: Probabilistic graphical model of our smoothed CLDA.
Smoothing with Prior Distributions on {mu ED To alleviate overfitting and prevent singu- larity in numerical computation, we impose priors distributions on μt; and Σ, to smooth the learning process (Fig. 3 Specifcally, we use Normal Inverse- wiieerr prior on μe, and D, as follows:
Σ, IW(Ao.
p. E NN Ex/no),
where Ao, Po. Mo: and Ko are hyperparameters for the prior distributions. With the prior distribution above, the update rules for ,r and E, become:
;:
tnft Nott
E.
Ao+s. + San
Ri nn (1,, -Mg)( 1OOTT
(9)
S. Σm..
mi Wm (Omy T.)lems m.)²
(10)
where n = Emm. Pmjjs Woij is the total virtual word counts used to estimate μt; and E, Eqn. is the smoothed version of Eqn. and Eqn. g respectively. From the Bayesian perfective, they correspond to the expectations of n4, 's and E.'s posterior distributions (see the Appendix for detailed derivation).
Online Learning μr, and Σ,. PL.Ms are deep neural networks trained using minibatches of data, while Eqn. 7 and Eqn. 8 need to go through the whole dataset before each update. We therefore use exponential moving average (EMA) to work with minibatchs (see the Appendix for details).
3.3 CLDA AS PL. INTERPRETERS AND REGULATORS
CLDA can be used either a PLM interpreter. which interprets model predictions at the concept level without any performance sacrifice, or a PLM regulator, which is jointly trained with PLMS during finetuning further improve performance. This is possible because both the word counts and contextual word embeddings in CLDA is continuous and differentiable. Below we start with more details on the differentiability and then introduce our CLDA interpreter and CLDA regulator.
Algorithm Algorithm for CLDA Regulators (w/o EMA)
Input: Pretrained (.e(.) and Sa(:), initialized g()), initialized (7m 1 documents (D) M m== number of epochs T.
( 1m=1 and {Ω) m==1
for t = 1: do
for m = 1: do
Update 4.m and Ym.. using Eqn. 5 and Eqn. respectively.
Update fae()). (+), fc(-), and g(-) using Eqn.
Update Ω using Eqn. 9
Differentiable Continuous Word Counts and Contextual Word Embeddings. One of CLDA's advantage is that handle continuous word counts and word embeddings. Such continuity translates to better differentability, and therefore is particularly desirable when ones wants to jointly train CLDA and PLM to further improve PLM performance.
As shown in Fig. 2 CLDA connects to a PLM through the attention weights amj (related to the word counts Wmj) and contextual word embeddings emj- Therefore, if the CLDD leaming objective

---

Under review as a conference paper at ICLR 2023
MRPC
clarlily cided
RTE
soli cared bith Takl
Dewesierally
seled (S89) wis racc Bix
dil (CST
Fraghe nnd raat
Like ee Shach Praviit
Anrgeess
Cargras odle Clisic Senda
Oonceg (1
aecretary
Contegt TTole 20 deeguutn Tours 1igl
Sropin farntl iagor siteri on peodlc
Carponert
8f tair coultty
large mas
"Siin conpecttionn undwoo
2
lo argo bue reviloo syylen,
Fil lparce sariices ln te bast Ve iosenct sad popey noe
Donieer Piseci jus suilnenng
enstrrmeefe
Haroo shoppin Hut Toiir
196 tephyriuahe bn (Nsppct
poohs
Jastih
zest
Dirivet (Togke 20 Margant May.
Buressed sucuu lecart penerett colan. [SPP) Tao prokece dinanie
Compeneit
Cornect (Tipkl
Comperent
Concapt (Tapkc W0, phypligg maaal
Figure 4: Visualization of CLDA's learned topies of contextual word embeddings. Left: MRPC's dataset-level interpretation with two example documents. Concept 83 is relatively far from the other three concepts in the embedding space; therefore we omit it on the left panel for better readability. Right: RTE's dataset-level interpretation with three example documents.
(Eqn. 2 differentiable w.r.t. amj and emjy these gradients can then be propagated to the PLM parameters and help finetune the PLM. The derivative of the ELBO in Eqn. 2 w.r.t. emy is: aL Jemj
Ei Wyissmms ( emy)- Similarly we can get the derivative w.r.t. Omj using the chain rule, where Oumi depends on the choice of schemes for computing Wmj from amj (described inSec. 3.1 Dams and is:
DL
Σ;
-mm)) (1I: - -emj)-
Ow 0rdm
CLDA as PLM Interpreter. Using CLDA as PLM interpreter is straightforward. One only needs to first learn the global concept parameters μ, and Σ, according to Sec. (3..23. and then infer document-level concept vectors O.. and word-level concept indices Zmy.. Together, they provide dataset-level, document-level, and word-level conceptual interpretations for PLM predictions.
CLDA as PLM Regulator. One could also use CLDDA a regulator (or regularizer) when finetuning a PLM. Assume PLM that produces attention weights and contextual word embeddings for document m, i.e., (am, em) fae (Pm). ). as well as the CLS embedding Cm = fe(Cmm &m. em) here c. is the CLS embedding of the second-last layer. To better see the connection between CLDA as:
and PLMs, we can rewrite the ELBO in Eqn.
The 7m 4m P2 am, Cm = Tic (7 Pmn S2 Sac(Pm))
where i.. A (dmm))
Fr is the collection of word-level concept parameters for document m, and Ω = (m, Ex) is the global concept parameters. Assuming document-level predictor g(cm) and denoting the ground-truth label as Von we have the PLM loss during finetuning:
Lp (glfelimm Sae (D ym)
Putting them together, we have the joint loss:
L; = Lo [)), Bm) + ALo (m 2m S2 fac (Dm)))
(11)
When finetuning a PI.M, one can iterate between updating (1) the PLM parameters in sae fac()), fa(), and g()). (2) the CLDA global concept parameters Z2 and (3) the CLDA document-level and word- level concept parameters 7 and 4 m. Note that f-e contains most of the parameters in PLMs and appears in the CLDA loss term L,; therefore CLDA can improve the finetuning process in PLMs. Alg. [ shows an overview of CLDA training when used as a PLM regulator (to prevent clutter, we show the version without EMA).
Theoretical Analysis. In the Appendix, we provide theoretical guarantees that under mild assump- tions our CLDA can learn concept-level interpretations for PLMSs especially in noisy data.
7

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENTS
4.1 EXPERIMENT SETUP
Datasets. We use the GLUE benchmark Wang et al. (2018) evaluate our methods. This benchmark includes multiple sub-tasks of predictions, with the paired sentences as inputs. In this paper, we use six datasets from GLUE (CoLA, MRPC, STS-B, QQP, RTE, and SST-2) to perform evaluation.
Implementation Our training/validationn al. (2018). All PL.S are base models from vanilla settings, with the hidden dimension of 768. The BERT Models are optimized by Adam W Kingmaa & Ba (2014) Optimizer, using a learning rate of 10 4 with linear warmup and linear learning rate decay. We finetune the models until metrics on validation sets get the highest score (see the Appendix for more implementation details).
data split of GLUE datasets follows exactly Devlin
Evaluation Metrics. We follow previous work Devlin et al. (2018) on the GLUE benchmark, and evaluation metries for the 4 datasets, i.e., Matthew's 's correlation
the provided testing scores for CoLA, Fl score and accuracy for MRPC and QQP, Pearson/Spearman correlation for STS-B, B, and accuracy for RTE and SST-2.
We compare our methods with the following four baselines: BERT Devlin et al. (018)), BART Lewis et al. (2019), RoBERTa Liu al. (2019). and DeBERTa He al. (2021) The basic settings of augA-augeented PLMs at the BERT side are kept consistent with the Base models for fair comparison (see the Appendix for details).
Baselines.
4.2 CONCEPTUAL INTERPRETATION (MORE RESULTS IN THE APPENDIX)
Dataset-Level Interpretations. To showcase CLDA's capability as a PLM interpreter, we use CLDA trained on MRPC and RTE, respectively, sample 3 concepts (topics) for each dataset, and plot the word embeddings of the top words (closest to the center μ4;) in these concepts using PCA. Fig. left) shows the concepts from MRPC. We can observe Concept 20 mostly about policing. including words such as 'suspect', , 'police', and 'house'. Concept 24 is mostly about politics, including words such as 'capital', 'Congress'. and 'Senate". Concept 27 contains mostly names such as 'Margaret" and 'Mary". Similarly, Fig. right) shows the concepts from RTE. We can observe Concept 67 is related to Islam and includes words such as 'Quran' and Pashaa Concept 13 is related to Europe and includes European countries/names such 'Prussia' and 'Sallburg'. Concept 91 is healtheare-related and includes words such as 'physiology' and 'insulin'.
Document-Level Interpretations. For document-level conceptual interpretations, we sample two example documents from MRPC (Fig. (eeft) and three from RTE (Fig. Aright)), where each document contains pair of sentences. The MRPC task is predict whether one sentence paraphrases the other. For example, in the first document of MRPC, we can see that our CLDA correctly interprets the model prediction "True" with Concept 24 (politics). The RTE task is predict whether one sentence entail the other. For example, in the second document of RTE, CLDA correctly interprets the model prediction 'True' with Concept 13 (countries).
Word-Level Interpretations. For word-level conceptual interpretations, we can observe that CLDA interpret the PLM's prediction on MRPC's first document (Fig. leftt) using words such as 'senate' and "bitty" that are related to politics. Note that the word "bitty" is commonly used (with 'little") by politicians to refer to the small size of tax relief/cut plans. Similarly, for RTE's first document (Fig. righht)) CLDA correctly identifies Concept 67 (Islam) and interprets the model prediction "False" by distinguishing between keywords such as 'Jihad' and 'Al Qaeda'.
4.3 QUANTITATIVE RESULTS
To evaluate CLDA PLM regulator, we use BERT, RoBERTa, BART, and DeBERTa as base models for our CLDA, leading four different CLDA models, BERT-CLDA, RoBERTa-CLDA BART- CLDA, and LABBERT--CLDA respectively. Table shows the performance of our CLDA variants and the correspondingly base PLMS on six benchmark datasets, CoLA, MRPC STS-B, B, QQP, RTE, and SST-2
8

---

Under review as a conference paper at ICLR 2023
Table I: Results on GLUE benchmark datasets.
Dataset
CoLA
MRPC F1/Acc.
STS-B
QQP F1/Acc.
RTE
SST-2
Metrics
Matthew's s Corr.
Pear:/Spea. Corr.
Acc.
Acc.
BERT-Base BERT-CLDA (Ours)
51.2
73.3/66.6
82.5/81.2
66.0/85.1
57.1
88.2
52.2
82.2/74.6
84.0/82.8
70.3/88.7
60.3
92.2
RoBERTa-Base
59.0
79.9/66.5
83.3/82.2
52.5/78.1 1
67.4
93.8
RoBERTa-CIDA (Ours)
59.1
78.2/71.4
83.5/82.4
69.5/88.1
70.1
94.7
BART-Base
46.9
85.4/79.8
87.6/86.5
70.5/88.8
62.6
91.9
BART-CLDA (Ours)
49.5
85.6/80.0
87.6/86.6
70.7/88.9
64.7
92.6
DeBERTa-Bart
57.2
86.3/81.8
89.6/88.8
70.4/88.9
64.8
92.4
DeBERTa-CLDA (Ours)
59.8
86.7/82.1
89.8/89.0
71.5/89.1
68.4
94.0
Average- Base
53.7
81.2/73.7
85.8/84.7
64.9/85.2
63.0
91.6
Average-CLDA (Ours)
54.9
83.2/77.0
86.2/85.2
70.5/88.7
65.9
93.4
The last two rows show the average predictive performance across different PI.M base models for different datasests. We can observe that on average, our CLDA significantly outperforms the baselines in all datasets. Notably, the largest dataset, QQP, our CLDA improves upon the baselines by 5.6% in terms of Fl score and by 3.5% in terms of accuracy. Moreover, even for 'difficult" natural language inference tasks such RTE, CLDA 2.9%. When using RoBERTa as the base model, CLDA achieves absolute improvements of 17.0% and 10.0% for F1 score and accuracy, respectively. Note that STS-B and SST-2 relatively "easy" datasets, and even the BERT-Base model could achieve correlation higher than 82% and accuracy higher than 88%, respectively. In this case the room for improvement is minimal. However, our CLDA could still lead to slight improvement in terms of Pearson correlation and Spearman correlation for STS-B, as well reasonable accuracy improvement for SST-2.
still improve the average accuracy by large margin of
4.4 ABLATION STUDIES
To evaluate different schemes of computing the virtual word counts Wmy from attention weights (as introduced in Sec. (.1). we perform abla- tion studics on the CoLA dataset using differ- PLM base models. Table 2 shows the re- sults on the base model and CLDA with identi- attention-based
Table Ablation studies for the CoLA dataset in terms of Matthew's correlation.
Model
BERT RoBERTa BART DeBERTa
Base
51.2
59.0
46.9
57.2
cal weights (CLDA-Identical) weights with variable length --DA--aaiabll), and attention-based weights with fixed length (CLDA-Fixed).
CLDA-Identical
50.4
55.6
47.0
51.9
CLDA-Variable
52.2
57.5
47.4
59.5
CLDAA-Fieed
52.2
59.1
49.5
59.8
One observation is that CL.DA-Identical tends to underperform the PLM base models, while both CLDA Variable and CLDA-Fixed can significantly outperform the base models. This verifies the importance of using attention weights compute the virtual continuous word counts. Another interesting observation is that CLDA-Fixed slightly outperforms CLDA-Variable Note that both CLDA-Fixed and D-Varibbll use attention weights to compute virtual continuous word counts; the difference is that in -LDA-larablle assigns longer documents more (total) weights when learning the global concept parameters (Eqn. and Eqn.(), while CLDA-FiXed treats each document fairly (as if they had fixed length). Therefore Table shows that it is beneficial to assume different documents have fixed total virtual word counts for CLDA.
CONCLUSION
We develop CLDA as a genearal framework to interpret pretrained word embeddings at the concept level. Our CLDA is compatible with any attention- based PLMs. It can not only interpret how PLMs make predictions, but also help improve contextual word embeddings in an end-to-end manner, thereby boosting predictive performance.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Kayhan Batmanghelich, Ardavan Saeedi, Karthik Narasimhan, and Sam Gershman. Nonparametric spherical topic modeling with word embeddings. In Proceedings of the conference. Association for Computational Linguistics. Meeting, volume 2016, pp. 537. NIH Public Access, 2016.
David M Blei. Probabilistic topic models. Communications of the ACM, 55(4):77-84, 2012.
David M Blei and John D Lafferty. Dynamic topic models. In Proceedings of the 23rd international conference on Machine learning, pp. 113-120, 2006.
David M Blei, Andrew Ng, and Michael Jordan. Latent dirichlet allocation. the Journal of machine Learning research, 3:993-1022, 2003.
Stefan Bunk and Ralf Krestel. Welda: enhancing topic models by incorporating local word context. In Proceedings of the 18th ACM/IEEE on Joint Conference on Digital Libvaries, pp. 293-302, 2018.
Dallas Card, Chenhao Tan, and Noah A Smith. A neural framework for generalized topic models. arXiv preprint arXiv:17055 170.0999,,,,, 2017.
Jonathan Chang and David Blei. Relational topic models for document networks. In Artificial intelligence and statisfics, pp. 81-88. PMLR, 2009.
Kevin Clark, ThangTThag Aa Luong, Quoe V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint vrXiv:2003.10555 2020.
Rajarshi Das. Manzil Zaheer, and Chris Dyer. Gaussian Ida for topic models with word embeddings. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Lonng Papers), pp. 795-804, 2015.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 18X0v:/810.000 2018.
Adji B Dieng, Francisco JR Ruiz, and David M Blei. Topic modeling in embedding spaces. Transac- tions of the Association for Computational Linguistics, 8:439-453, 2020.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanee bert with disentangled attention. In ICLR, 2021.
Matthew Hoffman, Francis Bach, and David Blei. Online learning for latent dirichlet allocation. advances in neural information processing systems, 23:856-864, 2010.
Pengfei Hu, Wenju Liu, Wei Jiang. and Zhanlei Yang. Latent topic model based gaussian-ld, for audio retrieval. In Chinese Conference on Pattern Recognition, pp. 556-563. Springer, 2012.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv: /909. 10351, 2019.
Michael Jordan, Zoubin Ghahramani, Tommi Jaakkola, and Lawrence Saul. An introduction to variational methods for graphical models. In Learning in graphical models, pp. 105-161. Springer, 1998.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 1rXiv:1412.6980, 2014.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv: / /942, 2019.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-te requence-to-sequeeee -eeeu-ueeeeeeeeeeeeeeeeeeee-ee-e pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.1 191.11911.. 2019.
10

---

Under review as a conference paper at ICLR 2023
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 19XX:::997..119999 2019.
2019.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representa- tions in vector space. arXiv preprint arXiv: 1301.381, 2013.
Aaron den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. arXiv preprini /tt1::/111.00000 2017.
Nicole Peinelt, Dong Nguyen, and Maria Liakata. tbert: Topic models and bert joining forces for semantic similarity detection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7047-7055, 2020.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:/910.0/108, 2019.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint TrXiv:1804.07466. 2018.
Chong Wang, David Blei, and David Heckerman. Continuous time dynamic topic models. arXiv preprint
2012.
Chen Xing, Wei Wu, Yu Wu, Jie Liu, Yalou Huang, Ming Zhou, and Wei- Ma. Topic aware neural response generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
A DETAILS ON LEARNING CLDA
Update Rules. Similar to Sec. 3.2.1 of the main paper, we expand the ELBO in Equ. of the main paper, take its derivative w.r.t. n, and set it to O: dL yielding the update rule for learning pe;
ap, Em, PmjtWmm E, (ems (1,)) 0,
E Em Pengi 1mmg p. Em. Ponsetinn
(12)
where E,- is canceled out. Similarly, setting the derivatives w.r.t.
E to 0, iec.
Em. @mjeWmms E, Σ (Cmm (m)( ". 1) we have
Σ 2im Pmmji (Omm) (4,) 14,)
(13)
Em. PmyeWmy
Online Learning of μ.; and Σ. Note that PLMS are deep neural networks trained using minibatches of data, while Eqn. and Eqn. need to go through the whole dataset before each update. Inspired by Hoffman et al. (2010) Oord et al. (2017). we using exponential moving average (EMA) to work with minibatchs. Specifically, we update them as:
p4 +1 p) B P. #. < p. N P.N.
Σ, + p. N. P. E, (1) BB Σ,. N p. N (1 p). B,
Σ # # N E N
where B is the minibatch size, N is running count, and E (0, 1) is the momentum hyperparameter. i. and Σ; the updated μa and E; after applying Eqn. 7 and Eqn. 8 onnly the current minibatch.
11

---

Under review as a conference paper at ICLR 2023
Table 3: Example concepts RTE dataset learned by CLDA.
Concepts
Top Words
bio-chem
cigarette
biological bolivian
ozone fiji
cardiovascular
chemist
liver
chemical
toxin
citizenship
indies
surrey bush
jamaican dutch
latino
caribbean
names
mozart
spielberg
einstein disperse
kurt liszt hilton
lynn
conspiracy
secretly
corrupt
infected ill prosecutor prosecutor
hidden illegally
sniper
administration
reagan interior
ambassador sheriff
diplomat
legislative
spokesman
embassy
crime
fraud
laundering
corruption
fool
robber
greed
Paired Sentences as a Document. Many modern natural language processing tasks involve predicting a label from a pair of sentences (for example, given two sentences, predict whether one sentence paraphrases the other). In this case, one document may contain a pair of sentences (with length Lm and Linz and Lm Lom + Lmma) as PLM inputs, and y of each sentence can be inferred as:
Tmgs = 02 + Jmm Omjilljjs Tmggi OG + +Zjmm, Wmjj
B EXPERIMENTAL SETTINGS AND IMPLEMENTATION DETAILS
We will release all code, models, and data. Below provide more details the experimental settings and practical implementation.
Data Preprocessing, Our rrainingvalidation//ee data split of GLUE datasets follows exactly Devlin et al. (2018). We train our model on the training data, perform model selection (select hyperpa- rameters) using the validation data, and evaluate methods the test data. Our tokenization and bert-configurations follow previous PLMs that we compare with. For fair comparison, we use low- ercase tokenization both in base models and our CLDA models. According to different versions of CLDA weighting. we can choose whether to calculate TF-IDF scores in documents. If we use an identical-weight CLDA, additional computing of TF-IDF scores is necessary to filter words with little information for our CLDA-base Abased topic models.
Implementation. All PLMs are base models from vanilla settings, with the hidden dimension of 768. We initialize the models with the seed 2021. The BERT Models are optimized by Adam KKngmma & Ba 2014) Optimizer, using learning rate of 10-4 with linear warmup and linear learning rate decay. We finetune the models until metrics on validation sets get the highest score. We treat the training batch-size, CLDA prior parameters, and (in Eqn. of the main paper) as hyperparameters, and run grid-search in training and validation to search for models with the highest possible performance. To alleviate overfitting of CLDA during joint training, we periodically include the CLDA loss term Le in the joint loss Ly along epochs, i.e., using the CLDA term every 1/3/5 epochs (as training hyperparameter as well), along with original base PLM finetuning loss. We use the fixed scheme for CLDA training by default to produce the results in Table of the main paper. We use the penultimate-layer word embeddings for CLDA in Eqn. 3 of the main paper, because our preliminary results show that using the penultimate layer instead of the output layer improves performance.
Baselines. To ensure fair comparison, during fine-tuning, we select the epoch for both baselines and CLDA entirely based on validation accuracy and report the test accuracy; in contrast, [Devlin et al., 2018] directly chooses the 3rd epoch; we argue that this is not rigorous and potentially 'overfits' test sets. Also, as aforementioned, We follow the convention of topic models and preprocess the documents into lower-case words for both baselines and CLDA; in contrast, [Devlin et al., 2018] keeps the words unchanged. Nevertheless, note that our CLDA can interpret any PLMs without accuracy sacrifice; therefore the exact accuracy for BERT-base is less relevant in our case.
Visualization Postprocessing. For better showease the dataset-level concepts as Fig. 4 of the main paper, we may employ simple linear transformations on the embedding of words after the aforementioned PCA step, in order to scatter all the informative words on the same figures. However, for some datasets such STS-B, this is not necessary
don't use it.
Topic (Concept) Identification. Inspired by Blei et al. (2003), we identify meaningful topics by listing the top-5 topics for each word, computing the inverse document frequency (IDF), and filtering out topics with the lowest IDF scores. Note that although GLUE benchmark are datasets that consists of documents with small size, making it particularly challenging for traditional topic
12

---

Under review as a conference paper at ICLR 2023
models (such LDA) to learn topics; interestingly CLDA still do well in learning the topics. We contribute this to the following observations: (1) Compared to traditional LDA using discrete word representations, CLDA uses continnous word embeddings. In such continuous space, topics learned for one word can also help neighboring words; this alleviates the sparsity issue caused by short documents and therefore learns better topics. (2) CLDA's attention-based continuous word counts further improves sample efficiency. In CLDA, important words have larger attention weights and therefore larger continuous word counts. In this case, one important word sentence possesses statistical (sample) power equivalent to multiple words; this leads to better sample efficiency in CLDA.
C EXPANSION OF ELBO
We expand the ELBO in Eqn. 2 of the main paper
K
K
K
K V
Clt, Φ; a. [M]K, {E}x) log r
log
((a) +
Σ (a)-1)(V(())
yg))
1=1
i=1
j=1
K
Oj 1 (T()
1 >
Vk))
j=1
k=1
Σ Omj
{e - m:) " 14;1
mjji
-log[(2T)) d/2 | 11/2
K
K
K
K
log r 7j) + Σ
log I(r:) - -6. - 1)(((()))
V > 7;)) j=
3
im
im1
Øys log 03.
(14)
D DERIVATION ON SMOOTHED CLDA
To alleviate overfitting and prevent singularity in numerical computation, we impose priors distribu- tions on μ, and Σ, to smooth the learning process. Specifcally, we use a Normal-Inven Inverse-Wishart prior on μ, and E; as follows:
Σ, ~ IW vu), n. M,E ~ N(po Σx/Ko),
where Ao. Po. Po. and Ko hyperparameters for the prior distributions. With the prior distribution above, the update rules for the parameters of the posterior distribution NIW(H, E.)
(m) A (n) (n) (())) become:
(n) koft trft
(15)
kotn
A (n)
= Ao + SS
son
(Tg: Mp) Ho)
(16)
kotn
(1)
Ko + n,
(17)
(n) = vo + n,
(18)
S, = Σ m. OmijiWan (Cmj (.) - PT)
(19)
where n Em. Wnj is the total virtual word counts used estimate μ4 and E. Taking the expectations of μ. and Σ; over the posterior distibution NIW( Eilre (n) A (n) (n) ((n)).
13

---

Under review as a conference paper at ICLR 2023
STS-B
Demeradioaally Reduction Emrbeadngge with Topics
aigeria
Concupt (Tope) 65: boah,, flovers, aiss, garder, wil,, disset zofa, vase, carret souds, turt, plnygrund. froo
(Tappc)
topee
rilla
Goncept (Topic) BM guaenill,, NSA reploraege madded. Careeier, Croeel,, hosteiic
Concept (Toppe) dis Kiww Mexcow, residience, Aigera aghens, Ulraatinn, emerge. Gooda Brnal
hose Stations
ukrainian moscow
a worann watch (SEP)) bothing rope llko.
Syrio pgries to sumonddr chemical woapons (SEP)
parachute
Componen
sifle E
axe
logget sink
kiev
Document: Tople: 63,
Prediction: (4)
Devcument Topic: BM
Preckston 0/118
South Kimnwen soither reportedy frees ather killing comracos of bonder with North Korca [SEP) South uoddier commetee and lisses
floors
woman la pountig egg into flying pan: (SEP) A man is
Wase trash playground garden
palting twu doga.
Document Topl:
Prediction: 5.005
(4)
airs closet
Prosittonn 0:115
guerrilla historic
raided
call playing flooo. [SEP] A man is sllcing garlic.
csopecelton Rusoe beeche uandee arrest
canadian
groatia
Document Topls:
Document Toplc: 84
espionage
Prediction 0,100
140
Prediction
Component
Figure 5: Visualization of CLDA's learned topics of contextual word embeddings. We show STS-B's datasct-level interpretation with six example documents. The prediction of CLDA between the range of [0, 5].
have the update rules as:
#. ENIw(u,
ko Ka Ruton
(20)
Kon Σg + ENIW AotS.+
(i,, MMO -H,))
kntn
(21)
Mo+n K-1
S: = Σ m.j Pmjittmy (emj
T4,) )A:)
(22)
Eqn. and
and Eqn. are the smoothed version of Eqn. 7 and Eqn. g respectively. From the Bayesian perspective, they correspond to the expectations of μr, and Σ,'s posterior distributions.
E MORE CONCEPTUAL INTERPRETATION RESULTS
Dataset-Level Interpretations. As in the main paper, we leverage CLDA interpreter STS-B and QQP, respectively, sample 4 concepts (topics) for each dataset, and plot the word embeddings of the top words (closest the center (4,) in these concepts using PCA. Fig. 5 shows the concepts from STS-B. We can observe Concept 63 is mostly about household and daily life, including words such as 'trash', 'flowers', 'airs', and 'garden'. Concept 60 mostly about tools, including words such as 'stations', 'rope', 'parachute', and 'hose'. Concept 84 is mostly about national security, including words such as 'guerilla', 'NSA', 'espionage', and 'raided'. Concept 55 contains mostly countries and cities such as 'Kiev", 'Moscow', 'Algeria', and 'Ukrainian'. Similarly, Fig. G shows the concepts from QQP. We can observe Concept 12 mostly about negative attitude, including words such as "boring', 'criticism', and 'blame'. Concept 73 is mostly about Psychology, including words such as 'adrenaline', 'haunting', and 'paranoia'. Concept 34 mostly about prevention and conservatives, including words such as 'destroys', 'unacceptable', and 'prohibits'. Concept 64 is mostly about strategies, including words such as 'rumours', 'boycott', and 'deportation'.
Document-Levé Interpretations. For document-level conceptual interpretations, we sample six example documents from STS-B (Fig. [5) and eight from QQP (Fig. (), respectively, where each document contains a pair sentences. The STS-B task is to predict the semantic similarity between two sentences with the score range of [0, 5]. For example, in Document (a) of Fig. 5 we can see that CLDA correctly interpret the model's predicted similarity score '0.118' (which is relatively low.) with Concept 63 (household and daily life) and Concept 60 (tools). Similarly, in Document (f) of Fig. 5] we can see that CLDA correctly interpret the model's predicted similarity score '4.672' (which relatively high) with Concept 84 (national security). The QQP task is to predict whether
14

---

Under review as a conference paper at ICLR 2023
QQP Desesdorall of Emtediinge wi Topios
Concept (Taged 12, boring onlic on, poier, Maaee, oripeity pussierent, ortoo desting,
scandals
Concrpt (Topic prodanryy raunting parssci, twiching
Concapt (Taple) 34: ceetrogs aracoerrate profests, atriinng ruined
predatory
hodyguards
depodtation besiguarte compaling
par praapoaa
Paynting
ompathy
unere to wath vinun you hond?? (SFa) (Weraaa
ure Hi key Ceniett ziero Indidt (8P)) What
Tactions
critic
deportationbaylle
boy Edepours
criticim
houe mair wnttcher??
Duoument Togic 64
corigeling
Prodicion: Tow
destroys
Precidion: True
Component
Hia canl tuee idea lrth fundet tolewwod
chesping (SEP)
punishment blame
Why unaibbicdoaako colilooooukkeee logal for paople?
Topic
Document
Prodicioo: True
Predicios: Fallo
Witl crnseration party Whove the
Ara there any lamous casea of pronsia (ih
oppocala of panans no at (SEP) what ure some good pronsis oppoatle
fadiana of the consersttiv party?
Documentt Topic 73
Prodition: True
lal
unacreptable profibits
Wry incle salerage OPEC7 (BEP) why does Inda opposs CPEC2
Whlunc how to luaring hrudking newSi?
Straining
Document Topic
Donament Topr: 04
Component i
Prededio::
14
Prodeter: True
Figure 6: Visualization of CLDA's learned topics of contextual word embeddings. We show QQP's dataset-level interpretation with eight example documents.
the two questions are paraphrase of each other. For example, in Document (b) of Fig. 6 can that CLDA correctly interprets the model's predicted label "False" with Concept 73 (Psychology). Similarly, in Document (e) of Fig. [ we can see that CLDA correctly interprets the model's predicted label 'True' with Concept 64 (strategies).
Word-Level Interpretations. For word-level conceptual interpretations, we can observe that CLDA interprets PLM's prediction on Document (c) of Fig. using words such as 'cat', 'floor', and 'garlic' that are related to household and daily life. Also, CLDA interprets PLM's prediction on Document (e) of Fig. Jusing words such as 'soldier' and 'border' that are related to national security. Similarly, for QQP's Document (d) (Fig. ()) CLDA correctly interprets the model prediction 'True' by identifying keywords such as 'sabotage" and 'oppose' with similar meanings in the topic of strategies. For QQP's Document (g). (Fig. ()) CLDA interprets the words the both sentences with the same semantics, such 'conservative' that is related to prevention and conservatives (note that in politics, 'conservative" refers to parties that tend to prevent/block new policies or legislation), and thereby predicting the correct label 'Truc',
Example Concepts. Following Blei al. 2003), we show the learned concepts on the RTE dataset in Table 3 which is complementary
aforementioned explanations. We select several different topics from Fig. 4 of the main paper. As in Sec. 4.2 of the main paper, we obtain top words from each concept via first calculating the average of the each word's 's corresponding contextual embeddings over the dataset, and then getting the nearest words to each topic center (11;) in the embedding space. As we can see in Table CLDA can capture various concepts with profound and accurate semantics. Therefore, although PLM embeddings are contextual and continuous, our CLDA can still find conceptual patterns of words on the dataset-level.
F THEORETICAL ANALYSIS ON CONTINUOUS WORD COUNTS
Before going to the claims and proofs, first we specify some basic problem settings and assumptions.
Suppose there are K + 1 topic groups, each of which is regarded to be sampled from a parameterized multivariate Gaussian distribution. In specific, the K ++11 1 'th distribution of topic has much larger covariance, and in the same time, closed to the center of embedding space. The prementioned properties can be measured by a series of inequalities:
Σx+1l max R+1 Ed|.
(23)
15

---

Under review as a conference paper at ICLR 2023
The approximate marginal log-lik log-likelihood of word embeddings, i.e., the third term of the ELBO as mentioned in Eqn. 4 of the main paper, is:
C C(train)
- bm
j=1 Bgl Zmjj Emij Hams' )mmm))
PmjiWmyf {- [Cmm 1²) " (Cmj (,) (2 22
Σ 1/
(24)
The above equation is the training objective, yet for fair comparison of different training schemes, calculate the approximated likelihood with word count for all words.
(leval)
Lm
By llogg' ('mm)) [Zmi Mamss Bamy)l
j=1
Σ
@mjil - (Cmj - =,, 'Cmj - mi) - (2T) E 1/2 (25)
2/22)/2 E
m.j.c
F.1 GAUSSIAN MIXTURE MODELS
Suppose we have a ground truth GMM model with parameters π E RK and {uT: Σije with K different Gaussian distributions. In the dataset, let and N2 denote the numbers of non-stop-words and stop-words, respectively. Then the marginal log likelihood of learned GMM model on given data sample e can be written as
K
plll(1, Σ}, σ) = Σ TNN(ee Pk: Ex)-
(26)
Assuming dataset of N + N, words {er N and taking the associated weights wy for each word into account, the log-likelihood of the dataset can be written as
N
N+N, log
plallin Ex)
log
WE Ex)
Mi Me. Ee)-
(27)
Leveraging Jensen's 's inequality, obtain lower bound of the above quantity (denoting O the ]
collection of parameters {Kg
KK k=
and π):
N+
10 (w,))
Σ w. log Σ Me Exd
wy log
Wo Ex)
where Cs constant.
the following theoretical analysis, we consider the following three different configurations
weights w.
Definition F.1 (Weight Configurations). We define three different weight configurations as follows:
Identical Weights: wy N+N, i (1,2,.... N.}
+, iE (1, N} Ground-Truth Weights 0. iE (N 1,N +2, ......NN No)
( E I 1. iE (1,2....,, N) de (0, I iE (N +1,N ..... N.}
Attention-Based Weights: w;
Definition F.2 (Advanced Weight Configurations). We define three different weight configurations follows:
Identical Weights: wi N+N,, i (1,2,.... ... N No}
+. ie (1, N} 0, iE (N + 1, +2, 2, ....... N NN
Ground-Trus Weights : w;
16

---

Under review as a conference paper at ICLR 2023
N H.. i (1,2,.... NJ [0, NNN, J,
Aitention-Based Weights: wy E
Definition E.3 (Optimal Parameters). With Definition are then defined as follows:
F.1 the corresponding optimal parameters
+ = arg max L(6;; W + Identical),
(29)
G = arg max ( W - GT),
(30)
O A arg max L(O; w + Attention):
(31)
where w + Identical, w - GT, and w + Attention indicates that 'Identical Weights', "Ground-
Truth Weights', , and 'Attention-Based Weights' are used, respectively.
Lemma F4. Suppose we have two series functions {s,((r)} and (/2,(1)), with two non-negative
weighting parameters X1, X2 satisfying N\1 + N₂A2 1. We define the final objective function f())
as:
N,
f(x; 22) = A1
Σ f1)(()) +
Σ
f2,(2))
(32)
i=1
i=N+1
We assume two pairs of parameters (2) and (x): where
A 2 X1:
(33)
A S X/2
(34)
Defining the optimal values of the o for different
weighting parameters as
3 arg max fx; x1, X2), x
(35)
3' arg max f(z X2),
(36)
we then have that
f 0) > f(:: 3. 0).
(37)
Proof. We prove this theorem by contradiction. Suppose that we have
0) f( O). According to Eqn. 49 i.e., A1 2 X1, and the equation N\1 + N₂A2 = 1, we have
(38)
112/2 = \y 1-NX
N. X
- N. X/12.
(39)
According to Eqn. 36 we have the following equality:
[(:; X₁. X²) s f( X, X))
(40)
Combined with the aforementioned assumption in Eqn.
38
we have that
N
N,
[2 = -1 Σ
(.) + X22
Σ
f2.(7)
(41)
i=1
N=
N
N
=(x Σ (1) + X2A2
Σ f2.(())+ + (N X1d2) 1 Σ (l)
(42)
N=++1
=>e X1, X²) - N(A - (0)
(43)
<X2f(E' X1 x2) + N - X122)((T:11 1.
(44)
N.
=(X/2² Σ (T) + +/112
Σ
(z)( + (N(A11222 - -2122) 1 Σ
(z))
(45)
N=1-1
i=1
N
=\,i²2 Σ hi(T')+ + +226
Σ 1z)
(46)
1=++1
(47)
17

---

Under review as a conference paper at ICLR 2023
which contradicts the definition of 2 in Equ. 35 (i.e., E maximizes f(x; A1, A2)), completing the proof.
Lemma F.S. Suppose we have two series of functions {f1,(()}} and {f2,(x)), with two series of non-negative weighting parameters A [' 22 [22 satisfying E A EiN++ X2,5 = 1. We define the final objective function f(.) as:
f(; A1, 22) = (1)fia(z) +
Σ )2,122((1))
(48)
1=++1
We assume pairs of parameters (X1, X2) and N2), where
\14 2 X1,0 i {1, 2, ..., N}: 22,1 S N2,1. ie (N N +2, N₂).
(49)
(50)
Defining the optimal values of the objective function for different weighting parameters as
T arg max ff (z; X1 X2),
(51)
T' arg max (x; X1 X2)
(52)
x" arg max (z, N O).
(53)
Under the following Assumptions (with 1 and denoting vectors with all entries equal to and 0, respectively);
1. f(ī, O, A2) S f A2).
2. f(x; A, 2 f 0). if x =1 X20,
|||| 1.
we have that
f(3 0) 2 f O)). 0).
(54)
Proof. We start with proving the following equality by contradiction:
|| x - || - x=1 11-
(55)
Specifically, if
| x 1 xx" 1|.
(56)
leveraging the Assumption and above, we have that
f(;; A1 A2) = f(3; A1, 0) + f(2; 0. X2) < f( X1,
117 = f(
(57)
which contradicts Eqn. 51 Therefore, Eqn. 55 holds.
Combining Eqn. 55 and Assumption 2 above, we have that
f(;: 1 0) 2 f('; fE':
(58)
concluding the proof.
Based on the definitions and lemmas above, we have the following theorems:
Theorem F.6 (Advantage of @ in the Simplified Case). With Definition F. and Definition F3
comparing O,, Og. and θ by evaluating them on the marginal log- likelihood of non-stop non-stop-words, i.e., L(. w + GT), we have that
(O,M(O1; - GT) S (EMMO(OA GT) S LOMMM(GG; GT).
(59)
18

---

Under review as a conference paper at ICLR 2023
Proof.
First, by definition easily find that Og achieves the largest Z(: GT) among the
three:
ma GT), CaM GT)) mag LaMrr GT) Lowrr(Ba:w GT).
(60)
Next, we set {wiker
A and {wi NNN, to Ag, respectively; we rewrite (wiJi=N+1
log Lk=l k= TWN(e MEg Ex) as f1,(() for E 11, . and f2,(2) for e
{N + 1,,N N N + N.), where corresponds to θ (I,, (p En H=m)) By Lemma have that
+1,
F.4 we
(O GT) s LGMM GT).
(61)
Combining Eqn.
60 and Eqn. 61 concludes the proof.
Theorem F.7 shows that under mild assumptions, the attention-based weights can help produce better estimates of in the presence of noisy stop-words and therefore learns higher-quality topics from the corpus, improving both generalization performance and interpretability of PLMs.
Theorem F.7 (Advantage of @ in the General Case). With Definition F.2 and Definition A3
comparing i Oc and ④ A by evaluating them on the marginal agglikelihood of non-stop non-stop-words, i.e. LGMM(: w + GT), we have that
CGMM((1;w GT) (oM GT) S CGMM((Gs GT).
(62)
Proof. First, by definition one can easily find that Og achieves the largest L(: w - GT) among the three:
max[Famsr(@)::. RGMM GT) max COMMr(..w GT) LoMMr GT).
(3)
Next, we invoke Lemma F.5 by (1) setting [wi to A and {wi) N+N, to X2, respectively,
and (2) rewriting log Σk 1= (e, ugg Σx) as fI) for E .......... and f2,(() for + (N++1,
K
N 1,... N N.), where corresponds to θ (I, (Mr Σ k By Lemma F55 we
then have that
LGMM (O + GT) & CGMM (Oc: - GT).
(64)
Note that because fi,(()) and (2,(()) are Gaussian, therefore Assumption and in Lemma E5 hold naturally under mild regularity conditions.
Combining Eqn. and Eqn. 71 concludes the proof.
F.2 CLDA AS INTERPRETERS
As mentioned in Eqn. 4 of the main paper, the ELBO of the marginal likelihood (denoting θ the collection of parameters Φ y and (/4: 1K is as follows:
CO (w,+) = Σ/'1 Roloog(emm (mnzi may" Ezms
Amsil (ems m2) DT (ems p.) log[(2e) 4/3 13))
(65)
Based on the definitions and lemmas above, we have the following theorems:
Theorem F.8 (Advantage of @ in the Simplified Case). With Definition F. and Definition
P3
comparing @ @g. and O by evaluating them on the marginal log- agllikeldoodd of non-stop non-stop-words i.e., L( w + GT), we have that
(OLDA(0 - GT) LOLDA + GT) LOLDA GT).
(66)
Proof. First, by definition one can easily find that O achieves the largest C(: - GT) among the three:
man(Forna GT), Corpa GT)) max COLDA GT) CoLpA GT). (67)
19

---

Under review as a conference paper at ICLR 2023
Next, set Um Um( |=] to A1 and Um (mmj NmttNm. X2. respectively; rewrite E; Omjif H(omg Um 11, ...,, Nm3 and f2 for j Um (Nm + 1, Nm ,,.,,, Nm Nm. where cor-
J=NN++1
(1) ,,,";, (omj pj) 12T] 2/2 Σ 11/3]} as f1s(z) for e
responds to - (0.7, v (1 ER) By Lemma F.4 we have that
LOLDA AY GT) < LOLDA GT).
(68)
Combining Eqn. 67 and Eqn. 68 concludes the proof.
L
Theorem F.8 shows that under mild assumptions, the attention-based weights can help produce better estimates of O in the presence of noisy stop-words and therefore learns higher-quality topics from the corpus, improving both generalization performance and interpretability of PL.Ms.
Theorem F9 (Advantage of @ in the General Case), With Definition F.2 and Definition comparing , @g and O by evaluating them the marginal log- likelihood of non-stop-words, ie., LCLDA(:; w GT)), we have that
LCLDA - GT) S LCLDA - GT) S LOLDA(9G; GT).
(69)
Proof. First, by definition easily find that Og achieves the largest L(:; GT) among the three:
max(CenDA(0)::: - Ge.. GT)) s max COL.DA + GT) = COL.A GT).
(70)
Nm +.m. to X2 Next, we invoke Lemma F.5 by (1) setting Um (wm 1j Nm A1 and Um Wmj N=
respectively, and (2) rewriting E @mjif - (Cmm 1bg D (mjj (4) log (2 d/2
Σ 1/2]
as (z) for j E Um (1,2, ...,. Nn) and f2,( for j E Um (N + 1, Nm + 1,..., Nm N Nm,s)
where x corresponds to & (Ø. Pg. Σ 1x) By Lemma E5 we then have that
( GT) S LCLDA
- GT).
(71)
Note that because (i,(() and (., very close Gaussian, therefore Assumption 1 and in Lemma F.5 hold naturally under mild regularity conditions.
Combining Eqn. 70 and
and Eqn. 7I concludes the proof.
20


Reviews:
[{"review_id": "ub4W7SvfEQ", "reviewer": "Reviewer_UZmn", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose to apply Continuous LDA to pre-trained language models. The authors outline how to jointly train in a weighted loss function both the CLDA and the LM, making CLDA act as a regularizer for language models. In addition, CLDA can be used to interpret LMs, by using attention weights to group words at a concept and document level. The authors finally show that the CLDA actually improves the performance of LMs on the GLUE benchmark, particularly BERT-based ones.", "summary_of_the_review": "The method introduced in the paper proposes to train an LM jointly with CLDA, shows how to interpret the resulting CLDA that uses the LM's attention weights, and improves performance on the GLUE benchmark. These contributions are solid and benefit the community.", "strength_and_weaknesses": "Strength:\n- Novel method of training CLDA jointly with LMs\n- Interpreting method as a bonus, strong theoretical backing\n- Good results on GLUE benchmark\n\nWeaknesses:\n- No major weaknesses\n\nNotes:\n- Concept 67 references \"West Asia\" and not necessarily \"Islam\" as per the highlighted red words. For example, \"Pasha\" and \"Qaeda\" are not related to Islam.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ejG3QXgCO0", "reviewer": "Reviewer_YUbf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes adding topic model upon pretrained language models (PLMs), in order to increase its interpretability and downstream performance. More specifically, the authors design CLDA, a new LDA model compatible with continuous word counts, which are computed via aggregated attention scores of the last layer in PLM. Each topics/concept is modeled as Gaussian distribution in the contextual embedding space. The CLDA can be used for interpretation of the PLM outputs when decoupled with PLM training, or used for regularization when jointly trained with PLMs. Experiments shows that CLDA can distinguish topics from word level to document level, and improve several GLUE tasks with joint learning.", "summary_of_the_review": "The paper proposes a LDA topic model based on the PLM, which can serve as interpreter and regularizor of the PLM. The method is technically sound. However, the paper is lack of discussion about how the interpretation results can be used and evaluated. The experimental results are incomplete and not that convincing, which prevent further exploration of the proposed method.", "strength_and_weaknesses": "Strength: The proposed CLDA model is interesting and new as far as I know. The inference and learning algorithms are well designed.\n\nWeakness: The motivation of this paper is somehow weak to me. Why is it necessary to build LDA models upon the PLMs? What can we gain from the interpretation results such as those shown in Figure 4? How can we evaluate them? For example, if a PLM made wrong prediction on certain example, will the CLDA model give the interpretation and help to fix the error? The application scenarios of the CLDA (from the interpretation view) are completely missed in the paper.\nFor the quantitative results part, it is less discussed what makes the improvement over the baseline models. If it is the regularization effect, is it possible that the baseline models can be fine-tuned with finer but simpler regularization to achieve the similar results, considering that the bi-level optimization of PLM and CLDA is much more inefficient? \nBesides, why not include all the GLUE tasks rather than selecting part of them? And all the experiments and analysis are only tested in fine-tuning stage, it is unclear whether we can apply it in pretraining and what the results could be. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NLf8zMzRcd4", "reviewer": "Reviewer_iDsq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper extend conventional topic model methods to contextualized word embeddings from BERT-like pre-trained models. The authors propose several modifications including using attention weights as continuous word counts and modeling and considering a contextualized word embeddings to be drawn from a Gaussian distribution corresponding to its latent topic. The authors propose to use the modified topic model as either interpreter for improved interpretability and as a regulator for improved performance. ", "summary_of_the_review": "This paper extends topic models in PLMs and uses it as interpreter and regulator. However the experiments does not well support the main claims and the novelty of the technical contribution with respect to similar literature is unclear.", "strength_and_weaknesses": "Strengthes:\n1. The framework is technically sound and the adaptation of conventional topic model to pre-trained language models is interesting and can be helpful.\n2. The authors conduct both theoretical and empirical analysis of the proposed method.\n\nWeaknesses:\n1. The paper is not very well written and some parts of it are quite hard to follow/parse. Also, the citation style is strange and not reader-friendly.\n2. While the framework of  topic model with pre-trained language model is technically correct, the main claims of the paper (i.e., it can serve as good interpreter and regulator) are not well supported. First, it is unclear how finding the concepts and topics of different level will help users interpret the model's prediction. From this point of view, the proposed method is not as good as (or better) than existing interpretation methods for pre-trained models using either attention weights or shapley values. Second, for the regularization effect, the results on the GLUE benchmark in Table 1 is very poor compared to that in literature. For example, BERT-base can easily achieve around 57 for CoLA, 88 for MRPC in F1, and 66 in RTE, as well as on all other datasets. This is very different from the baseline results reported in Table 1. The experimental settings are also not very clearly described, making the effectiveness of the proposed method as regulator not convincing.\n3. There already exists several works on extending topic models with pre-trained models [1-2]. The authors do not discuss the contribution with respect to these works, making it hard to judge the actual technical contribution of the proposed framework.\n\n[1] BERTopic: Neural topic modeling with a class-based TF-IDF procedure\n[2] Topic Modeling with Contextualized Word Representation Clusters", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "HLaRpqo1nS", "reviewer": "Reviewer_c7kD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose to interpret contextual word embeddings by fitting a Gaussian mixture model with a Dirichlet prior over documents (CLDA) to the embeddings. They apply this framework to analyze the embeddings of fine-tuned language models on a number of tasks, and show that the model can be used as a regulator during fine-tuning.", "summary_of_the_review": "This paper proposes a novel interpretability approach using a Bayesian model fit to word embeddings. There are concerns about the experimental settings and intended use cases.", "strength_and_weaknesses": "Strengths:\n* The paper follows the interesting idea of fitting a hierarchical Bayesian model to contextual word embedding.\n* It features a detailed derivation of the probabilistic model and the variational inference algorithm.\n* The probabilistic model can be used as a regulator during fine-tuning.\n\nWeaknesses:\n* The fine-tuned BERT-base and RoBERTa- base baselines in Table 1 have consistently worse performance than the official figures from their respective papers, casting doubt on the experimental setting in this paper. Additionally, it is strange that their RoBERTa-base model performs so much worse for QQP than BERT-base.\n* The focus of the method is unclear: Are we looking for “interpretability” in the sense of (a) explaining a model’s decision boundary, (b) feature attribution for a particular model predictions, or (c) analyzing downstream corpora (e.g., like a topic model)? The authors seem to promise a mix of (a) and (b), but I can’t see how the inferred concepts help to give faithful or even plausible explanations. I would suggest that the authors feature a case study with user requirements & expectations about “interpretability” and then compare CLDA’s interpretations to the baselines. For example, claims of “improving readability and intuitiveness” need to be supported more explicitly by the experiments.\n* The authors should be clear about the target setting. The introduction and method only use the term pre-trained language models (PLMs), while in the experiments the method is applied to LMs fine-tuned on sentence classification datasets.\n* The paper should discuss the literature on attention-based explanations in BERT, see Bibal et al., 2022, “Is Attention Explanation? An Introduction to the Debate”, as well as other works that use contextualized embeddings for topic modeling, e.g., Grootendorst, 2022, “BERTopic: Neural topic modeling with a class-based TF-IDF procedure” and Bianchi et al., 2021, “Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence”.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject