DEIM2022 E41-3(day4 p10)

Topic Modeling using Jointly Fine-tuned BERT for Phrases andSentences
Zikai ZHOU† and Kei WAKABAYASHI††† College of Knowledge and Library Sciences, School of Informatics, University of Tsukuba
1–2 Kasuga, Tsukuba-shi, Ibaraki 355–8550, Japan†† Faculty of Library, Information and Media Science, University of Tsukuba
1–2 Kasuga, Tsukuba-shi, Ibaraki 355–8550, Japan
E-mail: †s1813024@s.tsukuba.ac.jp, ††kwakaba@slis.tsukuba.ac.jp
Abstract Using topic modeling to analyze large collections of documents has been a functional approach for many
years. However, traditional topic models such as Latent Dirichlet Allocation rely on the assumption of “Bag-of–
Words”, which ignores the connection and inner semantics between words in terms of phrases. Although phrases act
as important grammatical units in human language, due to the restriction on vocabulary size and model complexity,
less research has been conducted on phrase-level topic models. In this research, we propose a phrase-level topic
model based on pre-trained distributed representations of words, documents, and phrases. We build this model basedon the Top2vec topic model and BERT embeddings. Due to the fact that there is no existing BERT-based model
designed to produce embedding for both sentences and phrases, we propose a jointly ﬁne-tuned BERT model forsentences and phrases embeddings. Our experiments demonstrate that the jointly ﬁne-tuned BERT could produce
high-quality sentences and phrases embeddings, and the model could generate well-performed phrase-level topics.
Key words topic model, ﬁne-tuning, BERT, phrases, semantic space

1 Introduction

Autoregressive Distribution Estimation (DocNADE) [4] use
neural networks to generate more expressive topics.

Natural Language Processing (NLP) is a subﬁeld of com-

One of the major issues in topic modeling is handlingputer science concerned with the interactions between com-phrases. A phrase is a certain group of words that act to-puters and human language. Given an NLP model, we wantgether as a grammatical unit. In both syntax and grammar

the computer could somehow “understand” the context, andmeaning, the combination of certain words could lead to athen extract information to achieve tasks such as semantic

diﬀerent purpose. In general, phrases could carry more infor-

analysis or text classiﬁcation. One main problem of natural

mation than single words. Some phrases could have mean-

language processing is to deal with a large collection of text

ings that cannot be determined by any of their components,

which cannot be reasonably read and sorted by humans. We

and some words are even meaningless without related words.

may want to discover the theme or subject from a text, andIn topic modeling, phrases are likely to lead to ambiguity ifthen organize the whole collection automatically.

we only consider words as grammatical units. Words such

Topic models are designed to achieve this goal. A topic

as white and house appearance may lead to an architecture-

is considered as a latent semantic structure of a text, which

related document, while the phrase White House appear-

could tell us what is the text talking about. Topic models areance leads to a politics-related document. Many traditional

used to discover topics from a collection of documents andtopic models such as LDA are based on the popular assump-then be used to summarize documents, search documents bytion of Bag-of-Words (BOW), in which the ordering-based

queries or keywords, or categorize documents by topics. Tra-

semantics of words are ignored. There is some research suchditional unsupervised topic models are based on the Bayesianas Bigram Topic Model (BTM) [5], Topical N-gram Modelprobabilistic model. Models such as Latent Dirichlet Allo-(TNG) [6] and Phrase-based LDA [7] which consider phrases

cation (LDA) [1] and Probabilistic Latent Semantic Analysis

by simply discovering and adding phrases into the original(PLSA) [2] discover topics by ﬁnding statistical features ofvocabulary set, and learning those phrases just like normal

words. Some modern topic models such as Neural Varia-

words. This approach however would make an enormous vo-tional Document Model (NVDM) [3] and Document Neuralcabulary set, then lead to problems such as huge models andunstable learning. These problems become even more serious

sentences and phrases embeddings for the topic model, andon some well-performed neural-network-based models.

the model could generate better phrase-level topics.

An alternative way of topic modeling is to use distributed
representations of words and documents. Learning embed-

2 Related Workdings for representations of words and documents has been

2. 1 Topic models

a popular ﬁeld in NLP. This method is behind the idea of

LDA is a widely used generative statistical topic model in

distributional hypothesis, in which John Rupert Firth fa-

natural language processing that describes each document

mously said “You shall know a word by the company it

as a distribution of topics, and each topic as a distribu-

keeps” [8], indicates that similar words are often used intion of tokens. It is a generalized form of PLSA which adds

similar contexts. Word2vec [9] ﬁrstly introduced the con-a Dirichlet prior distribution on document-topic and topic-

tinuous skip-gram model and used it to capture distributed

word distributions. Based on the idea of topic distributions,word representations, while later models such as GloVe [10]diﬀerent generative topic models such as Correlated Topicand Bidirectional Encoder Representations from Transform-

Model (CTM) [17] and Structured Topic Model (STM) [18]ers (BERT) [11] produced even better word embeddings us-

are proposed. Some other approaches such as NVDM anding diﬀerent methods. Compared to traditional approaches,

DocNADE use modern neural networks to generate similar

each of them produced state-of-the-art results on many NLP

distributions. However, these models are all based on thetasks. Top2vec [12] introduced a new way of topic modeling

Bag-of-Words assumption, and therefore the ordering andby using jointly embedded documents and word vectors. Af-

combination of words are ignored.

ter creating semantic embeddings for both documents andBeyond the Bag-of-Words assumption, some research con-words, Top2vec applies dimensionality reduction and clus-

duct n-gram models with existing topic models. BTM ﬁrstly

tering to document embeddings and generates topic vectors

introduce bigram tokens into LDA models, while TNG use

in the same semantic space. This method provides beneﬁts

a combination of unigrams and bigrams for longer phrase

such as automatically ﬁnding the number of topics and ﬁnd-tokens. Due to the limitation of vocabulary size, these mod-ing more informative and representative topics of the corpus

els are limited to bigram models. Other models constrainover traditional topic models such as LDA and PSLA.

phrases to limit the size of the vocabulary. Yu et al. [7] con-The original paper of Top2vec chose Doc2vec [13] to embed

duct a C-value for phrase importance and apply a thresholddocuments and words into the same semantic space. How-

onto the topic model, while Li et al. [19] integrates a regu-

ever, using pre-trained encoding models such as Universal

lar expression constraint condition to ﬁlter phrases. Despite

Sentence Encoder (USE) [14] or BERT could also be help-

these models successfully introducing meaningful phrases

ful when it comes to eﬃciency and performance on small

into topic models, the constraint ignores the semantics ofdatasets.

phrases and may discard some informative phrases.

Pre-trained models are often trained on mas-sive datasets and could produce high-quality embeddings forTop2vec is a topic model which leverages joint document

downstream tasks. While USE is pre-trained to produce sen-

and word semantic embedding to ﬁnd topic vectors. By using

tence embeddings, BERT is only pre-trained for word embed-

distributed representations of words and documents, it over-dings originally. An alternation of BERT called Sentence-

comes the weaknesses of BOW representations of documents

BERT [15], which is ﬁne-tuned to encode sentences, could bein terms of latent semantics. While the original model learns

used instead to embed documents and words into the samea Doc2vec model for embeddings, the model also shows thesemantic space.

potential of being a fast model by directly using pre-trained

To conduct a phrase-level Top2vec model, phrases shouldmodels such as USE and BERT. Our proposed model is aalso be embedded into the same semantic space as well asphrase-level extension based on the Top2vec model using pre-words and documents. Phrase-BERT [16], which is based on

trained BERT models.

the Sentence-BERT architecture, is a ﬁne-tuned BERT-base

2. 2 Embeddings

model that shows state-of-art results on phrase embeddings.

Our work relates to the idea of learning dense representa-

However, after the ﬁne-tuning for phrases, the model lost its

tions for semantic units, particularly on the modern neural-ability to embed documents and create meaningful clusters.

network-based pre-trained models.

To solve this problem, we propose a joint ﬁne-tuned BERT

Skip-gram and C-BOW models for learning word embed-

model by training the BERT model with mixed datasets

dings, and further research [20] shows its potential for phrase

and conducting a phrase-level topic model which generates

embeddings. The idea is also implemented by diﬀerent mod-phrase-level topics based on semantics.

Our experiments

els such as GloVe and fastText [21]. Furthermore, Doc2vec,show that the jointly ﬁne-tuned BERT could produce both

USE, and Embeddings from Language Models (ELMo) [22]Word2vec introduced

Figure 1 Basic structure for BERT embeddings. Although not

Figure 2 Multipel Negative Ranking Loss with hard negatives.

shown in the graph, [CLS] and [SEP] tokens will be

As shown, p1 is pulling towards p1+ and pushing away

added to the begin and end of the sentence.

+−−−from p+
2 , p3 , p1 , p2 , and p3 .

use diﬀerent approaches for generating sentence embeddings.

meaningful and could be compared with cosine-similarity.

The advent of huge-scale pre-trained language models such

Consider a sentence or phrase as a semantic unit, the net-

as BERT opens a new way for word and sentence embed-

work is trained on training data tuple which contains an an-dings. It is shown that the pre-trained BERT model beneﬁts

chor unit p, a positive example p+ , and a negative example

many neural language processing tasks [23], while the modi-

p− . We assume that in general, p and p+ will have a similarﬁed Sentence-BERT shows its ability to produce high-quality

meaning, while p and p− will have a diﬀerent meaning. The

embeddings for unsupervised tasks such as text comparison.

model is encouraged to produce close embeddings for p andHowever, Yu and Ettinger [24] show that BERT struggles

p+ while pushing the embeddings for p and p− apart.

to produce meaningful embeddings for short semantic units

The network uses Multiple Negative Ranking Loss [26] assuch as words or phrases. Phrase-BERT proposes a phrase-

the loss function for the weight update. Given a single train-

speciﬁc version of BERT and shows that it produces mean-

ing data batch as

ingful phrase embeddings while also promoting a lexically diverse semantic space. To achieve our goal of generating joint

−
+
−
+
−
(p, p+ , p− ) = {(p1 , p+
1 , p1 ), (p2 , p2 , p2 ), ..., (pn , pn , pn )},document, phrase, and word embeddings for topic modeling,

we compute the similarity between two units, for example p

we propose a joint ﬁne-tuning process based on Sentence-and p′ , as S(p, p′ ). Speciﬁcally, the cosine-similarity between

BERT and Phrase-BERT.

two embeddings are used for the calculation of similarity.

3 Proposal

S(p, p′ ) = cos (E(p), E(p′ )) =E(p) · E(p′ )∥E(p)∥∥E(p′ )∥We propose a joint ﬁne-tuning task on top of BERT for
both sentences and phrases, which relies on contrastive objectives similar to Sentence-BERT. The training dataset contains triplets for both sentence and phrase.
3. 1 BERT Embeddings
For sentence and phrases embeddings, we follow the procedure of the Sentence-BERT model [15], which takes the meanof all output vectors as the ﬁnal embedding. Given an input
X of length N tokens, by adding a mean pooling layer on
top of the BERT model, the ﬁnal representation E(X) could
be computed as:

examples p+| i) as negative. In addition, since we usej (j =
triplet for training, all negative example p−n would also be
considered as negative. The embedding will be pulling towards embedding of the positive example and pushing awayfrom embeddings of negative examples, as shown in Figure
2. The cross-entropy loss would then be computed as:
J(p, p+ , p− ) = −

n
n
∑
+
1∑
[S(pi , p+eS(pi ,pj )i ) − log
n i=1
j=1,j =i|∑N
E(X) =For the eﬃciency of the model, we treat the corresponding positive example p+i as positive, and all other positive

i=1 BERT(Xi )

N

− log

,

where BERT(Xi ) indicates the ﬁnal-layer token-level output
for token Xi . This structure is depicted in Figure 1.
3. 2 Fine-tuning Approach
We use the triplet network [25] to update the model
weights so that the embeddings produced by the model aren∑−eS(pi ,pk ) ],k=1

and would be minimized through back propagation.
3. 3 Sentence Training Data
Following the Sentence-BERT [15] settings, the sentence
training data is the combination of the SNLI [27] and theMulti-Genre NLI [28] dataset. These two datasets containsentence pairs and one of the three labels: entailment, neutral, and contradiction.

The pairs labeled as entailment

would be considered as positive pairs and the contradiction

Dataset Example pairSTSB

3. 4 Phrase Training Data
Following the Phrase-BERT [16] settings, the phrase train-

5.0

Birdie is washing itself in the water basin.)

Turneywould be added as hard negatives.

Score(The bird is bathing in the sink.,(learned person, pundit)

match

BiRD
(business development, economic growth)
0.586
Table 1 Datasets and example pairs used in our embedding evaluation.

ing data contains two diﬀerent objectives for high-quality
phrase embeddings.
The ﬁrst objective mainly aims at lexically diverse
phrasal paraphrases. Top 100K phrases are extracted fromthe WikiText-103 Corpus [29] using the SR-parser from
CoreNLP [30]. Given the phrase p, positive example p+ iscreated by passing p through the GPT2-based diverse paraphrasing model [31] and decoding the using nucleus samplingwith the nucleus probability mass of 0.8 [32] with lexical constraints. Negative sample p− is created by passing a randomly sampled phrase to the paraphraser.
The second objective mainly aims at phrases in context.
Top 100K phrases of length less than 10 tokens are extracted
from the Books3 Corpus [33] along with its context of length
120 tokens. Given the phrase p, positive example p+ is created by replacing the occurrence of p within the context witha [MASK] token, while p− is a randomly sampled context fromthe corpus.

[36].

Following the linear warm-up setup of Sentence-

BERT [15], the initial 10% of training steps are used as
warm-up steps.

For comparison, three models are sepa-

rately trained based on BERT, Sentence-BERT, and PhraseBERT with the same training setups and datasets, named
as joint-BERT-based, joint-Sentence-BERT-based, andjoint-Phrase-BERT-based respectively.

For phrase-level

topic model, following the default setup [12], we use the
UMAP as the dimensionality reduction tool and HDBSCAN
as the clustering tool based on Python implementations.
For UMAP parameters [37], we use n neighbors=15 andn components=25. For HDBSCAN [38], we use leaf as thecluster selection method.

4 Experiment
In this part, we show the performance of our proposed

3. 5 Topic Model
Our proposed phrase-level topic model is based on Top2vec[12] model structure. The Top2vec model generates topic,
document, and word vectors that are all jointly embedded,
with the distance between them representing semantic similarity. While the origin Top2vec model uses Doc2vec embeddings which are trained from the corpus, our proposed model
uses a pre-trained BERT model for embeddings. Here is thebasic procedures of the phrase-level topic model.
（ 1 ） Mine NP, VP, ADJP, and ADVP phrases of length

between 2 to 10 words through CoreNLP SR-parser.
（ 2 ） Create embeddings for words, phrases, and docu-

ments using pre-trained BERT models.
（ 3 ） Reduce dimensions of document embeddings using

Uniform Manifold Approximation and Projection for Dimension Reduction (UMAP) [34].
（ 4 ） Find clusters of document embeddings using Hierar-

chical Density-Based Spatial Clustering of Applications with
Noise (HDBSCAN) [35].
（ 5 ） Find centroid vectors of document clusters in origi-

nal dimension as topic vectors.
（ 6 ） Assign n-closest words to the topic vectors as topic

words.

models through several experiments. We compare our threemodels with BERT, Sentence-BERT, and Phrase-BERT as
baselines. Since the research [15] shows that using the meanpooled representation over the ﬁnal-layer outputs outperforms other methods such as using the [CLS] representation,a mean-pooling layer is added to the original BERT model(other models have already implemented this feature).
4. 1 Embedding Evaluation
We evaluate both sentence and phrase embeddings on different tasks (Table 1).
For sentence embeddings, we use the common Semantic Textual Similarity (STS) tasks, in speciﬁc the STSbenchmark [39] dataset. STS datasets provide a series ofsentence pairs and labels between 0 and 5 on their semantic relatedness. Following the setup of Sentence-BERT [15],with no STS-speciﬁc training, the Spearman’s rank correlations between the cosine-similarity of the sentence embeddings and the gold labels are computed for comparison.
For phrase embeddings, two diﬀerent phrase-level semantic relatedness tasks are used following previous works on
evaluating phrase embeddings. Turney [40] contains groups
of ﬁve unigrams and a corresponding bigram. The modelis asked to tell which of the ﬁve unigrams has the closest

3. 6 Implementation Details
We ﬁne-tune the joint model on two NVIDIA GTX
1080ti GPU parallelly for 1 epoch.

is used along with a learning rate of 2e−5 with Adam

A batch size of 16

meaning with the given bigram. The cosine-similarity of unigram and bigram embeddings is computed and compared toModelSTSB

TurneyBiRD

Baseline Models

Topic

Model

NumberCoherencePhraseCoherence

BERT

0.47290.42610.4437

Baseline Models

Sentence-BERT

0.85050.51830.6869BERT

10-586.87

Phrase-BERT

0.77820.57200.6880Sentence-BERT

85-549.60-135.71

Phrase-BERT

11-523.00-109.5390-504.12-128.2788-494.33-128.84

Proposed Models
joint-BERT-based

0.82200.57890.7086

Proposed Models

joint-Sentence-BERT-based

0.82740.58030.7063

joint-BERT-based

joint-Phrase-BERT-based
0.8316
0.56380.7043
Table 2 Results on embedding evaluation.

joint-SentenceBERT-based-138.38

joint-Phrase-ﬁnd the answer. The ﬁnal accuracy is used for comparison.
BiRD [41] is a correlation task consisting of pairs of bigramphrases and a human-rated similarity between 0 and 1. The

91-494.13-131.08
BERT-based
Table 3 Results on topic evaluation. For coherence score, top 20
words of each topic are used. For phrase coherence, top
10 phrases of each topic are used.

Pearson correlation coeﬃcient between the gold label and thecosine-similarity of the bigrams is computed for comparison.
The results are shown in Table 2. For the STS sentence

T = {t1 , t2 , ..., tN }, the UMass topic coherence for topic t
could be computed as follow:

embedding evaluation, the best score among the joint modelis 0.8316, while the Sentence-BERT produces the overall bestC(t; V (t) ) =score of 0.8505. As the performance is slightly worse com-

M m−1
∑
∑

(t)

log

m=2 l=1(t)

D(vm , vl ) + 1(t)
D(vl ),

pared to the Sentence-BERT, all three joint models give awhere V (t) = {v1 , v2 , ..., vM } denotes a list of the top M

signiﬁcant improvement compared to the Phrase-BERT scorewords for topic t, D(v) denotes the document frequency ofof 0.7782.

For the phrase level Turney task, two of thethe word v (i.e., the number of documents which v appears at

joint models produce higher scores compared to the base-least once), and D(v, v ′ ) denotes the co-document frequency

line phrase model Phrase-BERT, given that the best score isof the words v and v ′ (i.e., the number of documents which

0.5803 compared to the Phrase-BERT score of 0.5720. For

both v and v ′ appear at least once). To prevent calculat-

the phrase correlation task of BiRD, all three joint models

ing the logarithm of 0, a smoothing count of 1 is provided.

produce signiﬁcantly higher scores compared to the base-Since more related words appear in the same topic means

lines, given that the best score is 0.7086 compared to thebetter the topic concludes a subset of documents, a higher

best baseline score of 0.6880.

coherence score refers to higher topic quality.

4. 2 Topic Model Evaluation
Following the procedure of section 3.5, we construct the

(t)

(t)

The average of each topic’s coherence would be served as
the ﬁnal coherence score of the topic model:

phrase-level topic model using three baseline models andthree proposed joint BERT models. We choose the wellknown 20 News Groups Dataset [42] as the training datasets

(t)

1 ∑
C(ti ; V (ti ) ),N i=1
NC(T ; V ) =While the 20 News Groups

where N is the number of topics and is automatically foundDataset contains 18,831 posts with labels they are posted in,by the topic model, thus models that produce diﬀerent num-

none of the gold labels are used in the training of the topic

bers of topics could be compared.

for topic model evaluation.

model. The phrases are extracted in advance and added to

In the experiment, we employ the top 20 topic words which

the vocabulary. In addition, we set frequency thresholds forcontain phrases for each topic. The overall coherence scores

the frequency of words and phrases in order to avoid unstable

along with the topic number found by each model are dis-results. While documents would be directly embedded by theplayed in Table 3. In addition, in order to see how wellembedding model, only words appearing more than 25 times

phrases alone perform in topics, we calculate the coherence

and phrases appearing more than 10 times are embedded forscores for the top 10 phrases for each topic as well. For

topic word representation.

topic numbers generated by each model, the BERT and the

For topic evaluation, we use UMass topic coherence [43]Phrase-BERT model produce signiﬁcantly lower numbers as

metric. It is shown that UMass topic coherence conducts

10˜, while the Sentence-BERT and all three joint BERT

similar results with the human rating for topic quality, andmodels produce more than 80 topics. For the overall coher-

implements the idea of similar words are often used in sim-ence score, the baseline models give the best score of -523.00

ilar contexts. Given topic t belongs to the set of all topics

produced by Phrase-BERT. While all three joint models produce better scores compared to baseline models, the bestModelOverlap Ratefact that it mainly targets phrase embeddings. Nevertheless,

BERT

0.024

the joint model outperforms other baseline models, proving

Sentence-BERT

0.034

that it successfully obtain the ability to produce joint embed-

Phrase-BERT

0.019

dings for documents and phrases through joint ﬁne-tuning.

joint-BERT-based

0.029

Overall, the results meet our expectations on the jointly ﬁne-joint-Sentence-BERT-based

0.034

joint-Phrase-BERT-based
0.030
Table 4 Lexical overlap rates for each model in top 20 topic
words.

tuned BERT models.
In addition, we take a deeper look at our proposed topic
model. One of the big diﬀerences between other phrase-level
topic models [5] [6] [7] [19] and our proposed model is thatscore given by the proposed models is -494.13. For the co-instead of tokenized corpus to the combination of phrases

herence score of the top 10 topic phrases, the best score is

and words, our proposed model directly mines phrases andproduced by the Phrase-BERT model of -109.53. The joint

adding them into the vocabulary. While having the bene-

models outperform the BERT and the Sentence-BERT modelﬁt of not restricting the type and form of phrases, this ap-

with the best score of -128.27.

proach does bring the concern of lexical overlaps in words
and phrases. Since we use the UMass coherence score to

5 Discussion

evaluate topics that involve the counting of co-document ap-

According to the result, we show that our joint model

pearance, there is the possibility that more lexical overlaps in

could produce high-quality embeddings for both sentences

topic words produce a better coherence score. Therefore, we

and phrases. Since STS data is included in the training data

calculate the percentage of overlapping word pairs in all top-

for Sentence-BERT, it yields the best score among all mod-ics, which is shown in Table 4. The results show that basedels. Phrase-BERT loses some of its ability to produce sen-on the structure of mean-pooling BERT outputs, there aretence embeddings since there is no sentence-targeting data

actually few lexical overlapping appearances in topic words,

involved in its training dataset. Besides, by adding NLI data

which brings minimal inﬂuence to the ﬁnal result.

to the training dataset, the proposed joint model success-One of the goals of our proposed topic model is to build afully maintains its ability to produce sentence embeddings,

phrase-level topic model which relies on semantic similarity.

no matter which base model is chosen for the last-step ﬁne-

Although it is proved that BERT-based models have advan-

tuning. For phrase embeddings, the joint models outper-

tages in terms of semantic interpretation [11], we still wantform not only Sentence-BERT but also Phrase-BERT, which

to know how well they perform in the phrase-level topics.

means that the sentence training data may also improve theThus, we generate Wordclouds for the top 20 topic words

quality of phrase embeddings when the correct proportionof some presentive topics, and the result is shown in Fig-is mixed with phrase training data. Speciﬁcally, ﬁne-tuning

ure 3. It is easy to tell that these topics relate to religion,

based on BERT or Sentence-BERT seems to produce bettermedicine, cryptology, sports, politics, and spaceﬂight respec-

results compare to Phrase-BERT.

tively. Proper nouns such as NASA, NHL, NSA and phrases

Furthermore, the topic evaluation results show that thesuch as the White House, the Stanley Cup, copy protection,

joint model does improve topic quality in terms of phrase-

the space program are successfully embedded into the correct

level topic modeling. Since the topic model conducts cluster-

topics, meaning that the joint model captures the semantics

ing on document embeddings, the model’s ability of embed-of those words and utilizes them into the topic model, whichding sentences is highly related to the topic number. Due to

traditional statistical topic models failed to achieve.

the fact that BERT and Phrase-BERT have not conducted

The future challenges for our research focus on improving

speciﬁc training for sentences embeddings, the document em-

the quality and enhancing the topic model structure. Since

beddings are mixed together, thus few topics are discovered

the main goal of our proposed joint model is to embed sen-by the model. Sentence-BERT as well as the proposed mod-tences, words, and phrases into the same semantic space, we

els on the other hand produce quality sentence embeddings

choose the joint ﬁne-tuning approach based on two previous

for clustering and generate reasonable numbers of topics fromworks [15] [16]. There are other approaches worth trying suchthe corpus. The coherence scores show that under similar cir-as using diﬀerent tagging tokens for phrases and words, train-

cumstances, the joint models outperform Sentence-BERT in

ing the model with more and better datasets that oﬀer mean-terms of topic quality. We also observe the relationship be-

ingful hard negatives, and using advanced base models such

tween phrases and topics with phrase coherence scores. The

as RoBERTa [44] and MPNet [45]. Besides topic modeling,

results show that Phrase-BERT has an advantage due to thethere are still plenty of possibilities for the joint BERT-based
language model. For topic modeling, we expect the proposedmodel ﬁnds topics based on the semantics of documents,
phrases, and words. Furthermore, by conducting topics from
diﬀerent kinds of BERT-based models and comparing the result on topic coherence, we showed that our proposed topic
model successfully generated phrase-level topics and outperformed other baseline models.

References

Figure 3 Wordclouds for top 20 topic words of topic No.8, No.24,
No.35, No.46, No.55, No.58 from joint-BERT-based
topic model.

model to show eﬀectiveness on datasets other than 20 NewsGroups Dataset. Moreover, the problem of unstable topic
numbers remains. Although the model could automaticallyﬁnd the topic number, the number is still related to hyperparameters of HDBSCAN and UMAP. The future challenge
would be using prior knowledge or topic-speciﬁc ﬁne-tuning
to enhance our topic models.

6 Conclusion
In this research, we proposed a jointly ﬁne-tuned BERT
model for phrase and sentence embeddings and utilized it ona phrase-level topic model. We showed that previous models
focus on either sentence or phrase embeddings, which is not
optimal for topic models based on the semantics of these different grammatical units. Our proposed model took advantage of the joint ﬁne-tuning process and performed well on
both sentence and phrase embedding evaluations. Diﬀerent
from traditional statistical topic models, the proposed topic[1] David M Blei, Andrew Y Ng, and Michael I Jordan. Latent dirichlet allocation. the Journal of machine Learning
research, Vol. 3, pp. 993–1022, 2003.
[2] Thomas Hofmann. Probabilistic latent semantic indexing.
In Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, SIGIR ’99, pp. 50–57, New York, NY, USA,
August 1999. Association for Computing Machinery.
[3] Yishu Miao, Lei Yu, and Phil Blunsom. Neural variational
inference for text processing. November 2015.
[4] Stanislas Lauly Hugo Larochelle. A neural autoregressive
topic model. In Advances in Neural Information Processing Systems 25.
[5] Hanna M Wallach. Topic modeling: beyond bag-of-words.
In Proceedings of the 23rd international conference on Machine learning, ICML ’06, pp. 977–984, New York, NY,
USA, June 2006. Association for Computing Machinery.
[6] Xuerui Wang, Andrew McCallum, and Xing Wei. Topical
N-Grams: Phrase and topic discovery, with an application
to information retrieval. In Seventh IEEE International
Conference on Data Mining (ICDM 2007), pp. 697–702.
ieeexplore.ieee.org, October 2007.
[7] Zhiguo Yu, Todd R Johnson, and Ramakanth Kavuluru.
Phrase based topic modeling for semantic information processing in biomedicine. Proc. Int. Conf. Mach. Learn. Appl.,
Vol. 2013, pp. 440–445, December 2013.
[8] Henry Widdowson. J.R. ﬁrth, 1957, papers in linguistics
1934–51. Vigo Int. J. Appl. Linguist., Vol. 17, No. 3, pp.
402–413, November 2007.
[9] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeﬀrey Dean.
Eﬃcient estimation of word representations in vector space.
January 2013.
[10] Jeﬀrey Pennington, Richard Socher, and Christopher D
Manning. Glove: Global vectors for word representation.
In Proceedings of the 2014 conference on empirical methods
in natural language processing (EMNLP), pp. 1532–1543.
aclweb.org, 2014.
[11] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina
Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. October 2018.
[12] Dimo Angelov. Top2Vec: Distributed representations of
topics. August 2020.
[13] Quoc V Le and Tomas Mikolov. Distributed representationsof sentences and documents. May 2014.
[14] Daniel Cer, Yinfei Yang, Sheng-Yi Kong, Nan Hua,
Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario
Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan
Sung, Brian Strope, and Ray Kurzweil. Universal sentence
encoder. March 2018.
[15] Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using siamese BERT-Networks. August
2019.
[16] Shufan Wang, Laure Thompson, and Mohit Iyyer. PhraseBERT: Improved phrase embeddings from BERT with anapplication to corpus exploration. September 2021.
[17] David Blei and John Laﬀerty. Correlated topic models. Adv.

Neural Inf. Process. Syst., Vol. 18, p. 147, 2006.
[18] Lan Du, Wray Buntine, and Mark Johnson. Topic segmentation with a structured topic model. In Proceedings of the2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language
Technologies, pp. 190–200. aclweb.org, 2013.
[19] Baoji Li, Wenhua Xu, Yuhui Tian, and Juan Chen. A phrase
topic model for large-scale corpus. In 2019 IEEE 4th International Conference on Cloud Computing and Big Data
Analysis (ICCCBDA), pp. 634–639, April 2019.
[20] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado,and Jeﬀrey Dean. Distributed representations of words andphrases and their compositionality. October 2013.
[21] Piotr Bojanowski, Edouard Grave, Armand Joulin, andTomas Mikolov. Enriching word vectors with subword information. Transactions of the Association for Computational
Linguistics, Vol. 5, pp. 135–146, 2017.
[22] Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt
Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of NAACL-HLT, pp. 2227–2237. cs.ubc.ca, 2018.
[23] Yaru Hao, Li Dong, Furu Wei, and Ke Xu. Visualizing andunderstanding the eﬀectiveness of BERT. August 2019.
[24] Lang Yu and Allyson Ettinger. Assessing phrasal representation and composition in transformers. October 2020.
[25] Florian Schroﬀ, Dmitry Kalenichenko, and James Philbin.
Facenet: A uniﬁed embedding for face recognition andclustering. In Proceedings of the IEEE conference on
computer vision and pattern recognition, pp. 815–823. cvfoundation.org, 2015.
[26] Matthew Henderson, Rami Al-Rfou, B Strope, Yun-Hsuan
Sung, László Lukács, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and R Kurzweil. Eﬃcient natural language response
suggestion for smart reply. ArXiv, 2017.
[27] Samuel R Bowman, Gabor Angeli, Christopher Potts, andChristopher D Manning. A large annotated corpus forlearning natural language inference. In Proceedings of the2015 Conference on Empirical Methods in Natural Language Processing, pp. 632–642, Lisbon, Portugal, September
2015. Association for Computational Linguistics.
[28] Adina Williams, Nikita Nangia, and Samuel Bowman. A
Broad-Coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1112–1122, New Orleans,Louisiana, June 2018. Association for Computational Linguistics.
[29] Stephen Merity, Caiming Xiong, James Bradbury, andRichard Socher. Pointer sentinel mixture models. September 2016.
[30] Christopher D Manning, Mihai Surdeanu, John Bauer,
Jenny Rose Finkel, Steven Bethard, and David McClosky.
The stanford CoreNLP natural language processing toolkit.
In Proceedings of 52nd annual meeting of the association
for computational linguistics: system demonstrations, pp.
55–60. aclweb.org, 2014.
[31] Kalpesh Krishna, John Wieting, and Mohit Iyyer. Reformulating unsupervised style transfer as paraphrase generation.
October 2020.
[32] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin
Choi. The curious case of neural text degeneration. April
2019.
[33] Leo Gao, Stella Biderman, Sid Black, Laurence Golding,Travis Hoppe, Charles Foster, Jason Phang, Horace He,
Anish Thite, Noa Nabeshima, Shawn Presser, and Connor
Leahy. The pile: An 800GB dataset of diverse text for language modeling. December 2020.

[34] Leland McInnes, John Healy, and James Melville. UMAP:
Uniform manifold approximation and projection for dimension reduction. February 2018.
[35] Leland McInnes, John Healy, and Steve Astels. hdbscan: Hierarchical density based clustering. J. Open Source
Softw., Vol. 2, No. 11, p. 205, March 2017.
[36] Diederik P Kingma and Jimmy Ba. Adam: A method forstochastic optimization. December 2014.
[37] Umap: Uniform manifold approximation and projection for
dimension reduction. https://umap-learn.readthedocs.
io/en/latest/. Accessed: 2022-01-10.
[38] The hdbscan clustering library.
https://hdbscan.
readthedocs.io/en/latest/index.html. Accessed: 202201-10.
[39] Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio,
and Lucia Specia. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop
on Semantic Evaluation (SemEval-2017), Stroudsburg, PA,
USA, 2017. Association for Computational Linguistics.
[40] P D Turney. Domain and function: A Dual-Space model ofsemantic relations and compositions. J. Artif. Intell. Res.,
Vol. 44, pp. 533–585, July 2012.
[41] Shima Asaadi, Saif Mohammad, and Svetlana Kiritchenko.
Big BiRD: A large, ﬁne-grained, bigram relatedness dataset
for examining semantic composition. In Proceedings of the2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 1 (Long and Short Papers), pp. 505–
516. aclweb.org, 2019.
[42] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort,
Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu
Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg,and Others. Scikit-learn: Machine learning in python. theJournal of machine Learning research, Vol. 12, pp. 2825–
2830, 2011.
[43] David Mimno, Hanna Wallach, Edmund Talley, Miriam
Leenders, and Andrew McCallum. Optimizing semantic coherence in topic models. In Proceedings of the 2011 conference on empirical methods in natural language processing,pp. 262–272. aclweb.org, 2011.
[44] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar
Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. July 2019.
[45] Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan
Liu. MPNet: Masked and permuted pre-training for language understanding. April 2020.