Venue: NAACL 2022

# Seed-Guided Topic Discovery with Out-of-Vocabulary Seeds

 Yu Zhang1, Yu Meng1, Xuan Wang1, Sheng Wang2, Jiawei Han1

###### Abstract

Discovering latent topics from text corpora has been studied for decades. Many existing topic models adopt a fully unsupervised setting, and their discovered topics may not cater to users' particular interests due to their inability of leveraging user guidance. Although there exist seed-guided topic discovery approaches that leverage user-provided seeds to discover topic-representative terms, they are less concerned with two factors: (1) the existence of out-of-vocabulary seeds and (2) the power of pre-trained language models (PLMs). In this paper, we generalize the task of seed-guided topic discovery to allow out-of-vocabulary seeds. We propose a novel framework, named SeeTopic, wherein the general knowledge of PLMs and the local semantics learned from the input corpus can mutually benefit each other. Experiments on three real datasets from different domains demonstrate the effectiveness of SeeTopic in terms of topic coherence, accuracy, and diversity.1

Footnote 1: The code and datasets are available at [https://github.com/yuzhimanhua/SeeTopic](https://github.com/yuzhimanhua/SeeTopic).

+
Footnote †: _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 291 - 302

July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

Automatically discovering informative and coherent topics from massive text corpora is central to text analysis through helping users efficiently digest a large collection of documents Griffiths and Steyvers (2004) and advancing downstream applications such as summarization Wang et al. (2009, 2022), classification Chen et al. (2015); Meng et al. (2020b), and generation Liu et al. (2021).

Unsupervised topic models have been the mainstream approach to topic discovery since the proposal of pLSA Hofmann (1999) and LDA Blei et al. (2003). Despite their encouraging performance in finding informative latent topics, these topics may not reflect user preferences well, mainly due to their unsupervised nature. For example, given a collection of product reviews, a user may be specifically interested in product categories (e.g., "_books_", "_electronics_"), but unsupervised topic models may generate topics containing different sentiments (e.g., "_good_", "_bad_"). To consider users' interests and needs, seed-guided topic discovery approaches Jagarlamudi et al. (2012); Gallagher et al. (2017); Meng et al. (2020a) have been proposed to find representative terms for each category based on user-provided seeds or category names.2 However, there are still two less concerned factors in these approaches.

Footnote 2: In this paper, we use “seeds” and “category names” interchangeably.

**The Existence of Out-of-Vocabulary Seeds.** Previous studies Jagarlamudi et al. (2012); Gallagher et al. (2017); Meng et al. (2020a) assume that all user-provided seeds must be **in-vocabulary** (i.e., appear at least once in the input corpus), so that they can utilize the occurrence statistics or SkipGram embedding methods Mikolov et al. (2013) to model seed semantics. However, user-interested categories can have specific or composite descriptions, which may never appear in the corpus. Table 1 shows three datasets fro

\begin{table}
\begin{tabular}{p{86.7pt}|p{113.8pt} p{113.8pt}} \hline
**Dataset** & \multicolumn{2}{c}{**Category Names (Seeds)**} \\ \cline{2-3}  & cardiovascular diseases & hepatitis th/o/e \\
**SolDes** & chronic respiratory diseases & mental disorders \\
**Scientific Papers** & diabetes mellitus & musculoskeletal disorders \\
**Larsson** & digestive diseases & neoplasms (cancer) \\
**Amazon** (Product Reviews) & apps fed android beds eds and vinyl clds and vinyl cldethins, shoes and jeverty and editors video games \\ \hline
**Twitter** & food shop and service travel and transport poltesfets & outdoors and recreation arts and entertainment professional and other places \\
**Media Posts** & **the** nightlife spot & \\ \hline \end{tabular}
\end{table}
Table 1: Three datasets Cohan et al. (2020); McAuley and Leskovec (2013); Zhang et al. (2017) from different domains and their topic categories (i.e., seeds). Red: Seeds never seen in the corpus (i.e., out-of-vocabulary). In all three datasets, a large proportion of seeds are out-of-vocabulary.



entific papers, product reviews, and social media posts. In each dataset, documents can belong to one or more categories, and we list the category names provided by the dataset collectors. These seeds should reflect their particular interests. In all three datasets, we have a large proportion of seeds (45% in SciDocs, 60% in Amazon, and 78% in Twitter) that never appear in the corpus. Some category names are too specific (e.g., "_chronic respiratory diseases_", "_nightlife spot_") to be exactly matched, others are the composition of multiple entities (e.g., "_hepatitis ab/c/e_", "_neoplasms (cancer)_", "_clothing, shoes and jewelry_").3

Footnote 3: One possible idea to deal with composite seeds is to split them into multiple seeds. However, there are many possible ways to express the conjunctions (e.g., “\(i\)”, “\(j\)”, ”\(,\cdot\)” and “_and_” in Table 1), which may require manual tuning. Besides, simple chunking rules will induce splits that break the semantics of the original composition (e.g., “_professional and other places_” may be split into “_professional_” and “_other places_”). Moreover, even after the split, some seeds are still out-of-vocabulary. Therefore, we propose to use PLMs to tackle out-of-vocabulary seeds in a unified way. In experiments, we will show that our model is able to tackle composite seeds. For example, given the seed “_hepatitis ab/b/c/e_”, we can find terms relevant to “_hepatitis b_” and “_hepatitis c_” (see Table 4).

**The Power of Pre-trained Language Models. Techniques used in previous studies are mainly based on LDA variants Jagarlamudi et al. (2012) or context-free embeddings Meng et al. (2020). Recently, pre-trained language models (PLMs) such as BERT Devlin et al. (2019) have achieved significant improvement in a wide range of text mining tasks. In topic discovery, the generic representation power of PLMs learned from web-scale corpora (e.g., Wikipedia or PubMed) may complement the information a model can obtain from the input corpus. Moreover, out-of-vocabulary seeds usually have meaningful in-vocabulary components (e.g., "_night_" and "_life_" in "_nightlife spot_", "_health_" and "_care_" in "_health and personal care_"). The optimized tokenization strategy of PLMs Sennrich et al. (2016); Wu et al. (2016) can help segment the seeds into such meaningful components (e.g., "_nightlife_" \(\rightarrow\) "_night_" and "_##life_"), and the contextualization power of PLMs can help infer the correct meaning of each component (e.g., "_#life_" and "_care_") in the category name. Therefore, PLMs are much needed in handling out-of-vocabulary seeds and effectively learning their semantics.

**Contributions.** Being aware of these two factors, in this paper, we study seed-guided topic discovery in the presence of out-of-vocabulary seeds. Our proposed SeeTopic framework consists of two modules: (1) The _general_ representation module uses a PLM to derive the representation of each term (including out-of-vocabulary seeds) based on the general linguistic knowledge acquired through pre-training. (2) The _seed-guided local_ representation module learns in-vocabulary term embeddings specific to the input corpus and the given seeds. In order to optimize the learned representations for topic coherence, which is commonly reflected by pointwise mutual information (PMI) Newman et al. (2010), our objective implicitly maximizes the PMI between each word and its context, the documents it appears, as well as the category it belongs to. The learning of the two modules is connected through an iterative ensemble ranking process, in which the general knowledge of PLMs and the term representations specifically learned from the target corpus conditioned on the seeds can complement each other.

To summarize, this study makes three contributions. (1) _Task_: we propose to study seed-guided topic discovery in the presence of out-of-vocabulary seeds. (2) _Framework_: we design a unified framework that jointly models general knowledge through PLMs and local corpus statistics through embedding learning. (3) _Experiment_: extensive experiments on three datasets demonstrate the effectiveness of SeeTopic in terms of topic coherence, accuracy, and diversity.

## 2 Problem Definition

As shown in Table 1, we assume a seed can be either a single word or a phrase. Given a corpus \({\cal D}\), we use \({\cal V}_{\cal D}\) to denote the set of **terms** appearing in \({\cal D}\). In accordance with the assumption of category names, each term can also be a single word or a phrase. In practice, given a raw corpus, one can use existing phrase Chunking tools Manning et al. (2014); Shang et al. (2018) to detect phrases in it. _After_ phrase chunking, if a category name is still not in \({\cal V}_{\cal D}\), we define it as **out-of-vocabulary**.

**Problem Definition.**_Given a corpus \({\cal D}=\{d_{1},...,d_{|{\cal D}|}\}\) and a set of category names \({\cal C}=\{c_{1},...,c_{|{\cal C}|}\}\) where some category names are out-of-vocabulary, the task is to find a set of invocabulary terms \({\cal S}_{i}=\{w_{1},...,w_{S}\}\subseteq{\cal V}_{\cal D}\) for each category \(c_{i}\) such that each term in \({\cal S}_{i}\) is semantically close to \(c_{i}\) and far from other categories \(c_{j}~{}(\forall j\neq i)\)._

## 3 The SeeTopic Framework

In this section, we first introduce how we model general and local text semantics using a PLM mod 

ule and a seed-guided embedding learning module, respectively. Then, we present the iterative ensemble ranking process and our overall framework.

### Modeling General Text Semantics using a PLM

PLMs such as BERT (Devlin et al., 2019) aim to learn generic language representations from web-scale corpora (e.g., Wikipedia or PubMed) that can be applied to a wide variety of text-related applications. To transfer such general knowledge to our topic discovery task, we employ a PLM to encode each category name and each in-vocabulary term to a vector. To be specific, given a term \(w\in{\cal C}\cup\mathcal{V_{D}}\), we input the sequence "[CLS] \(w\)[SEP]" into the PLM. Here, \(w\) can be a phrase containing multiple words, and each word can be out of the PLM's vocabulary. To deal with this, most PLMs use a pre-trained tokenizer (Sennrich et al., 2016; Wu et al., 2016) to segment each unseen word into frequent subwords. Then, the contextualization power of PLMs will help infer the correct meaning of each word/subword, so as to provide a more precise representation of the whole category.

After LM encoding, following (Sia et al., 2020; Thompson and Mimno, 2020; Li et al., 2020), we take the output of all tokens from the last layer and average them to get the term embedding \(\mathbf{e}_{w}\). In this way, even if a seed \(c_{i}\) is **out-of-vocabulary**, we can still obtain its representation \(\mathbf{e}_{c_{i}}\).

### Modeling Local Text Semantics in the Input Corpus

The motivation of topic discovery is to discover latent topic structures from the input corpus. Therefore, purely relying on general knowledge in the PLM is insufficient because topic discovery results should adapt to the input corpus \(\mathcal{D}\). Now, we introduce how we learn another set of embeddings \(\{\mathbf{u}_{w}|w\in\mathcal{V_{D}}\}\) from \(\mathcal{D}\).

Previous studies on embedding learning assume that the semantic of a term is similar to its local context (Mikolov et al., 2013), the document it appears (Tang et al., 2015; Xun et al., 2017a), and the category it belongs to (Meng et al., 2020). Inspired by these studies, we propose the following embedding learning objective.

\[\begin{split}\mathcal{J}=&\underbrace{\sum_{d\in \mathcal{D}}\sum_{w_{i}\in d}\sum_{w_{j}\in(w_{i},h)}p(w_{j}|w_{i})}_{\text{context}}\\ &+\underbrace{\sum_{d\in\mathcal{D}}\sum_{w\in d}p(d|w)+\underbrace {c_{i}\in\mathcal{C}}\sum_{v\in\mathcal{S}_{i}}p(c_{i}|w)}_{\text{category}}, \end{split} \tag{1}\]

where

\[p(z|w)=\frac{\exp(\mathbf{u}^{r}_{w}\mathbf{v}_{s})}{\sum_{x_{i}}\exp(\mathbf{u}^{r}_{w} \mathbf{v}_{x_{i}})},~{}~{}(z\text{ can be $w_{j},d,~{}or~{}c_{i}$}). \tag{2}\]

In this objective, \(\mathbf{u}_{w_{i}}\) (and \(\mathbf{v}_{w_{j}}\)), \(\mathbf{v}_{d}\), \(\mathbf{v}_{c_{i}}\) are the embedding vectors of terms, documents, and categories, respectively. \(\mathcal{C}(w_{i},h)\) is the set of context terms of \(w_{i}\) in \(d\). Specifically, if \(d=w_{1}w_{2}...w_{L}\), then \(\mathcal{C}(w_{i},h)=\{w_{j}|i-h\leq j\leq i+h,j\neq i\}\), where \(h\) is the context window size.

Note that the last term in Eq. (1) encourages the similarity between each category \(c_{i}\) and its representative terms \(\mathcal{S}_{i}\). Here, we adopt an iterative process to gradually update category-representative terms. Initially, \(\mathcal{S}_{i}\) consists of just a few invocabulary terms similar to \(c_{i}\) according to the PLM. At each iteration, the size of \(\mathcal{S}_{i}\) will increase to contain more category-discriminative terms (the selection criterion of these terms will be introduced in the next section), and we need to encourage their proximity with \(c_{i}\) in the next iteration.

Directly optimizing the full softmax in Eq. (2) is costly. Therefore, we adopt the negative sampling strategy (Mikolov et al., 2013) for efficient approximation.

Interpreting the Objective.In topic modeling studies, pointwise mutual information (PMI) (Newman et al., 2010) is a standard evaluation metric for topic coherence (Lu et al., 2014; Roder et al., 2015). Levy and Goldberg (2014) prove that the Skip-Gram embedding model is implicitly factorizing the PMI matrix. Following their proof, we can show that maximizing Eq. (1) is implicitly doing the following factorization:

\[\mathbf{U}_{w}^{T}[\mathbf{V}_{w};\mathbf{V}_{d};\mathbf{V}_{c}]=[\mathbf{X}_{ ww};\mathbf{X}_{ud};\mathbf{X}_{wc}], \tag{3}\]

where the columns of \(\mathbf{U}_{w}\), \(\mathbf{V}_{w}\), \(\mathbf{V}_{d}\), \(\mathbf{V}_{c}\) are \(\mathbf{u}_{w_{i}}\), \(\mathbf{v}_{w_{j}}\), \(\mathbf{vd}\), \(\mathbf{vc}_{c_{i}}\), respectively (\(w_{i},w_{j}\in\mathcal{V_{D}}\), \(d\in\mathcal{D}\), \(c_{i}\in\mathcal{C}\)); \(\mathbf{X}_{ww}\), \(\mathbf{X}_{wd}\), and \(\mathbf{X}_{wc}\) are PMI matrices.

\[\begin{split}\mathbf{X}_{ww}&=\left[\log\Big{(} \frac{\#_{\mathcal{D}}(w_{i},w_{j})\cdot\lambda_{\mathcal{D}}}{\#_{\mathcal{ D}}(w_{i})\cdot\#_{\mathcal{D}}(w_{j})\cdot b}\Big{)}\right]_{w_{i},w_{j}\in \mathcal{V_{D}}},\\ \mathbf{X}_{wd}&=\left[\log\Big{(}\frac{\#_{d}(w) \cdot\lambda_{\mathcal{D}}}{\#_{\mathcal{D}}(w)\cdot d_{\cdot}b}\Big{)} \right]_{w\in\mathcal{V_{D}},~{}d\in\mathcal{D}},\\ \mathbf{X}_{wc}&=\big{[}x_{w,c_{i}}\big{]}_{w\in \mathcal{V_{D}},~{}c_{i}\in\mathcal{C}},~{}~{}\text{where}\\ x_{w,c_{i}}&=\begin{cases}\log\frac{|\mathcal{C}|}{b}, &\text{if $w\in\mathcal{S}_{i}$},\\ -\infty,&\text{if $w\in\mathcal{S}_{j}$ ($\forall j\neq i$}).\end{cases}\end{split} \tag{4}\]

Here, \(\#_{\mathcal{D}}(w_{i},w_{j})\) denotes the number of co-occurrences of \(w_{i}\) and \(w_{j}\) in a context window in \(\mathcal{D};\#_{\mathcal{D}}(w)\) denotes the number of occurrences of 

in \({\cal D}\); \(\lambda_{\cal D}\) is the total number of terms in \({\cal D}\); \(\#_{d}(w)\) denotes the number of times \(w\) occurs in \(d\); \(\lambda_{d}\) is the total number of terms in \(d\); \(b\) is the number of negative samples. (For the derivation of Eq. (3), please refer to Appendix A.)

To summarize, the learned local representations \(\mathbf{u}_{w}\) are implicitly optimized for topic coherence, where term co-occurrences are measured in context, document, and category levels.

### Ensemble Ranking

We have obtained two sets of term embeddings that model text semantics from different angles: \(\{\mathbf{e}_{w}|w\in{\cal C}\cup{\cal V}_{\cal D}\}\) carries the PLM's knowledge, while \(\{\mathbf{u}_{w}|w\in{\cal V}_{\cal D}\}\) models the input corpus as well as user-provided seeds. We now propose an ensemble ranking method to leverage information from both sides to grab more discriminative terms for each category.

Given a category \(c_{i}\) and its current term set \({\cal S}_{i}\), we first calculate the scores of each term \(w\in{\cal V}_{\cal D}\).

\[\begin{split}&{\rm score}_{G}(w|{\cal S}_{i})=\frac{1}{|{\cal S}_{ i}|}\sum_{v^{\prime}\in{\cal S}_{i}}\cos(\mathbf{e}_{w},\mathbf{e}_{w^{\prime}}),\\ &{\rm score}_{L}(w|{\cal S}_{i})=\frac{1}{|{\cal S}_{i}|}\sum_{ v^{\prime}\in{\cal S}_{i}}\cos(\mathbf{u}_{w},\mathbf{u}_{w^{\prime}}).\end{split} \tag{5}\]

The subscript "\(G\)" here means "general", while "\(L\)" means "local". Then, we sort all terms by these two scores, respectively. Each term \(w\) will hence get two rank positions \({\rm rank}_{G}(w)\) and \({\rm rank}_{L}(w)\). We propose the following ensemble score based on the reciprocal rank:

\[{\rm score}(w|{\cal S}_{i})=\Big{(}\frac{1}{2}\big{(}\frac{1}{{\rm rank}_{G}( w)}\big{)}^{\rho}+\frac{1}{2}\big{(}\frac{1}{{\rm rank}_{L}(w)}\big{)}^{\rho} \Big{)}^{1/\rho}. \tag{6}\]

Here, \(0<\rho\leq 1\) is a constant. In practice, instead of ranking all terms in the vocabulary, we only check the top-\(M\) results in the two ranking lists. If a term \(w\) is not among the top-\(M\) according to \({\rm score}_{G}(w)\) (resp., \({\rm score}_{L}(w)\)), we set \({\rm rank}_{G}(w)=+\infty\) (resp., \({\rm rank}_{L}(w)=+\infty\)). In fact, when \(\rho=1\), Eq. (6) becomes the _arithmetic_ mean of the two reciprocal ranks \(\frac{1}{{\rm rank}_{G}(w)}\) and \(\frac{1}{{\rm rank}_{L}(w)}\). This is essentially the mean reciprocal rank (MRR) commonly used in ensemble ranking, where a high position in one ranking list can largely compensate a low position in the other. In contrast, when \(\rho\to 0\), Eq. (6) becomes the _geometric_ mean of the two reciprocal ranks (see Appendix B), where two ranking lists both have the "veto power" (i.e., a term needs to be ranked as top-\(M\) in both ranking lists to obtain a non-zero ensemble score). In experiment, we set \(\rho=0.1\) and show it outperforms MRR (i.e., \(\rho=1\)) in our topic discovery task.

After computing the ensemble score \({\rm score}(w|{\cal S}_{i})\) for each \(w\), we update \({\cal S}_{i}\). To guarantee that each \({\cal S}_{i}\) is category-discriminative, we do not allow any term to belong to more than one category. Therefore, we gradually expand each \({\cal S}_{i}\) by turns. At the beginning, we reset \({\cal S}_{1}=...={\cal S}_{|{\cal C}|}=\emptyset\). When it is \({\cal S}_{i}\)'s turn, we add one term \({\cal S}_{i}\) according to the following criterion:

\[{\cal S}_{i}\leftarrow{\cal S}_{i}\cup\{\mathop{\arg\max}_{w\in{\cal V}_{\cal D }({\cal S}_{1}\cup...\cup{\cal S}_{|{\cal C}|})}{\rm score}(w|{\cal S}_{i})\}. \tag{7}\]

```
Input:A text corpus \({\cal D}=\{d_{1},..,d_{|{\cal D}|}\}\), a set of seeds \({\cal C}=\{c_{1},...,c_{|{\cal C}|}\}\), and a PLM. Output:\(({\cal S}_{1},....,{\cal S}_{|{\cal C}|})\), where each \({\cal S}_{i}\) is a set of category-discriminative terms for \(c_{i}\).
1 Compute \(\{\mathbf{e}_{w}|w\in{\cal C}\cup{\cal V}_{\cal D}\}\) using the PLM;
2// Initialize \({\cal S}_{i}\);
3\({\cal S}_{1},...,{\cal S}_{|{\cal C}|}\gets\emptyset\);
4forn\(t\gets 1~{}l\)o \(N\)do
5fori\(t\gets 1~{}l\)o \(|{\cal C}|\)do
6\(w_{n}\leftarrow\underset{w\in{\cal V}_{\cal D}({\cal S}_{1}\cup...\cup{\cal S} _{|{\cal C}|})}{\operatorname*{arg\max}_{w\in{\cal V}_{\cal D}({\cal S}_{1} \cup...\cup{\cal S}_{|{\cal C}|})}}\cos(\mathbf{e}_{w},\mathbf{e}_{c_{i}})\);
7\({\cal S}_{i}\leftarrow{\cal S}_{i}\cup\bigcup\{w_{n}\}\);
8// Update \({\cal S}_{i}\) forT’ iterations;
9for\(t\gets l\)o T do
10 Learn \(\{\mathbf{u}_{w}|w\in{\cal V}_{\cal D}\}\) from the input corpus \({\cal D}\) and the up-to-date representative terms \({\cal S}_{1},...,{\cal S}_{|{\cal C}|}\) according to Eq. (1);
11score\((w|{\cal S}_{i})\) and \({\rm score}_{L}(w|{\cal S}_{i})\leftarrow\text{Eq.~{}\eqref{eq:conj}}\);
12score\((w|{\cal S}_{i})\leftarrow\text{Eq.~{}\eqref{eq:conj}}\);
13\({\cal S}_{1},...,{\cal S}_{|{\cal C}|}\leftarrow\emptyset\);
14forn\(t\gets 1~{}l\)o \((t+1)N\)do
15fori\(t\gets 1~{}l\)o \(|{\cal C}|\)do
16\({\cal S}_{i}\leftarrow{\cal S}_{i}\cup\left\{\mathop{\arg\max}_{w\in{\cal V}_{ \cal D}({\cal S}_{1}\cup...\cup{\cal S}_{|{\cal C}|})}{\operatorname*{arg\max}_{ w\in{\cal V}_{\cal D}({\cal S}_{1}\cup...\cup{\cal S}_{|{\cal C}|})}}{ \operatorname*{score}(w|{\cal S}_{i})}\right\}\). Return \(({\cal S}_{1},...,{\cal S}_{|{\cal C}|})\);
```

**Algorithm 1****SeeTopic

### Overall Framework

We summarize the entire SeeTopic framework in Algorithm 1. To deal with **out-of-vocabulary** category names, we first utilize a PLM to find their nearest in-vocabulary terms as the initial category-discriminative term set \({\cal S}_{i}\) (lines 1-7). After initialization, \(|{\cal S}_{i}|=N\) (\(\forall 1\leq i\leq|{\cal C}|\)). Note that for an in-vocabulary category name \(c_{i}\in{\cal V}_{\cal D}\), itself will be added to the initial \({\cal S}_{i}\) as the top-1 similar in-vocabulary term.

After getting the initial \({\cal S}_{i}\), we update it by \(T\) iterations (Lines 8-16). At each iteration, according to the up-to-date \({\cal S}_{1},S_{2},...,{\cal S}_{|{\cal C}|}\), we relearn embeddings \(\mathbf{u}_{w}\), \(\mathbf{v}_{w}\), \(\mathbf{v}_{d}\), and \(\mathbf{v}_{c_{i}}\) using Eq. (1) (Line 10). The two set of embeddings, \(\{\mathbf{e}_{w}|w\in{\cal C}\cup{\cal V}_{\cal D}\}\) (computed at Line 1) and \(\{\mathbf{u}_{w}|w\in{\cal V}_{\cal D}\}\) (updated at Line 10), are then leveraged to perform ensemble ranking (Lines 11-12). Based on the 

ensemble score \({\rm score}(w|\mathcal{S}_{i})\), we update \(\mathcal{S}_{i}\) using Eq. (7) (Lines 13-16). After the \(t\)-th iteration, \(|\mathcal{S}_{i}|=(t+1)N\) (\(\forall 1\leq i\leq|\mathcal{C}|\)).

**Complexity Analysis.** The time complexity of using the PLM is \(\mathcal{O}((|\mathcal{C}|+|\mathcal{V}_{\mathcal{D}}|)\alpha_{\mathrm{PLM}})\), where \(\alpha_{\mathrm{PLM}}\) is the complexity of encoding one term via the PLM. The total complexity of local embedding is \(\mathcal{O}(T\lambda_{\mathcal{D}}(h+|\mathcal{C}|)b)\) because in each iteration \(1\leq t\leq T\), every \(w\in\mathcal{D}\) interacts with every other term in the context window of size \(h\), its belonging document, and each category \(c_{i}\in\mathcal{C}\), and each update involves \(b\) negative samples. The total complexity of ensemble ranking is \(\mathcal{O}(T|\mathcal{V}_{\mathcal{D}}||\mathcal{C}||\mathcal{S}_{i}|)\) as in each iteration \(1\leq t\leq T\), we compute scores between each \(w\in\mathcal{V}_{\mathcal{D}}\) and each \(w^{\prime}\in\mathcal{S}_{i}\) (\(\forall c_{i}\in\mathcal{C}\)).

## 4 Experiments

### Experimental Setup

**Datasets.** We conduct experiments on three public datasets from different domains: (1) **SciDocs**[25]4 is a large collection of scientific papers supporting diverse evaluation tasks. For the MeSH classification task [80], about 23K medical papers are collected, each of which is assigned to one of the 11 common disease categories derived from the MeSH vocabulary. We use the title and abstract of each paper as documents and the 11 category names as seeds. (2) **Amazon**[9] contains product reviews from May 1996 to July 2014. Each Amazon review belongs to one or more product categories. We use the subset sampled by Zhang et al. zhang2020learning, which contains 10 categories and 100K reviews. (3) **Twitter**[10]6 is a crawl of geo-tagged tweets in New York City from August 2014 to November 2014. The dataset collectors link these tweets with Foursquare's POI database and assign them to 9 POI categories. We take these category names as input seeds.

Footnote 4: [https://github.com/allenai/scidocs](https://github.com/allenai/scidocs)

Footnote 5: [http://jmcauley.ucsd.edu/data/amazon/index.html](http://jmcauley.ucsd.edu/data/amazon/index.html)

Footnote 6: [https://github.com/franticnerd/geoburst](https://github.com/franticnerd/geoburst)

Seeds used in the three datasets are shown in Table 1. Dataset statistics are summarized in Table 2. For all three datasets, we use AutoPhrase [10]7 to perform phrase chunking in the corpus, and we remove words and phrases occurring less than 3 times.

Footnote 7: [https://github.com/shangjingbo1226/AutoPhrase](https://github.com/shangjingbo1226/AutoPhrase)

Previous studies [10, 11] have tried some other datasets (e.g., RCV1, 20 Newsgroups, NYT, and Yelp). However, the category names they use in these datasets are all picked from **in-vocabulary** terms. Therefore, we do not consider these datasets for evaluation in our task settings.

Footnote 7: [https://github.com/sharingjingbo1226/AutoPhrase](https://github.com/sharingjingbo1226/AutoPhrase)

Following [10], we adopt a 60-40 train-test split for all three datasets. The training set is used as the input corpus \(\mathcal{D}\), and the testing set is used for calculating topic coherence metrics (see evaluation metrics for details).

**Compared Methods.** We compare our SeeTopic framework with the following methods, including seed-guided topic modeling methods, seed-guided embedding learning methods, and PLMs. (1) **seededLDA**[10]8 is a seed-guided topic modeling method. It improves LDA by biasing topics to produce input seeds and by biasing documents to select topics relevant to the seeds they contain. (2) **AnchoredCorEx**[10]9 is a seed-guided topic modeling method. It incorporates user-provided seeds by balancing between compressing the input corpus and preserving seed-related information. (3) **LabeledETM**[11] is an embedding-based topic modeling method. It incorporates distributed representation of each term. Following [11], we retrieve representative terms according to their embedding similarity with the category name. (4) **CatE**[11] is a seed-guided embedding learning method for discriminative topic discovery. It takes category names as input and jointly learns term embedding and specificity from the input corpus. Category-discriminative terms are then selected based on both embedding similarity with the category and specificity. (5) **BERT**[12]12 is a PLM. Following Lines 1-7 in Algorithm 1, we use BERT to encode each input category name and each term to a vector, and then perform similarity search to directly find all repre

\begin{table}
\begin{tabular}{c|c|c|c} \hline
**Dataset** & **SciDocs** & **Amazon** & **Twitter** \\ \hline \#Documents & 23,473 & 100,000 & 135,529 \\ \hline \#In-vocabulary Terms (After Phrase Chuning) & 55,897 & 56,942 & 17,577 \\ \hline Avg Doc Length & 239.8 & 119.0 & 6.7 \\ \hline \#Secels & 11 & 10 & 9 \\ \hline \#Out-of-vocabulary Seeds (After Phrase Chunking) & 5 & 6 & 7 \\ \hline \end{tabular}
\end{table}
Table 2: Dataset Statistics.

 

sentative terms. (6) **BioBERT**[12]13 is a PLM. It is used in the same way as BERT. Since BioBERT is specifically trained for biomedical text mining tasks, we report its performance on the SciDocs dataset only. (7) **SeeTopic-NoIter** is a variant of our SeeTopic framework. In Algorithm 1, after initialization (Lines 1-7), it executes Lines 9-16 only once (i.e., \(T=1\)) to find all representative terms.

Footnote 13: [https://huggingface.co/dmis-lab/biobert-vl_1](https://huggingface.co/dmis-lab/biobert-vl_1)

Here, all seed-guided topic modeling and embedding baselines (i.e., SeededLDA, Anchored CorEx, CatE, and Labeled ETM) can only take **in-vocabulary** seeds as input. For a fair comparison, we run Lines 1-7 in Algorithm 1 to get the initial representative in-vocabulary terms for each category, and input these terms as seeds into the baselines. In other words, all compared methods use BERT/BioBERT to initialize their term sets.

**Evaluation Metrics.** We evaluate topic discovery results from three different angles: topic coherence, term accuracy, and topic diversity.

(1) **NPMI**[8] is a standard metric in topic modeling to measure _topic coherence_. Within each topic, it calculates the normalized pointwise mutual information for each pair of terms in \({\cal S}_{i}\).

\[\mathrm{NPMI}=\frac{1}{|{\cal C}|}\sum^{|{\cal C}|}_{i=1}\frac{1}{{{(S_{i}^{ *})}\choose{j<k}}}\sum_{\begin{subarray}{c}w_{j},v_{k}\in{\cal S}_{i}\\ j<k\end{subarray}}\frac{\log\frac{P(w_{j},w_{k})}{P(w_{j})P(w_{k})}}{-\log P( w_{j},w_{k})}, \tag{8}\]

where \(P(w_{j},w_{k})\) is the probability that \(w_{j}\) and \(w_{k}\) co-occur in a document; \(P(w_{j})\) is the marginal probability of \(w_{j}\).14

Footnote 14: When calculating Eqs. (8) and (9), to avoid \(\log 0\), we use \(P(w_{j},w_{k})+\epsilon\) and \(P(w)+\epsilon\) to replace \(P(w_{j},w_{k})\) and \(P(w)\), respectively, where \(\epsilon=1/|{\cal D}|\).

(2) **LCP**[5] is another standard metric to measure _topic coherence_. It calculates the pairwise log conditional probability of top-ranked terms.

\[\mathrm{LCP}=\frac{1}{|{\cal C}|}\sum^{|{\cal C}|}_{i=1}\frac{1}{{{(S_{i}^{ *})}\choose{2}}}\sum_{\begin{subarray}{c}w_{j},w_{k}\in{\cal S}_{i}\\ j<k\end{subarray}}\log\frac{P(w_{j},w_{k})}{P(w_{j})}. \tag{9}\]

Note that **PMI**[10] is also a standard metric for topic coherence. We do observe that SeeTopic outperforms baselines in terms of PMI in most cases. However, since our local embedding step is implicitly optimizing a PMI-like objective, we no longer use it as our evaluation metric.

(3) **MACC**[11] measures _term accuracy_. It is defined as the proportion of retrieved terms that actually belong to the corresponding category according to the category name.

\[\mathrm{MACC}=\frac{1}{|{\cal C}|}\sum^{|{\cal C}|}_{i=1}\frac{1}{|{\cal S}_{ i}|}\sum_{w_{j}\in{\cal S}_{i}}{\bf 1}(w_{j}\in c_{i}), \tag{10}\]

where \({\bf 1}(w_{j}\in c_{i})\) is the indicator function of whether \(w_{j}\) is relevant to category \(c_{i}\). MACC requires human evaluation, so we invite five annotators to perform independent annotation. The reported MACC score is the average MACC of the five annotators. A high inter-annotator agreement is observed, with Fleiss' kappa [13] being 0.856, 0.844, and 0.771 on SciDocs, Amazon, and Twitter, respectively.

(4) **Diversity**[14] measures the _mutual exclusivity_ of discovered topics. It is the percentage of unique terms in all topics, which corresponds to our task requirement that each retrieved term is discriminatively close to one category and far from the others.

\[\mathrm{Diversity}=\frac{||U|_{i=1}^{|{\cal C}|}\,{\cal S}_{i}|}{\sum^{|{\cal C }|}_{i=1}|{\cal S}_{i}|}. \tag{11}\]

**Experiment Settings.** We use BioBERT as the

\begin{table}
\begin{tabular}{r|l l l l|l l l|l l l l} \hline \multirow{2}{*}{Methods} & \multicolumn{4}{c|}{ScilDocs} & \multicolumn{4}{c|}{Amazon} & \multicolumn{4}{c}{Twitter} \\ \cline{2-13}  & NPMI & LCP & MACC & Div. & NPMI & LCP & MACC & Div. & NPMI & LCP & MACC & Div. \\ \hline SeededLDA & 0.111\({}^{***}\) & **-1.232** & 0.156\({}^{*}\) & 0.140\({}^{**}\) & -1.505\({}^{*}\) & 0.147\({}^{**}\) & 0.393\({}^{*}\) & 0.026\({}^{*}\) & -4.508\({}^{**}\) & 0.195\({}^{*}\) & 0.696\({}^{**}\) \\ AnchoredCorEx & 0.2132\({}^{**}\) & -2.180\({}^{**}\) & -0.264\({}^{**}\) & **-1.000** & 0.268\({}^{**}\) & -1.963\({}^{*}\) & -0.333\({}^{**}\) & **-1.000** & 0.180\({}^{**}\) & -3.384\({}^{**}\) & 0.233\({}^{**}\) & **-1.000** \\ Labeled ETM & 0.669\({}^{*}\) & -1.549\({}^{**}\) & 0.458\({}^{**}\) & 0.961\({}^{**}\) & 0.616\({}^{**}\) & -2.103\({}^{**}\) & 0.585\({}^{**}\) & **-1.000** & 0.610\({}^{*}\) & -2.197\({}^{**}\) & 0.268\({}^{**}\) & 0.989 \\ CatE & 0.691\({}^{*}\) & -1.451\({}^{**}\) & 0.633\({}^{**}\) & **-1.000** & 0.633\({}^{**}\) & -1.688\({}^{**}\) & 0.856\({}^{**}\) & **-1.000** & **0.711** & **-1.653** & 0.483\({}^{**}\) & **1.000** \\ \hline BERT & 0.626\({}^{*}\) & -1.682\({}^{**}\) & 0.740\({}^{**}\) & 0.891\({}^{**}\) & 0.588\({}^{**}\) & -2.186\({}^{**}\) & 0.832\({}^{**}\) & **-1.000** & 0.626\({}^{**}\) & -2.088\({}^{**}\) & 0.627\({}^{**}\) & 0.944\({}^{**}\) \\ BiBBERT & 0.618\({}^{**}\) & -1.704\({}^{**}\) & **0.938** & 0.982\({}^{**}\) & **–** & **–** & **–** & **–** & **–** & **–** & **–** \\ \hline SeeTopic-Noller & 0.681\({}^{**}\) & -1.535\({}^{**}\) & 0.887 & **1.000** & 0.643\({}^{**}\) & -1.971\({}^{**}\) & 0.892 & **1.000** & 0.636 & -2.008\({}^{**}\) & 0.618 & **1.000** \\ SeeTopic & **0.717** & -1.267 & 0.909 & **1.000** & **0.684** & **-1.391** & **0.904** & **1.000** & 0.640 & -1.813 & **0.633** & **1.000** \\ \hline \end{tabular}
\end{table}
Table 3: NPMI, LCP, MACC, and Diversity of compared algorithms on three datasets. NPMI and LCP measure topic coherence; MACC measures term accuracy; Diversity (abbreviated to Div.) measures topic diversity. **Bold**: the highest score. Underline: the second highest score. \({}^{*}\): significantly worse than SeeTopic (p-value \(<0.01\)).

 

PLM on SciDocs, and BERT-base-uncased as the PLM on Amazon and Twitter. The embedding dimension of \({\mathbf{u}_{w}}\) is 768 (the same as \({\mathbf{e}_{w}}\)); the number of negative samples \(b=5\). In ensemble ranking, the length of the general/local ranking list \(M=100\); the hyperparameter \(\rho\) in Eq. (6) is set as \(0.1\); the number of iterations \(T=4\); after each iteration, we increase the size of \({\mathcal{S}_{i}}\) by \(N=3\). We use the top-10 ranked terms in each topic for final evaluation (i.e., \(|{\mathcal{S}_{i}}|=10\) in Eqs. (8)-(11)). Experiments are run on Intel Xeon E5-2680 v2 @ 2.80GHz and one NVIDIA GeForce GTX 1080.

### Performance Comparison

Table 3 shows the performance of all methods. We run each experiment 3 times with the average score reported. To show statistical significance, we conduct a two-tailed unpaired t-test to compare SeeTopic and each baseline. (The performance of BERT and BioBERT is deterministic according to our usage. When comparing SeeTopic with them, we conduct a two-tailed Z-test instead.) The significance level is also marked in Table 3.

We have the following observations from Table 3. (1) Our SeeTopic model performs consistently well. In fact, it achieves the highest score in 8 columns and the second highest in the remaining 4 columns. (2) Classical seed-guided topic modeling baselines (i.e., SeededLDA and Anchored CorEx) perform not well in respect of NPMI (topic coherence) and MACC (term accuracy). Embedding-based topic discovery approaches (i.e., Labeled ETM and CatE) make some progress, but they still significantly underperform the PLM-empowered SeeTopic model on SciDocs and Amazon. (3) SeeTopic consistently performs better than SeeTopic-NoIter on all three datasets, indicating the positive contribution of the proposed iterative process. (4) SeeTopic guarantees the mutual exclusivity of \({\mathcal{S}_{1}},...,{\mathcal{S}_{|{\mathcal{C}}|}}\). In comparison, SeededLDA, Labeled ETM, and BERT cannot guarantee such mutual exclusivity.

**In-vocabulary vs. Out-of-vocabulary.** Figure 1 compares the MACC scores of different seed-guided topic discovery methods on in-vocabulary categories and out-of-vocabulary categories. We find that the performance improvement of SeeTopic upon baselines on out-of-vocabulary categories is larger than that on in-vocabulary ones. For example, on Amazon, SeeTopic underperforms CatE on in-vocabulary categories but outperforms CatE on out-of-vocabulary ones; on Twitter, the gap between SeeTopic and baselines becomes much more evident on out-of-vocabulary categories. Note that all baselines in Figure 1 do not utilize the power of PLMs, so this observation validates our claim that PLMs are helpful in tackling out-of-vocabulary seeds.

### Parameter Study

We study the effect of two important hyperparameters: \(\rho\) (the hyperparameter in ensemble ranking) and \(T\) (the number of iterations). We vary the value of \(\rho\) in \(\{0.1,0.3,0.5,0.7,0.9,1\}\) (SeeTopic uses \(\rho=0.1\) by default) and the value of \(T\) in \(\{1,2,3,4,5\}\) (SeeTopic uses \(T=4\) by default, and SeeTopic-NoIter is the case when \(T=1\)). Figure 2 shows the change of model performance measured by NPMI and LCP.

Figure 1: MACC of seed-guided topic discovery methods on in-vocabulary categories and out-of-vocabulary categories.

Figure 2: Parameter study of SeeTopic measured by topic coherence.

 

According to Figures 2(a) and 2(b), in most cases, the performance of SeeTopic deteriorates as \(\rho\) increases from 0.1 to 0.9. Thus, setting \(\rho=0.1\) always leads to competitive NPMI and LCP scores on the three datasets. Although \(\rho=1\) is better than \(\rho=0.9\), its performance is still suboptimal in comparison with \(\rho=0.1\). This finding indicates that replacing the mean reciprocal rank (i.e., \(\rho=1\)) with our proposed Eq. (6) is reasonable. According to Figures 2(c) and 2(d), SeeTopic usually performs better when there are more iterations. On SciDocs and Twitter, the scores start to converge after \(T=4\). Besides, more iterations will result in longer running time. Overall, we believe setting \(T=4\) strikes a good balance.

### Case Study

Finally, we show the terms retrieved by different methods as a case study. From each of the three datasets, we select an out-of-vocabulary category and show its topic discovery results in Table 4. We mark a retrieved term as correct () if at least 3 of the 5 annotators judge the term as relevant to the seed. Otherwise, we mark the term as incorrect ().

For the category "_hepatitis__u/b/c/e_" from SocDocs, SeededLDA and Anchored CorEx can only find very general medical terms, which are relevant to all seeds in SciDocs and thus inaccurate; Labeled ETM and CatE find terms about "_alanine aminotransferase_", whose elevation suggest not only hepatitis but also other diseases like diabetes and heart failure, thus not discriminative either; BioBERT and SeeTopic, with the power of a PLM, can accurately pick terms relevant to "_hepatitis__b_" and "_hepatitis__c_". For the category "_ports and outdoors_" from Amazon, SeededLDA and Anchored CorEx again find very general terms, most of which are not category-discriminative; Labeled ETM and CatE are able to pick more specific terms such as "_ears and tracks_", but they still make mistakes; BERT, as a PLM, can accurately find terms that have lexical overlap with the category name (e.g., "_outdoorsmen_", "_sorting events_"), meanwhile such terms are less diverse; SeeTopic-NoIter starts to discover more concrete terms than BERT (e.g., "_indoor soccer_", "_bike riding_") by leveraging local text semantics; the full SeeTopic model, with an iterative updating process, can find more specific and informative terms (e.g., "_canoing_", "_picnicis_", and "_rafting_"). For the category "_travel and trans

\begin{table}
\begin{tabular}{|p{85.4pt}|p{71.1pt}|} \hline
**Method** & \multicolumn{2}{c|}{**Top-5 Representative Terms**} \\ \hline  & **Dataset:****SciDocs**, **Category Name: hepatitis **u/b/c/e \\ \hline \hline ScededLDA & patients (✗), treatment (✗), placebo (✗), study (✗), group (✗) \\ \hline Anchored CorEx & expression (✗), sense (✗), cells (✗), genes (✗), genetic (✗) \\ \hline Labeled ETH & hepatitis b virus hbw dna (✓), serum hwq dna (✓), serum alanine aminotransferase (✗), alanine aminotransferase al (✗), \\  & alanine aminotransferase alt (✗), below detection limit (✗) \\ \hline CatE & chronic hepatitis by virus hw infection (✗), hepatitis b e antigen hebag (✗), hepatitis b virus hbv dna (✓), normal alanine aminotransferase (✗), hpage-negative chronic hepatitis b (✓) \\ \hline BioBERT & hepatitis b virus hby dna (✓), chronic hepatitis b virus bw infection (✓), hepatitis b eantigen hebag (✓), hepatitis b virus  hbyr infection (✓), chronic hepatitis c virus hcv (✓) \\ \hline SeeTopic-Noller & hepatitis b virus hbv dna (✓), hepatitis b e antigen hebag (✓), chronic hepatitis b virus hbv infection (✓), hepatitis b surface antigen hbag (✓), ”hage-negative chronic hepatitis b (✓) \\ \hline SeeTopic & chronic hepatitis b viruses hbv infection (✓), hedge-negative chronic hepatitis b (✗), hepatitis c virus hcv-infected (✓), hepatitis b virus hbv dna (✓), chronic hepatitis c virus hcv (✓) \\ \hline  & Dataset:****Amazon**, **Category Name: sports and outdoors \\ \hline SeededLDA & use (✗), good (✗), one (✗), product (✗), like (✗) \\ \hline Anchored CorEx & sports (✗), use (✗), size (✗), wear (✗), fit (✓) \\ \hline Labeled ETM & cars and tracks (✓), tracks and cars (✓), search options (✗), championships (✗), cool bosses (✗) \\ \hline CatE & outdoorism (✓), outdoor activities (✓), cars and tracks (✓), foot support (✗), offers plenty (✗) \\ \hline BERT & cars and tracks (✓), outdoor activities (✓), outdoorsmen (✓), sports (✓), sporting events (✓) \\ \hline SeeTopic-Noller & outdoorism (✓), outdoor activities (✓), cars and tracks (✓), indoor soccer (✓), bike riding (✓) \\ \hline SeeTopic & cancoing (✓), picincis (✗), bike frides (✓), bike riding (✓), rafting (✓) \\ \hline \hline  & Dataset:****Twitter**, **Category Name:****travel** and **transport** \\ \hline SeedLDA & nye (✗), new york (✗), line (✗), high (✗), the square (✓) \\ \hline Anchored CorEx & new york (✗), post photo (✓), new (✗), day (✗), today (✗) \\ \hline Labeled ETM & tourism (✓), their (✗), morning view (✓), gma (✗) \\ \hline CatE & maritime (✓), tourism (✓), natural history (✗), scenery (✗), elevate (✗) \\ \hline BERT & maritime (✓), tourism (✓), natural history (✗), olive oil (✗), hpage claim (✓) \\ \hline SeeTopic-Noller & maritime (✓), tourism (✓), natural history (✗), scenery (✓), navy (✗) \\ \hline SeeTopic & wildlife (✗), scenery (✓), maritime (✓), highlighters (✗), aquarium (✓) \\ \hline \end{tabular}
\end{table}
Table 4: Top-5 representative terms retrieved by different algorithms for three out-of-vocabulary categories from SciDocs, Amazon, and Twitter. ‘✓’: at least 3 of the 5 annotators judge the term as relevant to the seed.

 

_port_" from Twitter, both BERT and CatE make mistakes by including the term "_natural history_"; SeeTopic-NoIter, without an iterative update process, also includes this error; the full SeeTopic model finally excludes this error and achieves the highest accuracy in the retrieved top-5 terms among all compared methods.

## 5 Related Work

**Seed-Guided Topic Discovery.** Seed-guided topic models aim to leverage user-provided seeds to discover underlying topics according to users' interests. Early studies take LDA Blei et al. (2003) as the backbone and incorporate seeds into model learning. For example, Andrzejewski et al. (2009) consider must-link and cannot-link constraints among seeds as priors. SeededLDA Jagarlamudi et al. (2012) encourages topics to contain more seeds and encourages documents to select topics relevant to the seeds they contain. Anchored CorEx Gallagher et al. (2017) extracts maximally informative topics by jointly compressing the corpus and preserving seed relevant information. Recent studies start to utilize embedding techniques to learn better word semantics. For example, CatE Meng et al. (2020a) explicitly encourages distinction among retrieved topics via category-name guided embedding learning. However, all these models require the provided seeds to be in-vocabulary, mainly because they focus on the input corpus only and are not equipped with general knowledge of PLMs.

**Embedding-Based Topic Discovery.** A number of studies extend LDA to involve word embedding. The common strategy is to adapt distributions in LDA to generate real-valued data (e.g., Gaussian LDA Das et al. (2015), LFTM Nguyen et al. (2015), Spherical HDP Battanghelich et al. (2016), and CGTM Xun et al. (2017b)). Some other studies think out of the LDA backbone. For example, TWE Liu et al. (2015) uses topic structures to jointly learn topic embeddings and improve word embeddings. CLM Xun et al. (2017a) collaboratively improves topic modeling and word embedding by coordinating global and local contexts. ETM Dieng et al. (2020) models word-topic correlations via word embeddings to improve the expressiveness of topic models. More recently, Sia et al. (2020) show that directly clustering word embeddings (e.g., word2vec or BERT) also generates good topics; Thompson and Mimno (2020) further find that BERT and GPT-2 discover high-quality topics, but RoBERTa does not. These models are unsupervised and hard to be applied to seed-guided settings. In contrast, our SeeTopic framework joint leverages PLMs, word embeddings, and seed information.

## 6 Conclusions and Future Work

In this paper, we study seed-guided topic discovery in the presence of out-of-vocabulary seeds. To understand and make use of in-vocabulary components in each seed, we utilize the tokenization and contextualization power of PLMs. We propose a seed-guided embedding learning framework inspired by the goal of maximizing PMI in topic modeling, and an iterative ensemble ranking process to jointly leverage general knowledge of the PLM and local signals learned from the input corpus. Experimental results show that SeeTopic outperforms seed-guided topic discovery baselines and PLMs in terms of topic coherence, term accuracy, and topic diversity. A parameter study and a case study further validate some design choices in SeeTopic.

In the future, it would be interesting to extend SeeTopic to seed-guided hierarchical topic discovery, where parent and child information in the input category hierarchy may help infer the meaning of out-of-vocabulary nodes.

## Acknowledgments

We thank anonymous reviewers for their valuable and insightful feedback. Research was supported in part by US DARPA KAIROS Program No. FA8750-19-2-1004, SocialSim Program No. W911NF-17-C-0099, and INCAS Program No. HR001121C0165, National Science Foundation IIS-19-56151, IIS-17-41317, and IIB 17-04532, and the Molecule Maker Lab Institute: An AI Research Institutes program supported by NSF under Award No. 2019897, and the Institute for Geospatial Understanding through an Integrative Discovery Environment (I-GUIDE) by NSF under Award No. 2118329. Any opinions, findings, and conclusions or recommendations expressed herein are those of the authors and do not necessarily represent the views, either expressed or implied, of DARPA or the U.S. Government.

## References

* Andrzejewski et al. (2009) David Andrzejewski, Xiaojin Zhu, and Mark Craven. 2009. Incorporating domain knowledge into topic modeling via dirichlet forest priors. In _ICML'09_, pages 25-32 

Kayhan Batmanghelich, Ardavan Saeedi, Karthik Narasimhan, and Sam Gershman. 2016. Nonparametric spherical topic modeling with word embeddings. In _ACL'16_, pages 537-542.
* Blei et al. (2003) David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. _JMLR_, 3:993-1022.
* Chen et al. (2015) Xingyuan Chen, Yunqing Xia, Peng Jin, and John Carroll. 2015. Dalatless text classification with descriptive lda. In _AAAI'15_, pages 2224-2231.
* Cohan et al. (2020) Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S Weld. 2020. Specer: Document-level representation learning using citation-informed transformers. In _ACL'20_, pages 2270-2282.
* Coletti and Bleich (2001) Margaret H Coletti and Howard L Bleich. 2001. Medical subject headings used to search the biomedical literature. _JAMIA_, 8(4):317-323.
* Das et al. (2015) Rajarshi Das, Manzil Zaheer, and Chris Dyer. 2015. Gaussian lda for topic models with word embeddings. In _ACL'15_, pages 795-804.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In _NAACL-HLT'19_, pages 4171-4186.
* Dieng et al. (2020) Adji B Dieng, Francisco JR Ruiz, and David M Blei. 2020. Topic modeling in embedding spaces. _TACL_, 8:439-453.
* Fleiss (1971) Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. _Psychological Bulletin_, 76(5):378.
* Gallagher et al. (2017) Ryan J Gallagher, Kyle Reing, David Kale, and Greg Ver Steeg. 2017. Anchored correlation explanation: Topic modeling with minimal domain knowledge. _TACL_, 5:529-542.
* Griffiths and Steyvers (2004) Thomas L Griffiths and Mark Steyvers. 2004. Finding scientific topics. _PNAS_, 101(suppl 1):5228-5235.
* Hofmann (1999) Thomas Hofmann. 1999. Probabilistic latent semantic indexing. In _SIGIR'99_, pages 50-57.
* Jagaflamudi et al. (2012) Jagadeesh Jagaflamudi, Hal Daume, and Raghavendra Udupa. 2012. Incorporating lexical priors into topic models. In _EACL'12_, pages 204-213.
* Lau et al. (2014) Jey Han Lau, David Newman, and Timothy Baldwin. 2014. Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality. In _EACL'14_, pages 530-539.
* Lee et al. (2020) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. _Bioinformatics_, 36(4):1234-1240.
* Levy and Goldberg (2014) Omer Levy and Yoav Goldberg. 2014. Neural word embedding as implicit matrix factorization. _NIPS'14_, pages 2177-2185.
* Li et al. (2020) Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. 2020. On the sentence embeddings from pre-trained language models. In _EMNLP'20_, pages 9119-9130.
* Liu et al. (2015) Yang Liu, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. 2015. Topical word embeddings. In _AAAI'15_, pages 2418-2424.
* Liu et al. (2021) Zequn Liu, Shukai Wang, Yiyang Gu, Ruiyi Zhang, Ming Zhang, and Sheng Wang. 2021. Graphine: A dataset for graph-aware terminology definition generation. In _EMNLP'21_, pages 3453-3463.
* Manning et al. (2014) Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David McClosky. 2014. The stanford corenlp natural language processing toolkit. In _ACL'14, System Demonstrations_, pages 55-60.
* McAuley and Leskovec (2013) Julian McAuley and Jure Leskovec. 2013. Hidden factors and hidden topics: understanding rating dimensions with review text. In _RecSys'13_, pages 165-172.
* Meng et al. (2020a) Yu Meng, Jiaxin Huang, Guangyuan Wang, Zihan Wang, Chao Zhang, Yu Zhang, and Jiawei Han. 2020a. Discriminative topic mining via category-name guided text embedding. In _WWW'20_, pages 2121-2132.
* Meng et al. (2020b) Yu Meng, Yunyi Zhang, Jiaxin Huang, Chenyan Xiong, Heng Ji, Chao Zhang, and Jiawei Han. 2020b. Text classification using label names only: A language model self-training approach. In _EMNLP'20_, pages 9006-9017.
* Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In _NIPS'13_, pages 3111-3119.
* Mimno et al. (2011) David Mimno, Hanna Wallach, Edmund Talley, Miriam Leenders, and Andrew McCallum. 2011. Optimizing semantic coherence in topic models. In _EMNLP'11_, pages 262-272.
* Newman et al. (2010) David Newman, Jey Han Lau, Karl Grieser, and Timothy Baldwin. 2010. Automatic evaluation of topic coherence. In _NAACL-HLT'10_, pages 100-108.
* Nguyen et al. (2015) Dat Quoc Nguyen, Richard Billingsley, Lan Du, and Mark Johnson. 2015. Improving topic models with latent feature word representations. _TACL_, 3:299-313.
* Qiu et al. (2018) Jiezhong Qiu, Yukiao Dong, Hao Ma, Jian Li, Kuansan Wang, and Jie Tang. 2018. Network embedding as matrix factorization: Unifying deepwalk, line, pte, and node2vec. In _WSDM'18_, pages 459-467 

Michael Roder, Andreas Both, and Alexander Hinneburg. 2015. Exploring the space of topic coherence measures. In _WSDM'15_, pages 399-408.
* Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In _ACL'16_, pages 1715-1725.
* Shang et al. (2018) Jingbo Shang, Jialu Liu, Meng Jiang, Xiang Ren, Clare R Voss, and Jiawei Han. 2018. Automated phrase mining from massive text corpora. _IEEE TKDE_, 30(10):1825-1837.
* Xia et al. (2020) Suzanna Xia, Ayush Dalmia, and Sabrina J Mielke. 2020. Tired of topic models? clusters of pretrained word embeddings make for fast and good topics too! In _EMNLP'20_, pages 1728-1736.
* Tang et al. (2015) Jian Tang, Meng Qu, and Qiaozhu Mei. 2015. Pte: Predictive text embedding through large-scale heterogeneous text networks. In _KDD'15_, pages 1165-1174.
* Thompson and Mimno (2020) Laure Thompson and David Mimno. 2020. Topic modeling with contextualized word representation clusters. _arXiv preprint arXiv:2010.12626_.
* Wang et al. (2009) Dingding Wang, Shenghuo Zhu, Tao Li, and Yihong Gong. 2009. Multi-document summarization using sentence-based topic models. In _ACL'09_, pages 297-300.
* Wang et al. (2022) Mu-Chun Wang, Zixuan Liu, and Sheng Wang. 2022. Textomics: A dataset for genomics data summary generation. In _ACL'22_.
* Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. _arXiv preprint arXiv:1609.08144_.
* Xun et al. (2017a) Guangxu Xun, Yaliang Li, Jing Gao, and Aidong Zhang. 2017a. Collaboratively improving topic discovery and word embeddings by coordinating global and local contexts. In _KDD'17_, pages 535-543.
* Xun et al. (2017b) Guangxu Xun, Yaliang Li, Wayne Xin Zhao, Jing Gao, and Aidong Zhang. 2017b. A correlated topic model using word embeddings. In _IJCAI'17_, pages 4207-4213.
* Zhang et al. (2017) Chao Zhang, Keyang Zhang, Quan Yuan, Fangbo Tao, Luming Zhang, Tim Hanratty, and Jiawei Han. 2017. React: Online multimodal embedding for recency-aware spatiotemporal activity modeling. In _SIGIR'17_, pages 245-254.
* Zhang et al. (2022) Yu Zhang, Shweta Garg, Yu Meng, Xiusi Chen, and Jiawei Han. 2022. Motifclass: Weakly supervised text classification with higher-order metadata information. In _WSDM'22_, pages 1357-1367.
* Zhang et al. (2020) Yu Zhang, Yu Meng, Jiaxin Huang, Frank F Xu, Xuan Wang, and Jiawei Han. 2020. Minimally supervised categorization of text with metadata. In _SIGIR'20_, pages 1231-1240.

## Appendix A The Embedding Learning Objective

In Section 3.2, we propose the following embedding learning objective:

\[\begin{split}\mathcal{J}=&\underbrace{\sum_{d\in \mathcal{D}}\sum_{u_{i}\in d}~{}\sum_{w_{j}\in(w_{i},h)}\frac{\exp(\mathbf{u}^{T}_{ w_{ij}}\mathbf{v}_{w_{i^{\prime}}})}{\sum_{v^{\prime}\in\nabla_{\mathcal{D}}}\exp(\mathbf{ u}^{T}_{w_{i^{\prime}}}\mathbf{v}_{w^{\prime}}})}_{\mathcal{J}_{\mathrm{context}}}+\\ &\underbrace{\sum_{d\in\mathcal{D}}\sum_{w\in d}\frac{\exp(\mathbf{u} ^{T}_{w}\mathbf{v}_{d})}{\sum_{d^{\prime}\in\mathcal{D}}\exp(\mathbf{u}^{T}_{w}\mathbf{v} _{d^{\prime}})}}_{\mathcal{J}_{\mathrm{document}}}+\\ &\underbrace{\sum_{c_{i}\in\mathcal{C}}\sum_{w\in\mathbf{S}_{i}} \frac{\exp(\mathbf{u}^{T}_{w}\mathbf{v}_{c_{i}})}{\sum_{c^{\prime}\in\mathcal{C}}\exp( \mathbf{u}^{T}_{w}\mathbf{v}_{c^{\prime}})}}_{\mathcal{J}_{\mathrm{category}}}.\end{split} \tag{12}\]

Now we prove that maximizing \(\mathcal{J}\) is implicitly performing the factorization in Eq. (3).

Levy and Goldberg (2014) have proved that maximizing \(\mathcal{J}_{\mathrm{context}}\) is implicitly doing the following factorization.

\[\begin{split}\mathbf{u}^{T}_{w_{i}}\mathbf{v}_{w_{j}}=&\log \big{(}\frac{\#_{\mathcal{D}}(w_{i},w_{j})\cdot\lambda_{\mathcal{D}}}{\#_{ \mathcal{D}}(w_{i})\cdot\#_{\mathcal{D}}(w_{j})\cdot b}\big{)},\\ \text{i.e.,}&\quad\mathbf{U}^{T}_{w}\mathbf{V}_{w}= \mathbf{X}_{ww}.\end{split} \tag{13}\]

We follow their approach to consider the other two terms \(\mathcal{J}_{\mathrm{document}}\) and \(\mathcal{J}_{\mathrm{category}}\) in Eq. (12). Using the negative sampling strategy to rewrite \(\mathcal{J}_{\mathrm{document}}\), we get

\[\sum_{w\in\mathcal{V}_{\mathcal{D}}}\sum_{d\in\mathcal{D}}\#_{d}(w)\Big{(}\log \sigma(\mathbf{u}^{T}_{w}\mathbf{v}_{d})+b\mathbb{E}_{d^{\prime}}\big{[}\log\sigma(-\mathbf{ u}^{T}_{w}\mathbf{v}_{d^{\prime}})]\Big{)}, \tag{14}\]

where \(\sigma(\cdot)\) is the sigmoid function. Following Levy and Goldberg (2014); Qiu et al. (2018), we assume the negative sampling distribution \(\propto\lambda_{d}\).15 Then, the objective becomes

Footnote 15: In practice, the negative sampling distribution \(\propto\lambda_{d}^{3/4}\), but related studies Levy and Goldberg (2014); Qiu et al. (2018) usually assume a linear relationship in their derivation.

\[\begin{split}&\sum_{w\in\mathcal{V}_{\mathcal{D}}}\sum_{d\in \mathcal{D}}\#_{d}(w)\log\sigma(\mathbf{u}^{T}_{w}\mathbf{v}_{d})+\\ &\sum_{w\in\mathcal{V}_{\mathcal{D}}}\#_{\mathcal{D}}(w)\sum_{d^{ \prime}\in\mathcal{D}}\frac{b\cdot\lambda_{d^{\prime}}}{\lambda_{\mathcal{D}}} \log\sigma(-\mathbf{u}^{T}_{w}\mathbf{v}_{d^{\prime}}).\end{split} \tag{15}\]

For a specific term-document pair \((w,d)\), we consider its effect in the objective:

\[\mathcal{J}_{w,d}=\#_{d}(w)\log\sigma(\mathbf{u}^{T}_{w}\mathbf{v}_{d})+\#_{\mathcal{ D}}(w)\frac{b\cdot\lambda_{d}}{\lambda_{\mathcal{D}}}\log\sigma(-\mathbf{u}^{T}_{w}\mathbf{ v}_{d}). \tag{16}\]

Let \(x_{w,d}=\mathbf{u}^{T}_{w}\mathbf{v}_{d}\). To maximize \(\mathcal{J}_{w,d}\), we should have

\[0=\frac{\partial\mathcal{J}_{w,d}}{\partial x_{w,d}}=\#_{d}(w)\sigma(-x_{w,d})- \frac{\#_{\mathcal{D}}(w)\cdot b\cdot\lambda_{d}}{\lambda_{\mathcal{D}}}\sigma(x_ {w,d}).\] (17) 

That is,

\[e^{2x_{w.d}-\big{(}\frac{\#_{d}(w)\cdot\lambda_{\mathcal{D}}}{\#_{ \mathcal{D}}(w)\cdot b\cdot\lambda_{d}}-1\big{)}e^{x_{w.d}}-\frac{\#_{d}(w)\cdot \lambda_{\mathcal{D}}}{\#_{\mathcal{D}}(w)\cdot b\cdot\lambda_{d}}=0. \tag{18}\]

Therefore, \(e^{x_{w.d}}=-1\) (which is invalid) or \(e^{x_{w.d}}=\frac{\#_{d}(w)\cdot\lambda_{\mathcal{D}}}{\#_{\mathcal{D}}(w) \cdot b\cdot\lambda_{d}}\). In other words,

\[\begin{split}& u_{w}^{T}v_{d}=x_{w,d}=\log\Big{(}\frac{\#_{ d}(w)\cdot\lambda_{\mathcal{D}}}{\#_{\mathcal{D}}(w)\cdot b\cdot\lambda_{d}} \Big{)},\\ &\text{i.e., }\mathbf{U}_{w}^{T}\mathbf{V}_{d}=\mathbf{X}_{wd}. \end{split} \tag{19}\]

Similarly, for \(\mathcal{J}_{\mathrm{category}}\), the objective can be rewritten as

\[\begin{split}&\sum_{w\in\mathcal{V}_{\mathcal{D}}}\sum_{c_{i}\in \mathcal{C}}\textbf{I}_{w\in\mathcal{S}_{t}}\log\sigma(\mathbf{u}_{w}^{T}\mathbf{v}_{ c_{i}})+\\ &\sum_{w\in\mathcal{V}_{\mathcal{D}}}\textbf{I}_{w\in\mathcal{ S}_{1}\cup...\cup\mathcal{S}_{|\mathcal{C}}}\sum_{c^{\prime}\in\mathcal{C}}\frac{b}{| \mathcal{C}|}\log\sigma(-\mathbf{u}_{w}^{T}\mathbf{v}_{c^{\prime}}).\end{split} \tag{20}\]

Following the derivation of \(\mathcal{J}_{\mathrm{document}}\), we get

\[\begin{split}&\mathbf{u}_{w}^{T}\mathbf{v}_{c_{i}}=x_{w,c_{i}}=\log \Big{(}\frac{\textbf{I}_{w\in\mathcal{S}_{i}}|\mathcal{C}|}{\textbf{I}_{w\in \mathcal{S}_{1}\cup...\cup\mathcal{S}_{|\mathcal{C}|}}\cdot b}\Big{)},\\ &\text{i.e., }\mathbf{U}_{w}^{T}\mathbf{V}_{c_{i}}=\mathbf{ X}_{wc}.\end{split} \tag{21}\]

Putting Eqs. (13), (19), and (21) together gives us Eq. (3).

## Appendix B The Ensemble Ranking Function

In Section 3.3, we propose the following ensemble ranking function:

\[\operatorname{score}(w|\mathcal{S}_{i})=\bigg{(}\frac{1}{2}\big{(}\frac{1}{ \operatorname{rank}_{G}(w)}\big{)}^{\rho}+\frac{1}{2}\big{(}\frac{1}{ \operatorname{rank}_{L}(w)}\big{)}^{\rho}\bigg{)}^{1/\rho}. \tag{22}\]

Now we prove this ranking function is a generalization of the _arithmetic_ mean reciprocal rank (i.e., MRR) and the _geometric_ mean reciprocal rank:

\[\begin{split}&\lim_{\rho\to 1}\operatorname{score}(w|\mathcal{S}_{ i})=\frac{1}{2}\Big{(}\frac{1}{\operatorname{rank}_{G}(w)}+\frac{1}{ \operatorname{rank}_{L}(w)}\Big{)};\\ &\lim_{\rho\to 0}\operatorname{score}(w|\mathcal{S}_{ i})=\Big{(}\frac{1}{\operatorname{rank}_{G}(w)}\cdot\frac{1}{ \operatorname{rank}_{L}(w)}\Big{)}^{1/2}.\end{split} \tag{23}\]

The case of \(\rho\to 1\) is trivial. When \(\rho\to 0\), we aim to show that

\[\begin{split}\lim_{\rho\to 0}\log\operatorname{score}( w|\mathcal{S}_{i})=\log\Big{(}\frac{1}{\operatorname{rank}_{G}(w)}\cdot\frac{1}{ \operatorname{rank}_{L}(w)}\Big{)}^{1/2}.\end{split} \tag{24}\]

In fact, let \(r_{G}=\frac{1}{\operatorname{rank}_{K}(w)}\) and \(r_{L}=\frac{1}{\operatorname{rank}_{L}(w)}\).

\[\begin{split}\lim_{\rho\to 0}\log\operatorname{ score}(w|\mathcal{S}_{i})&=\lim_{\rho\to 0}\log\Big{(}\frac{1}{2} r_{G}^{\rho}+\frac{1}{2}r_{L}^{\rho}\Big{)}^{1/\rho}\\ &=\lim_{\rho\to 0}\frac{\log\big{(}\frac{1}{2}r_{G}^{\rho}+\frac{1}{2}r_{ L}^{\rho}\big{)}}{\rho}\\ &=\lim_{\rho\to 0}\frac{\frac{1}{2}r_{G}^{\rho}\log r_{G}+s \frac{1}{2}r_{L}^{\rho}\log r_{L}}{\frac{3}{2}r_{G}^{\rho}+s\frac{1}{2}r_{L}^{ \rho}}\\ &=\frac{\lim_{\rho\to 0}\left(r_{G}^{\rho}\log r_{G}+r_{L}^{\rho}\log r _{L}\right)}{\lim_{\rho\to 0}\left(r_{G}^{\rho}+r_{L}^{\rho}\right)}\\ &=\frac{\log r_{G}+\log r_{L}}{2}\\ &=\log(r_{G}\cdot r_{L})^{1/2}.\end{split} \tag{25}\]

The third line is obtained by applying L'Hopital's rule.