SlowGuess/NAACL_2022_OCR huggingface dataset:
Yu Zhang $^{1}$ , Yu Meng $^{1}$ , Xuan Wang $^{1}$ , Sheng Wang $^{2}$ , Jiawei Han $^{1}$
{yuz9,yumeng5,xwang174,hanj}@illinois.edu swang@cs.washington.edu
Discovering latent topics from text corpora has been studied for decades. Many existing topic models adopt a fully unsupervised setting, and their discovered topics may not cater to users' particular interests due to their inability of leveraging user guidance. Although there exist seed-guided topic discovery approaches that leverage user-provided seeds to discover topic-representative terms, they are less concerned with two factors: (1) the existence of out-of-vocabulary seeds and (2) the power of pretrained language models (PLMs). In this paper, we generalize the task of seed-guided topic discovery to allow out-of-vocabulary seeds. We propose a novel framework, named SEE-TOPIC, wherein the general knowledge of PLMs and the local semantics learned from the input corpus can mutually benefit each other. Experiments on three real datasets from different domains demonstrate the effectiveness of SEETOPIC in terms of topic coherence, accuracy, and diversity. $^{1}$
Automatically discovering informative and coherent topics from massive text corpora is central to text analysis through helping users efficiently digest a large collection of documents (Griffiths and Steyvers, 2004) and advancing downstream applications such as summarization (Wang et al., 2009, 2022), classification (Chen et al., 2015; Meng et al., 2020b), and generation (Liu et al., 2021).
Unsupervised topic models have been the mainstream approach to topic discovery since the proposal of pLSA (Hofmann, 1999) and LDA (Blei et al., 2003). Despite their encouraging performance in finding informative latent topics, these topics may not reflect user preferences well, mainly due to their unsupervised nature. For example, given a collection of product reviews, a user may be specifically interested in product categories
Table 1: Three datasets (Cohan et al., 2020; McAuley and Leskovec, 2013; Zhang et al., 2017) from different domains and their topic categories (i.e., seeds). Red: Seeds never seen in the corpus (i.e., out-of-vocabulary). In all three datasets, a large proportion of seeds are out-of-vocabulary.
Dataset	Category Names (Seeds)
SciDocs (Scientific Papers)	cardiovascular diseases	hepatitis a/b/c/e
chronic kidney disease	mental disorders
chronic respiratory diseases	musculoskeletal disorders
diabetes mellitus	neoplasms (cancer)
digestive diseases	neurological disorders
hiv/aims
Amazon (Product Reviews)	apps for android	health and personal care
books	home and kitchen
cds and vinyl	movies and tv
clothing, shoes and jewelry	sports and outdoors
electronics	video games
Twitter (Social Media Posts)	food	residence
shop and service	outdoors and recreation
travel and transport	arts and entertainment
college and university	professional and other places
nightlife spot
(e.g., "books", "electronics"), but unsupervised topic models may generate topics containing different sentiments (e.g., "good", "bad"). To consider users' interests and needs, seed-guided topic discovery approaches (Jagarlamudi et al., 2012; Gallagher et al., 2017; Meng et al., 2020a) have been proposed to find representative terms for each category based on user-provided seeds or category names. However, there are still two less concerned factors in these approaches.
The Existence of Out-of-Vocabulary Seeds. Previous studies (Jagarlamudi et al., 2012; Gallagher et al., 2017; Meng et al., 2020a) assume that all user-provided seeds must be in-vocabulary (i.e., appear at least once in the input corpus), so that they can utilize the occurrence statistics or SkipGram embedding methods (Mikolov et al., 2013) to model seed semantics. However, user-interested categories can have specific or composite descriptions, which may never appear in the corpus. Table 1 shows three datasets from different domains: sci
2In this paper, we use "seeds" and "category names" interchangeably.
1The code and datasets are available at https://github.com/yuzhimanhua/SeeTopic.
Human Language Technologies, pages 291 - 302
entific papers, product reviews, and social media posts. In each dataset, documents can belong to one or more categories, and we list the category names provided by the dataset collectors. These seeds should reflect their particular interests. In all three datasets, we have a large proportion of seeds (45% in SciDocs, 60% in Amazon, and 78% in Twitter) that never appear in the corpus. Some category names are too specific (e.g., "chronic respiratory diseases", "nightlife spot") to be exactly matched, others are the composition of multiple entities (e.g., "hepatitis a/b/c/e", "neoplasms (cancer)", "clothing, shoes and jewelry").<sup>3</sup>
The Power of Pre-trained Language Models. Techniques used in previous studies are mainly based on LDA variants (Jagarlamudi et al., 2012) or context-free embeddings (Meng et al., 2020a). Recently, pre-trained language models (PLMs) such as BERT (Devlin et al., 2019) have achieved significant improvement in a wide range of text mining tasks. In topic discovery, the generic representation power of PLMs learned from web-scale corpora (e.g., Wikipedia or PubMed) may complement the information a model can obtain from the input corpus. Moreover, out-of-vocabulary seeds usually have meaningful in-vocabulary components (e.g., "night" and "life" in "nightlife spot", "health" and "care" in "health and personal care"). The optimized tokenization strategy of PLMs (Sennrich et al., 2016; Wu et al., 2016) can help segment the seeds into such meaningful components (e.g., "nightlife"  $\rightarrow$  "night" and "#lifel"), and the contextualization power of PLMs can help infer the correct meaning of each component (e.g., "#lifel" and "care") in the category name. Therefore, PLMs are much needed in handling out-of-vocabulary seeds and effectively learning their semantics.
Contributions. Being aware of these two factors, in this paper, we study seed-guided topic discovery in the presence of out-of-vocabulary seeds. Our proposed SEETOPIC framework consists of two modules: (1) The general representation module
uses a PLM to derive the representation of each term (including out-of-vocabulary seeds) based on the general linguistic knowledge acquired through pre-training. (2) The seed-guided local representation module learns in-vocabulary term embeddings specific to the input corpus and the given seeds. In order to optimize the learned representations for topic coherence, which is commonly reflected by pointwise mutual information (PMI) (Newman et al., 2010), our objective implicitly maximizes the PMI between each word and its context, the documents it appears, as well as the category it belongs to. The learning of the two modules is connected through an iterative ensemble ranking process, in which the general knowledge of PLMs and the term representations specifically learned from the target corpus conditioned on the seeds can complement each other.
To summarize, this study makes three contributions. (1) Task: we propose to study seed-guided topic discovery in the presence of out-of-vocabulary seeds. (2) Framework: we design a unified framework that jointly models general knowledge through PLMs and local corpus statistics through embedding learning. (3) Experiment: extensive experiments on three datasets demonstrate the effectiveness of SEETOPIC in terms of topic coherence, accuracy, and diversity.
As shown in Table 1, we assume a seed can be either a single word or a phrase. Given a corpus  $\mathcal{D}$ , we use  $\nu_{\mathcal{D}}$  to denote the set of terms appearing in  $\mathcal{D}$ . In accordance with the assumption of category names, each term can also be a single word or a phrase. In practice, given a raw corpus, one can use existing phrase chunking tools (Manning et al., 2014; Shang et al., 2018) to detect phrases in it. After phrase chunking, if a category name is still not in  $\nu_{\mathcal{D}}$ , we define it as out-of-vocabulary.
Problem Definition. Given a corpus  $\mathcal{D} = \{d_1, \dots, d_{|\mathcal{D}|}\}$  and a set of category names  $\mathcal{C} = \{c_1, \dots, c_{|\mathcal{C}|}\}$  where some category names are out-of-vocabulary, the task is to find a set of invocabulary terms  $S_i = \{w_1, \dots, w_S\} \subseteq \mathcal{V}_{\mathcal{D}}$  for each category  $c_i$  such that each term in  $\mathcal{S}_i$  is semantically close to  $c_i$  and far from other categories  $c_j$  ( $\forall j \neq i$ ).
3 The SEETOPIC Framework
In this section, we first introduce how we model general and local text semantics using a PLM mod
One possible idea to deal with composite seeds is to split them into multiple seeds. However, there are many possible ways to express the conjunctions (e.g., “/”, “()”, “,” and “and” in Table 1), which may require manual tuning. Besides, simple chunking rules will induce splits that break the semantics of the original composition (e.g., “professional and other places” may be split into “professional” and “other places”). Moreover, even after the split, some seeds are still out-of-vocabulary. Therefore, we propose to use PLMs to tackle out-of-vocabulary seeds in a unified way. In experiments, we will show that our model is able to tackle composite seeds. For example, given the seed “hepatitis  $a/b/c/e$ ”, we can find terms relevant to “hepatitis  $b$ ” and “hepatitis  $c$ ” (see Table 4).
ule and a seed-guided embedding learning module, respectively. Then, we present the iterative ensemble ranking process and our overall framework.
3.1 Modeling General Text Semantics using a PLM
PLMs such as BERT (Devlin et al., 2019) aim to learn generic language representations from web-scale corpora (e.g., Wikipedia or PubMed) that can be applied to a wide variety of text-related applications. To transfer such general knowledge to our topic discovery task, we employ a PLM to encode each category name and each in-vocabulary term to a vector. To be specific, given a term  $w \in \mathcal{C} \cup \mathcal{V}_{\mathcal{D}}$ , we input the sequence "[CLS]  $w$  [SEP]" into the PLM. Here,  $w$  can be a phrase containing multiple words, and each word can be out of the PLM's vocabulary. To deal with this, most PLMs use a pre-trained tokenizer (Sennrich et al., 2016; Wu et al., 2016) to segment each unseen word into frequent subwords. Then, the contextualization power of PLMs will help infer the correct meaning of each word/subword, so as to provide a more precise representation of the whole category.
After LM encoding, following (Sia et al., 2020; Thompson and Mimno, 2020; Li et al., 2020), we take the output of all tokens from the last layer and average them to get the term embedding  $e_w$ . In this way, even if a seed  $c_i$  is out-of-vocabulary, we can still obtain its representation  $e_{c_i}$ .
3.2 Modeling Local Text Semantics in the Input Corpus
The motivation of topic discovery is to discover latent topic structures from the input corpus. Therefore, purely relying on general knowledge in the PLM is insufficient because topic discovery results should adapt to the input corpus  $\mathcal{D}$ . Now, we introduce how we learn another set of embeddings  $\{\pmb{u}_w|w\in \mathcal{V}_{\mathcal{D}}\}$  from  $\mathcal{D}$ .
Previous studies on embedding learning assume that the semantic of a term is similar to its local context (Mikolov et al., 2013), the document it appears (Tang et al., 2015; Xun et al., 2017a), and the category it belongs to (Meng et al., 2020a). Inspired by these studies, we propose the following embedding learning objective.
$$
\begin{array}{l} \mathcal {J} = \underbrace {\sum_ {d \in \mathcal {D}} \sum_ {w _ {i} \in d} \sum_ {w _ {j} \in \mathcal {C} \left(w _ {i} , h\right)} p \left(w _ {j} \mid w _ {i}\right)} _ {\text {c o n t e x t}} (1) \\ + \underbrace {\sum_ {d \in \mathcal {D}} \sum_ {w \in d} p (d \mid w)} _ {\text {d o c u m e n t}} + \underbrace {\sum_ {c _ {i} \in \mathcal {C}} \sum_ {w \in \mathcal {S} _ {i}} p \left(c _ {i} \mid w\right)} _ {\text {c a t e g o r y}}, (1) \\ \end{array}
$$
$$
p (z \mid w) = \frac {\exp \left(\boldsymbol {u} _ {w} ^ {T} \boldsymbol {v} _ {z}\right)}{\sum_ {z ^ {\prime}} \exp \left(\boldsymbol {u} _ {w} ^ {T} \boldsymbol {v} _ {z ^ {\prime}}\right)}, \quad (z \text {c a n b e} w _ {j}, d, \text {o r} c _ {i}). \tag {2}
$$
In this objective,  $\mathbf{u}_{w_i}$  (and  $\mathbf{v}_{w_j}$ ),  $\mathbf{v}_d$ ,  $\mathbf{v}_{c_i}$  are the embedding vectors of terms, documents, and categories, respectively.  $\mathcal{C}(w_i, h)$  is the set of context terms of  $w_i$  in  $d$ . Specifically, if  $d = w_1w_2\ldots w_L$ , then  $\mathcal{C}(w_i, h) = \{w_j | i - h \leq j \leq i + h, j \neq i\}$ , where  $h$  is the context window size.
Note that the last term in Eq. (1) encourages the similarity between each category  $c_{i}$  and its representative terms  $S_{i}$ . Here, we adopt an iterative process to gradually update category-representative terms. Initially,  $S_{i}$  consists of just a few invocabulary terms similar to  $c_{i}$  according to the PLM. At each iteration, the size of  $S_{i}$  will increase to contain more category-discriminative terms (the selection criterion of these terms will be introduced in the next section), and we need to encourage their proximity with  $c_{i}$  in the next iteration.
Directly optimizing the full softmax in Eq. (2) is costly. Therefore, we adopt the negative sampling strategy (Mikolov et al., 2013) for efficient approximation.
Interpreting the Objective. In topic modeling studies, pointwise mutual information (PMI) (Newman et al., 2010) is a standard evaluation metric for topic coherence (Lau et al., 2014; Roder et al., 2015). Levy and Goldberg (2014) prove that the Skip-Gram embedding model is implicitly factorizing the PMI matrix. Following their proof, we can show that maximizing Eq. (1) is implicitly doing the following factorization:
$$
\mathbf {U} _ {w} ^ {T} \left[ \mathbf {V} _ {w}; \mathbf {V} _ {d}; \mathbf {V} _ {c} \right] = \left[ \mathbf {X} _ {w w}; \mathbf {X} _ {w d}; \mathbf {X} _ {w c} \right], \tag {3}
$$
where the columns of  $\mathbf{U}_w, \mathbf{V}_w, \mathbf{V}_d, \mathbf{V}_c$  are  $\pmb{u}_{w_i}$ ,  $\pmb{v}_{w_j}$ ,  $\pmb{v}_d$ ,  $\pmb{v}_{c_i}$ , respectively ( $w_i, w_j \in \mathcal{V}_{\mathcal{D}}$ ,  $d \in \mathcal{D}$ ,  $c_i \in \mathcal{C}$ );  $\mathbf{X}_{ww}, \mathbf{X}_{wd}$ , and  $\mathbf{X}_{wc}$  are PMI matrices.
$$
\begin{array}{l} \mathbf {X} _ {w w} = \left[ \log \left(\frac {\# _ {\mathcal {D}} (w _ {i} , w _ {j}) \cdot \lambda_ {\mathcal {D}}}{\# _ {\mathcal {D}} (w _ {i}) \cdot \# _ {\mathcal {D}} (w _ {j}) \cdot b}\right) \right] _ {w _ {i}, w _ {j} \in \mathcal {V} _ {\mathcal {D}}}, \\ \mathbf {X} _ {w d} = \left[ \log \left(\frac {\# _ {d} (w) \cdot \lambda_ {\mathcal {D}}}{\# _ {\mathcal {D}} (w) \cdot \lambda_ {d} \cdot b}\right) \right] _ {w \in \mathcal {V} _ {\mathcal {D}}, d \in \mathcal {D}}, \tag {4} \\ \mathbf {X} _ {w c} = \left[ x _ {w, c _ {i}} \right] _ {w \in \mathcal {V} _ {\mathcal {D}}, c _ {i} \in \mathcal {C}}, \quad \text {w h e r e} \\ x _ {w, c _ {i}} = \left\{ \begin{array}{l l} \log \frac {| C |}{b}, & \text {i f} w \in \mathcal {S} _ {i}, \\ - \infty , & \text {i f} w \in \mathcal {S} _ {j} (\forall j \neq i). \end{array} \right. \\ \end{array}
$$
Here,  $\#_{\mathcal{D}}(w_i, w_j)$  denotes the number of co-occurrences of  $w_i$  and  $w_j$  in a context window in  $\mathcal{D}$ ;  $\#_{\mathcal{D}}(w)$  denotes the number of occurrences of  $w$
in  $\mathcal{D}$ ;  $\lambda_{\mathcal{D}}$  is the total number of terms in  $\mathcal{D}$ ;  $\#_d(w)$  denotes the number of times  $w$  occurs in  $d$ ;  $\lambda_d$  is the total number of terms in  $d$ ;  $b$  is the number of negative samples. (For the derivation of Eq. (3), please refer to Appendix A.)
To summarize, the learned local representations  $\pmb{u}_w$  are implicitly optimized for topic coherence, where term co-occurrences are measured in context, document, and category levels.
3.3 Ensemble Ranking
We have obtained two sets of term embeddings that model text semantics from different angles:  $\{e_w|w\in \mathcal{C}\cup \mathcal{V}_D\}$  carries the PLM's knowledge, while  $\{\pmb {u}_w|w\in \mathcal{V}_D\}$  models the input corpus as well as user-provided seeds. We now propose an ensemble ranking method to leverage information from both sides to grab more discriminative terms for each category.
Given a category  $c_{i}$  and its current term set  $S_{i}$ , we first calculate the scores of each term  $w \in \mathcal{V}_{\mathcal{D}}$ .
$$
\operatorname {s c o r e} _ {G} (w | \mathcal {S} _ {i}) = \frac {1}{| \mathcal {S} _ {i} |} \sum_ {w ^ {\prime} \in \mathcal {S} _ {i}} \cos \left(\boldsymbol {e} _ {w}, \boldsymbol {e} _ {w ^ {\prime}}\right), \tag {5}
$$
$$
\mathrm {s c o r e} _ {L} (w | \mathcal {S} _ {i}) = \frac {1}{| \mathcal {S} _ {i} |} \sum_ {w ^ {\prime} \in \mathcal {S} _ {i}} \cos (\boldsymbol {u} _ {w}, \boldsymbol {u} _ {w ^ {\prime}}).
$$
The subscript  $G$  here means "general", while  $L$  means "local". Then, we sort all terms by these two scores, respectively. Each term  $w$  will hence get two rank positions  $\mathrm{rank}_G(w)$  and  $\mathrm{rank}_L(w)$ . We propose the following ensemble score based on the reciprocal rank:
$$
\operatorname {s c o r e} (w | \mathcal {S} _ {i}) = \left(\frac {1}{2} \left(\frac {1}{\operatorname {r a n k} _ {G} (w)}\right) ^ {\rho} + \frac {1}{2} \left(\frac {1}{\operatorname {r a n k} _ {L} (w)}\right) ^ {\rho}\right) ^ {1 / \rho}. \tag {6}
$$
Here,  $0 < \rho \leq 1$  is a constant. In practice, instead of ranking all terms in the vocabulary, we only check the top- $M$  results in the two ranking lists. If a term  $w$  is not among the top- $M$  according to  $\mathrm{score}_G(w)$  (resp.,  $\mathrm{score}_L(w)$ ), we set  $\mathrm{rank}_G(w) = +\infty$  (resp.,  $\mathrm{rank}_L(w) = +\infty$ ). In fact, when  $\rho = 1$ , Eq. (6) becomes the arithmetic mean of the two reciprocal ranks  $\frac{1}{\mathrm{rank}_G(w)}$  and  $\frac{1}{\mathrm{rank}_L(w)}$ . This is essentially the mean reciprocal rank (MRR) commonly used in ensemble ranking, where a high position in one ranking list can largely compensate a low position in the other. In contrast, when  $\rho \to 0$ , Eq. (6) becomes the geometric mean of the two reciprocal ranks (see Appendix B), where two ranking lists both have the "veto power" (i.e., a term needs to be ranked as top- $M$  in both ranking lists to obtain a non-zero
ensemble score). In experiment, we set  $\rho = 0.1$  and show it outperforms MRR (i.e.,  $\rho = 1$ ) in our topic discovery task.
After computing the ensemble score score  $(w|S_{i})$  for each  $w$ , we update  $S_{i}$ . To guarantee that each  $S_{i}$  is category-discriminative, we do not allow any term to belong to more than one category. Therefore, we gradually expand each  $S_{i}$  by turns. At the beginning, we reset  $S_{1} = \ldots = S_{|\mathcal{C}|} = \emptyset$ . When it is  $S_{i}$ 's turn, we add one term  $S_{i}$  according to the following criterion:
$$
\mathcal {S} _ {i} \leftarrow \mathcal {S} _ {i} \cup \left\{\underset {w \in \mathcal {V} _ {\mathcal {D}} \backslash (\mathcal {S} _ {1} \cup \dots \cup \mathcal {S} _ {| \mathcal {C} |})} {\arg \max } \operatorname {s c o r e} (w | \mathcal {S} _ {i}) \right\}. \tag {7}
$$
3.4 Overall Framework
We summarize the entire SEETOPIC framework in Algorithm 1. To deal with out-of-vocabulary category names, we first utilize a PLM to find their nearest in-vocabulary terms as the initial category-discriminative term set  $S_{i}$  (Lines 1-7). After initialization,  $|\mathcal{S}_i| = N$  ( $\forall 1 \leq i \leq |\mathcal{C}|$ ). Note that for an in-vocabulary category name  $c_{i} \in \mathcal{V}_{\mathcal{D}}$ , itself will be added to the initial  $S_{i}$  as the top-1 similar in-vocabulary term.
After getting the initial  $S_{i}$ , we update it by  $T$  iterations (Lines 8-16). At each iteration, according to the up-to-date  $\mathcal{S}_1, \mathcal{S}_2, \dots, \mathcal{S}_{|\mathcal{C}|}$ , we relearn embeddings  $\pmb{u}_w, \pmb{v}_w, \pmb{v}_d$ , and  $\pmb{v}_{c_i}$  using Eq. (1) (Line 10). The two sets of embeddings,  $\{e_w | w \in \mathcal{C} \cup \mathcal{V}_{\mathcal{D}}\}$  (computed at Line 1) and  $\{\pmb{u}_w | w \in \mathcal{V}_{\mathcal{D}}\}$  (updated at Line 10), are then leveraged to perform ensemble ranking (Lines 11-12). Based on the
ensemble score score  $(w|\mathcal{S}_i)$ , we update  $\mathcal{S}_i$  using Eq. (7) (Lines 13-16). After the  $t$ -th iteration,  $|\mathcal{S}_i| = (t + 1)N$  ( $\forall 1 \leq i \leq |\mathcal{C}|$ ).
Complexity Analysis. The time complexity of using the PLM is  $\mathcal{O}((|\mathcal{C}| + |\mathcal{V}_{\mathcal{D}}|)\alpha_{\mathrm{PLM}})$ , where  $\alpha_{\mathrm{PLM}}$  is the complexity of encoding one term via the PLM. The total complexity of local embedding is  $\mathcal{O}(T\lambda_{\mathcal{D}}(h + |\mathcal{C}|)b)$  because in each iteration  $1 \leq t \leq T$ , every  $w \in \mathcal{D}$  interacts with every other term in the context window of size  $h$ , its belonging document, and each category  $c_i \in \mathcal{C}$ , and each update involves  $b$  negative samples. The total complexity of ensemble ranking is  $\mathcal{O}(T|\mathcal{V}_{\mathcal{D}}||\mathcal{C}||\mathcal{S}_i|)$  as in each iteration  $1 \leq t \leq T$ , we compute scores between each  $w \in \mathcal{V}_{\mathcal{D}}$  and each  $w' \in \mathcal{S}_i$  ( $\forall c_i \in \mathcal{C}$ ).
Datasets. We conduct experiments on three public datasets from different domains: (1) SciDocs (Cohan et al., 2020) $^4$  is a large collection of scientific papers supporting diverse evaluation tasks. For the MeSH classification task (Coletti and Bleich, 2001), about 23K medical papers are collected, each of which is assigned to one of the 11 common disease categories derived from the MeSH vocabulary. We use the title and abstract of each paper as documents and the 11 category names as seeds. (2) Amazon (McAuley and Leskovec, 2013) $^5$  contains product reviews from May 1996 to July 2014. Each Amazon review belongs to one or more product categories. We use the subset sampled by Zhang et al. (2020, 2022), which contains 10 categories and 100K reviews. (3) Twitter (Zhang et al., 2017) $^6$  is a crawl of geo-tagged tweets in New York City from August 2014 to November 2014. The dataset collectors link these tweets with Foursquare's POI database and assign them to 9 POI categories. We take these category names as input seeds.
Seeds used in the three datasets are shown in Table 1. Dataset statistics are summarized in Table 2. For all three datasets, we use AutoPhrase (Shang et al., 2018)<sup>7</sup> to perform phrase chunking in the corpus, and we remove words and phrases occurring less than 3 times.
Previous studies (Jagarlamudi et al., 2012; Meng et al., 2020a) have tried some other datasets (e.g., RCV1, 20 Newsgroups, NYT, and Yelp). However, the category names they use in these datasets are
Table 2: Dataset Statistics.
Dataset	SciDocs	Amazon	Twitter
#Documents	23,473	100,000	135,529
#In-vocabulary Terms (After Phrase Chunking)	55,897	56,942	17,577
Avg Doc Length	239.8	119.0	6.7
#Seeds	11	10	9
#Out-of-vocabulary Seeds (After Phrase Chunking)	5	6	7
all picked from in-vocabulary terms. Therefore, we do not consider these datasets for evaluation in our task settings.
Following (Sia et al., 2020), we adopt a 60-40 train-test split for all three datasets. The training set is used as the input corpus  $\mathcal{D}$ , and the testing set is used for calculating topic coherence metrics (see evaluation metrics for details).