SlowGuess/ACL_2020_OCR huggingface dataset:
2. SetExpan (Shen et al., 2017): This method iteratively selects skip-gram context features from the corpus and develops a rank ensemble mechanism to score and select entities.
3. SetExpander (Mamou et al., 2018): This method trains different embeddings based on different types of context features and leverages additional human-annotated sets to build a classifier on top of learned embeddings to predict whether an entity belongs to the set.
4. CaSE (Yu et al., 2019b): This method combines entity skip-gram context feature and embedding features to score and rank entities once from the corpus. The original paper has three variants and we use the CaSE-W2V variant since it is the best model claimed in the paper.
5. MCTS (Yan et al., 2019): This method bootstraps the initial seed set by combing the Monte Carlo Tree Search algorithm with a deep similarity network to estimate delayed feedback for pattern evaluation and to score entities given selected patterns.
6. CGExpan: This method is our proposed Class-Guided Set Expansion framework, using BERT (Devlin et al., 2019) as the pre-trained language model. We include two versions of our full model, namely CGExpan-Comb and CGExpan-MRR, that use the combination of score and mean reciprocal rank for rank aggregation, respectively.
7. CGExpan-NoCN: An ablation of CGExpan that excludes the class name guidance. Therefore, it only incorporates the average BERT representation to select entities.
8. CGExpan-NoFilter: An ablation of CGExpan
Table 2: Mean Average Precision on Wiki and APR. “ $\nabla$ ” means the number is directly from the original paper.
CGExpan vs. Other	MAP@10	MAP@20	MAP@50
vs. SetExpan	100%	94.5%	87.3%
vs. CGExpan-NoFilter	100%	94.5%	58.2%
vs. CGExpan-NoCN	100%	94.5%	70.9%
Table 3: Ratio of seed entity set queries on which the first method reaches better or the same performance as the second method.
that excludes the negative class name selection step and uses only the single positive class name in the entity selection module.
Evaluation Metric. We follow previous studies and evaluate set expansion results using Mean Average Precision at different top  $K$  positions (MAP@  $K$ ) as below:
$$
\operatorname {M A P} @ K = \frac {1}{| Q |} \sum_ {q \in Q} \mathrm {A P} _ {K} \left(L _ {q}, S _ {q}\right),
$$
where  $Q$  is the set of all seed queries and for each query  $q$ , we use  $\mathrm{AP}_K(L_q, S_q)$  to denote the traditional average precision at position  $K$  given a ranked list of entities  $L_q$  and a ground-truth set  $S_q$ .
Implementation Details. For CGExpan, we use BERT-base-uncased $^4$  as our pre-trained LM. For parameter setting, in the class name generation module (Sec. 3.1), we take top-3 predicted tokens in each level of beam search and set the maximum length of generated class names up to 3. When calculating the similarity between an entity and a class name (Eq. 1), we choose  $k = 5$ , and will later provide a parameter study on  $k$  in the experiment. Also, since MAP@ $K$  for  $K = 10, 20, 50$  are typically used for set expansion evaluations, we follow the convention and choose 50 as the target set size in our experiments. $^5$
4In principle, other masked LMs such as RoBERTa and XLNet can also be used in our framework.
5The code and data are available at https://github. com/yzhan238/CGExpan
8156
Methods	Wiki
MAP@{10/20/50}	APR
MAP@{10/20/50}
Oracle-Full	0.991/0.976/0.891	1.000/1.000/0.964
Oracle-NoFilter	0.994/0.983/0.887	0.988/0.966/0.894
CGExpan	0.995/0.978/0.902	0.992/0.990/0.955
Table 4: Compared to oracle models knowing ground truth class names, CGExpan automatically generates class names and achieves comparative performances.
Overall Performance. Table 2 shows the overall performance of different entity set expansion methods. We can see that CGExpan along with its ablations in general outperform all the baselines by a large margin. Comparing with SetExpan, the full model CGExpan achieves  $24\%$  improvement in MAP@50 on the Wiki dataset and  $49\%$  improvement in MAP@50 on the APR dataset, which verifies that our class-guided model can refine the expansion process and reduce the effect of erroneous entities on later iterations. In addition, CGExpan-NoCN outperforms most baseline models, meaning that the pre-trained LM itself is powerful to capture entity similarities. However, it still cannot beat CGExpan-NoFilter model, which shows that we can properly guide the set expansion process by incorporating generated class names. Moreover, by comparing our full model with CGExpan-NoFilter, we can see that negative class names indeed help the expansion process by estimating a clear boundary for the target class and filtering out erroneous entities. Such an improvement is particularly obvious on the APR dataset. The two versions of our full model overall have comparable performance, but CGExpan-MRR consistently outperforms CGExpan-Comb. To explain such a difference, empirically we observe that high-quality entities tend to rank high in most of the ranked lists. Therefore, we use the MRR version for the rest of our experiment, denoted as CGExpan.
Fine-grained Performance Analysis. Table 3 reports more fine-grained comparison results between two methods. Specifically, we calculate the ratio of seed entity set queries (out of total 55 queries) on which one method achieves better or the same performance as the other method. We can see that CGExpan clearly outperforms SetExpan and its two variants on the majority of queries. In Table 4, we further compare CGExpan with two "oracle" models that have the access to ground truth class names. Results show that CGExpan can
Figure 3: Performance for different  $k$  values on Wiki (left) and APR (right).
achieve comparative results as those oracle models, which indicates the high quality of generated class names and effectiveness of CGExpan.
Parameter Study. In CGExpan, we calculate the similarity between an entity and a class name based on its  $k$  occurrences that are most similar to the class name (cf. Eq. (1)). Figure 3 studies how this parameter  $k$  would affect the overall performance. We find that the model performance first increases when  $k$  increases from 1 to 5 and then becomes stable (in terms of MAP@10 and MAP@20) when  $k$  further increases to 10. Overall, we find  $k = 5$  is enough for calculating entity-class similarity and CGExpan is insensitive to  $k$  as long as its value is larger than 5.
4.3 Case Studies
Class Name Selection. Table 5 shows some results of our class name ranking module for several queries from different semantic classes in the Wiki dataset. We see that CGExpan is able to select the correct class name and thus injects the correct semantics in later entity selection module. Moreover, as shown in the last column, CGExpan can identify several negative class names that provide a tight boundary for the target semantic class, including sports and competition for sport league class, as well as city and country for Chinese province class. These negative class names help CGExpan avoid adding those related but erroneous entities into the set.
From Table 5 we can see that it happens when the predicted positive class name is not exactly the ground true class name in the original dataset. However, since we use both the generated class names and currently expanded entities as guidance and select new entities according to the context features in the provided corpus, those imperfect class names can still guide the set expansion process and perform well empirically.
Also, in principle, synonyms of the positive class name can be wrongly selected as negative class names, which also happens but very rarely in our experiments. However, since these synonyms con
8157
Seed Entity Set	Ground True Class Name	Positive Class Name	Negative Class Names
{"Intel", "Microsoft", "Dell"}	company	company	product, system, bank, ...
{"United States", "China", "Canada"}	country	country	state, territory, island, ...
{"ESPNews", "ESPN Classic", "ABC"}	tv channel	television network	program, sport, show, ...
{"NFL", "NFL", "American league"}	sports league	professional league	sport, competition, ...
{"democratic", "labor", "tories"}	party	political party	organization, candidate, ...
{"Hebei", "Shandong", "Shanxi"}	Chinese province	chinese province	city, country, state, ...
{"tuberculosis", "Parkinson's disease", "esophageal cancer"}	disease	chronic disease	symptom, condition, ...
{"Illinois", "Arizona", "California"}	US state	state	county, country, ...
Table 5: Class names generated for seed entity sets. The  $2^{\text{nd}}$  column is the ground true class name in the original dataset. The  $3^{\text{rd}}$  and  $4^{\text{th}}$  columns are positive and negative class names predicted by CGExpan, respectively.
sistently rank lower than the positive one for the initial seeds based on the given corpus, they are indeed not good class names for this specific corpus. Thus, misclassifying them will not have much influence on the performance of our model.
Entity Selection. Table 6 shows expanded entity sets for two sample queries. After correctly predicting true positive class names and selecting relevant negative class names, CGExpan utilizes them to filter out those related but erroneous entities, including two TV shows in television network class and three entities in political party class. As a result, CGExpan can outperform CGExpan-NoFilter.
Entity Set Expansion. Traditional entity set expansion systems such as Google Sets (Tong and Dean, 2008) and SEAL (Wang and Cohen, 2007, 2008) typically submit a query consisting of seed entities to a general-domain search engine and extract new entities from retrieved web pages. These methods require an external search engine for online seed-oriented data collection, which can be costly. Therefore, more recent studies propose to expand the seed set by offline processing a corpus. These corpus-based set expansion methods can be categorized into two general approaches: (1) one-time entity ranking which calculates entity distributional similarities and ranks all entities once without back and forth refinement (Mamou et al., 2018; Yu et al., 2019b), and (2) iterative bootstrapping which aims to bootstrap the seed entity set by iteratively selecting context features and ranking new entities (Rong et al., 2016; Shen et al., 2017; Yan et al., 2019; Zhu et al., 2019; Huang et al., 2020). Our method in general belongs to the later category. Finally, there are some studies that incorporate extra knowledge to expand the entity set, including negative examples (Curran et al., 2007; McIntosh and Curran, 2008; Jindal and Roth, 2011), semi-structured web table (Wang et al., 2015), and ex
ternal knowledge base (Yu et al., 2019a). Particularly, Wang et al. (2015) also propose to use a class name to help expand the target set. However, their method requires a user-provided class name and utilizes web tables as additional knowledge, while our method can automatically generate both positive and negative class names and utilize them to guide the set expansion process.
Language Model Probing. Traditional language models aim at assigning a probability for an input word sequence. Recent studies have shown that by training on next word or missing word prediction task, language models are able to generate contextualized word representations that benefit many downstream applications. ELMo (Peters et al., 2018) proposes to learn a BiLSTM model that captures both forward and backward contexts. BERT (Devlin et al., 2019) leverages the Transformer architecture and learns to predict randomly masked tokens in the input word sequence and to classify the neighboring relation between pair of input sentences. Based on BERT's philosophy, RoBERTa (Liu et al., 2019) conducts more careful hyper-parameter tuning to improve the performance on downstream tasks. XLNet (Yang et al., 2019) further combines the ideas from ELMo and BERT and develops an autoregressive model that learns contextualized representation by maximizing the expected likelihood over permutations of the input sequence.
Aside from generating contextualized representations, pre-trained language models can also serve as knowledge bases when being queried appropriately. Petroni et al. (2019) introduce the language model analysis probe and manually define probing queries for each relation type. By submitting those probing queries to a pre-trained LM, they show that we can retrieve relational knowledge and achieve competitive performance on various NLP tasks. More recently, Bouraoui et al. (2020) further analyze BERT's ability to store relational knowledge by using BERT to automatically select high-
8158
Seed Entity Set	CGExpan	CGExpan-NoCN	CGExpan-NoFilter
{“ESPN”,“Discovery Channel”,“Comedy Central”}	1	“Pb”	1	“NBC”	1	“Pb”
2	“ABC”	2	“CBS”	2	“Mtv”
3	“CBS”	3	“Disney Channel”	3	“ABC”
...	...	...
35	“Telemundo”	35	“ESPN Radio”*	35	“MyNetworkTV”
36	“Fox Sports Net”	36	“BBC America”	36	“ESPN2”
37	“Dateline NBC”	37	“G4”	37	“the Today Show”*
38	“Channel 4”	38	“Sirius Satellite Radio”*	38	“Access Hollywood”*
39	“The History Channel”	39	“TNT”	39	“Cartoon Network”
...	...	...
{“democratic party”,“republican party”,“labor party”}	1	“republican”	1	“national party”	1	“republican”
2	“likud”	2	“labour party”	2	“likud”
3	“liberal democrats”	3	“gop establishment”*	3	“liberal democrats”
...	...	...
40	“komeito”	40	“republican jewish coalition”*	40	“young voters”*
41	“centrist liberal democrats”	41	“british parliament”*	41	“bjp”
42	“aipac”*	42	“tea party patriots”*	42	“religious”*
43	“aam aadmi party”	43	“centrist liberal democrats”	43	“congress”*
44	“ennahda”	44	“federal government”*	44	“lib dem”
...	...
Table 6: Expanded entity sets for two sample queries, with erroneous entities colored red and marked with a “*”.
quality templates from text corpus for new relation prediction. Comparing with previous work, in this paper, we show that probing pre-trained language model works for entity set expansion task, and we propose a new entity set expansion framework that combines corpus-independent LM probing with corpus-specific context information for better expansion performance.
In this paper, we propose a new entity set expansion framework that can use a pre-trained LM to generate candidate class names for the seed set, rank them according to the provided text corpus, and guide the entity selection process with the selected class names. Extensive experiments on the Wiki and APR datasets demonstrate the effectiveness of our framework on both class name prediction and entity set expansion. In the future, we plan to expand the method scope from expanding concrete entity sets to more abstract concept sets. For example, we may expand the set {"machine translation", "information extraction", "syntactic parsing"} to acquire more NLP task concepts. Another interesting direction is to generate a class name hierarchy via language model probing.
Research was sponsored in part by US DARPA KAIROS Program No. FA8750-19-2-1004 and SocialSim Program No. W911NF-17-C-0099, National Science Foundation IIS 16-18481, IIS
17-04532, and IIS-17-41317, and DTRA HDTRA11810026. Any opinions, findings, and conclusions or recommendations expressed herein are those of the authors and should not be interpreted as necessarily representing the views, either expressed or implied, of DARPA or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for government purposes notwithstanding any copyright annotation hereon. We thank anonymous reviewers for valuable feedback.
Zied Bouraoui, Jose Camacho-Collados, and Steven Schockaert. 2020. Inducing relational knowledge from bert. In AAAI.
James R. Curran, Tara Murphy, and Bernhard Scholz. 2007. Minimising semantic drift with mutual exclusion bootstrapping. In PAACL.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In *NAACL-HLT*.
Marti A Hearst. 1992. Automatic acquisition of hy- ponyms from large text corpora. In COLING.
Wen Hua, Zhongyuan Wang, Haixun Wang, Kai Zheng, and Xiaofang Zhou. 2017. Understand short texts by harvesting and analyzing semantic knowledge. In TKDE.
Jiaxin Huang, Yiqing Xie, Yu Meng, Jiaming Shen, Yunyi Zhang, and Jiawei Han. 2020. Guiding corpus-based set expansion by auxiliary sets generation and co-expansion. In WebConf.
Prateek Jindal and Dan Roth. 2011. Learning from negative examples in set-expansion. In ICDM.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692.
Jonathan Mamou, Oren Pereg, Moshe Wasserblat, Alon Eirew, Yael Green, Shira Guskin, Peter Izsak, and Daniel Korat. 2018. Term set expansion based nlp architect by intel ai lab. In EMNLP.
Tara McIntosh and James R. Curran. 2008. Weighted mutual exclusion bootstrapping for domain independent lexicon and template acquisition. In ALTA.
Patrick Pantel, Eric Crestan, Arkady Borkovsky, Ana Maria Popescu, and Vishnu Vyas. 2009. Web-scale distributional similarity and entity set expansion. In EMNLP.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke S. Zettlemoyer. 2018. Deep contextualized word representations. In NAACL-HLT.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. 2019. Language models as knowledge bases? In EMNLP.
Xin Rong, Zhe Chen, Qiaozhu Mei, and Eytan Adar. 2016. Egoset: Exploiting word ego-networks and user-generated ontology for multifaceted set expansion. In WSDM.
Jingbo Shang, Jialu Liu, Meng Jiang, Xiang Ren, Clare R. Voss, and Jiawei Han. 2018. Automated phrase mining from massive text corpora. TKDE.
Jiaming Shen, Zeqiu Wu, Dongming Lei, Jingbo Shang, Xiang Ren, and Jiawei Han. 2017. Setexpan: Corpus-based set expansion via context feature selection and rank ensemble. In ECML/PKDD.
Jiaming Shen, Zeqiu Wu, Dongming Lei, Chao Zhang, Xiang Ren, Michelle T. Vanni, Brian M. Sadler, and Jiawei Han. 2018a. Hiexpan: Task-guided taxonomy construction by hierarchical tree expansion. In KDD.
Jiaming Shen, Jinfeng Xiao, Xinwei He, Jingbo Shang, Saurabh Sinha, and Jiawei Han. 2018b. Entity set
search of scientific literature: An unsupervised ranking approach. In SIGIR.
Simon Tong and Jeff Dean. 2008. System and methods for automatically creating lists. US Patent 7,350,187.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS.
Chi Wang, Kaushik Chakrabarti, Yeye He, Kris Ganjam, Zhimin Chen, and Philip A. Bernstein. 2015. Concept expansion using web tables. In WWW.
Richard C. Wang and William W. Cohen. 2007. Language-independent set expansion of named entities using the web. In ICDM.
Richard C. Wang and William W. Cohen. 2008. Iterative set expansion of named entities using the web. In ICDM.
Richard C. Wang, Nico Schlaefer, William W. Cohen, and Eric Nyberg. 2008. Automatic set expansion for list question answering. In EMNLP.
Chenyan Xiong, Russell Power, and James P. Callan. 2017. Explicit semantic ranking for academic search via knowledge graph embedding. In WWW.
Lingyong Yan, Xianpei Han, Le Sun, and Ben He. 2019. Learning to bootstrap for entity set expansion. In EMNLP.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In NeurlPS.
Jifan Yu, Chenyu Wang, Gan Luo, Lei Hou, Juan-Zi Li, Zhiyuan Liu, and Jie Tang. 2019a. Course concept expansion in moocs with external knowledge and interactive game. In ACL.
Puxuan Yu, Zhiqi Huang, Razieh Rahimi, and James D Allan. 2019b. Corpus-based set expansion with lexical features and distributed representations. In SI-GIR.
Wanzheng Zhu, Hongyu Gong, Jiaming Shen, Chao Zhang, Jingbo Shang, Suma Bhat, and Jiawei Han. 2019. Fuse: Multi-faceted set expansion by coherent clustering of skip-grams. ArXiv, abs/1910.04345.