SlowGuess/ACL_2020_OCR huggingface dataset:
Yunyi Zhang $^{1}$ , Jiaming Shen $^{1}$ , Jingbo Shang $^{2}$ , Jiawei Han $^{1}$
<sup>1</sup> University of Illinois at Urbana-Champaign, IL, USA
$^{2}$  University of California San Diego, CA, USA
$^{1}\{\text{yzhan238, js2, hanj}\} \text{@illinois.edu}$ $^{2}\text{jshang@ucsd.edu}$
Entity set expansion, aiming at expanding a small seed entity set with new entities belonging to the same semantic class, is a critical task that benefits many downstream NLP and IR applications, such as question answering, query understanding, and taxonomy construction. Existing set expansion methods bootstrap the seed entity set by adaptively selecting context features and extracting new entities. A key challenge for entity set expansion is to avoid selecting ambiguous context features which will shift the class semantics and lead to accumulative errors in later iterations. In this study, we propose a novel iterative set expansion framework that leverages automatically generated class names to address the semantic drift issue. In each iteration, we select one positive and several negative class names by probing a pre-trained language model, and further score each candidate entity based on selected class names. Experiments on two datasets show that our framework generates high-quality class names and outperforms previous state-of-the-art methods significantly.
Entity set expansion aims to expand a small set of seed entities (e.g., {"United States", "China", "Canada"}) with new entities (e.g., "United Kingdom", "Australia") belonging to the same semantic class (i.e., Country). The entities so discovered may benefit a variety of NLP and IR applications, such as question answering (Wang et al., 2008), query understanding (Hua et al., 2017), taxonomy construction (Shen et al., 2018a), and semantic search (Xiong et al., 2017; Shen et al., 2018b).
Most existing entity set expansion methods bootstrap the initial seed set by iteratively selecting context features (e.g., co-occurrence words (Pantel et al., 2009), unary patterns (Rong et al., 2016), and coordinational patterns (Mamou et al., 2018)),
Figure 1: Examples of class-probing and entity-probing queries generated based on Hearst patterns.
while extracting and ranking new entities. A key challenge to set expansion is to avoid selecting ambiguous patterns that may introduce erroneous entities from other non-target semantic classes. Take the above class Country as an example, we may find some ambiguous patterns like “* located at” (which will match more general Location entities) and “match against *” (which may be associated with entities in the Sports Club class). Furthermore, as bootstrapping is an iterative process, those erroneous entities added at early iterations may shift the class semantics, leading to inferior expansion quality at later iterations. Addressing such “semantic drift” issue without requiring additional user inputs (e.g., mutually exclusive classes (Curran et al., 2007) and negative example entities (Jindal and Roth, 2011)) remains an open research problem.
In this study, we propose to empower entity set expansion with class names automatically generated from pre-trained language models (Peters et al., 2018; Devlin et al., 2019; Yang et al., 2019). Intuitively, knowing the class name is "country", instead of "state" or "city", can help us identify unambiguous patterns and eliminate erroneous entities like "Europe" and "New York". Moreover, we can acquire such knowledge (i.e., positive and negative class names) by probing a pre-trained language
8151
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8151-8160 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
model automatically without relying on human annotated data.
Motivated by the above intuition, we propose a new iterative framework for entity set expansion that consists of three modules: (1) The first, class name generation module, constructs and submits class-probing queries (e.g., "[MASK] such as USA, China, and Canada.") in Fig. 1) to a language model for retrieving a set of candidate class names. (2) The second, class name ranking module, builds an entity-probing query for each candidate class name and retrieves a set of entities. The similarity between this retrieved set and the current entity set serves as a proxy for the class name quality, based on which we rank all candidate class names. An unsupervised ensemble technique (Shen et al., 2017) is further used to improve the quality of final ranked list from which we select one best class name and several negative class names. (3) The third, class-guided entity selection module, scores each entity conditioned on the above selected class names and adds top-ranked entities into the currently expanded set. As better class names may emerge in later iterations, we score and rank all entities (including those already in the expanded set) at each iteration, which helps alleviate the semantic drift issue.
Contributions. In summary, this study makes the following contributions: (1) We propose a new set expansion framework that leverages class names to guide the expansion process and enables filtration of the entire set in each iteration to resolve the semantic drift issue; (2) we design an automatic class name generation algorithm that outputs high-quality class names by dynamically probing pretrained language models; and (3) experiments on two public datasets from different domains demonstrate the superior performance of our approach compared with state-of-the-art methods.
In this section, we provide background on language models and define the entity set expansion problem.
2.1 Language Model
A standard language model (LM) inputs a word sequence  $\mathbf{w} = [w_{1},w_{2},\dots ,w_{n}]$  and assigns a probability  $\mathbf{P}(\mathbf{w})$  to the whole sequence. Recent studies (Peters et al., 2018; Devlin et al., 2019; Yang et al., 2019) found that language models, simply trained for next word or missing word prediction,
can generate high quality contextualized word representations which benefit many downstream applications. Specifically, these language models will output an embedding vector for each word appearance in a specific context that is usually the entire sentence where the target word occurs, rather than just words appearing before the target word. Therefore, we can also view a LM as a model that inputs a word sequence  $\mathbf{w}$  and outputs a probability  $\mathbf{P}(w_i) = \mathbf{P}(w_i | w_1, \dots, w_{i-1}, w_{i+1}, \dots, w_n)$  to any position  $1 \leq i \leq n$ . Currently, Devlin et al. (2019) propose BERT and train the language model with two objectives: (1) a cloze-filling objective which randomly substitutes some words with a special [MASK] token in the input sentence and forces LM to recover masked words, and (2) a binary classification objective that guides LM to predict whether one sentence directly follows another (sentence). BERT leverages Transformer (Vaswani et al., 2017) architecture and is learned on English Wikipedia as well as BookCorpus. More LM architectures are described in Section 5.
We first define some key concepts and then present our problem formulation.
Entity. An entity is a word or a phrase that refers to a real-world instance. For example, "U.S." refers to the country: United States.
Class Name. A class name is a text representation of a semantic class. For instance, country could be a class name for the semantic class that includes entities like “United States” and “China”.
Probing Query. A probing query is a word sequence containing one [MASK] token. In this work, we utilize Hearst patterns (Hearst, 1992) to construct two types of probing queries: (1) A class-probing query aims to predict the class name of some given entities (e.g., “[MASK] such as United States and China”), and (2) an entity-probing query aims to retrieve entities that fit into the mask token (e.g., “countries such as [MASK] and Japan”).
Problem Formulation. Given a text corpus  $\mathcal{D}$  and a seed set of user-provided entities, we aim to output a ranked list of entities that belong to the same semantic class.
Example 1. Given a seed set of three countries {"United States", "China", "Canada"}, we aim to return a ranked list of entities belonging to the same country class such as "United Kingdom", "Japan", and "Mexico".
8152
3 Class-Guided Entity Set Expansion
We introduce our class-guided entity set expansion framework in this section. First, we present our class name generation and ranking modules in Sections 3.1 and 3.2, respectively. Then, we discuss how to leverage class names to guide the iterative expansion process in Section 3.3.
3.1 Class Name Generation
The class name generation module inputs a small collection of entities and generates a set of candidate class names for these entities. We build this module by automatically constructing class-probing queries and iteratively querying a pretrained LM to obtain multi-gram class names.
First, we notice that the class name generation goal is similar to the hypernymy detection task which aims to find a general hypernym (e.g., "mammal") for a given specific hyponym (e.g., "panda"). Therefore, we leverage the six Hearst patterns (Hearst, 1992)<sup>1</sup>, widely used for hypernymy detection, to construct the class-probing query. More specifically, we randomly select three entities in the current set as well as one Hearst pattern (out of six choices) to construct one query. For example, we may choose entities {"China", "India", "Japan"} and pattern " $NP_{y}$  such as  $NP_{a}, NP_{b}$ , and  $NP_{c}$ " to construct the query "[MASK] such as China, India, and Japan". By repeating such a random selection process, we can construct a set of queries and feed them into pre-trained language models to obtain predicted masked tokens which are viewed as possible class names.
The above procedure has one limitation—it can only generate unigram class names. To obtain multi-gram class names, we design a modified beam search algorithm to iteratively query a pretrained LM. Specifically, after we query a LM for the first time and retrieve top  $K$  most likely words (for the masked token), we construct  $K$  new queries by adding each retrieved word after the masked token. Taking the former query "[MASK] such as China, India, and Japan" as an example, we may first obtain words like "countries", "nations", and then construct a new query "[MASK] countries such as China, India, and Japan". Probing the LM again with this new query, we can get words like "Asian" or "large", and obtain more fine-grained class names like "Asian countries" or "large coun
tries". We repeat this process for maximum three times and keep all generated class names that are noun phrases<sup>2</sup>. As a result, for each Hearst pattern and randomly selected three entities from the current set, we will obtain a set of candidate class names. Finally, we use the union of all these sets as our candidate class name pool, denoted as  $\mathcal{C}$ . Note that in this module, we focus on the recall of candidate class name pool  $\mathcal{C}$ , without considering its precision, since the next module will further rank and select these class names based on the provided text corpus.
3.2 Class Name Ranking
In this module, we rank the above generated candidate class names to select one best class name that represents the whole entity set and some negative class names used in the next module to filter out wrong entities. A simple strategy is to rank these class names based on the number of times it has been generated in the previous module. However, such a strategy is sub-optimal because short unigram class names always appear more frequently than longer multi-gram class names. Therefore, we propose a new method below to measure how well each candidate class name represents the entity set.
First, we introduce a corpus-based similarity measure between an entity  $e$  and a class name  $c$ . Given the class name  $c$ , we first construct 6 entity-probing queries by masking the hyponym term in six Hearst patterns $^3$ , and query a pre-trained LM to obtain the set of six [MASK] token embeddings, denoted as  $X_{c}$ . Moreover, we use  $X_{e}$  to denote the set of all contextualized representations of the entity  $e$  in the given corpus. Then, we define the similarity between  $e$  and  $c$ , as:
$$
M ^ {k} (e, c) = \frac {1}{k} \max  _ {X \subseteq X _ {e}, | X | = k} \sum_ {\mathbf {x} \in X} \max  _ {\mathbf {x} ^ {\prime} \in X _ {c}} \cos (\mathbf {x}, \mathbf {x} ^ {\prime}), \quad (1)
$$
where  $\cos (\mathbf{x},\mathbf{x}^{\prime})$  is the cosine similarity between two vectors  $\mathbf{x}$  and  $\mathbf{x}^{\prime}$ . The inner max operator finds the maximum similarity between each occurrence of  $e$  and the set of entity-probing queries constructed based on  $c$ . The outer max operator identifies the top- $k$  most similar occurrences of  $e$  with the queries and then we take their average as the final similarity between the entity  $e$  and the class name  $c$ . This measure is analogous to finding
2Therefore, class names likes "and countries" and ", countries" are filtered out.
3For example, a query for class name "countries" is "countries such as [MASK]".
For example, the pattern  $NP_{y}$  such as  $NP_{a}$  indicates that noun phrase  $y$  is a hypernym of noun phrase  $a$ .
8153
Figure 2: Overview of one iteration in CGExpan framework.
$k$  best occurrences of entity  $e$  that matches to any of the probing queries of class  $c$ , and therefore it improves the previous similarity measures that utilize only the context-free representations of entities and class names (e.g., Word2Vec).
After we define the entity-class similarity score, we can choose one entity in the current set and obtain a ranked list of candidate class names based on their similarities with this chosen entity. Then, given an entity set  $E$ , we can obtain  $|E|$  ranked lists,  $L_{1}, L_{2}, \ldots, L_{|E|}$ , one for each entity in  $E$ . Finally, we follow (Shen et al., 2017) and aggregate all these lists to a final ranked list of class names based on the score  $s(c) = \sum_{i=1}^{|E|} \frac{1}{r_{c}^{i}}$ , where  $r_{c}^{i}$  indicates the rank position of class name  $c$  in ranked list  $L_{i}$ . This final ranked list shows the order of how well each class name can represent the current entity set. Therefore, we choose the best one that ranks in the first position as the positive class, denoted as  $c_{p}$ .
Aside from choosing the positive class name  $c_{p}$ , we also select a set of negative class names for the target semantic class to help bound its semantics. To achieve this goal, we assume that entities in the initial user-provided seed set  $E^{0}$  definitely belong to the target class. Then, we choose those class names that rank lower than  $c_{p}$  in all lists corresponding to entities in  $E^{0}$ , namely  $\{L_{i}|e_{i}\in E^{0}\}$ , and treat them as the negative class names. We refer to this negative set of class names as  $C_N$  and use them to guide the set expansion process below.
3.3 Class-Guided Entity Selection
In this module, we leverage the above selected positive and negative class names to help select new entities to add to the set. We first introduce two entity scoring functions and then present a new rank ensemble algorithm for entity selection.
The first function utilizes the positive class name  $c_p$  and calculates each entity  $e_i$ 's score :
$$
\operatorname {s c o r e} _ {i} ^ {l o c} = M ^ {k} \left(e _ {i}, c _ {p}\right), \tag {2}
$$
where  $M^k$  is defined in Eq. (1). We refer to this score as a local score because it only looks at top- $k$  best occurrences in the corpus where the contextualized representation of entity  $e_i$  is most similar to the representation of class name  $c_q$ .
The second scoring function calculates the similarity between each candidate entity and existing entities in the current set, based on their context-free representations. For each entity  $e$ , we use the average of all its contextualized embedding vectors as its context-free representation, denoted as  $\mathbf{v}_e$ . Given the current entity set  $E$ , we first sample several entities from  $E$ , denoted as  $E_s$ , and calculate the score for each candidate entity  $e_i$  as:
$$
\operatorname {s c o r e} _ {i} ^ {g l b} = \frac {1}{| E _ {s} |} \sum_ {e \in E _ {s}} \cos \left(\mathbf {v} _ {e _ {i}}, \mathbf {v} _ {e}\right). \tag {3}
$$
Note here we sample a small set  $E_{s}$  (typically of size 3), rather than using the entire set  $E$ . Since the current entity set  $E$  may contain wrong entities introduced in previous steps, we do not use all the entities in  $E$  and compute the candidate entity score only once. Instead, we randomly select multiple subsets of entities from the current set  $E$ , namely  $E_{s}$ , obtain a ranked list of candidate entities for each sampled subset, and aggregate all ranked lists to select the final entities. Such a sampling strategy can reduce the effect of using wrong entities in  $E$ , as they are unlikely to be sampled multiple times, and thus can alleviate potential errors that are introduced in previous iterations. We refer to this score as a global score because it utilizes context-free representations which better reflect entities' overall positions in the embedding space and measure the entity-entity similarity in a more global sense. Such a global score complements the above local score and we use their geometric mean to finally rank all candidate entities:
$$
\operatorname {s c o r e} _ {i} = \sqrt {\operatorname {s c o r e} _ {i} ^ {l o c} \times \operatorname {s c o r e} _ {i} ^ {g l b}}. \tag {4}
$$
As the expansion process iterates, wrong entities
8154
may be included in the set and cause semantic drifting. We develop a novel rank ensemble algorithm that leverages those selected class names to improve the quality and robustness of entity selection. First, we repeatedly sample  $E_{s}$  (used for calculating score $_i^{glb}$  in Eq. (3))  $T$  times from current entity set  $E$ , and obtain  $T$  entity ranked lists  $\{R^m\}_{m=1}^T$ . Second, we follow the class name ranking procedure in Section 3.2 to obtain  $|E|$  class ranked lists  $\{L^n\}_{n=1}^{|E|}$ , one for each entity  $e_i \in E$ . Note here each  $L^n$  is actually a ranked list over  $\{c_p\} \cup C_N$ , namely the set of selected one positive class name and all negative class names. Intuitively, an entity belonging to our target semantic class should satisfy two criteria: (1) it appears at the top positions in multiple entity ranked lists, and (2) within its corresponding class ranked list, the selected best class name  $c_p$  should be ranked above any one of the negative class name in  $C_N$ . Combining these two criteria, we define a new rank aggregation score as follows:
$$
\begin{array}{l} S (e _ {i}) = \sum_ {t = 1} ^ {T} \left(\mathbb {1} \left(e _ {i} \in E\right) + s ^ {t} \left(e _ {i}\right)\right) \\ \times \mathbb {1} \left(r _ {c _ {p}} ^ {i} <   \min  _ {c ^ {\prime} \in C _ {N}} r _ {c ^ {\prime}} ^ {i}\right), \tag {5} \\ \end{array}
$$
where  $\mathbb{1}(\cdot)$  is an indicator function,  $r_c^i$  is the rank of class name  $c$  in entity  $e_i$ 's ranked list  $L_{c}^{i}$ , and  $s^t (e_i)$  the individual aggregation score of  $e_i$  deduced from the ranked list  $R^t$ , for which we test two aggregation methods: (1) mean reciprocal rank, where
$$
s ^ {t} \left(e _ {i}\right) = \frac {1}{r _ {i} ^ {t}} \tag {6}
$$
and  $r_i^t$  is the rank of entity  $e_i$  in the  $t$ -th ranked list  $R^t$ ; and (2) the combination of scores (CombSUM), where
$$
s ^ {t} \left(e _ {i}\right) = \frac {\operatorname {s c o r e} _ {i} ^ {t} - \min  _ {e _ {j} \in R ^ {t}} \operatorname {s c o r e} _ {j} ^ {t}}{\max  _ {e _ {j} \in R ^ {t}} \operatorname {s c o r e} _ {j} ^ {t} - \min  _ {e _ {j} \in R ^ {t}} \operatorname {s c o r e} _ {j} ^ {t}} \tag {7}
$$
is the ranking score of  $e_i$  in the ranked list  $R^t$  after min-max feature scaling.
To interpret Eq. 5, the first summation term reflects our criterion (1) and its inner indicator function ensuring an entity in the current set  $E$  prone to have a large rank aggregation score if not been filtered out below. The second term reflects our criterion (2) by using an indicator function that filters out all entities which are more similar to a negative class name than the positive class name. Note here we calculate the aggregation score for all entities in
Dataset	# Test Queries	# Entities	# Sentences
Wiki	40	33K	1.50M
APR	15	76K	1.01M
the vocabulary list, including those already in the current set  $E$ , and it is possible that some entity in  $E$  will be filtered out because it has 0 value in the second term. This makes a huge difference comparing with previous iterative set expansion algorithms which all assume that once an entity is included in the set, it will stay in the set forever. Consequently, our method is more robust to the semantic drifting issue than previous studies.
Summary. Starting with a small seed entity set, we iteratively apply the above three modules to obtain an entity ranked list and add top-ranked entities into the set. We repeat the whole process until either (1) the expanded set reaches a pre-defined target size or (2) the size of the set does not increase for three consecutive iterations. Notice that, by setting a large target size, more true entities belonging to the target semantic class will be selected to expand the set, which increases the recall, but wrong entities are also more likely to be included, which decreases the precision. However, as the output of the set expansion framework is a ranked list, the most confident high-quality entities will still be ranked high in the list.
Datasets. We conduct our experiments on two public benchmark datasets widely used in previous studies (Shen et al., 2017; Yan et al., 2019): (1) Wiki, which is a subset of English Wikipedia articles, and (2) APR, which contains all news articles published by Associated Press and Reuters in 2015. Following the previous work, we adopt a phrase mining tool, AutoPhrase (Shang et al., 2018), to construct the entity vocabulary list from the corpus, and select the same 8 semantic classes for the Wiki dataset as well as 3 semantic classes for the APR dataset. Each semantic class has 5 seed sets and each seed set contains 3 entities. Table 1 summarizes the statistics for these datasets.
Compared methods. We compare the following corpus-based entity set expansion methods.
1. Egoset (Rong et al., 2016): This is a multifaceted set expansion system using context features and Word2Vec embeddings. The original
8155
Methods	Wiki	APR
MAP@10	MAP@20	MAP@50	MAP@10	MAP@20	MAP@50
Egoset (Rong et al., 2016)	0.904	0.877	0.745	0.758	0.710	0.570
SetExpan (Shen et al., 2017)	0.944	0.921	0.720	0.789	0.763	0.639
SetExpander (Mamou et al., 2018)	0.499	0.439	0.321	0.287	0.208	0.120
CaSE (Yu et al., 2019b)	0.897	0.806	0.588	0.619	0.494	0.330
MCTS (Yan et al., 2019)	0.980▼	0.930▼	0.790▼	0.960▼	0.900▼	0.810▼
CGExpan-NoCN	0.968	0.945	0.859	0.909	0.902	0.787
CGExpan-NoFilter	0.990	0.975	0.890	0.979	0.962	0.892
CGExpan-Comb	0.991	0.974	0.895	0.983	0.984	0.937
CGExpan-MRR	0.995	0.978	0.902	0.992	0.990	0.955
framework aims to expand the set in multiple facets. Here we treat all expanded entities as in one semantic class due to little ambiguity in the seed set.