SlowGuess/NAACL_2022_OCR huggingface dataset:
Yutong Shao $^{1*}$ , Nikita Bhutani $^{2\dagger}$ , Sajjadur Rahman $^{2\dagger}$ , Estevam Hruschka $^{2}$
<sup>1</sup>University of California, San Diego <sup>2</sup>Megagon Labs
yshao@eng.ucsd.edu, {nikita,sajjadur,estevam}@megagon.ai
Entity set expansion (ESE) aims at obtaining a more complete set of entities given a textual corpus and a seed set of entities of a concept. Although it is a critical task in many NLP applications, existing benchmarks are limited to well-formed text (e.g., Wikipedia) and well-defined concepts (e.g., countries and diseases). Furthermore, only a small number of predictions are evaluated compared to the actual size of an entity set. A rigorous assessment of ESE methods warrants more comprehensive benchmarks and evaluation. In this paper, we consider user-generated text to understand the generalizability of ESE methods. We develop new benchmarks and propose more rigorous evaluation metrics for assessing performance of ESE methods. Additionally, we identify phenomena such as non-named entities, multifaceted entities, vague concepts that are more prevalent in user-generated text than well-formed text, and use them to profile ESE methods. We observe that the strong performance of state-of-the-art ESE methods does not generalize well to user-generated text. We conduct comprehensive empirical analysis and draw insights from the findings.
Entities are integral to applications that require understanding natural language text such as semantic search (Inan et al., 2021; Lashkari et al., 2019), question answering (Chandrasekaran et al., 2020; Cheng and Erk, 2020) and knowledge base construction (Goel et al., 2021; Al-Moslmi et al., 2020). To this end, entity set expansion (ESE) is a crucial task that uses a textual corpus to enhance a set of seed entities (e.g., 'mini bar', 'tv unit') with
Figure 1: Example concepts and entities from Wiki vs. Tripadvisor. We highlight example multifaceted entities in blue, non-named entities in green and vague entities in magenta.
new entities (e.g., 'coffee', 'clock') that belong to the same semantic concept (e.g., room features).
Since training data in new domains is scarce, many existing ESE methods expand a small seed set by learning to rank new entity candidates with limited supervision. Broadly speaking, there are two types of such low-resource ESE methods: (a) corpus-based methods (Shen et al., 2018; Huang et al., 2020a; Yu et al., 2019a) that bootstrap the seed set using contextual features and patterns, and (b) language model-based methods (Zhang et al., 2020a) that probe a pre-trained language model with prompts to rank the entity candidates.
Despite the recent progress, reported success of ESE methods is largely limited to benchmarks focusing on named entities (e.g., countries, diseases) and well-written text such as Wikipedia. Furthermore, the evaluation is limited to top 10-50 predictions regardless of the actual size of the entity set. As a result, it is unclear whether the reported effectiveness of ESE methods is conditional to datasets, domains, and/or evaluation methods.
In this paper, we conduct a comprehensive study to investigate the generalizability of ESE methods in low-resource settings. Specifically, we focus on
*Work done during an internship at Megagon Labs.
Equal author contribution.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1343 - 1353
domains with user-generated text such as reviews. User-generated text data is abundant and is largely unlabeled. Enabling NLP applications including semantic search and question answering (Li et al., 2019; Bhutani et al., 2020; Dai and Song, 2019) over user-generated text requires entities mined from these largely unlabeled data. Furthermore, user-generated text has distinctive characteristics than well-written text, making it appropriate for this study. Due to lack of benchmarks on user-generated text, we create new benchmarks from three domains - hotels, restaurants and jobs.
We found that these benchmarks exhibit characteristics (illustrated in Figure 1) distinct from existing benchmarks: (a) multifaceted entities (entities that belong to multiple concepts — e.g., ‘venice beach’ can belong to concepts location and nearby attractions); (b) non-named entities (entities that are typically noun phrases but not proper names — e.g., ‘coffee’); and (c) vague entities (human annotators have subjective disagreement on their concept labels — e.g., ‘casino’ for nearby attraction). We explain why these characteristics emerge in user-generated text in Section 3.
We found that user-generated text can have up to  $10 \times$  more multifaceted entities and  $2 \times$  more non-named entities compared to well-curated benchmarks. Furthermore, concepts that do not have well-defined semantics result in vague entities. We use these characteristics to profile ESE methods, showing that the performance difference between well-curated and user-generated text can partially be attributed to these characteristics.
Contributions. To summarize, our key contributions include: a) identifying and verifying several important new characteristics in user-generated text that are not explored in evaluation of existing ESE methods, b) constructing three new user-generated text benchmarks (we publicly release two $^1$ ), c) proposing new metrics for evaluating ESE methods, d) deriving insights through a cross-domain (user-generated text vs. well-curated) comparison study on different ESE methods.
Key findings. Our main findings are listed below:
- Widely used evaluation metrics such as (mean average precision (MAP) at  $k \leq 20$ ) is an inadequate indicator of the performance of
ESE methods on both well-curated and user-generated text. Evaluating top- $k_{g}^{2}$  predictions is potentially more robust, especially for benchmarking.
- Performance of state-of-the-art (SOTA) ESE methods drops dramatically on user-generated text compared to well-curated text.
- Deviating from prior observations, simple corpus-based and language model-based methods that underperform SOTA methods on well-curated text can outperform SOTA methods on user-generated text.
- Simple rank-based ensemble methods can provide further improvements on user-generated text. The degree of overlap of correct predictions from candidate methods is indicative of the effectiveness of their ensemble.
We now introduce the task of entity set expansion (ESE), existing paradigms and evaluation methods.
Given a textual corpus and a user-defined seed set of entities (e.g., 'coffee', 'table') of concepts (e.g., room features), the task of ESE is to output a ranked list of entities (e.g., 'clock', 'tv') that belong to the same concept. Following previous work, we focus on the low-resource setting where the seed set is small (3-10 entities per concept).
2.2 Entity Set Expansion Paradigms
To expand the seed set, ESE methods rank candidate entities extracted from a textual corpus (Shang et al., 2018). We limit our scope to low-resource setting and exclude methods (Mao et al., 2020; Takeoka et al., 2021) that require large training examples sub-concepts hierarchy or external knowledge from ontologies and knowledge bases. We organize ESE methods into the following categories.
Corpus-based Methods. These methods (Huang et al., 2020b; Shen et al., 2017, 2018; Yu et al., 2019a) obtain contextual features and distributed representations of entity candidates from the corpus and use them to estimate similarity of candidates to entities in the seed set. This is either done in a single step (Mamou et al., 2018; Yu et al.,
${}^{2}{k}_{g}$  denotes the actual entity set size of a concept.
$^{1}$ https://github.com/megagonlabs/esebench
Characteristics	Examples	Comments
Multifaceted entities	R1: Be sure to book in advance an early morning trip to Alcatraz, go to fisherman's wharf ...
R2: I would not stay here again. I'd rather pay more and stay by fisherman's wharf ...	R1 refers to concept nearbyattraction while R1 refers to location. Entities that fall into multiple semantic concepts might influence other entity candidates for a target concept.
Vague entities	R1: . . . see the majestic Frenchy-looking civic center surrounded since 8pm by a crowd ...
R2: The Monticello Inn is five to ten minute cab ride from civic center ...	R1 indicates the entity of interest is a nearby attraction but R2 is vague. Popular concepts such as nearby attractions in user generated text can be inherently subjective.
Non-named entities	R1: There was tea and coffee available round the clock in the lounge.
R2: The room rate included a large and varied continental breakfast with excellent coffee. {concept: room features}	Concepts of interest in user-generated text domain often exhibit non-named entities. In fact, user-generated Tripadvisor dataset has 1.7× more non-named entities (such as “coffee”) compared to well-curated Wiki dataset.
Table 1: Exploring different characteristics of well-curated and user-generated text domains.
2019a) or iteratively (Shen et al., 2018; Huang et al., 2020b; Yan et al., 2021).
Language Model-based Methods. Studies have shown that pre-trained language models (LMs) can be used as knowledge bases when queried with prompts (Petroni et al., 2019; Liu et al., 2021). Following this, ESE methods (Zhang et al., 2020a; Takeoka et al., 2021) probe an LM to rank entity candidates. These methods rely on knowledge stored in LMs instead of using them to obtain contextualized representations of entities in the corpus.
Ensemble Methods. CaSE (Yu et al., 2019b) combines context feature selection with pre-trained word embeddings to compute similarities between entities. A similar mechanism, mean reciprocal ranking (MRR) ensemble, has been shown to be effective in combining rankings from different features, views or subsets of seeds (Shen et al., 2017; Zhang et al., 2020b; Huang et al., 2020b).
2.3 Benchmark and Evaluation Metrics
Widely-used benchmarks for ESE, such as Wiki and APR (Shen et al., 2017), are based on well-formed text corpora like Wikipedia and focus only on well-defined concepts such as countries, US states, and diseases. Furthermore, the ranked expansion results are evaluated against the ground truth using Mean Average Precision (MAP) at different top- $k$  positions where  $k$  is much smaller than the size of entity set. For example, there are 195 countries but only 10-50 predictions are evaluated. In following sections, we argue that existing benchmarks and evaluation metrics may not be adequate enough to estimate the real-world performance of the ESE methods and introduce new benchmarks and evaluation metrics to address their limitations.
3 Case Study
Existing work suggest that user-generated text differs from well-curated text in writing style (Bražinskas et al., 2020; Huang et al., 2020d) and cleanliness (Van der Wees et al., 2015; Dey et al., 2016). In this section, we discuss one of the use cases of ESE for a downstream NLP application and highlight new characteristics in user-generated text that are particularly relevant to the ESE task.
Let us consider a scenario where Tajin, a data scientist at an online travel company (similar to TripAdvisor), has to develop a semantic search feature that helps users explore relevant reviews corresponding to their queries. For example, when a user searches for ‘amenities’ at a hotel, the feature should display reviews with mentions of different amenities highlighted. Since the reviews are unlabeled, Tajin first consults an expert to compile a list of frequently queried concepts and corresponding example entities that may appear in the reviews (similar to Figure 1). To discover more entities for each concept, she formulates it as an ESE task, where the goal is to achieve a high coverage of entities in the reviews.
Given a review corpus and seed, Tajin employs a state-of-the-art ESE method that has been evaluated on well-curated text. She finds that it does not perform well in achieving a high coverage of entities in the hotel domain and wonders why. To explain her findings, we explore the characteristics of the TripAdvisor (Miao et al., 2020) and Wiki datasets next and discuss the potential factors that may impact the performance of the SOTA method.
Multifaceted entities. Unlike in Wiki benchmark where concepts are well-defined, concepts in Triadvisor are domain-specific and can have overlapping semantics (see Figure 1). As a result, an entity can belong to multiple concepts. For example, in Table 1, the entity 'fisherman's wharf' can be both location of and nearby attraction to a hotel. We refer to such entities as multifaceted entities (Rong et al., 2016). The overlapping semantics of entities can pose challenges to ESE methods that expand multiple concepts simultaneously.
Vague entities. Concept definitions in Wiki benchmark are strict (e.g., countries, states) and ground truth about concept-entity pairs can be obtained by referring to external resources or commonsense. However, some concepts in Tripadvisor are open-ended and subjective, leading to vagueness in interpretation. For example, in Table 1, the terms "nearby" and "attraction" in the concept nearby attractions are subjective. An entity 'civic center' may be neither an attraction nor a nearby one depending on the context in the review. As a result, human annotators independently labeling the entity may disagree on the ground truth label. We refer to entities with subjective disagreements between annotators as vague entities. Intuitively, ESE methods may find it difficult to learn to disambiguate the context of vague entities.
Non-named entities. Non-named entities (e.g., 'coffee' and 'tv unit') are typically noun phrases that are not proper names (Paris and Suchanek, 2021). Recent studies (Mbouopda and Melatagia Yonta, 2020; Bamman et al., 2019) have identified that non-named entities are prevalent even in well-curated domains and yet are ignored in existing benchmarks. Non-named entities are even more prevalent in user-generated text. As shown in Table 1, Tripadvisor benchmark contains almost  $2 \times$  non-named entities than Wiki. Since non-named entities are not canonicalized and can have broader semantics, they can make the ESE task more challenging.
Evaluation Metric. Existing evaluation metrics only consider top 10-50 entities for each target concept (Shen et al., 2017; Zhang et al., 2020a). There are multiple limitations of these metrics. First, there may not be sufficient representation of multi
faceted, vague, and non-named entities in a small set (<50 entities). Second, the actual number of correct entities per concept (referred to as concept size) may be much larger or smaller than 50. For example, both Tripadvisor and Wiki have larger and varying concept sizes with the median size being 121 and 205, respectively (check Table 2 for more detailed statistics). As a result, focusing only on precision of a small, fixed set of predictions may not reflect the recall of correct entities with respect to concept size.
While multifaceted, vague, and non-named entities can be present in well-curated data, the corresponding benchmarks and downstream applications target real-world named entities and ignore non-named entities (Paris and Suchanek, 2021). In contrast, in most user-generated text domains, the concepts of interest for downstream applications (semantic search feature as discussed above) are not limited to named-entities only and may exhibit multifaceted, vague, and non-named entities (e.g., facts about a hotel such as amenities and attractions). With the increasing use of user-generated text in NLP applications (Xu et al., 2021), it is therefore important to investigate the impact of the aforementioned characteristics on the performance of the ESE methods.
We now outline our experiment set-up designed to explore the suitability of existing benchmarks, metrics, and methods.
We first describe the ESE methods we evaluate. Following prior work (Shen et al., 2018; Zhang et al., 2020a), we use AutoPhrase (Shang et al., 2018) to generate candidate entity lists from the corpus of a given domain. We then use the following representative publicly available ESE methods from different paradigms to expand the seed set<sup>3</sup>.
SetExpan. SetExpan (Shen et al., 2017) is a SOTA corpus-based method that iteratively ranks entity candidates by filtering out noisy skip-gram features. It incorporates other context features such as POS tags and syntactic head tokens in ranking.
3All methods were released under Apache 2.0 license.
Embedding baseline (Emb-Base). In order to make use for more robust context embeddings, we develop a simple baseline that uses a pre-trained language model (LM) to derive context embeddings of entity candidates. To derive an entity embedding, we average context embedding of the sentences that mention the entity using BERT (Devlin et al., 2018). We compute concept embeddings by averaging embeddings of its seed entities, and rank entity candidates based on the cosine similarity of concept and entity embeddings.
CGExpan. CGExpan (Zhang et al., 2020b) is a SOTA LM-based method that iteratively uses Hearst patterns (Hearst, 1992) as prompts to obtain scores for ranking candidates. In addition, it considers how a candidate in turn ranks the target concept name to improve the quality of rankings.
LM Probing Baseline (LM-Base). We develop a simpler baseline that also uses Hearst patterns to prompt LMs and obtain scores for entity candidates. However, it does not include any other mechanisms such as concept name guidance and iterative expansion like CGExpan.
Ensemble Methods. We use mean reciprocal rank (MRR) as the representative for ensemble methods since it does not require any additional training data. Given the rankings from multiple methods, we compute MRR score of each entity:  $MRR(e) = \frac{1}{n} \sum_{i=1}^{n} \frac{1}{r_i(e)}$ , where  $n$  is the number of methods combined, and  $r_i(e)$  is the ranking of entity  $e$  under method  $i$ . We then re-rank all entities based on their MRR score. In this work, we study combinations of two ESE methods leading to 6 ensembles. We study 4 settings that offer interesting combinations across different paradigms:
- MRR-Baseline: Emb-Base + LM-Base.
- MRR-SOTA: SetExpan + CGExpan.
- MRR-Corpus: SetExpan + Emb-Base.
- MRR-LM-Probe: CGExpan + LM-Base.
We use widely adopted well-curated benchmarks: Wiki and APR (Zhang et al., 2020b; Shen et al., 2017). In addition, we create 3 new benchmarks based on user-generated text from Yelp (Huang et al., 2020c), Tripadvisor (Miao et al., 2020) and a proprietary Jobs dataset. All the datasets are in English. We first select concepts for the seed by referring to the features on the corresponding web
sites, to ensure their relevance for immediate downstream tasks. For example, we select concepts from various facets such as room type, amenities, and distance from attractions that help visitors search hotels on the Tripadvisor website<sup>4</sup>. Table 2 shows selected concepts in the benchmarks.
Data Collection and Annotation. In order to collect ground-truth to construct benchmarks for new domains, we collect top 200 predictions for each concept from each of the ESE methods described in Section 4.1. The first three authors of the paper labeled the predictions, 1 if a concept-entity pair is correct and 0 otherwise. We consider the majority vote as the final label for a concept-entity pair. For entities with rank  $>200$ , we label the corresponding concept-entity pairs to be all negatives based on our preliminary observations that most of them are incorrect. We release the new benchmarks except for the Jobs dataset.
In order to profile the benchmarks, we compute multifacetedness  $(m)$  as the fraction of entities in a benchmark that have been assigned to more than one concept. We compute non-named rate  $(r)$  as the fraction of non-named entities in the benchmark. We use Spacy<sup>5</sup> to identify named entities in the benchmarks. To avoid bias in estimating vagueness, we hire two additional in-house annotators who are unfamiliar with the concept definitions and entities. They label the ground truth concept-entity pairs — 1 if correct and 0 otherwise<sup>6</sup>. We compute vagueness  $(\kappa)$  in a benchmark using Fleiss' Kappa (Falotico and Quatto, 2015) which measures agreement among the annotators.
Since the benchmarks we constructed are intended to be comprehensive, we propose to estimate mean average precision (MAP) at gold-  $k$ $(k_{g})$  which equals the concept size, i.e., number of entities in the concept. In comparison to smaller and fixed  $k$ , evaluation at  $k_{g}$  has several advantages: (a) it can adapt to different concept sizes and (b) it gives an estimate of recall<sup>7</sup> which is crucial to estimate effectiveness in real-world settings with commonly large concept sizes. Intuitively, using
4https://tripadvisor.com
5https://spacy.io/usage/linguistic-features
$^{6}$ Labeling instructions are included in benchmark release.
${}^{7}P@{k}_{g} = R@{k}_{g} = F1@{k}_{g}$  because the number of predicted positives and true positives both equal to  ${k}_{g}$  .
Corpus	Seed	Benchmark
Dataset	#Docs	Example concepts (# concepts)	Avg. seed size	# entity cands	Concept size	κ	m	r
Wiki	973k	countries, parties, us states, china provinces, companies, tv channels, diseases, sports leagues (8)	9.875	203322	{51, 205, 446}	-†	0.0141	0.4143
APR	1043k	countries, us states, parties (3)	8.333	78870	{89, 202, 301}	-†	0.0000	0.3649
Yelp	757k	restaurant name, restaurant type, seating arrangement, food category, parking, ambience (14)	4.429	23527	{15, 99, 353}	0.0252	0.0369	0.7995
Tripadvisor	18k	location, property type, style, amenities, room features, room type, nearby attractions, staff (8)	6.625	6842	{31, 121, 244}	-0.1252	0.0908	0.7043
Jobs	318k	company, dress code, job position, pay schedule, benefits, payment option (14)	5.143	8028	{36, 100, 316}	-0.1902	0.0837	0.7957
Table 2: Statistics of datasets: no. of documents, example concepts in the seed, avg. no. of entities in the seed, no. of entity candidates and concept size  $\{min, median, max\}$  across different concepts. Statistics of benchmarks: multifacetedness  $(m)$ , non-named rate  $(r)$  and vagueness  $(\kappa)$ . †: for well-curated datasets, there is no subjective disagreement since the concept-entity pairs are factually verifiable.
$k_{g}$  would include more instances of multifaceted, vague and non-named entities that would otherwise be ignored in small  $k$ . Notice that in certain real-world scenarios, e.g., developing ESE methods for a new domain, estimating  $k_{g}$  may be difficult, thus previous metrics with smaller and fixed  $k$  can be useful. However, for other scenarios, especially for evaluation on benchmarks where the goal is to stress test methods, evaluation at  $k_{g}$  is more appropriate.
5 Findings
We next share our findings from analyzing the ESE methods. Note that all the results are obtained from single run of each experiment.
5.1 Appropriateness of Existing Benchmark and Metrics
Q1. Do we need new benchmarks based on user-generated text?
Table 2 compares the characteristics of the various benchmarks using measures described in Section 4.3. As can be seen, user-generated text benchmarks exhibit a higher degree of multifacetedness  $(m)$  and non-named rate  $(r)$  compared to well-curated Wiki and APR benchmarks. Moreover, poor agreement between annotators  $(\kappa < 0)$  indicates the presence of vagueness or subjectivity in user-generated text which does not exist in well-curated benchmarks. While all benchmarks exhibit diversity in concept sizes, the diversity is higher in user-generated text than well-curated benchmarks.
Takeaway 1 User-generated text benchmarks exhibit more multifaceted entities, non-named entities, and vagueness than well-curated benchmarks.
Q2. Do existing evaluation metrics accurately estimate the performance of ESE methods?