SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 3 shows the  $\%$  drop in MAP of different ESE methods when  $k$  is increased from 20 to  $k_{g}$ . The performance drop is consistent across both well-curated and user-generated text benchmarks with the largest being  $62\%$  for CGExpan on the Jobs benchmark. This indicates that existing metrics overestimate the real-world performance of all ESE methods. However, simpler baselines, Embase and LM-Base, tend to show lower performance drop than more sophisticated counterparts on user-generated text. This indicates that existing well-curated benchmarks do not reliably capture progress in this field.
Method/Datasets	Jobs	Yelp	TripAdvisor	Wiki	APR
SetExpan	-36.66%	-41.74%	-43.16%	-40.76%	-14.25%
CGExpan	-61.99%	-54.37%	-42.82%	-39.67%	-38.21%
Emb-Base	-54.10%	-43.96%	-36.41%	-35.45%	-56.33%
LM-Base	-36.17%	-35.15%	-34.47%	-56.40%	-43.96%
Table 3: Drop in performance of different ESE methods from MAP@20 to MAP@  $k_{g}$ . Largest drops in each dataset are highlighted in bold.
We further observed that, across all the benchmarks, the performance drops are higher for concepts with large entity sets. We show two such cases in Figure 2 — one with user-generated text (Figure 2a) and another with well-curated text (Figure 2b) — which illustrate precision curves at different values of  $k$  for concepts with large  $k_{g}$  in various benchmarks. As shown, two ESE methods that may show similar performance at  $k = 20$  (widely adopted metric) have much larger performance margins at  $k_{g}$ . Thus, evaluation results on only top 20 predictions may be an incomplete depiction of method robustness, especially for concepts with large entity sets.
(a) TripAdvisor: amenities
(b) Wiki: tv channel
Figure 2: Precision@  $k$  of LM-Base and CGExpan for example concepts. Dashed line indicates  $k = 20$  and dotted line indicates  $k_{g}$ . Performance margins at  $k = 20$  and  $k_{g}$  vary significantly.
Takeaway 2 Existing evaluation metrics tend to overestimate the real-world performance of ESE methods and may be unreliable for evaluating concepts with large entity sets.
5.2 Performance on new benchmarks
Q3. How effective SOTA methods are for entity set expansion on user-generated text benchmarks?
(a) user-generated text.
Figure 3: Overall MAP@  $k_{g}$  performance of each method on (a) user-generated text and (b) well-curated benchmarks. SOTA methods (CGExpan, SetExpan) are outperformed by simple baselines and ensemble methods on user-generated text.
(b) well-curated datasets.
Given new benchmarks and evaluation metrics, we now compare the performances of various ESE methods. Figure 3 shows that SOTA method CGExpan outperforms other methods on existing well-curated benchmarks which aligns with the reported success of the method. Surprisingly, simpler baseline methods (Emb-Base and LM-Base) that were not optimal on well-curated benchmarks, significantly outperform their SOTA counterparts (SetExpan and CGExpan, respectively) on user-generated text benchmarks, with LM-Base obtaining the best performance. We also observe that ensemble-based methods tend to perform better than or at least similar to the ESE methods they combine.
Takeaway 3 Performance of SOTA methods do not generalize to user-generated text benchmarks. Ensemble-based methods may improve over the corresponding standalone methods.
We now examine why SOTA approaches may underperform on user-generated text. Given the success of LM-based contextual representations, it is expected that Emb-Base may outperform lexical feature-based SetExpan. Furthermore, as SetExpan eliminates noisy features of a candidate entity before ranking candidates, it may disregard some context features of multifaceted and vague entities that are mentioned in diverse contexts in user-generated text, leading to sub-optimal ranking of entities. Similarly, CGExpan, which scores each candidate entity by selecting one positive concept and multiple negative concepts, may penalize entities belonging to multiple concepts (multifaceted entities) or mentioned in different contexts (vague entities). Therefore, many of the carefully designed approaches useful on well-curated domains may not generalize to user-generated text.
Takeaway 4 SOTA methods implement techniques that avoid selecting ambiguous context of an entity. Such a design choice potentially penalizes multifaceted and vague entities when ranking entity candidates for concepts.
Q4. How do characteristics of user-generated text affect performance of ESE methods?
We now discuss how different characteristics of user-generated text impact the behavior of ESE methods. To understand this, we compare the recall of entities that exhibit one of the target characteristics (multifaceted/non-named/vague) with recall of entities that do not exhibit any of the characteristics. This enables us to analyze the influence of a target characteristic independent of other characteristics. To compute recall, we consider an entity as retrieved if it is ranked in the top- $k_{g}$  predictions.
Figure 4 compares the recall of entities across different characteristics. For ease of visualization, we combine entities across the 3 benchmarks. As shown, almost all methods show lower recall of entities that exhibit challenging characteristics than entities without these characteristics, and SOTA methods suffer larger drops than simple methods. This supports our hypothesis that characteristics of user-generated text negatively affect performances,
(a) vague entities
(b) multifacetedness
(c) non-named entities
Figure 4: Recall@  $k_{g}$  of an ESE method typically drops with the presence of (a) vague, (b) multifaceted, and (c) non-named entities in user-generated text (show by red bars) compared to the case when none of these entities are present (shown by green bars).
Figure 5: Example concepts on which CGExpan and LM-Base have similar (company) or different (seating arrangement) performances (indicated in parenthesis). Incorrect entities are shown in red.
especially for SOTA methods which tend to penalize entities with diverse contexts. Future work may investigate how to overcome these challenges.
To provide a qualitative comparison between the behaviors of SOTA methods (e.g., CGExpan) and our proposed baselines (e.g., LM-Base), we show their predictions on two representative concepts in Figure 5. CGExpan and LM-Base have comparable performance on well-formed concepts (e.g., company) in Jobs. However, LM-Base outperforms CGExpan for concepts (e.g., seating arrangement) with entities having characteristics of user-generated text. CGExpan retrieves entities that co-occur frequently with seating arrangement.
Takeaway 5 Due to the presence of challenging characteristics in user-generated text, performance of all ESE methods are negatively impacted with SOTA methods exhibiting larger drops.
5.3 Improvement Opportunities
Q5. How do we design ensemble methods for benchmarks with user-generated text?
We analyze ensemble methods further since they tend to outperform other ESE methods (Figure 3). It is trivial that ensemble methods perform well when both combined methods are strong. We are more interested in other factors that may impact
performance. Specifically, we investigate what influences the effectiveness of a MRR method that combines two ESE methods. An MRR combination is more effective when it outperforms both candidate methods by a larger margin. We define effectiveness of combining methods as:
$$
E f f (m _ {1}, m _ {2}) = \frac {S (m _ {1} + m _ {2})}{\max  (S (m _ {1}) , S (m _ {2}))} - 1 \tag {1}
$$
where  $S(m)$  means the performance (MAP@ $k_{g}$  in our study) of method  $m$ , and  $m_{1} + m_{2}$  means the MRR combination of method  $m_{1}, m_{2}$ .
As discussed in Section 5.2, multifaceted and vague entities may appear in diverse contexts which SOTA approaches fail to capture, leading to lower recall. Intuitively, it is advantageous to combine methods that capture differing contexts and in the process predict collections of correct entities with minimal overlap. In other words, in order for a MRR method to achieve higher recall, the ESE methods must be compatible. We measure compatibility of two ESE methods as:
$$
\operatorname {C o m p} \left(m _ {1}, m _ {2}\right) = \frac {\| P \left(m _ {1}\right) \cup P \left(m _ {2}\right) \|}{\max  \left(\| P \left(m _ {1}\right) \| , \| P \left(m _ {2}\right) \|\right)} - 1 \tag {2}
$$
where  $P(m)$  is the set of correct entity predictions of method  $m$ , i.e. positive benchmark entities ranked among top-  $k_{g}$  by  $m$ .  $\| \cdot \|$  denotes the size of a set. When one of the correct prediction set of  $m_{1}, m_{2}$  is a subset of the other, their compatibility is 0. When the two methods find two disjoint sets of correct entities, their compatibility is 1.
We illustrate the correlation between compatibility of method pairs and effectiveness of their MRR combination in Figure 6 using a scatter plot. Each of the points represent the compatibility and effectiveness of the four ensemble methods (MRR-
Figure 6: Scatter plot showing the positive correlation between compatibility of methods and effectiveness of their MRR combination on three user-generated text benchmarks. The least compatible and effective combination results (in dashed circle) are from MRR-LM-Probe.
SOTA, MRR-Base, MRR-Corpus, and MRR-LM-probe) on all three user-generated text datasets. We observe that LM-based methods are least compatible due to their similarity in design. The resulting ensemble, MRR-LM-probe, has poor effectiveness (highlighted by the dashed ellipse in Figure 6). Other method pairs have less homogeneity in their design and the resulting ensembles often show higher effectiveness. The corresponding compatibility and effectiveness values have a strong positive correlation (Pearson correlation,  $R = 0.69$ ). Therefore, compatibility can be a useful metric for deciding whether combining two methods method may improve performance or not.
Takeaway 6 Two effective ESE methods on user-generated text with high compatibility (diversity in correct predictions) may achieve higher performance when combined using rank-based ensemble.
We now discuss the implications of the proposed benchmark, metrics, and experiment observations.
Capturing the silent majority. Recent work (Paris and Suchanek, 2021) shows that the majority of the entities in Wikipedia articles — which feeds knowledge-bases such as DBpedia (Auer et al., 2007) and YAGO (Suchanek et al., 2007) — are non-named and recommends adding the silent majority to these KBs for completeness. To this end, our proposed benchmark highlights the importance of capturing multifaceted, vague, and non-named entities present in user-generated text. For example, domain-specific KBs such as
the Amazon Product Knowledge Graph (Karamanolakis et al., 2020) rely on user-generated text to collect entities for concepts of interest. These KBs power many downstream tasks such as semantic search, question answering, and conversational AI. Therefore, these KBs would remain incomplete without capturing the different types of entities identified in our benchmark.
Practical usage. The goal of our evaluation metrics (evaluation at  $k_{g}$ ) is to characterize the performance of ESE methods in the presence of entity types that are typically present in user-generated text. Note that we do not recommend replacing the existing metric  $\mathrm{MAP} @ K = 20$ . Our proposed  $\mathrm{MAP} @ K_{g}$  metric is complementary and is designed to stress test ESE methods in scenarios where coverage is an important criteria (e.g., KB population.) Top-20 predictions do not have enough representation of non-named, multifaceted, and vague entities. Therefore, when evaluating ESE methods designed for user-generated text on our benchmark, the proposed evaluation at  $k_{g}$  metric may help practitioners measure the suitability of a method.
Towards domain-specific ESE. Our study highlights that compared to simple ESE baselines, SOTA methods exhibit poor performance on user-generated text. On the other hand, for well-curated text, SOTA methods outperform the baselines. However, the purpose of this study is not to show that there are better approaches than SOTA methods. Instead, we draw attention to the fact that, there is potential for future research on developing methods for user-generated text domain.
We conduct a comprehensive study to analyze the performance of ESE in user-generated text. We observe that user-generated text has characteristics that are not captured in existing benchmarks, and propose new benchmarks and evaluation metrics. Our findings indicate that state-of-the-art methods are not very effective in user-generated text and are often outperformed by simpler baselines.
We thank Tom Mitchell and Yoshihiko Suhara for their valuable feedback on the work.
Tareq Al-Moslmi, Marc Gallofre Ocaña, Andreas L Opdahl, and Csaba Veres. 2020. Named entity extraction for knowledge graphs: A literature overview. IEEE Access, 8:32862-32881.
Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007. Dbpedia: A nucleus for a web of open data. In The semantic web, pages 722-735. Springer.
David Bamman, Sejal Popat, and Sheng Shen. 2019. An annotated dataset of literary entities. In Proc. NAACL-HLT' 2019, pages 2138-2144, Minneapolis, Minnesota. Association for Computational Linguistics.
Nikita Bhutani, Aaron Traylor, Chen Chen, Xiaolan Wang, Behzad Golshan, and Wang-Chiew Tan. 2020. Sampo: Unsupervised knowledge base construction for opinions and implications. In Proc. AKBC' 2020.
Arthur Bražinskas, Mirella Lapata, and Ivan Titov. 2020. Few-shot learning for opinion summarization. In Proc. EMNLP'2020.
Ramji Chandrasekaran, Harsh Nilesh Pathak, and Tae Yano. 2020. Deep neural query understanding system at expedia group. In Proc. IEEE Big Data' 2020, pages 1476-1484. IEEE.
Pengxiang Cheng and Katrin Erk. 2020. Attending to entities for better text understanding. In Proc. AAAI' 2020, 05, pages 7554-7561.
Hongliang Dai and Yangqiu Song. 2019. Neural aspect and opinion term extraction with mined rules as weak supervision. arXiv preprint arXiv:1907.03750.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Kuntal Dey, Ritvik Shrivastava, and Saroj Kaushik. 2016. A paraphrase and semantic similarity detection system for user generated short-text content on microblogs. In Proc. COLING' 2016, pages 2880-2890, Osaka, Japan. The COLING 2016 Organizing Committee.
Rosa Falotico and Piero Quatto. 2015. Fleiss' kappa statistic without paradoxes. *Quality & Quantity*, 49(2):463-470.
Karan Goel, Laurel Orr, Nazneen Fatema Rajani, Jesse Vig, and Christopher Ré. 2021. Goodwill hunting: Analyzing and repurposing off-the-shelf named entity linking systems. In Proceedings of the 2021 Conference of the North American Chapter of the
Association for Computational Linguistics: Human Language Technologies: Industry Papers, pages 205-213, Online. Association for Computational Linguistics.
Marti A. Hearst. 1992. Automatic acquisition of hyponyms from large text corpora. In Proc. COLING' 1992.
Jiaxin Huang, Yiqing Xie, Yu Meng, Jiaming Shen, Yunyi Zhang, and Jiawei Han. 2020a. Guiding corpus-based set expansion by auxiliary sets generation and co-expansion. In Proc. WWW'2020, pages 2188-2198.
Jiaxin Huang, Yiqing Xie, Yu Meng, Jiaming Shen, Yunyi Zhang, and Jiawei Han. 2020b. Guiding corpus-based set expansion by auxiliary sets generation and co-expansion. Proc. WWW' 2020.
Jiaxing Huang, Yiqing Xie, Yu Meng, Yunyi Zhang, and Jiawei Han. 2020c. Corel: Seed-guided topical taxonomy construction by concept learning and relation transferring. Proc. SIGKDD '2020.
Rongtao Huang, Bowei Zou, Yu Hong, Wei Zhang, Aiti Aw, and Guodong Zhou. 2020d. Nut-rc: Noisy user-generated text-oriented reading comprehension. In Proc. COLING' 2020, pages 2687-2698.
Emrah Inan, Paul Thompson, Tim Yates, and Sophia Ananiadou. 2021. Hsearch: Semantic search system for workplace accident reports. In Proc. ECIR 2021, volume 12657 of Lecture Notes in Computer Science, pages 514-519. Springer.
Giannis Karamanolakis, Jun Ma, and Xin Luna Dong. 2020. Txtract: Taxonomy-aware knowledge extraction for thousands of product categories. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8489-8502.
Fatemeh Lashkari, Ebrahim Bagheri, and Ali A Ghorbani. 2019. Neural embedding-based indices for semantic search. Information Processing & Management, 56(3):733-755.
Yuliang Li, Aaron Xixuan Feng, Jinfeng Li, Saran Mumick, Alon Halevy, Vivian Li, and Wang-Chiew Tan. 2019. Subjective databases. arXiv preprint arXiv:1902.09661.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pretrain, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586.
Jonathan Mamou, Oren Pereg, Moshe Wasserblat, Alon Eirew, Yael Green, Shira Guskin, Peter Izsak, and Daniel Korat. 2018. Term set expansion based nlp architect by intel ai lab. arXiv preprint arXiv:1808.08953.
Yuning Mao, Tong Zhao, Andrey Kan, Chenwei Zhang, Xin Luna Dong, Christos Faloutsos, and Jiawei Han. 2020. Octet: Online catalog taxonomy enrichment with self-supervision. In Proc. SIGKDD'2020, pages 2247-2257.
Michael Franklin Mbouopda and Paulin Melatagia Yonta. 2020. Named entity recognition in low-resource languages using cross-lingual distributional word representation. Revue Africaine de la Recherche en Informatique et Mathematiques Appliquées, 33.
Zhengjie Miao, Yuliang Li, Xiaolan Wang, and Wang-Chiew Tan. 2020. Snippext: Semi-supervised opinion mining with augmented data. In Proc. WWW' 2020, pages 617-628.
Pierre-Henri Paris and Fabian M Suchanek. 2021. Non-named entities-the silent majority. In Proc. ESWC' 2021 Poster and Demo Track.
Fabio Petroni, Tim Roktaschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. 2019. Language models as knowledge bases? arXiv preprint arXiv:1909.01066.
Xin Rong, Zhe Chen, Qiaozhu Mei, and Eytan Adar. 2016. Egoset: Exploiting word ego-networks and user-generated ontology for multifaceted set expansion. Proc. WSDM' 2016.
Jingbo Shang, Jialu Liu, Meng Jiang, Xiang Ren, Clare R. Voss, and Jiawei Han. 2018. Automated phrase mining from massive text corpora. Proc. TKDE' 2018, 30:1825-1837.
Jiaming Shen, Zeqiu Wu, Dongming Lei, Jingbo Shang, Xiang Ren, and Jiawei Han. 2017. Setexpan: Corpus-based set expansion via context feature selection and rank ensemble. In Proc. ECML PKDD' 2017, pages 288-304. Springer.
Jiaming Shen, Zeqiu Wu, Dongming Lei, Chao Zhang, Xiang Ren, Michelle T Vanni, Brian M Sadler, and Jiawei Han. 2018. Hiexpan: Task-guided taxonomy construction by hierarchical tree expansion. In Proc. SIGKDD'2018, pages 2180-2189.
Fabian M Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2007. Yago: a core of semantic knowledge. In Proceedings of the 16th international conference on World Wide Web, pages 697-706.
Kunihiro Takeoka, Kosuke Akimoto, and Masafumi Oyamada. 2021. Low-resource taxonomy enrichment with pretrained language models. In Proc. EMNLP'2021, pages 2747-2758.
Marlies Van der Wees, Arianna Bisazza, and Christof Monz. 2015. Five shades of noise: Analyzing machine translation errors in user-generated text. In Proc. Workshop on Noisy User-generated Text' 2015, pages 28-37.
Wei Xu, Alan Ritter, Tim Baldwin, and Afshin Rahimi, editors. 2021. Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021). Association for Computational Linguistics, Online.
Lingyong Yan, Xianpei Han, and Le Sun. 2021. Progressive adversarial learning for bootstrapping: A case study on entity set expansion. In Proc. EMNLP'2021, pages 9673-9682, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Puxuan Yu, Zhiqi Huang, Razieh Rahimi, and James Allan. 2019a. Corpus-based set expansion with lexical features and distributed representations. In Proc. SIGIR' 2019, pages 1153-1156.
Puxuan Yu, Zhiqi Huang, Razieh Rahimi, and James Allan. 2019b. Corpus-based set expansion with lexical features and distributed representations. Proc. SIGIR' 2019.
Yunyi Zhang, Jiaming Shen, Jingbo Shang, and Jiawei Han. 2020a. Empower entity set expansion via language model probing. arXiv preprint arXiv:2004.13897.
Yunyi Zhang, Jiaming Shen, Jingbo Shang, and Jiawei Han. 2020b. Empower entity set expansion via language model probing. ArXiv, abs/2004.13897.