SlowGuess/NAACL_2022_OCR huggingface dataset:
from a masked language model. In ACL/IJCNLP, pages 1790-1799.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of wikipedia: Knowledge-powered conversational agents. In ICLR. OpenReview.net.
Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyang Gao, and Dilek Hakkani-Tür. 2019. Multiwoz 2.1: Multi-domain dialogue state corrections and state tracking baselines. CoRR, abs/1907.01669.
Thibault Févry, Livio Baldini Soares, Nicholas FitzGerald, Eunsol Choi, and Tom Kwiatkowski. 2020a. Entities as experts: Sparse memory access with entity supervision. In EMNLP, pages 4937-4951.
Thibault Févry, Livio Baldini Soares, Nicholas FitzGerald, Eunsol Choi, and Tom Kwiatkowski. 2020b. Entities as experts: Sparse memory access with entity supervision. In EMNLP. Association for Computational Linguistics.
Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378.
Octavian-Eugen Ganea and Thomas Hofmann. 2017. Deep joint entity disambiguation with local neural attention. In EMNLP, pages 2619-2629.
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. In EMNLP (Findings), pages 3356-3369.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. REALM: retrieval-augmented language model pre-training. CoRR, abs/2002.08909.
Benjamin Heinzerling and Kentaro Inui. 2021. Language models as knowledge bases: On entity representations, storage capacity, and paraphrased queries. In EACL, pages 1772-1791.
Carlos M Herrera. 1976. A trophic diversity index for presence-absence food data. *Oecologia*, 25(2):187-191.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In ACL, pages 328-339. Association for Computational Linguistics.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In EMNLP, pages 6769-6781.
Sungdong Kim, Minsuk Chang, and Sang-Woo Lee. 2021. Neuralwoz: Learning to collect task-oriented dialogue via model-based simulation. In ACL, pages 3704-3717.
Adarsh Kumar, Peter Ku, Anuj Kumar Goyal, Angeliki Metallinou, and Dilek Hakkani-Tür. 2020. MA-DST: multi-attention-based scalable dialog state tracking. In AAAI, pages 8107-8114.
Hwaran Lee, Jinsik Lee, and Tae-Yoon Kim. 2019. SUMBT: slot-utterance matching for universal and scalable belief tracking. In ACL, pages 5478-5483.
Oliver Lemon, Kallirroi Georgila, James Henderson, and Matthew N. Stuttle. 2006. An ISU dialogue system exhibiting reinforcement learning of dialogue policies: Generic slot-filling in the TALK in-car system. In EACL.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020a. BART: denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In ACL, pages 7871-7880.
Patrick S. H. Lewis, Ethan Perez, Aleksandra Pik-tus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Roektaschel, Sebastian Riedel, and Douwe Kiela. 2020b. Retrieval-augmented generation for knowledge-intensive NLP tasks. In NeurIPS.
Shuyang Li, Jin Cao, Mukund Sridhar, Henghui Zhu, Shang-Wen Li, Wael Hamza, and Julian J. McAuley. 2021. Zero-shot generalization in dialog state tracking through generative question answering. In EACL, pages 1063-1074.
Zhaojiang Lin, Bing Liu, Seungwhan Moon, Paul A. Crook, Zhenpeng Zhou, Zhiguang Wang, Zhou Yu, Andrea Madotto, Eunjoon Cho, and Rajen Subba. 2021. Leveraging slot descriptions for zero-shot cross-domain dialogue state tracking. CoRR, abs/2105.04222.
Xiao Ling and Daniel S. Weld. 2012. Fine-grained entity recognition. In AAAI. AAAI Press.
Dayiheng Liu, Yeyun Gong, Jie Fu, Yu Yan, Jiusheng Chen, Daxin Jiang, Jiancheng Lv, and Nan Duan. 2020. Rikinet: Reading wikipedia pages for natural question answering. In ACL, pages 6762-6771.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Robert L. Logan IV, Nelson F. Liu, Matthew E. Peters, Matt Gardner, and Sameer Singh. 2019. Barack's wife hillary: Using knowledge graphs for fact-aware language modeling. In ACL, pages 5962-5971.
Yinquan Lu, Haonan Lu, Guirong Fu, and Qun Liu. 2021. KELM: knowledge enhanced pre-trained language representations with message passing on hierarchical relational graphs. CoRR, abs/2109.04223.
Joan Lucariello, Amy Kyratzis, and Katherine Nelson. 1992. Taxonomic knowledge: What kind and when? Child development, 63(4):978-998.
Bodhisattwa Prasad Majumder, Shuyang Li, Jianmo Ni, and Julian J. McAuley. 2020. Interview: Large-scale modeling of media dialog with discourse patterns and knowledge grounding. In EMNLP, pages 8129-8141.
Pierre-Emmanuel Mazare, Samuel Humeau, Martin Raison, and Antoine Bordes. 2018. Training millions of personalized dialogue agents. In EMNLP, pages 2775-2779.
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The natural language decathlon: Multitask learning as question answering. CoRR, abs/1806.08730.
Yasumasa Onoe and Greg Durrett. 2020. Interpretable entity representations through large-scale typing. In Findings of EMNLP, pages 612-624.
German Ignacio Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. 2019. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71.
Matthew E. Peters, Mark Neumann, Robert L. Logan IV, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019. Knowledge enhanced contextual word representations. In EMNLP, pages 43-54.
Fabio Petroni, Tim Rocttäschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. 2019. Language models as knowledge bases? In EMNLP, pages 2463-2473.
Bharadwaj Pudipeddi, Maral Mesmakhosroshahi, Jinwen Xi, and Sujeeth Bharadwaj. 2020. Training large neural networks with constant memory using a new execution algorithm. CoRR, abs/2002.05645.
Yujia Qin, Yankai Lin, Ryuichi Takanobu, Zhiyuan Liu, Peng Li, Heng Ji, Minlie Huang, Maosong Sun, and Jie Zhou. 2021. ERICA: improving entity and relation understanding for pre-trained language models via contrastive learning. In ACL, pages 3350-3363. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 21:140:1-140:67.
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020. Schemaguided dialogue state tracking task at DSTC8. CoRR, abs/2002.01359.
Ahmad Sakor, Kuldeep Singh, Anery Patel, and Maria-Esther Vidal. 2020. Falcon 2.0: An entity and relation linking tool over wikidata. In CIKM, pages 3141-3148. ACM.
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. In EMNLP (Findings), pages 3784-3803.
Sameer Singh, Amarnag Subramanya, Fernando Pereira, and Andrew McCallum. 2012. Wikilinks: A large-scale cross-document coreference corpus labeled via links to Wikipedia. Technical Report UM-CS-2012-015, University of Massachusetts, Amherst.
Tianxiang Sun, Yunfan Shao, Xipeng Qiu, Qipeng Guo, Yaru Hu, Xuanjing Huang, and Zheng Zhang. 2020a. Colake: Contextualized language and knowledge embedding. In COLING. International Committee on Computational Linguistics.
Yu Sun, Shuohuan Wang, Yu-Kun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. 2020b. ERNIE 2.0: A continual pre-training framework for language understanding. In AAAI. AAAI Press.
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In NIPS, pages 3104-3112.
Raghuveer Thirukovalluru, Mukund Sridhar, Dung Thai, Shruti Chanumolu, Nicholas Monath, Sankaranarayanan Ananthakrishnan, and Andrew McCallum. 2021. Knowledge informed semantic parsing for conversational question answering. In RepL4NLP, pages 231-240, Online.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS, pages 5998-6008.
Denny Vrandecic. 2012. Wikidata: a new platform for collaborative data collection. In WWW, pages 1063-1064. ACM.
Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. 2021. KEPLER: A unified model for knowledge embedding and pre-trained language representation. TACL, 9:176-194.
Zhuoran Wang and Oliver Lemon. 2013. A simple and generic belief tracking mechanism for the dialog state tracking challenge: On the believability of observed information. In SIGDIAL, pages 423-432.
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zac Kenton, Sasha Brown, Will Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendricks,
William S. Isaac, Sean Legassick, Geoffrey Irving, and Jason Gabriel. 2021. Ethical and social risks of harm from language models. CoRR, abs/2112.04359.
Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019. Transferable multi-domain state generator for task-oriented dialogue systems. In ACL, pages 808-819.
Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. 2020a. Scalable zero-shot entity linking with dense entity retrieval. In EMNLP, pages 6397-6407.
Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. 2020b. Scalable zero-shot entity linking with dense entity retrieval. In EMNLP, pages 6397-6407.
Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. LUKE: deep contextualized entity representations with entity-aware self-attention. In EMNLP. Association for Computational Linguistics.
Ikuya Yamada, Koki Washio, Hiroyuki Shindo, and Yuji Matsumoto. 2019. Global entity disambiguation with pretrained contextualized embeddings of words and entities. CoRR, abs/1909.00426.
Pengcheng Yang, Xu Sun, Wei Li, Shuming Ma, Wei Wu, and Houfeng Wang. 2018. SGM: sequence generation model for multi-label classification. In COLING, pages 3915-3926.
Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019. Docred: A large-scale document-level relation extraction dataset. In ACL, pages 764-777.
Yang You, Jing Li, Sashank J. Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. 2020. Large batch optimization for deep learning: Training BERT in 76 minutes. In ICLR. OpenReview.net.
Taolin Zhang, Chengyu Wang, Nan Hu, Minghui Qiu, Chengguang Tang, Xiaofeng He, and Jun Huang. 2021. DKPLM: decomposable knowledge-enhanced pre-trained language model for natural language understanding. CoRR, abs/2112.01047.
We use the June 2021 Wikidata database file from https://www.wikidata.org/wiki/Wikidata:Database_download for raw KG data. We use English Wikipedia article HTML crawled from the same time period. While Wikidata contains multilingual definitions and labels for each node, in this paper we use only English entity and type names.
Wikipedia data was collected under the original terms of release which allow free usage of such materials for non-commercial purposes. We will release WikiWiki under the same license.
When creating questions for pre-training tasks, if a question has multiple answers (e.g. multiple chemists in Table 2), the answers are a comma- and and-delimited sequence, in order of appearance in the context. For the entity typing question, we use the order that types appear in the Wikidata page.
We train all of our models on a node with eight Nvidia V100 GPUs (comprising 256 GB total VRAM) and 768 GB of RAM. We optimize using Deepspeed Stage 1 (Pudipeddi et al., 2020) using FP16 and the Lamb optimizer (You et al., 2020). Experimental results, where applicable, are reported as median of 3 experiments.
Hyperparameters For pre-training, we use a learning rate of 1e-4 with a linear warm-up for the first  $10\%$  of training iterations, using an effective batch size of 960. Our models were trained on a single pass of our pre-training dataset of 50M questions, totaling 52K steps. We fine-tune models using the same learning rate schedule, using an effective batch size of 2560 and early stopping for a maximum of 10 epochs based on validation loss. We aim to establish the general ability of our pre-training scheme to instill type awareness, and thus fix hyperparameters for generative language models trained with our method without hyperparameter tuning.
As mentioned in Section 4, the RoBERTa-based classifier for entity typing on WikiWiki required significantly more hyperparameter tuning; we performed a hyperparameter sweep on batch size (512 to 2048), learning rate (1e-3 to 1e-5), optimizer
$^{5}$ https://en.wikipedia.org/wiki/Wikipedia:Copyrights
# Params	R	H	A	T	X
GPT2-DST + SGD	355M	26.2	24.4	31.3	29.1	59.6
355M	27.7	24.9	42.4	41.1	60.3
Ours (Base)	139M	40.4	36.5	39.8	36.1	70.9
Ours (Large)	406M	46.7	38.8	49.8	37.7	72.1
(Adam vs. Lamb), and whether to freeze the encoder. We achieved best performance (as in Table 6) with a learning rate of 1e-4, the Adam optimizer, an effective batch size of 960, and with gradual unfreezing (Howard and Ruder, 2018) over 5K steps. We found gradual unfreezing to be critical for model performance, with fully frozen and fully unfrozen RoBERTa models achieving entity typing F1 scores of  $\leq 10.0$ .
C Dialog State Tracking Notes
As discussed in Section 4, our method is orthogonal to and thus can be used simultaneously with techniques for creating synthetic in-domain training data for DST (Campagna et al., 2020; Kim et al., 2021). For slot queries, we use templated questions of the form: What [domain] [slot] is the user interested in?.
We compare our models against SOTA models for zero-shot DST on MultiWOZ 2.1. We affirm the observations of Lin et al. (2021) that while T5-DST achieves strong DST performance on the 2.0 version of the dataset, performance degrades on the 2.1 benchmark.
Li et al. (2021) also present results for GPT2-DST when training is augmented with additional DST data from a wider pool of domains—the Schema-Guided Dialog dataset (Rastogi et al., 2020). In the interest of fairness, we do not compare this setting in Table 4 as our models do not have access to any conversational data in pre-training and—like the other baseline models—cannot access additional DST data in fine-tuning. Despite the lack of exposure to conversational data, in Table 7 we show that our Small and Large models out-perform GPT2-DST + SGD in 3/5 domains (with absolute per-domain gain of  $5.5\%$  and relative gain of  $18.3\%$ ) and 4/5 domains (with absolute
Table 7: Zero-shot domain adaptation JGA  $(\%)$  on MultiWOZ 2.1 test set on the (R)estaurant, (H)otel, (A)ttraction, (T)rain, and Ta(X)i domains. Compared to GPT2-DST (Li et al., 2021) augmented with out-of-domain DST data (+SGD), our Base model out-performs the augmented model in 3/5 domains and our Large model out-performs it in 4/5 domains.
# Params	R	H	A	T	X
BART-base	139M	29.6	31.5	38.7	35.0	70.5
Ours (Base)	139M	41.3	33.6	42.5	36.6	71.9
Ours (Large)	406M	46.4	37.6	52.3	38.0	72.1
Table 8: Zero-shot domain adaptation JGA (%) on MultiWOZ 2.1 validation set on the (R)estaurant, (H)otel, (A)ttraction, (T)rain, and Ta(X)i domains.
gains of  $9.7\%$  and relative gains of  $30.6\%$ , respectively. We additionally present zero-shot DST performance (JGA) on the MultiWOZ 2.1 validation set in Table 8.
We perform our evaluation using the Amazon Mechanical Turk platform. To ensure high quality annotations, we recruit only crowd workers with Master qualification—indicating a history of high quality accepted work—and who are native English speakers. Crowd-workers remained anonymous outside of their qualifications and we did not collect any additional demographic information. Workers were informed that their type accuracy judgements were to be used in an academic research setting, with an option to opt-out and reject the task.
As both gold types and predicted types could be complex and require domain knowledge, evaluators were instructed to search any relevant additional material (textbooks, sites, papers) to ensure they made a high confidence judgment of type accuracy. Based on the average time spent evaluating each article, our pay rate worked out to above Federal minimum wage in the United States.
In Figure 2 we display the example instructions given to a human evaluator for assessing the accuracy of a type for an entity referenced in a context. In Figure 3 we show sample instructions given to a human evaluator to choose which of two types (predicted or gold label in random order) is more suitable / applies more accurately to the referenced entity.
E Ethics
As with all models capable of generating arbitrary text sequences, models trained with our framework and tasks run the risk of outputting toxic or offensive text (Gehman et al., 2020). However, our training aims to instill type knowledge for type
<sup>6</sup>https://www.mturk.com/
<sup>7</sup>https://www.mturk.com/worker/help
354P/LINEAR (formerly P/2010 A2 (LINEAR)) is a small Solar System body that displayed characteristics of both an asteroid and a comet, and thus, was initially given a cometary designation. Because it has the orbit of a main-belt asteroid and showed the tail of a comet, it was listed as a main-belt comet. But within a month of discovery, an analysis of images by the Hubble telescope suggested that its tail was generated by dust and gravel resulting from a recent head-on collision between asteroids rather than from sublimation of cometary ice. This was the first time a small-body collision had been observed; since then, minor planet 596 Scheila has also been seen to undergo a collision, in late 2010. The position of the nucleus was remarkable for being offset from the axis of the tail and outside the dust halo, a situation never before seen in a comet. The tail is created by millimeter-sized particles being pushed back by solar radiation pressure.
Question: Is "Hubble telescope" an example of "space observatory"?
Figure 2: Example of human evaluation question where the judge is asked to assess whether a predicted / ground truth type accurately applies to the entity referenced.
She was made a doctor of music by the University of Cambridge in 1976, and became a Dame Commander of the Order of the British Empire -LRB- DBE -RRB- in 1992.
Question: Which of the following best describes "a Dame Commander of the Order of the British Empire -LRB- DBE -RRB-"?
Figure 3: Example of human evaluation question where the judge is asked to assess to determine the relative suitability and quality of two different types for the entity referenced.
order of chivalry
O occupation
Both equally descriptive
$\bigcirc$  Neither are accurate
Cannot Tell
and concept-reliant downstream tasks. As such, we expect that our pre-training does not heighten the risk of offensive outputs compared to other general-purpose pre-training schemes on wide internet corpora.
The primary risk of instilling models with type knowledge lies in the potential for misinformation (Weidinger et al., 2021). For example, if our model is used to extend existing taxonomies, it runs the risk of hallucinating false types. We observe in Table 6 that while our model achieves high typing precision and recall for seen and unseen types in new documents, we are not at the point where it can be used in isolation to discover and add knowledge to existing knowledge graphs. In parallel with developing better methods for verifying type ontologies and assignments, it is important to incorporate domain experts or crowd-source verification when language models are used to discover facts or type relationships in new documents.
We also advocate for more careful inspection of racial, gender, and socioeconomic biases in existing type ontologies, as it is possible for type-aware models to propagate such biases (e.g. associating people with certain patterns of names with specific occupations).