WEBIE: Faithful and Robust Information Extraction on the Web
Year: 2023

Abstract:
Extracting structured and grounded fact triples from raw text is a fundamental task in Information Extraction (IE). Existing IE datasets are typically collected from Wikipedia articles, using hyperlinks to link entities to the Wikidata knowledge base. However, models trained only on Wikipedia have limitations when applied to web domains, which often contain noisy text or text that does not have any factual information. We present WEBIE, the first large-scale, entity-linked closed IE dataset consisting of 1.6M sentences automatically collected from the English Common Crawl corpus. WEBIE also includes negative examples, i.e. sentences without fact triples, to better reflect the data on the web. We annotate ∼21K triples from WEBIE through crowdsourcing and introduce mWEBIE, a translation of the annotated set in four other languages: French, Spanish, Portuguese, and Hindi. We evaluate the in-domain, out-of-domain, and zero-shot cross-lingual performance of generative IE models and find models trained on WEBIE show better generalisability. We also propose three training strategies that use entity linking as an auxiliary task. Our experiments show that adding Entity-Linking objectives improves the faithfulness of our generative IE models1.

heading: 1 Introduction
text: Information Extraction (IE) is the task of extracting structured information from unstructured text, usually in the form of triples <subject, relation, object>. It is essential for many Natural Language Processing applications such as knowledge base population, question answering, faithful summarisation, and fake news detection (Trisedya et al., 2019; Huguet Cabot and Navigli, 2021; Narayan et al., 2021; Whitehouse et al., 2022).
∗ Work conducted as Research Intern at Amazon Alexa AI. ∗∗ Now at Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, UAE. 1Dataset, code and additional materials are available at https://github.com/amazon-science/WebIE.
Typically, two pieces of information are needed for training closed IE2 systems: (i) the entities mentioned in the text and (ii) the relations that exist between each pair of entities. Obtaining such information requires expensive annotations, therefore most existing IE datasets, such as WikiNRE (Trisedya et al., 2019) or REBEL (Huguet Cabot and Navigli, 2021), are built using Wikipedia, as entity information is available through hyperlinks and relation information can be automatically extracted via distant supervision (DS) approach (Mintz et al., 2009) using a knowledge base (KB) such as Wikidata. The DS approach assumes that if two entities are connected through a relation in a KB, then the sentences that mention both entities together express the relation.
While models trained only on this fact-rich domain3 have shown to be useful for IE applications, they have limited capacity when applied to extracting information in other web domains, which often contains noisy text or text without any factual information. Take AllenAI’s C4 dataset4, an open-sourced version of Google’s C4 (Raffel et al., 2020) dataset based on Common Crawl, as an example. Our analysis using the DS approach reveals that less than 15% of the sentences contain triples (§2.1), whereas we observe that a state-ofthe-art (SOTA) generative IE model, GenIE (Josifoski et al., 2022), which is trained on REBEL, the largest IE dataset to date (which includes only positive examples), tends to generate triples for every sentence, resulting in a high rate of false positives and issues with hallucination.
To address these issues and facilitate future work on IE on the web, we present WEBIE, the first large-scale, entity-linked closed IE dataset collected from web sources. The WEBIE dataset is
2Closed IE refers to the extraction of triples with the entity and relation defined by a knowledge base.
3We use the term domain to refer to the URL domain. 4We use the dataset from https://huggingface.co/
datasets/allenai/c4.
7734
collected from the 200 most frequent URL domains from the C4 dataset. First, we use ReFinED (Ayoola et al., 2022), a state-of-the-art Entity Linking (EL) model to identify mention spans of the entities and link them to Wikidata. We then apply the DS approach to extract triples and use a Natural Language Inference (NLI) model to filter out triples not expressed by the sentence. We also include negative examples, i.e., sentences without any factual information, to better reflect the data on the web. Our final dataset consists of 1.6M sentences, and we annotate a subset of ∼21K triples through crowdsourcing. The annotated set is exclusively used as part of the test set to allow more reliable evaluation. Finally, we introduce mWEBIE, which contains human-corrected translations of the annotated version of WEBIE in four languages: French, Spanish, Portuguese, and Hindi.
Previous works have shown that compared to discriminative pipelines which often suffer from accumulative errors due to separate Entity Linking and Relation Extraction (RE) steps (Mesquita et al., 2019; Trisedya et al., 2019; Josifoski et al., 2022), generative models achieve superior performance in many closed IE tasks. Therefore we primarily benchmark WEBIE with generative, transformerbased encoder-decoder models, BART (Lewis et al., 2020) and mBART (Tang et al., 2021). The latter is used to evaluate the zero-shot cross-lingual transfer performance on mWEBIE.
We further propose three training strategies (§3.2) that use entity linking as an auxiliary task
for generative IE, namely joint generation with the linked-entity prompt (ENTITY-PROMPT), multitask learning with distinguished artificial prompt tokens (ARTIFICIAL-PROMPT), and training with an additional task-specific language model (LM) head (2LM-HEADS). We find that training with EL as an auxiliary task overall leads to better and more faithful IE results. An illustration of these training strategies is provided in Figure 1.
Our experiments show that compared to models trained only on Wikipedia datasets, models also trained on WEBIE are more robust and generalisable, achieving a new SOTA performance on REBEL (§5) and competitive zero-shot performance on WikiNRE. We demonstrate that WEBIE serves as a complementary dataset to existing datasets based on Wikipedia, and show that including negative examples is crucial for addressing false positives in generative IE.
Our main contributions are as follows: (1) We present (m)WEBIE, the first large-scale, entitylinked IE dataset on the web, where a subset is further annotated by humans and translated into four other languages; (2) We propose and study the effectiveness of using entity linking as an auxiliary task for generative IE with various training strategies; (3) Our comprehensive experiments demonstrate that models trained on WEBIE exhibit better generalisability in Information Extraction on the web domain, including competitive zero-shot performance on IE tasks on Wikipedia.

heading: 2 (m)WEBIE
text: In this section, we provide a detailed explanation of the dataset collection process for (m)WEBIE.

heading: 2.1 Collecting WEBIE
text: Data Preprocessing We start with the English portion of the AllenAI’s C4 dataset and keep the most frequent 200 URL domains5. We randomly sample 1M documents and use SpaCy6 for sentence segmentation. Sentences with fewer than 10 words are removed, resulting in ∼20M sentences.
Entity Linking and DS Dataset Next, we run ReFinED (Ayoola et al., 2022), a state-of-the-art EL model on the sentences to identify entity spans and link them to their corresponding Wikidata ID. Besides named entities, ReFinED also extracts numerical entities that do not have Wikidata ID. In this work, we only consider numerical entities that express dates, and map them to the corresponding year for simplicity7. Some examples of ReFinED processed output are included in Appendix B.
After obtaining the entity-linked sentences, we apply the DS paradigm to retrieve the set of relations that exist between each pair of entities in each sentence using Wikidata (September 2022 dump) as our KB and build a DS dataset. After the above steps, we obtain WEBIE DS dataset consisting of 21.2M entities and 4.8M triples.
Entailment Filtering One major drawback of the DS approach is that the triples extracted may or may not be expressed by the source sentence (Riedel et al., 2010). Following previous work on obtaining a cleaner version of the DS dataset (Huguet Cabot and Navigli, 2021; Vania et al., 2022), we apply an NLI model,
5See Appendix D for URL domains included in WEBIE. 6https://spacy.io/ 7For example, “October 10, 2018” will be mapped to “2018”.
nli-deberta-v3-large8, that is trained on SNLI (Bowman et al., 2015) and MultiNLI (Williams et al., 2018), to filter out triples that do not entail the sentence.
Each source sentence is treated as the premise and we use manually created templates (similar to Vania et al. (2022)) to convert a DS triple to one or more hypotheses. We then obtain the entailment probability score for each premise-hypothesis pair and take the maximum score for cases with multiple converted hypotheses. We set the threshold to be 0.7, similar to Huguet Cabot and Navigli (2021), and only keep triples with an entailment score above the threshold. We retain 2.1M triples (44% of the previous DS triples, see Table 1) after this filtering process.
Negative Examples After the DS creation and NLI filtering steps, only less than 10% of the original sentences contain triples. To train models for extracting facts from the web and alleviate false positives, we include two kinds of negative examples in WEBIE: (i) sentences with one or zero entities, and (ii) sentences with two or more entities, but without any factual information (i.e., no relation between the entities). We randomly sample negative instances covering both cases evenly and add them to WEBIE. In the end, WEBIE consists of 1.6M sentences, where 50% are negative examples. A summary of the statistics of WEBIE with a comparison with other datasets is shown in Table 1. The dataset is randomly split into train/validation/test sets using a 90/5/5 split.

heading: 2.2 Human Annotation
text: Existing IE datasets, such as REBEL, are often automatically annotated using the DS approach, hence the labels can be noisy. To allow more reliable evaluation of WEBIE, we randomly sample
8https://huggingface.co/cross-encoder/ nli-deberta-v3-large achieved superior results among the models we evaluated in our preliminary experiments.
∼21K triples from the most frequent 200 relations and annotate them with MTurk. Given a sentence, each HIT (Human Intelligence Task) is designed to verify if a DS triple is correctly expressed in the sentence9. First, the annotators are asked to verify if the head entity (subject) and tail entity (object) are linked correctly. For each entity, we provide its Wikipedia title and link to its Wikidata page as additional context. After that, the annotators are asked to verify if the triple relation is correctly inferred from the sentence. Here, we provide the relation descriptions and example use cases of each relation. We ask three MTurk workers to annotate each DS triple and take the majority vote as the final label for each triple. A triple is considered valid if both entities are linked to the correct Wikidata entities and the relation is inferred10 by the sentence. An annotation interface is shown in Appendix C.
To ensure the annotation quality, we set qualifications with additional requirements for MTurk workers (see Appendix C for details). The agreement among the three annotators is high: 99.4% for the head entities, 99.2% for the tail entities, and 76.1% for the relations have all three annotators agreeing on the same label. After the majority vote, 92.1% of the triples are labelled as inferred and therefore kept as valid triples.

heading: 2.3 Multilingual WEBIE
text: To enable zero-shot cross-lingual transfer evaluation on WEBIE, we further extend the annotated subset, with additional negative examples, to four other languages: French, Spanish, Portuguese, and Hindi. First, we use a neural machine translation model, the distilled 1.3B variant11 of NLLB-200 (Costa-jussà et al., 2022), to translate the English sentences into the target languages. We then use MTurk to verify the translation and add entity span information in the translated sentences. We provide the English sentence (with the entity spans highlighted) and its translation, and first, ask the annotators to correct the translation. After that, MTurk workers are asked to mark the corresponding entity spans in the target language. We ask two annotators to complete the aforementioned HIT, and an additional worker to select the bet-
9We ensure all DS triples in a selected sentence are annotated. 10We ask for inferred instead of explicit expression since some relations may not be explicitly expressed in the sentence, e.g. “located in” (London, UK) or “date of birth” XX (1986-2022).
11https://huggingface.co/facebook/ nllb-200-distilled-1.3B
ter translation, which is used in our final dataset. To obtain translations with higher quality, we restrict the region of the workers to countries where the target language is the official language12. The final mWEBIE consists of 9K instances in each language, which corresponds to roughly 90% of the 21K annotated triples.

heading: 3 Generative Information Extraction
text: This section describes the training strategies that we use for benchmarking (m)WEBIE.

heading: 3.1 Sentence-to-Triples Generation
text: We use BART and mBART for all of our experiments. Given a sentence s as input, we train the model to autoregressively generate the linearised triples t as an output. Following the practice from Huguet Cabot and Navigli (2021) and Josifoski et al. (2022), we linearise a triple ti by converting it into “<sub> head entity label <rel> relation <obj> tail entity label <et>”, where the tags in brackets represent subject, relation, object, and the end of triple, respectively. Head/tail entity label refers to the Wikipedia title that the mention span in the sentence is mapped to, which also has a one-to-one correspondence with the Wikidata ID13.
For each sentence, we order its linearised triples accordingly to the order in which they appear in the input sentence; first by the order of the appearance of the head entity, and then by the order of the tail entity (for cases when the head entities are the same). The conditional probability of generating t is formulated as p(t|s) = ∏Nt=0 p(ti|t<i, s). We use the standard cross-entropy loss and maximise the output sequence likelihood with teacher forcing (Sutskever et al., 2014). An example of input and output can be seen in the top left of Figure 1.

heading: 3.2 Entity-Linking as an Auxiliary Task
text: The standard linearised triples output only contains the label of the entity and not the span. As a result, it may be difficult to trace back from which input span an entity is generated, especially in the case when the model hallucinates (e.g., by generating an entity that is not mentioned in the sentence). To encourage models to generate faithful and interpretable output, we also experiment with models that are jointly optimised for generating triples and
12See details for mWEBIE annotations in Appendix C. 13For example, a mention span of “UK” is linked to Wikipedia
title “United Kingdom” and mapped to Q145 in Wikidata.
EL. The goal of the EL task is to identify and extract entity spans from the input sentence and link them to their corresponding KB entities. We posit that adding the EL task as an additional training objective will teach the model to put attention to the input spans when generating the output. We experiment with the following three approaches.
ENTITY-PROMPT Narayan et al. (2021, 2022) have shown that generation with entity-chain planning, i.e. generating the desired entities first before the actual output, is effective in improving the faithfulness and controlling hallucinations in text generation tasks such as abstractive summarisation. For generative IE tasks, EL can be used as an intermediate plan to ground the generation of the linearised triples. We define the Entity-Linking target in the format of “Mention Span1 # Entity Label1 | Mention Span2 # Entity Label2 | ...”, where the entity spans are ordered as they appear in the text. We then prepend the EL target to the linearised triples target, using special symbols as separators, i.e., “[ENTITY] Entity-Linking target [TRIPLE] Linearised Triples Target”, where “[ENTITY]” is the start symbol before generating the EL output, and “[TRIPLE]” is the start symbol before generating the linearised triples. Given an input sentence, we essentially train the decoder to first generate the EL chain and then generate the triples, conditioned on both the input sentence and the EL output14.
ARTIFICIAL-PROMPT Artificial Prompt tokens are symbols placed in front of the input sequence, which has previously been explored for neural machine translation to distinguish the language of the target output translation (Johnson et al., 2017), and visual question answering for joint answer and explanation generation (Whitehouse et al., 2023). We adapt this approach for jointly training our models for EL and generative IE. Specifically, we use an artificial prompt token <#el#> at the beginning of the input sentence when training for the EntityLinking target, and use <#tri#>15 for linearised output target. Training instances for both tasks are mixed and randomly shuffled for training.
2LM-HEADS Finally, inspired by Gontier et al. (2022), the third approach that we experiment with
14The EL target only includes mention spans that contribute to valid triples, consistent with the triples that are later generated conditioned on the linked entities.
15Both artificial prompt tokens are added as the special tokens to the tokenizer to avoid bias from pre-trained embeddings, but are intended to be biased to the associated task.
is the addition of a second language model (LM) head in the decoder, which is initialised with the same weights as the first (standard) LM head. The first LM head is optimised for generating the linearised triples while the second LM head is optimised for the EL task, thus each training instance has two different target outputs. During training, the input sentence is fed to the encoder once, and different target outputs are given to the same decoder. Each task-specific LM head is then responsible for generating output targeted for it. The training loss is then formulated as a weighted sum of the losses from both tasks: L = αLIE + (1− α)LEL.

heading: 3.3 Inference with a Constraint Trie
text: In addition to standard beam search decoding, we experiment with constraint decoding by restricting the generated output to be valid Wikipedia titles and Wikidata relations using a prefix Trie, following the ideas proposed in GENRE (Cao et al., 2021) and GenIE (Josifoski et al., 2022). We use two constraint Tries: an entity Trie and a relation Trie. The entity Trie is built using all Wikipedia titles (as the entity labels), and the relation Trie is built using all Wikidata relation property labels. We refer the readers to Cao et al. (2021) for more details on constructing the Trie.
We use four special symbols, <sub>, <rel>, <obj> and <et> to define the state of the generation. We apply both constraint Tries as follows. We adopt the constraint Trie so that, in the very first decoding state, the model is allowed to either (i) return an empty string for a negative example, or (ii) generate <sub>, which is the start symbol for generating a triple. If the <sub> symbol is generated, then we generate the head entity using the entity Trie, i.e., only valid entities will be considered. Once the generation of the head entity is completed, the model proceeds to generate <rel> (i.e., the start symbol for generating relation string) and then subsequently generate allowed tokens from the relation Trie which is built from the relations in Wikidata. After that, the model generates <obj> and the tail entity, in the same manner, using the entity Trie. After generating the full triple (indicated by <et> generated after the tail entity), the decoder can either stop the generation or start a new iteration for generating the next triple.
For the ENTITY-PROMPT models, since the entity mention spans are text from the input sentences and usually are not the same as the entity labels in
Wikidata, we propose a partial constraint generation approach. Specifically, we start the standard beam search for the EL target output and only activate the Trie constraints after that when generating the linearised triples.

heading: 4 Experiments
text: In this section, we explain the datasets used in the experiments and the detailed modelling setup.

heading: 4.1 Dataset
text: In addition to our proposed WEBIE dataset, we also use the following datasets for our experiments.
WikiNRE (Trisedya et al., 2019) is an IE dataset based on Wikipedia which is automatically constructed by aligning Wikipedia sentences to Wikidata triples using the DS approach. The authors apply a coreference resolution model (Clark and Manning, 2016) to obtain sentences with implicit entity names, and use a paraphrase detection model (Ganitkevitch et al., 2013; Grycner and Weikum, 2016) to filter out sentences that do not express the DS triples. In our experiments, we only use WikiNRE for zero-shot evaluation.
REBEL (Huguet Cabot and Navigli, 2021) is a large-scale IE dataset constructed automatically from Wikipedia abstracts. Using the Wikipedia hyperlinks in the abstracts, as well as numerical
values and dates, they map the entity spans to their corresponding Wikidata entities. They then use the DS approach to identify triples in each sentence. To filter out false positives, the authors use an NLI model by concatenating the entities and the relation as the hypothesis. In our experiment, we use the REBEL dataset that is sub-sampled by Josifoski et al. (2022), where 857 relations are considered. Both WikiNRE and REBEL do not contain negative examples and are not annotated by humans.

heading: 4.2 Models
text: We experiment with BART using two settings: BARTPLM with the pre-trained weights from Lewis et al. (2020)16, and BARTRAND, using the same configuration and architecture but randomly initialised weights. Across the two settings, Josifoski et al. (2022) find that BARTRAND generates better results than BARTPLM on REBEL. For mWEBIE, we experiment with the mBART-5017 model (for simplicity we refer to it as mBART in this paper).
To compare models trained on different datasets, we train both BARTPLM and BARTRAND on REBEL (R), WEBIE (W), and both datasets together (R+W). We evaluate the performance of the generated triples by parsing the linearised output to a list
16https://huggingface.co/facebook/ bart-large
17https://huggingface.co/facebook/ mbart-large-50
of triples and comparing it to the gold label to calculate precision, recall, and F1 scores. For WEBIE, we also calculate the accuracy of the prediction of negative instances, where a prediction is considered correct if the model accurately generates empty strings rather than hallucinating triples.
For training with EL as an auxiliary task, we primarily experiment with the BARTRAND. We prepare the training instances as described in §3.2, and train separate models on REBEL and on WEBIE. For the 2LM-HEADS, we conduct experiments with different values of the α parameter in the combined loss function, specifically, we set it to 0.5 and 0.75.
We use 8 GPUs, each with 32G VRAM, for all experiments. We set the batch size to 8 and accumulate gradient batches to 32. We follow the hyperparameters settings from Josifoski et al. (2022) and set the learning rate to 3e−5, weight decay to 0.01, and warmup steps to 5K18. We train for up to 30 epochs with early stopping (patience 10), validate twice per epoch, and take the last checkpoint for evaluation. Training one epoch takes ∼1.5 hours for BART and ∼2 hours for mBART.

heading: 5 Results and Analysis
text: We now present the main results of (m)WEBIE and compare different training strategies.

heading: 5.1 Main Results
text: Table 2 shows our benchmarking results on WEBIE. We report results with the constraint Trie in decoding since it overall achieves better results19. Contrary to the findings from Josifoski et al. (2022), we find that BART models with pre-trained weights are better than initialised weights. Constraint Trie decoding benefits REBEL, WikiNRE, and the recall performance of WEBIE, but may compromise
18For BARTPLM (W) we find it is necessary to use a lower learning rate 5e−6 for more stable training.
19See Table 5 in Appendix A for detailed comparison.
the precision since the models are also trained to handle negative examples.
Models trained on both REBEL and WEBIE (R+W) obtain overall better F1 scores on the two datasets compared to models trained on each dataset separately. Similar performance can also be observed in the zero-shot performance on WikiNRE. Models trained solely on the REBEL dataset (Wikipedia-domain) show poor generalisability on WEBIE20 and always generate false positives thus resulting in 0% accuracy for negative instances in WEBIE. This indicates that Wikipedia-domain data only is not adequate for training robust models for the web, and the absence of negative examples in these datasets leads to a prominent issue of hallucination when applied to the web.
BARTPLM (R+W) also achieves a new state-ofthe-art F1 score of 71.87 on REBEL, surpassing the performance of 68.93 from GenIE (Josifoski et al., 2022) and 70.74 from KnowGL (Rossiello et al., 2023), the latter of which trains with additional information including entity type. The results demonstrate the benefit of WEBIE, which contributes to the generalisability of the models.

heading: 5.2 Cross-lingual Transfer with mBART
text: We train mBART on the training set of WEBIE and evaluate the zero-shot cross-lingual transfer on mWEBIE. Similar to prior experiments, results in Table 3 show that constraint Trie decoding obtains higher performance than standard decoding21.
For English, mBART achieves higher overall performance than BARTPLM (see Table 2). The zero-shot results reveal that Hindi has a significant decline in performance compared to the other three non-English languages, French, Spanish, and Portuguese. Since these three languages utilise the
20For positive examples it only achieves 20 F1 points. 21We report results using EN as the source language token for mBART, as it produces better performance compared to the actual source language token. See more details in Appendix A.
Latin script as in English, which may result in an overlap of entity surface forms. In contrast, the transfer is more difficult for Hindi as it employs a different writing system. Manual analysis indicates that mBART tends to produce a high rate of false negatives in Hindi examples, where the correct extraction mostly occurs when the entities in the sentences share similar surface forms with the English counterparts.

heading: 5.3 Results with Additional EL Training
text: Table 4 shows the results of training with EntityLinking as an auxiliary task. For REBEL, the best results are achieved with the 2LM-HEADS approach, where the α parameter is set to 0.75. For WEBIE with negative examples, all EL training models achieve better F1 performance than BARTRAND, with ENTITY-PROMPT particularly resulting in better recall. This shows the benefit of joint training with EL to improve the faithfulness of web domain data. ARTIFICIAL-PROMPT achieves the best precision in WEBIE but does not show significant differences in performance compared to BARTRAND. Nevertheless, all three approaches provide better interpretability, i.e., the information of the mention spans in the text that contributes to the IE prediction.
ENTITY-PROMPT and ARTIFICIAL-PROMPT do not require additional architectural adaptation over the standard model. ENTITY-PROMPT also does not introduce training overhead, whereas the other two models may require twice the training time. 2LM-HEADS offers the flexibility of adapting the weighted combination of the main task and the auxiliary task by adjusting α in the joint loss formula, which allows more emphasis on the main target.

heading: 6 Related Work
text: IE Datasets The term Information Extraction has been used for different tasks in the literature. Most existing IE datasets are collected from Wikipedia articles aligned with Wikidata, including sentence-level IE datasets such as REBEL, WikiNRE, FewRel (Han et al., 2018), T-REx (Elsahar et al., 2018); document-level Relation Extraction22 datasets, e.g., DocRED (Yao et al., 2019), CodRED (Yao et al., 2021). SMiLER (Seganti et al., 2021) is a multilingual sentence-level IE dataset that is also based on Wikipedia, covering 14 languages
22We consider RE dataset as the ones that focus on extracting relations but without entity spans and/or linking information.
and 36 relations. These sentence-level IE datasets typically do not contain negative examples.
Datasets such as TACRED (Zhang et al., 2017), RE-TACRED (Stoica et al., 2021), and WebRED (Ormandi et al., 2021) have negative relation examples but they are not linked to knowledge bases. Our proposed dataset WEBIE is distinct from the existing datasets in that it is on the web domain, entity-linked, and with negative examples.
IE Approaches IE approaches can be classified into two categories: pipeline systems with discriminative models, and sequence-to-sequence systems with generative models. Pipeline models typically include separate modules for Named Entity Recognition (NER), Entity Linking and Relation Extraction (Chaganty et al., 2017; Yamada et al., 2020). Systems that jointly train NER, EL, and RE, have also been explored, taking advantage of the information shared between the tasks (Ji et al., 2020; Eberts and Ulges, 2020).
In recent years, generative IE has gained a lot of attention. Nayak and Ng (2020) utilise an LTSM model and propose a pointer network-based decoding. More recent approaches, e.g. as introduced in REBEL and GenIE, train a transformer-based encoder-decoder model with standard maximumlikelihood objectives to convert sentences to linearised output. KnowGL (Rossiello et al., 2023) improves upon REBEL with additional entity type information added to the linearised output. Our work extends GenIE and experiments with three different approaches where we incorporate explicit EL information as an auxiliary task with adapted constraint Trie decoding.

heading: 7 Conclusions
text: We present (m)WEBIE, the first large-scale, entitylinked closed IE dataset on the web. A subset of the dataset is further annotated by humans and translated into four other languages, French, Spanish, Portuguese, and Hindi, via crowd-sourcing.
We benchmark WEBIE with generative models and compare the models trained on WEBIE and REBEL (Wikipedia-domain). Our results show that models trained on WEBIE have competitive zero-shot performance when applied to REBEL and WikiNRE, whereas models trained only on REBEL have 0% accuracy on the negative examples in WEBIE. This highlights the importance of including negative examples for training more robust models and reducing hallucination in genera-
tive IE on the web. Models trained on both REBEL and WEBIE achieve the best performance on both datasets, as well as zero-shot results on WikiNRE, showing that WEBIE serves as a complementary dataset to existing Wikipedia-domain datasets.
Investigating the approaches with entity linking as an auxiliary task, we find that adding an additional task-specific LM head achieves the overall best performance. The ENTITY-PROMPT approach shows the most significant improvement on WEBIE with the constraint Trie. We primarily benchmark transformer-based encoder-decoder models on WEBIE, but future work could also explore pipeline frameworks and larger language models for few-shot performance.
Limitations
We identify several limitations in this work: (i) False negatives: Our current automatic triple extraction pipeline is built using the DS approach followed by filtering using an NLI model. However, Wikidata is not complete (Tan et al., 2022). While some triples may not be completely available in WEBIE, we expect models trained on this dataset can still discover new triples that do not exist in Wikidata. (ii) Limited relations in annotation: the human annotation is only conducted on the most frequent 200 relations. (iii) Limited languages in mWEBIE: As discussed in §2.3 and Appendix C, the languages in mWEBIE are limited to official languages from geographical regions where there is a reasonable amount of MTurk workers to accept the job. An alternative solution would be to use professional translators, especially for low-resource languages. (iv) Fixed dataset: Facts might change in the world (and Wikidata). This can lead to a degraded real-world performance if a system relies exclusively on WebIE for evaluation when the dataset is not updated accordingly.

heading: Acknowledgements
text: We would like to thank Jens Lehmann for the helpful feedback on the paper draft, and Balkarn Hayre for helping with the MTurk experiments. We also thank the anonymous reviewers for their valuable comments that improved the paper.

heading: A Additional Results
text: We show the full results in Table 5 for BARTRAND and BARTPLM trained on REBEL and WEBIE, using both beam search with and without constraint Trie decoding.
We show in Table 6 the results for non-English languages for mWEBIE when specifying the source language and using the default (English) for the mBART tokenizer. These results are from beam search without constraint Trie. We can see that specifying the source language mostly harms the performance (except French), especially for Portuguese. We hypothesise that due to the model being trained solely on English as the source token, mBART may have difficulty handling other languages.

heading: B Examples of ReFinED Output
text: We show examples of the sentences processed by ReFinED in Table 7. For each input sentence, ReFinED identifies the set of entities in that sentence, and outputs mention span, Wikidata id, and Wikipedia title for each entity. For our experiments, we use the wikipedia_model_with_numbers model with wikipedia entity set.

heading: C MTurk Annotation Details
text: In this section, we describe the detailed settings for annotating (m)WEBIEwith MTurk.
C.1 WEBIE The first annotation task (HIT) is to verify the correctness of the triples automatically created from the DS approach and filtered by the NLI model. The guidance and the interface are shown in Figure 2 and Figure 3, respectively.
In each HIT, we provide a sentence with its entities highlighted (head entity in blue and tail entity in green) and the URL of the web page which the sentence is extracted from. For the first EL annotation job, we provide both links to the Wikipedia and
Wikidata pages. Annotators are asked to choose if the highlighted spans are linked correctly to the KB. Next, the annotators are asked to verify if a relation (highlighted in orange) can be inferred from the sentence. We provide the description of the relation and an example use case to facilitate the annotation.
Each triple is annotated by three workers, and we pay $0.2 per HIT. We hire MTurk workers with Masters Qualification and set additional requirements including (i) having done 2,000 HITs and (ii) having a job approval rate ≥99%.
C.2 mWEBIE
Figure 4 and Figure 5 illustrates the interface for correcting machine-translated sentence and identifying corresponding entities in them. As it is challenging to find qualified crowd workers for the translation task23, we set the geographical regions for each language to the countries where the language is one of the official languages. We find
23Preliminary results where we include the USA for the mWEBIE annotation task indicate that MTurk workers with limited or no knowledge of the target language (or English) still accept the job, despite our specific requirement for proficiency in both English and the target language.
that only India and countries in America have an adequate number of MTurk workers, which highly restricts the options for our target languages. In the end, the countries we set for the target languages are as follows: Portuguese: AO, BR, CV, ST, GW, GQ, MZ; Spanish: ES, MX, CO, PE, CL; CA for French, and IN for Hindi24. It was also necessary to remove the Masters Qualification requirement for MTurk workers (except Hindi) to find adequate annotators. We then conduct pilot annotations, where we deliberately introduce errors in the reference machine translation to verify if the workers under our requirement settings are able to correct them.
We provide the English sentence paired with the original machine-translated sentence for the actual HIT. The English sentence is highlighted with its entity spans, and we instruct the workers to correct the translation while ensuring that the entities are correctly translated. After confirming the translation, workers are then asked to highlight the corresponding entities in the target language (in green). For negative sentences without entity spans, the longest noun phrases were highlighted instead to prevent workers from simply copying the reference translations. We pay $0.35 per HIT for positive sentences and $0.25 for negative sentences (since most sentences in negative examples have only one highlighted entity/noun phrase and it is
24For the mapping between country codes and countries, please refer to https://docs.aws.amazon. com/AWSMechTurk/latest/AWSMturkAPI/ ApiReference_LocaleDataStructureArticle. html
considered an easier task). Two MTurk workers are asked for the translation task, and an additional worker was asked to select the better translation, for which $0.10 per HIT was paid.

heading: D Domains in WEBIE
text: The 200 URL domains included in WEBIE are shown in Table 8.

heading: E Relations in the Annotated Set
text: Table 9 shows the details of the 200 relations that are covered in the human-annotated set of WEBIE.
COUNT PID RELATION DESCRIPTION
1359 P17 country sovereign state of this item (not to be used for human beings) 910 P131 located in the administra-
tive territorial entity the item is located on the territory of the following administrative entity. Use P276 for specifying locations that are non-administrative places and for items about events. Use P1382 if the item falls only partially into the administrative entity.
776 P530 diplomatic relation diplomatic relations of the country 684 P47 shares border with countries or administrative subdivisions, of equal level, that this item borders, either by land or water. A single common point is enough. 655 P27 country of citizenship the object is a country that recognizes the subject as its citizen 588 P161 cast member actor in the subject production .use "character role" (P453) and/or "name of the character role" (P4633) as qualifiers, use "voice actor" (P725) for voice-only role 580 P577 publication date date or point in time when a work was first published or released 546 P527 has part(s) part of this subject 480 P54 member of sports team sports teams or clubs that the subject represents or represented 438 P800 notable work notable scientific, artistic or literary work, or other work of significance among subject’s works 437 P463 member of organization, club or musical group to which the subject belongs. Do not use for membership
in ethnic or social groups, nor for holding a political position, such as a member of parliament (use P39 for that).
430 P108 employer person or organization for which the subject works or worked 426 P127 owned by owner of the subject 400 P361 part of object of which the subject is a part (if this subject is already part of object A which is a part of object B, then please only make the subject part of object A) 378 P1830 owner of entities owned by the subject 370 P102 member of political party the political party of which a person is or has been a member or otherwise affiliated 364 P150 contains the administra-
tive territorial entity (list of) direct subdivisions of an administrative territorial entity
359 P749 parent organization parent organization of an organization, opposite of subsidiaries 340 P178 developer organization or person that developed the item 314 P159 headquarters location city, where an organization’s headquarters is or has been situated. Use (P276) qualifier for specific building 310 P57 director director(s) of film, TV-series, stageplay, video game or similar 299 P118 league league in which team or player plays or has played in 297 P1376 capital of country, state, department, canton or other administrative division of which the municipality is the governmental seat 296 P449 original broadcaster network(s) or service(s) that originally broadcast a radio or television program 293 P36 capital seat of government of a country, province, state or other type of administrative territorial entity 285 P2936 language used language widely used (spoken or written) in this place or at this event 280 P355 has subsidiary subsidiary of a company or organization; generally a fully owned separate corporation 279 P175 performer actor, musician, band or other performer associated with this role or musical work 267 P166 award received award or recognition received by a person, organization or creative work 267 P569 date of birth date on which the subject was born 262 P641 sport sport that the subject participates or participated in or is associated with 258 P26 spouse the subject has the object as their spouse (husband, wife, partner, etc.). Use "unmarried partner" (P451)) for non-married companions 247 P571 inception time when an entity begins to exist; for date of official opening use P1619 241 P176 manufacturer manufacturer or producer of this product 234 P40 child subject has object as child. Do not use for stepchildren 233 P170 creator maker of this creative work or other object (where no more specific property exists) 227 P3373 sibling the subject and the object have at least one common parent (brother, sister, etc. including
half-siblings); use "relative" (P1038) for siblings-in-law (brother-in-law, sister-in-law, etc.) and step-siblings (step-brothers, step-sisters, etc.)
227 P50 author main creator(s) of a written work (use on works, not humans); use P2093 when Wikidata item is unknown or does not exist 226 P570 date of death date on which the subject died 224 P276 location location of the object, structure or event. In the case of an administrative entity as containing
item use P131. For statistical entities use P8138. In the case of a geographic entity use P706. Use P7153 for locations associated with the object.
204 P674 characters characters which appear in this item (like plays, operas, operettas, books, comics, films, TV series, video games) 203 P1412 languages spoken, written or signed language(s) that a person or a people speaks, writes or signs, including the native language(s) 201 P1441 present in work this (fictional or fictionalized) entity or person appears in that work as part of the narration (use P2860 for works citing other works, :P361/P1433 for works being part of other works, P1343 for entities described in non-fictional accounts) 201 P945 allegiance country (or other power) that the person or group serves 197 P58 screenwriter person(s) who wrote the script for subject item 197 P37 official language language designated as official by this item 193 P137 operator person, profession, or organization that operates the equipment, facility, or service 193 P162 producer person(s) who produced the film, musical work, theatrical production, etc. (for film, this does not include executive producers, associate producers, etc.) 185 P1411 nominated for award nomination received by a person, organisation or creative work
COUNT PID RELATION DESCRIPTION
184 P1056 product or material produced material or product produced by a government agency, business, industry, facility, or process 183 P35 head of state official with the highest formal authority in a country/state 180 P206 located in or next to body
of water body of water on or next to which a place is located
180 P1001 applies to jurisdiction the item (institution, law, public office, public register...) or statement belongs to or has power over or applies to the value (a territorial jurisdiction: a country, state, municipality, ...) 180 P144 based on the work(s) used as the basis for subject item 177 P156 followed by immediately following item in a series of which the subject is a part, preferably use as qualifier of P179 176 P112 founded by founder or co-founder of this organization, religion or place 174 P155 follows immediately prior item in a series of which the subject is a part, preferably use as qualifier of P179 171 P488 chairperson presiding member of an organization, group or body 169 P279 subclass of this item is a subclass (subset) of that item; all instances of these items are instances of those
items; different from P31 (instance of), e.g.: K2 is an instance of mountain; volcano is a subclass of mountain (and an instance of volcanic landform).
169 P169 chief executive officer highest-ranking corporate officer appointed as the CEO within an organization 168 P86 composer person(s) who wrote the music [for lyricist, use "lyrics by" (P676) 164 P140 religion or worldview religion of a person, organization or religious building, or associated with this subject 163 P750 distributed by distributor of a creative work; distributor for a record label; news agency; film distributor 161 P974 tributary watercourse that flows into an other one (for lake inflows use P200) 159 P6087 coach of sports team sports club or team for which this person is or was on-field manager or coach 157 P197 adjacent station the stations next to this station, sharing the same line(s) 156 P1344 participant in event in which a person or organization was/is a participant 155 P272 production company company that produced this film, audio or performing arts work 154 P461 opposite of item that is the opposite of this item 152 P1365 replaces person, state or item replaced. Use "structure replaces" (P1398) for structures. 152 P277 programmed in the programming language(s) in which the software is developed 151 P19 place of birth most specific known (e.g. city instead of country, or hospital instead of city) birth location of a person, animal or fictional character 150 P1366 replaced by other person or item which continues the item by replacing it in its role. Use P156 ("followed by") if the item is not replaced nor identical, but adds to the series (e.g. books in a series). 148 P585 point in time time and date something took place, existed or a statement was true 148 P710 participant person, group of people or organization (object) that actively takes/took part in an event or
process (subject). Preferably qualify with "object has role" (P3831)). Use P1923 for participants that are teams.
147 P466 occupant person or organization occupying property 144 P7047 enemy of opponent character or group of this fictive character or group 143 P580 start time time a time period starts 138 P403 mouth of the watercourse the body of water to which the watercourse drains 135 P400 platform platform for which a work was developed or released, or the specific platform version of a software product 134 P1327 partner in business or
sport professional collaborator
134 P22 father male parent of the subject. Not stepfather 134 P414 stock exchange exchange on which this company is traded 133 P306 operating system operating system (OS) on which a software works or the OS installed on hardware 129 P1346 winner winner of a competition or similar event, NOT to be used for awards (instead use "award received" on awardee’s item, possibly qualified with "for work" or for wars or battles 128 P1889 different from item that is different from another item, with which it may be confused 128 P4969 derivative work new work of art (film, book, software, etc.) derived from major part of this work 127 P31 instance of that class of which this subject is a particular example and member; different from ’subclass
of’; for example: K2 is an instance of mountain; volcano is a subclass of mountain (and an instance of volcanic landform)
127 P30 continent continent of which the subject is a part 124 P397 parent astronomical body major astronomical body the item belongs to 122 P607 conflict battles, wars or other military engagements in which the person or item participated 120 P2789 connects with item with which the item is physically connected 120 P1038 relative family member (qualify with "type of kinship"; for direct family member please use specific property) 119 P1891 signatory person, country, or organization that has signed an official document 118 P1029 crew member(s) person(s) that participated operating or serving aboard this vehicle 115 P937 work location location where persons or organisations were actively participating in employment, business or other work 114 P495 country of origin country of origin of this item (creative work, food, phrase, product, etc.) 113 P1557 manifestation of inherent and characteristic embodiment of a given concept 113 P6 head of government head of the executive power of this town, city, municipality, state, country, or other governmental
body
COUNT PID RELATION DESCRIPTION
112 P451 unmarried partner someone with whom the person is in a relationship without being married. Use "spouse" (P26) for married couples 112 P725 voice actor performer of a spoken role in a creative work such as animation, video game, radio drama, or dubbing over 112 P123 publisher organization or person responsible for publishing books, periodicals, printed music, podcasts, games or software 111 P264 record label brand and trademark associated with the marketing of subject music recordings and music videos 111 P737 influenced by this person, idea, etc. is informed by that other person, idea, etc., e.g. “Heidegger was influenced by Aristotle” 110 P706 located in/on physical feature located on the specified (geo)physical feature. Should not be used when the value is only political/administrative (P131) or a mountain range (P4552). 109 P3095 practiced by type of agents that study this subject or work in this profession 108 P1716 brand commercial brand associated with the item 106 P115 home venue home stadium or venue of a sports team or applicable performing arts organization 106 P3461 designated as terrorist by country or organization that has officially designated a given group as a terrorist organization 105 P136 genre creative work’s genre or an artist’s field of work. Use main subject P921 to relate creative works to their topic 105 P69 educated at educational institution attended by subject 105 P1532 country for sport country a person or a team represents when playing a sport 104 P172 ethnic group subject’s ethnicity (consensus is that a VERY high standard of proof is needed for this field to
be used. In general this means 1) the subject claims it themselves, or 2) it is widely agreed on by scholars, or 3) is fictional and portrayed as such)
103 P205 basin country country that have drainage to/from or border the body of water 103 P20 place of death most specific known (e.g. city instead of country, or hospital instead of city) death location of a person, animal or fictional character 102 P1923 participating team like ’Participant’ (P710) but for teams. For an event like a cycle race or a football match you
can use this property to list the teams and P710 to list the individuals (with ’member of sports team’ P54 as a qualifier for the individuals)
101 P398 child astronomical body minor body that belongs to the item 100 P179 part of the series series which contains the subject 100 P450 astronaut mission space mission that the subject is or has been a member of (do not include future missions) 99 P25 mother female parent of the subject. Not stepmother 99 P8345 media franchise this creative work belongs to this media franchise 98 P582 end time time a time period ends 97 P2341 indigenous to place or ethnic group where a language, art genre, cultural tradition or expression, cooking style or food, or biological species or variety is found (or was originally found) 95 P194 legislative body legislative body governing this entity; political institution with elected representatives, such as a parliament/legislature or council 95 P840 narrative location the narrative of the work is set in this location 95 P287 designed by person(s) or organization which designed the object 94 P103 native language language or languages a person has learned from early childhood 94 P7959 historic county traditional, geographical division of Great Britain and Ireland 93 P113 airline hub airport that serves as a hub for an airline 93 P121 item operated equipment, installation or service operated by the subject 91 P6886 writing language language in which the writer has written their work 91 P6379 has works in the collec-
tion collection that has works of this person or organisation (use archive location P485 for the archives)
90 P126 maintained by person or organization in charge of keeping the subject (for instance an infrastructure) in functioning order 90 P647 drafted by which team the player was drafted by 90 P551 residence the place where the person is or has been, resident 90 P3342 significant person person linked to the item in any possible way 89 P1431 executive producer executive producer of a movie or TV show 88 P1416 affiliation organization that a person or organization is affiliated with (not necessarily member of or employed by) 88 P138 named after entity or event that inspired the subject’s name, or namesake (in at least one language) 87 P2031 work period (start) start of period during which a person or group flourished in their professional activity 87 P241 military branch branch to which this military unit, award, office, or person belongs, e.g. Royal Navy 87 P2541 operating area geographic area or jurisdiction an organisation or industry operates in, serves, or has responsibility for 86 P676 lyrics by author of song lyrics 86 P1191 date of first performance date a work was first debuted, performed or live-broadcasted 85 P190 twinned administrative
body twin towns, sister cities, twinned municipalities and other localities that have a partnership or cooperative agreement, either legally or informally acknowledged by their governments
84 P598 commander of for persons who are notable as commanding officers, the units they commanded 84 P84 architect person or architectural firm responsible for designing this building 83 P1336 territory claimed by administrative divisions that claim control of a given area 82 P199 business division organizational divisions of this organization (which are not independent legal entities)
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations

heading: 3 A2. Did you discuss any potential risks of your work?
text: Limitations

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 4

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 5

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 2,4,5
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 2

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? 2, appendix B

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Appendix B
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. we use MTurk platform
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. we use MTurk platform

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Appendix B

heading: Future Work
text: For an event like a cycle race or a football match you
can use this property to list the teams and P710 to list the individuals (with ’member of sports team’ P54 as a qualifier for the individuals)
101 P398 child astronomical body minor body that belongs to the item 100 P179 part of the series series which contains the subject 100 P450 astronaut mission space mission that the subject is or has been a member of (do not include future missions) 99 P25 mother female parent of the subject. Not stepmother 99 P8345 media franchise this creative work belongs to this media franchise 98 P582 end time time a time period ends 97 P2341 indigenous to place or ethnic group where a language, art genre, cultural tradition or expression, cooking style or food, or biological species or variety is found (or was originally found) 95 P194 legislative body legislative body governing this entity; political institution with elected representatives, such as a parliament/legislature or council 95 P840 narrative location the narrative of the work is set in this location 95 P287 designed by person(s) or organization which designed the object 94 P103 native language language or languages a person has learned from early childhood 94 P7959 historic county traditional, geographical division of Great Britain and Ireland 93 P113 airline hub airport that serves as a hub for an airline 93 P121 item operated equipment, installation or service operated by the subject 91 P6886 writing language language in which the writer has written their work 91 P6379 has works in the collec-
tion collection that has works of this person or organisation (use archive location P485 for the archives)
90 P126 maintained by person or organization in charge of keeping the subject (for instance an infrastructure) in functioning order 90 P647 drafted by which team the player was drafted by 90 P551 residence the place where the person is or has been, resident 90 P3342 significant person person linked to the item in any possible way 89 P1431 executive producer executive producer of a movie or TV show 88 P1416 affiliation organization that a person or organization is affiliated with (not necessarily member of or employed by) 88 P138 named after entity or event that inspired the subject’s name, or namesake (in at least one language) 87 P2031 work period (start) start of period during which a person or group flourished in their professional activity 87 P241 military branch branch to which this military unit, award, office, or person belongs, e. g. Royal Navy 87 P2541 operating area geographic area or jurisdiction an organisation or industry operates in, serves, or has responsibility for 86 P676 lyrics by author of song lyrics 86 P1191 date of first performance date a work was first debuted, performed or live-broadcasted 85 P190 twinned administrative
body twin towns, sister cities, twinned municipalities and other localities that have a partnership or cooperative agreement, either legally or informally acknowledged by their governments
84 P598 commander of for persons who are notable as commanding officers, the units they commanded 84 P84 architect person or architectural firm responsible for designing this building 83 P1336 territory claimed by administrative divisions that claim control of a given area 82 P199 business division organizational divisions of this organization (which are not independent legal entities)
ACL 2023 Responsible NLP Checklist