CORE: A Few-Shot Company Relation Classification Dataset for Robust Domain Adaptation.
Authors: Philipp Borchert
Jochen De Weerdt
Kristof Coussement
Arno De Caigny
Marie-Francine Moens
Venue: EMNLP 2023 Main (2023)
Keywords: Few-shot learning
Relation classification
Business application of NLP
Information extraction
TL;DR: We present a high-quality dataset for few-shot relation classification and find that models trained on our dataset showcase improved out-of-domain performance.

Abstract: We introduce CORE, a dataset for few-shot relation classification (RC) focused on company relations and business entities. CORE includes 4,708 instances of 12 relation types with corresponding textual evidence extracted from company Wikipedia pages. Company names and business entities pose a challenge for few-shot RC models due to the rich and diverse information associated with them. For example, a company name may represent the legal entity, products, people, or business divisions depending on the context. Therefore, deriving the relation type between entities is highly dependent on textual context. To evaluate the performance of state-of-the-art RC models on the CORE dataset, we conduct experiments in the few-shot domain adaptation setting. Our results reveal substantial performance gaps, confirming that models trained on different domains struggle to adapt to CORE. Interestingly, we find that models trained on CORE showcase improved out-of-domain performance, which highlights the importance of high-quality data for robust domain generalization. Specifically, the information richness embedded in business entities allows models to focus on contextual nuances, reducing their reliance on superficial clues such as relation-specific verbs. In addition to the dataset, we provide relevant code snippets to facilitate reproducibility and encourage further research in the field. The CORE dataset and code are publicly available at \url{https://anonymous.4open.science/r/CORE-D377}.

CORE: A Few-Shot Company Relation Classification Dataset for Robust Domain Adaptation.
Philipp Borchert .ochen De Weerdt Kristof Coussement', Arno De Caigny', Sharie-Francing Moens IESEG School of Management, Univ. Lille, CNRS, UMR 9221 LEM Lille Economie Management, F-59000 Lille, France Research Centre for Information Systems Engineering, KU Leuven, Belgium 'Department of Computer Science, KU Leuven, Belgium
Abstract
advertising is pervasive, as one of Woodruff's s stated goals was to ensure that everyone on Earth drank Coca- Cola as their preferred beverage.
Coke's E1
We introduce CORE, dataset for few-shot re- lation classification (RC) focused on company relations and business entities. CORE includes 4,708 instances of 12 relation types with corre- sponding textual evidence extracted from com- pany Wikipedia pages. Company names and business entities pose a challenge for few-shot RC models due to the rich and diverse infor- mation associated with them. For example, company name may represent the legal entity, products, people, business divisions depend- ing on the context. Therefore, deriving the relation type between entities is highly depen- dent on textual context. To evaluate the per- formance of state-of-the-art RC models on the CORE dataset, we conduct experiments the few-shot domain adaptation setting. Our results reveal substantial performance gaps, confirm- ing that models trained different domains struggle to adapt to CORE. Interestingly, we find that models trained on CORE showcase improved out-of-domain performance, which highlights the importance of high-quality data for robust domain adaptation. Specifically, information richness embedded in business tities allows models to focus on contextual nu- ances, reducing their reliance on superficial clues such as oelation-specifit verbs. In ad- dition to the dataset, we provide relevant code snippets facilitate reproducibility and encour- age further research in the field.'
Legal Entity
Brand
Product
Figure Example for information richness embedded in named business entities.
et al., 2019; Sabo et al., 2021), and out-of-domain adaptation (Gao al., 2019). However, most isting RC datasets comprise a mixture of domains, and the sets of entities across different domains are often easily distinguishable from one another. The main challenge with these general domain datasets is that few-shot models may take shortcuts during training, given the limited set of possible relation types associated with specific entities (Liu et al., 2014; Lyu and Chen, 2021). For instance, the rela- tion type "place of birth" requires pair of entities consisting of entity types "Person" and "Geoloca- tion". This can lead to poor generalization perfor- mance when evaluated on out-of-domaan data.
To address these challenges, we introduce CORE, a high-quality dataset that focuses specif- ically on company relations. By narrowing the scope company relations and business entities, CORE presents more difficult task for models, as business entities embody a variety of informa- tion and may be used interchangeably to represent the legal entity, products or services, and brands, depending on the context (see Figure 1). As a re- sult, inferring the relation type solely based the entities becomes more challenging. Additionally, business entities are well-suited minimize entity type related clues due to the richness of associated information they possess. By minimizing these clues, CORE aims to incentivize models to more
1 Introduction
Relation classification (RC) is a fundamental task in natural language processing that involves identi- fying the corresponding relation between a pair of entities based on the surrounding textual context. Several studies highlight the need for more realis- tic dataset configurations that include None-of-the- Above (NOTA) detection (Gao et al., 2019; Sabo et al., 2021), fhw- evaluation (Baldini Soares
'The CORE dataset and code are publicly available at https://github com/ httm//pbbrrherre chorrt/certc

---

Published
weekly, the Commercial & Financial Chronicle E1 was deliberately modeled to be an American take on the popular business newspaper The Economist E2 which had been founded in England in 1843.
competitor of
acquired by
E2
merged with Relation type annotation
Entity position annotation
Relation direction annotation
Figure 2: CORE example for an undirected business relation.
effectively leverage textual context and improve predictive performance on out-of-domain data.
CORE itself outperform other fomain--dapteed models. Lastly, our study highlights the critical role of high-quality datasets in enabling robust domain adaptation for few-shot RC. Models evaluated on out-of- of-oomain data commonly rely on heuristic pat- terns and face challenges in adapting to contextual information. In summary, this paper makes sig- nificant contribution by introducing CORE, novel and high-quality dataset for relation classification. In the context of few-shot domain adaptation, our study demonstrates the challenges faced by state- of of-thh-arr art models when dealing with CORE, while highlighting the remarkable performance models trained on CORE itself.
The few-shot settings commonly used in dis- tantly supervised datasets are built upon unrealistic assumptions (Sabo et al., 2021). These assump- tions include unrealistic relation types, NOTA and entity distributions, as well as narrow focus on named entities while disregarding common nouns and pronouns. The study shows that models trained on more realistic few-shot evaluation settings ex- hibit significant decline in predictive performance. In contrast, CORE overcomes these limitations as it is human-annotated dataset, not bound by the relations or named entity types present in knowl- edge bases. It encompasses relation types that are absent in knowledge bases, and includes en- tity types beyond named entities, such as common nouns and pronouns. On the contrary, distant su- pervision relies on the extraction of textual context based subiecc--elltionnojjjee triples in knowl- edge bases (Han al., 2018), which restricts the scope to named entities (Sabo al., 2021).
Related work
Recent research datasets for few-shot RC has been focused on improving data quality. Several up- proaches have been proposed, including extensions and augmentations of existing datasets to incorpo- rate features such as NOTA detection or domain adaptation, as well as correction labeling errors (Alt et al., 2020; Gao et al., 2019).
We contribute to the field of relation classifica- tion in three significant ways through study. First, we introduce CORE, a high-quality dataset that encompasses company relations and extends domain coverage in RC. By focusing on company names and business entities, CORE creates a more demanding benchmark for relation classification models. These entities embody diverse information and can be used interchangeably to represent legal entities, products services, and brands, depend- ing on the context. Secondly, our benchmark of state-of-the-art models on CORE uncovers substan- tial performance gaps few-shot domain adap- tation. RC models trained on different domains struggle to adapt to CORE, whereas models trained
In this section, present overview of recent RC datasets that encompass sentence-level textual context. One of such datasets is TACRED, which a numan-annotated dataset based on online news, focusing on people and organizations as entities (Zhang et al., 2017). ). It preserves a realistic rela- tion type distribution and includes 79.5% NOTA instances. Alt et al. (2020) re-evaluated the original dataset, analyzing labeling errors, and investigating sources for incorrect model predictions. Another prominent dataset is FewRel, introduced by Han et al. (2018), which consists of text passages ex- tracted from English Wikipedia pages. is anno- tated using distant supervision and the Wikidata

---

knowledge base. The model evaluation is primarily focused on the K-Way N-Shot setting. Gao al. (2019) extended the dataset by incorporating NOTA examples and introducing The CrossRE dataset specifically targets the do- main adaptation setting in relation classification (Bassignana and Plank, 2022). It incorporates text from Wikipedia and news sources, covering six do- mains. Notably, the dataset excludes the business domain and is limited to named entities. In terms of relation types, CrossRE adopts a shared approach across domains, omitting any domain- -ssuaiii-pcciiccce re- lation types. Khaldi et al. (2022) acknowledge the value of business relations extracted from multilin- gual web sources and present their dataset, which is collected using distant supervision. This dataset focuses primarily named entities and subsequent human annotation. It contains five types of undi- rected business relationships.
Large (Liu et al., 2019) that was fine-tuned on the CoNLL2003 dataset retain entities tagged as or- ganizations or miscellaneous. The model achieved state-of oftte-of-th--e the-art performance on CoNLL2003 with a FI-Score of 97% on the full dataset and 96% on entities labeled as organizations. This extraction procedure allows the company corresponding to the Wikipedia page take any entity type. includ- ing named entities, pronouns, and nouns. We use spaCy to create a dictionary of nouns and pronouns in the dataset and manually validate entity types (Honnibal and Montani, 2017). For each candidate entity pair, we extract the corresponding paragraph as supporting context characterizing the relation between the entities. We remove duplicate entity pairs with overlapping context reduce the num- ber of candidate samples and manually filter out miscellaneous entities that do not belong to the cat- egory of business entities, such "German" "Dutch". Finally, we randomly sampled 14,000 candidate samples for subsequent annotation.
test sets.
Domain adaptation refers to the task of transfer- ring information from a source domain to a specific target domain. Domain adaptation models learn to extract domain-invariant information from the source domain, which can be applied to the target domain. Popular approaches include domain ad- versarial networks (Han et al., 2021a; Gao al., 2019) and leveraging external information such as task descriptions or knowledge bases (Zhang et al., 2021; Zhenzhen et al., 2022). Recent work has also explored prompt tuning approaches (Gao et al., 2021; Schick and Schütze, 2021; Wu and Shi, 2022). In this study, we address the problem of domain adaptation by exposing the model to limited number of instances from the target domain using few-shot learning. To achieve effective do- main adaptation, models combine domain-invariant information learned from the source domain with the information available from the target domain.
The annotation platform is created by the au- thors, offering a similar interface commercial services such as MTurk (see Appendix A.5). The dataset annotated by 50 students in business and management who are presented with candidate en- tity pairs and the corresponding textual context. The annotation consists of selecting one of the predefined relations (see Appendix A.1) and the textual context relevant determine the selected relation (see Figure 2). Since relations such as "ac- quired by" imply a direction, annotators are also asked to indicate whether the entity order is re- versed: (e1, (2) - (e2, r, e1). The relation anno- tation and the position of el and c2 in the textual context are validated in the subsequent annotation step. Thereby, the validation process includes the removal of candidate instances that contain non- business entities, incorrect relation type selection, insufficient context, or inappropriate data.
3 CORE
3.1
Data collection
The annotators were recruited through an open application process. For their contribution to the annotation process, the students received reim- bursement of $11 per hour. The total compensa- tion for data annotation amounted approximately $6,600.
To gather corporate Wikipedia pages in English, we conducted query on DBpedia (Lehmann et al., 2015) using the ontology class "dbo:Company", which corresponds to the Wikidata identifier "Q4830453". The dataset was constructed using Wikipedia content corresponding to April 2022.
3.2 Quality control
We extract candidate entity pairs based on the company corresponding to the Wikipedia page and named entities the text. We use named en- tity recognition (NER) model based on RoBERTa-
Data quality is particularly important in low re- source settings with models exposed to only few supporting instances. Alt et al. (2020) find that

---

800
715
659
600
Count
129
400
302
240
216
200
160
92
85
produca or service
undefined
collaboration
aequired by
client of shareholder of subsidiary of
traded on
competitar of brand of merged with regulated by
Figure 3: Relation types and distribution in the training set.
incorrectly labeled instances substantially reduce predictive performance in RC models. In close consideration of recent literature, we maintain high data quality in CORE (Alt et al.. 2020; Gao et al.. 2019; Sabo et al., 2021). The annotation process involves at least three different annotators, with the annotator agreement computed based relation type annotation, relation direction annotation, and entity annotation. The corresponding Fleiss Kappa is 0.636. We only retain instances with at least 66% intr-annotator agreement.
Category
# Train
# Test
Instances Relations
4,000
708
Directed Undirected
4
Total
12
12
Entity types
Named entity
6,777
1,214
Noun
721
114
Pronoun
502
88
Avg. tokens
34
34
Table
Dataset characteristics
We educate annotators by providing a mandatory step-by-step tutorial along with guiding documen- tation on the platform. Further quality control mea- sures include control set annotated by the authors. Consequently, excluded annotations from 6 an- notators who did not pass the quality control check. We employ consistency checks during the anno- tation dialog on the platform. For each example, the annotators select business entity types, such as "Company" or "Brand", corresponding to el and e2, The platform validates corresponding relations and entity types, i.e., annotators are required to indicate one of the entities "product service" when selecting the "product service of" relation.
of shared instances and relation types. We utilized cosine similarity based on tf-idf values to quantify the textual similarity between instances. Subse- quently, we manually analyzed 76 instance pairs that exhibited similarity greater than 50%. Our analysis revealed nine pairs with matching context, among which one instance pair shared entities and relation type annotations. The remaining pairs fo- cused on differentt entities and served as examples for various relation types. Examples of these in- stances are provided in Table 2.
Furthermore, we performed an analysis of the Wikidata propertics used in FewRel, including P449 (original broadcaster). P123 (publisher), P178 (developer), P176 (manufacturer) and P750 (distributed by). We observed that all of these properties align with CORE's "product service of" relation type. Furthermore, found that the properties P355 (has subsidiary) and P127 (owned by) partially overlap with "subsidiary of and "acquired by" in CORE. However, these properties include non-business entities and have a broader interpretation of "subsidiary" and "ownership", encompassing concepts like "part-of", physical location, or control. Examples illustrating this overlap are shown in Appendix A.2.
3.3 Dataset properties and analysis
The dataset's key properties are presented in Table and discussed in the subsequent section. CORE consists of annotated relation types that were predefined by the authors, as shown in Figure 3. Among these, there is an "undefined" relation that signifies relationship between el and e2 that does not match any predefined relation type. Thus, "un- defined" corresponds to the NOTA category. How- ever, examples containing non-business entities or missing entities were excluded from the dataset.
To better understand the distinctions between CORE and FewRel, which are both based on Wikipedia, we examined their similarities in terms

---

Text
CORE
Few has subsidiary
On 4 September 2014, they changed their business model coupon client of
store and served their clients.
Flipkart
Jabong,
Myntra
Zovi, Koovs Zivame
Its
games include
Microsoft Solitaire Collection
which comes
undefined
operating system
loaded
Windows 10
Windows and mobile devices.
Other of the company's s hits were Bonequinha de Seda in 1936 product or service of followed by
Estudantes
in
1935 Ganga Bruta in 1933, and Limite 1931.
Table 2: Shared instances between CORE and FewRel with corresponding entities highlighted in blue and orange, respectively.
3.4 Dataset splits
of support instances per relation, disregarding the relation distribution in the dataset. Considering the relation distribution in the model evaluation, we sample N K query instances randomly (Sabo et al., 2021). All instances included in training episodes are sampled from the training set. For test episodes we sample support instances from the training set and query instances from the test set.
The annotated instances were randomly divided into a training set (4,000 instances) and (708 instances), each split containing all available relation types. We do not include a dedicated de- velopment set, but use part of the training set for model development (Mukherjee et al., 2021).
4 Experiments
4.3 Models
4.1 Task formulation
To facilitate the comparison of evaluation results, benchmark SOTA models in few-shot RC litera- ture, considering performance and prior application in the domain adaptation setting (Sabo et al., 2021; Gao et al., 2021). We utilize BERT-Base with 110 million parameters (Devlin et al., 2019) as an en- coder for all benchmarked models. This prevents information disparities between language models, which are commonly trained on Wikipedia data. Prototypical Networks (Proto) create relation type prototypes using the support examples (Snell et al., 2017). Query instances are matched to rela- tion prototypes by computing similarity scores. In the BERT-Pair approach, query instances are concatenated with support instances computing a similarity probability to determine whether the query instances share the same relation type as the support instances (Gao et al., 2019).
We adopt the N-way K-shot evaluation setting, where the model is exposed to limited number of instances N X K during training. The episodes sampled from the dataset consist of two parts: (1) support set S {xi KxN containing input æ along with N randomly sampled relations r R
as well as K instances for each relation and (2) query set Q = {æ,} /=1 containing instances the N relations not included in the support set (Gao et al., 2019, 2021).
In line with recent literature, emphasize the importance of NOTA instances in the few-shot eval- uation setting (Gao al., 2019). By including NOTA as an additional relation type, we effectively sample N relations in each episode. Thereby, the NOTA relation consists the "undefined" rela- tion type and any relations not used for the training and test episodes (Sabo et al., 2021).
In the BERT with Entity Markers (BERTEM) approach, the entities are embedded with entity marker tokens at the respective start and end posi- tions the context (Baldini Soares et al., 2019). The concatenated representation of the entity start markers is used as the example representation. Query instances are classified based on similarity to the support instances.
4.2 Episode sampling
The sampling procedure accommodates overlap- ping relation types in the dedicated training and test sets. We consider two disjoint sets of relation types Tuain Ø for training and test episodes, respectively. We uniformly sample N x K support instances from the corresponding set relation types. This exposes models to the same number
For the BERT-Prompi approach, we express

---

the relation classification task a masked lan- guage modeling (MLM) problem (Gao al., 2021;
Domain
Model
In-domain
Proto-BERT
26.54/
31.96/43.87
BERT-Pair
26.77/25.00
34.78 34,
Schick and Schütze, 2021). Given a template T. each input is converted into Proompt = -x)) con- taining at least one [MASK] token. Similarly to the MLM task performed during pretraining of language model M, the masked token is predicted based on the surrounding context. To accommodate differences in model vocabularies, the relations mapped individual tokens included in the vocab- ulary: R + VM- Given input æ; the probability of predicting relation r is denoted as:
BERTEM
36.27 00,27/44..8 44.82
44.53
56.04
BERT-Prompt
53.27 / 67.30
74.16
88.00
HCRP
34.82/38.69
47.40 50.15
FewRel
Proto-BERT
19.46/2199
16.71/24.55
BERT-Pair
18.74 18.86
16.58 22.79
BERTEM
26,65 33.997
30.44 34.22
BERT-Prompt
25.01 3.36.5
33.26 35.70
HCRP
37.44
25.41 41.67
TACRED
Proto-BEKT
17,93 18.66
15.13 / 21.14
BERT-Pair
17.50 17.00
16.77 16.95
BERTEM
26.47 30.38
27.38 28.79
BERT BERT-Prompl
29.61 44.12
37.88 42.60
HCRP
27.23 32.85
41.91 44161
p(r |x) = p( [MASK] = Wi Tprrmpp) exp (w, h(MASK))
Table 3: Micro Fl Macro Fl for in-domain and out-of-
domain models evaluated on CORE.
Evev exp (we h(MASK))
5 Results and discussion
where h[MASK] is the hidden vector of [MASK] and to denotes the pre-softmax vector corresponding to E V (Gao et al., 2021; Han et al., 2022).
We evaluate the performance of the models using two evaluation metrics: micro Fl and macro F1. Micro Fl provides overall assessment of model performance by treating instances equally and ac- counting for the impact of an imbalanced NOTA ratio. In contrast, macro F1 focuses on individual class performance, treating each class equally and remaining unaffected by the NOTA ratio. We re- port micro Fl and macro F1 scores in percentage format in Tables 3, 4, and 5. The best results in each table are highlighted in bold, while the best performance measures for each domain under- lined. To ensure robust evaluation, the models were evaluated on sets of training and test relations for each dataset and N- N-Way K-Shot setting. The resulting evaluation metrics were then averaged.
The Hybrid Contrastive
(HCRP) approach is designed to learn from rela- tion types that are challenging to distinguish. It em- ploys a hybrid approach, creating prototypes and integrating additional data in the form of relation descriptions. HCRP incorporates a task-adaptive focal loss and utilizes contrastive learning tech- niques to enhance the discriminative power of the prototypes (Han al., 2021b).
4.4 Experimental setup
We evaluate RC models the few-shot domain adaptation setting for CORE, FewRel and TA- CRED. Our evaluation process consists of two steps. First, we evaluate the in-domain model per- formance. In the second step, we evaluate the out- of-domain performance on the respective test sets, We keep the number of training episodes consis- tent across datasets to prevent information dispar- ity across domains. For each dataset, we sampled 22,500 training episodes, which corresponds to ap- proximately 10 epochs for = 1 on CORE. In the case of FewRel, which has an artificial relation type distribution, we followed the approach described in (Gao et al., 2019). They uniformly sample sup- port and query instances from relation types, which also obscures the distribution of NOTA instances in the underlying data source. To address this, selected 50% NOTA rate (Gao et al., 2019) and applied the same sampling procedure to all models trained on FewRel, as in (Sabo et al., 2021).
5.1 Results CORE
The in-domain results presented in Table illus- trate the superior performance of the BERT-Prompt model across all few-shot settings, surpassing other approaches in terms of both micro Fl and macro F1 scores. Additionally, our findings reveal that the BERTEM model exhibits enhanced performance compared to the Proto-BERT and BERT-Pair mod- cls, as indicated by significant improvements in both miero F1 and macro Fl metries.
5.2 Results domain adaptation
Out-of-domain evaluation tests models' ability transfer information across different domains. Fol- lowing the approach in Gao et al. (2019), models learn to establish mappings between support and

---

query samples using the source domain data and corresponding labels (Tuain). In-domain evaluation is performed on feest, where even within this con- text, label spaces do not overlap. To accomplish this, the model is exposed to N x K support ex- amples from the target domain. Out-of-domain evaluation follows the same procedure, but with Iesst and support samples from a different domain.
Domain
Model
In-domain
Proto-BERT
20.48/25.40
19.86 20.75
BERT-Pair
21.66/21.32
/3.87/33.77
BERTEM
35.25/48.04
46.43 51.50
BERT-Prompt
56.02/
59.80 86.81
HCRP
36.33/51.07
52.09/ 63.11
CORE
Proto-BERT
26.19/31.64
29.82 / 39.52
BERT-Pair
18.99 22.98
BERTEM
40,27/52.44
54.03 65.45
BERT-Prompt
45.08
51.82 39.00
HCRP
51.56/57.55
61.24 67.96
Table 3 presents the performance of models trained FewRel and TACRED, evaluated on CORE. These results reveal substantial perfor- mance gaps between out-of-domain models and their in-domain counterparts, underscoring the chal- lenges of generalizing from RewRel and TACRED to CORE. Notably, BERT-Prompt trained on TA- CRED, achieves the highest performance among the evaluated models, though it still falls signifi- cantly short of the performance of the in-domain models. Interestingly, the performance differences among the other out-of-domain models are mini- mal, implying that all modeling approaches strug- gle to generalize to CORE. Furthermore, Proto- BERT, BERT-Pair, and BERT FM models trained on TACRED consistently demonstrate lower macro Fl values compared to the models trained on FewRel.
TACRED
Proto-BERT
20.28 23.53
20.95 27.62
BERT-Pair
16.44/16.5
13.78 17.67
BERTEM
34.19/43.95
38.88 /47.29
BERT BERT-Prompl
48.18
53.82 50.50
HCRP
49.15 61.99/ 63.33
Table 4: Micro Fl Macro Fl for in-domain and out-of- domain models evaluated on FewRel. The models are evaluated on the development set corresponding to the Wikipedia data.
Domain
Model
5-1
5-5
In-domain
Proto-BERT
28.50 14.68
25.33 20.38
BERT-Pas Pair
24.25 17.71
30.57
18.72
BERTEM
25.19/
60.39
BERT-Prompt
35.70 63.00
67.81 / 74.02
HCRP
29.58 26.43
$7.28 / 35.61
CORE
Proto-BERT
24.67
11.26 29. 20
BERT-Pair
14.47 17.48
11.44 21.89
BERTEM
16,32/ 38.65
18.80 48.39
To investigate the performance of in-domain models out-of-domain datasets, compared them with out-of-domain results for FewRel and TACRED, presented in Tables 4 and 5. Our findings show that the Proto-BERT, BERTEM and HCRP models trained on CORE consistently out- perform the in-domain models trained on FewRel, while the in-domain BERT-Prompt model remains the best performing model. The best performing out-of-domain models are HCRP trained on CORE and HCRP trained on TACRED.
BERT-Prompp
36,08 45.00
65.46 35.53
HCRP
34.04/ 37.54 54
47.20 44.76
FewRel
Proto-BERT
16,36 22.29
10.74 26.70
BERT-Pair
13.16 18.47
8.75 / 19.69
BERTEM
12,81 40.07
15.77 44.55
BERT-Prompt
18.71 / 49.10
38.21 45.67
HCRP
13.72 / 33.76
33.17 40.03
Table 5: Micro F1 Macro F1 for in-domain and out-of- domain models evaluated TACRED.
ing those trained on FewRel, struggle to adapt to TACRED when considering the micro F1 results. Wher When examining the macro Fl scores, we find that the BERT and BERT-
It is worth noting that both CORE and FewRel are based on examples extracted from Wikipedia, While lexical similarity can partially account for the performance differences between the models trained on CORE and TACRED in Table 4, models trained on FewRel fail to outperform the models trained on TACRED, as shown in Table 3.
models trained on FewRel demonstrate relatively better performance compared to the models trained CORE. How- ever, this improvement is reflected in the micro Fl scores.
To demonstrate our model's performance on domain-specific datasets, as opposed to general domain datasets, we included PubMed (Gao et al., 2019) and SCIERC (Luan et al., 2018) in our eval- uation. PubMed contains examples extracted from biomedical literature, featuring 10 unique relation types. Notably, both FewRel and PubMed are au- thored by Gao et al. (2019) and share certain char- acteristics, including artificially balanced relation
Regarding domain adaptation results on the TA- CRED dataset, we observe that the micro Fl per- formance of the BERT-Promme model trained the CORE dataset is comparable that of the in- domain model trained on TACRED. However, the macro F1 scores of the out-of-domain models are substantially lower. Furthermore, it is evident that Proto, BERT-Pair, and BERTEM FM models, includ-

---

Domain
Model
CORE
FewRel
TACRED
CORE
Proto BERT
22.25/21.38
28.48
CORE
11.75 %
20.72 72 %
BERT-Pair
17.79/ 19.31
/
FewRel
53.41 %
37.90 %
BERTEM
38.87
56.13/
62.50
62.50
BERT-Prompt
24.68 14.68/25..8 25.18
35.85/41.00
TACRED
44.85 %
15.12 %
HCRP
29.74/52.35
34.96
FewRel
Proto-BEKI
24.63/25.11
26.36 27.44
'able 8: Average performance decrease between best
BERT-Pair
17.39 18.91
24.53 /2,000
performing in-domain models (columns) and out-of-
BE
42,01 / 34.11
52.80 62.81
domain models (rows).
BERT-Prompt
28.27 41.556
HCRP
32.41/440.
37.99 36.50
TACRED
Proto-BEKT
19.37 20.26
19.64/21.61
performs BERT-Prompt models trained on FewRel and TACRED by 7.14% and 2.25%, respectively. In summary, findings suggest that out-of- domain models face challenges in adapting the CORE dataset. These models consistently exhibit lower performance, both terms of micro and macro Fl scores. To provide an overview of the performance differences, we illustrate the average performance gaps between the best performing in-domain model and the out-of-domain models across all evaluation metrics in Table 8. Notably, the models trained CORE show the best erage out-of-domain performance, indicated by lower average performance differences in compar- ison to in-domain models. Furthermore, our ex- periments show that models trained on CORE out- perform those trained on FewRel when evaluated on datascts with narrow domain focus, such as PubMed and SCIERC. Several factors contribute to these performance differences. First, the informa- tion richness in the training data allows the models to focus on contextual nuances, reducing their re- liance on superficial clues such as relation-specific verbs. Second, the broader entity coverage, includ- ing named entities, nouns, and pronouns, exposes models trained on CORE to more linguistically diverse instances for the different relation types. These observations align with the findings of Mao et al. (2022), who report significant out-of-domain performance improvements in low-resource set- tings when training on high-quality data. Consid- ering the benchmark evaluation, the prompt tuning approach emerges as the best-performing model.
BERT-Pair
15.61 16.05
1 16.28
BERTEM
32.12 36.99
38.72/ 40.32
BERT-Prompt
26.00
34.60 / 35.14
HCRP
32.00/45.13
37.14/
Table 6: Micro Fl Macro Fl for out-of-domain models evaluated on PubMed.
Domain CORF
Model
5-1
5-$
Proto BERT
17.71
19.37/ 19.83
BERT-Pair
1
20.42 20.03
BERTEM
22.20 24.94
29.28 33.69
BERT-Promp
41.31/ 28.13
47.88/ / 26.55
HCRP
12.34/22.31
18.41 /23.43
FewRel
Proto BERT
16.10/ 17.27
16.21 18.00
BERT-Pair
/6.51/
17.13 19.57
BERTEM
21.78/ 21.41/21.63
25.57
BERT-Prompt
32.06 23.95
23.95
36.15 / 23.18
HCRP
/
15.40 / 24.87
TACRED
Proto-BERT
15.31 16.93
16.54 17.06
BERT-Pair
15.46
16.04 15.77
BERTEM
17.74
17.31 / 20.97
BERT-Prompt
39.59 24.67
48.08 22.56
HCRP
21.03 23.25
33.66 24.98
Table 7: Micro Fl / Macro Fl for out-of-domain models evaluated on SCIERC.
distributions. SCIERC comprises examples from scientific literature within the Al domain, including 7 unique relation types. It provides a distinctive set of relation types and follows a different data ex- traction procedure, adding diversity to our bench- marked datasets. Both of these datasets have a limited number of relation types and do not include annotated NOTA category. Therefore, following the procedure outlined in Section 4, the inclusion of in-domain models is not feasible.
5.3 Error analysis
The oft-oi-domain evaluation on PubMed, pre- sented in Table 6, demonstrate that, on average, the BERT-EM model trained on CORE outperforms BERT-EM models trained on FewRel and HCRP trained TACRED by 2.40% and 9.61%, %, respec- tively. The out-of-domain evaluation results for models evaluated on SCIERC are provided in Table 7. The BERT- PERT-Prompt model trained on CORE out-
We conducted a qualitative analysis to identify error patterns in the domain adaptation setting. We used prediction examples from the BERT-Prompy mod- els trained different domains, which achieved the highest average performance in the previous section. From these models, we randomly sampled 100 erroncous and 100 correct predictions, focus- ing on the 5-Way 5-Shot evaluation setting.

---

E1 is listed on For models adapted to CORE (as shown in Table 3), we find that correct predictions often included (i) verbs closely related to the target relation, such as "sold", "bought", "merge" or (ii) combinations of entities that revealed the relation type, such as "it is listed on the New York Stock Exchange" or "the company was cited by the Food and Drug Ad- ministration". While there were no discernible pat- terns in the context, the domain-adapted models performed better on NOTA instances compared to instances belonging to the N relations of interest that did not exhibit patterns (i) and (ii). Selected examples are displayed below.
Conclusion
In this paper, we introduced CORE, a few-shot RC dataset that specifically focuses on company relations and business entities. To the best of our knowledge, there are no other datasets that specif- ically target company relations for few-shot RC, making CORE valuable addition to the existing benehmark resources. By extending the availability of datasets for few-shot domain adaptation settings, CORE contributes towards more robust and realis- tic RC evaluation results.
Despite the progress made in addressing chal- lenges in RC datasets, such as incorporating more realistic entity types and relation distributions, there is still need for further research and attention in adapting models to out-of-domain datasets us- ing diverse datasets and modeling approaches. Our findings illustrate that models trained on other RC datasets often rely on shortcuts that fail to general- ize to CORE. Moreover, models trained on CORE display superior out-of-domain performance. The information richness in the training data enables models focus contextual nuances, reducing their reliance superficial clues such as relation- specific verbs. This underscores the significance of high-quality datasets that include challenging entity types and require a deeper contextual under- standing. In conclusion, CORE serves as a valuable resource for the few-shot domain adaptation setting in RC. It highlights the importance of incorporating diverse and challenging datasets and encourages future research efforts to focus improving model adaptation to out-of-domain data. By addressing these challenges, we can advance the development of more effective and robust RC models.
Erroneous predictions for out-of-domain mod- els evaluated on CORE.
competitor of: In 2003, Toronto dominion bank held talks to merge Waterhouse
E1 with
E*TRADE, which would have created the sec- ond - largest discount broker in the united states E2 but the two sides could not come agreement control of the merged entity.
after Charles Schwab
product or service of: On July 4, 2011, Cope-Com E2 had converted their famed and legendary Amiga E1 game called battle squadron iOS devices titled battle squadron one and published the game through apple app store.
Correct predictions for out-of-domain models evaluated on CORE.
listed on: Sopheon
the alternative investment market of the London Stock Exchange E2
Limitations
acquired by: In 2008 the business was restructured with the creation of new Irish resident holding company, Charter International ple. E1 the com-
CORE instances solely extracted from company Wikipedia pages, which introduces limitation in terms of lexical similarity within the contextual information. To address this constraint, diversify- ing the dataset by incorporating company relations from various data sources, such as business news, would introduce linguistic variety and establish a more rigorous benchmark for evaluation.
E2 pany acquired by Colfax corporation an American company, in January 2012.
Our analysis revealed that models trained on CORE recognize typical patterns based verbs and entity types, well as more complex textual examples that require contextual understanding. Such examples include instances without verbs, instances involving multiple possible relations, or ambiguous entity types, as illustrated Figure
The datasct's practical applications are restricted to low-resource settings due to the limited number of instances and relations it contains. Expanding the dataset by including more instances and rela- tionships incurs substantial costs, as it necessitates human annotation efforts. Out of the 14,000 an-

---

notated candidate instances, only 30% contain any of the predefined relations. However, the current inclusion of the "undefined" category fails to accu- rately reflect the distribution of invalid candidate instances. This discrepancy primarily arises from the ubsence of entities within the context or entities that fall outside the scope of business entities. Another limitation lies in the dataset being mono- lingual, as it is currently limited to English. The transfer of annotations from the English Wikipedia corpus to other languages would require additional human supervision and allocation of resources.
Chengcheng Han, Zeqiu Fan, Dongxiang Zhang, Minghui Qiu, Ming Gao, and Aoying Zhou. 2021a. Meta-learning adversarial domain adaptation network for few-shot text classification. In Findings of the Association for Computational Linguisties: ACL- IJCNLP 2021, page 1664-1673, Online. Association for Computational Linguistics.
Jiale Han, Bo Cheng, and Wei Lu. 2021b. Exploring task difficulty for few-shot relation extraction. In Pro- ceedings of the 2021 Conference Empirical Meth- ods in Natural Language Processing, pages 2605- 2616, Online and Punta Cana, Dominican Republic. Association for Computational Linguisties.
Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun. 2022. Pur: Prompt tuning with rules for text classification. Al Open, 192...19..
References
Christoph Alt, Aleksandra Gabryszak, and Leonhard Hennig. 2020. TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task. In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics. pages 1558- 1569, Association for Computational Linguisties.
Xu Han, Hao Zhu, Pengfei Yu. Ziyun Wang. Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2018. FewRel: A Large-Scale Supervised Few-Shot Relation Clas- sification Dataset with Statc-of-the-Art Evaluation. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 4803-4809 4809, Brussels, Belgium. Association for Com- putational Linguistics.
Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling,
and Tom Kwiatkowski. 2019. Matching the Blanks: Distributional Similarity for Relation Learning. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 2895- 2905, Florence, Italy. Association for Computational Linguistics.
Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embed- dings, convolutional neural networks and incremental parsing. To appear.
Hadjer Khaldi, Farah Benamara, Camille Pradel, Gré- goire Sigel, and Nathalie Aussenac-Gilles. 2022. How's w business going worldwide? a multilingual annotated corpus for business relation extraction. In Proceedings of the 13th Conference on Language Resources and Evaluation, Marseille.
Elisa Bassignana and Barbara Plank. 2022. CrossRE: A cross cooss-domain dataset for relation extraction. In Find- ings the Association for Computational Linguistics: EMNLP 2022, pages 3592-3604, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Jacob Devlin, Ming-We Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. Proceedings of the 2019 Conference of the North American Chapter of the Association for Compula- tional Linguistics (NAACL), page 4171-4186.
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N. Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick van Kleef, Sören Auer, and Christian Bizer. 2015. Dbpedia- large-scale, multilingual knowledge base extracted from wikipedia. Semantic Web, 6(2):167-195.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Mak- ing Pre-trained Language Models Better Few-shot Learners. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume 1: Long Papers), pages 3816 3816-3830, Online. Association for Computa- tional Linguistics.
Yang Liu, Kang Liu, Liheng Xu, and Jun Zhao. 2014. Exploring fine-grained entity type constraints for dis- tantly supervised relation extraction. In Proceed- ings of COLING 2014, the 25th International Con- ference on Computational Linguistics: Technical Pa- pers, pages 2107-2116, Dublin, Ireland. Dublin City University and Association for Computational Lin- guistics.
Tianyu Gao. Xu Han, Hao Zhu, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou, 2019. FewRel 2.0: Towards More Challenging Few-Shot Relation Clas- sification. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP) pages 6250-6255, Hong Kong, China, Association for Com- putational Linguistics.
Yinhan Liu, Myle Oit, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretrain- ing Approach. arxiv:1907.11692 (cs). ArXiv: 1969.
Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-task identification of entities,

---

relations, and coreference for scientific knowledge graph construction. In Proceedings of the 2018 Con- ference Empirical Methods in Natural Language Processing, pages 3219-3232, Brussels, Belgium. Association for Computational Linguistics.
In Proceedings of the 2017 Conference on Empiri- cal Methods in Natural Language Processing, pages 35-45, Copenhagen, Denmark. Association for Com- putational Linguistics.
Li Zhenzhen, Yuyang Zhang. Jian-Yun Nie, and Dong- sheng Li. 2022. Improving few-shot relation classi- fication by prototypical representation learning with definition text. In Findings of the Association for Computational Linguistics: NAACL 2022,,022, pages 454- 464, Seattle, United States. Association for Computa- tional Linguistics.
Shengfei Lyu and Huanhuan Chen. 2021. Relation clas- sification with entity type restriction. In Findings of the Association for Computational Linguistics: ACL-
IJCNLP 2021, page 390-395, Online. Association for Computational Linguistics.
Yuning Mao, Ming Zhong, and Jiawei Han. 2022. Cite- Sum: Citation text-guided scientific extreme sum- marization and domain adaptation with limited su- pervision. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, pages 10922 10935, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
A Appendix
A.1 Relations
In the following, we provide textual descriptions of the relations included in the dataset. The rela- tions and their descriptions were predefined by the authors.
Subhabrata Mukherjee, Xiaodong Liu, Guoqing Zheng, Saghar Hosseini, Hao Cheng, Greg Yang, Christo- pher Meek, Ahmed Hassan Awadallah, and Jianfeng Gao. 2021. Clues: Few-shot learning evaluation in natural language understanding. 35th Conference on Neural Information Processing Systems (NeurIPS 2021) Track on Datasets and Benchmarks.
Acquired by: e2 purchases controlling stake el. The relation is directed. The inverted relation is best described by the same relation type.
Brand of: c2 offers products or services of el (Brand). The relation is directed. The inverted relation is best described by the same relation type.
Ofer Sabo, Yanai Elazar, Yoav Goldberg, and Ido Da- gan. 2021. Revisiting Few-shot Relation Classifica- tion: Evaluation Data and Classification Schemes. Transactions of the Association for Computational Linguistics, 9:691-706.
Client of: el uses (and presumably pays for) products or services offered by e2. The relation is directed. The inverted relation is best described by "Supplier of".
Timo Schick and Hinrich Schütze. 2021. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings the 16th Conference of the European Chapter of the As- sociation for Computational Linguistics: Main Vol- ume, page 255-269, Online. Association for Compu- tational I. Linguistics.
Collaboration: el and e2 collaborate in (parts of their) business activities. The relation is undi- rected.
Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical Networks for Few-shot Learning. Ad- vances in Neural Information Processing Systems, page 11.
Competitor of: el competes for resources with e2. The relation is undirected.
Merged with: el and e2 merged (parts of) their business activitics. The relation is undirected.
Hui Wu and Xiaodong Shi. 2022. Adversarial soft prompt tuning for cross-domain sentiment analysis. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 2438-2447, Dublin, Ireland. Association for Computational Linguistics.
Product or service of: el is offered for commer- cial distribution by e2. The relation is directed. The inverted relation is best described by the same relation type.
Jiawen Zhang, Jiaqi Zhu, Yi Yang. Wandong Shi, Cong- cong Zhang, and Hongan Wang. 2021. Knowledge- enhanced domain adaptation in few-shot relation clas- sification. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, KDD '21, page 2183-2191, New York, NY, USA. Association for Computing Machinery.
Regulated by: e2 regulates (parts of) the busi- ness activity of el. The relation is directed. The inverted relation is best described by the same relation type.
Shareholder of: el owns shares in c2. The re- lation is directed. The inverted relation is best described by the same relation type.
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-awes Attention and Supervised Data Improve Slot Filling.

---

owned by: After they returned to the Subsidiary of: e2 legally owns el. The relation is directed. The inverted relation is best described by "Parent of
Parameter
Values
Encoder
BERT-Base
Context length
{256, 512)
Training episodes
22,500
Traded on: Shares of el are listed on e2 (Stock exchange). The relation is directed. The inverted relation is best described by "lists".
Learning Rate
2e-5
Batch Size
64
Optimizer
Adam
undefined: el and c2 are business entities. The relation between the entities cannot be described by any other defined relation. The relation can be directed undirected, however directed relations are not annotated as such.
Table 9: Overview training parameters
A.3 Model training
The model parameters are shown in Table 9. We adjust the context length according to the dataset and model. All models were trained on NVIDIA RTX A6000 48 GB GPU. Due to memory limita- tions when training the model variants, train BERT-Pair models with half-precision.
A.2 Dataset Analysis
In contrast to the CORE relation "subsidiary of" and "acquired by". the FewRel relation P355 (has subsidiary) encompasses non- non-business entities and offers a broader interpretation of the term "sub- sidiary" beyond the corporate parent or subsidiary relationship.
A.4 Prompt tuning
We apply heuristic approach in creating the tem- plate T and vocabulary mapping R VM used in the BERT-Prompt model. The template was selected by the authors and was applied consis- tently across datasets, as follows: "[TEXT]. [E1] is [MASK] by [E2].". Similarly, the tokens mapped to the target relations for each benchmarked dataset are selected by the authors. Prompt engineering, including automated search procedures for optimal templates and vocabulary mappings, remains active research area. Acknowledging that heuristic approaches are suboptimal, we encourage further research to improve the prompt tuming approach. Our template and mapping files are available at https: / trub.hu..ccc com wwanm//boorceer//Cre
has subsidiary: The 2017-18 UMass E1
Minutewomen basketball team represents the
University of Massachusetts Amherst E2
during the 2017-18 college baskethall season.
Davidson County is also served by
has
subsidiary:
Davidson County Community College E2 comprehensive community col- that is a member school of the North Carolina Community College System E1
lege
The FewRel relation P127 (owned by) exhibits partial overlap with the CORE relations "subsidiary of and "acquired by". However, it introduces a different perspective on ownership, encompassing range of interpretations that extend beyond con- ventional notions of mere possession, including the notion of being constituent part of an entity, physical location, and even physical control.
A.5 Annotation Interface
The annotation platform provided clear instructions and mandatory tutorials to guide the annotators throughout the annotation process. These instruc- tions included detailed explanations of the anno- tation task and interface, which was designed be intuitive and efficient. Each component of the interface was explained in detail to ensure that an- notators were able to use it effectively. The follow- ing screenshots illustrate the interface provided to the annotators. Figures 5 and 6 displays the rela- tion annotation task, which included entity types, relation directions, and relevant context. Figure 7 shows the entity position annotation step, which verified the relation annotation completed in the previous step. Both of these steps were completed by different annotators.
owned by: The school is located in the city of Mannheim, Baden Württemberg E2 in Ger-
many at Mannheim Palace one of the largest baroque castles in Europe.
Polanski E2 residence on Cielo Drive E1 Patricia Krenwinkel, Susan Atkins and Charles "Tex" Watson entered the home.

---

CoRE Company Relation Extraction
Tutaril
Completed unu
Instructions
Your task is idundffy and desermine the contectual relation beoween teo entiiso, ie. bivily and "iiilyy at Ate entiy is defined a def catagary ind uding tel slon,, brands, insiltat stock unaheragu insustrio, Thrrafar wiien bed unipper voluacted from the Wilpedd of enlly comnasonnding different enityy
Logout
which der
The task comsics of the following 5 steps.
iapul care cally
2. identily "Ently 2" the tast and varily cornect the name in the cornspording input box.
3. Selvct vrille fox "Busity and Enily 2",
the drapdown ist.
that aui information ceteriin bulnwerr
Guamedaa
Relation Annotation
Entity Pocition Annotation
Your idartify and determina contastuel velation batowen two entriie, in. "tntiyy I" and "tntiyy
innc varily the annocated and idemily "Enity 1" contest.
Tutorial
Figure 4: Annotation Interface: Instructions.
You cas intert the nencin' by bycrtiishingg the page or cliing fomen Refreth
Microsoft
Wel Eapo
Inatrudiatt
Exttyy croooott
Entle
Dangerinncul
Invert
Relation type ?
Seloct the rolasion betwoon the tw
comporibv acenttin syete es although has lost the mugoriy evera
other co to mer and encerpiise software for
realit dlcuc computing
Stove replaced Gates atbuaag..
services
unfolded axquiring Dangeer computer pr eduction market for the firs time in June 2012 with launch of Micicsoft Surfice lire tabiet computers, liwe forming crouth Mobile through
entaring the pensonal
Figure S: Annotation Interface: Relation type annotation.

---

You can restart tutorial by refreshing the page or clicking this button:
Refresh
Microsoft
Wikipedia Page
Microsoft
Entity type:
Company
Entity 1:
Danger Inc.
Entity type:
Company
Entity
Relation type:
acquired by
Invert:
Danger Inc.
acquired
Microsoft
As of 2015, Microsoft is market dominant in the IBM PC compatible operating system market and the office software market, although it has lost the majority the overall operating system market to Android.
The company also produces a range of other consumer and enterprise software for desktops, laptops, tabs, gadgets, and servers, including Internet search (with Bing), the digital services market (through MSN), mixed reality (HoloLens), cloud computing (Azure), and software development (Visual Studio).
Steve Ballmer replaced Gates as CEO in 2000, and later envisioned 'devices and services' strategy.
This unfolded with Microsoft acquiring Danger Inc. in 2008, entering the personal computer production market for first June 2012 with the launch of the Microsoft
Surface line of tablet computers, and later forming Microsoft Mobile through the
Figure 6: Annotation Interface: Relation type annotation.

---

You can restart the tutorial by refreshing the page clicking this button:
Refresh
Entity 1: Deutsche Bank
Entity type: Company
Entity 2: DAX
Entity type: Stock Exchange
Relation type: traded on Invert: 0
Exclude Example:
E1 in context:
As the largest German banking institution, DB component of the DAX stock market index.
Context
The snippet only contains sentences selected by the annotator. The colors help you identify the entities the text.
000000
Next
Figure 7: Annotation Interface; Position annotation and validation.


Reviews:
[{"review_id": "jqOsRNeRo4", "reviewer": "Reviewer_qrfS", "summary": "", "questions": "Although it presented the differences between CORE and FewRel, maybe an explanation is needed why this work was started while FewRel already existed. There is overlapping, maybe a combination of the two datasets would be better?\nThe in-domain results presented in shows that the proposed model surpassed other approaches in few-shot settings while out-of-domain evaluation reveals models trained on CORE outperform the in-domain models trained on FewRel.\nThe work is interesting and the dataset could be a valuable resource for few-shot domain-adaptation in relation classification.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our manuscript. We appreciate your feedback and suggestions for improving the manuscript.\n\n> Although it presented the differences between CORE and FewRel, maybe an explanation is needed why this work was started while FewRel already existed. There is overlapping, maybe a combination of the two datasets would be better?\n\nWe acknowledge an overlap between CORE and FewRel and aim to clarify the key differentiating factors that uniquely positions CORE among other relation classification datasets:\n\n- A key distinction lies in **CORE's focus on challenging entity types that require contextual understanding** and cannot be derived based on entity type information. As evidenced by our experiments, this enhances the out-of-domain performance of models trained on CORE. We believe that this **uniquely positions the scope of CORE among other relation classification datasets**. \n- The datasets fundamentally differ in their data collection and annotation methodologies, even though both datasets are based on Wikipedia. CORE is human-annotated and therefore contains **subject-object-relation triples that are not present in existing knowledge bases**. In contrast, FewRel is distantly-annotated employing external knowledge bases with subsequent human-validation.\n- We further emphasize the relation type distribution reflecting the original data source and presence of an **annotated NOTA (none-of-the-above) category in CORE**, which is essential for the application of the dataset in end-to-end relation extraction models. \n\nThis results in a dataset that is substantially different in terms of content and scope. Our experimental results showcase that domain-adaptation models trained on CORE perform better in comparison to other benchmarked datasets. For applications beyond the low-resource setting evaluated in this work, we acknowledge that training on multiple datasets can be beneficial, as suggested by the reviewer.\n\n> The work is interesting and the dataset could be a valuable resource for few-shot domain-adaptation in relation classification.\n\nWe appreciate your recognition of the significance of our research and the value of the CORE dataset in the field of relation extraction. We firmly believe that the CORE dataset brings a unique contribution to this domain.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Although it presented the differences between CORE and FewRel, maybe an explanation is needed why this work was started while FewRel already existed. There is overlapping, maybe a combination of the two datasets would be better?\nThe in-domain results presented in shows that the proposed model surpassed other approaches in few-shot settings while out-of-domain evaluation reveals models trained on CORE outperform the in-domain models trained on FewRel.\nThe work is interesting and the dataset could be a valuable resource for few-shot domain-adaptation in relation classification.", "reasons_to_reject": "Although it presented the differences between CORE and FewRel, maybe an explanation is needed why this work was started while FewRel already existed. There is overlapping, maybe a combination of the two datasets would be better?\nThe in-domain results presented in shows that the proposed model surpassed other approaches in few-shot settings while out-of-domain evaluation reveals models trained on CORE outperform the in-domain models trained on FewRel.\nThe work is interesting and the dataset could be a valuable resource for few-shot domain-adaptation in relation classification.", "questions_to_authors": "Although it presented the differences between CORE and FewRel, maybe an explanation is needed why this work was started while FewRel already existed. There is overlapping, maybe a combination of the two datasets would be better?\nThe in-domain results presented in shows that the proposed model surpassed other approaches in few-shot settings while out-of-domain evaluation reveals models trained on CORE outperform the in-domain models trained on FewRel.\nThe work is interesting and the dataset could be a valuable resource for few-shot domain-adaptation in relation classification.", "paper_topic_and_main_contributions": "In this paper, a new dataset for few-shot relation extraction, specifically company relations, is presented, which includes 4,708 instances and 12 relation types. The authors claim that models trained on different domains struggle to adapt to CORE, however, models trained on CORE show improved out-of-domain performance.\nThe dataset was built by gathering information from Wikipedia pages, using a state-of-the-art named entity recognition. Relations were annotated with a reasonable annotator agreement.\nAlthough it presented the differences between CORE and FewRel, maybe an explanation is needed why this work was started while FewRel already existed. There is overlapping, maybe a combination of the two datasets would be better?\nThe in-domain results presented in shows that the proposed model surpassed other approaches in few-shot settings while out-of-domain evaluation reveals models trained on CORE outperform the in-domain models trained on FewRel.\nThe work is interesting and the dataset could be a valuable resource for few-shot domain-adaptation in relation classification.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fH43BCH3jO", "reviewer": "Reviewer_J5cv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive evaluation of our paper and agree with your assessment that diverse and challenging datasets such as CORE are required for robust domain adaptation. We incorporated your feedback and suggestions to strengthen the coherence and significance of our work.\n\n> The scope of the paper seems a bit narrow. The dataset focuses only on company relations, but makes strong claims about generalization of RC models. Perhaps that claim needs to be supported with datasets on a few other domains.\n\nIn response to your suggestion, we **extended the benchmark datasets to include PubMed** (Gao et al. 2019), which contains **relations extracted from biomedical literature**. In the table below, we compare the out-of-domain performance of models trained on CORE, FewRel and TACRED on the PubMed dataset. We have included the evaluation results (Micro F1 / Macro-F1) for the best-performing models, namely BERT-Prompt and HCRP (Han et al., 2021). The results in the table below are averaged across three random seeds. \n\n| Domain | Model       | 5-1           | 5-5           |\n| ------ | ----------- | ------------- | ------------- |\n| CORE   | BERT-Prompt | 24.68 / 25.18 | 35.85 / 41.09 |\n|        | HCRP        | 29.74 / **52.35** | 34.96 / **55.66** |\n| FewRel | BERT-Prompt | 25.88 / 41.81 | 28.27 / 41.56 |\n|        | HCRP        | **32.41** / 40.21 | **37.99** / 36.50 |\n| TACRED | BERT-Prompt | 29.36 / 36.00 | 34.60 / 35.14 |\n|        | HCRP        | 32.00 / 45.13 | 37.14 / 48.63 |\n\nWe summarize the key findings in the bullet points below:\n- The HCRP model displays the best average out-of-domain performance for all datasets.\n- In the following we evaluate the average performance across the micro-F1 and macro-F1 metrics. In the 5-1 setting, the HCRP model trained on **CORE outperforms HCRP models trained on FewRel and TACRED by 4.75% and 2.48%**, respectively.\n- In the 5-5 setting, the HCRP model trained on **CORE outperforms HCRP models trained on FewRel and TACRED by 8.06% and 2.42%**, respectively.\n\nOur manual error analysis indicates that models trained on FewRel and TACRED rely on shortcuts, such as inferring the relation type based on entity types or verbs closely related to the target relation (\"sold\" or \"bought\" for acquired by relation, \"merge\" for merged with relation). Hence, we conclude that the **narrow domain focus of CORE contributes to creating a challenging benchmark for few-shot domain-adaptation models** and is a valuable contribution to the field.\n\nThank you for highlighting your concern about the assertion of strong generalization claims.\nWe revised the manuscript to clarify that the out-of-domain generalization concerns the dataset included in our analysis. We include the adjustments along with the full evaluation results on the PubMed dataset in the camera-ready version of the paper.\n\n> The dataset is limited to company relations from Wikipedia pages. This may limit the diversity of the dataset. It may also not be representative of real-world data.\n\nWe agree that including various data sources increases linguistic variety and provides a more realistic evaluation benchmark, as mentioned in line 629. However, it's important to emphasize that **expanding the dataset while maintaining the high-quality data extraction standards we uphold comes with significant costs**.\n\nWe are committed to publish our dataset under open-source licenses that enable other researchers to modify, copy and publish results based on our work. Even though it comes with limitations, it's worth noting that the **data extracted from Wikipedia encompasses a broad spectrum of business-related textual content curated from various primary sources** (https://en.wikipedia.org/wiki/Wikipedia:Reliable_sources).\n\nFurthermore, we adhere to experimental setups that reflect real-world data (Sabo et al., 2021), and include various entity types, such as named entities, nouns and pronouns. Additionally, the relation distribution in CORE is not artificially balanced.\n\n> The few-shot RC models considered in the paper are not state-of-the-art models (e.g. https://aclanthology.org/2022.coling-1.205.pdf, https://ieeexplore.ieee.org/abstract/document/10032649/). How does the performance compare to relation extraction/generation models in few-shot settings.\n\nThank you for this comment, we **included the HCRP model (Han et al. 2021) in our benchmark**. The updated results are presented below, reflecting both in-domain and out-of-domain performance across different datasets. **These outcomes support our earlier conclusions**: models trained on other domains struggle to adapt to CORE, while models trained on CORE showcase superior out-of-domain performance. This conclusion holds even when considering only the HCRP model.\n\nTo facilitate a concise model comparison, we have included the best-performing models alongside HCRP in the tables. All evaluation results are averaged for ten sets of training and test relations for each dataset. The table below contains the evaluation results (Micro F1 / Macro-F1) for models evaluated on **CORE**:\n\n| Domain    | Model       | 5-1           | 5-5           |\n| --------- | ----------- | ------------- | ------------- |\n| In-domain | BERT-Prompt | **53.27** / **67.30** | **74.16** / **88.07** |\n|           | HCRP        | 34.82 / 38.69 | 47.40 / 50.15 |\n| FewRel    | BERT-Prompt | 25.01 / 36.35 | 33.26 / 35.70 |\n|           | HCRP        | 23.54 / 37.49 | 25.41 / 41.67 |\n| TACRED    | BERT-Prompt | 29.61 / 44.12 | 37.88 / 42.60 |\n|           | HCRP        | 27.23 / 32.85 | 41.91 / 41.61 |\n\nThe table below contains the evaluation results (Micro F1 / Macro-F1) for models evaluated on **FewRel**:\n\n| Domain    | Model       | 5-1           | 5-5           |\n| --------- | ----------- | ------------- | ------------- |\n| In-domain | BERT-Prompt | **56.02** / **71.71** | 59.80 / **86.81** |\n|           | HCRP        | 36.33 / 51.07 | 52.09 / 63.11 |\n| CORE      | BERT-EM     | 45.08 / 48.01 | 51.82 / 39.05 |\n|           | HCRP        | 51.56 / 57.55 | 61.24 / 67.96 |\n| TACRED    | BERT-Prompt | 48.18 / 54.80 | 53.82 / 50.50 |\n|           | HCRP        | 49.15 / 53.89 | **61.99** / 63.33 |\n\nThe table below contains the evaluation results (Micro F1 / Macro-F1) for models evaluated on **TACRED**:\n\n| Domain    | Model       | 5-1           | 5-5           |\n| --------- | ----------- | ------------- | ------------- |\n| In-domain | BERT-Prompt | 35.70 / **63.00** | **67.81** / **74.02** |\n|           | HCRP        | 29.58 / 26.43 | 57.28 / 35.61 |\n| CORE      | BERT-Prompt | **36.08** / 45.07 | 65.46 / 35.53 |\n|           | HCRP        | 34.04 / 37.54 | 47.20 / 44.76 |\n| FewRel    | BERT-Prompt | 18.71 / 49.10 | 38.21 / 45.67 |\n|           | HCRP        | 13.72 / 33.76 | 33.17 / 40.03 |\n\nWe summarize the key findings in the bullet points below:\n- The in-domain BERT-Prompt model consistently demonstrates superior performance over the in-domain HCRP models for all N-Way K-Shot settings.\n- Regarding the domain adaptation to CORE, the differences between BERT-Prompt and HCRP are not as pronounced, with HCRP frequently outperforming BERT-Prompt on individual metrics. However, models trained on CORE still substantially outperform out-of-domain models.\n- For the domain-adaptation to FewRel, the HCRP models trained on CORE and TACRED are the best performing models in their category. Notably, the HCRP model trained on CORE outperforms the HCRP model trained on TACRED in three out of the four evaluation metrics.\n- For the domain adaptation to TACRED, the HCRP models do not outperform the BERT-Prompt models.\n\nWe agree that the paper benefits from the addition of these results in the camera-ready version.\n\n> The dataset contains only a limited number of relations and examples, which may limit its usability in different scenarios.\n\nWe acknowledge the assessment that the contributions of this work and its practical applications are limited to low-resource settings, as mentioned in line 637. We believe that few-shot evaluation scenarios and domain-adaptation are valuable to the field of information extraction.\n\nWe carefully curated the set of relations in CORE. It's important to emphasize that **increasing the number of relation types may inadvertently dilute the inherent complexity of the relation classification task. The inclusion of new relation types can introduce entity type-related and linguistic clues that potentially simplify the task**. An apt example would be the \"location_headquarter\" relation type, necessitating specific entity types such as \"Company\" and \"Geolocation\". As previously mentioned, significant costs are associated with expanding the dataset, particularly when maintaining the high-quality data extraction procedure.\n\nReferences:\n- Tianyu Gao, Xu Han, Hao Zhu, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2019. FewRel 2.0: Towards More Challenging Few-Shot Relation Classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6250–6255, Hong Kong, China. Association for Computational Linguistics.\n- Ofer Sabo, Yanai Elazar, Yoav Goldberg, and Ido Dagan. 2021. Revisiting Few-shot Relation Classification: Evaluation Data and Classification Schemes. Transactions of the Association for Computational Linguistics, 9:691–706.\n- Jiale Han, Bo Cheng and Wei Lu. 2021. Exploring Task Difficulty for Few-Shot Relation Extraction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2605–2616, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Proposes a new dataset for few-shot relation classification in company domain. The dataset requires more challenging entity types and a deeper understanding of the context compared to existing few-shot relation classification datasets. \n2. Shows the dataset is challenging for out-of-domain generalization. \n3. Provides insights into how diverse and challenging datasets such as CORE are required for robust domain adaptation. Encourages future research to consider these factors when generating datasets to advance research on robust few-shot RC models. \n", "reasons_to_reject": "1. The scope of the paper seems a bit narrow. The dataset focuses only on company relations, but makes strong claims about generalization of RC models. Perhaps that claim needs to be supported with datasets on a few other domains. \n2. The dataset is limited to company relations from Wikipedia pages. This may limit the diversity of the dataset. It may also not be representative of real-world data. \n3. The dataset contains only a limited number of relations and examples, which may limit its usability in different scenarios. \n4. The few-shot RC models considered in the paper are not state-of-the-art models (e.g. https://aclanthology.org/2022.coling-1.205.pdf, https://ieeexplore.ieee.org/abstract/document/10032649/). How does the performance compare to relation extraction/generation models in few-shot settings. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a few-shot relation classification (RC) dataset, CORE, that focuses on company relations and business entities. The dataset includes 4708 instances of 12 relation types extracted from various company Wikipedia pages. The dataset is targeted for robust domain adaptation. The paper shows the existing few-shot RC models struggle to generalize well to the CORE dataset as they often rely on shortcuts. However, models trained on CORE dataset display better out-of-domain performance. The paper concludes the significance of high-quality datasets such as CORE to improve model adaptation to out-of-domain data. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xJTwWnH6Lc", "reviewer": "Reviewer_kwUM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback, and we appreciate your recognition of the value that CORE brings to the field of relation classification.\n\n> As mentioned by the authors in the limitations sections, the fact that the dataset is based solely on Wikipedia data limits the variety of styles used in different texts in the data.\n\nWe agree that including various data sources increases linguistic variety and provides a more realistic evaluation benchmark. It is important to emphasize that **expanding the dataset while maintaining the high-quality data extraction standards we uphold comes with significant costs**. Moreover, including samples from different data sources requires a careful evaluation to **minimize the introduction of linguistic clues** inherent to the data sources.\n\nWe are committed to publishing our dataset under open-source licenses that enable other researchers to modify, copy and share results based on our work. Even though it comes with limitations, it's worth noting that the data extracted from Wikipedia encompasses a broad spectrum of business-related textual content curated from various primary sources (https://en.wikipedia.org/wiki/Wikipedia:Reliable_sources).\n\n> Since the authors are presenting a dataset for the few-shot setting, a dataset with a larger number of distinct relation labels would have been more useful to evaluate models on a more diverse range of classes.\n\nWe acknowledge the assessment that contributions of this work and its practical applications are limited to low-resource settings, as mentioned in line 637. We carefully curated the set of relations considered in CORE to obtain a dataset that contains relations that are diverse, challenging and relevant in the business domain. Therefore, **increasing the number of relation types may inadvertently dilute the inherent complexity of the relation classification task**. The inclusion of new relation types can introduce entity type-related and linguistic clues that potentially simplify the task. An apt example would be the \"location_headquarter\" relation type, necessitating specific entity types such as \"Company\" and \"Geolocation\". As previously mentioned, significant costs are associated with expanding the dataset, particularly when maintaining the high-quality data extraction procedure.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The introduction of a new dataset is an important contribution to the field. In particular, a dataset containing containing the NOTA relation label is particularly valuable. Additionally, the fact that relations in the chosen domain would be hard to distinguish solely basing the reasoning on the entity types is a valuable characteristic of a dataset for RC.", "reasons_to_reject": "As mentioned by the authors in the limitations sections, the fact that the dataset is based solely on Wikipedia data limits the variety of styles used in different texts in the data.\nSince the authors are presenting a dataset for the few-shot setting, a dataset with a larger number of distinct relation labels would have been more useful to evaluate models on a more diverse range of classes.\n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces CORE, a novel and high-quality dataset for few-shot relation classification (RC), and demonstrates the challenges faced by state-of-the-art models when dealing with it, while highlighting the remarkable performance of models trained on the dataset itself. The study also highlights the critical role of high-quality datasets in enabling robust domain adaptation for few-shot RC. The paper provides a benchmark evaluation of several models and approaches for few-shot RC on CORE, and conducts a qualitative analysis to identify error patterns in the domain adaptation setting.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a few-shot relation classification dataset called CORE, focusing on company relations and business entities. The paper shows that the dataset is challenging for robust domain adaption, but the scope of the company scenario is a bit narrow.
Decision: Accept-Main