SlowGuess/ACL_2020_OCR huggingface dataset:
Method	Partial Match	Exact Match
ACE04	NYT10-HRL	NYT11-HRL	Wiki-KBP
Prec.	Rec.	F1	Prec.	Rec.	F1	Prec.	Rec.	F1	Prec.	Rec.	F1
Chan and Roth (2011)	42.9	38.9	40.8	-	-	-	-	-	-	-	-	-
MultiR (Hoffmann et al., 2011)	-	-	-	-	-	-	32.8	30.6	31.7	30.1	53.0	38.0
DS-Joint (Li and Ji, 2014)	64.7	38.5	48.3	-	-	-	-	-	-	-	-	-
FCM (Gormley et al., 2015)	-	-	-	-	-	-	43.2	29.4	35.0	-	-	-
SPTree (Miwa and Bansal, 2016)	-	-	-	49.2	55.7	52.2	52.2	54.1	53.1	-	-	-
CoType (Ren et al., 2017)	-	-	-	-	-	-	48.6	38.6	43.0	31.1	53.7	38.8
Katiyar and Cardie (2017)	50.2	48.8	49.3	-	-	-	-	-	-	-	-	-
NovelTagging (Zheng et al., 2017)	-	-	-	59.3	38.1	46.4	46.9	48.9	47.9	53.6	30.3	38.7
ReHession (Liu et al., 2017)	-	-	-	-	-	-	-	-	-	36.7	49.3	42.1
CopyR (Zeng et al., 2018)	-	-	-	56.9	45.2	50.4	34.7	53.4	42.1	-	-	-
HRL (Takanobu et al., 2019)	-	-	-	71.4	58.6	64.4	53.8	53.8	53.8	-	-	-
PA-LSTM-CRF (Dai et al., 2019)	-	-	-	-	-	-	-	-	-	51.1	39.3	44.4
CASREL	57.2	47.6	52.0	77.7	68.8	73.0	50.1	58.4	53.9	49.8	42.7	45.9
Table 5: Relational triple extraction results of different methods under Partial Match and Exact Match metrics.
et al., 2019; Li et al., 2019; Fu et al., 2019); (2) The stricter but less popular one is Exact Match adopted by Dai et al. (2019), where an extracted relational triple (subject, relation, object) is regarded as correct only if the relation and the heads and tails of both subject and object are all correct.
Since some works like (Zeng et al., 2018) can't handle multi-word entities and can only be evaluated under the Partial Match metric and some works like (Dai et al., 2019) are not open-source, it's hard to use a unified metric to compare our model with existing models. To properly compare our model with various baselines, we adopt the Partial Match metric for ACE04, NYT10-HRL and NYT11-HRL, and adopt the Exact Match metric for Wiki-KBP.
ACE04 We follow the same 5-fold cross-validation setting as adopted in previous works (Li and Ji, 2014; Miwa and Bansal, 2016; Li et al., 2019) and use the code released by (Miwa and Bansal, 2016) to preprocess the raw XML-style data for fair comparison. Eventually, it has 2,171 valid sentences in total and each sentence contains at least one relational triple.
NYT10-HRL & NYT11-HRL NYT corpus has two versions: (1) the original version of which both the training set and test set are produced via distant supervision by Riedel et al. (2010) and (2) a smaller version with fewer relation types, where the training set is produced by distant supervision while the test set is manually annotated by Hoffmann et al. (2011). Here we denote the original one and the smaller one as NYT10 and NYT11, respectively.
These two versions have been selectively adopted and preprocessed in many different ways among various previous works, which may be confusing sometimes and lead to incomparable results if not specifying the version. To fairly compare these models, HRL (Takanobu et al., 2019) adopted a unified preprocessing for both NYT10 and NYT11, and provided a comprehensive comparison with previous works using the same datasets. Here we denote the preprocessed two versions as NYT10-HRL and NYT11-HRL.
For fair comparison, we use the preprocessed datasets released by Takanobu et al. (2019), where NYT10-HRL contains 70,339 sentences for training and 4,006 sentences for test and NYT11-HRL contains 62,648 sentences for training and 369 sentences for test. We also create a validation set by randomly sampling  $0.5\%$  data from the training set for each dataset as in (Takanobu et al., 2019).
Wiki-KBP We use the same version as Dai et al. (2019) adopted, where the training set is from (Liu et al., 2017) and the test set is from (Ren et al., 2017). It has 79,934 sentences for training and 289 sentences for test. We also create a validation set by randomly sampling  $10\%$  data from the test set as Dai et al. (2019) suggested.
Dataset Study As stated beforehand, these datasets are not suitable for testing the overlapping problem. To further explain this argument, we analyze the datasets in detail and the statistics are shown in Table 6. We find that the test data in these datasets suffer little from the so-called overlapping triple problem since the sentences contain few overlapping triples. Even worse, we also find that the
$^{6}$ https://github.com/tticoin/LSTM-ER
Category	ACE04	NYT10-HRL	NYT11-HRL	Wiki-KBP
ALL	Train	Test	Train	Test	Train	Test
Normal	1604	59396	2963	53395	368	57020	265
EPO	8	5376	715	2100	0	3217	4
SEO	561	8772	742	7365	1	21238	20
ALL	2171	70339	4006	62648	369	79934	289
Table 6: Statistics of datasets. Note that a sentence can belong to both EPO class and SEO class.
annotated triples are not always the ground truth, which may affect the evaluation of our model. Take a real instance from the test set of NYT11-HRL for example, for the sentence "Mr. Gates, who arrived in Paris on Tuesday, was the first American defense secretary to visit France in nearly a decade . . .", the annotated triple set only contains one relational triple:
$\{(Paris, / location/administrative_division/country, France)\}$ . However, the output of our model has three triples:  $\{(Paris, / location/administrative_division/country, France), (France, / location/country/administrative_divisions, Paris), (France, / location/location/contains, Paris)\}$ . The last two triples should have been annotated in the sentence but were omitted, which will significantly affect the values of both precision and recall when quantifying the performance of our model.
This observation demonstrates that our CASREL model could extract more relational triples than the manually annotated ones in some cases due to the imperfect annotation. For this reason, the performance on such dataset like NYT11-HRL can only partially reflect the potential of the proposed model and probably underestimate the real value. Nonetheless, the CASREL model still achieves a competitive performance, showing the effectiveness of the proposed novel cascade binary tagging framework in relational triple extraction.
We also note that there is a significant gap (from 53.9 to 89.6 in terms of F1-score) between the performance on NYT11-HRL that preprocessed by Takanobu et al. (2019) and on NYT11-CopyR<sup>7</sup> that preprocessed by Zeng et al. (2018). Though both of the above two versions are adapted from the original NYT11 dataset (Hoffmann et al., 2011), there are two key differences in the NYT11-CopyR version as Dai et al. (2019) pointed out. First, instead of using the manually annotated test set, Zeng et al. (2018) randomly select 5000 sentences from the training data as the test data. The reason
is that the manually annotated data contains few overlapping triples and thus not suitable for testing the overlapping triple problem; Second, Zeng et al. (2018) only annotated the last word of an entity in both training and test sets because their model can not handle multi-word entities. Hence, any entity in their dataset is taken as a single-word entity, leading to that the Partial Match and Exact Match evaluation metrics make no difference. Moreover, such setting makes it much easier for our CASREL model to detect the span of an entity since the start and end positions are actually the same. We attribute the significant gap to these different settings between the above two versions. Noticeably, multi-word entities are common in real-world scenarios, so evaluating on a more proper dataset like NYT10-HRL (which also contains overlapping triples in test set) may better reveal the model's real value in relational triple extraction than on the ad-hoc one.
<sup>7</sup>We denote the version preprocessed by Zeng et al. (2018) as NYT11-CopyR for disambiguation, which is also referred to as "NYT" in the main experiment.