SlowGuess/NAACL_2022_OCR huggingface dataset:
We first observe that GenIE is always largely better than the baseline. The baseline suffers from the same difficulty as previous works; classifying among a large set of relations is hard with large imbalances. GenIE and the baseline have similar absolute decrease in performance when the number of relations increases, corresponding to a more considerable relative decrease for the baseline. More concretely, GenIE's micro F1-score goes from  $70.36\%$  for the top 100 relations, to  $68.82\%$  and  $68.93\%$  for the top 400 and 857 relation setups, respectively. This translates to a relative decrease of  $2\%$  only in the first step. For the baseline, the absolute score of  $47.67\%$  first falls to  $44.25\%$  and subsequently to  $42.5\%$  as the number of relations grows. This in turn, is an overall relative drop of almost  $11\%$ .
Notably, when looking at precision and recall separately (cf. Table 6 in Appendix G), GenIE
has a slight proportional decrease of 1-2 absolute points, both in precision and recall, which reflects the increased difficulty of the task due to larger number of relations. The baseline exhibits a similar drop in precision, but a much more significant drop in the recall of almost 10 absolute or 16 point relative. This suggests that the baseline simply ignores most of the relations with lower occurrence counts, which is consistent with the results in Sec. 2, and the hypothesis that the relation classification task is a bottleneck for effectively scaling the baseline system to a large number of relations.
We already have to deploy several techniques to help the baseline better deal with these issues (see Sec. 4.3), while GenIE, thanks to its generative autoregressive formulation, can effectively scale and manage the inherent imbalances of the task much more naturally.
Wiki-NRE	Geo-NRE
Precision	Recall	F1	Precision	Recall	F1
Pipeline baselines
AIDA + MinIE	36.72	48.56	41.82	35.74	39.01	37.30
NeuralEL + MinIE	35.11	39.67	37.25	36.44	38.11	37.26
AIDA + ClauseIE	36.17	47.28	40.99	35.31	39.51	37.29
NerualEL + ClauseIE	34.45	37.86	36.07	35.63	37.91	36.73
AIDA + CNN	40.35	35.03	37.50	37.15	31.65	34.18
NeuralEL + CNN	36.89	35.21	36.03	37.81	30.05	33.49
Encoder-decoder baselines
Single Attention	45.91	38.36	41.80	40.10	39.12	39.60
Single Attention (+pre-trained)	47.25	40.53	43.63	43.14	43.11	43.12
Single Attention (+beam)	60.56	52.31	56.13	58.69	48.51	53.12
Single Attention (+triplet classifier)	73.78	50.13	59.70	67.04	53.01	59.21
Transformer	46.28	38.97	42.31	45.75	46.20	45.97
Transformer (+pre-trained)	47.48	40.91	43.95	48.41	48.31	48.36
Transformer (+beam)	58.29	50.25	53.97	61.81	61.61	61.71
Transformer (+triplet classifier)	73.07	48.66	58.42	71.24	57.61	63.70
Our pipeline baseline	67.43	54.22	60.11	64.60	64.05	64.32
Table 5: Baselines comparison. All results are taken from from Trisedya et al. (2019). Encoder-decoder baseline are proposed by the authors and other pipeline baseline include an NER and an ED system AIDA (Hoffart et al., 2011) or NeuralEL (Kolitsas et al., 2018) and then a relation extraction system CNN (Lin et al., 2016), MiniE (Gashteovski et al., 2017), or ClausIE (Corro and Gemulla, 2013). Best results are highlighted in bold and second best are underlined. Our pipeline baseline scores the best or on pair among these other methods.
REBEL (top 100 Relations)	REBEL (top 400 Relations)	REBEL (857 Relations)
Precision	Recall	F1	Precision	Recall	F1	Precision	Recall	F1
Micro
GenIE	68.76 ± 0.12	72.05 ± 0.13	70.36 ± 0.10	67.10 ± 0.13	70.62 ± 0.15	68.82 ± 0.12	68.02 ± 0.15	69.87 ± 0.14	68.93 ± 0.12
SotA Pipeline	44.76 ± 0.17	50.99 ± 0.17	47.67 ± 0.16	38.98 ± 0.13	51.18 ± 0.12	44.25 ± 0.11	43.30 ± 0.15	41.73 ± 0.13	42.50 ± 0.13
Micro
GenIE	52.26 ± 0.25	54.13 ± 0.27	52.75 ± 0.24	41.50 ± 0.66	38.53 ± 0.57	38.12 ± 0.51	33.90 ± 0.73	30.48 ± 0.65	30.46 ± 0.62
SotA Pipeline	27.41 ± 0.27	31.05 ± 0.18	25.87 ± 0.15	16.94 ± 0.63	19.00 ± 0.36	14.73 ± 0.37	12.20 ± 0.35	10.44 ± 0.22	9.48 ± 0.21
Table 6: Impact of the number of relations in the schema on REBEL. The schema is constrained at both training and testing time.