SlowGuess/NAACL_2022_OCR huggingface dataset:
5.1.3 Support Instance Evaluation (SIE)
MNAV being based on prototypical networks (Snell et al., 2017) means that the embeddings of all support instances of a relation type are averaged
For the pooling step we use mean pooling as, during preliminary experiments, it performed better than the logsumexp pooling used by Zhou et al..
<sup>7</sup>We also examined the option of resampling the training samples such that the amount of negative examples in each training step roughly matches the amount of positive examples (Han and Wang, 2020), but found this to be less effective.
into a single prototype. While this has proven to be an effective strategy, we argue that it may not be ideal during inference in a cross-domain setting where the change in data distribution may break the assumption that the mean of support instances provides a good prototype. In SIE we therefore use all individual support instances during inference, instead of their average.
5.1.4 Support Based NOTA Vectors (SBN)
In treating NOTA as a relation type and learning a persistent set of vectors during training, MNAV works on the assumption that the NOTA distribution during testing will match that seen during training. While this assumption is warranted and seems to work well for in-domain few-shot learning, we argue that this may not be the case for cross-domain settings. For this reason, we additionally add NOTA instances from the support documents to our set of NOTA vectors during training and inference. Instead of randomly sampling NOTA vectors from the support documents we sample the most similar  $k = 5$  NOTA instances measured via their dot product for each relation prototype. During inference in a new domain, we use only the NOTA vectors sampled from the support document and ignore the learned vectors.
5.2 Sampling Training & Development Episodes
We compare 2 different ways of sampling episodes during training. First we sample training and development episodes in the same way as for the test sets. In order to get sufficient coverage to calculate representative macro  $F_{1}$  scores on the development set, we sample 4k episodes. As an alternative we modify the query sampling by ensuring that for each episode at least one of the query documents contains an instance of the relation type  $r_{s}$ . This way we increase the amount of non-NOTA examples the model sees during training. Another effect is that we need fewer development episodes (we use 500) to calculate macro  $F_{1}$  scores.
Model	macro F1 [%]
1-Doc	3-Doc
Random Sampling	5.77	5.29
Ensure Positive	7.26	9.37
Table 3: Results for different query sampling strategies on the in-domain task
6 Analysis of Results
All our models are based on BERT<sub>BASE</sub> (Devlin et al., 2019) implemented using Hugging-face's Transformers (Wolf et al., 2020) and trained using mixed precision. We follow Zhou et al. (Zhou et al., 2021) in using AdamW (Loshchilov and Hutter, 2019) as optimizer (learning rates  $\in$  [1e-5, 3e-5, 5e-5, 1e-4], of which 1e-5 generally performs best) and training using linear warmup (1k/2k steps) (Goyal et al., 2017) followed by a linear learning rate decay. We use gradient clipping of 1.0. We train each model for 50k episodes and perform early stopping based on the macro  $F_{1}$  score on the development set which we measure every 1k/2k steps (when random sampling/ensuring positive examples). Each 1-doc training episode consists of 1 support document and 3 query documents, 3-doc training episodes contain 3 support documents and 1 query document. During training we limit the size of  $R_{\text{episode}}$  to 1. We run each model 5 times using different random seeds and select the learning rate with the highest mean macro  $F_{1}$  score on the development set for testing. For test scores we report the mean and standard deviation of macro  $F_{1}$  scores for models trained using 5 different random seeds. For this model we report the macro  $F_{1}$  score on the test set. Results are shown in tables 3, 4, and 5. All models were trained on either NVIDIA V100 or NVIDIA 3090 GPUs.
6.2 Comparing Sampling Strategies
In table 3 we compare the test macro  $F_{1}$  scores of the best models (chosen according to development set) trained using the 2 sampling strategies described in section 5.2 on the 1-Doc challenge using the model DL-MNAV. We find that ensuring positive query documents during training and development helps increase the  $F_{1}$  scores. Due to the better performance, as well as the reduced computation time needed for validation (thanks to the smaller development set), we use the latter strategy
The sampled NOTA representations do not persist across episodes.
<sup>9</sup>Since, with SIE, we do not use prototypes at inference time, we then perform this sampling step for each relation instance rather than for prototypes and increase  $k$  to 20.
Model	1-Doc	3-Doc
Precision [%]	Recall [%]	F1 [%]	Precision [%]	Recall [%]	F1 [%]
Baseline	0.36	9.69	0.60	0.60	10.75	0.89
DL-MNAV	6.26 ± 0.22	21.08 ± 2.71	7.05 ± 0.18	7.71 ± 0.69	22.80 ± 3.82	8.42 ± 0.64
DL-MNAVSIE	5.57 ± 0.04	23.12 ± 1.69	7.06 ± 0.15	5.16 ± 0.19	33.61 ± 2.68	6.77 ± 0.21
DL-MNAVSIE+SBN	1.02 ± 0.05	22.94 ± 1.87	1.71 ± 0.04	1.75 ± 0.16	23.41 ± 0.76	2.79 ± 0.24
Table 4: Results for FREDo in-domain task. Reported results are macro averages across relation types.
Model	1-Doc	3-Doc
Precision [%]	Recall [%]	F1 [%]	Precision [%]	Recall [%]	F1 [%]
Baseline	1.34	3.04	1.76	1.84	2.47	1.98
DL-MNAV	2.30 ± 0.45	0.58 ± 0.12	0.84 ± 0.16	3.02 ± 2.38	0.29 ± 0.13	0.48 ± 0.21
DL-MNAVSIE	1.77 ± 0.60	2.08 ± 0.34	1.77 ± 0.60	2.51 ± 0.66	2.52 ± 0.31	2.51 ± 0.66
DL-MNAVSIE+SBN	2.26 ± 0.11	4.37 ± 0.13	2.85 ± 0.12	3.47 ± 0.14	4.24 ± 0.21	3.72 ± 0.14
in all following experiments.
6.3 Baseline Results
As expected for a baseline which is not fine-tuned to the task at hand, the resulting macro  $F_{1}$  scores are very low. We argue, however, that the baseline is nevertheless relevant for 2 reasons. For the indomain challenge, the baseline proves that the tasks are not trivially solvable by using a pre-trained language model out of the box. For the cross-domain challenge, our baseline lets us see whether models overfit on the training domain.
6.4 In-Domain Setting
Test scores for the in-domain challenge are shown in table 4. We observe large improvements in  $F_{1}$  scores over the baseline, especially for DL-MNAV which reaches  $7.05\%$  in the 1-Doc and  $8.42\%$  in the 3-Doc task. SIE does not seem to affect the accuracy of the model in the 1-Doc task; in the 3-Doc task, however, the  $F_{1}$  score drops by 1.65 percentage points. SBN, on the other hand causes the  $F_{1}$  scores to drop by more than  $5\%$ . This performance drop following the removal of learned NOTA vectors clearly illustrates their effectiveness for in-domain tasks.
In table 6 we compare the best  $F_{1}$  scores of different few-shot relation extraction benchmarks. Overall, compared to scores for benchmarks such as FewRel (Han et al., 2018) FewRel 2.0 (Gao et al., 2019), the  $F_{1}$  scores are considerably lower, illustrating the difficulty of such a realistic challenge. When compared to the more realistic sentence-level benchmark FS-TACRED (Sabo et al., 2021) for
Table 5: Results for FREDo cross-domain task. Reported results are macro averages across relation types.
Benchmark	input length	realistic NOTA	best F1 [%]
FewRel	sentences	X	97.85
FewRel 2.0	sentences	X	89.81
FS-TACRED	sentences	✓	12.39
FREDo (ours)	documents	✓	7.06
Table 6: A comparison highlighting the levels of difficulty of different few-shot relation extraction benchmarks. For all sentence-level benchmarks, we report the highest  $F_{1}$  scores (at the time of writing) in the 5-way 1-shot setting. For FREDo we report the 1-Doc setting. For FewRel and FewRel 2.0, we report the highest scores found at the respective CodaLab competition websites.
which Sabo et al. report  $F_{1}^{10}$  scores of  $12.39\%$  (1-shot) and  $30.04\%$  (5-shot) MNAV, these results are in-line with our expectations for an even more realistic (and thereby evidently more difficult) challenge. Notably, the scaling behavior seen in FS-TACRED between the 1-shot and the 5-shot setting is not as pronounced for FREDo. We hypothesize that this is due the fact that the change in  $K$  is not as large (see table 2), meaning that (1) our 1-Doc setting does not correlate perfectly to the 1-shot setting, and (2) due to the way that additional support documents are sampled, the 3-Doc setting does not guarantee additional support examples for infrequently occurring relation types.
6.5 Cross-Domain Setting
Test scores for the cross-domain challenge are shown in table 5. For DL-MNAV we see a significant drop in  $F_{1}$  scores over the baseline, il
10The reported results are micro  $F_{1}$  scores
lustrating the problem with learned NOTA vectors in a cross-domain setting. SIE brings the scores back to the baseline level, illustrating that the distribution of support examples is no longer well represented by their mean values. Switching to SBN (DL-MNAVSIE+SBN), we find that our model exceeds the baseline scores, suggesting that the NOTA distribution on sciERC seems to be sufficiently different to cause an overfitting effect for learned NOTA vectors. While SBN improves the results over the naive baseline, even the improved  $F_{1}$  scores are extremely low. This is, however, unsurprising given the increase in difficulty over the previous setting.
6.6 Scalability of DL-MNAV
Although our methods show improvements over the proposed baseline in both tasks, the results are currently severely lacking, especially compared to the state-of-the-art supervised learning approaches on both data sets (65.92%  $F_{1}$  for DocRED (Xu et al., 2021) and 52.0%  $F_{1}$  for sciERC (Ye et al., 2022)). This performance gap raises the question of whether our models will achieve similar performance if given enough support documents. In order to assess the scalability of DL-MNAV when given amounts of annotated data resembling the supervised setting, we initialize a trained model with the full DocRED training corpus as support documents (96 classes, 3053 documents) and evaluate the performance on the full development set (96 classes, 998 documents). We measure an increase in recall to 45.75% combined with a drop in precision to 5.75%, resulting in a  $F_{1}$  score of 8.86%.
While a direct comparison of this score with the few-shot settings assessed in FREDo is not appropriate, due to the nature of how the task is posed (different relation types are examined), the score can be compared to results obtained from supervised learning. Here we see clearly that, when trained in a few-shot setting, DL-MNAV does not scale well to the supervised setting. We hypothesize that a key factor inhibiting the scalability is that the learned NOTA vectors are independent from the support documents in DL-MNAV. As a result the NOTA representations of the model are unaffected by added support documents. We do not experiment with SIE or  $\mathrm{SIE + SBN}$  as the amount of support instances would result in prohibitively large model sizes.
6.7 Limitations
Regarding the limitations of the proposed benchmark, FREDo, we believe that while it represents a good basis for model development, it will be beneficial to add other cross-domain data sets from a greater variety of domains in the future. With the current, low  $F_{1}$  scores seen in our tests, overestimating the performance of approaches does not seem to be too critical a danger. We are, however, hopeful that new methods might achieve significantly better results. At that point we suggest a reassessment of how representative this benchmark is of cross-domain performance in general. For the time being, however, we are confident that our tasks represent a valuable contribution to advancing the field.
In order to encourage the development of few-shot relation extraction approaches which are useful in real world scenarios, we propose FREDo, a few-shot document-level relation extraction benchmark. By moving to the document-level, the settings become more realistic, a problem which existing benchmarks are struggling with. For both in-domain and cross-domain tasks we present an approach which performs better than a simple baseline. Our experiments confirm that, even though some existing benchmarks imply that impressive, even superhuman performance can already be achieved in few-shot relation extraction, realistic tasks are very difficult using current approaches and that significant advances are required for few-shot relation extraction approaches to be usable in a real word scenario. In providing a benchmark that reveals this performance gap, we look to pave the way towards new methods with a potentially high impact on domain-specific and cross-domain relation extraction at scale.
This work was partially supported by the German Federal Ministry of Education and Research (BMBF) as part of the project IIDI (01IS21026D) and the Smart Data Innovation Lab as part of the Smart Data Innovation Challenges (01IS19030A). The authors acknowledge support by the state of Baden-Württemberg through bwHPC.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Tianyu Gao, Xu Han, Hao Zhu, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2019. FewRel 2.0: Towards more challenging few-shot relation classification. pages 6251-6256.
Priya Goyal, Piotr Dólar, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. 2017. Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour.
X. Han and L. Wang. 2020. A Novel Document-Level Relation Extraction Method Based on BERT and Entity Information. IEEE Access, 8:96912-96919. Conference Name: IEEE Access.
Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2018. FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 4803-4809. Association for Computational Linguistics.
Sarthak Jain, Madeleine van Zuylen, Hannaneh Hajishirzi, and Iz Beltagy. 2020. Scirex: A challenge dataset for document-level information extraction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
Jiao Li, Yueping Sun, Robin J. Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J. Mattingly, Thomas C. Wiegers, and Zhiyong Lu. 2016. BioCreative V CDR task corpus: a resource for chemical disease relation extraction. Database, 2016:baw068.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. In Proceedings of the International Conference on Learning Representations 2019, page 18.
Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3219-3232, Brussels, Belgium. Association for Computational Linguistics.
Ofer Sabo, Yanai Elazar, Yoav Goldberg, and Ido Dagan. 2021. Revisiting Few-shot Relation Classification: Evaluation Data and Classification Schemes. Transactions of the Association for Computational Linguistics, 9:691-706.
Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical Networks for Few-shot Learning. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019. Matching the Blanks: Distributional Similarity for Relation Learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2895-2905, Florence, Italy. Association for Computational Linguistics.
Thy Thy Tran, Phong Le, and Sophia Ananiadou. 2021. One-shot to Weakly-Supervised Relation Classification using Language Models. In *Automated Knowledge Base Construction* (2021).
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koryakavukcuoglu, and Daan Wierstra. 2016. Matching Networks for One Shot Learning. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc.
Denny Vrandecic and Markus Krötzsch. 2014. Wikidata: a free collaborative knowledgebase. Communications of the ACM, 57(10):78-85.
Haoyu Wang, Ming Tan, Mo Yu, Shiyu Chang, Dakuo Wang, Kun Xu, Xiaoxiao Guo, and Saloni Potdar. 2019. Extracting Multiple-Relations in One-Pass with Pre-Trained Transformers. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1371-1377, Florence, Italy. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Ye Wu, Ruibang Luo, Henry C. M. Leung, Hing-Fung Ting, and Tak-Wah Lam. 2019. RENET: A Deep Learning Approach for Extracting Gene-Disease Associations from Literature. In Research in Computational Molecular Biology, Lecture Notes in Computer Science, pages 272-284, Cham. Springer International Publishing.
Yuxin Xiao, Zecheng Zhang, Yuning Mao, Carl Yang, and Jiawei Han. 2022. SAIS: Supervising and Aug-mentation Intermediate Steps for Document-Level Relation Extraction. In NAACL.
Benfeng Xu, Quan Wang, Yajuan Lyu, Yong Zhu, and Zhendong Mao. 2021. Entity Structure Within and Throughout: Modeling Mention Dependencies for Document-Level Relation Extraction. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 14149-14157. AAAI Press.
Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019. DocRED: A Large-Scale Document-Level Relation Extraction Dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 764-777, Florence, Italy. Association for Computational Linguistics.
Deming Ye, Yankai Lin, Peng Li, and Maosong Sun. 2022. Pack Together: Entity and Relation Extraction with Levitated Marker. In Proceedings of ACL 2022.
Dian Yu, Kai Sun, Claire Cardie, and Dong Yu. 2020. Dialogue-based relation extraction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
Ningyu Zhang, Xiang Chen, Xin Xie, Shumin Deng, Chuanqi Tan, Mosha Chen, Fei Huang, Luo Si, and Huajun Chen. 2021. Document-level Relation Extraction as Semantic Segmentation. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 3999-4006. International Joint Conferences on Artificial Intelligence Organization. Main Track.
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-aware Attention and Supervised Data Improve Slot Filling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 35-45, Copenhagen, Denmark. Association for Computational Linguistics.
Wenxuan Zhou, Kevin Huang, Tengyu Ma, and Jing Huang. 2021. Document-Level Relation Extraction with Adaptive Thresholding and Localized Context Pooling. In Proceedings of the AAAI Conference on Artificial Intelligence.
A Overlap of DocRED and sciERC
In table 7 we show the mapping of sciERC relation types onto Wikidata (Vrandecic and Krötzsch, 2014) properties and whether these relation types are contained in DocRED.
B Relation Types in in-domain dataset
In tables 8-11, we list the relation types in the different datasets based on DocRED.
sciERC ID	Wikidata ID	DocRED
hyponym-of	P279	✓
part-of	P361	✓
used-for	P366	✗
compare	P2210	✗
evaluate-for	P5133	✗
feature-of	-	-
conjunction	-	-
Table 7: Overlap of relation types in sciERC and DocRED
Wikidata ID	Description	Number of instances
P131	located in the administrative territorial entity	4193
P577	publication date	1142
P175	performer	1052
P569	date of birth	1044
P570	date of death	805
P161	cast member	621
P264	record label	583
P527	has part	632
P19	place of birth	511
P54	member of sports team	379
P40	child	360
P30	continent	356
P69	educated at	316
P26	spouse	303
P607	conflict	275
P159	headquarters location	264
P22	father	273
P400	platform	304
P1344	participant of	223
P206	located in or next to body of water	194
P127	owned by	208
P170	creator	231
P178	developer	238
P20	place of death	203
P1412	languages spoken, written or signed	155
P155	follows	188
P710	participant	191
P6	head of government	210
P108	employer	196
P276	location	172
P156	followed by	192
P166	award received	173
P123	publisher	172
P800	notable work	150
P449	original network	152
P58	screenwriter	156
P706	located on terrain feature	137
P162	producer	119
P37	official language	119
P241	military branch	108
P31	instance of	103
P403	mouth of the watercourse	95
P580	start time	110
P585	point in time	96
P749	parent organization	92
P937	work location	104
P36	capital	85
P576	dissolved, abolished or demolished	79
P172	ethnic group	79
P205	basin country	85
P1376	capital of	76
Table 8: Relation types present in training data (continued on next page).
Wikidata ID	Description	Number of instances
P171	parent taxon	75
P740	location of formation	62
P840	narrative location	48
P676	lyrics by	36
P1336	territory claimed by	33
P551	residence	35
P1365	replaces	18
P737	influenced by	9
P190	sister city	4
P807	separated from	2
P1198	unemployment rate	2
Table 9: Relation types present in training data (continued).
Wikidata ID	Description	Number of instances
P27	country of citizenship	2689
P150	contains administrative territorial entity	2004
P571	inception	475
P50	author	320
P1441	present in work	299
P57	director	246
P179	series	144
P137	operator	95
P112	founded by	100
P86	composer	79
P176	manufacturer	83
P355	subsidiary	92
P136	genre	111
P488	chairperson	63
P1366	replaced by	36
P1056	product or material produced	36
Table 10: Relation types present in development/Validation data.
Wikidata ID	Description	Number of instances
P17	country	2831
P361	part of	194
P495	country of origin	212
P102	member of political party	98
P463	member of	113
P3373	sibling	134
P1001	applies to jurisdiction	83
P118	league	56
P674	characters	74
P194	legislative body	56
P140	religion	82
P35	head of state	51
P364	original language of work	30
P272	production company	36
P279	subclass of	36
P25	mother	15
P582	end time	23
P39	position held	8
Table 11: Relation types present in test data.