SlowGuess/NAACL_2022_OCR huggingface dataset:
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Michihiro Yasunaga, Jungo Kasai, Rui Zhang, Alexander R Fabbri, Irene Li, Dan Friedman, and Dragomir R Radev. 2019. Scisummnet: A large annotated corpus and content-impact models for scientific paper summarization with citation networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7386-7393.
A Reliability of the Mention Annotation
To compute the reliability of the automatic mention annotation, we let annotators manually identify all references to the cited work including citation markers, scientific entity names, and other co-references, e.g., "The authors ...", in a small sample of 262 publication pairs. We then compute the agreement with the gamma tool (Mathet et al., 2015) and obtain a mean score of 0.60 gamma macro averaged over the publications. We therefore explicitly instruct our annotators to use the highlighting as a rough guidance but to manually check for other mentions and co-references.
B Intent Labeling Scheme
We describe each intent of our labeling scheme in Table 5.
C Annotation Interface
A screenshot of the annotation interface is shown in Figure 4.
D Detailed Data Analysis
We show the detailed context length distribution and intent distribution in Figures 5. While most citations contexts are only a single sentence, our analysis shows a substantial long tail of context lengths. The distribution of citation functions mirrors results seen in similar annotation schemes such as Jurgens et al. (2018) and Pride and Knoth (2020). However, note that the Differences and Similarities classes occur with different frequencies, with authors more likely to highlight distinguishing features of their own work. This finding helps support our choice in explicitly splitting the CompareOr-Contrast category of Jurgens et al. (2018) in the labeling scheme we use for MULTICITE.
E Detailed Model Descriptions
SciBERT is only available in base configuration (12 layers, 12 attention heads, 768 as hidden size, cased vocabulary with size 31, 116). For RoBERTa, we employ the large version (24 layers, 16 attention heads, hidden size 1024, cased vocabulary with size 50, 265). Links to code base and pretrained models are given in Table 6.
F Full table for §5
Intent	Description
Background Motivation	The target paper provides relevant information for this domain. The target paper provides motivation for the source paper. For instance, it illustrates the need for data, goals, methods etc.
Uses Extends Similarities	The source paper uses an idea, method, tool, etc. of the target paper. The source paper extends an idea, method, tool, etc. of the target paper. The source paper expresses similarities towards the target paper. Either similarities between the source and the target paper or similarities between another publication and the target paper.
Differences Future Work	The source paper expresses differences towards the target paper. Either differences between the source and the target paper or differences between another publication and the target paper. The target paper is a potential avenue for future research. Often corresponds to hedging or speculative language about work not yet performed.
Table 5: Our citation intent labeling scheme based on Jurgens et al. (2018). We differ from their scheme by splitting their ComparisonOrContrast into separate categories—Similarities or Differences—which are represented in other annotation schemes
Figure 4: The interface of our dedicated annotation platform: on the left hand side, the annotator can browse through their assigned papers; in the center, each sentence (choosable via checkboxes) of the citing paper is displayed with citation mentions of the target reference paper highlighted in yellow; on the right hand side, available rhetorical function labels (choosable via checkboxes).
(a) Context length distribution (log scale).
(b) Distribution of rhetorical function classes.
Figure 5: Corpus-level statistics from analysis in §4. We use the following function abbreviations: Background (bg), Motivation (mot), Uses (use), Similarities (sim), Extends (ext), Differences (diff), Future Work (fw).
Codebase	Model	URL
Transformers	-	https://github.com/huggingface/transformers
SciBERT	https://huggingface.co/allenai/scibert_scivocab_uncased
RoBERTa	https://huggingface.co/roberta-large
Table 6: Links to codebases and pretrained models used in this work.
train on:	size = 1
support = 2795	size = 2
support = 335	size = 3
support = 112	size = 4
support = 39	all
support = 3313
weak	strict	weak	strict	weak	strict	weak	strict	weak	strict
SciBERT	1	0.78	0.69	0.45	0.28	0.47	0.24	0.51	0.18	0.74	0.62
3	0.74	0.64	0.59	0.39	0.54	0.29	0.62	0.23	0.72	0.60
5	0.71	0.61	0.50	0.33	0.46	0.27	0.54	0.18	0.68	0.57
7	0.62	0.54	0.43	0.28	0.48	0.27	0.51	0.15	0.60	0.50
9	0.56	0.50	0.37	0.25	0.37	0.21	0.56	0.18	0.53	0.46
gold	0.80	0.70	0.68	0.46	0.66	0.39	0.64	0.26	0.78	0.66
RoBERTa	1	0.80	0.69	0.46	0.29	0.46	0.25	0.56	0.18	0.75	0.63
3	0.78	0.66	0.59	0.41	0.50	0.27	0.62	0.18	0.75	0.61
5	0.75	0.63	0.54	0.39	0.54	0.32	0.59	0.21	0.72	0.59
7	0.73	0.62	0.53	0.37	0.44	0.24	0.56	0.21	0.70	0.58
9	0.71	0.59	0.54	0.36	0.46	0.26	0.54	0.15	0.68	0.55
gold	0.81	0.69	0.70	0.50	0.67	0.45	0.59	0.28	0.79	0.66
Table 7: Expanded results from Table 2 to include RoBERTa scores. The conclusions drawn in §5 are the same.