SlowGuess/NAACL_2022_OCR huggingface dataset:
MULTICITE: Modeling realistic citations requires moving beyond the single-sentence single-label setting
Anne Lauscher $^{1}$  Brandon Ko $^{2}$  Bailey Kuehl $^{3}$  Sophie Johnson $^{3}$  Arman Cohan $^{3}$  David Jurgens $^{4}$  Kyle Lo $^{3}$
$^{1}$ MilaNLP, Bocconi University, Italy,  $^{2}$ University of Washington, Seattle WA,  $^{3}$ AI2, Seattle WA,  $^{4}$ School of Information, University of Michigan, Ann Arbor MI  
 $^{1}$ anne.lauscher@unibocconi.it,  $^{2}$ bk36@cs.washington.edu,  
 $^{3}$ armanc,baileyk,sophiej,kylel] @allenai.org,  $^{4}$ jurgens@umich.edu
Citation context analysis (CCA) is an important task in natural language processing that studies how and why scholars discuss each other's work. Despite decades of study, computational methods for CCA have largely relied on overly-simplistic assumptions of how authors cite, which ignore several important phenomena. For instance, scholarly papers often contain rich discussions of cited work that span multiple sentences and express multiple intents concurrently. Yet, recent work in CCA is often approached as a single-sentence, single-label classification task, and thus many datasets used to develop modern computational approaches fail to capture this interesting discourse. To address this research gap, we highlight three understudied phenomena for CCA and release MULTICITE, a new dataset of 12.6K citation contexts from 1.2K computational linguistics papers that fully models these phenomena. Not only is it the largest collection of expert-annotated citation contexts to-date, MULTICITE contains multi-sentence, multi-label citation contexts annotated throughout entire full paper texts. We demonstrate how MULTICITE can enable the development of new computational methods on three important CCA tasks. We release our code and dataset at https://github.com/allenai/multicite.
Citations connect the current paper to the broader discourse of science (e.g., Garfield, 1955; Siddharthan and Teufel, 2007), help signal future impact and uses (e.g., McKeown et al., 2016), and, in downstream applications, can aid in summarizing a work's contributions (e.g., Qazvinian and Radev, 2008; Cohan and Goharian, 2015; Lauscher et al., 2017a). The study of the role and purpose of citations, known as citation context analysis (CCA
Swales, 1986), has uncovered traces of how ideas have influenced a field, collaboration and competition among peers, and trends in scientific fields.
Computational approaches to CCA have largely focused on classifying the intent or purpose of a particular citation. With the advent of deep neural models, recent CCA research efforts have focused on increasing the size of the published resources (Cohan et al., 2019; Tuarob et al., 2019; Pride and Knoth, 2020). However, larger data sets have come at the expense of oversimplifying the rich discourse patterns surrounding citations, especially as complexity of annotation is often traded for large-scale data collection. The aforementioned recent large-scale resources, for instance, have approached CCA as a single-sentence, single-label classification task, overlooking the richness and nuance with which citations are used in discourse.
Contributions. In this work, we aim to fuel and inspire research into new or understudied tasks in CCA and their respective computational methods motivated by complex phenomena in scholarly citations. Our contributions are three-fold.
1) First, we identify three phenomena of citation behavior in scholarly literature that remain understudied by prior work in CCA (§2). We then propose future CCA research also consider efforts into three tasks well-motivated by these phenomena: (§5) multi-label classification given a variable-length citation context, (§6) context identification given a citation, and the novel (§7) evidence-based assessment of citing-cited paper relationships.
2) Second, to support these tasks and other new forms of CCA research, we introduce a novel resource (§4), the Multi-Sentence Multi-Label Multi-Mention Citation (MULTICITE) corpus, an expert-annotated collection of 1.2K full-text English-language publications from computational linguistics with 12.6K labeled citation contexts. MULTICITE substantially opens up new op
* Part of the work was conducted during an internship at the Allen Institute for AI and while being affiliated to the University of Mannheim.
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1875 - 1889 July 10-15, 2022 ©2022 Association for Computational Linguistics
portunities in CCA by being the only dataset which captures all three understudied phenomena and is large enough to support development of modern computational methods, i.e., deep neural models.
3) Finally, we lay the groundwork for future CCA research by demonstrating how one can use our new resource to develop modern computational methods to tackle each of the proposed analysis tasks. Our experiments establish strong baselines against which future work can compare.
2 Understudied Phenomena in CCA
Using examples, we highlight and motivate three understudied phenomena in CCA, showing that each reflects natural ways in which authors cite.
Multi-sentence contexts. While a citation appears in a particular sentence, its discussion may span that sentence and beyond. For example, consider the sentence:
"Gliozzo et al., (2005) succeeded eliminating this requirement by using the category name alone as the initial keyword, yet obtaining superior performance within the keyword-based approach."
This sentence alone provides background information about a previous approach and consequently, one could describe its function as Background from that text. However, the subsequent sentence continues the discussion of the citation:
"The goal of our research is to further improve the scheme of text categorization from category name, which was hardly explored in prior work."
Only by including this sentence can we identify the underlying intent of the authors: the cited publication is used as Motivation for the presented research. Through our annotation process (§4) we find that  $17.1\%$  of citation contexts involve multiple sentences. Thus, we argue that correctly modeling the precise scope of a citation – that is, its associated context – is necessary for understanding the discourse role it plays in the citing paper.
Multiple interpretations of function. A single citation may have multiple concurrent interpretations for its function. Not to be dismissed as simply being ambiguous, these multiple interpretations can stem from extended discussion of how the citing paper relates to the cited paper. Consider the sentence:
"In our experiments we use the same definition of structural locality as was proposed for the ISBN dependency parser in (Titov and Henderson, 2007b)."
This sentence can be labeled as Similarities. However, another possibility is to label this sentence as Uses, as the authors are adopting a definition of the cited work. Further, a citation can exhibit multiple interpretations through partially overlapping or entirely different contexts:
"Results Table 1 compares the published BERT BASE results from Devlin et al. (2019) to our reimplementation with either static or dynamic masking. We find that our reimplementation with static masking performs similar to the original BERT model, and dynamic masking is comparable or slightly better than static masking."
Here, the published results from the well-known BERT paper are a research artefact that is Used as a baseline (sentence 1). Then, the authors compare their reimplementation as well as their extension to these results (sentence 2), resulting in expressed Similarities as well as Differences. Through our annotation process (\$4) we find that  $19.3\%$  of citations have multiple interpretations. These multiple concurrent interpretations partially explain the existence of citation contexts that extend beyond the sentence boundary, and consequently are necessary to fully model how a particular citation contributes to the scientific discourse within a paper.
Multiple mentions densely throughout paper. A single reference may be cited (or mentioned) multiple times throughout a paper, potentially with each citation's occurrence serving a different function(s) depending on what the citing author is trying to communicate. Modeling these different citation mentions is critical to understanding all the ways a single reference paper may influence another. For instance, in §4, our paper Extends the citation labeling scheme of Jurgens et al. (2018) and then reports Similarities in dataset patterns in a separate mention. Through our annotation process in §4, we find referenced papers can be mentioned throughout a citing paper on average 9.1 times, yielding an average of 3.8 distinct interpretations over the course of a citing paper. Recognizing all the myriad functions a single reference can play throughout a citing paper is key to fully understanding the complex relationship between papers.
Motivated by these phenomena, we curate a novel resource (§4) from which we derive and formalize three CCA tasks that account for these phenomena. Systems developed to tackle these tasks can be viewed as answering the following CCA questions: (1) Given a citation context, what are all the different possible interpretations of its function or role within the citing paper (§5)? (2) Given
a citation mention, what constitutes the relevant, surrounding passage that comprises its "context" ( $\S 6$ )? (3) Given a pair of citing-cited works, what function(s) does the cited work serve within the citing work and what are the evidential passages supporting this assessment ( $\S 7$ )?
The importance and role of citations in understanding scholarly work has been recognized across multiple disciplines, from sociology (e.g., Garfield et al., 1964, 1970) to computer science (e.g., McKeown et al., 2016; Yasunaga et al., 2019; Lauscher et al., 2019; Daquino et al., 2020). Prior work in CCA has largely focused on classifying interpretations of a citation along varying dimensions, such as function (e.g., Teufel et al., 2006; Lauscher et al., 2017b), sentiment (e.g., Athar, 2011; Jha et al., 2016), or relative importance to a paper (Valenzuela et al., 2015). However, these works have frequently simplified their analyses, overlooking the important phenomena in  $\S 2$ .
We describe prior computational work on CCA with respect to the three phenomena pursued here. Table 1 overviews their corresponding published resources<sup>1</sup> and shows the variation in which datasets support different lines of citation inquiry.
Mostly single-sentence or fixed-width contexts.
While social scientists studying citations have acknowledged the broader context needed to understand citations (Swales, 1986), computational work has largely treated the citing sentence as the only context needed for CCA methods (e.g., Athar, 2011; Dong and Schäfer, 2011). In particular, while annotators in many works had access to larger context, the resulting resources still labeled only a single sentence. A handful of works have acknowledged the importance of multi-sentence, precise, and flexible or variable-length context windows (e.g., Elkiss et al., 2008; Kaplan et al., 2009; Athar and Teufel, 2012; Abu-Jbara et al., 2013; Kaplan et al., 2016, inter alia); yet, the resources associated with these works have either primarily focused on sentiment, rather than the more complex notion of citation function, or have opted for an arbitrary fixed-sized window as the correct context. In contrast, our work provides precisely-defined contexts for each citation function present. The
exception is Abu-Jbara et al. (2013), who also annotate variable-length contexts, but unlike us, first extract a fixed window of 4 sentences and do not consider how citations with multiple functions can each have their respective (different) contexts.
Few capture multiple interpretations. Prior computational work has largely assumed a citation has only a single rhetorical function, or when multiple intents are present, that there is only one primary function that warrants annotation. Indeed, in some of the preceding works, ambiguous citations were reportedly removed (e.g., Cohan et al., 2019), leading to an artificial simplification of the task. Of computational work, only Athar and Teufel (2012) has attempted to model multiple interpretations. In their work, up to four sentences surrounding a citation are each assigned one sentiment-related label. In contrast, our new resource recognizes the existence of citations with multiple concurrent labels (possibly with varying contexts for a single citation) and each label is annotated with the specific context associated with that interpretation.
Lack intentional sampling for dense citations.
Prior works have employed different strategies for selecting which citations they annotate. Works focused on citation sentiment have tended to label all citations within a paper or all mentions of a particular paper. However, works focusing on more complex interpretations like rhetorical function have included more selective sampling, e.g., targeting or up-sampling certain sections. For instance, while Abu-Jbara et al. (2013) model more complex citation contexts, they focus only on citations to a small specific set of highly-cited references found within a much larger set of papers. This sampling strategy can result in many references occurring only once within the citing paper, which yields datasets not much different from ones from works that do not consider multiple mentions at all (Cohan et al., 2019). Such datasets would not support CCA methods that attempt to learn holistic interpretations of how papers relate via access to many within-paper mentions. Our work is the first to consider a dedicated sampling strategy that identifies papers likely to be densely populated with myriad mentions of a particular reference ( $\S 4$ ).
4 MULTICITE: A New Resource for CCA
We describe the curation process and present dataset statistics for MULTICITE, the first resource
For detailed reviews of CCA, we refer to Iqbal et al. (2020) and Hernandez-Alvarez and Gomez (2016).
Author/ Year	Concept†	Size	Context?	Multi-label?	Dense sampling?
Pride and Knoth (2020)	Purpose & Influence	11,233	Single sentence	X	X
Cohan et al. (2019)	Intent	11,020	Single sentence	X	X
Ravi et al. (2018)	Sentiment	8,925	Single sentence	X	X
Tuarob et al. (2019)	Algorithm's Function	8,796	3 sentences	X	X
Athar (2011)	Sentiment	8,736	Single sentence	X	X
Abu-Jbara et al. (2013)	Purpose & Polarity	3,271	Variable within ±4 sent	X	X
Teufel et al. (2006)	Function	2,829	Surrounding paragraph	X	X
Jochim and Schütze (2012)	Citation Facets	2,008	Did not annotate contexts	X	X
Jurgens et al. (2018)	Function	1,969	Surrounding 300 chars	X	X
Athar and Teufel (2012)	Sentiment	1,741	Variable up to 4 sents	1 label/sent.	X
MULTICITE (this work)	Function	12,653	Variable, per-label	✓	✓
Table 1: Existing CCA datasets compared to this work along three dimensions: (1) How they define citation Contexts, (2) Whether they handle Multiple Labels per citation, and (3) Whether they employed a Sampling strategy to obtain Dense citations of a reference throughout the citing paper.
(a) Label distribution per context length.
(b) Pointwise mutual information (PMI) between labels.
Figure 1: Results of corpus-level analysis in §4 revealing interactions between phenomena present in MULTICITE. We use the following abbreviations for the functions: Background (bg), Motivation (mot), Uses (use), Similarities (sim), Extends (ext), Differences (diff), Future Work (fw).
for CCA that captures all three phenomena of interest and is large enough to support development of modern neural approaches—It is the largest collection of expert-annotated citation contexts to-date.
4.1 Curation process
Sampling. We procure an initial corpus of 50K full-text papers from the ACL Anthology and arXiv (with cs.CL category) using S2ORC (Lo et al., 2020), a large collection of papers released to support computing research. To find papers that likely exhibit our phenomena of interest, we employ the following strategy: For each paper's references, we compute the number of distinct paragraphs in which the reference's citation marker(s) appear, normalized by the total number of paragraphs. Re
trieving the top  $k$  paper-reference pairs yields papers in which the target reference is cited many times (hopefully in many different ways).
Labeling scheme. We extend the labeling scheme of Jurgens et al. (2018) for classified citations by their rhetorical function, mirroring the approach of Teufel (2014) to differentiate between citations that identify Similarities versus those identifying Differences. The full scheme with examples is shown in Supplemental Table 5.
Annotation protocol. Annotators were given a citing paper's full text and a target reference whose citations to consider. They were instructed to (1) read the text surrounding each mention of the target reference (highlighted automatically), (2) consider all rhetorical function labels associated with the
2Like S2ORC, our data is licensed as CC-BY-NC 2.0.
mention, (3) and for each candidate label, to indicate every sentence $^{3}$  belonging to the context for that label. To reduce ambiguity around what does (or doesn't) belong in a citation context, we trained annotators to resolve non-citation coreferences to the cited paper as the dominant (but not only) way to observe a multi-sentence context and to continue reading until they felt confident about the labels assigned. Furthermore, for difficult cases, we instructed them to temporarily remove context sentences to see whether the label could still be inferred. Furthermore, annotators were encouraged to skip and leave comments for difficult cases which were routed to two experienced annotators for adjudication.
Recruiting and training. We hired nine NLP graduate students via Upwork. Each student went through an hour of one-on-one training and another hour of independent annotations, which were manually reviewed and used for another hour of one-on-one training focused on feedback and correcting common mistakes. Annotators were then allowed to work independently on batches of 20 papers at a time with manual annotation review after each batch for quality control. Annotators were paid between $25-35 USD per hour and understood that their annotations would be publicly-released as a resource for research.
Inter-annotator agreement (IAA). Producing a single measure of IAA is difficult for data collected in this manner: annotators might agree on labels but disagree on the choice of context, vice versa, or disagree on both fronts. While some prior work has developed IAA measures that capture both context selection and labeling, e.g.,  $\gamma$  by Mathet et al. (2015), such methods aren't widely adopted in NLP and thus resulting IAA values can be difficult to interpret. We opted instead to recruit two new annotators to perform two tasks: (a) identify all function labels when shown a gold context, and (b) identify context sentences when shown a citation mention and gold label pair. For (a), the two achieved an average accuracy of 0.76 when counting any gold label match as correct, and 0.70 when only counting cases when all predicted labels match the gold annotations as correct ( $n = 54$ ). For (b), the two achieved an average sentence-level F1 score of 0.64, 0.63 and 0.65, respectively for
gold contexts of length 1, 2 or  $3+$  sentences, and an overall Cohen's Kappa of 0.65 ( $n = 120$ ).
4.2 Corpus statistics
MULTICITE consists of 1,193 papers from CL and NLP (with an average of 139.7 sentences per paper) with 12,653 annotated citation contexts. We highlight three key aspects. (1) We find that over one in six contexts  $(17.1\%)$  extends beyond a single sentence. MULTICITE provides 2,167 multi-sentence contexts, with 161 reaching  $5+$  sentences. See Supplemental Figure 5a for further breakdown. (2) Nearly one in five citations  $(19.3\%)$  also have multiple interpretations. Of the 2,084 multi-labeled citations, most have two labels but a handful have up to four. Our label distribution is skewed with the Background and Uses appearing most frequently and Future Work the least; this is in-line with prior work (e.g., Pride and Knoth, 2020). See Supplemental Figure 5b for a full label distribution. (3) Regarding mention occurrence, we find that our sampling strategy successfully finds interesting paper-reference pairs as the average of citation occurrences in a paper is 9.1. In addition, our intuition about mentions exhibiting different interpretations is supported by an average of 3.8 unique functions across all mentions of a given reference.
Looking at interactions between phenomena, we observe two interesting findings. First, we observe in Figure 1a that while every label is capable of being expressed with only a few sentences, there are surprisingly few single-sentence Motivation contexts. Second, pointwise mutual information (PMI) between the labels (Figure 1b) shows high co-occurrence in Extends and Differences.
While our new resource is the largest citation dataset, our design choices introduce some limitations. Due to task complexity, annotators labeled only direct citations but not indirect mentions (e.g. entity names used without citation). Our choice in open data, including the full texts, potentially allows future work to add these to our corpus. Further, our citation annotation scheme features seven classes, which itself is a simplification of the complex ways in which citations are used. While others have proposed 12-class (Teufel, 2014) or even 35-class (Garzone and Mercer, 2000) annotation schemes to capture long-tail citation uses, we ultimately opted for a simpler scheme that facilitates scalable annotation and easier model development.
$^{3}$ Identified using ScispaCy (Neumann et al., 2019).
4Median wage for similar work on the platform, and well above minimum wage in ANONYMIZED.
Figure 2: Shift in task form for citation context classification. Within a document (grey), each citation mention (orange) is associated with a surrounding context. Multi-sentence setting can contain both single-sentence (blue) and multi-sentence (purple) contexts. Multi-label scores need not sum to one as in the single-label setting.
5 Citation Context Classification
The first task we consider makes use of both the multi-sentence and multi-label phenomena described in §2: multi-label classification. One can view this as the multi-label extension to the traditional multi-class classification task common in prior work (see Figure 2), but we show experimentally that models using variable-length gold contexts yield superior results. This motivates (1) a shift in computational research away from assuming fixed-window (often 1-sentence) inputs, and (2) additional investment in understudied tasks like citation context identification (§6).
5.1 Modeling approach
Using Transformer (Vaswani et al., 2017) models, which have demonstrated success on many text classification tasks and datasets, we train both multiclass and multi-label predictors. For multi-class predictors (which output a single label), we follow common practices in adding a linear classification layer over the [CLS] token at the beginning of each input context and applying a softmax for prediction. For multi-label classification, we deviate by replacing the softmax with a sigmoid operation per label to produce multi-label outputs; we set our sigmoid prediction threshold to 0.5.
We conduct a series of experiments aligned with previous works (e.g., Jha et al., 2016), in which
we feed varying amounts of context sentences to a text classification model. These experiments aim to understand the importance of variable-length inputs and to establish computational baselines to support future research on MULTICITE.
Data processing. We perform a train-validation-test split of MULTICITE at a paper-level; to avoid leakage, all examples from the same paper are assigned to the same split resulting in 5,491 training, 2,447 validation, and 3,313 test instances. Because citation contexts can contain mentions of multiple papers, we remove task ambiguity by tagging the target paper's mention with [CITE] tokens.