SlowGuess/NAACL_2022_OCR huggingface dataset:
Training and optimization. We use pretrained SciBERT-base and RoBERTa-large weights from Huggingface Transformers (Wolf et al., 2020). We optimize the models using the average over binary cross-entropy losses for each label using Adam (Kingma and Ba, 2015) with a batch size of 32. We use grid search based on validation set performance to set the learning rate (1e-5 or 2e-5) and number of epochs (between 1 and 9).
Evaluation. We compute two types of accuracies: a strict version, in which a prediction is correct iff all predicted labels match exactly the gold annotation; and a weak version, in which a prediction is correct if at least one of the predicted labels matches the gold classes. The weak measure reflects an upper bound on performance (i.e., whether the model can detect any of the correct intents) and allows us to compare our multi-label models with models that only return single-labels. We perform this evaluation both across all MULTICITE test examples as well as broken down by specific gold context sizes (up to 4 sentences).
We present single run results on the test set in Table 2, and observe two important findings:
Variable-length contexts improve performance. Despite high occurrence of 1-sentence contexts in our corpus, a model trained on variable-length gold contexts still outperforms that trained only on single-sentence inputs (see rows "1" versus "gold"). Surprisingly, the variable-length model even outperforms the single-sentence model even on single-sentence test examples (see column "size = 1").
For brevity, we leave RoBERTa-large scores in the supplemental Table 7 as we observe the same patterns as SciBERT and arrive at the same conclusions.
train on:	size = 1	size = 2	size = 3	size = 4	all
weak	strict	weak	strict	weak	strict	weak	strict	weak	strict
1	0.78	0.69	0.45	0.28	0.47	0.24	0.51	0.18	0.74	0.62
3	0.74	0.64	0.59	0.39	0.54	0.29	0.62	0.23	0.72	0.60
5	0.71	0.61	0.50	0.33	0.46	0.27	0.54	0.18	0.68	0.57
7	0.62	0.54	0.43	0.28	0.48	0.27	0.51	0.15	0.60	0.50
9	0.56	0.50	0.37	0.25	0.37	0.21	0.56	0.18	0.53	0.46
gold	0.80	0.70	0.68	0.46	0.66	0.39	0.64	0.26	0.78	0.66
Table 2: Weak and strict accuracy scores of multi-label classifiers on MULTICITE. Column "train on:" reflects the size of contexts seen at training time; rows "1", "3", "5", "7" and "9" are fixed-width window sizes that contain gold citation context, while row "gold" means training with variable-length gold contexts. Columns "size = ?" correspond to the size of gold contexts supplied at test time. Best performances per column are bolded.
We hypothesize that due to the training on variable length gold contexts, the model is able to better capture the fine-grained differences between the many intent classes, and thus, performs better than a model that has seen restricted contexts only. For instance, as the label distribution varies across the different context lengths (cf. Figure 1), the input context length might serve as an additional signal to the variable-length classification model, while a model trained on a fixed window can not make any use of such length-dependent signals.
Fixed-width windows don't cut it. One cannot simply train on fixed-width windows and hope to achieve the same performance boost as we see with variable-length contexts. In fact, the variable-length model outperforms fixed-width models even on test examples where the gold context and fixed-width window exactly match (see rows "3" versus "gold", column "size = 3").
6 Identifying Citation Contexts
Motivated by the need for variable-length gold contexts, the second task we consider is citation context identification (Abu-Jbara et al., 2013), which has seen little research since the advent of neural models. We demonstrate experimentally how MULTICITE can be used to train modern neural models for this understudied task, thereby setting first Transformer baselines to support future study.
Task formulation. We adopt the task formulation from Abu-Jbara et al. (2013): given a window of sentences around a target paper citation mention, predict the sentences belonging to that citation's context—a set of sentences that are sufficient for identifying that citation's label(s), e.g., intent.
Modeling approach. We consider models for sentence-level sequence tagging (Dernoncourt and Lee, 2017; Cohan et al., 2018), which have been used successfully to group sentences in scholarly abstracts by their discourse roles. We establish a first computational baseline on this task by adapting the approach from Cohan et al. (2018)—a SciBERT (Beltagy et al., 2019) model trained to receive as input a window of sentences separated by separator tokens [SEP], and apply a linear classification to these tokens to output sentence-level predictions.
Data processing. To study how model performance may depend on window sizes, we generate examples from every gold citation context in MULTICITE with windows of 2, 4, 6, 8 and 10 sentences. For each window size, we follow four desiderata: (1) As in §5, we reduce task ambiguity by tagging the target paper's citation mention with [CITE] tokens. (2) Windows must contain nonzero context and non-context sentences. (3) Windows must contain gold contexts entirely without truncation. This results in fewer total examples for small window sizes. For instance, for windows of 2 and 6 sentences, we create 10,453 and 12,506 examples, respectively, a  $20\%$  increase in data from multi-sentence contexts. (4) To prevent exploiting positional information, gold contexts cannot always be centered in the window. We construct windows by, randomly appending non-context sentences around golds until the desired window size.
We perform 5-fold cross validation with a 70-10-20 train-validation-test data split. To avoid leakage, all examples from the same paper are assigned to the same split in a given fold.
Due to the 512 SciBERT token limit, some examples may have fewer than the stated number of sentences. We ensure examples contain whole sentence inputs without truncation.
Window	Micro-F1	Macro-F1
2	75.17	80.17
4	81.16	85.61
6	79.91	84.35
8	76.64	81.56
10	72.14	77.42
Table 3: Test results for citation context identification across models trained with various input window size configurations. Best performing model row is bolded.
Training and optimization. We use pretrained weights for SciBERT-base available on Huggingface Transformers (Wolf et al., 2020), which we fine-tune with a binary (context or not) cross-entropy loss over the sentence-level predictions. We use the Adam (Kingma and Ba, 2015) optimizer with a linear learning rate scheduler (with max learning rate of 3e-5 after 100 warmup steps, batch size of 36, and up to a maximum of 5 epochs). We manually chose these hyperparameters using validation performance across all folds.
Evaluation. We evaluate these models using two sentence-level F1 metrics. The micro-averaged F1 evaluates the performance across all sentences across the entire corpus as to whether they were correctly (or incorrectly) classified as belonging to the context of the target citation mention. The macro-averaged F1 averages F1-scores across sentences first computed within each paper.
We take special care to ensure fair comparison across models trained on different window sizes. For instance, the Window  $= 2$  processing only contains single-sentence contexts and model performance on only considering these examples reaches high 90 F1 scores. Instead, our evaluation is processing-agnostic—small window models are penalized for defaulting to "not context" predictions for sentences beyond their window.
As we increase window size, performance on both metrics increases as shown in Table 3, but only to a point. Large context windows create difficulty in the model's training due to the increased number of non-context sentences. While the performance is high enough that our model can likely serve as an effective preprocessing step for other CCA models needing variable-sized windows, our results point to the challenge and potential for new models to improve upon in identifying contextual boundaries of citation discourse.
Figure 3: Evidence-based assessment of citing-cited paper relationships. Given a document with all citation mentions (orange) of a target paper identified, we aim to answer questions about the function of the cited paper ([17]) within the citing work, and provide snippets as supporting evidence (pink).
7 Evidence-based Assessment of Citing-Cited Paper Relationships
Dense citations to a single reference reveal a multifaceted relationship between the citing and cited papers. The dense annotations in MULTICITE allow development of CCA methods that model these document-level relationships, e.g., assessing holistically how the cited work has influenced the design or outcome of the citing work. Consider an application scenario where a user wants to know why a given paper cites another. A hypothetical system might provide a paper-level assessment of their relationship and citation contexts as supporting evidence from the citing paper's full text.
Motivated by this aspirational application, we propose a CCA task requiring document-level understanding of dense citations: Answering questions about citing-cited paper relationships by operating across multiple mentions within paper full-text. Figure 3 provides a schematic of this task, which demonstrates the novel research potential of MULTICITE at supporting higher-level CCA, and by casting CCA as a form of question answering (QA), highlights compatibility with modern attempts on scientific QA (Dasigi et al., 2021).
While building such a system is beyond the scope of our work, we demonstrate through experiments how MULTICITE can support development of QA
models behind such applications.
Task formulation. We adapt the scientific QA task form in Dasigi et al. (2021) to our CCA setting by mapping each citation function to a question-answer template. For instance, Background becomes a question "Does the paper cite [TARGET] for background information?" and answer "Yes", and the Background citation context then becomes evidence for that answer.
Modeling approach. We consider the Qasper (Dasigi et al., 2021) document-grounded QA model pretrained to answer questions about NLP papers with supporting evidence. Qasper is based on the Longformer-Encoder-Decoder (LED) (Beltagy et al., 2020) architecture, which enables it to encode an entire paper's full text as a single input string. Like the model in §6, input sentences are separated by tokens  $\langle /s \rangle$ , which are used to predict binary labels of Evidence (or not) via a linear classification layer. Unlike the model in §6, LED also generates strings, which we train to produce answer strings "Yes" or "No".
Data processing. We use the same train-validation-test split from §5. For each citing-cited paper pair, we create seven questions (one for each of our seven rhetorical functions). To create positive examples, for all functions with at least one gold citation context, we create a "Yes"-answer and provide the first<sup>7</sup> gold context as evidence. To create negative examples, we create a "No"-answer without evidence. This results in 4,074 training, 1,764 validation, and 2,499 test question-answer pairs.
Training and optimization. We use code and pretrained weights from Dasigi et al. (2021), which we finetune on our derived QA pairs using a joint answer-evidence loss (and within-batch loss scaling for class imbalance) from the original work. We train using Adam (Kingma and Ba, 2015) for up to 5 epochs. We use the validation set for early stopping and grid search over batch sizes 2, 4, 8 or 16 and max learning rate of 3e-5 or 5e-5.
Evaluation. Following Dasigi et al. (2021), we evaluate the model using the Answer-F1 and Evidence-F1 metrics defined in that work, where Answer-F1 captures correctness of the generated
Answer F1	Evidence F1
Majority SL	0.61	0.48
Majority ML	0.72	0.48
Qasper	0.75	0.48
Table 4: Test results of the Qasper model finetuned on document-level QA examples from MULTICITE.
answer span and Evidence-F1 captures performance in extracting gold citation context sentences.
Baselines. We compare our model to two heuristic baselines. In MAJORITY SINGLE LABEL (SL), we always predict "Yes" for the majority class—Background. In MAJORITY MULTIPLE LABELS (ML), we predict for each of the seven question templates the majority answer ("Yes" or "No").
We present single run results on the test set in Table 4. While the Qasper model outperforms the heuristics at assessing citing-cited paper relationships (Answer F1), the gains from supervision are minor, and citation contexts retrieval (Evidence F1) does not outperform simple heuristics. This highlights the difficulty in this document-level understanding task and the potential for future research into powerful models capable of higher-level CCA.
Aiming to inspire novel research in CCA, we have acknowledged the existence of three understudied phenomena, and present MULTICITE, a novel resource that is both the largest corpus of citation contexts to-date and captures all three phenomena. We employ MULTICITE in three experiments demonstrating the importance of these phenomena, establishing strong baselines, and showcasing ways of conducting novel CCA. In the future, we intend to increase diversity of the dataset by extending it to cover more disciplines of science. We make all code and data publicly available at https://github.com/allenai/multicite.
This work was supported in part by NSF Convergence Accelerator Award #2132318. We thank Sam Skjonsberg and Mark Neumann for their help on the annotation interface. We would also like to thank Noah Smith for helping us connect with collaborators on this project and the anonymous reviewers for their insightful feedback.
This follows from Dasigi et al. (2021)'s approach to breaking ties between multiple valid evidences.
Amjad Abu-Jbara, Jefferson Ezra, and Dragomir Radev. 2013. Purpose and polarity of citation: Towards nlp-based bibliometrics. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 596-606. Association for Computational Linguistics (ACL).
Awais Athar. 2011. Sentiment Analysis of Citations Using Sentence Structure-based Features. In Proceedings of the ACL 2011 Student Session, HLT-SS '11, pages 81-87, Stroudsburg, PA, USA. Association for Computational Linguistics.
Awais Athar and Simone Teufel. 2012. Context-enhanced citation sentiment detection. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 597-601, Montreal, Canada. Association for Computational Linguistics.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciBERT: A pretrained language model for scientific text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv:2004.05150.
Arman Cohan, Waleed Ammar, Madeleine van Zuylen, and Field Cady. 2019. Structural scaffolds for citation intent classification in scientific publications. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics. Association for Computational Linguistics.
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 615-621, New Orleans, Louisiana. Association for Computational Linguistics.
Arman Cohan and Nazli Goharian. 2015. Scientific article summarization using citation-context and article's discourse structure. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 390-400, Lisbon, Portugal. Association for Computational Linguistics.
Marilena Daquino, Silvio Peroni, David Shotton, Giovanni Colavizza, Behnam Ghavimi, Anne Lauscher, Philipp Mayr, Matteo Romanello, and Philipp Zumstein. 2020. The opencitations data model. In *The
Semantic Web - ISWC 2020, pages 447-463, Cham. Springer International Publishing.
Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. A dataset of information-seeking questions and answers anchored in research papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4599-4610, Online. Association for Computational Linguistics.
Franck Dernoncourt and Ji Young Lee. 2017. Pubmed 200k RCT: a dataset for sequential sentence classification in medical abstracts. CoRR, abs/1710.06071.
Cailing Dong and Ulrich Schäfer. 2011. Ensemble-style self-training on citation classification. In Proceedings of the 5th International Joint Conference on Natural Language Processing, pages 623-631. Association for Computational Linguistics.
Aaron Elkiss, Siwei Shen, Anthony Fader, Güneş Erkan, David States, and Dragomir Radev. 2008. Blind men and elephants: What do citation summaries tell us about a research article? J. Am. Soc. Inf. Sci. Technol., 59(1):51-62.
Eugene Garfield. 1955. Citation indexes for science. Science, 122(3159):108-111.
Eugene Garfield, Irving H Sher, and Richard J Torpie. 1964. The use of citation data in writing the history of science. Technical report, Institute for Scientific Information Inc Philadelphia PA.
Eugene Garfield et al. 1970. Citation indexing for studying science. Nature, 227(5259):669-671.
Mark Garzone and Robert E Mercer. 2000. Towards an automated citation classifier. In *Conference of the Canadian Society for computational studies of intelligence*, pages 337-346. Springer.
Myriam Hernández-Alvarez and José Gomez. 2016. Survey about citation context analysis: Tasks, techniques, and resources. *Natural Language Engineering*, 22(3):327-349.
Sehrish Iqbal, Saeed-Ul Hassan, Naif Radi Aljohani, Salem Alelyani, Raheel Nawaz, and Lutz Bornmann. 2020. A decade of in-text citation analysis based on natural language processing and machine learning techniques: An overview of empirical studies. arXiv preprint arXiv:2008.13020.
Rahul Jha, Amjad-Abu Jbara, Vahed Qazvinian, and Dragomir R. Radev. 2016. NLP-driven citation analysis for scientometrics. Natural Language Engineering, pages 1-38.
Charles Jochim and Hinrich Schütze. 2012. Towards a Generic and Flexible Citation Classifier Based on a Faceted Classification Scheme. In Proceedings of the 24th International Conference on Computational Linguistics (COLING 2012).
David Jurgens, Srijan Kumar, Raine Hoover, Dan McFarland, and Dan Jurafsky. 2018. Measuring the evolution of a scientific field through citation frames. Transactions of the Association for Computational Linguistics, 6:391-406.
Dain Kaplan, Ryu Iida, and Takenobu Tokunaga. 2009. Automatic extraction of citation contexts for research paper summarization: A coreference-chain based approach. In Proceedings of the 2009 Workshop on Text and Citation Analysis for Scholarly Digital Libraries (NLPIR4DL), pages 88-95, Suntec City, Singapore. Association for Computational Linguistics.
Dain Kaplan, Takenobu Tokunaga, and Simone Teufel. 2016. Citation block determination using textual coherence. Journal of Information Processing, 24(3):540-553.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, Conference Track Proceedings, San Diego, CA, USA.
Anne Lauscher, Goran Glavaš, and Kai Eckert. 2017a. University of mannheim@ clscisumm-17: Citation-based summarization of scientific articles using semantic textual similarity. In CEUR workshop proceedings, volume 2002, pages 33-42. RWTH.
Anne Lauscher, Goran Glavaš, Simone Paolo Ponzetto, and Kai Eckert. 2017b. Investigating convolutional networks and domain-specific embeddings for semantic classification of citations. In Proceedings of the 6th International Workshop on Mining Scientific Publications, WOSP 2017, page 24-28, New York, NY, USA. Association for Computing Machinery.
Anne Lauscher, Yide Song, and Kiril Gashteovski. 2019. Minscie: citation-centered open information extraction. In 2019 ACM/IEEE Joint Conference on Digital Libraries (JCDL), pages 386-387. IEEE.
Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. 2020. S2ORC: The semantic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4969-4983, Online. Association for Computational Linguistics.
Yann Mathet, Antoine Widlocher, and Jean-Philippe Métivier. 2015. The unified and holistic method gamma  $(\hat{1}^3)$  for inter-annotator agreement measure and alignment. Computational Linguistics, 41(3):437-479.
Kathy McKeown, Hal Daume, Snigdha Chaturvedi, John Paparrizos, Kapil Thadani, Pablo Barrio, Or Biran, Suvarna Bothe, Michael Collins, Kenneth R. Fleischmann, Luis Gravano, Rahul Jha, Ben King, Kevin McInerney, Taesun Moon, Arvind Neelakantan, Diarmuid O'Seaghdha, Dragomir Radev, Clay Templeton, and Simone Teufel. 2016. Predicting the impact of scientific concepts using full-text features. J. Assoc. Inf. Sci. Technol., 67(11):2684-2696.
Mark Neumann, Daniel King, Iz Beltagy, and Waleed Ammar. 2019. ScispaCy: Fast and robust models for biomedical natural language processing. In Proceedings of the 18th BioNLP Workshop and Shared Task, pages 319-327, Florence, Italy. Association for Computational Linguistics.
David Pride and Petr Knoth. 2020. An authoritative approach to citation classification. In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries (JCDL). Association for Computing Machinery.
Vahed Qazvinian and Dragomir R. Radev. 2008. Scientific paper summarization using citation summary networks. In Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 689-696, Manchester, UK. Coling 2008 Organizing Committee.
Kumar Ravi, Srirangaraj Setlur, Vadlamani Ravi, and Venu Govindaraju. 2018. Article citation sentiment analysis using deep learning. In 2018 IEEE 17th International Conference on Cognitive Informatics & Cognitive Computing (ICCI* CC), pages 78-85. IEEE.
Advaith Siddharthan and Simone Teufel. 2007. Whose idea was this, and why does it matter? attributing scientific work to citations. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, pages 316-323, Rochester, New York. Association for Computational Linguistics.
John Swales. 1986. Citation analysis and discourse analysis. Applied linguistics, 7(1):39-56.
Simone Teufel. 2014. Scientific argumentation detection as limited-domain intention recognition. In *ArgNLP*.
Simone Teufel, Advaith Siddharthan, and Dan Tidhar. 2006. Automatic Classification of Citation Function. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, EMNLP '06, pages 103-110, Stroudsburg, PA, USA. Association for Computational Linguistics.
Suppawong Tuarob, Sung Woo Kang, Poom Wettayakorn, Chanatip Pornprasit, Tanakitti Sachati, Saeed-Ul Hassan, and Peter Haddawy. 2019. Automatic classification of algorithm citation functions in scientific literature. IEEE Transactions on Knowledge and Data Engineering.
Marco Valenzuela, Vu Ha, and Oren Etzioni. 2015. Identifying meaningful citations. In AAAI workshop: Scholarly big data.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.