SlowGuess/NAACL_2022_OCR huggingface dataset:
reasonable because dominant spans are often full sentences anyway. In contrast, the average score of reference spans is significantly higher than that of sentences containing reference-type citations; reference spans are shorter and contain highly concentrated key information derived from their cited papers. Thus, using CORWA citation spans as queries for ROUGE-based cited sentence retrieval is superior for reference-type citations and comparable for dominant-type citations.
5.2 Span-based Related Work Generation
Existing neural network-based, abstractive related work generation systems generate citation sentences given the surrounding context sentences (Xing et al., 2020; Ge et al., 2021; Luu et al., 2021) or generate entire paragraphs containing multiple citations (Chen et al., 2021). These task settings neglect the fact that the citation text corresponding to a cited paper is not necessarily in the form of a sentence, but could be a portion of a sentence or a block of multiple sentences. Our span-based annotation scheme identifies the citation tokens that are directly derived from the cited papers.
As Figure 6 shows, reference spans are not full sentences, while dominant spans can cover multiple sentences. For reference-type citations, using a full sentence as the generation target includes potentially unrelated tokens outside the citation span that do not refer to the cited paper. For dominant-type citations, using a single sentence as the generation target can result in 1) information loss when not all sentences describing the cited paper are included in the target, and the model never learns to generate them, or 2) information leak when sentences that actually describe the cited pa
per are used as context sentences instead of target sentences. Thus, we propose a span-level citation text generation task and present a pilot study using a Longformer-Encoder-Decoder (LED) (Beltagy et al., 2020) baseline model.
5.2.1 Experimental Setting
The common Transformer-based language models (Devlin et al., 2019; Liu et al., 2019; Lewis et al., 2020; Raffel et al., 2020) have a limited input window size (typically 512 or 1024 tokens), which presents a major challenge for tasks like related work generation that use multiple long documents as inputs. LED (Beltagy et al., 2020) addresses this challenge by using a local self-attention mechanism, rather than global self-attention, handling in input context windows of up to 16k tokens. We present an LED-based baseline model for the citation span generation task.
We first pretrain the LED-base model on the masked language modeling (MLM) task (Devlin et al., 2019) using related work sections from S2ORC papers in the computer science domain, as well as on the cross-document language modeling (CDLM) task (Caciularu et al., 2021), which aligns masked citation sentences with their context sentences and the full text of their cited papers. We further pretrain the LED encoder with the three CORWA sub-tasks (Supplementary Table 6). All pretraining strictly excludes the texts from test set.
For the citation span generation task, we input the concatenation of {the target paper's introduction (following Luu et al. (2021)), the partial related work paragraph excluding the target citation span, and the concatenation of {explicit citation mark, title, and abstract} of each cited paper in the target span}; the generation target is the ground truth citation span from CORWA. We provide the explicit citation mark (e.g. Devlin et al., 2018) because it is simple to extract but cannot be inferred from the paper text alone. Just as a human reader may remember the content of the frequently cited papers or the research topics of frequently cited authors, so the citation mark tokens may carry information about the cited paper and its authors.
In addition to the CORWA training set, we use the distantly supervised labels predicted by our joint related work tagger (§4.2) for training. We use the default hyper-parameters of the Huggingface LED implementation (Wolf et al., 2020).
<sup>6</sup>We indicate whether the target span is dominant or refer
Models	Dominant	Reference
R-1	R-2	R-L	R-1	R-2	R-L
LED-base w/o pretrain	0.220	0.060	0.183	0.228	0.091	0.223
LED-base Span	0.230	0.062	0.186	0.244	0.107	0.240
LED-base Sentence	0.244	0.075	0.202	0.193	0.050	0.151
Table 4: Performance of citation span/sentence generation using LED-base (Beltagy et al., 2020). Citation marks are excluded from the scores since they are trivial to generate and bring up the scores unintentionally. Note that the performance of span/sentence generations are NOT directly comparable due to different generation targets.
Flu.	Rel.	Coh.	Overall
Dominant
Gold Span	4.61	3.53	4.17	3.64
Span	4.92	4.07	4.20	3.99
Sentence	4.83	4.03	4.17	4.02
Reference
Gold Span	4.87	4.04	4.18	4.00
Span	4.68	4.24	4.26	3.96
Sentence	4.86	3.64	4.09	3.70
Table 5: Average fluency, relevance, coherence and overall scores, rated by human judges.
As Table 4 shows, the ROUGE scores of our LED-base models for citation span/sentence generation are similar to previous sentence-level citation text generation models (Xing et al., 2020; Ge et al., 2021), and our pretraining improves the citation span generation performance. Compared to sentence-level generation, span-level generation has lower scores for dominant citations, but higher scores for reference citations. However, because the span- and sentence-level tasks have different generation targets, their scores cannot be directly compared.
We perform a human evaluation following the setting of Xing et al. (2020); Ge et al. (2021). We sample 15 instances each for dominant and reference citations and compare their corresponding span- and sentence-based generation outputs, as well as the gold spans from the original related work sections. Each citation text is rated by three NLP graduate students who are fluent in English on a 1 (very poor) to 5 (excellent) point scale, with respect to four aspects: fluency (whether a citation span/sentence is fluent), relevance (whether a citation span/sentence is relevant to the cited paper(s)), coherence (whether a citation span/sentence is coherent within its context), and overall quality.
Table 5 shows human evaluation results, with moderate inter-annotator agreement (Kendall's  $\tau$  of 0.298, 0.205, and 0.172 among three annotators). All citation texts are judged to be highly fluent.
Interestingly, in previous studies (Xing et al.,
2020; Ge et al., 2021) the scores of gold sentences are higher than those of generated texts, but our gold spans have a significantly lower relevance scores than the generated spans. This is likely because the gold spans contain information derived from the body sections of the cited papers, which are not provided to either the models or to the human judges. As a result, some gold spans appear to be irrelevant to the human judges, echoing our earlier finding in §5.1 that citation spans contain more focused information. This observation also suggests that gold citation spans are not necessarily the best target for all task settings.
We also see that, while dominant sentences and spans receive similar scores, the reference sentences have lower relevance scores than the spans. This result makes sense because reference citation spans are short and focused, so the full sentences include tokens unrelated to the cited paper(s). Overall, the generated spans are rated slightly higher than the generated sentences by the human judges, confirming that span-level citation text generation is preferable to sentence-level generation.
6 Toward Full Related Work Generation
Existing extractive related work generation systems (Hoang and Kan, 2010; Hu and Wan, 2014; Chen and Zhuge, 2019; Wang et al., 2019) select sentences from the target paper and/or the cited papers, which can be concatenated to form a full related work section; neural network-based, abstractive related work generation systems generate individual citation sentences (Xing et al., 2020; Ge et al., 2021; Luu et al., 2021) or paragraphs (Chen et al., 2021). However, none of these prior works address the ordering of the extracted/generated sentences or the grouping of sentences into paragraphs, nor are they able to produce rhetorical sentences to smooth the transitions between citations. No prior work bridges the gap from generating individual citation texts to generating a full related work section.
We suggest a bottom-up, iterative approach to generate full related work sections. The process would begin with generating citation spans under
ence type, as well as the type of each citation in the span.
the settings proposed in §5.2. Then, multiple generated citation spans would be aggregated and rewritten into citation text blocks in either the summarization or narrative style. These blocks would be further aggregated and rewritten into paragraphs by generating transition and reflection sentences.
Generating and rewriting in this pipeline fashion has the following benefits: (1) It mitigates the practical issue of computational resource limitations, given that state-of-the-art models do not perform well on long text generation. (2) The auxiliary inputs, such as citation functions or discourse tags, may vary for each stage of generation. (3) As a practical system to assist researchers, it is crucial to allow user involvement in the iterative generation process. Due to the large search space, consisting of multiple valid related work section candidates with different writing styles, it is extremely challenging to precisely generate a satisfying text with a one-shot, end-to-end system. A human-in-the-loop approach allows the user to significantly prune the search space and simultaneously reduces the error-propagation issue caused by the pipeline design.
7 Other Related Tasks
7.1 Scientific Document Understanding
Besides summarization, scientific document understanding also plays an important role in related work generation.
Citation Analysis. Citations are the core of related work sections. There has been a line of research on citation analysis, including citation function (Teufel et al., 2006; Dong and Schäfer, 2011; Jurgens et al., 2018; Tuarob et al., 2019), citation intent (Cohan et al., 2019; Lauscher et al., 2021), citation sentiment (Athar, 2011; Athar and Teufel, 2012; Ravi et al., 2018; Vyas et al., 2020), etc. These studies annotate citations with different labeling schemes to study the various usages and purposes of citations.
Discourse Analysis. Scientific discourse analysis studies the rhetorical components of clauses, sentences, or text spans that are not limited to citations, uncovering how authors persuade expert readers with their claims. There is a significant amount of prior work proposing discourse schemes and developing models for discourse tagging for scientific articles (Teufel and Moens, 1999, 2002; Hirohata et al., 2008; Liakata, 2010; Liakata et al., 2012; Guo et al., 2010; De Waard and Maat, 2012;
Burns et al., 2016; Dernoncourt and Lee, 2017; Huang et al., 2020; Li et al., 2021a).
Our CORWA discourse tagging task focuses on distinguishing the source of the information in each related work sentence, which is complementary to the discourse tagging work listed above.
7.2 Cited Text Span
AbuRa'ed et al. (2020) extend Hoang and Kan (2010)'s RWSData dataset by annotating the Cited Text Span (CTS) (Wang et al., 2019). They annotate the specific sentences in cited papers that each citation in the target paper is based on. For each cited paper, they further collect a set of papers that co-cite this cited paper. Jaidka et al. (2018, 2019) propose the CL-Scisumm shared task, which includes identifying the CTS in reference papers for each citation instance. This shared task provides a valuable dataset for the precise generation of citation texts from a CTS, in contrast to most recent work, which uses the cited paper's abstract or introduction.
7.3 Studies of Literature Reviews
From an information studies perspective, Khoo et al. (2011) largely classify literature reviews into two styles: integrative and descriptive. Descriptive literature reviews summarize individual studies and provide detailed information on each, such as methods, results, and interpretation; integrative literature reviews provide fewer details of individual studies, instead focusing on synthesizing ideas and results extracted from these papers. Jaidka et al. (2010, 2011, 2013) analyze the properties of these two types of literature reviews.
We present the CORWA dataset of three interrelated annotation tasks: discourse tagging, citation span detection, and citation type recognition. We demonstrate the significance of CORWA with analyses from multiple perspectives, such as writing style and discourse patterns. We propose a strong baseline model that can automatically propagate the CORWA annotation scheme to massive unlabeled related work sections. Furthermore, we show that citation spans are a better alternative to citation sentences for both the relevant sentence retrieval and citation generation tasks. Finally, we discuss a novel framework for human-in-the-loop iterative abstractive related work generation.
Ahmed AbuRa'ed, Horacio Saggion, and Luis Chiruzzo. 2020. A multi-level annotated corpus of scientific papers for scientific document summarization and cross-document relation discovery. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 6672-6679, Marseille, France. European Language Resources Association.
Awais Athar. 2011. Sentiment analysis of citations using sentence structure-based features. In Proceedings of the ACL 2011 student session, pages 81-87.
Awais Athar and Simone Teufel. 2012. Context-enhanced citation sentiment detection. In Proceedings of the 2012 conference of the North American chapter of the Association for Computational Linguistics: Human language technologies, pages 597-601.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciBERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3615-3620, Hong Kong, China. Association for Computational Linguistics.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv:2004.05150.
Gully APC Burns, Pradeep Dasigi, Anita de Waard, and Eduard H Hovy. 2016. Automated detection of discourse segment and experimental types from the text of cancer pathway results sections. Database, 2016.
Avi Caciularu, Arman Cohan, Iz Beltagy, Matthew E Pe- ters, Arie Cattan, and Ido Dagan. 2021. CdIm: Crossdocument language modeling. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2648-2662.
Ziqiang Cao, Furu Wei, Li Dong, Sujian Li, and Ming Zhou. 2015. Ranking with recursive neural networks and its application to multi-document summarization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29.
Rich Caruana. 1997. Multitask learning. Machine learning, 28(1):41-75.
Jingqiang Chen and Hai Zhuge. 2019. Automatic generation of related work through summarizing citations. Concurrency and Computation: Practice and Experience, 31(3):e4261.
Xiuying Chen, Hind Alamro, Mingzhe Li, Shen Gao, Xiangliang Zhang, Dongyan Zhao, and Rui Yan. 2021. Capturing relations between scientific papers: An abstractive model for related work section generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the
11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6068-6077, Online. Association for Computational Linguistics.
Arman Cohan, Waleed Ammar, Madeleine van Zuylen, and Field Cady. 2019. Structural scaffolds for citation intent classification in scientific publications. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3586-3596, Minneapolis, Minnesota. Association for Computational Linguistics.
Anita De Waard and Henk Pander Maat. 2012. Epistemic modality and knowledge attribution in scientific discourse: A taxonomy of types and overview of features. In Proceedings of the Workshop on Detecting Structure in Scholarly Discourse, pages 47-55. Association for Computational Linguistics.
Franck Dernoncourt and Ji Young Lee. 2017. Pubmed 200k RCT: a dataset for sequential sentence classification in medical abstracts. CoRR, abs/1710.06071.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Cailing Dong and Ulrich Schäfer. 2011. Ensemble-style self-training on citation classification. In Proceedings of 5th international joint conference on natural language processing, pages 623-631.
Yubin Ge, Ly Dinh, Xiaofeng Liu, Jinsong Su, Ziyao Lu, Ante Wang, and Jana Diesner. 2021. BACO: A background knowledge- and content-based framework for citing sentence generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1466–1478, Online. Association for Computational Linguistics.
Yufan Guo, Anna Korhonen, Maria Liakata, Ilona Silins Karolinska, Lin Sun, and Ulla Stenius. 2010. Identifying the information structure of scientific abstracts: an investigation of three different schemes. In Proceedings of the 2010 Workshop on Biomedical Natural Language Processing, pages 99-107. Association for Computational Linguistics.
Jiawei Han, Jian Pei, Behzad Mortazavi-Asl, Helen Pinto, Qiming Chen, Umeshwar Dayal, and Meichun Hsu. 2001. Prefixspan: Mining sequential patterns efficiently by prefix-projected pattern growth. In proceedings of the 17th international conference on data engineering, pages 215-224. CiteSeer.
Kenji Hirohata, Naoaki Okazaki, Sophia Ananiadou, and Mitsuru Ishizuka. 2008. Identifying sections in scientific abstracts using conditional random fields. In Proceedings of the Third International Joint Conference on Natural Language Processing: Volume-I.
Cong Duy Vu Hoang and Min-Yen Kan. 2010. Towards automated related work summarization. In *Coling* 2010: Posters, pages 427-435.
Yue Hu and Xiaojun Wan. 2014. Automatic generation of related work sections in scientific papers: an optimization approach. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1624-1633.
Ting-Hao Kenneth Huang, Chieh-Yang Huang, Chien-Kuang Cornelia Ding, Yen-Chia Hsu, and C. Lee Giles. 2020. CODA-19: Using a non-expert crowd to annotate research aspects on  $10,000+$  abstracts in the COVID-19 open research dataset. In Proceedings of the 1st Workshop on NLP for COVID-19 at ACL 2020, Online. Association for Computational Linguistics.
Kokil Jaidka, Muthu Kumar Chandrasekaran, Sajal Rustagi, and Min-Yen Kan. 2018. Insights from cl-scisumm 2016: the faceted scientific document summarization shared task. International Journal on Digital Libraries, 19(2):163-171.
Kokil Jaidka, Christopher Khoo, and Jin-Cheon Na. 2010. Imitating human literature review writing: An approach to multi-document summarization. In International Conference on Asian Digital Libraries, pages 116-119. Springer.
Kokil Jaidka, Christopher SG Khoo, and Jin-Cheon Na. 2013. Literature review writing: how information is selected and transformed. In Aslib Proceedings. Emerald Group Publishing Limited.
Kokil Jaidka, Michihiro Yasunaga, Muthu Kumar Chandrasekaran, Dragomir Radev, and Min-Yen Kan. 2019. The cl-scisumm shared task 2018: Results and key insights. arXiv preprint arXiv:1909.00764.
Kokil Jaidka Jaidka, Christopher Khoo Khoo, and Jin-Cheon Na Na. 2011. Literature review writing: a study of information selection from cited papers/kokil jaidka, Christopher khoo and jin-cheon na.
David Jurgens, Srijan Kumar, Raine Hoover, Dan McFarland, and Dan Jurafsky. 2018. Measuring the evolution of a scientific field through citation frames. Transactions of the Association for Computational Linguistics, 6:391-406.
Min-Yen Kan, Kathleen R. McKeown, and Judith L. Klavans. 2001. Applying natural language generation to indicative summarization. In Proceedings of the ACL 2001 Eighth European Workshop on Natural Language Generation (EWNLG), Toulouse, France. Association for Computational Linguistics.
Christopher SG Khoo, Jin-Cheon Na, and Kokil Jaidka. 2011. Analysis of the macro-level discourse structure of literature reviews. Online Information Review.
Anne Lauscher, Brandon Ko, Bailey Kuhl, Sophie Johnson, David Jurgens, Arman Cohan, and Kyle Lo. 2021. Multicite: Modeling realistic citations requires moving beyond the single-sentence single-label setting. arXiv preprint arXiv:2107.00414.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Chun Li and Jianyong Wang. 2008. Efficiently mining closed subsequences with gap constraints. In Proceedings of the 2008 SIAM International Conference on Data Mining, pages 313-322. SIAM.
Xiangci Li, Gully Burns, and Nanyun Peng. 2021a. Scientific discourse tagging for evidence extraction. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2550-2562.
Xiangci Li, Gully A Burns, and Nanyun Peng. 2021b. A paragraph-level multi-task learning model for scientific fact-verification. In SDU@AAAI.
Xiangci Li and Jessica Ouyang. 2022. Automatic related work generation: A meta study. arXiv preprint arXiv:2201.01880.
Maria Liakata. 2010. Zones of conceptualisation in scientific papers: a window to negative and speculative statements. In Proceedings of the Workshop on Negation and Speculation in Natural Language Processing, pages 1-4. Association for Computational Linguistics.
Maria Liakata, Shyamasree Saha, Simon Dobnik, Colin Batchelor, and Dietrich Rebholz-Schuhmann. 2012. Automatic recognition of conceptualization zones in scientific articles and two life science applications. Bioinformatics, 28(7):991-1000.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. 2020. S2ORC: The semantic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4969-4983, Online. Association for Computational Linguistics.