SlowGuess/NAACL_2022_OCR huggingface dataset:
Kelvin Luu, Xinyi Wu, Rik Koncel-Kedzierski, Kyle Lo, Isabel Cachola, and Noah A. Smith. 2021. Explaining relationships between scientific documents. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2130-2144, Online. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Kumar Ravi, Srirangaraj Setlur, Vadlamani Ravi, and Venu Govindaraju. 2018. Article citation sentiment analysis using deep learning. In 2018 IEEE 17th International Conference on Cognitive Informatics & Cognitive Computing (ICCI* CC), pages 78-85. IEEE.
Erik F Sang and Jorn Veenstra. 1999. Representing text chunks. In Proceedings of the ninth conference on European chapter of the Association for Computational Linguistics, pages 173-179. Association for Computational Linguistics.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083, Vancouver, Canada. Association for Computational Linguistics.
Pontus Stenetorp, Sampo Pyysalo, Goran Topic, Tomoko Ohta, Sophia Ananiadou, and Jun'ichi Tsujii. 2012. Brat: a web-based tool for nlp-assisted text annotation. In Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 102-107.
Simone Teufel and Marc Moens. 1999. Discourse-level argumentation in scientific articles: human and automatic annotation. Towards Standards and Tools for Discourse Tagging.
Simone Teufel and Marc Moens. 2002. Summarizing scientific articles: experiments with relevance and rhetorical status. Computational linguistics, 28(4):409-445.
Simone Teufel, Advaith Siddharthan, and Dan Tidhar. 2006. Automatic classification of citation function. In Proceedings of the 2006 conference on empirical methods in natural language processing, pages 103-110.
Suppawong Tuarob, Sung Woo Kang, Poom Wettayakorn, Chanatip Pornprasit, Tanakitti Sachati, Saeed-Ul Hassan, and Peter Haddawy. 2019. Automatic classification of algorithm citation functions in scientific literature. IEEE Transactions on Knowledge and Data Engineering, 32(10):1881-1896.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Vishal Vyas, Kumar Ravi, Vadlamani Ravi, V Uma, Srirangaraj Setlur, and Venu Govindaraju. 2020. Article citation study: Context enhanced citation sentiment detection. arXiv preprint arXiv:2005.04534.
Panchcheng Wang, Shasha Li, Haifang Zhou, Jintao Tang, and Ting Wang. 2019. Toc-rwg: Explore the combination of topic model and citation information for automatic related work generation. IEEE Access, 8:13043-13055.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Xinyu Xing, Xiaosheng Fan, and Xiaojun Wan. 2020. Automatic generation of citation texts in scholarly papers: A pilot study. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6181-6190.
Michihiro Yasunaga, Jungo Kasai, Rui Zhang, Alexander R Fabbri, Irene Li, Dan Friedman, and Dragomir R Radev. 2019. Scisummnet: A large annotated corpus and content-impact models for scientific paper summarization with citation networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7386-7393.
Michihiro Yasunaga, Rui Zhang, Kshitijh Meelu, Ayush Parek, Krishnan Srinivasan, and Dragomir Radev. 2017. Graph-based neural multi-document summarization. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 452-462.
A.1 Training Configurations
For the joint related work tagger training, we use GeForce GTX 1080 11 GB GPUs. The training process lasts 2.5 hours on a single GPU using Huggingface's (Wolf et al., 2020) SciBERT, BERT-base or Roberta-base as the paragraph encoders, and it lasts 6.5 hours using LED-base encoder. We train the models for 15 epochs. It takes approximately one week to run the hyper-parameter search using five-fold cross-validation for all language models, using 8 GPUs in total.
For training the citation span generation model, we use Tesla V100s-PCIE-32GB GPUs. The training process lasts for 2 days on a single GPU. We run the training for a maximum of 3 epochs with early stopping based on the validation loss.
A.2 Ethical Considerations
We present a new dataset that is derived from the S2ORC dataset (Lo et al., 2020), which is released under CC BY-NC 2.0 license. The Huggingface models (Wolf et al., 2020) we develop upon are released under Apache License 2.0.
Our annotators were compensated for their work at a rate of double the minimum wage in our local area.
Models	Five-fold cross-validation scores	Test-set scores
Disc	CT	CS	Disc	CT	CS
SciBERT (Beltagy et al., 2019)	0.900 (0.0099)	0.961 (0.0038)	0.926 (0.0059)	0.898	0.959	0.930
Roberta-base (Liu et al., 2019)	0.886 (0.0050)	0.956 (0.0036)	0.922 (0.0048)	0.885	0.956	0.929
BERT-base (Devlin et al., 2019)	0.879 (0.0070)	0.954 (0.0055)	0.910 (0.0064)	0.875	0.952	0.915
LED-base (Pretrained)	0.872 (0.0253)	0.948 (0.0117)	0.905 (0.0088)	0.869	0.910	0.907
LED-base (Beltagy et al., 2020)	0.865 (0.0090)	0.922 (0.0128)	0.907 (0.0074)	0.842	0.874	0.909
Table 6: Micro-F1 scores for the joint related work tagger using different language models as the encoder. The tasks are discourse tagging (Disc), citation type recognition (CT), and citation span detection (CS). Five-fold cross-validation scores are reported as the mean (standard deviation) across all folds. The pretraining of LED is explained in §5.2.1.
Discourse Subsequence
transition, narrative_cite, single_summ
Functionalities
Introducing an approach and providing background knowledge.
Examples
1. Joint POS tagging with parsing is not a new idea.
2. In PCFG-based parsing (Collins, 1999; Charniak, 2000; Petrov et al., 2006), POS tagging is considered as a natural step of parsing by employing lexical rules.
3. For transition-based parsing, Hatori et al. (2011) proposed to integrate POS tagging with dependency parsing.
Discourse Subsequence
single_summ, reflection
Functionalities
Comparing the prior work to the current work.
Examples
1. Haghighi et al. (2009) confirm and extend these results, showing BLEU improvement for a hierarchical phrase-based MT system on a small Chinese corpus.
2. As opposed to ITG, we use a linguistically motivated phrase-structure tree to drive our search and inform our model.
Discourse Subsequence
reflection, single_summ
Functionalities
Supporting the current work with a previous work.
Examples
1. Our baseline semi-supervised model can be viewed as an extension of these approaches to a reading comprehension setting.
2. Dai et al. (2015) also explore initialization from a language model, but find that the recurrent autoencoder is superior, which is why we do not consider language models in this work.
Discourse Subsequence
transition, narrative_cite, transition
Functionalities
Topic sentence, narration of prior work followed by critique.
Examples
1. Traditional work on relation classification can be categorized into feature-based methods and kernel-based methods.
2. The former relies on a large number of human-designed features (Zhou et al., 2005; Jiang and Zhai, 2007; Li and Ji, 2014) while the latter leverages various kernels to implicitly explore a much larger feature space (Bunescu and Mooney, 2005; Nguyen et al., 2009).
3. However, both methods suffer from error propagation problems and poor generalization abilities on unseen words.
Table 7: Frequent discourse label subsequences detected by applying PrefixSpan (Han et al., 2001) and Gap-Bide algorithm (Li and Wang, 2008).
Discourse Subsequence
single_summ, single_summ, transition
Commenting previous works summarized.
1. Walker et al. (2012) extract rules representing characters from their annotated movie subtitle corpora.
2. Miyazaki et al. (2015) propose a method of converting utterances using rewriting rules automatically derived from a Twitter corpus.
3. These approaches have a fundamental problem to need some manual annotations, which is a main issue to be solved in this work.
narrative_cite, transition, single_summ
Criticizing the previously cited work and citing an improved work.
1. There have also been several classical studies based on nonneural approaches to headline generation (Woodsend et al., 2010; Alfonseca et al., 2013; Colmenares et al., 2015),
but they basically addressed sentence compression after extracting important linguistic units such as phrases.
2. In other words, their methods can still yield erroneous output, although they would be more controllable than neural models.
3. One exception is the work of Alotaiby (2011), where fixed-sized substrings were considered for headline generation.
Describing an idea following by a comment and then citations implementing the idea.
1. One of the classes of errors in the Helping Our Own (HOO) 2011 shared task (Dale and Kilgarriff, 2011) was punctuation.
2. Comma errors are the most frequent kind of punctuation error made by learners.
3. Israel et al. (2012) present a model for detecting these kinds of errors in learner texts.
Table 8: Frequent discourse label subsequences detected by applying PrefixSpan (Han et al., 2001) and Gap-Bide algorithm (Li and Wang, 2008), continued.