Venue: NAACL 2021

# On the Use of Context for Predicting Citation Worthiness of Sentences in Scholarly Articles

Rakesh Gosangi, Ravneet Arora, Mohsen Gheisarieha, Debanjan Mahata, Haimin Zhang

Bloomberg, New York

{rgosangi,rarora62,mgheisarieha,dmahata}@bloomberg.net, raymondzhang@gmail.com

###### Abstract

In this paper, we study the importance of context in predicting the citation worthiness of sentences in scholarly articles. We formulate this problem as a sequence labeling task solved using a hierarchical BiLSTM model. We contribute a new benchmark dataset containing over two million sentences and their corresponding labels. We preserve the sentence order in this dataset and perform document-level train/test splits, which importantly allows incorporating contextual information in the modeling process. We evaluate the proposed approach on three benchmark datasets. Our results quantify the benefits of using context and contextual embeddings for citation worthiness. Lastly, through error analysis, we provide insights into cases where context plays an essential role in predicting citation worthiness.

+
Footnote †: _Corresponding of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4539–4545

## 1 Introduction and Background

Citation worthiness is an emerging research topic in the natural language processing (NLP) domain, where the goal is to determine if a sentence in a scientific article requires a citation1. This research has potential applications in citation recommendation systems Strohman et al. (2007); Kucuktunc et al. (2014); He et al. (2010), and is also useful for scientific publishers to regularize the citation process. Providing appropriate citations is critical to scientific writing because it helps readers understand how the current work relates to existing research.

Footnote 1: For example, in the first excerpt in Table 1, the goal is to predict that the first, third, and fourth sentences would require citations but the second does not.

Citation worthiness was first introduced by Sugiyama et al. (2010), where the authors formulated as a sentence-level binary classification task solved using classical machine learning techniques like Support Vector Machines (SVMs). Subsequent works from Farber et al. (2018); Bonab et al. (2018) use similar approach but employ deep learning models like Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). More recently, Zeng et al. Zeng and Acuna (2020) proposed a Bidirectional Long short-term memory (BiLSTM) based architecture and demonstrated that context, specifically the two adjacent sentences, can help improve the prediction of citation worthiness.

Citation worthiness is closely related to citation recommendation (suggest a reference for a sentence in a scientific article), which is often approached as a ranking problem solved using models that combine textual, contextual, and document-level features Strohman et al. (2007); He et al. (2010). More recent works employ deep learning models Huang et al. (2015); Ebesu and Fang (2017) and personalization Cai et al. (2018); Yang et al. (2018); Cai et al. (2018). Citation analysis Athar and Teufel (2012) and citation function Teufel et al. (2006); Li et al. (2013); Hernandez-Alvarez et al. (2017), other closely related domains, aim to predict the sentiment and motivation of a citation respectively. Researchers have used many supervised approaches like sequence labeling Abu-Jbara et al. (2013), structure-based prediction Athar (2011), and multi-task learning Yousif et al. (2019) to address these problems.

In this paper, we want to investigate two research question about citation worthiness. First, we posit that citation worthiness is not purely a sentence-level classification task because the surrounding context could influence if a sentence requires a citation. This context could include not only adjacent sentences but also information about the section titles, paragraphs, and other included citations. Previous work Bonalp et al. (2018) has explored using the adjacent two sentences; we predict that citation worthiness models would improve with access to more contextual information. To pursue this hypothesis, we propose two new formulations: (a) sentence pair classification and (b) sentence se

quence modeling. For the latter formulation, we propose a new hierarchical architecture, where the first layer provides sentence-level representations, and the second layer predicts the citation worthiness of the sentence sequence. We also introduce a new dataset mostly because the prior datasets (Bonab et al., 2018; Zeng and Acuna, 2020) do not have sufficient contextual information to study this research question.

The second research objective is to understand if contextual embedding models would help citation worthiness. Recent developments in language modeling, specifically contextual embedding models Liu et al. (2019); Devlin et al. (2019), have already demonstrated significant improvements in various NLP research tasks. We expect to observe similar gains in citation worthiness. Following is a summary of the main contributions of this work:

* We propose two new formulations for citation worthiness: sentence-pair classification and sentence sequence modeling.
* We contribute a new dataset containing significantly more context, and we expect it to serve as another benchmark.
* Through rigorous experimental work, we demonstrate the benefits of sequential modeling and contextual embeddings for citation worthiness.
* We obtain new state-of-the-art results on three benchmark datasets.

## 2 Methodology

### Problem Statement

Let \(d=\{s_{1},s_{2},...,s_{n}\}\) be a scientific article, where \(s_{i}\) is the \(i^{th}\) sentence. The problem of citation worthiness is to assign each sentence \(s_{i}\) one of two possible labels \(L=\{l_{c},l_{n}\}\), where \(l_{c}\) denotes that sentence requires a citation and \(l_{n}\) means otherwise. We present three different formulations here to investigate our main research objectives.

### Sentence Classification

Our first formulation (Figure 1 (a)) approaches citation worthiness as a sentence level classification task similar to prior works of (Bonab et al., 2018; Farber et al., 2018). Given a sentence \(s_{i}\), we map it to a fixed-size dense vector \(x_{i}\) using contextual embedding models (e.g. BERT (Devlin et al., 2019)). We then feed \(x_{i}\) to a feed-forward layer to obtain the citation worthiness label. We fine-tune this entire architecture by optimizing the weights of the final layer.

### Sentence-Pair Classification

Our second approach (Figure 1 (b)) is to formulate citation worthiness as a sentence-pair classification task, where the pair consists of the given sentence and a sentence-like representation of the context. Namely, for a given sentence \(s_{i}\), we define the context \(c_{i}\) as the concatenation of the previous \(s_{i-1}\), \(s_{i}\), and the next sentence \(s_{i+1}\):

\[c_{i}=[s_{i-1};s_{i};s_{i+1}] \tag{1}\]

We then concatenate \(s_{i}\) with \(c_{i}\) separated by the [SEP] token, pass it through the embedding layer to obtain a vector representation \(x_{i}\). This vector is then passed through a feed-forward layer to obtain the class label. This approach is similar to (Zeng and Acuna, 2020), where the authors used Glove embeddings (Pennington et al., 2014) to obtain sentence representations, and BiLSTMs for context representations. This formulation has also been used previously for question-answering (Devlin et al., 2019) and passage re-ranking (Nogueira and Cho, 2019). In our sentence-pair classification approach, we defined \(c_{i}\) to include only two adjacent sentences, but it could easily include more. However, if we included too many sentences, the context might be too long for most transformer-based models.

\begin{table}
\begin{tabular}{|p{367.5pt}|} \hline The baseline TTS systems in the project utilize the HTS toolkit which is built on top of the HTK framework **Cite**. The HMM-based TTS systems have been developed for Finnish, English, Mandarin and Japanese **No Cite**. The systems include an average voice model for each language trained over hundreds of speakers taken from standard ASR corpora, such as Speech **Ctle**. Using speaker adaptation transforms, thousands of new voices have been created and new voices can be added using a small number of either supervised or unsupervised speech samples **Cite**. \\ \hline The LexRank method addressed was very successful in generic multi-document summarization **Ctle**. A topic-sensitive LexRank is proposed **Ctle**. As in LexRank, the set of sentences in a document cluster is represented as a graph, where nodes are sentences and links between the nodes are induced by a similarity relation between the sentences **No Ctle**. \\ \hline \end{tabular}
\end{table}
Table 1: The excerpts are obtained from (Kurimo et al., 2010) and (Chali et al., 2009). The actual citations have been removed but we include citation worthiness labels.

 



### Sentence Sequence Modeling

The third formulation addresses citation worthiness as a sequence labeling task solved using a hierarchical BiLSTM architecture (Figure 1 (c)). We first map each sentence \(s_{i}\) and context \(c_{i}\) (eq. 1) to a fixed-size dense vector \(x_{i}\) using the same approach as in section 2.3. Thus the given document \(d\) is represented as a sequence of vectors \(x=\{x_{1},x_{2},...,x_{n}\}\). We then feed these vectors to a BiLSTM model to capture the sequential relations between the sentences. The hidden state of the BiLSTM \(h_{i}\) provides a vector representation for sentence \(s_{i}\) that incorporates information from the surrounding sentences. Thus the sequence modeling approach captures long term dependencies between the sentences without us needing to explicitly encode them as extra features. We then use a feed-forward layer to map the BiLSTM output to the citation worthiness labels. We have also experimented with using only the sentence \(s_{i}\) to construct the vector \(x_{i}\). However, we observed that using the context \(c_{i}\) helped improve the model performance.

## 3 Experimental Work

### Datasets

Prior works in citation worthiness presented two benchmark datasets: SEPID-cite Bonab et al. (2018) and PMOA-cite Zeng and Acuna (2020). SEPID-cite contains 1,228,053 sentences extracted from 10,921 articles2 but does not contain the source of the sentences (e.g. paper id) or the sentence order. PMOA-cite contains 1,008,042 sentences extracted from 6,754 papers from PubMed open access. PMOA-cite also contains the preceding sentence, the next sentence, and the section header. However, the authors of PMOA-cite did data splits at sentence-level, which means sentences from the same research paper could be part of train and test datasets. Since we cannot use either one of these datasets directly for sequence modeling, we chose to process the ACL Anthology Reference Corpus Bird et al. (2008)3 (ACL-ARC) while preserving the sentence order, and then split the data at document-level.

Footnote 2: [http://pars.ie/lr/sepid-corpus](http://pars.ie/lr/sepid-corpus)

Footnote 3: [https://www.aclweb.org/anthology/](https://www.aclweb.org/anthology/)

The latest version of ACL-ARC Bird et al. (2008), released in 2015, contains 22,878 articles. Each article here contains the full text and metadata such as author names, section headers, and references. We first processed this corpus to exclude all articles without abstracts because they typically were conference cover sheets. Then, for each section in an article, we extracted paragraph information based on newlines. Then, we split the paragraphs into constituent sentences4 and processed the sentences to obtain citation labels based on regular expressions (Appendix A). We then sanitized the sentences to remove all the citation patterns.

Footnote 4: [https://github.com/fnl/seqtok](https://github.com/fnl/seqtok)

The resulting new corpus (**ACL-cite5**) contained 2,706,792 sentences from 17,440 documents of which 305,733 sentences (11.3%) had citations. Lastly, we performed document-level splits: training (10,464 docs, 1,625,268 sentences), validation (3,487 docs, 539,085 sentences), and test (3,487 docs, 542,081 sentences). To validate our citation regular expressions, we manually annotated a random sample of 500 sentences and observed only one error in the extracted labels. Table 3 provides some basic statistics on the three datasets.

[FOOTNOTE

Figure 1: The three proposed architectures.





### Experimental settings

We applied the sentence classification (SC) model on all three datasets, sentence-pair classification (SPC) model on PMOA-cite and ACL-cite, and sentence sequence modeling (SSM) approach on ACL-cite. This is because SEPID-cite does not have any context to apply SPC or SSM, and PMOA-cite does not have sufficient context for SSM.

To obtain sentence representations, we also explored the idea of pooling word-level embeddings obtained using CNNs. However, we observed no significant difference in the model performance when compared to using the [CLS] token. We also experimented with the choice of contextual embeddings: BERT Devlin et al. (2019), SciBERT Beltagy et al. (2019), Roberta Liu et al. (2019), and XLnet Yang et al. (2019) and observed that the Roberta model consistently gave the best results; therefore, we only report those numbers.

We used a batched training approach for the SSM models: split each article into sequences of \(m\) with an overlap of \(m/2\) sentences. For example, consider a document with 32 sentences and \(m=16\), we create three training sequences; first sequence: sentences 1 to 16, second sequence: sentences 9 to 24, and so on. During inference, for a given sentence, we include the preceding \(m/2\) sentences and the succeeding \(m/2-1\) sentences6. We trained and evaluated models at different values of \(m=4,8,16\). We trained all the models using the Adam optimizer with a batch size of 16, a learning rate of 1e-5, a maximum of 4 epochs to optimize for cross-entropy loss. The hidden layers in the BiLSTM models were set to 128 units. The models were trained on a GPU machine with 6 cores and each training epoch took approximately 4 hours. More details on the experimental settings are available in the Appendix.

Footnote 6: We used zero-padding in cases with insufficient context, e.g., beginning or end of a document.

### Results

Table 2 summarizes the results in terms of the precision, recall, F1 score for \(l_{c}\), and overall weighted F1 score. The baseline numbers reported here are either from prior works Farber et al. (2018); Bonab et al. (2018); Zeng and Acuna (2020) or based on architectures very similar to those used in these prior works. On the SEPID-cite dataset, our SC model obtained significantly better performance than the state-of-the-art results from Zeng and

\begin{table}
\begin{tabular}{|p{71.1pt}|r|r|r|r|} \hline
**Model** & **P** & **R** & **F1** & **W-F1** \\ \hline \multicolumn{5}{|c|}{**SEPID-cite**} \\ \hline CNN + Glove (f) & 0.196 & 0.269 & 0.227 & N/A \\ CNN + w2v (b) & 0.448 & 0.405 & 0.426 & N/A \\ BiLSTM + Glove (z) & **0.720** & 0.391 & 0.507 & N/A \\ SC + Roberta & 0.674 & **0.595** & **0.629** & 0.950 \\ \hline \multicolumn{5}{|c|}{**PMOA-cite**} \\ \hline BiLSTM + Glove (z) & 0.883 & 0.795 & 0.837 & N/A \\ SC + Roberta & **0.887** & **0.814** & **0.849** & 0.942 \\ \hline BiLSTM + Glove + context (z) & **0.907** & 0.811 & 0.856 & N/A \\ SPC + Roberta & 0.905 & **0.843** & **0.873** & 0.962 \\ \hline \multicolumn{5}{|c|}{**ACL-cite**} \\ \hline SC + Roberta & 0.782 & 0.496 & 0.607 & 0.921 \\ SPC + Roberta & 0.820 & 0.562 & 0.667 & 0.932 \\ SSM (4 sentences) + Roberta & **0.821** & 0.635 & 0.716 & 0.940 \\ SSM (8 sentences) + Roberta & 0.816 & 0.643 & 0.719 & 0.940 \\ SSM (16 sentences) + Roberta & 0.809 & 0.654 & 0.723 & 0.941 \\ SSM (16 sentences + section) + Roberta & 0.813 & **0.662** & **0.730** & **0.942** \\ \hline \end{tabular}
\end{table}
Table 2: SC: Sentence Classification, SPC: Sentence-Pair Classification, SSM: Sentence Sequence Modeling. (f) denotes that the numbers reported in Farber et al. (2018), (b) in Bonab et al. (2018), and (z) in Zeng and Acuna (2020). P, R, and F1 are precision, recall, and F1 scores for the cite class (\(l_{c}\)). W-F1 is the weighted-F1 on entire test dataset.

\begin{table}
\begin{tabular}{|p{71.1pt}|r|r|r|} \hline  & **SEPID-cite** & **PMOA-cite** & **ACL-cite** \\ \hline Articles Sections & 10,921 & 6,754 & 17,440 \\ Sertions & – & 32,198 & 130,603 \\ Paragraphs & – & 20,047 & 934,502 \\ Sentences & 1,228,053 & 1,008,042 & 2,706,792 \\ S w/o c & 1,142,275 & 811,659 & 2,401,059 \\ S w v c & 85,778 & 196,383 & 305,733 \\ A c/s & 131 & 132 & 141 \\ A w/s & 22 & 20 & 22 \\ \hline \end{tabular}
\end{table}
Table 3: Statistics of the three datasets. S w/o c: Sentences without citation, S w c: Sentences with citation, A c/s: Average characters per sentence, A w/s: Average words per sentene.



Acuna, 2020) with the F1 score increasing by more than 12%. On the PMOA-cite dataset, we obtain an F1 gain of 1.2% for sentence-level and 1.7% for contextual models. We indicate that the numbers from Zeng and Acuna (2020) use additional hand-crafted contextual features, including labels of surrounding sentences, but our models only use textual features.

The results on the ACL-cite dataset clearly show the importance of context in this domain. The use of surrounding two sentences boosted the performance by nearly 6% points, and the performance continues to improve with added context increasing by another 5.6% points for 16 sentences. The model performance improves by another 0.7% with the inclusion of section headers in the context.

Table 4 compares the performance of the SC and SSM models for different sections in the papers. The F1 score improves for all but most prominent for _Abstract_ and _Conclusion_ sections because of significant improvements in the precision.

### Subjective Analysis

We observed some interesting trends during the error-analysis of the SC and SSM models. We categorized these trends into three groups and selected an example from each to illustrate the impact of context (Table 1).

\(\bullet\)_Prior works_: In the first excerpt, the last sentence could be interpreted as the author's contribution if no context was available. The preceding sentences in the paragraph seem to help the model understand that this sentence requires a citation because it refers to prior work.

\(\bullet\)_Sections_: In the second excerpt, the second sentence could be interpreted as an introduction or conclusion. Once again, the context provides information to infer the section correctly and, therefore, the correct label.

\(\bullet\)_Topic sentences_: Context is essential to understand if a sentence is the first statement about a topic, typically when researcher provide citations, or continuation of a discussion. In the second excerpt, the model does not predict \(l_{c}\) for the last sentence because the authors already introduced the concept _LexRank_ in previous sentences.

## 4 Conclusions

In this paper, we study the impact of context and contextual models on citation worthiness. We propose two new formulations for this problem: sentence-pair classification and sentence sequence modeling. We contribute a new benchmark dataset with document-level train/dev/test splits, which enables to incorporate contextual information better. We propose a hierarchical BiLSTM approach for sequence modeling, but we could also consider a transformer-based approach and further improve with a CRF layer. Likewise, we also want to consider some of the newer language models Zaheer et al. (2020); Beltagy et al. (2020) that handle longer sentences.

We expect citation worthiness would be an important part of developing writing assistants for scientific documents. We studied the citation worthiness of sentences in scholarly articles in this paper, but we believe these findings are relevant to other domains like news, Wikipedia, and legal documents.

## References

* Abu-Jbara et al. (2013) Amjad Abu-Jbara, Jefferson Ezra, and Dragomir Radev. 2013. Purpose and polarity of citation: Towards NLP-based bibliometrics. In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 596-606, Atlanta, Georgia. Association for Computational Linguistics.
* Athar (2011) Awais Athar. 2011. Sentiment analysis of citations using sentence structure-based features. In _Proceedings of the ACL 2011 Student Session_, pages 81-87, Portland, OR, USA. Association for Computational Linguistics.
* Mourou et al. (2017)

\begin{table}
\begin{tabular}{|p{56.9pt}|r r|r|r|r|r|} \hline  & \multicolumn{3}{c|}{**SC**} & \multicolumn{3}{c|}{**SSM**} \\ \hline Section & P & R & Fl & P & R & F1 \\ \hline Abstract & 0.340 & 0.505 & 0.407 & 0.543 & 0.576 & **0.559** \\ Acknowledgments & 0.874 & 0.361 & 0.511 & 0.759 & 0.480 & **0.588** \\ Conclusion & 0.585 & 0.459 & 0.515 & 0.711 & 0.560 & **0.626** \\ Evaluation & 0.770 & 0.538 & 0.633 & 0.808 & 0.659 & **0.726** \\ Introduction & 0.833 & 0.514 & 0.636 & 0.831 & 0.645 & **0.726** \\ Methods & 0.791 & 0.525 & 0.631 & 0.803 & 0.650 & **0.718** \\ Related Work & 0.901 & 0.707 & 0.792 & 0.918 & 0.827 & **0.870** \\ \hline \end{tabular}
\end{table}
Table 4: Comparison of the F1 score of SC and SSM models by section.

 

Awais Athar and Simone Teufel. 2012. Context-enhanced citation sentiment detection. In _Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 597-601, Montreal, Canada. Association for Computational Linguistics.
* Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciBERT: A pretrained language model for scientific text. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3615-3620, Hong Kong, China. Association for Computational Linguistics.
* Beltagy et al. (2020) Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. _arXiv preprint arXiv:2004.05150_.
* Bird et al. (2008) Steven Bird, Robert Dale, Bonnie Dorr, Bryan Gibson, Mark Joseph, Min-Yen Kan, Dongwon Lee, Brett Powley, Dragomir Radev, and Yee Fan Tan. 2008. The ACL Anthology reference corpus: A reference dataset for bibliographic research in computational linguistics. In _Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)_, Marrakech, Morocco. European Language Resources Association (ELRA).
* Bonab et al. (2018) Hamed Bonab, Hamed Zamani, Erik Learned-Miller, and James Allan. 2018. Citation worthiness of sentences in scientific reports. In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval_, pages 1061-1064.
* Cai et al. (2018) X. Cai, J. Han, W. Li, R. Zhang, S. Pan, and L. Yang. 2018. A three-layered mutually reinforced model for personalized citation recommendation. _IEEE Transactions on Neural Networks and Learning Systems_, 29(12):6026-6037.
* Cai et al. (2018) Xiaoyan Cai, Junwei Han, and Libin Yang. 2018. Generative adversarial network based heterogeneous bibliographic network representation for personalized citation recommendation. _Proceedings of the AAAI Conference on Artificial Intelligence_, 32(1).
* Chali et al. (2009) Yllias Chali, Shafiq R. Joty, and Sadid A. Hasan. 2009. Complex question answering: Unsupervised learning approaches and experiments. _Journal of Artificial Intelligence Research_, 35(1):1-47.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Ebesu and Fang (2017) Travis Ebesu and Yi Fang. 2017. Neural citation network for context-aware citation recommendation. In _Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval_, SIGIR '17, page 1093-1096, New York, NY, USA. Association for Computing Machinery.
* Farber et al. (2018) Michael Farber, Alexander Thiemann, and Adam Jaitowt. 2018. To cite, or not to cite? detecting citation contexts in text. In _Advances in Information Retrieval_, pages 598-603, Cham. Springer International Publishing.
* He et al. (2010) Qi He, Jian Pei, Daniel Kifer, Prasenjit Mitra, and Lee Giles. 2010. Context-aware citation recommendation. In _Proceedings of the 19th International Conference on World Wide Web_, WWW '10, page 421-430, New York, NY, USA. Association for Computing Machinery.
* Hernandez-Alvarez et al. (2017) Myriam Hernandez-Alvarez, Jose M Gomez Soriano, and Patricio Martinez-Barco. 2017. Citation function, polarity and influence classification. _Natural Language Engineering_, 23(4):561.
* Huang et al. (2015) Wenyi Huang, Zhaohui Wu, Chen Liang, Prasenjit Mitra, and C. Lee Giles. 2015. A neural probabilistic model for context based citation recommendation. In _Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence_, AAAI'15, page 2404-2410. AAAI Press.
* Kucuktunc et al. (2014) Onur Kucuktunc, Erik Saule, Kamer Kaya, and Umit V. Catalyurek. 2014. Diversifying citation recommendations. _ACM Trans. Intell. Syst. Technol._, 5(4).
* Kurimo et al. (2010) Mikko Kurimo, William Byrne, John Dines, Philip N. Garner, Matthew Gibson, Yong Guan, Teemu Hirsimaki, Reimna Karhila, Simon King, Hui Liang, Keiichiro Oura, Lakshmi Saheer, Matt Shannon, Sayaka Shiota, Jilei Tian, Keiichi Tokuda, Mirjam Wester, Yi-Jian Wu, and Junichi Yamagishi. 2010. Personalising speech-to-speech translation in the emime project. In _Proceedings of the ACL 2010 System Demonstrations_, ACLDemos '10, page 48-53, USA. Association for Computational Linguistics.
* Li et al. (2013) Xiang Li, Yifan He, Adam Meyers, and Ralph Grishman. 2013. Towards fine-grained citation function classification. In _Proceedings of the International Conference Recent Advances in Natural Language Processing RANLP 2013_, pages 402-407, Hissar, Bulgaria. INCOMA Ltd. Shoumen, BULGARIA.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. _arXiv preprint arXiv:1907.11692_.
* Nogueira and Cho (2019) Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage re-ranking with bert. _arXiv preprint arXiv:1901.04085_ 

Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
* Strohman et al. [2007] Trevor Strohman, W. Bruce Croft, and David Jensen. 2007. Recommending citations for academic papers. In _Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_, SIGIR '07, page 705-706, New York, NY, USA. Association for Computing Machinery.
* Sugiyama et al. [2010] K. Sugiyama, T. Kumar, M. Kan, and R. C. Tripathi. 2010. Identifying citing sentences in research papers using supervised learning. In _2010 International Conference on Information Retrieval Knowledge Management (CAMP)_, pages 67-72.
* Taufel et al. [2006] Simone Taufel, Advaith Siddharthan, and Dan Tidhar. 2006. Automatic classification of citation function. In _Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing_, EMNLP '06, page 103-110, USA. Association for Computational Linguistics.
* Yang et al. [2018] L. Yang, Y. Zheng, X. Cai, H. Dai, D. Mu, L. Guo, and T. Dai. 2018. A lstm based model for personalized context-aware citation recommendation. _IEEE Access_, 6:59618-59627.
* Yang et al. [2019] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLnet: Generalized autoregressive pretraining for language understanding. In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_, pages 5754-5764.
* Yousif et al. [2019] Abdallah Yousif, Zhendong Niu, James Chambua, and Zahid Younas Khan. 2019. Multi-task learning model based on recurrent convolutional neural networks for citation sentiment and purpose classification. _Neurocomputing_, 335:195-205.
* Zaheer et al. [2020] Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020. Big bird: Transformers for longer sequences. In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
* Zeng and Acuna [2020] Tong Zeng and Daniel E. Acuna. 2020. Modeling citation worthiness by using attention-based bidirectional long short-term memory networks and interpretable models. _Scientometrics_, 124(1):399-428