Venue: NAACL 2018

# Content-Based Citation Recommendation

Chandra Bhagavatula

Allen Institute for AI

chandrab@allenai.org Sergey Feldman

Data Cowboys 1

sergey@data-cowboys.com Russell Power

Independent Researcher

russell.power@gmail.com Waleed Ammar

Allen Institute for AI

waleeda@allenai.org

Footnote 1: Work done while on contract with AI2

###### Abstract

We present a content-based method for recommending citations in an academic paper draft. We embed a given query document into a vector space, then use its nearest neighbors as candidates, and rerank the candidates using a discriminative model trained to distinguish between observed and unobserved citations. Unlike previous work, our method does not require metadata such as author names which can be missing, e.g., during the peer review process. Without using metadata, our method outperforms the best reported results on PubMed and DBLP datasets with relative improvements of over 18% in F1@20 and over 22% in MRR. We show empirically that, although adding metadata improves the performance on standard metrics, it favors self-citations which are less useful in a citation recommendation setup. We release an online portal for citation recommendation based on our method,1 and a new dataset OpenCorpus of 7 million research articles to facilitate future research on this task.

Footnote 1: [http://labs.semanticscholar.org/citeomatic/Work](http://labs.semanticscholar.org/citeomatic/Work) done while at AI2

+
Footnote †: Web dooms available from [https://www.wolfestream.com/Web](https://www.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from [https://web.wolfestream.com/Web](https://web.wolfestream.com/Web) dooms available from https:wolfestream.com/Web dooms available from https:wolfestream.



previous datasets used for citation recommendation, and
* a scalable web-based literature review tool based on this work.2 Footnote 2: [https://github.com/allenai/citeomatic](https://github.com/allenai/citeomatic)

## 2 Overview

We formulate citation recommendation as a ranking problem. Given a query document \(d_{q}\) and a large corpus of published documents, the task is to rank documents which should be referenced in \(d_{q}\) higher than other documents. Following previous work on citation recommendation, we use standard metrics (precision, recall, F-measure and mean reciprocal rank) to evaluate our predictions against gold references provided by the authors of query documents.

Since the number of published documents in the corpus can be large, it is computationally expensive to score each document as a candidate reference with respect to \(d_{q}\). Instead, we recommend citations in two phases: (i) a fast, recall-oriented candidate selection phase, and (ii) a feature rich, precision-oriented reranking phase. Figure 1 provides an overview of the two phases using a toy example.

Phase 1 - Candidate Selection:In this phase, our goal is to identify a set of candidate references for \(d_{q}\) for further analysis without explicitly iterating over all documents in the corpus.3 Using a trained neural network, we first project all published documents into a vector space such that a document tends to be close to its references. Since the projection of a document is independent of the query document, the entire corpus needs to be embedded only once and can be reused for subsequent queries. Then, we project each query document \(d_{q}\) to the same vector space and identify its nearest neighbors as candidate references. See SS3 for more details about candidate selection.

Footnote 3: In order to increase the chances that all references are present in the list of candidates, the number of candidates must be significantly larger than the total number of citations of a document, but also significantly smaller than the number of documents in the corpus.

Figure 1: An overview of our Citation Recommendation system. In Phase 1 (NNSelect), we project all documents in the corpus (7 in this toy example) in addition to the query document \(d_{q}\) into a vector space, and use its (K=4) nearest neighbors: \(d_{2},d_{6},d_{3},\) and \(d_{4}\) as candidates. We also add \(d_{7}\) as a candidate because it was cited in \(d_{3}\). In Phase 2 (NNRank), we score each pair \((d_{q},d_{2}),(d_{q},d_{6}),(d_{q},d_{3}),(d_{q},d_{4}),\) and \((d_{q},d_{7})\) separately to rerank the candidates and return the top 3 candidates: \(d_{7}\), \(d_{6}\) and \(d_{2}\).

 

Phase 2 - Reranking:Phase 1 yields a manageable number of candidates making it feasible to score each candidate \(d_{i}\) by feeding the pair \((d_{q},d_{i})\) into another neural network trained to discriminate between observed and unobserved citation pairs. The candidate documents are sorted by their estimated probability of being cited in \(d_{q}\), and top candidates are returned as recommended citations. See SS4 for more details about the reranking model and inference in the candidate selection phase.

## 3 Phase 1: Candidate Selection (NNSelect)

In this phase, we select a pool of candidate citations for a given query document to be reranked in the next phase. First, we compute a dense embedding of the query document \(d_{q}\) using the document embedding model (described next), and select \(K\) nearest neighbor documents in the vector space as candidates.4 Following Strohman et al. (2007), we also include the outgoing citations of the \(K\) nearest neighbors as candidates.

Footnote 4: We tune \(K\) as a hyperparameter of our method.

The output of this phase is a list of candidate documents \(d_{i}\) and their corresponding scores \(\textsc{NNSelect}(d_{q},d_{i})\), defined as the cosine similarity between \(d_{q}\) and \(d_{i}\) in the document embedding space.

Document embedding model.We use a supervised neural model to project any document \(d\) to a dense embedding based on its textual content. We use a bag-of-word representation of each textual field, e.g., \(d[\text{title}]=\{\text{'content-based', `citation'', 'recommendation'}\}\), and compute the feature vector:

\[\mathbf{f}_{d[\text{title}]}=\sum_{t\in d[\text{title}]}w^{\text{mag}}_{t} \frac{\mathbf{w}^{\text{dir}}_{t}}{\|\mathbf{w}^{\text{dir}}_{t}\|_{2}}, \tag{1}\]

where \(\mathbf{w}^{\text{dir}}_{t}\) is a dense direction embedding and \(w^{\text{mag}}_{t}\) is a scalar magnitude for word type \(t\).5 We then normalize the representation of each field and compute a weighted average of fields to get the document embedding, \(\mathbf{e}_{d}\). In our experiments, we use the title and abstract fields of a document \(d\):

Footnote 5: The magnitude-direction representation is based on Salimans and Kingma (2016) and was found to improve results in preliminary experiments, compared to the standard “direction-only” word representation.

\[\mathbf{e}_{d}=\lambda^{\text{title}}\frac{\mathbf{f}_{d[\text{title}]}}{\| \mathbf{f}_{d[\text{title}]}\|_{2}}+\lambda^{\text{abstract}}\frac{\mathbf{ f}_{d[\text{abstract}]}}{\|\mathbf{f}_{d[\text{abstract}]}\|_{2}},\]

where \(\lambda^{\text{title}}\) and \(\lambda^{\text{abstract}}\) are scalar model parameters.

Training.We learn the parameters of the document embedding model (i.e., \(\lambda^{*},w^{\text{mag}}_{*},\mathbf{w}^{\text{dir}}_{*}\)) using a training set \(\mathcal{T}\) of triplets \(\langle d_{q},d^{+},d^{-}\rangle\) where \(d_{q}\) is a query document, \(d^{+}\) is a document cited in \(d_{q}\), and \(d^{-}\) is a document not cited in \(d_{q}\). The model is trained to predict a high cosine similarity for the pair \((d_{q},d^{+})\) and a low cosine similarity for the pair \((d_{q},d^{-})\) using the per-instance triplet loss Wang et al. (2014):

\[\text{loss}=\max\big{(}\alpha+s(d_{q},d^{-})-s(d_{q},d^{+}),0\big{)}, \tag{2}\]

where \(s(d_{i},d_{j})\) is defined as the cosine similarity between document embeddings cos-sim\((\mathbf{e}_{d_{i}},\mathbf{e}_{d_{j}})\). We tune the margin \(\alpha\) as a hyperparameter of the model (see Appendix B for more details). Next, we describe how negative examples are selected.

Selecting negative examples.Defining positive examples is straight-forward; we use any \((d_{q},d^{+})\) pair where a document \(d_{q}\) in the training set cites \(d^{+}\). However, a careful choice of negative training examples is critical for model performance. We use three types of negative examples:

1. **Random:** any document not cited by \(d_{q}\).
2. **Negative nearest neighbors:** documents that are close to \(d_{q}\) in the embedding space, but are not cited in it.6 Footnote 6: Since the set of approximate neighbors depend on model parameters, we recompute a map from each query document to its \(K\) nearest neighbors before each epoch while training the document embedding model.
3. **Citation-of-citation:** documents referenced in positive citations of \(d_{q}\), but are not cited directly in \(d_{q}\). Negative examples belong to at least one of these types that serve different, and complementary purposes. Selecting a paper from the corpus at random as a negative example typically results in easy negative examples. Selecting nearest neighbor documents in the embedding space used for candidate selection enables the re-ranking phase (described in SS4) to fix some of the mistakes made in the candidate selection step. Finally, using citations-of-citations as negative examples is based on the assumption that the authors would have included them as positive examples if they were relevant for the query paper. In Appendix SSA, we describe the number of negative examples of each type used for training. Next, we describe how to rerank the candidate documents.

 



## 4 Phase 2: Reranking Candidates

(NNRank)

In this phase, we train another model which takes as input a pair of documents (\(d_{q}\), \(d_{i}\)) and estimates the probability that \(d_{i}\) should be cited in \(d_{q}\).

Input features.A key point of this work is to assess the feasibility of recommending citations without using metadata, but we describe all features here for completeness and defer this discussion to SS5. For each document, we compute dense feature vectors \(\mathbf{f}_{\text{df[field]}}\) as defined in Eq. 1 for the following fields: title, abstract, authors, venue and keyphrases (if available). For the title and abstract, we identify the subset of word types which appear in both documents (intersection), and compute the sum of their scalar weights as an additional feature, e.g., \(\sum_{t\in\cap\text{title}}w^{\cap}_{t}\). We also use _log_ number of times the candidate document \(d_{i}\) has been cited in the corpus, i.e., _log_(\(d_{i}\)[in-citations]). Finally, we use the cosine similarity between \(d_{q}\) and \(d_{i}\) in the embedding space, i.e., cos-sim\((\mathbf{e}_{d_{q}},\mathbf{e}_{d_{i}})\).

Model architecture.We illustrate the NNRank model architecture in Figure 2. The output layer is defined as:

\[s(d_{i},d_{j}) =\text{FeedForward}(\mathbf{h}), \tag{3}\] \[\mathbf{h} =\Big{[}\mathbf{g}_{\text{title}};\mathbf{g}_{\text{abstract}}; \mathbf{g}_{\text{authors}};\mathbf{g}_{\text{venue}};\] \[\quad\mathbf{g}_{\text{keyphrases}};\text{cos-sim}(\mathbf{e}_{ d_{q}},\mathbf{e}_{d_{i}});\] \[\quad\sum_{t\in\cap\text{title}}w^{\cap}_{t};\sum_{t\in\cap\text {abstract}}w^{\cap}_{t};\] \[\quad d_{i}[\text{in-citations}]\Big{]},\] \[\mathbf{g}_{\text{field}} =\text{cos-sim}(\mathbf{f}_{d_{q}[\text{field}]},\mathbf{f}_{d_{ q}[\text{field}]}),\]

where 'FeedForward' is a three layer feed-forward neural network with two exponential linear unit layers (Clevert et al., 2015) and one sigmoid layer. ',' indicates concatenation.

Training.The parameters of the NNRank model are \(w^{\text{mag}}_{*},\mathbf{w}^{\text{trk}}_{*},w^{\cap}_{*}\) and parameters of the three dense layers in 'FeedForward'. We reuse the triplet loss in Eq. 2 to learn these parameters, but redefine the similarity function \(s(d_{i},d_{j})\) as the sigmoid output described in Eq. 3.

At test time, we use this model to recommend candidates \(d_{i}\) with the highest \(s(d_{q},d_{i})\) scores.

## 5 Experiments

In this section, we describe experimental results of our citation recommendation method and compare it to previous work.

Datasets.We use the DBLP and PubMed datasets (Ren et al., 2014) to compare with previous work on citation recommendation. The DBLP

Figure 2: NNRank architecture. For each of the textual and categorical fields, we compute the cosine similarity between the embedding for \(d_{q}\) and the corresponding embedding for \(d_{i}\). Then, we concatenate the cosine similarity scores, the numeric features and the summed weights of the intersection words, followed by two dense layers with ELU non-linearities The output layer is a dense layer with sigmoid non-linearity, which estimates the probability that \(d_{q}\) cites \(d_{i}\).



dataset contains over 50K scientific articles in the computer science domain, with an average of 5 citations per article. The PubMed dataset contains over 45K scientific articles in the medical domains, with an average of 17 citations per article. In both datasets, a document is accompanied by its title, abstract, venue (i.e. journal or conference where the document was published), authors, citations (i.e. other documents in the corpus that are referenced in the given document) and keyphrases (i.e. phrases considered important by automated extraction methods). We replicate the experimental setup of Ren et al. (2014) by excluding papers with fewer than 10 citations and using the standard train, dev and test splits.6

Footnote 6: The dataset characteristics reported here are different from those in Table 3 in Ren et al. (2014) because we report the size of the filtered datasets while they report the size of the datasets before filtering.

We also introduce OpenCorpus,7 a new dataset of 7 million scientific articles primarily drawn from the computer science and neuroscience domain. Due to licensing constraints, documents in the corpus do not include the full text of the scientific articles, but include the title, abstract, year, author, venue, keyphrases and citation information. The mutually exclusive training, development, and test splits were selected such that no document in the development or test set has a publication year less than that of any document in the training set. Papers with zero citations were removed from the development and test sets. We describe the key characteristics of OpenCorpus in Table 1.

Footnote 7: [http://labs.semanticscholar.org/corpus/](http://labs.semanticscholar.org/corpus/)

Baselines.We compare our method to two baseline methods for recommending citations: ClusCite and BM25. ClusCite Ren et al. (2014) clusters nodes in a heterogeneous graph of terms, authors and venues in order to find related documents which should be cited. We use the ClusCite results as reported in Ren et al. (2014), which compared it to several other citation recommendation methods and found that it obtains state of the art results on the PubMed and DBLP datasets. The BM25 results are based on our implementation of the popular ranking function Okapi BM25 used in many information retrieval systems. See Appendix SSD for details of our BM25 implementation.

Evaluation.We use Mean Reciprocal Rank (MRR) and F1@20 to report the main results in this section. In Appendix SSF, we also report additional metrics (e.g., precision and recall at 20) which have been used in previous work. We compute F1@20 as the harmonic mean of the corpus-level precision and recall at 20 (P@20 and R@20). Following Ren et al. (2014), precision and recall at 20 are first computed for each query document then averaged over query documents in the test set to compute the corpus-level P@20 and R@20.

Configurations.To find candidates in NNSelect, we use the approximate nearest neighbor search algorithm Annoy8, which builds a binary-tree structure that enables searching for nearest neighbors in \(O\)(log _n_) time. To build this tree, points in a high-dimensional space are split by choosing random hyperplanes. We use 100 trees in our approximate nearest neighbors index, and retrieve documents using the cosine distance metric.

Footnote 8: [https://github.com/spotify/annoy](https://github.com/spotify/annoy)

We use the hyperopt library9 to optimize various hyperparameters of our method such as size of hidden layers, regularization strength and learning rate. To ensure reproducibility, we provide a detailed description of the parameters used in both NNSelect and NNRank models, our hyperparameter optimization method and parameter values chosen in Appendix SSA.

Main results.Table 2 reports the F1@20 and MRR results for the two baselines and three variants of our method. Since the OpenCorpus dataset is much bigger, we were not able to train the ClusCite baseline for it. Totti et al. (2016) have also found it difficult to scale up ClusCite to larger datasets. Where available, we report the mean \(\pm\) standard deviation based on five trials.

The first variant, labeled "NNSelect," only uses the candidate selection part of our method (i.e., phase 1) to rank candidates by their cosine

\begin{table}
\begin{tabular}{l|c}
**Statistic** & **Value** \\ \hline \# of documents in corpus & 6.9 million \\ \# of unique authors & 8.3 million \\ \# of unique keyphrases & 823,677 \\ \# of unique venues & 23,672 \\ avg. \# of incoming citations & 7.4 (\(\pm\) 38.1) \\ avg. \# of outgoing citations & 8.4 (\(\pm\) 14.4) \\ \hline size of training set [years 1991 to 2014] & 5.5 million \\ size of dev set [years 2014 to 2015] & 689,000 \\ size of test set [years 2015 to 2016] & 20,000 \\ \end{tabular}
\end{table}
Table 1: Characteristics of the OpenCorpus.

 

similarity to the query document in the embedding space as illustrated in Fig. 1. Although the document embedding space was designed to efficiently select candidates for further processing in phase 2, recommending citations directly based on the cosine distance in this space outperforms both baselines.

The second variant, labeled "NNSelect+ NNRank," uses the discriminative model (i.e., phase 2) to rerank candidates selected by NNSelect, without encoding metadata (venues, authors, keyphrases). Both the first and second variants show that improved modeling of paper text can significantly outperform previous methods for citation recommendation, without using metadata.

The third variant, labeled "NNSelect+ NNRank+ metadata," further encodes the metadata features in the reranking model, and gives the best overall results. On both the DBLP and PubMed datasets, we obtain relative improvements over 20% (for F1@20) and 25% (for MRR) compared to the best reported results of ClusCite.

In the rest of this section, we describe controlled experiments aimed at analyzing different aspects of our proposed method.

Choice of negative samplesAs discussed in SS3, we use different types of negative samples to train our models. We experimented with using only a subset of the types, while controlling for the total number of negative samples used, and found that using negative nearest neighbors while training the models is particularly important for the method to work. As illustrated in Table 3, on the PubMed dataset, adding negative nearest neighbors while training the models improves the F1@20 score from 0.306 to 0.329, and improves the MRR score from 0.705 to 0.771. Intuitively, using nearest neighbor negative examples focuses training on the harder cases on which the model is more likely to make mistakes.

Valuable featuresWe experimented with different subsets of the optional features used in NNRank in order to evaluate the contribution of various features. We found intersection features, NNSelect scores, and the number of incoming citations to be the most valuable feature. As illustrated in Table 3, the intersection features improves the F1@20 score from 0.296 to 0.329, and the MRR score from 0.653 to 0.771, on the PubMed dataset. The numerical features (NNSelectscore and incoming citations) improve the F1@20 score from 0.314 to 0.329, and improves the MRR score from 0.735 to 0.771. This shows that, in some applications, feeding engineered features to neural networks can be an effective strategy to improve their performance.

Performance across venuesWe studied the variability of performance of our model for papers from different venues. Figure 3 shows the F1@20 score of NNRank for papers belonging to the top

\begin{table}
\begin{tabular}{l|l l|l l|l l}
**Method** & \multicolumn{2}{c|}{**DBLP**} & \multicolumn{2}{c|}{**PubMedMed**} & \multicolumn{2}{c}{**OpenCorpus**} \\  & F1@20 & MRR & F1@20 & MRR & F1@20 & MRR \\ \hline BM25 & 0.119 & 0.425 & 0.209 & 0.574 & 0.058 & 0.218 \\ ClusCite & 0.237 & 0.548 & 0.274 & 0.578 & – & – \\ \hline NNSelect & 0.282\(\pm\)0.002 & 0.579\(\pm\)0.007 & 0.309\(\pm\)0.001 & 0.699\(\pm\)0.001 & 0.109 & 0.221 \\   + NNRank & 0.302\(\pm\)0.001 & 0.672\(\pm\)0.015 & 0.325\(\pm\)0.001 & 0.754\(\pm\)0.003 & **0.126** & 0.330 \\   + metadata & **0.303\(\pm\)**0.001 & **0.689\(\pm\)**0.011 & **0.329\(\pm\)**0.001 & **0.771\(\pm\)**0.003 & 0.125 & **0.330** \\ \end{tabular}
\end{table}
Table 2: F1@20 and MRR results for two baselines and three variants of our method. BM25 results are based on our implementation of this baseline, while ClusCite results are based on the results reported in Ren et al. (2014). “NNSelect” ranks candidates using cosine similarity between the query and candidate documents in the embedding space (phase 1). “NNSelect+ NNRank” uses the discriminative reranking model to rerank candidates (phase 2), without encoding any of the metadata features. “+ metadata” encodes the metadata features (i.e., keyphrases, venues and authors), achieving the best results on all datasets. Mean and standard deviations are reported based on five trials.

\begin{table}
\begin{tabular}{l|l l|l l}  & F1@20 & \(\Delta\) & MRR & \(\Delta\) \\ \hline Full model & 0.329 & & 0.771 & \\ \hline without intersection & 0.296 & **0.033** & 0.653 & **0.118** \\ without -ve NNs & 0.306 & 0.016 & 0.705 & 0.066 \\ without numerical & 0.314 & 0.008 & 0.735 & 0.036 \\ \end{tabular}
\end{table}
Table 3: Comparison of PubMed results of the full model with model without (i) intersection features, (ii) negative nearest neighbors in training samples, and (iii) numerical features.

 

ten venues (by their paper count) in the Pubmed corpus. NNRank's performance is robust across venues.

Encoding textual features.We also experimented with using recurrent and convolutional neural network to encode the textual fields of query and candidate documents, instead of using a weighted sum as described in Eq. 1. We found that recurrent and convolutional encoders are much slower, and did not observe a significant improvement in the overall performance as measured by the F1@20 and MRR metrics. This result is consistent with previous studies on other tasks, e.g., Iyyer et al. (2015).

Number of nearest neighbors.As discussed in SS3, the candidate selection step is crucial for the scalability of our method because it reduces the number of computationally expensive pairwise comparisons with the query document at runtime. We did a controlled experiment on the OpenCorpus dataset (largest among the three datasets) to measure the effect of using different numbers of nearest neighbors, and found that both P@20 and R@20 metrics are maximized when NNSelect fetches five nearest neighbors using the approximate nearest neighbors index (and their out-going citations), as illustrated in Table 4.

Self-citation bias.We hypothesized that a model trained with the metadata (e.g., authors) could be biased towards self-citations and other well-cited authors. To verify this hypothesis, we compared two NNRank models - one with metadata, and one without. We measured the mean and max rank of predictions that had at least one author in common with the query document. This experiment was performed with the OpenCorpus dataset.

A lower mean rank for NNRank + Metadata indicates that the model trained with metadata tends to favor documents authored by one of the query document's authors. We verified the prevalence of this bias by varying the number of predictions for each model from 1 to 100. Figure 4 shows that the mean and max rank of the model trained with metadata is always lower than those for the model that does not use metadata.

## 6 Related Work

Citation recommendation systems can be divided into two categories - _local_ and _global_. A local citation recommendation system takes a few sentences (and an optional placeholder for the candidate citation) as input and recommends citations based on the local context of the input sen

\begin{table}
\begin{tabular}{l|c|c|c} \# of neighbors & R@20 & P@20 & Time(ms) \\ \hline
1 & 0.123 & 0.079 & 131 \\
**5** & **0.142** & **0.080** & 144 \\
10 & 0.138 & 0.069 & 200 \\
50 & 0.081 & 0.040 & 362 \\ \end{tabular}
\end{table}
Table 4: OpenCorpus results for NNSelect step with varying number of nearest neighbors on 1,000 validation documents.

Figure 4: Mean and Max Rank of predictions with varying number of candidates.

Figure 3: F1@20 of NNRank on the Pubmed dataset across top ten venues



tences (Huang et al., 2015; He et al., 2010; Tang and Zhang, 2009; Huang et al., 2012; He et al., 2011). A global citation recommendation system takes the entire scholarly article as input and recommends citations for the paper (McNee et al., 2002; Strohman et al., 2007; Nallapati et al., 2008; Kataria et al., 2010; Ren et al., 2014). We address the global citation recommendation problem in this paper.

A key difference of our proposed method compared to previous work is that our method is content-based and works well even in the absence of metadata (e.g. authors, venues, key phrases, seed list of citations). Many citation recommendation systems crucially rely on a query document's metadata. For example, the collaborative filtering based algorithms of McNee et al. (2002); Jia and Saule (2017); Liu et al. (2015) require seed citations for a query document. (Ren et al., 2014; Yu et al., 2012) require authors, venues and key terms of the query documents to infer interest groups and to extract features based on paths in a heterogeneous graph. In contrast, our model performs well solely based on the textual content of the query document.

Some previous work (e.g. (Ren et al., 2014; Yu et al., 2012)) have addressed the citation recommendation problem using graph-based methods. But, training graph-based citation recommendation models has been found to be expensive. For example, the training complexity of the ClusCite algorithm (Ren et al., 2014) is cubic in the number of edges in the graph of authors, venues and terms. This can be prohibitively expensive for datasets as large as OpenCorpus. On the other hand our model is a neural network trained via batched stochastic gradient descent that scales very well to large datasets (Bottou, 2010).

Another crucial difference between our approach and some prior work in citation prediction is that we build up a document representation using its constituent words only. Prior algorithms (Huang et al., 2015, 2012; Nallapati et al., 2008; Tanner and Charniak, 2015) learn an explicit representation for each training document separately that isn't a deterministic function of the document's words. This makes the model effectively transductive since a never-before-seen document does not have a ready-made representation. Similarly, Huang et al. (2012)'s method needs a candidate document to have at least one in-coming citation to be eligible for citation - this disadvantages newly published documents. Liu et al. (2015) form document representations using citation relations, which are not available for unfinished or new documents. In contrast, our method does not need to be re-trained as the corpus of potential candidates grows. As long as the new documents are in the same domain as that of the model's training documents, they can simply be added to the corpus and are immediately available as candidates for future queries.

While the citation recommendation task has attracted a lot of research interest, a recent survey paper (Beel et al., 2016) has found three main concerns with existing work: (i) limitations in evaluation due to strongly pruned datasets, (ii) lack of details for re-implementation, and (iii) variations in performance across datasets. For example, the average number of citations per document in the DBLP dataset is 5, but Ren et al. (2014) filtered out documents with fewer than 10 citations from the test set. This drastically reduced the size of the test set. We address these concerns by releasing a new large scale dataset for future citation recommendation systems. In our experiments on the OpenCorpus dataset, we only prune documents with zero outgoing citations. We provide extensive details of our system (see Appendix SSA) to facilitate reproducibility and release our code11. We also show in experiments that our method consistently outperforms previous systems on multiple datasets.

Footnote 11: [https://github.com/allenai/citeomatic](https://github.com/allenai/citeomatic)

Finally, recent work has combined graph node representations and text-based document representations using CCA (Gupta and Varma, 2017). This sort of approach can enhance our text-based document representations if a technique to create graph node representations at test-time is available.

## 7 Conclusion

In this paper, we present a content-based citation recommendation method which remains robust when metadata is missing for query documents, enabling researchers to do an effective literature search early in their research cycle or during the peer review process, among other scenarios. We show that our method obtains state of the art results on two citation recommendation datasets, even without the use of metadata available to the 

baseline method. We make our system publicly accessible online. We also introduce a new dataset of seven million scientific articles to facilitate future research on this problem.

## Acknowledgements

We would like to thank Oren Etzioni, Luke Zettlemoyer, Doug Downey and Iz Beltagy for participating in discussions and for providing helpful comments on the paper draft; Hsu Han and rest of the Semantic Scholar team at AI2 for creating the OpenCorpus dataset. We also thank Xiang Ren for providing the data used in their experiments on the DBLP and Pubmed datasets. Finally, we thank the anonymous reviewers for insightful comments on the draft.

## References

* Beel et al. (2016) Joern Beel, Bela Gipp, Stefan Langer, and Corinna Breitinger. 2016. Research-paper recommender systems: a literature survey. _International Journal on Digital Libraries_ 17(4):305-338.
* Bottou (2010) Leon Bottou. 2010. Large-scale machine learning with stochastic gradient descent. In _Proceedings of COMPSTAT'2010_, Springer, pages 177-186.
* Bromley et al. (1993) Jane Bromley, James W Bentz, Leon Bottou, Isabelle Guyon, Yann LeCun, Cliff Moore, Eduard Sackinger, and Roopak Shah. 1993. Signature verification using a siamese time delay neural network. _International Journal of Pattern Recognition and Artificial Intelligence_ 7(04):669-688.
* Caragea et al. (2014a) Cornelia Caragea, Florin Adrian Bulgarov, Andreea Godea, and Sujatha Das Gollapalli. 2014a. Citation-enhanced keyphrase extraction from research papers: A supervised approach. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, pages 1435-1446.
* Caragea et al. (2014b) Cornelia Caragea, Florin Adrian Bulgarov, Andreea Godea, and Sujatha Das Gollapalli. 2014b. Citation-enhanced keyphrase extraction from research papers: A supervised approach. In _EMNLP_. volume 14, pages 1435-1446.
* Clevert et al. (2015) Djork-Arne Clevert, Thomas Unterthiner, and Sepp Hochreiter. 2015. Fast and accurate deep network learning by exponential linear units (elus). _arXiv preprint arXiv:1511.07289_ .
* Gupta and Varma (2017) Shashank Gupta and Vasudeva Varma. 2017. Scientific article recommendation by using distributed representations of text and graph. In _WWW_.
* He et al. (2011) Qi He, Daniel Kifer, Jian Pei, Prasenjit Mitra, and C. Lee Giles. 2011. Citation recommendation without author supervision. In _WSDM_.
* He et al. (2010) Qi He, Jian Pei, Daniel Kifer, Prasenjit Mitra, and Lee Giles. 2010. Context-aware citation recommendation. In _Proceedings of the 19th international conference on World wide web_. ACM, pages 421-430.
* Huang et al. (2012) Wenyi Huang, Saurabh Kataria, Cornelia Caragea, Prasenjit Mitra, C Lee Giles, and Lior Rokach. 2012. Recommending citations: translating papers into references. In _Proceedings of the 21st ACM international conference on Information and knowledge management_. ACM, pages 1910-1914.
* Huang et al. (2015) Wenyi Huang, Zhaohui Wu, Liang Chen, Prasenjit Mitra, and C Lee Giles. 2015. A neural probabilistic model for context based citation recommendation. In _AAAI_. pages 2404-2410.
* Iyyer et al. (2015) Mohit Iyyer, Varun Manjunatha, Jordan L. Boyd-Graber, and Hal Daume. 2015. Deep unordered composition rivals syntactic methods for text classification. In _ACL_.
* Jia and Saule (2017) Haofeng Jia and Erik Saule. 2017. An analysis of citation recommender systems: Beyond the obvious. In _ASONAM_.
* Kataria et al. (2010) Saurabh Kataria, Prasenjit Mitra, and Sumit Bhatia. 2010. Utilizing context in generative bayesian models for linked corpus. In _AAAI_.
* Liu et al. (2015) Haifeng Liu, Xiangjie Kong, Xiaomei Bai, Wei Wang, Teshome Megersa Bekele, and Feng Xia. 2015. Context-based collaborative filtering for citation recommendation. _IEEE Access_ 3:1695-1703.
* Lopez and Romary (2010) Patrice Lopez and Laurent Romary. 2010. Humb: Automatic key term extraction from scientific articles in grobid. In _Proceedings of the 5th international workshop on semantic evaluation_. Association for Computational Linguistics, pages 248-251.
* Manning et al. (2014) Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David McClosky. 2014. The stanford corenlp natural language processing toolkit. In _ACL_.
* McNee et al. (2002) Sean M McNee, Istvan Albert, Dan Cosley, Prateep Gopalakrishnan, Syong K Lam, Al Mamunur Rashid, Joseph A Konstan, and John Riedl. 2002. On the recommending of citations for research papers. In _Proceedings of the 2002 ACM conference on Computer supported cooperative work_. ACM, pages 116-125.
* Nallapati et al. (2008) Ramesh Nallapati, Amr Ahmed, Eric P. Xing, and William W. Cohen. 2008. Joint latent topic models for text and citations. In _KDD_.
* Ren et al. (2014) Xiang Ren, Jialu Liu, Xiao Yu, Urvashi Khandelwal, Quanquan Gu, Lidan Wang, and Jiawei Han. 2014. Cluscite: Effective citation recommendation by information network-based clustering. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_. ACM, pages 821-830 

Tim Salimans and Diederik P. Kingma. 2016. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. _CoRR_ abs/1602.07868. [http://arxiv.org/abs/1602.07868](http://arxiv.org/abs/1602.07868).
* Strohman et al. (2007) Trevor Strohman, W. Bruce Croft, and David D. Jensen. 2007. Recommending citations for academic papers. In _SIGIR_.
* Tang and Zhang (2009) Jie Tang and Jing Zhang. 2009. A discriminative approach to topic-based citation recommendation. In _PAKDD_.
* Tanner and Charniak (2015) Chris Tanner and Eugene Charniak. 2015. A hybrid generative/discriminative approach to citation prediction. In _HLT-NAACL_.
* Totti et al. (2016) Luam C. Totti, Prasenjit Mitra, Mourad Ouzzani, and Mohammed J. Zaki. 2016. A query-oriented approach for relevance in citation networks. In _WWW_.
* Wang et al. (2014) Jiang Wang, Yang Song, Thomas Leung, Chuck Rosenberg, Jingbin Wang, James Philbin, Bo Chen, and Ying Wu. 2014. Learning fine-grained image similarity with deep ranking. In _Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition_. IEEE Computer Society, Washington, DC, USA, CVPR '14, pages 1386-1393. [https://doi.org/10.1109/CVPR.2014.180](https://doi.org/10.1109/CVPR.2014.180).
* Yang et al. (2015) Cheng Yang, Zhiyuan Liu, Deli Zhao, Maosong Sun, and Edward Y Chang. 2015. Network representation learning with rich text information. In _Proceedings of the 24th International Joint Conference on Artificial Intelligence, Buenos Aires, Argentina_. pages 2111-2117.
* Yu et al. (2012) Xiao Yu, Quanquan Gu, Mianwei Zhou, and Jiawei Han. 2012. Citation prediction in heterogeneous bibliographic networks. In _SDM_.

 

Hyperparameter Settings

Neural networks are complex and have a large number of hyperparameters to tune. This makes it challenging to reproduce experimental results. Here, we provide details of how the hyperparameters of the NNSelect and NNRank models were chosen or otherwise set. We chose a subset of hyperparameters for tuning, and left the rest at manually set default values. Due to limited computational resources, we were only able to perform hyperparameter tuning on the development split of the smaller DBLP and Pubmed datasets.

For DBLP and PubMed, we first ran Hyperopt12 with 75 trials. Each trial was run for five epochs of 500,000 triplets each. The ten top performing of these models were trained for a full 50 epochs, and the best performing model's hyperparameters are selected. Hyperparameters for NNSelect were optimized for Recall@20 and those for the NNRank model were optimized for F1@20 on the development set. The selected values for DBLP are reported in Table 6 and for PubMed are reported in Table 7.

Footnote 12: [https://github.com/hyperopt/hyperopt](https://github.com/hyperopt/hyperopt)

OpenCorpus hyperparameters were set via informal hand-tuning, and the results are in Table 9. A few miscellaneous parameters (not tuned) that are necessary for reproducibility are in Table 8.

We briefly clarify the meaning of some parameters below:

* The triplet loss has variable margins for the three types of negatives: \(0.1\gamma,0.2\gamma,\text{and}~{}0.3\gamma\). We treat \(\gamma\) as a hyperparameter and refer to it as the margin multiplier.
* For the majority of our experiments, we use a _Siamese_ model [Bromley et al., 1993]. That is, the textual embeddings for the query text and abstract share the same weights. However, we had a significantly larger amount of data to train NNRank on OpenCorpus, and found that non-Siamese embeddings are beneficial.
* We estimate word embeddings on the titles and abstracts of OpenCorpus using Word2Vec implemented by the gensim Python package13. Footnote 13: [https://radimrehurek.com/gensim/](https://radimrehurek.com/gensim/)

## Appendix B Margin Loss Details

When computing the margins for the triplet loss, we use a boosting function for highly cited documents. The full triplet loss function is as follows:

max \[\big{(} \gamma\alpha(d^{-})\] \[+s(q_{q},d^{-})+B(d^{-})\] \[-s(q_{q},d^{+})-B(d^{+})\] \[,0\big{)}\] where \(\gamma\) is the margin multiplier, and \(\alpha(d^{-})\) varies based on the type of negative document:

* \(\alpha(d^{-})=0.3\) for random negatives
* \(\alpha(d^{-})=0.2\) for nearest neighbor negatives
* \(\alpha(d^{-})=0.1\) for citation-of-citation negatives.

The boosting function is defined as follows:

\[B(d)=\frac{\sigma\left(\frac{d[\text{in-citations}]}{100}\right)}{50}\]

where \(\sigma\) is the sigmoid function and \(d[\text{in-citations}]\) is the number of times document \(d\) was cited in the corpus. The boosting function allows the model to slightly prefer candidates that are cited more frequently, and the constants were set without optimization.

## Appendix C Nearest Neighbors for Training Details

When obtaining nearest neighbors for negative examples during training, we use a heuristic to find a subset of the fetched nearest neighbors that are sufficiently wrong. That is, these are non-citation samples that look dissimilar in the original text but similar in the embedding space. This procedure is as follows for each training query:

1. Compute the Jaccard similarities between a training query and all of its true citations using the concatenation of title and abstract texts.
2. Compute the bottom fifth percentile Jaccard similarity value. I.e. the value below which only the bottom 5% most least textually similar true citations fall. For example, if the Jaccard similarities range from 0.2 to 0.9, the fifth percentile might plausibly be 0.3.
3. Use the Annoy index computed at the end of the previous epoch to fetch nearest neighbors for the query document.

 

4. Compute the textual Jaccard similarity between all of the nearest neighbors and the query document.
5. Retain nearest neighbors that have a smaller Jaccard similarity than the fifth percentile. Using the previous example, retain the nearest neighbors that have a lower Jaccard similarity than 0.3.

## Appendix D BM25 Details

Okapi-BM25 is a popular ranking function. We use BM25 as an IR-based baseline for the task of citation recommendation. For the DBLP and Pubmed datasets, BM25 performance is provided in Ren et al. (2014). To create a competitive BM25 baseline for OpenCorpus, we first created indexes for the DBLP and Pubmed datasets and tuned the query to approximate the performance reported in previous work. We used Whoosh14 to create an index. We extract the key terms (using Whoosh's key_terms_from_text method) from the title and abstract of each query document. The key terms from the document are concatenated to form the query string. Table 5 shows that our BM25 is a close approximation to the BM25 implementation of previous work and can be reliably used as a strong IR baseline for OpenCorpus. In Table 2, we report results on all three datasets using our BM25 implementation.

Footnote 14: [https://pypi.python.org/pypi/Whoosh/](https://pypi.python.org/pypi/Whoosh/)

## Appendix E Key Phrases for OpenCorpus

In the OpenCorpus dataset, some documents are accompanied by automatically extracted key phrases. Our implementation of automatic key phrase extraction is based on standard key phrase extraction systems - e.g. Caragea et al. (2014, 2014); Lopez and Romary (2010). We first extract noun phrases using the Stanford CoreNLP package Manning et al. (2014) as candidate key phrases. Next, we extract corpus level and document level features (e.g. term frequency, document frequency, n-gram probability etc.) for each candidate key phrase. Finally, we rank the candidate key phrases using a ranking model that is trained on author-provided key phrases as gold labels.

## Appendix F Detailed Experimental Results

Table 10 compares NNRank with previous work in detail on DBLP and Pubmed datasets. ClustCite Ren et al. (2014) clusters nodes in a heterogeneous graph of terms, authors and venues in order to find related documents which should be cited. ClusCite obtains the previous best results on these two datasets. L2-LR Yu et al. (2012) uses a linear combination of meta-path based linear features to classify candidate citations. We show that NNRank (with and without metadata) consistently outperforms ClusCite and other baselines on all metrics on both datasets.

\begin{table}
\begin{tabular}{l|c c|c c}  & \multicolumn{2}{c|}{**DBLP**} & \multicolumn{2}{c}{**PubMedMed**} \\  & F@20 & MRR & F@20 & MRR \\ \hline Ren et al. (2014) & 0.111 & 0.411 & 0.153 & 0.497 \\ Our approximation & 0.119 & 0.425 & 0.209 & 0.574 \\ \end{tabular}
\end{table}
Table 5: Results of our BM25 implementation on DBLP and Pubmed datasets.

 



\begin{table}
\begin{tabular}{l|c c l|c l} \hline  & \multicolumn{3}{c|}{**NNSelect**} & \multicolumn{2}{c}{**NNRank**} \\
**Hyperparameter** & **Range** & **Chosen Value** & **Range** & **Chosen Value** \\ \hline learning rate & [1e-5, 1e-4, ..., 1e-1] & 0.01 & [1e-5, 1e-4, ..., 1e-1] & 0.01 \\
12 regularization & [0, 1e-7, 1e-6, ..., 1e-2] & 0 & [0, 1e-7, 1e-6, ..., 1e-2] & 1e-3 \\
11 regularization & [0, 1e-7, 1e-6, ..., 1e-2] & 1e-7 & [0, 1e-7, 1e-6, ..., 1e-2] & 1e-4 \\ word dropout & [0, 0.05, 0.1, ..., 0.75] & 0.60 & [0, 0.05, 0.1, ..., 0.75] & 0.35 \\ margin multiplier & [0.5, 0.75, 1.0, 1.25, 1.5] & 1.0 & [0.5, 0.75, 1.0, 1.25, 1.5] & 0.5 \\ dense dimension & [25, 50, ..., 325] & 300 & [25, 50, ..., 325] & 175 \\ metadata dimension & - & - & [5, 10, ..., 55] & 45 \\ use pretrained & [true, false] & true & [true, false] & false \\ finetune pretrained & [true, false] & true & [true, false] & - \\ number ANN neighbors & - & 10 & - & - \\ triplets per batch size & - & \(256\) & - & \(256\) \\ triplets per epoch & - & \(500000\) & - & \(500000\) \\ triplets per training & - & \(2500000\) & - & \(2500000\) \\ use Siamese embeddings & - & true & - & true \\ \hline \end{tabular}
\end{table}
Table 6: DBLP hyperparameter tuning results. Note that the dense dimension when using pretrained vectors is fixed to be 300. A ’-’ indicates that the variable was not tuned.

\begin{table}
\begin{tabular}{l|c l|l l} \hline  & \multicolumn{2}{c|}{**NNSelect**} & \multicolumn{2}{c}{**NNRank**} \\
**Hyperparameter** & **Range** & **Chosen Value** & **Range** & **Chosen Value** \\ \hline learning rate & [1e-5, 1e-4, ..., 1e-1] & 0.001 & [1e-5, 1e-4, ..., 1e-1] & 0.001 \\
12 regularization & [0, 1e-7, 1e-6, ..., 1e-2] & 0 & [0, 1e-7, 1e-6, ..., 1e-2] & 0 \\
11 regularization & [0, 1e-7, 1e-6, ..., 1e-2] & 1e-6 & [0, 1e-7, 1e-6, ..., 1e-2] & 1e-6 \\ word dropout & [0, 0.05, 0.1, ..., 0.75] & 0.55 & [0, 0.05, 0.1, ..., 0.75] & 0.1 \\ margin multiplier & [0.5, 0.75, 1.0, 1.25, 1.5] & 0.5 & [0.5, 0.75, 1.0, 1.25, 1.5] & 1.5 \\ dense dimension & [25, 50, ..., 325] & 325 & [25, 50, ..., 325] & 150 \\ metadata dimension & - & - & \([5,10, ..., 55]\) & 40 \\ use pretrained & [true, false] & false & [true, false] & false \\ finetune pretrained & [true, false] & - & [true, false] & - \\ number ANN neighbors & - & 10 & - & - \\ triplets per batch size & - & \(256\) & - & \(256\) \\ triplets per epoch & - & \(500000\) & - & \(500000\) \\ triplets per training & - & \(2500000\) & - & \(2500000\) \\ use Siamese embeddings & - & true & - & true \\ \hline \end{tabular}
\end{table}
Table 7: PubMed hyperparameter tuning results. Note that the dense dimension when using pretrained GloVe vectors is fixed to be 300. A ‘-’ indicates that the variable was not tuned.





\begin{table}
\begin{tabular}{l|c c c c c|c c c c c} \hline  & \multicolumn{6}{c|}{**DBLP**} & \multicolumn{5}{c}{**PubMedMed**} \\
**Method** & **P@10** & **P@20** & **R@20** & **F1@20** & **MRR** & **P@10** & **P@20** & **R@20** & **F1@20** & **MRR** \\ \hline BM25 & 0.126 & 0.0902 & 0.1431 & 0.11 & 0.4107 & 0.1847 & 0.1349 & 0.1754 & 0.15 & 0.4971 \\ L2-LR & 0.2274 & 0.1677 & 0.2471 & 0.200 & 0.4866 & 0.2527 & 0.1959 & 0.2504 & 0.2200 & 0.5308 \\ CluseCite & 0.2429 & 0.1958 & 0.2993 & 0.237 & 0.5481 & 0.3019 & 0.2434 & 0.3129 & 0.274 & 0.5787 \\ \hline NNSelect & 0.287 & 0.230 & 0.363 & 0.282 & 0.579 & 0.388 & 0.316 & 0.302 & 0.309 & 0.699 \\ + NNRank & 0.339 & 0.247 & 0.390 & 0.302 & 0.672 & 0.421 & 0.332 & 0.318 & 0.325 & 0.754 \\ + metadata & **0.345** & **0.247** & **0.390** & **0.303** & **0.689** & **0.429** & **0.337** & **0.322** & **0.329** & **0.771** \\ \hline \end{tabular}
\end{table}
Table 10: Comparing NNRank with ClusCite. (Ren et al., 2014) have presented results on several other topic-based, link-based and network-based citation recommendation methods as baselines. For succinctness, we show results for the best system, Cluscite, and two baselines BM25 and L2-LR.

\begin{table}
\begin{tabular}{l|l l} \hline
* & \(25\) \\ use pretrained & false & false \\ number ANN neighbors & 5 &
- \\ triplets per batch size & \(256\) & \(32\) \\ triplets per epoch & \(2500000\) & \(2500000\) \\ triplets per training & \(25000000\) & \(1000000000\) \\ use Siamese embeddings & true & false \\ \hline \end{tabular}
\end{table}
Table 8: Per-dataset parameters. These were hand-specified. *LazyAdamOptimizer is part of TensorFlow. **NAdam is part of Keras.

\begin{table}
\begin{tabular}{l|l l} \hline
**Hyperparameter** & **PubMed/DBLP Value** & **OpenCorpus Value** \\ \hline title/abstract vocabulary size & \(200000\) & \(200000\) \\ maximum title length & \(50\) & \(50\) \\ maximum abstract length & \(500\) & \(500\) \\ training triplets per query & \(6\) & \(6\) \\ min # of papers per author included & 1 & 10 \\ min # of papers per venue included & 1 & 10 \\ min # of papers per keyphrases included & 5 & 10 \\ max authors per document & 8 & 8 \\ max keyphrases per document & 20 & 20 \\ minimum true citations per document & 2/1 & 2 \\ maximum true citations per document & 100 & 100 \\ optimizer & LazyAdamOptimizer* & Nadam** \\ use magnitude-direction embeddings & true & true \\ reduce learning rate upon plateau & false & true \\ \hline \end{tabular}
\end{table}
Table 9: Hyperparameters used for OpenCorpus