SlowGuess/NAACL_2022_OCR huggingface dataset:
Sheshera Mysore $^{1*}$  Arman Cohan $^{2,3}$  Tom Hope $^{2,3}$
<sup>1</sup>University of Massachusetts Amherst, MA, USA <sup>2</sup>University of Washington, WA, USA
<sup>3</sup>Allen Institute for Artificial Intelligence, WA, USA
smysore@cs.umass.edu {tomh,armanc}@allenai.org
We present a new scientific document similarity model based on matching fine-grained aspects of texts. To train our model, we exploit a naturally-occurring source of supervision: sentences in the full-text of papers that cite multiple papers together (co-citations). Such cocitations not only reflect close paper relatedness, but also provide textual descriptions of how the co-cited papers are related. This novel form of textual supervision is used for learning to match aspects across papers. We develop multi-vector representations where vectors correspond to sentence-level aspects of documents, and present two methods for aspect matching: (1) A fast method that only matches single aspects, and (2) a method that makes sparse multiple matches with an Optimal Transport mechanism that computes an Earth Mover's Distance between aspects. Our approach improves performance on document similarity tasks in four datasets. Further, our fast single-match method achieves competitive results, paving the way for applying fine-grained similarity to large scientific corpora.<sup>1</sup>
The ability to identify similarity across documents in large scientific corpora is fundamental for many applications, including recommendation (Bhagavatula et al., 2018), exploratory or analogical search (Hope et al., 2017, 2021b; Lissandrini et al., 2019), paper-reviewer matching (Mimno and McCallum, 2007; Berger et al., 2020) and many more uses.
Scientific papers often describe multifaceted arguments and ideas (Hope et al., 2021a; Lahav et al., 2022), suggesting that models capable of matching specific aspects can better capture overall document relatedness, too. For example, sentences in research abstracts can often be categorized as descriptions of objectives, methods, or findings (Kim
et al., 2011; Chan et al., 2018), centrally important discourse structures of scientific texts.
In this paper, we propose a new model for document similarity that makes aspect-level matches across papers and aggregates them into a document-level similarity. We focus on sentence-level aspects of paper abstracts, and train multi-vector representations of papers in terms of their contextualized sentence embeddings. To train our models, we leverage a readily available data source: sentences that co-cite multiple papers. Unlike recent work that used citation links for learning scientific document similarity (Cohan et al., 2020), we observe that papers cited in close proximity provide a more precise indication of relatedness. Furthermore, the citing sentences typically describe how the co-cited papers are related, in terms of shared aspects (e.g., similar methods or findings, related challenges or directions, etc.). Building on this observation, we leverage these textual descriptions as a novel source of textual supervision, using them to guide our model to learn which sentence-aspects match without any direct sentence-level supervision. Guidance for the document similarity model is obtained via an auxiliary sentence encoder model that is used for aligning abstract sentences by finding pairs most similar to the citing sentence text.
Our document similarity objective is modeled as a function of similarity between sentence-level matches. We explore two strategies to aggregate over sentence-level distances between documents. First, a single-match method with minimum L2 distances between document aspect vectors. This approach readily supports approximate nearest neighbor search methods for large-scale retrieval. Second, a multi-match method that computes an Earth Mover's Distance between documents' aspect vectors by solving an Optimal Transport problem. This yields a soft sparse matching of aspect vectors, which when combined with their L2 distances gives a document-level distance.
* Part of the work done during internship at AI2.
1Code, data, and models available at: https:// github.com/allenai/aspire
Human Language Technologies, pages 4453 - 4470
Finally, as an additional benefit of our representation, our models also support a finer aspect-conditional retrieval task (Hope et al., 2017, 2021a; Chan et al., 2018; Mysore et al., 2021) where aspects can be specified by selecting abstract sentences — for example, selecting sentences describing methods and retrieving papers using similar methods. As we show, naively encoding sentences without their context leads to subpar results in this task, and our representation that does take context into account dramatically improves results.
Extensive empirical evaluation on four English scientific text datasets and seven similarity tasks at the level of documents and sentences demonstrates the effectiveness of our models. These include biomedical document retrieval tasks and a recent faceted query-by-example corpus of computer science papers (Mysore et al., 2021). This latter dataset is used for evaluating retrieval conditioned on specific aspects in context (e.g., for finding papers with similar methods to a query document), demonstrating that our model can be used in this challenging and important setting. In summary, we make the following main contributions:
1. Multi-Vector Document Similarity Model: We present ASPIRE, a multi-vector document similarity model that flexibly aggregates over fine-grained sentence-level aspect matches.
2. Co-Citation Context Supervision: We exploit widely-available co-citation sentences as a new source of training data for document similarity and provide a method using a novel form of textual supervision to guide representation learning for aspect matching.
3. State of the Art Results: Our ASPIRE models outperforms strong baseline methods across four datasets for the abstract and aspect-conditional similarity tasks.
Given query document  $Q$  and a candidate amongst a set of documents  $C \in \mathcal{C}$ , where documents consist of  $N$  sentences  $\langle S_1, S_2, \ldots, S_N \rangle$  we aim to leverage fine-grained document similarity in two problem settings. An abstract level retrieval task (Brown et al., 2019; Cohan et al., 2020) and an aspect-level retrieval task (Mysore et al., 2021):
Def 1. Retrieval by abstracts: Given query and candidate documents -  $Q$  and  $\mathcal{C}$  a system must output the ranking over  $\mathcal{C}$ .
Def 2. Aspect-level retrieval by sentences: Given query and candidate documents  $-Q$  and  $\mathcal{C}$ , and a subset of sentences  $S_{Q} \subseteq Q$  conditional on which to retrieve documents, a system must output the ranking over  $\mathcal{C}$ .
Modeling Desiderata: Next, we also outline key desired properties we require from models developed for task definitions 1 and 2. We follow these desiderata when building our methods (§3.1).
1. Allowing specification of optional fine-grained aspects: We would like models to allow the ability to specify fine-grained query aspects in a query document based on which retrievals should be made. These may be obtained automatically (e.g., with a discourse tagging method) or via user specification.
2. Scalable to large corpora and efficient inference: State of the art retrieval systems often rely on expensive cross-attention mechanisms on query-document pairs making training and inference expensive (Zamani et al., 2018; Lin et al., 2021). This is exacerbated for longer scientific documents requiring specific transformer models (Caciularu et al., 2021). We require our methods to leverage large training corpora and allow efficient inference at scale.
3 Proposed Approach: ASPIRE
In this section we describe our approach to document similarity - ASPIRE. We model finer-grained matches between documents at the level of sentences via contextual representations and aggregating over matches to obtain similarities between whole documents. We leverage co-citation sentences as a source of document similarity and also as implicit textual supervision describing related aspects of co-cited documents. We formulate our multi-vector models (Luan et al., 2021; Humeau et al., 2020) that can support scalable inference as novel multiple-instance learning (MIL) models.
3.1 Fine-grained Document Similarity
We assume to be given a training set consisting of sets of documents  $\mathcal{P}$  which are weakly-labeled for similarity. We leverage widely-available sets of papers co-cited together in the same sentence as similar (see Figure 1). This builds on the observation that co-citations in close proximity (e.g., in the same sentence) are strong indicators for paper relatedness (Gipp and Beel, 2009).
We follow the contrastive learning framework, commonly used for learning semantic similarity (Reimers and Gurevych, 2019; Cohan et al., 2020).
2 ASPIRE: Aspectual Scientific Paper Relations.
Figure 1: Example illustrating the signal in co-citations. Of all the sentences in co-cited abstracts  $\text{a}$  and  $\text{c}$ , the sentences shown are each individually aligned to co-citation context  $\text{b}$  as per embeddings from  $\text{BERT}_{\mathcal{E}}$  (\$3.2.2). Consequently these sentences in  $\text{a}$  and  $\text{c}$  are treated as sharing aspects between the co-cited papers and our fine-grained similarity model for single matching is trained to align them.
We train models on triples of the form  $(p, p', n)$  where  $p, p' \in \mathcal{P}$  and  $n \notin \mathcal{P}$  is a randomly selected negative, using the triple margin ranking loss  $\mathcal{L}_f(p, p', n) = \max [f(p, p') - f(p, n) + m, 0]$ , where  $f(\cdot, \cdot)$  is a distance between documents. All pairwise-combinations  $p, p' \in \mathcal{P}$  are treated as positive pairs in-turn. In this work, we parameterize  $f$  based on the distances between finer-grained document aspects  $\mathcal{A}$ . Given documents  $p$  and  $p'$ , we focus on a family of functions  $f$  of the form:
$$
f (p, p ^ {\prime}) = \sum_ {(i, i ^ {\prime}) \in \mathcal {A} _ {p} \times \mathcal {A} _ {p ^ {\prime}}} w _ {i, i ^ {\prime}} \cdot d _ {i, i ^ {\prime}}. \tag {1}
$$
Here,  $\mathcal{A}_p\times \mathcal{A}_p^\prime$  represents the space of alignments between aspects of document  $p$  and  $p^{\prime}$ ,  $d_{i,i^{\prime}}$  denotes a distance between two aspects  $i,i^{\prime}$ , and  $w_{i,i^{\prime}}$  represents a weight indicating the contribution of the aspect similarity to the overall document similarity. Unlike previous work (Neves et al., 2019; Jain et al., 2018; Hope et al., 2017), we make no assumption on specific aspect semantics in deriving a model architecture, and focus on aspects in the form of general subsets of document sentences.
For learning, we only assume to be given document-level supervision (sets of documents  $\mathcal{P}$ ), and no gold supervision on aspect-level similarity as in other related work, eg. Jain et al. (2018). Our task thus consists of learning  $w_{i,i'}$  and  $d_{i,i'}$  via indirect supervision. We cast this problem setting as a novel type of multi-instance learning (MIL) (Ilse et al., 2018) problem. Prior work in MIL broadly aims to learn instance level classifiers given labels for a bag of instances, this bears resemblance to our setting, where instances are aspects  $\mathcal{A}$ . However, unlike prior MIL work we focus on learning similarity rather than classification. We formulate two variants of  $f$  in Equation 1:
(1) A single match model (§3.2.2) which considers documents similar based on the single most similar alignment  $\hat{i}_p, \hat{i}_{p'} \in \mathcal{A}_p \times \mathcal{A}_p'$ . This assumes  $w = 1$  for the best alignment and  $w = 0$  elsewhere.
(2) A multi match model (§3.2.3) which makes multiple alignments between documents. We find aspect importance weights  $w_{i,i'}$ , by solving an Optimal Transport (OT) problem (Peyre et al., 2019).
In both variants, during training we learn contextualized aspect embeddings that minimize the contrastive loss paramertized with  $f$ , described further in §3.2.
Co-citation Contexts as Supervision: Finally, we present a method for incorporating implicit natural language supervision during training, presented by co-citation sentences which describe specific relations between co-cited documents. For example, Figure 1 shows a case explaining the similarity between the co-cited papers' methods. We leverage this textual supervision to find a "best" alignment  $\hat{i}_p, \hat{i}_{p'}$  in the single-alignment variant (1), and for guiding the optimal transport plan in variant (2). We describe the specific model components next. Fig 2a presents a schematic for our approach.
3.2.1 Document Encoder
We leverage a pre-trained BERT-based language model as a document encoder as the base of all our methods. Our encoder is mainly intended to output contextualized sentence representations. Given a document title and abstract, this is achieved as:
$$
\mathbf {S} = \operatorname {B E R T} _ {\theta} ([ \text {C L S} ] \text {T i l t e} [ \text {S E P} ] \text {A b s t r a c t}) \tag {2}
$$
where  $\mathbf{S} \in \mathbb{R}^{N \times d}$  represents contextualized sentences  $\mathbf{s}_1 \ldots \mathbf{s}_N$  stacked into a matrix. Here, each
(a) Learning fine-grained document similarity using co-citations.
(b) The single best match,  $\hat{i}_p,\hat{i}_{p'}$  , is computed from textual supervision in the co-citation context.
(c) Multi-aspect matches via a sparse transport plan.
Figure 2: Approach overview. (a) We train fine-grained similarity models using papers co-cited in the same sentence in research papers. (b) Single-match models are learned from implicit supervision in co-citation contexts. (c) Multi-match models are learned by aligning aspect representations by solving an Optimal Transport problem.
s is obtained by mean-pooling word-piece embeddings from the final layer of  $\mathrm{BERT}_{\theta}$  for the sentence tokens. Pairwise distances between sentences  $d_{i,i'}$  in Eq 1 for  $p, p'$ , are represented as a matrix  $\mathbf{D} \in \mathbb{R}^{N \times N'}$  of L2 distances between  $\mathbf{S}_p$  and  $\mathbf{S}_{p'}$ .
3.2.2 Single Match & Textual Supervision
Our single match model makes the assumption that document similarity is explained by a single best match, giving  $f_{\mathrm{TS}}(p,p^{\prime}) = \mathbf{D}[\hat{i}_{p},\hat{i}_{p^{\prime}}]$ . Here, we leverage weak supervision from co-citation contexts for training. This is done by using an auxiliary sentence encoder to compute a maximally aligned sentence  $\hat{i}_p$  in co-cited paper  $p$  to the co-citation context, similarly  $\hat{i}_{p'}$  aligns a sentence in  $p^\prime$  to the co-citation context. Then the two context aligned sentences are treated as aligned to each other, for training. In practice, the same papers  $\mathcal{P}$  can be co-cited in multiple different papers (in  $\sim 30\%$  of co-cited papers) giving us a set of co-citation sentences,  $e\in \mathcal{E}$  and training data of the form  $(\mathcal{E},\mathcal{P})$ . Alignments of the sentences in  $p$  and  $p^\prime$  to the co-citation contexts  $e\in \mathcal{E}$  are computed as:
$$
\begin{array}{l} \hat {i} _ {p}, \hat {k} _ {p} = \underset {i = 1 \dots N, k = 1 \dots N ^ {\prime}} {\operatorname {a r g m a x}} \mathbf {R} _ {p} \mathbf {R} _ {\mathcal {E}} ^ {T} \\ \hat {i} _ {p ^ {\prime}}, \hat {k} _ {p ^ {\prime}} = \underset {i = 1 \dots N, k = 1 \dots N ^ {\prime}} {\operatorname {a r g m a x}} \mathbf {R} _ {p ^ {\prime}} \mathbf {R} _ {\mathcal {E}} ^ {T} \tag {5} \\ \end{array}
$$
Here  $\mathbf{R}_p, \mathbf{R}_{p'}$ , and  $\mathbf{R}_{\mathcal{E}}^T$  are independent sentence representations for  $p, p'$  and  $e$ , respectively, obtained from a auxiliary sentence encoder BERT $_{\mathcal{E}}$  (details below), and  $\hat{i_p}, \hat{i}_{p'}$  represent the single best alignment of sentences across  $p, p'$  "anchored" on textual supervision sentences  $\mathcal{E}$ . Importantly, this supervision is only used during training time to guide learning. This procedure is depicted in Figure 2b with Figure 1 showing an example.
Co-citation Context Encoder The encoder  $\mathrm{BERT}_{\mathcal{E}}$  represents a ScIBERT based sentence encoder pre-trained for scientific text similarity. We train  $\mathrm{BERT}_{\mathcal{E}}$  on sets of co-citation contexts referencing the same set of papers (i.e.  $\mathcal{E}$ ) in a contrastive learning setup with random in-batch negative samples. This set,  $\mathcal{E}$ , can be considered as paraphrases since co-citation sentences citing the same papers often describe similar relations between the papers. This model is similar to Sentence-BERT (Reimers and Gurevych, 2019) and we refer to it as CoSent-Bert. In training document encoder  $\mathrm{BERT}_{\theta}$ , we keep  $\mathrm{BERT}_{\mathcal{E}}$  frozen. Appendix C presents more detail on  $\mathrm{BERT}_{\mathcal{E}}$  design.
3.2.3 Multiple Matches & Optimal Transport
While a single best sentence alignment  $\hat{i}_p, \hat{i}_{p'}$  may sufficiently explain document similarity for some documents and applications, documents often have
a stronger and weaker alignments. So, in computing sentence alignments between documents we would like a sparse matching that aptly weights alignments while ignoring non-alignments — corresponding to learning weights  $w_{i,i'}$  in Eq 1. To model this intuition we leverage optimal transport.
Optimal Transport The OT problem is constituted by two sets of points,  $\mathbf{S}_p$  and  $\mathbf{S}_{p'}$  as in our case, and distributions  $\mathbf{x}_p$  and  $\mathbf{x}_{p'}$  according to which the set of points is distributed. The OT problem involves computation of a transport plan  $\hat{\mathbf{P}}$ , which converts  $\mathbf{x}_p$  into  $\mathbf{x}_{p'}$  by transporting probability mass while minimizing an aggregate cost computed from the pairwise costs  $\mathbf{D}$  of aligning the points in  $\mathbf{S}_p$  and  $\mathbf{S}_{p'}$ .  $\hat{\mathbf{P}}$  is constrained such that its columns and rows marginalize respectively to  $\mathbf{x}_p$  and  $\mathbf{x}_{p'}$  (so that all mass is accounted for). Specifically, the computation of  $\hat{\mathbf{P}}$  takes the form of a constrained linear optimization problem:
$$
\begin{array}{l} \mathcal {W} = \min  _ {\mathbf {P} \in \mathcal {S}} \langle \mathbf {D}, \mathbf {P} \rangle (4) \\ = \min  _ {\mathbf {P} \in \mathcal {S}} \sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {N ^ {\prime}} \mathbf {D} [ i, j ] \mathbf {P} [ i, j ] (5) \\ \end{array}
$$
$$
\mathcal {S} = \left\{\mathbf {P} \in \mathbb {R} _ {+} ^ {N \times N ^ {\prime}} \mid \mathbf {P} \mathbf {1} _ {N ^ {\prime}} = \mathbf {x} _ {p}, \mathbf {P} ^ {T} \mathbf {1} _ {N ^ {\prime}} = \mathbf {x} _ {p ^ {\prime}} \right\} \tag {6}
$$
where  $\mathcal{W}$  refers to the Wasserstein or Earth Movers Distance and  $\hat{\mathbf{P}}$  is the minimizer resulting from solving Eq 5. Of interest here is an established result which shows  $\hat{\mathbf{P}}$  to be sparse with  $\mathcal{O}(N + N')$  non-zero entries (Swanson et al., 2020). Therefore,  $\hat{\mathbf{P}}$  represents a soft sparse alignment of sentences and can be used as weights  $w_{i,i'}$  in Eq 1, with document distances computed as  $f_{\mathrm{OT}}(p,p') = \langle \mathbf{D},\hat{\mathbf{P}}\rangle$ . Fig 2c presents a schematic for this approach.
Note that  $\mathbf{x}_p$  and  $\mathbf{x}_{p'}$  allow control over importance of sentences in  $p$  and  $p'$  in the form of relative probability mass. We compute these distributions using pairwise distances as  $\mathbf{x} = \text{softmax}(-\mathbf{s} / \tau)$  where  $\mathbf{s}_p = \min_i \mathbf{D}$  and  $\mathbf{s}_{p'} = \min_j \mathbf{D}$ , and  $\tau$  is a softmax temperature hyper-parameter.
For our neural network models trained with automatic differentiation, we leverage an entropy regularized version of the Wasserstein distance in Eq 5 (Cuturi, 2013). Here computation of  $\hat{\mathbf{P}}$ , is achieved via Sinkhorn iterations, a set of iterative linear updates allowing training with autodiff libraries and leveraging GPU computation. Finally, Cuturi (2013) show that computing  $\mathcal{W}$  with Sinkhorn iterations shows an empirical quadratic complexity, i.e.  $\mathcal{O}(N^2)$  — similar to that of attention as in a model for late interaction (Humeau et al., 2020).
Multi-task model: To leverage training signals used in both the single and multi-match models, we train a multi match model supervised with textual supervision in a multi-task setup:  $\mathcal{L}_{f_{\mathrm{TS}}} + \mathcal{L}_{f_{\mathrm{OT}}}$ .
As outlined in §2, we are interested in a wholeabstract based retrieval (Def 1) and an aspect level retrieval (Def 2). In both setups given a query  $Q$  and candidate  $C$  documents we denote sentence representations from a trained model by  $\mathbf{S}_Q$  and  $\mathbf{S}_C$ . For both tasks, we compute distances for ranking while controlling the aspects  $\mathcal{A}_Q$  (i.e  $\mathcal{A}_p$ ) over which the weighted sum of Eq 1 is performed.
Whole abstract retrieval: This corresponds to a setup where all aspects of the query document  $\mathcal{A}_p$  are used in computing distances between documents. In the single-alignment models, candidates  $C$  are ranked based on their maximally aligned sentence with  $Q$  using distances from a trained model:  $\hat{i}_p, \hat{i}_{p'} = \operatorname{argmin}_{i,j} \mathbf{D}$ . The multi match model ranks candidates using the distance  $\langle \mathbf{D}, \hat{\mathbf{P}} \rangle$ , where  $\hat{\mathbf{P}}$  is the solution to transport problem of Eq 5.
Aspect level retrieval: In aspect-level retrieval, a subset of sentences  $\mathcal{A}_q\subset \mathcal{A}_Q$  is used for query document  $Q$ ; for candidate documents  $C$ , we do not assume to be given specific aspects, and matching is done across all sentences in each  $C$ . In the single alignment models, we only consider a subset of the pairwise sentence distances to determine the maximally aligned sentences, giving  $\mathbf{D}^{\mathcal{A}} = \mathbf{D}[\mathcal{A}_q,:]$ . This corresponds to finding the maximally aligned candidate sentence to the query sentences in  $\mathcal{A}_q$ . Similarly, in the multiple-alignment model we compute the plan  $\hat{\mathbf{P}}^{\mathcal{A}_q}$  based on the subset of sentences corresponding to  $\mathcal{A}_q$  and generate rankings by  $\langle \mathbf{D}^{\mathcal{A}_q},\hat{\mathbf{P}}^{\mathcal{A}_q}\rangle$ . Note that  $\mathbf{S}_Q$  in  $Q$  is still contextualized, capturing document context of sentences not explicitly used in  $\mathcal{A}_q$ .
Scaling Inference: Our multi-vector model for single matching performs retrievals via minimum L2 distance. Therefore, this method is amenable to approximate nearest neighbour (ANN) search methods for large-scale retrieval (Andoni et al., 2018; Luan et al., 2021). Retrieval with our single-match model would involve  $|\mathcal{A}_Q|$  and  $|\mathcal{A}_q|$  calls to an ANN structure for the whole abstract and aspect-level tasks respectively.
On the other hand, as stated earlier our multimatch model using Sinkhorn iterations involves a  $\mathcal{O}(N^2)$  computation (Cuturi, 2013), which is
similar to late interaction methods. Humeau et al. (2020) show late interaction models to be significantly cheaper than cross-encoders while retaining most of their performance in ad-hoc search setups. While quadratic, OT computation in practice can be time-consuming, however, recent work of Backurs et al. (2020) has seen development of fast ANN methods for Wasserstein distances with practical run-times significantly smaller than quadratic ones. This promises the use of ANN methods in large-scale retrieval with our multi-match model
In our results we refer to our text supervised single match method as TSASPIRE, optimal transport multi match method as OTASPIRE, and the multitask trained multi aspect method as TS+OTASPIRE.
Evaluation data: We evaluate the proposed methods on datasets for whole abstract document similarity and fine-grained document similarity. We overview these below. Appendix B provides detail.