SlowGuess/NAACL_2022_OCR huggingface dataset:
1. RELISH: An expert annotated dataset of biomedical abstract similarity (Brown et al., 2019).
2. TRECCOVIDRF: The original TRECCOVID dataset is labelled for ad-hoc search by experts (Voorhees et al., 2021). We reformulate the dataset for abstract similarity, treating all abstracts relevant to one ad-hoc query as similar to each other and dissimilar from abstracts relevant to other queries.
3. SciDOCS: A benchmark suite of tasks intended for evaluating abstract-level scientific document representations (Cohan et al., 2020).
4. CSFCUBE: Fine-grained retrieval is evaluated using the recent dataset of Mysore et al. (2021), an expert-annotated dataset of machine learning and NLP abstracts labelled against candidates for relevance to one of 3 broad aspects capturing the main components of methodological research: background/objective, method, result. Relevance is labelled for query sentences corresponding to those aspects, while considering the broader relevance of the sentences' abstract context.
Baselines: We compare the proposed approaches to three classes of methods. We overview these classes and associated models below, with Appendix D presenting further detail: 1. Sentence models: Sentence embedding models present reasonable baselines since we consider fine-grained matches at the sentence level. These are represented by MPNET-1B, a sentence model
trained on over 1 billion text pairs<sup>3</sup>, Sentence-Bert (SENTBERT) (Reimers and Gurevych, 2019), SIM-CSE (Gao et al., 2021), cosentbert of §3.2.2, and ICTSENTBERT (Lee et al., 2019).
2. Abstract models: The abstract level model SPECTER (Cohan et al., 2020), represents a SOTA model for scientific document similarity trained on cited abstract pairs. We also train a variant of this model on co-cited papers: SPECTER-COCITE. Finally, we also compare to ScINCL, introduced in recent concurrent work of Ostendorff et al. (2022). SCINCL presents a bi-encoder model similar to SPECTER with improvements to its contrastive learning procedure - presenting a complementary direction to our approach.
3. Sentence level models modified for whole abstract similarity: Here we combine the SOTA sentence encoder MPNET-1B with the optimal transport (§3.2.3) for aggregating sentence level matches giving OTMPNET-1B.
Sentence models use the same inference procedure as our single match method, abstract models rank using L2 distances between papers embeddings, and the modified sentence model uses the multi match inference procedure. All reported model hyper-parameters are tuned, trained on 1.3M co-citation triples, and initialized with SPECTER unless noted otherwise. In reporting results, we report standard retrieval metrics Mean Average Precision (MAP) and NDCG at rank K. For NDCG@K we follow Wang et al. (2013), and choose  $\mathrm{K} = p*\left|\mathcal{C}\right|$  where  $p\in (0,1)$ . NDCG%20 therefore refers to NDCG computed at 20% of the pool size for a query. This is apt since queries have varying pool sizes. Appendices A, E, and F detail training data, algorithms, and hyper-parameters. Next, we present our main results comparing proposed approaches to baselines.
Fine-grained similarity: Table 1 presents results on CSFCUBE. We report performance on the three facets background, method, and result annotated in the dataset, and aggregated across all facets. We first make some observations about baseline methods: 1. MPNET-1B outperforms all other sentence level models and a SOTA abstract representation, SPECTER, indicating the value of sentence-level information for capturing fine-grained
3MPNET-1B: https://bit.ly/2Zbm2Iq
4 Initialization indicated via subscript in tables.
CSFCUBE facets →
Models	Aggregated	Background	Method	Result
MAP	NDCG%20	MAP	NDCG%20	MAP	NDCG%20	MAP	NDCG%20
MPNET-1B	34.64	54.94	41.06	65.86	27.21	42.48	36.07	54.94
SENTBERT-PP	26.77	48.57	35.43	60.80	16.19	33.40	29.16	48.57
SENTBERT-NLI	25.23	45.39	30.78	54.23	15.02	31.10	30.27	45.39
UNSIMCSE-BERT	24.45	42.59	30.03	51.59	14.82	31.23	28.76	42.59
SUSIMCSE-BERT	23.24	43.45	30.52	55.22	13.99	30.88	25.58	43.45
CoSentBert	28.95	50.68	35.78	61.27	19.27	38.77	32.15	50.68
ICTSENTBERT	28.61	48.13	35.93	59.80	15.62	35.91	34.42	48.13
OTMPNET-1B	36.41	56.91	43.23	67.60	28.69	43.49	37.76	60.30
SPECTER	34.23	53.28	43.95	66.70	22.44	37.41	36.79	56.67
SCINCL	39.37	59.24	49.64	70.02	27.14	46.61	41.83	61.70
SPECTER-COCITEscib	37.90	58.16	48.40	68.71	26.95	46.79	38.93	59.68
SPECTER-COCITEspec	37.39	58.38	49.99	70.03	25.60	45.99	37.33	59.95
TSASPIRESpec	40.26	60.71	49.58	70.22	28.86	48.20	42.92	64.39
OTASPIRESpec	40.79	61.41	50.56	71.04	27.64	46.46	44.75	67.38
TS+OTASPIRESpec	40.26	60.86	51.79	70.99	26.68	47.60	43.06	64.82
Table 1: Test set results for baseline and proposed methods on CSFCUBE, an expert annotated fine-grained similarity dataset of computer science papers. Our approaches outperform strong prior models OT/MPNET-1B and SPECTER by 5-6 points, and the concurrently introduced SCINCL model by 1.5-2 points aggregated across queries. Metrics (MAP, NDCG%20) are computed based on a 2-fold cross-validation and averaged over three re-runs of models. Here, TSASPIRE: Text supervised single-match method, OTASPIRE: Optimal Transport multi-match method and TS+OTASPIRE: Multi-task multi aspect method.
similarities. With OTMPNET-1B indicating the value of modeling multiple matches, 2. SPECTER-COCITEscib, which is identical to SPECTER but trained on co-citations outperforms it, showing the value of co-citations for fine-grained similarity.
Next, we examine performance of the proposed methods: 1. First we note that all of the proposed approaches consistently outperform perform-. mant prior work, OT/MPNET-1B and SPECTER, by about 5-6 points, and concurrent work of SCINCL by about 1.5-2 points aggregated across queries. 2. Next, we note that the proposed approaches outperform SPECTER-COCITE $_{\text{Spec}}$ , trained on co-citations by 2-3 points aggregated across queries. 3. Our single match model trained with textual supervision, TSASPIRE consistently outperforms baselines. 4. Finally, our multi-match model OTASPIRE, while outperforming baselines sees aggregate performance similar to single match methods. This is reasonable given the aspect-specific annotation of CSFCUBE where we expect gains from modeling fine-grained (contextualized) matches rather than aggregating multiple matches.
Now, we examine facet-specific performance: 1. Performance on background sees higher performance in general and the smallest gains for the proposed approaches. This may be attributed to background similarity being captured in coarse-grained topical similarity, a trait largely cap
tured in existing baselines. 2. method similarity in CSFCUBE presents significant challenges (Mysore et al., 2021, Sec 6) since it relies upon procedural similarities across steps of a method and on domain knowledge based similarities - this is often captured in co-citation data (Fig 1 presents one such complex paraphrase example). We see strongest performance for TSASPIRE here. 3. Finally, given that paper results interpretations are often dependent on all aspects of a given paper, result similarity often depends on similarity across the whole abstract. This leads OTASPIRE which models multiple matches to see strong performance.
Whole-abstract similarity: Table 2 presents results on  $\mathrm{TRECCOVID}_{\mathrm{RF}}$  and RELISH. At the outset, we note that while being annotated for whole-abstract relevance, these datasets present different characteristics. While  $\mathrm{TRECCOVID}_{\mathrm{RF}}$  presents queries centered on a very specific topic, RELISH presents a much more diverse set of queries. Further,  $\mathrm{TRECCOVID}_{\mathrm{RF}}$  pairs queries with pools of about 9000 candidates while RELISH has about 60 candidates per query. Next, we examine baselines.
1. In contrast to fine-grained similarity datasets the best sentence level model MPNET-1B, significantly underperforms an abstract level model, SPECTER, indicating the need for whole abstract representations for these datasets. Aggregating sentence matches as in OTMPNET-1B, drastically im
Models	TRECCOVIDRF	RELISH
MAP	NDCG%20	MAP	NDCG%20
MPNET-1B	17.35	43.87	52.92	69.69
SENTBERT-PP	11.12	34.85	50.80	67.35
SENTBERT-NLI	13.43	40.78	47.02	63.56
UNSIMCSE-BERT	9.85	34.27	45.79	62.02
SUSIMCSE-BERT	11.50	37.17	47.29	63.93
CoSentBert	12.80	38.07	50.04	66.35
ICTSENTBERT	9.80	33.62	47.72	63.71
OTMPNET-1B	27.46	58.70	57.46	74.64
SPECTER	28.24	59.28	60.62	77.20
SCINCL	28.73	59.16	62.09	78.72
SPECTER-COCITEScib	30.60	62.07	61.43	78.01
SPECTER-COCITESpec	28.59	60.07	61.43	77.96
TSASPIRESpec	26.24	56.55	61.29	77.89
OTASPIRESpec	30.92	62.23	62.57	78.95
TS+OTASPIRESpec	30.90	62.18	62.71	79.18
Table 2: Test set results for baseline and proposed methods on  $\mathrm{TRECCOVID}_{\mathrm{RF}}$  and RELISH, expert annotated abstract similarity datasets of biomedical papers. Our approaches outperform or match a strong prior model, SPECTER, and the concurrently introduced SCINCL by 2-3 points across metrics (MAP, NDCG%20). These are computed as averages over three model re-runs. Method names map similarly to Table 1.
proves MPNET-1B. 2. Next, similar to results in Table 1, a model identical to SPECTER, but trained on co-citations, SPECTER-COCITE $_{\text{Spec}}$ , outperforms SPECTER indicating the value of co-citation signal for whole-abstract similarity too. 3. Finally, we also note that while SCINCL sees an expected stronger performance to SPECTER in RELISH, it sees comparable performance in TRECCOVID $_{\text{RF}}$  - indicating the influence of the candidate pool size on its performance.
In examining performance of our proposed methods, we note the following: 1. Across datasets, our method for single matches, TSASPIRE, outperforms context-independent sentence baselines by several points indicating the value of contextualization. However, this method still underperforms abstract-level baselines. 2. However, methods modeling multiple matches, OTASPIRE and TS+OTASPIRE, substantially outperform TSASPIRE as well as baseline prior work SPECTER and OTMPNET-1B. This performance indicates the strength of OT based aggregation of fine-grained matches for abstract level similarity. The proposed methods additionally match or outperform the concurrent approaches of ScINCL. Note here, that given the complementary approach presented in ScINCL - strong models are likely to result from combining both approaches.
We present results demonstrating the value of
the proposed approach on the SciDOCS benchmark in Appendix G. Further, we also present a set of ablations in Appendix H. These ablations establish the value of textual supervision over the encoder  $(\mathrm{BERT}_{\mathcal{E}})$  used for encoding the text, the value of optimal transport compared to attention alternatives, and alternative single-match models trained without co-citation contexts.
Aspect-based paper representations: A large body of work learns structured representations of scientific papers. Jain et al. (2018) present an approach which learns pre-defined aspect (PICO) encoders for biomedical papers. Similarly work of Neves et al. (2019), Chan et al. (2018), and Kobayashi et al. (2018) each label paper texts and then compute aspect-specific embeddings for document classification or ranking using existing methods. This line of work often relies on pre-defined aspects and building aspect-specific methods. Finally, work of Ostendorff et al. (2020) and Luu et al. (2021) present an approach with some similarities to the ones presented above – these approaches leverage classification or language generation models to output fine-grained relationships between pairs of papers. Our work leverages co-citation contexts to supervise free-text aspects with a new model for document retrieval, that is also not tied to a specific schema of labels.
Fine-grained document representations: Another similar line of work is modeling fine-grained document-document similarity at the level of words or latent topics. Examples include early work El-Arini and Guestrin (2011) presenting paper recommendation methods with unigram-level similarity between papers using authorship and citation links or using latent document topics (Gong et al., 2018; Yurochkin et al., 2019; Dieng et al., 2020).
Our approach represents documents via sentences, a common and intuitive structure for reasoning about scientific document facets (Chan et al., 2018; Zhou et al., 2020). Ginzburg et al. (2021) present a self-supervised model for contextual sentence representations in long documents similar to our ICT baseline (Lee et al., 2019).
Ad-hoc Search: A range of recent work in information retrieval presents multi-vector models intended to capture different aspects of candidate documents with score aggregation relying on summations, max, or attention functions (Khattab and
Zaharia, 2020; Luan et al., 2021; Humeau et al., 2020), these however focusing on short-text queries seen in search or question answering (QA). Mitra et al. (2017) explore an approach to model term-level fine-grained similarities with neural networks, Liu et al. (2018) model fine-grained matches at the level of entity spans, and Akkalyoncu Yilmaz et al. (2019) model document relevance by aggregating sentence relevance. Similarly, recent work of Lee et al. (2021) models fine-grained matches for QA at the phrase level. Importantly, these methods rely on supervision from knowledge bases or QA datasets, limiting applicability to specific span definitions and areas with these resources, often not present in the scientific domain (Hope et al., 2021a).
A range of modeling approaches in the context of other tasks resemble elements of our approach. We describe these in Appendix J.
We presented ASPIRE, a scientific document similarity model that is trained by leveraging co-citation contexts for learning fine-grained similarity. We use co-citation contexts as a novel form of textual supervision to guide the learning of multi-vector document representations. Our model outperformed strong baselines on seven document similarity tasks across four English scientific text datasets. Moreover, we showed that a fast single-match method achieves competitive results, enabling fine-grained document similarity in large-scale scientific corpora. A future direction is the interactive use of our methods, with a system allowing users to highlight specific aspects of papers and retrieve contextually-relevant matches. Another promising application is for finding analogies — structural matches between texts describing ideas, as in scientific papers, to boost discovery (Hope et al., 2017, 2021b; Chan et al., 2018).
We are grateful to the members and fellow interns of Semantic Scholar at AI2, and members of the IESL and CIIR labs at the University of Massachusetts Amherst for helpful discussions. This work was supported in part by the National Science Foundation under Grant Number IIS-1922090, the Chan Zuckerberg Initiative under the project Scientific Knowledge Base Construction, the NSF Convergence Accelerator Award #2132318, and using high performance computing equipment ob
tained under a grant from the Collaborative R&D Fund managed by the Massachusetts Technology Collaborative.
Zeynep Akkalyoncu Yilmaz, Wei Yang, Haotian Zhang, and Jimmy Lin. 2019. Cross-domain modeling of sentence-level evidence for document retrieval. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3490-3496, Hong Kong, China. Association for Computational Linguistics.
Alexandr Andoni, Piotr Indyk, and Ilya Razenshteyn. 2018. Approximate nearest neighbor search in high dimensions. In Proceedings of the International Congress of Mathematicians: Rio de Janeiro 2018, pages 3287-3318.
Stefanos Angelidis and Mirella Lapata. 2018. Multiple instance learning networks for fine-grained sentiment analysis. Transactions of the Association for Computational Linguistics, 6:17-31.
Arturs Backurs, Yihe Dong, Piotr Indyk, Ilya P. Razenshteyn, and Tal Wagner. 2020. Scalable nearest neighbor search for optimal transport. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 497-506. PMLR.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computation.
Mark Berger, Jakub Zavrel, and Paul Groth. 2020. *Effective distributed representations for academic expert search*. In *Proceedings of the First Workshop on Scholarly Document Processing*, pages 56-71, Online. Association for Computational Linguistics.
Chandra Bhagavatula, Sergey Feldman, Russell Power, and Waleed Ammar. 2018. Content-based citation recommendation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 238-251, New Orleans, Louisiana. Association for Computational Linguistics.
Peter Brown, RELISH Consortium, and Yaoqi Zhou. 2019. Large expert-curated database for benchmarking document similarity detection in biomedical literature search. Database, 2019. Baz085.
Avi Caciularu, Arman Cohan, Iz Beltagy, Matthew Peters, Arie Cattan, and Ido Dagan. 2021. CDLM: Cross-document language modeling. In *Findings of the Association for Computational Linguistics*:
EMNLP 2021, pages 2648-2662, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Joel Chan, Joseph Chee Chang, Tom Hope, Dafna Shahaf, and Aniket Kittur. 2018. Solvent: A mixed initiative system for finding analogies between research papers. Proceedings of the ACM on Human-Computer Interaction, 2(CSCW):1-21.
Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel Weld. 2020. SPECTER: Document-level representation learning using citation-informed transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2270-2282, Online. Association for Computational Linguistics.
Marco Cuturi. 2013. Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pages 2292-2300.
Adji B. Dieng, Francisco J. R. Ruiz, and David M. Blei. 2020. Topic modeling in embedding spaces. Transactions of the Association for Computational Linguistics, 8:439-453.
Khalid El-Arini and Carlos Guestrin. 2011. Beyond keyword search: discovering relevant scientific literature. In Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Diego, CA, USA, August 21-24, 2011, pages 439-447. ACM.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894-6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Dvir Ginzburg, Itzik Malkiel, Oren Barkan, Avi Caci-ularu, and Noam Koenigstein. 2021. Self-supervised document similarity ranking via contextualized language models and hierarchical inference. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3088-3098, Online. Association for Computational Linguistics.
Bela Gipp and Jöran Beel. 2009. Citation proximity analysis (cpa): A new approach for identifying related work based on co-citation analysis. In *ISSI'09: 12th international conference on scientometrics and informetrics*, pages 571-575.
Hongyu Gong, Tarek Sakakini, Suma Bhat, and JinJun Xiong. 2018. Document similarity for texts of varying lengths via hidden topics. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages
2341-2351, Melbourne, Australia. Association for Computational Linguistics.
Braden Hancock, Paroma Varma, Stephanie Wang, Martin Bringmann, Percy Liang, and Christopher Ré. 2018. Training classifiers with natural language explanations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1884-1895, Melbourne, Australia. Association for Computational Linguistics.
Austin W. Hanjie, Ameet Deshpande, and Karthik Narasimhan. 2022. Semantic supervision: Enabling generalization over output spaces.
Tom Hope, Aida Amini, David Wadden, Madeleine van Zuylen, Sravanthi Parasa, Eric Horvitz, Daniel Weld, Roy Schwartz, and Hannaneh Hajishirzi. 2021a. Extracting a knowledge base of mechanisms from COVID-19 papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4489–4503, Online. Association for Computational Linguistics.
Tom Hope, Joel Chan, Aniket Kittur, and Dafna Shahaf. 2017. Accelerating innovation through analogy mining. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Halifax, NS, Canada, August 13 - 17, 2017, pages 235-243. ACM.
Tom Hope and Dafna Shahaf. 2016. Ballpark learning: Estimating labels from rough group comparisons. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 299-314. Springer.
Tom Hope and Dafna Shahaf. 2018. Ballpark crowdsourcing: The wisdom of rough group comparisons. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, WSDM 2018, Marina Del Rey, CA, USA, February 5-9, 2018, pages 234-242. ACM.
Tom Hope, Ronen Tamari, Hyeonsu Kang, Daniel Herschovich, Joel Chan, Aniket Kittur, and Dafna Shahaf. 2021b. Scaling creative inspiration with fine-grained functional facets of product ideas.
Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2020. Poly-encoders: Architectures and pre-training strategies for fast and accurate multi-sentence scoring. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Maximilian Ilse, Jakub M. Tomczak, and Max Welling, 2018. Attention-based deep multiple instance learning. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 2132-2141. PMLR.