SciRepEval: A Multi-Format Benchmark for Scientific Document Representations
Authors: Amanpreet Singh
Mike D'Arcy
Arman Cohan
Doug Downey
Sergey Feldman
Venue: EMNLP 2023 Main (2023)
Keywords: Scientific tasks benchmark
multi task learning
task specific embeddings
TL;DR: 24 Task benchmark for evaluating scientific document embeddings and task-format specific multiple embedding generation with SPECTER2 adapters..

Abstract: Learned representations of scientific documents can serve as valuable input features for downstream tasks without further fine-tuning. However, existing benchmarks for evaluating these representations fail to capture the diversity of relevant tasks. In response, we introduce SciRepEval, the first comprehensive benchmark for training and evaluating scientific document representations. It includes 24 challenging and realistic tasks, 8 of which are new, across four formats: classification, regression, ranking and search. We then use this benchmark to study and improve the generalization ability of scientific document representation models. We show how state-of-the-art models like SPECTER and SciNCL struggle to generalize across the task formats, and that simple multi-task training fails to improve them. However, a new approach that learns multiple embeddings per document, each tailored to a different format, can improve performance. We experiment with task-format-specific control codes and adapters and find they outperform the existing single-embedding state-of-the-art by over 2 points absolute. We release the resulting family of multi-format models, called SPECTER2, for the community to use and build on.

SciRepEval: A Multi-Format Benchmark for Scientific Document Representations
Amanpreet Singh
Mike D'Arey² Arman Cohan
Doug Downey Sergey Feldman'
Allen Institute for Artificial Intelligence, Seattle, WA, USA
2 Northwestern University, II, USA
Yale University, CT, USA (amanpreets, miked, armanc, dougd, sergey))allenai.
Abstract
can be indexed and consumed later by lightweight downstream models without additional fine-tuning.
Learned representations of scientific docu- ments can serve as valuable input features for downstream tasks without further fine-tuning. However, existing benchmarks for evaluating these representations fail to capture the diver- in-
While there has been significant progress in eval- uating generalizability of NLP models (Ye et al., 2021; Sanh al., 2021), evaluation of scientific document representations has remained limited. Existing benchmarks either focus on document similarity (Mysore et al., 2021; Voorhees et al., 2021) or tasks that are highly correlated and not diverse (Cohan et al., 2020). Further, shown in our experiments, model's good performance on general-purpost text embedding benchmarks like the MTEB (Muennighoff et al., 2022) may not translate as well to scientific tasks.
sity of relevant tasks. In response. troduce SciRepEval, the first comprehensive benchmark for training and evaluating scien- tific document representations. It includes 24 challenging and realistic tasks, of which are new, across four formats: classification, regres- sion, ranking and search. We then use this benchmark to study and improve the general- ization ability of scientific document represen- tation models. We show how state-of-the-art models like SPECTER and SciNCL struggle to generalize across the task formats, and that sim- ple multi-task training fails to improve them. However, new approach that learns multiple embeddings per document, each tailored to different format, can improve performance. We experiment with tosk-forma-sppecffic control codes and adapters and find they outperform the existing single-embedding state-of-the-art by over points absolute. We release the re- sulting family of multi-format models, called SPECTER2, for the community to use and build on.
We introduce SciRepEval, the first bench- mark for comprehensive evaluation of document- representation models in the scientific domain. Un- like prior work, SciRepEval is large and includes a collection highly diverse tasks, thus encouraging research on generalization (instance-lewe) cross- task and cross-domain). It consists of 24 realistic tasks that reflect practical use cases of scientific document representations across four formats: text classification, regression, proximity- based ranking (e.g., nearest-neighhor), and ad-hoc search. Eight of these are new contributions. SciRepEval con- tains standard sets of both training and evaluation datasets to simplify and standardize comparisons between methods evaluated on the benchmark.
1 Introduction
Learning representations of documents is critical for a variety of NLP tasks such as classification, search, and recommendation (Cohan et al., 2020). Recent work has shown how pre-trained language models (e.g. (Devlin et al., 2019; Raffel et al., 2020; Brown et al., 2020)) can be tailored to produce high-quality document representations with con- trastive learning (Xu al., 2021; Gao et al., 2021; Neelakantan al., 2022). In the scientific domain, contrastive learning of cross-document links (e.g. citations) has led to improved document-level rep- resentations (Cohan et al., 2020; Ostendorff et al., 2022b; Mysore et al., 2022). These representations
Further, we use this benchmark to investigate and improve the generalization ability of document representation models. Following recent work (Co- han et al., 2020; Ostendorff et al., 2022b; Mysore et al., 2022) we pre-fine- tre-fine-tune a transformer model originally trained on citation triplets to produce high-quality representations for downstream tasks. We hypothesize that condensing all relevant infor- mation of document into single vector might not be expressive enough for generalizing across a wide range of tasks. Prior work addresses a similar challenge in the context document similarity and

---

learns multiple representations associated with dif- ferent aspects of a paper (c.g. task, method, results)
com/Vallenaai/scrreeeva..
The SPECTER2
models are released as well: https://githun: com/ Comm/allenaai/PECTER..
(Mysore et al., 2022; Ostendorff et al., 2022a). In contrast, we aim to learn effective representations for multiple downstream task formats.
Background
With the success of multi-task learning in NLP (Ye et al., 2021; Sanh et al., 2021), we explore it in the context of scientific document represen- tations by optimizing a suitable objective for ev- ery task format Sval,,Eval, i.e., cross-entropy for classification, triplet margin for proximitylad- hoc search, and mean squared error for regression. We explore two state-of-the -ortt---rttttttt methods to gener- ate format-specific representations: control codes (Keskar et al., 2019; Raffel et al., 2020) as input signal; and parameter-efficient adapter methods (Pfeiffer et al., 2021; Stickland and Murray, 2019), where separate network module is introduced for every task format as in Figure 1.
Representing Scientific Documents Prior works have produced large scale language models pre- trained on scientific corpora (Beltagy et al., 2019; Yasunaga et al., 2022; Trewartha et al., 2022). These tend to perform better than general purpose models on scientific domain tasks, and serve as foundation for learning dense embeddings of scien- tific documents. Cohan al. (2020) and Ostendorff et al. (2022b) fine-tune SciBERT (Beltagy et al., 2019) with a triplct loss that encourages papers cit- ing each other to have similar embeddings, using the title and abstract of research papers as the input. Both Cohan et al. (2020) and Ostendorff et al. (2022b) are evaluated on SciDoes. However, as discussed in section and Appendix G, this benchmark has important limitations. contrast, SciRepEval provides more challenging and diverse tasks to help motivate methods for producing scien- tific document representations that can generalize well across tasks. We attempt to learn task-specific embeddings of documents by pre-fine-tuning multiple objectives simultaneously. Ostendorff et al. (2022a) and Mysore et al. (2022) study the orthogonal task of generating multiple embeddings per paper for different "facets," while we aim to learn general embeddings for mutliple task formats.
Our experiments investigate (i) existing docu- ment representation methods generalize to a highly diverse set of tasks, (ii) if multi-task training on diverse data can improve document representation models, and (iti) if task-format-i -specfff representa- tions can improve generalization. Through exten- sive analysis we find that existing state-of-the-art scientific document representation models such as SPECTER (Cohan al., 2020) and SciNCL. (Os- tendorff et al., 2022b) struggle with generalizing to multiple task types. We interestingly find that sim- ple multi-task training on large set of tasks does not lead to significant improvements. However, we learn that multiple task format-specific representa- tions substantially improve generalization.
Multi-Tasse Learning Across Formats Multi- task learning (Caruana, 1993) with deep neural networks has shown improvements upon single- task training for related objectives (Liu et al., 2015, 2019h). Though unrelated tasks can lead to neg- ative transfer, recent work shows that simply in- creasing number of tasks yields better performance in multi-task learning (Aghajanyan al., 2021; Aribandi et al., 2022: Padmakumar et al., 2022). Aghajanyan et al. (2021) pre-fine pre-fine-tune language models on 46 tasks from 4 task types before fine- tuning on the downstream task. Aribandi et al. (2022) pre-train T5 (Raffel et al., 2020) on C4 span denoising and 107 other tasks across 8 task families. Ye et al. (2021) introduce ontology of 160 tasks for few shot multi-task training. Unlike these task families, which are divided primarily by semantics (e.g. classifying sentiment vs entailment), the train- ing tasks in SciRepEval consist of 8 large scientific datasets across the four task formats. We wish to
To summarize, our contributions are:
SciRepEval, new comprehensive bench- mark of 24 highly diverse and practical tasks for scientific document representation tech- niques across four different formats, of which 8 are made available for the first time, and six are explicitly designed for training. (ii) An extensive investigation the generaliz- ability of state-of-the-art scientific document representation models.
(iii) SPECTER2, a set of new multi-task document representation models that, unlike existing methods, can produce representations tailored to different task formats. The new methods show improved generalization, outperforming prior work by over 2 points absolute.
We release the benchmark and code to encourage further research in this area: https:/ / //g thub.

---

evaluate final document representations, so rather than fine-tune on downstream tasks as in above ap- proaches, we directly apply the representations as features to the tasks (Cohan et al., 2020).
between methods evaluated on the benchmark. The previous scholarly paper embedding bench- mark SciDoes (Cohan et al., 2020) includes two classification, one recommendation, and four near- est neighbors tasks. SciRepEval includes SeiDocs as a subset, but addresses several key limitations:
Adapters for Multiple Tasks Adapters were in- troduced by Houlsby et al. (2019) for parameter efficient training of transformers (Vaswani al.,
(i) The four nearest neighbor tasks in SciDocs are built to distinguish related papers from random negatives given query paper, which might be too easy and not representative of real tasks in schol- arly information retrieval. SciRepEval has more realistic tasks like search, author disambiguation, and paper-reviewer matching among others.
2017). small number of trainable parameters are added to each layer, while freezing the base en- coder. This similar to ELMo (Peters et al., 2018)) which learned task-specific weightings for the biL- STM layers. To use adapters in a multi-task setup, Pfeiffer et al. (2021) define a two-step process they call Fusion. First, individual adapter modules trained for every task. The second step adds fusion modules at each layer which attend to (i.e. fuse) all the previously pre-fine-tuned adapters, keeping them fixed. Similarly, Stickland and Murray (2019) introduced Projected Attention Layers (PALs) with adapters and self-attenting modules for every task, but the entire network is trained simultaneously.
(ii) For the methods evaluated in section 5, we found that the SciDoes recommendations task was noisy and had limited power to distinguish different embeddings. The test set includes only 1000 click- through events, and the use of propensity weighting means an even fewer examples dominate test per- formance. While SeiRepEval includes SciDocs subset, we exclude the recommendations task. (iii) The tasks in SciDocs were constructed to be used only for evaluation, and have few-enough samples that training on SeiDoes is impractical (see Table 1). In SeiRepEval, eight of the largest tasks across the four formats are intended for training, while the rest the out-of-train tasks are reserved for evaluation. This enables the study of multi- task approaches, rather than relying solely on the citation signal. The training data in SciRepEval also has large scale representation across multiple domains as discussed in Appendix E.
Control Codes Control codes can be defined as token(s) pre-pended to the input additional sig- nals for the model. Keskar et al. (2019) use control codes as prompts govern style, content, and task- specific behavior for conditional text generation. Tay et al. (2022) use control codes to switch be- tween three de-noising modes during pre-training, and associate every downstream task with mode during fine-tuning. Zhang et al. (2022) apply con- trol codes to dense retrieval and produce multiple representations covering different aspects of the same document, allowing them to match queries written from multiple perspectives. In contrast to this, we use control codes to indicate the target task format for an embedding output by the model, and demonstrate how this is effective for producing paper embeddings across different formats.
(iv) Four of the tasks in SciDocs have very high prrfel-erffrmmacc correlations among them (over 0.99), indicating that the diversity of the tasks is limited. See Appendix for more details.
The tasks in SciRepEval are summarized in Ta- ble 1. They are a mixture of existing and new datasets. Datasets with at least 200,000 instances (triplets for proximity/ad-hor search) are in-train datasets used for training while others are out-of- train used only for evaluation.
3 SciRepEval
We introduce SciRepEval, a benchmark of 24 tasks across four formats to train and evaluate multi-task embeddings of scholarly papers. SciRepEval aims to enable comprehensive evaluation of paper em- beddings with: (1) a highly diverse set of tasks spanning multiple formats such as classification, re- gression, proximity and ad-hoc search to challenge the general- perrpoe-purpose applicability of embeddings, (2) realistic tasks that reflect practical use cases of paper embeddings, and (3) a standard set of train- ing and evaluation datasets to simplify comparisons
Next, briefly describe each of the task for- mats and their component tasks. Full details are provided in Appendix A. All the tasks save one use paper embeddings created from a combination of paper title and abstract as the input. Search re- quires additional metadata (subsection 4.1) which is concatenated to the title and abstract.
Ad-Hoe Search In ad-hoc search tasks, we are given a short textual query and the aim is to rank a

---

Task Format
Nume
Train Dev
Test
Eval Metric
Source
In-Troin
MeSH Descriptors
2.328,
258,687
Macro Macro F1
This work
CLF
Fields of study (FoS)
676.524
471 G
This work
RGN
Citation count Year Publication
202,774
30,058
Kendall's T Kendall's T
This work
218,864
30,000
This work
Stmee Author Detection
Q: 76,489 P: 673.170
Q: 13,585 P: 123,430
MAP MAP
(Subracnaninn et al., 2021)
PRX
Highly Inftuential Citations
Q: 65,982 P: 2,004,688
Q: 1,199 P: 58,255
This work
Citation Prodiction Triplets
819,836
"noe used eval
(Cohan al., 2020)
SRCII
Search
Q: 528,497 P: 5,284,970
Q: 2.585 P: 25,850
aDGC
This work
Out-of-Wrain
Riomimicry
10,991
Rinary Fl
(Shyam et al.,, 2019)
DRSM
955
Macro
(Bums, 2022)
CLF
SciDoes MAG
23,540
Macro H1
(Cohan ot al, 2020)
SciDoes McSH Diseases
25,003
Macro
(Cohan al., 2020)
Peer Score
10,210
Kendall's
This work
RGN
h-Index of Authors
8,438
Kendall's i T
This work
Tweet Mentions
25,655
Kendall's s T
(Jain and Singh, 2021)
S2AND
X: 68,968 Y: 10,922
B² Fl
(Subramanian et ali, 2021)
(Mimno and McCallum, 2007)
Paper-Reviewer Matching
Q:107 P: 1,729
P@:5, P@10
(Liu et 2014)
(Zhno et al., 2022)
RELISH
Q: 3190 P: 191,245
nDCG
(Brown 2019)
PRX
SciDoes Corview
Q: 1.000 P: 29,978
MAP, sDOG
(Coban al., 2020)
SoilDess Co-read
Q: 1,000 29.977
MAP, sDXG
(Crhaa al., 2020)
SciDues Cite
Q: 1,000 P: 29,928
MAP, =DOG
(Cohan et al., 2020)
SciDoes Co-cite
29.949
nDOU
(Cohan al., 2020)
NFCorgus
#DCG #DCG
(Rotean et 2016)
SRCH
TREC-CUVID CUVID
Q: 50 P: 69,318
(Voorhees et al., 2021)
Table 1: Summary of SciRepEval tasks across the four formats classification (CLF), regression (RGN), proximity (PRX) and adhoc search (SRCH). The models section are trained on the in-train tasks and then henchmarked on their held-out sets well as the 16 test tasks. Information retrieval tasks have Q queries with P candidate pairs and S2AND has X clusters with Y author-paper pairs. S: Silver, G: Gold. SciDoes is evaluated per Cohan et al. (2020).
set of candidate papers by their relatedness to the query. Ad-hoc search is critical mechanism for paper discovery in practice, and we gather multi- ple real-world data sets for training and evaluation. We include TREC-CoVID (Voorhees et al., 2021) and NFCorpus (Boteva et al., 2016) as test tasks for this format. We also introduce Search, a large training set of over 500K clickthrough events from Semantic Scholar, scholarly search engine.
include total of ten proximity-based tasks, in- cluding four nearest neighbor tasks from SciDocs (predicting citations, co-citations, co-vit viewed or co- read papers), and three others from previous work: the S2AND author disambiguation task (Subrama- nian et al., 2021) with paper similarity features; Paper-Reviewer Matching (Mimno and McCallum, 2007; Liu et al., 2014; Zhao et al., 2022), where candidate reviewers tors based on the similarity of their papers the query paper to be reviewed; and finally RELISH (Brown al., 2019): document recommendation task consisting of ground truth labels collected by a consortium of over 1500 scientists worldwide. We also introduce three new large-scale training datasets aimed at predicting auth-a, citations (via triplets) as in Cohan et al. (2020), and influ- ential citations, which we define as four more citations of the same paper in the text of single paper.
ranked by expert annota-
To evaluate the ad-hoe search tasks, candidate papers are ranked by increasing Euclidean distance between the query and the candidate embeddings. Pytrec_eval (Van Gysel and de Rijke, 2018) is used to calculate the ranking metrics. Normalized Discounted Cumulative Gain (nDCG) is used for Search and TREC-CoVID as the true relevance score can be > 1.
Proximity Similar to ad-hoc search, proximity tasks involve ranking set of candidate papers by their relatedness to query, except the query in this case is a paper as well. Proximity-based tasks form a basis for paper-based retrieval and recom- mendation, and for estimating paper similarity for use in applications like author disambiguation. We
For evaluation, we again rank candidate em- beddings by Euclidean distance, using MAP and nDCG as the scoring metric except for S2AND with B Fl (Bagga and Baldwin, 1998), and Paper-

---

Reviewer Matching, which uses precision(5 and @10.
Control Codes
Adapters
[CLF]
Classification Classifying papers into topical cat- egories a foundational task for document organi- zation and discovery. Apart from the two SciDocs tasks (MAG and MeSH Diseases), we have four others; including binary task to predict whether a paper is relevant to biomimicry (Shyam et al., 2019), two biomedical classification tasks, namely DRSM from Burns (2022) and MeSH Descriptors classification (Lipscomb, 2000), and new large- scale field of study (FoS) multi-label training set of more than 500K papers with silver FoS labels based on publication venues.
[ORN]
[RGN]
+
doc
Language Model
PRO ORY
PRX
cjtt
RGN
Clasaification
Proximity
Regression
Generating multi-format embeddings. A task format is either associated with a control code appended to the input, or adapter blocks attached the model.
Figure I:
We evaluate embeddings on classification by scoring their performance as features within lin- ear support vector classifiers. Results for these tasks are evaluated using binary/macro Fl score. To better understand how embeddings perform in data-scarce regimes, we also construct two few- shot versions each from Biomimiery, DRSM and FoS dataset subset for which we have manually annotated gold labels.
and also incurs significant storage costs scaling with the number of tasks. We propose method for learning distinct document embedding for each task format, using a multi-task learning setup.
We assume are given labeled data from a set of tasks for our four formats (ad-hoc search, proximity, classification, and regression), and we learn models capable of producing an embedding for any given (paper, format) pair. Our goal is for these embeddings to be used in lightweight classiffersrregresorrs as well as in nearest neighbor tasks, which we evaluate both on held-out data from the training tasks, and new held-out tasks.
Regression We also consider regression tasks where the goal is to predict a continuous quan- tity for a given paper. For evaluation, we consider predicting three numeric attributes related to promi- nence or quality: Tweet Mentions (Jain and Singh, 2021), and two new datasets predicting peer review rating and maximum h-index of authors for a set of ICLR papers from OpenReview!. For training, we introduce two additional datasets; predicting citation count and publication year of papers.
To help build intuition for why different embed- ding sets for different task formats may be helpful, Figure 1 illustrates the qualitative distinctions be- tween the task formats. In general, an embedding space yielding good results for one task format may be less suited to others; for example, the classifi- cation space provides an error-free linear classifier, but its nearest neighbor pairs are not always of the same class. Empirically, we find that learning specialized embeddings per format improves per- formance, and that embeddings trained on a format tend to perform better on held-out tasks with the same format (see Table 3). Further, partitioning randomly (as discussed in section 7) was less effec- tive than the format-based partitioning. Nonethe- less, format-based partitioning is just one choice of many and experimenting with other partitioning schemes is an important item of future work.
We evaluate embeddings on regression by scor- ing their performance as features within linear sup- port vector regression models. The reported results are computed as Kendall's 7 rank correlation be- tween the true and predicted labels.
4 Multi-format representation learning
Typical approaches for learning document embed- dings produce a single embedding for every task (Cohan et al., 2020; Ostendorff et al., 2022b). We hypothesize that a single embedding is insufficient for generalizing across multiple downstream tasks. At the other extreme, learning embeddings for each task separately limits generalization to new tasks
4.1 Model
openreveew.eeee
We follow Cohan et al. (2020) in using a pretrained transformer encoder as our base model. A scientific document is given as input to the encoder as a con- catenation of its title and abstract separated by the
"We found in our experiments that Pearson's and Kendall's is r produced similar relative results between models. We did not use MSF because its unbounded values could skew the overall average.

---

[SEP] token.² Unlike Cohan al. (2020), we use three different types of training objectives suitable for each format to further train the model as de- scribed in subsection 4.2. We explore two methods to learn separate embeddings for each task form: control codes and adapters as shown in Figure 1. Control Codes In this approach, we prepend a special token per-format (TTble the appendix) to the input and pass it to the transformer, taking the final layer embedding corresponding to this token as the document representation and feeding it to the task-specific head (described in subsection 4.2). Adapters Adapters have been shown to be effec- tive for multi-task learning, therefore we explore Adapter Fusion (Pfeiffer et al., 2021) and PALS (Stickland and Murray, 2019) in our work. Both these methods introduce task-specific adapters and attention modules at each transformer layer. Since we aim to learn different embeddings for different task formats, we create modules for each task for- mat rather than each task, and the final embedding the [CLS] token output via the adapter is taken as the corresponding representation of the document.
5 Experiment Setup
Training Data Our multi-format models are trained on the 8 large in-train tasks detailed in Ta- ble I. For proximity and ad-hoc search, we create up to instances per query by sampling positive and negative papers from its candidate pool. We limit the number of training samples from each task to 600K, resulting in training and validation sets of of 3.27M and 446K instances respectively.
Transformer Baselines As first step, eval- uate existing document representation methods on SciRepEval. These include SciBERT (Belt- agy al., 2019), language model pre-trained on scientific corpora, and paper-embedding methods SPECTER (Cohan et al., 2020), ASPIRE (Mysore et al., 2022), and SciNCL (Ostendorff et al., 2022b) which is the state-of-the-art on SciDocs. ASPIRE produces representations for aspect-based match- ing between query and candidate papers which is similar setting to proximity tasks, so we only evaluate and report its results for that subset in Appendix C. Finally, we also evaluate two recent general-purpose purpose text embedding methods, E5 v2 (Wang et al., 2022) and MPNet (Song et al., 2020), These methods are pre- pre-trained on a set of over IB query-candidate pairs including scientific text, and are two of the best BERT-base sized models on the recent MTEB benchmark (Muennighoff et al., 2022) for general-purpose text embeddings.
4.2 Training
We train our models in a multi-task setup with task-heterogeneous batching (Aghajanyan et al., 2021). For classification and regression, we use linear heads atop the base transformer." Cross En- tropy and Binary Cross Entropy loss with sigmoid activation are used for multi-class and multi-label classification respectively. For regression we mini- mize the Mean Squared Error loss.
SPECTER2 Models Although SPECTER and SciNCL are strong baselines, about 70% of their pre- pre-training data is from just two domains as shown in Table 9. This leads to poor generalization as out- lined in Medić and Snajder (2022) where BM25 outperformed both models across all the domains except Computer Science and BioMed on cita- tion recommendation task. To increase the domain coverage we pre-train new model similar Co- han al. (2020), but with 10x more data spanning 23 fields of study and term it SPECTER2 Base, Each query paper in our training set can have up to 10 triplets where the positive candidates are taken from the direct citations of the query. 6 easy neg- atives are sampled based on the field of study (4 same the query: 2 different) while hard nega- tives come from papers cited by one of the query paper's citations but not by the query itself. We also include the SciNCL triplets as a subset. This
For proximity and ad-hoc search tasks we use the triplet margin loss from Cohan et al. (2020). For these task forms, given a query, a relevance score accompanies each candidate. Hence, each training instance in this setup is triplet consisting of query Q in the form of paper (for which wish find similar documents) or raw text, a positive candidate paper P+ and a negative candidate P-, where P+ has a higher score than P-, Then, we optimize the triplet loss:
(1) where d is the Euclidean distance used as a measure of similarity between the query embedding QE and candidate embeddings pt and P F. e is the margin hyperparameter whose value is I chosen based on preliminary experiments.
Levipter = max (d( (() d(QR, PT) +,
'In the Search task. publishing venue and year is also provided.
Perffrraacc with smaller dataset samples max 400K samples/tasks was relatively poor.
'The lincar heads are discarded after training.

---

Model
In- Train
Out-of-Train
Average
benchmark includes diverse tasks with a variety of different evaluation metries, and following previ- ous work (e.g. Wang et al. (2019)) we report an av- erage of the individual metrics (each ranging from 0-100). Among the vanilla models, even though ES v2 and MPNet perform better than SciBERT; SPECTER, SciNCL and SPECTER2 Base outper- form them suggesting the need for domain and task specific embeddings to do well on SciRepE- val. The pre-fine-tuned multi-format variants of SPECTER2 outperform the baseline models on average. However, SPECTER2 Base and MTL CLS are on par even though the latter is trained on in-domain datasets. This is where the task for- mat based training helps. We find that all the ap- proaches that produce multiple representation types outperform the MTL CLS model, which learns only single representation shared for all tasks by 1.4 to 2 points. The adapter variants are better than MTL CTRL overall, and result an improvement of up to 0.8 points on the out-of-train tasks with the fusion adapters performing the best.
Transformer Baselines
E2-base-v2
55.7
70.9
67.0
MPNet
49.0
71.0
65.3
SciBERT
51.5
60.2
58.0
SPECTER
54.7
72.0
67.5
SciNCL
55.6
73.4
68.8
SPECTER2
Base
56.3
73.6
69.1
MTL CLS
60.2 (044)
72.1 (021)
69.0 (0.1)
MTL CTRL
62.4 (oo8)
73.1 (0.8)
70.4 1.
Adapters
62.4 (006)
73.9 (013))
70.9 (000)
PALs
61.8
72.6 (127)
69.9 (0.2)
Fusion
62.4 (0.0))
73,9 (007))
70.9 (0.00
Adapters
62.9 (..9))
74,1
71.2 (0.19)
MTL CTRL
Table 2: Evaluation results on SeiRepEval in multiple settings. SPECTER2 Models are further categorized based on the training setup. Base is trained only cita- tions as mentioned earlier; MTL CLS generates a single embedding for all tasks, MTL CTRL (control codes) and Adapter variants (Adapters, PALs, and Adapter Fu- sion) produce an embedding per task format. We also consider ensemble approach that averages the MTL CTRL and Adapter embeddings. For models we trained, we report the mean and standard deviation (in parenthe- ses) across 5 runs with different seeds. The best results are highlighted in bold. We conduct one way analysis of variance (ANOVA) with Tukey's 's test (Haynes, 2013) for a 0.05 across multiple settings and underline those not statistically significantly different from the best.
Further, as shown in Table the control codes and adapters are the most efficient in terms of model size and computation runtime. Hence, we try to improve upon each by combining represen- tations from both models by averaging them?, and we find that these combined embeddings outper- form the individual models consistently across the in-train, out-of-train, and overall average settings.
results in 6.2M training and I 76k validation triplets, and we release this data to the community.
Next, for our multi-format experiments, we pre- fine-tune the base model on SciRepEval in-train tasks both with (MTL CTRL) and without (MTL CLS) the control codes. Finally, to compare the control codes approach with adapters, we train the base model with BERT PALS and Fusion archi- tectures. Fusion, being two step process, first introduces task format specific adapters and then fusion modules. The MTL CTRL. and adapter ap- proaches produce multiple representations per doc- ument while MTL CLS produces a single repre- sentation. We use the PyTorch implementations of the models by HuggingFace The specific training configurations are described in Appendix B.
Alternative Base Models To test the consistency of our findings for the proposed training techniques, we also train the MTL CLS, MTL CTRL and adapters variants with SPECTER and SciNCL as the base models. Table 8 in Appendix D shows that the MTL CTRL token and the adapters approaches still substantially outperform the MTL CLS ap- proach, suggesting that the efficacy of using an embedding per task format instead of a single em- bedding per document is consistent across a range of base model types.
Analyses
Specialization of Control Code Embeddings Our hypothesis is that by training embedding spaces on particular task formats, they will become more accurate for tasks of that format than for oth- ers. We test this by sampling one in-train and one
6 Results
Table 2 shows the evaluation of all our transformer baselines producing both single and multiple rep- resentations per document on SciRepEval. Our
"We also tried concatenating both the emheddings, which yielded similar results with double the embedding size.
"htnss//huggingffecccoommddd

---

Control Code Used
Parameters per Task Form
Training Time
Inference
Task format
Model
Time
CLF
RGN
PRX
QRY
MTL CTRL
768
1x
1x
Classification
43.3
29.4
32.7
31.1
PALS
2M
1.42x
1.29x
Regression
29.8
46.8
43.3
43.1
Adapters
1M
0.96x
1.05x
Proximity
87.4
78.9
88.8
87.5
Fusion
22M
1.32x
1.69x
Search
73.4
72.6
76.1
78.5
Adapters
IM
1.96x
2.05x
(a) in-train
MTL CTRL
Table 4: Parameter and (relative) runtime efficiency of SPECTER2 models. MTL. CTRL. and Adapters sim- ilar in runtime, but PALs, Fusion and ensemble variants add significant computation costs.
Control Code Used
Task format
CLF
RGN
PRX
QRY
Classification
64.8
63.6
62.8
63.7
Regression
16.9
22.2
17.8
16.1
Proximity
43.8
40.5
45.1
45.2
Ad-hoc search
87.4
83.1
90.3
90.9
Model
MAP
R@5
(b) out-of-train
BM-25
33.7
28.5
SPECTER2 Base
38,0
32.4
Table 3: Cross task analysis for control codes. The best results for each task format across all control codes is underlined. These are represented in the diagonal for both in-train and out-of-train tasks suggesting that format based partitions in multi-task training produce representations suitable for the corresponding format.
SPECTER2 MTL CLS
34.6
24.9
SPECTER2 MTL CTRL
36.5
30.7
SPECTER2 Adapters
33.0
SPECTER2 Adapters MTL CTRL
38.4
32.9
Table 5: Comparison of SPECTER2 models with BM25 on the MDCR benchmark. As in the original paper. report MAP and Recall@5 scores. The best results obtained are highlighted in bold.
out-of-train task of every format (for ease of com- putation) and evaluating them with all the control codes. As shown in Table 3, the control codes trained a task format yield best results for tasks of that format, for both in-train and out-of-train.
has impact training time. PALs, in contrast, introduce attention layers and train the entire network, increasing runtime, while Adapters add and only train half as many parameters as PALs. Fu- sion layers have 10x as many parameters as PALs leading to 2x more time on inference. The ensem- ble model Adapters + MTL has the highest train- ing and inference cost, so the cheaper Adapters, achieving only slightly lower task performance (Ta- ble 2), may be preferable in many cases. Train- ing and inference times measured with 1 and 10k samples, respectively. As such, we release the SPECTER2 Base model and our best task-format Adapters publicly for further use.?
As an extension to this we also analyze how well the control code representations work when the training tasks are randomly grouped together as op- posed to by task format. From an evaluation across 5 random partitions, we found that task-format- based partitions were better by over 2.7 points on average (both in-train and out-of-train) across the formats, suggesting that representations specific to each task format do lead better results overall. Finally, to study training affinity among the task formats themselves, we pre-fine-tune on at most two formats at once. Appendix reveals that com- bined multi-task training on similar task formats like mroximity'adhoe-searre search results in performance gains, but only the related tasks. Training all the tasks yields better results on average across the task formats.
MDCR Medić and Šnajder (2022) introduced the benchmark showing that BM25 outperforms neural models on the task of retrieving papers cited by query paper from a large candidate pool across multiple scientific fields. Table shows that our multi-task format based training establishes a new state of the art on the benchmark, with the Adapters yielding the best results. We report results broken out by different fields of study in Appendix E.
new MDCR citation-recommenddaee
Efficiency While the variants producing task- format based representations are strong baselines on SciRepEval shown in Table 2, efficiency is an important practical consideration. As shown in Table 4, the control code approach only requires one new control code embedding per format, and
*In-train: FoS, Citation Count, Same Author Detection, Search; Out-of-train: DRSM, Peer Review Score, Peer- Reviewer Matching, TREC-CoVID
Phttps://githud http:///ftttbbboommmmmmmm///////

---

8 Conclusion
The evaluation for this task requires setting up a separate project repository (Sub- ramanian et al., 2021). Our current implemen- tation only produces the input embeddings for this task, but future work could integrate the entire evaluation workflow within the bench- mark.
S2AND:
We introduce SciRepEval, benchmark for sci- entific document representation methods with 24 tasks across four task formats. On this benchmark. we show that learning a separate document rep- resentation for each task format substantially im- proves task performance compared to learning a single representation for all tasks. Future work could address limitations of our work by evaluating partitioning schemes beyond task format, crafting higher-fidelity metrics to account for the diversity of tasks in SciRepEval (which vary in sensitivity and in relevance downstream applications), or further exploring how accuracy varies with compu- tational and storage cost.
Peer Review Score, h-Index of Authors: The data for these tasks is sourced from Open- Review and consists of a static set of papers from 2017-2022. The data could be periodi- cally updated to avoid staleness, and ana- lyze yearly trends.
Cite (SciDocs): About 1700 positively la- beled citation pairs in the task (Cohan et al., 2020) are in fact citations of citations. Since it is an existing benchmark, we include it without any changes, but we could extend SciRepEval by including MDCR (Medié and Snajder, 2022) which covers more fields of study, but we do report MDCR results in this work.
Limitations
Dependence on short text features All the tasks and training techniques described our work de- pend upon the textual features of a scientific paper, namely titles and abstracts. Depending upon the document source the abstract may not always be present. To be robust to this case, when pre-training SPECTER2 Base we ensure that for 5% of the pa- pers we only process the titles. However, the model performance may be subpar in the absence of the paper abstract.
More task formats: Although proximity em- beddings from the SPECTER2 models can be used for any task formats not covered by SciRepEval, we could extend the benchmark to include more formats such as question an- swering and generation.
Further, we do not explore methods that use full text of papers since this is less consistently available (and also increases computational cost of wransformer-based models), but exploring this could be helpful in future work, Moreover, scien- tific documents are associated with rich metadata other than the text contents such as authors, cita- tions, venue, etc. We use this data in some tasks like Search and Citation Prediction, but not in oth- ers. The data can be useful in the absence of paper text, and as part of future work we can explore supplementing the textual features.
Confirming benchmark findings in real- world applications: Our experiments show that the new embeddings we introduce raise performance on our benchmark: would be helpful to determine whether the higher scores on our fixed data sets actually translate to bet- ter application metrics, by performing e.g. an online A/B test with an application that con- sumes the embeddings (paper recommenda- tion, search, etc.).
Other task partitioning methods We partition the SciRepL'val training tasks as per task formats based on the intuition in section 4 and although it gives better performance than random partition runs as reported in section 7, there may be other task partitions that can yield even better results. Clearly, including the task format to be evaluated on during training helps as seen in Table 11, but this still does not return the best overall results on our benchmark. During our preliminary experi- ments, we had an additional format specialized author-based tasks which included Same Author
Potential benchmark extensions A number of improvements our benchmark are possible, in- cluding:
Fields of study: The large training set con- tains silver labels automatically derived from publication venues with only a few hundred manually annotated samples for evaluation.
The gold label proportion could be increased, and/or improved silver data can be obtained from, for example, high-quality large lan-
guage models.

---

Detection, but found that incorporating this task into the proximity tasks instead led to better perfor- mance. Evaluating other task partitions an item of future work.
Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Iyya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Acknowledgements
We would like to thank Jonathan Bragg. Zhipeng Hou, and the anonymous reviewers for helpful com- ments, suggestions and feedback. We would also like to acknowledge the support of NASA AATT Project for funding the PeTaL research and con- tributing the biomimiery dataset. This work was supported in part by NSF Grant 2033558.
Gully Burns. 2022. Drsm-corpus vl.
Rich Caruana. 1993. Multitask learning: A knowledge- based source of inductive bias. In Proceedings of the Tenth International Conference on International Conference on Machine Learning, ICML'93, page 41-48, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
References
Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava,
Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. 2021. Muppet: Massive multi-task representations with pre-finetuning In Proceedingy of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pages 5799-5811, Online and Punta Cana, Dominican Republic. Association for Com- putational Linguistics.
Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. 2018. GradNorm: Gradient nor- malization for adaptive loss balancing in deep mul- titask networks. In Proceedings of the 35th Interna- tional Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 794-803. PMLR.
Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Hon- glei Zhuang, Vinh Q. Tran, Dara Bahri, Jianmo Ni, Jai Gupta, Kai Hui, Sebastian Ruder, and Donald Metzler. 2022. Ext5: Towards extreme multi-tast task scaling for transfer learning. In International Confer- ence on Learning Representations.
Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel Weld. 2020. SPECTER: Document-level representation learning using eitation-informee transformers. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pages 2270-2282, Online. Association for Computational Linguistics.
Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In The first international conference language resources and evaluation workshop on linguistics coreference.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Pupers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Beltagy, Kyle Lo, and Arman Cohan. 2019. SciB- ERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- pages 3615-
guage Processing 3620, Hong Kong, China. Association for Computa- tional Linguistics.
Christopher Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. 2021. Efficiently identifying task groupings for multi task learning. In NewrIPS.
Vera Boteva, Demian Gholipour Ghalandari, Artem Sokolov, and Stefan Riezler. 2016. A full-text learn- ing to rank dataset for medical information retrieval. In European Conference on Information Retrieval.
Tianyu Gao, Xingeheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Process- ing, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Peter Brown, Ameya Sadguru Kulkarni, Osama Re- fai, and Yaoqi Zhou. 2019. Large expert-curated database for benchmarking document similarity de- tection in biomedical literature search. Database: The Journal of Biological Databases and Curation, 2019.
Winston Haynes. 2013. Tukey's 's Test, pages 2303-2304. Springer New York, New York, NY.
Matthew Honnibal, Ines Montani, Sofie Van Lan- deghem, and Adriane Boyd. 2020. spacy: Industrial- strength natural language processing in python.
Tom Brown. Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind

---

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parfmeter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Leurning, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR.
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2022. Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316 16.
Sheshera Mysore, Arman Cohan, and Tom Hope. 2022. Multi-vector models with textual guidance for fine- grained scientific document similarity. In Proceed- ings of the 2022 Conference of the North American Chapter of the Association for Computational Lin- guisties: Human Language Technologies. Seattle, United States. Association for Computational Lin- guistics.
Naman Jain and Mayank Singh. 2021. Tweetpap: A dataset to study the social media discourse scien- tific papers. In 2021 ACM/IEEE Joint Conference on Digital Libraries (JCDL), pages 328-329.
Sheshera Mysore, Tim O'Gorman, Andrew McCallum, and Hamed Zamani. 2021. Csfcube-a test collec- tion of computer science research articles for faceted query by example. In Thirty-fitth Conference on Neu- ral Information Processing Systems (NeurIPS).
Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong. and Richard Socher. 2019. Ctrl: A conditional transformer language model for control- lable generation. ArXiv, abs/1909.0558 999.08888.
Carolyn E Lipscomb. 2000. Medical subject headings (mesh). Bulletin of the Medical Library Association, 88(3):265.
Arvind Neelakantan, Tao Xu, Raul Puri, Alec Rad- ford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas A. Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke. Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger, David P. Schnurr, Felipe Petroski Such, Kenny Sai-Kin Hsu, Madeleine Thompson, Tabarak Khan, Toki Sherbakov, Joanne Jang, Peter Welinder. and Lilian Weng. 2022. Text and code embeddings by contrastive pre- tre-training. ArXiv, abs/2201. 10005..
Shengchao Liu, Yingyu Liang, and Anthony Gitter. 2019a. Loss-balanced task weighting to reduce neg- ative transfer in multi-task learning. Proceedings of the AAAI Conference on Artificial Intelligence. 33701)99777-9978
Xiang Liu, Torsten Suel, and Nasir Memon. 2014. A robust model for paper reviewer assignment. In Proceedings of the 8th ACM Conference on Recom- mender Systems, RecSys 14, page 25-32, New York, NY, USA. Association for Computing Machinery.
Malte Ostendorff, Till Blume. Terry Ruas, Bela Gipp. and Georg Rehm. 2022a, Specialized document em- beddings for aspect-bases similarity of research pa- pers. In Proceedings of the 22nd ACM/TEEE Joint Conference on Digital Libraries, JCDI. '22, New York, NY, USA. Association for Computing Machin- ery.
Xiaodong Liu, Jianfeng Gao, Xiaodong He, Li Deng, Kevin Duh, and Ye-yi Wang. 2015. Representation learning using multi-task deep neural networks for semantic classification and information retrieval. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 912-921, Denver, Colorado. Association for Computational Linguistics.
Malte Ostendorff, Nils Rethmeier, Isabelle Augenstein. Bela Gipp, and Georg Rehm. 2022b. Neighborhood Contrastive Learning for Scientific Document Rep- resentations with Citation Embeddings. In The 2022 Conference Empirical Methods in Natural Lan- guage Processing (EMNLP 2022), Abu Dhabi. Asso- ciation for Computational Linguistics. 7-11 Decem- ber 2022. Accepted for publication.
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jian- feng Gao. 2019b. Multi-task deep neural networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4487-4496, Flo- rence, Italy. Association for Computational Linguis- tics.
Vishakh Padmakumar, Leonard Lausen, Miguel Balles- teros. Sheng Zha, He He. and George Karypis. 2022. Exploring the role of task transferability in large- scale multi-task lcarning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2542-2550, Seattle, United States. Association for Computational Lin- guistics.
Tyy Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Confer- ence on Learning Representations (ICLR).
Zoran Medié and Jan Snajder. 2022. Large-scale eval- uation of transformer- bassed article encoders on the task of citation recommendation. In SDP.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018 Deep contextualized word repre- sentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long Papers), pages 2227-2237, New Orleans. Louisiiaa. Association for Computa- tional Linguistics.
David Mimno and Andrew McCallum. 2007. Expertise modeling for matching papers with reviewers. In Proceedings of the 13th ACM SIGKDD International Conference Knowledge Discovery and Data Min- ing, KDD '07, page 500-509, New York, NY, USA. Association for Computing Machinery.

---

Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2021. AdapterFusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Con- ference of the European Chapter of the Association for Computational Linguisties: Main Volume, pages 487-503, Online. Association for Computational Lin- guistics.
Marco Valenzuela, Vu A. Ha, and Oren Etzioni. 2015. Identifying meaningful citations. In AAAI Workshop: Scholarly Big Data.
Christophe Van Gysel and Maarten de Rijke. 2018. Pytrec_evar _erre_eval: An extremely fast python interface to trec_eval. In The 41st International ACM SIGIR Con- Serence on Research and Development in Information Retrieval, SIGIR '18, page 873-876, New York, NY, USA. Association for Computing Machinery.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu, 2020. Exploring the limits of transfer learning with a unified text-to-text transfomeer. Journal of Machine Learning Research,
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N. Gomez. Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Sys- tems, NeurlPS 17, page 6000 Red Hook, NY, USA. Curran Associates Inc.
(1(14)):1667
Victor Sanh. Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, et al. 2021. Multitask prompted training enables zero-shot task generalization. In International Con- ference on Learning Representations.
Ellen Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman William R. Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. 2021. Trec-covid: Constructing pandemic information retrieval test collection. SIGIR Forum, 54(1).
Vikram Shyam, Lauren Friend, Brian Whiteaker, Nicholas Bense, Jonathan Dowdall, Bishoy Bok- tor, Manju Johny, Isaias Reyes, Angeera Naser, Nikhitha Sakhamuri, Victoria Kravets, Alexandra Calvin, Kaylee Gabus, Delonte Goodman, Herbert Schilling. Calvin Robinson, Robert Omar Reid II, and Colleen Unsworth. 2019. Petal (periodic table of life) and physiomimetics. Designs, 3(3).
Alex Wang. Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understand- ing systems. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu, 2020. Mpnet: Masked and permuted pre- training for language understanding. arXiv preprint arXiv:2004.09297.
Kuansan Wang, Iris Shen, Charles Huang, Chich-Han
Wu, Yuxiao Dong, and Anshul Kanakia. 2020a. Mi- crosoft academic graph: when experts are not enough. Quantitative Science Studies, 1(1):396-413,
Asa Cooper Stickland and lain Murray. 2019. BERT and PALs: Projected attention layers for efficient adaptation in multi-task learning. In Proceedings of the 36th International Conference on Machine Learn- ing, volume 97 of Proceedings of Machine Learning Research, pages 5986-5995, PMLR.
Liang Wang, Nan Yang. Xiaolong Huang, Binxing Jiao, Linjun Yang. Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. Text embeddings by weakly- supervised contrastive pre- pre-training arXiv preprint arXiv:2212.03533.
Shivashankar Subramanian, Daniel King, Doug Downey, and Sergey Feldman. 2021. S2AND: Benchmark and Evaluation System for Author Name Disambiguation. In JCDL '21: Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2021, JCDL. '21, New York, NY, USA. Association for Computing Machinery.
Lucy Lu Wang, Kyle Lo. Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang. Doug Burdick, Darrin Eide, Kathryn Funk. Yannis Katsis, Rodney Michael Kinney, Yunyao Li, Ziyang Liu, William Merrill, Paul Mooney, Dewey A. Murdick, Devvret Rishi, Jerry Sheehan, Zhihong Shen, Brandon Stilson, Alex D. Wade, Kuansan Wang. Nancy Xin Ru Wang, Christopher Wilhelm, Boya Xie, Douglas M. Ray- mond, Daniel S. Weld, Oren Etzioni, and Sebastian Kohlmeier. 2020b. CORD-19: The COVID-19 open research dataset. In Proceedings of the 1st Work- shop on NLP for COVID-19 al ACL 2020, Online. Association for Computational Linguistics.
Yi Tay, Mostafa Dehghani, Vinh Quang Tran, Xavier Garefa, Dara Bahri, Tal Schuster, Huaixiu Zheng. Neil Houlsby, and Donald Metzler. 2022. Unifying language learning paradigms. ArXiv, 0bs/2205.05111..
Amalic Trewartha, Nicholas Walker, Haoyan Huo, Sanghoon Lee, Kevin Cruse, John Dagdelen, Alexan- der Dunn, Kristin A. Persson, Gerbrand Ceder, and Anubhav Jain. 2022. Quantifying the advantage of domain-specific pre-training named entity recognition tasks materials science. Patterns, 1044:
Peng Xu, Xinchi Chen, Xiaofci Ma, Zhiheng Huang, and Bing Xiang, 2021. Contrastive document rep- resentation learning with graph attention networks. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3874-3884, Punta Cana, Dominican Republic, Association for Compu- tational Linguistics.

---

Michihiro Yasunaga, Jure Leskovec, and Percy Liang. 2022. LinkBERT: Pretraining language models with document links. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Pupers), pages 8003-8016, Dublin, Ireland. Association for Computational Lin- guistics.
Each query consists of a short title, question ask- ing the required information and a narrative de- scribes briefly exactly the type of information that the results should have. For our evaluation we com- bine these fields into single text separated by the [SEP] token.
Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. 2021. CrossFit: A few-shot learning challenge for cross- task generalization in NLP. In Proceedings of the 2027 Conference on Empirical Methods in Natural Language Processing, pages 7163-7189, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
A.2 Proximity
S2AND and Same Author Detection The S2AND dataset (Subramanian et al., 2021) contains signatures (author-paper pairs) that are clustered cording which author mentions refer the same person. Due the high resource requirements of running the original S2AND evaluation, we create S2AND-mini, a version of S2AND with only 1000 blocks from each of S2AND's ' dataset sources and at most 500 signatures per block. Our evaluation of S2AND-mini follows the original evaluation of S2AND; that is, method's document embed- dings are used along with author and paper meta- data create features for clustering algorithm that consists of a pairwise scoring model followed by greedy agglomerative clustering. We use B Fl (Bagga and Baldwin, 1998) as in the original paper for evaluation.
Shunyu Zhang, Yaobo Liang, Ming Gong, Daxin Jiang, and Nan Duan. 2022. Multi-view document repre- sentation learning for open-domain dense retrieval. In Proceedings of the 60th Annual Meeting the A Association for Computational Linguistics (Volume I: Long Papers), pages 5990-6000, Dublin, Ireland. Association for Computational Linguistics.
Yue Zhao, Ajay Anand, and Gaurav Sharma. 2022. Re- viewer recommendations using document vector em- beddings and a publisher database: Implementation and evaluation. JEEE Access. 10:21798-21811.
A SciRepEval Tasks
A.1 Ad-hoc search
Search We used click through data from Seman- tie Scholar, an academic search engine. Only search queries with at least 10 results were in- cluded, and a set of heuristic rules were applied to exclude likely noise and bots. We removed au- thor queries when the query was judged to con- tain any person tokens by named entity recognition (Honnibal al., 2020).
We also use S2AND to create the data for our same-author detection task. Unlike the original S2AND evaluation, our same-author task uses only paper embeddings without any additional author or paper metadata, which allows to directly train the embedding model on the data. Same-author detection is formulated as triplet ranking task; given three papers of which two share an author, the goal is find the matching pair.
NFCorpus This is set of non technical English queries and associated results compiled from Nutri- tionFacts.org (Boteva et al., 2016). The candidate document are selected from PubMed and the rele- vance score represents whether the candidates are linked from the query article webpage. A score of 2 represents a direct link and 1 represents link from a directly linked article. We use the test sub- set of 323 queries for evaluation. To generate the negative candidates, we randomly sample at most 100 documents for each query from the remaining Pubmed corpus.
An acronym for RElevant LIterature SearcH, this task is joint annotation effort of more than 1500 scientist across 84 countries (Brown et al., 2019). It consists of over 190k relevance labels for PubMed articles with regards to a given query (seed) article. The articles cover 76% of MeSH descriptors. The labelling scheme has 3 scores: 2-relevant, 1-partially relevant and 0- irrelevant. We use the complete dataset for evaluat- ing our methods.
RELISH
TREC-COVID TREC-COVID was introduced by Voorhees et al. (2021) as a biomedical literature search task relevant COVID-19. The dataset con- sists of 50 search queries and candidate literature from the CORD-19 corpus (Wang et al., 2020b) along with their relevance scores on a scale of 0-2.
Peer Reviewer Matching In this task the goal is to judge whether given paper is relevant to a potential reviewer. As data for this task is hard to obtain at scale due the double-blind nature of many conferences and journals, we combine multiple existing reviewer-pane papeer matching datasets:

---

Mimno and McCallum (2007), with 393 paper-review relevance ratings from a corpus of 148 NeurlPS 2006 papers and 364 review- ers, annotated by nine human experts.
up to five triplets, where the positives are sampled from papers directly cited by the query and neg- atives are chosen either randomly (easy) or from citations of citations (hard). easy and 2 hard difficult are chosen for each query. To evaluate the effectiveness of this pre-training we follow Co- han et al. (2020) and use SciDocs for evaluation, excluding the recommendations task.
Liu et al. (2014), an extension of Mimno and McCallum (2007) which adds 766 additional paper-review annotations.
Zhao et al. (2022), with 694 paper-reviewer relevance ratings from corpus of 75 papers and 1833 reviewers from the IEEE ICIP 2016 conference, annotated by human experts.
A.3 Classification
MeSH Descriptors Medical Subject Headings (MeSH) (Lipscomb, 2000) indexes biomedical pub- lications into categorical hierarchy consisting of descriptors which refer to topie headings and spe- cific aspect related to topic respectively, The dataset is a collection of scientific documents be- longing to the 30 most frequently occurring top level MeSH descriptors and having exactly one qualifier. We filter out the records that don't 't have an associated qualifier. The descriptors thus serve as the labels in the multi-class classification task.
All datasets have been annotated on the same 0-3 relevance rating scale. The candidate reviewers are all researchers, and we embed all the papers written by them using our models. To obtain the model's score for each candidate reviewer, we compute the cosine similarity between the query paper and each of the candidate's papers, and take the mean of the top 3 similarities the score. We consider two ways to map the 0-3 relevance judgements to binary labels-hard and soft decision-wher for the soft decision a score of or is considered relevant and for hard decision only score of is considered relevant. Precision at 5 (P@5) and 10 (P@ 10) results are used as the final metric, which ultimately results in four numbers (P@5 and P@ 10 for each of hard and soft decisions), which are averaged to produce the single number reported in final results for this task.
Fields of Study (FoS) The FoS task multi- label classification problem where each scientific document is assigned one or more classes out of 23 possible fields. For gold test data, we manually la- beled 471 papers into at most three fields-of offlsuddyy For silver training data, assumed that paper within venue generally falls within narrow set of fields and manually assigned FoS labels to publica- tion venues. We then propagated the venue labels to the papers published therein.
To evaluate different data sizes, we obtain the F1 score on the gold data in three settings: 5-shot, 10- shot, and the complete gold test set. The average of these scores is treated as the score for this task when computing the overall average score for the benchmark.
Highly Influential Citations In this task, given a paper and paper B, we aim to predict whether B is highly influenced by A. As measuring influence is subjective and human annotation is expensive, we approximate influence by counting the number of times A is cited in the text of B. If A is cited at least 4 times, we consider it to be highly influen- tial (a positive example in our triplet-based loss); otherwise, we consider it to be a negative example. During evaluation, we sample query papers which have at least positive candidates and compute the L2 distance for similarity ranking. Note that our definition of 'influential' differs from that in Valenzuela et al. (2015).
Disease Research State Model (DRSM) DRSM (Burns, 2022) is a collection of Pubmed papers that deal with six specific aspects of rare diseases. The gold data is annotated by in-house experts and used for evaluation, while the silver data is gener- ated by annotation service providers with medical expertise.
Similar to FoS, we obtain the Fl score on 24- shot, 64-shot, and full data, then average the results before computing the final benchmark score.
Citation Prediction (SPECTER Pre-training Triplets) This is the task and dataset used for pre-training in Cohan al. (2020). based on citation links between scientific documents where each instance a triplet consisting of a query, a positive and a negative paper. Each query can have
Biomimicry We sample tags for of papers in the PeTal. database (Shyam et al., 2019) to create a binary classification dataset with labels indicating whether each paper is about biomimicry. The data

---

is unbalanced, with only 13% positive samples. We evaluate 16-shot, 64-shot, and full-data setup and take the mean to get the final score.
Task form
Input format
Classification
concat doc)
Regression
(onca)([RGN]] doc)
Proximity
([PXX] doc)
Ad-hoe Search (IOC3 query//doc))
A.4 Regression
Citation Count We sample a collection of scien- tific articles published in 2016 from the set of pa- pers in the search datasct described in section A.I, so that 5 year period has passed for them to col- lect citations. Each article has at least one citation, and the citation counts are converted to log scale.
Table 6: Assigned input formats and control codes for each task form. [CLF], [RGN], [PRX] and [QRY] are special tokens, doc is the input.
the feeds datasets have been replaced with REL- ISH and NFCorpus. Both are recommendation tasks similar to feeds - RELISH, a proximity task with 60 candidatcs/query and NFCorpus, an adhoc- search task with 138 candidates/query on average.
Year of Publication The aim of this task is to determine research trends by predicting the year of publication of a scientific article. We sample publications from the search dataset with publi- cation date after the year 2005 and scale the years so that their values are between 0 and 1. Further, since this task is used for training along with cita- tion count prediction, and to align the loss scales, the labels are scaled by the mean of the labels in citation count for parity.
B Implementation details
During pre-training, all the tasks with the same format share their task-format specific parameters. The control code based paradigm introduces four new (randomly-imitialzzed) special tokens to the vocabulary. We try initializing these additional parameters randomly, with the [CLS] token and a combination of [CLS] with some noise. How- ever, it has little impact on the resulting model per- formance with random initialization being better average. Further, we also tried loss weighting strategies (Chen et al., 2018 8; Liu et al., 2019a) but our preliminary experiments produced better re- sults without any scaling so we didn't explore it further. All the base models are trained for two epochs on two 48GB NVIDIA Quadro RTX 8000 GPUs with 16 bit precision, an effective batch size of 256, and maximum input length of 512 to- kens. Each batch is sampled with an equal number of examples from each task." 12 We use Adam (Loshchilov and Hutter, 2019) with e = 1e-8. The learning rate follows an inverse square root sched- ule with a linear warmup of 700 steps and peak of 5e-5.
Peer Review Score We use the OpenReview "PI to collect paper metadata and correspond- ing review scores for ICLR conferences from 2017 to 2022. Each reviewer in ICLR assigns final rating in the range [0-10], and we take the mean rating as the label for every paper.
h-Index of Authors In this task the goal is to predict the maximum h-Index of any of the authors of scientific publication. We re-use the peer re- view score dataset, obtain the h-Index of all the authors for each paper using the Semantic Scholar AP and pick the max as the label. The labels are normalized to lie between [0.1].
Tweet Mentions The goal of this task is to pre- dict the combined number of paper's mentions and retweets. We post-process the dataset created by Jain and Singh (2021) containing tweets about Arxiv papers between 2010-19. The sum of nor- malized counts of mentions and retweets is finally considered as the score to be predicted.
The adapter approaches follow the two step train- ing process and learning rate configurations de- scribed in Pfeiffer et al. (2021). One adapter per task family is attached to the base model in both single adapter and fusion stages and is trained for a maximum of 6 and 4 epochs respectively. For PALs one layer is added per task format and the entire network is trained for epochs in Stickland and Murray (2019).
The prior versions of SciRepEval included three feeds datasets. We observed high correlation in model performance among these tasks, similarly to SciDocs in Appendix G. Moreover, each query was associated with 10 candidates on average, which is small for a real recommendation engine. For a more robust evaluation with larger candidate pools,
10 "https://api.ppeent http:///ppp.peeereiie..ntttt il "ttnps//ppi.ccmattisccoo..
We experimented with mixed and task sequential batching as well which did not yicld good results.

---

Model
In- Train
Out-of-Train
Avg
Model
In- Train
Out-of-Trrin
Average
TS ASPIREes
65.0
86.3
82.7
SPECTER
TS ASPIREai
65.5
86.0
82.6
MTL CLS
60.0
71.6
68.6
OT ASPIREcs
64.5
86.4
82.8
MTL CTRL
61.9
72.7
69.9
OT ASPIRFaio
65.0
86.0
82.5
Adapters
61.5
73.2
70.2
SPECTER2
Adapters +
62.2
73.6
70.6
MTL CTRL
67.0
83.3
MTL CTRL
Adapters
67.7
SeiNCI.
MTL CLS
60.1
71.9
68.8
Table 7: Comparison of SPECTER2 multi-format meth- ods with ASPIRE on proximity tasks. The best results for each base model are underlined. TS: Text Supervi- sion, OT: Optimal Transport
MTL CTRL
62.1
72.9
70.1
Adapters
61.9
73.8
70.7
Adanters +
62.5
74.0
71.0
MTL CTRL
B.1 Evaluation
Table 8: Results for multi formia training with SPECTER and SciNCL, as hase models. For brevity, report only the single adapters results due their advantage of computation efficiency. The best results for each base model are underlined.
For classification and regression, we train lin- ear SVM on each downstream task using the em- beddings as input, and we tune the regularization parameter C via grid search. Multi-class and multi- label classification are configured under the one vs all classifier setting.
E SviRepEval Domain Distribution and MDCR Evaluation
ASPIRE Evaluation
We study the domain diversity of SciRepEval and display the results in Table 9. To compare against the training data for SciDocs, we consider the ci- tation prediction triplets on which SPECTER is trained which also a subset of the SciRepEval in-train tasks. Even though Medicine and Com- puter Science papers still form bulk of the data, SciRepEval has 105x more documents on average per domain compared to the SPECTER triplets.
ASPIRE (Mysore et al., 2022) produces representa- tions for the dense retrieval of scientific documents based on matching multiple aspects between the query and candidates. To evaluate these representa- tions under the settings they are designed for, we only report the results on the proximity tasks in Ta- ble 7. We use the model implementations available HuggingFace which have been pre-trained on documents from the Computer Science (CS) and Biomedical (Bio) domains. The models variants can be further sub-categorized retrieval based on best aspect matching (TS ASPIRE) and a weighted sum of the similarity score among all the aspects based on Optimal Transport (OT ASPIRE) between the query and candidates. Both our multi-format approaches with control codes and adapters pro- duce better results overall and out-of-train tasks, Note however, since ASPIRE models are trained on co-citations, they perform much better on average on the citation based tasks from SeiDoes and thus achieve relatively higher scores on out-of-train.
Further, as shown in Table 5, our task format based models outperform BM25 on the MDCR benchmark (Medić and Snajder, 2022) and estab- lish the new state of the art. Table 10 displays the breakdown of the results by fields of study. Apart from Geology and History, the ensemble model is equivalent or better than BM25 on all the scientific domains.
F SPECTER Objective
Lastly, we perform an ablation study to better understand the importance of the unsupervised citation-based training objective. We used SciB- ERT as the base model for this ablation since both SPECTER and SciNCL were trained with the ci- tation objective. Removing the citation objective and its accompanying data from SeiBERT MTL CTRL, find that the in-train performance drops from 61.9 61.8, while out-of-train drops from 57.9 to 57.5, hinting that the citation objective may be helpful for generalization to new tasks.
D Robustness of multi-format training
Table 8 shows comparison of SciRepEval results between MTL CLS, MTL CTRL and Adapter vari- ants trained with SciNCL and SPECTER the base models. The multiple embedding approaches of control codes and adapters are better than simple multi task learning that produces a single document embedding across the board.

---

Field of study
SviRepEval (A)
SciDoes (B)
Increase Ratio (A/B)
Medicine
3,201,323
74,685
43
Computer Science
1,187,689
199,664
Biology
882.357
13.377
66
Chemistry
508.056
3,813
133
Psychology
492,071
22.590
22
Materials Seience
271,865
7,681
Engineering
254,826
31,444
Mathematics
231,482
25,800
Physics
217,670
7,285
Business
217,585
5,450
Sociology
156,128
2,305
Political Science
154,388
1,032
150
Economics
123,357
2,705
Environmental Science
91,682
1,136
Art
89,527
206
435
Geography
83,688
1,491
56
Philosophy
61,996
151
411
Geology
51,103
640
History
46,430
159
292
Table 9: Data domain distribution in SciRepEval for the training tasks and comparison with SeiDocs. We group the unique documents in both the benchmarks by their MAG (Wang al., 2020a) fields of study and present the counts in columns 2 and and the absolute increase per field in column4.
SPECTER2
SPECTER2 Basc
SPECTER2 MTL CTRL
SPECTER2 Adapters
BM25
Adapters
MTL CTRL
FoS
MAP
R R@5
MAP
R@5
MAP
R@5
MAP
R@5
MAP
R R@5
Art
38.2
32.3
43.4
37.8
39.9
32.6
44.7
37.4
43.5
36.7
Biology
38.3
33.6
39.9
33.3
40.6
33.9
41.6
36.0
42.2
37.1
Business
28.1
22.5
35.0
30.5
32.7
28.6
35.3
29.6
34.7
29.3
Chemistry
38.0
32.6
39.7
33.5
38.9
32.4
41.4
35.7
40.8
35.3
Computer Science
34.8
30.5
38.5
33.4
38.9
32.8
39.3
33.5
39.9
34.7
Economies
30.5
26.0
33.7
28.5
31.9
26.3
33.7
27.3
33.7
29.5
Engineering
34.6
29.3
35.4
30.3
36.5
30.3
35.6
29.8
36.6
30.1
Environmental Science
31.6
26.2
35.0
27.9
36.8
30.7
36.7
30.5
38.1
32.5
Geography
31.8
27.8
37.1
31.8
34.0
29.2
37.2
33.3
36.7
32.6
Geology
33.1
28.0
33.4
27.7
32.5
25.8
33.6
28.4
34.1
28.4
History
38.1
32.9
41.9
34.7
37.3
31.3
41.4
36.1
40.3
34.7
Materials Science
36.1
30.7
39.7
34.0
39.4
32.8
39.9
34.3
40.7
34.2
Mathematics
35.3
28.3
40.8
34.2
39.0
33.6
41.7
35.7
40.8
34.2
Medicine
38.6
32.5
43.8
39.0
43.5
38.2
45.5
40.0
45.3
39.6
Philosophy
30.2
25.7
37.2
32.0
33.3
28.6
36.3
32.4
36.2
31.6
Physics
35.1
30.2
37.6
32.5
37.2
30.8
37.6
32.7
39.0
33.9
Political Science
28.6
23.1
35.7
31.6
32.0
26.9
35.0
29.5
34.0
29.0
Psychology
32.5
28.9
38.8
33.2
37.1
31.6
39.5
34.4
39.4
34.1
Sociology
26.8
20.5
34.6
29.8
31.2
26.2
34.3
29.9
33.5
28.0
Avg
33.7
28.5
38.0
32.4
36.5
30.7
38.4
33.0
38.4
32.9
Table 10: MDCR benchmark results breakdown by Fields study (FoS). Best results are highlighted in bold.

---

G Cross-Task Correlation Analysis
Figure 2 show's s Pearson correlations of model per- formance metrics between tasks in SciRepLival. To compute the correlations, we include all of the individual task results of the model runs shown in Table 2 and Table 8, excluding the ensembles. The correlations between tasks in SeiDocs (bot- tom right) are highest, while correlations between tasks in the entirety of SciRepEval span a larger range. Notably, DRSM, Biomimiery and S2AND are uncorrelated with most other tasks. In short, between-task diversity is larger in SciRepEvval than in SciDocs.
H Related Tasks for MTL Training
When training multiple tasks simultancously, it is important to choose a combination of tasks that does not display negative transfer (Aribandi et al., 2022; Padmakumar et al., 2022; Fifty et al., 2021). Given T tasks, it may be computationally prohibitive to train all 2T - 1 task combinations find the best one. Alternatively, recent work suggests pre-fine-tuning on large collection of tasks simultaneously offsets the negative transfer between subset of those tasks (Aghajanyan et al., 2021; Aribandi et al., 2022). Padmakumar al. (2022) show that pre-fine-tuning on a small set of tasks related to the downstream task more effi- cient than large scale multi-task training and yields similar results. As shown in Table 11, we study the training affinity among our task formats by pre-fine- tuning on individual task formats well pairwise combinations in multi-task setup. Supporting the findings in Padmakumar et al. (2022), pre-fine- tuning on the tasks from each individual format's training data gives better or similar performance on downstream tasks when compared to training on all the formats at once, except on out-of -ft-of- classification and search. Surprisingly, regression when combined with proximity and search gives the best result both in-train and out-of-train tasks. Additionally, related tasks like proximity and ad-hoe search also provide one another boost when trained together. However, training on all the tasks simultancously yields the best results on average, and individual results are within point of the best combinations task format, except for out-of-train regression and in-train classification.

---

Task Format Evaluated On
Task Format(s)
Trained On
Train
Oftiof-Train Train
CL.
RGN
PRX
SRCH
Avg
CLFFF
RGN
PRX
SRCH
Avg
Avg
CLF
69.1
16.3
58.4
71.3
54.9
70.7
18.3
79.7
73.4
68.0
63.7
RGN
48.8
45.8
53.1
69.6
52.7
63.9
20.4
61.5
67.4
56.4
55.3
PRX
57.5
33.8
66.8
73.1
53.3
69.5
17.6
87.3
78.5
72.4
68.0
SRCH
53.2
32.0
59.8
78.1
53.4
63.5
14.6
77.6
78.0
65.44
62.0
CLF+RGN
68.4
41.9
60.9
71.8
54.6
71.5
18.0
79.0
73.9
67.8
65.6
CLF+PRX
65.9
31.9
65.7
73.0
56.5
70.9
16.3
86.6
75.7
71.8
68.0
CLF+SRCH
66.6
20.3
60.8
78.4
53.7
70.9
13.9
81.5
72.0
68.3
64.4
RGN+PRX
58.4
44.0
67.4
72.4
57.4
69.5
18.7
87.3
78.2
72.6
69.0
RGN+
$4.6
44.4
60.2
78.8
55.6
67.2
18.4
77.2
78.3
66.5
63.9
PRX+SRCH
57.5
33.2
66.8
78.1
56.5
68.2
18.4
86.8
79.3
72.1
67.9
All
67.5
44.8
67.0
78.3
71.7
19.3
87.2
79.7
73.1
70.4
Table 11: Task relatedness analysis for choosing a sub-group of tasks to train on to obtain optimum performance. SPECTER2 Base model is trained on one or more task formats (rows) and then evaluated a comparison with MTL CTRL (last row). Both per task format and overall average performance reported (columns). The best training combination for every task highlighted in bold. The best single and combined training results for every evaluated task format respectively are underlined.
1.0
Biomimicry
DRSM
RELISH
NFCorpus
0.8
TREC-COVID
S2AND
Peer Review Matching
0.6
Review Score
Max h-index
Tweet Mentions
0.4
MeSH
FoS
Same Author
0.2
Search
Highly Cited
Citation Count
0.0
Pub Year
Scidocs MAG
Scidocs MeSH
-0.2
Scidocs Cite
Scidocs Co-View
Scidocs Co-Cite
-0.4
Scidocs Co-Read
Biomimicry
DRSM
RELISH
NFCorpus
TREC-COVID
S2AND
Peer Review Matching
Review Score
Max h-index
Tweet Mentions
MeSH
Fos
Same Author
Search
Highly Cited
Citation Count
Pub Year
Scidocs MAG
Scidocs MeSH
Scidocs Cite
Scidocs Co-View
Scidocs Co-Cite
Scidocs Co-Read
Figure Correlations of model performances between tasks SeiRepEval.


Reviews:
[{"review_id": "N6KLE5BCtA", "reviewer": "Reviewer_vkEw", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the helpful comments!\n\nWe agree that SciRepEval is an incremental improvement, but we feel it is a necessary one and appropriate for the “Resources and Evaluation” track of EMNLP. Unlike existing benchmarks, we do not restrict the evaluation to candidate ranking tasks but expand to 3 more task types, leading to the creation of 11 new datasets. Our proposed benchmark with 25 tasks, fulfills the need for comprehensive evaluation of scientific document embeddings where it was earlier difficult to move the needle due to lack of task coverage.\n \n\nWe would like to cite contemporary general embedding benchmarks MTEB (Muennighoff et al., EACL 2023) and BEIR (Thakur et al., NeurIPS 2021) as similar works that introduce a large collection of datasets as a standard benchmark for consistent evaluation across models. We try to fulfill the need for the same in the scientific document embedding space. We also show that our approach of learning separate embedding spaces for different task formats (nearest-neighbor, classification, regression) is effective, which is novel to our knowledge.\n\nWe thank the reviewer for pointing out the grammatical mistake. We will fix it in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper is very well written, clearly explaining the area of focus and contributions as well as providing in-depth explanation and relevant information/metadata for the steps described. \n\n* The work presents strong reason and need for a diverse benchmark for scientific document representation, and takes a step towards that goal by providing a highly diverse benchmark.\n\n* The paper also provides a new format-based embedding approach which showcases better performance in case of generalizability as compared to other state-of-the-art models.", "reasons_to_reject": "* The benchmark, although diverse and better than existing benchmarks in various areas as discussed in the paper, can be termed as an incremental work at best and does not have much novelty\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The work presents a benchmark, SciRepEval, which can be termed as a combination of new tasks as well as existing benchmarks and datasets such as SciDocs. The benchmark consists of 25 tasks out of which 11 are new. These tasks are split across 4 different formats, namely, classification, regression, ranking and search. The paper also presents a new approach called Multi-Format embedding, where instead of using a single embedding across the whole scientific document, they utilize different embedding based on different formats. \n\nThe paper clearly outlines the area of focus to be the following:\n* If existing documents representations generalize to a highly diverse range of tasks\n\n* If multi-task training on diverse data can improve documentation representation models\n\n* If task-format-specific embedding representation can improve generalizability\n\nThe paper also clearly outlines the area of contributions:\n* They present a new comprehensive as well as diverse benchmark for document representation techniques across 4 different formats\n\n* An extensive investigation of generalizability of state-of-the-art scientific document representation models\n\n* A set of new multi-task document representation models which can produce task-format specific representations. This showcases boost in generalizability and outperforms prior works by 2 points absolute\n\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "On page 5 on line 383 there is a typing error, maybe the authors intended to write \"... and we learn _**model's capability**_ of producing an embedding for any given ...\""}, "extra_text": {}}, {"review_id": "2PWusq1jqY", "reviewer": "Reviewer_Tbar", "summary": "", "questions": "1. Where are the 25 task datasets from, especially these datasets from 'this work'?\n2. Is there any rationale why learn the task format into the representation that can help the generalization? ", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful comments.\n\nWith regards to novelty, in contrast to the existing widely used benchmarks, we do not restrict to candidate ranking tasks but expand to 3 more task types, leading to the creation of 11 new datasets. Our proposed benchmark with 25 tasks, fulfills the need for comprehensive evaluation of scientific document embeddings where it was earlier difficult to move the needle due to lack of task coverage. \nFurther, we propose and compare novel modeling techniques which not only use multi-task training, but also generate different effective embeddings for the same input based simply on the selection of the appropriate task.\n\nAnswers from Authors:\n\n1. Table 1 cites the sources for the existing datasets. The 11 new datasets were collated by us using public apis and academic search engines most prominently OpenReview and Semantic Scholar which provide paper metadata like title, abstracts, citations etc. We describe more about each of our tasks in Appendix A.\n\n2. We build our work on the foundational idea that large scale multi-task training helps improve model generalization ability (Ye et al., 2021; Sanh et al., 2021) . Regarding why to partition by task format in particular, Figure 1 sketches the intuition of why different task formats may benefit from different embedding spaces.  For example, the embedding space for classification (left-hand pane) returns perfect classification performance for a linear classifier (separating triangles and circles perfectly).  However, that embedding space would not be as effective for a nearest-neighbor proximity task format, because there are some triangles whose nearest neighbors are circles, and vice-versa, which would create errors for a proximity task.  By contrast, the proximity embedding space (middle pane) always has nearest neighbors of the same class, so would be better for proximity tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper introduces a comprehensive benchmark for Scientific Document Representations evaluation.\n2. The paper points out that current approaches cannot generalize well on the benchmark, but the proposed method with format code can do better.\n", "reasons_to_reject": "Lack of novelty. ", "questions_to_authors": "1. Where are the 25 task datasets from, especially these datasets from 'this work'?\n2. Is there any rationale why learn the task format into the representation that can help the generalization? ", "paper_topic_and_main_contributions": "The manuscript introduces the SciRepEval benchmark, including the 25 challenging and realistic tasks across four formats. It also provides insights into how state-of-the-art models perform on these tasks and the limitations of existing approaches. The paper also introduces a new approach that can improve performance and provides access to the best SciRep models publicly for other practitioners to use and build on. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None", "Typos_Grammar_Style_and_Presentation_Improvements": "none"}, "extra_text": {}}, {"review_id": "3zeAf7yQQ9", "reviewer": "Reviewer_U5Yz", "summary": "", "questions": "- Maybe I missed something, but I find the motivation of multi-format representing learning lacking. Could the authors elaborate on why this is important? Thanks!\n- While there are benefits to learning separate representations for each task format, the cost of storing separate models could be substantial. The authors elaborated on the inference cost in Section 7, but I'm wondering if there are other impacts on, for example, storage.\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the helpful comments from the reviewer.\n\nAnswers from Authors:\n\n1. The motivation for multi-format representation learning stems from the need for a single scientific document representation model that can generalize well across a wide range of tasks formats without further expensive fine-tuning. As also noted in response to review 2, Figure 1 sketches the intuition of why different task formats may benefit from different embedding spaces.  For example, the embedding space for classification (left-hand pane) returns perfect classification performance for a linear classifier (separating triangles and circles perfectly).  However, that embedding space would not be as effective for a nearest-neighbor proximity task format, because there are some triangles whose nearest neighbors are circles, and vice-versa, which would create errors for a proximity task.  By contrast, the proximity embedding space (middle pane) always has nearest neighbors of the same class, so would be better for proximity tasks. Task formats are just one example of categorizing downstream tasks into meaningful groups for producing general-purpose embeddings; there could be more effective partitions which can be explored as part of future work. \n\n2. While the usage of all the adapters together would require 4x the cost of storing a single embedding (or Nx depending upon number of adapters trained), we believe that storage is not a bottleneck today, being considerably cheaper than both GPU/CPU training and inference. Further, the storage cost of general-purpose embeddings would be considerably lesser compared to per-task embeddings which would scale with the number of required downstream tasks. Having said that, our approach does not increase the space cost for any individual given task. Given a task specification, users would be expected to store and generate (on demand if need be) only the actual relevant embeddings for that particular task.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- A new and solid benchmark that could be useful for the research community.\n- Well-executed experiments.\n- The authors also introduce methods that improve existing methods on the proposed benchmarks.\n", "reasons_to_reject": "- This is not necessarily a reason for rejection, but this paper is packed with contents (dataset, evaluation, and new methods). The authors could consider splitting the content of the paper into separate papers.\n", "questions_to_authors": "- Maybe I missed something, but I find the motivation of multi-format representing learning lacking. Could the authors elaborate on why this is important? Thanks!\n- While there are benefits to learning separate representations for each task format, the cost of storing separate models could be substantial. The authors elaborated on the inference cost in Section 7, but I'm wondering if there are other impacts on, for example, storage.\n", "paper_topic_and_main_contributions": "This paper focuses on learning representations of scientific documents and introduces the first comprehensive benchmark for training and evaluating document representations. The authors use this benchmark to evaluate existing methods, demonstrating the tasks' challenges. \n\nI'm not well-versed in scientific documents representation literature, so my comments should be taken with a grain of salt.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a comprehensive benchmark for Scientific Document Representations evaluation. It comprises  25 tasks out of which 11 are new. A detailed analysis of the generalizability of state-of-the-art scientific document representation models is presented along with a new method called Multi-Format embedding, where the authors  utilize different embedding based on different formats which shows better performance as compared to other state-of-the-art models.
Decision: Accept-Main