SciRepEval: A Multi-Format Benchmark for Scientific Document Representations
Authors: Amanpreet Singh
Mike D'Arcy
Arman Cohan
Doug Downey
Sergey Feldman
Venue: Submitted to ICLR 2023 (2023)

Abstract: Learned representations of scientific documents can serve as valuable input features for downstream tasks, without the need for further fine-tuning. However, existing benchmarks for evaluating these representations fail to capture the diversity of relevant tasks. In response, we introduce SciRepEval, the first comprehensive benchmark for training and evaluating scientific document representations. It includes 25 challenging and realistic tasks across four formats: classification, regression, ranking and search. We then use the benchmark to study and improve the generalization ability of scientific document representation models.  We show how state-of-the-art models struggle to generalize across task formats, and that simple multi-task training fails to improve them.  However, a new approach that learns multiple embeddings per document, each tailored to a different task format, can improve performance.
We experiment with task-format-specific control codes and adapters in a multi-task setting and find that they outperform the existing single-embedding state-of-the-art by up to 1.5 points absolute. 

Under review as a conference paper at ICLR 2023
SCIREPEVAL: A MULTI-FORMAT BENCHMARK FOR SCIENTIFIC DOCUMENT REPRESENTATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Learned representations of scientific documents serve as valuable input fea- tures for downstream tasks, without the need for further fine-tuning. However, existing benchmarks for evaluating these representations fail to capture the diver- sity of relevant tasks. In response, we introduce SciRepEval, the first comprehen- sive benchmark for training and evaluating scientific document representations. It includes 25 challenging and realistic tasks, 11 of which are new, across four for- mats: classification, regression, ranking and search. We then use the benchmark study and improve the generalization ability of scientific document representation models. We show how state-of-the-an models struggle to generalize across task formats, and that simple multi-task training fails to improve them. However, a new approach that learns multiple embeddings per document, each tailored to a differ- ent format, improve performance. We experiment with format-speciftt control codes and adapters in multi-task setting and find that they outperform the existing single-embedding state-of-the-art by up to 1.5 points absolute.
INTRODUCTION
Learning representations of whole documents is critical for a variety of NLP tasks including classi-
fication, search, and recommendation (Cohan et al. 2020 Recent work has shown how pretrained language models (e.g., (Devlin et al.) 2019: Raffel et al. 2020; Brown et al. (020 can be tailored to produce high-quality representations of documents with contrastive learning (Xu al., [ Gao et al.] 2021 Neelakantan et al., 2022]. In the scientific domain, training objectives based on contrastive learning cross-document links (e.g., citations) have shown further improvements in document-level representation learning (Cohan et al.) 2020 Ostendorff et al.] 2002b; Mysore al., 2022). These methods are especially useful because the representations they produce can be indexed and later efficiently consumed by lightweight downstream models without additional fine-tuning.
While there has been significant progress in evaluating generalizability of NLP models (Ye et al,, 2021 Sanh et al. (2021), evaluation of scientific document representation models has remained limited. Existing benchmarks either focus on document similarity (Mysore et al., 2021 Voorhees
al. 202T or include tasks that are highly correlated and not diverse (Cohan al. 2020
We introduce SciRepEval, the first benchmark for comprehensive evaluation of document- representation learning models in the scientific domain. Unlike prior work, SciRepEval is large and includes a collection of highly diverse tasks, thus encouraging research on generalization (in- cluding instance-level, cross-task and cross-domain generalization). It consists of 25 realistic tasks that reflect practical use cases of scientific document representations across four formats: text classi- fication, regression, proximity- based ranking (e.g., nearest-neighhet and ad ho-hoe search. Eleven of these tasks are new contributions. SciRepEval contains standard sets of both training and evaluation datasets to simplify and standardize comparisons between methods evaluated on the benchmark.
We then use this new benchmark to investigate and improve the generalization ability of docu- ment representation models. Following recent work (Cohan et al. 2020 Ostendorff et al. 2022b Mysore al. 2022) we further pre-fine-tune transformer language model (SciNCL; Ostendorff et al. 2022b) to produce high-quality representations for downstream tasks. We hypothesize that condensing all relevant information of the document into single vector representation might not be expressive enough for generalization across a wide range tasks. Prior work addresses a similar challenge in the context of document similarity by learning multiple finer-grained representations,

---

Under review as a conference paper at ICLR 2023
each associated with different aspect of paper (e.g., task, method, results, etc) MMyore al. 2022 Ostendorff et al., (022a). In contrast, we aim to learn effective representations for multiple downstream task formats.
Following recent success in multi-task learning in NLP (Ye et al. 2021 Sanh et al.; 2021). I. we explore large-scale multi-task training in the context of scientific document representations, where we apply suitable optimization objectives for the various task formats in SciRepEval. i.e., cross- entropy loss for classification, triplet loss for proximity/ad-hoe search, and mean square error loss for regression. We explore two state-of-the-art techniques for generating format-specific document representations: using control codes (Keskar et al. 2019; Raffel al. (2200) as input indicating the format, and ceuumeter-efficeen adapter methods (Houlsby et al. [2019 Pfeiffer et al. [2021 Sttckland
& Murray 2019) in which a separate network module introduced for every task format.
Our experiments investigate: (i) if existing document representation methods have the ability to generalize to a highly diverse set of tasks, (ii) if multi-task training on diverse data can improve document representation models, and (iii) if fask-format--pecific representations can improve gen- eralization. Through extensive analysis we find that existing state-of-the-art scientific document rep- resentation models such as SPECTER (Cohan et al. (020)) and SciNCL (Ostendorff et al.) (022b) struggle with generalizing to all task types. We interestingly find that simple multi-task training on large set of tasks is not able significantly improve the results. However, learn that multiple task format-specific representations can substantially improve generalization.
To summarize, contributions
(i) SciRepEval, a new comprehensive benchmark of 25 highly diverse and practical tasks for scientific document representation techniques across four different formats, of which 11 tasks are made available for the first time, and six of the tasks are explicitly designed for training.
(ii) An extensive investigation on the generalization ability of state-of-the-art scientific document representation models.
(iii) A set of new multi-task document representation models that, unlike existing methods, can produce representations tailored different task formats. The new methods show improved generalization over previous work, outperforming prior methods by up to 1.5 points absolute.
2 BACKGROUND
Representing Scientific Documents Earlier work aimed document embeddings used word vec-
tors (J et al. 2016 Le & Mikolov 2014 Wu et al.
2018).
convolutions (Liu et al. 2017 Zamani 2017) and BERT-based methods (Reimers &
ct al. 2018), bi-encoder networks (Conneau et al.
Gurevych 2019). Recent works have produced large scale language models pre-trained scientific corpora (Beltagy et al. 2019 Yasunaga et al. 2022. Trewartha et al. 2022). These tend to perform better than general purpose models scientific domain tasks, and serve as foundation for learn- ing dense embeddings of scientific documents. [Cohan et al.] (020) and Ostendorff et al.] (2022b) fine-tune SciBERT (Beltagy et al. (019)) with a triplet loss that encourages papers citing each other to have similar embeddings, using the title and abstract of research papers as the input.
Both Cohan al. (2020) and Ostendorff al. (2022b) are evaluated on the SeiDoes benchmark. However, of the 7 tasks in SciDoes are overly-simplistic in that the goal is to distinguish 5 real citations from 20 randomly chosen non-citations (further limitations of SciDoes are discussed in ssection 3 and Apppnndi F). Hence, the existing techniques work reasonably well on SciDocs. In contrast, SciRepEval provides a more challenging and diverse set of tasks, for both training and evaluation to help motivate methods for producing scientific document representations that can do well across multiple task formats. As a first step in this direction, we attempt to learn task-specific embeddings of the documents by pre-fine-tuning on multiple objectives simultaneously. Related to our approach, there are techniques learning multiple embeddings per paper (Ostendorff et al., 2022at Mysore et al.) 2022). These methods are, however, orthogonal to ours in that they generate an embedding per paper "facet", while we focus on learning separate embeddings per task format. In addition, these techniques focus only on finer-grained paper similarity, while our aim is producing general embeddings applicable to a variety of task formats.
Multi- Task Learning Across Formats Multi-tase task learning (Caruana 1993) with deep neural net- works has been shown to improve performance over single-task training for related objectives (Liu
2

---

Under review as a conference paper at ICLR 2023
et al. 2015:
2019b). Though unrelated tasks can lead to negative transfer, recent work has shown
that simply incressing the number of tasks tends to yield better performance in multi-tase tastiitakkkk learning
(Aghajanyan et al. 2021; Aribandi et al., 2022 Padmakumar et al. 2022) Aghajanyan et al. (2021)
pre-fine-tune pre-trained language models simulataneously on 46 tasks across 4 task types before
fine-tuning on the downstream task. [Aribandi et al. (2022) pre-train T5 (Raffel et al.; (020)
combination of C4 span denoising and 107 other tasks across 8 task families, Ye et al. (2021) intro- duce an ontology of 160 tasks for few shot multi-task training. Unlike these task families, which are divided primarily by semanties (e.g., classifying sentiment vs classifying entailment), the training tasks in ScilepEval consist of 8 large-scale scientific datasets across the four task formats. Since our goal is to evaluate final document representations, rather than fine-tune on individual downstream tasks like the above approaches, we follow SPECTER (Cohan et al. 2020) and directly apply the representations as features to the tasks.
Adapters for Multiple Tasks Adapters were introduced by Houlsby et al. (2119) for parameter efficient training of transformers (Vaswani et al., (017). A small number trainable parameters are added to each layer, while freezing the base encoder. This strategy is similar that of ELMo (Peters) let al. 2018), which learned task-specific weightings for the SiL.TMM layers. To apply adapters to multi-task learning, Pfeiffer et (202I) define two-step process they call Fusion. First, individual adapter modules are trained for every task. The second step introduces task-specific fusion modules at each layer which attend to (i.e. fuse) all the previously pre-fine-tunes adapters, keeping them fixed. Similarly, (Stickland & Murray (2019) introduced Projected Attention Layers (PALs) with adapters and self-attention modules for every task, but the entire network is trained simultaneously.
Control Codes Control codes can be defined as token(s) pre-pended to the input to serve as addi- tional signals to the model. Keskar et al. (2019) use control codes as prompts to govern style, con- tent, and tasc-specific behavior for conditional text generation. [Ta et al. (2022) use control codes to switch between three de-noising modes during pre-training, and associate downstream task with particular mode during fine-tuning. [Zhang et al. (2022) apply control codes in the context of dense retrieval to produce multiple representations covering different aspects of the same document, allow- ing them to match queries written from multiple perspectives. In contrast to this past work, we use control codes indicate target task format for the embedding output by the model, and demonstrate how this is effective for producing paper embeddings across different formats.
3 SCIREPEVAL
multiple task We introduce SciRepEval, a benchmark suite of 25 tasks across four formats for training and eval- uating multi-task embeddings of scholarly papers. SciRepEval aims to enable comprehensive eval- uation of paper embeddings by providing (1) highly diverse set of tasks- formats such classification, regression, proximity and ad-hoc search challenge the general- purpose applicability of embeddings, (2) realistic tasks that reflect practical use cases of paper em- beddings, and (3) standard set of both training and evaluation datasets to simplify comparisons between methods evaluated on the benchmark.
The previous scholarly paper embedding benchmark is SeiDocs (Cohan et al. 2020), which includes two classification tasks, four nearest neighbors tasks, and one recommendation task. SciRepEval includes SeiDoes subset, but addresses several key limitations:
(i) The four nearest neighbor tasks in SciDocs are constructed to distinguish related document from randomly selected negatives given a query document, which might be too easy and not representative of real tasks in scholarly information retrieval. SciRepEval has more realistic tasks such as search, author disambiguation, and paper-reviewer matching among others.
(ii) For the methods evaluated in Section 5 we found that the SciDoes recommendations task was noisy and had limited power to distinguish between different embeddings. The test set includes only 1000 clickthrough events, and the use of propensity weighting means that an even smaller number of examples dominate test set performance. While SciRepEval includes SciDoes subset, we exclude the recommendations task.
(iii) The tasks in SciDocs were constructed to be used only for evaluation, and have few-enough samples that training on SciDoes is impractical (see Table 1). In SciRepEval, eight of the largest tasks across the four formats are used for training, while the rest out-of-train tasks are reserved for evaluation. This enables the study of multi-task approaches, rather than relying
3

---

Under review as a conference paper at ICLR 2023
Table 1: Summary of the SvilepEval benchmark tasks across the four formats classification (CLF), regression (RGN), proximity (PRX) and adhoc search (SRCH). The models in (ecciinn 6 are first trained on the in-train tasks and then benchmarked on their held-out sets as well as the 17 test tasks. Information retrieval tasks have Q queries with P candidate pairs and the S2AND task has X clusters with Y author-paper pairs. S: Silver, G: Gold. SciDocs is evaluated as per Cohan et al. (2020).
Tink Format
Name
Train Dev
Test
Eval Metric
Source
In-Trale
CLF
McSII Desuriptors
Fields of study (FrS)
2.328. 179
676,52488
471
258, 687
Macro Fl Macro Fl
This work
This work
RGN
Citation count Year of Publication
202,774
218,864
30,058
30,000
Kendall's x T
Kendall's T
This work
This work
Some Author Detectiin
Q: 76,489 P: 673, 170
Q: 13,585 P:: 123.430
MAP
Suhmmanion et al
2021
PRX
Highly Inffucntial Citations
Q: 65,982 P: 2,004,668 ,888
Q: 1,199 Ps 54,255
MAP
This wark
Chution Prediction Triplets
819,836
*nol used eor
Colun 2020
SACH
Search
Q: 723,343 1,253,300
2,585
25,850
BDGC
This work
Out-af-Traie
CLF
Hlominicry
11,057
Binnryy Fl
Shvam 2019
DRSM
7.520 S: 955
Macro
RGIN
Pozr Review Soore
I-lddss
10,210
Kendall's T Kendull's
This work
work
Tweet Mentions
25,655
Kendall's x T
& Singh
PRX
SZAND
X: 68.968 V: 10,942
B F1
Suhmmanina et e
221
Mulnne MN
2007
Puper-Roviewer Marching
1:107 1,779
PO.5, P 10
7
Feods-1 Feods-M M
Q: 425 Pr: 4,222 Q: 9025 Ph: 87,528
MAP
This work
MAP
This work
SRCH
Feods Tille TREC-CoVID
Q: 50 P: 69,318
Q: 424 Ph 4,233
MAP #DOG
Noothets
This work
202
SeiDAess
CLF
MAG McSII Discases
23,5:00
Macro Fl
Cohun et al. 2020
25,003
Macro Fl
0020
Co-vgew
Q: 1,000 Ph: 29.978
MAP, aDCG
Coban et al. 2020
PRX
Co-rcad
1,000 29,977
aDCG
1,000 29.928
aDCG
Co-cite
Q: 1,000 Ph: 29.949
MAP, aDCG
Coban 21 al. 40.00
solely on the citation signal. The training data in SciRepEval also has large scale representa- tion in multiple domains as discussed in |Apppedix D
(iv) Four of the tasks in SciDocs have very high porfo-pperfommacce correlations between them (greater than 0.99), indicating that the diversity of the tasks is limited. See Appendix F for more details.
The tasks Sval are summarized in Table I
They are a mixture of existing and new datasets. Datasets with at least 100,000 instances (triplets for proximity/ad-hoc search) are in-train datasets used for training while others are out-of-train used only for evaluation. Although SciDocs tasks are used as out-of-training evaluation tasks, we report their performance in separate category.
Next, we briefly describe each of the task formats and their component tasks. Full details are pro- vided in Appendix A Except for Search, all the tasks use paper embeddings created from combi- nation of paper title and abstract as the input. Search requires additional metadata (subsection 4.1l which is concatenated to the title and abstract before producing the paper representation.
Ad-Hoe Search In ad-hoc search tasks, we are given textual query and the task rank a set of candidate papers by relatedness to the query. Ad-hoc search is a critical mechanism for paper discovery in practice, and we gather multiple real-worte world data sets for training and evaluation. One evaluation dataset comes from previous work, TREC-CoVID (Voorhees et al. (021). biomedical challenge task that ranks papers from CORD-19 (Wang et al., (020b) in response to textual search queries. Two other datasets are newly introduced in our work: 'feeds" dataset taken from a schol- arly paper recommendation system, where we treat the user-specified feed name as the topic query, and the goal is to rank the papers the user has annotated as relevant to the feed above those annotated as irrelevant. Finally, for training, we release a new large data set of more than 700,000 clickthrough events from scholarly search engine which we term as the Search task.
4

---

Under review as a conference paper at ICLR 2023
To evaluate embedding set ad-hoc search, rank candidate papers by increasing Euclidean distance between the query embedding and the candidate paper embeddings. Pytrec_ed _evlll Van Gysel & de Rijke 2018) is used to calculate the ranking metrics. Normalized Discounted Cumulative Gain (nDCG) is used for Search and TREC-COVID tasks as the true relevance score can be 1. For the feeds tasks which have binary labels, use Mean Average Precision (MAP).
Proximity Similar to ad-hoc search, proximity tasks involve ranking a set of candidate papers by their relatedness to a query, except the query in this case is not textual but instead paper. Proximity- based tasks form a basis for paper-based retrieval and recommendation, and for estimating paper similarity for use in applications like author disambiguation. We include total of eleven proximity- based tasks, including four evaluation tasks from SciDoes (predicting citations and co-citations, and predicting co-viewed co-read papers), and two others from previous work: the S2AND author disambiguation task (Subramanian et al.) (021) I) with paper similarity features, and Paper-Reviewer Matching, where candidate reviewers are ranked by expert annotators based on the similarity of their papers to the query paper to be reviewed. The Paper- Reviewer Matching task combines three
existing datasets (Mimno & McCallum 2007; [iu al., 2014 Zhao et al. (2222) which we describe in more detail in subsection A.2 We also introduce five new proximity tasks including two feeds evaluation tasks from the recommender discussed above, where one or multiple relevant papers serve as queries. For training, we include three large-scale datasets aimed at predicting same-authors, citations (via triplets) (Cohan et al. 2020), and influential citations, which we define as four or more citations of the same paper in the text of single paper.
For evaluating embeddings in proximity tasks, we rank candidates by Euclidean embedding distance, with MAP as the evaluation metric except for S2AND, which uses B F1 (Bagga & Baldwin 1998) and Peer Review Matching, which uses precision @5 and @ 10.
Classification Paper classification, in which the input is paper and the output is a topical cate- gory, is a foundational task for document organization and discovery. Apart from the two SciDocs classification tasks (MAG and MeSH Diseases), we take four additional classification tasks, includ- ing binary task to predict whether a paper is relevant to biomimiery (Shyam et al., 2019), two biomedical classification tasks, namely DRSM from Burns cation (Lipscomb) 2000), and a new large-scale field of study (FoS) multi-label training set of more than 500K papers with silver FoS labels based on publication venue.
(2022) and McSH Descriptors classifi-
We evaluate embeddings on classification by scoring their performance as features within linear support vector classifiers. Results for these tasks are evaluated using Fl score (which may be micro- or macro-F1 depending on the dataset, indicated in Table T). To better understand how embeddings perform in data-scarce regimes, we also construct two few-shot versions each from both aut-of-train classification datasets and the FoS dataset subset for which we have manually annotated gold labels.
Regression We also consider a set of regression tasks where the goal is to predict continuous quantity for given paper. For evaluation, we consider predicting three numeric attributes related to prominence or quality: Tweet Mentions (Jain & Singh 2021). and the peer review rating and maximum h-index of authors for a collection of ICLR papers obtained from OpenReview (fommnn two new datasets). For training, introduce two additional datasets of more than 200K examples each, predicting citation count and year of publication.
We evaluate embeddings regression tasks by scoring their performance when used as features within linear support vector regression models. Results for these tasks are evaluated using the Kendall's rank correlation between the true and predicted labels ?
MULTI-FORMAT REPRESENTATION LEARNING
Typical approaches for learning document embeddings produce a single embedding for every task (Cohan et al.] 2020 Ostendorff et al.] 2022b). We hypothesize that a single embedding will be in- sufficient for generalizing across diversity of downstream tasks when the embeddings are used as features in lightweight classifiers. At the other extreme, learning embeddings for each task sepa-
httts://api.opeenevvieee
We found in our experiments that Pearson's p and Kendall's produced similar relative results between models. We did not use MSE because its values are unbounded and could skew the overall average across the datascts in the benchmark.
5

---

Under review as a conference paper at ICLR 2023
Control Codes
Adapters
[CLF]
[QRY]
PRX
QRY
doc
Language Model
[RGN]
CLF
RGN
[PRX]
PRX QRY
CLF
RGN
Classification
Proximity
Regression
Figure Generating multi-format embeddings. task format is either associated with a task- specific control code supplied with input document, or adapter blocks attached to the model.
rately does not allow generalization with the number of tasks. We propose method for learning a distinct document embedding for each task format, using multi-task learning framework.
tasks and also incurs significant storage costs scaling
We assume we are given labeled data from a set of tasks for our four formats (ad-hoc search, proxim- ity, classification, and regression), and learn models capable of producing embedding for any given (paper, format) pair. Here, papers are represented in terms of their title and abstract. Our goal for these embeddings be used in lightweight classifietsrrggresooss as well as in nearest neighbor tasks, which we evaluate both on held-out data from the training tasks, and on new held-out tasks.
To help build intuition for why different embedding sets for different task formats may be helpful, Figure/ T illlustrates the qualitative distinctions between the task formats. In general, an embedding space performing well for one task format may be less suited to the others; for example, the clas- sification space provides error-free linear classifier, but its nearest neighbor pairs are not always of the same class. Empirically, we find that learning specialized embeddings per format improves performance, and that embeddings trained on format tend to perform better held-out tasks with the same format (sec Table 4). Further, partitioning randomly (as discussed in Section 7 was less ef- fective than the format-basee partitioning. Nonetheless, format-based partitioning is just one choice of many and experimenting with other partitioning schemes is an important item of future work.
4.1 MODEL
We follow Cohan et al. (2020) in using a pretrained transformer encoder as our base model. A sci- entific document is given input the encoder a concatenation of its title and abstract separated by the [SEP] token Unlike Cohan et al. (2020), we use three different types of training objectives suitable for each format to train the model as described in in/subsection 4.2) We explore two methods to learn separate embeddings for each task form: control codes and adapters as shown in Figure
Control Codes In the control code approach, prepend special per- per-format token (see Table in the appendix) to the input and pass it to the transformer model, taking the final layer representation corresponding to this token as the document embedding and feeding it input the task-specific head (described in Section 4.2).
We also experiment with adapters which have been shown to be effective for multi-task learning. In particular, we explore Adapter Fusion (Pfiiffer et al., 202T and PALs (Stickland & Murray 2019) methods, each of which introduces task-specific adapters and attention modules at every transformer layer. Since our goal is to learn different embeddings for different task formats,
Adapters
For the Search task, additional metadata like paper venue and year of publishing is also supplied.

---

Under review as a conference paper at ICLR 2023
create modules for each task format rather than each task, and the final hidden representation of the [CLS] token output via the adapter is taken as the corresponding embedding of the document.
4.2 TRAINING
We train the model in a multi-task setup with task-heteroggeneous batching (Aghajanyan et al. 202 For classification and regression tasks, we use a linear head atop the base transformer encode We train on both multi-class and multi-label tasks, using Cross Entropy loss for the former and Binary Cross Entropy (BCE) with sigmoid activation for the latter. For regression we minimize the Mean Square Error (MSE) loss.
For proximity and ad-hoc search tasks we use the triplet loss as in Cohan et al. (2020). For these task forms, given query, relevance score accompanies each candidate. The query can be a document (for which we wish to find similar documents) or a raw textual query. Each training instance in this setup is a triplet consisting of a paper or plain text query 2, a positive candidate paper P+ and negative candidate P-, where P+ has a higher score than P-. Then, we optimize the triplet loss:
Ltriplet max [d( @z. PB)
dl Qr. PE) + c.
where d is the Euclidean distance used as a measure of similarity between the query embedding QE and candidate embeddings P/ and Pg. and is the margin hyperparameter whose value chosen as based on preliminary experiments.
5 EXPERIMENT SETUP
Training Data We train our multi-format models on the 8 large scale in-train tasks detailed in Table I For the proximity and ad-hoc search tasks, we create up 5 examples for each query by sampling positive and negative papers from its candidate pool. We limit the number of training samples from each task at most 600K The The resultant training and validation data sets consist of total of 3.27M and 446K instances respectively.
Transformer Baselines As first step, we evaluate the existing document representation meth- ods on our benchmark. These include the transformer encoders SciBERT (Beltagy et al. 2019) language model pre-trained on scientific corpora: and SPECTER (Cohan et al. 2020), SciNCL (sstenorrff et al. (222b)) and ASPIRE (Mysore et al.) 2022). ASPIRÉ produces representations for aspect-based matching between query and candidate papers which is similar setting as our prox- imity tasks. Hence we only evaluate it on that specific subset and report the results in Appendix C Next, for our multi-format baselines, we initialize with SciNCL which is the state of the art on Sci- Docs, and then further train it in multi-task setup the in-train tasks both with (MTL CTRL) and without the control codes (MTL CLS). Finally, to compare the control codes-based approach with the adapter techniques, we experiment with the BERT PALS and Fusion architectures, keeping SciNCL as the base model in both. Fusion being a two step process, first introduces task format specific adapters (Adapters) and then the fusion modules (Adapter Fusion). The MTL CTRL and adapter approaches produce multiple representations per document while MTL CLS produces a sin- gle representation similar existing methods. We use the PyTorch implementations of the models by HuggingFaco" The specific training configurations are described in Appendix B
6 RESULTS
Table shows the evaluation of all our transformer baselines producing both single and multiple representations per document on SeiRepEval. Our benchmark includes diverse tasks with a variety of different evaluation metrics, and following previous work (e.g.. WWan et al., 2019) we report an average of the individual task metrics (which each range from 0- 100). The rrefiinetuned multi- format variants outperform the vanilla models on average, and we also find that all the approaches that produce multiple representation types outperform, by up to 1.5 points, the MTL CLS model, which learns only a single representation shared for all tasks. The adapter variants are better than MTL CTRL overall, and result in an improvement of 0.6-1.3 points on the out-of-train tasks with task-format specific adapters performing the best.
"The linear heads thrown away after training. *Performance with smaller dataset samples- max 400K samples/tasks was relatively poor. "hurs://uuginngfcceccm co/models ww/mme/e/aa.ss/eesesss.....
7

---

Under review as a conference paper at ICLR 2023
Table Evaluation results on SciRepEval in multiple settings. MTL CLS generates a single embed- ding for all tasks, MTL CTRL (control codes) and Adapter variants (Adapters, PALs, and Adapter Fusion) produce an embedding per task format. We consider an ensemble approach that averages the MTL CTRL and Adapter embeddings. For models we trained, we report the mean and standard deviation (in parentheses) across runs with different seeds. The best results are highlighted bold. We conduct one way analysis of variance (ANOVA) with Tukey's 's test (Haynes 2013) for 0.05 across multiple settings and underline those not statistically significantly different from the best.
Model
In-Train
Out-of-Train
SciDocs
Average
SciBERT
$1.5
52.5
69.0
58.2
SPECTER
54.7
57.4
89.1
68.0
SciNCL
55.6
57.8
90.8
69.0
SciNCL MTL CLS
60.0 (0.11)
57.0 (028)
89.5 (0.08)
69.4 (0.12)
SciNCL + MTL CTRL
61.9 (0.22)
57.8 (0.4)
89.7 (0.16)
70.2 (0.23)
SciNCI. + Adapters
61.9 (0.11)
59.1 (006)
90.3 (0.07)
70.9 (0.03)
SciNCL + PALs
61.9 (0.26)
58.4 (0.28)
90.1 (0.09)
70.6 (0.11)
SciNCL Adapter Fusion
62.0 (0.04)
58.8 (0:26)
90.1 (0.11)
70.8 (0.14)
SciNCL + Adapters + MTL CTRL
62.5 (0.01)
59.1 (0.15)
90.6 (0.01)
71.2 (0.03)
Table 3: Results for multi-format training with SciBERT and SPECTER as base models. For brevity, we report only the single adapters results due to their additional advantage of computation efficiency. The best results for each base model are underlined.
Model
In-Train
Out-of-
SciDocs
Average
SciBERT MTL CLS
59.4
57.1
88.9
69.0
SciBERT MTL CTRL
62.5
57.8
89.6
70.4
SciBERT Adapters
62.6
58.0
90.1
70.6
SciBERT Adapters + MTL CTRL
62.9
58.6
90.4
71.1
SPECTER + MTL CLS
60.0
56.5
89.0
69.0
SPECTER + MTL CTRL
61.9
58.0
89.4
70.2
SPECTER + Adapters
61.5
58.9
89.6
70.5
SPECTER + Adapters + MTL CTRL.
62.2
59.1
89.9
70.9
the control codes and adapters are the most efficient in terms of model size and computational efficiency. Hence, we try to improve upon each by combining representations from the Adapter model and the MTL. CTRL model by averaging them and we find that these combined embeddings outperform the individual ones consistently across the in-train, out-of-train, and SciDoes settings. All the models except SciBERT (not pre-trained with citation objective) perform well on SciDocs, with vanilla SciNCL being the best. ASPIRE, as reported in Appendix performs well on SciDocs but not on other similar tasks in SciRepEval.
Further, as shown in Table 5
Alternative Base Models To confirm that our findings hold across multiple base models, we com- pare MTL CLS, MTL CTRL and adapters with SPECTER and SciBERT as the base models. Taall 3) shows that the MTL CTRL token and the adapters approaches still substantially outperform the MTL CLS approach, suggesting that the efficacy of using an embedding task format instead of single embedding per document holds across range of base model types.
ANALYSES
Specialization of Control Code Embeddings Our hypothesis is that by training embedding spaces on particular task formats, they will become more accurate for tasks of that format than for others. We test this hypothesis by sampling in-train and one out-of-train task of every for- mat (for ease of computation) and applying all the control codes to them for evaluation. As shown
We also tried concatenating the embeddings in preliminary experiments, which yielded similar results but doubled the embedding size.
In-train: FoS, Citation Count, Same Author Detection, Search; Out-of-train: DRSM, Peer Review Score, Revr-Reviewer Matching, TREC-CoVID
8

---

Under review as a conference paper at ICLR 2023
Table 4: Cross task analysis for control codes. The best results for each task format across all control codes is underlined. These are represented in the diagonal for both in-train and out-of-train tasks suggesting that format based partitioning multi-task training produces effective document representations suitable for the corresponding format.
Control Code Embeddings Used
Task format
Ouaio
In-Train
CLF
RGN
PRX
QRY
CLF
RGN
PRX
QRY
Classification
66.9
63.5
65.0
64.5
38.7
32.9
31.9
31.0
Regression
16.2
20.4
18.1
17.5
29.7
46.8
43.7
42.9
Proximity
43.8
43.1
44.7
44.7
87.1
82.2
89.0
88.3
Ad-hoc search
86.7
85.2
85.3
90.5
73.9
75.6
77.5
78.5
Table 5: Parameter and (relative) runtime efficiency comparison of models. MTL CTRL and Adapters similar in runtime, but the PALs and Fusion variants add significant computation costs.
Parameters per
Training Time
Inference Time
Model
Task Form
MTL CTRL
768
1x
1x
PALS
2M
1.42x
1.29x
Adapters
1M
0.96x
1.05x
Adapter Fusion
22M
1.32x
1.69x
in Table 4, the control codes trained on task format perform best for tasks of that format, for both in-train and out-of-train
As an extension to this experiment we also analyze how well the control code representations work when the encoder is trained tasks which randomly grouped together opposed by task format. We take the mean evaluation metrics produced from 5 random partition runs. On the out- of-train tasks, the corresponding control codes for classification, regression, proximity and ad-hoc search show gain of +0.2, +3.9, +4.5 and +2.2 points respectively over random partitioning. Simi- larly, for in-train tasks the control codes are better by +5.2, +3.8, +1.2 and +1.3 points respectively. The results suggest that representations specific to each task format do lead to better results overall.
Finally, study training affinity among the task formats, we pre-fine-tune on a maximum of two formats at once. Appendix G reveals that combined multi-task training on similar task formats like regression/classffcatoon and proximityladhoc--e related tasks. Training on all the tasks yields better results average across the task formats.
results in performance gains, but only on
Efficiency While the variants producing representations based on task-format serve as strong base- lines on the SciRepEval benchmark shown in Table 2] efficiency is another important consider- ation in practice. As shown in Table 5] the control code approach only requires one new control code embedding per format, and does not affect training time. PALs, by contrast, introduces new attention layers and trains the entire network, increasing training time, and Adapters adds and only trains half as many parameters as PALs. Fusion layers introduce 10x as many parameters as PALS leading to 2x more time on inference. Training and inference times are measured on runs with Ik and 10k samples, respectively.
8 CONCLUSION
We introduce SciRepEval, benchmark for scientific document representation methods with 25 tasks across four task formats. On this benchmark, we show that learning a separate document representation for each task format substantially improves task performance compared to learning single representation for all tasks. Future work could address limitations of our work by evaluating partitioning schemes beyond task format, crafting figher-iielity metrics to account for the diversity of tasks in SciRepEval (which vary in sensitivity and in relevance to downstream applications), or further exploring how accuracy varies with computational and storage cost.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. Muppet: Massive multi-task representations with pre-finetuning. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pp. 5799-5811, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguis- tics. doi: 1t118/65/11122111mmlllloool
UtL http///aaaatttttc www.llnnhhhloyy... org/2021.
emn p- i...ain44.. 468
Vamsi Aribandi, Yi Tay. Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang. Vinh Q. Tran, Dara Bahri, Jianmo Ni, Jai Gupta, Kai Hui, Sehastian Ruder, and Donald Metzler. Ext5: Towards extreme multi-task scaling for transfer learning. In International Conference on Learning Representations, 2022. URLhttps: BRLLhttps:///peneeee http///ppppeeeieeeeeeeeeeeeeeeem net forum?
id LddVzhhBBg
Amit Bagga and Breck Baldwin. Algorithms for scoring coreference chains. In The first inter- national conference on language resources and evaluation workshop on linguistics coreference. 1998.
Iz Beltagy, Kyle Lo,, and Arman Cohan. SciBERT: pretrained language model for scientific text. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3615-3620, Hong Kong, China, November 2019, Association for Computational Linguistics. doi: 10//53//1/11111300 URL http:p://aeeeaantoooorooooooo
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 1877-1901, Curran Asso- httppps.cccppppee http:/pss..ccpppee//////////
ciates, Inc., 2020. URL /
Gully Burns.
Drsm-corpus vl, 2022. URL
DRSM- corpus
Rich Caruana. Multitask learning: A knowledge-hased source of inductive bias. In Proceedings of the Tenth International Conference on International Conference on Machine Learning, ICMI'99, pp. 41-48, San Francisco, CA, USA, 1993. Morgan Kaufmann Publishers Inc ISBN 1558603077.
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. GradNorm: Gradient normalization for adaptive loss balancing in deep multitask networks. Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 794-803. PMLR, 10-15 Jul 2018. URL https::/proceet
/acess//v////// mlr presss/ v80/ chenl8a.htn.
Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel Weld. SPECTER:
Document-level representation learning using citation-informed transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2270-2282, On- line, July 2020, Association for Computational Linguistics. doi: 10. 10.16653 acl-man..07. URL httpp:://accaathhhaeeeeee
org. wtwpg///000000 acl.main..22mm. 207
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. Supervised learning of universal sentence representations from natural language inference data. In Proceed- ings of the 2017 Conference Empirical Methods in Natural Language Processing, pp. 670- 680, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. 10.18653/1/DD1-10770 URI WtRlhhtts::/aar antho ogy. ory.org/D17-1100
10

---

Under review as a conference paper at ICLR 2023
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies, Volume oong and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 18653/11/N199144/ URL https: /ac.aannolooy..rrgg//aoooo.o
Christopher Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn, Efficiently identifying task groupings for multi-task learning. In NeurIPS, 2021.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sen- tence embeddings. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing. Online and Punta Cana, Dominican Republic, November 2021. Associ- ation for Computational Linguistics. doi: (0.186633v1/2021. emnlp- http:///2222...mmmmma...5555.52
URL https:
//aclanthol org. emn 1p..pai..52222
Winston Haynes. Tukey's Test, pp. 2303-2304. Springer New York, New York, NY, 2013. ISBN 978-1-4419-9863-7, doi: 10. 11-111/097781--11-988-5--
URL /ttpps::s////
1212
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spacy: Industrial- strength natural language processing in python. 2020. doi: 10.5281/zenodo. 121oodd..1113000 1212333.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski. Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th In- ternational Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2790-2799. PMLR, 09-15 Jun 2019. URL https: /ttpss///prcceeeeee ngs mml. press/
htt//v9//hou press/v97/how aby1 1 9a. html
Ganesh J, Manish Gupta, and Vasudeva Varma. Doc2sent2vec: A novel two-phase approach for learning document representation. In Pruceedings of the 39th International ACM SIGIR Con- ference on Research and Development in Information Retrieval, SIGIR '16, pp. 809-81 812, New York, NY, USA. 2016. Association for Computing Machinery. ISBN 978 9781450340694. N0.1145/2911451.1 14717. URL httpl:tttt://tttt www.:oggg1000. 111.11..111222880 hatppp////aal.oooo0..111///////1 1tt..//01...1//11115555531111311
doi: 221141199777
Naman Jain and Mayank Singh. Tweetpap: A dataset to study the social media discourse of scientific papers. In 2021 ACM/IEEE Joint Conference on Digital Libraries (JCDL), pp. 328-329, 2021.
doi: 10. N0.11099/CDLS25003 200...22202000055
Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. ArXiv, hass/990,05858, 2019.
Quoc Le and Tomas Mikolov. Distributed representations of sentences and documents. In Proceed- ings of the 31st International Conference on International Conference on Machine Learning Volume 32, ICML'14, pp. 11-1188-11-1196. .orgRorgg 2014.
Carolyn E Lipscomb. Medical subject headings (mesh). Bulletin of the Medical Library Association, 88(3):265, 2000.
Chundi Liu, Shunan Zhao, and Maksims Volkovs. Unsupervised document embedding with cnns. ArXiv, abs/1711 abs/1771.00416,, .4160, 2017.
Shengchao Liu, Yingyu Liang, and Anthony Gitter. Loss balanced task weighting to reduce negative transfer in multi-task learning. Proceedingy of the AAAI Conference on Artificial Intelligence, 33 (01):9977-9978, Jul. 201 9a. doi: 10. arg/index.ppp/ AAAT/ LatAAA//ariii
3330193379 URI. httpt:ttts///sss.. aaa.......aa.................... 5125
Xiang Liu, Torsten Suel, and Nasir Memon. A robust model for paper reviewer assignment. In the Sth ACM Conference on Recommender Systems, RecSys '4, pp. 25-32, New York, NY. USA, 2014. Association for Computing Machinery. ISBN 978 1.8346636 doi: 10. http:///S1122444444AAAAAASAS//// 1145/2645710.2645744 URL Com turing. htw.aa......aaa........eeeeeeee northwester..
Proceedings of the
cdl./10.1144//245710000

---

Under review as a conference paper at ICLR 2023
Xiaodong Liu, Jianfeng Gao, Xiaodong He, Li Deng, Kevin Duh, and Ye-yi Wang. Representation learning using multi-task deep neural networks for semantic classification and information re- trieval. In Proceedings of the 2015 Conference the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 912-921, Denver, Colorado, May-June 2015. Association for Computational Linguistics. doi: 10.3115/v 1/922- URL wttp://o.....g/111111000100 http://ca..aaacolooooooro///-/-.
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. Proceedings of the 57th Annual Meeting the Association for Computational Linguisties, pp. 4487-4496, Florence, Italy, July 2019b. Association for Com- putational Linguistics. doi: 10.18653/v1/P19-11 URL https: http:://aaaaatttto orgg
P19- 1441
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations (ICLR), 2019.
David Mimno and Andrew McCallum. Expertise modeling for matching papers with reviewers. In Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD '07. pp. 500-509, New York, NY, USA, 2007. Association for Com- puting Machinery. ISBN 9781595936097. doi: 10. 10.1145/1281 192. 192. URL https:// doi moii-org.om turing. library........ nor wwwtwwwstert..tommmm edu / 0tte://...1111111111 11 1281247
Sheshera Mysore, Tim O' O'Gommnn, Andrew McCallum, and Hamed Zamani. Csfcube-a test collec- tion of computer science research articles for faceted query by example. In Thirty-fifth Conference on Neural Information Processing Systems (NeurIPS), 2021.
Sheshera Mysore, Arman Cohan, and Tom Hope. Multi-vector models with textual guidance for fine-grained scientific document similarity. In Proceedings of the 2022 Conference of
the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Seattle, United States, July 2022. Association for Computational Linguis- tics. doi: 10. (00.1883//11/022.0o www.nnaaalmammaa333 URL httt:p/://cclatteooooo.oooo/////
naacl-ma in..a.n... 331
Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas A. Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger, David P. Schnurr, Fe- lipe Petroski Such, Kenny Sai-Kin Hsu, Madeleine Thompson, Tabarak Khan, Toki Sherbakov, Joanne Jang, Peter Welinder, and Lilian Weng. Text and code embeddings by contrastive pre- training. ArXiv, abs/2201. 10005, 2022.
Malte Ostendorff, Till Blume, Terry Ruas, Bela Gipp, and Georg Rehm. Specialized doc- ument embeddings for aspect-based similarity of research papers. In Proceedings of the 22nd ACM/IEEE Joint Conference on Digital Libraries, JCDL '22, New York, NY, USA, 2022a. Association for Computing Machinery. ISBN 9781450393454. doi: 10.1145/3529372. 3530912. URL https://doi-org turing. hww..aa..rrrr..oottteeteee.e. edu/10. 1145/3529372 3530912
Malte Ostendorff, Nils Rethmeier, Isabelle Augenstein, Bela Gipp, and Georg Rehm. Neighborhood Contrastive Learning for Scientifie Document Representations with Citation Embeddings. In The 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP 2022), Abu Dhabi, December 2022b. Association for Computational Linguisties. doi: 10.48550/arxiw.2222. 06671. 7-11 December 2022. Accepted for publication.
Vishakh Padmakumar, Leonard Lausen, Miguel Ballesteros, Sheng Zha He He, and George Karypis. Exploring the role of task transferability in large-scale multi-tasi learning. In Pro- ceedings of the 2022 Conference of the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technologies, pp. 2542-2550, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v 1/0.3//////2222aaaaa maacc-main.nieeemm 183, URL
wrt/2o222222222 2 naacl ma intainn.. 183
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018
12

---

Under review as a conference paper at ICLR 2023
Conference the North American Chapter the Association for Computational Linguisties: Hu- man Language Technologies, Volume (Long Papers). pp. 2227-222 2277, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/11/N188 1202. URL http//aalaanttoogoggooooooo
1200
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapter-
Fusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Con- ference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 487-503, Online, April 2021. Association for Computational Linguistics. doi: 10.18653/v1/
2021 1221.ecc-mmin.39 URL _ https::////.... //ac wawehoanthhoooooooo logy. .rg./...r//o.. 2021 we0.1.eaalll... -main.nnn..mm
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to- text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL http:
7/jml wttp://///pppps////20000-00
html
Nils Reimers and Iryna Gurevych. Centence-BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pp. 3982-3992, Hong Kong, China, November 2019. Association for Com- putational Linguistics. doi: 10188661/111114444 URL https: http://////aaa/thhoooooooooo ogy.. org//
D19-1410
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, al. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2021.
Vikram Shyam, Lauren Friend, Brian Whiteaker, Nicholas Bense, Jonathan Dowdall, Bishoy Boktor, Manju Johny, Isaias Reyes, Angeera Naser, Nikhitha Sakhamuri, Victoria Kravets, Alexandra Calvin, Kaylee Gabus, Delonte Goodman, Herbert Schilling, Calvin Robinson, Robert Omar Reid II, and Colleen Unsworth. Petal (periodic table of life) and physiomimetics. Designs, (3), 2019. doi: 10.3390/dessggns333004.. 3/43
www http//wwwwppppp.
Asa Cooper Stickland and lain Murray. BERT and PALs: Projected attention layers for ef- ficient adaptation in multi-task learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learming Research, pp. 5986-5995. PMLR, 09-15 Jun 2019. URL https::/proceecingg wwt.smml..ppr... /vt.sss/v///// wtteeessvv///siccaanddeee /ttssstickkaanddna 9a. html
Shivashankar Subramanian, Daniel King, Doug Downey, and Sergey Feldman. S2AND: A Bench- mark and Evaluation System for Author Name Disambiguation. In JCDL '21: Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2021, JCDL '21, New York, NY, USA, 2021. Association for Computing Machinery.
Yi Tay, Mostafa Dehghani, Vinh Quang Tran, Xavier Garefa, Dara Bahri, Tal Schuster, Huaixiu Zheng, Neil Houlsby, and Donald Metzler. Unifying language learning paradigms. ArXiv, abs/2205.05131, 2022.
Amalic Trewartha, Nicholas Walker, Haoyan Huo, Sanghoon Lee, Kevin Cruse, John Dagdelen, Alexander Dunn, Kristin A. Persson, Gerbrand Ceder, and Anubhav Jain. Quantifying the ad- vantage of domain-specific pre-training on named entity recognition tasks in materials science. Patterns, 304)8100488, 2022. doi: http:://oo.oogg//10..00.//paeppp 100488 URL https:
7/
rect cww http:///c..eeeeerttaaeaaa///////
Marco Valenzuela, Vu A. Ha, and Oren Etzioni. Identifying meaningful citations. In AAAI Work- shop: Scholarly Big Data, 2015.
Christophe Van Gysel and Maarten de Rijke. Pytrec_eval: An extremely fast python interface to trec trec_eval. In The 41st International ACM SIGIR Conference on Research and Development in In- formation Retrieval, SIGIR 18. pp. 873-876, New York, NY. USA, 2018. Association for Com- puting Machinery. ISBN 9781450356572. doi: 10.1145/320977882210065 URL https://
doi-org. hrgpa///g...o.ooo..cooooooonnnnn
13

---

Under review as a conference paper at ICLR 2023
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st In- ternational Conference on Neural Information Processing Systems, NeurlPS' 17, pp. 6000-6010, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860989
Ellen Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fuusman, William R. Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. Trec-covid: Constructing pandemic in- formation retrieval test collection. SIGIR Forum, 54(1), feb 2021. ISSN 0163-5840. doi: 10.
1145/345196443451966 http::tts:///// edd.10.1114/345196644 3451965
org. turing. library w.b.aay.norttwwssterr. nowt.westtt.......
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language un- derstanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. URL wtRlhtttss//ppoceddiaee nww.:eewwd.............ee..ee... http://wwpp.cccpppp////////////. 4496bf24afe
pdf
Kuansan Wang, Iris Shen, Charles Huang, Chieh-Han Wu, Yuxiao Dong, and An- shul Kanakia. Microsoft academic graph: when experts are not enough. Quanti-
tative Science Studies, 1(1):396 February 2020a. doi:
000010
URL
mà www.ooootcccatooooo grp. are
Lucy Lu Wang, Kyle Lo, Yogamand Chandrasekhar, Russell Reas, Jiangjiang Yang. Doug Burdick, Darrin Eide, Kathryn Funk, Yannis Katsis, Rodney Michael Kinney, Yunyao Li, Ziyang Liu, William Merrill, Paul Mooney, Dewey A. Murdick, Devvret Rishi, Jerry Sheehan, Zhihong Shen, Brandon Stilson, Alex D. Wade, Kuansan Wang, Nancy Xin Wang. Christopher Wilhelm, Boya Xie, Douglas M. Raymond, Daniel S. Weld, Oren Etzioni, and Sebastian Kohlmeier. CORD- 19: The COVID-19 open research dataset. In Proceedings of the Ist Workshop on NLP for COVID-19 at ACL 2020, Online, July 2020b. Association for Computational Linguistics. URL bettps://acanttinn
org 2020.
Lingfei Wu, Ian En- Hsu Yen, Kun Xu, Fangli Xu, Avinash Balakrishnan, Pin- Yu Chen, Pradeep Ravikumar, and Michael J. Withrock. Word mover's ' embedding: From Word2Vec to document embedding. Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pp. 4524-4534, Brussels, Belgium, October-November 2018. Association for Com- putational Linguistics. doi: 10. 10.18653/v1/D18 1482. URL https: /ac.:::o///llathhooog.... orgg
D18 1482
Peng Xu, Xinchi Chen, Xiaofei Ma, Zhiheng Huang, and Bing Xiang. Contrastive document rep- resentation learning with graph attention networks. In Findings of the Association for Computa- tional Linguistics: EMNLP 202/, pp. 1874-3884, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/0211 URL
ooy.. org. 2021 findings 2t025///1/221...ndnn--emmmme findinng-emmllp332..
327
Michihiro Yasunaga, Jure Leskovec, and Percy Liang. LinkBERT: Pretraining language mod- els with document links. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8003-8016, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18663/1/22022.acclloogg TEtts::/acclattee
URL.
Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. CrossFit: A few-shot learning challenge for cross-task generalization in NLP. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7163-7189, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 200553//1/222.emmlpppp main.572. URL https://aclantthooogg wwtp:llooooooooooo0/ 20222 emnlp-main. 572
Hamed Zamani, Mostafa Dehghani, W. Bruce Croft, Erik Learned-Min Millerr, and Jaap Kamps. From neural re-ranking neural ranking: Learning sparse representation for inverted indexing. In
14

---

Under review as a conference paper at ICLR 2023
Proceedings of the 27th ACM International Conference Information and Knowledge Manage- ment, CIKM '18, pp. 497-506, New York. NY, USA, 2018. Association for Computing Machin- ery. ISBN 978 9781450360142. doi: 10.1144/3226220632777 www.....21188888 1800. 1145/3269208 3271800
httttps::/doi.ce/
Shunyu Zhang, Yaobo Liang, Ming Gong, Daxin Jiang. and Nan Duan. Multi-view document repre- sentation learning for open-domain dense retrieval. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 5990-6000, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.1 33655/v1/2022.allllog 414. URL ht
org 2002 wtt.:///222...oooooooooooo ac1.222.c1-ooo..44
Yue Zhao, Ajay Anand, and Gaurav Sharma. Reviewer recommendations using document vector embeddings and a publisher database: Implementation and evaluation. IEEE Access, 10:21798- 21811, 2022. doi: 10.1 I0.1009AACCEESS22222156666
A SCIREPEVAL TASKS
A.1 AD-HOC SEARCH
Search We used clickthrough data from an academic search engine, Only search queries with at least 10 results were included, and a of heuristic rules were applied to exclude likely noise and bots. We removed author queries when the query was judged to contain any person tokens by named entity recognition (Honnibal et al. 2020
Feeds Research feeds help researchers maintain a library of papers they are currently reading and also index them by topics. We use anonymized research feeds data from an academic search engine that recommends papers based on user's library. This data includes information on whether users found the recommendations relevant or not. The data contains 430 feeds which have more than five positive and two negative paper annotations from users. We use this to create the Feeds-1, Feeds-M and Feeds Title tasks. The first two are proximity tasks and are described in selsscttio 3]
The title of research feed provided by the user usually indicates the topic of scientific articles in the feed. While the other two datasets have papers as query and belong to the proximity family, this dataset is classified ad-hoe search, as the query is short text snippet rather than paper. We remove feeds with generic titles like 'My field' and 'Final Project'; replace abbreviations with their long forms where possible and filter out feeds with non- non-English titles.
Feeds Title:
TREC-COVID TREC-COVID was introduced by Voorhees et al. (202 as biomedical literature search task relevant COVID-19. The dataset consists of 50 search queries and candidate literature from the CORD-19 corpus (Wang et al. 2020b) along with their relevance scores on a scale of 0-2. Each query consists of short title, a question asking the required information and narrative describes briefly exactly the type of information that the results should have. For our evaluation we combine these fields into a single text separated by the [SEP] token.
A.2 PROXIMITY
S2AND and Same Author Detection The S2AND dataset (Subramanian et 2021 contains signatures (author-paper papper pairs) that are clustered according to which author mentions refer to the same person. Due to the high resource requirements of running the original S2AND evaluation, we create S2AND-mini, version of S2AND with only 1000 blocks from each of S2AND's dataset sources and at most 500 signatures per block. Our evaluation of S2AND-mini follows the original evaluation of S2AND; that is, method's document embeddings are used along with author and paper metadata to create features for clustering algorithm that consists of a pairwise scoring model followed by greedy agglomerative clustering. We use B3 F1 (Bagga & Baldwin 1998) in the original paper for evaluation.
We also S2AND to create the data for authrr detection task. Unlike the original S2AND evaluation, our same-aute author task uses only paper embeddings without any additional author or paper metadata, which allows to directly train the embedding model the data. Sath-author
15

---

Under review as a conference paper at ICLR 2023
detection is formulated as triplet ranking task; given three papers of which two share an author, the goal is to find the matching pair.
Feeds-1 We re-purpose the feeds dataset from Section A. for this and the next task. The first paper added feed chronologically serves the query. The next 5 positive user annotations considered relevant and 5 negative candidates are sampled either from user annotations or randomly.
Feeds-M Given K positive papers annotated in a feed (assuming K > 5), we use the first M K as queries. For every query, the positive candidates are sampled from all the papers the user positively annotated after the query paper was added to their feed, and negative candidates are sampled from user annotations or randomly.
Peer Reviewer Matching In this task the goal is to judge whether given paper is relevant to a potential reviewer. As data for this task is hard to obtain at scale due the double-blind nature of many conferences and journals, we combine multiple existing reviewer-paper matching datasets:
Mimno McCallum (2007). with 393 paper-review relevance ratings from corpus of 148 NeurlPSS 2006 papers and 364 reviewers, annotated by nine human experts.
Liu al. 2014), an extension of Mimno & McCallum (007) which adds 766 additional paper-review annotations.
Zhao et al. 2022), with 694 paper-reviewer relevance ratings from corpus of 75 papers and 1833 reviewers from the IEEE ICIP 2016 conference, annotated by human experts.
All datasets have been annotated the same 0-3 relevance rating scale. The candidate reviewers all researchers, and we embed all the papers written by them using our models. To obtain the model's score for each candidate reviewer, we compute the cosine similarity between the query paper and each of the candidate's s papers, and take the mean of the top 3 similarities as the score. We consider two ways map the 0-3 relevance judgements binary labels-hard and soft decision-wwere for the soft decision score of or is considered relevant and for hard decision only a score of is considered relevant. Precision at 5 () and 10 (P@ 10) results are used as the final metric, which ultimately results in four numbers (P@5 and P@ 10 for each of hard and soft decisions). which are averaged to produce the single number reported in our final results for this task.
Highly Influential Citations this task, given a paper A and paper B, we aim to predict whether B is highly influenced by A. As measuring influence is subjective and human annotation is expen- sive, we approximate influence by counting the number times A is cited in the text of B. If A is cited at least 4 times, consider it to be highly influential (a positive example in triplet-based loss); otherwise, we consider it to be a negative example. During evaluation, we sample query pa- pers which have least positive candidates and compute the L2 distance for similarity ranking. Note that our definition of "influential" differs from that in Valenzuela et al. (2015).
Citation Prediction (SPECTER Pre-training Triplets) This is the task and dataset used for pre- training in Cohan et al. (2020). It is based on citation links between scientific documents where each instance is triplet consisting of query, positive and negative paper. Each query can have up to five triplets, where the positives are sampled from papers directly cited by the query and negatives chosen either randomly (casy) or from citations of citations (hard). 3 casy and 2 hard difficult are chosen for each query. To evaluate the effectiveness of this pre- to-tarainig we follow Cohan et al. 2020) and use SciDocs for evaluation, excluding the recommendations task.
A.3 CLASSIFICATION
McSH Descriptors Medical Subject Headings (MeSH) (Lipscomb) 2000) indexes biomedical publications into categorical hierarchy consisting of descriptors which refer to topic headings and specific aspect related to a topic respectively. The dataset is a collection of scientific documents belonging to the 30 most frequently occurring top level McSH descriptors and having exactly one qualifier. We filter out the records that don't have an associated qualifier. The descriptors thus serve the labels in the multi-class classification task.
16

---

Under review as a conference paper at ICLR 2023
Fields of Study (FoS) The FoS task is multi-label classification problem where each scientific document is assigned one or more classes out of 23 possible fields. For gold test data, we manually labeled 471 papers into most three fields-of-study. For silver training data, assumed that paper within a venue generally falls within a narrow set of fields and manually assigned FoS labels to publication venues. We then propagated the venue labels to the papers published therein.
To evaluate different data sizes, we obtain the F1 score on the gold data in three settings: 5-shot, 10-shot, and the complete gold test set. The average of these scores is treated as the score for this task when computing the overall average score for the benchmark.
Disease Research State Model (DRSM) DRSM (Burns) 2022) is a collection Pubmed papers that deal with six specific aspects of rare diseases. The gold data annotated by in-house experts and used for evaluation, while the silver data is generated by annotation service providers with medical expertise.
Similar to FoS, we obtain the F1 score on 24-shot, 64-shot, and full data, then average the results before computing the final benchmark score.
Biomimiery We sample tags for set of papers in the PeTaL database (Shyam et al. 2019 to create a binary classification dataset with labels indicating whether each paper is about biomimicry. The data is unbalanced, with only 13% positive samples. We evaluate 16-shot, 64-shot, and full-data setup and take the mean to get the final score.
A.4 REGRESSION
Citation Count We sample a collection of scientific articles published 2016 from the set of papers in the search dataset described in A.1 that year period has passed for them collect citations. Each article has at least one citation, and the citation counts are converted to log scale.
Year of Publication The aim of this task to determine research trends by predicting the year of publication of scientific article. We sample publications from the search dataset with publication date after the year 2005 and scale the years so that their values are between 0 and Further, since this task is used for training along with citation count prediction, and to align the loss scales, the labels are scaled by the mean of the labels in citation count for parity.
Peer Review Score We use the OpenReview AP collect paper metadata and corresponding review scores for ICLR conferences from 2017 to 2022. Each reviewer in ICLR assigns a final rating in the range [0-10], and we take the mean rating as the label for every paper.
h-Index of Authors In this task the goal is predict the maximum h- b-Index of any of the authors of a scientific publication. We re-use the peer review score dataset, obtain the h- of all the authors for each paper using the Semantic Scholar AP and pick the max as the label. The labels are normalized to lie between [0.1].
Tweet Mentions The goal of this task is to predict the combined number paper's mentions and retweets. We post-process the dataset created by Jain Singh (202I) containing tweets about Arxiv papers between 2010-19. The sum of normalized counts of mentions and retweets is finally considered the score be predicted.
B IMPLEMENTATION DETAILS
During pre-tree mre-training, all the tasks with the same format share their fask-format specific parameters. The control code based paradigm introduces four cauddomlyinitializn special tokens to the vocabulary. We try initializing these additional parameters randomly, with the [CLS] token and combination of (CLS) with some noise. However, has little impact the resulting model
Phttss//api.opemeevie.ww hhaps://pp.ccmantscchoolarggg
17

---

Under review as a conference paper at ICLR 2023
Table Assigned input formats and control codes for each task form. [CLF], [RGN], [PRX] and [QRY] are special tokens, doc is the input.
Task form
Input format
Classification
concat ( ([CLF] doc)
Regression
concat ( [RGN] doc)
Proximity
concat ( [PRX], doc)
Ad-hoc Search concat ( ([QRY]/[PPR) query/ /do)
performance with random initialization being better on average. Further, we also tried loss weighting strategies (Chen et al.] 2018; Liu et al., 2019a) but our preliminary experiments produced better results without any scaling so we didn't explore it further. All the base models are trained for two epochs two 48GB NVIDIA Quadro RTX 8000 GPUs with 16 bit precision, effective batch size of 256, and a maximum input length of 512 tokens. Each batch is sampled with an equal number of examples from each task[ We use AdamW (Loshchilov & Hutter, 2019) with 1e-8. The learning rate follows an inverse square root schedule with a linear warmup of 700 steps and peak of 5e-5.
The adapter approaches follow the two step training process and learning rate configurations de- scribed in Pfeiffer et al. (202T). One adapter per task family attached the base model both single adapter and fusion stages and is trained for maximum of 6 and 4 epochs respectively. For PALS one layer added per task format and the entire network is trained for epochs as in SSickllnnd & Murray 2019).
B.1 EVALUATION
For classification and regression, we train linear SVM each downstream task using the em- beddings as input, and we tune the regularization parameter C via grid search. Multi-class and multi-label classification are configured under the one vs all classifier setting.
C ASPIRE EVALUATION
Table 7: Comparison of the our SciNCI. multi-format methods with ASPIRE on proximity tasks.
The best results for each base model are underlined. TS: Text Supervision, OT: Optimal Transport
Model
In- Train
Out-of Train
SciDocs
Average
TS ASPIRECS
91.3
79.9
TS ASPIREaio OT ASPIREcS OT ASPIREBIO SciNCL + MTL CTRL
65.5
65.6
90.9
79.8
64.5
65.5
91.2
79.8
65.0
65.7
90.5
79.6
66.9
66.9
80.7
SciNCL + Adapters
67.0
67.5
91.5
81.1
ASPIRE (Mysore et al. 2022) produces representations for the dense retrieval of scientific doeu- ments based matching multiple aspects between the query and candidates. To evaluate these representations under the settings they are designed for, we only report the results on the proximity tasks in Table 7| We use the model implementations available on HuggingFace which have been pre- trained on documents from the Computer Science (CS) and Biomedical (Bio) domains. The models variants can be further sub-categorized and a weighted sum of the similarity score among all the aspects based on Optimal Transport (OT ASPIRE) between the query and candidates. Both multi-format approaches with control codes and adapters produce better results overall and on out-of-train tasks. Note however, since ASPIRE models are trained on co-citations, they perform much better on average on the citation based tasks from SeiDocs.
retrieval based best aspect matching (TS ASPIRE)
We experimented with mixed and task sequential batching as well which did not yicld good results.
18

---

Under review as a conference paper at ICLR 2023
D SCIREPÉVAL DOMAIN DISTRIBUTION
Table 8: Data domain distribution in SciRepEval for the training tasks and comparison with SeiDocs. We group the unique documents in both the benchmarks by their MAG (Wang al. 2020a fields of study and present the counts in columns 2 and 3 and the absolute increase per field in column4.
Field of study
SciRepEval (A)
SciDocs (B)
Increase Ratio (A/B)
Medicine
3,201,323
74,685
43
Computer Science
1,187,689
199,664
Biology
882,357
13,377
66
Chemistry
508,056
3,813
133
Psychology
492,071
22,590
22
Materials Science
271,865
7,681
35
Engineering
254,826
31,444
8
Mathematics
231,482
25,800
Physics
217,670
7,285
30
Business
217,585
5,450
40
Sociology
156,128
2,305
68
Political Science
154,388
1,032
150
Economics
123,357
2,705
46
Environmental Science
91,682
1,136
81
Art
89,527
206
435
Geography
83,688
1,491
56
Philosophy
61,996
151
411
Geology
51,103
640
80
History
46,430
159
292
We study the domain diversity of SciRepEval and display the results in Table 8 To compare against the training data for SciDocs, we consider the citation prediction triplets on which SPECTER is trained which is also subset of the SciRepEEvl in-train tasks. Even though Medicine and Computer Science papers still form bulk of the data, SciRepEval has 105x more documents on average per domain compared to the SPECTER triplets.
E SPECTER OBJECTIVE
Lastly, perform ablation study to better understand the importance of the unsupervised citation-based training objective. We used SciBERT as the base model for this ablation since both SPECTER and SciNCL were trained with the citation objective. Removing the citation objective and its accompanying data from SciBERT + MTL CTRL, we find that the in-train performance drops from 61.9 to 61.8, while out-of-train drops from 57.9 57.5, hinting that the citation objective may be helpful for generalization to new tasks.
F CROSS-TASK CORRELATION ANALYSIS
In Figure 2 we show Pearson correlations of model performance metrics between tasks in SciRepE- val. To compute the correlations, we include all of the individual task results of the model runs shown in Table 2 Table 3. excluding the ensembles. The correlations between tasks in SciDocs (bottom right) are highest, while correlations between tasks the entirety of SciRepEval span larger range. Notably, DRSM-Complete and S2AND are uncorrelated with most other tasks. This shows that the overall task diversity is larger in SciRepEval than in SciDocs.
G TASK RELATEDNESS FOR MULTI-TASK TRAINING
To train on multiple tasks simultaneously. care has to be taken to choose a combination of tasks such that negative transfer is avoided, but finding this optimum combination of tasks is hard (Aribandi
19

---

Under review as a conference paper at ICLR 2023
Correlation Belwoon Taska
[CLF) MoSH Dosoritorc F1
[CLF) Completa
Fee Gold Shot-F1
(CLF) Fax 5 Shel
[DLF] Biominicry Completo
[CLF] (eommaay Shet-F1
[CLF) Blomaay 16 Shot F1
[[LF] DRSM 64 Shot F1
[EX P] neeM Stet
Foode:1 MMP
[PXx] Feeds-Mt MAP
(PR)] Paper Matching--
intuentaal Citaions- MKP
[PRX] Bara Author Dataction MAP
[PRog R0AND-RS
[SRCH] Feeds Tiko MAP
[SRCH] Search nDCG
[RUN] Meer Heveow Score K
(RGN)
piCN) Tweee Mentonns
[RON] Citaton C Tax
[RON] Publication
(CLF) Does] MAG F1
(CLF) [Bo Doca] MeGil Deeeses-
[PRX] [BdDera Coviie
[PRx [SoDees] cold nDC8
(Ptd] [Settocs] nDCG
[Feo (SoiDezs)
(cu (aliy Fox Cold to Shat- FS
(CLF Fos cod 5 Shot Fi Blommiyy Complele
TOLFI alminicy 64 Shat- F1
(OLF) (CLF) DRSM et Shat F1
Pryy reecs MMP
MPP
(PR2S) Highyy trruar munnnaa Citatons MAP
(PFAA Same fush or Delecio MMP
(PPO) SRAND B3
jeniery Feede life MAP
(SRCH) Boardh 1000
[ERCH] TRE covao 1DoG
DINN their tor one score K. ia
(Rov: Toost Menioas K Thu
(TOH)] prok) Citstton Count i Tou
F1
(roog (Sethoo Co- seeee 100G
pPX (% ihsa Commel i800
(S Dved iDc0
(PRX] Doooa Co- 1000
[eir] Meed Neoo peoerapton [CLF] Fox Dorppilte
prine
[CLF] Bemmiiay 18 Stor
(CLF) (CLF)DRBM DRDMR complle
(CLF) LLLIDRMMM shat
(PPO0] OrooiFoodss Foods M.
(Prong Papper deweeeee narcinng
[HON) Mox maschinde
(CLF) (SaiDoe)) (o0os))M MMG
(CLF] secieca MAss Dissoco
[ecod pnstication
Figure 2: Correlations of model performances between tasks SciRepEval.
al. 2022, Padmakumar et al. 2022; Fifty al., (021). For given set of tasks, it's often not feasible train on all 2 1 task combinations to find the best one. Hence, rather than coming up with an optimum combination, recent work suggests pre-fine-tuning on large collection of tasks simultancously to offset the negative transfer between subset of those tasks(Aghajnyyan et al. 2021 Aribandi et al. 2022). Padmakumar et al. (2022) show that pre- fee-finettttngg on a small set of tasks related the downstream task is more efficient than large scale multi-task training and yields similar results. As shown in Table 9) we study this training affinity among our task formats by pre- fine-tuning on individual task formats
well their combinations in multi-task setup. Supporting the findings in Padmakumar et al. (2022), pre-fine-tuning on all the tasks using each individual format's training data leads better or similar performance related downstream tasks when compared to training all the formats at once for all the cases except out-of-traie classification. Moreover, combining related task groups like classification and regression during pre-fine-tur preffinetuuing results in better task transfer than training them individually. This may be due to the fact that the learned representations of both are consumed as features by downstream linear SVM. Similarly, proximity and ad-hoc search seem to give one another boost when trained together. However, training on the tasks simultaneously yields the best results on average, and individual results are within point of the best combinations per task format, except for out-of-train regression and in in-train classification.
20

---

Under review as a conference paper at ICLR 2023
Table 9: Task relatedness analysis for choosing a sub-group tasks to train on so as to obtain opti- mum performance. The base SeiNCL model(Ostendorft et al. (002b) is trained formats (rows) and then evaluated for a comparison with SciNCL CTRL (last row). Both per task format and overall average performance is reported (columns). The best training combination for every task is highlighted in bold. The best single and combined training results for every evaluated task format respectively are underlined.
more task
Task Format Evaluated On
Task Format(s)
Trained On
Out-of Train
In- Train
All
CLF
RGN
PRX
SRCH
Avg
CLF
RGN
PRX
SRCH
Avg
Avg
CLF
56.8
15.4
70.0
77.5
54.1
68.2
21.3
59.3
71.1
$2.7
64.6
RGN
56.6
21.4
68.1
74.1
54.4
53.2
45.0
55.2
68.7
53.7
59.8
PRX
55.4
19.6
73.7
84.5
$7.6
S7.8
32.1
67.2
73.2
55.3
68.6
SRCH
46.6
15.7
71.7
83.8
54.1
53.2
28.1
60.4
78.0
51.6
63.3
CLF RGN
58.7
20.5
68.4
75.0
54.8
68.8
45.1
57.3
68.7
58.7
63.8
CLF + PRX
73.4
82.2
32.6
66.4
43.8
58.1
CLF + SRCH
58.8
17.2
71.6
83.5
56.6
67.7
31.0
59.9
78.4
56.5
66.3
RGN + PRX
51.8
20.9
73.5
78.2
56.1
50.4
44.3
67.1
71.7
56.5
67.7
RGN SRCH
52.0
17.8
72.1
84.0
55.8
54.8
43.6
60.6
78.4
56.6
65.3
PRX SRCH
53.2
17.8
73.8
85.1
56.8
57.9
32.1
66.6
78.2
55.9
68.3
All
58.9
18.2
73.4
84.7
57.7
66.7
44.3
66.9
78.5
62.1
70.3
21


Reviews:
[{"review_id": "POSOW6sPRC", "reviewer": "Reviewer_d6LN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors of this paper introduce a benchmark for scientific representation learning consisting of 25 tasks in 4 formats (classificaiton, regression, ranking, search) and evaluate multiple general-purpose scientific representation learning methods (SciBERt, SPECTER, SciNL) alongside adaptation methods to learn task-specific and format-specific representations. The results suggest that fine-grained scientific document representations significantly out-perform general-purpose representations.", "summary_of_the_review": "The authors identify an area of opportunity relating to current datasets and methods to evaluate scientific representation learning models and propose a well-grounded, larger benchmark in SciRepEval. The ablation study is comprehensive and there is some interesting discussion of cross-task analysis, although I would have liked to see that section expanded.", "strength_and_weaknesses": "Strengths:\n- Introduces a more comprehensive benchmark for scientific document representation learning\n- Clearly distinguishes SciRepEval from previous benchmarks (SciDocs) and justifies why a new benchmark is needed (low-power of the recommendation task, a need for more realistic task settings, and lack of training corpus)\n- Comprehensive ablation setup targeting fine-grained representation learning, multi-task learning with a strong base model\n\nWeaknesses:\n- It would have been nice to see an explicit section detailing what additional challenges remain in benchmarking scientific representation learning and how these limitations apply to the results seen with multi-task approaches.\n- Would have liked to see more discussion about task relatedness when discussing benchmarks, especially since SciRepEval seems to be designed in order to improve breadth and coverage of common real-world applications of the representations. For example, it would be interesting to see the relationship as applied to the task of choosing which sub-tasks to group for multi-task learning (see Efficiently Identifying Task Groupings for Multi-Task Learning; Fifty et al. NeurIPS 2021).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NZac4_IDVY", "reviewer": "Reviewer_tJad", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces SciRepEval, which is a novel benchmark for training and evaluating scientific document representations. Additionally, the struggles of multi-task learning with regards to generalization in these tasks is explored. An alternative that explicitly encodes the task type is offered and evaluated.\n", "summary_of_the_review": "This paper extends an existing benchmark substantially, and offers a new approach to multi-task learning meant to deal with the multitude of tasks in a way that is not wholly novel, and beats the baseline, although only slightly. \n", "strength_and_weaknesses": "Strengths\n\n- Not only are we treated to a single task in this benchmark, but 25, including classification, regression, and recommendation, each of which have appropriate measures for evaluation (though other measures may also have been appropriate).\n- Given the variety of tasks, a ‘multi-format’ method of representation learning is proposed which is not entirely novel on its own, but is additively useful within this paper more ostensibly about the benchmarking suite.\n- Suitable baselines (SciBERT, SPECTER) and several extensions (e.g., CTRL-code and embedding adapters) provide a suitable evaluation suite.\n\n\nWeaknesses\n\n- Although much of the information is found in the Appendices, more detail on the tasks themselves are expected within the main body, especially with regards to the source of the data, which could likely be added to Table 1, in some form.\n- Tables 2 and 3 and their discussion in the text require some additional context to evaluate these results, especially with regards to the units of measure and highlighting that the first two columns are not exactly comparable with the third (SciDocs). Given the relative similarities of the performance between SPECTER and SCiNCL (especially considering the variants of each), statistical significance tests should be performed.  \n\n\nMinor\n\n- The use of task-specific control codes is reminiscent (somewhat) of ELMo, and a comparison or acknowledgement should be made to some of the literature around that model.\n- Some further explanation around the deficiencies of SciDocs in Sec 2 (especially with regards to the correlations and easiness of negative candidates) really should be provided, especially as SciRepEval’s superiority is meant to be evaluated in contrast to the former.\n- There is a higher-than-usual amount of repetitiveness, especially around the 25 tasks in SciRepVal in the first few pages, but this is a very minor complaint\n- Some deeper dive into how performance is affected by covariates such as the field of study (Table 8) after one filters out the effect of data set size would be interesting.\n- Be sure to check the formatting of your references\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AKYRUOCmpRh", "reviewer": "Reviewer_uueE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new benchmark (SciRepEval) for scientific representation learning consisting of 25 tasks in 4 formats (classification, regression, ranking, and search). It shows that learning a separate document representation for each task format would improve the task performance compared to learning a single representation for all tasks.", "summary_of_the_review": "This paper is written in above-average quality and provides a new benchmark with a comprehensive analysis", "strength_and_weaknesses": "Strength\n- The authors provide a comprehensive analysis on 25 tasks in 4 formats. \n- The authors use strong baseline models to make the results more convincing.\n\nWeakness\n- There is not enough information on the tasks in the main content. Would be better to provide some high-level info there and left the majority in the Appendix.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "196nCVxVANN", "reviewer": "Reviewer_RNDk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new benchmark for scientific document representations, including 25 tasks across classification, regression, ranking and search. The paper also includes investigations on performances of existing models on the introduced tasks. ", "summary_of_the_review": "While the paper provides a useful benchmark, there is a strong component of novelty missing from the paper. ICLR does not seem like the correct venue for this paper to be presented. ", "strength_and_weaknesses": "Strengths: \n- The paper consolidates useful tasks to make one benchmark for scientific document representations\n\nWeaknesses: \n- The models mentioned in the paper are mostly created by others in the community\n- The benchmark, while useful, is a consolidation of tasks that were already existing \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject