Pre-training Multi-task Contrastive Learning Models for Scientific Literature Understanding
Authors: Yu Zhang
Hao Cheng
Zhihong Shen
Xiaodong Liu
Ye-Yi Wang
Jianfeng Gao
Venue: EMNLP 2023 Findings (2023)
Keywords: scientific literature understanding
multi-task learning
contrastive learning
language model pre-training

Abstract: Scientific literature understanding tasks have gained significant attention due to their potential to accelerate scientific discovery. Pre-trained language models (LMs) have shown effectiveness in these tasks, especially when tuned via contrastive learning. However, jointly utilizing pre-training data across multiple heterogeneous tasks (e.g., extreme multi-label paper classification, citation prediction, and literature search) remains largely unexplored. To bridge this gap, we propose a multi-task contrastive learning framework, SciMult, with a focus on facilitating common knowledge sharing across different scientific literature understanding tasks while preventing task-specific skills from interfering with each other. To be specific, we explore two techniques -- task-aware specialization and instruction tuning. The former adopts a Mixture-of-Experts Transformer architecture with task-aware sub-layers; the latter prepends task-specific instructions to the input text so as to produce task-aware outputs. Extensive experiments on a comprehensive collection of benchmark datasets verify the effectiveness of our task-aware specialization strategy, where we outperform state-of-the-art scientific pre-trained LMs. Code, datasets, and pre-trained models can be found at https://scimult.github.io/.

Pre-training Multi-task Contrastive Learning Models for Scientific Literature Understanding
Yu Zhang Hao Cheng Zhihong Shen Xiaodong Liu Ye-Yi Jianfeng Gao
University of Illinois at Urbana-
Urbana-Champaigg Champaign Microsoft Research Microsoft Search, Assistant and Intelligence {chehao, zhi zhihosh, xiaodl, yeyiwang jfgao f
yuz9eillinois.en edu
Abstract
models (LMs) (Beltagy et al., 2019; Gu al., 2021; Liu et al., 2022) in these tasks as they generate high- quality scientific text representations, especially when the LMs are further tuned via contrastive learning. For example, MICoL (Zhang et al., 2022) proposes metadata-inducee contrastive learning that can perform extreme multi-label paper classifi- cation with more than 10,000 classes; SPECTER (Cohan et al., 2020) and SciNCL (Ostendorff et al., 2022) leverage citation information to create train- ing pairs and achieve remarkable performance in predicting various types of links between papers. Nevertheless, jointly using data across different scientific literature understanding tasks for LM pre- training remains largely unexplored. Intuitively, there are some common knowledge and skills that can be shared across related tasks. For example, accurately identifying fine-grained topic classes of a paper not only helps classification but also provides hints to link prediction and search. There- fore, a multi-task Icarning framework is expected to be beneficial with improved parameter efficiency. However, if all parameters of the backbone LM are shared across tasks (Liu al., 2019), the model is observed to suffer from the undesirable task inter- ference (Ma et al., 2023), that is, the model sacri- fices the performance others when jointly trained certain extent. This is because specialized skills are still required in different tasks competing for the limited shared parameter space. For instance, the encoder for extreme multi-label paper classification should fo- cus more on fine-graine ficlds-of-study entities in each paper, while the encoder for citation pre- diction should put more effort into understanding citation intents. Mixing these two skills may result in a negative transfer across the two tasks.
Scientific literature understanding tasks have gained significant attention due to their po- tential to accelerate scientific discovery. Pre- trained language models (LMs) have shown effectiveness in these tasks, especially when tuned via contrastive learning. However, jointly utilizing pre-training data across multiple het- erogeneous tasks (e.g., extreme multi-label pa- per classification, citation prediction, and liter- ature search) remains largely unexplored. To bridge this gap, we propose a multi-task con- trastive learning framework, SciMult, with focus on facilitating common knowledge shar- ing across different scientific literature under- standing tasks while preventing ssecifie skills from interfering with each other. To be specific, we explore two techniques task- aware specialization and instruction tuning. Trans-
The former adopts a Mixureeoff former architecture with task-aware sub-layers; the latter prepends task-specific instructions to the input text so to produce task-aware outputs. Extensive experiments on a compre- hensive collection of benchmark datasets ver- ify the effectiveness of our task aware special- ization strategy, where we outperform state- of fi-the-ar art scientific pre-traines LMs. Code, datasets, and pre- trained models can be found https::/sciii mul thub.io/
tasks boost the
1 Introduction
Scientific literature understanding tasks, such as paper classification (Zhang al., 2023b), citation prediction (Bhagavatula et al., 2018), scientific lit- erature search (Voorhees al., 2021), and recom- mendation (Kanakia et al., 2019), have received increasing attention because they can be broadly applied to academic service platforms (Tang et al., 2008; Sinha et al., 2015; Ammar et al., 2018), 8), and more importantly, uncover knowledge structures to accelerate scientific discovery (Naik et al., 2022; Chandak et al., 2023). Recent studies have demon- strated the effectiveness of pre-trained language
In this paper, to mitigate task interference in multi-task scientific literature understanding, we propose to consider two techniques: task-aware specialization and instruction tuning. Task-aware specialization, inspired by the Mixture-of-Expert (MoE) Transformer architecture (Fedus et al., 2022;
Equal contribution
Work done during an internship at Microsoft Research.

---

Du al., 2022; Zhou et al., 2022; Cheng et al.. 2023), modifies the Transformer block in the LM to have multiple parallel sub-layers, each of which is dedicated for one task. When performing differ- ent tasks, the input will be routed to different sub- layers based on task types. In this way, the encoder contains both shared parameters and task-specific parameters, making it capable of producing task- aware outputs when tapping into shared knowledge. In contrast, instruction tuning adopts one encoder for all tasks, but it prepends task-specific instruc- tions (Wei et al., 2022; Sanh et al., 2022; Ouyang et al., 2022; Wang et al., 2022; Chung et al., 2022: Asai et al., 2023) to the input text during training so that the encoder can learn to produce task-aware representations. These two ideas are different from the techniques (e.g.. adapters (Houlsby et al., 2019) and control codes (Keskar al., 2019)) explored in previous studies (Singh et al., 2022) for multi- task scientific text representation learning. Indeed, far as we know, this is a pioneering study that explores the effect of the MoE Transformer archi- tecture and instruction tuning in scientific NLP.
2 Background
We consider three widely studied tasks in scien- tific literature understanding: classification, link prediction, and search.
(Extreme Multi-Label) Classification. Classi- fying academic papers to their relevant label(s) is a fundamental task in scientific text mining. It can help organize papers according to their fields/themes and benefit downstream applications such trend analysis of scientific topics (Prab- hakaran et al., 2016; Jin et al., 2021). The label space L can be either coarse-grained (Cohan et al. 2020; e.g., predicting whether a paper belongs to "Computer Science" or "Biology") or fine- grained (Peng et al. 2016; Xun et al. 2019; Ye et al. 2021; Zhang et al. 2021, 2023b; e.g., predicting whether a paper is relevant to "Alphacoronavirus or "Betacoronavirus" or both). When & is large and fine-grained (e.g., with more than 10,000 la- bels), it natural to assume that each paper p can be relevant to more than one label. This task is called extreme multi-label classification (Liu al., 2017; Prabhu et al., 2018; Chang al., 2020), which aims rank all labels L according how likely p is relevant to l.
To validate the efficacy of our proposed tech- niques. we conduct a comprehensive empiri- cal study using datasets from multiple sources (Kanakia al., 2019; Cohan et al., 2020; Thakur et al., 2021; Zhao et al., 2022; Singh et al., 2022; Zhang et al., 2023b) for evaluating various scien- tific literature understanding tasks. For each task, models will be tested on not only in-domain but also cross-domain evaluation datasets. Specifically, for extreme multi-label text classification, mod- els trained on computer science and biomedicine papers will be tested in the geography and psychol- ogy fields (Zhang et al., 2023b); for link prediction, models trained on citation signals (Cohan et al., 2020) need to be evaluated on patient summary retrieval (Zhao et al., 2022) and paper recommen- dation (Kanakia et al., 2019); for search, models will be tested on datasets specific to COVID-19 (Voorhees et al., 2021) or related to claim verifi- cation (Wadden et al., 2020) which are not seen during pre-training. Experimental results show that SciMul SciMult outperforms competi- tive scientific prentr LMs (Cohan et al., 2020; Ostendorff et al., 2022; Singh et al., 2022) on most datasets and achieves the new state-of-the-aar performance on the leaderboard of PMC-Patients (Zhao et al., 2022). Ablation studies further prove that task-aware specialization can effectively mit- igate task interference, while the improvement brought by instruction tuning is not consistent across all tasks.
Link Prediction. Link prediction aims to predict if certain type of link exists between query paper PQ and candidate paper PC- Narrowly, "links" refer to citation links (Bhagavatula et al. 2018; Wright and Augenstein 2021; i.e., Po cites Pc). Broadly, "links" can be defined as relations that PQ and PC are co-viewed frequently by users, co-cited frequently by other papers, and so on (Co- han et al., 2020; Zhao et al., 2022). An accurate link prediction model can benefit tasks like paper recommendation (Kanakia et al., 2019) and help identify the potential use scientific literature (Yin et al., 2022; Lin et al., 2023).
Search. Scientific literature search helps re- searchers track their interested fields-of-study and prevents them from drowning in the whole litera- ture. Given search query q and a pool of papers P, the task is to find papers p E P that are relevant to q. Search also serves as the initial step of more complex scientific text mining tasks, such claim verification (Wadden et al., 2020) and open-domain question answering (Jin et al., 2019).
3 Models
3.1 Multi-task Contrastive Learning
One can observe that the three tasks bear a common feature - a "query" q and a pool of "candidates"

---

C C- is the paper p to be classified in classification, the query paper PQ in link prediction, and the search query q in literature search; C is the label space L in classification, the set of candidate papers PC in link prediction, and the candidate paper pool P in search. This motivates us to jointly train a multi-task model that is applicable to all tasks.
C2,..., c2,......
) are given. To be specific, q
Add Norm
Add & Norm
FFN
FFN
FFN
Router
MHA
MHH 2a0k Produclen
MHA
Add & Norm
Router
MHA
Mixture Experts Transformer with Task-Specific MHA Sublsyers
Mixture of Exparts Transformer with Task-Specilic FFN Sublayers
To implement this idea, our proposed SciMult framework is built upon a Bi-Encoder architec- ture, where two encoders (whose parameters are shared) encode queries and candidates indepen- dently. Following Karpukhin et al. (2020), we adopt maximum inner product search (MIPS) in the embedding space to find positive candidates for each query. Formally, the similarity is calculated as follows:
Figure I: Two different types of Mixture-of---we Experts Transformer architecture. They will route the imput to different MHA and FFN sub-layers, respectively, when considering different tasks.
citation intents. To tackle this issue, we consider two different strategies, task- specialization and instruction tuning to learn task-specific repre- sentations of scientific text.
sim(g, c) B(q) E(c).
(1)
3.2 Task-Aware Specialization
where E(-) is a text encoder (e.g., a Transformer- based LM) to be learned. We assume the text in- formation of a paper is its title and abstract (i.e., "[CLS] [title} } [SEP] (abstract) [SEP]"). Follow- ing Zhang et al. (2022), we assume that each label's name and definition are available', which consti- tute the label text information (i.e., "[CLS] [name) [SEP] [definition] J [SEP]"). For search queries, we naturally form their text input as "[CLS] {query) [SEP]". The output of E(-) is the representation of [CLS] after the last layer.
Inspired by recent Mixture-of of--Exerts (MoE) mod- els (Fedus et al., 2022: Du et al., 2022; Zhou et al., 2022; Cheng al., 2023; Ma et al., 2023), we pro- pose to adopt task-specific Transformer blocks in the LM architecture. Specifically, a typical Trans- former block (Vaswani et al., 2017) contains multi-heat attention (MHA) sub-layer and a feed- forward network (FFN) sub-layer. Fedus et al. (2022) propose
cittrreoffExperrt Transformer block with multiple FFN sub-layers stacked upon shared MHA sub-layer. As shown in Figure I (right), we adopt this architecture and let each FFN sub-layer correspond to one particular task t (t E [classification, link prediction, search)) ). For example, if the encoder E(-) is trained/tested on a classification task, the input will be routed to the classification FFN. Different from Fedus et al. (2022), Ma et al. (2023) propose to specialize the MHA sub-layer and observe better performance in open-domain question answering. We test this architecture as well, where, as shown in Figure (left), shared FFN sub-layer is stacked upon task- specific MHA sub-layers. We again use the same task-dependent routing for this variant.
The Bi- E--oddde is trained via a contrastive learn- ing objective by pulling relevant q and c₁ together while pushing irrelevant q and e- apart:
exp(simlg, c+)) min- log exp(sim(g,c.)) + Σ., exp(sim(q.c.)) EG
(2)
Although it has been shown that model pa-
rameter sharing improves the performance of Bi- Encoder retrievers (Xiong et al., 2021), simply shar- ing all parameters of the backbone LM (Liu et al., 2019) may suffer from task interference and lead suboptimal performance. This is because the semantics implied by relevant (7, c) pairs for dif- ferent tasks vary. For example, the encoder for fine-grained paper classification needs to pay more attention to fields-of-study entities, while the en- coder for link prediction focuses on understanding
In SciMult, following Du et al. (2022), we stack typical Transformer blocks and task-specific Trans- former blocks alternately, i.e.. for a base-size LM with 12 Transformer blocks, there will be 6 typical Transformer blocks and 6 task-specific Transformer blocks interleaved with each other. In this way, the encoder will have both parameters 0, characterizing specific skills for task l and parameters 0 charac- terizing the common knowledge shared across all tasks. Thus, we denote the encoder as E(p.e,) (.).
'This assumption holds for benehmark scientific label spaces such as fields-of-stddy in the Microsoft Academic Gruph (MAG) (Shen et al., 2018) and terms in Medical Sub- ject Headings (MeSH) (Coletti and Bleich, 2001). Meanwhile, if label definitions are not available, our model can take lahel names as the only input, the performance of which is studied in Appendix E.4.

---

and the task-aware similarity between the query q and the candidate will be modified
Task
Instruction
Classification
Tag a scientific paper with relevant scientifie topic classes.
sime (q. c) (q) (c).
(3)
Link Prodiction
Find a patr of scientific papers that one paper cites other.
Note that the query encoder and the candidate en- coder for the same task still share their parameters.
Seurch
Retrieve scientific that is relevant the query.
Table I: Instructions used for the three tasks.
3.3 Instruction Tuning
query/candidate encoder Es(-), and let them share parameters (i.e., ø = 0). Second, we can keep the querylcandidate encoder frozen and optimize the instruction encoder only, which bears similarities with prefix-tuning (Li and Liang, 2021) by treat- ing instructions prefixes. We will evaluate both approaches in our experiments.
Training LMs with task-specific instructions (Wei et al., 2022; Sanh et al., 2022; Ouyang al., 2022; Wang et al., 2022; Chung al., 2022; Asai al., 2023) has been extensively studied with remarkable progress achieved. However, the effect of instruc- tion tuning on scientific literature understanding tasks has remained elusive. Moreover, the major focus of previous studies is on effective zero-shot few-shot model transfer to tasks rather than mitigating task interference. As result, head-to- head comparison between instruction tuning and MoF. in multi-task LM pre-training is missing, so we aim to bridge the gap in this paper.
3,4 Negative Sampling
Previous studies on contrastive learning with sci- entific text (Cohan et al., 2020; Ostendorff et al.. 2022) have emphasized the importance of hard neg- atives. Specific to citation prediction, Cohan et al. (2020) propose way to derive hard negatives: Given positive query- where PQ cites PC i. if there exists paper pC such that (1) PC+ cites pC- but (2) PQ does not cite pC-, then pc- is hard negative.
Different from task-aware specialization which trains (+) for each task instruction tun- ing keeps one encoder E-(-) with all its param- eters 0 shared across different tasks. Each task l is instead characterized by natural language instruction Tt- The instructions we use for the three tasks are shown in Table 1. We can prepend the representations of I; to the query and can- didate texts to get their task-aware embeddings. To be specific, suppose It contains tokens { 1K k=1- We first use an instruction encoder Ex(() to encode It and get its token representa- tions (a(
pair (pQ- PC+)
We generalize this idea to the other tasks. For extreme multi-label classification, given a positive paper-label pair (p. l). we consider paper p' cited by p. sharing a common author with p. or published in the same venue as p, in which case p' should be semantically close to p. If p' has a label I' that is irrelevant to p, then is treated as hard negative for (p, 1). For literature search, we use the training data from Singh et al. (2022) which contains a short list of papers returned by an academic search engine for each query q. The papers clicked by users will be treated as positives p+. and the others in the list will be viewed hard negatives p... Related studies (Cohan et al., 2020: Ostendorff et al., 2022) show that combining easy negatives (i.e., negatives randomly sampled from the entire candidate pool C) and hard negatives leads to better performance. Thus, during pre-training, for each positive pair (g, c.), we sample one hard negative and treat all in ba-batch negatives (Karpukhin et al., 2020) as casy negatives, both of which are com- bined as e_ 's to optimize Equation 2.
(n) ===1 after each layer n E (0, 1, ..., N). Then, we use the query/candidate encoder E-(-) (-) to encode
4 4A and e CI.....C At layer n E {1, 2, .... N). the output representations of and c will take the instruction token representations corresponding to that layer as context. Formally,
Transformer( (n-1)
(n-1)
LA
1.1
(n-1)
(n-1)
(4)
(n)
c
Transformer
(an (x-1)
1.1 2e
1)
(n-11)
The task-aware similarity between y and will then be modified as
(N) (N) sim, (g. c) StCLsT "CCss]"
(5)
4 Experiments
4.1 Datasets
There are two ways to train the model. First, we can update the entire architecture, includ- ing both the instruction encoder (-)()) and the
We adopt a comprehensive collection of benchmark datasets for model evaluation. Each task has its pre-

---

Task
Pre-training
In-domain Evalustion
Crus--domain Evaluation
MAPLE (Zhang al, 20236)
MAPLE (Zhing al, 20236)
(CS-Conference, Chemisty-McSH)
MAPLE (Zhang al., 20256)
Chosilieation
[CS Joumal, Biology-MeSIII McSHI Medicine MeS1f)
SeiDoes (Cahan et al., 2020)
(Geographyy, Psychology)
[MAG Fields, McSH Discases]
Link Prodiction
Citation Prediction Triplets (Cohan al.,
SeiDoos (Coltan of 2020)
Recommendation (Kanakia at. 2019)
[Co-viow, Co-read, Citce Co-cite]
PMC-Patients (Zhno et sl., 2022)
TREC-COVID (Voothees al. 2021),
Search
SciRepEvvl--eeaee (Singh et al., 2022)
AciReppEvll-Seuee (Singh et al., 2022)
SciFuct (Wadden et al., 2020),
NFCurpus (Boteva et 2016)
Table 2: Datasets used for pre-training, in-domain evaluation, and cross-domain evaluation.
training, in-domain evaluation, and cross-domain evaluation datasets?, which will be briefly intro- usage of
(Singh al., 2022) with 528,497 queries. It
also has a hold-out testing set with 2,637 queries, which employ for in-domain evaluation. For cross-domain evaluation, we adopt TREC-COVID (Voorhees et al., 2021), SciFact (Wadden et al., 2020), and NFCorpus (Boteva et al., 2016), all of which are from the popular BEIR benchmark (Thakur et al., 2021). Note that TREC-COVID has two different versions in SciRepEval and BEIR. The BEIR version has more simplified queries and a larger pool of candidate papers. We will report model performance on both versions.
duced below. Table 2 summarizes these datasets with more details in Appendix A.
Classification. We consider the MAPLE bench- mark (Zhang et al., 2023b), which consists of 23 fine-graines multi-label paper classification datasets across 19 scientific fields. Each paper in MAPLE is tagged with its relevant MAG fields- of-study (Shen et al., 2018) and MeSH terms (Co- letti and Bleich, 2001). Among the 23 datasets, CS-Journal, Biology-MeSH, and Medicine-MeSH are selected for pre-training: CS-Conference and Chemistry-MeSH are used for in-domain evalua- tion; Geography and Psychology. whose candidate label spaces are not seen during pre-training, are uti- lized for cross-doman evaluation. Besides, we use the MAG and McSH datasets in the SciDoes bench- mark (Cohan al., 2020) as in-domain evaluation datasets for grarse-grained paper classification.
4.2 Compared Methods
We consider the following LM baselines: SciB- ERT (Beltagy et al., 2019), SentBERT (Reimers and Gurevych, 2019), SPECTER (Cohan et al., 2020), PubMedBERT (Gu et al., 2021), LinkBERT (Yasunaga et al., 2022), BioLinkBERT (Yasunaga et al., 2022), OAG-BERT (Liu al., 2022), SciNCL (Ostendorff al., 2022), and SPECTER 2.0 (Singh et al., 2022). Details about the basclines can be found in Appendix B.
Link Prediction. For pre-training, we leverage more than 819K citation prediction triplets released in Cohan al. (2020), which were used pre-train SPECTER (Cohan et al., 2020). For in-domain evaluation, we make of the SciDoes benchmark (Cohan et al., 2020), which evaluates the predic- tion of four link types: Co-view, Co-read, Cite, and Co-cite. For cross-domain evaluation, we use (1) the PMC-Patients dataset (Zhao et al., 2022) where each query is patient summary and the task is to find its linked research articles and patient summaries, and (2) the Recommendation dataset (Kanakia et al., 2019) collected via an online sur- vey, where the participants are authors of query pupers, and they need to judge the relevance be- tween the query paper and some candidate papers on a scale
Besides LM baselines, we also report the per- formance some task-specific classical methods, such as Citeomatic (Bhagavatula et al., 2018) for citation prediction. Kanakia et (2019) for rec- ommendation, and BM25 (Robertson and Walker, 1994) for search.
For our SciMult model, we pre-train five model variants: toMMult-annill.. SciMult MMAAlleertH SciMul t-Pefix, and SciMult- All of them are initialized from P Among them, SciMult- MHAExpert and CcFMuxt-FFNEExeer adopt the task-aware specialization strategy presented in subsection 3.2 with task-specific MHA and FFN sub-layers, respectively; SciMult-Prefix and SciMul MeMMult-Instruune Instrruccio leverage instruction tuning presented in subsection 3.3, the former of which tunes the instruction encoder only while the latter
Search. For pre- training, we exploit the Search dataset released in the SciRepEval benchmark
IIn-domain evaluation datasets share the same data source and properties (e.g.. the label space, the link type) with pre- training data; cross-domain evaluation datasets are otherwise and evaluate model generalizability.
hnttps://huggingenn ubMedBERT base- www.TT-bbaeuunccdd--ee unnased-abstrace

---

MAPLE (Zangg at al.. 20236)
Finerge Fieegrained classi Fication
CS-Comferance
Chemistry-MMSS
Gongraphyy
Psychology
R00100
R:020
RG
R:050
RF
RG050
RG:100
Average
SclBERT(Bellagy et al., 2019)
12.01
42.84
43.87
30.53
31.46
32.15
52.01
$4.53
58.11
43.07
44.02
45.22
43.32
SemBERT (Reimers and Gurevydh, 2019)
42.79
44,34
45.96
30.75
31.75
32.44
53.54
57.23
61.11
43.33
44.
46.17
SPECTER (Coban et al., 2020)
47.38
53.18
58.43
34,26
29.35
43:41
59.12
65.33
70.75
47.07
51.30
56.17
52.15
PU (CRu et al, 20121)
41,95
42.50
43.24
30.46
31.46
31.83
52.19
54,82
56.88
43.93
46.25
49.27
43.74
LinkBERT (Yasutaga
2023)
30.52
31.56
32.37
42.62
43.23
Riel inKBERT (Yaounag et al,, 2022)
42,90
42.61
43.57
30.37
31.15
31.48
50.36
50.54
50.86
42. 20.30
42.55
42.79
41.74
OAG-BERT (Liu 2022) 2022)
42.59
43.79
44.93
30.58
31.97
32.62
51.44
52.25
53,16
42.63
42.95
43.30
42.68
SeiNCL, (Ouennefff et al. 2022)
17.92
53.57
58. 29
34.99
40.50
64.64
59.00
65.49
71.41
48.
54.21
50.50 84
53.23
SPECTER 2.0 (Singh et al., 2022)
18.63
55.09
60.68
36.17
43.06
11.225
62.87
70.30
76,37
50.50
58.27
65. 66.66
56.33
Sc -amiill
$3.40
64.70
74.09
50.42
56.58
63.17
ScaMkllt MHAExpettt
$4.02
65.49
75.07
39.41
50.92
59.59
65.94
75.01
81,93
$1.77
59.55
67.86
62.21
SFiNult-FFNEspen
53.73
63.79
72.46
38.01
48.76
57.43
61.90
70.69
78.81
50,09
56.91
61.28
59.74
t-Petix
33.68
72.07
37.97
48.95
57.56
79.71
50.10
57.25
60000
ScaMult Instraction
53.78
63.99
72.72
38.81
50.12
58.95
63:26
71.74
79.52
50.85
58.47
66.46
60.72
Table 3: Fine-grained classification performance on MAPLE (Zhang ct al., 2023b),
Blue : In-domain evaluation
datasets. Red: Cross-domain evaluation datasets.
Gray Better than all baselines. Bold: The best score.
tunes the whole architecture: SciMult- uses neither of the two strategies. All variants utilize hard negatives introduced in subsection 3.4 during contrastive learning. Hyperparameter configurations of SciMult can be found in Appendix C.
here: (I) The Bi-Encoder setting (abbreviated "BiEne" in Table 4) follows our practice in fine- grained classification where calculate sim, (p, 1) given the description of each I (without relying on any training data after LM pre-training). (2) The Linear Classifier setting (abbreviated to "Linear" in Table 4) follows the practice in Cohan et al. (2020), which takes the embedding vector E(p) as the input feature of each paper and trains a linear SVM for classification. This setting requires labeled training and validation data after LM pre-training but no longer needs label descriptions.
4.3 Fine-grained Classification
Following Zhang et al. (2023b), we consider a sim- ple heuristic when ranking all candidate labels: la- bels whose name appears in the query paper p should be ranked higher than those not appear- ing in p. In other words, we first rank all labels / according to sim(p, 1) and then reorder all labels appearing in p in front of all other labels. Here. we evaluate model performance using Recall @ (k: = 20, 50, 100), i.e., the proportion of gold la- bels found in the top-k retrieved results to all gold labels of a paper.
The results of both settings are shown in Table 4. We find that, on average, SciMult variants can al- ways beat all baselines with more pronounced out- bf- advantage (i.e., the "BiEnc" setting). On the other hand, when further fine-tuned with linear classifier, most pre-trained models perform very similarly with better performance achieved by the multi-task models (SPECTER 2.0 and our SciMult variants), indicating the advantage of multi-task training for graseeer classification.
The results are shown in Table 3. We also con- duct experiments without using this heuristic, the results of which are shown in Table A3, where all models perform consistently worse. From Ta- ble 3, we observe that: (1) SciMul C-MAAxxen and SciMult tolutt-Inne Instrructionn outperform all baselines on all four datasets. The other three SciMult vari- ants have significant advantages over all baselines on in-domain evaluation datasets, but their edges on cross-domain datasets are not consistent. (2) Com- paring among all the SciMult variants, SciMult- MHAExpert is always the best on cross-domain datasets, indicating its generalizability to unseen label spaces.
4.5 Link Prediction
In the link prediction task, we also consider two settings: retrieval and reranking.
For the retrieval setting, given a query paper pQ= the task is to find all candidate papers pC that are linked to PQ (via the relation "Cite", "Co-cite", etc. ) from the whole dataset. The link prediction per- formance of compared methods under the retrieval setting is shown in Table 5, where the evaluation metrics are Recall @ 20. 50, and 100. As expected, SciMult variants significantly outperform all base- lines in almost all cases.
4.4 Coarse-grained Classification
Following Cohan et al. (2020) and Ostendorff et al. (2022), we directly predict the most likely coarse label of each paper and use Macro-F1 as the evalu- ation metric. Two different settings are considered
As for the cross-domain PMC-Patients dataset (Zhao et al., 2022), besides evaluating the zero- shot retrieval performance (Table 5), we also test

---

Claassficacton
Guass--gaained
Linear
Fiells
Bilaa
Linear
al., 2021
Dienses
Bllbac
Avemage
Prediction
Lak
[Petrieva3)
R
SciDaes (Clakan et 2020)
R::100
R::20
R,550
Coecle
RR 100
RR-21
PMIC-Pulints (2022)
Putient-n--
R::50
R0:100
Average
BMIS
BMSS
12:31
24.06
10.49
14.15
11.95
18.01
48.00
53.15
34:85
SOBERT
79.21
1881
90.71
12.25
47.85
SOBERT
2.45
3.33
5.32
5.36
7,87
9.18
17.15
20.17
22.90
10:39
50.51
16.26
09.11
9.01
19.36
5.45
6:20
11:13
9.19
12.65
15.91
8.20
10.46
12.50
10:42
SPECTER
62.07
55.09
96.4
57.70
T0.40
SPECTER
21.59
31.58
40.02
31.31
43.41
52.90
27.02
34,52
$3.43
35,96
PURMSBBERT
77.4
10.67
$1.45
1.14
42.65
PERMoBEERT
4,42
6.45
8.4
8.42
11.21
13.77
21
24.48
27.18
13.96
Lanchiiee
92.51
14.00
56.75
1.25
19.50
2.58
540
4.00
500
6.35
7.60
11.37
13.06
2.81
BioLinkBERT
76.21
7.37
$5.50
BRLINBERTT
12.86
OAG-BERT
SANCL
60.12
58.67
72.20
36.57
48.50
39.30
40.74
SPECTER
82,66'
50.00
56.03
SPECTER
23.53
34.47
43.54
46.46
30.36
35.10
36.38
11.31
65.63
38.01
75.54
46.33
36.40
52.05
50.20
45.49
SciMal1-MMAn -MMAAEs-
67.38
66.72
16.31
Scimtul -MHAAEgen
$4.31
40.31
59.71
54.16
64.50
51.25
58.25
50.46
FFNNEppt
32:25
69.25
$8.07
63.57
15.79
36.34
51.31
61.97
38.12
$2:49
52.91
63.36
52.31
59.05
50.87
$2.48
69 02
58.62
06.53
16.67
Scamul --irrii
54.16
40.27
60.52
17.00
52.21
82.85
43.54
52.25
59.00
57.17
Sc3N.It Insorctionn
$2.67
69.79
58.71
63.82
76.24
SctPult Instrcttion
53.58
47.94
58.37
36.22
99.95
04.20
43.37
52.10
58.72
48.95
Table 4: Coarse-grained classification per- formance (Macro-F1 scores) on SciDocs and PMC-Patients (Zhao et al., 2022) under the retrieval setting. (Cohan et al., 2020). Scores with f are re- ported in Cohan et al. (2020), Ostendorff et al. (2022), and Singh et al. (2022).
Table 5: Link prediction performance on SeiDocs (Cohan et al., 2020)
Pationt-to- Retrieval
MRR
P0
1DCG
RØ1K
candidate is not binary and ranges from to 5. As result, the model needs to rank candidate papers with higher scores in front of those with lower scores, and we use nDCG and nDCG@k (k 5, 10) as the metrics. (MAP cannot be used in Recommendation because relevance is not binary.) The performance is demonstrated in Table 7.
DPR (SciMult MHHAEpprrt
64.44
22.12
28.62
69.09
DPR (NAR
42.96
16
19.51
63.40
DPR (SPECTER)
46.41
15.99
19.70
57.98
DPR (BioLinkBERT)
40.89
15:33
18:47
62.44
BM25
48.22
9.97
15.28
30.64
Conteever
15.03
3.41
4,62
16.74
SemBEET
10.58
3.53
Pationt--a- Pationl Retrieval
MRR
10
nDCG9910 10
Rø1K
(SciMult- MHAAEpprr
25.35
22.39
From Table 7, we observe that: (1) On Sci- Does, SciMult t variants outperform SPECTER in most cases. Note that for the link prediction task, SciMult uses exactly the same pre-training data (including hard negatives) as SPECTER (Cohan et al., 2020). Therefore, this observation implies that exploiting pre-training data from other tasks can benefit the link prediction performance, which validates our motivation to develop a multi-task lcarning framework. (2) On SciDocs, SciMult t vari- ants can rarely outperform SciNCL and SPECTER 2.0 (but the gaps are not evident). This is possi- bly because SciNCL uses more complicated hard negative sampling strategies, and SPECTER 2.0 ex- ploits more diverse pre-traiming data and tasks. In fact, Ostendorff et al. (2022) have pointed out the data leakage issue that 40.5% of SeiDoes papers appear in the pre-training data. This motivates to incorporate the cross-domain Recommendation dataset, which SciMult t- Vanilla and SciMult- MHAExpert consistently outperform all baselines.
BM25
22,86
4.67
18.29
69.66
DPR (BooLInHERRT)
21.20
5.59
18.06
WI.49
DPR (PoRMccDERT)
19.37
5.05
16.30
79.35
DPP (SPECTER)
15.08
3.79
12.27
73.01
Conmnever SemBERT
10.50
2.24
9,01
52.64
5.28
1.17
3.88
37.55
Table 6: Comparison between SciMul MiMMuut-MMMn t-Apee and models on the leaderboard of PMC-Patients (Zhao et al., 2022) Patient-to-Artin Retrieval and Patient-to- Patient Retrieval tasks. MrMMult-MHAExpert achieves the new state-of-the -ftte--rt-h---ttteee performance.
supervised SciMult by further fine-tuning it on the provided training data. Specifically, we pick SciMult-MHAExee (because of its overall good performance according to Table 9) and use DPR (Karpukhin et al., 2020) to fine-tune it. The compar- ison between DPR(SciMult MrHMMMuuttHHAEperrr and ex- isting models on the leaderboard of PMCCPatients* is shown in Table 6. Our model outperforms all ex- isting models and achieves the new sfate-of-the-an art,
For the reranking setting, we follow the original evaluation protocol of SciDocs (Cohan et al., 2020): For each query paper PQ. a small set of candidate papers {pC1, ..., is given, which contains up to positives and 25 random negatives. The task aims to rank the positives higher than the negatives, so we use MAP and nDCG as evaluation metrics. The Recommendation dataset (Kanakia et al., 2019) has similar format, except that the score of each
4.6 Search
We use nDCG @ 10, the primary metric of BEIR (Thakur et al., 2021), to measure models' perfor- mance on the search task. The results are shown in Table 8. We find that, on the four cross-domain evaluation datasets and on average, four SciMult variants can outperform all baselines. In particular, on NFCorpus, only SciMult can beat BM25, while
Phttps://pmec http::///mmmmppppss...tt

---

SciDoes (Cohan et al., 2020)
Kanakia et (2019)
1irk Prediction (Berarking)
Covien
Ca-read
Cile
Coeeile
Recommendation
MAP
BDCG
nDCG
MAP
BDCG
MAP
aDCG
mDCGs5
nDCC@10
RDCG
Average
Citeomatic (Bhagavttula et al., 2018)
$1.1
90.21
80.51
90,21
56.3
94.1
81.41
92.6'
Kanskia (2019)
83,881
K7.71
93.59'
ScIBERT (Beltagy et 2019)
50.77
73.11
47.31
71.1"
48.3"
71.77
49.71
72.6'
77.17
82.49
90.85
66.3 66.86
SemBERT (Reimers and Gurevych, 2019)
68.21
K3.31
64.88
81.31
03.51
81.6
66.41
82.88
76.75
81,49
90.80
76.45
SPECTER (Cohan 2020)
$3.6'
91.51
92.47
$8.3"
94,97
94.88
PobMedRERT (Gi ol., 2021)
59.43
78.25
55.59
75.63
51.81
73.43
58.19
77
77.30
82.21
91.09
71.97
LinkBERT (Yzsunaga al., 2022)
44.21
67.76
41.04
65.31
39.33
63.91
42.84
67.18
76.10
80.89
90.47
61.73
BioLinkBERT (Yasumga et al., 2022)
56.46
76.36
50.76
72. 18
47.73
70.55
52.94
74:44
77.02
81.78
90.73
68.
OAG-BERT (1.uu et al., 2022)
64.61
81.50
60.13
78.65
57.35
77.60
62.47
80.92
76.75
82.12
90.96
73.91
SeiNCI (f 2022)
92.31
96.41
34
91.45
SPECTER (Singh et al., 2022)
85.18|
92.27
86.951
93.53
92.23
96.84
91.13
96.261
86.05
89.12
94.59
91.29
Seimilt-Vimilla
83:99
91.68
$6.66
93.67
91.37
96.26
91.50
96.45
87.32
89,32
94,88
91.19
83.92
91.60
93.55
92.58
96.92
91.47
96,
86.68
89.45
94.77
91.25
ScoM.Lt-FFNNxpeer PFNEppt
83:23
91.25
85.61
93:20
93:77
91.42
90.39
95.94
55.75
88.45
94,29
90.85
SoiM.1 L-Profis
83.43
94.28
86.05
94.66
91:13
Sc.Hu11
82.13
90.88
84.14
92.36
92.63
96.91
59.27
95.43
86.49
88.81
94.51
90.32
Table 7: Link prediction performance SciDoes (Cohan et al., 2020) and Recommendation (Kanakia et al., 2019)
under the reranking setting. Scores with i are reported in Cohan et al. (2020). Ostendorff et al. (2022), and Singh et (2022). Scores with $ are calculated from the model output released by Kanakia et al. (2019).
Scilkephival (Singh at. 2022)
BEIR (Thakur et al., 2021)
Search
Scarch
(2002)
TREC-COVID
(2121)
TREC-COVID
(2021)
SciFact
(2020)
NF Carppp
(2016)
MIlARspert PENExpper
Prefte
Instruction
&DCG
aDCG@11
aDCG010
aDOG 10
aDCGG
Avcrage
Clveifization
(Fisc)
+3.15%
.0.5%
-0.51%
+0.6550
RM2S
73.47
55.85
$7.79
65.65
30.00
56.55
Tuble 3
SaBERT
71.39
40.95
0.58
21.84
20.74
9:40
Clasitication (Crora))
SPECTER
73.42
49.74
15.83
47.07
+1 1.005
+0.3355
+1.0%%
+0.95
PVAMedBERT
70777
45:25
7.56
0.30
1.09
25:00
Table
LiaLBRRT
71,65
52.45
2.28
0.49
1.73
25.73
Lisk Prediction
BidLiAABER
71.18
36,01
3.17
0.12
1.988
22,29
(Remeval)
+1.05
+4.0115
+3.45%
HUMMS
OAG-SERT
72.17
55.09
7.11
18:35
8:48
52.24
Tible
SANCL
73.75
13.50
$6.51
52.16
76,221
Link Prodiction
(Revasking) Tible 7
+0.07%
-0.37%
-0.00%
-0.5%
SctPule: Vanilla
70.76
SeiPul E-MMiAFagen
26.
56
71.18
20.67
90.79
67.08
ScsPut:: HINElspeeT
76.12
12.32
52.15
63.57
27.48
60.31
Seorch Tible 8
40.869
-9.25%
-1.25%
-3.328
SeiPul P-Poofi
76.55
$2.83
88.15
20.30
30.00
65.88
SciPull E-lestratiinn 75.86
$3.59
61.05
70.62
30.25
64.27
Table 9: Relative performance change of different SciMult variants in comparison with SciMult- Vanilla in terms of the aver- age evaluation metric.
Table 8: Search performance SeiRepEEval-Saarch (Singh al., 2022), TREC-COVID (Voorhees et al., 2021), SeiFact (Wadden et al., 2020), and NFCorpus (Boteva et al., 2016). The score with t is reported in Singh et al. (2022).
all baseline LMs are lacking by clear margin.
conduct more analysis through controlled exper- iments, which can be found in Appendix E. To briefly summarize, we observe that: (1) Using hard negatives in multi-task contrastive learning helps produce higher-quality text representations in gen- eral and benefit all tasks. (2) When pre-training non- Vanilla variants, warming up the LM by train- ing a Vanilla variant during initial steps yields bet- ter performance, (3) Using some other reasonable instructions during inference does not significantly affect model performance. (4) Although label def- initions are used for paper classification during pre-training and are beneficial the classification performance during inference, our model can still outperform baselines in classification by taking la- bel names as the only input.
4.7 Overall Analysis
To summarize, in Tables 3. 4. 5, and 8, all SciMult variants except SciMult-FFNEE FFFNxxeett can always beat all baselines in terms of the average metric. Meanwhile. to systematically examine whether our proposed techniques can mitigate task interfer- ence, we calculate the relative performance change of the four non-Varn Vanilla variants in comparison with SciMul t- Vanilla in each task. Table 9 shows the results. We observe that SciMult-MHAN proves SciMult- Vanilla across all tasks, which im- plics that the MoE architecture with task-specific MHA sub-layers effectively overcome task inter- ference during multi-task pre-training. By contrast, other proposed techniques are advantageous a subset of tasks, such as coarse-grained classifica- tion and link prediction under the retrieval setting.
im-
5 Related Work
Scientific Literature Understanding. Recent LMs pre-trained on domain-specific scientific texts,
To validate the design choices of SciMult, we

---

from Transformer-based ones such as SciBERT (Beltagy al., 2019), BioBERT (Lee al., 2020), ChemBERT (Guo et al., 2021), and PubMed- BERT (Gu et al., 2021) to GPT-based ones such as SciGPT2 (Luu et al., 2021) and BioGPT (Luo et al., 2022), aim to learn high-quality contextualized text representations for scientific literature understand- ing. Subsequent studies have utilized these LMs to fine-grained paper classification (Zhang et al.. 2022, 2023a), cite-worthiness detection (Wright and Augenstein, 2021), scientific claim verification (Wadden et al., 2020), and so on.
(Houlsby et al., 2019) and control codes (Keskar et al., 2019) to produce task-aware paper repre- sentations. We believe our study is orthogonal to Singh et al. (2022) as the techniques considered by us, including the Mixture-of-Exxper Transformer (Fedus et al., 2022) and instruction tuning (Wei et al., 2022), are distinct from theirs.
Conclusions
In this work. we propose to pre-train scientific LMs via multi-task contrastive learning. To mit- igate task interference, we adopt two strategies: Task-aware specialization considers a Mixture-of- Experts Transformer architecture so that each task has its unique components, while instruction tun- ing relies on task-specific instructions to produce task-aware text representations. Extensive exper- iments on a comprehensive collection of bench- mark datasets demonstrate the advantages of our models against competitive scientific LMs in ex- treme multi-label classification, link prediction, and search. In particular, we achieve the new state-of- ofate-of-the-art performance on the PMC-Patients leaderboard. We also show that task-specific MHA sub-layer are beneficial the model performance across all examined tasks, whereas the benefit of other proposed techniques is not consistent. Further analysis validates some of our design choices, such as hard negative mining in extreme classification.
Besides text information, metadata associated with scientific papers also broadly considered. For example, Citeomatic (Bhagavatula et al., 201 88,, SPECTER (Cohan al., 2020), BioLinkBERT (Yasunaga et al., 2022), and SciNCL (Ostendorff et al., 2022) leverage citation links between papers; OAG-BERT (Liu et al., 2022) models venues, au- thors, ffelds-o-sstdyy and affiliations during LM pre-training; S2AND (Subramanian et al., 1021) further utilizes year, email, and position informa- tion for author name disambiguation. Nevertheless, all aforementioned models either consider typical LM pre-trane pre-training tasks (e.g., MLM and NSP) only or focus on one additional task during model training (e.g., citation prediction). In comparison, SciMult exploits data from heterogeneous sources and pro- poses a multi-task learning framework that can be applied to a wide range of tasks.
Contrastive Learning and Multi-task Learning in the Scientific Domain. SPECTER (Cohan et al., 2020) and SciNCL (Ostendorff et al., 2022) are pioneering studies on using contrastive learning to enhance scientific LMs. They propose to derive positive and negative contrastive pairs from cita- tion triplets and demonstrate the power of mining hard negatives. MICoL (Zhang et al., 2022) and CitationSum (Luo et al., 2023) adopt contrastive learning to multi-label classification and summa- rization of scientific papers, respectively. As for multi-task learning, Luan et al. (2018) propose a multi-task scientific knowledge graph construction framework by jointly identifying entities, relations, and coreference; Wang et al. (2019) treat multiple biomedical named entity recognition datasets (with different types of entities annotated) multiple tasks so that they can mutually benefit each other. However, these studies do not have specific designs to tackle task interference. To the best of our knowl- edge, the recent work by Singh et al. (2022) is the most relevant one to SciMult, which pre-trains a scientific LM on various tasks and uses adapters
Limitations
In comparison with previous studies on instruction tuning considering tens of (Asai et al., 2023) to thousands of (Wang et al., 2022) tasks, we do not explore that many different scientific literature un- derstanding tasks. As result, we may not unleash the power of instruction tuning the utmost extent. It is of our interest to collect more datasets in the scientific domain, such as author name disambigua- tion (Subramanian al., 2021) and biomedical question answering (Jin et al., 2019), and then ex- plore whether the instruction tuning model trained on more tasks can mitigate task inference and sup- port zero-shot transfer to new tasks. Also, our study focuses on the Bi-Encoder architecture only. It would be meaningful to investigate other types of architectures, such Cross-Encoders and late- interaction models (Humeau et al., 2020; Khattab and Zaharia, 2020), and study how to apply task- aware specialization and instruction tuning to them.

---

References
Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. Domain-specific lan- guage model pretraining for biomedical natural lan- guage processing. ACM Transactions on Computing for Healthcare, 3(1):1-23
Waleed Ammar, Dirk Groeneveld, Chandra Bhagavat- ula. Iz Beltagy, Miles Crawford, Doug Downey, Ja- Dunkelberger, Ahmed Elgohary, Sergey Feldman, Vu Ha, et al. 2018. Construction of the literature graph in semantic scholar. In NAACL-HLT pages 84-91.
Jiang Guo, A Santiago Ibanez-Lopez. Hanyu Gao, Vic- tor Quach, Connor W Coley, Klavs Jensen, and Regina Barzilay. 2021. Automated chemical reaction extraction from scientific literature. Journal of Chem- ical Information and Modeling, 62(9):2035-2045.
Akari Asai, Timo Schick, Patrick S. H. Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannanch Hajishirzi, and Wen-tau Yih. 2023, Task-aware re- trieval with instructions. pages 3650-3675.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: pretrained language model for scientific text. In EMNLP' 19, pages 3615-3620.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficien transfer learning for nip. ICML' 19, pages 2790-2799.
Chandra Bhagavatula, Sergey Feldman, Russell Power, and Waleed Ammar. 2018. Content-based citation recommendation. In NAACL-HLI' 18, pages 238- 251.
Samuel Humeau, Kurt Shuster, Maric-Anne Lachaux, and Jason Weston. 2020. Poly-encoders: Architec- tures and pre-training strategies for fast and accurate multi-sentence scoring. In TCLR'20.
Vera Boteva, Demian Gholipour, Artem Sokolov, and Stefan Riezler. 2016. A full-text learning to rank dataset for medical information retrieval. In FCIR' '6, pages 716-722.
Ching Jin, Yifang Ma. and Brian Uzzi. 2021. Scien- tific prizes and the extraordinary growth of scientific topics. Nature Communicationss 12(1):5619.
Payal Chandak, Kexin Huang, and Marinka Zitnik. 2023. Building a knowledge graph enable pre- cision medicine. Scientific Data, 10(1):67.
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. Pubmedqa: A dataset for biomedical research question answering. In EMNLP' 19, pages 2567-2577
Wei-Cheng Chang, Hsiang-Fu Yu, Kai Zhong, Yiming Yang, and Inderjit Dhillon. 2020. Taming pre- trained transformers for extreme multi-label text clas- sification. In KDD'20. pages 3163-3171.
Anshul Kanakia, Zhihong Shen, Darrin Eide, and Kuansan Wang. 2019, A scalable hybrid research paper recommender system for microsoft academic. In 79, pages 2893-2899.
Hao Cheng, Hao Fang, Xiaodong Liu, and Jianfeng Gao. 2023. Task-aware specialization for efficient and robust dense retrieval for open-domain question answering. pages 1864-1875.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In EMNLP'20, pages 6769-6781.
Hyung Won Chung, Hou, Shayne Longpre, Bar- ret Zoph. Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling insruuctionfinettuee language models. arXiv preprint arXiv:22 11416.
Nitish Shirish Keskar, Bryan McCann, Lav R Varshney. Caiming Xiong, and Richard Socher. 2019. Ctrl: A conditional transformer language model for control- lable generation. arXiv preprint wcxvv:1909.0085..
Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S Weld. 2020. Specter: Document-level representation learning using citation-informed transformers. ACI. 20, pages 2270-2282
Omar Khattab and Matei Zaharia. 2020. Colbert: Effi- cient and effective passage search via contextualized late interaction over bert. In SIGIR 20, pages 39-48.
Margaret H Coletti and Howard I. Bleich. 2001. Med- ical subject headings used to search the biomedical literature. JAMIA, 8(4):317-323.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4): 1234-1200.
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu. Orhan Firat, et al. 2022.
Glam: Efficient scaling of language models with mixture-of-expprts. In /CML'22 22, pages 5547-5569.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuninge Optimizing continuous prompts for generation. In ACI,21, pages 4582-4597.
William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling trillion parameter models with simple and efficient sparsity. JMLR, 23( 23(1):5232-5270
Zihang Lin, Yian Yin, Lu Liu, and Dashun Wang. 2023. Sciscinet: large-scale open data lake for the science of science research. Scientific Data, 10( 10115.15.

---

Jingzhou Liu, Wei-Cheng Chang. Yuexin Wu, and Yim- ing Yang. 2017. Deep learning for extreme multi- label text classification. In SIGIR 17, pages 1 11-124.
for improving large-scale mesh indexing. Bioinfor- matics, 32(12))170-19.
Vinodkumar Prabhakaran, William L. Hamilton, Dan McFarland, and Dan Jurafsky. 2016. Predicting the rise and fall of scientific topics from trends in their rhetorical framing. In ACL' 76, pages 1170-1180.
Xiao Liu, Da Yin, Jingnan Zheng, Xingjian Zhang, Peng Zhang, Hongxia Yang. Yuxiao Dong, and Tang. 2022. Oag-bert: Towards a unified backbone lan- guage model for academic knowledge services. In KDD 22, pages 3418-3428.
Yashoteja Prabhu, Anil Kag. Shilpa Gopinath, Kunal Dahiya, Shrutendra Harsola, Rahul Agrawal, and Manik Varma. 2018. Extreme multi-label learning with label features for warm-start tagging, ranking & recommendation. In WSDM 18, pages 441-449.
Xiaodong Liu, Pengeheng He, Weizhu Chen, and Jian- feng Gao. 2019. Multi-task deep neural networks for natural language understanding. In ACL'19, pages 4496.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert Sentence embeddings using siamese bert-networks. In EMNLP'S 19, pages 3982-3992.
Nya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In ICLR"19. 19.
Yi Luan, Luheng He. Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-task identification of enti- ties, relations, and coreference for scientific knowl- edge graph construction. In 118 pages 3219- 3232.
Stephen E Robertson and Steve Walker. 1994. Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In SI- GIR 99, pages 232-241.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler. Arun Raja, Manan Dey, et al. 2022. Multitask prompted training enables zero-shot task generalization. In ICLR'22 22.
Renqian Luo. Liai Sun, Yingee Xia, Tao Qin, Sheng Zhang. Hoifung Poon, and Tie Yan Liu. 2022. Biogpt: generative pre-trained transformer for biomedical text generation and mining. Briefings in Bioinformatics, 23(6).
Zhihong Shen, Hao Ma, and Kuansan Wang. 2018. A web-scale system for scientific knowledge explo- ration. In ACI'18 System Demonstrations, pages 87-92.
Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. Citationsum: Citation-aware graph contrastive learning for scientific paper summarization. In WwW'25. 23, pages 1843-1852.
Amanpreet Singh, Mike D' Arcy, Arman Cohan, Doug Downey, and Sergey Feldman. 2022. Seirepeval: A multi-format benchmark for scientific document representations. arXiv preprint arXiv22211.13308
Kelvin Luu, Xinyi Wu, Rik Koncel-Kedziongs Kyle Lo. Isabel Cachola, and Noah A Smith. 2021. Ex- plaining relationships between scientific documents. In ACI,21, pages 2130-2144.
Kaixin Ma. Hao Cheng, Yu Zhang, Xiaodong Liu, Eric Nyberg, and Jianfeng Gao. 2023. Chain-of-skills: configurable model open-domain question an- swering. In ACL 23, pages 1599-1618.
Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Dar- rin Eide, Bo-June Hsu, and Kuansan Wang. 2015. An overview of microsoft academic service (mas) and applications. In WWW"15 Companion, pages 243-246.
Aakanksha Naik, Sravanthi Parasa, Sergey Feldman, Lucy Wang, and Tom Hope. 2022. Literature- augmented clinical outcome prediction. Findings of NAACL'22, pages 438 453.
Shivashankar Subramanian, Daniel King, Doug Downey, and Sergey Feldman. 2021. S2and: A benchmark and evaluation system for author name disambiguation. JCDL'21, pages 170-179.
Malte Ostendorff, Nils Rethmeier, Isabelle Augenstein, Bela Gipp. and Georg Rehm. 2022. Neighborhood contrastive learning for scientific document repre- sentations with citation embeddings. In EMNN.P'22, pages 11670-1 1688.
Jie Tang, Jing Zhang, Limin Yao, Juanzi Li, Li ZiZang and Zhong Su. 2008. Arnetminer: extraction and mining of academic social networks. KDD '08. pages 990-998.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models follow instruc- tions with human feedback. In NewrlPS pages 27730-27744.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Ab- hishek Srivastava, and Iryna Gurevych. 2021. Beir: A heterogeneous benchmark for zero-shot evalua- tion of information retrieval models. In N) Datasets and Benchmarks Track.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS'17, pages 5998-6008.
Shengwen Peng, Ronghui You, Hongning Wang. Chengxiang Zhai, Hiroshi Mamitsuka, and Shanfeng Zhu. 2016. Deepmesh: deep semantic representation

---

Ellen Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fure Fushmer. William R Hersh, Kyle Lo, Kirk Roberts, lan Soboroff, and Lucy Lu Wang. 2021. Trec-covid: constructing pandemic information re- trieval test collection. SIGIR Forum, 14..):1-12.
Yu Zhang, Bowen Jin, Qi Zhu, Yu Meng, and Jiawei Han. 2023b. The effect of metadata on scientific literature tagging: A cross-field cross-model study. In WwW pages 1626-1637.
Yu Zhang, Zhihong Shen, Yuxiao Dong, Kuansan Wang, and Jiawei Han. 2021. Match: Metadata-aware text classification in large hierarchy. In WWW221, pages 3246 3257.
David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction; Verifying scientific claims. In EMNLPP'0, pages 7534-7550.
Zhang, Zhihong Shen, Chieh-Han Wu, Boya Xie. Junheng Hao. Yc-Yi Wang. Kuansan Wang, and Ji- awci Han. 2022. Metadata-induced contrastive leam- ing for zero-shot multi-label text classification. WWW'22, pages 3162-3173.
Xuan Wang, Yu Zhang, Xiang Ren, Yuhao Zhang, Marinka Zitmik, Jingbo Shang, Curtis Langlotz, and Jiawei Han. 2019. Cross-type biomedical named en- tity recognition with deep multi-task learning. Bioin- formatics, 35(10)
7hengyun Zhao, Qiao Fangyuan Chen, Tuorui Peng, and Sheng Yu. 2022. Pme-patients, large-scale dataset of patient summaries and relations for bench- marking cetrieval-based clinical decision support sys- tems. arXiv preprint arXiv:2202. 13876
Yizhong Wang, Swaroop Mishra, Pegah Alipoormo- labashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, An- jana Arunkumar, David Stap, et al. 2022. Super- naturalinstructions: Generalization via declarative in- structions on 1600+ nlp tasks. In EMNLP'22, pages 5085-5109.
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew Dai, Quoe Le, James Laudon, ct al. 2022. Mixture-of-experts with expert choice routing. In NeurlPS'22. 22, pages 7103- 7114.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2022. Finetuned language models zero-shot learners. In ICLR'22 22.
Dustin Wright and Isabelle Augenstein. 2021. Cite- worth: Cite worthiness detection for improved scien- tific document understanding. In Findings of ACI,21, pages 1897-
Wenhan Xiong, Xiang Li. Srini Iyer, Jingfei Du, Patrick Lewis, William Yang Wang, Yashar Mehdad, Scott Yih, Scbastian Riedel, Douwe Kicla, et al. 2021. An- swering complex open- poen-domain questions with multi- hop dense retrieval. In ICLR'21.
Guangxu Xun, Kishlay Jha, Ye Yuan, Yaqing Wang, and Aidong Zhang. 2019. Meshprobenet: self- attentive probe net for mesh indexing. Bioinformat- ics, 3399333399933.
Michihiro Yasunaga, Jure Leskovec, and Percy Liang. 2022. Linkbert: Pretraining language models with document links. In ACL 22, pages 8003-8016.
Chenchen Ye, Linhai Zhang, Yulan He, Deyu Zhou, and Jie Wu. 2021. Beyond text: Incorporating metadata label structure for multi-label document classifi- cation using heterogeneous graphs. In EMNLP'2I, pages 1171.
Yian Yin, Yuxiao Dong, Kuansan Wang, Dashun Wang, and Benjamin Jones. 2022. Public use and pub- lic funding of science. Nature Human Behaviour. 6(10):1344 1350.
Yu Zhang, Bowen Jin, Xiusi Chen, Yanzhen Shen, Yunyi Zhang. Yu Meng, and Jiawei Han. 2023a. Weakly supervised multi-label classification of full-text sci- entifie papers. In KDD'23 23, pages 3458-3469.

---

A Dataset Details
MAPLE (Chemistry-MeSII) The candidate la- bel space of Chemistry-MeSH for evaluation is the same as that of Biology-MeSH and Medicine- MeSH during pre-training, which is a superset of the original label space of ChemistryMMeHH
Statistics of the pre-training and evaluation datasets are summarized in Table Al and Table A2, respec- tively.
We lowercase the text all datasets.
MAPLE (Geography and Psychology). We di- rectly use the original data. Since we do not have la- bel definitions in these two fields, only label names are used as the text information.
A.1 Classification
The MAPLE benchmark (Zhang et al., 2023b) is available at https: thub wwt.//ooooammiiiiooe imanhua /MAPLE. The SciDocs benchmark (Cohan et al., 2020) is available at https: htgp::/ com/all enai/scidocs.
SciDocs (MAG Fields and MeSH Diseases). Un- der the Linear Classifier setting, we directly use the original data and their train-val-test split. Under the Bi-Encoder setting. because no training and val- idation samples are needed, we merge the training, validation, and testing sets for evaluation.
MAPLE (CS-Journal). Because label definitions are not released in MAPLE, we adopt the defini- tions of 15,808 CS fields-of-study in Zhang et al. (2022). This label space is not identical to the orig- inal label space of CS-Journal (with 15,540 labels), and we remove papers that do not have any relevant labels in the adopted label space. Because some general labels (e.g., "Machine Learning") are rel- evant to a large proportion of papers in the original dataset, we perform downsampling to ensure each label appears in at most 100 positive (p. 1) pairs the pre-training data, In this way, the LM will not be overwhelmed by cases of general labels and can learn more semantics of specific labels (e.g., "Lagrangian Support Vector Machine").
A.2 Link Prediction
The SciDocs benchmark (Cohan et al., 2020) is available at
com/allenai /sci docs. The PMC-Patients dataset (Zhao et al., 2022) is available at https: thwb..t..tttggggg////////g/. wwwpwccmcmpmp//ccccc
The Recommendation dataset (Kanakia et al., 2019) is available at
Citation Tripless We directly use the data from http:://wwuginngfaae..cco/aaaaas httpa////icceeeeaaaateeeetetttte
MAPLE (Biology-MeSH and MeSi).ne-MeSII. We combine these two datasets and use shared label space of 30,194 McSH terms in the 2022 ver- sion of MeSH. The definition of each McSH term is its "Scope Note". Same as above, we perform downsampling to ensure each label appears in at most 100 positive (p,1) pairs in the pre-training data.
n. In each triplet (PQ, JPQ.PC+PPC ). PQ cites PC+ and PQ does not cite pC-. With probability of about 40%, PC- is a hard negative.
SciDoes (Co-view, Co-read, Cite, and Co-cite). Under the reranking setting, we directly use the original data. Under the retrieval setting, for "Cite" links, a query-candidate paper pair (PQ, Pc) is viewed as positive if PQ cites Pc according to the dataset; for "Co-cite" links, query-candidat, pair (PQ. Pc)) is viewed as positive if pQ and pc are co-cited by at least 10 papers in the dataset; for "Co-view" and "Co-read" links, we cannot derive version for retrieval because we do not know whether (PQ,Pc) is positive when PC is not in the reranking candidate pool of po in the original data. PMC-Patients. In Table 5, because no training and validation samples are needed, merge the train- ing, validation, and testing sets of the Patient-to- Patient Retrieval task for evaluation. In Table 6, compare SciMult with other models on the PMC- Patients leaderboard, we adopt the original train- val-test split.
MAPLE (Coarse). There are 19 fields in MAPLE, For each of the 18 non-CS fields, we sample 50,000 papers, label it as the corresponding coarse field (e.g., a paper p from the Mathematics field will form the laber-label pair (p. Mathematics)), and put it into the pre- training data. These data can train the LM to perform coarse-grained classification.
MAPLE (CS-Conferencee. The candidate label space of CS Conference for evaluation is the same that of CS-Jouaal for pre-training (i.e., 15,808 labels from Zhang et al. (2022)). Because this label space is not identical to the original label space of OS-Conference (with 13,613 labels), we remove papers that do not have any relevant labels in our adopted label space.

---

Dataset
#Queries
#Positive (Query, Candidate) Pairs
Classification
MAPLE (CS Joumal) (Zhang al., 20236)
358,447
1,242,855
MAPLE BBillogyMeSSH+ Medicine-M:SH) (Zhang et 20230)
1,345, 128
1,976,858
(Coarse) (Zhang et 20236)
Link Prediction
Citatioo Prediction Triplets (Cohan et al., 2021)
165,340
819,836
SciRepEvall-Seni Seurch (Singh et al., 2022)
528.497
620,033
Table A1: Statistics of Pre-training Data.
#Queries
#Candidates
Fine-grained Classi ficacionn
MAPLE (CX-Conference) (Zhang et al., 20236)
261,781
15,808
MAPLE (Chemistry-Most (Zhang 20236)
762.129
MAPLE (Geography) (Zhing el 20236)
73,883
3,265
(Psychology) (Zhang et 2023b)
372,954
7,641
Coarse-grai fication
SeiDoss (MAG Fields) (Cohan al., 2020)
25,001
19
SciDocs (MeHH Diseases) (Cohan al., 2020)
Link Prediction (Retrieval)
SeiDoes (Gite) (Cohan al., 2020)
92,211
142,009
SciDocs (Co (Coban 2020)
PMC-Patients (Patient-to-PPinn Po-Paie Retrieval, Zero-shot) (Zhao ot 2023)
100,327
155,151
PMC PCCCeassn (Patient to Article Retrieval, Supervised) (Zhan al., 2022)
5,959
1.413,087
PMC Patients (Patient Puiient Retrieval, Supervised) (Zhao al., 2022)
2,812
155,151
Link Prediction (Beranking)
SeiDoes (Cu-view) (Cohan et al., 2020)
1,000
teranking, 98 for each query om average
SeiDocs (Co-rcul) (Colan 2020)
1,000
teranking, 29.98 for each query average
SeiDees (Cite) (Cohan al., 2020)
1,000
remnking. 93 for each query on avernge
SciDocs (Coban 2020)
terunking. for query average
Recomnnndationn (Kanakia et al., 2019)
137
renanking. 16.28 for each query avernge
Search
SciRapEvall Seurch (Singh
2,637
teranking, 10,00 query average
TREC COVID in SciRepEval (Voorhees al., 2021)
50
reranking, 1386.36 for each query avernge
TREC-CO BEIR (Voorhees al., 2021)
SciFact (Wadken 2020)
1,109
5,183
NFCorpus (Boteva er al., 2016)
3,237
3,633
Table A2: Statistics of Evaluation Datasets.
Recommendation. In the original dataset, each paper only has its MAG ID (Sinha al., 2015), while its title and abstract are not included. We find the title and abstract of most papers a version the Microsoft Academic Graph downloaded in 2021. Those papers whose titles and abstracts are not found are removed from evaluation.
data, where treat all (q, p) pairs with positive score positive (g,p) pairs. The testing set of SciRepEval-Seume is utilized for in-domain evalu- ation.
TREC-COVID. We directly use the original test- ing set. the SciRepEval version, each query has multiple segments separated by [SEP].
A.3 Search
SciFact and NFCorpus. We directly use the origi- nal data. Because no training and validation sam- ples are needed, we merge the training, validation, and testing sets for evaluation.
The SciRepEval benchmark (Singh et al., 2022) is available at https: //githb.. http:/ttu....malleeeeeeeeeeeeeee com/allenai/s ci cirepeval. The BEIR benchmark (Thakur et al.,
2021) is available https://githhe com/bei r-cellar/beir.
B Baseline Details
SeiBERT (Beltagy et al., 2019) is an LM pre- trained on scientific text using masked language modeling (MLM) and next sentence prediction (NSP).
SeiRepEval-Search. The original data scores each query-paper pair (q.p) in the range of ( 14 ac- cording user tlick-through events from a schol- arly search engine. The training and validation sets SciRepEval-Serd Search are put into our pre-training
SentBERT (Reimers and Gurevych, 2019) is a

---

general-domain LM that leverages negative sam- pling fine-tune BERT for producing better sen- tence embeddings.
use the classification adapter; for link prediction. we use the proximity adapter, for search, we use the adhoc query adapter to encoder queries and the proximity adapter to encode candidate papers. For fine-grained classification, we test both the classification adapter and the proximity adapter, the latter of which achieves better performance on average, we choose the proximity adapter.
SPECTER (Cohan et al., 2020) uses paper cita- tions to generate positive and negative samples for contrastive fine-tuning of SciBERT.
PubMedBERT (Gu et al., 2021) a biomedical LM pre-trained on PubMed papers using MLM and NSP. We use the checkpoint pre-trained on abstracts rather than that on full texts because the former performs better on the majority of our evaluation tasks.
C Hyperparameter Configurations of SciMult
For pre-training, we use the AdamW optimizer (Loshchilov and Hutter, 2019) with (31, (2) (0.9, 0.999) and warm up the learning rate for the first 5% of the steps. We train the model for 20 epochs with peak learning rate of 3e-4 and weight decay 0.01. The training is on 48 V100 GPUs with fp16, and the batch size per GPU is 32. When pre-training the four non- Vanilla model variants, we adopt two-stage strategy: We first train SciMult t- Vanilla checkpoint to learn com- mon knowledge shared across different tasks. Then, starting from that checkpoint, leverage task- aware specialization or instruction tuning for tinual pre-training, expecting the model to learn task-specific skills.
LinkBERT and BioLinkBERT (Yasunaga et al., 2022) leverage Cross-Encoder architecture that concatenates two linked text segments to- gether and are trained through MLM and NSP on Wikipedia and PubMed, respectively.
OAG-BERT (Liu et al., 2022) is an entity- augmented scientific LM pre-trained on both aca- demic texts and their associated metadata entities (e.g,, venues, authors) through masked entity pre- diction.
SciNCL (Ostendorff' et al., 2022) advances the sampling strategy of SPECTER to create higher- quality positives and negatives for neighborhood contrastive learning.
D More Results on Fine-grained Classification
In this section, show the fine- ane-grained classifica- tion performance of compared methods on MAPLE (Zhang et al., 2023b) without the label name match- ing heuristic. The results are Table A3. We find that: (1) In comparison with the results in Table 3, after removing the label name matching heuristic, all models perform consistently worse. This obser- vation validates the effectiveness of the heuristic proposed in Zhang al. (2023b). (2) Most find- ings drawn from Table still hold in Table A3. For example, all SciMult variants can outperform all baselines in terms the average metric; SciMult- MHAExpert is always the best on cross-domain evaluation datasets.
SPECTER 2.0 (Singh al., 2022) is the succes- sor SPECTER pre-trained on much larger collection of citation prediction triplets and more diverse tasks from the SciRepEval benchmark (Singh et al., 2022). We adopt SPECTER 2.0- Adapters to generate task-specific embeddings for different tasks.
For all baselines, set the similarity func- tion si c) cos(g, e) where q and query and candidate embeddings, respectively, af- ter LM encoding. (Except for reranking tasks on SciDocs, where the evaluation code explicitly sets c) - || cll2.)
When using SentBERT and BERT, we take the average of all token embeddings to represent the entire input sequence because this leads to sig- nificantly better performance; when using other baselines above, take the [CLS] embedding. For SPECTER 2.0, we try to use the most proper variant in each task. To be specific, following Singh et al. (2022), for charse-grained classification, we
E
Analysis
In this section, we analyze some design choices of SciMult through controlled experiments.
E.1 Effect of Hard Negatives
We first demonstrate the contribution hard nega- tives in our contrastive learning framework. Since

---

MAPLE (Zang et al.. 20236)
Finerg Tinegrainee classi Fication
CS- CS-Collrnnce
Chemistry-MeSH
Gongraphyy
Psychology
R:
RG:100
Average
BM25 (Robettoon and Walker, 1991))
17:41
23.73
29.12
7.78
10:75
13:41
15.81
24.15
35.99
13,00
18.45
21.90
19.51
SciBERT (Bellagy al.,
3.63
1.72
3.48
5.84
SemBEKT (Reimers and Gureyych, 2019)
2,94
5.24
7.97
1.20
2.21
3.45
7.68
14.26
20.97
2,62
4.79
7.67
6.75
SPECTER (Cobann et tl., 2020)
16.28
24.50
$2.00
11.80
17.94
23.85
18.12
28.02
37.36
15.18
23.46
32.24
23.45
FRT
RIRT (Y et ol., 2022)
1.38
2,55
4.14
0.68
1.58
2.72
0.57
1.16
2.34
0.65
1.13
1.72
1.71
BioLinkBERT (Yammaga
2022)
1.02
3.05
0.46
0.87
1,40
0.68
0.14
0.39
0.79
OAG-BERT (Liu al., 2022)
2002)
2,61
4.24
6.09
1.72
2.68
3.74
1.96
3.22
4.75
0.79
1.31
1.97
2.92
SciNCL (Ostendortf al., 2022)
17:42
25.32
32.82
13.96
20.33
26.22
21.50
32.96
43.02
22.76
32.85
12.30
27.65
SPECTER 2.0 (Singh 2022)
56.85
ScaMMul Vanalla
31.52
46.04
55.61
26.96
39.63
50.37
32:20
40.96
59.12
30,02
20.45
50.31
42.09
SciMult- MeHMult-MHAAeperr
35.63
51.00
64.02
26.11
38.65
19.89
37,70
52.44
63.99
32,69
44,80
56.28
46.12
Sc3Mul L-ENExpee
22.83
34.93
41.37
55.03
27.95
ScaMult Prefix
38.42
52.58
64.51
22.67
34,88
45.95
28 28.25
43.00
56.55
26.09
38.48
49.71
41.76
SciMul
Table A3: Fine-grained classification performance on MAPLE (Zhang et al., 2023b) when the compared methods not use the "label name matching" heuristic.
Vanilla Hard Negative)
Vanilla (wlo Hard Negative)
Expert (w/ Warm-up)
MHA Expert (wlo Warm-up)
Classitication
Classitication
(Fine)
60.31
39,89
62.21
59.98
Clavsification
Clovolication
(Coame)
75.51
77.44
(Couse)
76.28
76.08
Link Prodiction
Link Prodiction
(Retricval)
4H.49
45,90
(Retrieva))
$0.46
47.59
Prediction
Prodiction
(Renanking)
(Reranking)
Sench
66.48
65.87
Soarch
67.85
62.34
Table A4: Average metrics of SciMult Vanilla with and without hard negatives in each evaluation task.
Table A5: Average metrics SciMul t- t-MHArxpert with different pre-training strategies in each evaluation task.
the benefit of using hard negatives in citation pre- diction has been reported in Cohan et al. (2020) and Ostendorff et (2022), we mainly show how hard negative mining in fine-grained classification (proposed in subsection 3.4) improves the perfor- mance. Table A4 shows the average metrics of SciMult Vanilll when trained with hard negatives of classification and without them (but still with hard negatives of the other tasks). We find that SciMult-Vanilly (with Hard Negative) outperforms SciMult- Vanilla (without Hard Negative) in most tasks, including not only classification but also link prediction and search. This observation indicates that the general quality of the learned text represen- tations is enhanced after utilizing hard negatives of classification.
SciMult-MHAEwt with an ablation version: We skip the warm-up stage and directly use Pub- MedBERT as the initial checkpoint to start the second stage. Table A5 demonstrates the perfor- mance of SciMul t-MHAExpert (with Warm-up) and SciMult-MHARR find that the warm-up strategy has a positive contri- bution across all tasks.
(without Warm-up). We
E.3 Effect of Instructions
We now explore the effect of using different instruc- tions during inference. As shown in Table 1, during pre-training, we use the following instruction for the link prediction task:
"Find pair of scientific papers that one paper cites the other."
E.2 Effect of the Pre-training Strategy
We call it the 'Cite" instruction. Different from the pre-training data in which links are always cita- tion links, the evaluation datasets consist of other link types. Therefore, we consider the following instructions:
As mentioned in Appendix C, when pre-training non- von-llnilla variants of SciMult, we follow two- stage strategy: We first warm up the LM by train- ing Vanilla variant and then apply task-aware specialization instruction tuning, hoping the first stage learns common knowledge and the sec- ond stage focuses on specifc skills. We now explore the effect of this strategy by comparing
"Find pair of scientific papers that are co-viewed frequently."
"Find a pair of scientific papers that

---

"Co-viev" Instruction
"Co-read" Instruction
"Cite" Ina truction
par with each CS CS-Conference
"Co-cite" Instruction
R:0:20
R#5
R@100
SPECTER 2.0 (Name Only)
48.14
53,93
58.98
Co-vlew
81.16
$1.47
82.13
$2.08
(Name Detinition)
Prediction
SciMul t- E-Mperrrrri (Name Ouly)
$2.57
61.27
67.84
Co-rad
MHHAEeen Delinition)
65.49
Prediction
Chemistry-Most MeSH
R
R050
RC 100
Cie
92.47
92.63
Only)
42.19
SPECTER 2.0 (Name-Defmiion)
36,17
43.06
48.26
Prediction
Oo-citz
$7.49
$3.05
89.27
$8.83
I-MAAApet (Name
46.47
SciMul:: MHAAExper DetintDeiitiion
29.41
50.92
29.59
Table A6: MAP scores of SciMult- Innttructnnt with different instructions the SciDocs (Cohan al., 2020) link prediction task under the reranking setting.
Table A8: Fine-grained classification performance of SPECTER 2.0 and SciMul t-MHAExpert when taking label names only or label namestdefnitions as input.
"Related" Instruction
Instruction
Instruction
and Psychology.
Link Prodiction
On CS-Conference and Chemistry-MeSH, we conduct additional experiments by removing la- bel definitions and letting the model use label names only during inference. The performance of MciMult-MHAEExprr and SPECTER 2.0 (the strongest baseline in Table 3) is demonstrated in Table A8. We can observe that, after removing la- bel definitions, although the performance of both SPECTER 2.0 and SciMul t--HA Axpert drops (which intuitive because we remove some useful information), SciMult-MHAExper consistently performs better than SPECTER 2.0. fact, even SPECTER 2.0 uses label definitions, SciMult- MHAExpert still outperforms SPECTER 2.0 with label names only.
(Ratriosal)
Link Prediction
90.32
90.31
90.35
Table A7: Average metrics of SciMul t- Instrucction with different instructions link prediction tasks.
co-read frequently." "
"Find pair of scientific papers that are co-cited frequently."
"Find a pair scientific papers that one paper is related to the other."
We call these four instructions "Co-view", "Co- read", "Co-cite", and "Related", respectively. Table A6 shows the MAP scores of SciMult- Instruction with different instructions in the Sci- Does (Cohan et al., 2020) link prediction task under the reranking setting. We find that the "Co-view" and "Co-read" instructions underperform the "Cite" and "Co-cite" instructions in most cases, even in Co-view and Co-read link prediction.
Table A7 shows the average metrics of SciMult- Instruction with different instructions in link pre- diction tasks. We find that the "Cite", "Co-cite", and "Related" instructions other.
E.4 Availability of Label Definitions in Classification
Although both label names and definitions are used for paper classification during the pre-training of SciMult, we would like to emphasize that label definitions are not required but optional during in- ference. For example, mentioned in Appendix A, the Geography and Psychology datasets do not have label definitions, thus classification on these two datasets relies on label names only. According to Table 3 and Table A3, SciMul M-HHEerrrrt out- performs all baselines consistently on Geography


Reviews:
[{"review_id": "yaIusL6DUD", "reviewer": "Reviewer_keSz", "summary": "", "questions": "A. Given a specific dataset (task), does jointly pre-training on multiple datasets from different tasks improve upon training SciMult only on the one dataset (task)? For example, in Table 3, if we only train SciMult on CS-Conference, how does it perform on CS-Conference compared to SciMult-Vanilla which is jointly pre-trained on multiple tasks?\n\nB. When training SciMult checkpoint, do you start from a random initialized Transformer or pre-trained ones such as SciBERT?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful review and feedback! We discuss the raised points as follows:\n\n1. **SciMult underperforming or performing on par with SPECTER 2.0 on SciDocs** \n\nFirst, we would like to mention the data leakage problem of SciDocs pointed out by Ostendorff et al. in [1]: “**40.5%** of SciDoc’s papers leaking into SPECTER’s training data”. This is also mentioned in our submission (Lines 1205-1210, Page 15). Because SPECTER, SciNCL, SPECTER 2.0, and SciMult all use SPECTER’s training data (i.e., the 819K citation prediction triplets) during pre-training, it becomes questionable how indicative the performance comparison can be if SciDocs is used for evaluation.\n\nBecause of this, we exploit some other public datasets (i.e., PMC-Patients and Recommendation) to evaluate the link prediction task. Indeed, **none** of the texts in the Patient-to-Patient task and only **0.2%** of the papers in the Patient-to-Article task in PMC-Patients can be found in SPECTER’s training data. On these datasets, SciMult-MHAExpert consistently outperforms SPECTER 2.0 and other baselines, which is more persuasive in our view. Moreover, in fine-grained classification and search tasks where SciDocs is not used, SciMult-MHAExpert outperforms all baselines in almost all cases, with SciRepEval-Search as the only exception. This also implies the generalizability of our proposed model. We will make this clearer in our revision.\n\n2. **Improvement upon training SciMult on only one dataset/task**\n\nThank you for the great suggestion. To answer this question, we pre-train SciMult on CS-Journal only for the classification task. (We do use “CS-Conference” because it is one of our evaluation datasets, while “CS-Journal” is in our pre-training data.) We adopt the SciMult-Vanilla architecture, which is equivalent to SciMult-MHAExpert when there is only one task/dataset. The performance comparison on the classification task is shown as follows:\n\n| | CS-Conf | CS-Conf | CS-Conf | Chem-MeSH | Chem-MeSH | Chem-MeSH | Geo | Geo | Geo | Psych | Psych | Psych | Average |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| | **R@20** | **R@50** | **R@100** | **R@20** | **R@50** | **R@100** | **R@20** | **R@50** | **R@100** | **R@20** | **R@50** | **R@100** | |\n| SciMult-Vanilla (CS-Journal Only) | 54.44 | 64.11 | 72.29 | 31.99 | 35.50 | 38.67 | 53.59 | 55.76 | 58.11 | 45.44 | 47.36 | 49.42 | 50.56 |\n| SciMult-Vanilla (All Data) | 53.40 | 64.70 | 74.09 | 39.78 | 51.31 | 59.75 | 62.08 | 70.65 | 77.79 | 50.42 | 56.58 | 63.17 | 60.31 |\n| SciMult-MHAExpert (All Data) | 54.02 | 65.49 | 75.07 | 39.41 | 50.92 | 59.59 | 65.94 | 75.01 | 81.93 | 51.77 | 59.55 | 67.86 | 62.21 |\n\nWe can observe that SciMult-Vanilla (CS-Journal Only) underperforms SciMult-Vanilla (All Data) and SciMult-MHAExpert (All Data) in all columns except R@20 on CS-Conference. This validates the motivation of multi-task learning that jointly training the model on multiple relevant tasks/datasets (even with the Vanilla architecture) can often outperform single-task learning. If we further devise architectures that can mitigate task interference (e.g., MHAExpert), the performance can be even higher. We will add this discussion in our revision.\n\n3. **Initialization of SciMult**\n\nSorry that we forgot to mention this in our submission. SciMult is initialized by the pre-trained PubMedBERT-abstract model (https://huggingface.co/microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract). We will add this in our revision.\n\n**References**\n\n[1] Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings. EMNLP 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper conducted extensive experiments on a comprehensive collection of benchmarks. The experimental results show that the proposed method SciMult outperforms state-of-the-art scientific language models under in-domain settings and cross-domain settings. \n\n2. The paper innovatively unifies different scientific literature understanding tasks into a contrastive learning framework.\n\n3. The paper is clearly written and easy to follow.", "reasons_to_reject": "1. According to Table 4 and Table A4, the proposed method underperforms or performs on par with SPECTER 2.0 on SciDocs (an in-domain dataset) when the evaluation setting aligns with SPECTER 2.0 (Linear and Reranking). This suggests that the performance improvement may be subject to the contrastive learning setting.", "questions_to_authors": "A. Given a specific dataset (task), does jointly pre-training on multiple datasets from different tasks improve upon training SciMult only on the one dataset (task)? For example, in Table 3, if we only train SciMult on CS-Conference, how does it perform on CS-Conference compared to SciMult-Vanilla which is jointly pre-trained on multiple tasks?\n\nB. When training SciMult checkpoint, do you start from a random initialized Transformer or pre-trained ones such as SciBERT?", "paper_topic_and_main_contributions": "The paper proposes SciMult, a multi-task contrastive learning framework, to jointly pretrain language models on datasets across multiple scientific literature understanding tasks (e.g., paper classification and literature search). The authors explore two different strategies, task-aware specialization, and instruction tuning, to mitigate interference between different tasks. The authors conduct comprehensive experiments on datasets across literature understanding tasks. Experimental results show that SciMult outperforms state-of-the-art language models and the effectiveness of task-aware specialization.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NlEgoNuaxj", "reviewer": "Reviewer_UpiD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful review and feedback! We discuss the raised points as follows:\n\n1. **Assumption on the availability of label names and definitions**\n\nThis is a great question. First, we would like to clarify that **label definitions** are **not required** but optional during inference. In fact, in our experiments, the Geography and Psychology datasets do not have label definitions at all, thus classification on these two datasets relies on label names only. This is mentioned in the appendix (Lines 1003-1006, Page 12), and we will make this clearer in our revision.\n\nFor model pre-training, because Zhang et al. [1] have released a large number of label definitions in CS and biomedicine domains, we leverage such information because we would like our pre-trained model to encode more semantic signals and to be more powerful. When applying our pre-trained model to downstream classification tasks, the model is flexible enough to take label names only. As shown in Table 3, on Geography and Psychology (where only label names are available), SciMult-MHAExpert outperforms all baselines consistently. On CS-Conference and Chemistry-MeSH, we conduct additional experiments by removing label definitions and let the model use label names only during inference. The results are as follows:\n\n| | CS-Conference | CS-Conference | CS-Conference | Chemistry-MeSH | Chemistry-MeSH | Chemistry-MeSH |\n| --- | --- | --- | --- | --- | --- | --- |\n| | **R@20** | **R@50** | **R@100** | **R@20** | **R@50** | **R@100** |\n| SPECTER 2.0 (Name Only) | 48.14 | 53.93 | 58.98 | 35.73 | 42.19 | 46.96 |\n| SPECTER 2.0 (Name+Definition) | 48.63 | 55.09 | 60.68 | 36.17 | 43.06 | 48.26 |\n| SciMult-MHAExpert (Name Only) | 52.57 | 61.27 | 67.84 | 37.83 | 46.47 | 52.59 |\n| SciMult-MHAExpert (Name+Definition) | 54.02 | 65.49 | 75.07 | 39.41 | 50.92 | 59.59 |\n\nWe can observe that: after removing label definitions (the “Name Only” rows), although the performance of both SPECTER 2.0 and our SciMult-MHAExpert drops  (which is intuitive because we remove some useful information), SciMult-MHAExpert still consistently outperforms SPECTER 2.0, **even if SPECTER 2.0 uses label definitions**, supporting that our model is a performant general backbone model for scientific understanding.\n\nSecond, the assumption of having **label names** in zero-shot text classification is widely accepted (e.g., [2, 3, 4]). After all, we need some information to characterize each label when annotated data is not available.\n\n2. **Technical novelty of SciMult**\n\nWe would like to clarify that the goal of this paper is not simply combining multi-task learning and contrastive learning. This is just a starting point of SciMult, fully covered by Section 3.1. In Sections 3.2 and 3.3, we make tremendous efforts to solve a general challenge in multi-task learning: task interference. Although the used techniques, including Mixture-of-Experts Transformers and instruction tuning, have been proposed in previous studies, they were not invented to mitigate task interference originally. In other words, we are using these techniques in a novel way to tackle a general problem. Then, in Section 3.4, we propose to sample hard negatives for each scientific literature understanding task. Note that we adopt different strategies for different tasks from the task nature perspective. In particular, we propose a new hard negative sampling strategy for paper classification based on citation/author/venue information, which is domain-specific and task-specific. This idea is not considered in previous studies as far as we know, and its effectiveness has been validated in Appendix F.1 and Table A5.\n\nMoreover, we believe a comprehensive benchmark study of scientific language models presented in this submission is also a contribution. Our study covers a wide range of tasks, compares various models in a controlled way, and provides insightful observations. In particular, our model achieves the new state-of-the-art performance on PMC-Patients, which is not included in the previous SciRepEval benchmark but has far-reaching applications in the real world, such as matching patients with clinical trials for precision health.\n\n**References**\n\n[1] Metadata-Induced Contrastive Learning for Zero-Shot Multi-Label Text Classification. WWW 2022.\n\n[2] Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach. EMNLP 2019.\n\n[3] X-Class: Text Classification with Extremely Weak Supervision. NAACL 2021.\n\n[4] TaxoClass: Hierarchical Multi-Label Text Classification Using Only Class Names. NAACL 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors considered the problem of undesirable backbone parameters-sharing, avoiding suffering from the task interference.\n2. The idea of using instruction tuning is reasonable for further enhancing the performance in the proposed basic unified-framework.\n3. For each task, the paper tested on not only in-domain but also cross-domain evaluation datasets.\n4. The paper is well written and easy to follow.", "reasons_to_reject": "1. The assumption of the method is somewhat strong, that each label's name and definition must be both available.\n2. The idea of multi-task learning and contrastive learning are not new, and there are no specific new insights for the task itself from the task nature perspective, which leads to a marginal contribution for the research field.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposed a unified framework to solve all three kinds of scientific literature understanding tasks, which includes multi-label classification, link prediction and search. They adopted task-specific Transformer blocks for the unified model to overcome the limitations of backbone parameters-sharing. Specifically, two different types of Mixture-of-Experts are introduced for the Transformer architecture, which can route the input to different MHA and FFN sub-layers, respectively, when considering different tasks. Experimental results demonstrate the effectiveness of the proposed method.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EveKeJulb9", "reviewer": "Reviewer_BsFu", "summary": "", "questions": "A. Better to clarify the differences between this manuscript and SciRepEval. The contributions of this manuscirpt seems limited.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful review and feedback! We discuss the raised points as follows:\n\n1. **Comparison between SciMult and SciRepEval/SPECTER 2.0**\n\nWe fully acknowledge that the SciRepEval/SPECTER 2.0 paper [1] is earlier than our submission. As evidence, in our submission, we cite the paper, compare our SciMult model with SPECTER 2.0 in all tasks, and discuss the difference between SciMult and SPECTER 2.0 in our Related Work section (Lines 611-621, Page 8). We would like to emphasize the differences in more detail here from three aspects.\n\n(1) **High-level focus**: This is reflected in the titles of the two papers. We focus on an empirical study of multiple techniques for mitigating task interference and a more performant scientific language model across the board for scientific understanding tasks. By contrast, the SciRepEval paper emphasizes its contribution to constructing a novel and comprehensive benchmark for scientific document representations in its title.\n\n(2) **Techniques**: This is already mentioned by the reviewer. To be specific, SPECTER 2.0 uses adapters and control codes, while SciMult uses Mixture-of-Experts Transformers and instruction tuning. Although the general goal of all these techniques is to produce task-specific embeddings, we argue that the two papers propose significantly different approaches to implementing this general goal. We do not think that one paper should be viewed as lacking motivation simply because it shares a general goal with another paper, especially when the goal of producing task-specific embeddings is common in multi-task learning (e.g., [2, 3]).\n\n(3) **Empirical performance**: We conduct a head-to-head performance comparison between SciMult and SPECTER 2.0 across various tasks. Experimental results show that SciMult outperforms SPECTER 2.0 in most cases (Tables 3, 4, 5, 7, and A4). The exception happens on the SciDocs dataset, where SciMult-MHAExpert is on par with or slightly underperforms SPECTER 2.0. However, we need to mention the data leakage problem of SciDocs pointed out by Ostendorff et al. in [4]: “**40.5%** of SciDoc’s papers leaking into SPECTER’s training data”. Because SPECTER, SciNCL, SPECTER 2.0, and SciMult all use SPECTER’s training data (i.e., the 819K citation prediction triplets) during pre-training, it becomes questionable how indicative the performance comparison can be if SciDocs is used for evaluation. In comparison, in the PMC-Patients dataset we use, **none** of the texts in the Patient-to-Patient task and only **0.2%** of the papers in the Patient-to-Article task can be found in SPECTER’s training data.\n\n(4) **Datasets**: We agree with the reviewer that SciRepEval has a wider range of tasks, some of which (e.g., regression, author name disambiguation) are not explored in SciMult. However, there are also tasks and datasets studied in SciMult but not in SciRepEval, such as PMC-Patients, Recommendation, SciFact, and NFCorpus. In particular, SciMult-MHAExpert achieves the new state-of-the-art performance on the PMC-Patients leaderboard, which suggests potential far-reaching applications of SciMult in the real world, such as matching patients with clinical trials for precision health.\n\nFinally, we need to argue that the contributions of a language model pre-training paper include not only its proposed new techniques but also its released model, which can be used by the public. Therefore, we feel the superior performance of a pre-trained language model on specific datasets is also an important criterion. As evidence, after BERT, many studies focus on developing more performant domain-specific backbone models – SciBERT [5], LEGAL-BERT [6], PhoBERT [7], and DarkBERT [8], to name just a few – were accepted by *ACL conferences. We fully support this and hope similar **empirical** contributions in our submission can be acknowledged as well.\n\n**References**\n\n[1] SciRepEval: A Multi-Format Benchmark for Scientific Document Representations. arXiv 2022.\n\n[2] Prefix-Tuning: Optimizing Continuous Prompts for Generation. ACL 2021.\n\n[3] Task-aware Retrieval with Instructions. Findings of ACL 2023.\n\n[4] Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings. EMNLP 2022.\n\n[5] SciBERT: A Pretrained Language Model for Scientific Text. EMNLP 2019.\n\n[6] LEGAL-BERT: \"Preparing the Muppets for Court”. Findings of EMNLP 2020.\n\n[7] PhoBERT: Pre-trained language models for Vietnamese. Findings of EMNLP 2020.\n\n[8] DarkBERT: A Language Model for the Dark Side of the Internet. ACL 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A. It proposes four model variants to produce task-aware representations in order to learn common knowledge sharing across diverse tasks while preventing task-specific skills from interfering with each other.\n\nB. Extensive experiments verify the effectiveness of SciMult, especially the task-aware specialization strategy, which outperforms state-of-the-art scientific LMs in both in-domain and corss-domain tasks.\n\nC. The manuscript is easy to follow.", "reasons_to_reject": "A. The motivation is not strong enough, although this manuscript notes that its techniques are distinct from the recent work SciRepEval[1] which shares the most relevant idea with SciMult. (Note that SciRepEval is first submitted 23 November, 2022.)\n\n(1) Both of them proposed multi-task pretrained model for scientific literature understanding, while SciRepEval explore more scientific literature understanding tasks than SciMult.\n\n(2) The pretraining strategies adopted by these two works are almost the same except specific model details, including task-aware adapters and instruction tuning, which aim to obtain task-specific representation. Although SciMult generalizes hard negatives to other tasks, which is of significant importance in citation prediction, the innovation is not that strong.\n\n(3) Both of them consider not only in-domain but also cross-domain evaluationon different tasks and datasets.\n\n[1] Amanpreet Singh, Mike D’Arcy, Arman Cohan, Doug Downey, and Sergey Feldman. 2022. Scirepeval: A multi-format benchmark for scientific document representations. arXiv preprint arXiv:2211.13308.", "questions_to_authors": "A. Better to clarify the differences between this manuscript and SciRepEval. The contributions of this manuscirpt seems limited.", "paper_topic_and_main_contributions": "This manuscript proposes a multi-task contrastive learning based language model pretraining framework for scientific literature understanding tasks, namely SciMult, which explores both task-aware specialization and instruction tuning, focusing on facilitating common knowledge sharing across different tasks. Extensive experiments on a comprehensive collection of benchmark datasets verify the effectiveness of the task-aware specialization strategy, where outperform state-of-the-art scientific LMs.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Summary:
The paper presents a unified framework for addressing three scientific literature understanding tasks: multi-label classification, link prediction, and search. To overcome the limitations of backbone parameter sharing, the authors employ task-specific Transformer blocks within the unified model. They introduce two types of Mixture-of-Experts in the Transformer architecture to route inputs to different Multi-Head Attention (MHA) and Feed-Forward Network (FFN) sub-layers based on the task at hand. Experimental results confirm the method's effectiveness.

Reason To Accept:
1.The paper is well-structured and presents its ideas in a clear and comprehensible manner.
2.The paper introduces an innovative approach by unifying various scientific literature understanding tasks within a contrastive learning framework. It devises four model variants to generate task-aware representations, facilitating shared knowledge acquisition across diverse tasks while mitigating interference from task-specific skills.
3.The authors address the issue of unwanted backbone parameters-sharing, preventing task interference, and propose a reasonable approach involving instruction tuning to further enhance the performance within the proposed unified framework.
4.The paper conducts extensive experiments across a wide range of benchmarks. The results demonstrate that the proposed SciMult method surpasses state-of-the-art scientific language models in both in-domain and cross-domain settings.

Reason To Reject:
1.According to Table 4 and Table A4, the proposed method underperforms or performs on par with SPECTER 2.0 on SciDocs (an in-domain dataset) when the evaluation setting aligns with SPECTER 2.0 (Linear and Reranking). This suggests that the performance improvement may be subject to the contrastive learning setting.
2.The assumption of the method is somewhat strong, that each label's name and definition must be both available.
3.The idea of multi-task learning and contrastive learning are not new, and there are no specific new insights for the task itself from the task nature perspective, which leads to a marginal contribution for the research field.
Decision: Accept-Findings