Multi-Task Knowledge Distillation with Embedding Constraints for Scholarly Keyphrase Boundary Classification
Authors: Seo Yeon Park
Cornelia Caragea
Venue: EMNLP 2023 Main (2023)
Keywords: keyphrase boundary classification
multi-task learning
knowledge distillation

Abstract: The task of scholarly keyphrase boundary classification aims at identifying keyphrases from scientific papers and classifying them with their types from a set of predefined classes (e.g., task, process, or material). Despite the importance of keyphrases and their types in many downstream applications including indexing, searching, and question answering over scientific documents, scholarly keyphrase boundary classification is still an under-explored task. In this work, we propose a novel embedding constraint on multi-task knowledge distillation which enforces the teachers (single-task models) and the student (multi-task model) similarity in the embedding space. Specifically, we enforce that the student model is trained not only to imitate the teachers’ output distribution over classes, but also to produce language representations that are similar to those produced by the teachers. Our results show that the proposed approach outperforms previous works and strong baselines on three datasets of scientific documents.

Multi-Task Knowledge Distillation with Embedding Constraints for Scholarly Keyphrase Boundary Classification
Seo Yeon Park and Cornelia Caragea
Computer Science
University of Illinois Chicago
spark31 3@uic. eecuicc.edu edu cornelia@uic.ed.
Abstract
ommendation (Chen et al., 2015); topie classifica- tion (Sadat and Caragea, 2022; Onan al., 2016 6; Caragea et al., 2015); and, more broadly, data aug- mentation for NLP tasks (Li et al., 2023a,b).
The task of scholarly keyphrase boundary clas- sification aims at identifying keyphrases from scientific papers and classifying them with their types from a set of predefined classes (c.g., task, process, or material). Despite importance of keyphrases and their types in many downstream applications including indexing, searching, and question answering over scientific documents, scholarly keyphrase boundary classification is still an under- exder-explord task. In this work, we propose a novel embedding constraint on multi- task knowledge distillation which enforces the teachers (single-task models) and the student (multi-task, model) similarity in the embedding space. Specifically, we enforce that the student model trained not only to imitate output distribution over classes, but also to pro- duce language representations that are similar those produced by the teachers. Our results show that the proposed approach outperforms previous works and strong baselines on three datasets of scientific documents.
The scholarly keyphrase boundary classification (SKBC) task shares similarities with the named entity recogmition (NER) task. Specifically, NER aims to identify entities" boundaries and classify them into pre-defined categories (c.g.,'U.N. is 'organization" entity) (Collobert al., 201 Sim- ilarly, SKBC aims identify the boundary of keyphrases or entities in scientific text and clas- sify them into a set of pre-defined classes (c.g., "WebVision is 'dataset' entity and 'object detec- tion' is a 'task' entity). NER (Tjong Kim Sang and De Meulder, 2003; Ratinov and Roth, 2009; Col- lobert et al., 2011: Lample et al., 2016) is treated fundamental task in NLP because the named entities often convey the key information of text. Similar NER, the key information of the scien- tific discourse available in research papers revolves around scientific entities (e.g., datasets, tasks, mod- els). Despite that much research has been done on NER (Bodapati et al., 2019; Yu et al., 2020; Chen et al., 2020; Tabassum et al., 2020; Jia and Zhang, 2020; Li et al., 2020; Yan et al., 2021; Zhou et al., 2022: Zhang al., 2022: Liu and Ritter, 2023), there are only very few works that exist on SKBC (Augenstein and Søgaard, 2017: Luan et al., 2017, 2018; Park and Caragea, 2020; Jain et al., 2020), and hence, this task of scholarly keyphrase bound- ary classification is very much under-explored. One potential reason for this under-exploration is the lack of large annotated datasets for this task and the difficulty of annotations which require human experts in scientific field.
teachers'
1 Introduction
Scholarly keyphrase boundary classification is the task of identifying highly summative phrases from scientific papers and classifying them into set of pre-defined classes (Augenstein et al.. 2017: Augenstein and Søgaard, 2017). In scientific domain, keyphrases and their classes, e.g., task, process, or material, are critical for many down- stream applications including effectively mining, searching, and analyzing the scientific literature (Augenstein et al., 2017); promoting an efficient understanding of what methods, processes, tasks, resources are being used or proposed in given paper and how to track them over time (Uban al., 2021);; scientific paper summarization (Abu- Jbara and Radev, 2011: Qazvinian et al., 2010); keyphrase-based question answering (Quarteroni and Manandhar, 2006); machine comprehension (Subramanian et al., 2018); scientific paper rec-
In this paper, we propose novel framework for SKBC, called multi-task knowledge distilla- tion with cosine embedding constraints. Clark et al. (2019b) showed the effectiveness of the multi- task knowledge distillation framework which dis-

---

tills knowledge from multiple teachers (single-task models) the student model (multi-task model). However, in the process of learning from multiple teachers corresponding to multiple related auxil- lary tasks, the student model may become biased towards a particular auxiliary task (or subset of these tasks) and may "erase" all the "good" infor- mation (linguistic features that "matter") from the other remaining tasks. To overcome this, pro- pose a way of imposing similarity constraints in the embedding space between the student and all the teachers to ensure that the student's final rep- resentations are not biased towards some auxiliary task(s), and hence, the student's representations do not diverge (too much) from those of any of the teachers. Consequently, our proposed novel embedding constraint lets the student model be focused on the target task by optimally using the knowledge learned from all auxiliary tasks. No- tably, we achieve state-of-the-art performance on three datasets with our proposed multi-tast knowl- edge distillation with embedding constraints. To facilitate future research on scholarly keyphrase classification, we release our code.'
were among the first to create a dataset annotated for terminology extraction and classification from ACL Anthology papers to stir research this area. Augenstein et al. (2017) proposed a shared task on SciencelE. at SemEval 2017, which includes keyphrase identification and keyphrase classifica- tion as Subtask A and Subtask B, respectively. This shared task attracted large number of participat- ing teams and achieved a highest F1-score of 56% and 44% for Subtasks A and B, respectively, us- ing an RNN with a CRF layer on top (Augenstein et al., 2017). These low Fl scores proved the dif- ficulty the subtasks. Other works participating in this shared task used hand-crafted features for the keyphrase identification and classification. For example, Liu et al. (2017) used combined features of pretrained word embeddings and set of linguis- tic features with Support Vector Machines. Lee et al. (2017) and Marsi et al. (2017) used CRF with linguistically motivated features such as Part-of- Speech tags and word lemmas.
One observation that was made apparent was that the existing annotated datasets for this task are small in size. In order to overcome the small data size problem, Augenstein and Søgaard (2017) pro- posed to transfer knowledge from data-rich tasks through deep multi-task learning where keyphrase boudary classification represents the main task and several related tasks such as FrameNet, semantic supersense tagging. chunking, and multiword ex- pression identification serve as auxiliary tasks (one at a time) to help guide the main task. Still, the results of this approach are low (Augenstein and Søgaard, 2017), e.g., yielding an F1 of 45.49% on the SemEval 2017 dataset (Augenstein et al.. 2017), and an F1 of 58.95% on the ACL dataset (QasemiZadeh and Schumann, 2016) for keyphrase classification. Park and Caragea (2020) proposed to use intermediate task transfer learning using pre-trained language models (Pruksachatkun al., 2020) to overcome the data scarcity problem by us- ing one intermediate task at time. In another line of research, Luan et al. (2017) aimed at address- ing the same small data size problem and proposed graph-based semi-supervised algorithm with a data selection strategy to leverage unannotated arti- cles. Lai et al. (2020) introduced semi-supervised approach for scholarly keyphrase identification em- ploying self-knowledge distillation. Ammar et al. (2017) also used sunirvipde learning for en- tity and relation extraction from scientific papers.
Our contributions are summarized as follows:
We propose novel cosine embedding con- straint added to multi-task knowledge distil- lation for keyphrase boundary classification that enforces teacher-student similarity in the embedding space.
We conduct comprehensive evaluation on three datasets. Experiments show that our proposed approach significantly outperforms strong baselines and prior works and achieves new state-of-the-an on this task.
We provide a detailed analysis using entropy to understand our proposed model's behav- ior (uncertainty) in predicting keyphrases, to- gether with an error analysis.
2 Related Work
The task of automatically identifying and classi- fying keyphrases from scientific papers has long been under-explored but has recently started to gain attention in the research community. For example, QasemiZadeh and Schumann (2016) motivated the importance of this task for many applications such as knowledge acquisition and topic tracking, and

---

Target Input
Nuclear
theory
devoted
major
efforts
to
describe
thermalization
Identification
B
0
O
B
Target Output
Classification
B-Task
I-Task
O
B-Process
Table I: An example instance of scholarly keyphrase boundary classification with BIO schema.
Student (Multi-tack Learning)
Tarpe Thet Oztput
Autilaar Thoks
Target
Curpet
Ausillar Tadk
Target Task Teacher
1-1
Fadly Chemrie Caiitor
1-A Auxiliary Task Teacher
StapkTiskk Model (Tencher)
SingleTask Model (Teachet)
Torget Taak Chipus
Aureitary Taoks Chipur
for Tagget
for
Ausiliary Task
K
Language
Trilkig
laput Source Document
Kounialy Chepes
Knewladge Darikaaio with Enbeddingg Constraion
Figure 1:
An overview of our proposed approach using plate notation. Plates (the grey boxes in the figure indexed by K) refer to repetitions of knowledge distilling steps for K auxiliary tasks. The single task model is used a teacher model for our multi-task student model. "Trm Output" refers to the last layer of pre-trained language models.
Other works used supervised or seme-super keyphrase extraction or generation (Kulkarni et al., 2022; Patel and Caragea, 2021, 2019; Alzaidy et al.,
ensures the student's final representations do not diverge (too much) from those of the teachers. Ac- cordingly, we aim to preserve the beneficial in- formation from the teachers that may not be re- tained in the multi-task knowledge distillation due to catastrophic forgetting (Kirkpatrick al., 2016). Along similar lines, Mahmoud and Hajj (2022) pre- sented oujti-obeecctve learning which leveraged knowledge distillation in a multi-task autoencoder network to minimize catastrophic forgetting of pre- trained tasks. Meshgi et al. (2022) introduced a regularization term in multi-task learning, utilizing task uncertainty to enhance model generalization and mitigate the risk of catastrophic forgetting.
2019; Florescu and Caragea, 2017; Ye and Wang,
2018; Chowdhury et al., 2022; Garg et al., 2023,
2022; Ye et al., 2021; Huang al., 2021; Wu et al.,
2023). Morcover, Ammar et al. (2018) focused on building literature (heterogeneous) graph to organize published scientific literature for easy manipulation and discovery. Luan et al. (2018) created a new dataset annotated for three tasks, sentence-level scientific entity recognition, rela- tion extraction, and coreference resolution, and used multi-task learning framework (ScilE) to jointly learn from all three tasks. Jain al. (2020) extended sentence-level information extraction to decument-level, creating a comprehensive dataset for extracting information from scientific articles. Pan et al. (2023) introduced a scholarly benchmark for dataset mention detection to address the limita- tions of existing corpora in terms of size, diversity of dataset mentions, and entity linking informa- tion. In contrast, propose cosine embedding constraint added multi-task knowledge distil- lation that enforces teacher-student similarity in the embedding space. Similar to our work, some works (Sun et al., 2019; Aguilar et al., 2020) ex- plored teacher-student similarity at the internal rep- resentations. However, their focus is on model compression (to approximate the teacher with smaller student model) while our work is focused teacher-student semantic space similarity that
Target Task
Problem Definition The task objective is to iden- tify and classify keyphrase boundaries from an in- put text sequence. Let x = (#1, .n) be an in- put sequence of size n. where 2;, ... n. are words from a vocabulary V. and let L be a set of la- bels. The goal is learn function (or a classifier) f (xx) + 3/1 that classifies each word 2; from x into one of the available labels y: E L. We do this by minimizing the cross-entropy loss. The set L fol- lows the BIO schema for keyphrase identification, where 'B' refers to the beginning of a keyphrase, 'T' to the inside of keyphrase, 'O' is any word that is not part of keyphrase (we used a similar labeling scheme for keyphrase classification, e.g.. 'B-Task', 'I-Task', 'B-Process', 'I-Process', 'O'). Table 1 shows an input- sequence pair of both keyphrase identification and classification.

---

4 Approach
related tasks can improve the performance of mod- els that use MTL with only one related task. We describe MTL+KD (Clark et al., 2019b) as follows: Given a set of K + 1 tasks, i.e,, a target task and K auxiliary tasks (see Figure 1), each with its own training set D k 1,... K + 1, we train single-task (teacher) model on each task, denoted 6^. Then we use the single-task models & to teach a multi-task shared (student) model with parameters 9 using cross-entropy loss Lce given as follows:
In this section, we first describe our baseline multi- task learning model ($4.1). We then present multi- task learning that distills knowledge from multi- ple related auxiliary single-tasks (teachers) into multi-task (student) model ($4.2). After that, we describe our multi-task knowledge distillation with embedding constraints ($4.3). Last, we describe the auxiliary tasks ($4.4).
4.1 Multi-Task Learning
multi-task learning (MTL) a target task is learned by leveraging knowledge from multiple related auxiliary tasks (Liu et al., 2019b).
CCE(0)=
K+1
Σ
El (1 f)()),0)) 0) (1) (E
Σ
Our baseline MTL model is similar to that of Liu et al. (2019b) and is shown in Figure (mid- dle part). All our models are built top of pre- trained language models, namely BERT (Devlin et al., 2019) and SciBERT (Beltagy et al., 2019).
where (l denotes the output for task l pro- duced by neural model with parameters $ How- ever, Clark et al. (2019b) pointed out that there is a limitation that the student might not be able to overcome the teacher model's performance and suggested to use teacher annealing KD, which com- bines gold labels with predictions. Thus, the cross- entropy loss with teacher annealing becomes:
As shown in the figure, the lower layers (corre- sponding to a pre-trained language model M) are shared across all tasks, whereas the top layers are task-specific. The model jointly learns the target task with different types of NLU tasks-natural language inference tasks, single-sentence classifi- cation, pairwise text classification, and sequence tagging tasks. For the target task, the model M yields a sequence of contextual embeddings, one for each token x; in the input sequence x as h, = =Max) E Rd (of dimension d). The ob- tained representation of each token is then sent through fully connected layer with softmax for classification, i.e., û softmar roft), where W Rdx[[]; is a weight matrix to be learned, and h, is the contextual embedding of the 7t token x, in the input sequence x. For the NLU classification tasks, we use the [CLS] token for classification. In MTL, we train the model to minimize the sum of cross-entropy loss for all tasks. Unlike previous work (Augenstein and Søgaard, 2017), we use all auxiliary tasks at once in our MTL model.
CCE K11
Σ
Σ
El Dyt + (1- (af B*),
l=
(x (EEDR/EEE
5^( 0)
(2)
where linearly increased from through training (see Figure 1).
4.3 Multi-Tass Knowledge Distillation with Embedding Constraints
We now present our multi-task knowledge distilla- tion with embedding constraints framework which enforces fteacher-student similarity in the embed- ding space in addition to the student imitating the teachers' output distribution. That is, we constrain the multi-task student model to learn language rep- resentations that are similar in the semantic space with those of the teachers.
4.2 Multi-Task Learning with Knowledge Distillation
In standard knowledge distillation (KD) (Hinton et al., 2015), student model is trained to imi- tate teacher model's output distribution under the assumption that this output distribution provides better signal to the student than the gold label itself. Clark et al. (2019b) showed that using knowledge distillation with MTL when learning from multiple
Despite that large pre-trained language models (Devlin al., 2019) fine-tuned on supervised tasks perform remarkably well due to their capability to capture the structure of language (Clark et al., 2019a) and other linguistics patterns, e.g., aspects of syntax (Shi et al., 2016; Blevins et al., 2018), in

---

Task Name
Irainl
iDevl
IClassl
Task Type
Domain
Supersense
8,006
1,186
Sequence tagging
Newswire, Novel
MWFs
11,890
1,146
57
Sequence tagging
Online review
FrameNet
9,334
4,000
92
Sequence tagging
Open domain
Chunking
104,054
11,588
12
Sequence tagging
Wall Street Journal
POS Tagging (POS)
384,180
54,138
15
Sequence tagging
Wikipedia, Talks, Literature
NER
13,996
3,241
12
Sequence tagging
Wall Street Journal
SciTail
23,596
1,304
Natural language inference
Science exams
WNLI
634
146
Natural language inference
Fiction books News, Wikipedia
RTE
2,490
277
Natural language inference
MRPC
3,688
1,725
Paraphrase Acceptability
News
CoLAA
8,551
1,043
Books, Journal articles
Table 2:
The overview of auxiliary tasks in our experiments.
Task Name
ITriinl
IDevl
iTsstl
IClassl
lAvg KP per Doel
lAvg Doe Word Counti
Domain
Computer Science, Physies, and Material Science Natural Language Processing Artificial Intelligence
SemEval 2017
5,992
1,076
1,817
3
18
162
ACL RD-TEC 2.0
1,930
214
1,088
7
12
107
ScilE
5,712
641
1.677
6
120
Table 3: The statisties of target tasks in our experiments.
MTL+KD, some information from the pre-trained teachers M may not be retained in the MTL model possibly due to catastrophic forgetting (Kirkpatrick al., 2016). To preserve the "good" information from the teachers, c.g., linguistic features that "mat- ter," we constrain the model to enforce the similar- ity between the word embeddings produced by the student with those produced by the teachers.
natural language inference, sequence tagging, para- phrase detection, and grammatical acceptability tasks. These tasks have proven beneficial in multi- task learning (Liu et al., 2019b). We show the statistics of our auxiliary tasks in Table In de- tail, we use the following tasks: Supersense (Jo- hannsen et al., 2014), MWES (Schneider and Woot- ers, 2017), FrameNet (Das et al., 2014), Chunk- ing (Tjong Kim Sang and Buchholz, 2000), POS Tagging (McDonald et al., 2013), NER (Tjong Kim Sang and De Meulder, 2003), SciTail (Khot et al., 2018), WNLI (Wang et al., 2019), RTE (Wang al., 2019), MRPC (Dolan and Brockett, 2005), and CoLA (Warstadt et al., 2019). Further details about these tasks are presented in Appendix A.I.
We calculate the similarity between the hidden representations (i.e., the last transformer layer out- puts) by produced by the student with those pro- duced by the teachers hg* via cosine similarity. Thus, we define an additional loss term to penalize the student model from diverging from the hidden representations of the teachers:
K
Loos (0)
Σ
Σ
1 - cos( hg
hy
5 Target Task Datasets
(af) JDD
To evaluate models' performance on our target task, i.e., keyphrase boundary identification and classi- fication, we use three datasets, as described below. Table shows the details of each dataset.
(3)
where he" M(xt) E Rd is the hidden repre- sentation for sample af of task & produced by the corresponding teacher M (similarly for h²). Our final loss consists of (i) cross-entropy loss from Eg.. 2; and (ii) cosine embedding loss from Eq. 3 as:
SemEval 2017 Task 10 SemEval 2017 (Augen- stein et al., 2017) is scientific keyphrase bound- ary identification and classification dataset. The dataset has three pre-defined classes (keyphrase types) which are Process, Task, and Material.
(()) CCE(0) + aLcos(() (0),
(4)
where a is hyper-parameter.
ACL RD-TEC 2.0 ACL (QasemiZadeh and Schumann, 2016) dataset for term and entity categorization in scientific text. The ACL dataset has seven pre-defined classes, which are Tool, Mea- surement, Language Resources (LR), Language Resources Product (Lr-prod), Model, Technology
4.4 Auxiliary Tasks
Training on multiple tasks known to help reg- ularize multi-task models (Ruder, 2017). Here, we select K = 11 auxiliary tasks across a range of GLUE benchmarks (Wang et al., 2019), e.g.,

---

(Tech), and OtherScientifeTTem (Other). Note that the class Other here is different from the class Other ('O') in the BIO scheme that represents non- keyphrases.
To compare our method with existing NER ap- proaches, we consider the following two NER methods that utilize a backbone BiLSTM and BERT, respectively.
ScilE ScilE (Luan al., 2018) is dataset for the detection of scientific entities, their relations, and coreference clusters. The SciTE dataset has six pre- defined classes, which are Material, Method, Task, OtherScientiiiTerm (Other), Metric, and Generic.
Biaffine NER (Yu et al., 2020): A framework that has biaffine model top of multi- layer BiLSTM for graph- based dependency parsing to provide a global view on the input for NER.
Unified MRC for NER (Li et al., 2020): A unified framework that has BERT as a back- bone which treats the task of NER as ma- chine reading comprehension (MRC) task.
6 Experiments
6.1 Baselines
We use the following baselines for comparison.
BiLSTM (Augenstein and Søgaard, 2017): Single target task learning of 3-layer BiLSTM with pretrained SENNA embeddings.?
6.2 Implementation
For the SemEval 2017 and ScilE, we use the pub- lished train, validation, and test splits. For the ACL RD-TEC 2.0 dataset, we perform 60/10/30 split to create the train, validation, and test splits, re- spectively. We estimate hyper-parameters for each deep neural model via grid search over combina- tions. The detailed information on the best hyper- parameters setting shown in Appendix A.2.
BiLSTM MTL (Augenstein and Sø- gaard. 2017): Multitask learning of 3-layer BiLSTMs with SENNA embeddings. Note that * corresponds to one auxiliary task.
BERT (Devlin et al., 2019) BERT fine-tuning on the target task.
BERT + ITTL (Park and Caragea, 2020): BERT with intermediate task transfer learning which fine-tunes BERT on a single auxiliary task before fine-tuning on the target task.
6.3 Main Results
We show the main experimental results in Table 4. We perform both keyphrase boundary identi- fication (Identif) and keyphrase boundary classi- fication (Classif). We evaluate the performance of models using phrase-level micro-averaged F1- score and make the following observations.
Self-Distill (Lai et al., 2020): A semi- supervised learning that trains a single teacher BERT model on labeled data and is used to generate pseudo-labels on unlabeled data. Af- terward, it combines the labeled and pscudo- labeled data train single-task student BERT model that is initialized by the teacher BERT model.
First, we observe that the SciBERT-based mod- els generally achieve higher performance com- pared with the previous models based on BiL- STM especially on keyphrase classification. Fur- thermore, we observe that SciBERT-baseh models perform better than BERT-based models. For ex- ample, vanilla SciBERT achieves 66. 70/48.21 FI SemEval keyphrase iddntificationneaassffiatit whereas vanilla BERT achieves 60.40/46.87 FI. We posit that the lower performance of BERT in part due to distributional shifts (i.c., domain differ- ences from the general to the scientific domain). Second, we observe that SciBERT can be im- proved further by using multi-task knowledge dis- tillation. Distilling knowledge from the teach- the student by training the student imitate the teachers' output probability for all tasks, i.e., the target and all auxiliary tasks (SciB- ERT-MTL-KD+ALL) improves the Fl score to 74.17/55.81 (A = 7.47/7.60 over SciBERT)
SciBERT (Beltagy et al., 2019): SciBERT fine-tuning on the target task.
SciBERT + ITTL (Park and Caragea, 2020): SciBERT with intermediate task transfer leamn ing which fine-tunes SciBERT on single aux- iliary task before fine-tuning SciBERT on the target task.
SciBERTMMKD (Liu et al., 2019a): A multi- task knowledge distillation framework distill- ing knowledge from a multi-task teacher SciB- ERT to a multi-task student SciBERT, utiliz- ing the same auxiliary tasks our proposed method for training both teacher and student multi-task models.
httmps//tooan.colllbtt.tcco

---

SemEval 2017 Task 10
ACL RD-TEC 2.0
SciLE
Identif
Classif
Identif
Classif
Identif
Classif
Results on BiLSTM-bused Models
BiLSTM (Augenstein Søgaard, 2017)
67.70
38.01
81.85
58.51
72.33
58.05
BiLSTM MTL + Supersense
63.93
43.54
81.36
58.95
72.65
54.33
BiLSTM MTT.+ + MWES
72.42
45.49
80.69
56.87
72.92
55.21
BiLSTM MTL + FrameNet
65.18
45.24
81.68
58.89
70.44
52.60
BiL.TMM MTI. + Chunking
63.96
42,86
81.37
57.84
75.40
59.43
BiLSTM MTL + ALL
67.06
43.75
77.89
55.10
80.51
63.80
Biaftine- NER (Yu al., 2020)
68.76
47.56
82.04
59.49
80.58
63.76
Results on BERT-based Models
BERT (Devlin al., 2019)
60.40
46.82
79.50
51.67
81,02
65.44
BERT ITTI. (Park and Caragea, 2020)
65.89
49.02
77.72
45.77
75.87
57.40
Self-Distill (Lai et al., 2020)
55,40
41.56
76,66
50.38
73.57
56.95
Unitied MRC for NER (Li al., 2020)
65.29
50.05
81.88
55.36
85.41
68.81
BERT + MTL + ALL
62.73
47.18
79.92
56.64
83.99
66.72
BERT MTI. + KD + Target
67.91
51.95
83.33
63.65
87.27
69.44
BERT MTL + KD + TTrrgtt Cosine
66.21
51.17
78.60
57.52
87.17
68.68
BERT MTL. KD ALL
65.81
49.44
80.77
55.55
85.88
67.54
BERT MTL + KD ALL Cosine
66.47
49.67
78.45
54.60
87.49
68.18
Results on SciBERT buseeR--bssed Models
SciBERT (Beltagy al., 2019)
66.70
48.21
79.77
65.11
81.02
67.44
SciBERT ITTL (Park and Caragea, 2020)
73.89
56.90
88.01
69.90
82.65
75.23
SciBERT-MKD
al., 2019a)
73.69
54.82
83.34
68.85
81.39
70.06
SciBERT MTL ALL
69.04
52.30
85.71
68.85
87.30
72.99
SciBERT MTL KD 'Target
70.47
53.38
87.07
70.18
88.72
73.61
SciBERT MTL. KT) Target + Cosine
70.91
54.60
85.69
69.72
90.49
76.23
SciBERT MTL KD
ALL
74.17
55.81
88.37
71.33
89.58
16.18
SciBERT MTL. KD AI.I+ Cosine
75.08
57.08
89.62
73.09
90.87
77.30
Table 4: Fl-score results of proposed models based on BERT and SciBERT in comparison with the previous work. For pre-trained language models with intermediate task transfer learning (ITTL), we use the best-reported results from prior work. Underlined scores are best within each group and bold scores are best overall.
on SemEval. Remarkably, the best result is achieved with our proposed model, which en- forces teacher-student similarity in the embed- ding space. Precisely, proposed model (SciB- ERT+MTL--K++ALL-Coi improves the F1 score further 75, 555.08/77.08 (A 8.38/8 8.38/8.87 /8.87 over SciBERT) on SemEval. While SciBERT with intermediate task transfer learning (SciBERT + ITTL) shows competitive performance compared with vanilla SciBERT, our proposed model outper- forms SciBERT + ITTL on all three datasets. Our method also outperforms existing multi-task knowl- edge distillation baseline methods (i.e., Delf-Distill and SciBERT-MKD) all three datasets. For ample, our method achieves better Fl score of 75.08/57.09 .88 on SemEival, compared to SciBERT- MKD which achieves an Fl score of 73.69/54.82 82 on keyphrase
to SciBERT, BERT returns the best performance when distilling knowledge by training the student to imitate the teacher's 's output applied only for the target task. While BERT+MTL+KD++ models im- prove vanilla BERT, there is no improvement when teacher-student similarity is enforced in the embedding space for BERT. A potential explana- tion is vocabulary shifts (i.c., domain differences with our target task) between BERT and our sci- entific domain datasets. Specifically, BERT vo- cabulary is from general domain whereas the vocabulary of our target task is from scientific domain. In contrast to BERT, SciBERT vocabulan is from scientific domain. The token overlap be- tween BERT vocabulary and SciBERT vocabulary is only 42%, which implies that there is a substan- tial difference in frequently used words between scientific and general domain texts (Beltagy et al., 2019). We conclude that SciBERT has more cover- age in handling scholarly texts and results in better language representations for scholarly keyphrase classification than BERT.
This proves the effectiveness of our proposed method. In addition, using knowledge distillation only on the target task, KD+Target, notice performance decrease from KD+ALL. Interestingly, in contrast

---

SemEval 2017 Task 10
ACL RD-TEC
SeilE
Identif
Classif
Identif
Classif
Identif
Classif
SciBERT MIL Chunking
72.66
55.17
86.27
71.87
89.28
76.76
SciBERT MTL Chunking KD Both
72.50
53.88
85.36
71.27
87.49
75.58
SeiBERT MTT. Chunking + KD Both Cosine
72.12
54.93
87.92
70.23
86.93
71.46
SciBERT MTI. + NER
69.83
53.81
84.06
70.11
87.89
75.17
SciBERT MILL + NER KD+ + Both
66.42
50.46
86.37
71.58
89.76
75.94
SciBERT MILL + NER KD + Both Cosine
65.42
48.53
87.03
69.87
90.43
76.39
SeiBERT MIL + MWES
62.54
50.86
82.95
68.13
90.78
75.95
SciBERT MTTI + MWES + KD + Both
65.51
52.63
83.49
64.32
90.56
76.22
SciBERT MITL + MWES + KD + Both Cosine
66.93
52.95
82.98
65.57
90.84
76.89
SciBERT MIL SciTail
64.96
49.42
81.58
66.78
86.33
71.71
SeiBERT MIL SciTail KD Both
67.18
51.35
88.09
69,79
90.77
75.25
SciBERT MTTL SciTail KD Both Cosine
69.94
54.87
87.04
69.09
89.95
76.03
SciBERT MTL + ALL
69.04
52.30
85.71
68.85
87.30
72.99
SciBERT MIL+ + + ALL + Cosine
75.08
$7.08
89.62
73.09
90.87
77.30
Table 5: Fl-score of SciBERT based multi-task models using one auxiliary task time in comparison with our proposed model SciBERI-MTL-LLLL-D--CCO
'Both' refers to using an auxiliary task and the target task.
BeeeEva 2017 Tosk
AOL RD-TEO (.)K.)))))))
SilE
SobteTT
ALL- Costre
Figure 2: Entropy distribution per class between our proposed model the baseline model (SciBERT). The x-axis shows the pre-defined classes in each dataset and the y-axis shows the entropy value. The black lines indicate the median of each distribution.
and
We also evaluate the performance of MTL that uses all auxiliary tasks at once using single auxiliary task time to understand the benefits of using all auxiliary tasks at once. We report the results in Table We selected Chunking. NER, MWES and SciTail as representative auxiliary tasks (to avoid clutter). We observe from Table that our proposed model that combines all 11 auxiliary tasks at once performs better compared with the models that just one auxiliary task time. To further investigate the performance gains sin- gle and multiple auxiliary tasks (i.e., even subsets of the eleven auxiliary tasks), we provide the full results MTL with single or multiple auxiliary tasks in Appendix A.3. We also find that MTL with single or multiple auxiliary tasks (but less than 11) yield lower performance compared with our pro- posed model all datasets. This suggests that all auxiliary tasks necessary achieve the best per- formance. We also investigate the role of auxiliary tasks' dataset sizes our proposed model by per- forming an experiment on varying the amounts of
data all auxiliary tasks. We show these results in Appendix A.4. We find that the larger the auxiliary task dataset sizes, the better the performance.
6.4 Analysis
6.4.1 Entropy Analysis
To gain more insight into our proposed model's prediction behavior in identifying and classifying scientific keyphrases, we investigate the entropy of the prediction decisions made by model SeiB- EETT-MIL-KD+ALL-CO
and contrast with the vanilla SciBERT. Entropy standard mea- surement to determine uncertainty of a probability distribution. Given a random variables with n outcomes 31, 22, Ey, the entropy of X is de- fined as H(X) Σn P(x:))ogP(zz)) (Brown et al., 1992). High entropy means high uncertainty.
We aim understand how certain our models are in classifying keyphrases, i.e.. how different behaviors emerge in models' output distribution. We do so by using entropy to explore prediction un- certainty. Figure shows a histogram of entropy

---

Article from SemEval 2017 Task
In this paper a comparison between two popular feature extraction methods presented. Scale-Invariant feature transform (or SIFT) is the first method. The Speeded up robust features (or SURF) is presented as second. These two methods are tested on set of depth maps. Ten defined gestures of left hand are in these depth maps. The Microsoft Kinect camera is used for capturing the images [I]. The Support vector machine (or SVM) is used as classification method. The results are accuracy of SVM prediction on selected images.
Gold Key
"Scale-invariant feature transform', 'Process') ) ('SIFT" Process' ) ('Speeded up robust features'. 'Process') ('SURF", 'Process") " (' ('depth maps', 'Material") ('Support vector machine', 'Process") ('SVM' 'Process") ('comparison between two popular feature extraction methods'. "Task") ("Microsoft Kinect camera'. 'Process')
('classification method', 'Process")
Keyphrase outputs of SSSIBEETAMTL-KDD-A
('Scale-invariant feature transform "rocess") ("SIFT, "rrocess)(("UUS)) "Proeess') ("depth maps', 'Material), ('Support vector machine', 'Process') ('SVM prediction', 'Process') ('comparison between two popular', 'Task") ('feature extraction methods', 'Process') ('Microsoft Kinect camera', , 'Process') ), ('defined gestures', 'Process") ) ('capturing', 'Process),, ('images', 'Material") ('classification method", , 'Process')
Table 6: The comparison between gold and predicted keyphrases of our model on paper from SemEval 2017. Underlined Texts mark error predictions.
each model's output distribution shown per class in each dataset. Most notably, our proposed model has much lower entropy values in predicted keyphrases across all the classes in each dataset. For exam- ple, on ScilE, our proposed model's entropies are aligned in similar range of low values, whereas the baseline model produces broader range of high entropy values. We also observe similar pat- terns the other two datasets. This result suggests that our proposed model is not only more accurate than the baseline model but also it is more certain in the predictions made.
Conclusion
In this paper, we addressed the task of keyphrase boundary classification using multi-task knowledge distillation. We further enhance this approach by proposing a novel extension in which the student model not only imitates teachers' output distribu- tion over classes but also produces similar language representations. The results of experiments on three datasets showed that our model achieves state- of-the-ar results. In the future, we plan to expand our method to various settings such few-shot learning and semi-supervised settings.
Limitations
6.4.2 Error Analysis
We proposed knowledge distillation with embed- ding constraints for keyphrase boundary classifica- tion in which the student (multi-task model) not only learns from the teachers (single-task models) output distributions but also learns hidden language representations that do not diverge from those of the teachers. While we achieved competitive perfor- mance compared to strong baselines, there is one limitation of our method in which our proposed method struggles in classifying longer keyphrases.
We manually investigate test errors to understand potential drawbacks of our proposed model. Table 6 presents an example article from the SemEval dataset and the comparison between gold labels and our proposed model's predicted keyphrases. As shown in the table, our proposed model fails identify and classify the longest keyphrase in the article, which is 'comparison between two pop- ular feature extraction' annotated as Task. Our proposed model returns two different keyphrases which are 'comparison between two popular' as Task and 'feature extraction methods' as Process. It is reasonable to infer 'feature extraction methods' as Process. The model has also trouble predicting whether the output is Task or Process because there is a subjectivity between the two classes. Similar types of errors can be observed in other documents of the dataset and across the datasets. This type of crror is not necessarily& a shortcoming of our model, but rather of the subjectivity of human annotations.
Acknowledgements
This research is supported in part by NSF CA- REER award 1802358, NSF TIS award 2107518. 18, and UIC Discovery Partners Institute (DPI) award. Any opinions, findings, and conclusions expressed here are those of the authors and do not necessarily reflect the views of NSF We also thank our anony- mous reviewers for their constructive feedback and comments, which helped improve our paper.

---

References
Short Papers), pages 341-346, Vancouver, Canada. Association for Computational Linguistics.
Amjad Abu-Jbara and Dragomir Radev. 2011. Coher- ent citation-bas based summarization of scientific papers. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 500-509, Portland, Oregon, USA. Association for Computational Lin- guistics.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019, SciB- ERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Procexsing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP), pages 3615- 3620, Hong Kong. China. Association for Computa- tional Linguistics.
Gustavo Aguilar, Yuan Ling, Yu Zhang. Benjamin Yao, Xing Fan, and Chenlei Guo. 2020. Knowledge dis- tillation from internal representations. In The Thirty- Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applica- lions of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAT Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY. USA, February 7-12, 2020, pages 7350-7357. AAAI Press.
Terra Blevins, Omer Levy, and Luke Zettlemoyer 2018. Deep RNNs encode soft hierarchical syntax. In Pro- ceedings of the 56th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 2: Short Papers), pages 14-19, Melbourne, Australia. Associ- ation for Computational Linguistics.
Sravan Bodapati, Hyokun Yun, and Yaser Al-Onaizan. 2019. Robustness to capitalization errors in named entity recognition. In Proceedings of the 5th Work- shop on Noisy User-generated Text (W-NUT 2019), pages 237-242 Hong Kong, China, Association for Computational Linguistics.
Rabah Alzaidy, Cornelia Caragea, and C. Lee Giles. 2019. Bi-lstm-orf sequence labeling for keyphrase extraction from scholarly documents. In The World Wide Web Conference, WWW 11, page 2551-2557, New York, NY, USA. Association for Computing Machinery.
Peter F. Brown, Stephen A. Della Pictra, Vincent J. Della Pietra, Jennifer C. Lai, and Robert L. Mereer. 1992. An estimate of an upper bound for the entropy of English. Computational Linguistics, 18(1)33 11:31-40..
Waleed Ammar, Dirk Groeneveld, Chandra Bhagavat- ula, Iz Beltagy, Miles Crawford, Doug Downey, Ja- son Dunkelberger. Ahmed Elgohary. Sergey Feld- man, Vu Ha. Rodney Kinney, Sebastian Kohlmeier, Kyle Lo, Tyler Murray, Hsu-Han Ooi, Matthew Pe- ters, Joanna Power, Sam Skjonsherg, Lucy Lu Wang, Chris Wilhelm, Zheng Yuan, Madeleine van Zuylen, and Oren Etzioni. 2018. Construction of the litera- ture graph in semantic scholar. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Indus- try Papers), pages 84-91, New Orleans- - Louisiana. Association for Computational Linguisties.
Cornelia Caragea, Florin Bulgarov, and Rada Mihalcea. 2015. Co-training for topic classification of scholarly data. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. pages 2357-2366, Lisbon, Portugal. Association for Computational Linguistics
Hung-Hsuun Chen, II Ororbia, G Alexander, and C Lee Giles. 2015. Expertseer: A keyphrase based expert recommender for digital libraries. arXiv preprint arXiv:1511.02058.
Jiaao Chen, Zhenghui Wang, Ran Tian, Zichao Yang, and Diyi Yang. 2020. Local additivity based data augmentation for semi-supervised NER. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1241-1251, Online. Association for Computational Linguisties.
Waleed Ammar, Matthew E. Peters, Chandra Bhagavat- ula, and Russell Power. 2017. The A12 system at SemEval-201 task 10 (SciencelE): supervisd end-to sod-to-ent entity and relation extraction. In Pro- ceedings of the 11th International Workshop on Se- mantic Evaluation (SemEval-20177 pages 592-596. Vancouver, Canada. Association for Computational Linguistics.
Jishnu Ray Chowdhury. Seoyeon Park, Tuhin Kundu, and Comella Caragea. 2022. KPDROP: improving absent keyphrase generation. In Findings of the Asso- ciation for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arah Emirates, December 7-11, 2022, pages 4853-4870. Association for Computa- tional Linguistics.
Isabelle Augenstein, Mrinal Das, Sehastian Riedel, Lakshmi Vikraman, and Andrew McCallum. 2017. SemEval 2017 task 10: SciencelE extracting keyphrases and relations from scientific publications. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-22) 2017), pages 546- 555, Vancouver, Canada. Association for Computa- tional Linguistics.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019a. What does BERT look at? an analysis of BERT's attention. In Pro- ceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276-286, Florence, Italy, Association for Com- putational Linguistics.
Isabelle Augenstein and Anders Søgaard. 2017. Multi- task learning of keyphrase boundary classification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2:

---

Kevin Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher D. Manning, and Quoe V. Le. 2019b. BAM! bom-agaai multi-task networks for natural language understanding. Annual Meeting of the Association for Computational Linguistics, pages 5931 1-5937, Florence, Italy. Asso- ciation for Computational Linguistics.
AAAI Conference on Artificial Intelligence, AAAI 2021. Thirty-Thire Conference on Innovative Ap- plications of Artificial Intelligence, IAAT 2021, The Eleventh Symposium on Educational Advances in Ar- tificial Intelligence, EAAI 2021, Viriual Event, Febru- ary 2-9, 2021, pages 13082-13899 AAAT Press.
Proceedings of the 57th
Sarthak Jain, Madeleine van Zuylen, Hannanch Ha- jishirzi, and Iz Beltagy. 2020. SciREX: A challenge dataset for document-level information extraction. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 7506- 7516, Online. Association for Computational Lin- guistics.
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel P. Kuksa. 2011. Natural language processing (almost) from scratch, J. Mach. Learn. Res., 12:2493-2537.
Dipanjan Das, Desai Chen, André F. T. Martins, Nathan Schneider, and Noah A. Smith. 2014. Frame-semantic parsing. Computational Linguistics, 40(1):9-56.
Chen Jia and Yue Zhang. 2020. Multi-cell composi- tional LSTM for NER domain adaptation. Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 5906- 5917, Online. Association for Computational Lin- guistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisticx: Human Language Tech- nologies, NAACL-HHLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers). pages
Anders Johannsen, Dirk Hovy, Héctor Martínez Alonso, Barbara Plank, and Anders Søgaard. 2014. More or less supervised supersense tagging of twitter. In Proceedings of the Third Joint Conference on Lexi- cal and Computational Semantics, *SEM@ COLING 2014, August 23-24, 2014, Dublin, Ireland, pages 1-11. The *SM 2014 Organizing Committee.
Association for Computational
Linguistics.
William B. Dolan and Chris Brockett. 2005. Automati- cally constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing, IWP@ IJCNLP 2005, Jeju Island, Korea, October 2005, 2005. Asian Federation of Nat- ural Language Processing.
Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. Scitail: textual entailment dataset from science question answering. In Proceedings of the Thirty- Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Arti- ficial Intelligence (IAAI-18), and the 8th AAAI Sym- posium on Educational Advances in Artificial Intel- ligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 5189-5197. AAAI Press.
Corina Florescu and Cornelia Caragea. 2017. Posi- tionRank: An unsupervised approach to keyphrase extraction from scholarly documents, In Proceedings of the 55th Annual Meeting of the Association for Computcational Linguistics (Volume I: Long Papers), pages 1105-1115 15, Vancouver, Canada. Association for Computational Linguistics.
James Kirkpatrick, Razvan Pascanu, Neil C. Rabi- nowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Ag- nieszka Grabska- -aaosskaaaaaiiskaaa..mm Demis Hassabis, Clau- dia Clopath. Dharshan Kumaran, and Raia Hadsell. 2016. Overcoming catastrophic forgetting in neural networks. CoRR, htns/1622.00996
Krishna Garg, Jishnu Ray Chowdhury, and Cornelia Caragea. 2022. Keyphrase generation beyond the boundaries of title and abstract. In Findings of the Association for Computational Linguisties: EMNLP 2022, pages 5809-582 5821, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Mayank Kulkarni, Debanjan Mahata, Ravneet Arora, and Rajarshi Bhowmik. 2022. Learning rich repre- sentation of keyphrases from text. In Findings of the Association for Computational Linguistics: NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 891-908 Association for Computational Lin- guistics.
Krishna Garg, Jishnu Ray Chowdhury, and Cornelaa Caragea. 2023. Data augmentation for low-resource keyphrase generation. In Findings of the Associa- tion Computational Linguistics: ACI. 2023, pages 8442-848 8455, Toronto, Canada. Association for Com- putational Linguistics.
Tuan Manh Lai. Trung Bui, Doo Soon Kim, and Quan Hung Tran. 2020. A joint learning approach based self-distillation for keyphrase extraction from scientific documents. In Proceedings of the 28th International Conference on Computational Lin- guistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 649-656. International Committee on Computational Linguistics.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. CoRR, abs/1503.00533 025311.
Xiaoli Huang, Tongge Xu, Lvan Jiao, Yueran Zu, and Youmin Zhang. 2021. Adaptive beam search decod- ing for discrete keyphrase generation. In Thitt

---

Guillaume Lample, Miguel Ballesteros. Sandeep Sub- ramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Compula- tional Linguistics: Human Language Technologies. pages 260-270,, San Diego, California. Association for Computational Linguistics.
Xiaodong Liu, Pengcheng He. Weizhu Chen, and Jian- feng Gao. 2019b. Multi-task deep neural networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4487-4496, Flo- rence, Italy. Association for Computational Linguis- tics.
Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi task identification of entities. relations, and coreference for scientific knowledge graph construction. Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing, pages 3219-3232 Brussels, Belgium. Association for Computational Linguistics.
Lung-Har Hao Lee, Kuei-Ching Lee, and Yuen-Hsien Tseng. 2017. The NTNU system at SemEval-2011 task 10: Extracting keyphrases and relations from scientific publications using multiple conditional random fields. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-20117 pages 951- 955, Vancouver, Canada, Association for Computa- tional Linguistics.
Yi Luan, Mari Ostendorf, and Hannaneh Hajishirzi. 2017. Scientific information extraction with semi- supervised neural tagging. In Proceedings of the 2017 Conference on Empirical Methods in Natu- ral Language Processing, pages 2641-2651, Copen- hagen, Denmark. Association for Computational Lin- guistics.
Hector J. Levesque, Ernest Davis, and Leora Morgen- stem. 2012. The winograd schema challenge. In Principles of Knowledge Representation and Rea- soning: Proceedings of the Thirteenth International Conference, KR 2012, Rome, Italy, June 10-14, 2012. Pres..
Reem A. Mahmoud and Hazem Hajj 2022. Multi- objective learning to overcome catastrophic forget- ting in time-series applications. ACM Trans. Knowl. Discov. Data, 16(6).
Xiaoya Li, Jingrong Feng. Yuxian Meng, Qinghong Han, Fei Wu, and Jiwei Li. 2020. A unified MRC framework for named entity recognition. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5849-5859, On- line. Association for Computational Linguistics.
Erwin Marsi, Utpal Kumar Sikdar, Cristina Marco. Biswanath Barik, and Rune Swtre. 2017. NTNU- @ Sciencell: at SemEval-2211 task 10: Identifying and labelling keyphrases with conditional random fields. In Proceedings the 11hh International Work- shop on Semantic Evaluation (SemEval-2017) pages 938-941, Vancouver, Canada. Association for Com- putational Linguistics.
Yingjie Li, Krishna Garg, and Cornelia Caragea. 2023a, A new direction in stance detection: Target-stance extraction in the wild. In Proceedings of the 61 An- nual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 10071- 10085, Toronto, Canada. Association for Computa- tional Linguisties.
Ryan McDonald, Joakim Nivre, Yvonne Quirmbach- Brundage, Yoav Goldberg, Dipanjan Das, Kuzman Ganchev, Keith Hall, Slav Petrov, Hao Zhang. Oscar Täckström, Claudia Bedini, Núria Bertomeu Castelló, and Jungmee Lee. 2013. Universal dependency an- notation for multilingual parsing. In Proceedings of the 51st Annual Meeting of the Association for Com- putational Linguistics, pages 92-97, Sofia, Bulgaria. Association for Computational Linguistics.
Yingjie Li, Chenye Zhao, and Cornelia Caragea. 2023b. Tis: A target-hased teacher-student framework for zero-shot stance detection. In Proceedings of the ACM Web Conference 2023, WWW '23, page 1500-1509, New York, NY, USA. Association for Computing Machinery.
Linqing Liu, Huan Wang, Jimmy Lin, Richard Socher, and Caiming Xiong. 2019a. Mkd: a tusk knowledge distillation approach for pretrained lan- guage models. arXiv preprint arXiv: 19111.358..8
Kourosh Meshgi, Maryam Sadat Mirzaei, and Satoshi Sekine. 2022. Uncertainty regularized multi-task learning. In Proceedings of the 12th Workshop Computational Approaches to Subjectivity, Sentiment & Social Media Analysis, pages 78-88, Dublin, Ire- land. Association for Computational Linguistics.
Shuheng Liu and Alan Ritter. 2023. Do CONLL-2003 named entity taggers still work well in 2023? In Proceedings of the 61st Annual Meeting of the As- sociation for Computational Linguistics ( (Volume 1: Long Papers), pages 2254-8271 Toronto, Canada. Association for Computational Linguistics.
Aytug Onan, Serdar Korukoglu, and Hasan Bulut. 2016. Ensemble of keyword extraction methods and classi- fiers in text classification. Expert Syst. Appl., 57:232- 247.
Sijia Liu, Feichen Shen, Vipin Chaudhary, and Hong- fang Liu. 2017. MayoNLP SemEval 2017 task 10: Word embedding distance pattern for keyphrase clas- sification in scientific publications. In Proceedings of the 1tth International Workshop on Semantic Eval- uation (oanEvvll-207) pages 6666960, Vancouver, Canada, Association for Computational Linguistics.
Huitong Pan. Qi Zhang, Eduard Dragut, Cornelia Caragea, and Longin Jan Latecki. 2023. DMDD: A Large-Scale Dataset for Dataset Mentions Detection. Transactions of the Association for Computational Linguistics, 11:1 1-1132-1144 146.

---

Seoyeon Park and Comella Caragea. 2020. Scientific keyphrase identification and classification by pre- trained language models intermediate task transfer learning. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5409-5419, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Mobashir Sadat and Cornelia Caragea. 2022. Hierarchi- cal multi-label classification of scientific documents. In Proceedings of the 2022 Conference Empiri- cal Methods in Natural Language Processing, pages 8923-8937, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Nathan Schneider and Chuck Wooters. 2017. The NLTK FrameNet API: Designing for discoverabil- ity with rich linguistic resource. In Proceedings of the 2017 Conference on Empirical Methods in Nat- ural Language Processing: System Demonstrations. pages 1-6, Copenhagen, Denmark. Association for Computational Linguistics.
Krutarth Patel and Cornelia Caragea. 2019. Exploring word embeddings in crf-based keyphrase extraction from research papers. In Proceedings of the 10th International Conference on Knowledge Capture, K- CAP '19, page 37-44, New York, NY, USA. Associ- ation for Computing Machinery.
Krutarth Patel and Comella Caragea. 2021. Exploit- ing position and contextual word embeddings for keyphrase extraction from scientific papers. In Pro- ceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guistics: Main Volume, pages 1585-1591, Online. Association for Computational Linguistics.
Xing Shi, Inkit Padhi, and Kevin Knight. 2016. Does string- based neural MT leam source syntax? In Pro- ceedings of the 2016 Conference Empirical Meth- ods in Natural Language Processing, pages 1526 1534, Austin, Texas. Association for Computational Linguisties.
Sandeep Subramanian, Tong Wang, Xingdi Yuan, Saizheng Zhang. Adam Trischler, and Yoshua Ben- gio. 2018. Neural models for key phrase extraction and question generation. In Proceedings of the Work- shop on Machine Reading for Question Answering, pages 78-88, Melbourne. Australia, Association for Computational Linguistics.
Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang. Clara Vania, Katharina Kann, and Samuel R. Bow- man. 2020. Intermediate-taase transfer learning with pretrained language models: When and why does it work? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5231-5247, Online. Association for Computational Linguisties.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. Patient knowledge distillation for BERT model com- pression. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Nat- ural Language Processing, F.MNIP P-ICCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 4322- 4331. Association for Computational Linguistics.
Behrang QasemiZadeh and Anne-Kathrin Schumann. 2016. The ACL RD-TEC 2.0: A language resource for evaluating term extraction and entity recognition methods. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16) 16), pages 1862-1868, Portorož, Slovenia. European Language Resources Association (ELRA).
Jeniya Tabassum, Mounica Maddela, Wei Xu, and Alan Ritter. 2020. Code and named entity recognition in StackOverflow. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 4913-4926, Online, Association for Computational Linguistics.
Vahed Qazvinian, Dragomir R. Radev, and Arzu- can Ozgür. 2010. Citation summarization through keyphrase extraction. In COLING 2010, 23rd Inter- national Conference on Computational Linguistics, Proceedings of the Conference, 23-27 August 2010, Beijing, China, pages 895-903. Tsinghua University Press.
Erik F. Tjong Kim Sang and Sabine Buchholz. 2000. In- troduction to the CoNLL-2000 shared task chunking. In Fourth Conference on Computational Natural Lan- guage Learning and the Second Learning Language in Logic Workshop.
Silvia Quarteroni and Suresh Manandhar, 2006. User modeling for adaptive question answering and infor- mation retrieval. In Proceedings of the Nineteenth International Florida Artificial Intelligence Research Society Conference, Melbourne Beach, Florida, USA, May 11-13, 2006, pages 776-781. AAAI Press.
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independe independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT AALT-NAACLA 2003, pages 142- 147.
Lev Ratinov and Dan Roth. 2009. Design challenges and misconceptions in named entity recognition. In Proceedings of the Thirteenth Conference on Compu- tational Natural Language Learning (CoNLL--2009, pages 147-155, Boulder, Colorado. Association for Computational Linguistics.
Ana Sabina Uban, Cornelia Caragea, and Liviu P. Dinu. 2021. Studying the evolution of scientific topics and their relationships. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1908-1922, Online, Association for Computa- tional Linguistics.
Sebastian Ruder. 2017. An overview of multi task learn- ing in deep neural networks. CoRR, abs/1700.05500

---

Alex Wang, Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. In 7th In- ternational Conference on Learning Representations. ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
2251-2262, Dublin, Ireland. Association for Compu- tational Linguistics.
A Suuplementary Materials
A.1 Auxiliary Tasks
In this section, we provide details of our auxiliary tasks by giving a specific example from each task.
Alex Warstadt, Amanpreet Singh, and Samuel R. Bow- man. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625 641.
Supersense Supersense tagging (Johannsen al., 2014) is the task of assigning high-level ontologi- cal classes to open-class words using Semcor 3.0 corpus." In this work, we use the most common top 3 noun supersense classes which are Person, Loca- tion and Group. For example, in given sentence 'Clara Harris, one of the guests in the box, stood up and demanded water', the supersense of 'Clara Harris' is Person and the supersense of 'guests' is Person.
Di Wu, Wasi Uddin Ahmad, and Kai-Wei Chang. 2023. Rethinking model selection and decoding for keyphrase generation with pre- trained sequence-to- sequence models. Xrviv preprint ar Xiv:23110006374
Hang Yan, Tao Gui, Junqi Dai, Qipeng Guo, Zheng Zhang, and Xipeng Qiu. 2021. A unified generative framework for various NER subtasks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Procexsing (Volume 1: Long Papers). pages 5808 5822, Online. Association for Computational Linguistics.
MWEs Multiword Expressions Identification and Classification (MWEs) is the task of detecting and classifying the span of single and multiword noun and verb expressions to high-level ontological semantic classes using Streusle corpus' (Schnei- der and Wooters, 2017). In this task, we focus on both noun and verb high-level semantic classes. For example, in given sentence 'I offer compas- sionate approachable and personalized counseling services', the high-level ontological semantic class of compassionate" is verb.social, 'personalized" is verb.social, and 'counseling services' is noun.act.
Hai Ye and Lu Wang. 2018. Semi-supervised learn- ing for neural keyphrase generation. In Proceed- ings of the 2018 Conference Empirical Methods Natural Language Processing, pages 4142-4153, Brussels, Belgium. Association for Computational Linguisties.
Jiacheng Ye, Tao Gui, Yichao Luo, Yige Xu, and Qi Zhang. 2021. One2set: Generating diverse keyphrases as a set. In Proceedings of the 59th An- nual Meeting of the Association for Computationa Linguistics and the 11th International Joint Confer- ence on Natural Language Processing, ACL/IJCNPP 2021. (Volume 1: Long Papers). Virtual Event, Au- gust 1-6, 2021, pages 4598-4608, Association for Computational Linguistics.
FrameNet FrameNet Target Identification (Das et al., 2014) is the task of deciding which word span evoke a semantic frame in a given sentence of FrameNet 1.7 (Schneider and Wooters, 2017). FrameNet is a corpus of lexical and predicate- argument semantics in English. In this task, a frame is conceptual structure describing events, relations, objects and the participants in it. For example, the target "moist" in sentence evokes frame Being state.
Juntao Yu, Bernd Bohnet, and Massimo Poesio. 2020. Named entity recognition as dependency parsing. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 6470- 6476, Online. Association for Computational Lin- guistics.
Shuai Zhang, Yongliang Shen, Zeqi Tan, Yiquan Wu, and Weiming Lu. 2022. De-bias for generative ex- traction in unified NER task. In Proceedings of the 60th Annual Meeting of the Association for Compu- lational Linguistics (Volume 1: Long Papers), pages 808-818, Dublin, Ireland. Association for Computa- tional Linguistics.
Chunking Text Chunking is the task of detecting the chunks of words in CoNLL-2000 shared dataset (Tjong Kim Sang and Buchholz, 2000). The chunk tags include various grammatical classes such noun phrase, verb phrase, prepositional phrase and these tags follow the BIO schema. For example, a sentence 'He reckons the current account deficit will narrow to only #1.8 billion in September' can
Ran Zhou, Xin Li, Ruidan He, Lidong Bing, Erik Cam- bria, Luo Si, and Chunyan Miao. 2022. MELM: Data augmentation with masked entity language model- ing for low-resource NER. In Proceedings of the 60th Annual Meeting of the Association for Compu- lational Linguistics (Volume 1: Long Papers), pages
hhtps://web.cees..u wwwch. edul mihalcea/downllods.hto hhtps://gituub.co/mrrttttt

---

be annotated as, 'NP He] IVP reckons] INP the current account deficit] [VP will narrow] [PP to] INP only # 1.8 billion] [PP in] INP SeptemberJ', where NP refers to noun phrase, VP refers to verb phrase, PP refers to prepositional phrase.
CoLA The Corpus of Linguistic Acceptability (Warstadt et al., 2019) the corpus targeting confirm grammatical acceptability judgements a given sentence. The corpus are drawn from books and journal articles on linguistic theory.
POS Tagging POS Tagging is the task of label- ing each word in sentence with its part of speech tag. In this work, we employ the English POS tag- ging annotation collection of universal dependency parsing (McDonald et al., 2013). For example, the following sentence, 'Aesthetic appreciation and spanish art.', has the grammar class sequence as I'ADJ', 'NOUN', 'CCONJ', 'ADJ', J.OUN'J.
A.2 Training Details
Our implementation is based on the Huggingface implementation of BERT (bert- porrt-bse--und (acesed and SciBERT scciber-scivoocab-unceee All the texts chopped span no longer than 512 tokens. We train all models using the SGD optimizer. We esti- mate aaper-parameter for each deep neural model via a grid search over combinations. We use the fol- lowing range of values to determine the best hyper- parameters: batch size 1,,32,,4) learning rate [0.01, 0.005J, and momentum [0.0, 0.9]. The information on the best hyper-parameters setting is provided follows. For all multi-task learn- ing models including knowledge distillation based models, and vanilla BERT, we use momentum as 0.0, whereas vanilla SciBERT uses momentum 0.9. The training is stopped once convergence is deter- mined on the validation dataset with 10 epochs. For each target dataset, we use batch size 1. The aux- iliary tasks of Supersense, MWEs, FrameNet use batch size 4, whereas Chunking uses 32, POS Tag- ging uses 64, NER uses 32, SciTail uses 16, WNLI uses 16, RTE uses 1, MRPC uses 16, CoLA uses 16. Aside from these details, we follow the SciBERT paper for all other training hyper-parameters. We set a = to calculate a final training objective. All experiments are done on an NVIDIA V100 GPU.
Named Entity Recognition the task of classifying named entities that are present in a text. The entity tags include Person, Organization and Place (Tjong Kim Sang and De Meulder, 2003).
NER
SciTail SciTail (Khot al., 2018) is the task of recognizing the entailment of hypothesis that is constructed from a science question and its corresponding answer by employing the premise. The dataset is collected by crowd-sourcing and by multiple-choice science questions from 4th -rtdd and 8th-grade exams. For example, the relation between the following two sentences, 'Neurons re- ceive information from dendrites which are then passed to the soma cell body' and 'Dendrites from the cell body receives impulses from other neurons.' is labeled as Entail.
WNLI The Winograd Schema Challenge (Levesque et ul., 2012) is the reading comprehen- sion task where system read sentence with pronoun and select the proper pronoun referent from a list of choices. Wang et al. (2019) transform this challenge as a inference task that predicts if the sentence with the pronoun substituted is entailed by the original sentence.
A.3 Effect of Auxiliary Tasks
To explore the effect of auxiliary tasks in our method, we investigate our proposed model by us- ing single or a subset of auxiliary tasks (i.e., less than eleven auxiliary tasks). Accordingly, we aim to: (1) understand the performance gains on mod- els varying the number of auxiliary tasks and (2) explore the necessity of using all auxiliary tasks our proposed model.
The Recognizing Textual Entailment (RTE) is the task of recognizing entailment between two sentences. The dataset collected from a series of annual textual entailment challenges (Wang et al., 2019).
RTE
We first employ auxiliary tasks one at a time on SciBER++MTL+ KD+*+Cosine (with * be- ing a single auxiliary task) to find which auxil- iary task contributes more to improve performance and show the results in the left two columns of Ta- ble under the heading "Single Aux Task". We sort auxiliary tasks decreasing order by clas- sification tasks' Fl-score. Interestingly, we ob- serve that Chunking improves the most in perfor-
MRPC The Microsoft Research Paraphrase Cor- pus (Dolan and Brockett, 2005) is the corpus aim- ing to determine whether sentences in a pair are semantically equivalent or paraphrasing one an- other.

---

Adding Aux Task
SemEval 2017 Task 10
Single Aux Task
Sequentially
Ideniifi
Classiji
SemEval
2017
Task 10
Chunking
54.93
Identif
Classif
Identif
Classif
Chanking + POS
72. 38
56.23
Chunking
72.12
54,93
72.12
54.93
Chunking POS NER
72.41
55.16
SeiTail
69.94
54.87
68.77
52.09
Chanking POS NER + KIE
73.45
56.77
Supersense
72.62
54,43
71.12
54.15
ACL RD-TEC
MRPC
66.36
53.64
60.91
48.65
Chanking
87.92
70.23
MWES
66.93
$2.95
63,91
49.46
Chunking
CoLA
68.08
52.23
69.28
53.72
Chunking NER MRRC
86.42
71.02
Chanking NER MRPC MWEs
86.54
71.53
FrameNet
67.30
52.12
68.39
53.02
Chunking NER MRPC + MWEs FrameNet
88.11
72.54
WNLI
67.66
51.02
68.94
54.26
ScilE
POS Tagging
66.75
50.92
72.11
55.79
NER
65.42
48.53
73.97
56.38
MWES
90.84
76.89
NER
76.31
RTR
61.77
47.25
75.08
57.08
MWES NER Supersense
90.23
76.39
ACL RD-
TEC 2.0
NER Supersense
76.61
Identif
Classif
Identifi
Classif
MWES NER Supersense POS
Chunking
90.58
76.87
Chunking
87.92
70.23
87.92
70.23
Table 8: Fl-score of MOTERT++MTI+DDE Cossine,
NER
87.03
69.87
87.43
72.38
CoLA
87.67
69.44
85.54
68.57
where * a set sorted auxiliary tasks that increase the target task performance in the initial and the final phase on multi-task learning of sequentially adding auxiliary
Supersense
88.85
69.17
85.04
67.42
SciTail
87.04
69.09
81.83
64.11
WNLI
88.72
68.18
86.49
67.72
tasks.
RTE
87.58
67.47
85.38
65.37
POS Tagging
87.78
67.28
86.82
68.86
MRPC
86.35
66.57
87.47
70.44
MWEs
82.98
65.57
88.41
72.63
FrameNet
85.41
64.61
89.62
73.09
ScilE
Identifi
Classifi
Identifi
Classifi
MWES
90.84
76.89
90.84
76.89
1327
NER
90.43
76.39
90.66
76.57
SeiTail
89.95
76.03
88.80
74.65
20000
WNLI
89.75
75.72
88.38
73.01
Figure 3: Fl-score on the impact of auxil- iary tasks data size on target task keyphrase boundary senntfffcttoolllflllllllsiiiiiti mance in SciBERT-MTTL-KD+ALL-- ted lines refer to the best performance of SciB- TRTT-MTI-KD--LLL-COO model on each dataset.
Supersense
90.53
75.37
90.63
76.35
CoLA
88.47
73.96
89.35
75.56
MRPC
87.96
73.32
89.72
74.95
perfor- Dot-
88.20
72.87
86.89
72.33
Tagging
89.44
72.59
90.78
76.14
FrameNet
87.70
71.94
90.10
73.65
Chunking 86.93
71.46
90.87
77.30
Table 7: FI-score of SciBERT=MTL.KKE *Cosine models where * refers to single auxiliary task or subset of them.
auxiliary tasks generally achieve competitive per- formance. However, none of the models achieve performance as good as our proposed model which uses all auxiliary tasks at once. This supports that all auxiliary tasks are required to achieve competi- tive performance for the keyphrase boundary clas- sification task.
mance SemEval and ACL., whereas Chunking improves the least in performance on ScilE. To fur- ther investigate the impacts of the auxiliary tasks, we sequentially add auxiliary tasks one at time on SciBERT+MTL--DD Coosine model follow- ing the order and show results the right two columns of Table under the heading "Adding Aux Task Sequentially". We observe that the per- formance generally increases in the initial and the
A.4 Effect of Auxiliary Tasks Dataset Size
We investigate the impact of dataset size of the auxiliary tasks on our SciB- ERT+MIT--KD+ALL+CCoIn
model's per- formance. Specifically, we conduct a set of experiments on varying amounts of eleven auxiliary tasks' data size on the proposed model and show the results in Figure 3. Interestingly, we observe that our proposed model obtains the largest performance improvements when using a large amount of auxiliary tasks' data size even though we use a small amount of target task
final phase of adding sorted auxiliary tasks on the SciBERT-MTL-KD+** Coosinc (with * being set of auxiliary tasks) model. Hence, we additionally run of experiments using only a few auxil- iary tasks selected from the top and the end of the ranked list. The results are reported Table 8. We find that combining initial and final ranked

---

Tech Metel
6:11
6:27
6013
0:88
0.227
Prediced Naterna
0018
0.993
6.65
6037
0.23
0.23
0.32
6.0078
0.8
6.035
0.024
0.44
Prediied Class
MAtTC Other
Taak
0.033
Prediced
Tool other
0:35
0076
0.62
0.099
B:04
0.0099
6.02
0.02
0.8
0.76
0.12
6:37
0.67
0.7
Meas
8:45
023
6023
0.54
6:7
Metted Nuturnll
unod
0.22
0.43
0.65
60096
6.0988
0.5
60
-00
Haterial
Task True Class
Mutorial Mechod
True
True Class
Figure 4: The keyphrase classification results confusion matrix visualization of each target dataset. The x-axis refers to true classes and the y-axis refers to predicted classes in each target task. The numbers are normalized by the count of predicted keyphrases
training samples.
A.5 Error Analysis
We visualize confusion matrices of our proposed model output (SIBERRTAATTL++LLLLDD-CCCO
in Figure 4. In Figure 4, numbers of each cell rep- resent how many classified keyphrases belong to each true class. For example, on SemEval 2017 Task 10 dataset confusion matrix, the cell corre- sponding to row Process and column Task refers to the ratio of keyphrases predicted as Task but which should be classified as Process to the total number of keyphrases that classified Task. Consequently, each row in every confusion matrix sums up 1. We also confirm the subjectivity of pre-defined classes poses a challenge to keyphrase boundary classification task. For the SemEval 2017 Task 10 dataset, we observe our proposed model incorrectly classifies keyphrases between Task and Process because there is subjectivity between these two classes. For the ACL RD-TEC 2.0 dataset, proposed model mis-classifies keyphrases as Tech. However, contrast the above two target datasets, on the ScilE dataset, we observe that our proposed model generally performs well.


Reviews:
[{"review_id": "7CXxjHMhlT", "reviewer": "Reviewer_7E6J", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review of our paper. We appreciate your feedback. We address your concerns below and will incorporate them in the paper. \n\n> Novelty\n\nThank you for pointing out the paper by Lai et al (2020). We will cite and discuss it in our work. We would like to highlight some differences and the novel contributions of our work in comparison with the work by Lai et al (2020). Unlike their work, which focuses solely on keyphrase extraction (or identification), we focus on both keyphrase identification and classification (here keyphrase classification is defined as the task to determine the category of each identified keyphrase, e.g., “WebVision” is a ‘dataset’, and “object detection” is a ‘task’, not just simply keyphrases of a scientific paper). While keyphrase identification (or extraction) has been extensively studied, keyphrase classification has received relatively little attention, despite its importance in many applications. Our work aims to encourage further research on this important topic. \n\nFurthermore, our work is fundamentally different from Lai et al (2020). While they explored ‘a single-task teacher-student model’ in semi-supervised learning, our work focuses on ‘multiple-tasks teacher-student model’ in supervised learning. Specifically, Lai et al (2020) first trained a single teacher model on labeled data (e.g., the SemEval 2017 dataset) and used it to generate pseudo-labels on unlabeled data (i.e., KP20k). They then combined the labeled and pseudo-labeled data to train a single-task student model that is initialized by the teacher model. They did not consider using any auxiliary tasks. In contrast to this, we propose to leverage multiple auxiliary tasks. Specifically, we propose a method to distill knowledge from multiple teacher models. However, in the process of learning from multiple teachers corresponding to multiple related auxiliary tasks, the student model may become biased towards a particular auxiliary task (or a subset of these tasks) and may “erase” all the “good” information (linguistic features that “matter”) from the other remaining tasks. This is known as catastrophic forgetting. To overcome this, we proposed a way of imposing similarity constraints in  the embedding space between the student and all the teachers to ensure that the student’s final representations are not biased towards some auxiliary task(s), and hence, the student’s representations do not diverge (too much) from those of any of the teachers. Our proposed novel embedding constraints help to retain beneficial information that could be easily lost due to catastrophic forgetting in multi-task learning. \n\nMoreover, we show that our method produces more confidence prediction (please see Section 4.2.1) which is also important in training deep learning models compared to existing methods. We believe that these aspects of our method make it a significant contribution to the field. \n\n(Lai et al. 2020) A Joint Learning Approach based on Self-Distillation for Keyphrase Extraction from Scientific Documents. Lei et al. 2020.  https://arxiv.org/abs/2010.11980 \n\n>Comparison with existing teacher-student models\n\nOur method outperforms the reported results of Lai et al (2020) on the SemEval 2017 dataset, achieving an F1 score of 75.80 compared to their score of 55.4 for keyphrase identification. In addition to this comparison, we performed another comparison with the work by Liu et al (2020) on the Semeval 2017 dataset. Specifically, Liu et al (2020) proposed a method of distilling knowledge from the multi-task teacher model to the multi-task student model, whereas our method distills knowledge from the multiple single-task teacher models to the multi-task student model with proposed embedding constraints. The results of our experiments show that our method achieves a better F1 of 75.08 on the SemEval 2017 dataset, compared to their method which achieves an F1 of 73.69 on keyphrase identification. On keyphrase classification, our approach achieves an F1 of 57.08 compared with  54.82 obtained by their approach. This proves the effectiveness of our proposed method. We will include these results in our paper.\n\n(Lai et al. 2020) A Joint Learning Approach based on Self-Distillation for Keyphrase Extraction from Scientific Documents. Lei et al. 2020.  https://arxiv.org/abs/2010.11980 \n\n(Liu et al. 2020) MKD: a Multi-Task Knowledge Distillation Approach for Pretrained Language Models. Liu et al. 2020. https://arxiv.org/abs/1911.03588 \n\n>Empirical results to show why multi-task knowledge distillation (KD) performs better than a single-task knowledge distillation (KD)\n\nThank you for your comment. Our intuition was that multi-task knowledge distillation allows different teacher models to provide the student model with a diverse set of knowledge, in the same way that ensembles benefit from a diverse set of models, and hence, can equip the student model with the knowledge from all auxiliary tasks and perform better than simply using one auxiliary task. We evaluated this and showed the results in Table 7 in Appendix A.3. The results of experiments with only one auxiliary task at a time are shown in the first two columns of the table, whereas the last two columns show results of experiments when adding auxiliary tasks one by one with the last row in each dataset block showing the performance with all auxiliary tasks. We can see from these results that using all auxiliary tasks performs much better compared to the setting that uses only one auxiliary task at a time. We will clarify this in the paper. \n\n\n>Which tasks allow the student model to learn richer representation?\n\nTo find out which auxiliary tasks help the student model learn richer representations for target tasks in our method, we investigate the F1 scores of using each single auxiliary task one at a time. We show the results in Appendix A.3. Specifically, we hypothesize that if there is a large improvement in F1 score on the target task when using a specific auxiliary task, then that auxiliary task contributes to generating the richest representations for the target task and hence improves the F1 score. Interestingly, we observe that Chunking improves the most in performance on SemEval 2017 and ACL, whereas MWEs improves the most in performance on SciIE. More details can be found in Table 7 in Appendix A.3. Specifically, we sort auxiliary tasks in decreasing order by classification tasks’ F1-score in Table 7 under the line ‘Single Aux Task’, with a performance of using each auxiliary task one at a time to find which auxiliary task contributes more to improving the target task performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper is well written and easy to follow. The authors provide sufficient background about the task and the problem statement (the lack of training data).\n\n* The paper addresses an interesting problem regarding limited labeled data in keyphrase boundary classification, which has important implications for downstream applications such as indexing, searching, and question answering over scientific documents.\n\n* The evaluation results demonstrate the effectiveness of the proposed approach on different datasets compared to other baselines.", "reasons_to_reject": "* The novelty of this paper is limited; a similar approach, with a teacher-student model, for this task has been previously proposed in [1]. \n* There is no comparison against other teacher-student models (e.g., [1] to benchmark the performance of these semi-supervised models proposed for this task.\n* In addition, I find this study needs to demonstrate empirically why multi-task knowledge distillation (via an ablation study) is necessary rather than a single task, and which task allows the student model to learn richer representations.\n\n[1] https://arxiv.org/abs/2010.11980", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper addresses the task of keyphrase boundary classification in scholarly documents, which involves identifying the boundaries of keyphrases that represent the main topics of a document. There is a limited amount of labeled data available for this task, making it difficult to train accurate models. \nTo address this challenge, the authors propose multi-task knowledge distillation with embedding constraints, which involves training a student model to imitate the output of multiple teacher models that are trained on related tasks. The student model also learns to produce similar language representations as the teacher models, resulting in improved performance on three datasets of scientific documents. \nThe experiments show that the proposed approach achieves outperforming results on the three datasets.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "A Joint Learning Approach based on Self-Distillation for Keyphrase Extraction from Scientific Documents\nhttps://arxiv.org/abs/2010.11980"}, "extra_text": {}}, {"review_id": "fIN17O4DKk", "reviewer": "Reviewer_bBkg", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review of our paper. We appreciate your feedback. We address your concerns below and will incorporate them in the paper.\n\n> Effectiveness of our proposed embedding constraints & validation on our hypothesis of proposed embedding constraints\n\nAs we can see from Table 4, when adding embedding constraints to multi-task knowledge distillation, (i.e., comparing SciBERT+MTL+KD+ALL+Cosine with SciBERT+MTL+KD+ALL), we observe improvements in performance on both keyphrase identification and keyphrase classification on all three datasets. For example, on keyphrase classification, we see an improvement from 76.18 to 77.30 on SciIE, from 71.33 to 73.09 on ACL-RD, and from 55.81 to 57.08 on SemEval 2017.  \n \nOur reasoning behind enforcing that \"the student’s final representations do not diverge from those of the teachers\" is to preserve all the “good” information (linguistic features that “matter”) from all the auxiliary tasks. We do so by imposing similarity constraints in the embedding space between the student and all the teachers to ensure that the student’s final representations are not biased towards some auxiliary task(s), and hence, the student’s representations do not diverge (too much) from those of any of the teachers. Currently, we verify this through our results which show that without these similarity constraints the performance drops. However, we agree with you that a further quantitative analysis will shed light into our findings and we will explore this in our future work. However, we will include some preliminary results in the final version of our paper (e.g., using influence functions as in Zaremoodi and Haffari (2019)).   \n\n(Zaremoodi and Haffari, 2019) Adaptively Scheduled Multitask Learning: The Case of Low-Resource Neural Machine Translation. Zaremoodi and Haffari. 2019. https://aclanthology.org/D19-5618.pdf \n\n\n\n\n>Related work on uncertainty mitigation to avoid catastrophic forgetting in MTL\n\nWe will add the work about uncertainty mitigation in multi-task learning in related work. Thanks for your valuable suggestions. \n\n>Adding alternative MTL + distillation method as baseline\n\nOur method outperforms the reported results of Lai et al. (2020) on the SemEval 2017 dataset, achieving an F1 score of 75.80 compared to their score of 55.4 for keyphrase identification. In addition to this comparison, we performed another comparison with the work by Liu et al. (2020) on the Semeval 2017 dataset. Specifically, Liu et al. (2020) proposed a method of distilling knowledge from the multi-task teacher model to the multi-task student model, whereas our method distills knowledge from the multiple single-task teacher models to the multi-task student model with proposed embedding constraints. The results of our experiments show that our method achieves a better F1 of 75.08 on the SemEval 2017 dataset, compared to their method which achieves an F1 of 73.69 on keyphrase identification. On keyphrase classification, our approach achieves an F1 of 57.08 compared with  54.82 obtained by their approach. This proves the effectiveness of our proposed method. We will include these results in our paper.\n\n(Lai et al. 2020) A Joint Learning Approach based on Self-Distillation for Keyphrase Extraction from Scientific Documents. Lei et al. 2020.  https://arxiv.org/abs/2010.11980 \n\n(Liu et al. 2020) MKD: a Multi-Task Knowledge Distillation Approach for Pretrained Language Models. Liu et al. 2020. https://arxiv.org/abs/1911.03588 \n\n\n> SciIE performs similarly when using all auxiliary tasks or using across different auxiliary tasks, where performance gains obtained are after 2 decimal places.\n\nIndeed, SciIE performs similarly regardless of whether we use all auxiliary tasks or different auxiliary tasks in Table 5. We hypothesize that this is because SciBERT already performs well on SciIE even without fine-tuning. For example, Beltagy et al (2019) reported that SciBERT without fine-tuning achieved a F1 score of 65.77 on SciIE keyphrase classification task, which is slightly lower than a F1 score of 67.44 achieved by SciBERT with fine-tuning. We performed similar experiments on the SemEval 2017 using SciBERT. In contrast to SciIE, we observe a F1 score of 22.59 for the SemEval 2017 keyphrase classification on SciBERT without fine-tuning, which is significantly worse than a F1 score of 48.21 on SciBERT with fine-tuning. Accordingly, we posit that SciBERT already possesses beneficial knowledge that is required to perform well on keyphrase classification on SciIE. Therefore, involving some auxiliary tasks on multi-task knowledge distillation can improve performance because the retained knowledge in SciBERT proves beneficial for acquiring additional information. However, there is a chance that the performance will plateau if we incorporate more auxiliary tasks, as the knowledge that a model can obtain from additional auxiliary tasks may be less meaningful. We will explore this further in the future. \n\n(Beltagy et al. 2019) SciBERT: A Pretrained Language Model for Scientific Text. Beltagy et al. 2019. https://arxiv.org/abs/1903.10676", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper has three major strengths\n- After long time, scholarly key-phrase area is seeing some traction especially with usage of MTL + distillation. The current work is solid in that premise with simple cosine tweak to show performance improvement,\n- The results also open up multiple other research areas , especially benefit of auxiliary task, relation of auxiliary task and the dataset (especially the results show large variance)\n- The exhaustive experimentation will be helpful in understanding the task difficulty.", "reasons_to_reject": "The paper has two major weakness\n- The conjecture thrown that cosine constraint improves results is debatable. Especially because the results of SCIIE vs others we can see that despite cosine the results are almost similar and gains obtained are after 2 decimal places.\n- The authors original start by saying \"the student’s final representations do not diverge from those of the teachers\", however results nowhere verified this hypothesis. Rather the authors pivot to showing improvement in results of SCKB.\n\nOtherwise following are few improvements that could be considered\n\n- Line 006 - 012 - the sentence could be simplified to mention the underexploration\n- Line 013 - Embedding constraint, was added to maintain embeddding space, not labeled data. \n- Line 021 - Numerical Results could be introduced to keep the paper attractive to the readers\n- Line 065 - While underexploration is the key, one can also argue on the difficulty of task compared to other NLP applications. This line requires citation or else its speculative.\n- Line 069 - 095: Since the argument is bais towards task and need of cosine embedding constraint, can quantitative results be introduced as contribution to show benefit of the method with respect to reducing such bias?\n- Line 112- 184: While the related work is ample, there are many works such as uncertainity mitigation to avoid catastrophic forgetting in MTL settings along with multiobjective learning in other areas of AI. Some references could be drawn to these.  see https://dl.acm.org/doi/10.1145/3502728 and https://aclanthology.org/2022.wassa-1.8.pdf\n- Line 362 - While the baselines selected are somewhat related. The method selected in this work uses MTL + distillation. To this end, alternative MTL + distillation methods are needed for fair comparison. Moreover keyphrase in other domains have also been explored. Those methods would have been more ideal baseline to compare against. Moreover the relative improvement with the change is around 1% which suggest the method may not seem as effective or the problem conjectured of regularization in lieu of teacher embeddings makes little sense. For example from Table 4, we can see that with more alterntive tasks (ALL) we can see a large jump, which suggests that semantics learnt from such complex objective helps tackle the problem but not using cosine?\n- Section 6.3: Table 5 shows interpretation of using KD + all auxilary task helps. But we can see the results for SCI=IE is very close across differnt auxiliary task? Can you elaborate on such behavior, otherwise it may argued that the dataset behavior is key not the method\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper focuses on scholarly key phrase extraction\n- To this end, the paper propose a form of embedding regularization under setting of multitask learning + distillation.\n- The paper effectively argues that maintaining relative mapping of embeddings in student model like teacher will help it perform well.\n- The authors use multiple based including NER's to justify the benefit of such constraint. The evaluation is fairly comprehensive where it has considered all the tasks.\n- The authors effectively conclude that using wide variety of auxiliary task helps in improving the overall results, which is somewhat debatable especially with mixture of results across dataset. However the paper is very useful in moving the field forward.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9Lf5Ay3pca", "reviewer": "Reviewer_evsV", "summary": "", "questions": "1. What is the basis for claiming that automatically identifying and classifying keyphrases from scientific papers has long been under-explored when there seems to be a good wealth of work on this? At minimum, the QasemiZadeh and Schumann 2014 paper \"The ACL RD-TEC: a dataset for benchmarking terminology extraction and classification in computational linguistics\" will expose more history than appears to be acknowledged here.\n2. Are there alternatives to softmax, and have they been explored? (why use softmax?)\n3. Similarly, why is cross-entropy suitable for loss, and then why just \"add\" a cosine value to it?\n4. As 2/3, explain rationale and options for other selections where not already offered.\n5. Finally, is \"boundary\" really an appropriate term? Boundaries tend to denote limits and so the boundaries of a 4-gram would seem to be either the two outer words or the positions immediately before or after these; consider, for example, word boundaries as relate to regular expressions that would relate the non-word outer positions. ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review of our paper. We appreciate your feedback. We address your concerns below and will incorporate them in the paper. \n\n>Basis for saying under-exploration on keyphrase identification and classification\n\nWe admit that the keyphrase ‘identification/extraction’ gained a lot of attention starting back in the 1980s. However, there is still low attention in ‘keyphrase classification’ despite that it is essential to various applications. There is no available large dataset for ‘keyphrase classification’, which aims to classify each keyphrase to a predefined class (e.g., ‘KP20k’ can be a ‘data’ keyphrase). We would like to stress that the task of keyphrase classification in scholarly documents is very important for downstream applications such as question answering systems, paper recommendation systems, machine comprehension systems, and knowledge graph construction. However, only very few works exist on this topic . Our motivation is to spur future research on the topic of keyphrase classification. \n\n>Why use softmax? \n\nWe have not yet tried functions other than softmax in our method. The softmax function is an ideal choice for our token-level classification task because it converts the logits for each token into a probability distribution over the possible class labels. We then apply post-processing to obtain phrase level prediction with their predicted class labels. We follow the same evaluation protocol as the prior work by Augenstein and Søgaard (2017). \n\n>Why use cross-entropy loss?\n\nWe use cross-entropy loss to encourage the predicted probabilities to match the true distribution of classes, which is suitable for our task that is a token-level classification problem. Again, we wanted to have consistency with the work by Augenstein and Søgaard (2017).\n\n>Why add the embedding constraints loss and cross-entropy loss?\n\nWe used the cross-entropy loss as in the work by Augenstein and Søgaard (2017). In addition, we incorporated the embedding constraints loss to the cross-entropy loss to enforce that \"the student’s final representations do not diverge from those of the teachers\" and hence, the student preserves all the “good” information (linguistic features that “matter”) from all the auxiliary tasks.\n\n>Is ‘boundary’ an appropriate term?\n\nTo avoid confusion and be consistent with prior work by Augenstein and Søgaard (2017), we use their problem definition and terminology for the keyphrase boundary identification/classification task.\n\n(Augenstein and Søgaard. 2017) Multi-Task Learning of Keyphrase Boundary Classification.  Augenstein and Søgaard. 2017. https://aclanthology.org/P17-2054/", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Term extraction and classification for scientific texts has been explored extensively. Approaches that can demonstrate improvement over such a body of literature are important to elaborate upon and understand.  ", "reasons_to_reject": "The paper considers a limited historical perspective, requires clarity over selections made - for example, alternatives rejected, and has been stretched to 15 pages with additional material.", "questions_to_authors": "1. What is the basis for claiming that automatically identifying and classifying keyphrases from scientific papers has long been under-explored when there seems to be a good wealth of work on this? At minimum, the QasemiZadeh and Schumann 2014 paper \"The ACL RD-TEC: a dataset for benchmarking terminology extraction and classification in computational linguistics\" will expose more history than appears to be acknowledged here.\n2. Are there alternatives to softmax, and have they been explored? (why use softmax?)\n3. Similarly, why is cross-entropy suitable for loss, and then why just \"add\" a cosine value to it?\n4. As 2/3, explain rationale and options for other selections where not already offered.\n5. Finally, is \"boundary\" really an appropriate term? Boundaries tend to denote limits and so the boundaries of a 4-gram would seem to be either the two outer words or the positions immediately before or after these; consider, for example, word boundaries as relate to regular expressions that would relate the non-word outer positions. ", "paper_topic_and_main_contributions": "Paper poses combinations of extant methods to address scientific term extraction and classification and claims improvement over other extant approaches. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Authors should consider other prior work - QasemiZadeh and Schumann 2014 would offer a start point for this."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Summary: This paper examines the problem of determining keyphrase boundaries in scientific documents. To tackle the problem that there only exists limited training data, the authors suggest a solution called multi-task knowledge distillation with embedding constraints. This method trains a student model to replicate the outputs of several teacher models that were trained on similar tasks. The student model also learns to generate comparable language representations as the teacher models. The experimental results demonstrate that this approach achieves superior results on all three datasets.

Strengths: All the reviewers unanimously agree that this is a solid contribution with interesting implications. The paper is well-written and easy to follow. The approach used in this study, which involves a basic cosine adjustment, effectively enhances performance. Additionally, the findings reveal several other potential avenues for research, such as the advantages of auxiliary tasks and their relation to the dataset. The comprehensive experimentation conducted in this study will greatly contribute to understanding the complexity of the task.

Weaknesses: I don't think there are any major weaknesses -- some of the weaknesses have been addressed during the discussion phase. The authors should take the feedback into account when preparing for the final version of the paper. For example, Point 3 raised by Reviewer evsV has not been addressed in full.
Decision: Accept-Main