Unifying Question Answering, Text Classification, and Regression via Span Extraction
Authors: Nitish Shirish Keskar
Bryan McCann
Caiming Xiong
Richard Socher
Venue:  (2020)
Keywords: NLP
span-extraction
BERT

Abstract: Even as pre-trained language encoders such as BERT are shared across many tasks, the output layers of question answering, text classification, and regression models are significantly different. Span decoders are frequently used for question answering, fixed-class, classification layers for text classification, and similarity-scoring layers for regression tasks. We show that this distinction is not necessary and that all three can be unified as span extraction. A unified, span-extraction approach leads to superior or comparable performance in supplementary supervised pre-trained,  low-data, and multi-task learning experiments on several question answering, text classification, and regression benchmarks.

Under review as a conference paper at ICLR 2020
UNIFYING QUESTION ANSWERING, TEXT CLASSIFI- CATION, AND REGRESSION VIA SPAN EXTRACTION
Anonymous authors Paper under double-blind review
ABSTRACT
Even as pre-trained language encoders such as BERT are shared across many tasks, the output layers of question answering, text classification, and regression models are significantly different. Span decoders are frequently used for question answering, fixed-class, classification layers for text classification, and similarity- scoring layers for regression tasks, We show that this distinction is not necessary and that all three can be unified as span extraction. A unified, span-extraction ap- proach leads to superior or comparable performance in supplementary supervised pre- trainrdd,,, low-, and multi-task learning experiments on several question swering, text classification, and regression benchmarks.
INTRODUCTION
Pre-trained natural language processing (NLP) systems (Radford et al. 2019; Devlin et al. 2018 Radford et al. 2018 Howard & Ruder 2018 Peters et al. 2018 McCann et al. 2017 7 Liu et al. 2019b) have been shown to transfer remarkably well downstream tasks including text classifi- cation, question answering, machine translation, and summarization (Wang et al. 2018 Rajpurkar eet al. 2016; Conneau al. 2018). Such approaches involve pre-training phase followed by the addition of task-specific layers and a subsequent re- re-training or fine-tuning of the conjoined model. Each task-specific layer relies on an inductive bias related the kind of target task. For question answering, a task-specific span-decoder is often used to extract a span of text verbatim from por- tion of the input text (Xiong et al. (016)) For text classification, tpec-fpe classification layer with fixed classes is typically used instead. For regression, similarity- squares and cosine similarity are employed. These task-specific inductive biases are unnecessary. On several tasks predominantly treated as text classification or regression, we find that reformulat- ing them as span-extraction problems and relying on span-decoder yields superior performance to using a task-specific layers.
layers such as least-
For text classification and regression problems, pre-trained NLP systems can benefit from supple- mentary training on internediate-labeled tasks (STILTs) (Phang et al.] 2018), i.e. supplementary supervised training. We find this is similarly true for question answering. classification, and regres- eptanextraction. Because rely only the span-extractive inductive
sion when reformulated bias, we are able to further explore previously unconsidered combinations datasets. By doing this, find that question answering tasks benefit from text classification tasks and classification tasks can benefit from question answering ones.
The success of pre-training for natural language processing systems affords the opportunity re- examine the benefits of our inductive biases. Our results on common question answering, text classification, and regression benchmark tasks suggest that is advantageous to discard the inductive bias that motivates task-specific, fixed-class, classification and similarity-scorin layers in favor of the inductive bias that views all three as span-extraction problems.
1.1 CONTRIBUTIONS
Summarily, we demonstrate the following:
1. Span-extoen Span-extraction an effective approach for unifying question answering, text classifica- tion, and regression.

---

Under review as a conference paper at ICLR 2020
Input
Extracted span
Output
SQuAD
Nikola Tesla 1858 7 January 1943) Serbian American inventor... (SEP) year bom?
10 July 1856
SST
Positive negalive? [SEP] The movie is slow, very very slow.
negative
The
anough. Entallment, contradiction neutral? [SEP] The rights recently put place nowhere near enough.
MNLI
contradiction
STS
waman riding 0.0 guitar.
(SEP) playing
Figure I: Illustration of our proposed approach using the BERT pre-trained sentence encoder. Text classification tasks are posed as those of span extraction by appending the choices the ipput. Simi- larly, regression tasks are posed by appending bucketed values to the input. For question answering, no changes over the BERT approach are necessary. The figure includes four examples from the SQuAD, SST, MNLI, and STS datasets, respectively.
2. Span-extraction benefits as much from intermediate-task training as more traditional text classification and regression methods.
3. Span-extraction allows for combinations of question answering and text classification datasets in intermediate-task training that outperform using only one or the other.
4. Span-extractive multi-task learning yield stronger multi-task models, but weaker single- task models compared to intermediate-tass training.
5. Span-extraction with intermediate-tase training proves robust in the presence of lim- ited training data than the corresponding task-specific versions.
2 RELATED WORK
Transfer Learning. The use of pre-trained encoders for transfer learning NLP dates back to Collobert & Weston (2008): Collobert et al. (201T) but has had resurgence in the recent past. BERT (Devlin et 2018)) employs the recently proposed Transformer layers (Vaswani et al. 2017) in conjunction with masked language modeling objective
pre-trained sentence encoder. Prior to BERT, contextualized word vectors (McCann et al. 2017) were pre-trained using machine trans- lation data and transferred text classification and question answering tasks. ELMO (Peters al., 2018) improved contextualized word vectors by using a language modeling objective instead of machine translation. ULMFit (Howard & Ruder 2018) and GPT (Radford al.) 2018) showed how traditional, causal language models could be fine-tuned directly for a specific task, and GPT- 2 (Radford et al. (209)) showed that such language models indirectly learn tasks like machine translation, question answering, and summarization.
Intermediate-task and Multi-task Learning. The goal of unifying NLP is not new (Collobert & Weston 2008) Collobert et al., 2011). In [Phang et al. (018)), the authors explore the efficacy of supplementary training on intermediate tasks, framework that the authors abbreviate STILT'. Given target task 7' and a pre-trained sentence encoder, they first fine-tune the encoder on an inter- mediate (preferably related) task / and then finally fine-tune the task T. The authors showed that such an approach has several benefits including improved performance and better robustness to hy- perparameters. The authors primarily focus on the GLUE benchmark (Wang et al.) 2018 Liu al. (2019)) explore the same task and model class (viz., BERT) in the context of multi-tasking Instead of using supplementary training, the authors choose to multi-task on the objectives and, similar to BERT on STILTs, fine-tune on the specific datasets in the second phase. Further improvements can be obtained through heuristics such as knowledge distillation as demonstrated in [larrk et al. (2019). All of these approaches require different classifier head for each task, e.g., a two-way classifier for SST and three-way classifier for MNLI. Two recent approaches: decaNLP (MCCann et al. 2018) and GPT-2 Radford et al. (201)) propose the unification of NLP question answering and language modeling. respectively. As investigated in this work, the task description is provided in natural language instead of fixing the classifier a-priori.
2

---

Under review as a conference paper at ICLR 2020
Task
Dataset
Source Text
Auxiliary Text
Sentence sification
Clas-
SST
positive negative?
it's slow very, very slow
Sentence Pair
MNLI
I don't know lot about camping. tailment, contradiction, or neutral?
know exactly.
Classification
Sentence Pair Classification
RTE
The capital of Slovenia is Ljubljana, Slovenia has 270,000 inhabi- with 270,000 inhabitants. entailment or tants. not?
Sentence Regression
Pair
STS-B
woman riding horse. 0.0 0.25 0.5 0.75 1.0 ... 5.0.
A man playing guitar.
Question
SQuAD
Nikola Tesla (10 July 1856 7 January 1943) was Serbian American inventor
When was Tesla born?
Answering
Table 1: Treating different examples as forms of span-extraction problems. For sentence pair clas- sification datasets, one sentence is present in each of the source text and auxiliary text. The possible classification labels are appended to the source text. For single sentence classification datasets, the source text only contains the possible classification labels. For question answering datasets, no changes to the BERT formulation is required; the context is presented as source text and the question auxiliary text.
METHODS
We propose treating question answering, text classification, and regression as span-extractive tasks. Each input is split into two segments: source text which contains the span to be extracted and an auxiliary text that used to guide extraction. Question answering often fits naturally into this frame- work by providing both question and context document that contains the answer to that question. When treated as span-cxtraction, the question is the auxiliary text and the context document the source text from which the span is extracted. Text classification input text most often does not tain a natural language description of the correct class. When is more natural to consider the input text one whole, we treat it as the auxiliary text and use list of natural language descriptions of all possible classification labels as source text. When the input text contains two clearly delimited segments, one is treated as auxiliary text and the other as source text with appended natural language descriptions of possible classification labels. For regression, we employ a process similar to classi- fication; instead of predicting a doating-point number, we bucket the possible range and classify the text instead.
Our proposal is agnostic to the details of most common preprocessing and tokenization schemes for the tasks under consideration, so for ease of exposition we assume three phases: preprocessing, encoding, and decoding. Preprocessing includes any manipulation of raw input text; this includes tokenization. An encoder is used to extract features from the input text, and an output layer is used to decode the output from the extracted features. Encoders often include a conversion of tokens to distributed representation followed by application of several layers of LSTM, Transformer, convo- lutional neural network, attention, or pooling operations. In order to properly use these extracted features, the output layers often contain more inductive bias related to the specific task. For many question answering tasks, a span-decoder uses the extracted features to select a start and end token in the source document. For text classification, a linear layer and softmax allow for classification of the extracted features. Similarly, for regression, a linear layer and a similarity-scorte such as cosine distance or least-squares employed. We propose to use span nan-deeoders the output layers for text classification and regression in place of the more standard combination of linear layer with task-specific objectives.
objective
3.1 SPAN-EXTRACTIVE BERT (SPEX-BERT)
In our experiments, we start with a pre-trained BERT as the encoder with preprocessing as described in Devlin et al.] (2018). This preprocessing takes the source text and auxiliary text and outputs a sequence of p + 2 tokens: special CLS token, the tokens of the source text, separator
3

---

Under review as a conference paper at ICLR 2020
token SEP, and the auxiliary tokens. The encoder begins by converting this sequence of tokens into a sequence of p vectors in IR Each of these vectors is the sum of a token embedding, a positional embedding that represents the position of the token in the sequence. and segment embedding that represents whether the token is in the source text or the auxiliary text as described in Devlin et al. (2018). This sequence is stacked into matrix Xo E IRxd so that it can be processed by several Transformer layers (Vaswani et al. 2017). The ith layer first computes a² by applying self- attention with & heads over the previous layer's outputs:
a² (X,) = [A1; W.. where h, = a , X
XX
a(X,Y,2) softmax
Z
Va
A residual connection (He et al. 2016) and layer normalization (Ba et al. 2016 merge information from the input and the multi-head attention:
H, LayerNorm(a" (X.) + X,)
This is followed by a feedforward network with ReLU activation (Nair & Hinton 2010: Vaswani al. 2017), another residual connection, and a final layer normalization. With parameters U and E R/xd:
X LayerNorm(max H,UVV + H.))
(4)
Let Xaf E R represent the final output of these Transformer layers. At this point, task-specific head usually uses some part of Xxf to classify, regress, or extract spans. Our proposal is to use a span-decoder limited to X whenever classification similarity-scoring layer is typically used. In this case, we add only two trainable parameter vectors datarz and dend following Devlin et al. 2018). and we compute start and end distributions over possible spans by multiplying these vectors with H and applying softmax function:
Pstart = softmax (attmaxXX.dsttarl
Pend = softmax (onitmax(Xxffeee (Xe)dend)
(5)
During training, we are given the ground truth answer span (a",b" as pair of indices into the source text. The summation of cross-entropy losses over the start and end distributions then gives an overall loss for a training example:
Estart =- -EHaa =i] log Pataret (i)
Lend = -In log Pend (i)
(6)
With L = Estart Lend and at inference, extract span (a, b) as
=
arg max Pstare
b = arg max Pend (i)
(7)
EXPERIMENTAL SETUP
4.1 TASKS, DATASETS AND METRICS
We divide our experiments into three categories: classification, regression, and question answer- ing. For classification and regression, we evaluate on all the GLUE tasks (Wang al. 2018). This includes the Stanford Sentiment Treebank (SST) (Socher et al. 2013), MSR Paraphrase Corpus (MRPC) (Dolan& & Brockett| 2005), Quora Question Pairs (QQP), Multi-genre Natural Language ference (MNLI) /Williams et al. 2017), Recognizing Textual Entailment (RTE) (Dagan et al.) 2010
Bar-Haim et al. 2006 Giampicollo et al. 2007 Bentivogli et al. 2009), Question -aatrionnnnnseiingggg as NLI (QNL)) (Rajpurkar al. 2016), and Semantic Textual Similarity (STS-B) Cer et al. (2017). The
Winograd schemas challenge as NLI (WNLI) Levesque et al. (2012) was excluded during training
because of known issues with the dataset. As with most other models the GLUE leaderboard,
report the majority class label for all instances. With the exception of STS-B, which is a regression
dataset, all other datasets classification datasets. For question answering, we employ 6 popular
4

---

Under review as a conference paper at ICLR 2020
SST 67k
MRPC
QQP
MNLI 393k
RTE 2.5k
QNLI 105k
CoLA 8.5k
STS
GLUE
Train Ex.
3.7k
364k
7k
Leaderboard Score
Development Set Scores
BERT
92.5
89.0
91.5
86.2
70.0
92.3
62.1
90.2
MNNLI
93.2
89.5
91.4
86.2
83.4
92.3
59.8
90.9
-SNLI
92.7
88.5
90.8
80.1
57.0
90.7
SAEX-BERTLARGE
93.7
88.9
91.0
86.4
69.8
91.8
64.8
89.5
+SQUAD
93.7
86.5
90.9
86.0
74.7
91.8
57.8
90.1
->TriviaQA (Web)
93.3
85.0
90.5
85.7
73.6
91.7
60.2
89.9
+TriviaQA (Wiki)
94.4
86.5
90.6
85.6
71.5
91.6
59.9
90.1
-MNLI
94.4
90.4
91.3
86.4
85.2
92.0
60.6
90.9
+MNLI -SUAA
93.7
89.5
91.1
86.4
84.1
92.3
60.5
90.2
Test Set Scores (both on STILTs)
BERTLANE
94.3
86.6
89.4
86.0
80.1
92.7
62.1
88.5
82.0
SpEx-B LERGE
94.5
87.6
89.5
86.2
79.8
92.4
63.2
89.3
82.3
Table 2: Performance metrics on the GLUE tasks. We use Matthew's correlation for CoLA,, an average of the Pearson and Spearman correlation for STS, and exact match accuracy for all others. Bold marks the best performance for a task in section delimited by double horizontal lines. Scores for MNLI averages of matched and mismatched scores. (-> A) indicates that model was fine-tuned on A as an intermediate task before fine-tuning on a target task (the task header for any particular column). In cases where A and the target task are the same, additional fine-tuning is done. The phrase on STILTs indicates that test set scores on the target task are the result of testing with the best (-+ A) according development scores.
datasets: the Stanford Question Answering Dataset (SQuAD) (Rajpurkar et 2016 QA Zero-shot Relationship Extraction (ZRE; we use the 0th split and append the token unanswe wew.bbrb............ rable le to all ex- amples so it can be extracted as span) (Levy al., (017)) QA Semantic Role Labeling (SRL) (He) et al. 2015), Commonsense Question Answering (CQA; we use version 1.0) Taamor et al. 2018) and the two versions (Web and Wiki) of TriviaQA (Joshi al., (017)). Unless specified otherwise, all scores are on development sets. Concrete examples for several datasets can be found in Table
4.2 TRAINING DETAILS
For training the models, closely follow the original BERT setup Devlin al. (2018) and Phang et al. (2018). We refer to the 12-layer model as BERT BASE and the 24-layer model as BERT LARGE. Unless otherwise specified, we train all models with batch size of 20 for 5 epochs. For the SQuAD and QQP datasets, we train for 2 epochs. We coarsely tune the learning rate but beyond this, do not carry out any significant hyperparameter tuning. For STILTs experiments, we re-initialize the Adam optimizer with the introduction of each intermediate task. For smaller datasets, BERT (especially BERT,ARGE) known to exhibit high variance across random initializations. repeat the experiment 20 times and report the best score is common in prior work (Phang et al. 2018 Devlin et al. 2018). The model architecture, including the final layers, stay the same across all tasks and datasets no task-specific classifier heads adaptations are necessary.
these cases, we
4.3 MODELS AND CODE
Pre-trained models and code can be found available in PyTorch Paszke et al. (017).
MASKED
We rely on the BERT training library
RESULTS
Next, we present numerical experiments to buttress the claims presented Section L.1

---

Under review as a conference paper at ICLR 2020
SQuAD 87.6k
ZRE
SRL
CQA
SST
MRPC
RTE
Training Examples
840k
6.4k
9.5k
At most 1k training examples
BBEX-BERTLARGE
84.0
69.1
90.3
60.3
BERTLARGE MNNLI
91.1
83.8
69.0
.MNII
84.5
71.6
90.7
56.7
90.5
85.5
82.7
ZRE
84.0
69.1
90.8
61.3
SQuAD
84.0
82.5
91.7
63.8
SpEX-BERTLARE
91.3
82.5
67.1
TriviaQA (Web)
84.5
75.3
91.3
63.8
MNNLI
91.2
86.5
82.7
TriviaQA (Wiki)
84.3
74.2
91.4
64.4
MNLI - SQuAD
84.5
80.1
91.5
65.7
(a) Exact match scores on the development set for a set of question answering tasks. Bold marks the best per- formance for a task. Note that SpEx BERT and BERT equivalent for the question answering task.
(b) Development set accuracy scores on three of GLUE tasks when fine-tuned only on a constrained sub- set of examples. Bold indicates best score for task.
Table 3
Span-extraction is similar superior to task-specific heads (classification regression). Ta- ble shows our results comparing BERT (with and without STILT's) with the corresponding variant of SpEx-BERT on the GLUE tasks Wang al. (2018). For almost all datasets, the performance for SpEX-BERT is better than that of BERT, which is perhaps especially surprising for the regres- sion task (STS-B). One can reasonably expect model performance improve by converting such problems into span-extraction problem over natural language class descriptions.
SpEx-BERT improves on STILTs. As in the case of [haan al. (2018), we find that using supplementary tasks for pre-training improves the performance the target tasks. We follow the setup of Phang et al. (2018) and carry out a two-stage training process. First, we fine-tune the BERT model with a span-extraction head on the target task with a fresh instance the optimizer. Note that Phang et al. (2018) require a new classifier head when switching between tasks that have different numbers of classes or task, but no such modifications are necessary when SpEx-BERT is applied. SpEx-BERT also allows for seamless switching between question answering, text classification, and regression tasks.
intermediate task. Next, we fine-tune this model
In T we present the results for SpEx-BERT on STILTs. In majority of cases, the performance of SpEX-BERT matches outperforms that of BERT. This is especially pronounced for datasets with limited training data, such as MRPC and RTE with SpEx-BERTi, ARRGE and BERTHARGE:: 85.2 vs 83.4 for RTE, and 90.4 vs 89.5 for MRPC). We hypothesize that this increase due the fact that the class choices are provided to the model in natural language, which better utilizes the pre- trained representations of large language model like BERT. Finally, we note, perhaps surprisingly, that question answering datasets (SQuAD and TriviaQA) improve performance of some of the clas- sification tasks. Notable examples include SST (pre-trained from the Wiki version of TriviaQA) and RTE (pre-trained from any of the three datasets).
STILTs improves question answering as well. Table 3a shows similar experiments on popu- lar question answering datasets. The transferability of question answering datasets is well-known. Datasets such as TriviaQA, SQuAD and ZRE have been known to improve each other's scores and have improved robustness to certain kinds of queries (Devlin al. 2018 McCann et al. 2018). We further discover that through the formulation of SpEX-BERT, classification datasets also help question answering datasets. In particular, MNLI improves the scores of almost all datasets over their baselines. For SQuAD, the benefit of STILTs with the classification dataset MNLI is almost as much as the question answering dataset TriviaQA.
STILTs can be chained. Pre-training models using intermediate tasks with labeled data has been shown to useful in improving performance. |Phan et al. (2018) explored the possibility of using one intermediate task to demonstrate this improvement. We explore the possibility of chaining multiple intermediate tasks in Table 3a Conceptually. improved performance on SQuAD during the first stage of fine-tuning leads to improved performance for the target task of CQA, improving performance of SQuAD through in turn pre- pre-training on MNLI would improve the eventual goal of CQA. Indeed, experiments suggest the efficacy of chaining intermediate tasks in this way. CQA

---

Under review as a conference paper at ICLR 2020
Model
RTE
Natural language description MNLI
BERT - RTE
70.0
Proposed Approach
84.7
BERTLARGE + MNLI + RTE
83.4
segmentation of input text
83.2
terse class descriptions
84.4
SpEX-BERTLAROS LARGE RTE
69.8
SpEX-BEKTLARGE LARGE MNLI RTE
85.2
SpEx- SEX-BERRTARCE MMNLI, RTB)
75.0
1pEx-BERTLARE + MMNLI, RTE) + RTB
75.8
(a) Development set accuracy on the RTE dataset with STILTss and multi-tasking. We denote the process of multi tasking on datasets A and B by (A, B}. For each progression (represented by -)) we reset the optimizer but retain model weights from the previous stage.
(b) Development set accuracy using the SpEx- BERT approach on three versions of the MNLI dataset: (1) with the hypothesis and premise separated source and auxiliary text (see Section [ for details) and terse class descriptions; (2) with both hypothesis and premise treated entirely as auxiliary text; and (3) with segmented input but including one-sentence descrip- tion of the classes (entailment, contradiction, neutral) based on definitions and common synonyms.
Table 4
obtains score of 63.8 when fine-tuned from SQuAD model (of score 84.0) and obtains score of 65.7 when fine-tuned on SQuAD model that was itself fine-tuned using MNLI (of score 84.5) as an intermediate task.
Multi-task STILTs yields stronger multi-task models, but weaker single-task models. also experiment with multi-task learning during intermediate-task training. We present the results for such mnermediate-multi-ttssk. multi-task training on RTE in Table 4a In intermediate-mullte eycle through one batch for each of the tasks until the maximum number of iterations is reached. No special consideration is made for the optimizer or weighing of objectives. The results show that tiermediate-multi-tee
training, we
training improves performance over the baseline for RTE, but this improve- ment is less than when only MNLI is used for intermediate-tate training. Though not desirable if RTE the only target task, such intermediate-muliittaskk training yields a better multi-task model that performs well on both datasets: the joint (single) model achieved 75.0 on RTE and 86.2 on MNLI, both of which are better than their single-task baselines. some cases, the increased per- formance for one task (MNLI) might be preferable to that on another (RTE). We note that this observation similar to the one of Phang et al. 2018).
SpEX-BERT on STILTs is more robust than BERT on STILTs when training data is limited. In Table 3b we present results for the same models (BERT and with sub-sampled versions the dataset. For this experiment, we follow Phang et al. (2018) and subsample 1000 data points random without replacement and choose the best development set accuracy across several random restarts. The rest of the experimental setup remains unchanged. When used in conjunction with STILTs, the performance improves as expected and, in majority of cases, significantly exceeds that of the corresponding baseline that does not use span-extraction.
being tunett
DISCUSSION
6.1 PHRASING THE QUESTION
As described Section 3 when converting any of the classification or regression problems into a span- wann-xxtaaction one, the possible classes or bucketed values need to be presented in natural language as part of the input text. This leaves room for experimentation. We found that separation of naturally delimited parts of the input text into source and auxiliary text was crucial for best performance. Re- call that for question answering, the natural delimitation is to assign the given context document as the source text and the question the auxiliary text. This allows the span-decoder to extract a span from the context document, as expected. For single-sentence problems, there is no need for delimita- tion and the correct span is typically not found in the given sentence, it is treated auxiliary text.

---

Under review as a conference paper at ICLR 2020
Natural language descriptions of the classes or allowable regression values are provided source text for span extraction. For two- sentence problems, the natural delimitation suggests treating one sentence as source text and the other auxiliary. The classification or regression choices must be in the source text, but it was also the case that one of the sentences must also be in the source text. Simply concatenating both sentences and assigning them as the source text was detrimental for tasks like MNLI.
For the case of classification, when experimenting with various levels of brevity, we found that simpler is better. Being terse eases training since the softmax operation over possible start and end locations is over a relatively smaller window. While more detailed explanations might elaborate on what the classes mean or otherwise provide additional context for the classes, these potential benefits were outstripped by increasing the length of the source text. We present these results on the development set of the MNLI dataset with BERT in Table 4b] For regression, there exists a trade-off between brevity and granularity of the regression. We found that dividing the range into 10 20 buckets did not appreciably change the resulting correlation score for STS-B.
6.2 A FULLY JOINT MODEL WITHOUT
PARAMETERS
Unlike similar approaches using task- speciffi heads Liu et al. 2019a), SpEX-BERT allows for a sin- gle model across a broader set of tasks. This makes possible a single, joint model with all parameters shared. We present the results of this experiment in Table in the Appendix; we multi-task over all datasets considered far. Multi-task performance exceeds single-tast performance for many of the question answering datasets (ZRE, SRL, CQA) as well as the classification dataset RTE. In some cases, these improvements are drastic (over 9% accuracy). Unfortunately, the opposite true for the two tasks that are the greatest source of transfer, MNLI and SQuAD, and the remaining GLUE tasks. Understanding why such vampiric relationships amongst datasets manifest, why any partieu- lar dataset appears beneficial, neutral, or detrimental to the performance of others, and why question answering tasks appear more amenable to the fully-joint setting remain open questions. Nonethe- less, a purely ppan-extractive approach has allowed us observe such relationships more directly than in settings that use multiple task-specific heads or fine-tune separately on each task. Because some tasks benefit and others suffer, these results present trade-off. Depending on which tasks and datasets are more pertinent, multi-task learning might be the right choice, especially given the ease of deploying single architecture that does not require any task-specific modifications.
Joint models for NLP have already been studied Collobert et al. (2011); McCann et al. (018);; Radford et al.) (219)) with a broad set of tasks that may require text generation and more general architectures. These approaches have yet to perform as well as task-specific models on common benchmarks, but they have demonstrated that large amounts of unsupervised data, curriculum learn ing, and task sampling strategies can help mitigate the negative influence multitasking tends to have on datasets that are especially good for transfer learning. This work represents connection between those works and work that focuses on task-specific fine-tuning of pre-trained architectures.
7 CONCLUSION
With the successful training of supervised and unsupervised systems that rely on increasingly large amounts of data, more of the natural variation in language is captured during pre- pre-training. This suggests that less inductive bias in the design of task-specife architectures might be required when approaching NLP tasks. We have proposed that the inductive bias that motivates the use taek-specific layers is no longer necessary. Instead, span-extractive approach, common question answering, should be extended to text classification and regression problems as well. Experiments comparing the traditional approach with BERT SpEX-BERT have shown that the span-extractive approach often yields stronger performance as measured by scores on the GLUE benchmark. This reduces the need for architectural modifications across datasets or tasks, and opens ways for applying methods like STILTs to question answering or combination of text classification, regression, and question answering datasets to further improve performance. Experiments have further shown that span- extraction proves more robust in the presence of limited training data. We hope that these findings will promote further exploration into the design of unified architectures for a broader set of tasks.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Jimmy Ba, Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. CoRR, abs/1607/ (0s/0660,, 2016.
Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second pascal recognising textual entailment challenge. In Proceedings of the second PASCAL challenges workshop on recognising textual entailment, volume pp. 6-4. Venice, 2006.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. In TAC, 2009.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task I: Semantic textual similariy-mulliinguuis multilingua and cross-lingual focused evaluation. arXiv preprint arXiv:1708.0000 2017.
Kevin Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher D Manning, and Quoc V Le. Bam! born-again multi-task networks for natural language understanding. arXiv preprint arXiv: arXiv:/907.044 /907.9997.49 2019.
Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pp. 160-167. ACM, 2008
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural language processing (almost) from scratch. Journal of machine learning research, 12(Aug):243322533 2011.
Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. Xnli: Evaluating cross-lingual sentence representations. arXiv preprint arXiv:1809.050 05055,, 2018.
Ido Dagan, Bill Dolan, Bernardo Magnini, and Dan Roth. Recognizing textual entailment: Rational, evaluation and approaches -rraatm Natural Language Engineering, 16(1):105-105: 2010.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 048000 2018.
William Dolan and Chris Brockett. Automatically constructing a corpus sentential paraphrases. In Proceedings of the Third International Workshop Paraphrasing (IWP2005), 2005.
Danilo Giampiccolo. Bernardo Magnini, Ido Dagan, and Bill Dolan. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pp. 1.9. Association for Computational Linguistics, 2007.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Luheng He, Mike Lewis, and Luke S. Zettlemoyer. Question-answer driven semantic role labeling: Using natural language to annotate natural language. In EMNLP, 2015.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. arXiv preprint arXiv: cet.:06111.066 2018
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv: M.Xi::170.. fttp::/75.035// 2017.
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. Thir- teenth International Conference the Principles of Knowledge Representation and Reasoning, 2012.
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading comprehension. arXiv preprint arXiv: 1706.041151 2017.

---

Under review as a conference paper at ICLR 2020
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. arXiv preprint arXiv:1901.11500 2019a.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 19Xh::1907.11699 2019b.
Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. Learned translation: Contextualized word vectors. In Advances in Neural Information Processing Systems, pp. 6294- 6305, 2017.
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv: /806, 08730, 2018.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings the 27th International Conference on Machine Learning (ICML-10), pp. 807-814, 2010.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W, W. 2017.
Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations, arXiv preprint arXiv:1802.00 180.00566, 2018.
Jason Phang, Thibault Févry, and Samuel R Bowman. Sentence encoders stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv: 1.010088 2018.
Alec Radford, Karthik Narasimhan, Tim Salimans, and llya Sutskever. Improving language under- standing by generative pre- training. URL htsts:://3-ussttteee/ west - amazonass com/ apenai-assetss
/gguunsupervised/laaaaa anguage.understarn ing.paper.pds 2018.
Alec Radford, Jeffrey
Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. URL
https: ftttps///chhmofpssss....oooooooo (better- htw.://niiaaeeeooeeeeee//nnaaaaa /
ers. pds. 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05220 2016.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language pro- cessing, pp. 1631-1642, 2013.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsensegat: question answering challenge targeting commonsense knowledge. arXiv preprint 101Xiv:881.000 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neu- ral Information Processing Systems 30, pp. 5998-6008. Curran Associates, Inc., 2017. URL 8- all -you-need pdf
www..cc......o cc happpapper/// /71 www.:aatttttiiiioo
Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 1804 07461, 2018.
10

---

Under review as a conference paper at ICLR 2020
Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arxi::7044.0522,, 2017.
Caiming Xiong, Victor Zhong, and Richard Socher. Dynamic coattention networks for question answering, arXiv preprint arXiv:161 2016.
A MULTITASKING RESULTS
Below is the table that supports the commentary in Section 6.2
SST
MRPC
QQP
MNLI
RTE
QNLI CoLA
STS
SQuAD
ZRE
SRL
CQA
Individual Models
BERTLARGE LARGE
92.5
89.0
91.5
86.2
70.0
92.3
62.1
90.9
84.0
69.1
90.3
60.3
SpEX-BERT LERG
93.7
88.9
91.0
86.3
69.8
91.8
64.8
89.5
84.0
69.1
90.3
60.3
Multi-task Models (best joint single model)
SpEx-BERT LARGE
92.4
87.5
90.9
85.0
71.1
91.3
58.8
89.2
80.4
75.0
97.7
61.0
-NNLI
93.2
87.0
90.9
85.6
81.2
91.3
57.9
90.1
80.5
76.6
97.7
61.5
+SQUAD
92.2
87.0
91.0
85.3
80.9
91.2
52.0
90.1
80.6
78.8
97.7
~MNLI SSQAD
92.3
90.9
90.8
85.2
84.1
90.9
52.1
90.2
80.6
75.3
97.8
61.5
Multi-tass task Models (maximum individual score for each dataset during the course of training)
SpEx-BERT 93.0
88.5
91.0
85.2
73.3
91.4
59.8
88.9
81.9
77.8
97.7
64.7
-MNLI
93.2
90.8
85.7
84.1
91.6
59,9
89.8
81.4
97.7
63.3
-SQuAD
92.9
89.2
91.1
85.4
84.1
91.4
56.1
90.1
82.8
79.6
97.8
65.3
+MNLI-SQUAD
92.7
91.4
90.8
85.4
85.2
91.2
57.5
90.2
83.2
77.5
97.8
64.8
Table 5: Development set performance metrics on a single (joint) model obtained by multi-tasking on all included datasets. We include best single-task performances (without STILTs), labeled as individual models, for the sake of easier comparison. We divide the remaining into two parts in the first, the scores indicate the performance on a single snapshot during training and not individual maximum scores across the training trajectory. the second, include the best score for every dataset through the training; note that this involves inference multiple model snapshots. For the models trained with STILT', the SpEx-BERS model is first fine-tuned on the intermediate task by itself after which the model is trained in multi-tasking fashion. Bold implics best in each column (i.c., task).


Reviews:
[{"review_id": "rkl2c_vTtB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a method for converting sentence pair classification tasks and sentence regression tasks into span into span extraction tasks, by listing all the possible classes (entailment, contradiction, neural) or the discretized scores (0.0, 0.25 ...) and concatenating them with the source text. With this formulation, one can train a BERT-based span-extraction model (SpEx-BERT) on classification, regression, and QA tasks without introducing any task-specific parameters. The purposed SpEx-BERT model achieves moderate improvement (0.3 points) over the BERT-large baseline on the GLUE test set when fine-tuned on intermediate STILTs tasks (Phang et al., 2018).\n\nStrengths:\n- Extensive finetuning/intermediate-finetuning experiments on a range of NLP tasks.\n- The paper is mostly well-written and easy to follow.\n\nWeaknesses:\n- This paper presents a lot of experiments. But it seems that the most useful / head-to-head comparison against the BERT model are the last 2 rows in Table 2 with the GLEU results, where the improvement is moderate.\n- The idea of expressing various NLP tasks (including textual entailment and text classification) as question-answer has been well-explored in decaNLP (McCann et al., 2018). It would be nice if the authors could elaborate more on how the proposed method differs from theirs.\n\nOther comments/suggestions:\n- Likely typo in abstract: \"fixed-class, classification layers for text classification\"\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "S1gveMimYH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper asks whether it works to remove task-specific heads and treat classification and regression problems as span extraction, by formatting problems in such a way that a single span extraction model can be used.  This is a reasonable question to ask, and the authors performed a very large number of experiments attempting to answer this question.  The authors claim that using span extractive models instead of task-specific heads yields improved performance over separate heads.\n\nMy main concern with this work is actually with something that would otherwise be a strength - the very large number of experiments.  Looking at the results tables, I come to a different conclusion from the authors: there does not appear to be a significant difference between using a single head or using multiple heads (this is still an interesting conclusion).  The numbers presented all appear to be within the bounds of random fluctuations, which are not controlled for at all with standard statistical testing methodologies.  And with the very large number of experiments, there are bound to be some that stand out.  This is especially true given the methodology used for small datasets - even if it was used by prior work, it is still not statistically sound to take the max of 20 samples from a training distribution and report the difference without any other statistical testing.\n\nAs an example, look at table 3b.  This is claimed as a big win for SpExBERT when comparing the MNLI fine-tuned version.  But if you look at the other rows in the table, SpExBERT is worse than BERT by a *larger margin* than it is better in the MNLI case (contradicting the claimed result from table 2).  And this general trend is seen across the tables - the differences are small and inconsistent, making it look very much like we are just seeing random fluctuations.  This must be controlled for statistically in order to make any valid conclusions.  The one possible exception here seems to be SST.  Those results on the dev set do indeed seem to be more consistent, which is interesting, and hints at the utility of using \"positive\" and \"negative\" as natural language descriptions, as the authors claim.  It's not very convincing, however, as the test set difference is very small, and SpExBERT had more opportunities to find a good dev set number, as it had more experiments.\n\nMy second major concern is with the experimental set up.  The authors want to claim that using a unified span extraction format yields superior performance to having separate heads.  But there are baselines missing to really demonstrate this claim.  The multiple head setup isn't really evaluated as a baseline in most of the experiments (e.g., using SQuAD / other QA datasets as an intermediate task in table 2, using BERT for any of the QA datasets in table 3).  So, even if the above issue of statistical testing were solved, it would still be very hard to evaluate the claim, as the proper comparisons are not present in the majority of cases.\n\nMy main conclusion from reading this paper is that it does not appear to matter what head you use for these particular datasets.  This is an interesting result, though it is not the one that is claimed in the paper.  I think it would be very challenging to extend this approach to a broader set of tasks, however, as the authors suggest towards the end of the paper.  How do the authors propose handling cases where it is not feasible to put all possible outputs in the context?  This includes any generative task (including generative QA) and any kind of structured output (like parsing, tagging, etc.), or a regression task that does not lend itself well to bucketing.\n\nMinor issues:\n\nI would be careful about claiming that you've successfully captured regression in a span extraction format.  You have one regression task, and it's one where the original labels were already bucketed, so the bucketing makes sense.  I am very skeptical that this would actually work for more general regression.\n\nRe the paragraph titled \"SpEx-BERT improves on STILTs\": Note that SpExBERT requires additional data preprocessing and hand-written templates when switching between tasks, which is not necessary in the method of Phang et al.  There are always tradeoffs.  Neither using separate heads nor doing your additional processing are very hard, so I wouldn't make as big a deal out of this as you do.  If you want to talk about one of the methods using more \"human inductive bias\", or whatever, the hand-written templates in yours might actually be _more_ \"incidental supervision\" than a very small difference in model architecture.  But again, the difference here is very small, not worth noting in the paper.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BkldDbITtH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submission proposes a span-extraction approach to unify QA, text classification and regression tasks. The extensive empirical studies presented in this paper demonstrate that the proposed method can improve the performance by a small margin across multiple datasets and tasks. Overall, I find that the idea of unifying QA, text classification and regression is interesting by itself, but the experiments cannot justify their claims well mainly due to the mixed results.   \n\nI have the following concerns:\n\n0. Compared to decalNLP (https://github.com/salesforce/decaNLP), this new approach seems unable to handle as many types of tasks as decalNLP. It is not clear to me what is the main advantage. As discussed in the Related Work Section, decalNLP needs to fix the classifier a-priori, but this submission's method needs a natural language description, which seems more difficult to implement in practice. \n1. The results are mixed. For example, SpEx-BERT underperforms BERT on RTE and QNLI in Table 2 and Span-extractive multi-task learning results in weaker single-task models as shown in Table 4a. Furthermore, an error analysis would be helpful here. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: (I acknowledge reading authors' recent note on decaNLP.)

This paper proposes a span extraction approach (SpExBERT) to unify question answering, text classification and regression. Paper includes a significant number of experiments (including low-resource and multi-tasking experiments) on multiple benchmarks. The reviewers are concerned about lack of support on author's claims from the experimental results due to seemingly insignificant improvements and lack of analysis regarding the results. Hence, I suggest rejecting the paper.
Decision: Reject