Unifying Question Answering, Text Classification, and Regression via Span Extraction

1 INTRODUCTION . Pre-trained natural language processing ( NLP ) systems ( Radford et al. , 2019 ; Devlin et al. , 2018 ; Radford et al. , 2018 ; Howard & Ruder , 2018 ; Peters et al. , 2018 ; McCann et al. , 2017 ; Liu et al. , 2019b ) have been shown to transfer remarkably well on downstream tasks including text classification , question answering , machine translation , and summarization ( Wang et al. , 2018 ; Rajpurkar et al. , 2016 ; Conneau et al. , 2018 ) . Such approaches involve a pre-training phase followed by the addition of task-specific layers and a subsequent re-training or fine-tuning of the conjoined model . Each task-specific layer relies on an inductive bias related to the kind of target task . For question answering , a task-specific span-decoder is often used to extract a span of text verbatim from a portion of the input text ( Xiong et al. , 2016 ) . For text classification , a task-specific classification layer with fixed classes is typically used instead . For regression , similarity-measuring layers such as leastsquares and cosine similarity are employed . These task-specific inductive biases are unnecessary . On several tasks predominantly treated as text classification or regression , we find that reformulating them as span-extraction problems and relying on a span-decoder yields superior performance to using a task-specific layers . For text classification and regression problems , pre-trained NLP systems can benefit from supplementary training on intermediate-labeled tasks ( STILTs ) ( Phang et al. , 2018 ) , i.e . supplementary supervised training . We find this is similarly true for question answering , classification , and regression when reformulated as span-extraction . Because we rely only on the span-extractive inductive bias , we are able to further explore previously unconsidered combinations datasets . By doing this , we find that question answering tasks can benefit from text classification tasks and classification tasks can benefit from question answering ones . The success of pre-training for natural language processing systems affords the opportunity to reexamine the benefits of our inductive biases . Our results on common question answering , text classification , and regression benchmark tasks suggest that it is advantageous to discard the inductive bias that motivates task-specific , fixed-class , classification and similarity-scoring layers in favor of the inductive bias that views all three as span-extraction problems . 1.1 CONTRIBUTIONS . Summarily , we demonstrate the following : 1 . Span-extraction is an effective approach for unifying question answering , text classification , and regression . 2 . Span-extraction benefits as much from intermediate-task training as more traditional text classification and regression methods . 3 . Span-extraction allows for combinations of question answering and text classification datasets in intermediate-task training that outperform using only one or the other . 4 . Span-extractive multi-task learning yield stronger multi-task models , but weaker singletask models compared to intermediate-task training . 5 . Span-extraction with intermediate-task training proves more robust in the presence of limited training data than the corresponding task-specific versions . 2 RELATED WORK . Transfer Learning . The use of pre-trained encoders for transfer learning in NLP dates back to Collobert & Weston ( 2008 ) ; Collobert et al . ( 2011 ) but has had a resurgence in the recent past . BERT ( Devlin et al. , 2018 ) employs the recently proposed Transformer layers ( Vaswani et al. , 2017 ) in conjunction with a masked language modeling objective as a pre-trained sentence encoder . Prior to BERT , contextualized word vectors ( McCann et al. , 2017 ) were pre-trained using machine translation data and transferred to text classification and question answering tasks . ELMO ( Peters et al. , 2018 ) improved contextualized word vectors by using a language modeling objective instead of machine translation . ULMFit ( Howard & Ruder , 2018 ) and GPT ( Radford et al. , 2018 ) showed how traditional , causal language models could be fine-tuned directly for a specific task , and GPT2 ( Radford et al. , 2019 ) showed that such language models can indirectly learn tasks like machine translation , question answering , and summarization . Intermediate-task and Multi-task Learning . The goal of unifying NLP is not new ( Collobert & Weston , 2008 ; Collobert et al. , 2011 ) . In Phang et al . ( 2018 ) , the authors explore the efficacy of supplementary training on intermediate tasks , a framework that the authors abbreviate as STILTs . Given a target task T and a pre-trained sentence encoder , they first fine-tune the encoder on an intermediate ( preferably related ) task I and then finally fine-tune on the task T . The authors showed that such an approach has several benefits including improved performance and better robustness to hyperparameters . The authors primarily focus on the GLUE benchmark ( Wang et al. , 2018 ) . Liu et al . ( 2019a ) explore the same task and model class ( viz. , BERT ) in the context of multi-tasking . Instead of using supplementary training , the authors choose to multi-task on the objectives and , similar to BERT on STILTs , fine-tune on the specific datasets in the second phase . Further improvements can be obtained through heuristics such as knowledge distillation as demonstrated in Clark et al . ( 2019 ) . All of these approaches require a different classifier head for each task , e.g. , a two-way classifier for SST and a three-way classifier for MNLI . Two recent approaches : decaNLP ( McCann et al. , 2018 ) and GPT-2 Radford et al . ( 2019 ) propose the unification of NLP as question answering and language modeling , respectively . As investigated in this work , the task description is provided in natural language instead of fixing the classifier a-priori . 3 METHODS . We propose treating question answering , text classification , and regression as span-extractive tasks . Each input is split into two segments : a source text which contains the span to be extracted and an auxiliary text that is used to guide extraction . Question answering often fits naturally into this framework by providing both a question and a context document that contains the answer to that question . When treated as span-extraction , the question is the auxiliary text and the context document is the source text from which the span is extracted . Text classification input text most often does not contain a natural language description of the correct class . When it is more natural to consider the input text as one whole , we treat it as the auxiliary text and use a list of natural language descriptions of all possible classification labels as source text . When the input text contains two clearly delimited segments , one is treated as auxiliary text and the other as source text with appended natural language descriptions of possible classification labels . For regression , we employ a process similar to classification ; instead of predicting a floating-point number , we bucket the possible range and classify the text instead . Our proposal is agnostic to the details of most common preprocessing and tokenization schemes for the tasks under consideration , so for ease of exposition we assume three phases : preprocessing , encoding , and decoding . Preprocessing includes any manipulation of raw input text ; this includes tokenization . An encoder is used to extract features from the input text , and an output layer is used to decode the output from the extracted features . Encoders often include a conversion of tokens to distributed representation followed by application of several layers of LSTM , Transformer , convolutional neural network , attention , or pooling operations . In order to properly use these extracted features , the output layers often contain more inductive bias related to the specific task . For many question answering tasks , a span-decoder uses the extracted features to select a start and end token in the source document . For text classification , a linear layer and softmax allow for classification of the extracted features . Similarly , for regression , a linear layer and a similarity-scoring objective such as cosine distance or least-squares is employed . We propose to use span-decoders as the output layers for text classification and regression in place of the more standard combination of linear layer with task-specific objectives . 3.1 SPAN-EXTRACTIVE BERT ( SPEX-BERT ) . In our experiments , we start with a pre-trained BERT as the encoder with preprocessing as described in Devlin et al . ( 2018 ) . This preprocessing takes in the source text and auxiliary text and outputs a sequence of p = m+n+2 tokens : a special CLS token , the m tokens of the source text , a separator token SEP , and the n auxiliary tokens . The encoder begins by converting this sequence of tokens into a sequence of p vectors in Rd . Each of these vectors is the sum of a token embedding , a positional embedding that represents the position of the token in the sequence , and a segment embedding that represents whether the token is in the source text or the auxiliary text as described in Devlin et al . ( 2018 ) . This sequence is stacked into a matrix X0 ∈ Rp×d so that it can be processed by several Transformer layers ( Vaswani et al. , 2017 ) . The ith layer first computes αp ( Xi ) by applying selfattention with k heads over the previous layer ’ s outputs : αk ( Xi ) = [ h1 ; · · · ; hk ] Wo ( 1 ) where hj = α ( XiW 1j , XiW 2 j , XiW 3 j ) α ( X , Y , Z ) = softmax ( XY > √ d ) Z ( 2 ) A residual connection ( He et al. , 2016 ) and layer normalization ( Ba et al. , 2016 ) merge information from the input and the multi-head attention : Hi = LayerNorm ( αp ( Xi ) +Xi ) ( 3 ) This is followed by a feedforward network with ReLU activation ( Nair & Hinton , 2010 ; Vaswani et al. , 2017 ) , another residual connection , and a final layer normalization . With parameters U ∈ Rd×f and V ∈ Rf×d : Xi+1 = LayerNorm ( max ( 0 , HiU ) V +Hi ) ) ( 4 ) LetXsf ∈ Rm×d represent the final output of these Transformer layers . At this point , a task-specific head usually uses some part of Xsf to classify , regress , or extract spans . Our proposal is to use a span-decoder limited to Xsf whenever a classification or similarity-scoring layer is typically used . In this case , we add only two trainable parameter vectors dstart and dend following Devlin et al . ( 2018 ) , and we compute start and end distributions over possible spans by multiplying these vectors with Hf and applying a softmax function : pstart = softmax ( Xsfdstart ) pend = softmax ( Xsfdend ) ( 5 ) During training , we are given the ground truth answer span ( a∗ , b∗ ) as a pair of indices into the source text . The summation of cross-entropy losses over the start and end distributions then gives an overall loss for a training example : Lstart = − ∑ i I { a∗ = i } log pstart ( i ) Lend = − ∑ i I { b∗ = i } log pend ( i ) ( 6 ) With L = Lstart + Lend and at inference , we extract a span ( a , b ) as a = argmax i pstart ( i ) b = argmax i pend ( i ) ( 7 )

Summary: This paper introduces a method for converting sentence pair classification tasks and sentence regression tasks into span into span extraction tasks, by listing all the possible classes (entailment, contradiction, neural) or the discretized scores (0.0, 0.25 ...) and concatenating them with the source text. With this formulation, one can train a BERT-based span-extraction model (SpEx-BERT) on classification, regression, and QA tasks without introducing any task-specific parameters. The purposed SpEx-BERT model achieves moderate improvement (0.3 points) over the BERT-large baseline on the GLUE test set when fine-tuned on intermediate STILTs tasks (Phang et al., 2018).