Unifying Question Answering, Text Classification, and Regression via Span Extraction

1 INTRODUCTION . Pre-trained natural language processing ( NLP ) systems ( Radford et al. , 2019 ; Devlin et al. , 2018 ; Radford et al. , 2018 ; Howard & Ruder , 2018 ; Peters et al. , 2018 ; McCann et al. , 2017 ; Liu et al. , 2019b ) have been shown to transfer remarkably well on downstream tasks including text classification , question answering , machine translation , and summarization ( Wang et al. , 2018 ; Rajpurkar et al. , 2016 ; Conneau et al. , 2018 ) . Such approaches involve a pre-training phase followed by the addition of task-specific layers and a subsequent re-training or fine-tuning of the conjoined model . Each task-specific layer relies on an inductive bias related to the kind of target task . For question answering , a task-specific span-decoder is often used to extract a span of text verbatim from a portion of the input text ( Xiong et al. , 2016 ) . For text classification , a task-specific classification layer with fixed classes is typically used instead . For regression , similarity-measuring layers such as leastsquares and cosine similarity are employed . These task-specific inductive biases are unnecessary . On several tasks predominantly treated as text classification or regression , we find that reformulating them as span-extraction problems and relying on a span-decoder yields superior performance to using a task-specific layers . For text classification and regression problems , pre-trained NLP systems can benefit from supplementary training on intermediate-labeled tasks ( STILTs ) ( Phang et al. , 2018 ) , i.e . supplementary supervised training . We find this is similarly true for question answering , classification , and regression when reformulated as span-extraction . Because we rely only on the span-extractive inductive bias , we are able to further explore previously unconsidered combinations datasets . By doing this , we find that question answering tasks can benefit from text classification tasks and classification tasks can benefit from question answering ones . The success of pre-training for natural language processing systems affords the opportunity to reexamine the benefits of our inductive biases . Our results on common question answering , text classification , and regression benchmark tasks suggest that it is advantageous to discard the inductive bias that motivates task-specific , fixed-class , classification and similarity-scoring layers in favor of the inductive bias that views all three as span-extraction problems . 1.1 CONTRIBUTIONS . Summarily , we demonstrate the following : 1 . Span-extraction is an effective approach for unifying question answering , text classification , and regression . 2 . Span-extraction benefits as much from intermediate-task training as more traditional text classification and regression methods . 3 . Span-extraction allows for combinations of question answering and text classification datasets in intermediate-task training that outperform using only one or the other . 4 . Span-extractive multi-task learning yield stronger multi-task models , but weaker singletask models compared to intermediate-task training . 5 . Span-extraction with intermediate-task training proves more robust in the presence of limited training data than the corresponding task-specific versions . 2 RELATED WORK . Transfer Learning . The use of pre-trained encoders for transfer learning in NLP dates back to Collobert & Weston ( 2008 ) ; Collobert et al . ( 2011 ) but has had a resurgence in the recent past . BERT ( Devlin et al. , 2018 ) employs the recently proposed Transformer layers ( Vaswani et al. , 2017 ) in conjunction with a masked language modeling objective as a pre-trained sentence encoder . Prior to BERT , contextualized word vectors ( McCann et al. , 2017 ) were pre-trained using machine translation data and transferred to text classification and question answering tasks . ELMO ( Peters et al. , 2018 ) improved contextualized word vectors by using a language modeling objective instead of machine translation . ULMFit ( Howard & Ruder , 2018 ) and GPT ( Radford et al. , 2018 ) showed how traditional , causal language models could be fine-tuned directly for a specific task , and GPT2 ( Radford et al. , 2019 ) showed that such language models can indirectly learn tasks like machine translation , question answering , and summarization . Intermediate-task and Multi-task Learning . The goal of unifying NLP is not new ( Collobert & Weston , 2008 ; Collobert et al. , 2011 ) . In Phang et al . ( 2018 ) , the authors explore the efficacy of supplementary training on intermediate tasks , a framework that the authors abbreviate as STILTs . Given a target task T and a pre-trained sentence encoder , they first fine-tune the encoder on an intermediate ( preferably related ) task I and then finally fine-tune on the task T . The authors showed that such an approach has several benefits including improved performance and better robustness to hyperparameters . The authors primarily focus on the GLUE benchmark ( Wang et al. , 2018 ) . Liu et al . ( 2019a ) explore the same task and model class ( viz. , BERT ) in the context of multi-tasking . Instead of using supplementary training , the authors choose to multi-task on the objectives and , similar to BERT on STILTs , fine-tune on the specific datasets in the second phase . Further improvements can be obtained through heuristics such as knowledge distillation as demonstrated in Clark et al . ( 2019 ) . All of these approaches require a different classifier head for each task , e.g. , a two-way classifier for SST and a three-way classifier for MNLI . Two recent approaches : decaNLP ( McCann et al. , 2018 ) and GPT-2 Radford et al . ( 2019 ) propose the unification of NLP as question answering and language modeling , respectively . As investigated in this work , the task description is provided in natural language instead of fixing the classifier a-priori . 3 METHODS . We propose treating question answering , text classification , and regression as span-extractive tasks . Each input is split into two segments : a source text which contains the span to be extracted and an auxiliary text that is used to guide extraction . Question answering often fits naturally into this framework by providing both a question and a context document that contains the answer to that question . When treated as span-extraction , the question is the auxiliary text and the context document is the source text from which the span is extracted . Text classification input text most often does not contain a natural language description of the correct class . When it is more natural to consider the input text as one whole , we treat it as the auxiliary text and use a list of natural language descriptions of all possible classification labels as source text . When the input text contains two clearly delimited segments , one is treated as auxiliary text and the other as source text with appended natural language descriptions of possible classification labels . For regression , we employ a process similar to classification ; instead of predicting a floating-point number , we bucket the possible range and classify the text instead . Our proposal is agnostic to the details of most common preprocessing and tokenization schemes for the tasks under consideration , so for ease of exposition we assume three phases : preprocessing , encoding , and decoding . Preprocessing includes any manipulation of raw input text ; this includes tokenization . An encoder is used to extract features from the input text , and an output layer is used to decode the output from the extracted features . Encoders often include a conversion of tokens to distributed representation followed by application of several layers of LSTM , Transformer , convolutional neural network , attention , or pooling operations . In order to properly use these extracted features , the output layers often contain more inductive bias related to the specific task . For many question answering tasks , a span-decoder uses the extracted features to select a start and end token in the source document . For text classification , a linear layer and softmax allow for classification of the extracted features . Similarly , for regression , a linear layer and a similarity-scoring objective such as cosine distance or least-squares is employed . We propose to use span-decoders as the output layers for text classification and regression in place of the more standard combination of linear layer with task-specific objectives . 3.1 SPAN-EXTRACTIVE BERT ( SPEX-BERT ) . In our experiments , we start with a pre-trained BERT as the encoder with preprocessing as described in Devlin et al . ( 2018 ) . This preprocessing takes in the source text and auxiliary text and outputs a sequence of p = m+n+2 tokens : a special CLS token , the m tokens of the source text , a separator token SEP , and the n auxiliary tokens . The encoder begins by converting this sequence of tokens into a sequence of p vectors in Rd . Each of these vectors is the sum of a token embedding , a positional embedding that represents the position of the token in the sequence , and a segment embedding that represents whether the token is in the source text or the auxiliary text as described in Devlin et al . ( 2018 ) . This sequence is stacked into a matrix X0 ∈ Rp×d so that it can be processed by several Transformer layers ( Vaswani et al. , 2017 ) . The ith layer first computes αp ( Xi ) by applying selfattention with k heads over the previous layer ’ s outputs : αk ( Xi ) = [ h1 ; · · · ; hk ] Wo ( 1 ) where hj = α ( XiW 1j , XiW 2 j , XiW 3 j ) α ( X , Y , Z ) = softmax ( XY > √ d ) Z ( 2 ) A residual connection ( He et al. , 2016 ) and layer normalization ( Ba et al. , 2016 ) merge information from the input and the multi-head attention : Hi = LayerNorm ( αp ( Xi ) +Xi ) ( 3 ) This is followed by a feedforward network with ReLU activation ( Nair & Hinton , 2010 ; Vaswani et al. , 2017 ) , another residual connection , and a final layer normalization . With parameters U ∈ Rd×f and V ∈ Rf×d : Xi+1 = LayerNorm ( max ( 0 , HiU ) V +Hi ) ) ( 4 ) LetXsf ∈ Rm×d represent the final output of these Transformer layers . At this point , a task-specific head usually uses some part of Xsf to classify , regress , or extract spans . Our proposal is to use a span-decoder limited to Xsf whenever a classification or similarity-scoring layer is typically used . In this case , we add only two trainable parameter vectors dstart and dend following Devlin et al . ( 2018 ) , and we compute start and end distributions over possible spans by multiplying these vectors with Hf and applying a softmax function : pstart = softmax ( Xsfdstart ) pend = softmax ( Xsfdend ) ( 5 ) During training , we are given the ground truth answer span ( a∗ , b∗ ) as a pair of indices into the source text . The summation of cross-entropy losses over the start and end distributions then gives an overall loss for a training example : Lstart = − ∑ i I { a∗ = i } log pstart ( i ) Lend = − ∑ i I { b∗ = i } log pend ( i ) ( 6 ) With L = Lstart + Lend and at inference , we extract a span ( a , b ) as a = argmax i pstart ( i ) b = argmax i pend ( i ) ( 7 )

Summary: This paper asks whether it works to remove task-specific heads and treat classification and regression problems as span extraction, by formatting problems in such a way that a single span extraction model can be used.  This is a reasonable question to ask, and the authors performed a very large number of experiments attempting to answer this question.  The authors claim that using span extractive models instead of task-specific heads yields improved performance over separate heads.