Deepening Hidden Representations from Pre-trained Language Models

1 INTRODUCTION . Language representation is essential to the understanding of text . Recently , pre-trained language models based on Transformer ( Vaswani et al. , 2017 ) such as GPT ( Radford et al. , 2018 ) , BERT ( Devlin et al. , 2019 ) , XLNet ( Yang et al. , 2019 ) , and RoBERTa ( Liu et al. , 2019c ) have been shown to be effective for learning contextualized language representation . These models have since continued to achieve new state-of-the-art results on a variety of natural language processing tasks . They include question answering ( Rajpurkar et al. , 2018 ; Lai et al. , 2017 ) , natural language inference ( Williams et al. , 2018 ; Bowman et al. , 2015 ) , named entity recognition ( Tjong Kim Sang & De Meulder , 2003 ) , sentiment analysis ( Socher et al. , 2013 ) and semantic textual similarity ( Cer et al. , 2017 ; Dolan & Brockett , 2005 ) . Normally , Transformer-based models are pre-trained on large-scale unlabeled corpus in an unsupervised manner , and then fine-tuned on the downstream tasks through introducing task-specific output layer . When fine-tuning on the supervised downstream tasks , the models pass directly the output of Transformer encoder ’ s final layer , which is considered as the contextualized representation of input text , to the task-specific layer . However , due to the numerous layers ( i.e. , Transformer blocks ) and considerable depth of these pretrained models , we argue that the output of the last layer may not always be the best representation of the input text during the fine-tuning for downstream tasks . Devlin et al . ( 2019 ) shows diverse combinations of different layers ’ outputs of the pre-trained BERT result in distinct performance on CoNLL-2003 Named Entity Recognition ( NER ) task ( Tjong Kim Sang & De Meulder , 2003 ) . Peters et al . ( 2018b ) points out for pre-trained language models , including Transformer , the most transferable contextualized representations of input text tend to occur in the middle layers , while the top layers specialize for language modeling . Therefore , the onefold use of the last layer ’ s output may restrict the power of the pre-trained representation . In this paper , we propose an extra network component design for Transformer-based model , which is capable of adaptively leveraging the hidden information in the Transformer ’ s hidden layers to refine the language representation . Our introduced additional components include two main additional components : 1 . HIdden Representation Extractor ( HIRE ) dynamically learns a complementary representation which contains the information that the final layer ’ s output fails to capture . 2 . Fusion network integrates the hidden information extracted by the HIRE with Transformer final layer ’ s output through two steps of functionalities , leading to a refined contextualized language representation . Taking advantage of the robustness of RoBERTa by using it as our backbone Transformer-based encoder ( Liu et al. , 2019c ) , we conduct experiments on GLUE benchmark ( Wang et al. , 2018 ) , which consists of nine Natural Language Understanding ( NLU ) tasks . With the help of HIRE , our model outperforms the baseline on 5/9 of them and advances the state-of-the-art on SST-2 dataset . Keeping the backbone Transformer model unchanged on its architecture , pre-training procedure and training objectives , we get comparable performance with other state-of-the-art models on the GLUE leaderboard , which verifies the effectiveness of the proposed HIRE enhancement over Transformer model . 2 MODEL 2.1 TRANSFORMER-BASED ENCODER . Transformer-based encoder is responsible for encoding input text into contextualized representation . Let { w1 , . . . , wn } represent a sequence of nwords of input text , Transformer-based encoder encodes the input sequence into its universal contextualized representationR ∈ Rn×d : R = Encoder ( { w1 , . . . , wn } ) ( 1 ) where d is the hidden size of the encoder and R is the output of Transformer-based encoder ’ s last layer which has the same length as the input text . We call it preliminary representation in this paper to distinguish it with the one that we introduce in section 2.2 . Here , we omit a rather extensive formulations of Transformer and refer readers to Vaswani et al . ( 2017 ) , Radford et al . ( 2018 ) and Devlin et al . ( 2019 ) for more details . 2.2 HIDDEN REPRESENTATION EXTRACTOR . Transformer-based encoder normally has many structure-identical layers stacked together , for example , BERTLARGE and XLNetLARGE all contain 24 layers of the identical structure , either outputs from these hidden layers or the last layer , but not only limited to the latter , may be extremely helpful for specific downstream task . To make full use of the representations from these hidden layers , we introduce an extra component attached to the original encoder , HIdden Representation Extractor ( HIRE ) to capture the complementary information that the output of the last layer fails to capture . Since each layer does not take the same importance to represent a certain input sequence for different downstream tasks , we design an adaptive mechanism that can compute the importance dynamically . We measure the importance by an importance score . The input to the HIRE is { H0 , . . . , Hj , . . . , Hl } where l represents the number of layers in the encoder . Here H0 is the initial embedding of input text , which is the input of the encoder ’ s first layer but is updated during training and Hj ∈ Rn×d is the hidden-state of the encoder at the output of layer j . For the sake of simplicity , we call them all hidden-state afterwards . We use the same 2-layer Bidirectional Gated Recurrent Unit ( GRU ) ( Cho et al. , 2014 ) to summarize each hidden-state of the encoder . Instead of taking the whole output of GRU as the representation of the hidden state , we concatenate GRU ’ s each layer and each direction ’ s final state together . In this way , we manage to summarize the hidden-state into a fixed-sized vector . Hence , we obtain U ∈ R ( l+1 ) ×4d with ui the summarized vector ofHi : ui = Bi-GRU ( Hi ) ∈ R4d ( 2 ) where 0 ≤ i ≤ l. Then the importance value αi for hidden-stateHi is calculated by : αi = ReLU ( W Tui + b ) ∈ R ( 3 ) whereW ∈ R4d×1 and b ∈ R are trainable parameters . Letα= { αi } be normalized into a probability distribution s through a softmax layer : s = softmax ( α ) ∈ Rl+1 ( 4 ) where si is the normalized weight of hidden-state i when computing the representation . Subsequently , we obtain the input sequence ’ s new representationA by : A = l+1∑ i=0 siHi ∈ Rn×d ( 5 ) With the same shape as the output of Transformer-based encoder ’ s final layer , HIRE ’ s output A is expected to contain the additional useful information from the encoder ’ s hidden-states and we call it complementary representation . 2.3 FUSION NETWORK . This module fuses the information contained in the output of Transformer-based encoder and the one extracted from encoders ’ hidden states by HIRE . Given the preliminary representation R , instead of letting it flow directly into task-specfic output layer , we combine it together with the complementary representationA to yieldM , defined by : M = [ R ; A ; R+A ; R ◦A ] ∈ Rn×4d ( 6 ) where ◦ is element-wise multiplication ( Hadamard Product ) and [ ; ] is concatenation across the last dimension . Later , two-layer bidirectional GRU , with the output size of d for each direction , is used to fully fuse the information contained in the preliminary representation and the complementary representation . We concatenate the outputs of the GRUs in two dimensions together for the final contextualized representation : F = Bi-GRU ( M ) ∈ Rn×2d ( 7 ) 2.4 OUTPUT LAYER . The output layer is task-specific . The following are the concerned implementation details on two tasks , classification and regression . For classification task , given the input text ’ s contextualized representation F , following Devlin et al . ( 2019 ) , we take the first row c ∈ R2d of F corresponding to the first input token ( < s > ) as the aggregated representation . Let m be the number of labels in the datasets , we pass c through a feed-forward network ( FFN ) : q =W T2 · tanh ( W T1 c+ b1 ) + b2 ∈ Rm ( 8 ) with W1 ∈ R2d×d , W2 ∈ Rd×m , b1 ∈ Rd and b2 ∈ Rm the only parameters that we introduce in output layer . Finally , the probability distribution of predicted label is computed as : p = softmax ( q ) ∈ Rm ( 9 ) For regression task , we obtain q in the same manner with m = 1 , and take q as the predicted value . 2.5 TRAINING . For classification task , the training loss to be minimized is defined by the Cross-Entropy : L ( θ ) = − 1 T T∑ i=1 log ( pi , c ) ( 10 ) where θ is the set of all parameters in the model , T is the number of examples in the dataset , pi , c is the predicted probability of gold class c for example i . For regression task , we define the training loss by mean squared error ( MSE ) : L ( θ ) = 1 T T∑ i=1 ( qi − yi ) 2 ( 11 ) where qi is the predicted value for example i and yi is the ground truth value for example i . 3 EXPERIMENTS . 3.1 DATASET . We conducted the experiments on the General Language Understanding Evaluation ( GLUE ) benchmark ( Wang et al. , 2018 ) to evaluate our proposed method . GLUE is a collection of 9 diverse datasets1 for training , evaluating , and analyzing natural language understanding models . 3.2 MAIN RESULTS . 1All the datasets can be obtained from https : //gluebenchmark.com/tasks Table 1 compares our method with a list of Transformer-based models on the development set . Model parameter comparison is shown in Table 2 . To obtain a direct and fair comparison with our baseline model RoBERTa , following the original paper ( Liu et al. , 2019c ) , we fine-tune RoBERTa+HIRE separately for each of the GLUE tasks , using only task-specific training data . The single-model results for each task are reported . We run our model with five different random seeds but the same hyperparameters and take the median value . Due to the problematic nature of WNLI dataset , we exclude its results in this table . The results shows that RoBERTa+HIRE consistently outperforms RoBERTa on 4 of the GLUE task development sets , with an improvement of 1.7 points , 0.4 points , 0.5/0.2 points , 0.3 points on CoLA , SST-2 , MNLI and QNLI respectively . And on the MRPC , STS-B and RTE task , our model get the same result as RoBERTa . It should be noted that the improvement is entirely attributed to the introduction of HIdden Representation Extractor and fusion network in our model . Table 3 presents the results of HIRE enhancement and other models on the test set that have been submitted to the GLUE leaderboard . Following Liu et al . ( 2019c ) , we fine-tune STS-B and MRPC starting from the MNLI single-task model . Given the simplicity between RTE , WNLI and MNLI , and the large-scale nature of MNLI dataset ( 393k ) , we also initialize RoBERTa+HIRE with the weights of MNLI single-task model before fine-tuning on RTE and WNLI . We submitted the ensemble-model results to the leaderboard . The results show that RoBERTa+HIRE still boosts the strong RoBERTa baseline model on the test set . To be specific , RoBERTa+HIRE outperforms RoBERTa over CoLA , SST-2 , MRPC , SST-B , MNLI-mm and QNLI with an improvement of 0.8 points , 0.4 points , 0.7/0.9 points , 0.2/0.1 points , 0.2 points and 0.1 points respectively . In the meantime , RoBERTa+HIRE gets the same results as RoBERTa on QQP and WNLI . By category , RoBERTa+HIRE has better performance than RoBERTa on the single sentence tasks , similarity and paraphrase tasks . It is worth noting that our model obtains state-of-the-art results on SST-2 dataset , with a score of 97.1 . The results are quite promising since HIRE does not modify the encoder internal architecture ( Yang et al. , 2019 ) or redefine the pre-training procedure ( Liu et al. , 2019c ) , getting the comparable results with them .

Summary: This paper presents a new mechanism, called HIRE, to extract more information from the intermediate layers of pre-trained models, which will be further fused with the last layer of pre-trained models. The main contribution of this work is the newly proposed dynamic feature extractor HIRE and the fusion network. Experiments confirmed the effectiveness of the proposed method, and some interesting observations on the importance of different layers for different tasks were given (i.e. Figure 2).