StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding

1 INTRODUCTION . A pre-trained language model ( LM ) is a key component in many natural language understanding ( NLU ) tasks such as semantic textual similarity ( Cer et al. , 2017 ) , question answering ( Rajpurkar et al. , 2016 ) and sentiment classification ( Socher et al. , 2013 ) . In order to obtain reliable language representations , neural language models are designed to define the joint probability function of sequences of words in text with self-supervised learning . Different from traditional word-specific embedding in which each token is assigned a global representation , recent work , such as Cove ( McCann et al. , 2017 ) , ELMo ( Peters et al. , 2018 ) , GPT ( Radford et al. , 2018 ) and BERT ( Devlin et al. , 2018 ) , derives contextualized word vectors from a language model trained on a large text corpus . These models have been shown effective for many downstream NLU tasks . Among the context-sensitive language models , BERT ( and its robustly optimized version RoBERTa ( Liu et al. , 2019b ) ) has taken the NLP world by storm . It is designed to pre-train bidirectional representations by jointly conditioning on both left and right context in all layers and model the representations by predicting masked words only through the contexts . However , it does not make the most of underlying language structures . According to Elman ( Elman , 1990 ) ’ s study , the recurrent neural networks was shown to be sensitive to regularities in word order in simple sentences . Since language fluency is determined by the ordering of words and sentences , finding the best permutation of a set of words and sentences is an essential problem in many NLP tasks , such as machine translation and NLU ( Hasler et al. , 2017 ) . Recently , word ordering was treated as LM-based linearization solely based on language models ( Schmaltz et al. , 2016 ) . Schmaltz showed that recurrent neural network language models ( Mikolov et al. , 2010 ) with long short-term memory ( Hochreiter & Schmidhuber , 1997 ) cells work effectively for word ordering even without any explicit syntactic information . In this paper , we introduce a new type of contextual representation , StructBERT , which incorporates language structures into BERT pre-training by proposing two novel linearization strategies . Specifically , in addition to the existing masking strategy , StructBERT extends BERT by leveraging the structural information : word-level ordering and sentence-level ordering . We augment model pre-training with two new structural objectives on the inner-sentence and inter-sentence structures , respectively . In this way , the linguistic aspects ( Elman , 1990 ) are explicitly captured during the pre-training procedure . With structural pre-training , StructBERT encodes dependency between words as well as sentences in the contextualized representation , which provides the model with better generalizability and adaptability . StructBERT significantly advances the state-of-the-art results on a variety of NLU tasks , including the GLUE benchmark ( Wang et al. , 2018 ) , the SNLI dataset ( Bowman et al. , 2015 ) and the SQuAD v1.1 question answering task ( Rajpurkar et al. , 2016 ) . All of these experimental results clearly demonstrate StructBERT ’ s exceptional effectiveness and generalization capability in language understanding . We make the following major contributions : • We propose novel structural pre-training that extends BERT by incorporating the word structural objective and the sentence structural objective to leverage language structures in contextualized representation . This enables the StructBERT to explicitly model language structures by forcing it to reconstruct the right order of words and sentences for correct prediction . • StructBERT significantly outperforms all published state-of-the-art models on a wide range of NLU tasks at the time of model submission . This model extends the superiority of BERT , and boosts the performance in many language understanding applications such as semantic textual similarity , sentiment analysis , textual entailment , and question answering . 2 STRUCTBERT MODEL PRE-TRAINING . StructBERT builds upon the BERT architecture , which uses a multi-layer bidirectional Transformer network ( Vaswani et al. , 2017 ) . Given a single text sentence or a pair of text sentences , BERT packs them in one token sequence and learns a contextualized vector representation for each token . Every input token is represented based on the word , the position , and the text segment it belongs to . Next , the input vectors are fed into a stack of multi-layer bidirectional Transformer blocks , which uses self-attention to compute the text representations by considering the entire input sequence . The original BERT introduces two unsupervised prediction tasks to pre-train the model : i.e. , a masked LM task and a next sentence prediction task . Different from original BERT , our StructBERT amplifies the ability of the masked LM task by shuffling certain number of tokens after word masking and predicting the right order . Moreover , to better understand the relationship between sentences , StructBERT randomly swaps the sentence order and predicts the next sentence and the previous sentence as a new sentence prediction task . In this way , the new model not only explicitly captures the fine-grained word structure in every sentence , but also properly models the inter-sentence structure in a bidirectional manner . Once the StructBERT language model is pre-trained with these two auxiliary tasks , we can fine-tune it on task-specific data for a wide range of downstream tasks . 2.1 INPUT REPRESENTATION . Every input x is a sequence of word tokens , which can be either a single sentence or a pair of sentences packed together . The input representation follows that used in BERT ( Devlin et al. , 2018 ) . For each input token ti , its vector representation xi is computed by summing the corresponding token embedding , positional embedding , and segment embedding . We always add a special classification embedding ( [ CLS ] ) as the first token of every sequence , and a special end-of-sequence ( [ SEP ] ) token to the end of each segment . Texts are tokenized to subword units by WordPiece ( Wu et al. , 2016 ) and absolute positional embeddings are learned with supported sequence lengths up to 512 tokens . In addition , the segment embeddings are used to differentiate a pair of sentences as in BERT . 2.2 TRANSFORMER ENCODER . We use a multi-layer bidirectional Transformer encoder ( Vaswani et al. , 2017 ) to encode contextual information for input representation . Given the input vectors X = { xi } Ni=1 , an L-layer Transformer is used to encode the input as : Hl = Transformerl ( Hl−1 ) ( 1 ) where l ∈ [ 1 , L ] , H0 = X and HL = [ hL1 , · · · , hLN ] . We use the hidden vector hLi as the contextualized representation of the input token ti . 2.3 PRE-TRAINING OBJECTIVES . To make full use of the rich inner-sentence and inter-sentence structures in language , we extend the pre-training objectives of original BERT in two ways : 1© word structural objective ( mainly for the single-sentence task ) , and 2© sentence structural objective ( mainly for the sentence-pair task ) . We pre-train these two auxiliary objectives together with the original masked LM objective in a unified model to exploit inherent language structures . 2.3.1 WORD STRUCTURAL OBJECTIVE . Despite its success in various NLU tasks , original BERT is unable to explicitly model the sequential order and high-order dependency of words in natural language . Given a set of words in random order from a sentence , ideally a good language model should be able to recover this sentence by reconstructing the correct order of these words . To implement this idea in StructBERT , we supplement BERT ’ s training objectives with a new word structural objective which endows the model with the ability to reconstruct the right order of certain number of intentionally shuffled word tokens . This new word objective is jointly trained together with the original masked LM objective from BERT . Figure 1a illustrates the procedure of jointly training the new word objective and the masked LM objective . In every input sequence , we first mask 15 % of all tokens at random , as done in BERT ( Devlin et al. , 2018 ) . The corresponding output vectors hLi of the masked tokens computed by the bidirectional Transformer encoder are fed into a softmax classifier to predict the original tokens . Next , the new word objective comes into play to take word order into consideration . Given the randomicity of token shuffling , the word objective is equivalent to maximizing the likelihood of placing every shuffled token in its correct position . More formally , this objective can be formulated as : argmax θ ∑ logP ( pos1 = t1 , pos2 = t2 , . . . , posK = tK |t1 , t2 , . . . , tK , θ ) , ( 2 ) where θ represents the set of trainable parameters of StructBERT , and K indicates the length of every shuffled subsequence . Technically , a larger K would force the model to be able to reconstruct longer sequences while injecting more disturbed input . On the contrary , when K is smaller , the model gets more undisturbed sequences while less capable of recovering long sequences . We decide to use trigrams ( i.e. , K = 3 ) for subsequence shuffling to balance language reconstructability and robustness of the model . Specifically , as shown in Figure 1a , we randomly choose some percentage of trigrams from unmasked tokens , and shuffle the three words ( e.g. , t2 , t3 , and t4 in the figure ) within each of the trigrams . The output vectors of the shuffled tokens computed by the bidirectional Transformer encoder are then fed into a softmax classifier to predict the original tokens . The new word objective is jointly learned together with the masked LM objective in a unified pre-trained model with equal weights . 2.3.2 SENTENCE STRUCTURAL OBJECTIVE . The next sentence prediction task is considered easy for the original BERT model ( the prediction accuracy of BERT can easily achieve 97 % -98 % in this task ( Devlin et al. , 2018 ) ) . We , therefore , extend the sentence prediction task by predicting both the next sentence and the previous sentence , to make the pre-trained language model aware of the sequential order of the sentences in a bidirectional manner . As illustrated in Figure 1b , given a pair of sentences ( S1 , S2 ) as input , we predict whether S2 is the next sentence that follows S1 , or the previous sentence that precedes S1 , or a random sentence from a different document . Specifically , for the sentence S1 , 13 of the time we choose the text span that follows S1 as the second sentence S2 , 13 of the time the previous sentence ahead of S1 is selected , and 13 of the time a sentence randomly sampled from the other documents is used as S2 . The two sentences are concatenated together into an input sequence with the separator token [ SEP ] in between , as done in BERT . We pool the model output by taking the hidden state corresponding to the first token [ CLS ] , and feed the encoding vector of [ CLS ] into a softmax classifier to make a three-class prediction . 2.4 PRE-TRAINING SETUP . The training objective function is a linear combination of the word structural objective and the sentence structural objective . For the masked LM objective , we followed the same masking rate and settings as in BERT ( Devlin et al. , 2018 ) . 5 % of trigrams are selected for random shuffling . We used documents from English Wikipedia ( 2,500M words ) and BookCorpus ( Zhu et al. , 2015 ) as pre-training data , following the preprocessing and the WordPiece tokenization from ( Devlin et al. , 2018 ) . The maximum length of input sequence was set to 512 . We ran Adam with learning rate of 1e-4 , β1 = 0.9 , β2 = 0.999 , L2 weight decay of 0.01 , learning rate warm-up over the first 10 % of the total steps , and linear decay of the learning rate . We set a dropout probability of 0.1 for every layer . The gelu activation ( Hendrycks & Gimpel , 2016 ) was used as done in GPT ( Radford et al. , 2018 ) . We denote the number of Transformer block layers as L , the size of hidden vectors as H , and the number of self-attention heads as A . Following the practice of BERT , We primarily report experimental results on the two model sizes : StructBERTBase : L = 12 , H = 768 , A = 12 , Number of parameters= 110M StructBERTLarge : L = 24 , H = 1024 , A = 16 , Number of parameters= 340M Pre-training of StructBERT was performed on a distributed computing cluster consisting of 64 Telsa V100 GPU cards . For the StructBERTBase , we ran the pre-training procedure for 40 epochs , which took about 38 hours , and the training of StructBERTLarge took about 7 days to complete .

Summary: This paper proposed a new pre-trained language model based on BERT, called StructBERT. The key contributions are the two new pre-train objectives, (1) word structural objective, where the goal is to reconstruct the right order of intentionally shuffled word tokens, and (2) sentence structural objective, a three-class sentence-pair prediction, either the 2nd sentence precedes the 1st, the 2nd succeeds the 1st, or the 2nd is randomly selected. Unlike the original NSP (next sentence prediction) task, which is simple but tends out to be not so helpful in many downstream tasks, both proposed pre-train objectives seem to be rather useful in benchmarks tested in the paper, including GLUE, SNLI, and SQuAD.