StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding
Authors: Wei Wang
Bin Bi
Ming Yan
Chen Wu
Jiangnan Xia
Zuyi Bao
Liwei Peng
Luo Si
Venue:  (2020)

Abstract: Recently, the pre-trained language model, BERT (and its robustly optimized version RoBERTa), has attracted a lot of attention in natural language understanding (NLU), and achieved state-of-the-art accuracy in various NLU tasks, such as sentiment classification, natural language inference, semantic textual similarity and question answering. Inspired by the linearization exploration work of Elman, we extend BERT to a new model, StructBERT, by incorporating language structures into pre-training. Specifically, we pre-train StructBERT with two auxiliary tasks to make the most of the sequential order of words and sentences, which leverage language structures at the word and sentence levels, respectively. As a result, the new model is adapted to different levels of language understanding required by downstream tasks.

The StructBERT with structural pre-training gives surprisingly good empirical results on a variety of downstream tasks, including pushing the state-of-the-art on the GLUE benchmark to 89.0 (outperforming all published models at the time of model submission), the F1 score on SQuAD v1.1 question answering to 93.0, the accuracy on SNLI to 91.7.

Published as a conference paper at ICLR 2020
STRUCTBERT:
INCORPORATING LANGUAGE STRUCTURES INTO PRE- TRAINING FOR DEEP LANGUAGE UNDERSTANDING
Wei Wang, Bin Bi, Ming Yan, Chen Wu, Jiangnan Xia, Zuyi Bao, Liwei Peng and Luo Si Alibaba Group Inc.
fhebian.ww. ww, b.bi, yml19608 nuchen.we, jiangnan, xajng liwei .eng, luc. ea baba -nn.
ABSTRACT
Recently, the pre-trained language model, BERT (and its robustly optimized version RoBERTa), has attracted lot of attention in natural language understanding (NLU), and achieved state-of-the-an ofrtte-artt accuracy various NL. tasks, such as sentiment classification, natural language inference, semantic textual similarity and question answering. Inspired by the lincarization exploration work of Elman (Elman 1990), extend BERT to new model, StructBERT, by incorporating language structures into pre-training. Specifically, we pre-train StruetBERT with two auxiliary tasks make the most of the sequential order of words and sentences, which leverage language structures at the word and sentence levels, respectively. As result, the new model is adapted to different levels of language understanding required by downstream tasks.
The StructBERT with structural pre-training gives surprisingly good empirical results on a variety downstream tasks, including pushing the state-of-the-art on the GLUE benchmark to 89.0 (outperforming all published models at the time of model submission), the score on SQuAD v1.1 question answering to 93.0, the accuracy SNLI 91.7.
INTRODUCTION
A pre-trained language model (LM) is key component in many natural language understanding (NLU) tasks such as semantic textual similarity (Cer et al. 2017). question answering (Rajpurkar al. 2016) and sentiment classification (Socher et al. (013). 3. In order to obtain reliable language rep- resentations, neural language models are designed to define the joint probability function of sequences of words in text with self-supervised learning. Different from traditional word-specific embedding in which each token is assigned a global representation, recent work, such as Cove (McCann et al. 2017), ML (Peters et al. 2018), GPT (Radford et al.) (018) and BERT (Devlin et al.) 2018)) derives contextualized word vectors from language model trained on large text corpus. These models have been shown effective for many downstream NLU tasks.
Among the context-sensitive language models, BERT (and its robustly optimized version RoBERTa (Liu et al.) 2019b)) has taken the NLP world by storm. is designed pre-train bidirectional representations by jointly conditioning on both left and right context in all layers and model the representations by predicting masked words only through the contexts. However, does not make the most of underlying language structures.
According to Elman (Elman 1990)'s ss study, the recurrent neural networks was shown to be sensitive to regularities in word order in simple sentences. Since language fluency is determined by the ordering of words and sentences, finding the best permutation of set of words and sentences an essential problem in many NLP tasks, such as machine translation and NLU (Hasler et al. 2017). Recently, word ordering was treated as LM-based linearization solely based on language models (Schmaltz al. 2016). Schmaltz showed that recurrent neural network language models (Mikolov al. 2010) with long short-term memory (Hochreiter Schmidhuber. 1997) cells work effectively for word ordering even without any explicit syntactic information.

---

Published as a conference paper at ICLR 2020
In this paper, we introduce new type of contextual representation, StructBERT, which incorpo- rates language structures into BERT pre-training by proposing two novel linearization strategies. Specifically, in addition to the existing masking strategy, StructBERT extends BERT by leveraging the structural information: word-level ordering and sentence-level ordering. We augment model pre-training with respectively. In this way, the linguistic aspects (Elman 1990) are explicitly captured during the pre-training procedure. With structural pre-training, StruetBERT encodes dependency between words as well sentences in the contextualized representation, which provides the model with better generalizability and adaptability.
structural objectives the inner-sentence and inter-sentence structures,
StructBERT significantly advances the state-of-the-ase results on variety of NLU tasks, including the GLUE benchmark (Wang et al. 2018), the SNLI dataset (Bowman et al. 2015) and the SQuAD v1.1 question answering task (Rajjurkar et al. 2016). All of these experimental results clearly demonstrate StructBERT's exceptional effectiveness and generalization capability in language understanding.
We make the following major contributions:
We propose novel structural pre-training that extends BERT by incorporating the word structural objective and the sentence structural objective to leverage language structures in contextualized representation. This enables the StructBERT to explicitly model language structures by forcing to reconstruct the right order of words and sentences for correct prediction.
StructBERT sigmificantly outperforms all published state-of-the-as models on a wide range of NLU tasks at the time of model submission. This model extends the superiority of BERT, and boosts the performance many language understanding applications such as semantic textual similarity, sentiment analysis, textual entailment, and question answering.
2 STRUCTBERT MODEL PRE-TRAINING
StructBERT builds upon the BERT architecture, which uses a multi-layer bidirectional Transformer network (Vaswani et al., 2017). Given single text sentence pair of text sentences, BERT packs them in one token sequence and learns a contextualized vector representation for each token. Every input token is represented based on the word, the position, and the text segment it belongs to. Next, the input vectors are fed into a stack of multi-layer bidirectional Transformer blocks, which uses self-attention compute the text representations by considering the entire input sequence.
The original BERT introduces two unsupervised prediction tasks to pre-train the model: i.c., a masked LM task and next sentence prediction task. Different from original BERT, our StructBERT amplifies the ability of the masked LM task by shuffling certain number of tokens after word masking and predicting the right order. Moreover, better understand the relationship between sentences, StructBERT randomly swaps the sentence order and predicts the next sentence and the previous sentence a new sentence prediction task. In this way, the new model not only explicitly captures the fine-grained word structure in every sentence, but also properly models the inter-sentence structure in a bidirectional manner. Once the StructBERT language model is pre-trained with these two auxiliary tasks, we can fine-tune it on task-specific data for wide range of downstream tasks.
2.1 INPUT REPRESENTATION
Every input x a sequence of word tokens, which can be either a single sentence or pair of sentences packed together. The input representation follows that used in BERT (Devlin et al., all,20188 8). For each input token f;, its vector representation X; is computed by summing the corresponding token embedding, positional embedding, and segment embedding. We always add special classification embedding ([CLS]) as the first token of every sequence, and a special end-of-sequence ([SEP) token to the end of each segment. Texts tokenized subword units by WordPiece (Wu et al.) 2016) and absolute positional embeddings are learned with supported sequence lengths up to 512 tokens. In addition, the segment embeddings are used to differentiate a pair of sentences in BERT.
2

---

Published as a conference paper at ICLR 2020
Treadeemam Elarnderr
Teansfirer Rocrader
SEX
SL
Steelerr
Ri
Mi
Ai
Tranfarasr Enceder
Polteed
Pr
Pi
Pr
Corpar
14kmn
jrent
Trigren
Productos
(a) Word Structural Objective
(b) Sentence Structural Objective
Figure 1: Illustrations of the two new pre-training objectives
2.2 TRANSFORMER ENCODER
We multi layer bidirectional Transformer encoder (Vaswani et al. 2017) to encode contextual information for input representation. Given the input vectors [x, 1= an L-layer Transformer used encode the input as:
H' = Trans formero
(I)
where / E [1, LJ. H = X and H ....... h. We use the hidden vector by the contextualized representation of the input token ty.
2.3 PRE-TRAINING OBJECTIVES
To make full use of the rich inner-sentence and inter-sentence structures in language, extend the pre-training objectives of original BERT in two ways: ( word structural objective (mainly for the single-sentence task), and 2 sentence structural objective (mainly for the sentence-paie task). We pre-train these two auxiliary objectives together with the original masked LM objective in a unified model exploit inherent language structures.
2.3.1 WORD STRUCTURAL OBJECTIVE
Despite its success in various NLU tasks, original BERT is unable to explicitly model the sequential order and high-order dependency of words natural language. Given a set words in random order from sentence, ideally good language model should be able recover this sentence by reconstructing the correct order of these words. To implement this idea in StructBERT, we supplement BERT's training objectives with new word structural objective which endows the model with the ability to reconstruct the right order of certain number of intentionally shuffled word tokens. This new word objective jointly trained together with the original masked LM objective from BERT.
Figure aa illustrates the procedure of jointly training the new word objective and the masked LM objective. In every input sequence, we first mask 15% of all tokens random, as done in BERT (De- vlin et al. 2018 The corresponding output vectors hy of the masked tokens computed by the bidirectional Transformer encoder are fed into softmax classifier to predict the original tokens.
Next, the new word objective comes into play take word order into consideration. Given the randomicity of token shuffling, the word objective is equivalent to maximizing the likelihood of placing every shuffled token in correct position. More formally, this objective can be formulated as:
arg Max Σ log P(pos = I, pos, .,, pos LK (4,, ....... ,,.,,,,,,,0 Lx,
where 0 represents the set trainable parameters of StructBERT, and K indicates the length of every shuffled subsequence. Technically, larger K would force the model to be able to reconstruct longer sequences while injecting more disturbed input. On the contrary, when K is smaller, the model gets more undisturbed sequences while less capable of recovering long sequences. We decide to use trigrams (i.e., K = 3) for subsequence shuffling to balance language reconstructability and robustness of the model.
3

---

Published as a conference paper at ICLR 2020
Specifically, as shown in Figure la randomly choose some percentage of trigrams from unmasked tokens, and shuffle the three words (c.g.. 12, t3. and & in the figure) within each of the trigrams. The output vectors of the shuffled tokens computed by the bidirectional Transformer encoder are then fed into a softmax classifier to predict the original tokens. The new word objective is jointly learned together with the masked LM objective in unified pre-trained model with equal weights.
2.3.2 SENTENCE STRUCTURAL. OBJECTIVE
The next sentence prediction task is considered easy for the original BERT model (the prediction accuracy of BERT can easily achieve 97%-98% in this task (Devlin et al. 2018 We, therefore. extend the sentence prediction task by predicting both the next sentence and the previous sentence, to make the pre-trained language model aware of the sequential order of the sentences in a bidirectional manner.
As illustrated in Figure given pair of sentences (S,, 52) as input, we predict whether S2 the next sentence that follows Si, or the previous sentence that precedes Si, random sentence from a different document. Specifically, for the sentence S1, of the time we choose the text span that follows S1 as the second sentence S2, 1 of the time the previous sentence ahead of S1 is selected, 3
1 of the time a sentence randomly sampled from the other documents is used as Sy. The two sentences are concatenated together into an input sequence with the separator token [SEP] between, as done in BERT. We pool the model output by taking the hidden state corresponding to the first token [CLS], and feed the encoding vector of [CLS] into a softmax classifier make a three-class prediction.
and 3
2.4 PRE-TRAINING SETUP
The training objective function is a linear combination of the word structural objective and the sentence structural objective. For the masked LM objective, we followed the same masking rate and settings as in BERT (Devlin et al. 2018). 5% of trigrams are selected random shuffling.
We used documents from English Wikipedia (2,500M words) and BookCorpus (Zhu et al. (0155) as pre-training data, following the preprocessing and the WordPiece tokenization from (Devlin et al. 2018). The maximum length of input sequence was set to 512.
We ran Adam with learning rate le-4, B1 0.9, 32 = 0.999, L2 weight decay of 0.01, learning rate warm-up over the first 10% of the total steps. and linear decay of the learning rate. We set dropout probability of 0.1 for every layer. The gelu activation (Hendrycks & Gimpel 2016) was used as done in GPT (Radford et al. 2018).
We denote the number of Transformer block layers as L, the size of hidden vectors as H, and the number of self-attention heads as A. Following the practice of BERT, We primarily report experimental results on the two model sizes:
StructBERTBase. 12, H 768, A 12, Number of parameters= 110M
StructBERTLarge I, = 24, H = 1024, A = 16, Number of parameters= 340M
Pre-training of StructBERT was performed V100 GPU cards. For the StructBERTBasee we ran the pre-training procedure for 40 epochs, which took about 38 hours, and the training of StructBERTLarge took about 7 days complete.
distributed computing cluster consisting of 64 Telsa
3 EXPERIMENTS
In this section, we report results of StructBERT variety of downstream tasks including General Language Understanding Evaluation (GLUE benchmark), Standford Natural Language inference (SNLI corpus) and extractive question answering (SQuAD v1.1).
Following BERT's practice, during fine-tuning on downstream tasks, we performed a grid search or exhaustive search (depending the data size) the following sets of parameters and chose the model that performed the best on the dev set. All the other parameters remain the same as those in pre-training:
4

---

Published as a conference paper at ICLR 2020
Batch size: 16, 24, 32: Learning rate: 2e-5, 3e-S, Se-S; Number of epochs: 2, 3; Dropout rate: 0.05, 0.1
System
CoLA
SST-2
MRPC
STS-B
MNLI
QNLI
RTE
WNLI
Avg.
8.5k
67k
3.5k
5.7k
36.3
392k
108k
2.5k
634
Haman Baseline
66.4
97.8
86.3/80.8
92.7/22.6
59,5/80,4
92.0992.8
91.2
93.6
95.9
BERTLagge
89.385,4
87.6/86.5
1/89.
86.7/85.9
70.1
65.1
39.6
BERT STIL.T 2
62.1
94.3
90.2/86.6
88, 7/88.3
719999.4
86.4/85.6
92.7
80.1
65.1
28.3
82.0
SpanBERT
64.3
94.8
90.9/87.9
89. 99.9/89.1
71.9/89.5
88, 1/87.7
94.3
79.0
65.1
45.1
82.8
Snorkel McTaL
63.8
96.2
91.5/88.5
90. 1/89.7
73.1/89.9
87.6/87.2
93.9
80.9
65.1
39.9
83.2
MI-DNN++
65.4
95.6
91.1/88.2
89 61.66/8990 0
72.7/89.6
87. 9/87.4
95.8
85.1
65.1
41.9
83.8
MT-DNN=
92.2/89.55
73.7/89.9
87.9/87.4
42.8
StuctBER
57.2
94.7
89.986.1
88, 5/8.7/87.6
72.0/89.6
85,5784.6
92.66
76.9
65.1
39.0
80.9
StructEERTLaage
65.3
95.2
92.0/89.3
90.3/89 4
74.1/900.5
88.0/87.7
95.7
83.1
65.1
43.6
83,9
StmetBERTLagge
68.6
95.2
92.5/90.1
9/.1/90.6
74.4/90.7
88.2/87.9
65.1
43,9
XI.Net
67.8
96.8
93.0v90.7
91.6/91.1
74.2/90.3
90.2/89.8
98.6
86.3
90.4
47.5
88.4
RoBERTa"
67.8
96.7
92.3/89.8
92.2/919 9
74.3/90.2
90.8/90.2
9%.9
88.2
89.0
48.7
88.5
Ady-RoBERTa*
68.0
96.8
93, 1/908
92.4/92.2
74.8/90.3
91.1/90.7
98.8
88.7
89.0
50.1
88.8
StrccBEETTKBBIKC
69.2
97.1
93.6/91.5
92.8/92.4
74.4/90.7
90.7/90.3
99.2
87.3
89.7
47.8
89.0
Table Results of published models on the GLUE test set, which are scored by the GLUE evaluation server. The number below each task denotes the number of training examples. The state-of-the- art results are bold. All the results are obtained from https:: / /tlp:::////uubeeccmmaa
uebenchma rk com/ Lboardd (StructBERT submitted under different model name ALICE), iidicatess the
ensemble model. Model references: 1: (Devlin et al. 2018); 2: (Phang et al. 2018): )) 3, (Joshi et al.
2019 4: (Ratner et al. 2017): 5: (Liu et 2019a): Yang et al. 2019b); (Liu al. 2019b).
3.1
GENERAL LANGUAGE UNDERSTANDING
3.1.1 GLUE BENCIMARK
The General Language Understanding Evaluation (GLUE) benchmark (Wang et al. 2018 is
collection of nine NLU tasks, covering textual entailment (RTE (Bentivogli et al. 2009 and
MNLI (Williams al., 2017). crrestion-answer entailment (QNLI (Wang al. (0188), paraphrase (MRPC (Dolan & Brockett ). question paraphrase (QQP), textual similarity (STS-B (Cer et al.
2017)). sentiment (SST-2 (Socher et al. 2013)). linguistic acceptability (CoLA), and Winograd Schema (WNLI (Levesque et 2012))
On the GLUE benchmark, given the similarity of MRPC/RTE/STSS to MNLI, we fine-tuned StructBERT on MNLI before training on MRPC/RTE/STS- B data for the respective tasks. This follows the two-stage transfer learning STILTs introduced in (Phang et al. 2018). For all the other tasks (i.e., RTE, QNLI, QQP, SST-2, CoLA and MNLI), we fine-tuned StructBERT for each single task only on its in-domain data.
Table presents the results of published models on the GLUE test set obtained from the official benchmark evaluation server. Our StruetBERTLarg ensemble suppressed all published models (excluding RoBERTa ensemble and XLNet ensemble) the average score, and performed the best among these models in six of the nine tasks. In the most popular MNLI task, our StruetBERTLarg single model improved the best result by 0.3%/0.5% since we fine-tuned MNLI only its in-domain data, this improvement is entirely attributed to our new training objectives. The most significant improvement over BERT observed CoLA (4.8%), which may be due to the strong correlation between the word order task and the grammatical error correction task. In the SST-2 task, our model improved over BERT while performed worse than MT-DNN did, which indicates that sentiment analysis based on single sentences benefits less from the word structural objective and sentence structural objective.
Model
GPT
BERT
MT-DNN
SJRC
StruetBERTT
Dev
90.1
91.4
92.2
Test
89.9
90.8
91.1
91.3
91.7
Table 2: Accuracy (%) on the SNLI dataset.
5

---

Published as a conference paper at ICLR 2020
System
Dev set
Test set
EM
F1
EM
F1
Human
82.3
91.2
XLNe((single+DA) (Yang al. (019b)
88.9
94.5
89.9
95.0
BEXTTeensembe=+DA) (Devlmn et al. 2018)
86.2
92.2
87.4
93.2
(T-NET(single) (Yang et al. 2019a)
85.1
91.7
85.9
92.4
BERT(singic+DA) (Devlin ct al. 2018
84.2
91.1
85.1
91.8
QANettensemble--AJJ (Yu al. 2018)
84.5
90.5
StructBERTLarge (single)
85.2
92.0
StructBERTLarge (ensemble)
87.0
93.0
Table 3: SQuAD results. The StructBERTLarg ensemble is 10x systems which different pre- training checkpoints and ainettunngg seeds.
With pre-training on large corpus, XLNet ensemble and RoBERTa ensemble outperformed all published models including our StructBERTLarg ensemble. To take advantage of the large data which RoBERTa is trained on, we continued pre-training with our two new objectives from the released RoBERTa model, named StructBERTRoERTE. At the time model submission, our StructBERTRoBERT. ensemble, which was submitted under a different name ALICE, achieved the best performance among all published models including RoBERTa and XI.Net on the leaderboard, creating new state-of-te thee-of-hhe-art result of 89.0% on the average GLUE score. It demonstrates that the proposed objectives are able to improve language models in addition to BERT.
3.1.2 SNLI
Natural Language Inference (NLI) is one of the important tasks in natural language understanding. The goal of this task is to test the ability of the model reason the semantic relationship between two sentences. In order to perform well an NLI task, model needs to capture the semantics of sentences, and thus to infer the relationship between a pair of sentences: entailment, contradiction or neutral.
We evaluated our model on the most widely used NLI dataset: The Stanford Natural Language Inference (SNLI) Corpus (Bowman al., 2015), which consists of //366/9,84//9.../ premise- hypothesis pairs in train/dev/test sets and target labels indicating their relations. We performed a grid search on the sets of parameters, and chose the model that performed best the dev
Table shows the results on the SNLI dataset of our model with other published models. StructBERT outperformed all existing systems SNLI, creating
state-of-the-art results 91.7%, which amounts to 0.4% absolute improvement over the previous state-of-the-ae model SJRC and 0.9% absolute improvement
BERT. Since the network architecture of model is identical to that of BERT, this improvement is entirely attributed to the new pre-training objectives, which justifies the effectiveness of the proposed tasks of word prediction and sentence prediction.
3.2 EXTRACTIVE QUESTION ANSWERING
SQuAD v 1 is popular machine reading comprehension dataset consisting of 100,000+ questions created by crowd workers 536 Wikipedia articles (Rajpurkar et al. (016)) The goal of the task is to extract the right answer span from the corresponding paragraph given a question.
We
StructBERT language model on the SQuAD dataset for 3 epochs, and compared the result against the state-of-the-ar methods on the official leaderboard]] as shown in Table We can see that even without any additional data augmentation (DA) techniques, the proposed ERT model was superior to all published models except XLNe++DA on the dev set. (= With data augmentation and large corpus used during pre- pre-training, XLNet+DA outperformed our StructBERT which did
https: / / ra jpurkar iiggttthtn t.ub. i SC//QQUAI--eeS exp orer/
We have submitted the model under the name of ALICE to the SQuAD v1.1 CodaLab for evaluation on the test set. However, due to crash of the Codalab evaluation server, have not got our test result back yet at the time of paper submission. We will update the result it announced.

---

Published as a conference paper at ICLR 2020
Task
CoLA
SST-2
MNLI
SNLT
QQP
SQuAD
(Acc)
(Acc)
(Acc)
(Acc)
(Acc)
(FI)
StructBERT Base
85.8
92.9
85,4
91.5
91.1
90.6
-word structure
81.7
92.7
85.2
91.6
90.7
90.3
-sentence structure
84.9
92.9
84.1
91.1
90.5
89.1
BERTBase
80.9
92.7
84.1
91.3
90.4
88.5
Table 4: Ablation over the pre-training objectives using StructBERRBase architecture. Every result is the average score of 8 runs with different random seeds (the MNLI accuracy is the average score of the matched and mis-matched settings).
data augmentation or large pre-training corpus. It demonstrates the effectiveness of the proposed pre-trained StructBERT modeling the question-paragraan paragraph relationship for extractive question answering. Incorporating the word and sentence structures significantly improves the understanding ability in this fine-grained answer extraction task.
3.3 EFFECT OF DIFFERENT STRUCTURAL OBJECTIVES
We have demonstrated the strong empirieal results of the proposed model on variety of downstream tasks. In the StructBERT pre-training. the two structural prediction tasks are the most important components. Therefore, we conducted an ablation study by removing one structural objective from pre-training at time to examine how the two structural objectives influence the performance various downstream tasks.
Results are presented in Table From the table, were both critical to most of the downstream tasks, except for the word structural objective in the SNLI task. Removing any word sentence objective from pre-training always led to degraded performance in the downstream tasks. The StructBERT model with structural pre- pre-training consistently outperformed the original BERT model, which shows the effectiveness of the proposed structural objectives. (2) For the sentence-pan pair tasks such as MNLI, SNLI, QQP and SQuAD, incorporating the sentence structural objective significantly improved the performance. It demonstrates the effect of inter-sentence structures learned by pre- treiirainngg in understanding the relationship between sentences for downstream tasks. (3) For the single-sentence tasks such as CoLA and SST-2, the word structural objective played the most important role. Especially in the CoLA task, which is related to the grammatical eиrоr correction, the improvement was over 5%. The ability of reconstructing the order of words in pre-training helped the model better judge the acceptability of a single sentence.
see that: (1) the two structural objectives
We also studied the effect of both structural objectives during supe-supe pre-training. Figure 2 illustrates the loss and accuracy of word and sentence prediction the number of pre-training steps for StructBERTBass and BERTBase. From the two sub-figures on top, it is observed that
KouCaii Gata THE
Figure 2: Loss and accuracy of word and sentence prediction over the number of pre-training steps
7

---

Published as a conference paper at ICLR 2020
compared with BERT, the augmented shuffled token prediction in StructBERT's word structural objective had little effect on the loss and accuracy of masked token prediction. On the other hand, the integration of the simpler task of shuffled token prediction (lower loss and higher accuracy) provides StructBERT with the capability of word reordering. In contrast, the new sentence structural objective in StructBERT leads more challenging prediction task than that in BERT, as shown in the two figures at the bottom. This new pre-training objective enables StructBERT to exploit inter-sentence structures, which benefits sentence-pair downstream tasks.
4 RELATED WORK
4.1 CONTEXTUALIZEH LANGUAGE REPRESENTATION
word can have different semantics depending on the its context. Contextualized word representation is considered to be important part of modern NLP research, with various pre-trained language
models (McCann et al.] 2017 Peters et al., 2018 Radford et al.) [0188] Devlin et al., 2018) emerging recently. ELMo (Peters et al. 2018) learns two unidirectional LMs based on long short-term memory networks (LSTMs). A forward LM reads the text from left to right, and a backward LM encodes the text from right to left. Following the similar idea of ELMo, OpenAI GPT (Radford al. 2018) expands the unsupervised language model to much larger scale by training free text corpora. Different from ELMo, builds upon a multi-layer Transformer (Vaswani et al. 2017) decoder, and uses a left-to-righ Transformer to predict text sequence word-by-word
giant collection of
In contrast, BERT (Devlin al.) 2018) (as well as its robustly optimized version RoBERTa (Liu et al.] 2019b)) employs a bidirectional Transformer encoder to fuse both the left and the right context, and introduces two novel pre-training tasks for better language understanding. We base our LM on the architecture of BERT, and further extend it by introducing word and sentence structures into pre- pre-training tasks for deep language understanding.
4.2 WORD & SENTENCE ORDERING
The task of linearization aims to recover the original order of a shuffled sentence (Schmaltz et al. 2016). Part of larger discussion as to whether LSTMs are capturing syntactic phenomena linearization, standardized in a recent line of research as method useful for isolating the performance of text-to- text generation (Zhang& & Clark 2015) models. Recently, Transformers have emerged as powerful architecture for learning the latent structure of language. For example, Bidirectional Transformers (BERT) has reduced the perplexity for language modeling task. We revisit Elman's 's question by applying BERT to the word-ordering task, without any explicit syntactic approaches, and find that pre-trained language models are effective for various downstream tasks with linearization.
Many important downstream tasks such STS and NLI (Wang et al. 2018) are based understand- ing the relationship between two text sentences, which is not directly captured by language modeling. While BERT (Devlin al. 2018) pre-trains binarized next sentence prediction task to understand sentence relationships, we take one step further and treat it as a sentence ordering task. The goal of sentence ordering to arrange set of sentences into a coherent text in clear and consistent manner, which can be viewed as ranking problem (Chen et al. 2016). The task is general and yet challenging, and once is especially important for natural language generation (Reiter & Dale 1997). Text should be organized according to the following properties: rhetorical coherence, topical relevancy, chronological sequence, and cause-effect. In this work, we focus on what arguably the most basic characteristics of a sequence: their order. Most of prior work on sentence ordering was part of the study of downstream tasks, such as multi-document summarization (Bollegala et al.) 2010). We revisit this problem in the context of language modeling new sentence prediction task.
CONCLUSION
In this paper, we propose novel structural pre-training which incorporates word and sentence structures into BERT pre-training. A word structural objective and sentence structural objective are introduced as two new pre-training tasks for deep understanding of natural language in different granularities. Experimental results demonstrate that the new StructBERT model can obtain new
8

---

Published as a conference paper at ICLR 2020
results in variety of downstream tasks, including the popular GLUE benchmark, the SNLI Corpus and the SQuAD v1.1 question answering.
REFERENCES
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. In TAC, 2009.
Danushka Bollegala, Naoaki Okazaki, and Mitsuru Ishizuka. A bottom-up approach to sentence ordering for multi-document summarization. Information processing & management, 46(1): 89-109,2010
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. arXiv preprint arXiv: 05326, 2015.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task I: Semantic textual mammllarttymmllilinggen and cross- lingual focused evaluation. arXiv preprint wrXiv:1708.00053 2017.
Xinchi Chen, Xipeng Qiu, and Xuanjing Huang. Neural sentence ordering. arXiv:1607.06952 2016.
arXiv preprint
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1 14,000 2018.
Ming-Wei
William B Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005.
Jeffrey L Elman. Finding structure in time. Cognitive science, 14(2):1792211, 1990.
Eva Hasler, Felix Stahlberg, Marcus Tomalin, Adri de Gispert, and Bill Byrne. A comparison of neural models for word ordering. arXiv preprint 1rXiv:1708.000 8000, 2017.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv: 01166666661111 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. Spanbert: Improving pre-training by representing and predicting spans. arXiv preprint 1rXiv:1907.10529, 2019.
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning, 2012.
Xiaodong Liu, Pengeheng He, Weizhu Chen, and Jianfeng Gao. Multi-takk deep neural networks for natural language understanding. arXiv preprint 1rXiv:1901.11504 2019a.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint 1907.19997..192, 2019b.
Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems, pp. 6294- 6305, 2017.
Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černocký, and Sanjeev Khudanpur. Recurrent neural network based language model. Eleventh annual conference of the international speech communication association, 2010.

---

Published as a conference paper at ICLR 2020
Matthew E Peters, Mark Neumann, Mohit lyyer. Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint 1r0.v:1802.05000 2018.
Jason Phang, Thibault Févry, and Samuel R Bowman. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01080 2018.
Alec Radford, Karthik Narasimhan. Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre- vre-training. URL https://s3-us-westt amazonaws. com/openai- understanding paper: pdf. 2018.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv: arXiv:1606.055 1606.05250 2016.
Alexander Ratner, Stephen H Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. Snorkel: Rapid training data creation with weak supervision. Proceedings of the VLDB Endowment, 11(3):269-22) 282, 2017.
Ehud Reiter and Robert Dale. Building applied natural language generation systems. Natural Language Engineering, 3(1):57-87, 1997.
Allen Schmaltz, Alexander M Rush, and Stuart M Shieber. Word ordering without syntax. arXiv preprint FrXxv:6604.08633 2016.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631-1642, 2013.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint 1804. 2018
Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv: 170.:05420 2017.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc Le. Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao,, Qin Gao, Klaus Macherey, al. Google's 's neural machine translation sys- tem: Bridging the gap between human and machine translation. arXiv preprint arXiv: 609,088/4, 2016.
An Yang, Quan Wang, Jing Liu, Kai Liu, Yajuan Lyu, Hua Wu, Qiaoqiao She, and Sujian Li. Enhanc- ing pre-trained language representations with rich knowledge for machine reading comprehension. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.
2346-2357, 2019a.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint 17X1v:1906.08237, 2019..
Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoe V Le. Qanet: Combining local convolution with global self-attention for reading comprehension. arXiv preprint arXiv: 2018.
Yue Zhang and Stephen Clark. Discriminative syntax-based word ordering for text generation. Computational linguistics, 41(3):503-538, 2015.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralaa, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pp. 19-27, 2015.
10


Reviews:
[{"review_id": "rJer0FEJqB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a new pre-trained language model based on BERT, called StructBERT. The key contributions are the two new pre-train objectives, (1) word structural objective, where the goal is to reconstruct the right order of intentionally shuffled word tokens, and (2) sentence structural objective, a three-class sentence-pair prediction, either the 2nd sentence precedes the 1st, the 2nd succeeds the 1st, or the 2nd is randomly selected. Unlike the original NSP (next sentence prediction) task, which is simple but tends out to be not so helpful in many downstream tasks, both proposed pre-train objectives seem to be rather useful in benchmarks tested in the paper, including GLUE, SNLI, and SQuAD. \n\nThe paper is well written and understandable for anyone who has a basic background about BERT or pre-train. The experimental results are impressive. Some of my questions / suggestions:\n\n- The two auxiliary tasks are evidently helpful. I wonder what intuition/theory leads to the selection of these two tasks? If the authors have test multiple other tasks that were not as helpful, it is also interesting to know them. \n\n- The wording of the text should be revised to reflect the up-to-date leaderboard results. Personally, I don't think the leaderboard results are that critical, but just want to make sure the writing is accurate at the time of publishing.\n\n- Please also update the results from SQuAD 1.1 CodaLab. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJgNRaZ0tr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to use additional structures within and between sentences for pre-training BERT. The basic idea is to shuffle either some n-grams within sentences or the sentences in texts, then train the model to predict the correct orders. Experiments in this work show that, with this additional training objective, the proposed pre-trained model, StructBERT, obtains good performance on the tasks including natural language understanding and question answering.\n\nOverall, I think the experiments and results in this work are not sufficient enough to support the claim:\n\n- It is necessary to show the performance of BERT only trained with the proposed word and sentence objectives. Otherwise, it is not clear how much benefit the model can get from them and the work is basically incremental. \n- Some justification is needed about why choosing trigrams and why 5% is a good number of sampling trigrams from texts\n\nBesides, there are some recent work on analyzing why BERT encodes any linguistic properties of texts, for example \n\n- Goldberg. Assessing BERT's syntactic abilities. 2019\n- Tenny et al. BERT Rediscovers the Classical NLP Pipeline. ACL 2019\n- Tenny et al. What do you learn from context? ICLR 2019\n\nAll of them show positive results on BERT can capture some syntactic information from text automatically. Which makes me wonder why the simple additional training objective proposed in this work can still lead to performance improvement. Is there an explanation? \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "H1lYNb_e5S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces two new tasks for large scale language model pretraining: trigram word unscrambling and contextual sentence ordering. Using these tasks to pretrain on top of masked language modelling shows improvements when the resulting model is finetuned on downstream tasks. The proposed tasks are simple to implement, and particularly the sentence ordering task is an improvement over the original BERT next sentence task, which is widely regarded as too simple to drive learning good representations. For this reason, I recommend acceptance of this paper.\n\nSome minor quibbles:\n1) Structure in language usually means syntactic structure. How does unscrambling word trigrams help uncover syntactic structure? The references to Elman 1990 also don't serve to clarify anything, I suggest that they are removed.\n2) Some prior work on word ordering (e.g. [1] and older papers cited therein) is missing.\n3) The permutation objective seems very similar to the XLNet objective. Could the authors elaborate more on this in the paper?\n4) Did the authors try with other n-gram shuffling orders?\n5) The sentence ordering task has been used previously (e.g. [2]).\n6) Table 1 overhangs the right margin.\n\nReferences:\n[1] Discriminative Syntax-Based Word Ordering for Text Generation, Zhang and Clark 2015\n[2] Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning, Jernite et al. 2017", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a pair of complementary word- and sentence-level pretraining objectives for BERT-style models, and shows that they are empirically effective, especially when used with an already-pretrained RoBERTa model.

Work of this kind has been extremely impactful in NLP, and so I'm somewhat biased toward acceptance: If this isn't published, it seems likely that other groups will go to the trouble to replicate roughly these experiments. However, I think the paper is borderline. Reviewers were impressed by the results, but not convinced that the ablations and analyses were sufficient to motivate the proposed methods, suggesting that some variants of the proposed methods could likely be substantially better. In addition, I agree strongly with R3 that framing this work around 'language structure' is disingenuous, and actively misleads readers about the contribution to the paper.
Decision: Accept (Poster)