Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper introduces two new tasks for large scale language model pretraining: trigram word unscrambling and contextual sentence ordering. Using these tasks to pretrain on top of masked language modelling shows improvements when the resulting model is finetuned on downstream tasks. The proposed tasks are simple to implement, and particularly the sentence ordering task is an improvement over the original BERT next sentence task, which is widely regarded as too simple to drive learning good representations. For this reason, I recommend acceptance of this paper.

Some minor quibbles:
1) Structure in language usually means syntactic structure. How does unscrambling word trigrams help uncover syntactic structure? The references to Elman 1990 also don't serve to clarify anything, I suggest that they are removed.
2) Some prior work on word ordering (e.g. [1] and older papers cited therein) is missing.
3) The permutation objective seems very similar to the XLNet objective. Could the authors elaborate more on this in the paper?
4) Did the authors try with other n-gram shuffling orders?
5) The sentence ordering task has been used previously (e.g. [2]).
6) Table 1 overhangs the right margin.

References:
[1] Discriminative Syntax-Based Word Ordering for Text Generation, Zhang and Clark 2015
[2] Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning, Jernite et al. 2017
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: This paper proposed a new pre-trained language model based on BERT, called StructBERT. The key contributions are the two new pre-train objectives, (1) word structural objective, where the goal is to reconstruct the right order of intentionally shuffled word tokens, and (2) sentence structural objective, a three-class sentence-pair prediction, either the 2nd sentence precedes the 1st, the 2nd succeeds the 1st, or the 2nd is randomly selected. Unlike the original NSP (next sentence prediction) task, which is simple but tends out to be not so helpful in many downstream tasks, both proposed pre-train objectives seem to be rather useful in benchmarks tested in the paper, including GLUE, SNLI, and SQuAD. 

The paper is well written and understandable for anyone who has a basic background about BERT or pre-train. The experimental results are impressive. Some of my questions / suggestions:

- The two auxiliary tasks are evidently helpful. I wonder what intuition/theory leads to the selection of these two tasks? If the authors have test multiple other tasks that were not as helpful, it is also interesting to know them. 

- The wording of the text should be revised to reflect the up-to-date leaderboard results. Personally, I don't think the leaderboard results are that critical, but just want to make sure the writing is accurate at the time of publishing.

- Please also update the results from SQuAD 1.1 CodaLab. 
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper proposes to use additional structures within and between sentences for pre-training BERT. The basic idea is to shuffle either some n-grams within sentences or the sentences in texts, then train the model to predict the correct orders. Experiments in this work show that, with this additional training objective, the proposed pre-trained model, StructBERT, obtains good performance on the tasks including natural language understanding and question answering.

Overall, I think the experiments and results in this work are not sufficient enough to support the claim:

- It is necessary to show the performance of BERT only trained with the proposed word and sentence objectives. Otherwise, it is not clear how much benefit the model can get from them and the work is basically incremental. 
- Some justification is needed about why choosing trigrams and why 5% is a good number of sampling trigrams from texts

Besides, there are some recent work on analyzing why BERT encodes any linguistic properties of texts, for example 

- Goldberg. Assessing BERT's syntactic abilities. 2019
- Tenny et al. BERT Rediscovers the Classical NLP Pipeline. ACL 2019
- Tenny et al. What do you learn from context? ICLR 2019

All of them show positive results on BERT can capture some syntactic information from text automatically. Which makes me wonder why the simple additional training objective proposed in this work can still lead to performance improvement. Is there an explanation? 

review_title: Official Blind Review #1

Meta-review: This paper proposes a pair of complementary word- and sentence-level pretraining objectives for BERT-style models, and shows that they are empirically effective, especially when used with an already-pretrained RoBERTa model.

Work of this kind has been extremely impactful in NLP, and so I'm somewhat biased toward acceptance: If this isn't published, it seems likely that other groups will go to the trouble to replicate roughly these experiments. However, I think the paper is borderline. Reviewers were impressed by the results, but not convinced that the ablations and analyses were sufficient to motivate the proposed methods, suggesting that some variants of the proposed methods could likely be substantially better. In addition, I agree strongly with R3 that framing this work around 'language structure' is disingenuous, and actively misleads readers about the contribution to the paper.
Decision: Accept (Poster)