Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new way to pre-train language models. The main idea is to include sentence-level hierarchy information during the pre-training. Instead of considering only neighbor sentences, they consider more relations between sentences such as if they are in the same paragraph or if they are in the same document. By training this classification loss along with the original masked token loss, they have better performance on the SuperGLUE downstream tasks.
strength_and_weaknesses: Strength
- The motivation is clear and reasonable.
- The experiments support the claim and seem to be promising

Weaknesses
- To test the generalizability, I suggest to test on GLUE tasks as well.
- It's not clear what size of models is using for the ablation study in Table 2 
- What will happen if we do not shuffle the sentences?
clarity,_quality,_novelty_and_reproducibility: The proposed method is clearly described. The novelty is enough for me. They propose additional self-supervised loss to improve the language models.
summary_of_the_review: The motivation is clear and reasonable. Although the proposed method is simple, it's very effective. Ablation study supports their claim.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper argues for adding inter-sentence tasks to language model pretraining. In this direction, the paper introduces Text Structure Prediction (TSP), an improved version of BERT's Next Sentence Prediction where a LM must classify two sentences using one of six labels that track whether sentences belong to the same document, same paragraph, are neighbors and their order (A then B vs. B then A).
strength_and_weaknesses: ## Strengths

* Well-written and easy to follow.
* The models are properly trained before evaluation. Section 4.1.1 states, "To ensure fair comparisons, all models at the same scale had almost the same computational cost and number of parameters in both training and inference.". This is a detail people often forget, so I am happy to see that the paper takes computation into account.

## Weaknesses

* The introduction argues that "Without text structure, a text becomes a long continuous word sequence, which is hard to read and makes it difficult to identify key concepts and logical relationships." While this is probably, intuitively, true for humans, the paper uses the statement to argue against prior unsupervised sentence tasks and projects intuitive understanding of human attention and reading abilities onto machines (i.e., language models) without valid proof. This is a logical fallacy. I do not believe that this is a major weakness for the paper. The paragraph just needs to be rewritten to avoid the incorrect argument.
* The Related Work / Word-Level Pretraining section is a little too naroow. I understand that the paper's method only applies to transformer encoders, but I believe the paper would benefit from summarizing pretraining methods for encoder-decoders too.
* Section 4.1.1 cites "the sentence splitter of NLTK (Bird et al., 2009) for the use of TSP". NLTK's sentence splitter is Punkt (Kiss and Strunk, 2006), see also https://www.nltk.org/_modules/nltk/tokenize/punkt.html. The paper should cite the Punkt paper together with the NLTK citation. Giving credit to the method's authors is not only fair, but it explicitly names the sentence splitter used, thus improving clarity for readers.

clarity,_quality,_novelty_and_reproducibility: * Well-written and easy to follow.
* The TSP task is not hugely different from prior sentece-based pretraining methods, but it does work.
* See the point in Weaknesses and the Punkt sentece splitter.
* The paper contains all the hyperparameters needed to reproduce the experiments.

summary_of_the_review: A good paper with a couple of minor issues that can be easily addressed before a camera ready.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposed a new inter-sentence training task for language model pertaining, called textual structure prediction. The author designs six structure relationship labels for TSP prediction. The empirical results show that the TSP improves the performance of pretrained language models on NLU tasks.
strength_and_weaknesses: Strength:
1. The paper is clearly stated and easy to follow.

2. The performance improves significantly in some of the SuperGLUE downstream tasks.

Weaknesses:
1. The hierarchical relationship labels are not convincing enough. It is even difficult for humans to predict whether two sentences are from the same document. Enforcing the model to learn the "same document" label might cause the overfitting of the sentence representations. The ablation study also shows that order information is more important than hierarchical structure information.

2. The empirical results are not uniformly improved with the proposed TSP methods on the downstream tasks. 
clarity,_quality,_novelty_and_reproducibility: The clarity is good. The quality and novelty are fair. The experiments seem reproducible.
summary_of_the_review: The paper proposed hierarchical structure labels for training language models. The empirical results show performance improvements, while why the hierarchical labels work needs more reasonable explanations.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes an inter-sentence pretraining task, Text Structure Prediction (TSP), which encourages the encoder to classify sentence structure relationships. The relationships are manually defined to be among 6 types across two dimensions: hierarchical relation and ordering relation. The hierarchical relation contains three cases: neighbor, same paragraph and same document; the ordering relation contains two cases: original and reversed order. By randomly shuffling a subset of the sentences in a sequence, TSP can be used to train the encoder to classify sentence pairs into the pre-defined 6 relation types. The authors argue that such an inter-sentence pretraining task provides more high-level language understanding signals and thus improves using word-level pretraining tasks only. The major benefit of TSP compared to previously proposed inter-sentence pretraining task, as pointed out by the authors, is that TSP is able to offer performance gain for encoders of different sizes (small, base, large), whereas previous methods usually do not benefit large model sizes. The empirical evaluation is conducted on the SuperGLUE benchmark. The authors show that TSP is more effective than other compared sentence-level tasks when combined with MLM.
strength_and_weaknesses: Pros:
* The paper is clearly written and well-organized.
* The proposed pretraining task TSP is shown to outperform the compared sentence-level pretraining objectives (NSP, SOP and SSO) on SuperGLUE.

Cons:
* The novelty of the method is quite limited. The proposed objective TSP is essentially a combined (and slightly extended) version of NSP and SOP -- the "hierarchical relation" classification is extending NSP by further classifying the "non-neighbor" case into "same-paragraph" and "same-document"; the "order relation" classification is equivalent to SOP. In this sense, the proposed task is not new and should be compared with multi-task training that uses both NSP and SOP (this seems to be an important baseline that is missing from comparisons).
* The paper omitted discussions and comparisons with an important family of sentence-level pretraining tasks that use contrastive learning. For example, COCO-LM and CLEAR demonstrated that contrastive learning-based sequence-level tasks can effectively improve NLU task fine-tuning performance, and they can also be combined with word-level pretraining tasks. 
* The empirical evaluation is quite problematic. My major concern comes from the fact that the paper does not follow the conventional pretraining/downstream task settings that have been extensively used by previous studies. (1) The pretraining corpus used in this paper is OpenWebText whereas previous pretraining methods use either Wikipedia + BookCorpus (in BERT) or further expanded corpora that include a wide variety of datasets besides OpenWebText (in RoBERTa). The common pretraining corpora choice could be either of the two above. Even if BookCorpus is not publicly available, Wikipedia should at least be included as a formal language pretraining source. If the pretraining experiments are conducted only on OpenWebText, it's very unclear whether the observed performance gain of TSP can be generalized to common/larger pretraining corpora. (2) Most pretrained language models (e.g., BERT, RoBERTa, ELECTRA) have been evaluated on GLUE whereas this paper only reports SuperGLUE results. This makes it hard to compare with previous pretrained models. While it is good to have SuperGLUE results, I believe it's important to conduct experiments on GLUE as well to facilitate easy comparisons with prior work. (3) The evaluation results only have the best performing results on the test set but not median results across all tasks (Table 6 only has averaged scores). This is problematic because some tasks have very small test sets and the scores could vary a lot. The common practice is to report median results over several random seeds on dev sets of all tasks.

References:  
Meng, Yu et al. “COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining.” NeurIPS (2021).  
Wu, Zhuofeng et al. “CLEAR: Contrastive Learning for Sentence Representation.” ArXiv abs/2012.15466 (2020).

---
**Post-Rebuttal Updates**:  
I thank the authors for providing responses to my raised concerns. Unfortunately, I'm not convinced by the authors' arguments regarding my major concerns. My detailed comments are as below:
* Regarding novelty: I never implied in my review that a method needs to be complicated to be novel; instead, simple methods can be novel as long as they provide new perspectives/insights. My critique regarding the novelty of the paper is mainly due to the fact that the proposed objective TSP is very similar to previously proposed sentence-level objectives (NSP and SOP) in formulation. The major difference between TSP and NSP+SOP, according to the authors' response, is that TSP considers paragraph-level relationships. However, the motivation for this idea is very vague -- how can one accurately/unambiguously define the real difference between a sentence and a paragraph, and the difference between a paragraph and a document? For example, in novels, it's very common that one or two sentences can be a paragraph; other times, a paragraph can also be an entire document. If the distinction between sentences, paragraphs and documents is even hard to tell for humans, then I don't see how the so-called "paragraph-level relationships" newly introduced in TSP can actually make a big difference compared to a simple combination of previous methods (NSP+SOP).
* Regarding the comparisons to contrastive learning approaches: I'm not fully convinced by the argument that the comparisons between contrastive learning objectives and the proposed TSP objective can be omitted. I understand that the two sets of methods are based on different principles, but they are indeed for the same goal -- improving the sequence-level representations of PLMs. Therefore, unless TSP has other usages/benefits that contrastive learning fails to bring, I don't see why they cannot be fairly compared to each other. Such comparisons do not exist in prior work probably because it is quite well known that NSP/SOP-style methods do not really work well (also shown in this paper), and many PLMs simply ended up not using any sequence-level objectives (e.g., RoBERTa, ELECTRA), so the contrastive learning papers didn't compare with NSP/SOP (of course, I do hope to see those results in the contrastive learning papers). 
* The empirical gain of TSP does not seem convincing to me: Although TSP provides significant gains on GLUE/SuperGLUE average score, the gains are mainly from small/unstable tasks, like CoLA, MRPC, RTE and STS-B. Fine-tuning on those tasks are notoriously unstable and is subject to high variance. On large tasks like MNLI and SST-2 (note that they are more reliable than the previous tasks; RoBERTa paper uses these two tasks intead of the average GLUE score for ablation studies), the TSP objective barely provides any gain and even worsens the result. Therefore, I suspect that the performance gain of TSP is simply due to randomness in fine-tuning.
* Regarding the choice of pretraining corpus (OpenWebText): I'm indeed aware that OpenWebText is larger than Wikipedia, but Wikipedia is still a more standard choice for pretraining corpus if only one corpus is used (e.g., base model exploration in BERT, RoBERTa, ELECTRA all used Wikipedia + BookCorpus). This is probably because Wikipedia's text quality is arguably higher than OpenWebText which contains many short/casual/informal texts. Therefore, I'd suggest the authors follow the standard protocol for pretraining; otherwise, the findings and conclusions may not be reliable generalize well. This is not my major concern about the paper though. 

clarity,_quality,_novelty_and_reproducibility: * Clarity: The paper is clearly written and well-organized.
* Quality: The contribution of the paper is not convincing due to the problematic empirical evaluation (see cons above).
* Novelty: The novelty is quite limited. The proposed objective TSP is essentially a combined version of previous sentence-level tasks (see cons above).

summary_of_the_review: I believe the paper will need quite a lot major revisions to be convincing. (1) There should be more discussions and evaluations regarding what are the real differences from NSP and SOP. (2) The method needs to be compared with contrastive-learning based pretraining tasks. (3) The evaluation needs to follow conventional pretraining/fine-tuning settings.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents a new approach to pre-training language models, incorporating sentence-level hierarchy information during the pre-training process. The proposed method, called Text Structure Prediction (TSP), involves classifying sentences based on their relation to one another, such as whether they are in the same paragraph or document. The results of the experiments indicate that the TSP method leads to improved performance compared with other pretraining methods.

Overall, the paper has several strengths, including a clear motivation for the proposed approach and well-supported claims through experiments. However, some weaknesses have also been identified by the reviewers, including a lack of proper empirical evaluation and choice of a different pretraining dataset than those used in prior models, which doesn't allow for direct comparison with previously reported results. Additionally, the paper could benefit from a more comprehensive discussion and comparison with close related work (e.g., contrastive learning papers and other structured pretraining methods like SLM; Lee et al, 2020).
justification_for_why_not_higher_score: The paper doesn't really compare with previously reported results. They execution could be different and they could have picked a standard pretraining corpus like Books+Wiki and then directly compare with prior works. In addition, some of the very closely related work is missing both in discussion and comparison (https://arxiv.org/pdf/2205.01703.pdf). Some of the reviewers remain unconvinced after the author response.
Decision: Reject