Improving Language Model Pretraining with Text Structure Information
Authors: Yi-Siang Wang
Ryohei Sasano
Koichi Takeda
Venue: Submitted to ICLR 2023 (2023)
Keywords: Language Model Pretraining
Representation Learning

Abstract: Inter-sentence pretraining tasks learn from sentence relationships and facilitate high-level language understanding that cannot be directly learned in word-level pretraining tasks. However, we have found experimentally that existing inter-sentence methods for general-purpose language pretraining improve performance only at a relatively small scale but not at larger scales. For an alternative, we propose Text Structure Prediction (TSP), a more sophisticated inter-sentence task that uses text structure to provide more abundant self-supervised learning signals to pretraining models at larger scales. TSP classifies sentence pairs over six designed text structure relationships and it can be seen as an implicit form of learning high-level language understanding by identifying key concepts and relationships in texts. Experiments show that TSP provides improved performance on language understanding tasks for models at various scales. Our approach thus serves as an initial attempt to demonstrate that the exploitation of text structure can facilitate language understanding.

Under review as a conference paper at ICLR 2023
IMPROVING LANGUAGE MODEL PRETRAINING WITH TEXT STRUCTURE INFORMATION
Anonymous authors Paper under double-bling review
ABSTRACT
Inter-sentence pretraining tasks learn from sentence relationships and facilitate high-level language understanding that cannot be directly learned in word-level pretraining tasks. However, we have found experimentally that existing inter- sentence methods for general-purpose language pretraining improve performance only at relatively small scale but not at larger scales. For an alternative, we propose Text Structure Prediction (TSP), a more sophisticated inter-sentence task that uses text structure to provide more abundant self-supervised learning signals to pretraining models at larger scales, TSP classifies sentence pairs over six de- signed text structure relationships and it can be seen implicit form of learning high-level language understanding by identifying key concepts and relationships in texts. Experiments show that TSP provides improved performance language understanding tasks for models at various scales. Our approach thus serves as an initial attempt demonstrate that the exploitation of text structure can facilitate language understanding.
INTRODUCTION
General-purpose pretrained language models have been widely applied in natural language process- ing (NLP). The most representative model of these models BERT (Devlin al. 2019). which is pretrained simultaneously on two pretraining tasks: masked language model (MLM) task, and next sentence prediction (NSP) task. While MLM masks words and requires models to fill clozes, NSP is an inter-sentence task of predicting whether two texts are continuous. Inter-sentence tasks learn relationships between sentences and facilitate high-level language understanding that is not directly learned by word-level pretraining tasks (Devlin et al.) 2019). However, the representative inter-sentence task, NSP, has been found to fail to improve performance (Liu et al., 2019 Yang et al., 2019). Although few successors of NSP have been proposed, they are still not widely adopted and researched. this paper, we show that the existing inter-sentence methods for general-purpor language pretraining actually suboptimal; to improve those methods' weaknesses, we then propose an alternative that redefines what is learned from sentence relations.
The existing pereral-purpose inter-sentence pretraining tasks include NSP (Devlin al. 2019), which discriminates whether two texts come from different documents; sentence order prediction (SOP) (Lan et al. 2020), which discriminates whether two texts are swapped; and sentence structure objective (SSO) Wang et al., 2020), which discriminates whether two texts come from different documents are swapped. To investigate claims of improved performance, we experimented these methods at three different scales (Small, Base, and Large). which mainly followed the setting of BERT Devlin et al.) 2019). The model size and the amount of consumed data increased from Small to Base and then to Large scale (see Appendix A for the details). As seen in Figure our experimental results show that the existing methods improved performance only at the Small-scale whereas they undermined failed to improve performance for models larger scales.
In investigating the reason for little improvement by the existing methods at larger scales, we no- ticed that all of the aforementioned methods split an input text into two segments and learn from only the relationship between the two segments, thus ignoring the text's underlying structure. As illustrated in Figure 2] text structure can be seen the organization of information in texts. Without text structure, a text becomes a long continuous word sequence, which hard to read and makes it difficult identify key concepts and logical relationships for humans. This intuitive understand-

---

Under review as a conference paper at ICLR 2023
79
MLM + NSP
76
SOP
73
MLM SSO
SuperGLUE
MML TSP
67
64
58
Small
Base
Large
Figure 1: Experimental results on the language understanding benchmark SuperGI.UE (Wang al,, 2019) at the Small, Base, and Large scales. The experiment compared the effect on the performance of using different inter-sentence tasks (NSP, SOP, SSO, and our proposed task, TSP) when learned concurrently with the word-based MLM pretraining task. By taking advantage of text structure information, TSP outperforms pure MLM at different scales, whereas the other inter-sentence base- lines (NSP, SOP, and SSO) failed to improve the performance at larger scales in our experiments.
Cuantum Day was apparently yeslarday, and Google the occasion with the launch of The Qubit Game, spotted by 9to5Google. Created partnership Doublespaak games, "playful journey buliding quantum computer, one qubit at time," Google said. It also hopes the game, World Quantum Day, help generate
Paragraph (3 sentences]:
Brief introduction of The Qubit Game
Further describe
quantum computer. It's prefty straighfforward (you won't need to learn any quantum physics) the goal increasing Qubils while keeping them cool. The more Qubits you have, the more difficult gets. Eventually, you'll
game resolves Cubits,
builing
entanglement
Paragraph II sentences) The details of the game.
new upgrades, complete
projects and hopelully become a little more curious about how we're building quantum computers," wrote Google Quantum head education Astawi
Explain the reason
The goal drew attention quantum computing, because seems there's dearth of people working the field. To that end, Google bringing game to diassroom, hoping encourage educators to talk about subject and expand access quantum computing research.
Paragraph III (2 sentences)
Google's intention behind the game.
Conclude
students pursuing computers, and understanding what would be be a quantum scientist engineer," wrote Asfaw. "For that's what World Quantum Day about: showing everyone what quantum computing maily is and how they can get involved."
building using quentum
Paragraph IV sentences)
The meaning World Quantum Day.
Figure 2: Example text structure real text. The document comprises multiple paragraphs, where each paragraph is composed of multiple sentences. Each paragraph conveys specific concept constructed by the sentences that the paragraph composed of, while the order of concepts comes with clear intention and forms logical flow. This suggests that the combination of hierarchy and order in text structure hides valuable clues to high-level language comprehension.
ing of human reading ability inspires us to explore the possibility that text structure can provide models' language understanding ability abundant learning signals high-level semantics and their interaction, especially for models at larger scales. Hence, goal in this paper is to demonstrate that learning from text structure can improve seneral-purposs language model pretraining. We thus propose inter-sentence task that better exploits text structure to examine our hypothesis.
The proposed task, Text Structure Prediction (TSP), is our initial attempt to integrate text struc- ture information into general- purpose language pretraining. Regarding the task design, view text structure in terms of two axes: ordering and hierarchy. Hierarchies are nested groupings of sentences, such paragraphs sections, where each group conveys specific concept and its
2

---

Under review as a conference paper at ICLR 2023
perordinate group conveys more global concept. The order of sentences represents the flow of context and connections between information such as chronological process or cause-effect rela- tionship (Danes, 2015). Thus, the solution of the TSP task requires two abilities: understanding of the semantics of sentences to group them into nested concepts (reflected by hierarchical relation- ships), and identification of connective relations between them (reflected by ordering relationships). Specifically, we propose six inter-sentence relations for learning from text structure information, which combine three hierarchical levels and two ordering directions. As result, the TSP task re- quires a model to classify sentence pairs over the six defined text structure relationships. More details of TSP are given in Section 3 Through experiments we have shown the following: (1) Our proposed method continues to improve performance on language understanding benchmark at the Small, Base, and Large scales, whereas the inter-sentence baselines fail to do so. (2) Our proposed method is comparable to or better than the baselines on almost all language understanding tasks in the benchmarks. Accordingly, the results have shown the effectiveness and potential of exploiting text structure information for general- purpose language pretraining.
In summary, the contributions of this paper are as follows
1. We find that existing inter-sentence tasks for general-purpose language pretraining did not bring significant improvement in our settings. In our experiments, these approaches per- form well at a relatively small scale, but fail to improve or even degrade performance at larger scales.
2. We propose the use of text structure information to provide more valuable learning signals for
language pretraining. For our initial attempt at this, we designed the Text Strueture Prediction (TSP) task and show that TSP improves performance at all scales in our experiments. We thus demonstrate that the exploitation of text structure information is promising research direction for facilitating general-purposs language understanding.
2 RELATED WORK
2.1 WORD-LEVEL PRETRAINING
Self-supervised word representation learning has rapidly developed in the past several years. GPT (Radford et al. 2018) predicts the next token with the information from the previous tokens. ELMo (Peters al.j (2018) uses a bidirectional framework produce contextualized word representations by fusing the final hidden states of two LSTTM networks processing information in different direc- tions. BERT (Devlin al., 2019) achieves deep bidirectional pretraining via the MI.M task, which masks certain words in the input and then predicts the masked words. Since BERT's success, trans- fer learning has become mainstream in NLP. Many research initialize their models with pretrained models to reuse learned representations for their target tasks. Also, the development of word-level pretraining tasks has become prosperous. For example, ELECTRA (Clark et al.. 2020) detects re- placed tokens that come from incorrect predictions by another MLM model, while PMI-Masking (Levine et al. 2021] samples and masks collocations instead of words. In addition to encoder-based BERT and decoder-based GPT, word-level pretraining methods that based encoder-decoder such as T5 (Raffel et al.) 2020) and BART (Lewis et al. 2020) are also developed, which gain great success on text generation tasks. Our proposed approach is learned simultaneously with a word-level pretraining task, and it can be easily combined with any word-level pretraining task that is based on transformer encoder, with negligible additional computational and memory costs.
2.2 SENTRNCE-LEVEH PRETRAINING
Approaches for learning sentence representations mainly fall into three categories. The first type is sentence bottleneck, including aims to generate single or multiple sentence representations that are useful for word prediction ob-
(Kiros al., 2015) and CMLM (Yang et al. 2021) )
jectives. The sentence bottleneck methods compress the encoded sentence into bottleneck repre- sentations and thus benefiting from information bottleneck (Tishby et al. 1999). The second type is contrastive learning, including COCO-LM (Meng et al. (002)) and CLEAR (Wu et al. 2020), which makes representations of positive sentence pairs closer and vice versa. Contrastive learning methods improve uniformity and case the anisotropy problem that limits the expressiveness of em- beddings (Gao et al. 2021). The third type is inter-sentence learning. including NSP and SOP
3

---

Under review as a conference paper at ICLR 2023
described in Section I which learns sentence representations by capturing relationships between sentences. Since inter-sentence methods leam from interactions of sentences, they inherently also encode relationships between sentences and facilitate contextualization of multiple sentences. There currently no research comparing the three kinds of methods with each other, and each type ben- efits from different principles. However, the inter-sentence approach is the only one that explicitly encodes and contextualizes multiple sentences. Our proposed approach belongs to the inter-sentence category.
2.3 ETTER-SENEENCE TASKS
In terms the kinds of sentence relationships to be learned, most inter-sentence tasks are sentence ordering tasks that rearrange set of sentences into their original order (Cui et al. 2020 Kumar et al. 2020), whereas our proposed sentence relationships (Section 3.2) include ordering relationships but not restricted them. Regarding the creation of sentence representations, instead of encoding each sentence individually (Prabhumoye et al.) 2020), our proposed framework (Section 3.1 adopts an approach of concatenating sentences into input sequence, whereby information flows across sentences; thus, the created sentence representations are contextualized. In terms of semantic units, approaches such as NSP and SOP capture relationships between segments that are two halves of text sequence that may include many sentences. In contrast, our approach processes sentences as in many other approaches. Finally, for task objectives, some approaches aim to identify coherent text (Barzilay & Lapata 2008); Mesgar & Strube 2018) or learning discourse representations (Iter jet al., (2020), whereas our baselines (NSP, SOP, SSO) and our proposed method are for creating general-purpose language models.
PROPOSED METHOD
We explain our proposed method in two parts. Section 3.1 introduces the overall proposed frame- work for creating sentence embeddings and then performing six-class classification over sentence pairs using created sentence embeddings. Figure shows overview of this framework. Sec- tion 3.2 further elaborates on the six inter-sentence relationships that we designed for classification and learning over sentence pairs. Overall, the implementation of TSP comes with several advan- tages: (1) It is a self-supervised task that does not require external resources or human labeling. (2) can easily be combined with any pretraining task that outputs contextual word representations. example is provided in Appendix F] (3) It can be learned concurrently with other tasks in one for- ward pass. (4) only requires constructing an output head on top of a text encoder, with negligible extra parameters and computations.
3.1 PROPOSED FRAMEWORK
3.1.1 INPUT SEQURNCE
Given a text sequence, we mark the scope of each sentence in the sequence by using the NLTK toolkit (Bird al., 2009), which is based on Punkt (Kiss & Strunk 2006). The result is an sequence that can be described as x =< #1,1, (1,2 T1. .... 1k Tk,2, Te >, where T,, is the 1-th token in sentence n; the length of sentence i, and l is the number of sentences z. Next, the sequence's 's sentences are shuffled to enable the model to learn to identify their correct order. We have found that shuffling all of sequence's 's sentences introduces noise into the model; thus, we shuffle only limited percentage of the sentences (15% in this work, as explained in Appendix O while keeping the other sentences at their original positions. Additionally, to learn simultaneously from the MLM task, we follow the same procedure used in BERT to corrupt the sequence: 15% tokens are selected; then, 80% of them are replaced with a [MASK] token, 10% are replaced with random tokens, and 10% are unchanged. Finally, we add sentinel tokens [CLS] and [SEP] to the sequence as in BERT, and masked and partially shuffled input sequence is thus obtained for the model.
4

---

Under review as a conference paper at ICLR 2023
Sentence Embeddings
3
$2
S
Average
Average
Average
TSP
Token Hidden States
Transformer Encoder
6-Class Classification
Partially Shuffled Input Sequence
Sentence 1
Sentence 3
Sentence 2
S
S₂
S
Input Sequence
Sentence 1
Sentence 2
Sentence 3
Figure 3: Overview of the proposed Text Structure Prediction (TSP) task. (left) Before an input sequence is sent to the encoder, certain randomly selected sentences in the sequence are shuffled. After encoding, a sentence embedding obtained by averaging the encoded states the tokens belonging to that sentence. (right) The TSP loss is calculated for performing six-class classification over sentence pairs by their sentence embeddings, where the six classes are shown Figure
3.1.2 SENTENCE EMBEDDING
To obtain sentence representations, first calculate contextualized token hidden states:
h
fo(a'),
where fo is the transformer encoder function and x' is a corrupted and partially shuffled input token sequence. By averaging the hidden state vectors of tokens that belong to the sentence, we obtain its sentence embedding:
8; h4.
(2)
where ha is the hidden state vector of the t-th token in the i-th sentence and is the length of the i-th sentence. Additionally, through the transformer encoder's attention mechanism, each token interacts with all tokens in the input sequence rather than only the tokens in the sentence it belongs to. As result, the sentence embeddings are contextualized and aware of other sentences in the same sequence. This means that when compare sentences for their structure relationships. not only the sentences themselves but also their context are considered. Since identifying structure relationships using only information from the compared sentences be very hard, it is intuitive that models are motivated to encode high-level interactions between sentences the context into sentence representations.
3.1.3 PAIRWISE CLASSIFICATION
Next, we perform six-class classification over sentence pairs to learn the text structure relationship between sentences. Given a pair of sentences, we pass their sentence embeddings to a classifier to obtain prediction logits ži,j "YL(GLUU(W1 $ 8,))). @ denotes vector concatenation, the embedding of the k-th sentence, and GELU the activation function proposed by Hendryycs & Gimpel (2016). Finally, the loss in the Text Structure Prediction (TSP) task is the cross-entropy loss:
is
exp(ZiJ....)
Cisplin -log 56 E exp((
where Vi, is the correct label for the relationship between the i-th and j-th sentences, and 2i,.ee is the e-th element of the logit vector for the sentence pair. The final TSP loss Ltsp the average over the TSP losses for all sentence pairs in mini-batch, where the sentences in each sentence pair are in the same sequence. Overall, a model is optimized by minimizing the sum of MLM loss Lmtn and our TSP loss Ltspi
L Lrsp Luntm-
5

---

Under review as a conference paper at ICLR 2023
Next Sentence Prediction (NSP)
Text Structure Prediction (TSP)
Different Document
Neighbor
Same Document
Same Document
Reverse Order
Normal Order
Sentence Order Prediction (SOP)
Hierarchy
Neighbor
Neighbor
Same Paragraph
Same Paragraph Normal Order
Reverse Order
Normal Order
Reverse Order
Sentence Structure Objective (SSO)
Neighbor
Neighbor Normal Order
Different Document
Reverse Order
Neighbor
Neighbor
Reverse Order
Normal Order
Order
Figure 4: Inter-sentence relationship sets for (left) our baselines and (right) TSP. An inter-sentence task classifies pairs of texts into one of the relations in its own defined sets. For example, if two sentences are in the same paragraph (but not adjacent) and reverse order, the TSP task should classify the pair Same Paragraph & Reverse Order. Note that we leave Different Document for future work because of its lack of difficulty (Lan et al. 2020 and complexity of implementation. By comparing the defined relations, can that TSP takes better advantage of the text structure and is a more complex task that is expected to provide more valuable knowledge.
3.2 PROPOSED TEXT STRUCTURE RELATIONSHIPS
The key part of the training objective is design of six classes represent six text structure relationships between sentences. As illustrated on the right in Figure these relationships are the set of combination of three hierarchical relationships {neighbor, same paragraph, same document) and two ordering relationships (normal order, reverse order). Concretely, these relationships are defindd as follows. The hierarchical relationships include (1) neighbor (two sentences are adjacent), (2) same paragraph (two sentences are in the same paragraph but not adjacent), and (3) same document (two sentences are the same document but not adjacent or in the same paragraph). The ordering relationships include (1) normal order (two sentences are in the forward order), and (2) reversed order (two sentences are in the reversed order).
3.3 LEARNING EXPECTATION FROM HIERARCHY AND ORDER
The text hierarchy describes the semantics hierarchy and the interactions between the semantics. As shown in Figure the semantics of sentences are included in higher-level paragraph semantics, and paragraph semantics are included in higher-level document semantics. Relationships between semantics at different levels and the same level can also be seen in the example. To group sen- tences nestedly, models are expected to summarize semantics at different levels from the context, while capturing relationships and interactions between these semanties. Intuitively, it can provide unique learning signals for models' understanding ability on and between texts at different levels of granularity.
The order of sentences depends on the logical relations that connect the different concepts described in the sentences. These relations can be chronological relations, cause-effectt relations, and so Textual coherence is also an important clue for sentence order. A model must recognize relations between concepts and identify coherence rearrange the order of sentences. Moreover, learning from sentence order has long been shown to be useful for many downstream applications (Cui et al. 2020

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENTS
4.1 SETUP
4.1.1 PRETRAINING
All models in this paper were pretrained on Open WebText (Gokaslan & Cohen 2019p, an open- source recreation of the WebText corpus described in GPT2 (Radford et al. 2019). It includes 38GB of texts from 8,013,769 documents extracted from Reddit posts. Note that we did not use Wikipedia=BookkCnne
because of the public unavailability of official BookCorpus (Zhu et al., 2013). In preprocessing the pretraining data, we delimited paragraphs by blank lines and marked the scopes of sentences with the sentence splitter of NLTK (Bird et al. 2009) for the use of TSP.
For this paper, we experimented with pretraining models on word-based MLM with different inter- sentence tasks on MLM only. In particular, referred to the setting of ELECTRA (Clark et al.) 2020). another popular pre-trained model. We trained with sequence length of 512 and I million training steps at the Base and Large scales, or with a length of 128 and 1.45 million steps at the Small scale. The batch sizes vary with the scale, as described along with other details in Appendix A To ensure fair comparisons, all models the same scale had almost the same computational cost and number of parameters in both training and inference. Note that we did not perform any hyperparameter search, although we think it could help our proposed method to perform better.
mainly followed BERT's pretraining setting and
4.1.2 EVALUATION
Our pretrained models were evaluated on the widely adopted SuperGLUE benchmark (Wang et al, 2019) for evaluating general language understanding systems. SuperGL.UE contains tasks cover- ing natural language inference tasks RTE (Dagan et af.) 2006) and CB, multiple-choice reasoning task COPA (Melissa al.) 2011). gusstion-answering tasks BoolQ (Clark et al., 2019). MultiRC Khashabi et al.] 2018), ReCoRD (Zhang et al. 2018). and word sense disambiguation task WiC Pilehvar Camacho- Camacho-Collados) 2019). The metrics used here for these tasks are the same as those the ones described in the SuperGLUE paper.
Because some of the evaluation datasets are small and the scores on them could vary substantially depending on the random seed, we followed the ELECTRA setting and finetuned 10 runs for each task from the same pretrained checkpoint for every model. Here, we took the median score of 10 runs for our development set results (Appendix (D), and we used the best-performing model on the development set for evaluation on the test set. In contrast to certain related papers, we did not apply tricks such as intermediate finetuning ensembling during the stage of finetuning. Similarly, we excluded the WSC task because it involves intermediate finetuning and publicly unavailable training data to obtain decent results (Kocijan et al. 2019). More details about our finetuning are given in Appendix B
4.2 RESULTS
Table [ lists the averaged SuperGLUE scores. TSP improved on pure MLM and outperformed the inter-sentence baselines at all scales. These results show the effectiveness and potential of incor- porating text structure information into learning. The results also reveal two findings. First, NSP undermined or made little difference in the performance, which matches the claims of Liu et al. (019 Yang et al. (2019). Second, while SOP has been claimed improve NSP on the GLUE benchmark (Wang et al. 2018 Lan et al. 2020), we found that it failed to outperform NSP in our experiment.
Next, we analyze the scaling of the four inter-sentence tasks. Although the inter-sentence baselines (NSP, SOP, and SSO) improved on pure MLM at the Small scale, at the larger scales they failed to improve pure MLM by significant margin even undermined the performance. We suspect the reason to be the task difficulty. While the Small-scale models learned from the inter-sentence baseline tasks, those tasks may not have been difficult enough provide valuable learning signals to models at larger scales. We can also see that SSO (the fusion of NSP and SOP) is slightly more complicated than NSP and SOP, and it slightly outperformed both NSP and SOP at Base
7

---

Under review as a conference paper at ICLR 2023
Table I: Experiment results on the SuperGLUE test set at different scales. Overall, TSP outper- formed MLM at the different scales, whereas the other inter-sentence baselines (NSP, SOP, and SSO) failed to achieve improvement. Note that because the sizes and settings for the naming of scales (e.g., Small, Base, and Large) differ greatly in different papers, results may not be directly comparable between papers.
CB
COPA
MultRC
RTE
WIC
BoolQ
ReCoRD
Scale
Model
Avg.
Avg.
MLM
58.8
75.7/81.6
57.6
60.3/13.6
58.3
60.9
70.4
49,4/48.6
MLM+NSP (BERT)
61.1
76.2/82.0
58.4
61.1/14.7
65.5
65.5
71.5
50. 1/493
Small
MLM+SOP
77.7/80.4
59.0
61.7/14.5
71.7
48.3/47.4
MLM+SSO
60.8
74.4/81.6
57.2
61.1/14.3
66.1
66.1
71.8
49.3/48.6
MLM+TSP (ours)
61.5
83.0/86.0
59.4
62.4/14.3
66.
60.2
71.8
50.5/49.8
MLM
71.0
84.8/90.4
63.2
73.2/28.6
71.3
66.1
79.4
79.1/78.2
MLM+NSP (BERT)
70.8
84.5/90.8
66.0
73.8/30.6
73,2
68.8
70.9
77. 77.9/76,9
Base
MLM+SOP
71.0
83.5/90.4
70.0
71.7/25.4
70.8
68.1
78.3
74.9/74.0
MLM+S50
71.6
84.8/89.6
71.2
73.0/28.5
72.3
68.0
76.9
75.0/74.0
MLM+TSP (nurs)
73.6
84.2/91.2
72.4
75.8/33.5
74.1
66.6
80.6
79.2/78.5
MLMM
76.5
87.4/92.4
73.6
77.8/7.5
76.6
68.7
82.7
86.5/85.7
MLM+NSP
76.5
83.6/89.6
76.0
78.2/38.4
77.4
71.0
81.7
85. 1/84/4
Large
MLM+SOP
76.4
84.8/90.4
74.6
79.3/40.9
77.1
68.6
82.2
85.2/84.5
MLM+SS0
81.8/88.4
79.6/40.9
82.1
85.1/84.3
MLM+TSP (ours)
79.0
93 1/94/9
77.4
80.5/43.2
79.1
72.4
82.7
85.8/85.2
and Large scales. On the other hand, with the better use of text structure information, TSP was designed more complicated (six inter-sentence relations instead of two three inter-sentence relations for the inter-sentence baseline tasks, as shown in Figure(4) and more difficult. TSP provides valuable knowledge for models all scales, especially for Large-scale models, which have the largest capability and consumed data to better exploit the difficult TSP task, thus yielding the largest improvement over other baselines as compared to the other scales. Encouraged by this observation. we expect that models with even larger model sizes and training data would also benefit significantly from learning text structure information.
While TSP provided competitive or better performance on most of the tasks, we also found that performed well on the reasoning (CB, RTE, and COPA) and crestion-annweringg (MultiRC and BoolQ) tasks. We can view a question and an answer as two texts that share similar concept and can be placed in the same hierarchical group, such as a paragraph or a pair of consecutive sentences. Likewise, we can view reasoning in terms of whether two texts are coherent. From these view- points, the results are consistent with our interpretation that TSP facilitates language understanding by strengthening recognition and comparison of concepts and reasoning relationships (Section 3.2) On the other hand, although ReCoRD is QA task, it is more like prediction of masked entity in paragraph given the context, which may explain why pure MLL the leading model for ReCoRD at the Large scale. Regarding WIC, a word sense disambiguation task, TSP showed an interesting phenomenon. Compared
other baselines in the WiC task, TSP performed poorly the Small scale, got better at the Base scale, and became the best-performing model at the Large scale. We can relate this result to our surmise on the scaling effect mentioned above. Learning of the complicated TSP stretches models' capabilities, and small models may not have enough capability to encode in- formation that useful for word-level tasks like WiC, whereas large models have enough capability that complicated sentence-level learning can even help with word-level understanding. While more elaborate approaches will be needed to analyze the effect of learning text structure information on different types of downstream tasks, which is outside the scope of this paper, hope to have good starting point by providing interesting results and explanations for those who follow.
In addition to the main results, we also reported other experiment results. First, we reported scores on the SuperGLUE development set in Appendix D which showed similar results as Table [ Second, reported results the GLUE benchmark in Appendix El Finally, demonstrate that our proposed approach generally useful, we did an experiment showing that TSP can be easily combined with another word-based pretraining task (see Appendix [ for details). Additionally, this
8

---

Under review as a conference paper at ICLR 2023
Table Ablation results for the designed text structure relations.
#ordering relationship
#hierarchical relationship
Model
#total classes
SuperGLUE
MLM+TSP
2
3
73.9
MLM+ paragraph
4
2
2
73.2
MLM+TSP hierarchy
72.5
MLM+TSP- order
3
71.9
MLM
71.2
experiment also shows that TSP still improves performance in its preliminary results, which suggests that learning text structure information may generalize to pretraining tasks other than MLM.
5 ABLATION STUDY
To investigate whether the proposed TSP text structure relations are helpful, we experimented with different TSP variations in which some text structure relations were ablated. Specifically, MLM+TSP -hierarchy adopted only the ordering relationships (normal, reversed), MLM+TSP order adopted only the 3 hierarchical relationships (neighbor, paragraph, document), and MLM+ TSP -paragraph combined the 2 ordering relationships with only the 2 hierarchical relationships (neigh- bor, document), with discrimination of whether sentences were in the
paragraph.
The results on the SuperGLUE development set of models at the Base scale are shown in Table First, we found that the models pretrained with TSP benefitted from both the hierarchy relationships and the ordering relationships, as both MLM+TSP hierarchy and MLM+TSP order underper- formed MLM+TSP and outperformed MLM. Second, the performance slightly dropped when was not discriminated whether two sentences were in the same paragraph, which shows that the adja- cency of sentences is not the only important factor, and that discrimination of hierarchical groups at different levels matters. Although other design elements such as discrimination of sections and documents remain unexplored, we will encourage future work to explore further, whereas in this paper we focus uncovering the potential for integrating text structure information.
6 CONCLUSION
Despite their stated usefulness for high-level language understanding and continued development, inter-sentence tasks have not gained enough attention as expected. Our experiments have shown that existing inter-sentence approaches for general-purpose language pretraining can not improve performance or even degrade performance scales larger than Small. Furthermore, these existing approaches ignore large part of text structure information and oversimplify complicated text struc- tures into one relationship between two lengthy parts text sequence. The goal of this paper was thus to explore the potential of utilizing text structure information provide more valuable learning signals and maintain the improvement at larger scales. For an initial attempt to achieve this goal, proposed Text Structure Prediction (TSP), self-supervised inter-sentence pretraining task that can easily be combined with any encoder-based pretraining task. TSP redefines what is learned from sentence pairs via text structure relationships. Specifically, these text structure relationships combine hierarchical and ordering relationships, which can be seen as implicit learning signals for language understanding via recognition of concepts and reasoning relationships respectively. this paper, we have shown that (1) inter-sentence baselines (NSP, SOP, SSO) were not helpful for perfor- mance at larger scales in our settings, whereas TSP improved the overall performance of models at all scales in our experiments, which again demonstrates that inter-sentence tasks for learning from inter-sentence relationships can help language understanding. Moreover, (2) the good scaling and significant improvement the Large scale showed that the exploitation of text structure information can result in a sophisticated task that provides valuable knowledge for models with a large capability and data size. Finally, although there may still be much room for different ways to exploit text struc- ture information, we have shown the effectiveness and potential of exploiting such information, and expect that this work will encourage more research on learning from text structure information.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Regina Barzilay and Mirella Lapata. Modeling local coherence: An entity-based approach. Com- putational Linguistics, 34(1):1-34, 2008.
Steven Bird, Ewan Klein, and Edward Loper. Natural language processing with Python: analyzing text with the natural language toolkit. O'Reilly Media, Inc.". 2009.
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 2936.
Association for Computational Linguistics, June 2019.
Kevin Clark, Minh-Thang Luong, Quoe V. Le, and Christopher D. Manning. Electra: Pre-training text encoders as discriminators rather than generators. International Conference on Learning Representations, 2020.
Baiyun Cui, Yingming Li, and Zhongfei Zhang. BERT-enhanced relational sentence ordering net- work. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro- cessing (EMNLP), pp. 1310-6320 Association for Computational Linguisties, November 2020.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment chal- lenge. In Joaquin Quiñioneo- Candella, Ido Dagan, Bernardo Magnini, and Florence d' Alché Buc (eds.), Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classifi- cation, and Recognising Tectual Entailment, pp. 177-190. Springer Berlin Heidelberg, 2006.
Frantisek Danes. FUNCTIONAL PERSEETTENEEEEEPPEEEEETTII PERSPECTIVE THE ORGANIZATION OF THE TEXT. De Gruyter Mouton, 2015.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-4186. Association for Computational Linguisties, June 2019.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1894-6910 Association for Computational Linguistics, November 2021.
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http::/skky1ion007..grk io/
OpenWeeTeextCorpss 2019.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv: Learning. 2016.
Dan Iter, Kelvin Guu, Larry Lansing, and Dan Jurafsky. Pretraining with contrastive sentence ob- jectives improves discourse performance of language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4859 4870. Association for Com- putational Linguistics, July 2020.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Look- ing beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies. Volume 1 (Long Papers), pp. 252-262. Association for Computational Linguistics, June 2018
Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Asso- ciates, Inc., 2015.
Tibor Kiss and Jan Strunk. Unsupervised multilingual sentence boundary detection. Computational Linguistics, 32(4):485-525, 2006.
10

---

Under review as a conference paper at ICLR 2023
Vid Kocijan, Ana-Maria Cretu, Oana-Maria Camburu, Yordan Yordanov, and Thomas Lukasiewicz. A surprisingly robust trick for winograd schema challenge. CoRR, abbs/1905.0622 09050, 2019.
Pawan Kumar, Dhanajit Brahma, Harish Karnick, and Piyush Rai. Deep attentive ranking networks for learning to order sentences. Proceedings of the AAAI Conference on Artificial Intelligence, 34 (05):81 15 11--122, April 2020.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations, 2020.
Yoav Levine, Barak Lenz, Opher Lieber, Omri Abend, Kevin Leyton-Brown Moshe Tennenholtz, and Yoav Shoham. [MMl-masking Principled masking of correlated spans. In International Conference on Learning Representations, 2021.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Compulational Linguistics, pp. 7871-7880 Associa- tion for Computational Linguistics, July 2020.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692 16922, 2019.
Roemmele Melissa, Bejan Cosmin, Adrian, and Gordon Andrew, S. Choice of plausible alterna- tives: An evaluation of commonsense causal reasoning. In AAA/ Spring Symposium on Logical Formalizations of Commonsense Reasoning, pp. 21 Stanford University, March 2011.
Yu Meng, Chenyan Xiong, Payal Bajaj, saurabh tiwary, Paul Bennett, Jiawei Han, and XIA SONG. Coco-Im: Correcting and contrasting text sequences for language model pretraining. In M. Ran- zato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.). Advances in Neural Information Processing Systems, volume 34, pp. 23102-233 14. Curran Associates, Inc., 2021.
Mohsen Mesgar and Michael Strube. A neural local coherence model for text quality assessment. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4328 4339. Association for Computational Linguistics, October-November 2018. 8.
Matthew F. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Con- ference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pp. Linguistics, June 2018.
Association for Computational
Mohammad Taher Pilchvar and Jose Camacho-Collade WiC: the word-in-context dataset for eval- uating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pp. 1267-1273. Association for Computational Linguistics, June 2019.
Shrimai Prabhumoye, Ruslan Salakhutdinov, and Alan W Black. Topological sort for sentence order- ing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2783-2792. Association for Computational Linguistics, July 2020.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language under- standing by generative pre-training. 2018
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140): 2020.
11

---

Under review as a conference paper at ICLR 2023
Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method. In Proc. of the 37-th Anmual Allerton Conference on Communication, Control and Computing, pp- 368-377, 1999.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman, GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for Association for Computational Linguistics, November 2018.
NLP, pp. 553-355.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpor language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Wei Wang, Bin Bi, Ming Yan, Chen Wu, Jiangnan Xia, Zuyi Bao, Liwei Peng, and Luo Si. Struct- bert: Incorporating language structures into pre-training for deep language understanding, In International Conference Learning Representations, 2020.
Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. Clear: Contrastive learning for sentence representation, 2020.
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime Carbonell, Russ R Salakhutdinov, and Quoe V Le. Xlnet: Generalized autoregressive pretraining for language understanding. H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, Fox, and R. Garnett (eds.), Advances in Neu- ral Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Ziyi Yang, Yinfei Yang, Daniel Cer, Jax Law, and Eric Darve. Universal sentence representation learning with conditional masked language model. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6216-6228. Association for Computa- tional Linguistics, November 2021.
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. Record: Bridging the gap between human and machine commonsense reading comprehension, 2018.
Y. Zhu, R. Kiros, R. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, and S. Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 19-27. IEEE Computer Society, dec 2015.
12

---

Under review as a conference paper at ICLR 2023
A PRETRAINING DETAILS
The details given in Table 3 apply to all the models in this paper. In most cases, used the same hyperparameters as for BERT and ELECTRA. At the Large scale, we adopted a setting closer to BERT-Large because ELECTRA's setting for the Large scale requires much higher memory and computational cost than our computational resource could handle. For the Large scale, we also slightly increased the batch size 288, which we found that avoids divergence our experiments. Additionally, inspired by BERT, we trained with a sequence length of 128 for the first half of the training steps and then with sequence length of 512 for the rest while keeping the same number of tokens in batch to speed up pretraining in our Large-scale experiments. Note that we did not perform any hyperparameter search, although we think it could help our proposed method to perform better.
Table 3: Pretraining hyperparamettrr..
Hyperparameter
Small
Base
Large
Number of layers
12
12
24
Hidden size
256
768
1024
FFN inner hidden size
1024
3072
4096
Attention head size
4
12
16
Attention heads
64
64
64
Embedding size
128
768
1024
Mask percent
15
15
15
Learning rate decay
I Linear
Linear
Linear
Warmup steps
10000
10000
10000
Learning rate
5e-4
2c-4
le-4
Adam
le-6
le-6
le-6
Adam 31
0.9
0.9
0.9
Adam $2
0.999
0.999
0.999
Dropout
0.1
0.1
0.1
Weight decay
0.01
0.01
0.01
Batch size
128
256
288
Training steps (MLM/ELECTRA)
1.45M/IM
1M/-
1M/-
B FINETUNING DETAILS
Our finetuning setting (see Table for details) mostly followed that of ELECTRA. Most tasks were finetuned for 3 epochs. However, inspired by ELECTRA, which increases the number of epochs to 10 for the RTE task, we also found that increasing the number of epochs to 10 helped stabilize the finetuning performance for CB and BoolQ. Except for this, we did not perform any hyperparameter search for finetuning.
Table 4: Finetuning hyperparameters.
Hyperparameter
Value
Learning rate
3e-4 for Small, Ic-4 for Base, 5e-5 for Large
Adam epsilon
le-6
Adam befa.
0.9
Adam beta. 2
0.999
Layerwise LR decay
0.8 for Base/Small, 0.9 for Large
Learning rate decay
Linear
Warmup fraction
0.1
Dropout
0.1
Weight decay
0.1
Batch size
32
Training epochs
10 for RTE, CB, BoolQ; 3 for other tasks.
13

---

Under review as a conference paper at ICLR 2023
For the WiC and ReCoRD tasks, in which span embeddings for words entities required to compare spans with each other, we created span embeddings by applying average pooling to the final hidden representations of the tokens constructing span and compared spans by passing the concatenated span embeddings to the classifiers.
C SHUFFLING RATE
Table 5: Comparison of different sentence shuffling rates terms of the SuperGLUE development results for Small-scale MLM-based models. We choose 15% as our final choice for shuffling rate. The results show that TSP is not sensitive to the extent of shuffling, but shuffling is necessary for TSP.
Shuffling rate
SuperGLUU
0%
59.7
15%
60.3
30%
60.3
50%
60.2
100%
60.1
Among the sentences in a given input sequence, we randomly shuffled 15% of them while other sentences stay at their original positions, to prevent complete leakage of the original ordering and hierarchical information. We also experimented with different rates for such kind of partial shuffling, and the results are listed in Table The case of no shuffling and complete shuffling gave the worst results, while there was little difference in performance for shuffling of 15 ~ 50%. Accordingly, the results indicate that shuffling needed, but the proposed task not sensitive to the partial shuffling rate. We chose 15% as the partial shuffling rate in this paper because it was the lowest shuffling rate this analysis that gave good performance with minimal corruption of inputs.
D RESULTS ON THE SUPERGLUE DEVELOPMENT SET
For reference, Table lists experimental results the SuperGI.UR development
E RESULTS ON THE GLUE BENCHMARK
The GLUE benchmark is the predecessor of the SuperGLUE benchmark. Compared to the old GLUE benchmark, SuperGLUE comes with a new set of more difficult language understanding tasks and improved resources, which encourages to evaluate models on SuperGI.UF. as our main results. At the same time, we also provide GLUE results for readers' reference. As shown in Table the proposed task, which learns from both hierarchical and ordering relationships, also outperform other baselines on the GLUE benchmark at Small, Base, and Large scales.
F COMBINATION WITH OTHER WORD-LEVEL PRETRAINING METHOD
Because the proposed method can casily be added on top of other word-level pretraining methods, we conduct an experiment to evaluate whether the proposed method improves performance when it not combined with MI.M. ELECTRA (Clark al. 2020), another word-level pretraining method, was adopted as the alternative to MLM in this experiment. We chose ELECTRA because it is a discriminative pretraining method that performs binary classification to detect replaced words among all words, making largely different from MLM, which is generative and predicts masked words. Although ELECTRA comprises a generator to generate inputs for discrimination and a discriminator to detect replaced tokens in generated inputs, the generator is discarded after pre- training. and only the discriminator is used for finetuning. This is why, when combining TSP with EL ELECTRA, applied TSP only to the discriminator and the generation stage remained unchanged.
14

---

Under review as a conference paper at ICLR 2023
Table 6; Results of MLM-based models on the SuperGLUE development set.
CB
COPA
MultiRC
RTE
WIC
BoolQ
ReCoRD
Scale
Model
Avg.
Avg.
MLM
58.7
83.9/77.6
55.0
67.1/12.8
59.6
61.0
70.7
39.0/48.2
MLM+NSP (BERT)
60.4
70.7/80.4
58.0
67.0/14.5
66.4
73.0
38, 38.6/48.1
Small
MLM+SOP
60.0
71.7/78.6
55.0
67.2/14.8
68.6
63.5
73.2
39.1/48.2
MLM+SSO
60.0
65.0/75.0
56.0
67.3/143
66.4
68.8
73.2
39.3/49.1
MLM+TSP (ours)
72.9/80.4
66.6/15.2
62.1
73.5
89. 00.77499.
MLM
71.2
90.6/92.9
63.0
75.1/27.3
73.7
66.9
80.1
64.9/78.0
MLM+NSP (BERT)
71.6
89.9/92.9
63.0
75.8/28.7
73.8
70.1
79.9
64.1/76.9
Base
MLM+SOP
83.7/87.5
75.1/26.8
68.7
79.1
62.2/74.6
MLM+SSO
70.8
86.3/91.1
66.0
75.5/26.7
72.2
70.3
78.9
62,4/74.9
MLM+TSP (ours)
73.9
88. 1/92.9
72.0
79.0/84 19.0/334.1
77.6
68.3
80.6
65. 2/78./
MLM
76.4
93.0/94.6
70.0
80.3/35.8
83.3
71. 71.7/85.5
MLM+NSP (BERT)
76.7
93.7/94.6
70.55
81,5/41.6
81.2
69.7
82.5
70.7/84.2
Large
MLM+SOP
77.0
93.2/94.6
71.0
82.0/3.0
80.9
69.0
83.3
71.1/85.0
MLM+S50
77.3
93.6/94.1
71.5
82.9/43.1
82.0
70.2
83.3
70.6/84.2
MLM+TSP (ours)
78.0
94.3/96.4
72.0
83.3/45.4
82.0
70.5
83.9
71,4885.0
Table Results of MI-M-based models on the GLUE development set. "Pearson" stands for Pear- son correlation. "Spearman" stands for Spearman correlation.
CoLA
SST
MRPC
STS
QQP
MultiNLI
QNLI
RTE
Scale
Model
Avg.
Matthew's Corr
Acc.
F1/Ace
PeersonSpecrmaon
Fl/tet.
matched/unmutcced
Acc.
Acc.
MLM
77.3
47.0
90.0
89.4/85.0
83.6/83.2
84.6/88.5
79.2/80.3
85.2
59.6
MLM+NSP (BERT)
79.1
44.4
89.8
90.0/85.8
88. 4/88.0
85.4/89.1
79.9/81.0
88.3
Small
MLM+SOP
78.4
44.5
89.6
89.2/84.8
85.3/84.8
85.1/88.8
78.8/79.3
86.6
MLM+SS0
79.2
45.1
89.6
90.1/85.8 8
88.2/87.8
85.6/89.2
80.3/81.2
88.7
66.4
MLM+TSP (ours)
46.8
91.0/87.2
87.1/86.7
65.6/89.3
80.2/81.2
MLM
84.6
61.5
93.5
92.2/89.
89.7/89.5
87.9/91.0
86.6/86.7
91.8
73.7
MLM+NSP
84.3
60.4
93.1
91.5/88.2
90.4/90.1
87.9/91.0
85,3/85.7
91.7
73.8
Base
MLM+SOP
83.5
58.1
92.0
91.7/88.5
89.9/89.6
87.7/90.9
84.9/85.2
91.0
72.6
MLM+SSO
83.9
60.3
92.5
91.6/88.2
90.7/90.3
87.5/90.8
85,0/85.5
91.5
72.2
MLM+TSP (ours)
85.5
62.5
.9
92. 92.5/89.1
90. 7/90.4
87.8/91.0
86.3/86.6
92.8
77.6
MLM
86.5
62.5
95.0
92.0/88.8
90.8/90.7
89.0/91.7
88. 7/8.7/8..1
93.3
80.7
MLM+NSP
86.2
60.2
94.5
92.2/89.2
91.8/91.4
88.8/91.7
88,0/88.3
93.0
81.2
Large
MLM+SOP
86.5
61.7
91.6/91.4
88.9/91.7
88.3/88.6
MLM+SSO
86.5
61.5
94.5
92.2/88.8
91.8/91.6
88.8/91.6
88.4/88.5
93.0
82.0
MLM+TSP (ours)
87.4
64.7
95.1
93.4/90.9
91.8/91.6
89.1/91.9
88.3/89.0
94.1
82.0
Note that, as in the previous experiments, TSP added negligible parameters and computation to the ELECTRA pretraining.
The experimental results listed in Table 9 When combined with ELECTRA. the proposed method could still improve the SuperGLUE score. Because limited computational resources and time, we did not perform these experiments beyond the Small-scale, but expect that the proposed method would perform well in such cases. Overall, this experiment shows the capability of the proposed method to be combined with other word-level pretraining methods, thus suggesting the opportunity to improve such methods via text structure learning the future.
15

---

Under review as a conference paper at ICLR 2023
Table 8: Results of MLM-based models on the GLUE development set. "Pearson" stands for Pear- son correlation. "Spearman" stands for Spearman correlation.
CoLA
SST
MRPC
STS
QQP
MultiLI
QNLI
RTE
Scale
Model
Avg.
Phanon/ppammu
MLM
77.3
47.0
90.0
89.4/85.0
83.6/81.2
84.6/88.5
79.2/80.3
85.2
59.6
MLM
79.1
44.4
89.8
90.0/85.8
88.4/88.0
35.4/89.1
79.9/81.0
88.3
66.4
Small
MLM+SOP
78.4
44.5
89.6
89,2/84.8
85.3844.8
85,1/88.8
78.8/79.3
86.6
68.6
MLM+SSO
79.2
45.1
89.6
90.1/85.8
88.2/87.8
6..6/99.2
80.3/81.2 2
88.7
66.4
MLM+TSP (ouns)
79.8
46.8
89.7
91. 01.0/87.2 2
87.1/86.7
85 6/89.3
80.2/81 2
88.9
68.9
MLM
84.6
61.5
93.5
92.2/89.1
89.7/89.5
87. 97900
86. 96.6/86.7
91.8
73.7
MLM+NSP
84.3
60.4
93.1
91.5/88.2
90.4/90.1
87.9/91.0
85.3/85.7
91.7
73.8
Base
MLM+SOP
83,5
58.1
92.0
91.7/88.5
89.9/89.6 6
87.7/90.9
84.9/85.2
91.0
72.6
MLM+SSO
83.9
60.3
92.5
91.6/88.2
90.7/90.3
87.5/90.8
85.0V85.5
91.5
72.2
MI M-M+TSP (omrs)
85.5
62.5
93.9
92.5/89.7
90. 7/90.4
87.8/91.0
86.3/86.6
92.8
77.6
MLM
86.5
62.5
95.0
92.0/88.8
90.8/90.7
89.0/91.7
88. 7/89.1
93.3
80.7
MLM+NSP (ESII]
86.2
60.2
94.5
92.2/89.2
91.8/91.4
88.8/91.7
88.0/88.3
93.0
81.2
Large
MLM+SOP
86.5
61.7
94.6
92.6/89.7
91.6/91.4
88.9/91.7
88.3/88.6
93.0
80.9
MLM+SSO
86.5
61.5
94.5
92.2/88.8
8/996 6
88.8/916
88. 88.4/88.5
93.0
82.0
MLM+TSP (ouns)
87.4
61.77
95.1
93.4/90.9
91.8/91.6
89 89.1/91.9
88.3/89.0
9-1.
82.0
Table 9: Results of ELECTRA-based models on the SuperGLUE. development set. TSP performs well even when combined with another word-level pretraining method that vastly different from MLM.
Scale
Model
SuperGLUE
ELECTRA
57.4
ELECTRA#NSP
58.6
Small
ELECTRA+SOP
58.5
ELECTRA+SSO
60.6
ELECTRA+TSP
60.7
16


Reviews:
[{"review_id": "uvWN_xJJKx", "reviewer": "Reviewer_t9DH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a new inter-sentence training task for language model pertaining, called textual structure prediction. The author designs six structure relationship labels for TSP prediction. The empirical results show that the TSP improves the performance of pretrained language models on NLU tasks.", "summary_of_the_review": "The paper proposed hierarchical structure labels for training language models. The empirical results show performance improvements, while why the hierarchical labels work needs more reasonable explanations.", "strength_and_weaknesses": "Strength:\n1. The paper is clearly stated and easy to follow.\n\n2. The performance improves significantly in some of the SuperGLUE downstream tasks.\n\nWeaknesses:\n1. The hierarchical relationship labels are not convincing enough. It is even difficult for humans to predict whether two sentences are from the same document. Enforcing the model to learn the \"same document\" label might cause the overfitting of the sentence representations. The ablation study also shows that order information is more important than hierarchical structure information.\n\n2. The empirical results are not uniformly improved with the proposed TSP methods on the downstream tasks. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "fiJNBLL_3h6", "reviewer": "Reviewer_VZCD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new way to pre-train language models. The main idea is to include sentence-level hierarchy information during the pre-training. Instead of considering only neighbor sentences, they consider more relations between sentences such as if they are in the same paragraph or if they are in the same document. By training this classification loss along with the original masked token loss, they have better performance on the SuperGLUE downstream tasks.", "summary_of_the_review": "The motivation is clear and reasonable. Although the proposed method is simple, it's very effective. Ablation study supports their claim.", "strength_and_weaknesses": "Strength\n- The motivation is clear and reasonable.\n- The experiments support the claim and seem to be promising\n\nWeaknesses\n- To test the generalizability, I suggest to test on GLUE tasks as well.\n- It's not clear what size of models is using for the ablation study in Table 2 \n- What will happen if we do not shuffle the sentences?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WM1I_d1w5p", "reviewer": "Reviewer_LMAd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper argues for adding inter-sentence tasks to language model pretraining. In this direction, the paper introduces Text Structure Prediction (TSP), an improved version of BERT's Next Sentence Prediction where a LM must classify two sentences using one of six labels that track whether sentences belong to the same document, same paragraph, are neighbors and their order (A then B vs. B then A).", "summary_of_the_review": "A good paper with a couple of minor issues that can be easily addressed before a camera ready.\n", "strength_and_weaknesses": "## Strengths\n\n* Well-written and easy to follow.\n* The models are properly trained before evaluation. Section 4.1.1 states, \"To ensure fair comparisons, all models at the same scale had almost the same computational cost and number of parameters in both training and inference.\". This is a detail people often forget, so I am happy to see that the paper takes computation into account.\n\n## Weaknesses\n\n* The introduction argues that \"Without text structure, a text becomes a long continuous word sequence, which is hard to read and makes it difficult to identify key concepts and logical relationships.\" While this is probably, intuitively, true for humans, the paper uses the statement to argue against prior unsupervised sentence tasks and projects intuitive understanding of human attention and reading abilities onto machines (i.e., language models) without valid proof. This is a logical fallacy. I do not believe that this is a major weakness for the paper. The paragraph just needs to be rewritten to avoid the incorrect argument.\n* The Related Work / Word-Level Pretraining section is a little too naroow. I understand that the paper's method only applies to transformer encoders, but I believe the paper would benefit from summarizing pretraining methods for encoder-decoders too.\n* Section 4.1.1 cites \"the sentence splitter of NLTK (Bird et al., 2009) for the use of TSP\". NLTK's sentence splitter is Punkt (Kiss and Strunk, 2006), see also https://www.nltk.org/_modules/nltk/tokenize/punkt.html. The paper should cite the Punkt paper together with the NLTK citation. Giving credit to the method's authors is not only fair, but it explicitly names the sentence splitter used, thus improving clarity for readers.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TGZmIS1rCr", "reviewer": "Reviewer_SQUJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an inter-sentence pretraining task, Text Structure Prediction (TSP), which encourages the encoder to classify sentence structure relationships. The relationships are manually defined to be among 6 types across two dimensions: hierarchical relation and ordering relation. The hierarchical relation contains three cases: neighbor, same paragraph and same document; the ordering relation contains two cases: original and reversed order. By randomly shuffling a subset of the sentences in a sequence, TSP can be used to train the encoder to classify sentence pairs into the pre-defined 6 relation types. The authors argue that such an inter-sentence pretraining task provides more high-level language understanding signals and thus improves using word-level pretraining tasks only. The major benefit of TSP compared to previously proposed inter-sentence pretraining task, as pointed out by the authors, is that TSP is able to offer performance gain for encoders of different sizes (small, base, large), whereas previous methods usually do not benefit large model sizes. The empirical evaluation is conducted on the SuperGLUE benchmark. The authors show that TSP is more effective than other compared sentence-level tasks when combined with MLM.", "summary_of_the_review": "I believe the paper will need quite a lot major revisions to be convincing. (1) There should be more discussions and evaluations regarding what are the real differences from NSP and SOP. (2) The method needs to be compared with contrastive-learning based pretraining tasks. (3) The evaluation needs to follow conventional pretraining/fine-tuning settings.", "strength_and_weaknesses": "Pros:\n* The paper is clearly written and well-organized.\n* The proposed pretraining task TSP is shown to outperform the compared sentence-level pretraining objectives (NSP, SOP and SSO) on SuperGLUE.\n\nCons:\n* The novelty of the method is quite limited. The proposed objective TSP is essentially a combined (and slightly extended) version of NSP and SOP -- the \"hierarchical relation\" classification is extending NSP by further classifying the \"non-neighbor\" case into \"same-paragraph\" and \"same-document\"; the \"order relation\" classification is equivalent to SOP. In this sense, the proposed task is not new and should be compared with multi-task training that uses both NSP and SOP (this seems to be an important baseline that is missing from comparisons).\n* The paper omitted discussions and comparisons with an important family of sentence-level pretraining tasks that use contrastive learning. For example, COCO-LM and CLEAR demonstrated that contrastive learning-based sequence-level tasks can effectively improve NLU task fine-tuning performance, and they can also be combined with word-level pretraining tasks. \n* The empirical evaluation is quite problematic. My major concern comes from the fact that the paper does not follow the conventional pretraining/downstream task settings that have been extensively used by previous studies. (1) The pretraining corpus used in this paper is OpenWebText whereas previous pretraining methods use either Wikipedia + BookCorpus (in BERT) or further expanded corpora that include a wide variety of datasets besides OpenWebText (in RoBERTa). The common pretraining corpora choice could be either of the two above. Even if BookCorpus is not publicly available, Wikipedia should at least be included as a formal language pretraining source. If the pretraining experiments are conducted only on OpenWebText, it's very unclear whether the observed performance gain of TSP can be generalized to common/larger pretraining corpora. (2) Most pretrained language models (e.g., BERT, RoBERTa, ELECTRA) have been evaluated on GLUE whereas this paper only reports SuperGLUE results. This makes it hard to compare with previous pretrained models. While it is good to have SuperGLUE results, I believe it's important to conduct experiments on GLUE as well to facilitate easy comparisons with prior work. (3) The evaluation results only have the best performing results on the test set but not median results across all tasks (Table 6 only has averaged scores). This is problematic because some tasks have very small test sets and the scores could vary a lot. The common practice is to report median results over several random seeds on dev sets of all tasks.\n\nReferences:  \nMeng, Yu et al. “COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining.” NeurIPS (2021).  \nWu, Zhuofeng et al. “CLEAR: Contrastive Learning for Sentence Representation.” ArXiv abs/2012.15466 (2020).\n\n---\n**Post-Rebuttal Updates**:  \nI thank the authors for providing responses to my raised concerns. Unfortunately, I'm not convinced by the authors' arguments regarding my major concerns. My detailed comments are as below:\n* Regarding novelty: I never implied in my review that a method needs to be complicated to be novel; instead, simple methods can be novel as long as they provide new perspectives/insights. My critique regarding the novelty of the paper is mainly due to the fact that the proposed objective TSP is very similar to previously proposed sentence-level objectives (NSP and SOP) in formulation. The major difference between TSP and NSP+SOP, according to the authors' response, is that TSP considers paragraph-level relationships. However, the motivation for this idea is very vague -- how can one accurately/unambiguously define the real difference between a sentence and a paragraph, and the difference between a paragraph and a document? For example, in novels, it's very common that one or two sentences can be a paragraph; other times, a paragraph can also be an entire document. If the distinction between sentences, paragraphs and documents is even hard to tell for humans, then I don't see how the so-called \"paragraph-level relationships\" newly introduced in TSP can actually make a big difference compared to a simple combination of previous methods (NSP+SOP).\n* Regarding the comparisons to contrastive learning approaches: I'm not fully convinced by the argument that the comparisons between contrastive learning objectives and the proposed TSP objective can be omitted. I understand that the two sets of methods are based on different principles, but they are indeed for the same goal -- improving the sequence-level representations of PLMs. Therefore, unless TSP has other usages/benefits that contrastive learning fails to bring, I don't see why they cannot be fairly compared to each other. Such comparisons do not exist in prior work probably because it is quite well known that NSP/SOP-style methods do not really work well (also shown in this paper), and many PLMs simply ended up not using any sequence-level objectives (e.g., RoBERTa, ELECTRA), so the contrastive learning papers didn't compare with NSP/SOP (of course, I do hope to see those results in the contrastive learning papers). \n* The empirical gain of TSP does not seem convincing to me: Although TSP provides significant gains on GLUE/SuperGLUE average score, the gains are mainly from small/unstable tasks, like CoLA, MRPC, RTE and STS-B. Fine-tuning on those tasks are notoriously unstable and is subject to high variance. On large tasks like MNLI and SST-2 (note that they are more reliable than the previous tasks; RoBERTa paper uses these two tasks intead of the average GLUE score for ablation studies), the TSP objective barely provides any gain and even worsens the result. Therefore, I suspect that the performance gain of TSP is simply due to randomness in fine-tuning.\n* Regarding the choice of pretraining corpus (OpenWebText): I'm indeed aware that OpenWebText is larger than Wikipedia, but Wikipedia is still a more standard choice for pretraining corpus if only one corpus is used (e.g., base model exploration in BERT, RoBERTa, ELECTRA all used Wikipedia + BookCorpus). This is probably because Wikipedia's text quality is arguably higher than OpenWebText which contains many short/casual/informal texts. Therefore, I'd suggest the authors follow the standard protocol for pretraining; otherwise, the findings and conclusions may not be reliable generalize well. This is not my major concern about the paper though. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject