SlowGuess/ACL_2020_OCR huggingface dataset:
Dan Iter\*, Kelvin Guu\*, Larry Lansing\*, and Dan Jurafsky\*
*Computer Science Department, Stanford University
**Google Research
{daniter, jurafsky}@stanford.edu
{kguu, llansing}@google.com
Recent models for unsupervised representation learning of text have employed a number of techniques to improve contextual word representations but have put little focus on discourse-level representations. We propose CONPONO<sup>1</sup>, an inter-sentence objective for pretraining language models that models discourse coherence and the distance between sentences. Given an anchor sentence, our model is trained to predict the text  $k$  sentences away using a sampled-softmax objective where the candidates consist of neighboring sentences and sentences randomly sampled from the corpus. On the discourse representation benchmark DiscoEval, our model improves over the previous state-of-the-art by up to  $13\%$  and on average  $4\%$  absolute across 7 tasks. Our model is the same size as BERT-Base, but outperforms the much larger BERT-Large model and other more recent approaches that incorporate discourse. We also show that CONPONO yields gains of  $2\%-6\%$  absolute even for tasks that do not explicitly evaluate discourse: textual entailment (RTE), common sense reasoning (COPA) and reading comprehension (ReCoRD).
Pretraining large language models has become the primary method for learning representations from unsupervised text corpora. Since the initial improvements demonstrated by ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019), many alternative pretraining methods have been proposed to best leverage unlabeled data. These methods include bi-directional language modeling (Peters et al., 2018), masked language models (Devlin et al., 2019), word order permutation (Yang et al.,
2019), more robust training (Liu et al., 2019) and more efficient architectures (Lan et al., 2019). However, little focus has been put on learning discourse coherence as part of the pretraining objective.
While discourse coherence has been of great interest in recent natural language processing literature (Chen et al., 2019; Nie et al., 2019; Xu et al., 2019), its benefits have been questioned for pretrained language models, some even opting to remove any sentence ordering objective (Liu et al., 2019). However, in a recently published benchmark for evaluating discourse representations, Chen et al. (2019) found that the best performing model was surprisingly BERT, despite comparing against models specifically designed for discourse, such as DisSent (Nie et al., 2019) and a new recurrent network trained on a large range of sentence ordering objectives. We show that combining transformer encoders with our inter-sentence coherence objective, we can further improve discourse-level representations in language models.
We present a model that trains a sentence-level encoder to capture discourse relationships between sentences, including ordering, distance and coherence. The encoder is trained by using its output to predict spans of text that are some  $k$  sentences away from a context in either direction. The predictions are made discriminatively with a sampled-softmax that contrasts the correct target sentence against negatives, including hard examples sampled from the same paragraph. Our objective is inspired by the recently proposed Constrastive Predictive Coding (CPC) (van den Oord et al., 2018), but, among other differences, is applied on the sentence-level rather than the token-level and is bi-directional. We call this the CONtrastive Position and Ordering with Negatives Objective (CONPONO) $^2$ .
* Work done during internship at Google.
  1Code is available at https://github.com/google-research/language/tree/master/language/conpono and https://github.com/daniter-cu/DiscoEval
Also means arrange or order in Latin.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4859-4870 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
We evaluate our model on DiscoEval (Chen et al., 2019), a recently published benchmark for evaluating and probing for various aspects of discourse-level semantics in representations output by discourse models. We observe that the representations learned with CONPONO outperform BERT-Large and achieve a new state-of-the-art despite using fewer parameters and training on the same data. Furthermore, we show that our new objective improves model performance on other tasks including textual entailment, common-sense reasoning and reading comprehension. We compare CONPONO against BERT-Base on RTE (Giampiccolo et al., 2007; Bentivogli et al., 2009), COPA (Roemmele et al., 2011) and ReCoRD (Zhang et al., 2018), while controlling for model size, training data and training time.
1. We describe a novel sentence-level discourse objective that is used in conjunction with a masked language model for unsupervised representation learning for text. We show that this objective can leverage the cross-attention and pretrained weights of a transformer model to learn discourse-level representations.
2. We show that our model achieves a new state-of-the-art on DiscoEval, improving the results on 5 of the 7 tasks and increasing accuracy by up to  $13\%$  and an average of over  $4\%$  absolute across all tasks. We also show  $2\%-6\%$  absolute improvements over Bert-Base on RTE, COPA and ReCoRD as evidence that discourse pretraining can also improve model performance on textual entailment, commonsense reasoning and reading comprehension.
Figure 1 illustrates the CONPONO model. The intuition is that if the model is able to accurately predict the surrounding target sentences given some anchor text, then the vector representations for these sentences should also be useful for downstream tasks.
The input to the model is a paragraph that is split into sentences. A sentence is chosen at random as the anchor, and will be denoted as  $s_i$ . We encode  $s_i$  with a transformer encoder to produce a vector  $c_i$ . The surrounding sentences are denoted as  $s_{i+k}$  where  $k \in [-K..-1, 1..K]$ , meaning the maximum distance we use is  $K$ . We report results for  $K \in [1..4]$ . These sentences,  $s_{i+k}$ , are encoded
jointly with the anchor sentence. We use just a single encoder  $g_{\theta}$  so all text is encoded with the same weights. The encoded vectors are named  $t_{i+k}$  because these are the target vectors the model tries to identify given the anchor and a target distance  $k$ . Equation 1 defines  $t_{i+k}$  and  $c_{i}$  as a function  $g_{\theta}$  of the input sentences. Note that the CONPONO  $g_{\theta}$  is different from the encoder in CPC because we input both the anchor and the target into the encoder, rather than separate anchor and target encoders.
$$
t _ {i + k} = g _ {\theta} \left(s _ {i}, s _ {i + k}\right), \quad c _ {i} = g _ {\theta} \left(s _ {i}\right) \tag {1}
$$
Given the anchor and targets, we define a log-bilinear model in equation 2 to score the plausibility of target  $t_{i + k}$  being in position  $k$  from anchor  $c_{i}$ . The full set of parameters for our model is  $\theta$  for the encoder and a  $W_{k}$  for each  $k$ . CPC has the same bi-linear form as Equation 2 but the architecture for the encoders is different.
$$
f _ {k} (s _ {i + k}, s _ {i}) = \exp (t _ {i + k} ^ {T} W _ {k} c _ {i}) \tag {2}
$$
The loss for each  $k$  is given in equation 3 where the score for the correct target is contrasted to scores of random samples  $s_j$ , sampled from both in-document and random sentences from the corpus,  $S$ .
$$
\mathcal {L} _ {k} = - \mathbb {E} _ {S} \left[ \log \frac {f _ {k} \left(s _ {i + k} , s _ {i}\right)}{\sum_ {s _ {j} \in S} f _ {k} \left(s _ {j} , s _ {i}\right)} \right] \tag {3}
$$
To train CONPONO, we sample negative examples randomly from the corpus and from the same paragraph but different  $k$  as hard negatives. Note that when  $|k|$  is greater than 1, there will be sentences between the anchor sentence and target sentence that will be purposely omitted from the input. The missing context is intended to create a challenging objective where the model may not be able to rely on trivial signals that often appear in contiguous sentences.
2.1 Encoder Architectures
For each example we encode two text spans, the anchor and the target. There are three main options for encoding the two spans into  $c_{i}$  and  $t_{i+k}$ . The simplest method, and most similar to CPC is to encode the anchor and target separately, which we call isolated encoding. With this encoder, equation 1 will be  $t_{i+k} = g_{\theta}(s_{i+k})$ . The major drawback of this approach is that there is no token-level cross-attention between the anchor and the target, which has been shown to generally improve text encoding
Figure 1: During training, a text segment is selected as the anchor  $(S_i)$ . The anchor as well as all the targets,  $S_{i-2}\ldots S_{i+2}$  plus random samples  $S_r$  are encoded with the transformer masked language model. The encoded representation of the anchor is used to predict each target at its target distance. The  $S_i$  objects are raw text sentences, the encoder is the transformer model, and  $c_i$  and  $t_i$  are vectors.
(Vaswani et al., 2017). Cross-attention is the mechanism in neural networks that allows for attention to be shared between multiple inputs, in our case, two separate spans of text.
Alternatively, we can encode the anchor and target together and then dot product the latent vector with a learned vector representation for each distance  $k$ . We call this approach a uni-encoder. With this encoder, equation 2 will be  $f_{k}(s_{i + k}, s_{i}) = \exp(t_{i + k}^{T} w_{k})$ . The class matrix  $W_{k}$  in equation 2 is replaced by a class vector  $w_{k}$ , which has fewer parameters. This is similar to the ordering objectives in BERT and ALBERT where the pooled representation is used for a binary classification task and the learned vector representation for each distance  $k$  is just the softmax weights. The potential drawback to this method is that each pair of sentences is represented by a single vector. This encoder may learn a representation that is similar for all examples that have the same label but does not explicitly model the content of the input.
CONPONO implements the intersection of these two approaches. The targets are concatenated to the anchor when encoded, to make use of the cross-attention of the transformer encoder. The anchor, is encoded independently, though with the same weights. This objective allows for more freedom in the values of  $c_{i}$  and  $t_{i + k}$ , unlike the uni-encoder. Furthermore, since the encoder,  $g_{\theta}$ , can encode either one span  $(s_i)$  or two spans  $(s_i,s_{i + k})$ , it can be used for downstream tasks that have either single (eg. SSP) or double (eg. BSO) span inputs.
2.2 Comparing Inter-Sentence Modeling Objectives
There are different tasks that can be used for learning inter-sentence representations. BERT (Devlin et al., 2019) included a next sentence prediction (NSP) task. For NSP, two spans are fed into the model with the second span either being the next contiguous span of text from the source or  $50\%$  of the time it is replaced with a random span from the corpus. The task is a binary classification of whether the two spans are from the same source. ALBERT (Lan et al., 2019) compares the NSP approach to using no inter-sentence objective and to sentence order prediction, which for clarity we refer to as binary sentence ordering (BSO). For BSO, the input is two spans that are always contiguous and from the same source but  $50\%$  of the time are in reverse order. With CONPONO we capture the benefits of both learning ordering between coherent sentences and contrasting against random negatives. We make the objective even more challenging by also predicting order on spans that are multiple sentences apart, and using other sentences from the same paragraph as harder negatives.
2.3 Technical details
In practice, we use a 512 token input which is much larger than most two sentence pairs. To train on longer sequence lengths, we use 4 sentences as the anchor and 3 sentences as the target segment. We truncate longer sentences and pad tokens up to the sequence length as done for typical BERT input. There is no overlap between the two segments and
the  $k$  distance refers to the number of sentences omitted between the two segments. For example, for a paragraph we may choose  $s_7..s_{10}$  as the anchor and  $s_1..s_3$  as the target for  $k = -4$  because  $s_3$  is 4 positions behind  $s_7$ . Since most paragraphs are not long enough to have many sentences in both directions of a 4 sentence anchor, we randomly select 4 of the 8 possible  $k$  targets for a given paragraph. Because of the random sampling, we oversample shorter distances because they occur more consistently in the data.
We train with 32 input sentences, where 1 is the correct target, 3 are hard negatives from the same document and 28 are random sentences from other documents. For fair comparison, we train on the same data as BERT, using only Wikipedia and BooksCorpus (Zhu et al., 2015). We initialize our model with BERT-Base weights and train until the model has seen one-fourth as many segment pairs as the original BERT model (32M total), so the total compute and iterations of training are not significantly greater than BERT-Base. We also use a masked language model objective similar to BERT but dynamically mask during training for different masks each epoch. When jointly encoding two inputs, we concatenate the input tokens and separate the two spans with a “[SEP]” token to mimic the BERT format.
We evaluate our model on the DiscoEval benchmark (Chen et al., 2019) and on the RTE (Giampiccolo et al., 2007; Bentivogli et al., 2009), COPA (Roemmele et al., 2011) and ReCoRD (Zhang et al., 2018) datasets. We chose the DiscoEval benchmark because it is intended to evaluate a model's ability to represent the "role of a sentence in its discourse context". We also report results on RTE, COPA and ReCoRD because these tasks have a discourse or sentence ordering aspect to them but are not exclusively designed for discourse evaluation.
3.1 Discourse Evaluation
Tasks: DiscoEval (Chen et al., 2019) is a suite of tasks "designed to evaluate discourse-related knowledge in pretrained sentence representations". The benchmark is composed of seven tasks; four based on sentence ordering or coherence (Sentence position (SP), Binary sentence ordering (BSO), Discourse coherence (DC) and Sentence section prediction (SSP)) and three that are based on classi
fying the type of relationship between a pair of text sequences (Penn Discourse Tree Bank Explicit and Implicit (PDTB-E/I) and Rhetorical structure theory (RST)). PDTB (Prasad et al., 2008) and RST (Carlson et al., 2001) are human annotated datasets. Both are multi-class classification tasks where PDTB is classifying a pair of sentences whereas RST is predicting the class of a node in a document-level discourse tree. Both classes of tasks are critical aspects of understanding discourse.
Baselines: The previously best overall performing model from DiscoEval (Chen et al., 2019) was BERT-Large (Devlin et al., 2019). We also include the results for BERT-Base because our model is most comparable to BERT-Base in terms of parameter size, training data and training compute. We also evaluate RoBERTa-Base (Liu et al., 2019) because it was trained on more data, reported improvements over BERT-Base on other tasks but dropped the next sentence prediction objective entirely. We also compare against a BERT-Base model which we trained with binary sentence ordering (BERT-Base BSO) because this objective has been shown to be more useful than next sentence prediction (Lan et al., 2019). This BERT-Base BSO model was initialized with BERT weights and trained on the same data but only on contiguous spans of text where  $50\%$  of the time we switch the order. This model and CONPONO are initialized from the same weights and trained on the same number of segment pairs so that the two models can be compared fairly.
In Section 2.1 we describe different encoding approaches for generating the sentence-level representations. We report results from versions of CONPONO using each of these encoding approaches, labeled isolated to represent separate encoding and uni-encoder to represent joint encoding of the anchor and target without a separate anchor encoding. The final line in Table 1 is the combined approach that we describe in Section 2.
Modeling DiscoEval We reuse the code from DiscoEval and generally maintain the same process for collecting our results on the benchmark, such as freezing all weights and only training a logistic regression or one layer perceptron on top of the sentence encodings. Note that since we are only interested in the vector representations of the input, we drop the weight matrix  $W_{k}$  and only use the output of the encoder. We omit the details for
Model	SP	BSO	DC	SSP	PDTB-E	PDTB-I	RST-DT	avg.
BERT-Base	53.1	68.5	58.9	80.3	41.9	42.4	58.8	57.7
BERT-Large	53.8	69.3	59.6	80.4	44.3	43.6	59.1	58.6
RoBERTa-Base	38.7	58.7	58.4	79.7	39.4	40.6	44.1	51.4
BERT-Base BSO	53.7	72.0	71.9	80.0	42.7	40.5	63.8	60.6
CONPONO isolated	50.2	57.9	63.2	79.9	35.8	39.6	48.7	53.6
CONPONO uni-encoder	59.9	74.6	72.0	79.6	40.0	43.9	61.9	61.7
CONPONO (k=2)	60.7	76.8	72.9	80.4	42.9	44.9	63.1	63.0
CONPONO std.	±.3	±.1	±.3	±.1	±.7	±.6	±.2	-
Table 1: CONPONO improves the previous state-of-the-art on four DiscoEval tasks. The average accuracy across all tasks is also a new state-of-the-art, despite a small drop in accuracy for PDTB-E. BERT-Base and BERT-Large numbers are reported from Chen et al. (2019), while the rest were collected for this paper. We report standard deviations by running the evaluations 10 times with different seeds for the same CONPONO model weights.
the encoding logic for each task since that is explained in detail in Chen et al. (2019). Here we only mention our deviations from the Chen et al. (2019) methodology. The most salient difference is that we only use the pooled representation from our model rather than the average from multiple layers of the model for the SP, BSO and DC tasks.
For encoding individual tasks we prefer to encode pairs of sentences together. For SP we encode the first sentence concatenated with every other sentence instead of taking the point-wise difference and concatenate the 5 vectors. For BSO we also encode the two sentences together instead of separately. For DC we split the paragraph into pairs of sentences and encode those together. We concatenate the 3 output vectors. For RST instead of embedding each sentence and doing a mean of all the sentences in a subtree, we simply concatenate those sentences and encode them all together as a single text span. Any text segments longer than 512 tokens are truncated from the end.
Results: Table 1 shows that our model outperforms the previous state-of-the-art accuracy on DiscoEval overall. Our model excels in particular on the sentence ordering and coherence tasks (SP, BSO, and DC). Note that our model parameter count is the same as BERT-Base but it outperforms BERT-Large, which has significantly more parameters and has used much more compute for pretraining. From the discussion in Section 2.2, BERT represents using the NSP objective and we train BERT-Base BSO to compare NSP, BSO and CONPONO directly. BERT-Base BSO scores tend to fall between those of BERT-Base and our model, implying that the sentence ordering objective is improving the models for this benchmark, but that
binary sentence ordering is not sufficient to capture the added benefits of including more fine-grained ordering and negative examples.
We observe that CONPONO outperforms both the isolated encoding and uni-encoding approaches. CONPONO isolated preforms significantly worse than both other approaches, suggesting that crossattention between the anchor and the target is critical to learning stronger discourse representations. CONPONO uni-encoder results are closer to our combined encoding approach but still fall short on every task. This empirical result suggests that the separate encoding of the anchor during pretraining is important despite the fact that theoretically CONPONO could trivially reduce to the uni-coder representation by ignoring  $c_{i}$ .
3.2 RTE, COPA and ReCoRD
Tasks: DiscoEval was specifically designed to evaluate model performance on discourse tasks but there are many other benchmarks that could also benefit from pretraining for improved discourse coherence. We evaluate our model on three such tasks, Recognizing Textual Entailment (RTE) (Giampiccolo et al., 2007; Bentivogli et al., 2009), Corpus of Plausible Alternatives (COPA) (Roemmele et al., 2011) and Reading Comprehension with Common-sense Reasoning Dataset (ReCoRD) (Zhang et al., 2018). We report accuracy on the validation set provided by each dataset.
Each example in RTE is a pair of sentences. The model must classify whether or not the second sentence entails the first. Examples in COPA are composed of a single context sentence followed by two candidate sentences that are either a cause or effect of the context sentence. The model must select the
Context	Completions
ReCoRD
... Despite its buzz, the odds are stacked against Google's Chrome OS becoming a serious rival to Windows... Chrome OS must face the same challenges as Linux: compatibility and unfamiliarity. A big stumbling block for Google will be whether its system supports iTunes.	Google will also be under pressure to ensure [Chrome OS / iTunes / Linux] works flawlessly with gadgets such as cameras, printers, smartphones and e-book readers.
RTE
Rabies virus infects the central nervous system, causing encephalopathy and ultimately death. Early symptoms of rabies in humans are nonspecific, consisting of fever, headache, and general malaise.	Rabies is fatal in humans.
COPA
The women met for coffee.	They wanted to catch up with each other. The cafe reopened in a new location.
most "plausible" sentence of the two. Lastly, an example in ReCoRD is a paragraph from a news article, followed by several bullet points and with all the entities marked. The model is given a single sentence from later in the document with a single entity masked out and must select the entity from the context that fills the blank. Table 2 shows examples of each with correct choices in bold.
Baselines: We compare our model against BERT-Base because this is the closest model in terms of parameter size and training data. However, since our model is initialized with BERT-Base weights, we also report results from BERT-Base BSO because it was trained on the same number of text examples as CONPONO. We also compare against BERT-Large to contrast to a much larger language model. We provide results from Albert (Lan et al., 2019) when available to provide a state-of-the-art baseline that may have used more data, compute and parameters. The purpose of these results is not to compare against the current state-of-the-art but rather to better understand the improvements that can be found from adding a discourse coherence objective to BERT-Base without significantly increasing the model size or training data.
Results: We believe that the coherence and ordering aspects of these evaluation tasks are well fit to demonstrate the how our model can improve on strong baselines such as BERT-Base. Table 3 shows that our model achieves accuracies on RTE and COPA comparable to BERT-Large while having the same number of parameters as BERT-Base. Interestingly, we observe improvements over the baseline with BERT-Base BSO, showing that even
Table 2: These are examples from ReCoRD, RTE, and COPA that exhibit aspects of discourse coherence. For ReCoRD, candidate entities are in italics and replaced terms in the completion are underlined. True completions are bold.
Model	RTE	COPA
BERT-Base	66.4	62.0
BERT-Base BSO	71.1	67.0
CONPONO	70.0	69.0
BERT-Large	70.4	69.0
ALBERT	86.6	-