SlowGuess/ACL_2020_OCR huggingface dataset:
simple discourse-level objectives could lead to noticeable downstream effects. Though these improvements are modest compared to BERT-Large, they are meant to highlight that our model does not only improve on results for artificial sentence ordering tasks, but also on aspects of benchmarks used to generally evaluate pretrained language models and language understanding.
3.2.1 ReCoRD results and models
Table 3: Our model improves accuracy over BERT-Base for RTE and COPA benchmarks. Improvements are comparable to BERT-Large but still lag behind much larger models trained on more data, such as ALBERT. All scores are on the validation set.
Model	Accuracy
BERT-Base	61.2
CONPONO	63.2
BERT-Large	69.8 [EM]
Table 4: CONPONO is more effective at classifying the most plausible sentence from the extended context than BERT-Base. We report the BERT-Large exact match score, where the model selects only the target entity from the context, for reference. All scores are on the validation set.
The task for the ReCoRD dataset is to select the correct entity from those that appear in the context to fill in the blank in the target. Previous models for ReCoRD have used a similar structure to SQuAD (Rajpurkar et al., 2016) where the model outputs a vector for each token and the model learns the best start and end position of the answer span based on the softmax over all the tokens. We, instead, generate all possible target sentences by filling the blank with each marked entity and discriminatively choose the sentence most likely to be the true "plausible" sentence from the context. This modified task evaluates how our model compares to BERT-Base choosing the most coherent sentence from a set of nearly identical sentences. In Table 4 we show that CONPONO does achieve a boost over BERT-Base but is still well below BERT-Large exact match score on the harder task of selecting the entities in context. The strong results from BERT-Large imply that having a better representation of the text with a large model is able to subsume any improvement from learning plausible contexts for this task.
There are three aspects of our modeling choices that warrant a deeper understanding of their importance to the model:
- Window size: We ablate the 4 window sizes (ie. choices of k).  $\mathrm{k} = 1$  is effectively binary sentence ordering with negative samples.
- Masked Language Model Objective: We remove the MLM objective allowing the model to optimize only the CONPONO objective without maintaining a good token level representation.
- Model size: We train a smaller model that is also initialized with pretrained weights.
To measure the effects of each of these design decisions, we report DiscoEval scores for each model as well as accuracy on the CONPONO classification task on a held-out set of examples. This is to show how well the model is optimized as well as how well it performs on downstream tasks.
Table 5 shows the results on DiscoEval with our model and several key ablations. We observe that using a window size for our objective that is larger than 1 is key to seeing downstream improvements. We believe that this is due to the objective being
harder for the model because there is more variation farther from the anchor. At the same time, increasing the window size beyond 2 seems to result in similar performance. This may be because larger distances from the anchor also lead to more ambiguity. We see this reflected in the held-out classification accuracy being lower for examples with larger distance labels in Figure 2.
We also note that keeping the masked language model objective during pretraining also improves downstream performance. In Figure 2 we see that classification accuracy is consistently lower with the MLM objective compared to without. This is expected because during inference, many key terms may be masked out, making the task harder. However, keeping this objective during pretraining maintains a good token-level representation that is necessary for downstream tasks.
Lastly, we try training a smaller version of our model, with only 2 hidden layers, and a 512 intermediate size. The smaller model is able to train much faster, allowing us to train on many more examples and new data. However, we are unable to achieve similar results despite training on 24 times more examples, and including CCNews (Liu et al., 2019), a larger and higher quality data source.
3.4 Qualitative Analysis
To glean some insight into how CONPONO representations may differ from BERT-Base representations, we look at the occurrence of discourse markers in the BSO-Wikipedia task of DiscoEval. We choose this task because it is a simple binary classification task that has only 2 sentences as input and the domain is similar to the pre-training data. We look at the usage of discourse markers identified by Nie et al. (2017); but, when, if, before, because, while, though, after, so, although, then, also, still.
We extract examples from the test set on which CONPONO output the correct label and BERT-Base output the incorrect label and visa versa. For each set of examples, we measure the change in the occurrence of discourse markers relative to the training data counts. Since some markers are much more common than others, we take the weighted average of the change in appearance rate, where the weights are the training data counts of each marker.
3We omit and and as because they are very common in this corpus but often are not used as connectives between the two candidate sentences for the BSO task.
Model	SP	BSO	DC	SSP	PDTB-E	PDTB-I	RST-DT	avg.
k=4	59.84	76.05	73.62	80.65	42.28	44.25	63.00	62.81
k=3	60.47	76.68	72.74	80.30	43.40	44.28	62.56	62.92
k=2	60.67	76.75	72.85	80.38	42.87	44.87	63.13	63.07
k=1	47.56	66.03	72.62	80.15	42.79	43.55	62.31	59.29
- MLM	54.92	75.37	68.35	80.2	41.67	43.88	61.27	60.81
Small	45.41	61.70	67.71	75.58	35.26	36.18	46.58	52.63
Table 5: The ablation analysis shows the effects of different  $k$  values (ie. window sizes) in our objective, removing the MLM objective during pretraining and training with a small transformer encoder.
Per Position Accuracies on Unseen Examples
Figure 2: We can evaluate the accuracy on the CONPONO objective for each label (ie. distance between anchor and target sentence) on a set of 5,000 examples held-out from training. We observe that higher accuracy does not necessarily correlate with better downstream performance on DiscoEval.
We find that in the set of examples that CON-PONO classified correctly, the rate of discourse makers was  $15\%$  higher than in the training corpus. This is in contrast to  $11\%$  higher among the examples that BERT classified correctly. The standard deviation for random samples of the same size was about  $1\%$ . This suggests that both BERT and CON-PONO are relying heavily on discourse markers to solve the BSO-Wikipedia task.
While it is expected for shallow discourse markers to be strong features for sentence ordering, we expect CONPONO to also incorporate deeper features, such as anaphora, due to its pretraining objective. One indication of CONPONO relying on alternative features than BERT-Base is that there was a  $12\%$  relative increase in discourse markers in the CONPONO set when counting markers only in the first sentence whereas an  $8\%$  relative increase in the BERT set when counting markers only in the second sentences. The difference in the location of the discourse markers in the two sets of examples suggests that CONPONO and BERT utilize those features differently and that CONPONO may be less likely to incorrectly classify examples that use discourse markers in the first sentence of a BSO example. Manually inspecting a sample of examples hints that there are often strong corefer
ences between the two input sentences that indicate the ordering.
Table 6 shows two examples from the CONPONO correct set which is drawn from the BSO-Wikipedia test data. In both examples, the discourse marker appears in the first sentence but the second sentence contains anaphora referring to an antecedent in the first sentence.
Some of the largest improvements on benchmarks such as GLUE (Wang et al., 2018) have come from ELMO's large scale bi-directional language modeling (Peters et al., 2018), BERT's masked language models (Devlin et al., 2019), XLNET's generalized autoregressive pretraining (Yang et al., 2019), RoBERTa's robust training (Liu et al., 2019) and ALBERT's parameter reduction techniques (Lan et al., 2019). As discussed in Section 2.2, most language model were limited to NSP or BSO for inter-sentence representation learning. We showed that by comparing to BERT, which uses NSP and BERT-Base BSO which we train with the BSO objective that our objective is able to improve the discourse-level representations by training on more fine-grained sentence ordering, non-contiguous
In 1941 [1] Vaughn joined the United States National Guard for what had been planned as a one-year assignment, but when [2] World War II broke out, he was sent abroad until the war ended in 1945.
[1]He decided to make music a career when he was discharged from the army at the end of [2]the war, and attended Western Kentucky State College, now known as Western Kentucky University, majoring in music composition.
Although it lasted only twenty-three years ( 1933–1956 ) and enrolled fewer than 1,200 students , Black Mountain College was one of the most fabled experimental institutions in art education and practice .
It launched a remarkable number of the artists who spearheaded the avant-garde in the America of the 1960s.
Table 6: Two examples from the DiscoEval BSO-Wikipedia test set on which CONPONO made the correct prediction but BERT-base did not. Bold terms are discourse markers, underlined terms are co-referents. In both examples, the discourse marker appears in the first sentence but the second sentence has anaphora referring to an antecedent in the first sentence.
neighboring sentences and contrasting against random negatives.
Early approaches to sentence representation, such as Skip-Thought Vectors (Kiros et al., 2015), mimicked word embedding methods in addition to left-to-right language modeling to use unlabeled data to learn sentence level representations. DisSent (Nie et al., 2019) focused more on collecting data that could be used to train a supervised classification model on pairs of sentences. These and other innovations in sentence representation lead to the creation of more evaluations for discourse and coherence representation (Chen et al., 2019; Xu et al., 2019).
Like other unsupervised representation learning models, CONPONO is trained to generate a latent variable that encodes inter-sentence relationship and discourse coherence. Our objective is inspired by the Contrastive Predictive Coding (CPC) objective (van den Oord et al., 2018). CPC was originally designed to be a "universal unsupervised learning approach to extract useful representations from high-dimensional data" and was previously implemented on the token-level for text models. We utilize the k-distance predictions of CPC because it naturally captures discourse and sentence ordering properties when applied on the sentence-level. Furthermore, by combining our objective with a transformer encoder, our model is able to benefit from cross-attention between the anchor and the target sentences, which we show outperforms encoding the anchor and target separately, as implemented in CPC. In Section 3.3 we show that the cross-attention is an important factor in learning a good representation for downstream tasks and effectively optimizing our inter-sentence objective.
In this paper we present a novel approach to encoding discourse and fine-grained sentence ordering in text with an inter-sentence objective. We achieve a new state-of-the-art on the DiscoEval benchmark and outperform BERT-Large with a model that has the same number of parameters as BERT-Base. We also observe that, on DiscoEval, our model benefits the most on ordering tasks rather than discourse relation classification tasks. In future work, we hope to better understand how a discourse model can also learn fine-grained relationship types between sentences from unlabeled data. Our ablation analysis shows that the key architectural aspects of our model are cross attention, an auxiliary MLM objective and a window size that is two or greater. Future work should explore the extent to which our model could further benefit from initializing with stronger models and what computational challenges may arise.
We wish to thank the Stanford NLP group for their feedback. We gratefully acknowledge support of the DARPA Communicating with Computers (CwC) program under ARO prime contract no. W911NF15-1-0462
Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. 2009. The fifth PASCAL recognizing textual entailment challenge.
Lynn Carlson, Daniel Marcu, and Mary Ellen Okurovsky. 2001. Building a discourse-tagged corpus in the framework of rhetorical structure theory. In Proceedings of the Second SIGdial Workshop on Discourse and Dialogue.
Mingda Chen, Zewei Chu, and Kevin Gimpel. 2019. Evaluation benchmarks and learning criteria for discourse-aware sentence representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 649-662, Hong Kong, China. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pages 1-9. Association for Computational Linguistics.
Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in neural information processing systems, pages 3294-3302.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.
Allen Nie, Erin Bennett, and Noah Goodman. 2019. DisSent: Learning sentence representations from explicit discourse relations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4497-4510, Florence, Italy. Association for Computational Linguistics.
Allen Nie, Erin D Bennett, and Noah D Goodman. 2017. Dissent: Sentence representation learning from explicit discourse relations. arXiv preprint arXiv:1710.04334.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages
2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn discourse TreeBank 2.0. In LREC 2008.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD:  $100,000+$  questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium. Association for Computational Linguistics.
Peng Xu, Hamidreza Saghir, Jin Sung Kang, Teng Long, Avishek Joey Bose, Yanshuai Cao, and Jackie Chi Kit Cheung. 2019. A cross-domain transferable neural coherence model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 678-687, Florence, Italy. Association for Computational Linguistics.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237.
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. ReCoRD: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint 1810.12885.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19-27.
We include some fine-grained DiscoEval results that were reported as averages, as well as implementation and reproduction details for our experiments.
A.1 SP, BSO and DC breakdown
Table 7 shows the scores for each model per each dataset domain for the SP, BSO and DC tasks in DiscoEval.
A.2 CONPONO pretraining details
CONPONO is pretrained on 1.6 million examples randomly sampled from Wikipedia and BooksCorpus. We use the same number of training examples for all the ablations and training BERT-Base BSO. On example consists of a single anchor and 32 candidate targets, 4 losses (1 for each of the 4 randomly chosen true targets (ie.  $k$ )). We use a  $25\%$  warm up rate and a learning rate of 5e-5. The model is initialized with BERT-Base weights. We add a square interaction weight matrix that is the same size as model output dimensions (ie. 756) that is referred to as  $W_{k}$  in Section 2. There is one such matrix for each  $k$ . The maximum sequence length of the input is 512, though do to some preprocessing constraints, the maximum input seen by the model is 493.
Our CONPONO small model has a hidden size of 128, an intermediate size 512, and has 2 hidden layers. We train it on 38.4 million examples, including examples from CCNews. Samples are drawn from each source proportional to the size of the source, meaning that about  $70\%$  of training examples come from CCNews. Otherwise, we use all the same parameters as CONPONO.
A.3 Parameter counts
Table 8 shows the number of parameters in each model used.
A.4 RTE, COPA and ReCoRD details
RTE is trained for 3240 steps, with checkpoints every 750 steps and a learning rate of 8e-6. The warm-up proportion is  $10\%$  and the a maximum sequence length of 512
COPA is trained for 300 steps, with checkpoints every 50 steps and a learning rate of 1e-5. The warm-up proportion is  $10\%$  and the maximum sequence length of 512.
ReCoRD is trained for 8 epochs over the training data with a learning rate of 2e-5, warm-up proportion of  $10\%$  and a maximum sequence length of 512.
Model	Parameters
BERT-Base	110M
RoBERTa-Base	110M
CONPONO [All Variants]	110M
BERT-Large	335M
Table 8
Model	Wiki	SP
arxiv	ROC	Wiki	BSO
arxiv	ROC	DC
Wiki	Ubuntu
BERT-Large	50.7	47.3	63.4	70.4	66.8	70.8	65.1	54.2
RoBERTa-Base	38.35	33.73	44.00	60.19	55.16	60.66	62.80	53.89
BERT-Base BSO	49.23	50.92	60.80	74.67	68.56	72.22	88.80	56.41
CONPONO - MLM	50.95	51.90	61.92	77.98	71.45	76.68	86.70	50.00
CONPONO Small	44.90	41.23	50.10	65.03	58.89	61.19	78.10	57.32
CONPONO isolated	49.33	44.60	56.53	59.16	57.48	56.94	71.60	54.71
CONPONO uni-encoder	54.30	58.58	66.75	78.25	71.65	73.99	86.00	57.90
k=4	54.07	58.30	67.15	79.04	72.21	76.89	88.38	58.85
k=3	54.65	59.55	67.22	79.34	73.61	77.08	89.48	56.00
k=2	54.83	58.77	68.40	79.24	74.16	76.84	89.22	56.41
k=1	44.05	40.98	57.65	68.47	62.40	67.24	89.03	56.20