SlowGuess/ACL_2020_OCR huggingface dataset:
pages 977-986, Baltimore, Maryland. Association for Computational Linguistics.
Yangfeng Ji and Jacob Eisenstein. 2015. One vector is not enough: Entity-augmented distributed semantics for discourse relations. Transactions of the Association for Computational Linguistics, 3:329-344.
Man Lan, Jianxiang Wang, Yuanbin Wu, Zheng-Yu Niu, and Haifeng Wang. 2017. Multi-task attention-based neural networks for implicit discourse relationship representation and identification. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1299-1308, Copenhagen, Denmark. Association for Computational Linguistics.
Wenqiang Lei, Yuanxin Xiang, Yuwei Wang, Qian Zhong, Meichun Liu, and Min-Yen Kan. 2018. Linguistic properties matter for implicit discourse relation recognition: Combining semantic interaction, topic continuity and attribution. In Thirty-Second AAAI Conference on Artificial Intelligence.
Junyi Jessy Li and Ani Nenkova. 2014. Reducing sparsity improves the recognition of implicit discourse relations. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 199-207, Philadelphia, PA, U.S.A. Association for Computational Linguistics.
Ziheng Lin, Min-Yen Kan, and Hwee Tou Ng. 2009. Recognizing implicit discourse relations in the Penn Discourse Treebank. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 343-351, Singapore. Association for Computational Linguistics.
Yang Liu and Sujian Li. 2016. Recognizing implicit discourse relations via repeated reading: Neural networks with multi-level attention. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1224-1233, Austin, Texas. Association for Computational Linguistics.
Annie Louis, Aravind Joshi, Rashmi Prasad, and Ani Nenkova. 2010. Using entity features to classify implicit discourse relations. In Proceedings of the SIGDIAL 2010 Conference, pages 59-62, Tokyo, Japan. Association for Computational Linguistics.
Linh The Nguyen, Linh Van Ngo, Khoat Than, and Thien Huu Nguyen. 2019. Employing the correspondence of relations and connectives to identify implicit discourse relations via label embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4201-4207, Florence, Italy. Association for Computational Linguistics.
Allen Nie, Erin Bennett, and Noah Goodman. 2019. DisSent: Learning sentence representations from explicit discourse relations. In Proceedings of the
57th Annual Meeting of the Association for Computational Linguistics, pages 4497-4510, Florence, Italy. Association for Computational Linguistics.
Joonsuk Park and Claire Cardie. 2012. Improving implicit discourse relation recognition through feature set optimization. In Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 108-112, Seoul, South Korea. Association for Computational Linguistics.
Gary Patterson and Andrew Kehler. 2013. Predicting the presence of discourse connectives. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 914-923, Seattle, Washington, USA. Association for Computational Linguistics.
Emily Pitler, Annie Louis, and Ani Nenkova. 2009. Automatic sense prediction for implicit discourse relations in text. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 683-691, Suntec, Singapore. Association for Computational Linguistics.
Emily Pittler, Mridhula Raghupathy, Hena Mehta, Ani Nenkova, Alan Lee, and Aravind Joshi. 2008. Easily identifiable discourse relations. In *COLING 2008: Companion volume: Posters*, pages 87–90, Manchester, UK. COLING 2008 Organizing Committee.
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics (*SEM), pages 180-191, New Orleans, Louisiana. Association for Computational Linguistics.
Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn Discourse TreeBank 2.0. In LREC 2008.
Rashmi Prasad, Bonnie Webber, Alan Lee, and Aravind Joshi. 2019. Penn Discourse Treebank Version 3.0. In LDC2019T05. Philadelphia: Linguistic Data Consortium.
Lianhui Qin, Zhisong Zhang, Hai Zhao, Zhiting Hu, and Eric Xing. 2017. Adversarial connective-exploiting networks for implicit discourse relation classification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1006-1017, Vancouver, Canada. Association for Computational Linguistics.
Attapol Rutherford, Vera Demberg, and Nianwen Xue. 2017. A systematic study of neural discourse models for implicit discourse relation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume
1, Long Papers, pages 281-291, Valencia, Spain. Association for Computational Linguistics.
Marzieh Saeidi, Max Bartolo, Patrick Lewis, Sameer Singh, Tim Roktaschel, Mike Sheldon, Guillaume Bouchard, and Sebastian Riedel. 2018. Interpretation of natural language rules in conversational machine reading. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2087-2097, Brussels, Belgium. Association for Computational Linguistics.
Wei Shi and Vera Demberg. 2017. Do we need cross validation for discourse relation classification? In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 150-156, Valencia, Spain. Association for Computational Linguistics.
Wei Shi and Vera Demberg. 2019. Learning to explicitate connectives with Seq2Seq network for implicit discourse relation classification. In Proceedings of the 13th International Conference on Computational Semantics - Long Papers, pages 188-199, Gothenburg, Sweden. Association for Computational Linguistics.
Wen Ting Wang, Jian Su, and Chew Lim Tan. 2010. Kernel based discourse relation recognition with temporal ordering information. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 710-719, Uppsala, Sweden. Association for Computational Linguistics.
Nianwen Xue, Hwee Tou Ng, Sameer Pradhan, Rashmi Prasad, Christopher Bryant, and Attapol Rutherford. 2015. The CoNLL-2015 shared task on shallow discourse parsing. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning - Shared Task, pages 1-16, Beijing, China. Association for Computational Linguistics.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 5753-5763. Curran Associates, Inc.
A Dataset Statistics
We report the training, development and test set sizes for all dataset splits discussed in the paper (Table 6). These are counts of individual labeled span pairs in the dataset, not the counts of individual labels (development and test set examples can be doubly-annotated). Note that the count we provide
for the training split of Ji is one short of what has been reported in Shi and Demberg (2019) and also the count obtained by using Qin et al. (2017)'s preprocessing code. This is due to a duplicate example with label EXPANSION.ALTERNATIVE, which our preprocessing code does not generate.
Split	Train	Dev	Test
PDTB 2.0
Ji	12825	1165	1039
Lin	13366	515	766
P&K	13908	1165	1188
X-val	13676	1281	1273
L1 (Ji)	13046	1183	1046
PDTB 3.0
L2 X-val	19005	1756	1747
L2+L3 X-val	19005	1756	1747
L1 (Ji)	17854	1647	1471
Table 6: Dataset sizes for PDTB 2.0 and 3.0. Cross-validation counts are averaged across 12 folds.
Tables 7 and 8 list the label counts of each class in PDTB 3.0 and PDTB 2.0, respectively.
Label	n
Comparison	2298/2518
Contingency	6998/7583
Expansion	10062/10833
Temporal	1731/1828
Comparison.Concession	1494
Comparison.Contrast	983
Contingency.Cause	5785
Contingency.Cause+Belief	202
Contingency Condition	199
Contingency.Purpose	1373
Expansion.Conjunction	4386
Expansion.Equivalence	336
ExpansionInstantiation	1533
Expansion.Level-of-detail	3361
Expansion.Manner	739
Expansion.Substitution	450
Temporal.Asynchronous	1289
Temporal.Synchronous	539
Contingency.Cause.Result	2835
Contingency.CauseReason	2950
Expansion.Level-of-detail.Arg1-as-detail	256
Expansion.Level-of-detail.Arg2-as-detail	3105
Expansion.Manner.Arg1-as-manner	572
Expansion.Manner.Arg2-as-manner	167
Temporal.Asynchronous.Precedence	1081
Temporal.Asynchronous.Succession	208
Table 7: Label counts for PDTB 3.0 L1, L2 and directional senses of L3 that have more than 100 annotated instances. L1 classification is evaluated on Ji split, so we list both the label counts in Ji split and the total label counts in the whole dataset.
Label	n
Comparison	2291/2503
Contingency	3911/4255
Expansion	8249/8861
Temporal	909/950
Comparison.Concession	223
Comparison.Contrast	2120
Contingency.Cause	4172
Contingency.Pragmatic cause	83
Expansion.Conjunction	3534
ExpansionInstantiation	1445
Expansion.Alternative	185
Expansion.List	400
Expansion.Restatement	3206
Temporal.Asynchronous	697
Temporal.Synchrony	251
Table 8: Label counts for PDTB 2.0 L1 and 11 senses of L2 (label set commonly used in the literature for L2 classification). L1 classification is evaluated on Ji split, so we list both the label counts in Ji split and the total label counts in the whole dataset.
B List of Splits in Prior Work
We compile a (non-exhaustive) list of the Wall Street Journal sections used as training, development, test sets in published work to demonstrate the high variability. We mostly list works that do not explicitly specify the source of the splits, with some exceptions. Some of the works have overlapping sections across splits, which we suspect to be typos but cannot verify.
- Prasad et al. (2008) (officially recommended split): 2-21 (train), 22 (dev), 23 (test)
- Pittler et al. (2009); Ji and Eisenstein (2015): 2-20 (train), 0-1 (dev), 21-22 (test)
- Lin et al. (2009): 2-21 (train), 23 (test)
- Patterson and Kehler (2013): 2-22 (train), 0-1 (dev), 23-24 (test)
- Wang et al. (2010): 2-22 (train), 23-24 (test)
- Louis et al. (2010): 0-22 (train), 23-24 (test)
- Braud and Denis (2015): 2-21 (train), 0-1, 23-24 (dev), 21-22 (test)
- Li and Nenkova (2014): 2-19 (train), 20-24 (test)
Lei et al. (2018): 2-20 (train), 0-1, 23-24 (dev), 21-22 (test)
- Park and Cardie (2012): 2-20 (train), 0-2 (dev), 21-22 (test)
For all sentence encoder models, we fine-tuned each encoder for a maximum of 10 epochs with early stopping when the development set performance did not improve for 5 evaluation steps (step size=500), with a batch size of 8. We used a learning rate of  $5e - 6$  for all models except for XLNet-large, for which we used  $2e - 6$ . We used accuracy as the validation metric. We ran each model 5 times with different random initializations of the fine-tuning layer, and reported the average performance across the 5 runs.
D Top-level Sense Classification Results
Table 9 shows the performance on L1 classification for both PDTB 2.0 and PDTB 3.0.
Model	PDTB 2.0	PDTB 3.0
F1	Acc	F1	Acc
Majority class	17.4	54.9	15.2	47.3
Lan et al. (2017)	47.8	57.4	-	-
Dai and Huang (2018)	48.7	58.2	-	-
Bai and Zhao (2018)	51.1	-	-	-
Bai et al. (2019)	52.2	60.7	-	-
Nguyen et al. (2019)	53.0	-	-	-
BERT (base, uncased)	52.6	64.3	62.1	69.0
BERT (large, uncased)	59.1	68.7	66.8	72.4
XLNet (base, cased)	56.0	66.3	64.8	71.3
XLNet (large, cased)	54.3	67.2	68.3	73.8
Table 9: Accuracy and F1 on L1 classification (4-way) for PDTB 2.0 and 3.0, using Ji split for both. We report average performance across 5 random restarts.
E Single-span Baselines for L2 Classification
Table 10 lists the performance of single-span (either ARG1 or ARG2) baselines for PDTB 2.0. Results on PDTB 3.0 are reported in Table 4.
We additionally note that ARG2-only models consistently outperform ARG1-only models in both PDTB 2.0 and 3.0. For PDTB 3.0, the strong association between ARG2-initial to and CONTINENCE.PURPOSE was largely responsible for this discrepancy (see Section 4 also).
F Cross-validation and Randomized validation
Gorman and Bedrick (2019) have proposed validation over randomized splits using significance testing with multiple-comparisons correction. An
adaptation of this idea to our proposal of section-based evaluation would be randomized sampling of sections to create section-based splits. Given label sparsity and distributional skew across sections, cross-validation has an advantage of guaranteed coverage of label counts used for testing, although this may not be a large issue if sufficient number of random splits are sampled. Conversely, the main goal of evaluation on random splits—avoiding overfitting to the standard split—is partially mitigated by reporting the average performance over cross-validation splits. Still, if a standard cross-validation split is adopted, overfitting may still arise over time. Although we leave it to future work to decide which practice should be followed, we provide comparisons between the four models we tested, using our proposed cross-validation splits and random validation splits (both  $n = 12$ ). Random splitting was done section-wise instead of instance-wise; we randomly split the dataset into 21 train, 2 dev, 2 test sections 12 times. Table 11 shows the model comparison results.
G Additional Error Analyses
Figure 2 shows the confusion matrices generated from PDTB 2.0 L2 classification results produced by XLNet-large and BERT-large models. Figure 3 shows the confusion matrices of PDTB 3.0 L2 classification predictions, again from XLNet-large and BERT-large models (we did not observe immediate qualitative differences between XLNet and BERT, or between large and base models).
The figures aggregate the predictions from all test sets of the cross-validation experiment, so the datapoints shown span the full dataset except for WSJ section 22. The colors are normalized over each row; the darkest shade is the most frequently predicted label for the true label denoted by the row.
It was generally the case for both models that classes sharing the same L1 senses (e.g., CONTINGENCY.CAUSE and CONTINGENCY.PRAGMATIC CAUSE, or COMPARISON.CONTRAST and COMPARISON.CONCESSION) were confused. When such confusions occurred, the more frequent class often subsumed the prediction of the other class (e.g., CONTINGENCY.PRAGMATIC CAUSE was often classified as CONTINGENCY.CAUSE but not vice versa).
As noted in Section 4, TEMPO-RAL.SYNCHRONOUS (SYNCHRONY in PDTB
Model	Accuracy	X-Accuracy
Ji	Lin	P&K
Majority class	26.18	26.11	28.54	26.42
Adversarial Net (Qin et al., 2017)	46.23	44.65	-	-
Seq2Seq+MemNet (Shi and Demberg, 2019)	47.83	45.82	-	41.29
ELMo (Bai and Zhao, 2018)	48.22	45.73	-	-
ELMo, Memory augmented (Bai et al., 2019)	49.15	46.08	-	-
Multitask learning (Nguyen et al., 2019)	49.95	46.48	-	-
BERT+MNLI (Nie et al., 2019)	-	-	53.7	-
BERT+DisSent Books 5 (Nie et al., 2019)	-	-	54.7	-
BERT (base, uncased), ARG1-only	38.59 (±0.67)	36.11 (±1.01)	35.86 (±1.43)	36.66 (±1.26)
BERT (large, uncased), ARG1-only	39.31 (±0.70)	36.42 (±0.21)	37.71 (±1.42)	37.23 (±1.22)
XLNet (base, cased), ARG1-only	39.48 (±1.10)	35.40 (±1.06)	35.71 (±1.32)	37.38 (±1.76)
XLNet (large, cased), ARG1-only	39.77 (±1.58)	35.61 (±1.48)	36.20 (±1.77)	36.33 (±2.04)
BERT (base, uncased), ARG2-only	40.99 (±1.34)	40.99 (±1.34)	40.98 (±1.12)	40.60 (±1.48)
BERT (large, uncased), ARG2-only	44.27 (±1.00)	40.78 (±1.33)	42.34 (±1.21)	41.45 (±1.64)
XLNet (base, cased), ARG2-only	43.20 (±1.48)	40.84 (±0.99)	40.45 (±1.22)	40.46 (±1.45)
XLNet (large, cased), ARG2-only	42.00 (±1.24)	41.78 (±1.00)	41.48 (±1.14)	41.17 (±1.48)
Table 10: Single-span baseline performance on PDTB 2.0 L2 classification (11-way). All results are averages over 5 random restarts, except for cross-validation where we report averages over 12 folds.
X-validation	Randomized
BERT-base vs BERT-large	8	9
BERT-base vs XLNet-base	8	6
BERT-base vs XLNet-large	12	12
BERT-large vs XLNet-large	6	7
XLNet-base vs BERT-large	0	1
XLNet-base vs XLNet-large	6	10
Table 11: The number of splits out of twelve for which the second model had significantly higher accuracy than the first model after Bonferroni correction. We used McNemar's test following Gorman and Bedrick (2019).
2.0) was frequently confused with EXPANSION.CONJUNCTION (but not vice versa). The models generally had a tendency to predict CONTINGENCY.CAUSE across the board, likely due to it being the most frequent label.
Figure 2: Confusion matrices of XLNet-large and BERT-large models on PDTB 2.0 L2 classification task. The rows are true labels and the columns are predicted labels.
Figure 3: Confusion matrices of XLNet-large and BERT-large models on PDTB 3.0 L2 classification task. The rows are true labels and the columns are predicted labels.