SlowGuess/NAACL_2021_OCR huggingface dataset:
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797-1807, Brussels, Belgium. Association for Computational Linguistics.
Allen Nie, Erin Bennett, and Noah Goodman. 2019. DisSent: Learning sentence representations from explicit discourse relations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4497-4510, Florence, Italy. Association for Computational Linguistics.
Yannis Papanikolaou, Ian Roberts, and Andrea Pierleoni. 2019. Deep bidirectional transformers for relation extraction without supervision. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pages 67–75, Hong Kong, China. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Rishi Sharma, James Allen, Omid Bakhshandeh, and Nasrin Mostafazadeh. 2018. Tackling the story ending biases in the story cloze test. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 752-757, Melbourne, Australia. Association for Computational Linguistics.
Wei Shi and Vera Demberg. 2019. Next sentence prediction helps implicit discourse relation classification within and across domains. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5790-5796, Hong Kong, China. Association for Computational Linguistics.
Damien Sileo, Tim Van De Cruys, Camille Pradel, and Philippe Muller. 2019. Mining discourse markers for unsupervised sentence representation learning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3477-3486, Minneapolis, Minnesota. Association for Computational Linguistics.
Shiva Upadhye, Leon Bergen, and Andrew Kehler. 2020. Predicting reference: What do language models learn about discourse models? In Proceedings of
the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 977-982, Online. Association for Computational Linguistics.
Elena Voita, Rico Sennrich, and Ivan Titov. 2019. The bottom-up evolution of representations in the transformer: A study with machine translation and language modeling objectives. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4396-4406, Hong Kong, China. Association for Computational Linguistics.
Zining Zhu, Chuer Pan, Mohamed Abdalla, and Frank Rudzicz. 2020. Examining the rhetorical capacities of neural language models. In Proceedings of the Third Blackbox NLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 16-32, Online. Association for Computational Linguistics.
A Pretrained Language Models
The pretrained models are sourced from Huggingface (https://huggingface.co/), as detailed in Tables 3 and 4.
Model	Huggingface model
BERT	bert-base-uncased
BERT (large)	bert-large-uncased
RoBERTa	roberta-base
ALBERT	albert-base-v2
ELECTRA	electra-base-discriminator
GPT-2	gpt2
BART	bart-base
T5	t5-small
Table 3: List of English pretrained language models
Language	Huggingface model
Chinese	bert-base-chinese
German	bert-base-german-dbmdz-uncased
Spanish	bert-base-spanish-wwm-uncased
B Data Construction, Examples, and Training Configuration
B.1 Next Sentence Prediction
We use spaCy (https://spacy.io/) to perform sentence tokenization, and ensure that the distractor options in the training set do not overlap with the test set. For all languages and models, the training configurations are similar: the maximum tokens in the context and the next sentence are 450 and 50, respectively. If the token lengths are more than this, we truncate the context from the beginning of the sequence, and truncate the next sentence at the end of the sequence. We concatenate context with each option, and perform binary classification.
Other training configuration details: learning rate  $= 1\mathrm{e} - 3$ , Adam epsilon  $= 1\mathrm{e} - 8$ , maximum gradient norm  $= 1.0$ , maximum epochs  $= 20$ , warmup  $= 10\%$  of the training steps, and patience for early stopping  $= 5$  epochs.
B.2 Sentence Ordering
In generating sentence ordering data, we once again use spaCy (https://spacy.io/) to perform sentence tokenization. For all languages and models, the
Table 4: List of non-English BERT models.
#Sentence (context)	Total
2	2500
4	2500
6	2500
8	2500
Total	10000
Table 5: NSP data based on the number of sentences.
Context
s1: The Eastern Star, mostly carrying elderly tourists, capsized on 1 June near Jianli in Hubei province.
s2: Just 14 of the 456 passengers and crew are known to have survived.
Options
0: The channel recently said its signal was carried by 22 satellites
0: That step has become a huge challenge for opposition candidates
0: Six men were convicted and then acquitted of the atrocity and no-one has since been convicted of involvement in the bombing
1: A search is continuing for eight people who remain missing.
Table 6: Example of English NSP data with 2-sentence context. 1 indicates the correct next sentence.
#Sentence	Total
3	2000
4	2000
5	2000
6	2000
7	2000
Total	10000
Table 7: Sentence ordering data based on number of sentence.
training configurations are similar, with the maximum tokens in each sentence  $= 50$ , learning rate  $= 1\mathrm{e} - 3$ , Adam epsilon  $= 1\mathrm{e} - 8$ , maximum gradient norm  $= 1.0$ , training epochs  $= 20$ , warmup  $= 10\%$  of the training steps, and patience for early stopping  $= 10$  epochs.
B.3 Discourse Connective Prediction
As our Chinese and German data is extracted from discourse treebanks, the number of distinct connective words varies. For instance, in the Chinese discourse treebank, we find 246 unique connective
s0: West Mercia Police said the police do not encourage members of the public to pursue their own investigations.
s1: David John Poole, from Hereford, poses online as a 14-year-old girl and says he has been sent hundreds of explicit messages.
s2: He says his work has led to two arrests in four weeks.
Correct order: 2-0-1
Table 8: Example of English sentence ordering data
words. To simplify this, we set the connective word to OTHER if its word frequency is less than 12.
For all languages and models, the training configurations are: maximum token length of each sentence  $= 50$ , learning rate  $= 1\mathrm{e} - 3$ , Adam epsilon  $= 1\mathrm{e} - 8$ , maximum gradient norm  $= 1.0$ , maximum epochs  $= 20$ , warmup  $= 10\%$  of the training steps, and patience for early stopping  $= 10$  epochs.
B.4 RST-related Tasks
In Figures 7 and 8, we present the distribution of the nuclearity and relation labels for the 4 different discourse treebanks. The English treebank is significantly larger, with a strong preference for the NS (nuclear-satellite) relationship. Unlike other languages, the proportion of NN (nuclear-nuclear) relationships in the Chinese discourse treebank (CDTB) is the highest. We also notice that the relation label set in CDTB is the simplest, with only 4 labels.
Most of the training details for nuclearity and relation prediction are the same as for the NSP task, except we set the maximum token length of each sentence to 250. Particularly for EDU segmentation, we set the maximum token length in a document to 512.
B.5 Cloze Story Test
As discussed in Table 2, we use cloze story test version-1 (Mostafazadeh et al., 2016). Although version-2 (Sharma et al., 2018) is better in terms of story biases, the gold labels for the test set are not publicly available, which limited our ability to explore different layers of a broad range of pretrained language models (due to rate limiting of test evaluation).
For the data split, we followed previous work (Liu et al., 2018) in splitting the development set into a training and validation set. We perform bi
but: 2237, and: 2190, as: 1547, when: 1085, if: 993, before: 462, while: 358, because: 335, though: 229, after: 196, so: 180, although: 84, still: 38, then: 35, also: 31
Development:
but: 222, and: 192, as: 181, when: 116, if: 104, because: 44, before: 40, while: 28, though: 28, so: 16, after: 16, also: 5, although: 5, then: 2, still: 1
and: 211, but: 202, as: 153, when: 129, if: 97, before: 48, while: 44, because: 41, after: 20, though: 19, although: 11, so: 9, still: 7, also: 5, then: 4
other: 520, 并: 182, 其中: 131, 也: 118, 但: 60, 而: 60, 还: 55, 以: 47, 使: 43, 后: 42, 为: 41, 同时: 37, 由于: 34, 因此: 28, 如: 26, 又: 20, 为了: 19, 如果: 17, 而且: 16, 但是: 15, 因为: 15, 虽然-但: 13
other: 22, 并: 7, 也: 6, 而: 6, 其中: 4, 但: 4, 因为: 4, 为: 3, 还: 3, 而且: 3, 又: 2, 如果: 2, 同时: 2, 使: 2, 后: 2, 如: 1, 由于: 1, 虽然-但: 1, 为了: 1
other: 60, 其中: 18, 并: 18, 也: 10, 使: 10, 还: 9, 同时: 8, 而: 6, 以: 5, 但: 5, 为: 4, 又: 3, 因为: 2, 虽然-但: 2, 由于: 2, 为了: 2, 因此: 2, 而且: 1, 如: 1
other: 336, und: 191, doch: 62, wenn: 56, aber: 56, Denn: 36, dann: 23, auch: 23, sondern: 19, oder: 19, so: 18, also: 17, deshalb: 16, weil: 15, als: 13
other: 50, und: 32, doch: 12, wenn: 11, aber: 9, Denn: 6, dann: 5, so: 5, auch: 5, oder: 4, deshalb: 3, weil: 3, sondern: 2, als: 1
other: 69, und: 23, doch: 11, aber: 10, Denn: 9, wenn: 8, dann: 8, so: 4, weil: 4, auch: 4, sondern: 3, deshalb: 3, oder: 2, als: 1
Figure 5: Discourse connective word distribution.
nary classification similar to the NSP task, by first merging all 4-sentence stories into a single text (context). We limit the context to a maximum of 450 tokens, and each candidate sentence (as the story ending) is limited to 50 tokens. Other training details are the same as for the NSP task.
ENGLISH
S1: Two men nudged the door open.
S2: Slipped into the room with him.
Connective word: and
CHINESE
S1: 目前，约有十五万家外商投资企业在中国银行开立帐户，
S2: 二万多家获得中国银行的贷款支持。
Connective word: 其中
GERMAN
S1: der mann bezahlte viele handwerker nicht
S2: wurde voriges Jahr zu einer mehrjährigen haftstraßeverurteilt
Connective word: und
Figure 6: Discourse connective: data examples
ENGLISH
Train: NS: 10348, NN: 3853, SN: 2702
Development: NS: 1195, NN: 407, SN: 341
Test: NS: 1373, NN: 507, SN: 428
CHINESE
Train: NN: 3133, NS: 1784, SN: 1242
Development: NN: 432, NS: 219, SN: 158
Test: NN: 188, NS: 107, SN: 58
GERMAN
Train: SN: 752, NS: 733, NN: 407
Development: NS: 116, SN: 106, NN: 67
Test: NS: 150, SN: 133, NN: 72
SPANISH
Train: NS: 1011, SN: 570, NN: 461
Development: NS: 163, SN: 73, NN: 71
Test: NS: 211, SN: 121, NN: 89
Figure 7: Nuclearity label distribution.
ENGLISH
elab: 7830, attr: 3041, list: 1957, same: 1390, cont: 1108, evid: 967, back: 931, cause: 685, eval: 588, purp: 560, temp: 526, cond: 326, comp: 299, mann: 225, summ: 222, topic: 204, prob: 153, text: 142
CHINESE
并列类: 4144,解说类: 1630,因果类: 1333,转折类: 214
GERMAN
reason: 267, interpretation: 232, elaboration: 204, joint: 203, background: 163, list: 138, concession: 125, antithesis: 123, conjunction: 117, condition: 116, circumstance: 113, e-elaboration: 111, cause: 101, evidence: 99, preparation: 87, evaluation-s: 80, contrast: 49, result: 46, evaluation-n: 38, purpose: 30, sequence: 29, restatement: 17, means: 11, disjunction: 10, summary: 9, solutionhood: 7, justify: 4, otherwise: 3, enablement: 2, unless: 1, motivation: 1
SPANISH
elaboración: 625, preparación: 370, lista: 257, fondo: 178, unión: 168, medio: 135, resultado: 134, circunstancia: 122, propósito: 115, secuencia: 79, interpretación: 77, antítesis: 67, contraite: 61, causa: 57, evidencia: 49, condición: 47, concesión: 44, justificación: 39, same-unit: 33, soluciones: 26, motivación: 21, reformulación: 16, conjunción: 14, disyúncción: 9, evaluación: 9, resumen: 8, capacitatación: 5, alternatively: 3, unless: 2
Figure 8: Relation label distribution.
C Full Experimental Results
Layer	NSP	Sent. Ord.	Discourse Conn.	Nuclearity	Relation	EDU segment.	Cloze ST.
BERT (English); std = 0.00 – 0.02
1	0.36	0.28	0.28	0.62	0.38	0.30	0.58
2	0.45	0.28	0.31	0.64	0.41	0.42	0.61
3	0.79	0.28	0.30	0.62	0.40	0.49	0.63
4	0.95	0.32	0.35	0.65	0.44	0.52	0.60
5	0.97	0.31	0.44	0.75	0.54	0.55	0.66
6	0.97	0.35	0.53	0.78	0.60	0.56	0.72
7	0.96	0.33	0.57	0.80	0.65	0.56	0.72
8	0.96	0.32	0.57	0.81	0.65	0.54	0.73
9	0.96	0.34	0.59	0.80	0.65	0.52	0.72
10	0.97	0.33	0.58	0.80	0.64	0.47	0.75
11	0.97	0.31	0.59	0.79	0.63	0.44	0.76
12	0.99	0.32	0.56	0.76	0.59	0.39	0.76
RoBERTa (English); std = 0.00 – 0.02
1	0.78	0.29	0.46	0.72	0.55	0.68	0.72
2	0.86	0.31	0.48	0.73	0.56	0.92	0.73
3	0.88	0.30	0.49	0.75	0.58	0.90	0.74
4	0.95	0.34	0.51	0.77	0.59	0.88	0.75
5	0.96	0.37	0.51	0.79	0.60	0.91	0.78
6	0.96	0.37	0.52	0.79	0.61	0.86	0.78
7	0.96	0.39	0.52	0.78	0.61	0.85	0.83
8	0.95	0.37	0.54	0.78	0.61	0.87	0.86
9	0.94	0.37	0.54	0.79	0.61	0.87	0.86
10	0.94	0.36	0.54	0.78	0.61	0.88	0.86
11	0.93	0.35	0.53	0.77	0.59	0.87	0.85
12	0.90	0.31	0.48	0.75	0.56	0.73	0.82
ALBERT (English); std = 0.00 – 0.03
1	0.34	0.29	0.29	0.63	0.40	0.47	0.56
2	0.85	0.33	0.33	0.63	0.44	0.54	0.66
3	0.91	0.30	0.32	0.64	0.45	0.53	0.68
4	0.93	0.30	0.35	0.67	0.46	0.51	0.69
5	0.96	0.30	0.35	0.67	0.47	0.47	0.70
6	0.97	0.29	0.37	0.67	0.48	0.44	0.71
7	0.97	0.29	0.40	0.68	0.48	0.40	0.73
8	0.98	0.26	0.40	0.68	0.49	0.34	0.73
9	0.97	0.23	0.40	0.68	0.49	0.32	0.75
10	0.98	0.21	0.41	0.70	0.49	0.25	0.76
11	0.98	0.17	0.43	0.73	0.52	0.18	0.77
12	0.99	0.13	0.53	0.79	0.63	0.11	0.85
ELECTRA (English); std = 0.00 – 0.02
1	0.86	0.27	0.47	0.72	0.54	0.42	0.72
2	0.90	0.31	0.48	0.72	0.55	0.47	0.74
3	0.90	0.31	0.49	0.73	0.55	0.45	0.74
4	0.94	0.31	0.51	0.74	0.57	0.50	0.75
5	0.96	0.35	0.52	0.77	0.59	0.54	0.76
6	0.96	0.36	0.53	0.78	0.60	0.57	0.78
7	0.96	0.37	0.53	0.79	0.61	0.54	0.78
8	0.97	0.39	0.55	0.80	0.63	0.51	0.82
9	0.97	0.41	0.56	0.80	0.63	0.48	0.86
10	0.97	0.43	0.58	0.80	0.63	0.49	0.89
11	0.97	0.42	0.57	0.80	0.64	0.52	0.89
12	0.96	0.40	0.57	0.79	0.60	0.48	0.88
Table 9: Full results for BERT, RoBERTa, ALBERT, and ELECTRA over English.
Layer	NSP	Sent. Ord.	Discourse Conn.	Nuclearity	Relation	EDU segment.	Cloze ST.
GPT-2 (English); std = 0.00 – 0.02
1	0.86	0.26	0.47	0.72	0.55	0.35	0.73
2	0.87	0.26	0.48	0.73	0.56	0.37	0.73
3	0.88	0.28	0.48	0.73	0.56	0.40	0.74
4	0.90	0.30	0.51	0.75	0.57	0.40	0.76
5	0.91	0.32	0.51	0.75	0.57	0.41	0.75
6	0.93	0.33	0.52	0.77	0.59	0.42	0.76
7	0.93	0.33	0.52	0.76	0.60	0.42	0.77
8	0.92	0.34	0.51	0.77	0.59	0.41	0.77
9	0.92	0.33	0.50	0.76	0.59	0.41	0.77
10	0.91	0.31	0.49	0.75	0.58	0.41	0.77
11	0.91	0.30	0.49	0.74	0.57	0.42	0.75
12	0.85	0.28	0.47	0.73	0.55	0.38	0.72
BART (English); Layers 7–12 are the decoder; std = 0.00 – 0.01.
1	0.86	0.30	0.48	0.73	0.55	0.79	0.73
2	0.92	0.34	0.49	0.76	0.58	0.88	0.76
3	0.95	0.35	0.51	0.76	0.58	0.89	0.76
4	0.96	0.38	0.52	0.78	0.60	0.86	0.78
5	0.97	0.39	0.53	0.78	0.62	0.82	0.79
6	0.96	0.41	0.52	0.80	0.62	0.62	0.78
7	0.94	0.32	0.51	0.77	0.59	0.10	0.76
8	0.95	0.39	0.54	0.79	0.61	0.23	0.77
9	0.95	0.40	0.54	0.79	0.62	0.32	0.78
10	0.95	0.40	0.54	0.80	0.62	0.31	0.81
11	0.96	0.38	0.52	0.78	0.60	0.34	0.80
12	0.95	0.36	0.52	0.77	0.59	0.47	0.82
T5 (English); Layers 7–12 are the decoder; std = 0.00 – 0.03.
1	0.77	0.27	0.39	0.71	0.50	0.26	0.71
2	0.80	0.30	0.43	0.74	0.54	0.38	0.70
3	0.82	0.32	0.45	0.75	0.55	0.40	0.73
4	0.84	0.33	0.46	0.76	0.57	0.37	0.74
5	0.87	0.33	0.45	0.76	0.57	0.33	0.72
6	0.86	0.35	0.46	0.76	0.58	0.28	0.72
7	0.77	0.28	0.41	0.73	0.54	0.24	0.71
8	0.77	0.26	0.44	0.74	0.55	0.27	0.71
9	0.77	0.24	0.46	0.75	0.56	0.27	0.71
10	0.74	0.22	0.45	0.75	0.55	0.20	0.72
11	0.70	0.22	0.44	0.73	0.54	0.11	0.72
12	0.68	0.20	0.42	0.73	0.52	0.00	0.72
Table 10: Full results for GPT-2, BART, and T5 over English.
Layer	NSP	Sent. Ord.	Discourse Conn.	Nuclearity	Relation	EDU segment.
Chinese; std = 0.00 – 0.02.
1	0.30	0.38	0.36	0.53	0.58	0.70
2	0.47	0.38	0.36	0.53	0.58	0.75
3	0.47	0.42	0.36	0.53	0.58	0.79
4	0.83	0.42	0.36	0.53	0.58	0.79
5	0.90	0.44	0.35	0.59	0.60	0.81
6	0.93	0.44	0.39	0.60	0.64	0.83
7	0.94	0.45	0.42	0.64	0.67	0.83
8	0.94	0.44	0.43	0.64	0.66	0.83
9	0.94	0.43	0.46	0.63	0.68	0.83
10	0.96	0.43	0.44	0.64	0.68	0.83
11	0.96	0.42	0.46	0.61	0.66	0.81
12	0.98	0.40	0.44	0.62	0.66	0.78
German; std = 0.00 – 0.07.
1	0.43	0.58	0.44	0.48	0.16	0.55
2	0.60	0.58	0.43	0.49	0.18	0.56
3	0.77	0.59	0.43	0.47	0.17	0.67
4	0.76	0.61	0.43	0.44	0.21	0.71
5	0.98	0.63	0.38	0.48	0.26	0.73
6	0.99	0.65	0.43	0.54	0.29	0.74
7	1.00	0.65	0.43	0.58	0.31	0.76
8	0.99	0.64	0.43	0.60	0.35	0.75
9	1.00	0.64	0.44	0.59	0.33	0.69
10	0.99	0.64	0.45	0.58	0.35	0.65
11	1.00	0.63	0.43	0.58	0.33	0.58
12	1.00	0.63	0.38	0.58	0.33	0.59
Spanish; std = 0.00 – 0.02.
1	0.39	0.49	—	0.50	0.29	0.43
2	0.55	0.52	—	0.56	0.31	0.50
3	0.56	0.53	—	0.58	0.31	0.52
4	0.96	0.55	—	0.62	0.37	0.57
5	0.98	0.56	—	0.64	0.41	0.59
6	0.99	0.56	—	0.68	0.45	0.62
7	1.00	0.57	—	0.68	0.47	0.64
8	1.00	0.58	—	0.75	0.49	0.69
9	1.00	0.58	—	0.74	0.51	0.66
10	1.00	0.58	—	0.77	0.56	0.62
11	1.00	0.57	—	0.77	0.55	0.59
12	1.00	0.56	—	0.76	0.54	0.50
Table 11: Full results for the BERT monolingual models over Chinese, German, and Spanish.
Layer	NSP	Sent. Ord.	Discourse Conn.	Nuclearity	Relation	EDU segment.	Cloze ST.
BERT-Large (English); std = 0.00 – 0.02.
1	0.34	0.26	0.29	0.60	0.37	0.23	0.61
2	0.53	0.25	0.33	0.62	0.40	0.28	0.67
3	0.57	0.26	0.32	0.63	0.42	0.29	0.66
4	0.60	0.29	0.32	0.63	0.42	0.40	0.66
5	0.64	0.30	0.35	0.65	0.43	0.43	0.69
6	0.82	0.31	0.37	0.66	0.44	0.45	0.68
7	0.87	0.31	0.39	0.66	0.45	0.44	0.69
8	0.95	0.30	0.39	0.66	0.45	0.44	0.72
9	0.96	0.32	0.40	0.67	0.47	0.43	0.70
10	0.96	0.32	0.40	0.67	0.46	0.49	0.71
11	0.97	0.32	0.39	0.68	0.47	0.53	0.70
12	0.97	0.33	0.47	0.77	0.57	0.58	0.71
13	0.97	0.34	0.52	0.78	0.61	0.56	0.71
14	0.97	0.34	0.57	0.81	0.65	0.59	0.73
15	0.97	0.34	0.61	0.82	0.67	0.58	0.75
16	0.97	0.35	0.60	0.83	0.67	0.55	0.75
17	0.97	0.34	0.62	0.82	0.68	0.53	0.82
18	0.98	0.36	0.63	0.82	0.68	0.54	0.82
19	0.99	0.37	0.63	0.82	0.67	0.50	0.83
20	0.99	0.34	0.63	0.81	0.67	0.48	0.84
21	0.99	0.35	0.63	0.81	0.65	0.41	0.83
22	0.99	0.35	0.61	0.81	0.65	0.37	0.84
23	0.99	0.34	0.59	0.80	0.63	0.36	0.82
24	0.99	0.33	0.57	0.77	0.58	0.31	0.81
Table 12: Full results of English BERT-large.
D Frozen vs. Fine-tuned BERT Layers
Figure 9: A comparison of BERT with frozen vs. fine-tuned layers.
E Full Results of Models with Average Pooling
Figure 10: Full results of all models over English with average pooling on all tasks except in EDU segmentation (with the only differences over Figure 2 being for BERT and ALBERT, where we originally used [CLS] embeddings on two-text classification probing tasks).
F [CLS] vs. Average Pooling in English BERT-base Model
Average pooling generally performs worse than [CLS] embeddings in the last layers of BERT.
Figure 11: Comparison of [CLS] vs. average pooling embeddings for BERT-base across the five tasks for English. Please note that sentence ordering and EDU segmentation are always performed with average pooling embeddings and sequence labelling at the (sub)word level, respectively.