SlowGuess/ACL_2020_OCR huggingface dataset:
Type	Model	BA/PA
UNSUP	BERT	36.9/3.5
TR.XL	32.3/2.6
FT	BERT	71.7/29.9
SUP	PMI2	29.8/8.4
COHERE	23.3/1.1
INFST	55.8/18.4
HUMAN	-	87.1/56.3
Table 5: Test BA/PA of various model types with EXH decoding and AP+AN context.
understanding the relationship between two sentences, which is highly correlated with our task. Therefore, we use the pre-trained BERT-Large-uncasedd with its NSP layer to predict the most appropriate candidate for each blank given its context. Specifically, BERT is employed to predict the probability of the context and the candidate being consecutive.
Finetuning BERT A wide range of NLP tasks have greatly benefited from the pre-trained BERT model. Therefore, we also finetune the pre-trained BERT-Large model on our task through sequence pair classification schema. Specifically, for each blank, its correct candidate will be labelled as 0 and the label of all other wrong candidates is 1. Batch size and number of epochs for all models are 32 and 3. We employ Adam (Kingma and Ba, 2014) as the optimizer with three different learning rates  $\{1e^{-5}, 2e^{-5}, 3e^{-5}\}$ . Best model selection is based on validation performance. All BERT finetuning experiments including ablation study follow this training strategy.
5.1 Decoding Strategy
The decoding strategy decides how exactly we assign a candidate to each blank in the passage. Due to shared candidates, we have two strategies:
1. INC: Answering each blank from left to right in order. Once a blank is answered with a candidate, this candidate is unavailable for later blanks.
2. EXH: Exhaustively scoring all permutations of candidates to answer the blanks. The score of a permutation is simply the sum of each its
5673
Type	Model	P	N	AP	AN	P+N	AP+AN
UNSUP	BERT+INC	33.0/2.1	34.7/4.1	29.8/2.1	15.7/0.3	34.7/2.3	27.3/1.4
+EXH	34.2/3.2	40.2/4.7	31.5/2.6	14.7/0.0	40.2/4.7	36.9/3.5
FT	BERT+INC	44.3/6.8	48.0/9.6	50.4/9.9	56.9/16.1	61.0/20.4	66.6/25.1
+EXH	47.2/8.5	54.2/11.2	60.0/17.5	60.0/17.5	66.5/25.2	71.7/29.9
SUP	\(\overline{\mathsf{PMI}}_2+\mathsf{INC}\)	23.4/1.2	24.4/1.5	16.2/0.3	17.5/0.1	26.2/1.7	17.1/0.0
+EXH	24.7/1.5	28.2/1.5	20.6/0.9	13.3/0.0	29.7/2.6	25.2/0.6
Table 6: Test BA/PA of various model types unsupervised (UNSUP), finetuned (FT) and supervised (SUP) across varying context levels, with INC or EXH decoding.
BERT-Un	TR.XL	BERT-ft
RemoveDt	47.4/17.2	39.7/9.1	80.9/62.0
RandomDt	44.6/12.4	36.0/6.8	77.9/50.9
Unablated	40.2/4.7	32.3/2.6	71.7/29.9
Table 7: Test BA/PA with distractor ablations on test set. RemoveDt and RandomDt represent removing and sampling distractors respectively. BERT-Un and BERT-ft denotes pre-trained and finetuned BERT.
constituent blank-candidate pairs. The highest scoring permutation is the answer.
We design two metrics to evaluate models. Both of these metrics are reported as percentage.
Blank accuracy (BA): The fraction of blanks answered correctly, averaged over all passages.
Passage Accuracy (PA): PA is 1 iff the model gets all blanks in a passage correct, and 0 otherwise. The average of PA over all passages is reported.
5.3 Human Performance
We hire annotators from AMT to both answer and label difficulty for 144 randomly chosen test examples. Annotators are restricted to be from USA/UK and have Master designation on  $\mathrm{AMT^8}$  along with  $>90\%$  HIT approval rate. On average, each annotator spends 624 seconds to answer one example. Difficulty level is chosen from {VeryHard, Hard, Moderate, Easy, VeryEasy}.  $3.5\%$  of annotators find the task VeryHard, while  $8.3\%$  find it VeryEasy. The largest fraction of  $38.2\%$  find it to be Moderate. We note that SCDE contains a larger proportion of non-easy
questions  $(61.0\%)$ . Human performance is reported in Table 5. Annotators achieve BA of  $87\%$  which we take as the ceiling performance for models to match.
5.4 Model Performance
All models are trained with AP+AN context and decoded by  $\mathrm{ExH}^9$ . Results are shown in Table 5. Finetuning BERT achieves the best performance among other models, though it still lags behind human performance significantly. Unsupervised models could only solve one third of all blanks. Surprisingly,  $\overline{\mathrm{PMI}_2}$  and COHERE performs worse than the unsupervised models. We conjecture that it is difficult for COHERE, using syntactic regularities alone, to distinguish between the ground truth answer for a particular blank and another candidate which is a ground truth answer for another nearby blank. As noted,  $\overline{\mathrm{PMI}_2}$  suffers due to inability of incorporating larger context.
To explore effects of various context length and decoding strategies, models are trained with different context lengths and inferred by both decoding methods. Results are shown in Table 6.
INC vs EXH EXH is better than INC for most approaches, indicating that human created blanks are interdependent and need joint answering.
Context Length Increasing the context length, such as (P vs. AP), could significantly improve model performance, showing that this task needs discourse-level context to be answered. Furthermore, models with bidirectional context, such as  $(\mathrm{P} + \mathrm{N})$ , perform better than single-direction context, e.g., P, indicating that this task needs global context. Lastly, we observe that PMI-based approaches which do not explicitly encode sentences
Marked by AMT based on approval %, no. approved etc.
${}^{9}$  Unless stated otherwise, models decode with EXH and are trained with full context i.e AP+AN
5674
Figure 1: Test blank accuracy of BERT-ft and Human on each reasoning type category introduced in §3.3.
are unable to incorporate larger context levels, showing best performance with  $\mathrm{P + N}$
5.5 BERT-ft vs. Human
BERT after finetuning (BERT-ft) can perform reasonably well  $(72\%)$  but there is still a gap comparing with human performance  $(87\%)$ . In this section, we would like to analyze the strength and weakness of BERT-ft compared with HUMAN. Therefore, we analyze their performance across different reasoning categories on test set. From Figure 1, inference questions are the most difficult for both HUMAN and BERT-ft and questions needing WordMatch are relatively easy. Compared with human performance, BERT-ft could achieve comparable BA on WordMatch and paraphrasing problems. However, BERT-ft performs much worse on questions needing inference and summary. We also refer to some examples from Table 4.
In Example 4, BERT-ft prefers A but the answer is C. The reason why BERT-ft chooses A may be that "enjoy life" happens in the context, but summarizing the next sentence is necessary to achieve the correct answer. Therefore, it is necessary to improve the ability of BERT to represent meaning at the sentence level beyond representing individual words in context.
We also explore how the system performance corresponds to the human judgement of difficulty. Since evaluates rate the problems into 5 difficulty levels, we report the system BA/PA for each level in Table 8. For BA (blank-level accuracy), we see that, overall, the system accuracy decreases as difficulty increases from VeryEasy (0.75) to Very-Hard (0.68). However, the decrease is not exactly monotonic (there is a small increase from VeryEasy to Easy, as also from Moderate to Hard).
We conjecture that non-monotonicity could be
due to two reasons:
- Our difficulty annotations are at passage level rather than blank level. There might be some hard blanks in a passage marked overall "Easy". Conversely, there might be easy blanks in a passage marked overall "Hard".
- Since we've more examples marked with certain difficulty levels - e.g.  $30.5\%$  examples are "Easy" while only  $8.3\%$  are "VeryEasy". This might make system accuracy average for levels with more examples more stable (lower sample variance), leading to some non-monotonicity (e.g. for Easy and VeryEasy)
For PA (passage-level accuracy, i.e., getting all questions correct) also, we see a clear decrease as difficulty increases from VeryEasy (0.63) to Very-Hard(0.2). The decrease here is sharper than BA, with only one violation of monotonicity (increase from 0.29 to 0.35 on Moderate to Hard). The sharper trend for PA supports our first point above.
Difficulty	BA	PA
Very Easy	0.75	0.63
Easy	0.78	0.45
Moderate	0.71	0.29
Hard	0.72	0.35
Very Hard	0.68	0.20
Table 8: BERT-ft performance in terms of human judgement of difficulty.
5.6 Distractor Quality
An attractive aspect of this task is distractors designed by English teachers. We verify distractor quality through the following experiments.
Model Performance w/o Distractors All distractors in the test set are removed and models are evaluated on this non-distracted test set. Results are shown in Table 7. It is clear to see that after removing these distracting candidates, models can get better scores, showing that models find it hard to exclude distractors during prediction.
Randomly Sampled Distractors After removing human-created distractors, we further randomly sample sentences from other passages as new distractors. To mitigate sampling variance, we run this experiment with 8 seeds and report the
Model	Uni.	PMI2	BERT-ft	HUMAN
DE	1.429	1.204	0.661	0.375
Table 9: Distractor error on test set of different models. Uni. denotes the uniform model.
Training Strategy	PA	BA	DE
QA	65.2	26.1	0.792
QH	71.7	29.9	0.661
QA;QH	74.2	33.9	0.624
QA+QH	74.5	34.3	0.637
Table 10: Test performance of models with  $Q_{A}$  and  $Q_{H}$ .
averaged score in Table 7. Comparing with distractors designed by teachers, models could discern these distractors more easily.
Annotation artifacts of distractors Annotation artifacts (Gururanget al., 2018) occurs in many datasets created by human annotators. A potential artifact type for our task is whether we could detect distractors without passages. Therefore, we finetune BERT-Large as a binary classifier, the input of which is just distractors and other correct candidates. With this model, we could only obtain  $38\%$  F1 score on the test set, showing that it is difficult to filter distractors out without any context.
Distractor Error (DE) We define DE as the number of predicted answers per passage which are actually distractors. Through DE, we measure a model's ability to exclude distractors during prediction. Results are shown in Table 9. HUMAN has the lowest DE and BERT-ft could discern distractors to some extent. However, DE of  $\overline{\mathrm{PMI}_2}$  is more than 1, meaning that on average, there is at least one distractor in the predicted answer list.
In summary, distractors created by teachers are high quality and increase task difficulty.
5.7 Automatically Generated Sentence Cloze Questions
To explore automatic generation of examples for the task, we construct sentence cloze questions by randomly choosing five sentences in a passage as blanks. We defer automatically generating distractors to future work since non-trivial distractor generation is a hard problem in itself. Specifically, we extract all passages from RACE (Lai et al., 2017)
(which is also from exams) and filter out passages which have less than 10 sentences or more than 30 sentences. While choosing blank positions, we prevent three or more blanks consecutive to each other in generated questions. Finally, 16,706 examples are obtained automatically. Here, questions generated automatically and collected from examinations are called  $Q_{A}$  and  $Q_{H}$  respectively.
We leverage  $Q_{A}$  in three ways: 1). train models only on  $Q_{A}$ , 2) first train models on  $Q_{A}$  and finetune models on  $Q_{H}$ , i.e.,  $Q_{A}$ ;  $Q_{H}$ , 3) train models on the concatenation of  $Q_{A}$  and  $Q_{H}$ , i.e.,  $Q_{A} + Q_{H}$ . BERT-Large is finetuned through these ways and results are shown in Table 10. The model trained only on  $Q_{A}$  has worst performance and we attribute this to the difficulty of distinguishing distractors without seeing them during training. Therefore, this model has the highest DE. However, models trained on  $Q_{H}$  and  $Q_{A}$  could achieve better performance. We conjecture this is because  $Q_{A}$  assists the model to have better generalization.
We introduce SCDE, a sentence cloze dataset with high quality distractors carefully designed by English teachers. SCDE requires use of discourse-level context and different reasoning types. More importantly, the high quality distractors make this task more challenging. Human performance is found to exceed advanced contextual embedding and language models by a significant margin. Through SCDE, we aim to encourage the development of more advanced language understanding models.
We thank Qizhe Xie, Hiroaki Hayashi and the 3 anonymous reviewers for valuable comments.
Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to-beat baseline for sentence embeddings. ICLR.
Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Scott Wen-tau Yih, and Yejin Choi. 2019. Abductive commonsense reasoning. arXiv preprint arXiv:1908.05739.
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642.
Danqi Chen, Jason Bolton, and Christopher D Manning. 2016. A thorough examination of the cnn/daily mail reading comprehension task. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2358-2367.
Kenneth Ward Church and Patrick Hanks. 1990. Word association norms, mutual information, and lexicography. Computational linguistics, 16(1):22-29.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv:1705.02364.
Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Sandra S Fotos. 1991. The cloze test as an integrative measure of efl proficiency: A substitute for essays on college entrance examinations? Language learning, 41(3):313-336.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), volume 2, pages 107-112.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in neural information processing systems, pages 1693-1701.
Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2015. The goldilocks principle: Reading children's books with explicit memory representations. arXiv preprint arXiv:1511.02301.
Jennifer Hill and Rahul Simha. 2016. Automatic generation of context-based fill-in-the-blank exercises using co-occurrence likelihoods and google n-grams. In Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications, pages 23-30.
Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daume III. 2015. Deep unordered composition rivals syntactic methods for text classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), volume 1, pages 1681-1691.
Jon Jonz. 1991. Cloze item types and second language comprehension. Language testing, 8(1):1-22.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785-794.
Marco Marelli, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamparelli. 2014. Semeval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. In Proceedings of the 8th international workshop on semantic evaluation (SemEval 2014), pages 1-8.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839-849.
Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gimpel, and David McAllester. 2016. Who did what: A large-scale person-centered cloze dataset. arXiv preprint arXiv:1608.05457.
Denis Paperno, Germán Kruszewski, Angeliki Lazari-dou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez. 2016. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv:1606.06031.
5677
Anselmo Penas, Yusuke Miyao, Alvaro Rodrigo, Edward H Hovy, and Noriko Kando. 2014. Overview of CLEF QA Entrance Exams Task 2014. In CLEF (Working Notes), pages 1194-1200.
Alvaro Rodrigo, Anselmo Penas, Yusuke Miyao, Edward H Hovy, and Noriko Kando. 2015. Overview of CLEF QA Entrance Exams Task 2015. In CLEF (Working Notes).
Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. 2019a. Atomic: An atlas of machine commonsense for if-then reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3027-3035.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019b. Socialiaq: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09728.
Karin Sim Smith, Wilker Aziz, and Lucia Specia. 2016. Cohere: A toolkit for local coherence. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 4111-4114.
Wilson L Taylor. 1953. "cloze procedure": A new tool for measuring readability. Journalism Bulletin, 30(4):415-433.
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaeher Suleman. 2017. NewsQA: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 191-200.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
Qizhe Xie, Guokun Lai, Zihang Dai, and Eduard Hovy. 2018. Large-scale cloze test dataset created by teachers. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2344-2356.
Geoffrey Zweig and Christopher JC Burges. 2011. The microsoft research sentence completion challenge. Microsoft Research, Redmond, WA, USA, Tech. Rep. MSR-TR-2011-129.
A Problem Complexity
With  $|B| = 5$  blanks and  $|C| = 7$  candidates, the size of answer space,  $|A|$ , is number of permutations  $|B|$  objects taken  $|C|$  at a time, i.e.,  $\mathrm{P}(7,5) = 2520$ . Therefore, the probability of answering all blanks correctly is  $\frac{1}{2520} = 0.03\%$
What are the chances of getting answers partially correct? What are the chances of getting answers partially correct? If we have the same number of candidates as blanks, this is equivalent to  $|B|! - D_{|B|}$ , where  $D_{|B|}$  is the number of derangements $^{10}$  of  $|B|$  elements. In the presence of more candidates than blanks i.e distractors, this expression becomes more involved to derive. Therefore, here, we enumerate all the permutation of answer lists given a correct answer. With  $|C| = 7$  and  $|B| = 5$ ,  $\zeta(|C|, |B|) = 51.8\%$ . In other words, there is a  $48.2\%$  probability of being entirely wrong with a randomly chosen set of answers to each blank in the passage.
What are the chances of getting distractors as predicted answers? For the expectation of number of distractors choosing by uniform model, it should be  $\mathbb{E}[DE]$ , where  $DE$  denotes distractors errors.
$$
\sum_ {d = 0} ^ {2} p (D E = d) \times d \tag {1}
$$
where  $p(DE = d)$  denotes the probability of  $d$  predicated answers are distractors. Since there are two distractors in candidates, the maximum of  $d$  is 2. Furthermore,  $p(DE = 1)$  is
$$
\mathrm {P} (5, 4) \mathrm {C} (5, 4) \mathrm {C} (2, 1) / | A | = 0. 4 7 6 \qquad (2)
$$
and  $p(DE = 2)$  is
$$
\mathrm {P} (5, 3) \mathrm {C} (5, 3) \mathrm {A} (2, 2) / | A | = 0. 4 7 6 \tag {3}
$$
where  $\mathrm{C}(\cdot, \cdot)$  and  $\mathrm{P}(\cdot, \cdot)$  is combination and permutation respectively. Therefore, the expectation of number of distractors is 1.429.
B Additional Experiment Specifications
Specific BERT Model Used
We use uncased BERT models for all our experiments. We use the BERT models trained by the canonical pytorch implementation of Wolf et al. (2019).
C More examples
We show more examples belonging to different reasoning categories in Table 11. Also, some completed questions with strong distractors, multiblank logic and diverse reasoning types are shown in Table 12, 13 and 14.
10en.wikipedia.org/wiki/Derangement
5679
Table 11: More examples of reasoning categories.
5680
Dear David _______1 After I had spent a week with my English family, I slowly began to understand their English a little better. _______2 Students in my group are from different cities of Britain and their dialects are different too! Some of their accents are quite strong and they also have their own words and expressions. _______3 Before I came to England I had thought that fish and chips were eaten every day. That's quite wrong! I get rather annoyed now when I hear all the foolish words about typical English food. I had expected to see "London fog". Do you remember our texts about it? We had no idea that most of this "thick fog" disappeared many years ago when people stopped using coal in their homes. But the idea to speak about weather was very helpful. _______4 On the other hand, habits are different. People tell me what is typical British here in London is not always typical in Wales or Scotland. _______5 But what is ordinary for all British is that they follow traditions. Probably Britain has more living signs of its past than many other countries. And people have always been proud of having ancient buildings in capitals, big cities and the countryside. I will tell you more about Britain in my other letters. Love from Britain.
A. But it's not the language that's different and surprising.
B. Thanks for your nice letter.
C. I have difficulty in understanding my classmates.
D. The family I live with are friendly.
E. It's very different from what I learned at school.
F. Local habits and traditions are not the same as what we knew.
G. The weather in London is really changeable.
Answers:  $1\rightarrow \mathrm{B}$  2→E,3→A,4→G,5→F (C and D are distractors)
Discussion: C is a strong distractor - not only does it have strong word overlap with the contexts of many blanks - it also has words which can make it rank high in terms of the possible inferences (dialects are different implies difficulty in understanding. Though not as strong as C, D also has a key word matching and is similar in content to the topic.