SlowGuess/ACL_2020_OCR huggingface dataset:
Xiang Kong\*, Varun Gangal\*, Eduard Hovy
{xiangk,vgangal,hovy}@cs.cmu.edu
We introduce SCDE, a dataset to evaluate the performance of computational models through sentence prediction. SCDE is a human-created sentence cloze dataset, collected from public school English examinations. Our task requires a model to fill up multiple blanks in a passage from a shared candidate set with distractors designed by English teachers. Experimental results demonstrate that this task requires the use of non-local, discourse-level context beyond the immediate sentence neighborhood. The blanks require joint solving and significantly impair each other's context. Furthermore, through ablations, we show that the distractors are of high quality and make the task more challenging. Our experiments show that there is a significant performance gap between advanced models (72%) and humans (87%), encouraging future models to bridge this gap. $^{1}$
Cloze questions were first proposed by Taylor (1953) as a readability test, motivated by Gestalt psychology. They become an efficient way of testing reading for public exams, overtaking the dominant paradigm of subjective questions (Fotos, 1991; Jonz, 1991). Cloze datasets (Zweig and Burges, 2011; Hermann et al., 2015; Hill et al., 2015; Paperno et al., 2016; Onishi et al., 2016; Xie et al., 2018) became prevalent as question-answering (QA) benchmarks since they are convenient either to be generated automatically or by annotators. These datasets could be split into two clear types:
1. Where the context is a complete text, and there is an explicit question posed which is a statement with a cloze gap. The answer is either generated freely or is a span
from the context, e.g. Children's Books Test (CBT) (Hill et al., 2015).
2. Where the context itself comes with cloze gaps. There is no explicit question. The answer is generated freely or chosen from a set of candidates, e.g. CLOTH (Xie et al., 2018).
Herein, we focus on the 2nd category. A common property of these datasets is that they have gaps at the level of words, entities or short syntactic spans. The entity and span-based clozes may sometimes be multi-token, but they do not extend beyond a few tokens. Nevertheless, none of these datasets have cloze gaps at the level of full sentences. Since many syntactic and semantic cues are present in the same sentence, this makes the gap easier to fill compared to the sentence level cloze case where models would have to rely on "discourse" cues beyond the same sentence.
Besides lack of intra-sentence cues, sentence-level cloze may require comparing candidates of very different lengths. For instance, the example in Table 1 has a standard deviation of 7.6 with candidate lengths between 3 to 25. A model that only represents words well may not get comparable probabilities at sentence level for very different sentence lengths. Therefore, robust sentence representation models are also required to solve this question. In this paper, we present SCDE, a dataset of sentence-level cloze questions sourced from public school examinations. Each dataset example consists of a passage with multiple sentence-level blanks and a shared set of candidates. Besides the right answer to each cloze in the passage, the candidate set also contains ones which don't answer any cloze, a.k.a., distractors. Both cloze positions and distractors are authored by teachers who design the public school examinations carefully. §3.2 explains our data collection. A representative example from SCDE is shown in Table 1.
$^{1}$ Data: vgtomahawk.github.io/sced.html
$^{2}$ Code: https://github.com/shawnkx/SCDE
5668
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5668-5683 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
A student's life is never easy. And it is even more difficult if you will have to complete your study in a foreign land. The following are some basic things you need to do before even seizing that passport and boarding on the plane. Knowing the country. You shouldn't bother researching the country's hottest tourist spots or historical places. You won't go there as a tourist, but as a student. In addition, read about their laws. You surely don't want to face legal problems, especially if you're away from home. Don't expect that you can graduate abroad without knowing even the basics of the language. Before leaving your home country, take online lessons to at least master some of their words and sentences. This will be useful in living and studying there. Doing this will also prepare you in communicating with those who can't speak English. Preparing for other needs. Check the conversion of your money to their local currency. The Internet of your intended school will be very helpful in findings an apartment and helping you understand local currency. Remember, you're not only carrying your own reputation but your country's reputation as well. If you act foolishly, people there might think that all of your countrymen are foolish as well.
A. Studying their language.
B. That would surely be a very bad start for your study abroad program.
C. Going with their trends will keep it from being too obvious that you're a foreigner.
D. Set up your bank account so you can use it there, get an insurance, and find an apartment.
E. It'll be helpful to read the most important points in their history and to read up on their culture.
F. A lot of preparations are needed so you can be sure to go back home with a diploma and a bright future waiting for you.
G. Packing your clothes.
Answers with Reasoning Type:
$1\to \mathrm{F}$  (Summary),  $2\to \mathrm{E}$  (Inference),  $3\to \mathrm{A}$  (Paraphrase),  $4\to \mathrm{D}$  (WordMatch),  $5\to \mathrm{B}$  (Inference) (C and G are distractors)
Discussion:
Blank 3 is the easiest to solve, since "Studying their language" is a near-paraphrase of "Knowing even the basics of the language". Blank 2 needs to be reasoned out by Inference - specifically  $E$  can be inferred from the previous sentence. Note however that  $C$  is also a possible inference from the previous sentence - it is only after reading the entire context, which seems to be about learning various aspects of a country, that  $E$  seems to fit better. Blank 1 needs Summary  $\rightarrow$  it requires understanding several later sentences and abstracting out that they all refer to lots of preparations. Finally, Blank 5 can be mapped to B by inferring that people thinking all your countrymen are foolish is bad, while Blank 4 is a easy WordMatch on apartment to D. The other distractor G, although topically related to preparation for going abroad, does not directly fit into any of the blank contexts
Table 1: A Representative Example from SCDE.
Another salient aspect of our dataset is that more than  $40\%$  of blanks belong to the reasoning category "Inference" (more on this in §3.3 and Table 4) which require models to compare plausibility of competing hypotheses given a premise (whether the previous or last sentence(s), or even a combination of information from the two). Filling these blanks requires the model to reason by using commonsense knowledge, factual knowledge, time gaps, etc. Some of these can be thought of as simple entailment, but more generally, many of these can be seen as requiring abductive reasoning, which is of recent interest (Bhagavatula et al., 2019; Sap et al., 2019a,b) to the NLP community. In summary, our contributions are as follows
1. We introduce the task of sentence level cloze completion with multiple sentence blanks and a shared candidate set with distractors.
2. We release SCDE, a sentence level cloze dataset of  $\approx 6\mathrm{k}$  passages and  $\approx 30\mathrm{k}$  blanks.
3. We estimate human performance on SCDE, and benchmark several models, including state-of-the-art contextual embeddings (Table 5). We find a significant gap of  $>15\%$  for future models to close in order to match human performance.
4. Through several ablations described in  $\S 5.6$ , we show that distractors designed by English teachers are of high quality and make the task more challenging.
5. We show that extra sentence level cloze questions generated automatically from an external corpus can be used to further improve model performance through data augmentation (See §5.7).
Several cloze test datasets are collected to measure reading comprehension ability of machines. CNN/DailyMail (Hermann et al., 2015), an early dataset of current QA research, constructs cloze questions from article summaries, with article spans as answers. Their cloze gaps are entities and hence one or few tokens long at best. The LAM-BADA dataset (Paperno et al., 2016) constructs a corpus of word level cloze gaps, such that each gap is in the last passage sentence. CBT (Hill and Simha, 2016) creates word level cloze questions by removing a word in the last sentence of every consecutive 21 sentences, with the first 20 sentences being the context. Onishi et al. (2016) curate a dataset of who-did-what type sentences with
5669
Dataset	SL	MB	Distractors	Candidates	Position	|Context|w
SCDE	✓	✓	Human	Shared	Anywhere	319
ROCSTORIES (2016)	✓	×	Human	-	End	25
CLOTH (2018)	×	✓	Human	Separated	Anywhere	243
LAMBADA (2016)	×	×	Exhaustive	-	End	76
CBT (2015)	×	×	Automatic	-	End	465
MRSCC (2011)	×	×	Human	-	Anywhere	20
Table 2: Comparing SCDE with previous cloze datasets. Exhaustive denotes the case where the entire vocabulary is a candidate for a word level cloze. For the single-blank case, candidate sharing is irrelevant. SL and MB mean sentence level and multi-blanks respectively.  $\| \text{Context} \|_w$  is the average token length of the context.
word level blanks. The CLOTH (Xie et al., 2018) dataset collects word level cloze questions from English exams designed by teachers. MRSCC (Zweig and Burges, 2011) consists of 1,040 word level cloze questions created by human annotators.
Among recent cloze datasets, ROCStories (Mostafazadeh et al., 2016) is the closest we could find to a sentence level cloze dataset. In this task, the first 4 sentences of a 5-sentence story are provided, and the task is to choose the correct ending from a pair of candidate ending sentences. However, there are several key differences between SCDE and ROCStories. Firstly, there are multi-blanks in SCDE which are not in a fixed position and require learning cues from bidirectional contexts of varying lengths. Secondly, the endings in ROCStories have been found to contain "annotation artifacts" (Gururangan et al., 2018) which makes a large fraction of them predictable independent of context.
In contrast, SCDE is by design independent of artifacts, since a) given a blank, only some of our candidates are distractors, the rest being answers for other blanks. Even if one were to learn a classifier to distinguish distractors without context, the non-distractor candidates would be unresolvable without context. b) we further check how distinguishable our distractors are from non-distractors without context by training a strong classifier in this setting, as described in §5.6. The classifier obtains a reasonably low F1 score of 0.38.
In Table 2, we summarize the comparison of SCDE with cloze datasets from prior art to show its attractive aspects.
Public school examinations have been used as a data source by many earlier QA works, two prominent examples being the CLEF QA tracks (Penas et al., 2014; Rodrigo et al., 2015) and RACE (Lai et al., 2017).
3 SCDE Dataset
3.1 Sentence Cloze Test with distractors
In this task, each question consists of a passage,  $S$ , multiple sentence level blanks  $B$ , and a shared set of candidates  $C$  with distractors  $D$ , where  $D \subset C$ .
Problem Complexity<sup>3</sup> For our case, given the typical value of  $|C|$  and  $|B|$  being 7 and 5 respectively, the size of the answer space,  $|A|$  is 2520. Thus, the chance of guessing all blanks correctly at random is only  $0.04\%$ . Moreover, there is a  $48.2\%$  probability of being entirely wrong with randomly guessing. Finally, given an answer list chosen uniformly at random, the expectation of number of distractors in the answer list is 1.4, i.e. on average, roughly one and half answers are distractors.
3.2 Data Collection and Statistics
Raw sentence cloze problems are crawled from public websites<sup>4</sup> which curate middle and high school English exams designed by teachers. In total, 14,062 raw passages and 68,515 blank questions are crawled from these websites and the following steps are used to clean them. Firstly, duplicate passages are removed. Secondly, when the official answer to the problems are images, two OCR toolkits<sup>5</sup> are employed to convert these images to text and the questions with different results from these two programs will be discarded. Finally, we remove examples which have 1) answers pointing to non-existent candidates, 2) missing or null candidates, 3) number of blanks  $>$  number of candidates, 4) missing answers.
After cleaning, we obtain our SCDE dataset with 5,959 passages and 29,731 blanks. They are
<sup>3</sup>We defer the derivation to Appendix §1.
<sup>4</sup>http://www.21cnjy.com/; http://5utk.rs5u.com/; http://zujuan.xkw.com/; https://www.gzenxx.com/Html/rw/.
$^{5}$ tesseract; ABBYY FineReader
5670
Statistic	Value
Total Passages	5,959
Total Blanks	29,731
Blanks Per Passage	4.99
# Candidates Per Passage	6.79
Avg Candidates Per Blank	1.35
% Consecutive Blanks	1.28
# Words Per Passage	319.64
Vocabulary Size	48.6k
Var(Candidate Length)	19.54
Table 3: SCDE Statistics. For Consecutive Blanks, either of previous or next sentences is also a blank.
randomly split into training, validation and test sets with 4790, 511 and 658 passages respectively. The detailed statistics are presented in Table 3. We find that candidates have very different lengths and passages have long context.
3.3 In-Depth Analysis & Categorization
In order to evaluate students' mastery of a language, teachers usually design tests in a way that questions cover different aspects of a language.
Reasoning Types As illustrated with examples in Table 4, we set a four-fold categorization for the reasoning which leads to a ground truth candidate being assigned to a blank. Our reasoning type taxonomy is motivated by categorization of question types in earlier works in QA such as (Chen et al., 2016; Trischler et al., 2017)<sup>6</sup>. Strictly speaking, these reasoning types could co-exist. But for simplicity, we classify each blank into only one of the four.
- WORDMATCH: If the candidate has word overlap, especially of non-stopwords or infrequent phrases, with context around the blank.
- PARAPHRASE: If the candidate doesn't have an explicit word overlap with the context, but nevertheless contains words or phrases which are paraphrases of those in the context.
- INFERENCE: If the candidate is a valid hypothesis conditioned on the left context [as premise], or a necessary pre-condition/premise based on the right context. Note that the candidate in this case doesn't contain word overlap/paraphrases which would obviate need for inferential reasoning. The reasoning required needs not
be just strict entailment (Bowman et al., 2015; Marelli et al., 2014) but could also involve abductive reasoning (Bhagavatula et al., 2019), where the candidate is just one of many likely hypothesis (premise) given the left (right) context as premise (hypothesis).
- SUMMARY: If the candidate is a summary, introduction, or conclusion of multiple sentences before or after it. In this type, unlike INFERENCE, there is no requirement to deduce and reason about new hypotheses/possibilities not present in the premise - only consolidation and rearranging of information is required.
A sample of 100 passages containing 500 blanks are manually categorized into these four categories. Examples and statistics of these four types are listed in Table 4. More than  $40\%$  blanks need inference to be solved, denoting the high difficulty of our dataset.
4.1 Context Length
We experiment with giving our models different amounts of context. Through this, we can explore how context length affects model performance.
1. P(N): Immediate previous (next) sentence
2.  $\mathrm{P} + \mathrm{N}$  : Immediate previous and next sentence
3. AP(AN): All previous (next) sentences
4. AP+AN: All previous and next sentences
$\mathrm{AP + AN}$  is the unablated setting, where all passage sentences are available to the model.
4.2 PMI
Before exploring deep representational approaches, we would like to find how well symbolic ones perform at this task. Starting with works such as Iyyer et al. (2015) and Arora et al. (2017), it has become convention to first benchmark simple baselines of this kind. PMI merely encodes how likely it is for a word pair to occur in consecutive sentences. It does not consider the internal sentence structures, or the relative position of the words in their respective sentence. Intuitively, it can be called a "surface-level" approach. A high performance by PMI would indicate that candidates can be matched to blanks by simple ngram statistics, without requiring sentence representation, which would make SCDE uninteresting.
See Section 4.2 from both respective papers.
5671
Type	Examples with Excerpts From Blank Context
WM(18.47%)	1: One day, a teacher was giving a speech to his student. He held up a glass of water and asked the class _The students answers ranged from 20g to 500g.✓ Candidate: B. How heavy do you think this glass of water is?× Candidate: D. It does not matter on the weight itself.Explanation: WordMatch based on glass of water.
Para.(19.48%)	2: If you want time to have breakfast with your family, save some time the night before by setting out clothes, shoes and bags. _That's a quarter-hour more you could be sleeping if you bought a coffee maker with a timer.× Candidate: D. And consider setting a second alarm.× Candidate: F. Stick to your set bedtime and wake-up time, no matter the day.✓ Candidate: G. Reconsider the 15 minutes you spend in line at the cafe.Explanation: Need to match 15 minutes, quarter-hour and coffee, cafe.
Infer.(41.97%)	3: May is a great month. _You can have a good time with your family.× Candidate: E. All the students can come to their schools.✓ Candidate: F. From May 1st to 7th, we don't need to come to school.× Candidate: G. On May 20th, a famous sports star YaoMing comes to our school.Explanation: Need to infer that not coming to school → one is at home with family. Simply matching for words May or school will also match wrong candidates.
Sum.(20.08%)	4: How to Enjoy Life As a Teen? Are high school days equal to the "best years of your life"? Maybe not, but you can learn to make the most of your high school days _Whether it's having a computer, having friends, having a good supply of food, a bed to sleep on, family that loves you, having a decent education or simply being born in this world. Be happy, and life will reward you.× Candidate: A. Remember that the point of life is for you to enjoy it.✓ Candidate: C. Learn to appreciate small things.Explanation: After summarizing sentences after the blank [which describe a list of "small things)], the answer should be C. A is a strong distractor since both "enjoy" and "life" appear in the context, besides being pertinent to the topic. Indeed, our best-performing BERT-ft model chooses A as the answer.
Table 4: Blanks in a sample of 100 passages are manually categorized into four categories. For the ease of illustration, we've shown only limited context around the blanks , and 1-2 wrong candidates. WM, Para., Infer. and Sum denote WordMatch, Paraphrase, Inference and Summary respectively. More examples are in Appendix.
We estimate PMI counts (Church and Hanks, 1990) from all consecutive sentence pairs in our training split. Let  $f$  denote frequency
$$
\operatorname {P M I} \left(w _ {s}, w _ {c}\right) = \frac {\mathrm {f} \left(w _ {s} \in S , w _ {c} \in C\right)}{\mathrm {f} \left(w _ {s} \in S\right) \mathrm {f} \left(w _ {c} \in C\right)}
$$
Note that our PMI definition diverges from typical PMI since its asymmetric between  $w_{s}$  and  $w_{c}$ . Since  $S$  and  $C$  are the sets of non-terminating and non-starting sentences respectively, they overlap but aren't identical. For a pair of sentences, we find aggregate  $\overline{\mathrm{PMI}}(S, C)$  as:
$$
\overline {{\mathrm {P M I}}} (S, C) = \frac {1}{| C | | S |} \sum_ {w _ {c} \in C} \sum_ {w _ {s} \in S} \mathrm {P M I} (w _ {s}, w _ {c})
$$
This definition can be extended to all n-grams upto a certain  $n$ . We denote this by  $\overline{\mathrm{PMI}}_n$ . We notice that  $\overline{\mathrm{PMI}}_n$  performance saturates after  $n = 2$ . Hence, in our experiments, we use  $\overline{\mathrm{PMI}}_2$ .
4.3 Language Modelling
One intuitive way to solve this task is to generate the blank sentence given the context by advanced pre-trained language models (LM). Formally, suppose the blank is the  $i$ th sentence,
$s_i$ , and  $s_1, \ldots, s_{i-1}$ ,  $s_{i+1}, \ldots, s_n$  are the context. Our goal is to choose  $c_k$  from the candidate set which could maximize the joint probability  $p(s_1, \ldots, s_{i-1}, c_k, s_{i+1}, \ldots, s_n)$ .
Due to limited number of passages available to train a robust LM, Transformer-XL (TR.XL) Base (Dai et al., 2019), trained on WikiText-103, is employed to address this task. In order to make decoding time tractable, context length is limited to three sentences before and after the blank.
4.4 Coherence
Coherence models assign a continuous score to a sentence sequence indicative of its coherence. This score is usually unnormalized and not needed to be a probability [unlike language models].
We use the local coherence approaches implemented by the COHERE framework (Smith et al., 2016). Roughly, this model works on the intuition that successive sentences exhibit regularities in syntactic patterns. Specifically, it uses n-gram patterns on linearized syntactic parses (e.g. S NP VP ...) of consecutive sentences. Once
7github.com/karins/CoherenceFramework
trained, this model can return a "coherence score" for any sentence sequence.
The COHERE model is first trained on all ground-truth passages from our training set, with the ground truth answers filled into the blanks. At test-time, we score each possible answer permutation using the trained COHERE model and pick the highest scoring one. Note that decoding for COHERE is by definition exhaustive, and doesn't make any assumptions by answering the blanks in a particular order.
4.5 InferSent
Conneau et al. (2017) use textual inference supervision as a signal to train a shared sentence encoder for premises and hypotheses, which can later be used as a sentence representative. We refer to this approach as INFST. Context features of a given blank and one candidate feed to two encoders in INFST respectively and classify whether this candidate is suitable to this blank. The maximum tokens of context features is set as 256. Bi-directional LSTMs with the max pooling operation are employed as our encoders. We follow the training procedure described in Conneau et al. (2017).
4.6 BERT Models
Input Representations Let  $c_k$  denotes the  $k$ th candidate.  $s_{-i}$  and  $s_{+i}$  denote the  $i$ th sentence before and after the blank respectively and  $|P|$  and  $|N|$  represent total number of sentences before and after the current blank respectively. Following the input convention in Devlin et al. (2018), the input sequence given various context lengths and  $c_k$  is:
1. P: [CLS]  $s_{-1}$  [SEP]  $c_k$
2. N: [CLS]  $c_k$  [SEP]  $s_{+1}$
3. AP: [CLS]  $s_{-|P|}\ldots s_{-1}[\mathrm{SEP}]c_k$
4. AN: [CLS]  $c_k[\mathrm{SEP}]s_{+1}\ldots s_{+|N|}$
To retain sentence sequentiality, the order between the context and the candidate follows that in the original passage. Furthermore, for  $(\mathrm{A})\mathrm{P} + (\mathrm{A})\mathrm{N}$ , we create and score one input sample for each of the context directions during prediction. The average of these two scores is taken as the final score. The maximum tokens of input is set as 256 in our experiments and only the context is truncated to meet this requirement.
BERT Next Sentence Prediction (NSP) One of the objectives in BERT pre-training stage is