ClozEx: A Task toward Generation of English Cloze Explanation
Authors: Zizheng Zhang
Masato Mita
Mamoru Komachi
Venue: EMNLP 2023 Findings (2023)
Keywords: language assessment
english education
text generation
dataset
TL;DR: We proposed a task and corresponding large-scale dataset for generating explanations for English cloze questions.

Abstract: Providing explanations for cloze questions in language assessment (LA) has been recognized as a valuable approach to enhancing the language proficiency of learners. 
However, there is a noticeable absence of dedicated tasks and datasets specifically designed for generating language learner explanations.
In response to this gap, this paper introduces a novel task ClozEx of generating explanations for cloze questions in LA, with a particular focus on English as a Second Language (ESL) learners. 
To support this task, we present a meticulously curated dataset comprising cloze questions paired with corresponding explanations. 
This dataset aims to assess language proficiency  and facilitates language learning by offering informative and accurate explanations.
To tackle the task, we fine-tuned various baseline models with our training data, including encoder-decoder and decoder-only architectures. 
We also explored whether large language models (LLMs) are able to generate good explanations without fine-tuning, just using pre-defined prompts.
The evaluation results demonstrate that encoder-decoder models have the potential to deliver fluent and valid explanations when trained on our dataset.

ClozEx: A Task toward Generation of English Cloze Explanation
Zizheng Zhang and Masato Mita and Mamoru Komachi
Tokyo Metropolitan University. 6-6 Asahigaoka, Hino, Tokyo 191-0065, Japan CyberAgent, Inc. 2-24-12 Shibuya Shibuya-ku, Tokyo 150-6121, Japan Hitotsubashi University. 2-1 Naka, Kunitachi, Tokyo 186-8601, Japan ziznn-ziz tmu. ac. jp, mi mitaamasei aaa_msato@cyeerrgenttt co. jp. mamoru. komachi@r www-u.it.... jp
V
Abstract
Question:
The work was done the Rehabilitation Institute of Chicago under an $8-million grant from the Army.
Providing explanations for cloze questions in language assessment (LA) has been recognized as valuable approach to enhancing the lan- guage proficiency of learners. However, there is noticeable absence of dedicated tasks and datasets specifically designed for generating language learner explanations. In response to this gap, this paper introduces novel task ClozEx of generating explanations for cloze questions in LA, with particular focus En- glish as Second Language (ESL) learners. To support this task, we present meticulously cu- rated dataset comprising cloze questions paired with corresponding explanations. This dataset aims to assess language proficiency and facili- tates language learning by offering informative and accurate explanations. To tackle the task, we fine-tuned various baseline models with training data, including encoder-decte decoder and decoder-only architectures. We also explored whether large language models (LLMs) ablc to generate good explanations without fine- tuning, just using pre-defined prompts. The evaluation results demonstrate that encoder- decoder models have the potential to deliver fluent and valid explanations when trained on our dataset."
(A) at (B) down (C) round (D) of
Explanation:
Based on the context of the sentence, option (A) "at" is the appropriate choice for the cloze question. "At" indicates a specific location arrival at particular place position, such as "he is at the store."
Table I: Examples of data for the ClozEx task. A system is expected to receive Question as input and produce Explanation as output.
1988; Klein-Brale 1997). They are also widely employed in famous tests for English Second Language (ESL) learners, such the International English Language Testing System (IELTS) and Test of English as Foreign Language (TOEFL). Explanations for cloze questions play crucial role in language learning, particularly in self-study contexts. When learners encounter challenging cloze questions, having access to clear and con- cise explanations after answering the question can greatly aid their understanding of the correct an- swers (Williams et al., 2010). Explanations provide learners with insights into the reasoning behind the correct and incorrect choices, helping them iden- tify and rectify their own misconceptions. The pro- vision of high- explanations can empower learners, fostering deeper comprehension and long- term knowledge retention.
1 Introduction
Cloze questions (Taylor, 1953) a fundamental component of language assessment (LA). They typ- ically consist of a sentence or a passage with certain words or phrases omitted, and language Iearners are required to select or fill in the most appropri- ate word to complete the text. Cloze questions in language educational settings are usually used in language educational settings to evaluate lan- guage proficiency in terms of various aspects such grammatical knowledge (Rye, 1982; Alderson, 1979) and reading comprehension skills (Raymond,
However, despite its usefulness, there has been almost no work on generating high-quality expla- nations for given cloze questions. One essential reason is that no dataset for such a task is avail- able. Because of high costs terms of time and human effort, employing experts to create such a dataset, to provide abundant data, is difficult, al-
Dataset and codes are available at zz
com/

---

though it could guarantee the quality of the dataset. Furthermore, even if a dataset could be constructed, it would challenging to automatically generate human-like cloze explanation.
2 Related Work
Owing to the efficiency of cloze questions in lan- guage assessment, previous research focused on the automatic generation of cloze questions. In con- trast to early-period studies that employed naive approaches such as fixed ratio word deletion and random selection of distractor options (Rye, 1982; Bachman, 1985), current research places greater emphasis on ensuring the validity of generated questions. For example, Sakaguchi et al. (2013) cre- ated distractor options based on an English learner writing correction corpus. With this method, words that tend to be misused in given context would be selected as the distractor options. Therefore, ques- tions generated from their methods are expected to be more discriminative in measuring the lan- guage proficiency of learners. Additionally, pre- vious research (Goto et al., 2010; Correia et al., 2012; Hill and Simha, 2016; Jiang et al., 2020; Panda et al., 2022) investigated what aspects of features affect the validity of cloze questions, such as part of speech (POS), n-gram frequency, and sense of words. They then generated questions based such features. However, although ques- tion generation tasks do primarily aim generate plausible questions based on given texts, they of- ten do not explicitly address the generation of ac- companying explanations. The ability to generate explanations alongside cloze questions is crucial for providing comprehensive support to language learners. By shifting the focus from generating questions to generating explanations, this research introduces novel task that contributes to the ad- vancement of language learning technologies.
To address these challenges, we propose the task ClozEx of generating explanations for English cloze questions. Intuitively, a good explanation that helps answer cloze question should be easy to read and provide sufficient background knowledge. Therefore, fluency and informativeness should be considered in explanation generation. We also pro- vide large dataset comprising over 140k expert- quality-assureds (question, explanation) pairs, an example of which is shown in Table 1.
Finally, to investigate the factors that contribute to addressing the ClozEx task, we train various models baselines, including encoder-decode and decoder-only architectures. We also investi- gated the performance of large language model (LLMs) in a zero-shot prediction scenario, which we employed LLMs to generate explanations for given cloze questions without fine-tuning. The evaluation of baseline models indicated that both recoddrrdecoder and decoder-only models after fine-tuning are able to produce acceptable explana- tions. Meanwhile, LLMs generally good gen- erating fluent explanations, but in most cases, these explanations do not provide sufficient information for answering questions. Only providing LLMs with questions with naive prompts challenging to generate high-quality explanations efficiently. The main contributions of this work are summa- rized as follows:
We propose new task toward generation of fluent and valid English cloze explanation (ClozEx) for ESL learning.
Nagata (2019) proposed a task of feedback com- ment generation (FCG) for writing learning with corresponding dataset. The FCG task automatically generates feedback comments such hint explanatory note for writing learning for non-native learners of English. However, although this task contributes to grammar learning through writing correction, it has certain limitations in facilitating systematic grammar learning. Firstly, the FCG task primarily relies on free English composition, which adopts a bottom-up approach to provide grammat- ical knowledge. Consequently, it inevitably lacks comprehensive coverage of grammar items that learners need master. In contrast, cloze questions are meticulously designed by experts, adhering established learning guidelines, thereby ensuring a certain level of coverage of grammar items that
We create large-scale and expert-quality assured dataset for ClozEx task, including more than 140k instances generated by pattern-based method.
We investigate model performance trained on our dataset. We also explore the ability of LLMs of generating appropriate explanations in zero-shot scenario.
We examine the correlation between auto- matic evaluation metries and manual evalua- tion in the context of the ClozEx task, provid- ing insights into the reliability of these metrics for assessing the quality of generated explana- tions.

---

learners should be familiar with. Secondly, the primary focus of the FCG task lies in explaining the appropriateness of specific words within a sen- tence, rather than elucidating why certain plausi- ble expressions should be avoided. Furthermore, the commentaries in the FCG task stem from free composition, making it challenging to scale the production of high-quality commentaries without significant manual effort. In contrast, the ClozEx task builds "patterns" for each grammar item, as the cloze questions are constructed using a top-down approach. Consequently, it becomes feasible to automatically generate a considerable number of high-quality explanatory (details are explained in Section 3.2), as demonstrated by the generation of over 140k such sentences in this study.
of specific group of cloze questions. A pattern can also be used to create new cloze questions with explanations. Thus, we designed a pattern-based method for automatic cloze question and explana- tion generation. This method extracts patterns from expert-designes cloze questions and explanations to ensure the quality. Then these patterns are used to generate new questions and explanations.
The data creation process is outlined in Figure 1. This method involves the extraction of patterns from expert-designed cloze questions and their cor- responding explanations. These patterns serve as the foundation for generating new questions and explanations based on publicly available corpus. During the question creation phase, sentences from news corpus that align with given pattern are se- lected. Distractor options then generated based on which aspect of language is measured. For the explanation generation process, templates tailored to the question type are designed. These templates are populated with question and pattern informa- tion to yield initial explanations. Finally, we em- ploy LLMs to paraphrase the template-busee based expla- nations, enhancing their fluency and diversifying their expression
3 ClozEx Task and Dataset
3.1 Task Definition
Methods devised to address the ClozEx task are expected to operate on cloze question q as in- put. A cloze question comprises a sentence with a blank. denoted send, and a set of options OPT [opt1, opt2, ..., optn] (typically, n equals 4). The objective of the methods is to generate an explanation text ezp as output for the given ques- tion. The generated explanation should satisfy two critcria: (1) fluency (Cotter, 2012), mcaning that the explanation should be coherent and easily com- prehensible, because an explanation that is difficult to read would not effectively aid language learn- ing; (2) validity (Cross, 1991), indicating that the explanation should provide sufficient information, such relevant language knowledge, facilitate answering the question accurately.
3.3 Target Types
We begin by focusing on three specific types of cloze questions: affix, verb-tense, and preposi- tion. These question types have been selected based on their prominence in language assess- ment (Mochizuki and Aizawa, 2000; Collins, 2007; Chodorow et al., 2010), particularly in the context of the Test of English for International Communi- cation (TOEIC). Affix questions require ESL learn- to differentiate POS of options by analyzing prefixes suffixes. Verb-tense questions prompt learners to identify the appropriate tense of the sentence and options. Preposition questions ne- cessitate learners to comprehend the meaning of a sentence and consider the potential senses of the options (see Table 6 in Appendix A for examples). The comprehension of affix and verb tense ques- tions often relies narrower context within the sentence, allowing learners answer without nec- essarily reading the entire sentence. By contrast, preposition questions require a comprehensive un- derstanding of the sentence and an awareness of the various senses associated with prepositions. There-
3.2 Data Preparation
Experts in English education can be hired to write explanations for cloze questions to provide very high-quality data. However, because of the con- sumption of time and human effort, datasets ere- ated in such a way are scale-limited. To mitigate the considerable cost associated with manual expla- nation generation, we need to explore an automated method for creating both the questions and expla- nations in our dataset.
Experts design cloze questions in top-down manner, starting with a specific grammatical item. Subsequently, they designed various questions based on the grammatical item (Rye, 1982). Such grammatical items could be regarded as a pattern
"To avoid redundancy, or an excessive amount of irrelevant information, in the generated explanation, we set a maximum length for the explanation (128 words).

---

Public Corpus
Question
(Expert-desiencd)
Pattern Extractor
Pattern
Question Generator
Explanation
(Expert-designed)
Questions
Explanation Paraphraser
Initial Explanations
Explanation Generator
Explanations
Figure 1: Pipeline of data creation method. Yellow rectangles symbolize input to the pipeline, whereas blue rectangles represent output. Modules are depicted orange and their corresponding intermediate results are highlighted in green
fore, affix/tense and preposition questions neces- sitate different focal points for extracting patterns and generating informative explanations.
The Westchester Philharmonic received a national award its education program three years ago.
(a) Example of sent" red word represents the answer op- tion, and green ones denote hint words extracted from expert- designed explanation.
A Affix/Tense Questions Affix/tense questions ne- cessitate ESL learners to identify and analyze a specific context referred to as "hint words," which serve to modify or be modified by the word in the blank to answer the question accurately. To capture the patterns inherent in these questions, focus on the relationship between the hint words and the answer option.
punct
obi
num
ady
amod
case
mod
mod
...received a
national award for ... program three years ago NN IN...
VBD DT
JJ
NN CD NNS RB
(b) Partial dependoney parsing tree of sent in (a). Only nodes colored words are extracted as pattern Pattern Figure
To extract the pattern from each expert-designed question, we begin by inserting the answer option into the sentence, resulting in completed sen- tence denoted sen/ans, Next, we extract the hint words from the expert-designed explanation, and we mark their corresponding positions in sentans (see (a) in Figure 2). Subsequently, we employ dependency parsing on sentans to generate its de- pendency tree. Given that the hint words and the answer option play crucial roles in the question, we extract a sub-tree from the dependency tree that en- compasses all the hint words and the answer node. This sub-trec serves as the pattern for the question and is denoted as pallern (see (b) in Figure 2, the pattern could be summarized as "A noun works an object that is modified by an article and adjec- tive.").
con
amod
pllly nsubi
family hired
legal contractor had NN
ramp installed ...
DT NN VBD DT JJ
CC
VBD DT
NN
VBN
(e) Partial sent: and its dependency parsing tree. Because free: consists of pattern (marked in colored text), sent, could be used to generate question.
Question:
The family hired a legal and had the ramp installed front of their home Woodlands at Copperstone Brentwood. conttraccllll (B) contractor
(C) contracted (D) contractable
Initial Explanation: The word the blank should be the object of "hired". "a" is the determiner of the blank. "legal" is the the adjective modifier of the blank.
Thus, a Noun, singular mass is required. (A) contractual is a Adjective. (B) contractor is Noun, singular or mass. (C) contracted is a Verb, past tense. (D) contractable is Adjective.
After obtaining the pattern for a specific ques- tion, we utilize it to generate new questions. We parse all sentences, denoted as from publicly available news corpus quire their respective parsing trees, denoted [tree], ..., treem). We use a news corpus because news is formal writing and leads to fewer gram- matical errors. parsing tree, tree, includes
Therefore, the correct answer (B) contractor.
sentm,
(d) Example of generated question and corresponding initial explanation Initial Explanations in Figure 1).
Figure 2: Examples of process of generating new question with its explanation.

---

the extracted pattern patterny, we consider the corresponding sentence, sent,, as a suitable candi- date for generating a new question that belongs to patterny. It is important to note that our focus lies in capturing the modification relationship between the hint words and the answer option (e.g,, de- pendency relations), and their grammatical classes within the sentence (e.g., POS), rather than the spe- cific words used in the question generation process (see (c) in Figure 2).
language learning (ALTE, 2011). As highlighted by Srikumar and Roth (2013), prepositions sharing the same semantic relation often appear similar contexts. By utilizing prepositions with similar se- mantic roles as distractor options, we can enhance the difficulty level of preposition questions. To facilitate this, we construct dictionary to cluster prepositions based on their semantic roles, which aids in the selection of appropriate distractor op- tions.
To select distractors for the new question, we built candidate dictionaries for affix and verb-tense questions, respectively. Distractor options are se- lected from the corresponding dictionary. For ex- ample, if an affix question has the answer option "contractor", the distractor candidates could be in ["contractuall, "contraction", "contracted", "con- tractable"]. Similarly, distractor options for verb- tense questions are also selected from another pre- defined dictionary.
Finally, similar to the approach described in Sec- tion 3.2.1, we design a template to generate initial explanations, which are then refined by employ- ing an LLM to enhance their fluency and diversity. Details the implementation can be found in Ap- pendix B.
3.4 Dataset Analysis
To validate the quality and suitability of our created dataset for training models in the ClozEx task, we conducted a thorough manual quality assessment. As outlined in Section 3.1, the evaluation focused on two aspects: fluency and validity.
Finally, we design templates for specific types of questions to present all the necessary information for answering the question, including pattern and options (see (d) in Figure 2). To improve fluency and diversity, we employ LLM paraphrase the template-basee explanation. Details on the imple- mentation can be found in Appendix B.
For the fluency assessment, we enlisted the ex- pertise of two native English speakers from the university with which the authors are associated. These experts independently evaluated 100 ran- domly selected instances from our dataset using 5-point Likert scale (1 denotes the worst and 5 denotes the best), solely considering the fluency of the generated explanations and disregarding their validity. To evaluate the validity aspect, we re- cruited four advanced ESL learners from the uni- versity with which the authors are associated, be- cause these learners possess a strong understanding of textbook grammar (Glisan and Drescher, 1993). Similarly, these annotators used 5-point Likert scale assess the validity of 100 instances. To en- sure the independence between fluency and validity, we selected fluent instances in advance for the va- lidity estimation. The validity assessment aimed to determine whether the explanations provided the necessary information to answer the corresponding question. Further details regarding the estimation process can be found in Appendix C.
Prep. Questions Preposition questions require a comprehensive understanding of sentence meaning and the specific senses associated with the preposi- tion options. Consequently, the pattern for a prepo- sition question should incorporate the answer op- tion along with its corresponding sense within the given sentence. To achieve this, we employed preposition sense disambiguation (PSD) model to determine the sense of the answer option within a particular sentence, denoted as sentons
Subsequently, we consider the answer option together with its identified sense as the pattern, de- noted pallern. We then apply PSD to sentences extracted from a publicly available news corpus. If a sentence, denoted as sent,, contains the pattern patternj, it is considered a viable candidate for generating a new preposition question.
When selecting distractor options for preposi- tion questions, straightforward approach would involve randomly choosing prepositions from pool of available options. However, this method may yield simple questions that are easy to answer. Such simplistic questions fail to effectively gauge the language proficiency of ESL learners or aid in
To ensure robustness, each instance underwent double annotation for both fluency and validity. We performed the Pearson correlation test to assess the inter-annotan agreement between the different
'They hold public English test certificates to indicate they have a CEFR A2 level or higher.

---

A
Estimation Score
where "[OPT]" is special token that is used for concatenation among sentence and options. The output of the encoder-decoder models is the corre- sponding explanation. We explored different model sizes, including base and large, to assess their per- formance in the ClozEx task.
Pear.
p-value
Avg.
Med.
Var.
Flu.
0.82
<0.001
4.29
4.00
0.52
Val. 0.77
<0.001
4.51
4.50
0.45
Table 2: Inter-annotator agreement and manual esti- mation result. Pear. denotes Pearson's is correlation co- efficient. Avg. Med., and Var. indicate the average, median, and variance of scores, respectively. Flu. and Val. represent fluency and validity.
On the other hand, in the case of the decoder- only models for fine-tuning, and GPT2-medium (Radford et al., 2019). For decoder-only models, the input is a question with an explanation that is connected with a prompt. We then fine-tuned models with such input instances. Because LLMs have showen remarkable per- formance across diverse tasks in zero-shot scenar- ios (Kojima et al., 2022), to explore the potential of LLMs in solving the ClozEx task without the need for additional training data, we employed LLMs of different sizes and structures generate explana- tions without fine-tuning. We employed GPT2- large, GPT2-XL, GPT3.5-turbo 4 and LLaMa- 7B (Touvron et al., 2023) to generate explanations in the zero-shot scenario. The prompts used for the LLMs can be found in Appendix D.
selected GPT2
#(Q. E)
Q avg. len.
E Eag. len.
Train
102,930
28.99
58.53
Dev.
22.056
29.00
58.69
Test
22,057
28.95
58.47
Table 3: Statistics of our dataset. #(Q, E) represents number of (question, explanation) pairs. Q avg. len. and E avg. len. denote average lengths of questions and explanations (number of words), respectively.
annotators. Result of inter-annotator agreement and manual estimation shown in Table 2. The high correlation coefficients indicate strong agreement among the annotators, underscoring the reliability of our manual estimation. The scores for both fluency and validity exhibited high median values and low variance. These findings confirm the high quality of our dataset and support its publication as a reliable resource for the ClozEx task.
Evaluation Metries We engaged human annota- tors to estimate the fluency and validity of the gen- erated explanation, following the same estimation process as described in Section 3.4. We randomly selected 100 samples of generated explanations from each model to be estimated. All instances were estimated without reference explanations, suring reference-free evaluation.
For comprehensive understanding of our dataset, Table 3 presents a statistical analysis, pro- viding relevant insights into its characteristics.
To complement the manual annotation, which can be time-consuming and less generalizable, we also employed automatic metrics to assess the gen- erated explanations. For reference- we used BLEU-4 (Papineni et al., 2002) from the Huggingface Evaluate library to measure the sim- ilarity between the generated explanations and the reference labels. According to Wang al. (2023a), LLMs such as GPT3.5-turke can evaluate the qual- ity of generated text and exhibit a moderate correla- tion with human annotators. Therefore, we utilized GPT3.5 GGT33.5-tubb -ttbbo as a reference- metric to evaluate the fluency and validity of the generated explana- tions. The reliability of GPT evaluators will be discussed in Section 5.2. Samples used for the GPT evaluator are the same manual estimation. All metries except BLEU are based on the Likert 5- S-point scale. Prompts for the GPT evaluator can
4 Experiment
To address the ClozEx task, conducted an inves- tigation into baseline models under various scenar- ios and architectures. To evaluate the performance of these baseline models, we conducted thorough assessments using development and test data from our dataset, encompassing both manual and auto- matic evaluation metrics.
metrics,
4.1 Experiment Setup
Baseline Models As generation task, we em- ployed encoder-decoded and decoder-only models for fine-tuning. In the case of the encoder-decoder models, we performed fine-tuning on BART (Lewis al., 2020) and T5 (Raffel et al., 2020) archi- tectures. For fine-tuning, we tailored cloze ques- tions as input for the encoder-decoder models in the format of "(sent [OPT][ | (opta,)
hhttps://platform. http://pllatffr...ppae... wwwnai.ccom/ docs//odelsm hhttps://hugiingfacc co/ //docs/evallate/idees

---

be found in Appendix D.
ClozEx task, we will discuss the micro-averaged Pearson correlation coefficient between manual and automatic scores in Section 5.2.
4.2 Result
The evaluation results are presented in Table 4. With regard to the manual metrics, the encoder- decoder models generally exhibited the ability to generate fluent and valid explanations, except for T5-base. BART-large achieved the highest level of validity performance. By contrast, the decoder- only models based on GPT-2 produced acceptably fluent texts but did not effectively explain the ques- tions. Across all the fine-tuned models, the size of the model did not have a substantial impact on performance, except for T5 base and large, where it hindered the generation of more valid explanations. Although LLMs are capable of generating text with acceptable fluency thanks the large amount of pre-training data, they received low evaluations in terms of producing valid explanations. This high- lights the ongoing challenge of using LLMS to gen- crate cloze question explanations for LA, without mentioning the generation of a dataset specifically tailored for the ClozEx task. A detailed discussion regarding the performance of LLMs is included in Section 5.1.
5 Discussion
5.1 Do LLMs Explain Cloze Questions Well?
Given the remarkable performance of LLMs across various tasks without fine-tuning (Liu et al., 2023), there is reasonable expectation that they would excel in generating high-quality explanations for cloze questions. However, our experimental find- ings indicate that no LLM achieved an acceptable validity score in manual evaluation. Upon analyz- ing the explanations generated by we identified two critical shortcomings of LLMs in effectively explaining cloze questions.
Firstly, LLMs exhibit tendency generate fac- tual errors, thereby failing to ensure the accuracy of the generated texts. This deficiency exemplified in LLM-GPTT3.5turbe Question Appendix E, where an evident error is observed in the verb tense following the word "did not," a discrepancy that can have detrimental consequences in the context of LA.
Secondly, LLMs have the propensity to produce explanations that lack meaningful and informative content, failing to provide the necessary knowledge required for comprehending the reasons behind the answer options. As illustrated by LLM-GPT3.5- turbo Question 2 in Appendix E, such explana- tions leave ESL learners unaware of why the given answer option is necessary, while also failing clu- cidate the distinctions among the options resulting from affixes. Furthermore, these explanations may even present incorrect answers and flawed analyses. further diminishing their utility.
With regard to the automatic metrics, the BLEU score exhibited strong correlation with manual fluency and validity scores when evaluating mod- els fine-tuned with our training data. However, because the LLMs did not learn the distribution from our training data, the generated text varied from the reference. Because a good explanation for a cloze question is not unique, reference- metrics should focus on evaluating models trained with our data. In this regard, BART-large achieved the best performance once again.
The GPT evaluator demonstrated stability in terms of fluency. These GPT-Fluency scores showed a positive correlation with manual fluency scores. However, in terms of validity, the GPT eval- uator was less consistent, assigning varying scores to models that received similar validity scores from human annotators (such BaRT-base, BART- large, and T5-large). Notably, L.MM-GPT3.5-urbb was highlighted, because the GPT evaluator exhib- ited more leniency toward it than human annota- tors.
5.2 Are Automatic Metrics Reliable in ClozEx?
The evaluation of automatic metrics, specifically BLEU and GPT-Fluency scores, aligns with the trends observed in manual evaluation scores (Sec- tion 4.2). To ascertain the reliability of these met- rics in reflecting the quality of generated explana- tions, we computed the micro-averaged Pearson correlation coefficient between manual and auto- matic evaluation scores.
Finally, although these automatic scores showed some correlation with human evaluation, they were calculated under the macro average. To determine the reliability of these automatic metrics in the
As shown in Table the BLEU score is largely independent of the manual fluency score. How- ever, when excluding explanations generated by LLMs, the BLEU score exhibits a moderate cor-

---

Manual
Automatic GPT-Fluency
Fluency
Validity
BLEU
GPT--alidity
BART-base
4.13
4.38
25.64 25.53
4.88
3.75
BART-large
4.11
4.43
27.33 27.01
4.84
2.90
T5-base
2.03
1.52
7.62 77.59
2.53
1.32
T5-large
3.99
4.26
22.70 /22.68
4.95
2.31
GPT2
3.87
2.78
15,40 / 15.41
4.03
1.77
GPT2-medium
3.91
1.85
16.85 / 16.84
4.16
2.03
LLM-GGPT--arge
3.97
1.73
0.50 / 0.51
3.94
1.58
LLM-GPTT-XLL
3.97
1.70
0.60 0.60
4.00
1.58
L.MM-GPT3.5-turb
4.53
2.70
1.39 1.34
4.93
4.87
LLM-LLaMa-7B 3.81
1.78
1.06 1.08
3.81
1.44
Table Performance of baseline models. BLEU scores are based on dev. and test sets, respectively. evaluation metrics, GPT-Fluency and GPT-Validity indicate fluency and validity estimation, respectively, using GPT3.5-urbo. Prefix LLM- denotes -LM-ggeertted explanations. Except for BLEU, all scores are ranged in [],
Man. Fluency
Man. Validity
Conclusion
BLEU
0.04 0.11
0.08 / 0.11
w/o LLMs
0.39 / 0.43
0.44 / 0.47
GPT-Fluency
0.57 / 0.61
This paper introduced a novel task, ClozEx, aimed at generating fluent and valid explanations for En- glish cloze questions to support ESL learning.
GPT-Validity
0.03 0.05
Table 5: Pearson correlation coefficient between man- ual and automatic evaluation scores. The automatic scores yielded two correlated coefficients because each instance is assessed by two annotators.
We curated a comprehensive dataset compris- ing more than 140k instances of cloze questions paired with explanations. The dataset is created by pattern-based method. Patterns extracted from expert-designed cloze questions and explanations ensure the quality of generated questions and ex- planations. Expert evaluations confirmed the high quality and suitability of our dataset for the ClozEx task.
relation with the manual fluency score. The valid- ity correlation reported similar tendency. As reference-based metric, BLEU demonstrates limi- tations in recognizing explanations with different styles from our dataset, implying that low BLEU score does not necessarily indicate poor expla- nation. However, due the high quality our dataset, an explanation with high BLEU score can generally be considered good.
To address this task, fine-tuned various mod- els, including encoder-decoder and decoder-only architectures, generate explanations for the pro- vided questions. Additionally, we investigated the potential of LLMs to produce explanations in a zero-shot scenario. The experimental results high- lighted the capability of the encoder- mod- els to generate high-quality explanations. However. although LLMs excelled in generating fluent texts, they struggled to produce valid explanations. Thus, we analyzed the limitations of LLMs in generat- ing satisfactory explanations without fine-tuning, shedding light on the challenges they face in this context.
As reference-free metric, GPT-Fluency hibits strong correlation with manual fluency scores, even when considering LLM explanations. Unlike the correlation observed between GPT- Fluency and Manual Fluency, GPT-Validing fails to effectively reflect the manual validity score. Furthermore, for explanations generated by LLM- GPT3.5-turbo, as mentioned in Section 5.1, GPT- Validity tends to assign higher scores. In light of these findings, when reference-free evaluation is conducted, it is acceptable to employ LLMS such as GPT3.5-turbo to assess fluency in the ClozEx task. However, using LLMs to evaluate validity is not recommended.
Additionally, we explored the correlation be- tween manual and automatic evaluation metrics, discovering that automatic metrics exhibited some degree of reliability for the ClozEx task.

---

Limitations
the limitations of LLMs when it comes effec- tively clucidating cloze questions. Overcoming these challenges will necessitate the exploration of novel methodologies and strategies, such as in- corporating external grammatical knowledge, to enhance the ability of LLMs to generate explana- tions that are precise, informative, and contextually appropriate.
One limitation of study the ClozEx task, designed support ESL learning, is that our ex- periment did not investigate its effectiveness in im- proving language proficiency. Although the expert estimation of our dataset yielded positive results, it serves only as indirect evidence that the expla- nations contained within can aid ESL learning. To obtain direct evidence, further experiments are re- quired. For instance, conducting study where English proficiency of ESL learners is assessed before and after exposure to a batch of questions and explanations from our dataset would allow us to observe whether these materials contribute to proficiency enhancement.
A Acknowledgements
This work was supported by JST, the establishment of university fellowships towards the creation of science technology innovation, Grant Number JP- MJFS2139.
Another limitation of our study pertains to the question types included in the dataset. Initially, we constructed the dataset by employing pattern extraction methods that focused on three specific question types. However, is important to note that language assessment encompasses a wide range of question types. For instance, there are questions that require learners to identify the meanings of content words assess the usage of pronouns, con- junctions, and other linguistic elements. The pat- tern extraction methods utilized in our study were tailored address specific question types, which may limit the coverage of our dataset. To expand the scope of our dataset, future efforts would en- tail devising new methods to extract patterns from these specific question types.
References
Charles Alderson. 1979. The cloze procedure and proficiency in English as a foreign language. TESOL quarterly, pages 219-227.
ALTE. 2011. Manual for Language Test Development and Examining: For Use with the CEFR. Language Policy division, Council of Europe.
Lyle F Bachman. 1985. Performance on cloze tests with fixed-ratio and rational deletions. TESOL Quarterly, 19(3):535-556.
Martin Chodorow, Michael Gamon, and Joel Tetreault. 2010. The utility article and preposition error cor- rection systems for English language learners: Feed- back and assessment. Language Testing, 27(3):419- 436.
The third limitation pertains to the automatic evaluation metrics employed. Although we ob- served positive correlation between BLEU and GPT-Fluency scores and manual evaluation scores, certain issues arise when these metrics are utilized. BLEU, being reference-based encounters difficul- ties when confronted with situations where good explanations are not unique. Although reference- free metrics like LLMs, such GPT-Fluency, offer an alternative, their reliability is not always guar- anteed (Wang et al., 2023b). Additionally, mod- els such as GPT3.5-turbo which served as the backbone for GPT evaluators in our study, are not open-source, posing potential obstacles for future research endeavors.
Laura Collins. 2007. L1 differences and L2 similari- ties: teaching verb tenses in English. ELT Journal, 61(4)29953333
Rui Correia, Jorge Baptista, Maxine Eskenazi, and Nuno Mamede. 2012. Automatic generation of cloze ques- tion stems. In International Conference Computa- tional Processing of the Portuguese Language, pages 168-178. Springer.
Jennifer Cotter. 2012. Understanding relationship between reading fluency and reading comprehension: Flueney strategies focus for instruction.
Charles B. Cross. 1991, Explanation and the theory of questions. Erkenntnis (1975-), 34(2):237-25 (41))2)
Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. Multi-news: A large-scale multi- doctmed summarization dataset and abstrac- tive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1074-1084, Florence, Italy, Asso- ciation for Computational Linguistics.
Furthermore, despite demonstrating proficiency in paraphrasing explanations during the dataset cre- ation phase, I.LMs proved inadequate in generat- ing explanations for cloze questions without any prior information. These observations underscore

---

Eileen W Glisan and Victor Drescher. 1993. Textbook grammar: Does it reflect native speaker speech? The Modern Language Journal, 77(1):33-33.
Masamichi Mochizuki and Kazumi Aizawa. 2000, An aftix acquisition order for EFL learners: an ex- ploratory study. System, 28(2):291-304
Hongyu Gong, Jiaqi Mu, Suma Bhat, and Pramod Viswanath. 2018. Preposition sense disambiguation and representation. In Proceedings of the 2018 Con- ference Empirical Methods in Natural Language Processing, pages 1510-1521, Brussels, Belgium. Association for Computational Linguistics.
Ryo Nagata. 2019. Toward task of feedback comment generation for writing learning. In Proceedings of the 2019 Conference on Empirical Methods in Natu- rul Langwage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JCCNL",, pages 3206-3215, Hong Kong, China. Association for Computational Linguistics.
Takuya Goto, Tomoko Kojiri, Toyohide Watanabe, To- moharu Iwata, and Takeshi Yamada. 2010. Auto- matic generation system of multiple-choice cloze questions and its evaluation. Knowledge Man- agement & E-Learning: An International Journal, 2(3):210-224.
Subhadarshi Panda, Frank Palma Gomez, Michael Flor, and Alla Rozovskaya. 2022. Automatic genera- tion of distractors for fill-nntthe-llank exercises with round-trip neural machine translation. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics: Student Research Work- shop, pages 391-401 Dublin, Ireland. Association for Computational Linguistics.
Felix Hamborg, Norman Meuschke, Corinna Breitinger, and Bela Gipp. 2017. news-please: generic news crawler and extractor. In Proceedings of the 15th In- ternational Symposium of Information Science, pages 218-223.
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing 7hu. 2002. Bleu: a method for automatic evalu- ation of machine tramslation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 31 1-3188, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Jennifer Hill and Rahul Simha. 2016. Automatic gen- eration of context-based fll-in-the-blaak exercises using co-occurrence likelihoods and Google n-grams. In Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications, pages 33.30.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Zhenghao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Tex Kato. 2017. TOEIC L&R tesulo bunpo mondai deru 1000mon (TOEIC L&R Test Grammar Problems 1000 Questions). ASK Publishing.
Christine Klein-Braley. 1997. C-tests in the context of reduced redundancy testing: An appraisal. Language testing, 14(1):47-84
Patricia M Raymond. 1988. Close procedure in the teaching of reading. TESL Canada journal, pages 91-97.
Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Twasawa. 2022. Large lan- guage models are zero-shot reasoners. In Advances in Neural Information Processing Systems, volume 35, pages 22199-22213 Curran Associates, Inc.
James Rye. 1982 Cloze procedure and the teaching of reading. London.
Keisuke Sakaguchi, Yuki Arase, and Mamoru Komachi. 2013. Discriminative approach to fill-n-the-bllant quiz generation for language learners. In Proceed- ings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pa- pers), pages 238-242, Sofia, Bulgaria. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-seequence pre-training for natural language generation, tramslation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computationnal Linguistics, pages 7871-7880, Online. Association for Computa- tional Linguistics.
Vivek Srikumar and Dan Roth. 2013. Modeling mantic relations expressed by prepositions. Transac- tions of the Association for Computational Linguis- tics, 1:231-242.
Zhengliang Liu, Xiaowei Yu, Lu Zhang, Zihao Wu. Chao Cao, Haixing Dai, Lin Zhao, Wei Liu, Ding- gang Shen, Quanzheng Li, Tianming Liu, Dajiang Zhu, and Xiang Li. 2023. Deid-gpt: Zero-shot medi- cal text de-identification by gpt-4.
Wilson L Taylor. 1953, "Cloze procedure": A new tool for measuring readability. Journalism quarterly. 30(4):415-433.

---

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Fric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and efficient foundation language models.
Affix Question:
Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023a, ChatGPT good NLG evaluator? preliminary study.
As expected. the infectious period had a clear relationship with mean offspring num-
ber.
(A) positive (B) positively (C) positives (D)
Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin. Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023b. Large language models are not fair evalua- tors.
positivity
Explanation:
For this cloze question, you need to choose an adjective that modifies the word "rclationship." Option (A) "positive" is an adjective, while options (B), (C), and (D) are not adjectives. Therefore, the correct answer is option (A), "positive."
Joseph Jay Williams, Tania Lombrozo, and Bob Rehder. 2010. Why does explaining help learning? insight from explanation impairment effect. In Proceed- ings of the Annual Meeting of the Cognitive Science Society, volume 32.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text clas- sification. In Advances Neural Information Pro- cexsing Systems, volume 28. Curran Associates, Inc.
Verb Tense Question:
The couple were Saturday on disorderly conduct charges by officers investigating family dispute at their home in New Canaan, Conn.
A Examples of Questions in Dataset
(A) arresting (B) arrested (C) arrest (D) arrests
B Details of Dataset Creation
Explanation:
Patterns for affix/tense questions were extracted from a published TOEIC practice book (Kato, 2017). A total of 231 patterns were extracted from 432 affix questions, while 99 patterns were ex- tracted from 219 tense questions. For preposition questions, focused 34 prepositions used in the PSD dataset (Gong al., 2018) question patterns.
The blank in the cloze question requires a past participle verb, as indicated by the pas- sive auxiliary "were" and the passive nominal subject "couple." The options given are: (A) arresting (gerund or present participle), (B) ar- rested (past participle)
(C) arrest (base form), and (D) arrests (3rd person singular present). Based on this information, the correct answer is option (B), "arrested."
To generate new questions and explana- tions, we selected the news (Zhang et al., 2015), ce_news (Hamborg et al., 2017), and multi news (Fabbri et al., 2019) corpora from the public news corpus.
Preposition Question:
The work was done the Rehabilitation Institute of Chicago under an $8-million grant from the Army.
In the process of creating preposition ques- tions, employed BERT-PSD to identify the pattern present in each given sentence. Although BERT-PSD a tfate-of-the-ee model in the PSD task, it achieved an accuracy of only 90.84 80.84, lead- ing to potential noise in the dataset. To address this, we set a threshold of 0.8 for the model's prediction confidence. If the model predicted the pattern of a sentence with confidence equal to or higher than 0.8, we retained the sentence along with its pattern for producing tions. Otherwise, the sentence was discarded. With
(A) (B) down (C) round (D) of
Explanation:
Based on the context of the sentence, option (A) "at" is the appropriate choice for the cloze question. "At" indicates a specific location or arrival at particular place position, such as "he is at the store."
Table 6: Examples of questions with different types.
questions and explana-
hhttps://githhi cow/dirknnuuaeuserr preposition-sett sewpesittioo-sssediiiiii waannseddisamiiiieeeee dimbbiguttoniiiti

---

this threshold, the prediction accuracy improved to 97.78%.
3=Borderline: 'hhe explanation fell between unacceptable and acceptable fluency.
For creating distractor options in affix questions, we prepared a distractor candidate dictionary in advance. We collected words from English dic- tionary website' that share the same root but have different prefixes or suffixes. A similar process was followed for tense questions, where the distractor candidate dictionary focused specifically on verbs and their various tense forms. In the case of prepo- sition questions, the distractor candidate dictionary was created based on preposition semantic rela- tions (Srikumar and Roth, 2013). Prepositions that share the same semantic relations are considered distractor options for each other.
A=Acepptabl:: The explanation was clear and understandable, but with room for improve- ment.
5=Good: The explanation was fluent and easy to understand.
Validity. You are given instances of English fill- in-the-blank questions with corresponding expla- nations. Your task is estimate whether the ex- planation explains the question well. For batch, you need to estimate 45 instances. You need to estimate the validity using a 5-scale metric to score the explanation. The ratings are as follows:
To avoid ambiguous questions that have multi- ple correct answers, we utilized a GPT2-based LM a distractor option obtained a higher LM score than the answer, as determined by the scorer, the option was discarded. The templates used for generating initial explanations for ques- tions shown Table 7. These initial explana- tions were further paraphrased using GPT3.5-turbo. The prompt for paraphrasing provided in Ap- pendix D.
scorer 8
1=Bad: The explanation included factual er- rors or was unrelated to the question.
2 2=Unacceptable: The explanation related the question but provided knowledge that did not contribute to answering
3=Borderline: The explanation fell between unacceptable and acceptable validity.
Details of Manual Quality Estimation
Human evaluators were tasked with rating the qual- ity of generated explanations from each method in terms of fluency and validity using 1-5 scale. The following instructions and criteria were provided guide their ratings:
4=Aceeptable: The explanation provided some necessary knowledge for answering the question, but there were still some missing elements.
5=Good: The explanation provided sufficient language knowledge answer the question.
Fluency. You are given instances of English fill- in-the-blank questions with corresponding explana- tions. Your task is to estimate whether the expla- nation is fluent in English. For batch, you need estimate 45 instances. You need estimate the fluency using a 5-scale metric to score the explana- tion, and you do not need to identify whether the explanation explains the question correctly, please just focus on its fluency. The ratings are as follows:
The annotation approved by the ethical com- mittee in the authors' university prior conduct- ing this research (approval number: H21-041). All annotators were paid about $13.5 for every 45 stances (which takes about hour), while the mini- mum wage locally is about $7.5.
D Used Prompts
The prompt we used to paraphrase initial explana- tions with GPT3.5-turbo (the parameter of OpenAl API) is:
1=Bad: The explanation was unreadable.
2=Unaccepttalee The explanation was disflu- ent.
messages=[
Phttpss///ww, www.vvooabuarr.ccoo. "https://githur com/ www.oomrerimmmmeeem
'role' "system', 'content': 'You are an English teacher.))

---

Affix/tense
The word in the blank should be (RELATION _O__TTORET_TOOOO BONSWWEEFFANWWRRR ANSWER) "(PARENT OF ANSWERY.. is (RELATION CEATTINNTT__BST TO_CHILD_OO OF OF_ANSWERI Thus, [POS OF LANSANSEE
of "(CHILD. "AYSWERY" of the blank. is required. (OPTION; is [POS_OF SF_OOFOOO -FFOOPTION:). Therefore, the correct answer is (ANSWW ER). According the meaning of this sentence the option (ANSW ER) is suitable, which means "ISENSE_A OF SANSFANNSWEE E
Prep.
Table 7: Templates used for generate initial explanations.
{ ('role": 'user", 'content': fPParapraase the following explanation of cloze ques-
room for improvemeet.lSS--Codd::: The explanation fluent and easy to understand AnThe input is: [ezp)")
tion within 128 words: (crp)")
The prompt we used in training/inference with LLM-GPT2 family and LLM-LLaMa-7B is:
In GPT-Validity, the prompt is:
messages=[
Question: (sent) AnOptions: (A)
I'role': 'system', 'content': 'You are
[opti) (B) (opt2) (C) [opt3) (D)
linguistics expert.),
[opl4] /Nte://xttaaatoom(eer))
('role': 'user', 'content': f'Esstimtte whether the given explanation could explain the cloze question well.nHere is the score definition:lnI=Badd The explanation included factual errors or was unrelated to the ques- Non.in2=Unacceptable The explanation related to the question but provided knowledge that did contribute to answering B.33ooorrene: The expla- nation fell between unacceptable and acceptable validity.nA=Accepabbbe: The explanation provided some necessary knowledge for answering the question, but there were still some missing clements. An5=oodd: The explanation provided sufficient language knowledge answer the question.An An'Te input question is: [q]\nThe explanation is: {exp}'}
where eap is empty in the inference phase. The prompt we used in inference with GPT3.5- turbo is:
m
{ ('role': 'system', 'content': 'You are an English teacher:),
'user', 'content': f'Generate an explanation of the following cloze ques- tion: (sent) (NOppiioss))))) (opt1] (B) { (C) (opts) (D) {opta}'}
('role':
For GPT-evaluators, the prompt we used in GPT-
Fluency is:
messages=[
{ ('role':
'system', 'content'; 'You are
English teacher:).
('role': 'user', 'content': f'Estimate whether the given text is fnuee is the score definition::II-Bad:: The explanation was unread- able.In2=Unacceptaale The expla- nation was disfluent. in3=Borderline: The explanation fell between un- acceptable and acceptable fluency. An4=Acceptable: The explanation was clear and understandable, but with
E
Examples of Generated Explanations
BART
The repayment, which set- tles suit brought under international legal tribunal, is billions of dollars in frozen foreign ac- counts that Iran can now access after the end of nuclear sanctions.
Question:
from the tens of

---

(A) separates (B) separability (C) sepa- rateness (D) separate
Explanation (GPT2): Based on the def- inition provided, option (b) "down" is the appropriate choice for the blank in this sentence. This word indicates move- ment towards a lower position or level, as in the example "he descended the hill." This word can also be used to describe the action of
Explanation (BART-base): The blank in the cloze question is filled by the cop- ula "is". This means that adjective is needed complete the sentence. The options given include a verb in the third person singular present tense, singular or mass noun, an adjective, and another adjective. The correct answer is the ad- jective "separatc".
Explanation (GPT2-medium): The cor- rect answer for the cloze question is op- tion (b) down, which refers to the action of moving towards a lower point or level, such when someone goes down a hill. This is based on the sentence's meaning. For example, "he
Explanation (BART-larre): The blank in the cloze question is filled by the cop- ula "is". This means that an adjective needed to complete the sentence. The op- tions given are: (A) separates, which is verb in the 3rd person singular present tense; (B) separability, singular mass noun; (C) separateness, also noun in the same category; and (D) separate, an ad- jective. Based on this information, the correct answer is option (D), which "separate".
LLMs (except GPT3.5-turbo)
The repayment, which set- tles a suit brought under an international legal tribunal, is from the tens of billions of dollars in frozen foreign ac- counts that Iran
Question:
access after the end of nuclear sanctions.
(A) separates (B) separability (C) sepa- rateness (D) separate
T5
Question: South Korean shares slid on Friday, with foreign investors their biggest daily volume in nearly 20 months, hurt after aggressive easing from the Bank of Japan sent the yen reeling, well by the tension over North Korea.
Explanation (LLM-GPTT-large)) The first option is separate the accounts from the rest of the assets. This would allow the U.S. to keep the money in the accounts, but not the assets. The second option is to separate the accounts from the rest of the assets. This would allow the U
(A) selling (B) undersells (C) resell (D) sell
Explanation (T5-base): The stock market slid on Friday, with investors
Explanation (LLM-GPT2-XL); Sepa-
ration is the most common option. Sepa-
ration is the most common option. Sepa-
ration is the most common option. Sepa-
Explanation (T5-large): The correct an- swer for the cloze question (A) selling, which is verb the form of gerund or present participle. This is indicated by the fact that the word "selling" can be used as a verb in this context.
ration is the most common option. Sepa-
ration is the most common option. Sepa-
ration is the most common option. Sepa- ration is the most common option. Sepa- ration is the most common option. Sepa- ration is the most common option. Sepa- ration is the most common option.
GPT2
Explanation (LLM-LLaMa-7BB) The repayment, which settles suit brought under international legal tribunal, is from the tens of billions of dollars in frozen foreign accounts that Iran can now access after the end of nuclear sanctions.
The father of the former Meghan Markle says he wishes he could have walked her her wedding to Prince Harry.
Question:
the aisle during
(A) with (B) down (C) as (D) for

---

The repayment, which settles suit brought under an international le- gal tribunal, is from the tens of billions of dollars in frozen foreign accounts that Iran can now access after the end of nu- clear sanctions
Question:
LLM-GGTT.5-turbo
Question Isn't it just that didn't t
to sell the sets had produced?
(A) manage (B) manages (C) managing
(D) managed
The correct answer is (D) managed. The sentence is in the past tense and "did not" is the negative form of the auxiliary verb "did". Therefore, the next verb has to be in the past tense as well. The verb "managed" is the cor- rect form the past tense, meaning "suc- ceeded doing something despite dif- ficulties or obstacles". Example: "The company didn't 't manage to sell all of its products before the end of the year."
Explanation 1:
Question The innards of the machine were complex but
(A) manageable (B) manageably (C) un- manageably (D) mismanage
Explanation 2: The correct option (C) unmanageably. The word "innards" means the internal parts of a machine. The sentence suggests that the parts of the machine were complex and difficult to manage. The word "manageably" (op- tion B) means something that is able to be managed easily which contradicts the sentence. The word "mismanage" (option D) means to handle or operate wrongly or inefficiently, which does not fit in the sentence. Therefore, the correct answer is (C) unmanageably, meaning that the innards of the machine were dif- ficult to manage.


Reviews:
[{"review_id": "tPad9azgBB", "reviewer": "Reviewer_SEmd", "summary": "", "questions": "Line 490, \"Despite the large amount of training data...\" - why would acceptable fluency be in spite of training data? Would that help?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed feedback. \n\nIn Line 490, we meant despite the LLMs could yield fluent text, generated text is not suitable for explaining the question. In other words, the LLMs we used are all pre-trained with a large amount of data and have the capability to generate fluent text for any given instruction without fine-tuning. However, in spite of the generated text being fluent, it is not valid as an explanation.\n\nWe apologize for the ambiguous expression, we will modify it with a clear expression like: “Although LLMs are capable of generating text with acceptable fluency thanks to the large amount of pre-training data, they received low evaluations in terms of producing valid explanations.”", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The application is important and pertinent. The task of generating explanations for cloze tests would benefit language learners. It is apparently difficult enough that current models cannot master it, but conceivably could in the near future.\n- A new dataset is introduced. Though it is not manually generated, it is based on manually generated data and uses reasonable data augmentation techniques.\n- The methods for creating the dataset are interesting. They combine crafted rules incorporating linguistics and syntactic parsing but also use large language models in subtle and appropriate ways.\n- Benchmark systems are rigorously evaluated, including by humans. Current relevant models are tried, including fine-tuned sequence-to-sequence and generative models and zero-shot instruction-following models. Large language models are also employed for evaluation, but with appropriate skepticism and validation, leading to further interesting results for their use in evaluation.\n", "reasons_to_reject": "None.", "questions_to_authors": "Line 490, \"Despite the large amount of training data...\" - why would acceptable fluency be in spite of training data? Would that help?", "paper_topic_and_main_contributions": "The paper concerns the task of generating explanations for answers to multiple choice cloze questions, especially those aimed at language learners, which tend to have grammar-based distractors and explanations. It introduces an automatically generated dataset based on extracting question and explanation patterns from existing tests, applying these patterns to unlabeled text using syntactic parsing, and diversifying the language by using large language models to paraphrase. ", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3uvz5HwGfY", "reviewer": "Reviewer_xPm4", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful feedback. Here are responses to your questions:\n\n1. To Reasons To Reject: “To my knowledge...”.\n\n    Thank you for raising this point. We agree with you that while evaluating the model performance, manual evaluation is, of course, the most accurate metric. However, the manual evaluation is costly and lacks reproducibility, so, in addition to the manual evaluation,  we explored whether automatic metrics could represent the model performance as well.  It is not obvious to what extent BLEUs are reliable in this generation task, and we consider it significant and a contribution of this study to have clarified this point.\n    The result in Section 5.2 demonstrated that reference-based metric has indeed its own limitations and could only be performed in specific conditions.\n\n\n2. To Reasons To Reject: “In lines 439-449...”.\n\n    Thank you for taking your time to test some cases for our task! Because our paper is going to propose a new task and dataset, under the condition of limited pages, we prefer leaving more space for introducing the task and dataset, as well as analysis of several baseline models to provide insights to future work.\n\n    Another reason that we did not fine-tune LLMs was because we wanted to investigate whether LLMs could tackle the ClozEx task well in a zero-shot scenario --- if LLMs could do so  without any external data, the ClozEx training set is meaningless (i.e., only the dev/test sets are useful). Therefore, LLMs in our experiment were not expected to be compared with fine-tuned models.\n\n    You mentioned that ChatGPT could produce acceptable explanations in your experiment. We are interested in understanding the discrepancy between your experiment and ours. However, because we are not familiar with your experiment setting, it is not easy to answer what caused the inconsistency. Could you kindly share your experimental settings (actual prompts) and examples (if not from our data)? Constructive discussions around such differences can lead to valuable insights. Here we would like to share some of our opinions (just FYI, our zero-shot prompts are included in Appendix D).\n\n    In our experiment,  GPT3.5-turbo did not perform as poorly as other LLMs and obtained valid scores near the borderline. Furthermore, as indicated in Section 5.1,  GPT3.5-turbo suffered from two typical problems. We attach some examples of questions at the end of this response. These questions are from the ClozEx dataset and cause poor explanations when using GPT3.5-turbo API (results are reproducible because we set Temperature=0).\n\n    As for low BLEU scores of LLMs, as we indicated in Section 5.2 and Limitations, reference-based metrics are not appropriate to evaluate explanations from zero-shot generation, which is why LLMs generally did not reach high BLEU scores. We listed the BLEU scores of LLMs to demonstrate it is not suitable in such cases.\n\n\n3. To Reasons To Reject: “In Table 4, the authors use GPT-model...”.\n\n    Thank you for bringing our attention to the manual evaluation. As you pointed out, the number of manual evaluations was limited to 100 samples. However, we believe that the number was sufficient because inter-annotator agreements for fluency and validity were 0.82 and 0.77, respectively (statistically significant, Table 2). \n\n    We did not claim whether an automatic metric is convincing before calculating the correlation with the manual metric. To reduce the cost and improve the reproducibility of model evaluation, we explored whether automatic metrics are consistent with manual ones --- if automatic metrics can represent the model performance as well as manual metrics, future work will not have to compare models by manual annotation. In Section 5.2, the correlation indicated that not all automatic metrics are reliable in evaluating ClozEx methods, some are totally unusable and some can only work on limited conditions. We kindly ask the reviewer to note that only GPT3.5-turbo is using the same model with GPT-Fluency and GPT-Validity (although there may be overlap between the training dataset between GPT-based models).\n\n4. To Reasons To Reject: “Secondly, as shown in section 3.2...”.\n\n    Our questions are generated by expert-designed questions, which cover various grammatical items with different difficulties. We extract patterns from such questions and use extracted patterns in turn to generate questions, so generated questions could be regarded as an expansion of expert-designed questions. Instances in our dataset are from fixed patterns, but question sentences and options are selected randomly in limited conditions from three large corpora, which could avoid the domain dependency. Nonetheless, thanks for suggesting an experiment for additional experiments and we would love to include statistics showing the variety of the created dataset. In the meantime, we would like the reviewers to download our data accompanied with this submission.\n\n5. To reasons To Reject: “Thirdly, the paper constructs a dataset...”.\n\n    The pipeline of creating the dataset requires both expert-designed questions and explanations. It is hard to generate explanations using the pipeline by only providing questions. Therefore, the ClozEx task requires models to generate explanations with only questions as input.\n\nWe are sorry for the misunderstanding due to our unclear writing. We would modify the draft by emphasizing the above items.\n\n-----\n\nExamples of questions leading to poor explanation (Using GPT3.5-turbo API; Temperature=0; prompt is indicated in Appendix D in our paper; verified on 24/08/2023.)\n\nExample 1:\n> Question: The protective netting in place to protect fans from falling concrete at Wrigley Field must be replaced with a permanent solution if the Chicago Cubs _______ to play at the stadium next spring.\n>\n> Options: (A) wanting  (B) want  (C) wanter  (D) wants\n\nExample 2:\n> Question: Isn't it just that it didn't _______ to sell the sets it had produced?\n>\n> Options: (A) manage (B) manages (C) managing (D) managed\n\n\nExample 3:\n> Question: Please complete the _______ below with your enquiry and a member of our team will contact you shortly.\n> \n> Options: (A) reformists (B) reforms (C) deformed (D) form", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The task about Expalanation generation is interensting.\n2. The paper propose a dataset about explanation generation.", "reasons_to_reject": "At first, the experimental part of this paper is extremely imprecise:\n1. To my knowledge, explanation generation task is special so the BLEU metric is unsuitable. For the exmaple in figure 1, the explanation \"Based on the context of the sentence, option A, option B, option C, so the answer is \" can obtain high BLEU score, but this explanation is unhelpful. For this reason, all the BLEU in Table 4 is meaningless. For example, we can see T5-base only obtain 7.59 of BLEU score, but BART-base can obtain 25.53.\n2. In lines 439-449, the authors claim that they do not fine-tune the LLMs and compare the results with other baselines. I think it is so unfair comparisions, which makes me very curious about the actual performance of LLMs. So I test some cases about cloze explanation in ChatGPT. In fact, ChatGPT can provide good explanations for most cases. I think show a so low BLEU score in Table 4 for LLMs but lack details is a serious problem for this paper.\n3. In Table 4, the authors use GPT-model as a metric to evaluate the models including many GPT-based models, so these two metrics are unconvincing.\nSo all the results in Table 4 are meaningless except human evaluation, but human evaluation only test 100 samples as describe in line 454.\n\nSecondly, as shown in section 3.2, all the explanations and questions are generated by template, which are far away from real ESL education. One of the most prominent problems is the lack of difficulty. Although the paper provide a large-scale dataset including 140k samples, there is a high possibility that it contains a large number of similar samples. I think the authors should conduct some experiments  about this problem.\n\nThirdly, the paper constructs a dataset by constructing a corpus through the template and then polishing it with LLMs. We can see that the authors have been provided the pipeline to solve ClozEx dataset. It is meaningless to generate explanations by a end-to-end method for these data. \n\nI think it should at least provide a dataset where all questions and explanations are written by humans to be meaningful for research.\n\nFurthermore, the authors write so many text in section Limitations. Although the reviewing guidelines suggest me that I could not reject the paper base on the limitations. But I think I must put forward my point of view. If a method is not good, what researchers should do is not to use it and explore better methods. Rather than saying that my work uses this method, I admit that it is not good but it has nothing to do with my work. The method you use is also part of your work.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper propose a contribution about English Cloze Explanation. The authors propose a method to build explanation dataset and conduct some experiments showing that the explanation generation task is challenge.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8F70M6b2aJ", "reviewer": "Reviewer_zBsN", "summary": "", "questions": "1. Are the same annotators used for evaluating the LLM generated explanations at the dataset explanations? (And is there a time gap between those 2 annotations or were they annotated at the same time)\n2. Also, were the annotators paid for the annotators as part of their salary, paid for completing the annotations or volunteers?\n(You may want to add an ethics statement discussing pay and other ethics consideration of the annotations)", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and the concern about the ethics risk.\n\nIn the two phases of annotation, we hired the same six annotators, but there was an obligatory time gap between the two annotations. There were about two weeks between two annotations. During that period we needed to train baseline models and obtain results for the next annotation process.\n\nBecause all instances in annotation were automatically generated, and we did not leak them to annotators, there was no chance for the annotators to see the instances before. Instruction is listed at the end of our response.\n\nWe have considered using MTurk to perform the annotation, but we could not identify the English level of a learner. Hiring English learners from the university could solve it --- we checked annotators' English test certificates (e.g., IELTS, TOEFL, TOEIC) physically to make sure they have CEFR A2 level or higher.\n\nThe annotation was approved by the ethical committee in the authors' university prior to conducting this research. All annotators were paid about 13.5 USD for every 45 instances (takes about 1 hour), and the minimum wage locally is about 7.5 USD.\n\nWe appreciate your thoughtful consideration, and we will add the above points to our paper.\n\n-----\n\nInstruction of fluency:\nYou are given instances of English fill-in-the-blank questions with corresponding explanations (See example in Table 1). Your task is to estimate whether the explanation is fluent in English. For a batch, you need to estimate 45 instances. The reward for each batch is an Amazon gift card of \\$13.5.\n\nTable 1(Omitted)\n\nYou need to estimate the fluency using a 5-scale metric to score the explanation, and you don’t need to identify whether the explanation explains the question correctly, please just focus on its fluency.\n\nThe 5 (the best) represents the explanation was fluent and easy to understand.\n\nThe 4 represent the explanation was clear and understandable, but with room for improvement.\n\nThe 3 represent the explanation that falls between unacceptable and acceptable fluency.\n\nThe 2 represent the explanation was disfluent.\n\nThe 1 (the worst) represents the explanation was unreadable.\n\n----- \n\nInstruction of validity:\nYou are given instances of English fill-in-the-blank questions with corresponding explanations (See example in Table 2). Your task is to estimate whether the explanation can explain the question well. For a batch, you need to estimate 45 instances. The reward for each batch is an Amazon gift card of $13.5. \n\nTable 2 (Omitted)\n\nYou need to estimate the validity using a 5-scale metric to score the explanation.\n\nThe 5 (the best) represents the explanation that provided sufficient language knowledge to answer the question.\n\nThe 4 represents the explanation that provided some necessary knowledge for answering the question, but there were still some missing elements.\n\nThe 3 represents the explanation that falls between unacceptable and acceptable validity.\n\nThe 2 represents the explanation that was related to the question but provided knowledge that did not contribute to answering it.\n\nThe 1 (the worst) represents the explanation that included factual errors or was unrelated to the question.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The task may and be useful for developing language learning systems\n2. I think the insights into the relationship between automated evaluation methods and human evaluation methods is useful", "reasons_to_reject": "1. The 6 annotators are recruited from the author’s university. This may introduce bias towards questions in the dataset and makes the annotations difficult to reproduce. It is unclear if they knew the origin/context explanations they were annotating. It is also I think it would be performing annotations on a platform like M-Turk. I would also consider including the full instructions provided to annotators in Appendix C.\n\nEdit: \nThe authors have explained why they did not use a crowd source annotation platform like M-Turk and have provided the instructions shown to the annotators. However, not all annotations were performed at the same time (annotations were performed with a 2 week gap). It is possible that this could produce some bias in the annotations. This gap should be noted in the Limitations for this work.", "questions_to_authors": "1. Are the same annotators used for evaluating the LLM generated explanations at the dataset explanations? (And is there a time gap between those 2 annotations or were they annotated at the same time)\n2. Also, were the annotators paid for the annotators as part of their salary, paid for completing the annotations or volunteers?\n(You may want to add an ethics statement discussing pay and other ethics consideration of the annotations)", "paper_topic_and_main_contributions": "The authors propose a new task: ClozEx which focuses on generating explanations for cloze question in language assessment \nFor this task, they present a new dataset created from templates/pattern applied to a corpus of news text\nThe authors try training and prompting a number of large language models (LLM) to be used as baseline for the proposed task and dataset.\nThey access the questions created for the dataset and the questions generated by the language models using human annotations of Fluency (how easy the explanation is to comprehend) and Validity (is the explanation valid) on the Likert scale. Additionally, author evaluate the LLM generated questions using some automatic evaluation methods\nThe fine tuned encoder-decoder LLM methods seems to outperform decoder only method and prompting methods with the BART-large baseline archives the highest Validity score\nFinally, the authors compare the correlation of human and automatic evaluation scores\nGPT-Fluency and annotator Fluency are highly correlated. However, none of the automated evaluation methods were not highly correlated with annotator Validity.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 538: “no LLM achieved an acceptable validity score in manual evaluation”. I think you mean “no LLM achieved an acceptable validity score in manual evaluation without fine tuning” as it seems like BART-large explanations perform about the same at the original dataset explanations", "Justification_for_Ethical_Concerns": "Original review:\nNo mention of appropriate review board \nNo mention of if annotators were fairly compensated and how the compensation was determined to be fair\nPlease see “For papers presenting new datasets” in\nhttps://2023.emnlp.org/ethics/review-questions/\n\nEdit:\nThe authors have clarified that the annotation work was approved by the ethical committee and that annotators were paid above minimum wage to perform the annotations"}}]

Author rebuttal:

Meta-review: This paper presents a new task (and corpus) ClozEx which focuses on generating explanations for cloze question in language assessment with a focus on focus on English as a Second Language, ESL, learners. Various models have been fine-tuned to generate explanations for the questions. Extensive evaluations have been conducted in order to answer to two questions: (i) Do LLMs Explain Cloze Questions Well?, (ii) Are Automatic Metrics Reliable in ClozEx?.  The experiments are solid and the analysis interesting.
Some extra information could be provided concerning the annotators and the bias management.
Decision: Accept-Findings