Abstract: In this paper we formalize the problem automatic fill-in-the-blank question generation using two standard NLP machine learning schemes, proposing concrete deep learning models for each. We present an empirical study based on data obtained from a language learning platform showing that both of our proposed settings offer promising results.

Introduction: With the advent of the Web 2.0, regular users were able to share, remix and distribute content very easily. As a result of this process, the Web became a rich interconnected set of heterogeneous data sources. Being in a standard format, it is suitable for many tasks involving knowledge extraction and representation. For example, efforts have been made to design games with the purpose of semi-automating a wide range of knowledge transfer tasks, such as educational quizzes, by leveraging on this kind of data. In particular, quizzes based on multiple choice questions (MCQs) have been proved efficient to judge students’ knowledge. However, manual construction of such questions often results a time-consuming and labor-intensive task. Fill-in-the-blank questions, where a sentence is given with one or more blanks in it, either with or without alternatives to fill in those blanks, have gained research attention recently. In this kind of question, as opposed to MCQs, there is no need to generate a WH style question derived from text. This means that the target sentence could simply be picked from a document on a corresponding topic of interest which results easier to automate. Fill-in-the-blank questions in its multiple-choice answer version, often referred to as cloze questions (CQ), are commonly used for evaluating proficiency of language learners, including official tests such as TOEIC and TOEFL BIBREF0 . They have also been used to test students knowledge of English in using the correct verbs BIBREF1 , prepositions BIBREF2 and adjectives BIBREF3 . BIBREF4 and BIBREF5 generated questions to evaluate student’s vocabulary. The main problem in CQ generation is that it is generally not easy to come up with appropriate distractors —incorrect options— without rich experience. Existing approaches are mostly based on domain-specific templates, whose elaboration relies on experts. Lately, approaches based on discriminative methods, which rely on annotated training data, have also appeared. Ultimately, these settings prevent end-users from participating in the elaboration process, limiting the diversity and variation of quizzes that the system may offer. In this work we formalize the problem of automatic fill-in-the-blank question generation and present an empirical study using deep learning models for it in the context of language learning. Our study is based on data obtained from our language learning platform BIBREF6 , BIBREF7 , BIBREF8 where users can create their own quizzes by utilizing freely available and open-licensed video content on the Web. In the platform, the automatic quiz creation currently relies on hand-crafted features and rules, making the process difficult to adapt. Our goal is to effectively provide an adaptive learning experience in terms of style and difficulty, and thus better serve users' needs BIBREF9 . In this context, we study the ability of our proposed architectures in learning to generate quizzes based on data derived of the interaction of users with the platform.

Question: What is the size of the dataset?
Answer: 300,000 sentences with 1.5 million single-quiz questions