Difficulty-Controllable Neural Question Generation for Reading Comprehension using Item Response Theory
Year: 2023

Abstract:
Question generation (QG) for reading comprehension, a technology for automatically generating questions related to given reading passages, has been used in various applications, including in education. Recently, QG methods based on deep neural networks have succeeded in generating fluent questions that are pertinent to given reading passages. One example of how QG can be applied in education is a reading tutor that automatically offers reading comprehension questions related to various reading materials. In such an application, QG methods should provide questions with difficulty levels appropriate for each learner’s reading ability in order to improve learning efficiency. Several difficulty-controllable QG methods have been proposed for doing so. However, conventional methods focus only on generating questions and cannot generate answers to them. Furthermore, they ignore the relation between question difficulty and learner ability, making it hard to determine an appropriate difficulty for each learner. To resolve these problems, we propose a new method for generating question– answer pairs that considers their difficulty, estimated using item response theory. The proposed difficulty-controllable generation is realized by extending two pre-trained transformer models: BERT and GPT-2.

heading: 1 Introduction
text: Automatic question generation (QG) for reading comprehension is the task of automatically generating reading comprehension questions related to given reading passages. Various QG methods have been developed in the natural language processing (NLP) research field (Zhang et al., 2021). They have also been used in various educational systems, such as intelligent tutoring systems, writing support systems, and knowledge assessment systems (Ghanem et al., 2022; Kurdi et al., 2020; Le et al., 2014; Rathod et al., 2022; Zhang et al., 2021).
Early QG methods have relied on rule-based or template-based approaches, which use handcrafted rules or templates to generate an interrogative question text from a declarative text (Zhang et al., 2021). However, preparing those QG methods for a target application is time-consuming and labor-intensive because achieving high-quality QG requires well-designed rules and templates for each application (Chen et al., 2021; Zhang et al., 2021). End-to-end QG methods based on deep neural networks have received wide attention as a means of overcoming this limitation (Chan and Fan, 2019; Du et al., 2017; Ushio et al., 2022; Yu et al., 2023; Zhang et al., 2021). Earlier neural QG methods were designed as sequence-to-sequence (seq2seq) models based on recurrent neural networks (RNNs) and attention mechanisms (Du et al., 2017), while recent methods are based on pre-trained transformer models (Gao et al., 2019; Ghanem et al., 2022; Lee and Lee, 2022; Rathod et al., 2022; Ushio et al., 2022), including BERT (Bidirectional Encoder Representations from Transformers) (Devlin et al., 2019), GPT-2 (Generative Pre-trained Transformer 2) (Radford et al., 2019), BART (Bidirectional and Auto-Regressive Transformers) (Lewis et al., 2020), and T5 (Text-to-Text Transfer Transformer) (Raffel et al., 2022). Those methods have succeeded in generating fluent questions that are pertinent to given reading passages.
A representative application of how QG can be used for educational purposes is a reading tutor that automatically offers reading comprehension questions related to various reading materials (Kurdi et al., 2020; Le et al., 2014; Rathod et al., 2022; Zhang et al., 2021). This helps to focus learners’ attention on the reading materials and offers the opportunity to observe any misconceptions they might have (Kurdi et al., 2020), which supports the development of reading comprehension skills. To enhance such learning, it is useful to provide questions with difficulty levels appropriate for each
119
learner’s reading ability. Such adaptivity is a core component of recent AI-based intelligent tutoring systems.
Difficulty control of QG is a relatively new task (Cheng et al., 2021; Kurdi et al., 2020), and thus previous research on difficulty-controllable QG for reading comprehension is still limited (Chen et al., 2021; Cheng et al., 2021; Gao et al., 2019). There are currently only two conventional methods; the first uses an RNN-based seq2seq model in which hidden states before its encoder are modified to receive a difficulty as input that is categorized as either easy or hard (Gao et al., 2019), and the second is a multi-hop QG (Cheng et al., 2021) that takes the question difficulty to be the number of inference steps required to answer a question and aims to generate questions while controlling the number of required inference steps. However, both methods have the following limitations that prevent them from generating questions appropriate for a learner’s ability.
1. They ignore the relation between question difficulty and learner ability, making it difficult to determine an appropriate difficulty for each learner.
2. They are answer-aware QG methods, which generate questions given a reading passage and an answer text, as illustrated in Fig. 1, and thus cannot generate question–answer pairs. Without correct answers, systems cannot score learners’ answers automatically,
meaning adaptive systems will not work efficiently. Furthermore, controlling difficulty in answer generation is also important because difficulty is a property that generally depends on both questions and answers.
To resolve these problems, we propose a new method for generating question–answer pairs that considers the difficulty associated with learners’ ability. A unique feature of our method is that it uses item response theory (IRT) (Lord, 1980), a test theory based on mathematical models, to quantify the difficulty of each question–answer pair. IRT is based on statistical models that define the relation between question difficulty and learner ability, and thus it helps us to select a difficulty appropriate for each learner’s ability. For these reasons, we aim to generate question–answer pairs while considering their difficulty, quantified by IRT. For our QG method, we first propose a method for constructing a training dataset consisting of quadruplets (reading passage, question text, answer text, and IRT-based difficulty), based on the SQuAD dataset, which is the most popular benchmark dataset for the reading comprehension QG task. Then, we propose a difficulty-controllable generation method for question–answer pairs that can be trained using this dataset. Our generation method consists of two pre-trained transformer-based models, which are extended to take IRT-based difficulty values as input: a difficulty-controllable answer extraction model using BERT, and a difficulty-controllable answer-aware QG model using GPT-2.
To our knowledge, this is the first difficultycontrollable QG method aimed at generating question–answer pairs corresponding to IRT-based difficulty.

heading: 2 Task Definition
text: The task tackled in this study is to generate a reading comprehension question and a corresponding correct answer, given a reading passage and a target difficulty value. Here, we assume that a correct answer to each question consists of a segment of text from the corresponding reading passage, as in typical answer-aware QG tasks (Rajpurkar et al., 2016). Fig. 2 shows an outline of our task.
The detailed task definition is as follows. Let a given reading passage be a word sequence r = {ri | i ∈ {1, . . . , I}}, where ri represents the i-th word in the passage, and I is the passage text length. Similarly, let a question text q and an answer text a be word sequences q = {qj | j ∈ {1, . . . , J}} and a = {ak | k ∈ {1, . . . ,K}}, respectively, where qj is the j-th word in the question text, ak is the k-th word in the answer text, J is the question text length, and K is the answer text length. Note that the answer text a must be a subset of the word sequence in the reading passage r, namely, a ⊂ r. Using this notation, our task is to generate a question text q and an answer text a given a reading passage r and a target difficulty value b, where the difficulty value b is assumed to be quantified based on IRT, as explained in the introduction.

heading: 3 Item Response Theory
text: IRT (Lord, 1980) is a statistical framework used in psychometrics and educational measurement to analyze examinees’ responses to test items (items corresponds to questions in our study). One of the unique characteristics of IRT is that it estimates two types of latent factors from response data: examinee ability and item characteristics. Examinee ability refers to the latent trait or ability that the test is intended to measure, such as reading comprehension ability in our context. Item characteristics refer to the properties of test items, including their difficulty level and their ability to discriminate examinee ability. IRT uses probabilistic models, called IRT models, to estimate examinees’ abilities and item characteristics from response data that typically consist of a binary variable taking one if an examinee answers an item correctly and zero otherwise.
IRT has been widely used in various educational and psychological tests because it has the following typical benefits (Uto and Ueno, 2020) compared with classical test theory (a simple and traditional framework based on basic statistics such as mean, variance, and correlation coefficients): 1) IRT provides detailed information about item properties, including difficulty and discrimination, which helps test developers identify problematic items and improve test quality. 2) IRT provides accurate estimates of examinee ability and item properties. 3) The abilities of examinees who take different tests can be estimated on the same scale because examinee ability is estimated considering the effects of the items’ characteristics. 4) IRT is the basis for computerized adaptive testing (CAT), which can reduce test length and increase measurement precision by selecting appropriate items for a target examinee’s ability (van der Linden and Glas, 2010).
This study uses the Rasch model (a oneparameter logistic model), which is the most traditional and well-known IRT model. The Rasch model defines the probability that the m-th examinee correctly answers the n-th item as
pnm = exp(θm − bn)
1 + exp(θm − bn) , (1)
where bn represents the difficulty of the n-th item and θm represents the latent ability of the m-th examinee.
To explain the relationship between the latent ability θ and the difficulty parameter b in the Rasch model, Fig. 3 depicts item response curves (IRCs) of the Rasch model, which are drawn by plotting the probability pnm, for three different difficulty
values. In the figure, the horizontal axis shows θ, the vertical axis shows the probability pnm, and three solid curves show the IRC for three items with different difficulty values.
These IRCs show that examinees with higher θ have a higher probability of responding correctly to each item. We can also see that the IRC shifts to the right as the item difficulty value increases, reflecting the fact that higher ability is required to correctly answer items with high b. Furthermore, under the Rasch model, the probability that an examinee with ability θ correctly answers the question with difficulty b becomes 0.5 when θ = b.
The IRT model parameters are generally estimated in two phases, namely, item calibration and ability estimation, in order to guarantee asymptotic consistency. Item calibration estimates the item parameters from response data by marginalizing the examinee ability θ from the likelihood in order to ensure the asymptotic consistency of the item parameter estimates. Specifically, marginal maximum likelihood (MML) estimation using an expectationmaximization (EM) algorithm has been widely used for item calibration (Baker and Kim, 2004). Given calibrated item parameters, the ability estimation phase calculates the examinee’s ability θ. An expected a posteriori (EAP) estimation, a type of Bayesian estimation, is generally used for the ability estimation (Fox, 2010; Uto et al., 2023).
This study aims to quantify question difficulty based on the IRT. The next section explains how to prepare the dataset with IRT-based difficulty, which is required to train our QG model.

heading: 4 Creating a Dataset with IRT-based Question Difficulty
text: We require an appropriate dataset to construct our QG method for solving the difficulty-controllable QG task defined in Section 2. While several popular datasets have been developed for general reading comprehension QG tasks (Zhang et al., 2021), the most popular is SQuAD (Rajpurkar et al., 2016), which consists of over 100,000 question–answer pairs from Wikipedia articles. Specifically, SQuAD is a collection of triplets (r, q,a), where each answer a is a text fragment from a corresponding reading passage r and each reading passage r corresponds to a paragraph of a Wikipedia article. However, to construct a difficulty-controllable QG method, we require a dataset consisting of quadruplets (r, q,a, b). Thus, we first propose a method
for extending the SQuAD dataset by appending the IRT-based difficulty values for each question– answer pair. The details for doing so are as follows.
1. Collecting response data for each question– answer pair: We collect answers from multiple respondents to each question in the SQuAD dataset and grade those answers as correct or incorrect. Ideally, we should gather responses from a population of target learners, but this is highly expensive and time-consuming. Thus, we substitute actual learner responses with automated question– answering (QA) systems, in the same way that several previous difficulty-controllable QG studies have done (Chen et al., 2021; Gao et al., 2019).
2. Difficulty estimation using IRT: Using the collected response data, we estimate the question difficulty by using the Rasch model and the item calibration procedure introduced in Section 3. Note that the difficulty value generally depends on the contents of both the question and the answer.
3. Creating a dataset with difficulty estimates: We construct a dataset consisting of quadruplets (r, q,a, b) by appending the estimated difficulty values b into the triplets (r, q,a) of the SQuAD dataset.

heading: 5 Proposed Method
text: Our difficulty-controllable QG method, which is trained using the extended SQuAD dataset, is realized by performing the following two tasks in sequence: (1) difficulty-controllable answer extraction that extracts an answer text from a given reading passage while considering a target difficulty value, and (2) difficulty-controllable answer-aware QG that generates a question given a reading passage, an answer text, and a target difficulty value. Details of each are provided in the following sections.

heading: 5.1 Difficulty-Controllable Answer Extraction
text: We perform the difficulty-controllable answer extraction using BERT (Devlin et al., 2019). BERT is a pre-trained multilayer bidirectional transformer with 340M parameters, a transformer being a neural network architecture based on self-attention mechanisms. BERT is pre-trained on large amounts
of text data over two unsupervised learning tasks, masked language modeling and next-sentence prediction. The pre-trained BERT can be applied to various downstream tasks by fine-tuning the model with a task-specific supervised dataset after adding task-specific output layers. We use finetuned BERT for the answer extraction task because BERT has been widely used before in various text extraction tasks (Srikanth et al., 2020).
To perform answer extraction using BERT, we add output layers that predict the start and end positions of the answer text within a given reading passage. Specifically, we add two dense layers with softmax activation to transform each BERT output vector, which correspond to the words within a given reading passage, into probability values for whether the word is at the start or end position of the answer text. By extracting the word sequence within the start and end positions, which take the maximum probabilities, we can extract an answer text from a given reading passage.
We control the difficulty of the answer extraction by inputting a difficulty value with the reading passage. Specifically, the input for our model is defined as
b, [SEP], r1, r2, r3, . . . , rI , (2)
where [SEP] is the special token used to separate the difficulty value and the reading passage. This input is what enables the model to extract an answer text from a reading passage while considering the input difficulty value. Fig. 4 shows an outline of the answer extraction model.
We can fine-tune the answer extraction model by using a collection of triplets (r,a, b), which can be obtained from the extended SQuAD dataset explained in Section 4. This fine-tuning is performed
by minimizing cross-entropy loss between the predicted positions of the start and end of an answer text and their true positions.

heading: 5.2 Difficulty-Controllable Answer-Aware Question Generation
text: We use GPT-2 to perform difficulty-controllable answer-aware QG. GPT-2 is a transformer-based language model with more than 1.5 billion parameters, and it is pre-trained on more than 8 million documents using an unsupervised learning process called language modeling, which sequentially predicts the next word from the current word sequence. We use GPT-2 for the QG tasks because it has been widely used before in various text generation tasks.
Conventional answer-aware QG models based on pre-trained language models (Srivastava and Goodman, 2021), including GPT-2, are implemented by designing the model’s input as
r1, . . . , [A], a1, . . . , aK , [A], . . . , rI , [G], (3)
where [A] is a special token representing an answer’s start and end positions within a reading passage. [G] is also a special token representing the end of a reading passage. Conventional QG models receive this input and generate a question text after the special token [G].
To implement difficulty-control for the answeraware QG model, we concatenate a target difficulty value to the conventional input form above using
b, [Q], r1, . . . , [A], a1, . . . , aK , [A], . . . , rI , [G], (4) where [Q] is the special token used to separate the difficulty value and the given reading passage. Given this input, the model generates a question text based on a reading passage, an answer, and a
target difficulty value. Fig. 5 presents an outline of our QG model.
We can fine-tune the answer-aware QG model by using a dataset consisting of quadruplets (r, q,a, b), explained in Section 4. Specifically, we prepare the following format data and train GPT2 by maximizing the log-likelihood for question texts:
b, [Q], r1, . . . , [A], a1, . . . , aK , [A],
. . . , rI , [G], q1, . . . , qJ . (5)

heading: 5.3 Determining Appropriate Difficulty based on IRT
text: As explained in Section 1, IRT helps us to select a difficulty appropriate for each learner’s ability. Earlier studies on adaptive learning have demonstrated that offering questions with a difficulty at which the learner would have a 50% chance of answering correctly is the most effective approach for learning (Ueno and Miyazawa, 2018). As explained in Section 3, under the Rasch model, the probability that a learner with ability θ correctly answers the question with difficulty b becomes 0.5 when θ = b. Thus, we can generate questions with a difficulty appropriate for each learner using the following steps inspired by the framework of CAT (van der Linden and Glas, 2010).
1. Provide some questions randomly to a learner and collect response data.
2. Estimate the learner’s ability using the Rasch model and the response data.
3. Generate a question–answer pair by inputting the estimated ability value as the difficulty value into the proposed QG method.
Furthermore, by repeating procedures 2–3, we can enable adaptive QG.

heading: 6 Experiments
text: In this section, we demonstrate that our proposed method can generate questions and answers corresponding to target IRT-based difficulty values.

heading: 6.1 Data preparation
text: For our experiment, we first constructed an extended SQuAD dataset consisting of quadruplets (r, q,a, b) by following the procedures explained in Section 4. The original SQuAD dataset was divided into training data (90%) and test data (10%)
in advance. In this experiment, we trained QA models using the training data and constructed an extended dataset using the test data. The detailed procedures were as follows.
θ̂
BERT-based model 2.25 ALBERT-based model 1.28 Logistic regression 0.52 Logistic regression (selected features) -0.64 Sliding-window model -2.84
A larger variety of respondent abilities is generally effective for clearly distinguishing the difficulty among questions, suggesting that our use of these five QA systems in our experiment is reasonable. Note that ability and question difficulty are estimated assuming a standard normal distribution, meaning that these estimates distribute approximately on a scale with a mean of 0 and a standard deviation of 1.
4. Creating a dataset with difficulty estimates: We created a dataset D consisting of quadruplets (r, q,a, b) by integrating the obtained IRT-based difficulty values and SQuAD test data.

heading: 6.2 Experimental Procedures
text: We conducted the following experiment using the created dataset D and the original SQuAD training data.
1. Using the original SQuAD training data, we fine-tuned the proposed answer extraction model and the answer-aware QG model, ignoring the difficulty. This fine-tuning was done by removing the difficulty value from the input of the proposed models. Although this procedure is not mandatory, we applied it to improve the basic QG performance.
2. We randomly divided the dataset D into parts, one 90% (designated as D(train)) and the other 10% (designated as D(eval)). Then, using the 90% dataset D(train), we fine-tuned the difficulty-controllable answer extraction model and the difficulty-controllable answeraware QG model, where the initial model parameters were set to the values obtained in procedure 1.
3. We generated questions and answers for each reading passage in the remaining 10% dataset D(eval), given each of the six difficulty values (1, 29, 49, 64, 79, 100). Using the generated questions and answers, we conducted both an automatic evaluation and a human evaluation, which are explained below.
We used PyTorch and the Transformers library to implement the proposed models and the neural QA systems. Furthermore, we used R and the TAM package to perform the IRT parameter estimation.
Here, we summarize the basic statistics of the datasets D(train) and D(eval), which we developed in the above procedure 2 to train and evaluate our difficulty-controllable QG method. First, the number of reading passages in D(train) and D(eval) was 1,860 and 207, respectively. Next, the average number of questions per reading passage in D(train) and D(eval) was 5.21 and 4.28. Furthermore, Table 2 shows the number of questions corresponding to the six difficulty values in each dataset. From these results, we can confirm that the basic statistics and the difficulty distributions are similar between the two datasets, indicating that the dataset D was randomly divided into D(train) and D(eval) without bias.

heading: 6.3 Automatic Evaluation
text: We performed an automatic evaluation by calculating the percentage of correct answers given by the neural QA systems (BERT-based and ALBERTbased QA models) to the questions generated for each difficulty. Fig. 6 shows the results, which indicate that the correct answer rate of QA systems
Table 3: Examples of generated questions and answers for different difficulties.
Reading passage Much of the work of the Scottish Parliament is done in committee. The role of committees is stronger in the Scottish Parliament than in other parliamentary systems, partly as a means of strengthening the role of backbenchers in their scrutiny of the government and partly to compensate for the fact that there is no revising chamber. The principal role of committees in the Scottish Parliament is to take evidence from witnesses, conduct inquiries and scrutinise legislation. Difficulty 1 (easiest) Question Where is much of the work of the Scottish Parliament done? Answer committee Difficulty 100 (most difficult) Question What is the purpose of the chairman and member of the committee? Answer take evidence from witnesses, conduct inquiries and scrutinise legislation
decreases as the difficulty increases. This suggests that our proposed method generates questions that reflect the given difficulty.
Furthermore, we calculated the average word length in the generated answer texts for each difficulty. Fig. 7 shows the results, and these indicate that the average word length in the generated answer texts increases as the target difficulty values increase. Considering that questions with longer and more complex answers are generally difficult to correct perfectly, this result suggests that the proposed method extracts answers that reflect the specified difficulty.
Table 3 shows examples of the generated question–answer pairs when given the same reading text but different difficulty values, demonstrating that higher difficulty values correspond to longer answers.

heading: 6.4 Human Evaluation
text: For the human evaluation, we randomly selected ten reading passages from D(eval) and extracted question–answer pairs for the six difficulty values corresponding to each reading passage from the generated data obtained in experimental procedure 3. Then, the 60 question–answer pairs were evaluated by four human judges according to the following four evaluation metrics.
1. Difficulty: The subjective difficulty evaluation for each question–answer pair, graded on a scale from one to five, where smaller grades mean the question was easier.
2. Fluency: Evaluation of the grammatical correctness of generated questions, graded on a three-point scale: Yes, Acceptable, and No.
3. Relevance: Evaluation of the content relevance between generated questions and reading passages, graded on a binary scale: Yes and No.
4. Answerability: Evaluation of the answerability of each generated question–answer pair from a given reading passage, graded on a four-point scale: Yes, Partially, and No. Here, “Partially” indicates that the generated answer does not entirely match the correct answer for the generated question but partially includes the correct answer.
Fig 8 shows the relation between the input difficulty values and the averaged scores in the human difficulty evaluation for the generated questions. They indicate that the human subjects judged the questions generated with higher difficulty values to
be more difficult. This indicates that the proposed method can appropriately control the difficulty of generated question–answer pairs.
Table 4 gives the results for Fluency, Relevance, and Answerability. It shows that more than 90% of the questions were generated with correct or acceptable grammar, and about 90% appropriately reflected the content of the given reading passages. Furthermore, about 70% of generated question– answer pairs were completely answerable, and about 85% were partially appropriate. These results indicate that fluency and relevance are acceptable but further improvement might be required in terms of answerability, which is planned for future work.

heading: 7 Conclusion
text: In this study, we proposed a new neural QG method that generates question–answer pairs while considering their difficulty, estimated using IRT. We also evaluated the effectiveness of this method through experiments using SQuAD.
One limitation of this study is that we used only the SQuAD dataset in our experiments. The SQuAD dataset has often been criticized because it is overly dependent on the similarity of question/answer sentences rather than on human-type reasoning, meaning it requires only superficial read-
ing skills. Thus, examining the effectiveness of our proposed method by applying it to various other datasets will be an important future task.
Furthermore, in the human evaluation experiment presented in Section 6.4, we examined only 60 question–answer pairs generated through the proposed model from ten randomly selected reading passages. The relatively small scale of the experiment is due to the high workload required for people to carefully evaluate the various properties of a large number of questions. However, in the future, we aim to conduct a larger-scale human evaluation in order to increase the reliability of the experimental results.
Although the present study used only five QA systems, the use of a larger number of QA systems with different characteristics is expected to improve the accuracy of question-difficulty estimation and provide difficulty estimates with finer granularity. Therefore, examining the effects of increasing the number and variability of QA systems will be another future direction of this research.
We also need to confirm in greater detail whether QA systems can be substituted for human learners. A comparison between IRT-based question difficulties calibrated from the responses of QA systems as well as human learners might be a plausible approach.
Another future goal is to develop a method of transforming the scale of the IRT-based difficulty, estimated based on QA systems, into a scale appropriate for a population of target learners. Such a scaling adjustment is expected to be achievable by using equating, which is a well-established technique in IRT.
Furthermore, our QG method is easily extended to adaptive QG systems based on the framework of computerized adaptive testing, as mentioned in Section 5.3. Developing and evaluating such an adaptive system using our QG method will also be our focus in future work.

heading: Acknowledgements
text: This work was supported by 19H05663, and 21H00898.

heading: Future Work
text: Thus, examining the effectiveness of our proposed method by applying it to various other datasets will be an important future task. Furthermore, in the human evaluation experiment presented in Section 6. 4, we examined only 60 question–answer pairs generated through the proposed model from ten randomly selected reading passages. The relatively small scale of the experiment is due to the high workload required for people to carefully evaluate the various properties of a large number of questions. However, in the future, we aim to conduct a larger-scale human evaluation in order to increase the reliability of the experimental results. Although the present study used only five QA systems, the use of a larger number of QA systems with different characteristics is expected to improve the accuracy of question-difficulty estimation and provide difficulty estimates with finer granularity. Therefore, examining the effects of increasing the number and variability of QA systems will be another future direction of this research. We also need to confirm in greater detail whether QA systems can be substituted for human learners. A comparison between IRT-based question difficulties calibrated from the responses of QA systems as well as human learners might be a plausible approach. Another future goal is to develop a method of transforming the scale of the IRT-based difficulty, estimated based on QA systems, into a scale appropriate for a population of target learners. Such a scaling adjustment is expected to be achievable by using equating, which is a well-established technique in IRT. Furthermore, our QG method is easily extended to adaptive QG systems based on the framework of computerized adaptive testing, as mentioned in Section 5. 3. Developing and evaluating such an adaptive system using our QG method will also be our focus in future work