Recommendation for English Multiple-choice Cloze Questions
Based on Expected Test Scores
Tomoharu Iwata
NTT Communication Science Laboratories
iwata@cslab.kecl.ntt.co.jp
Tomoko Kojiri
Graduate School of Information Science, Nagoya University
kojiri@is.nagoya-u.ac.jp
Takeshi Yamada
NTT Communication Science Laboratories
Toyohide Watanabe
Graduate School of Information Science, Nagoya University
Abstract
When students study for multiple-choice cloze tests as the Test of English for International Communication (TOEIC), they tend to repeatedly tackle questions of the same type. In such situations, students
can effectively solve questions related to their incorrectly answered questions. However, since theyneed several different kinds of knowledge and a large vocabulary to derive answers, it is inappropriate
to statically define the relations among questions from various viewpoints beforehand. In this paper, we
propose a recommendation algorithm for English multiple-choice cloze questions that maximize students’ expected improvements of test scores based on the learning log data of other students. Effective
questions may be identical for most students who incorrectly answered the same questions. Therefore,in our approach, relations among questions in tests and questions studied during tests are determinedbased on the change from incorrect to the correct answers of the test questions. Questions that maximize
the expected test scores, which are calculated based on the input test scores using regression models,
are recommended for future students. Based on this method, students can acquire higher test scoreswith better learning efficiency. Experimental results show that our method yields major improvementsin performance compared with random material recommendation method.

1

Introduction

stored in e-Learning systems, and automatic question
generation systems have also been developed to generAs network and database technologies continue to ate new questions [14].
grow rapidly, e-Learning systems have become widelyused in various domains. English multiple-choiceIn e-Learning systems, providing suitable learning
cloze questions are one popular learning material in e- materials that teach unacquired knowledge is crucial. In
Learning. Such questions are applied in various tests, the e-Learning of English multiple-choice cloze quesnamely, the Test of English for International Commu- tions, students tend to repeatedly tackle the same type of
nication (TOEIC) or university entrance examinations questions to heuristically acquire the knowledge asked
in Japan. A large number of such questions are already in the questions. Since students need various kinds ofEnglish knowledge to solve questions, such as grammar and a large vocabulary, determining the important knowledge for answering each question is difficult. Moreover, because there are many questions, it isimpractical to statically define the relations among allquestions beforehand.
This research assumes students who are studying toacquire English knowledge within the given tests forEnglish multiple-choice cloze questions. They take English multiple-choice cloze tests that contain a specificnumber of questions and study using the same type ofquestions to acquire the knowledge of the incorrectly
answered questions. The objective of our research isto provide effective questions in the studying phase toincrease the scores of subsequent tests taken after the
studying phase. Questions can be selected in various
ways based on such supporting policies as giving manyquestions so that students deeply understand the unacquired knowledge and giving a minimum number ofquestions so that they can quickly pass the test. Our
research focuses on increasing test scores with great efficiency by giving a small set of questions that are adequate for gaining good test scores.
Question answers reflect the acquired/unacquired
knowledge of students. If students answered incorrectly
the same question, their unacquired knowledge probably is the same. Therefore, questions that help students
grasp the knowledge of questions may also be effective for other students who failed the same question. To
provide appropriate questions based on the tendency ofstudent understandings, this research proposes a recommendation algorithm for questions based on implicit relations among questions acquired by students’ learning
log data.
Although many personalized learning material recommendation algorithms have been proposed, they do
not directly support learning efficiency. Instead, they
consider student preferences or/and understanding levels [6, 5, 12, 13, 19, 20]. This research focuses on situations where students take the same test before and after
studying to quantify the expected improvement in testscores; appropriate questions must be recommended tomaximize such expected improvement.
In the research field of recommendation systems,
since the cognitive load on users to assign explicit ratings is heavy, gathering enough appropriate ratings isdifficult [16, 2]. Therefore, in our approach, test scores
are used to determine the relations between questions
and student explicit ratings are not applied.
Our method recommends questions by which stu-

dents can provide correct answers to questions that theyanswered incorrectly before studying. Effective questions for incorrectly answered questions may be identical for most students. If particular questions are studied
between tests and student test scores are increased after
such studying, they may provide knowledge about questions whose answers were changed and became correct.
The expected improvement of test scores based on questions is calculated by logistic regression models [9]. By
training the logistic regression models using student testscores and their learning log data, namely, the studied
questions, questions that improve test scores can be automatically extracted.
The remainder of this paper is organized as follows.
In Section 2, we briefly review related work. In Section 3, we explain our approach for providing effective
questions. In Section 4, we present our recommendation algorithm, and our method is evaluated using learning log data and test scores in Section 5.

2 Related Work
Traditional intelligent tutoring systems (ITS) evaluate
the learning situations of students and give learning materials that supply the unacquired knowledge of students [13, 10, 11, 1, 4]. In such systems, metadata
about materials are attached beforehand, including thedifficulty, topic, prerequisites, and the relationship between materials. The learning materials to be provided
are determined by traversing the databases of learningmaterials based on student answers, the metadata of thelearning materials, and the relations among them. Thismethod may be appropriate if the number of learningmaterials is not so large and their relations and metadata
can be defined uniquely. Our research focuses on situations where plenty of questions exist whose relations
are not defined beforehand. Moreover, the unacquired
knowledge of students is not uniquely determined by
the metadata. Therefore, attaching metadata to questions and determining questions based on it is impractical. Instead of statically preparing metadata of thelearning materials, our method grasps the implicit relations among questions based on the learning log dataof the students who tackle the same tests and provides
questions that may affect the incorrectly answered questions.
Many methods for recommending learning materials, which are also called curriculum sequencing, adaptive tutoring, or personalized learning path guidance,have been proposed [6, 5, 12, 13, 19, 20]. For example, the Personalized E-Learning system based on Item
Response Theory (PEL-IRT) [6] automatically adjusts
the difficulty of learning materials according to the student’s level of understanding based on item response
theory [8] to provide appropriate learning materials.
However, since PEL-IRT does not consider questions
that students need to tackle in the test, it does not necessarily provide learning materials that can improve testscores.
Recommendation systems for books, music, andmovies are used in many on-line stores [18], wherecollaborative filtering [17] is a common method. Collaborative filtering recommends products that are purchased by users with similar preferences. E-Learning
systems that reflect student preferences have also been
proposed [13]. However, if students only use their favorite materials their test scores may not improve.
Our method is related to the scheme described in [5]because both involve a test for each student prior tolearning. The method in [5] recommends materials considering their difficulty and incorrectly answered questions. However, the objective of this method is to makestudents understand unacquired knowledge. Since thismethod does not consider learning efficiency, students
need to cope with many questions before understanding
the questions that they answered incorrectly. Our research increases specific test scores previously tackled
by students with a limited number of questions.

3

correctly answered questions. If the important knowledge is identical for different questions, the explanations for deriving the answer for one question can beapplied to other questions. Such questions can be inferred by student learning log data. When two questions
share important knowledge and students cannot answer
one of the questions in the tests, they may answer correctly after studying with other questions. Of course,
some students may not notice the relations between thequestions. Effective questions are those that give hints
for deriving incorrectly answered questions. Therefore,studied questions are detected as effective for test questions whose answers are changed from incorrect to correct answer by many students.

ApproachThis research assumes that the student, who took a couple of tests in the past, is studying a set of multiplechoice cloze questions in the studying phase to take a
new test after the studying phase, where each test consists of a number of questions. While studying, students tackle different questions from those in the tests.
The aim of this research is not to give a large volumeof questions. Instead, it provides a limited number ofsuitable questions that can help increase test scores.
The increase of test scores corresponds to situations
where students can successfully acquire the knowledge
on which the questions focus in the tests.
Students need various grammar and vocabularyknowledge for answering English multiple-choice cloze
questions. Of course, important knowledge exists to determine answers. Effective questions for maximizing
test scores have the same important knowledge as in-

Figure 1: Framework of learning support system

Figure 1 shows the framework of our learning
support system. The mechanism for recommending
questions selects effective questions from the question database and provides appropriate study questions
based on the input test results of students. To evaluate
the questions, the expected improvement values are introduced that represent the expected test scores when
individual questions are posed for studying. The expected improvement values are calculated based on thelearning log data and the test results of many other stu- recommended questions z is written as follows:
dents. In the mechanism for estimating the relations be∑
Si P (i)P (xi = −1)P (yi = 1|xi = −1, z),
tween questions, the changes of the test results and the E(z) =i∈Vstudied questions are investigated and set as expected(4)
improvement values.
where Si represents the score allocated to question i,
P (i) represents the probability that question i is asked
in future test P (i) + P̄ (i) = 1 in which P̄ (i) repre4 Proposed Method
sents the probability that question i is not asked in future tests, P (xi = −1) represents the probability that
4.1 Problem setting
question i is incorrectly answered before the studying
phase, and P (yi = 1|xi = −1, z) represents the probThe goal of our method is to select questions to be usedability that question i is correctly answered after theduring the studying phase from a set of questions tostudying phase when question i is incorrectly answered
enhance learning efficiency, which is quantified by the
before the studying phase and questions z are recomexpected improvement in test scores. In our approach,mended in the studying phase.
the implicit relations between test questions and studied
If the test result before studying phase x is given, thequestions are learned.
expected improvement in the test score can be simpliLet zj be a variable that represents whether question fied as follows:
j is recommended in the studying phase as follows:∑E(z|x) =Si P (i)P (yi = 1|xi = −1, z). (5)


i:x
=−1
1
if
questionjisstudiedizj =(1) We use Eq. (5) as the expected improvement in testin the studying phase,
0 if question j is not studied.
scores in the following sections.
The studied questions are represented by vector z =(zj )j∈M , where M represents a set of all candidate
questions to study.
Let xi and yi be variables that represent whether
question i is correctly or incorrectly answered before
and after the studying phases, as follows:


1






xi = −1






0

if question i is correctly answered
before the studying phase,
if question i is incorrectly answered (2)before the studying phase,
if question i is not answered,

4.2 Recommendation algorithm
Appropriate questions may differ according to previously studied questions from the studying phase. Our
method sequentially selects a question that maximizes
the expected improvement from questions that have notyet been recommended as follows:ĵ = arg max E(z +j |x),j:zj =0(6)

where z = (zj )j∈M represents the currently studied questions and z +j represents the studied questions= 1
when question j is newly recommended, or zj+j0+j00
if j = j and zj 0 = zj 0 if j 6= j . Table 1 shows
our method’s question recommendation procedure. Ex
1if question i is correctly answered
amples of end conditions include those where the num
after the studying phase,
ber of studied questions, the expected improvement, or
yi = −1 if question i is incorrectly answered (3) the time period of the studying phase exceeds a certain
after the studying phase,threshold.

0We use a greedy algorithm to determine a set ofif question i is not answered,questions to be recommended as described above. If
The results of the set of test questions V before and the number of questions is fixed and known before theafter the studying phase are represented by vectors x = studying phase, we can identify the set of questions that(xi )i∈V and y = (yi )i∈V , respectively.
maximizes the improvement scores by calculating Eq.
The improvement in the test scores expected from (5) for all possible combinations of the fixed number ofTable 1: Question recommendation procedure with proposed method.

of students N :=1. Input the test result before studying x;
2. Initialize the studied question vector:z = (0, · · · , 0);
3. Select question ĵ which was recommended by
Eq. (6);
4. Update the studied question vector:z = z +ĵ ;5. Return to step 3 unless an end condition issatisfied.

L(Θ)
∑ ∑(

I(xni = −1 ∧ yni = 1)n∈N i∈V× log P (yni = 1|xni = −1, zn )+I(xni = −1 ∧ yni = −1))
× log P (yni = −1|xni = −1, zn )∑ ∑(=I(xni = −1 ∧ yni = 1)(µi + θi> zn )n∈N i∈V())−I(xni = −1) log 1 + exp(µi + θi> zn ) , (8)where xni and yni indicate whether question i was correctly or incorrectly answered by student n before andafter the studying phases andquestions. A greedy algorithm is used because it is fastand does not require the number of studied questions to
be fixed beforehand.

P (yni = −1|xni = −1, zn )= 1 − P (yni = 1|xni = −1, zn )1,=
1 + exp(µi + θi> zn )(9)represents the probability that question i is incorrectly
answered by student n when question i is incorrectly
answered before the studying phase and questions z arerecommended. In the experiments, we used a quasi4.3 Improvement model
Newton method [15] for the optimization and Gaussianpriors with zero means for the unknown parameters to
When recommending questions, our method requires avoid overfitting [7]. The Hessian of the log likelihood
improvement model P (yni = 1|xni = −1, zn ), which function with Gaussian priors with respect to the paramis the probability of the improvement of answering eters is positive definite. Therefore, the log likelihood
question i with study questions z. We model the im- is a convex function, and the global optimum solution
is guaranteed.
provement using logistic regression [9] as follows:

5 Experiments

1),
1 + exp −(µi + θi> zn ) 5.1 Setting(7)
where µi and θi = (θij )j∈M are unknown parameters. We evaluated our method using TOEIC multiple-choice
Although the logistic regression is widely used for bi- cloze questions in which students select appropriatenary classifiers, it is the first attempt to model the im- words from four options with the correct grammar for
provement of scores, to our knowledge. Intuitively, µi the blank in the sentence.
We implemented a web-based e-Learning system forrepresents the ease with which the answer to question i
is improved, and θij represents the influence of question the evaluation. In the experiment, students take tests
j on the improvement in the answer to question i. Un- before and after the studying phase, which we call preknown parameters Θ = {µi , θi }i∈V can be estimated test and post-test, to measure the effect of studying onby maximizing the following log likelihood, which con- improving test scores. One question is presented on one
sists of the learning log data and the test results for set web page, and students answer each question in a series.
P (yni = 1|xni = −1, zn ) =(“Correct”
Answer

Question
Explanation

Choices

User’s answer

Answer button
Button for next question

Figure 2: Web page posing a question generated by our
e-Learning system
Figure 3: Web page showing the solution and explanation generated by our e-Learning system
The 40 questions in the pre- and post-tests are identical,|V | = 40.
The recommended questions in the studying phaseare provided with solutions and explanations. One
question is recommended to each student on one web
page for studying, and the solution and explanation arepresented on another web page after the student has answered the question. Note that the students are not supplied with solutions and explanations in the pre- andpost-tests. There are 80 candidate questions for the
studying phase, |M | = 80, and 40 are recommended
to each student in the studying phase. The recommended questions are different from the questions in
the pre- and post-tests. However, about half are related to the test questions, for example, they involve
questions about identical idioms and grammatical rules.
They were heuristically selected by the authors.
Figures 2 and 3 are the web pages generated by our
e-Learning system. Fig. 2 is an web page that displays
a question sentence and a choice of fill in the blankanswers. The student selects one answer and pushes
the answer button. Fig. 3 is an web page showing thesolution and explanation for the studying phase. The
student’s answer is evaluated and its correctness is displayed at the top. After the student pushes the button,
the next question appears.

The proposed improved model (7) estimates the improvement probability by utilizing relationships among
questions. To evaluate the effect of the relationships,
we compared the proposed model with a model that
does not consider relationships among questions or the
Bernoulli model as follows:
1+yi

P (yi |xi = −1, z) = φi 2 (1 − φi )

1−yi
2

,

(10)

where φi represents the probability that question i iscorrectly answered in the post-test when question i isincorrectly answered in the pre-test. The Bernoulli
model assumes that the improvement does not dependon recommended materials z. Parameter φi can be estimated based on the maximum likelihood as follows:∑I(yni = 1 ∧ xni = −1)∑
φ̂i = n∈N.
(11)
n∈N I(xni = −1)

For the evaluation measurement, we used the
AUC [3] of the problem to predict whether questions
that were incorrectly answered in the pre-test are correctly answered in the post-test. AUC is the area under the Receiver Operating Characteristic (ROC) curve,
where the ROC curve is a graphical plot of the true positive rate versus the false positive rate. A higher AUC
represents better predictive performance. We computed
5.2 Evaluation of improvement models
AUC using leave-one-out cross-validation. We used 52
Our method requires improvement model P (yi = evaluation data sets, in each of which one student’s data
1|xi = −1, z). We constructed and evaluated improve- were used for the evaluation and the data of the otherment models using the log data of 52 students, such as 51 students were used for training. Table 2 shows the|N | = 52, with random material recommendations.
AUC, and Fig. 4 shows the ROC curve. The AUC ofTable 2: AUC of improvement models based on Table 3: Students who incorrectly answered question
Bernoulli distribution and logistic regression
xi1 that contained idiom ‘stop by’
pre-test studying phase post-test# ofBernoulli distribution
Logistic regressionxzystudents
i1
j1
i1
0.556
0.592
-1
1
1
15
-1
1
-1
2
1
-1
0
1
2
-1
0
-1
13
True Positive Rate

0.8
0.6
0.4
0.2

Bernoulli
Logistic Regression
0
0

0.2

0.4

0.6

0.8

1

False Positive Rate

Figure 4: ROC curves for improvement models basedon Bernoulli distribution and logistic regression

Algorithm 1 Simulation of student n with our recommendation
1: Set z ← 0, t ← 1
2: while t ≤ T do
3:ĵ = arg maxj:zj =0 E(z +j |xn )4:
Set z ← z +j
5:
Set t ← t + 1
6: end while
7: Output E(z|xn )the post-test are shown in Table 3. The probability ofstudents answering questions i1 correctly in the posttest when question j1 was recommended in the studying
phase was P̂ (yi1 = 1|xi1 = −1, zj1 = 1) = 15/17. In
contrast, the probability when question j1 was not recommended in the studying phase was P̂ (yi1 = 1|xi1 =−1, zj1 = 0) = 2/15. This result indicates that the recommendation of question j1 effectively improved the
responses to question i1 .

the improvement model based on the logistic regression is higher than that of the Bernoulli model, which
implies that the recommended questions are importantfor predicting improvements in the scores and that wecan predict them with the logistic regression model.
The computational time for learning the parametersof the proposed improvement model was 2.19 second
when using a computer with a Xeon5160 3GHz CPU.
5.4

5.3

Analysis of improvement models

Evaluation of recommendation algorithms by simulations

We examined the effectiveness of our recommendation
algorithm by simulation. Student behavior was simulated using the improvement model that was estimatedusing the log data of the 52 students described above.
The function of Algorithm 1 is to generate an expected
improvement of the test scores of student n, where T is
the number of recommendations set as 40.
We compared our algorithm with the following threealgorithms: Random, Mistakable, and Level. Random
randomly recommends a question. Mistakable recommends a question that is mistaken by many students.
Recommendation question ĵ is determined as follows:∑ĵ = arg max
I(znj = −1),
(12)

The highest and second highest θij were θi1 j1 = 0.388
and θi2 j2 = 0.208, where questions in test i1 and studying phase j1 contain the idiom ‘stop by’ and questions in test i2 and studying phase j2 contain the idiom‘across the street.’ This result is natural because recommendations of questions about identical idioms canimprove test scores. Even though our method does notuse information related to question contents, it automatically extracts the relationship between questions using
the learning log data and the test results.
We analyzed the relationship between questions in
the tests and in the studying phase using questions i1
and j1 that contain the idiom ‘stop by’ as an example.
j:zj =0n∈N
In the pre-test, 32 students answered question i1 incorrectly. The student results for the studying phase and where znj = −1 represents that student n incorrectlyTable 4: Average percentage of expected improvement Table 5: Average percentage improvement rates andrates and standard deviations with simulations
standard deviations with actual students
RandomMistakable
Level
ProposedRandom
Proposed
36.4 ± 4.9 38.6 ± 4.4 38.1 ± 4.1 52.7 ± 7.6
15.6 ± 22.9 27.5 ± 20.6

answered question j in the studying phase. Level rec- Table 6: Average scores and standard deviations in preommends a question whose difficulty is appropriate to test, studying phase, and post-test
pre-test studying phase post-test
the student’s understanding level by selecting recomRandom
72.8 ±
73.1 ±
77.3 ±
mendation question ĵ as follows:
15.7
14.4
14.1
ĵ = arg min |dj − ln |,(13)
Proposed
70.1
±
69.8
±
78.2
±
j:zj =0
13.2
12.7
13.1
∑
1
where dj = |Nn∈N I(znj = −1) is the difficulty|∑of question j and ln = |V1 | i∈V I(xni = −1) is thelevel of student n.
We evaluated the recommendation algorithm by theWe evaluated the recommendation algorithm by the improvement rate, which is the correctly answered
expected improvement rate, which is the expected num- questions after studying to the incorrectly answered
ber of correctly answered questions after studying to the questions before studying as follows:
number of incorrectly answered questions before study∑()
i∈V I(yni = −1)
ing as follows:∑
Rn = 1 −× 100.
(15)i∈V I(xni = −1)
E(z|xn )Sn = ∑
× 100.
(14) Table 5 shows the average improvement rates. Ouri∈V I(xni = −1)
algorithm corrected 27.5% of the mistakes and proHere, we assumed that questions correctly answered vided statistically significant increases compared with
before studying were also correct after studying. Ta- the random recommendation method (one-tailed t-test,ble 4 shows the expected improvement rates for each p < 0.007). The rates of real questions in Table 5 wererecommendation algorithm. The proposed algorithm smaller than those in the simulations in Table 4 because
outperformed the others for improving the expected test we assumed that questions correctly answered before
scores. The Mistakable and Level algorithms also im- learning were not incorrectly answered after studying
proved the scores more than the Random algorithm, al- in the simulations.
though the effect was smaller than that of our algorithm
Table 6 shows the average scores obtained in the prebecause they did not directly maximize test scores.
test, the studying phase, and the post-test. The maximum score is 100, which means the score assigned to
5.5 Evaluation of recommendation algo- one question is 2.5. Although the average pre-test scorewith our method is lower than that with the random recrithms by actual students
ommendation method, the average post-test score withWe evaluated the learning efficiency of our recommen- our method is higher than that with the random recomdation algorithm with actual students. Because the eval- mendation method. Moreover, since the deviation withuation of recommendation with many actual students our method is smaller than that with the random reccosts much, we compared the proposed method with the ommendation method, many students with our system
most basic random method. 38 students studied ques- successfully increased their test scores. This result intions recommended randomly, and 49 studied questions dicates that our method is superior to random recomrecommended by our method. The students included mendation methods.
Figure 5 shows the average improvements in themembers of the graduate and undergraduate schools andthe staffs of Nagoya and Kansai Universities. All had test scores in relation to the pre-test scores. The impreviously studied the basic English knowledge for the provement is in inverse proportion to the pre-test score.
Our method is superior to the random recommendation
provided questions in high school.

20

Table 9: Useful words/idioms in questions in studying
phase and number of students who cited themword/idiom# of students
10
stop
by
7
5
across the street
30
the day after tomorrow
2
due
to
2
−5
R(55−70) O(55−70) R(70−85) O(70−85) R(85−100) O(85−100)
launch2although
1
Figure 5: Average improvements in test scores with randespite
1
dom recommendation ’R’ and our method ’O’ in relafollow in one’s footsteps
1
tion to pre-test scores. Values in parentheses are pre-test
resign1
scoresin order to
1
among
1
Improvement

15

Table 7: Students who were/were not recommended related questions when questions that contain identical idioms were incorrectly answered in pre-tests
idiom
recommended
notrecommended
stop by30
0
due to14
1across the15
2streetthat related to the test questions. Table 9 shows a list ofwords and idioms in the recommended questions citedby students as useful for deriving the answers in the
post-test and the numbers of students who cited them.
The list contains six idioms and five words from various
parts of speech. This result indicates that our method
can extract relationships from various parts of speech.
The computational time for recommending a material
with the proposed method was 0.04 second. The computational efficiency indicates that the proposed method
Table 8: Questionnaire answers: “Did the questions in can be used for a real time recommender system.
the studying phase help in the post-test?”
answerfrequencyYes
42
6 Conclusion
No7
We proposed a method for recommending questions forEnglish multiple-choice cloze questions that maximizes
method regardless of the pre-test score.
learning efficiency based on the expected improvement
We also analyzed whether our method recommended in test scores. The experimental results suggest that ourquestions that can improve test scores when students question recommendation approach is promising andincorrectly answered questions in tests that included will become a useful tool for e-Learning.
the same idioms. Table 7 shows the number of stuAlthough we modeled the expected improvement in
dents who were/were not recommended related ques- test scores with simple logistic regression using only
tions when questions about idioms were incorrectly an- the learning log data and test results to simplify ourswered in the pre-test. This result shows only few cases framework’s novelty, we could also use more informawhere the related questions were not recommended tion about the questions and student attributes for thewith our method.
modeling.
Currently, we are focusing on English multipleAfter studying, students who learned with our
method received questionnaires about the recom- choice cloze questions. However, this recommendation
mended questions. Table 8 shows the answers to the method is not specific to such questions. It can alsoquestion “Did the questions help you in the post-test?” be applied to mathematics and physics, for example,
Since the questions were helpful for about 86% of the because implicit relations among questions for understudents, our method successfully provided questions standing may exist in other fields. In addition, learningmaterials tackled by students in the studying phase canalso be other types of materials. Students learn not onlywith questions but also with textbooks. Our approach
does not depend on questions; it can be applied to othertypes of materials. We would like to apply our proposed
method to other courses of learning and other types oflearning materials. We plan further verification of our
proposed method by comparing other methods with actual students.

References[1] M. E. S. A, E. Aimeur, and C. Frasson. Towards a casebased intelligent tutoring system for student modelling.
In ICCE ’98: Proceedings of the International Conference on Computers in Education, volume 1, pages 528–
535, 1998.
[2] G. Adomavicius and A. Tuzhilin. Toward the next generation of recommender systems: A survey of the stateof-the-art and possible extensions. In IEEE Trans. on
Knowledge and Data Engineering, volume 17, pages
734–739, 2005.
[3] A. Bradley. The use of the area under the roc curve inthe evaluation of machine learning algorithms. In Pattern Recognition, volume 30, pages 1145–1159, 1997.
[4] C. M. Buff and M. A. Williams. An intelligent tutoring system architecture based on knowledge representation and reasoning. In ICCE ’99: Proceedings of theInternational Conference on Computers in Education,
volume 1, pages 736–743, 1999.
[5] C.-M. Chen, C.-M. Hong, and M.-H. Chang. Personalized learning path generation scheme utilizing genetic
algorithm for web-based learning. WSEAS Transactions
on Information Science and Applications, 3(1):88–95,
2006.
[6] C.-M. Chen, H.-M. Lee, and Y.-H. Chen. Personalized
e-learning system using item response theory. Comput.
Educ., 44(3):237–255, 2005.
[7] S. F. Chen and R. Rosenfeld. A Gaussian prior forsmoothing maximum entropy models. Technical report,
CMUCS–99–108, 1999.
[8] R. K. Hambleton, H. Swaminathan, and H. J. Rogers.
Fundamentals of Item Response Theory. Sage Publications, Newburg Park, 1991.
[9] T. Hastie, R. Tibshirani, and J. Friedman. The Elements of Statistical Learning: Data Mining, Inference
and Prediction. Springer, New York, 2001.
[10] R. Hübscher. Logically optimal curriculum sequences
for adaptive hypermedia systems. In Adaptive Hypermedia and Adaptive Web-Based Systems, pages 121–
132, 2000.
[11] G. F. Knolmayer. Decision support models for composing and navigating through e-learning objects. In

HICSS ’03: Proceedings of the 36th Annual Hawaii International Conference on System Sciences, Washington, DC, USA, 2003. IEEE Computer Society.
[12] M.-G. Lee. Profiling students’ adaptation styles in webbased learning. Comput. Educ., 36:121–132, 2001.
[13] X. Li and S. K. Chang. A personalized e-learning system based on user profile constructed using information
fusion. In Proceedings of the 11th International Conference on Distributed Multimedia Systems, pages 109–
114, Banff, Canada, Sep. 2005.
[14] Y.-C. Lin, L.-C. Sung, and M. C. Chen. An automatic
multiple-choice question generation scheme for english
adjective understanding. In ICCE2007 Workshop Proceedings of Modeling, Management and Generation of
Problems / Questions in eLearning, pages 137–142,
2007.
[15] D. C. Liu and J. Nocedal. On the limited memory BFGS
method for large scale optimization. Math. Programming, 45(3):503–528, 1989.
[16] D. W. Oard and J. Kim. Implicit feedback for recommender systems. In AAAI Technical Report WS-98-08,
pages 81–83, 1998.
[17] P. Resnick, N. Iacovou, M. Suchak, P. Bergstrom, andJ. Riedl. Grouplens: an open architecture for collaborative filtering of netnews. In Proceedings of the ACM
Conference on Computer Supported Cooperative Work,pages 175–186, 1994.
[18] J. B. Schafer, J. A. Konstan, and J. Riedl. E-commerce
recommendation applications. Data Mining and Knowledge Discovery, 5:115–153, 2001.
[19] M. Stern and B. P. Woolff. Curriculum sequencing in a
web-based tutor. In ITS ’98: Proceedings of the 4th International Conference on Intelligent Tutoring Systems,
pages 574–583, London, UK, 1998. Springer-Verlag.
[20] T. Y. Tang and G. Mccalla. Smart recommendationfor evolving e-learning system. In Proceedings of the
11th International Conference on Artificial Intelligencein Education, Workshop on Technologies for Electronic
Documents for Supporting Learning, pages 699–710,
2003.