Recommendation Algorithm
for Learning Materials
that Maximizes Expected Test Scores
Tomoharu Iwata1 , Tomoko Kojiri2 ,Takeshi Yamada1 , and Toyohide Watanabe2
1

NTT Communication Science Laboratories,
2-4, Hikaridai, Seika-cho, “Keihanna Science City,” Kyoto, 619-0237, Japan,{iwata,yamada}@cslab.kecl.ntt.co.jp,
2
Graduate School of Information Science, Nagoya University,
Furo-cho, Chikusa-ku, Nagoya, 464-0803, Japan,{kojiri,watanabe}@watanabe.ss.is.nagoya-u.ac.jp

Abstract. We propose a recommendation algorithm for learning materials that enhances learning efficiency. Conventional recommendation
methods consider user preferences and/or levels, but they do not directly
consider the learning efficiency. With our method, the learning efficiency
is quantified by the expected improvement in the test score, and materials are recommended so as to maximize this expected improvement. The
expected improvement is calculated with logistic regression models that
employ the user’s test result obtained before learning as input. Experimental results using fill-in-the-blank exercises for English learning showthat our method yields major improvements in performance comparedwith random material recommendation.
Key words: e-learning, personalization, adaptive tutoring systems, logistic regression

1

Introduction

With the rapid growth of network and database technologies, e-learning systems
have become widely used in various domains. With e-learning systems, the recommendation of suitable materials or exercises for each user is important because
users have different learning levels, prior knowledge, and goals. In a textbook,the sequence of materials and exercises is fixed. This is very inefficient becausesome users find themselves undertaking exercises that are too easy or readingmaterials that are not examined in the target test.
In this paper, we propose a recommendation algorithm for learning materials
that directly enhances learning efficiency. Although many personalized learningmaterial recommendation algorithms have been proposed, they do not directly
maximize learning efficiency. Instead, they consider user preferences or/and levels [1–4]. With our method, the learning efficiency is quantified by the expectedimprovement in the test score, and materials are recommended so as to maximize the expected improvement. Intuitively speaking, our method recommends
materials so that users provide correct answers to questions that they answeredincorrectly prior to learning.
The expected improvement is calculated by logistic regression models [5] using the questions that the user answered incorrectly before learning as input.
By training the logistic regression models using the learning log data and user
test results, we can automatically extract information about which learning materials contribute to improvements in the test score. Since our method does not
require meta data of the materials, it is applicable to any course.

2

Proposed Method

2.1

Preliminaries

Let xi and yi be variables that represent whether question i is correctly orincorrectly answered before and after the learning phases, respectively, as follows:(1/ − 1 if question i is correctly/incorrectly answered before learning,xi =0
otherwise,(1)(
1/ − 1 if question i is correctly/incorrectly answered after learning,yi =0
otherwise,(2)
where xi = 0/yi = 0 means question i has not been answered before/after
the learning phase. The results of the set of test questions V before and after
the learning phase are represented by vectors x = (xi )i∈V and y = (yi )i∈V ,respectively.
Let zj be a variable that represents whether material j is recommended in
the learning phase as follows:(1 if material j is recommended in learning,zj =(3)0 otherwise.
The recommended materials are represented by a vector z = (zj )j∈M , where M
represents a set of learning materials.
2.2

Recommendation algorithm

The goal of our method is to select materials to be used in the learning phasefrom a set of materials in order to enhance learning efficiency. Learning efficiency
is quantified by the expected improvement in the test score.
The expected improvement in the test score given recommended materials z
is written as follows:X
E(z) =P (i)P (xi = −1)P (yi = 1|xi = −1, z),(4)i∈VTable 1. The material recommendation procedure with our method.
1.
2.
3.
4.
5.

Input the test result before learning x,Initialize the recommended material vector: z = (0, · · · , 0),
Select a material to be recommended ĵ by (6),Update the recommended material vector: z = z +ĵ ,Return to step 3 unless an end condition is satisfied.

where P (i) represents the probability that question i is asked in the test, P (i) +P̄ (i) = 1, in which P̄ (i) represents the probability that question i is not asked
in the test, P (xi = −1) represents the probability that question i is incorrectly
answered before learning, and P (yi = 1|xi = −1, z) represents the probability
that question i is correctly answered after the learning phase when the question
i is incorrectly answered before the learning phase and materials z are recommended. In (4), E(z) is regarded as the expected number of questions that areincorrectly answered before the learning phase and correctly answered after thelearning phase given recommended materials.
When the probabilities that questions are asked are uniform, and questions
that are incorrectly answered before learning are known, the expected improvement in the test score can be simplified as follows:E(z|x) ∝Xi∈V

I(xi = −1)P (yi = 1|xi = −1, z),(5)where I(A) represents an indicator function, i.e. I(A) = 1 if A is true, I(A) = 0
otherwise. We use (5) as the expected improvement in the test score in thefollowing sections for the simplicity.
Our method sequentially selects a material that maximizes the expected improvement from materials that have not yet been recommended as follows:ĵ = arg max E(z +j |x),j:zj =0(6)where z = (zj )j∈M represents currently recommended materials, and z +j repre=1sents recommended materials when material j is newly recommended, or zj+j
000
if j = j 0 and zj+j=zifj=6j.
Table
1showsthematerial
recommendation
0j
procedure with our method. Examples of end conditions include those where
the number of recommended materials, the expected improvement, or the timeperiod of the learning phase exceeds a certain threshold.

2.3

Improvement model

When recommending materials, our method requires improvement model P (yi =1|xi = −1, z), which is the probability of the improvement of question i given

recommended materials z. The improvement is modeled based on logistic regression [5] as follows:
P (yi = 1|xi = −1, z) =1,
1 + exp(−(µi + θi> z))(7)where µi and θi = (θij )j∈M are unknown parameters. Intuitively speaking, µirepresents the ease with which question i is improved, and θij represents theinfluence of material j on the improvement of question i.
The unknown parameters Θ = {µi , θi }i∈V can be estimated by maximizing
the following log likelihood using the learning log data and test results for a setof users N :X X³
L(Θ) =I(xni = −1 ∧ yni = 1) log P (yni = 1|xni = −1, zn )n∈N i∈V´+I(xni = −1 ∧ yni = −1) log P (yni = −1|xni = −1, zn ) ,(8)where xni and yni indicate whether question i is correctly or incorrectly answered by user n before and after the learning phases, respectively, and P (yni =−1|xni = −1, zn ) represents the probability that question i is incorrectly answered by user n after the learning phase when question i is incorrectly answered
before the learning phase and materials z are recommended. The global optimum solution is guaranteed because the above log likelihood based on logistic
regression models is a convex function.

3Experiments

3.1

Setting

We evaluated our method using fill-in-the-blank questions for English learning,which are employed in many tests designed to evaluate grammatical knowledge,
such as TOEIC and TOEFL. In the fill-in-the-blank questions used here, users
select appropriate words with the correct grammar for the blank in the sentencefrom four options.
We implemented a web-based e-learning system for the evaluation. In theexperiment, a user takes a test before and after the learning phase (pre-test andpost-test) to measure the effect of learning on improving the test score. One
question is presented to a user on one web page, and the user answers eachquestion in series. The questions in the pre- and post-tests are the same, andthere are 40 questions, |V | = 40.
The materials recommended in the learning phase are fill-in-the-blank questions with solutions and explanations. One question is recommended to each user
on one web page for learning, and the solution and explanation are presented
on one web page after the user has answered the question. Please note that theusers are not supplied with solutions and explanations in the pre- and posttests. There are 80 learning materials, or questions, |M | = 80, and 40 of theseTable 2. AUC of improvement models based on the Bernoulli distribution and thelogistic regression.
Bernoulli distribution
0.556

Logistic regression
0.592

are recommended to each user in the learning phase. The learning materials aredifferent from the questions in the pre- and post-tests. However, about half ofthe materials are related to test questions, for example they involve questions
about the same idioms and grammatical rules.
3.2

Evaluation of improvement models

Our method requires improvement model P (yi = 1|xi = −1, z). We constructed
and evaluated improvement models using the log data of 52 users with randommaterial recommendations, |N | = 52. We compared improvement models basedon the logistic regression in (7) with the Bernoulli distribution. The Bernoulli
model assumes that the improvement does not depend on the recommended
materials z. The parameters can be estimated based on the maximum likelihood.
For the evaluation measurement, we used the area under the ROC curve(AUC) of the problem to predict whether or not questions that were incorrectly
answered in the pre-test are correctly answered in the post-test. A higher AUC
represents better predictive performance. We computed AUC using leave-oneout cross-validation, which means that we used 52 evaluation data sets, in eachof which one user’s data are used for the evaluation and the data of the other51 users are used for the training. Table 2 shows the AUC. The AUC of theimprovement model based on the logistic regression is higher than that of theBernoulli model, which implies that the recommended materials are importantin terms of predicting the improvement in the score, and we can predict theimprovement in the test score with the logistic regression model.
The highest and second highest θij were θi1 j1 = 0.388 and θi2 j2 = 0.208,
respectively, where question i1 and material j1 are about the idiom ‘stop by’,and question i2 and material j2 are about the idiom ‘across the street’. Thisresult is natural because the recommendation of materials about the same idioms
can improve the test score. Even though our method does not use information
related to question or material content, it automatically extracts the relationshipbetween questions and materials using the learning log data and test results.
We analyzed the relationship between questions and materials using question
i1 and material j1 , which are about the idiom ‘stop by’, as an example. In thepre-test, 32 users answered question i1 incorrectly. The probability of the user answering question i1 correctly in the post-test when material j1 was recommended
was P̂ (yi1 = 1|xi1 = −1, zj1 = 1) = 15/17. In contrast, the probability whenmaterial j1 was not recommended was P̂ (yi1 = 1|xi1 = −1, zj1 = 0) = 2/15. This
result indicates that the recommendation of material j1 is effective in improving
the response to question i1 .

Table 3. Average improvements in the test scores with random recommendation andour method.
Random
4.474

3.3

Our method
8.125

Evaluation of recommendation algorithms

We evaluated the learning efficiency of the proposed recommendation algorithm
by comparing it with a random recommendation algorithm. 38 users studied materials recommended at random, and 49 users studied materials recommended
by our method. Table 3 shows the average improvements in the test scores on
a 100-point scale. Our method provided statistically significant increases compared with the random recommendation (one-tailed t-test, p < 0.04). In thequestionnaires, 86% of users answered that there were helpful materials in thelearning phase with our method.

4Conclusion

We proposed a method for recommending learning materials that maximizes
learning efficiency, or the expected improvement in the test score. The experimental results encourage us to believe that our learning material recommendation approach is promising and will become a useful tool for e-learning. Although
we modeled the expected improvement in the test score with logistic regression
using only the learning log data and test results, we could also use content information about the learning materials and test questions such as difficulties, anduser attributes such as levels. We plan a further verification of our method byapplying it to other courses of learning.

References
1. Chen, C.M., Hong, C.M., Chang, M.H.: Personalized learning path generation
scheme utilizing genetic algorithm for web-based learning. WSEAS Transactions
on Information Science and Applications 3(1) (2006) 88–95
2. Li, X., Chang, S.K.: A personalized e-learning system based on user profile constructed using information fusion. In: Proceedings of the 11th International Conference on Distributed Multimedia Systems, Banff, Canada (Sep. 2005) 109–114
3. Stern, M., Woolff, B.P.: Curriculum sequencing in a web-based tutor. In: ITS ’98:
Proceedings of the 4th International Conference on Intelligent Tutoring Systems,London, UK, Springer-Verlag (1998) 574–583
4. Tang, T.Y., Mccalla, G.: Smart recommendation for evolving e-learning system.
In: Proceedings of the 11th International Conference on Artificial Intelligence inEducation, Workshop on Technologies for Electronic Documents for Supporting
Learning. (2003) 699–710
5. Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning: Data
Mining, Inference and Prediction. Springer, New York (2001)