情報処理学会研究報告
IPSJ SIG Technical Report

マトリクス分解から見た項目反応理論の精度評価と実試験データへの適用
廣瀬 英雄1,a)

概要：マトリクス分解から見た項目反応理論の精度評価を実試験データによって確認した．項目反応理論から推定された応答マトリクスと同等なマトリクス分解による低ランクマトリクスのランクは低いが項目反応理論の予測能力は高い．
キーワード：項目反応理論, マトリクス分解, 特異値分解, フローベニウスノルム, 不完全マトリクス.

Matrix Decomposition Perspective for Accuracy Assessment of Item
Response Theory with Applications to Actual Examination Data
Hirose Hideo1,a)Abstract: This paper investigates the predictive eﬀectiveness of item response theory from matrix decomposition perspective. Comparing the diﬀerence in terms of matrix norm between the observed item response
matrix and the estimated item response matrix with that between the observed item response matrix andthe low-rank approximation matrix generated by the matrix decomposition method, it is found that the rankof the generated low-rank approximation matrix that is equivalent to the estimated item response matrix isvery low. However, the predictive ability of item response theory still seems to be high enough.
Keywords: Item response theory, matrix decomposition, singular value decomposition, Frobenius matrix
norm, incomplete matrix treatment.

1. Introduction

can be obtained using the maximum likelihood estimation
method.

Item response theory (IRT) ([1], [2], [6], [23]) is a the-The maximum likelihood estimators are known to be

ory based on a statistical parametric model that simulta-

consistent and asymptotically eﬃcient under certain con-neously assesses abilities of examinees and diﬃculties ofditions ([10]); that is, no consistent estimators have lower

problems. Because of its versatility and reliability, IRT

asymptotic mean squared errors other than the maximum

has been regarded as one of the standard methods in as-

likelihood estimators. This means that under the assumed

sessing the performance of examinees. For this reason,

mathematical model and its parameter space, the estima-

IRT is used in various oﬃcial examinations, including thetors are the best. However, there might be other models

TOFLE. To an observed item response matrix consistingthat are better than IRT. While the AIC (Akaike’s infor-of examinee user rows and problem item columns, esti-

mation criterion) is often used to compare the superiority

mates of IRT parameters and their confidence intervalsof parametric models, we use another criterion, the root
mean squared error (RMSE), in order to accomplish such1a)Bioinformatics Center, Kurume University Graduate Schoolof Medicine, Fukuoka Tokyo 830–0011, Japanhirose hideo@kurume-u.ac.jp

c 1959 Information Processing Society of Japan⃝a purpose. This is a primary challenge to see the prediction eﬀectiveness of item response theory from a new1

情報処理学会研究報告
IPSJ SIG Technical Report

perspective; we look at the item response matrix itself di-

the objective of the paper is to clarify the position of therectly from the matrix decomposition perspective. The

IRT performance in the sense of low-rank approximation

typical data cases corresponding to this new look appear

matrix equivalent to the estimated item response matrix

in the examination data in education.

using a number of actual examination data sets.

Using the estimates for parameters in IRT, the item
response matrix can be reconstructed; we call this theestimated item response matrix. Then, the diﬀerence be-

2. Item response theory
2.1 Mathematical model

tween the observed and estimated item response matrices

The standard IRT estimates proficiency parameters θican be computed using an appropriate matrix norm such

(i = 1, . . . , n) and problem parameters aj , bj , cj (j =as the Frobenius matrix norm. Thus, it is possible to mea-

1, . . . , m) simultaneously by using the observed item re-sure how close the observed item response matrix is to thesponse matrix. Usually, this item response matrix (ma-estimated item response matrix. This is the criterion totrix size is n×m) consists of 1/0 valued elements δij , withmeasure the diﬀerence between two matrices.

the value 1 for the (i, j) element corresponding to the caseMany researchers have proposed new methods toachieve superiority over the standard IRT performance.

where examinee i solved question j correctly and the value
0 for the case where he/she solved it incorrectly.

For example, multidimensional item response theory

Assume that the logistic probability function pij of ex-(MIRT) (see [15]) and knowledge tracing (KT) (seeaminee i correctly answering question j is expressed such[13], [24]) have been proposed to find examinee proficiency

thatusing parametric models. [25] describe the results of a

1 − cj,
1 + exp{−1.7aj (θi − bj )}(1)= 1 − qij (θi ; aj , bj , cj ),pij (θi ; aj , bj , cj ) = cj +performance comparison among those parametric models.

In the evaluation of parametric models, the log-

likelihood values are primarily used. In a nonparametric

where θi is called the ability for examinee i and aj , bj , cj

approach, [21] show how to predict student performanceare called the discrimination parameter, diﬃculty param-

using a recommender system. In addition, [20] use a rec-eter, and pseudo-guessing parameter, respectively; qij isommender system to predict student performance. Since

the probability that examinee i answers question j incor-

recommender systems often use a matrix factorization al-

rectly.

gorithm, the RMSE is used to evaluate the closeness ofthe two matrices. More complex cases have also been proposed, with [11] integrating KT and IRT, and [26] comparing deep learning approaches to simple IRT.
However, unlike papers that provide such new mathematical models, this paper intends to examine the effectiveness of IRT itself from a diﬀerent perspective as
slightly mentioned above using actual educational datacases.

To accomplish this, we use matrix decomposi-tion (MD) perspective. [7] introduced matrix completion(MC) and low-rank singular value decomposition (SVD)
to evaluate the diﬀerence between two matrices. By using SVD, we can obtain a low-rank matrix that is close to
the original matrix in the sense of Frobenius matrix norm.
The second challenge of this paper is to apply the method

2.2 Parameter estimation
Using the maximum likelihood estimation (MLE)
method, the maximum likelihood estimates θ̂i and âj , b̂j ,ĉj for parameters θi and aj , bj , cj can be obtained by maximizing the likelihood function,
n ∏
m (
)
∏
δ
1−δ
L=
pijij × qij ij .

(2)

i=1 j=1

When only diﬃculty parameter bj is considered, suchthe model is called the Rasch model. Usually, the twoparameter model (cj = 0) is the standard, and we willdeal with this case below. Also, in terms of recommendersystems, we will refer to examinees as users and questions
as items.

to a number of actual examination data cases performed

If we denote parameters θi and aj , bj , cj together by Θ,at universities. By applying the matrix decomposition andand the observed matrix by ∆ = (δij ), then the estimation

singular value decomposition methods to more than 40 examination data cases, ranging from small to large matrix

process is expressed as follows.
∆ ; Θ̂.

(3)

sizes, we were able to derive a very clear conclusion about
the eﬀective approximated low-rank matrix that is equivalent to the estimated item response matrix. Therefore,c 1959 Information Processing Society of Japan⃝

2.3 Estimated item response matrix
Applying the delta method to pij in equation (1) as a2

情報処理学会研究報告
IPSJ SIG Technical Reportfunction of Θ̂, we can obtain δ̂ij which is a continuous

indicator function; i.e., Iii = 1, and Iij = 0 (i ̸= j).

value in [0, 1]. This estimation process can be expressed

We make vector ui by ui = Avi /σi , (i ≥ r), where√σi = ξi . In addition, if we produce matrices U = (ui )such thatˆΘ̂ ; ∆,(4)ˆ the estimated item response matrix. The
and we call ∆value δ̂ij is corresponding to the probability of correctly

and V = (vj ), then A can be expressed as A = U ΣV T ,∑ror equivalently, A = l=1 σl ul vlT . Here, Σ is a diagonal
matrix using σi . This is the typical singular value decomposition (SVD) (see [5], [18], [19]).

answering the question using equation (1).
Considering such treatment, the δij value is extended
from a discrete value of 1/0 to a continuous value of [0, 1],

3.2 Generating the low-rank matrix
We define Ak such that

although δij takes values δij = 1 if the question is successfully answered and δij = 0 if it is not. We also deal with
the null value of the element (i, j), corresponding to thecase where the examinee i has not tackle problem j, orthe case that the response is unknown. How to deal withsuch cases is explained in [9], [16], [17].
Once, estimates âj and b̂j for aj and bj are obtained,
we can perform estimation procedure for θi to each i independently. If there is a sequence of random variables
Xl satisfying√dl[Xl − θi ] −→ N (0, σi2 ),(5)where θi and σi2 are finite valued constants, N is a normalddistribution, and −→ denotes convergence in distribution,then√dl[g(Xl ) − g(θi )] −→ N (0, σi2 · [g ′ (θi )]2 ),

Ak =

k
∑

σl ul vlT ,

(8)

l=1

using the first k columns in the matrices of U and V .
This procedure generates the low-rank matrix Ak for A as
shown below.
It is interesting to remind the following theorem ([3]).
Theorem1 (Eckart-Young)1) rank(Ak ) = k
2) For any m × n matrix B, (rank(B) ≤ k),n∑σl2 )1/2 ,||A − Ak ||F =min||A − B||F = (B,rank(B)≤kl=k+1

where || · ||F means the Frobenius matrix norm, i.e.,∑||(aij )||F = ( i,j |aij |2 )1/2 .
The theorem claims that Ak is best approximated to A
among all the matrices with rank of less than k + 1 in the(6)sense of matrix norm.

for any function g satisfying the property that g ′ (θ) exists
and is non-zero valued. According to this, and regarding
g as pij in equation (1), δ̂ij becomes optimal in the likeliˆ is optimal in the mathematicalhood sense. Therefore, ∆3.3 Construction of the low-rank item response
matrixWhen we regard A as the observed item response ma-model assuming equation (1) and the parameter space oftrix ∆, and we regard Ak as ∆k , we can construct theΘ.

low-rank item response matrix ∆k from ∆. When it isdesired to emphasize that ∆k is derived from SVD, it is

3. Singular value decomposition

denoted as ∆SVD
if necessary.
k

3.1 Singular value decomposition procedure

4. Matrix decomposition

Assuming that A = (aij ) is a m × n matrix. Then, AT A
becomes a n × n symmetric matrix, and AAT becomes a

4.1 Matrix decomposition procedurem × m symmetric matrix, where A denotes the trans-

SVD is a promising method to find an approximate ma-pose of A. Eigen values and eigen vectors to these twotrix that is close to a certain matrix in the sense of the ma-

matrices AT A and AAT are the same if they exist. We

trix norm. However, it requires that all the elements must

denote the eigen values and eigen vectors to matrix AT A

be occupied. Occasionally, one encounters cases where notas {ξ1 , ξ2 , · · · , ξn } and {v1 , v2 , · · · , vn }. That is,

all elements of the observed response matrix are occupied.

T

AT Avi = ξi vi .

In such cases, the matrix decomposition (MD) method(7)

Eigen values can be reordered such that ξ1 ≥ ξ2 ≥ · · · ≥ξr > 0, ξr+1 = · · · = ξn = 0, where r is the rank of A A.
T([12]) can be used; other methods, such as the imputation
method ([22]) or the matrix completion method ([7]) arealso used.

Since A A is symmetric, eigen vectors can be made as or-

MD is similar to SVD. The target matrix R ∈ Rm×n

thonormal system. That is, vi · vj = Iij , where Iij is thecan be constructed from two matrices U ∈ Rm×k andT

c 1959 Information Processing Society of Japan⃝3

情報処理学会研究報告
IPSJ SIG Technical ReportV ∈ Rn×k , but the decomposed form has not the diago-

We name this example case A.
The figure on the left in Figure 1 shows the observed

nal singular value matrix found in SVD. MD is described

item response matrix. In the figure, only the responses ofas

28 users are shown for clarity. This matrix is composed

R = U V T.

(9)of binary elements, with 1 for correct answers and 0 for

Using the values of the non-null elements of A, we find U

incorrect answers. The observed item response matrix isand V so that

denoted as ∆ = (δij ).

E=

m ∑
n
∑

Iij (aij − rij )2(10)

i=1 j=1

becomes small, where rij =∑k

l=1 uil vjl , and Iij is the in-dicator function such that Iij = 1 if aij is non-null and
Iij = 0 if aij is null. For stable computation, we use another function with penalty terms such that
W =

m ∑
n
∑

Iij (aij − rij )2 + ku

i=1 j=1m ∑k∑
i=1 l=1

u2il + kv

n ∑
k
∑

2
vjl
,

j=1 l=1

where, ku and kv are regularization factors to prevent
overfitting. To find the optimum value, we use the descent

図 1 Observed item response matrix and estimated item re-sponse matrix.

method ([4], [14]). From appropriately set initial values
(0)

(0)

(t+1)

− uil | and |vjl

of uil and vjl , we proceed the following iterations until
|uil

(t)

(t+1)

(t)

− vjl | are suﬃciently small.

∂W (t)
(t+1)
(t)
|
uil
← uil − λ
∂uil
(t+1)

vjl

(t)

← vjl − λ∂W (t)| ,∂vjl

5.2 Estimated item response matrix by using IRT
Applying the maximum likelihood estimation method
to this observed item response matrix ∆ yields the max-

(11)

where, λ is the learning coeﬃcient. This is a typical MD
procedure ([12]).
4.2 Construction of the low-rank item response
matrixWhen only k vectors are used for the matrices U and
V , we denote R in such a case as Rk . As in the SVD case,when we regard R as the observed item response matrix∆, and we regard Rk as ∆k , we can construct the low-rank
item response matrix ∆k from ∆. When it is desired toemphasize that ∆k is derived from MD, it is denoted as∆MD
if necessary.
k

5. Examination data analysis (complete
matrix treatment)
5.1 A typical example case of the observed item
response matrix
As a typical case for complete matrix treatment, we use

imum likelihood estimate Θ̂ for the parameter Θ. Using
this estimated value Θ̂, the estimated item response maˆ can be reconstructed. As explained earlier, this ∆ˆtrix ∆
is optimal in the sense of the likelihood principle. The
ˆ Comparing
figure on the right in Figure 1 shows this ∆.
ˆ and ∆ in Figure 1, we can roughly imagine the original∆
ˆ However, thisobserved item response matrix ∆ from ∆.
approximation appears to be inaccurate.
To see if this is correct, we will now use the Frobenius
matrix norm. Using this matrix norm, the proximity of
two equal-sized matrices A = (aij ) and B = (bij ) can be
expressed by the RMSE(A, B) such that
v
u
n ∑
m
u 1 ∑
(aij − bij )2
RMSE(A, B) = t
nm i=1 j=1√
1=(||A − B||F )2 .
nm

(12)

ˆ ∆) of the diﬀerence between
In this case, the RMSE(∆,an observed item response matrix obtained from a mathe-

the observed item response matrix ∆ and the estimatedˆ is computed to be 0.3915. This
item response matrix ∆matics midterm examination given at a certain university.

indicates that the distance between an observed δij and

The number of examinees n is 216 and the number of ques-

its estimated value δ̂ij lies on average around 0.3915. In-

tions m is 31. There are no missing data in this matrix.

tuitively, this value does not seem small.

c 1959 Information Processing Society of Japan⃝4

情報処理学会研究報告
IPSJ SIG Technical Report

5.3 Low-rank item response matrix
ˆ by IRTAs described above, the estimated matrix ∆does not seem to accurately reproduce the observed response matrix, although each estimate δ̂ij becomes optimal in the likelihood sense under the defined parameter
space and likelihood function. Therefore, we investigate
ˆ is in terms of MD and SVD.
how accurate ∆Applying the methods described in sections 3 and 4,
the low-rank item response matrices ∆SVD
and ∆MDcankk
be computed.

Table 1 shows the RMSE (∆SVD, ∆) andk

RMSE(∆MDk , ∆) in the cases of k

= 1, . . . , 5, 10, 20, 31.
ˆThe table also shows the RMSE(∆, ∆) corresponding to

図 2 Low-rank matrices ∆SVD
and ∆MD
reproduced from the22

oserved item response matrix.

would be necessary to collect other examination cases tosee if these properties hold true in other cases.

IRT estimation.
The table shows, first, that the performance of SVD and
MD are almost the same. In other words, MD catches up
well with SVD. This means that MD can be applied as analternative method when SVD cannot be used directly.
Such a case may occur in the case of incomplete matrix,
especially when the matrices are sparse.
ˆ ∆) obtained by IRT lies
Next, we see that the RMSE(∆,between the RMSE (∆1 , ∆) and RMSE(∆2 , ∆) in boththe SVD and MD cases. This is amazing in terms of maˆ
trix approximation. The estimated response matrix ∆using IRT would not exceed the accuracy obtained froma k = 2 low-rank response matrix generated from theobserved item response ∆. In this example, this value(k = 2) would be very small given that the rank of observed matrix ∆ is 31. In other words, the reproducibilityof the observed item response matrix appears to be low

5.4 42 examination cases
To make sure that the above mentioned properties holdtrue for other examination cases, 42 examination cases
were collected, including case A. For all examinations, answers were given as discrete values of 1/0 (1 for correct
answers and 0 for incorrect answers).
ˆ ∆) for the 42 examinaFigure 3 shows the RMSE(∆,tion cases. In the figure, the case id shown on the horizontal axis are arranged in ascending order of the magˆ ∆) shown on the vertical axis fornitude of the RMSE(∆,easy understanding. Also shown are the RMSE(∆SVD, ∆)k(k = 1, 2, 3) for each case id. Looking at the figure, weˆ ∆) is obtained in
see that RMSE(∆SVD , ∆) < RMSE(∆,2

all cases. This suggests that the eﬀectiveness of IRT issimilar to that of a very low-rank approximation matrix.

for IRT.
Such properties can also be seen in the pictures of the
low-rank response matrices. The figure on the left in Figure 2 shows ∆SVD
and that on the right ∆MD2
2 . These areˆ in Figure 1.
very similar to ∆表 1 RMSE of the diﬀerence of the two matrices between the

low-rank response matrix and the observed response mak

trix
, ∆)
RMSE(∆SVDk

RMSE(∆MDk , ∆)

1

0.4066

0.406720.38510.385430.36520.3656

40.34790.348550.33060.3314100.25620.2583

However, this is the result when all the element data of

20

0.13250.1400∆ are used, i.e., when the estimation is performed with

31

00.0570training data only. In this case, the estimated values δ̂ijˆ ∆)RMSE(∆,
0.3915ˆ ∆) and RMSE(∆SVD , ∆) (k = 1, 2, 3) for 42
図 3 RMSE(∆,kcomplete matrix using full element data.

may be overfitting values. In order to clarify whether
However, this is only the result of one case study. It

c 1959 Information Processing Society of Japan⃝the estimates obtained with IRT are really inaccurate, we

5情報処理学会研究報告
IPSJ SIG Technical Reportnext investigate the prediction accuracy in the case of IRT,of the RMSE(T̃kSVD , T ) and RMSE(T̃kMD , T ) for the boot-SVD and MD using the training data and test data.

strapped 10 cases are located near the mean value of the

6. RMSE for test data using incomplete
matrix treatment

RMSE(T̃ , T ).
Table 2 shows the mean values µ(RMSE(T̃kSVD , T )) andµ(RMSE(T̃kMD , T )) as well as the mean values of theWe, here, create two matrices S and T for the trainingµ(RMSE(T̃ , T ) using 10 bootstrapped cases, where k =and test data from the observed item response matrix ∆,1, . . . , 10. In the table, boldface type denotes the smallest

respectively. We denote the corresponding estimated ma-

value among the various k values. For MD, the smallest

trices for S and T as S̃ and T̃ in IRT. Similarly, we denote

RMSE(T̃kMD , T ) is obtained when k = 3, and for SVD, thethe low-rank matrices with rank k for S and T as S̃kSVD
and T̃kSVD in SVD, and as S̃kMD and T̃kMD in MD.

smallest RMSE(T̃kSVD , T ) is obtained when k = 5. It isconsidered that the RMSE(T̃kSVD , T ) or RMSE(T̃kMD , T )would not be smaller than a certain value, unlike the

6.1 Case A

monotonically decreasing behavior of the RMSE(S̃kSVD , S)

Using examination example case A in section 5, we com-or RMSE(S̃kMD , S). In this case, this lower limit is locatedpute the RMSE of the training and test data. To do this,in the neighborhood of the RMSE(S̃, S). This means that

10% of the elements are randomly selected from the orig-

the accuracy (reliability) of mimicking the original data ∆inal matrix as test data, and the remaining 90% elementsof IRT in terms of prediction can be explained by a very

are used as the training data. Then, the RMSE(S̃, S) for

low-rank matrix decomposed by SVD or MD.

the training data and RMSE(T̃ , T ) for the test data are
obtained. Since there may be fluctuations of the RMSE
due to the selection of test data, this is repeated (bootstrapped) 10 times and the average RMSE is expressed asµ(RMSE).
Figure 4 shows the RMSE for the training test datausing MD and SVD. Circles denote the RMSE(S̃kSVD , S)
and RMSE(T̃kSVD , T ) for each data selection, and similarly
triangles denote the RMSE(S̃kMD , S) and RMSE(T̃kMD , T ).
For reference, the µ(RMSE(S̃, S)) (mean value of the
RMSE(S̃, S)) and the µ(RMSE(T̃ , T )) (mean value of the
RMSE(T̃ , T )) using 10 bootstrapped cases reproduced by

図 4 RMSE for the test data and the training data via MD and

SVD (case A).

IRT are shown with horizontal dotted lines. In the figure,the cases of k = 1, . . . , 10 for MD and SVD are presented.
Looking at the figure, as k increases, the RMSE of

表 2 Smallest mean values of 10 bootstrapped RMSE for thethe training data shows monotonically decreasing characteristics as expected.

test data to each ∆kMoreover, the range of variakµ(RMSE(T̃kSVD , T ))µ(RMSE(T̃kMD , T ))also small. As explained earlier, the RMSE(S̃k , S) with1

0.41680.4160training data is found to lie between RMSE(S̃1SVD , S) and2

0.40850.4095

RMSE(S̃2SVD , S). This property is also found in the MD

3

0.4026

0.4053case.

40.40250.405950.39850.409660.40090.412670.40190.42568

0.40250.4326

curve (first decreasing and later increasing with increasing

90.40480.4403k). Such V -shaped curves due to model complexity are

100.40520.4556tion in RMSE among the 10 bootstrapped data sets is

However, for the test data, the RMSE(T̃kSVD , T ) curves
as a function of k were found to exhibit very diﬀerent
curve shape patterns. The RMSE follows a V -shapedˆ ∆)RMSE(∆,
0.4056

well known (e.g., [8]). In addition, the RMSE varies tosome extent among 10 bootstrapped data cases. Surpris-

Recall that the RMSE for IRT lies between the k = 1

ingly, near the bottom of the V -shaped curve, the values

RMSE and the k = 2 RMSE for MD or SVD with fullc 1959 Information Processing Society of Japan⃝6

情報処理学会研究報告
IPSJ SIG Technical Reportmatrix data, as indicated in Table 1. This superficial re-using 10 cases reproduced by IRT are shown using hori-sult seems to indicate that IRT’s ability to represent thezontal dotted lines. In the figure, cases of k = 1, . . . , 30

observed item response matrix is weak. This is becauseˆ is similar to that using a
the prediction ability using ∆for SVD are illustrated.

very low-rank approximation matrix, although the rank

using 10 bootstrapped cases by IRT lies between theof matrix ∆ is the same as the minimum number of users

RMSE(T̃1SVD , T ) and RMSE(T̃2SVD , T ).

and items, which in this case is 31.

µ(RMSE(S̃, S)) lies between the RMSE(S̃1SVD , S) andAs shown in the figure, the mean value µ(RMSE(T̃ , T ))In addition,

However, as seen above, the RMSE in the test case is

RMSE (S̃2SVD , S). This property is the same as the fact

also close to the RMSE of the best reconstructed low-rank

that the RMSE(S̃, S) using the training data lies between

matrix (rank lower than 5) using the matrix decomposi-

the values of RMSE(S̃1SVD , S) and RMSE(S̃2SVD , S) for one

tion. This tells us that the mathematical model of IRT

case. Such a result may be realized by chance, but the(two-parameter logistic model) is well defined to represent

property that the µ(RMSE(T̃kSVD , T )) is not much diﬀer-

the actual examination case (case A). However, this char-ent from the µ(RMSE(T̃ , T )) remains the same as in otheracteristic could be found only in certain cases. Therefore,

7 cases.

we next investigate whether such a characteristic applies
to other examination cases, using various data cases.

In other words, the potential ability of IRT to mimic
the observed response matrix ∆ is found to be equivalent
to the ability of the low-rank approximation matrix gen-6.2 8 examination cases among 42 cases

erated by matrix decomposition to mimic the observed re-

From 42 data cases, we picked 8 cases, including casesponse matrix, not only in the full training data use study

A, to verify whether the RMSE of the test cases in IRT isbut also in the training and test data use study. Althoughclose to the RMSE of the low-rank matrix in SVD. Table

the rank of the low-rank approximation response matrix
ˆ is extremely
by matrix decomposition corresponding to ∆3 shows µ(RMSE(T̃kSVD
opt , T )) and µ(RMSE(T̃ , T )), wherek opt means k for which µ(RMSE(T̃kSVD
opt , T )) is minimum
when k = 1, . . . , kmax .

smaller than the rank of the observed response matrix ∆,the predictive ability of IRT seems to be high enough since

Looking at the table, in almost all cases, we see that

the RMSE(T̃kSVD
opt , T ) is almost equal to µ(RMSE(T̃ , T )).

µ(RMSE(T̃kSVD
opt , T )) is close to µ(RMSE(T̃ , T )) and k opt

In other words, if the size of the item response matrix

is very small. The exception is case B (case id is 30), whereis moderate, i.e., less than 1000 users and less than 100

the matrix size is n = 1101, m = 84, and k opt = 16.

items, it would be diﬃcult to obtain more information

In this case, the number of examinee is large to somethan IRT produces from the observed item response ma-

extent and µ(RMSE(T̃kSVD
opt , T )) is clearly smaller thantrix alone using the matrix decomposition method.

µ(RMSE(T̃ , T )). Thus, we next consider the relation between the RMSE(T̃kSVD
opt , T ) and RMSE(T̃ , T ) in case B.
表 3 Mean values of 10 bootstrapped RMSE for the test data

to 8 data cases
case id µ(RMSE(T̃kSVD
opt , T ))k optµ(RMSE(T̃ , T ))5

0.293510.2908100.35471

0.359115

0.34571

0.334420

0.3801

1

0.3726250.370150.378930

0.344216

0.377135

0.39822

0.396440

0.405330.4068Figure 5 shows the RMSE for the training data and thetest data using SVD. Circles indicate the RMSE(S̃kSVD , S)

図 5 RMSE for the test data and the training data via MD and

SVD (case30).

7. Concluding remarks

and RMSE(T̃kSVD , T ) for each data selection. For refer-

This paper has investigated the predictive eﬀectiveness

ence, the µ(RMSE(S̃, S)) (mean value of the RMSE(S̃, S))of item response theory itself from matrix decomposition

and the µ(RMSE(T̃ , T )) (mean value of the RMSE(T̃ , T ))perspective.

c 1959 Information Processing Society of Japan⃝7

情報処理学会研究報告
IPSJ SIG Technical Report

If the observed item response matrix is given, the maximum likelihood estimates for parameters in item response
theory can be estimated, and the estimated item response
matrix is also reconstructed using the estimates. Then,

[10]
[11]

the diﬀerence between the observed and estimated item
response matrices can be determined in the sense of matrix
norm. Matrix decomposition and singular value decomposition methods can generate a low-rank approximation[12]

matrix from the observed item response matrix, and thediﬀerence between the observed and the generated low-

[13]

rank matrix can be measured in the sense matrix norm.
The eﬀectiveness of item response theory may be evaluated by comparing the two matrices between the estimated item response matrix obtained from the item re-

[14]
[15]

sponse theory and the low-rank approximation matrix obtained from matrix decomposition or singular value de-[16]

composition.
Applying item response theory, matrix decomposition,and singular value decomposition to many actual exam-[17]

ination data, it is found that the rank of the generated
low-rank approximation matrix that is equivalent to theestimated item response matrix is very low. However, the[18]predictive ability of IRT seems to be high enough since
the minimum root mean squared errors for test data us-[19]

ing matrix decomposition and singular value decomposition methods are almost equal to the root mean squared[20]

error from item response theory.
参考文献
[1]

[2]
[3]

[4]
[5]
[6]

[7]

[8]
[9]

Baker, F. B. and Kim, S.-H.: Item Response Theory: Parameter Estimation Technique, 2nd edn., Marcel Dekker (2004).
de Ayala, R.: The Theory and Practice of Item Response Theory, Guilford Press (2009).
Eckart, C. and Young, G.: The Approximation of One
Matrix by Another of Lower Rank, Psychometrika,
Vol. 1, pp. 211–218 (1936).
Fletcher, R.: Practical Methods of Optimization, Wiley(2000).
Golub, G. H. and Van Loan, C. F.: Matrix Computations, Johns Hopkins Univ. Press (2012).
Hambleton, R., Swaminathan, H. and Rogers, H. J.:
Fundamentals of Item Response Theory, Sage Publications (1991).
Hastie, T., Mazumder, R., Lee, J. D. and Zadeh, R.:
Matrix Completion and Low-Rank SVD via Fast Alternating Least Squares, Journal of Machine Learning Research, Vol. 16, pp. 3367–3402 (2015).
Hastie, T., Tibshirani, R. and Friedman, J.: The Elements of Statistical Learning, Springer (2009).
Hirose, H. and Sakumura, T.: Item Response Prediction
for Incomplete Response Matrix Using the EM-type Item
Response Theory with Application to Adaptive Online
Ability Evaluation System, IEEE International Confer-c 1959 Information Processing Society of Japan
⃝

[21]

[22]

[23]
[24]

[25]

[26]

ence on Teaching, Assessment, and Learning for Engineering, pp. 8–12 (2012).
Kendall, M. G. and Stuart, A.: Advanced Theory of
Statistics, Macmillan Pub Co (1983).
Khajah, M., Huang, Y., Gonzlez-Brenes, J. P., Mozer,
M. C. and Brusilovsk, P.: Integrating knowledge tracing and item response theory: A tale of two frameworks, 4th International Workshop on Personalization
Approaches in Learning Environments, pp. 7–15 (2014).
Koren, Y., Bell, R. M. and Volinsky, C.: Matrix Faxctorization Techniques for Recommender Systems, Computer, Vol. 42, pp. 30–37 (2009).
Piech, C., Bassen, J., Huang, J., Ganguli, S., Sahami,M., Guibas, L. J. and Sohl-Dickstein, J.: Deep knowledge tracing, Advances in Neural Information Processing Systems, pp. 505–513 (2015).
Polak, E.: Optimization : Algorithms and ConsistentApproximations, Springer (1997).
Reckase, D.: Multidimensional Item Response Theory,
Springer (2011).
Sakumura, T. and Hirose, H.: Making up the Complete
Matrix from the Incomplete Matrix Using the EM-type
IRT and Its Application, Transactions on Information
Processing Society of Japan (TOM), Vol. 72, pp. 17–26(2014).
Sakumura, T., Kuwahata, T. and Hirose, H.: An Adaptive Online Ability Evaluation System Using the Item
Response Theory, Education & e-Learning, pp. 51–54(2011).
Strang, G.: Multiplying and Factoring Matrices, The
American Mathematical Monthly, Vol. 125, pp. 223–230(2018).
Strang, G.: Introduction to Linear Algebra, WellesleyCambridge Press (2021).
Sweeney, M., Lester, J., Rangwala, H. and Johri, A.:
Next-term student performance prediction: A recommender systems approach, Journal of Educational Data
Mining, Vol. 8, pp. 22–51 (2016).
Thai-Nghe, N., Drumond, L., Horva, T., KrohnGrimberghe, A., Nanopoulos, A. and Schmidt-Thieme,
L.: Factorization techniques for predicting student performance, Educational recommender systems and technologies: Practices and challenges, pp. 129–153 (2011).
Troyanskaya, O., Cantor, M., Sherlock, G., Brown, P.,
Hastie, T., Tibshirani, R., Botstein, D. and Altman,
R. B.: Missing Value Estimation Methods for DNA Microarrays, Bioinformatics, pp. 520–525 (2001).
van der Linden, W. J.: Handbook of Item Response Theory, Chapman and Hall/CRC (2016).
Vie, J.-J.: Deep factorization machines for knowledge
tracing, Thirteenth Workshop on Innovative Use of
NLP for Building Educational Applications, pp. 370–
373 (2018).
Vie, J.-J. and Kashima, H.: Knowledge Tracing Machines: Factorization Machines for Knowledge Tracing,
The Thirty-Third AAAI Conference on Artificial Intelligence, pp. 750–757 (2019).
Wilson, K. H., Xiong, X., Khajah, M., Lindsey, R. V.,
Zhao, S., Karklin, Y., Van Inwegen, E. G., Han, B.,
Ekanadham, C., Beck, J. E., Heﬀernan, N. and Mozer,
M. C.: Estimating student proficiency: Deep learning
is not the panacea, Workshop on Machine Learning for
Education, Neural Information Processing Systems, pp.
1–8 (2016).

8