Probabilistic Modeling of Peer Correction
and Peer Assessment
Takeru Sunahase

Yukino Baba

Hisashi Kashima

Kyoto University

University of Tsukuba

stakeru@ml.ist.i.kyotou.ac.jp

baba@cs.tsukuba.ac.jp

Kyoto University;
RIKEN Center for AIP

ABSTRACT
Peer assessment is a promising solution for scaling up thegrading of a large number of submissions. The reliability ofevaluations is one of the critical issues in peer assessment;
several probabilistic models have been proposed for obtaining reliable grades from peers. Peer correction is a similar
framework, in which students are instructed to correct theerrors in submissions from other students. Peer correction
is typically performed simultaneously with peer assessment;a reviewer is instructed to correct the errors in a submission
and to provide a grade to it. We observe the occasional inconsistency between a grade and the correction; for example,
a reviewer provides a high grade for a submission but she
corrects many errors in it. Such inconsistencies can point tounreliable reviewers. In this paper, we propose probabilistic
models for peer correction, and the combination of the peer
correction models and the existing peer assessment models for capturing the inconsistency to accurately estimate
the reviewer reliability and the student ability. We conduct
experiments using the dataset of an actual peer correction
platform for language translation, and the results demonstrate that the combination of peer correction models andpeer assessment models improves the accuracy of the student
ability estimation.

Keywords
Peer correction, peer assessment, statistical models

1.

INTRODUCTION

MOOCs have changed education by oﬀering open access touniversity course materials; however, not everything performed in oﬄine classes is eﬀectively introduced in MOOCs.
An example is the ability assessment; in oﬄine classes,
teachers evaluate the student abilities by examining theirsubmitted assignments and decide how to improve the educational eﬃciency. In contrast, assessing the abilities oftens of thousands of students in MOOCs is not feasible forteachers.

Takeru Sunahase, Yukino Baba and Hisashi Kashima"Probabilistic Modeling of Peer Correction and Peer Assessment"
In: Proceedings of The 12th International Conference on
Educational Data Mining (EDM 2019), Collin F. Lynch, Agathe
Merceron, Michel Desmarais, & Roger Nkambou (eds.) 2019, pp.
426 - 431

kashima@i.kyoto-u.ac.jp

A promising solution for large-scale ability assessment is toallow students themselves to be involved in the evaluation;
instead of teachers, students grade the submissions fromother students. Such peer assessment approach is beneficial
for scaling-up the ability assessment and it has been applied
to several MOOCs courses [7]. However, the reliability ofevaluations is one of the critical issues in peer assessment
because some students may provide unreliable evaluations
owing to laziness or lack of evaluation skills. Several probabilistic models have been proposed for estimating the reliabilities of the reviewers in order to accurately assess thestudent abilities in peer assessment [7, 4, 11, 8, 13, 6]. Thesemodels are based on the assumption that students with highability are likely to provide reliable grades. The models areused to estimate the ability of a student as a test taker andthe reliability as a reviewer.
In a similar framework of peer assessment, called peer correction, students correct the errors in the submissions fromother students. Peer correction is helpful for teachers to
reduce their eﬀorts for providing feedback to the students.
Typically, peer correction is performed simultaneously withpeer assessment; a student is instructed to grade a submission and to correct its errors.
Although the outcomes of peer correction are naturally assumed to be informative for estimating the student abilities,
probabilistic models for peer correction have not yet been
investigated. Based on a natural assumption that a student
who receives fewer corrections are likely to have a higher
ability, we propose probabilistic models for peer correctionthat capture the relationship between the student abilities
and the correction outcomes.
Additionally, we noticed an inconsistency between the outcomes of peer correction and those of peer assessment. In
one case, a reviewer provides a high grade to a submission
but she corrects many errors in it; in another case, a reviewer
assigns a low grade but she does not make any corrections.
Our idea is that such inconsistencies are beneficial in determining unreliable reviewers; thus, we propose to combine
peer assessment models with our peer correction models.
This combination allows us to capture the inconsistency andto incorporate it into the estimation of the reviewer reliability and the student ability.
We conduct experiments using a peer correction dataset
about language translation. The results of the experiments

Proceedings of The 12th International Conference on Educational Data Mining (EDM 2019)426

show that our probabilistic models for peer correction are capable of estimating the student abilities, and the combinedmodels of peer correction and peer assessment demonstrate
a better performance in determining high-ability students
than the peer assessment models.
The contributions of this paper are twofold: (i) we propose
novel probabilistic models for peer correction that enable us
to estimate the student abilities from the received corrections (Section 4), and (ii) we propose to combine our peer
correction models and peer assessment models to exploit theinconsistencies among the outcomes of corrections and assessments (Section 6); the results of the experiments showthat the combined models are eﬃcient in accurately estimating the student abilities.

2.

PROBLEM DEFINITION

We begin with the formulation of peer assessment and peer
correction. We assume there is a set of students S. Whena student creates a submission for an assignment, other students (that we call reviewers) evaluate it and assign grades.
The grade for the student u ∈ S assigned by the reviewerv ∈ S is denoted by zuv ∈ R. Each reviewer is additionally
instructed to correct the errors in a submission. A correction result is denoted by yuv . If a reviewer does not provide
any correction for a submission, such information is also embedded in yuv . The representation of yuv is discussed in thenext section.
Given a set of peer assessment and peer correction outcomes,
D, each of which is represented by a tuple (u, v, zuv , yuv ), ourgoal is to estimate the true abilities of the students {su }u∈S ,where su ∈ R.

3.

DATASETIn this work, we use a peer assessment and peer correction
dataset collected from Conyac1 , which is a crowdsourcing
language translation platform. This platform employs peer
correction and peer assessment between translators for collaboratively improving their skills; thus, a translator on thisplatform can be considered as a student. When a student
submits a translation, other students evaluate its quality ona five-point scale (zero (low) to four (high)) and correct theerrors in it. Students are invited to high-reward jobs if they
have reviewed several submissions.
Students on Conyac can take a qualification test to demonstrate their skills. On this test, a student is instructed totranslate the given sentences and then the translations areevaluated by experts employed by the service provider. According to the score, a student is assigned one of five expertise levels (D, C, B, A, and A+). This level is used for thejob assignment and the default level is set to one. We consider the assigned levels as the ground truth of the student
abilities, that we aim to estimate from the outcomes of peer
assessment and peer correction.
We target the peer assessment and peer correction forJapanese to English translations on Conyac. Our dataset
contains 5,008 reviews for 413 students, and 135 students
1

https://conyac.cc/

427

provide at least one review. Figure 1(a) shows the distribution of the grades assigned to translations and Figure 1(b)illustrates the distribution of the students’ true expertiselevels.
We conduct exploratory data analysis to investigate how
the outcomes of peer correction can be used for estimating
student expertise levels. A natural expectation is that a student whose submissions are likely to be corrected would havelower ability. We calculate the correction ratio of each student, which is the number of corrected submissions divided
by the number of submissions. Figure 1(c) shows the average correction ratio of the students in each expertise level.
We observe that students with the highest level are likely tohave lower correction ratios than the others.
Additionally, we consider that students who have more errors in their submissions would be have lower ability. We calculate the number of corrected parts in each submission by
applying the Gestalt pattern matching [10]. We first obtainthe matched patterns in pre-correction and post-correction
submissions, and then count the number of unmatched patterns in the post-correction submissions. The examples of
the calculated numbers are shown in Table 1 and Figure 1(d)shows the distribution of the number of corrected parts ineach submission. We calculated the average number of corrected parts of each student and Figure 1(e) presents theaverage of the values at each level. We found that the students with higher levels are likely to have a lower number ofcorrected parts.
From these observations, we decide to use the following binary and numerical variables to represent a correction out(b)come: (1) yuv ∈ {0, 1}, which indicates whether the corre(b)sponding submission is corrected by the grader (yuv = 0) or(n)(b)
not (yuv = 1), (2) yuv ∈ {0, 1, 2, . . . , }, which indicates thenumber of parts corrected by the grader.

4.

PEER CORRECTION MODELS

We propose two peer correction models, PCb and PCn , forestimating the student true abilities. The models are illustrated in Figure 2(a).

4.1

PCb model(b)

We first present a generative model for yuv ∈ {0, 1}, which
is a binary indicator whether the submission has been corrected by the reviewer or not. We have two latent parameters into our model, that is, student true ability and reviewer
bias; each student is associated with the latent true ability,su ∈ R, which we aim to estimate, and each reviewer has adiﬀerent bias parameter, bv ∈ R, presuming that a reviewer
with a lower bias tends to review a submission negatively.
Following the observations, we assume that a submission
from a student is likely to be not corrected by a reviewer
if the student has high ability. In addition, a reviewer is
not likely to correct a submission if he/she has a higher
bias. These assumptions are represented as the following
generative model:
(
)
(b)
(b)
yuv
∼ Bern yuv
σ (su + bv + r) ,(1)
where σ(x) = 1/ (1 + exp(−x)), Bern(·) is the Bernoulli dis-

Proceedings of The 12th International Conference on Educational Data Mining (EDM 2019)

250

1500
1000
500
0

1.0
Mean correction rate

2000

Num. of students

Num. of observations

2500

50
D

4

C

Grade

0.6
0.4
0.2
0.0

A+

(b) Distribution of expertise levels

D

C

B
A
Expertise levelA+(c) Average and standard deviation ofcorrection ratios

Num. of observations

Mean num. of corrected parts(a) Distribution of submission grades

B
A
Expertise level

0.8

15

20

D

25

Num. of corrected parts(d) Distribution of the number of corrected parts

C

B
A
Expertise levelA+(e) Average and standard deviation of themean numbers of corrected parts

Figure 1: Statistics of our dataset

Table 1: Examples of the calculated number of corrected parts. The corrected parts are highlighted (modified parts arehighlighted in yellow, added parts are highlighted in pink, and removed parts are highlighted in blue). There were sevencorrected parts in the last example because there were three modified parts (“Current members,” “are” and “was working onthe”), two added parts (“female” and “,”), and two removed parts (“girls” and “the”).
Pre-correction

Post-correction

Please enter the title.
Please enter theapplication conditions .
41 people from major travel agencies of Japan and land
operators participated and had the business meetings with
suppliers about latest Thailand MICE circumstances.

Please enter the title.
Please enter theeligibility requirements .
41 people from major travel agencies of Japan and land
operators participated and had the business meetings with
suppliers about the latest Thailand MICE circumstances.
Kalafina is the female vocal band produced by Yuki Kajiura. Current members of Kalafina are WAKANA,KEIKO , and HIKARU. The group was formed in order to produce the main song when composer Yuki Kajimura was working on the music for the film “Boundaryof Emptiness”.

Kalafina is the vocal girls band produced by Yuki Ka-- -•jiura. Currently the member of Kalafina is WAKANA,KEIKO and HIKARU. The group was formed in orderto produce the main song when the composer Yuki Kajimura produced music for the film “Boundary of Emptiness”.

-(b)tribution, and r is a noise. Note that yuv = 1 indicatesthat the corresponding submission is not corrected by thereviewer. We denote this generative model by PCb model.
We can interpret su + bv + r as an apparent ability of thestudent u for the reviewer v at the time. The model indicates that a submission is likely to be not corrected when
the apparent ability is high.
In the same way as the existing peer assessment models thatwill be reviewed in the next section, we use normal distributions as priors for su , bv , and r:(Student ability)su∼ N (su |µ0 , 1/γ0 )(2)(Reviewer bias)(Noise)bvr∼ N (bv |0, 1/η0 )∼ N (r|0, 1/κ0 ),

(3)
(4)

where µ0 , γ0 , η0 , and κ0 are hyperparameters.

Num. ofcorrected
parts
1
1

7

4.2 PCn model
Our second model targets the number of corrected parts in(n)each correction, yuv ∈ {0, 1, 2, . . . , }. Following the observations from the actual dataset, we assume that a reviewer
corrects more parts of a submission when the student has alower ability. We use the Poisson distribution to represent
this assumption:
(
)
1
(n)
(n)
yuv
∼ Poisson yuv.
exp (su + bv + r)
Similar to the PCb model, su + bv + r is considered as theapparent ability of the student u to the reviewer v, and thismodel indicates that more parts of the submission is likely
to be corrected by the reviewer if the apparent ability ofthe student is lower. We call this model PCn . The priors
given in Eqs. (2), (3), and (4) are incorporated into the PCnmodel as well.

Proceedings of The 12th International Conference on Educational Data Mining (EDM 2019)428

00,01 (PG3)

1/oao, fJoµo, 'Yo

1/0

fJo (PG4)
fJo, >,. (PG5) µo,'Yosl<Q(a) PCb and PCn(b) PG1(c) PG3 , PG4 , and PG5(d) PG1 +PCb

Figure 2: Peer correction and peer assessment models, and combined models

5.

PEER ASSESSMENT MODELS

We next review the existing peer assessment models, which
are combined with our peer correction models in the next
section. In particular, we summarize the PG1 [7], PG3 [7],PG4 [6], and PG5 [6] models2 . These are the generativemodels for grades, zuv ∈ R. The peer assessment models
are illustrated in Figures 2(b) and 2(c).
The student ability and the reviewer bias parameters arealso incorporated in the peer assessment models. All themodels use the same priors given in Eqs. (2) and (3).

5.1

PG1 and PG3 models

In addition to the latent parameters incorporated in the peer
correction models (su and bv ), the peer assessment models
contain the reviewer reliability τv ∈ R+ . This parameter
indicates how likely a grade given by the reviewer contains
a noise. PG1 is defined as follows:(Reviewer reliability)(Outcome)(Outcome)

τv = θ1 sv + θ0zuv ∼ N (zuv |su + bv , 1/τv ) ,where θ0 and θ1 are hyperparameters.

5.2

PG4 and PG5 models

PG4 and PG5 are variations of PG3 and they incorporate
the relationship between the reviewer reliability and the reviewer ability into the priors of the reliability parameter.
The generative models of the reviewer reliability and outcome in PG4 are given as follows:(Reviewer reliability)(Outcome)τv ∼ Gamma(τv |sv , β0 )zuv ∼ N (zuv |su + bv , 1/τv ) ,and those in PG5 are given as follows:(Reviewer reliability)(Outcome)τv ∼ N (τv |sv , 1/β0 )zuv ∼ N (zuv |su + bv , λ/τv ) ,where β0 and λ are hyperparameters.
2
We do not include PG2 [7], which is almost similar to PG1
except it incorporates time-series factors.

429

We finally combine our peer correction models and the existing peer assessment models. By combining these two typesof models, we expect to capture an inconsistency between
the outcome of peer correction and that of peer assessment;
the inconsistency can be informative for estimating the reviewer reliabilities.
We use PG1 and PCb to explain the model combining and weterm the combined model as PG1 +PCb . We simply consider
that su and bv are shared between these two models; namely,
the generative model for PG1 +PCb is given as:(Student ability)(Reviewer reliability)(Reviewer bias)(Noise)τv ∼ Gamma(τv |α0 , β0 )zuv ∼ N (zuv |su + bv , 1/τv ) ,where α0 and β0 are hyper parameters. PG3 is an extension of PG1 , which incorporates the relationship between
the reviewer reliability and the ability of the reviewer (as astudent). PG3 is given as follows:(Reviewer reliability)

6. COMBINED MODELS FOR PEER CORRECTION AND PEER ASSESSMENT(Outcomes)su ∼ N (su |µ0 , 1/γ0 )τv ∼ Gamma(τv |α0 , β0 )bv ∼ N (bv |0, 1/η0 )r ∼ N (r|0, 1/κ0 )zuv ∼ N (zuv |su + bv , 1/τv ) , and
(
)
(b)
(b)
yuv
∼ Bern yuv
σ (su + bv + r) .

The PG1 +PCb model is illustrated in Figure 2(d). Other
combined models are defined similarly as PG1 +PCb .
When an inconsistency occurs between corrections andgrades, i.e., a reviewer provides a high grade to a submission
but makes many corrections in it, we consider that a large
noise occurs on the grade (zuv ) and thus the reliability of
the reviewer (τv ) is estimated as low. The combination ofpeer assessment models and peer correction models allows
us to leverage such inconsistencies to estimate the reviewer
reliabilities and the student abilities.

7. EXPERIMENTS
We conduct experiments using the actual peer assessment
and peer correction dataset about language translation. We
investigate the eﬀectiveness of the proposed methods to estimate the student abilities.

7.1 Baselines
We compare the proposed models (PCb , PCn , andPG{1,3,4,5} +PC{b,n} ) with the following baselines: (a) Correction ratio (PC♯b ): this is a naı̈ve version of PCb
and considers the correction ratio of each student as theability. Specifically,the() correction ratio is defined as
∑
(b)
(b)
(b)
− y(b) ∈Y (b) δ yuv = 0 /|Yu |, where Yu is the set ofuv

ucorrection outcomes for the student u, and δ (·) is the in-Proceedings of The 12th International Conference on Educational Data Mining (EDM 2019)Table 2: Average and standard deviation of AUC scores of each method on various classification boundaries. Each column
indicates the results for each classification boundary; for example, (D, C, B, A | A+) represents the results for classifying thestudents at A+ and the others. The winner for each boundary is bold-faced. The cases where a combined model outperforms
the corresponding peer assessment model (PG1 , PG3 , PG4 , or PG5 ) are underlined.
(D, C, B, A | A+)

AUC(D, C, B | A, A+) (D, C | B, A, A+)(D | C, B, A, A+)

0.713 ± 0.020
0.805 ± 0.037
0.714 ± 0.045
0.832 ± 0.050
0.794 ± 0.023
0.845 ± 0.016
0.751 ± 0.193
0.862 ± 0.069
0.842 ± 0.115
0.821 ± 0.020
0.841 ± 0.137
0.870 ± 0.019
0.914 ± 0.018
0.846 ± 0.019
0.788 ± 0.153
0.844 ± 0.038
0.888 ± 0.024

0.584 ± 0.015
0.628 ± 0.008
0.627 ± 0.013
0.710 ± 0.012
0.723 ± 0.015
0.739 ± 0.025
0.756 ± 0.033
0.774 ± 0.012
0.775 ± 0.029
0.755 ± 0.016
0.779 ± 0.026
0.764 ± 0.008
0.782 ± 0.016
0.737 ± 0.017
0.705 ± 0.044
0.753 ± 0.017
0.746 ± 0.033

0.498 ± 0.025
0.483 ± 0.034
0.611 ± 0.041
0.602 ± 0.046
0.810 ± 0.011
0.801 ± 0.015
0.786 ± 0.020
0.791 ± 0.011
0.759 ± 0.043
0.792 ± 0.011
0.789 ± 0.030
0.800 ± 0.016
0.782 ± 0.032
0.661 ± 0.047
0.710 ± 0.060
0.646 ± 0.054
0.686 ± 0.066

PC♯b
PCb
PC♯n
PCn
PG♯
PG1
PG3
PG4
PG5
PG1 +PCb
PG3 +PCb
PG4 +PCb
PG5 +PCb
PG1 +PCn
PG3 +PCn
PG4 +PCn
PG5 +PCndicator function. For assigning a higher ability for a student with less corrections, we multiply the value with −1.
(b) Mean number of corrected parts (PC♯n ): this isa naı̈ve version of PCn and considers the mean number ofthe corrected parts of each student as the ability. Specifically, the mean number of the corrected parts of the student
∑
(n)
(n)
(n)
u is defined as − y(n) ∈Y (n) yuv /|Yu |, where Yu is theuvu
set of correction outcomes for the student u. For assigninga higher ability for a student with less corrected parts, we
multiply with −1. (c) Mean grades (PG♯ ): this is a naı̈veversion of PG1 and considers the mean assigned grades ofeach∑ student as the ability. The mean grade is defined aszuv ∈Zu zuv /|Zu |, where Zu is the set of grades assigned tothe student u. (d) PG1 , PG3 , PG4 , and PG5 : existing
peer assessment models.

7.2

Experimental setup

We implemented the models using the No-U-Turn Sampler (NUTS) [3], which is a variation of the Hamiltonian
Monte Carlo. We executed four chains and they produce
5,000 samples in total. The initial 500 samples were ignored
and the average of the rest samples were used as the estimated parameters.
We randomly generated 150 sets of candidate hyperparameters for each method. A method with a set of candidate
hyperparameters produces the estimated student abilities.
Their performance was evaluated using the groundtruth of
20% of the students. We then decided the best set of hyperparameters for the method and the final result for eachmethod was evaluated by the remaining students. We performed this procedure five times and calculated the average.
Each method outputs the estimated ability of each student.
We use the expertise levels assessed by the experts as theground truth, and investigate how accurately each method

0.580 ± 0.013
0.604 ± 0.013
0.598 ± 0.010
0.690 ± 0.008
0.697 ± 0.013
0.742 ± 0.013
0.725 ± 0.020
0.743 ± 0.015
0.731 ± 0.015
0.736 ± 0.010
0.736 ± 0.011
0.730 ± 0.021
0.719 ± 0.026
0.726 ± 0.009
0.704 ± 0.022
0.724 ± 0.018
0.731 ± 0.012

classifies the students with high expertise and those withlow expertise. We specifically use the area under the ROC
curve (AUC) as an evaluation metric.

7.3 Results
Table 2 shows the AUC scores of each method on diﬀerent
classification boundaries. Our peer correction models (PCband PCn ) demonstrate better or comparative performanceto the existing peer grading models in detecting the students
at the highest level; this supports the eﬀectiveness of thepeer correction results for estimating student abilities. We
see that the “no-correction” cases only occur for high-ability
students and the correction information is helpful for distinguishing between the “perfect students” and “almost perfect
students”, both are likely to obtain the highest grades fromthe reviewers and the correction outcomes are required toclassify them.
In contrast, the performance of peer correction models becomes inferior for detecting the students at lower levels, andPG♯ achieves the best performance for detecting the students at the lowest level; the average of the obtained grades
is suﬃciently informative for detecting low-ability students.
Our methods would be beneficial for a situation where teachers aim to detect students who require advanced course materials or assignments.
The combined models of PG{1,3,4,5} +PCb outperform thecorresponding PG{1,3,4,5} in most cases; the outcomes ofpeer correction are useful for improving the student ability estimation. It is noteworthy that PG5 +PCb achieves an
AUC of 0.914 for classifying the students at A+ and theothers. This result is brought by the capability of the combined models for capturing the inconsistencies between theoutcomes of assessments and those of corrections.

Proceedings of The 12th International Conference on Educational Data Mining (EDM 2019)430

The number of corrected parts can be more informative than simply considering whether a submission is corrected; in fact, PCn is better than PCb and PC♯n performs
better than PC♯b ; however, PG{1,3,4,5} +PCb outperforms
PG{1,3,4,5} +PCn in our experiments. Because there are
more model variations in PCn than PCb , a more meticulous modeling for combining the PG models and the PCnmodel would be required.

8.

RELATED WORK

Peer assessment models are categorized into two groups:
models for cardinal peer assessment and models for ordinal
peer assessment. The former models target a situation where
the outcomes are assigned in explicit numerical scores, suchas five-point scores. In addition to the probabilistic models
reviewed in Section 5, Walsh proposed PeerRank [13], an
extension of PageRank for peer assessment. In ordinal peer
assessment, each grader is shown multiple submissions andinstructed to rank them. The Bradley–Terry model [2] hasbeen applied for ordinal peer assessment [11, 8] and Mi etal. proposed to use the cardinal peer assessment models forordinal peer assessment [6]. Although several probabilistic
models for peer assessment have been studied, peer correction has not yet been investigated.
The design of peer assessment frameworks has been attempted to improve the reliability of evaluation. Kulkarni
et al. ([4]) reported that the feedback about the grading bias
to graders was beneficial for improving the reliability. Another work proposed to design peer assessment as a multiple
choice task where a grader is instructed to choice the best
submission [5]. Peer assessment mechanisms based on gametheory have been introduced to derive accurate evaluations
from peers [14].
Our peer correction models are very related to the models
studied in the item response theory, which are for quantifying student abilities and item characteristics in educational
tests. One of the simple item response theory model is the
Rasch model [9] and our PCb (given in Eq.(1)) model has asimilar formulation to the Rasch model.
Besides peer assessment, probabilistic models for estimating grader reliability have been studied in crowdsourcing
as well. Specifically, a two-stage framework was proposed
where crowdsourcing workers in the first stage produce outputs, such as translations or logo designs, and another set ofworkers in the second stage evaluates the outputs [1]. Probabilistic models for estimating the reliability of each grader
and the quality of each output in this two-stage framework
have been proposed [1, 12]. Unlike peer assessment, the overlap between students (i.e., creators of outputs) and graders
is not assumed in crowdsourcing.

9.

CONCLUSIONSWe presented probabilistic models for peer correction, which
are used for estimating the student abilities. We proposed
two models: one considering whether a grader has correcteda submission, and the other utilizing the number of correctedparts in each submission. We also combined the peer correction models with the peer assessment models; this combination allows us to estimate the reliability of graders fromthe outcomes of peer corrections and those of peer assess-431

ment by considering the consistency between the corrections
and assessments. The experiments using the actual datasetof peer correction showed that the combination of peer correction models and peer assessment models was particularly
eﬀective in detecting high ability students.
In our models, we did not consider the importance of eachcorrected part; however, the importance levels diﬀer amongcorrected parts in which minor corrections (e.g., adding apunctuation mark) and major corrections (e.g., paraphrasing) exist. A major correction would indicate the low qualityof a submission and considering such factors is a promising
direction to improve the ability estimation accuracy.

10. ACKNOWLEDGMENTS
This work was supported by JSPS KAKENHI Grant Number 15H01704 and 18K18105.

11. REFERENCES[1] Y. Baba and H. Kashima. Statistical quality
estimation for general crowdsourcing tasks. In ACM
SIGKDD, pages 554–562, 2013.
[2] R. A. Bradley and M. Terry. The rank analysis ofincomplete block designs: I. the method of paired
comparisons. Biometrika, 39(3):324–345, 1952.
[3] M. D. Hoﬀman and A. Gelman. The No-U-turnsampler: adaptively setting path lengths in
Hamiltonian Monte Carlo. J. Mach. Learn. Res.,
15(1):1593–1623, 2014.
[4] C. Kulkarni, K. P. Wei, H. Le, D. Chia,
K. Papadopoulos, J. Cheng, D. Koller, and S. R.
Klemmer. Peer and self assessment in massive online
classes. ACM Trans. Comput.-Hum. Interact.,20(6):33:1–33:31, 2013.
[5] I. Labutov and C. Studer. JAG: a crowdsourcing
framework for joint assessment and peer grading. In
AAAI, pages 1010–1016, 2017.
[6] F. Mi and D.-Y. Yeung. Probabilistic graphical models
for boosting cardinal and ordinal peer grading in
MOOCs. In AAAI, pages 454–460, 2015.
[7] C. Piech, J. Huang, Z. Chen, C. Do, A. Ng, andD. Koller. Tuned models of peer assessment in
MOOCs. In EDM, pages 153–160, 2013.
[8] K. Raman and T. Joachims. Methods for ordinal peer
grading. In ACM SIGKDD, pages 1037–1046, 2014.
[9] G. Rasch. Probabilistic Models for Some Intelligenceand Attainment Tests. 1960.
[10] J. W. Ratcliﬀ and D. E. Metzener. Pattern matching:
the Gestalt approach. DDJ, 13(7):46, 1988.
[11] N. B. Shah, J. K. Bradley, A. Parekh, M. Wainwright,
and K. Ramchandran. A case for ordinal
peer-evaluation in MOOCs. In NIPS-DDE, 2013.
[12] T. Sunahase, Y. Baba, and H. Kashima. Pairwise
HITS: quality estimation from pairwise comparisons increator-evaluator crowdsourcing process. In AAAI,
pages 977–984, 2017.
[13] T. Walsh. The PeerRank method for peer assessment.
In ECAI, pages 909–914, 2014.
[14] W. Wu, C. Daskalakis, N. Kaashoek, C. Tzamos, and
M. Weinberg. Game theory based peer gradingmechanisms for MOOCs. In L@S, pages 281–286, 2015.

Proceedings of The 12th International Conference on Educational Data Mining (EDM 2019)