Date of publication xxxx 00, 0000, date of current version xxxx 00, 0000.
Digital Object Identifier xx.xxxx/xxxx.xxxx.DOI

Confidence-Aware Learning Assistant
SHOYA ISHIMARU1 , TAKANORI MARUICHI2 , ANDREAS DENGEL1 and KOICHI KISE.2
1
2

German Research Center for Artificial Intelligence (DFKI), Trippstadter Str. 122, 67663, Kaiserslautern, Germany
Graduate School of Engineering, Osaka Prefecture University, 1-1, Gakuen-cho, Naka-ku, Sakai, Osaka 599-8531, Japan

arXiv:2102.07312v1 [cs.HC] 15 Feb 2021

Corresponding author: Shoya Ishimaru (e-mail: shoya.ishimaru@dfki.de).

ABSTRACT Not only correctness but also self-confidence play an important role in improving the qualityof knowledge. Undesirable situations such as confident incorrect and unconfident correct knowledge prevent
learners from revising their knowledge because it is not always easy for them to perceive the situations.
To solve this problem, we propose a system that estimates self-confidence while solving multiple-choicequestions by eye tracking and gives feedback about which question should be reviewed carefully. We report
the results of three studies measuring its effectiveness. (1) On a well-controlled dataset with 10 participants,
our approach detected confidence and unconfidence with 81 % and 79 % average precision. (2) With thehelp of 20 participants, we observed that correct answer rates of questions were increased by 14 % and
17 % by giving feedback about correct answers without confidence and incorrect answers with confidence,
respectively. (3) We conducted a large-scale data recording in a private school (72 high school students
solved 14,302 questions) to investigate effective features and the number of required training samples.
INDEX TERMS Eye tracking, in-the-wild study, learning augmentation, self-confidence estimation.

I. INTRODUCTIONUANTIFIED learning – sensing learning behaviors forgiving effective feedback based on the contexts toeach learner – has high potential in the era of digitalized
education [1]. The appearance of smart sensors equipped ona personal computer, tablet, smartphone, chair, eyeglasses,
etc. has enabled researchers to estimate various internal statessuch as engagement, boredom, tiredness, and self-confidence
while learning [2], [3]. Among these internal states, theimportance of self-confidence has especially been investigated in educational research. Self-confidence is a base of
metacognitive judgments and the most common paradigm
in metacognitive domains ranging from decision-making andreasoning [4], [5] to perceptual judgments [6], [7] and memory evaluations [8], [9]. It is a manifestation of metacognitive
assessing of one’s own knowledge or scholastic ability andaffected by proficiency, achievement, cognitive anxiety, anddifficulty of a task [10]. Several studies have reported thata positive increment in self-confidence enhances learners’
engagement and performance [11], [12].

Q

One of the most critical cases where self-confidence plays
an important role is on multiple-choice questions (MCQ).
MCQ is a type of question that asks selecting the mostappropriate choice from given options. Since information
obtained from the answer to MCQ is only its correctness, it ishard to distinguish between the cases when a learner answers
VOLUME x, 202x

FIGURE 1. An overview of Confidence-Aware Learning Assistant (CoALA).
The system recommends questions which should be checked again on thebasis of the correctness and the self-confidence estimated by eye-tracking.

with confidence or when a learner answers randomly without
confidence. We consider that there are four levels of the quality of knowledge, from high to low: correct with confidence,
incorrect without confidence, correct without confidence, andincorrect with confidence. In particular, the last two cases
are serious. In the case of correct without confidence, theanswer will be treated as correct by chance, and the learnerloses a chance to acquire correct knowledge. In the case ofincorrect with confidence, the wrong knowledge may cause
further misunderstandings.
1

Ishimaru et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALSAs a solution to such serious cases, we propose a system
which estimates self-confidence on MCQ by analyzing eye
movements. Based on the estimation output, our systemgenerates a report suggesting which question should be reviewed as shown in Figure 1. We define such an intelligent
system that adapts learning materials for each learner basedon self-confidence as Confidence-Aware Learning Assistant(CoALA). The idea of estimating self-confidence on MCQ
by eye tracking has been originally proposed by Yamada etal. [13]. Compared to their study, we aim for (1) proposing
a user-independent approach considering a real scenario and(2) investigating the effectiveness as an end-to-end working
system including feedback.
This paper presents our self-confidence estimation algorithm and the results of three studies measuring its effectiveness. In the first study, which involves 10 undergraduate
students solving a total of 1,700 questions, we investigatedthe performance (precision and recall) of the self-confidence
estimation. Then we trained a confidence estimator using thisdataset and used it for the second study. This is due to arealistic scenario of applying the technology; an estimatoris trained with the data from different learners. The second
study consists of a pre-test, a review, and a post-test with thehelp of 20 participants. By comparing the results of the pretest and the post-test, we evaluated how much our system improves the quality of knowledge in terms of the correctness of
the answers and the students’ confidence. For the third study,
we deployed our system in a private school. During this fiveweek demonstration, 72 high school students solved 145,489
questions, and 14,302 questions were labeled by themselves.
In this wild dataset, we discuss the limitations and future
directions of our system. In summary, our contribution in thispaper includes:• User-independent gaze-based self-confidence estimation on multiple-choice questions, which detects confidence with 81 % and detects unconfidence with 79 %average precision, respectively.
• End-to-end confidence-based reviewing system, which
increases correct answer rates by 14 % for unconfident
correct answers and 17 % for confident incorrect answers compared to a controlled condition.
II. RELATED WORK
A. EYE TRACKING FOR LEARNING ASSISTANT

It has been proved that eye-gaze contains linguistic proficiency [14] and a degree of self-confidence. For example, thebehavior of a student who does not understand the contents
of a document is characterized by low reading speed andfrequent rereading [15]. Thai et al. reported that comprehension for a question of a student is appeared in his/her eye
movement, for example, rereading of a question [16]. Moreover, about the relation between the behavior of eyes andself-confidence, it has been proved that low self-confidence
is characterized by a frequent rereading of questions andlong gaze on choices [17]. Okoso et al. have proposed amethod of extracting difficult parts of a document for a reader
2

to comprehend and found some effective features [18]. Lee
et al. have proposed to build a virtual tutor to support thelearning of a student [19]. This work demonstrated that eye
communications with a virtual tutor enhance the efficiencyof learning. Oliver et al. have succeeded to estimate theEnglish skill of non-native English speaker from his/her eye
behaviors in English test. The contribution of their work is
to estimate the skill successfully with a small error by a few
documents [14]. Yamada et al. have tackled the automatic
estimation of self-confidence by sensing and analyzing learners’ problem solving behaviors through eye movements [13].
However, their method works well if the training can be donefor each learner with enough amount of data, which may not
be realistic. In other learning subjects, for instance, Ishimaru
et al. have investigated reading behaviors of students on
a textbook in Physics [20]. They have proposed Areas ofInterests based and subsequence based approaches to predictexpertise.
B. OTHER SENSING MODALITIES

Some researchers have measured the attention of students inlearning by Electroencephalogram (EEG) and investigated acorrelation between attention and self-efficacy, which refers
to the level of confidence of an individual with regard to theirability at task execution) [21], [22]. Though a method using
EEG can be a solution to estimate self-confidence, the device
disturbs a user engaging in a task for the reason that it isalways attached to his head. On this point, the eye tracker ispreferable because it can be attached to a display.
There has been a growing interest in the study of therelation between cognitive performance and the autonomic
nervous system (ANS). The activity of ANS can be alsomeasured by heart rate variability (HRV) [23], Electrodermal
activity (EDA) [24], [25] and so on. We did not utilize theseapproaches in this work because we had received commentsfrom students in the private school that wearing sensors while
studying requires a high physical workload. If we can record
the precise physiological signal with remote sensing, we
consider integrating it. For instance, the nose temperature,
which can be measured by a commercial infrared thermography camera, can be a nice candidate [26]. Abdelrahman
et al. have recorded nose and forehead temperatures under
different task difficulties and found significant changes [27].
Although mobile eye trackers appeared, there is still astrong gap between controlled behaviors in the laboratory
and natural behaviors in the wild. One of the critical issues
in this research field is how we can conduct experiments innatural settings for proposing robust methods. Towards thisobjective, several researchers have conducted long-term andlarge-scale experiments (e.g., over 80 hours of recording witha mobile eye tracker [28] and 780 hours of recording with acommercial Electrooculography glasses [29]). Our work is in
this context and has evaluated real learning behaviors.

VOLUME x, 202x

Ishimaru et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS

C. ROLE OF SELF-CONFIDENCE IN LEARNING

Several studies have mentioned correlations between selfconfidence or other cognitive states and behaviors of people
in specific tasks including achievement test of learning [30],cognitive test [31], and cooking [32].
According to work by Forbes-Riley et al., adapting auser’s self-confidence into the computer tutoring system
improves performance on learning efficiency and a user’s
satisfaction [33]. Kleitman et al. reported that a high levelof self-confidence predicted high grades for primary school
children [34]. Indeed, students who have self-confidence
awareness tend to be recognized in their performances, whichdevelops their level of self-confidence again. This positive
feedback loop motivates students to learn by themselves. In
another study, Stankov et al. showed that self-confidence canbe used to identify misconceptions [35]. The misconception
occurs when a learner feels confident with the knowledge andthinks that he/she is answering correctly but actually gives anincorrect answer.
Roderer et al. have gathered participants of several ages
and have found a correlation between the self-confidence ofparticipants and their age. Junior participants have tended
to get higher self-confidence than senior participants [36].
In contrast to this research, we gathered participants ofalmost the same age so as to investigate self-confidencewith only information in answering. The researches referredabove, however, only have proposed the importance of selfconfidence. On the other hand, our work is not only to findcorrelations but also to estimate self-confidence for practicalapplications.
D. POSITION OF OUR WORK

Most of the previous work focuses only on the scientificinvestigation about the importance of self-confidence. Only alimited number of research trials have tackled the automatic
estimation of confidence. Moreover, the use of estimatedconfidence to improve the quality of knowledge has not been
well attempted in the past. We consider that this is due to thefollowing two limitations.
No general estimation – It is difficult to establish selfconfidence estimation which is independent of environments,
subjects, and learners. In other words, estimation methods
may work well under a specific environment, for specificsubjects and learners, but may not if those conditions are nolonger satisfied. In the latter case, the estimation is unstable
and less reliable. The important research question here iswhether such estimation is still effective in improving thequality of knowledge.
No end-to-end system – Effectiveness should be evaluatedas an end-to-end system including sensing, estimation, andfeedback. It is often the case that parts work well but thesystem built by connecting them does not. Unfortunately,
most of the previous work focuses on parts, and little has
considered the end-to-end scenario. If the goal is to builda system capable of improving the quality of knowledge oflearners, this standpoint is mandatory.
VOLUME x, 202x(a) Answer with confidence(b) Answer without confidence

FIGURE 2. Examples of eye gaze on multiple-choice questions

In summary, our work’s main aim is to evaluate all themethods of sensing, estimation, and feedback not independently but as an end-to-end system to prove that it can
improve the quality of knowledge.
III. PROPOSED METHOD

The processing in our system consists of the following four
steps: data recording, feature calculation, feature selection,
classification, and feedback.
A. DATA RECORDING

The eye gaze of a user is recorded by a remote eye tracker
attached at the bottom of a display. The output of the eye
tracker includes coordinates of the gaze on display and theirtimestamps. Figure 2 shows the difference of eye gaze whilesolving MCQ with and without confidence. This preliminary
observation indicates that confusion of choices appears in eye
gaze as the transition between choices. The circle in the figureis the position the user is looking at, which is visualized
as a demonstration and calibration purpose, invisible whilesolving questions.
Eye movements are composed of two events: fixations andsaccades. A fixation indicates an event when the gaze pauses
at a certain position over a certain period, usually minimum
100 ms. A rapid movement between fixations is called asaccade. We classify raw gaze into fixations and saccades byusing an algorithm proposed by Buscher et al. [37]. A blink– rapid closing of the eyelid – is not analyzed in our method
because the time required to solve one question (10 - 60 sec.)
is too short of calculating statistical features. In addition, asmooth pursuit occurs when a person tracks a moving object
with slow speed. But this metric is not considered in thismethod because all information on a display is fixed.
B. FEATURE CALCULATION

We define Areas of Interest (AOIs) as rectangles covering aquestion and each choice in order to recognize deep behaviors (e.g., a ratio of reading-times on a question and choices,a process of the decision with comparisons of choices, etc.)
Fixations and saccades are automatically associated with thecorresponding AOIs in this step. Then we extract 30 features
shown in Table 1. Features 1–14 are related to fixations, andFeatures 15–28 are about saccades. We also use the readingtime and the correctness of the answer as features.
3

Ishimaru et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALSTABLE 1. The list of the features

21-24
25-28
29
30

FeatureFixation {count, ratio} on Choices
Fixation {count, ratio} on Question{Sum, mean, max, min} of fixation durations on Choices{Sum, mean, max, min} of fixation durations on Question
Variance of {x, y} coordinate of fixations{Sum, mean} of saccade length
Saccade count {all, on Question, between Choices,
between Question and Choices}{Sum, mean, max, min} of saccade durations{Sum, mean, max, min} of saccade speeds
Reading-timeCorrectness of the answer

1.0
0.9
Precision

No.
1-2
3-4
5-8
9-12
13-14
15-16
17-20

0.8
0.7
Conf. gaze and reading time: 0.81
Conf. reading time only: 0.80
Unconf. gaze and reading time: 0.79
Unconf. reading time only: 0.73

0.6
0.5
0.0

0.2

0.4

0.6

0.8

1.0

RecallC. FEATURE SELECTION

We select effective features from the above 30 candidates
because increasing the number of features does not always
increase classification performance. We utilize the following
simple hill-climbing strategy called forward stepwise. Firstly,
we create a subset of features. The subset is empty at theinitial state. Then we calculate average precision scores of estimations using each feature and insert one with the best feature to the subset. Performances of estimations with featuresin the subset and one new feature are calculated, and keep thebest combination again. These processes are repeated as longas the new subset performs better than the old one. Two-foldcross-validation is used for this feature selection. Note that
this step has proceeded only while training a classifier withtraining samples. Then selected features are used to classify
unknown samples.

FIGURE 3. 11-point precision-recall curves and average precisions ofself-confidence estimations

A. EXPERIMENTAL DESIGN

We invited 10 participants (male: 5, female: 5) to our laboratory for solving 170 MCQ about English vocabularies
and grammars. All the participants were first-year Japanese
undergraduate students. We utilized Tobii 4C remote 90 Hz
eye tracker for this data recording. Note that an upgrade key
provided by Tobii was applied to use this device for scientificpurposes. Participants answered the most appropriate word
for a blank in a question from choices. After answering
each question, they answered a survey “Do you have a
confidence in your decision?” with Yes or No. Answers to thisquestionnaire were used as ground truth labels (referred to astrue confidence in this paper). We applied the random oversampling in imbalanced-learn to create a balanced dataset.

D. CLASSIFICATION

We estimate the self-confidence of answers by a Support
Vector Machine (SVM) with the selected features. The Radial
Basis Function (RBF) kernel with penalty parameters C = 1
and γ = 0.125 were selected experimentally and are used for
the SVM. As a preliminary study, we tested other machinelearning techniques including Random Forest, and found that
SVM performs the best overall in our classification task.
E. FEEDBACK TO A LEARNER

By combining the correctness and the estimated confidence,answers of a learner are categorized into four groups: correctwith confidence, correct without confidence, incorrect withconfidence, and incorrect without confidence. As shown in
Figure 1, the system highlights questions that should bespecially reviewed. A learner can claim if the output iswrong. Then the data are stored to personalize the upcoming
estimation.
IV. STUDY 1: EVALUATION OF SELF-CONFIDENCE
ESTIMATIONIn this first study, we evaluate the performance of the selfconfidence estimation. We involved 10 participants with thesame background in creating a well-designed dataset. Thissection explains the procedures and the results.
4

B. SELF-CONFIDENCE ESTIMATION PERFORMANCE

Figure 3 shows 11-point precision-recall curve of the confidence detection and unconfidence detection among all participants. This result indicates that our confidence estimation
performs accurate enough, relatively better in confidence
detection compared to unconfidence detection (average precisions: 81 % and 79 %). Since the labels of confidence werebalanced, the chance ratio of the estimation is 50 %. Selectedfeatures from this recording were as follows: f5: sum of
fixation durations on choices, f13: variance of x coordinateof fixations, f19: the number of saccades between choice
areas, f21: sum of saccade durations, and f29: readingtime. Since some of the selected features are correlated witheach other, one feature, i.e., only reading-time might be ableenough to classify confident and unconfident. However, theeye gaze feature improves the performance, in particular for
the unconfident detection.
C. OBSERVATION OF MISCLASSIFICATIONS

We describe a difference in the eye gaze between the casea participant answered with confidence and without confidence. We display on Figure 4 some examples of the estimation results. The circles represent the fixations, and thediameter of the circle is proportional to fixation duration.
VOLUME x, 202x

Ishimaru et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS(a) True conf. estimated as conf.

(b) True unconf. estimated as conf.
FIGURE 5. The procedure of the feedback study.

RH1 – Questions answered correctly without confidence (vague knowledge) tend to be forgotten comparedto knowledge with confidence.
• RH2 – Questions answered incorrectly with confidence(misunderstandings) tend to be mistaken again compared to wrong knowledge without confidence.
• RH3 – Estimating self-confidence from learning behaviors and giving feedback (e.g., adding questions to areview list, highlighting them while reviewing) avoids
such scenarios.
•(c) True conf. estimated as unconf. (d) True uncof. estimated as unconf.
FIGURE 4. Examples of eye gaze on each classification result

Hence the longer a participant looked at a point, the larger
the diameter of the fixation is. The lines between circles
represent the saccades.
Figure 4 (a) is an example of the eye gaze of a participant
answered with confidence. Figure 4 (d) is an example of theeye gaze which a participant answered without confidence.
We can find that the confidence in answering is characterized
by the fewer eye movements and smaller diameter of the
fixations, on the other hand, the unconfidence is characterized by the complex eye movements and the longer fixation
durations.
In Figure 4 (b), a participant answered without confidence,but the classifier estimated as he answered with confidence.
We assume that he gave up to answer correctly to thisquestion because he did not have the necessary knowledge.
In such a case, we can find that the number of fixations issmall, and the participant took a short time to answer thisquestion. These characteristics are common to Figure 4 (a),which represents a confident decision. Therefore the classifier estimated as a confident decision.
In Figure 4 (c), a participant answered with confidence,but the classifier estimated as he answered without confidence. We assume that this participant decided his answer
carefully by eliminating irrelevant choices one by one. In
such a case, we find more fixations and frequent transitionsof eyes between rectangles. This characteristic is common to
Figure 4 (d), which represents an unconfident answer.
V. STUDY 2: EVALUATION OF SELF-CONFIDENCE
BASED FEEDBACK

To evaluate the effectiveness of feedback based on selfconfidence, we utilized the classifier of the first recording andprepared the end-to-end review feedback system as the second study. This section explains the details of the experimentand answers to the following our research hypotheses.
VOLUME x, 202x

A. EXPERIMENTAL DESIGN

We employed 20 participants (undergraduate and graduate
school students, age: 18–25, male: 14, female: 6) and monitoring the transition of their performance. For questions, weprepared three levels of MCQ about English grammar: Level
1 (easy, 170 questions), Level 2 (normal, 290 questions),and Level 3 (hard, 160 questions). Each question requires
the most appropriate word for a blank in a question fromfour choices. Eye movements on the questions were recordedby Tobii 4C remote 90 Hz eye tracker with an upgrade
key. Figure 5 shows the experimental procedure. We invited
participants for three days and asked for the following tasks.
One-day breaks were inserted between each task-day. Participants who completed tasks received 5,000 JPY.
Trial (the first day) – Each participant solved 10 questions with the three levels as a trial. Two reasons were behind
this trial: getting participants used to the MCQ interface andselecting an appropriate degree of difficulty. If the questions
are too easy or too difficult, the dataset will be unbalanced,
and we cannot show any transition of their performance.
Based on the results, we selected a suitable level whose
correct answer rate is closest to 50 %.
Pre-Test (the first day) – After choosing the suitablelevel, each participant answered 120 questions of the selectedlevel and reported his/her self-confidence after each decision.
Besides, the result page (see Figure 1) with correctness
and estimated self-confidence based on the training dataset
appears after answering every 10 questions. We instructedparticipants to press “Read the answer” button for self-review
except for the questions correctly answered with confidence.
We recorded 2,075 answers in total. Based on the correctness and estimated self-confidence, we categorized theminto four groups: (1) correct with confidence, (2) correct5Ishimaru et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS

1.0(4b) incorr. w/ conf. => feedback
0.8

14.1%

0.82

**

*

1.0

0.80

0.8

0.66

(4a) incor. w/ conf.

16.5%
53.5%

0.6

0.6

0.4

0.4

0.2

0.2

0.65

0.64

(3)

(4a)

**

0.81

(1) corr. w/ conf.

5.1%
5.1%
(3) incorr. w/o conf.
5.6%(2b) corr. w/o conf. => feedback(2a) corr. w/o conf.

0.0

FIGURE 6. The distribution of questions in the feedback study.

without confidence, (3) incorrect without confidence, and(4) incorrect with confidence. The role of our system is to
identify (2) and (4) for suggesting a learner review them. In
order to evaluate the effectiveness of the system, we gave
feedback to half of (2) and (4) (see Figure 6). In following,
the without feedback samples are called as controlled groups(2a) and (4a), and the with feedback samples are defined as
experiments groups (2b) and (4b).
Review (the third day) – Participants answered review
questions generated based on the first day’s feedback. Wronganswers (3) and (4) were inserted into the list of the review.
In addition, we added (2b) to the list. During the review,(2b) and (4b) were emphasized on the question page. After
solving each question, each participant reported his/her selfconfidence. The result page with correctness and estimatedself-confidence was shown for every 10 questions. We asked
participants to press “Read the answer” button again for
self-review, except for the questions correctly answered withconfidence. The order of questions and choices was shuffledfrom the pre-test.
Post-Test (the fifth day) – Participants solved the same
120 questions as the pre-test. They reported confidences on
decisions for each question and checked the result page every
10 questions as same as the first and the third day. The orderof questions and choices was shuffled from the review.
B. IMPORTANCE OF SELF-CONFIDENCE ESTIMATION

Figure 7 shows results of the effect of our review feedback.
For this investigation, we divided all questions into twogroups: answered correctly or incorrectly at the pre-test. Then
we compared their correctness at the post-test under eachcondition.
If a participant answered correctly at the pre-test, he/sheshould be able to select the right choices again when he/she
is asked the same questions. However, some answers were
wrong at the post-test. Figure 7 (a) reports how many questions were forgotten. As a result, the ratio of (2a) correct
answers without confidence was dropped 16 % compared to(1) answers with confidence (p < 0.01 evaluated by Welch’s
t-test). In other words, answers without confidence tend to
6

*

(1)

(2a)

(2b)

0.0

(4b)

(1) corr. w/ conf.
(2a) corr. w/o conf.
(2b) corr. w/o conf. => feedback(3) incorr. w/o conf.
(4a) incor. w/ conf.
(4b) incorr. w/ conf. => feedback(a) Correct at the pre-test(b) Incorrect at the pre-test

FIGURE 7. The mean of correct answer rates among 20 participants at the
post-test. The symbol ** and * indicates p<0.01 and p<0.05, respectively.

be forgotten in the near future, and therefore they should
be included in the review list. We observed that questions
answered without confidence could not always be answered
correctly if they are asked again (RH1 is true). There is
not much difference in the correctness of post-test between
wrong answers with and without confidence (RH2 is notalways true).
C. EFFECT OF FEEDBACK

Figure 7 also shows that feedback succeeded to improve themean correct answer rate. The performance of the experiment groups was 14 % higher than the performance of thecontrolled group (p < 0.05) for the feedback about correct
and unconfident questions (see Figure 7 (a)), and was 17 %higher for incorrect and confident questions (p < 0.01; see
Figure 7 (b)). Highlighting questions that were answeredincorrectly with confidence could increase the probability ofmaintaining the correct answers in mind (RH3 is true).
D. QUALITY OF KNOWLEDGE

Let us show how the quality of knowledge changes by the
feedback with the estimated confidence. Figure 8 represents
transitions of levels: correctness and reported confidence
between the pre-test and post-test. Controlled groups (randomly selected no feedback samples) are not included inthis chart. After the review, the number of correct answers
with confidence was increased compared to the other threegroups. In addition, an interesting finding from this chart isthat participants were able to assess their state of knowledge
better after the review. A lot of correct with unconfident
answers were changed to correct with confidence answers.
And percentages of correct with unconfident answers andincorrect with confident answers were decreased. From theresult mentioned above, the feedback is effective in improving the quality of knowledge.

VOLUME x, 202x

Ishimaru et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS

1.0

Precision

0.9
0.8
0.7
Conf. gaze and reading time: 0.79
Conf. reading time only: 0.76
Unconf. gaze and reading time: 0.78
Unconf. reading time only: 0.77

0.6
0.5
0.0

0.2

0.4

0.6

0.8

1.0

RecallFIGURE 8. Transitions of currentnesses and estimated confidences before(left) and after (right) the review.

FIGURE 9. 11-point precision-recall curves and average precisions ofself-confidence estimations trained by wild data

VI. STUDY 3: DEPLOYMENT IN THE WILD

In a real learning scenario, we are not able to ask students
to calibrate an eye tracker many times. They frequently moveahead and change a seat position. Therefore eye gaze inthe wild dataset was not precise compared to data in thelaboratory. It causes problems in our feature calculation because AOIs are predefined as absolute coordinates on display.
However, an interesting finding from scan path images is thata relative positional relationship between gazes on a questionand choices is still correct even if they are shifted. In orderto solve this issue, we decided to define AOIs with a new
approach. From all fixations in one recording, we calculate
the maximum and the minimum x and y coordinates. Then
AOIs are defined on the basis of relative positions in thisspace. In our question format, an area of question is the 34 %top part of the space, and areas of questions are divided intoa cross of the remaining 66 % bottom part.

This paper has demonstrated how does CoaLA estimate selfconfidence and how much does it improve learning performances. However, it is a common affair that unexpected
problems in the laboratory condition happen in the wild condition. In this section, we report findings from a deployment
in the real classroom environment.
A. EXPERIMENTAL DESIGN

We have collaborated with a private school and deployed oursystem in the school. Students solved MCQ about vocabularies in English on the system. Then they printed out a listof words involving incorrect answers and correct answers
with low self-confidence. The questions were prepared by
the private school. The main purpose of this deployment is
not to record data but to demonstrate the system in the realenvironment. Therefore, unlike the previous two studies, we
did not prevent students’ natural behaviors. Calibration of aneye tracker was performed once before a student starts using
the system. We asked the self-confidence of the decision(ground truth labels) once every five questions. Each student
has their own username in order to track who solved whichquestion with or without confidence. The number of solved
questions depends on the students. We utilized Tobii 4C
remote 90 Hz eye tracker with an upgrade key. The durationof this demonstration was around five weeks. 83 students
used our system, and we collected 145,489 solving behaviors
in total. We evaluated our proposed self-confidence estimation on this dataset with leave-one-participant-out crossvalidation.
B. PRE-PROCESSING

Since real recordings included many noisy behaviors, thefollowing filterers were applied to obtain a reliable dataset.
(1) We analyzed labeled data in this study. (2) Data withinvalid usernames (e.g., guest) are filtered out. (3) Data with
only a few eye gaze (a ratio of valid gaze coordinates is lessthan 80 % of one recording) are also ignored. Finally, the wilddataset consists of 14,302 valid samples from 72 students.
VOLUME x, 202x

C. CONFIDENCE ESTIMATION RESULTS IN THE WILD

We utilized the recorded data for training the estimatorand Figure 9 shows the estimation results. As same as thelaboratory study, our approach could detect confidence andunconfidence relatively better than the estimator using only
reading-time. f1: fixation count on choices, f8: minimum
fixation duration on choices, f12: minimum fixation duration
on question, f29: reading-time, and f30: correctness of theanswer were selected as features.
D. EFFECTIVE FEATURES

Figure 10 shows a list of features selected on the laboratory
dataset (the first study) and wild dataset (the third study). In
both conditions, f29: reading-time has a negative correlation
with self-confidence, and was selected as a feature. Most of
the calculated features are negatively correlated with selfconfidence. This is because the longer a learner takes time
to consider, the more fixations and saccades are observed.
Interestingly, a feature that is highly correlated with selfconfidence is not necessarily selected in a classifier. Furthermore, a feature that is not correlated individually can play animportant role in combining other features.
7

Ishimaru et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS

0.90
0.3
0.2
0.1
0.0
f30 f22 f1 f5 f16 f24 f29 f15 f13 f18 f8 f7 f25 f11 f12 f20 f9 f14 f19 f3 f4 f2 f6 f28 f26 f21 f23 f27 f17 f10(a) Laboratory dataset
0.3

Average precision

0.4

0.85
0.80
0.75
0.70
confidence detection
unconfidence detection

0.65

0.2

0.60

0.1

102

0.0

(b) Wild dataset
FIGURE 10. Pearson correlations between self-confidence and each feature.
Features selected by the forward stepwise are highlighted as red color. (circle:
positive, triangle: negative correlation; sorted by the absolute value)

E. THE NUMBER OF TRAINING SAMPLES

Figure 11 shows the relation between the number of training
samples and the performance. Average precisions increased
till the number of training samples reached 200. Increments
more than 200 did not contribute to the improvement, but themore training samples we had, the less standard deviation of
the result was obtained.
VII. DISCUSSION

Studies 1 and 3 have given us interesting findings to improve the system. In the first study, we evaluated our gazebased self-confidence estimation on MCQ. The combinationof gaze features and the reading-time could estimate selfconfidence better than the estimation by reading-time only
(average precisions: 0.81 % and 0.79 %). One possible reason
for the weaker contribution of gaze features compared to theprevious report [13] is that we aim to develop a system thatstarts with a user-independent estimation, although individual learners have their own characteristics of eye movements.
Our system has a function to collect feedback to the estimation results by learners (see Figure 1), and the personalizationof the estimation remains for future work.
The third study demonstrated that our self-confidence estimation works in the wild condition such as a real classroom
environment, where the system can not be frequently calibrated. Instead of utilizing self-calibration approaches [38],[39], calculating features from relative-position based AOIs
performed enough in our use case. The number of training
samples seems not to be an important matter in this task.
Rather than collecting similar answers, recording solving
behaviors on varied types and levels of questions with shortand long reading-times should improve the estimation.
Another limitation of our studies is that the characteristics
of questions in the two datasets were different. Since we
could not control the difficulty level of questions in thewild dataset (the third study), the questions seem to be easy
for participants, and there are more correct answers thanincorrect answers.
The research mentioned in Section II attempts are mainly
focused on their contexts and parts, and thus it is hard to find
8

103

104

The number of samples

f29 f17 f15 f9 f30 f1 f2 f5 f19 f20 f11 f18 f21 f23 f7 f25 f27 f24 f6 f12 f10 f22 f8 f13 f4 f3 f14 f28 f16 f26

FIGURE 11. Average precisions on different number of samples randomly
selected from the wild dataset.

the evaluation as a whole system in an end-to-end manner.
For example, this means that little has been known how
accurate the estimation should be to achieve the goal, whichis, in our case, to improve the quality of knowledge. Our
second study indicated that questions answered with vague
knowledge tend to be forgotten compared to knowledge withconfidence (decreased by 16 %), and our confidence-based
feedback avoided the drop.
An important issue is whether it is still meaningful to give
feedback based on the noisy estimation of self-confidence. In
order to establish a system that improves the learners’ performance, the end-to-end viewpoint must be incorporated into
the evaluation. Although there is still room for improvementin our self-confidence estimation, we observed improved
learning performances.
VIII. CONCLUSION

We have proposed Confidence-Aware Learning Assistant(CoALA), which estimates self-confidence on MCQ by analyzing eye movements and generates a report suggesting
which question should be reviewed. The self-confidence estimation algorithm was evaluated in the laboratory and thewild condition. By utilizing a pre-trained estimator on thelaboratory dataset, we conducted a user-study of the review
feedback. Our end-to-end confidence-based review increased
correct answer rates by 14 % for unconfident correct answers
and 17 % for confident incorrect answers compared to a controlled condition. By visualizing transitions of correctness
and reported self-confidence in a pre-test and a post-test, weobserved that the quality of knowledge was increased. We
conclude that CoALA is helpful for learners.
In future work, we will apply our method to different kindsof subjects involving Mathematics, Science, Society, etc. We
expect a successful estimation of self-confidence in an MCQ,
which a student can answer just by looking at a display andthinking about a question. Moreover, we aim to apply our
method to questions that do not include choices. In this work,
designing AOIs for a question and each choice has been
related to obtaining some effective features. We need to find
new features to solve this problem.
VOLUME x, 202x

Ishimaru et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALSREFERENCES[1] Andreas Dengel. Digital co-creation and augmented learning. In Proceedings of the The 11th International Knowledge Management in Organizations Conference on The changing face of Knowledge Management
Impacting Society, page 3. ACM, 2016.
[2] Rafael A Calvo and Sidney D’Mello. Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Transactions on Affective Computing, 1(1):18–37, 2010.
[3] Mark Warschauer and Carla Meskill. Technology and second language
teaching. Handbook of Undergraduate Second Language Education,
15:303–318, 2000.
[4] Rakefet Ackerman and Valerie A Thompson. Meta-reasoning. Reasoning
as memory, pages 164–182, 2015.
[5] Logan Fletcher and Peter Carruthers. Metacognition and reasoning. Phil.
Trans. R. Soc. B, 367(1594):1366–1378, 2012.
[6] Stephen M Fleming, Brian Maniscalco, Yoshiaki Ko, Namema Amendi,
Tony Ro, and Hakwan Lau. Action-specific disruption of perceptual
confidence. Psychological Science, 26(1):89–98, 2015.
[7] Megan AK Peters and Hakwan Lau. Human observers have optimal
introspective access to perceptual processes even for visually masked
stimuli. Elife, 4:e09651, 2015.
[8] John Dunlosky, Michael J Serra, Greg Matvey, and Katherine A Rawson.
Second-order judgments about judgments of learning. The Journal of
General Psychology, 132(4):335–346, 2005.
[9] Bridgid Finn and Janet Metcalfe. The role of memory for past test in theunderconfidence with practice effect. Journal of Experimental Psychology:
Learning, Memory, and Cognition, 33(1):238, 2007.
[10] Richard Clément, Zoltán Dörnyei, and Kimberly A Noels. Motivation,
self-confidence, and group cohesion in the foreign language classroom.
Language learning, 44(3):417–448, 1994.
[11] Elizabeth A Linnenbrink and Paul R Pintrich. The role of self-efficacy
beliefs instudent engagement and learning intheclassroom. Reading&Writing Quarterly, 19(2):119–137, 2003.
[12] Jon-Chao Hong, Ming-Yueh Hwang, Kai-Hsin Tai, and Chi-Ruei Tsai. An
exploration of students’ science learning interest related to their cognitive anxiety, cognitive load, self-confidence and learning progress using
inquiry-based learning with an ipad. Research in Science Education, pages
1–20, 2017.
[13] Kento Yamada, Koichi Kise, and Olivier Augereau. Estimation of confidence based on eye gaze: an application to multiple-choice questions. In
Proceedings of the 2017 ACM International Joint Conference on Pervasive
and Ubiquitous Computing and Proceedings of the 2017 ACM International Symposium on Wearable Computers, pages 217–220. ACM, 2017.
[14] Olivier Augereau, Hiroki Fujiyoshi, and Koichi Kise. Towards an automated estimation of english skill via toeic score based on reading analysis.
In Pattern Recognition, 2016 23rd International Conference on, pages
1285–1290. IEEE, 2016.
[15] Keith Rayner. Eye movements in reading and information processing: 20
years of research. Psychological bulletin, 124(3):372, 1998.
[16] Meng-Jung Tsai, Huei-Tse Hou, Meng-Lung Lai, Wan-Yi Liu, and FangYing Yang. Visual attention for solving multiple-choice science problem:
An eye-tracking analysis. Computers & Education, 58(1):375–385, 2012.
[17] Kazuaki Kojima, Keiich Muramatsu, and Tatsunori Matsui. Experimental
study toward estimation of a learner mental state from processes of solving
multiple choice problems based on eye movements. In Proceedings of 20th
International Conference on Computers in Education, pages 81–85, 2012.
[18] Ayano Okoso, Takumi Toyama, Kai Kunze, Joachim Folz, Marcus Liwicki, and Koichi Kise. Towards extraction of subjective reading incomprehension: Analysis of eye gaze features. In Proceedings of the 2015
CHI Conference on Human Factors in Computing Systems: Extended
Abstracts, pages 1325–1330. ACM, 2015.
[19] Hanju Lee, Yasuhiro Kanakogi, and Kazuo Hiraki. Building a responsive
teacher: how temporal contingency of gaze interaction influences wordlearning with virtual tutors. Royal Society open science, 2(1):140361,
2015.
[20] Shoya Ishimaru, Syed Saqib Bukhari, Carina Heisel, Nicolas Großmann,
Pascal Klein, Jochen Kuhn, and Andreas Dengel. Augmented learning on
anticipating textbooks with eye tracking. In Positive Learning in the Ageof Information, pages 387–398. Springer, 2018.
[21] Jerry Chih-Yuan Sun and Katherine Pin-Chen Yeh. The effects of attention
monitoring with eeg biofeedback on university students’ attention and selfefficacy: The case of anti-phishing instructional materials. Computers &Education, 106:73–82, 2017.
VOLUME x, 202x[22] Lu-Ho Hsia, Iwen Huang, and Gwo-Jen Hwang. Effects of different online
peer-feedback approaches on students’ performance skills, motivation andself-efficacy in a dance course. Computers & Education, 96:55–71, 2016.
[23] Antonio Luque-Casado, Mikel Zabala, Esther Morales, Manuel MateoMarch, and Daniel Sanabria. Cognitive performance and heart ratevariability: the influence of fitness level. PloS one, 8(2):e56935, 2013.
[24] Iuliia Brishtel, Shoya Ishimaru, Olivier Augereau, Koichi Kise, and Andreas Dengel. Assessing cognitive workload on printed and electronic
media using eye-tracker and eda wristband. In Proceedings of the 23rd International Conference on Intelligent User Interfaces Companion, page 45.
ACM, 2018.
[25] Hugo D Critchley. Electrodermal responses: what happens in the brain.
The Neuroscientist, 8(2):132–142, 2002.
[26] Shoya Ishimaru, Soumy Jacob, Apurba Roy, Syed Saqib Bukhari, Carina
Heisel, Nicolas Großmann, Michael Thees, Jochen Kuhn, and Andreas
Dengel. Cognitive state measurement on learning materials by utilizing
eye tracker and thermal camera. In Proceedings of the 14th IAPR International Conference on Document Analysis and Recognition, volume 8,
pages 32–36. IEEE, 2017.
[27] Yomna Abdelrahman, Eduardo Velloso, Tilman Dingler, Albrecht
Schmidt, and Frank Vetere. Cognitive heat: exploring the usage of thermal
imaging to unobtrusively estimate cognitive load. Proceedings of the ACM
on Interactive, Mobile, Wearable and Ubiquitous Technologies, 1(3):33,
2017.
[28] Julian Steil and Andreas Bulling. Discovery of everyday human activities
from long-term visual behaviour using topic models. In Proceedings ofthe 2015 acm international joint conference on pervasive and ubiquitous
computing, pages 75–85. ACM, 2015.
[29] Shoya Ishimaru, Kensuke Hoshika, Kai Kunze, Koichi Kise, and Andreas
Dengel. Towards reading trackers in the wild: detecting reading activities
by eog glasses and deep neural networks. In Proceedings of the 2017 ACM
International Joint Conference on Pervasive and Ubiquitous Computing
and Proceedings of the 2017 ACM International Symposium on Wearable
Computers, pages 704–711. ACM, 2017.
[30] Radka Jersakova, Richard J Allen, Jonathan Booth, Céline Souchay, andAkira R O’Connor. Understanding metacognitive confidence: Insightsfrom judgment-of-learning justifications. Journal of Memory and Language, 97:187–207, 2017.
[31] Sabina Kleitman and Jennifer Gibson. Metacognitive beliefs, selfconfidence and primary learning environment of sixth grade students.
Learning and Individual Differences, 21(6):728–735, 2011.
[32] Jennifer A Pooler, Ruth E Morgan, Karen Wong, Margaret K Wilkin, andJonathan L Blitstein. Cooking matters for adults improves food resource
management skills and self-confidence among low-income participants.
Journal of nutrition education and behavior, 49(7):545–553, 2017.
[33] Katherine Forbes-Riley and Diane J Litman. Adapting to student uncertainty improves tutoring dialogues. In AIED, pages 33–40, 2009.
[34] Sabina Kleitman, Lazar Stankov, Carl Martin Allwood, Sarah Young,and Karina Kar Lee Mak. Metacognitive self-confidence in school-aged
children. In Self-directed learning oriented assessments in the AsiaPacific, pages 139–153. Springer, 2012.
[35] Lazar Stankov, Jihyun Lee, Wenshu Luo, and David J Hogan. Confidence:
A better predictor of academic achievement than self-efficacy, self-concept
and anxiety? Learning and Individual Differences, 22(6):747–758, 2012.
[36] Thomas Roderer and Claudia M Roebers. Can you see me thinking(about my answers)? using eye-tracking to illuminate developmental differences in monitoring and control skills and their relation to performance.
Metacognition and learning, 9(1):1–23, 2014.
[37] Georg Buscher, Andreas Dengel, and Ludger van Elst. Eye movements as
implicit relevance feedback. In Proceedings of the 2008 CHI Conferenceon Human Factors in Computing Systems: Extended Abstracts, pages
2991–2996. ACM, 2008.
[38] Michael Xuelin Huang, Tiffany CK Kwok, Grace Ngai, Stephen CF Chan,and Hong Va Leong. Building a personalized, auto-calibrating eye tracker
from user interactions. In Proceedings of the 2016 CHI Conference on
Human Factors in Computing Systems, pages 5169–5179, 2016.
[39] Thiago Santini, Wolfgang Fuhl, and Enkelejda Kasneci. Calibme: Fastand unsupervised eye tracker calibration for gaze-based pervasive humancomputer interaction. In Proceedings of the 2017 chi conference on humanfactors in computing systems, pages 2594–2605, 2017.

9