情報処理学会 SIG-BTI シンポジウム

Basic experiment related to overconfidence detection method fromPC interaction data
Picard Sébastien、 上村拓也、
富士通株式会社

烏谷彰

memory task and computed features such as follows: the duration

1. Backgroundof keystrokes, the duration between keystrokes, the duration

Overconfidence can lead to poor task performance or failure tohovering over a button before clicking it. We hypothesized that theachieve one’s goals and can also prevent the implementation of

more the user will experience difficulty recalling a combination

optimal behavior change strategies. Overall, it can negatively

from memory, the larger the feature values will be. Figure 1

impact a various range of industries from health promotion toillustrates this hypothesis.

fitness and exercise, to business and learning among others.

2. Problems with existing work
Typically, the way to approach the issue for the abovementioned cases is as follows: analyze if the user has met their goalor not and compare that with a questionnaire about their
expectation of success with regard to the goal [1], this allows tomeasure to what extent the user is overconfident. After that it ispossible to take into account the overconfidence level for
subsequent objectives and customize programs. However insteadof waiting until the user has either achieved or failed at a first goal,
it is desirable to detect overconfidence before adverse outcomessuch as program abandonment or low achievement occur to be able
to implement countermeasures.

Figure 1): distribution of average duration in milliseconds
between consecutive keystrokes. Blue line = 1st iteration (whenthe user is experiencing no difficulty), Red line = last iteration(when user cannot remember recall the combination)
In this experiment we started from simple combinations of digits,

3. Proposed methodthus we could compute a baseline for each user (we assume theWe envisioned scenes where the user is interacting with a PC tousers experience no problem remembering the first easy

carry out their work or to study. For such use cases, we used acombinations).

definition of overconfidence similar to the over-estimation

Based on the data of all the other participants in the experiment wecategory from [2]. Specifically, we defined overconfidence as 1)

built a model to assess whether the user will achieve their goal

the user will not be able to perform their task correctly or solve(until how many digits they can remember combinations) or not

problems correctly 2) the user is confident of their future successbased on feature data relative to the baseline.

and do not realize they will fail.
For this paper we focused on 1).

Our hypothesis is that we caninfer whether the user will achieve their goal or not by detecting
behavior that indicates the user is experiencing difficulty.
We focused on the aspect that when a user is operating a PC, if
they hesitate because they are experiencing difficulty, the typing
patterns will be slower than usual, also the time needed to makedecisions will be longer (for example they will pause longer before
clicking validation buttons) than usual. We conducted a principle
experiment to verify the feasibility of our method. To reflect theabove-mentioned behavioral hypothesis, we implemented aⓒ2023 Information Processing Society of Japan

4. Experiment design
We devised a principle experiment to verify the feasibility of our
method to detect whether a user operating a PC in an office settingwith achieve their goal or not.
4.1 Task and experiment scenario:- Display a random combination of digits for 5 seconds- The user memorizes the combination and interacts with acustom UI requiring keyboard input and mouse
movement before submitting their answer (this is called1

情報処理学会 SIG-BTI シンポジウム

an iteration for the rest of the paper)- Starting from a combination with less than 5 digits, the6.3 Model

combination grows in length with a 1-digit increment,To estimate whether a user is going to achieve or fail at their goal

either to the left or right to the previous combinationof remembering a certain number of digits, we used a statistical- This iteratively continues until the user either gives up ormodel which takes multiple features described in the previous

fails to remember up to 3 times (we call this series ofsection as input and outputs a binary label where 1 indicates theiteration a sequence for the rest of the paper)

user will not meet their goal.

- Before the first combination is displayed the user is asked

We used a random forest algorithm after early results (notto input the expected length of the combination they canreported in this paper) showing an improvement over other model

remember (We call this goal for the rest of the paper)

tested (included KNN, logistic regression) and assumed that the in--built feature selection exhibited by random forest is responsible for

4.2 Example of features used as input for the model

the improvement of performance compared to other models. In thisKeyboard typing speed, average duration between keystrokes,experiment we adopted a randomized search for the selection oftime spent hovering on the submit button after typing the last digit.

hyper-parameters combined with a cross validation scheme.
Further, to minimize the impact of overfitting due to personal

5. Data collection

differences we put emphasis in selecting folds such that data fromparticipants cannot be both in the training and validation set.

16 users responded to a pre-experiment survey and indicated theirAdditionally, we reported results where data from one user isascent to participate in the experiment. Explanations about how to

exclusively used in the test set, this means that for user i in test, weinstall the program and run the user interface (UI) was provided totrained the model based on the remaining 10 users, then weall participants. The task consisted of running the UI daily for 2

repeated for i = 1 to 11 and aggregated the results.

weeks (excluding weekends) and report their performance in ashared file so as to elicit overconfidence. 11 participants engaged

7. Resultswith the program and we discarded the first sequence (used forFrom the computation of features related to the task where userspractice). From the data obtained, we restricted the analysis to theremember a combination of growing length, we evaluated ouriterations where the number of digits is lower that the goal, 66

method and reported results about the feasibility to predict whether

sessions and 525 iterations.

the user will achieve their goal (meet their expectations) or not.

6. Analysis
6.1 Features

At first, we focused on each iteration separately. This task is rather
ambitious but constitute the fundamental building block of ourmethod.

We computed a battery of features meant to reflect our hypotheses

We report the following results in figure 2): samples where thestated in 3. Proposed method section. Those features included

user does not meet their goal are labeled as 1 and samples wherefeatures computed from keyboard as well has mouse hovers overthe user meets their goal are labeled as 0. The rows of the confusion

buttons to highlight the slowing of typing patterns expected to arise

matrix indicate the true label and the columns the predicted labels.

when hesitation due to inability to remember kicked in.

We report the accuracy as well as balanced accuracy which is theaverage accuracy per class, we consider it is a better indicator in

6.2 Normalizationcase of imbalanced data sets such as we have here. The models are

Our hypothesis relies on comparing behavior with a normal level,also trained to maximize the balanced accuracy. For the rest of thewe believe that changes in the above features are more meaningful

paper, we report performance statistics rounded to the fist decimal.

than absolute values. To verify this, in the result section weFigure 2a) shows the results for the basic model with nocompared performance for absolute values and data normalized asnormalization. Figure 2b) show results with normalization wherefollows: a) the first 5 iterations of the second sequence (the first

the user goal, iteration number of current number of digits are alsosequence being excluded from the data) is used to compute theadded as features.

minimum and maximum values independently for each participant.
All features measuring durations (e.g. mean or standard deviationof the inter-keystroke duration) are normalized by subtracting theminimum and dividing by (maximum-minimum) independently
for each user.

ⓒ2023 Information Processing Society of Japan2

情報処理学会 SIG-BTI シンポジウム

Figure 2a) confusion matrix without normalization

Figure 3): sequence level results
Figure 3) shows more promising results with a balanced accuracy
reaching 66%, also recall for class 1 exceeds 50% with 8 / 14 =57.1% but decreases for class 0 with 39 / 52 = 75.0%. The results
obtained here show that it is possible to obtain better performanceat sequence level by harnessing the results from the first fewFigure 2b) confusion matrix with normalization

iterations.
We consider further tests with larger data sets are needed toThe results shown in figure 2a) confirm the feeling that theconfirm the reproducibility of the results and improvements inproblem is extremely ambitious. Without normalization theterms of prediction performance are required for various

balanced accuracy falls below 45% and reaches 56% with theapplications.

normalization scheme. The improvement in balanced accuracy issubstantial, especially in the latter case recall (41/136 = 30%) for
class 1 is much higher compared with the former case (10/136 =7.4%), however it is still much below 50%. For class 0, that is,when user will meet their goal, recall is 85 / 389 = 78.1% in thelatter case and 319 / 389 = 82%. The results obtained here confirm
the hypothesis that absolute feature values are less meaningful thanrelative values and we keep the normalization scheme for the restof the analysis.
To verify our claims, we then evaluated the performance of ourmethod to predict sequence labels, focusing on making predictionsat an early stage (when the number of digits is lower than the goal).
Due to the low number of sequences we decided to harness theiteration labels instead of computing features at sequence levels.
However, with binary labels the information loss would be too
significant, so a more fine-grained approach is preferable to predict
sequence labels from iteration labels. Finally, for ease ofcomparison with the results described in figure 2, we output thesequence classification results in binary format.

8. Conclusions and future work
Our experiment results indicate that it is possible to infer from PC
interaction data the first part of our overconfidence definition,
namely 1) the user will not be able to perform their task correctlyor solve problems correctly.
We need to address the second part of our definition of
overconfidence, namely 2) the user is confident of their future
success and do not realize they will fail.

We consider this may bedone using a questionnaire, when the user is expected to be unable
to reach their goal.
Overall, our vision is that overconfidence may be detected earlyenough to modify intervention strategies, benefit users and makesignificant impact for all sorts of applications. Future work
includes various steps towards the realization of this vision, forexample the implementation of the 2-step overconfidence
detection in real-time and the evaluation of countermeasures fordifferent applications.

The implementation steps are described here:
Step 1: modify the iteration model to output 4-class labels (L0:

9. References:outperformed goal by more than 10 digits, L1: by 5 to 10 digits,[1]

L2: met goal or exceeded by less than 4 digits, L3:user didn’t meetOverconfidence: A Laboratory

their goal).

Neuroscience, Psychology and Economics 7.3 (2014): 125-50.

Step 2: obtain the 4-level labels for the first 5 iterations of eachPrint.

sequence and return the maximum value as intermediate result.
Step 3: return to a binary label by converting L0, L1, L2 to label 0,Ifcher,John,andHoma

Zarghamee.

"AffectandInvestigation."

Journalof[2] Moore, Don A., and Paul J. Healy. "The trouble withoverconfidence." Psychological review 115.2 (2008): 502

L3 to label 1.

ⓒ2023 Information Processing Society of Japan3