Limitations of Emotion Recognition from Facial Expressions in
e-Learning Context
Agnieszka Landowska, Grzegorz Brodny and Michal R. Wrobel
Department of Software Engineering, Gdansk University of Technology, Narutowicza Str. 11/12, 80-233 Gdansk, Poland

Keywords:

e-Learning, Emotion Recognition, Facial Expression Analysis, Intelligent Tutoring Systems.

Abstract:

The paper concerns technology of automatic emotion recognition applied in e-learning environment. During
a study of e-learning process the authors applied facial expressions observation via multiple video cameras.
Preliminary analysis of the facial expressions using automatic emotion recognition tools revealed several
unexpected results, including unavailability of recognition due to face coverage and significant
inconsistency between the results obtained from two cameras. The paper presents the experiment on elearning process and summarizes the observations that constitute limitations of emotion recognition fromfacial expressions applied in e-learning context. The paper might be of interest to researchers andpractitioners who consider automatic emotion recognition as an option in monitoring e-learning processes.

1

INTRODUCTION

There are numerous emotion recognition algorithms
that differ on input information channels, output
labels or affect representation model andclassification method. From the perspective of
e-learning applications, the most importantclassification is based on input channel, as not allchannels are available in the target environment.
Proposed in the field of Affective Computing
algorithms differ on information sources they use(Landowska, 2015b). Therefore some of them have
limited availability in e-learning context. Assumingthat a learner works in a home environment, more
specialized equipment is not available, eliminatinge.g. physiological measurements as an observation
channel. However it can be expected that a home elearning environment will be equipped with amouse, a keyboard, a microphone and a low to
medium quality camera. Voice channel is an option
for synchronous classes and videoconferences. In
asynchronous e-learning observation channels
include: monitoring standard input devices usage,
facial expression analysis using cameras andscanning of textual inputs for sentiment (for free-textonly). Authors of the paper are aware of thesynchronous and blended model of e-learning,
however this study focuses on asynchronous
learning process in home environment.

Authors of the paper designed and conducted anexperiment that aimed at monitoring e-learning
process using automatic emotion recognition. Facial
expression was among the observation channels andwe have expected to reveal information on a learneraffect from automatic analysis. However, theanalysis of the channel led to unexpected results,
including unavailability of recognition due to facecoverage and significant discrepancy between the
results obtained from two cameras. This paper aims
at reporting the limitations of emotion recognition
from facial expressions applied in e-learning
context.
The main research question of the paper is given
as follows: What availability and reliability ofemotion recognition might be obtained from facial
expression
analysisine-learninghome
environment? The criteria for analysis will include
availability and reliability of emotion recognition.
The quasi-experiment of e-learning processmonitoring was performed to spot realistic
challenges in automatic emotion recognition. As aresult, a number of concerns were identified for
affect acquisition applied in e-learning context.
The paper is organized as follows. Section 2
provides previous research we based our study on.
Section 3 includes operationalisation of variables
and experiment design, while Section 4 – study
execution details and results. Section 5 provides

383
Landowska, A., Brodny, G. and Wrobel, M.
Limitations of Emotion Recognition from Facial Expressions in e-Learning Context.
DOI: 10.5220/0006357903830389
In Proceedings of the 9th International Conference on Computer Supported Education (CSEDU 2017) - Volume 2, pages 383-389
ISBN: 978-989-758-240-0
Copyright © 2017 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved

CSEDU 2017 - 9th International Conference on Computer Supported Education

summary of results and some discussion, followedby concluding remarks (Section 6).

2

RELATED WORK

Works that are mostly related to this research arestudies on emotion recognition from facial
expression analysis.
The most frequently used emotion recognition
methods that might be considered in monitoring elearning include facial expression analysis (Szwoch
and Pieniazek, 2015), audio (voice) signal analysis
in terms of modulation and textual input analysis(Kolakowska, 2015).
Video input is most commonly used channel for
emotion recognition, as it is universal and not
disturbing method of user monitoring. Algorithms
analyze face muscle movements in order to assessuser emotional state based on Facial Action Coding
System (FACS) (Sayette et al., 2001). There aremany algorithms that differ significantly on thenumber of features and methods of data extraction,
feature selection and classification process.
Classifiers are usually build on one of the known
artificial intelligence tools and algorithms, including
decision trees, neural networks, Bayesian networks,
linear discriminate analysis, linear logistic
regression, Support Vector Machine, Hidden
Markov Models (Kołakowska et al., 2013).
Depending on the classification method, input
channels and selected features, accuracy of affectrecognition differs significantly, rarely achieving
more than 90 percent. It is important to emphasisthat highest accuracies are obtained mainly for twoclass classifiers. As literature on affective computing
tools is very broad and has already been summarized
several times, for a more extensive bibliography onaffective computing methods, one may refer to Zenget al. (Zeng et al., 2009) or to Gunes and Schuller(2013).
The emotion recognition techniques provide
results in diverse models of emotion representation.
Facial expression analysis usually provide the results
using Ekman’s six basic emotions model extended
with neutral state – usually a vector of seven values
is provided, each value indicating an intensivenessof: anger, joy, fear, surprise, disgust, sadness, neutral
state (Kołakowska et al., 2015).
Emotion recognition from facial expressions issusceptible to illumination conditions and occlusionsof the face parts (Landowska, 2015b).
Facial expression analysis has a major drawback– mimics could be to some extent controlled by384

humans and therefore the recognition results might
be intentionally or unintentionally falsified(Landowska and Miler, 2016).
Self-report on emotions, although subjective, isfrequently used as a “ground truth” and thisapproach will be applied in this study. The second
approach from the literature is multi-channel
observation and consistency check (Bailenson et al.,2008). Another approach is manual tagging by
qualified observers or physiological observations,but this approach was not used in this study.
The abovementioned results influenced decisions
on the design of this study, especially use of morethan one observation channel and improving
illumination conditions. Detailed study design isreported in Section 3.

3

RESEARCH METHODOLOGY

In order to verify applicability of emotion
recognition in e-learning context a quasi-experiment
was conducted. It was based on a typical on-line
tutorial in using a software tool extended with
monitoring user emotion recognition channels. The
concept was to engage observation channels that areavailable in typical home environment, although theexperiment was held at lab setting.

3.1

Experiment Design

The aim of the experiment was to investigate
emotional states while learning using video tutorials.
Video tutorials, such as published on Youtube, arepopular, especially among the younger generation
form of gaining knowledge on how to use specifictools, perform construction tasks, and even playgames.
The experiment was held at Emotion Monitor
stand at Gdansk University of Technology. The
stand is a configurable setting allowing to multichannel observation of a computer user (Landowska,
2015a). The experiment hardware setting consistedof three computers, specialized lighting set and two
cameras. Software component included: Inkscape as a tool to learn by a participant, web browser as a main tool leading a participant(with a dedicated website developed to set tasks
and collect questionnaire data),
 Morae Recorder and Observer to record user's
actions,
 video recording software that might record two
cameras consecutively.

Limitations of Emotion Recognition from Facial Expressions in e-Learning Context

A participant of the study had one computer with
one monitor and standard input devices at disposal,
the other equipment were used for observation
purpose. There were two cameras fronting user face,
one located above and one below the monitor, bothat monitor center. The cameras were intentionally astandard computer equipment, as usually is available
at home desk and medium quality Logitech
webcams were used. There was one factor
uncommon for home environment: specialized
lighting set that allowed to maintain stable andadequate illumination conditions. The set on is aprerequisite of Noldus FaceReader, an emotion
recognition tool, to work properly, as defined by thesoftware producer. Recognition rates decease with
uneven and inadequate lighting and this condition
was explored before, therefore we have designed anexperiment rather to observe camera location
condition. The experimental setting is visualized in
Figure 1.

description, as was used in the following
questionnaires).
(2) Preliminary survey to fill-in, which included
questions about age, gender, level of familiarity
with the graphical software including Inkscapeand assessment of current emotional state (SAM
scale).
(3) Tutorial #1.
(4) Post-task questionnaire (SAM scale anddescriptive opinions).
(5) Tutorial #2.
(6) Post-task questionnaire (SAM scale anddescriptive opinions).
(7) Tutorial #3.
(8) Post-task questionnaire (SAM scale anddescriptive opinions).
(9) Final questionnaire summarizing the completed
course.
In this manner there were presented and evaluatedthree consecutive tutorials – the intention was tocapture reactions to tasks of diverse difficulty andduration. The first tutorial presented a relatively
simple operation in Inkscape (putting a text on acircle path) and lasted for 3 minutes. The second one
was the most complicated (a text formatting that
imitates carving in a wood) – it was 6:42 minutes
long, however users often had to stop and rewind thevideo in order to perform the task properly. The lasttutorial was moderately a complicated (drawing apaper folded in a shape of a plane) and it lasted for
6:32 minutes. While watching a tutorial, the user
was meant to perform operations shown in the film.
It was not required to achieve the final result in the
Inkscape, the user could move to the next stage,
when the tutorial video has finished.

3.2

Figure 1: Experimental setting design.

During the study, data were collected fromindependent channels, which allow to makeassumptions on emotional state of user: video, key
stroke dynamics, mouse movements and self-report.
The experiment procedure started with aninformed
consent
andfollowed
scenario
implemented as consecutive web pages:(1) Experiment instruction (contained information
on experiment procedure and also the SelfAssessment Manikin (SAM) emotional scale

Operationalisation of Variables

The main research question of the paper: Whatavailability and reliability of emotion recognition
might be obtained from facial expression analysis ine-learning home environment? was decomposed tomore detailed metrics that might be retrieved basedon experiment results.
Availability factor characterizes, to what extent
video observation channel is available throughout
time. There are several conditions of unavailability:
a face might be not well visible due to partial or total
occlusion, relocation of face position due to body
movements (camera position usually is set and face
might be partially visible, if a learner moves
intensively), a face angle towards camera might be
too high for an recognition algorithm to work
properly. Following metrics were proposed: (1)385

CSEDU 2017 - 9th International Conference on Computer Supported Education

percentage of time, when a face was notrecognizable at video recording (both overall and peruser, denoted UN1); (2) percentage of time, whenface was visible, but no emotion is recognizable atvideo recording (both overall and per user, denoted
UN2); (3) percentage of time-based availability ofemotion recognition recordings from video (bothoverall and per user, denoted AV). We assumed that
if overall and per-user availability is greater than
90% of time, the conditions for analysis are good,while we expect at least 70% availability (minimum
level) per user in order to make any conclusions
based on the emotion observations.
Reliability factor indicates, how trustworthy arerecognized emotional states – to what extent wemight assume, they are the actual emotions of alearner during the process. As there is no way toknow the ground truth regarding emotional state, inthe experiment we have employed an approach ofmulti-channel observation and consistency measures
to validate the reliability. There were two cameras
and the video recordings were analyzed
independently (after synchronization). The following
metric is proposed: (1) percentage of time whenemotion recognition results from the two cameras
are consistent – the same dominant emotion isrecognized (both overall and per user, denoted
REL1); (2) direct difference between recognizedstates in valence-arousal representation model (bothoverall and per user, denoted REL2).
For consistency analysis, the un-recognized faceand emotion condition frames are excluded. We
expect overall and per-user consistency to be greaterthan 70%, while 50% is the minimal consistency peruser in order to make any conclusions based on theemotion observations.

3.3

Data Analysis Methods and Tools

Video recordings were analyzed using Noldus
FaceReader software, that recognizes facial
expressions based on FACS. The facial expressions
are then interpreted as emotional state intensity. The
tool provides detailed results as intensiveness vector,
containing values (0-1) for: joy, anger, fear, disgust,surprise, sadness and neutral state, or, alternatively it
might provide the values of valence and arousal.
FaceReader might also provide discrete results –each frame is assigned a dominant emotion as alabel. Both result types were analyzed. From theperspective of the emotion recognition from facial
expression analysis, the following events would be
disturbing: looking around and covering part of the386

face with a hand. In order to apply automatic face
analysis, face position should be frontal to thecamera.
If a face is not found on a frame, FIND_FAILED
label is returned. If a face was found, but a program
was unable to recognize an emotional state a
FIT_FAILED label is returned. The error labels areused in this study in calculating availability rates.
Data pre-processing and analysis was performed
using Knime analytical platform. Significance tests
were performed, whenever necessary – the results
are provided in the following sections.

4

EXPERIMENT EXECUTION
AND RESULTS

The experiment was held in 2016 and 17 people took
part in it. Videos were recorded with 1280x720
resolution and 30 fps frequency. Two video
recordings were broken, therefore in this paper wereport results based on 15 participants. Among
those, 13 were male and 2 female, aged 20 to 21.
From the study execution the following
observations should be declared. Participants
differed in task execution duration – the shortest
study lasted 55 and the longest 103 minutes. Some
subjects did not achieve the final result in one or
multiple tasks. The participants were not advised on
this – the decision of proceeding to another task
before previous one was accomplished was up to
them.

4.1

Availability

In order to evaluate the quantitative distribution of
the availability over time, analysis of data exported
from FaceReader emotions recognition software has
been performed. Availability metrics UN1, UN2 andAV (for definitions see Section 3.2) were calculated
for upper and lower camera independently and forboth. The results are provided in Table 1. All means
are statistically significant, except for UN1 for upper
camera, which was denoted with an asterix.
Significance was confirmed by single sample t-test –95% confidence interval was assumed.
Upper camera was characterized by average
89,7% availability, which is close to thresholddefined as good analytical conditions. There were
only two participants that had availability below
70% of the recording time.

Limitations of Emotion Recognition from Facial Expressions in e-Learning Context

Table 1: Availability metrics (all means are statistically significant except for one marked with *).
Participant
P01
P03
P04
P05
P06
P07
P08
P09
P10
P11
P12
P14
P15
P16
P17
Mean
(SD)

UN1
0,1
0,3
1,7
2,5
4,9
0,8
0,2
30,0
0,9
0,0
0,3
19,2
0,3
0,5
1,4

Upper cam .
UN2
0,4
1,8
13,8
2,7
1,4
2,0
8,6
11,3
3,8
0,0
1,8
14,6
0,8
2,1
6,4

AV
99,5
97,9
84,5
94,7
93,7
97,1
91,2
58,7
95,2
99,9
97,8
66,2
98,9
97,4
92,3

UN1
0,9
1,0
2,6
26,4
0,0
1,1
0,7
0,2
2,4
0,0
0,0
38,0
6,1
21,7
0,9

5,1
(8,6)*

5,2
(4,9)

89,7
(12,3)

7,2(11,9)

Lower camera was characterized by average
78,2% availability, which is below the defined
threshold, however might be acceptable, as exceeds
70% of time. For the camera, 4 participants had low(under minimal) availability, meaning that inpractice they should be excluded from analysis. For
two participants availability of emotion recognition
through video channel was as low as 20-30 % oftime.
In most of the cases, when one camera washighly unavailable, the data from the other one wereavailable, which is an argument for using two.
Although there was a difference between averageavailability of the lower and upper camera, thedifferences for metrics UN1 and AV are notstatistically significant (only difference for UN2
metric is statistically significant), which was
confirmed with paired t-test, assuming confidence
interval of 95%.
A more detailed analysis of the cases with thelowest availability rates was performed. In the vast
majority of cases disturbance was caused by leaning
the chin on the hand. For example participant P14
held a hand near the face for more than half of therecording time. Such position is typical for highlevel of concentration or state of deep thoughts. In
art, for example, it is used to represent characters of
thinkers and philosophers. Figure 2 shows one of theexperiment participant among two most famous
sculptures of thinkers, Rodin's Le Penseur, andMichelangelo's Il Penseroso. However, this position
may also be associated with fatigue and boredom.

Low er cam .
UN2
AV
4,1
95,0
6,1
92,9
9,5
87,9
43,0
30,6
2,3
97,7
28,6
70,3
5,0
94,3
4,7
95,1
59,9
37,6
2,1
97,9
0,0
100,0
42,3
19,8
3,9
90,0
18,1
60,2
7,5
91,5
14,6
(18,7)

4.2

78,2(27,3)

UN1
0,5
0,6
2,1
14,5
2,5
1,0
0,4
15,1
1,7
0,0
0,2
28,6
3,2
11,1
1,2

Both cam eras
UN2
2,3
4,0
11,7
22,9
1,8
15,3
6,8
8,0
31,8
1,0
0,9
28,4
2,4
10,1
7,0

AV
97,3
95,4
86,2
62,7
95,7
83,7
92,8
76,9
66,5
98,9
98,9
43,0
94,4
78,8
91,9

6,2
(8,2)

9,9
(10,1)

83,9(16,2)

Reliability

Reliability metrics results are provided in Table 2.
Metric REL01 refers to consistency based on labelsof dominant emotions and for almost all participants
is below a threshold of 50%. For 4 participants theemotion labels are different for more than 90% oftime. Such huge discrepancy was the first ourobservation while analyzing results. More detailed
analysis indicate that upper camera tends tooverestimate anger (as eyebrows are recorded fromupper perspective, they seem more lowered than inzero angle position). The lower camera seems tooverestimate surprise, as eyebrows are recorded
from lower perspective, they seem more up than inzero angle position). Confusion matrixes based onrecognized labels show that also neutral state fromone camera is paired with another emotion from thesecond camera. As label-based consistency was very
low, we have decided to analyze consistency of theemotion recognition results in valence-arousal modelof emotions. Metric REL02 was calculated for bothdimensions and the results are provided in Table 2.

Figure 2: Hand by the face posture while thinking.

387

CSEDU 2017 - 9th International Conference on Computer Supported Education

insufficient or uneven illumination. When compared
to a questionnaire (self-report), all automatic
emotion recognition methods are more independent
on human will and therefore might be perceived as amore reliable source of information on affective
state of a user, however inconsistency rate isalarming.
The study results permit to draw a conclusionthat automatic emotion recognition from facial
expressions should be applied in e-learning
processes tests with caution, perhaps being
confirmed by another observation channel.
The authors acknowledge that this study andanalysis has some limitations. The main limitationsof the study include: limited number of participants
and arbitrarily chosen metrics and thresholds. Morecase studies as well as additional experiments thatpractically would validate the findings are plannedin the future research.
There are also issues that were not addressed andevaluated within this study, i.e. consistency withother emotion recognition channels and perhaps selfreport. Those factors require a much deeper
experimental project.

The consistency for arousal is high – in 13 out of
15 participants exceeds 90%, only 2 have theconsistency above 80%. Valence inconsistency issignificantly higher – 90% threshold is exceeded
only in one case, while another two are above 80%.
For majority of participants the consistency ofvalence recognition from the two camera location islower than 50%, and even for one is reported as 0.
Difference of valence is statistically significant,which was confirmed by paired t-test with 95%
confidence interval.

5

SUMMARY OF RESULTS

The presented study revealed the following results:
 availability of camera recordings in e-learning
environment is acceptable;
 upper camera availability is higher than for thelocation below the monitor; when one camera recording is unavailable,
recording from the second one is usuallyavailable, making an advantage of using two; when using two cameras the inconsistency of
emotion recognition is relatively high and for
majority of the participants below the acceptable
threshold; lower camera tends to overestimate surprise,
while upper one – anger.
All automatic emotion recognition algorithms are
susceptible to some disturbances and facial
expression analysis is not an exception – suffersfrom face oval partial cover, location of the camera,

6

CONCLUSIONS

There is a lot of evidence that human emotions
influence interactions with computers and software
products. No doubt that educational processes
supported with technologies are under that influence

Table 2: Reliability metrics.
Participant
P01
P03
P04
P05
P06
P07
P08
P09
P10
P11
P12
P14
P15
P16
P17

388

Valence
REL01 Upper Cam . Low er Cam .
Mean (SD)
Mean (SD)43,5 0,00 (0,02) -0,02 (0,03)
36,6 -0,46 (0,21) -0,18 (0,14)
19,9 -0,33 (0,20) -0,78 (0,17)
17,5 -0,50 (0,18) -0,19 (0,15)
50,2 -0,10 (0,14) -0,13 (0,12)
20,9 -0,86 (0,07) -0,29 (0,10)
7,4 -0,70 (0,19) -0,11 (0,16)
26,9 -0,20 (0,14) -0,53 (0,24)
9,6 -0,52 (0,25) -0,20 (0,18)
5,8 -0,75 (0,10) -0,01 (0,01)
8,4 -0,02 (0,03) -0,08 (0,13)
12,4 -0,56 (0,21) -0,27 (0,19)
17,5 -0,83 (0,14) -0,17 (0,12)
44,3 -0,92 (0,09) -0,50 (0,16)
37,0 -0,65 (0,14) -0,41 (0,13)

Diff
0,02
0,28
0,45
0,31
0,03
0,57
0,59
0,34
0,31
0,75
0,05
0,29
0,66
0,42
0,24

ArousalUpper Cam . Low er Cam .
REL02
Mean (SD)
Mean (SD)0,23 (0,06)
100,00 0,25 (0,05)
0,30 (0,04)
38,46 0,34 (0,05)
0,32 (0,08)
11,22 0,33 (0,08)
0,34 (0,05)
30,43 0,27 (0,07)
0,23 (0,08)
89,47 0,30 (0,03)
0,30 (0,04)
1,69 0,28 (0,05)
0,35 (0,07)
8,33 0,36 (0,06)
0,36 (0,08)
23,81 0,35 (0,06)
0,32 (0,05)
28,85 0,30 (0,09)
0,30 (0,03)
0,00 0,29 (0,03)
0,24 (0,05)
89,47 0,28 (0,03)
0,33 (0,08)
35,82 0,41 (0,0)
0,33 (0,05)
2,74 0,28 (0,04)
0,34 (0,05)
5,95 0,29 (0,06)
0,36 (0,05)
33,33 0,35 (0,04)

Diff
0,02
0,04
0,01
0,07
0,07
0,02
0,01
0,01
0,01
0,01
0,04
0,08
0,04
0,04
0,01

REL02
100,00100,00100,00
94,20
91,23
98,31
100,0080,95
92,31100,00100,0085,07
100,0097,62
100,00Limitations of Emotion Recognition from Facial Expressions in e-Learning Context

too. Therefore investigating emotions induced byeducational resources and tools is an object ofinterest of designers, producers, teachers andlearners, as well.
This study contributes to identifying practicalconcerns that should be taken into account when
designing e-learning processes monitoring and wheninterpreting the results of automatic emotion
recognition.

ACKNOWLEDGEMENTS
This work was supported in part by PolishNorwegian Financial Mechanism Small Grant
Schemeunderthecontractno
PolNor/209260/108/2015 as well as by DS Funds of
ETI Faculty, Gdansk University of Technology.

REFERENCES
Bailenson, J.N., Pontikakis, E.D., Mauss, I.B., Gross, J.J.,
Jabon, M.E., Hutcherson, C.A., Nass, C. and John, O.,
2008. Real-time classification of evoked emotions
using facial feature tracking and physiologicalresponses. International Journal of Human-Computer
Studies, 66(5), pp.303–317.
Binali, H., Wu, C. and Potdar, V., 2010. Computational
approaches for emotion detection in text. In 4th IEEE
International Conference on Digital Ecosystems andTechnologies. IEEE, pp. 172–177.
Gunes, H. and Piccardi, M., 2005. Affect Recognition
from Face and Body: Early Fusion vs. Late Fusion. In
2005 IEEE International Conference on Systems, Man
and Cybernetics. IEEE, pp. 3437–3443.
Gunes, H. and Schuller, B., Categorical and dimensional
affect analysis in continuous input: Current trends andfuture directions. Image and Vision Computing, 31(2),pp.120–136.
Wang, J., Yin, L., Wei, X. and Sun, Y., 2006. 3D Facial
Expression Recognition Based on Primitive Surface
Feature Distribution. In 2006 IEEE Computer Society
Conference on Computer Vision and Pattern
Recognition - Volume 2 (CVPR’06). IEEE, pp. 1399–
1406.
El Kaliouby, R. and Robinson, P., 2005. Real-Time
Inference of Complex Mental States from Facial
Expressions and Head Gestures. In 2004 Conference
on Computer Vision and Pattern Recognition
Workshop. IEEE, pp. 154–154.
Kolakowska, A., 2015. Recognizing emotions on the basisof keystroke dynamics. In Human System Interactions(HSI), 2015 8th International Conference on. pp. 291–
297.
Kołakowska, A., Landowska, A., Szwoch, M., Szwoch,
W. and Wróbel, M.R., 2013. Emotion Recognition and

its Application in Software Engineering. 6th
International Conference on Human System
Interaction, pp.532–539.
Kołakowska, A., Landowska, A., Szwoch, M., Szwoch,
W. and Wróbel, M.R., 2015. Modeling emotions for
affect-aware applications. In Information Systems
Development
andApplications.
FacultyofManagement, University of Gdańsk, pp. 55--69.
Landowska, A., 2015a. Emotion monitor-concept,
construction and lessons learned. In Computer Scienceand Information Systems (FedCSIS), 2015 Federated
Conference on. pp. 75–80.
Landowska, A., 2015b. Towards Emotion Acquisition in
IT Usability Evaluation Context. In Proceedings of the
Mulitimedia, Interaction, Design and Innnovation on
ZZZ - MIDI ’15. New York, New York, USA: ACM
Press, pp. 1–9.
Landowska, A. and Miler, J., 2016. Limitations of
Emotion Recognition in Software User Experience
Evaluation Context. In Proceedings of the 2016
Federated Conference on Computer Science andInformation Systems. pp. 1631–1640.
Neviarouskaya, A., Prendinger, H. and Ishizuka, M., 2009.
Compositionality Principle in Recognition of FineGrained Emotions from Text. In Proceedings of theThird International ICWSM Conference (2009). pp.
278–281.
Picard, R.W. and Daily, S.B., 2005. Evaluating affective
interactions: Alternatives to asking what users feel. In
CHI Workshop on Evaluating Affective Interfaces:
Innovative Approaches. pp. 2119–2122.
Sayette, M.A., Cohn, J.F., Wertz, J.M., Perrott, M.A. andParrott, D.J., 2001. A Psychometric Evaluation of theFacial Action Coding System for Assessing
Spontaneous Expression. Journal of Nonverbal
Behavior, 25(3), pp.167–185.
Szwoch, M. and Pieniążek, P., 2015. Facial emotion
recognition using depth data. In 2015 8th International
Conference on Human System Interaction (HSI).
IEEE, pp. 271–277.
Vizer, L.M., Zhou, L. and Sears, A., 2009. Automatedstress detection using keystroke and linguisticfeatures: An exploratory study. International Journalof Human-Computer Studies, 67(10), pp.870–886.
Zeng, Z., Pantic, M., Roisman, G.I. and Huang, T.S.,
2009. A Survey of Affect Recognition Methods:
Audio, Visual, and Spontaneous Expressions. IEEE
Transactions on Pattern Analysis and MachineIntelligence, 31(1), pp.39–58.
Zimmermann, P., Gomez, P., Danuser, B. and Schär, S.,
2006. Extending usability: putting affect into the userexperience. Proceedings of NordiCHI’06, pp.27–32.

389