Multimodal Tourists’ Emotion and Satisfaction
Estimation Considering Weather Conditions andAnalysis of Feature ImportanceRyoya Hayashi1,2 , Yuki Matsuda1,2,3 , Manato Fujimoto1,2 , Hirohiko Suwa1,2 , and Keiichi Yasumoto1,2 ,1Nara Institute of Science and Technology, Ikoma, Nara 630-0192, Japan
Email: {hayashi.ryoya.ho3, yukimat, manato, h-suwa, yasumoto}@is.naist.jp.
2
RIKEN Center for Advanced Intelligence Project AIP, Tokyo, Japan.
3
JST Presto, Tokyo, Japan.
Abstract—Smart tourism, which provides rich tourism support
to tourists, is becoming increasingly common, but does not reﬂect
the experiences of individual tourists. To provide richer tourism
support, we have to recognize the psychological state of eachtourist, such as emotions and satisfaction level. We previously
reported that we could estimate the psychological states of
tourists by measuring and analyzing their unconscious behaviors(head movements, body movements, facial expressions, and vocal
expressions) during sightseeing. In this paper, we propose a new
method using principal component analysis and support vector
machine, which achieves higher performance than the previous
method in estimating emotion and satisfaction. We evaluatedthe performance of the proposed method using the data of 46
participants. The result of the emotion estimation showed thatthe proposed method achieved unweighted average recall (UAR)of 69% compared to 51.3% for the previous method for thethree-class classiﬁcation task. The satisfaction level estimationachieved mean absolute error (MAE) of 1.008 for the proposed
method compared to 1.033 for the previous method for a sevenlevel regression task. In addition, we developed a new modelfor estimating emotion and satisfaction considering weather
conditions during sightseeing, assuming that the emotions andsatisfaction of tourists are affected by the weather conditions.
For the emotion estimation, UAR increased from 53.8% to 58.0%
and for the satisfaction levels estimation, MAE decreased from
1.244 to 1.238 compared to the previous model for the data of
24 participants. We believe that considering weather conditions
is effective in estimating tourists’ emotion and satisfaction.
Index Terms—smart tourism, emotion estimation, satisfaction
estimation, mobile sensing, wearable device, weather

I. I NTRODUCTION
Real-time urban environment information (e.g., congestion
levels, event information) is becoming more easily available
with the spread of various smart devices such as smartphones
and the advancement of sensing technology. Smart tourism,
which provides rich tourism support to tourists by utilizing these technologies, is attracting attention in the ﬁeld of
tourism. To make these tourism support systems more useful,
we need to consider the psychological states of tourists, suchas their emotions and satisfaction, which mean the feedback
from users. For example, even if a tourist visits the samesightseeing spot, his/her emotions and satisfaction may bedifferent from the average ones depending on his/her personality and preferences. If we estimate the psychological state of

tourists at any time, we can utilize this information in the onsite tourism support system, which dynamically recommends
sightseeing spots and routes during sightseeing [1].
The objective of our study is to investigate a method
for estimating the psychological state of tourists based on
quantitative data. The most common approaches to collecting
tourist emotions and satisfaction levels are review website
postings and surveys [2]–[5]. As is well known, these approaches have difﬁculties in motivating people to post reviews
or participate in surveys. Also, there are issues regarding
the comprehensiveness of the information and the effects ofpsychological bias also cannot be ignored. We assume that thepsychological state of a tourist appears in various unconscious
gestures such as head movement, body movement, facial
expression, and vocal expression during sightseeing. Thus, weestimate the psychological state of tourists by multimodally
sensing them with wearable devices and smartphones without
using questionnaire surveys that were often used in existing
approach [6].
In our previous study [6], we applied a neural network as themachine learning algorithm for estimating tourists’ emotions
and satisfaction levels (hereinafter called the previous method).
However, we believe that there is room for improvement in theestimation accuracy and computation time of the model. In
this paper, we propose a new method for estimating tourists’emotions and satisfaction levels. In the proposed method,First, we conduct principal component analysis (PCA) for
dimension reduction of the features used for the model. The
previous method used 188-dimensional features as input, but
the proposed method reduces it to 60 dimensional features
by PCA. Next, we apply support vector machine (SVM) as
a machine learning algorithm. We evaluated the performanceof the proposed method using the data of 46 participants. The
results of the emotions estimation showed that the proposed
method achieved unweighted average recall (UAR) of 69%
compared to 51.3% for the previous method for the threeclass classiﬁcation task (positive, negative and neutral). The
satisfaction level estimation achieved mean absolute error(MAE) of 1.008 for the proposed method compared to 1.033
for the previous method for a seven-level regression task.
In addition, we developed a new model for estimating emo-

tions and satisfaction levels considering weather conditions
during sightseeing, assuming that the emotions and satisfactionlevels of tourists are affected by the weather conditions. For
the emotion estimation, UAR increased from 53.8% to 58.0%
and for the satisfaction levels estimation, MAE increased from
1.244 to 1.238 compared to the previous model. On the other
hand, we conﬁrmed that if we construct a model for datasetthat contains information on various seasons and different
tourist destinations, the model does not improve its accuracy.
We believe that it is necessary to consider the season or
weather conditions before building the model to improve of
the estimation of tourists’ emotion and satisfaction levels.
The contributions of this paper are summarized as thefollowing two points. First, we conﬁrmed that the estimation performance of tourists’ emotion and satisfaction canbe improved by using dimensional reduction of features anddevising machine learning algorithms. Second, we revealed
that tourists’ emotion and satisfaction could be affected byseasons and weather conditions by constructing the model
considering weather conditions.
The rest of this paper is organized as follows. Section II
explains the existing work related to our study. Section III
describes the method for tourist psychological state estimation.
Section IV provides the results of modeling methods describedin Section III. Section V discusses the implications of considering weather conditions in the model. Finally, Section VI
concludes this paper.
II. R ELATED W ORK
In general, the individual psychological state data is collected by questionnaire surveys. However, questionnaires are
not always accurate, and the reliability of the results is
problematic. Moreover, when the number of questions islarge, it becomes a burden for the people who answer it. It
is not desirable to conduct the questionnaire survey during
sightseeing, because it will disturb the sightseeing.
In this study, we are investigating a method for automatically estimating an individual’s psychological state by sensing
his/her biological information, without placing a burden on the
respondent to answer. Actually, there are many researches on
methods of estimating psychological states by sensing. We
believe that these methods can apply to the estimation of
psychological states during sightseeing.
Resch et al. [7] proposed a system (Urban Emotions) thatcollects physical motion data of individuals and estimates
their emotions using it. They use a wristband-type wearable
device to collect the data. A method for estimating emotions
by sensing eye gaze is also proposed [8], [9]. A method
for estimating emotion by collecting people’s voices is also
proposed [10], [11]. These methods are based on laboratoryenvironment, and their accuracy is not high when applied
in noisy outdoor environment. However, Tzirakis et al. [12]reported that the combination of audio and video data has thepossibility to estimate emotions even in outdoor environments.
Howarth et al. [13] investigated the relationship between theweather and the mood of college students by asking them to❶  !" #
!"#$%!"#$-!"#$.

!"#$/;"!(62

9"3"5!
71#'4"5!)6&"3:0()

-0"1(23"!$#2'%27(81'2"!$7""4"5!)6&''()$*&''()$+&''()$,❷  
 

Is psychological stateaﬀected by weather?

"#

!
=>$(1)

?"7$@$5(7>(1)

%27($@$A7(G:3"23H$?27>H$%3$:3''23

=(()$@&"'<"#()='"()
F(7!

70 %
&"'<"#()$!1!

B1(#"!C<"#"!$D:3''()'E

Fig. 1. Workﬂow to estimate tourists=(()"!$'"2'

emotion and satisfaction

report their mood every day, and found that students tended
to concentrate less when the humidity was higher. Klimstra
et al. [14] reported that there are individual differences in
the relationship between weather and mood. Denissen et al.
[15] reported that mood changes with the season. In winter,high temperatures positively affect people’s mood, while insummer, it negatively affects people’s mood. However, thesestudies use psychological state data collected by questionnaire
surveys.
III. M ETHOD
In this section, ﬁrst, we explain the method to estimate
the psychological state of tourists we have proposed previously [6]. Second, we explain collection of weather conditions
during sightseeing, which is additionally included in this paper.
Third, we explain the datasets that are used to construct
the model for estimating the psychological state of tourists.
Finally, we explain the construction method of the new model
proposed in this paper.
A. Method Overview
Fig. 1 shows the workﬂow to estimate tourists’ emotion andsatisfaction. For sensing of tourist behaviors, the tourists wear
some wearable devices and have a smartphone. We deﬁned
one sightseeing spot as one session, and labeled each session
with one emotion state and one satisfaction level.
1) Collection of Sensing Data: Our method can collectthe following tourist behavior data. First, we collect eye
movements and eye gaze during sightseeing by using Pupil
Labs Eye Tracker [16], assuming that these data express
the tourists’ interest in sightseeing spots because tourists canobtain a lot of visual information during sightseeing. Second,
we collect the head and body movements during sightseeing

by using the SenStick multi-sensor board [17], assuming thatthese also express the tourist’s psychological state because
tourists are usually looking at various landmarks and movingaround during sightseeing. Third, we collect facial expressions
and vocal expressions by asking tourists to take selﬁes using
the smartphone, assuming that photos and movies that are
taken during sightseeing express their interest in sightseeing.
For more details about the way each feature is calculated,
please refer to our previous work [6].
2) Collection of Psychological State Labels: We collect theemotional state and satisfaction levels of tourists as a groundtruth using smartphone application that tourists can answer
on their own emotional state and satisfaction levels at theend of each session. For emotional states, we use the spatialmodel that expresses emotional states on two axes of Valence(Positive/Negative) and Arousal (Active/Passive), deﬁned byRussell et al. [18]. Based on this model, emotional states are
divided into three groups and nine categories: Positive group(Excited, Happy/Pleased, Calm/Relaxed), Neutral group (Neutral), and Negative group (Sleepy/Tired, Bored/Depressed,
Disappointed, Distressed/Frustrated, Afraid/Alarmed). For satisfaction levels, we use 7-point Likert scale. Tourists canchoose their own satisfaction level from 0 (unsatisﬁed) to 6
(satisﬁed). The initial level (before the start of sightseeing) is
set to 3.
B. Collection of Weather Information
We collect weather information during sightseeing as theenvironmental information. It is assumed that the psychological state during sightseeing is affected by the weather
conditions at that time. For example if it rains, tourists may not
enjoy sightseeing fully due to the limited visibility by using
an umbrella during sightseeing. Also, if it is sunny and thetemperature is very high, tourists may feel uncomfortable due
to sweating. In this study, we use temperature, humidity, andatmospheric pressure as the weather data that can be obtained
from SenStick [17] which is used to measure head and body
movements during sightseeing.
C. Dataset
The dataset we use in this study is collected from experiments in three sightseeing areas: Ulm in Germany, Naraand Kyoto in Japan. For details of sightseeing routes andinformation, please refer to our previous papers [6], [19].
The dataset of Ulm and Nara consists of 22 people, however,the period when we conducted the experiment is widelydifferent for each tourist. In fact, the data were collected
in December 2017, January 2018, April 2018, May 2018,
June 2018, August 2018, and September 2018. Therefore, theweather information such as temperature or humidity during
sightseeing is different for each tourist. On the other hand,the dataset of Kyoto consists of 24 participants, its collection
period is three consecutive days, from March 25 to 27, 2019.
Therefore, the weather conditions are similar for each tourist
during sightseeing in the dataset of Kyoto.

For these reasons, we report the results of two cases of
estimating emotion and satisfaction to investigate the effect ofweather conditions - one is when the weather conditions are
similar: only using the dataset collected in Kyoto, the otheris when the weather conditions are dissimilar: using all thedataset collected in Ulm, Nara, and Kyoto.
D. Modeling
The features used in this study are 188 dimensions (194dimensions if we include weather data). In this study, weused PCA to reduce the dimensions to approximately onethird, which is 60 dimensions, and then applied SVM. We
conducted grid search with radial basis function (RBF) as
kernel, C parameter in the range of 100 to 106 , and gamma
parameter in the range of 10−1 to 10−6 , and selected the best
performing model among them.
IV. R ESULTS
Based on the dataset we collected in the experiments
described in the previous section, we constructed a machinelearning model for estimating tourists’ emotions and satisfaction levels. In this paper, we explored two new approaches
in comparison with the previous studies. First, we propose a
new method for model construction that is different from theprevious methods. We applied neural Network as the machinelearning algorithm in [6], however, we newly applied PCA
and SVM to improve the estimation performance of the modeland to reduce the computational complexity. Secondly, weconstructed a model considering weather conditions during
sightseeing. In previous study, we used data on tourists’ unconscious behavior to construct models. In this paper, we newly
constructed models considering weather conditions in addition
to tourists’ unconscious behavior data, assuming that tourists’emotions and satisfaction are affected by weather conditions
during sightseeing. In this section, we report the results of
exploring these approaches. The results are summarized inTable I and Table II.
A. Comparison of Methods
In this subsection, we report the results of the differencesin the methods used to construct the model for estimating
tourist emotion and satisfaction. We compared the results ofthe case of using neural network that we applied in [6], andthe results of the case of using PCA and SVM that we appliedin this study. First, we report the results of emotion estimation.
Second, we report the results of satisfaction estimation.
1) Results of Emotion Estimation: Table I shows the resultsof emotion estimation. In this study, we treat the emotion
estimation as a three-class classiﬁcation task (positive, negative, and neutral). We use unweighted average recall (UAR)
as the evaluation metric. Note that the previous method uses
10-fold cross validation for the evaluation. Also, the proposed
method uses 5-fold cross validation to consider the small ofminority data. The results of constructing the model by using
all datasets collected in Ulm, Nara, and Kyoto, showed that the
proposed method using PCA and SVM achieves UAR of 0.69

TABLE I
R ESULTS OF E MOTION E STIMATION

Nural Network
Proposed Model

Kyoto
w/o weather
w/ weather
0.423
0.456
0.538
0.580

Ulm & Nara & Kyoto
w/o weather
w/ weather
0.513*
0.445
0.690
0.483

* The result is based on the model that includes the object detection and theEmpatica features. For more details, please refer to [19].
TABLE IIR ESULTS OF S ATISFACTION E STIMATION

Nural Network
Proposed Model

Kyoto
w/o weather
w/ weather
1.226
1.192
1.109
1.092

Ulm & Nara & Kyoto
w/o weather
w/ weather
1.033*
1.189
1.008
1.092

* The result is based on the model that includes the object detection and theEmpatica features. For more details, please refer to [19].

B. Considering Weather Information
In this subsection, we report the results of the model
construction with and without weather data for the estimationof tourists’ emotion and satisfaction. First, we report theresults of emotion estimation. Second, we report the resultsof satisfaction estimation.
1) Results of Emotion Estimation: Table I shows the resultsof emotion estimation. The results of constructing the modelusing only the dataset collected in Kyoto showed UAR of
0.456 when considering the weather conditions, whereas UAR
of 0.423 when not considering the weather conditions. On theother hand, the results of constructing the model by using all
the datasets collected in Ulm, Nara, and Kyoto, showed that
UAR are lower when considering the weather conditions thanwhen not considering them. These results are similar for boththe previous method (using neural network) and the proposed
method (using PCA and SVM).
2) Results of Satisfaction Estimation: Table II shows theresults of satisfaction estimation. The results of constructing
the model using only the dataset collected in Kyoto showed

0.20

Weight

0.15
0.10
0.05
0.00

1 6 5 12 44 35 19 17 42 3 11 9 0 22 8 30 52 16 4 34

Principal Component No.

Fig. 2. Permutation Importance of Best Emotion Estimation Model

Weightwhereas the previous method using neural network achieves
UAR of 0.513. When using only the dataset collected in Kyoto,
the proposed method achieved UAR of 0.538, compared to
UAR of 0.423 for the previous method.
2) Results of Satisfaction Estimation: Table II shows theresults of satisfaction estimation. In this study, we treat thesatisfaction level estimation as a regression task (seven levels).
We use mean absolute error (MAE) as evaluation metric. Notethat the previous method uses 10-fold cross validation for theevaluation. Also, the proposed method uses Leave one person
out cross validation to validate more appropriate generalization
performance. The results of constructing the model by using
all datasets collected in Ulm, Nara, and Kyoto, showed thatthe proposed method using PCA and SVM achieves MAE
of 1.008 whereas the previous method using neural network
achieves MAE of 1.033. When using only the dataset collectedin Kyoto, the proposed method achieved MAE of 1.109,
compared to MAE of 1.226 for the previous method.

0.25
0.20
0.15
0.10
0.05
0.00

1 4 5 8 0 7 19 3 6 22 12 44 48 35 33 11 47 2 40 15

Principal Component No.

Fig. 3. Permutation Importance of Best Satisfaction Estimation Model

MAE of 1.192 when considering the weather conditions,whereas MAE of 1.226 when not considering the weather
conditions. On the other hand, the results of constructing themodel by using all the datasets collected in Ulm, Nara, andKyoto, showed that MAE are lower when considering theweather conditions than when not considering them. These
results are similar for both previous method using neuralnetwork and proposed method using PCA and SVM.
V. D ISCUSSION
In this section, we discuss the ﬁndings based on the resultsof the previous section. First, we discuss the inﬂuence ofconsidering weather conditions in constructing emotion andsatisfaction estimation models. Second, we analyze the featureimportance in the emotion and satisfaction estimation model
constructed by considering the weather conditions.

TABLE III
P RINCIPAL C OMPONENT
Rank
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

1
2

PC 1
F0semitoneFrom27.5Hz sma3nz mean
F3amplitudeLogRelF0 sma3nz mean
F2amplitudeLogRelF0 sma3nz mean
shimmerLocaldB sma3nz mean
F1amplitudeLogRelF0 sma3nz mean
HNRdBACF sma3nz mean
F0semitoneFrom27.5Hz sma3nz std
jitterLocal sma3nz mean
HNRdBACF sma3nz std
F2amplitudeLogRelF0 sma3nz std
F3amplitudeLogRelF0 sma3nz std
F1amplitudeLogRelF0 sma3nz std
shimmerLocaldB sma3nz std
logRelF0-H1-H2 sma3nz std
logRelF0-H1-A3 sma3nz meanjitterLocal sma3nz std
mfcc2 sma3 std
logRelF0-H1-A3 sma3nz std
mfcc4 sma3 mean
logRelF0-H1-H2 sma3nz mean
ave std theta ww60
ave std theta ww10
ave std theta ww20
ave ave phi ww1
mfcc4 sma3 std
ave ave phi ww20
ave ave phi ww60
ave ave phi ww240
ave ave phi ww5
ave ave phi ww120

PC 6
slope500-1500 sma3 mean
AU14 r std
all countleft count
AU12 r meanright-left countmfcc3 sma3 mean
AU14 r meanright-left span std
left span mean
mfcc4 sma3 meanleft span std
AU12 r std
up count
up-down count
AU10 r meanright-left span mean
AU05 r std
counter p 4
humidity mean
AU17 r std
AU26 r std
mfcc3 sma3 std
humidity std
slope500-1500 sma3 std
counter p 3
AU05 r mean
27 right-left val std
right-left val mean
counter p 7

PC 5
alphaRatio sma3 std
mfcc2 sma3 meanhammarbergIndex sma3 std
slope0-500 sma3 std
mfcc1 sma3 std
std std phi ww1
ave ave theta ww240
ave ave theta ww120
ave ave theta ww180
ave ave theta ww60
ave ave theta ww1
ave ave theta ww20
ave ave theta ww5
ave ave theta ww10
F3frequency sma3nz std
AU12 r std
ave std phi ww10
F1frequency sma3nz std
slope0-500 sma3 mean
mfcc2 sma3 std
Loudness sma3 mean
F2frequency sma3nz std
AU14 r std
slope500-1500 sma3 std
ave std phi ww20
mfcc3 sma3 mean
counter p 1
F1bandwidth sma3nz mean
mfcc3 sma3 std
AU10 r mean

PC 12
Loudness sma3 mean
up-down val std
Loudness sma3 std
up-down span mean
up-down val mean
AU02 r std
AU02 r mean
spectralFlux sma3 meanhammarbergIndex sma3 stdpressure meanright-left val mean
mfcc3 sma3 std
humidity mean
temp std
right-left countleft span std
1AU01 r std
AU09 r std
down countright countleft span mean
AU01 r meanright-left span std
AU09 r meanright-left span mean
AU05 r std
F3frequency sma3nz std
up span meanleft count
AU05 r mean

PC 44
AU26 r std
AU26 r mean
humidity mean
down span mean
temp std
up-down span std
std ave theta ww240
down count
counter t 4
right-left span meanpressure std
up-down span mean
walk span meanup span mean
std std phi ww1
counter t 5
std ave phi ww180
mfcc4 sma3 std
std ave theta ww20
std std phi ww240
walk value mean
counter t 2
jitterLocal sma3nz std
counter p 1
walk span std
walk value std
F1bandwidth sma3nz std
right span mean
std std theta ww120
mfcc1 sma3 std

V :Weather V :Video V :Audio V :Eye V :Walk V :Tilt
The sufﬁx sma3 means that the data is ﬁltered with a moving average ﬁlter for time window 3 ( sma3nz is non-zero conditional).
Audio features are deﬁned in eGeMAPS [20].

A. Considering Weather Information

B. Feature Importance

As explained before, the dataset of Kyoto was collected
for three consecutive days from March 25 to 27, 2019, thus
the weather conditions are almost same. Since the performance of the model is higher when considering the weather
conditions for this dataset, we believe that tourists’ emotions and satisfaction during sightseeing are affected by theweather conditions during sightseeing. On the other hand, if
we constructed the model considering the weather conditions
for all datasets of Ulm, Nara, and Kyoto, the performanceof the model did not improve. As previously mentioned, thedatasets for Ulm and Nara were collected in December 2017,
January 2018, April 2018, May 2018, June 2018, August 2018,
and September 2018, which are different for each tourist,
therefore, the weather conditions during sightseeing are also
different for each tourist. In other words, it may not beappropriate to use directly the weather information obtained
for model constructing. For example, if the temperature during
sightseeing is the same, but the season is a different, thetourists may feel the difference in the feeling temperature.
They may feel that the temperature is comfortable, or they
may feel that the temperature is uncomfortable. Therefore, oneof the effective methods to construct a model considering theweather conditions is to construct each model for each season
independently. Another effective method is to construct one
model for all seasons by setting the explanatory parametersthat consider the effects of the seasons in advance.

We used Permutation Importance to evaluate the featureimportance. Permutation Importance is a technique to evaluate
the importance of a feature based on the difference in theperformance of the model when a feature is shufﬂed so thatit no longer contributes to the model. If the performance ofthe model is lowered when shufﬂing one kind of features, we
can consider that the feature is highly important. However, if
the performance of the model is not changed when shufﬂing
one kind of features, we can consider that the feature is notimportant because it does not affect the model.
Fig. 2 shows the feature importance of the emotion estimation model with the highest UAR score in the evaluation. The
horizontal axis represents each feature. Note that the proposed
method applied dimensionality reduction to the original feature
vectors in advance, thus it represents the principal component(PC) No. The vertical axis represents the weight of featureimportance. The higher weight of the feature importancemeans that the feature is important in the construction of
the model. Fig. 3 shows the feature importance of the satisfaction estimation model with the lowest MAE score in theevaluation. The horizontal and vertical axes represent the same
information as in Fig. 2. We can see that PC No. 1, No. 6,
No. 5, No. 12, and No. 44 are the most important featuresin order in emotion estimation. Table III lists the 30 original
features ranked in order of the contribution of each of the
ﬁve PCs mentioned above. For example, PC 1 represents ahigh contribution of audio features, PC 4 represents a highcontribution of video features. For details on how to calculate
each feature, please refer to the previous paper [6]. Note thatsome weather features are included, such as humidity mean,
which represents the average humidity of one session during
sightseeing. The results suggest that such weather information
contributes to the construction of the model.
VI. C ONCLUSIONS
Aiming to estimate tourists’ psychological states from theirunconscious behaviors, in this paper, we explored two new
approaches in comparison with our previous study. First,
we constructed a machine learning model using PCA andSVM to improve the estimation performance and reduce thecomputational complexity. The result of the emotion and satisfaction estimation showed that the proposed method achieved
higher performance than the previous method. Second, weconstructed the model considering weather conditions, assuming that tourists’ emotions and satisfaction are affectedby weather conditions during sightseeing. We conﬁrmed thatthe model considering weather conditions performs higher
than the model not considering weather conditions when thecollected datasets are in similar conditions. As future work,
we need to explore ways of considering weather conditions
that are more contextual for tourists’ feelings, to improve
the performance of the estimation of tourists’ emotion andsatisfaction.
ACKNOWLEDGEMENT
This study was supported in part by JSPS KAKENHI underGrant No. JP19K24345 and JST PRESTO under Grant No.
JPMJPR2039.
R EFERENCES[1] M. Hidaka, Y. Kanaya, S. Kawanaka, Y. Matsuda, Y. Nakamura,
H. Suwa, M. Fujimoto, Y. Arakawa, and K. Yasumoto, “On-site trip
planning support system based on dynamic information on tourism
spots,” Smart Cities, vol. 3, no. 2, pp. 212–231, 2020. [Online].
Available: https://www.mdpi.com/2624-6511/3/2/13[2] “Tripadvisor,” http://www.tripadvisor.com/, (accessed 6 May 2021).
[3] “Yelp,” https://www.yelp.com/, (accessed 6 May 2021).
[4] J. Alegre and J. Garau, “Tourist satisfaction and dissatisfaction,” Annalsof Tourism Research, vol. 37, no. 1, pp. 52–73, 2010.
[5] C. F. Chen and F. S. Chen, “Experience quality, perceived value,satisfaction and behavioral intentions for heritage tourists,” Tourism
Management, vol. 31, no. 1, pp. 29–35, 2010.
[6] Y. Matsuda, D. Fedotov, Y. Takahashi, Y. Arakawa, K. Yasumoto, andW. Minker, “EmoTour: Estimating emotion and satisfaction of users
based on behavioral cues and audiovisual data,” Sensors, vol. 18, no. 11,
2018. [Online]. Available: http://www.mdpi.com/1424-8220/18/11/3978[7] B. Resch, A. Summa, G. Sagl, P. Zeile, and J.-P. Exner, “Urban
emotions – geo-semantic emotion extraction from technical sensors,
human sensors and crowdsourced data,” in Progress in Location-Based
Services 2014, 11 2014, pp. 199–212.
[8] W. L. Zheng, B. N. Dong, and B. L. Lu, “Multimodal emotion recognition using eeg and eye tracking data,” in 2014 36th Annual International
Conference of the IEEE Engineering in Medicine and Biology Society,Aug 2014, pp. 5040–5043.
[9] Z. Zhang, Y. Song, L. Cui, X. Liu, and T. Zhu, “Emotion recognition
based on customized smart bracelet with built-in accelerometer,” PeerJ,
vol. 4, p. e2258, 2016.
[10] H. Kaya, A. A. Karpov, and A. A. Salah, “Robust acoustic emotion
recognition based on cascaded normalization and extreme learning
machines,” in Advances in Neural Networks - ISNN 2016, 2016, pp.
115–123.

[11] W. Y. Quck, D. Y. Huang, W. Lin, H. Li, and M. Dong, “Mobile acoustic
emotion recognition,” in 2016 IEEE Region 10 Conference (TENCON),
Nov 2016, pp. 170–174.
[12] P. Tzirakis, G. Trigeorgis, M. A. Nicolaou, B. W. Schuller, andS. Zafeiriou, “End-to-end multimodal emotion recognition using deepneural networks,” IEEE Journal of Selected Topics in Signal Processing,vol. 11, no. 8, pp. 1301–1309, 2017.
[13] E. Howarth and M. S. Hoffman, “A multidimensional approach to therelationship between mood and weather,” British Journal of Psychology,
vol. 75, no. 1, pp. 15–23, 1984.
[14] T. A. Klimstra, T. Frijns, L. Keijsers, J. J. Denissen, Q. A. Raaijmakers,
M. A. Van Aken, H. M. Koot, P. A. Van Lier, and W. H. Meeus, “Comerain or come shine: Individual differences in how weather affects mood.”
Emotion, vol. 11, no. 6, pp. 1495–1499, 2011.
[15] J. J. Denissen, L. Butalid, L. Penke, and M. A. Van Aken, “The effects
of weather on daily mood: a multilevel approach.” Emotion, vol. 8, no. 5,
p. 662, 2008.
[16] M. Kassner, W. Patera, and A. Bulling, “Pupil: An open source platform
for pervasive eye tracking and mobile gaze-based interaction,” in Proceedings of the 2014 ACM International Joint Conference on Pervasive
and Ubiquitous Computing: Adjunct Publication, ser. UbiComp ’14
Adjunct, 2014, pp. 1151–1160.
[17] Y. Nakamura, Y. Arakawa, T. Kanehira, M. Fujiwara, and K. Yasumoto,“Senstick: Comprehensive sensing platform with an ultra tiny all-in-one
sensor board for iot research,” Journal of Sensors, vol. 2017, 2017.
[18] J. A. Russell, “A circumplex model of affect,” Journal of personality
and social psychology, vol. 39, no. 6, pp. 1161–1178, 1980.
[19] D. Fedotov, “Contextual time-continuous emotion recognition based on
multimodal data,” Ph.D. dissertation, Ulm University, 2020.
[20] F. Eyben, K. R. Scherer, B. W. Schuller, J. Sundberg, E. André,C. Busso, L. Y. Devillers, J. Epps, P. Laukka, S. S. Narayanan, and K. P.
Truong, “The Geneva Minimalistic Acoustic Parameter Set (GeMAPS)
for Voice Research and Affective Computing,” IEEE Transactions on
Affective Computing, vol. 7, no. 2, pp. 190–202, 2016.