JMIR SERIOUS GAMES

Chun et al

Original Paper

Prediction of Specific Anxiety Symptoms and Virtual Reality
Sickness Using In Situ Autonomic Physiological Signals During
Virtual Reality Treatment in Patients With Social Anxiety Disorder:
Mixed Methods Study
Joo Young Chun1*, MS; Hyun-Jin Kim2*, MD; Ji-Won Hur3, PhD; Dooyoung Jung4, MD, PhD; Heon-Jeong Lee5,
MD, PhD; Seung Pil Pack6, PhD; Sungkil Lee7, PhD; Gerard Kim8, PhD; Chung-Yean Cho9, MFA; Seung-Moo Lee9,
MFA; Hyeri Lee9, BA; Seungmoon Choi10, PhD; Taesu Cheong1, PhD; Chul-Hyun Cho5, MD, PhD
1School of Industrial and Management Engineering, Korea University, Seoul, Republic of Korea

2

Department of Psychiatry, Chungnam National University Sejong Hospital, Sejong, Republic of Korea

3

School of Psychology, Korea University, Seoul, Republic of Korea

4

Department of Biomedical Engineering, Ulsan National Institute of Science and Technology, Ulsan, Republic of Korea

5

Department of Psychiatry, Korea University College of Medicine, Seoul, Republic of Korea

6

Department of Biotechnology and Bioinformatics, Korea University, Sejong, Republic of Korea

7

Department of Software, Sungkyunkwan University, Suwon, Republic of Korea

8

Department of Computer Science and Engineering, Digital Experience Laboratory, Korea University, Seoul, Republic of Korea

9

Department of Film & Multimedia, Korea National University of Arts, Seoul, Republic of Korea

10

Department of Computer Science and Engineering, Pohang University of Science and Technology, Pohang, Republic of Korea*

these authors contributed equally

Corresponding Author:
Chul-Hyun Cho, MD, PhD
Department of Psychiatry
Korea University College of Medicine
73 Goryeodae-ro, Seongbuk-gu
Seoul, 02841
Republic of Korea
Phone: 82 029205505
Fax: 82 0262805810
Email: david0203@gmail.com

Abstract
Background: Social anxiety disorder (SAD) is the fear of social situations where a person anticipates being evaluated negatively.
Changes in autonomic response patterns are related to the expression of anxiety symptoms. Virtual reality (VR) sickness caninhibit VR experiences.
Objective: This study aimed to predict the severity of specific anxiety symptoms and VR sickness in patients with SAD, using
machine learning based on in situ autonomic physiological signals (heart rate and galvanic skin response) during VR treatment
sessions.
Methods: This study included 32 participants with SAD taking part in 6 VR sessions. During each VR session, the heart rate
and galvanic skin response of all participants were measured in real time. We assessed specific anxiety symptoms using theInternalized Shame Scale (ISS) and the Post-Event Rumination Scale (PERS), and VR sickness using the Simulator Sickness
Questionnaire (SSQ) during 4 VR sessions (#1, #2, #4, and #6). Logistic regression, random forest, and naïve Bayes classification
classified and predicted the severity groups in the ISS, PERS, and SSQ subdomains based on in situ autonomic physiologicalsignal data.
Results: The severity of SAD was predicted with 3 machine learning models. According to the F1 score, the highest prediction
performance among each domain for severity was determined. The F1 score of the ISS mistake anxiety subdomain was 0.8421

https://games.jmir.org/2022/3/e38284

XSL• FO
RenderX

JMIR Serious Games 2022 | vol. 10 | iss. 3 | e38284 | p. 1
(page number not for citation purposes)

JMIR SERIOUS GAMES

Chun et alusing the logistic regression model, that of the PERS positive subdomain was 0.7619 using the naïve Bayes classifier, and thatof total VR sickness was 0.7059 using the random forest model.
Conclusions: This study could predict specific anxiety symptoms and VR sickness during VR intervention by autonomic
physiological signals alone in real time. Machine learning models can predict the severe and nonsevere psychological states ofindividuals based on in situ physiological signal data during VR interventions for real-time interactive services. These models
can support the diagnosis of specific anxiety symptoms and VR sickness with minimal participant bias.
Trial Registration: Clinical Research Information Service KCT0003854; https://cris.nih.go.kr/cris/search/detailSearch.do/13508(JMIR Serious Games 2022;10(3):e38284) doi: 10.2196/38284
KEYWORDS
social anxiety; virtual reality; autonomic physiological signals; machine learning; virtual reality sickness

Introduction

symptom levels and the appropriate application of personalized
therapeutic methods.

Anxiety involves uncertainty about the expectancy of a threat.
It is a normal emotional response that helps individuals heed
and cope with potential signs of danger [1]. Anxiety induces
anticipatory stress and various neurophysiological responses.
Previous research has shown that certain brain regions thatprocess threatening information can biologically trigger stressresponse systems, such as the autonomic nervous system andhypothalamic-pituitary-adrenal axis. Hyperactivity of theautonomic response can induce the expression of anxiety signals,
perception of sensations from inside the body, and central
periphery interactions [2].

However, VR sickness influences user experience when using
VR systems [10]. The symptoms of VR sickness include eyefatigue, disorientation, and nausea [11]. Sickness resulting from
VR is referred to by many names, including motion sickness,cybersickness, and simulator sickness. These uncomfortablefeelings disturb VR intervention; therefore, VR sickness is a
crucial problem that must be solved. The most popular method
to assess VR sickness is the Simulator Sickness Questionnaire(SSQ) [12]. The symptoms of simulator motion sickness
resemble those of motion sickness, but are less severe. Three
distinct symptom groups, namely nausea, oculomotor, anddisorientation, were clustered across 16 symptom variables.
Because simulator sickness–related symptoms differ frommotion sickness patterns, a measurement system was establishedthrough this grouping. The SSQ scores based on factor analysis
models provide a scale score as a good indicator of overallsimulator sickness severity. This quantifies simulator sickness
for activities that can lead to symptoms. However, the method
cannot be used for real-time measurements because it measures
psychological scales through a questionnaire after VRintervention. There are several technical approaches forovercoming VR sickness, and measuring and evaluating it inreal time is crucial. This is one of the factors determining thedifference in individual sensitivity and future compliance. If it
is not addressed during VR intervention, patient compliancecan be greatly reduced [13].

Social anxiety disorder (SAD) is the fear of negative evaluationand embarrassment in social situations. For instance, anxiety
can occur when thinking about being observed while eating andspeaking publicly [3]. The physiological symptoms of SAD
include heart palpitations, sweating, tremors, shaking, andblushing. Many existing studies have focused on
anxiety-induced heart reactions, and have discovered thatanxiety increases blood catecholamine concentrations [4] andcauses excess sympathetic activation and parasympathetic
withdrawal [5].
Existing treatment methods for SAD include pharmacotherapy
and psychotherapy, such as cognitive behavioral therapy (CBT)[6]. During CBT, an exposure technique allows the patient toexperience and participate in a feared situation to create a naturalprocess related to fear reduction [7]. However, individuals withsocial anxiety rarely request for support from professionals
owing to fear of the stigma of mental health treatment.
Therefore, the demand for accessibility to CBT via online
content has been increasing. Moreover, the range of virtual
reality (VR) treatments is expanding [8].
Recent VR intervention research has created virtualenvironments, such as restaurants. Individuals undergoing VRintervention showed reduced anxiety regarding daily social
interactions. They also showed a decrease in social interaction
anxiety and depression, and improved satisfaction with daily
life after the VR intervention [9].
Exposure therapy is the mainstay VR treatment for SAD.
Measuring anxiety symptoms in real time while applying theexposure technique enables the confirmation of actual SAD

https://games.jmir.org/2022/3/e38284

XSL• FO
RenderX

Several studies have proposed interventions for social anxiety
using VR treatment technology. The results showed that social
interaction anxiety and quality of life improved when treatment
was applied [9]. In particular, previous studies have shown that
exposure to scenarios of various social situations through VR
is effective for alleviating social anxiety symptoms [14].
Recently, a study applied VR treatment to anxiety disorders,such as SAD [15], proposed a VR intervention for SAD
treatment, and verified its clinical effect by reducing scores on
the Social Anxiety Disorder Scale. This method included CBTto enable participants to overcome SAD by exposing them to aself-introduction situation. The treatment had exposure therapy
for patients with SAD through VR for over 6 sessions.
Compared with previous studies that applied relatively
unidirectional VR content, this study applied a customized
approach that divided the difficulty according to the patient’s
JMIR Serious Games 2022 | vol. 10 | iss. 3 | e38284 | p. 2
(page number not for citation purposes)

JMIR SERIOUS GAMES
own symptom report. However, this study used a subjective
self-assessment scale, which was influenced by patient bias.
Furthermore, survey-oriented progress is lengthy and may havedifferent results based on subjective judgment.
We have attempted to investigate SAD in relation to VR
treatment using objective data such as brain imaging data [16].
Objective physiological signals, which reflect the degree ofanxiety symptoms or sickness symptoms, can be assessed inreal time during VR treatment sessions by several sensors. This
would provide a basis for stand-alone interactive content to beprovided during VR treatment for anxiety. Moreover, reducing
VR motion sickness could help enhance user experience andcompliance using physiological signals.
In this study, we developed a machine-learning model to predict
the severity of specific anxiety symptoms and VR sickness using
physiological signal data that were measured in situ during VR
treatment sessions in patients with SAD.

Methods
ParticipantsParticipants were recruited through advertisements on theuniversities’ online sites. The inclusion criteria for the SAD
group were set as follows: (1) men and women aged between
19 and 31 years using the Korean language; (2) condition
meeting the Diagnostic and Statistical Manual of Mental
Disorders, fourth edition (DSM-IV) criteria for SAD, asevaluated by the Mini-International Neuropsychiatric Interview[17]; (3) no psychiatric comorbidity, except for depression andpanic disorder, and no experience in psychotropic drug
treatment; (4) not currently receiving psychotherapy; (5) not
currently diagnosed with medical or neurological disorders; (6)no history of psychotic symptoms that can be triggered by VRinterventions; and (7) no susceptibility to visual stimuli. We set
the exclusion criteria as follows: (1) any history of organic brain
damage or intellectual disability; (2) a history of psychotic
symptoms that can be triggered by VR interventions; (3)vulnerability to visual stimuli; and (4) ineligibility forparticipation in a magnetic resonance imaging assessment(during the study, we performed a functional magnetic resonanceimaging evaluation for another project in the same subjects)[18].
Forty individuals with SAD participated in the study. Amongthem, 8 individuals quit for personal reasons (eg, time
limitations). Ultimately, 32 participants completed the study.
All participants completed all VR sessions. The number ofpeople who participated in the entire session was 32, but 6 ofthem, for whom the sensor data for a specific session were notcollected because of problems such as sensor errors, were
excluded from data analysis. Consequently, data from 26
individuals were used for specific anxiety symptom and VR
sickness predictions after the data qualification process.

Ethics Approval

Chun et alaccordance with the principles of the Declaration of Helsinki.
All participants were informed of the study procedure andprovided written informed consent before the experiment. This
study was registered on the Clinical Research Information
Service (KCT0003854).

Study Design
VR Intervention for SAD
The study design included VR sessions, psychological scaleevaluations, and in situ autonomic physiological signal datameasurements (Figure 1). A total of 6 VR sessions and 4
psychological scale assessments were performed. A recent study
found that the amount of anxiety that increased during actual
conversation and the anxiety that increased during a
conversation in a VR environment was similar [19]. In a study
comparing VR exposure therapy and real-world exposure
therapy for SAD [20], 97 participants with a major diagnosis
of SAD were randomly assigned to VR exposure therapy,
real-world exposure therapy, or a waiting list. A standardized
self-report scale showed a statistically significant improvementin those who completed active treatment than in those on thewaiting list. Thus, VR exposure therapy was as effective as
real-world exposure therapy for treating SAD. In this study,
participants completed VR treatment sessions that included
interactive content on social anxiety situations, and the VRintervention was performed face-to-face and provided
individually. The intervention proceeded in a VR environment,
and each participant entered a meeting room with several
nonplaying characters [15]. The content had introduction, core,and finishing stages. In the introduction phase, participants
selected their avatars and were informed how to manipulate the
VR device to progress through the stages [21]. Participants
performed a warm-up session with mediation to support theiradaptation. The core stage had an intervention in which college
student’s participated in social situations where they introduced
themselves to each other. Participants proceeded with thetreatment session by selecting the difficulty of the core level as
easy, medium, or hard. In the first session, all participants started
the VR intervention at the easy level. From the second session,
participants freely chose the level of difficulty they wanted
based on the difficulty they experienced during the treatment
process. This is related to the degree of difficulty in interacting
in social situations. The reaction of the nonplaying characters
was based on the difficulty level selected in the previous stage.
In the easy-level session, the nonplaying characters concentrated
well on the patients’ self-introduction. Nonplaying characters
appeared distracted in the hard level, such as yawning or
engaging in chatting. The finishing stage provided cognitiveand behavioral safety guidelines for SAD with text and audio
information during the VR intervention. The VR headset used
was VIVE (HTC Corporation). The in situ autonomic
physiological signals (heart rate [HR] and galvanic skin response[GSR]) of the participants were measured during the VRexperience.

The study was approved by the Institutional Review Board of
Korea University Hospital (2018AN0377). It was conducted in

https://games.jmir.org/2022/3/e38284

XSL• FO
RenderX

JMIR Serious Games 2022 | vol. 10 | iss. 3 | e38284 | p. 3
(page number not for citation purposes)

JMIR SERIOUS GAMES

Chun et al

Figure 1. Virtual reality (VR) treatment process with psychological scale measurement and physiological signal data extraction. Psychological scaleassessments included Beck Anxiety Inventory, State-Trait Anxiety Inventory, Social Phobia Scale, Social Interaction Anxiety Scale, Brief-Fear of
Negative Evaluation Scale, Internalized Shame Scale, Post-Event Rumination Scale, and Liebowitz Social Anxiety Scale. SSQ: Simulator Sickness
Questionnaire.

Data Collection and Separation
This study analyzed specific anxiety symptoms and in situ
autonomic physiological signal data. For this purpose,
physiological signal data were measured in real time during 4
VR sessions (#1, #2, #4, and #6), where psychological scale
tests were performed after VR treatment. Data were extracted
as independent variables. The SSQ was evaluated during all
treatment sessions; therefore, in situ physiological signal data
for all sessions were extracted. We created a data frame with
the HR and GSR data. Values below zero in the HR and GSR
measurements were considered outliers. Outliers were removed,
and a moving average was used to reduce the noise of the data.

https://games.jmir.org/2022/3/e38284

XSL• FO
RenderX

Measurement
Psychological Scales
We measured psychological scales using subjective surveys
after the VR sessions #1, #2, #4, and #6. The Beck Anxiety
Inventory [22], State-Trait Anxiety Inventory [23], Social Phobia
Scale [24], Social Interaction Anxiety Scale [25], Brief-Fear of
Negative Evaluation Scale [26], Internalized Shame Scale (ISS)[27], and Post-Event Rumination Scale (PERS) [28] were used.
After building a simple linear regression model for the scales,
their subdomains, and the SSQ, the best performing group ofscales based on the mean squared error of the models’ prediction
results was analyzed. Subsequently, we generated a machinelearning model to predict the survey results. The linear
regression coefficient was included as an independent variablefor prediction; therefore, the specific anxiety symptoms were
JMIR Serious Games 2022 | vol. 10 | iss. 3 | e38284 | p. 4
(page number not for citation purposes)

JMIR SERIOUS GAMES
selected in the order of the smallest root mean square error
deviation when predicting with simple linear regression. The
subdomains of the corresponding specific anxiety symptoms
were also included and used for prediction. The ISS and PERS
showed a good reflection of the physical response to SAD. In
this study, the machine learning model used the ISS and PERS
as independent variables. Multimedia Appendix 1 provides thecomplete regression results for our machine learning model.
The ISS assesses the patients’ shame and self-esteem [27]. It
has 30 items that measure shame and self-esteem (24 and 6
items, respectively). ISS data were measured using the ISS
written in the Korean language [29]. Factor analysis involved
the following 4 factors: inadequacy, emptiness, self-punishment,
and fear of mistakes, with 10, 5, 5, and 4 items, respectively.
The PERS measures postevent ruminations during social
situations [28,30]. It comprises 2 scales, negative and positive
rumination, with 5 and 9 items, respectively. Each answer wasmeasured on a numerical scale from 0 (low score) to 4 (highscore), with a higher score indicating that the individual
frequently experiences rumination.

VR Sickness Scale
The SSQ was developed for VR sickness, and is the gold
standard for assessing physical sickness after exposure to asimulator or VR environment [12]. The SSQ can measure 16
symptoms of VR sickness. The subdomains are grouped into
nausea (general discomfort, increased salivation, sweating,nausea, difficulty concentrating, stomach awareness, andburping), oculomotor (general discomfort, fatigue, headache,eyestrain, difficulty focusing, difficulty concentrating, andblurred vision), and disorientation (difficulty focusing, nausea,
fullness of the head, blurred vision, dizziness with eyes open,
dizziness with eyes closed, and vertigo). The total VR sickness
score was the sum of nausea, oculomotor, and disorientationgroup scores, with a higher score indicating a greater level ofsimulator sickness.

Chun et al

Physiological Signal Data
We measured all participants’ HR and GSR during the VR
sessions in real time. These values are closely related to in situ
physiological signaling responses related to anxiety. We
recorded the data using a Shimmer3 GSR+ Unit (Shimmer)with 3 channels. Two channels tracked the electrodermal skinconductance level signals via hand electrodes attached to thefirst and middle fingers. We monitored the fingers of thenondominant hand at a sampling frequency of 52 Hz. The thirdchannel recorded cardiac volume data from an infrared sensor
attached to the left earlobe. Data were converted into HR datausing a software system.
Preprocessing removed any sections where the HR and GSR
were not measured or had negative values. The noise was
reduced using the moving average method at 2-s intervals. We
extracted all the data from the time series data set of theparticipants. We also calculated the mean, standard deviation,
minimum, maximum, linear regression coefficient, maximum
difference, peak ratio, and mean difference for each in situ
physiological signal. These created the data frame. To calculate
the peak ratio [32], we divided the number of peaks by the lengthof the physiological signal data.
We used real data for the prediction model; therefore, generating
data was a challenge. The severe and nonsevere groups variedin size; therefore, we manipulated the data set to improve themodel performance. We did not consider the undersamplingtechnique because the number of participants was insufficient.
Conversely, we used the oversampling technique on the minority
group to improve classifier performance.
The SMOTE (Synthetic Minority Oversampling Technique)approach generated insufficient data [33]. This method helps
construct classifiers from imbalanced data sets (where theclassification categories are not approximately equallydistributed), such as real-world data sets with a small numberof abnormal cases.

Data Labeling and Processing

Statistical Analysis

We used k=2 to categorize the specific anxiety symptoms inthe severe and nonsevere groups, where k is the number of
median points of the clusters to be classified. Data are grouped
as per the number of hyperparameters k. In k-means clustering,
the algorithm can be executed when the user determines thenumber of clusters. K-means clustering is a representative
detached clustering algorithm that uses unsupervised learning[31].

Group-Based Analysis

We labeled the severe and nonsevere groups using k-meansclustering for each specific anxiety symptom and VR sickness.
The ISS cutoff values for dividing the severe groups were 48(total score), 11 (mistake anxiety), 9 (self-punishment), 10(emptiness), and 16 (inappropriate). The PERS cutoff values
were 41 (total score), 17 (positive), and 27 (negative). For the
SSQ, severity classification was performed using k-meansclustering via the sum of the response scores for each category.
The cutoff values for dividing the severe group were 9 (total
score), 7 (nausea), 8 (oculomotor), and 4 (disorientation).

https://games.jmir.org/2022/3/e38284

XSL• FO
RenderX

Differences between groups (severe and nonsevere) wereassessed using independent t tests in Python with SciPy version
1.5.2. Differences were considered statistically significant at
P<.05. Furthermore, we analyzed correlations between theextracted variables. The variables with the strongest positive
correlations were mean and peak GSR ratios. The variables withthe second strongest positive correlations were the mean andstandard deviation of the GSR. The variables with the strongest
negative correlations were the linear regression coefficient andstandard deviation of the GSR. The second most negatively
correlated variables were the GSR difference and HR linear
regression coefficient. All correlations are illustrated in
Multimedia Appendix 2. There was a high correlation between
the mean and peak GSR (0.92); however, removing this feature
decreased the prediction score. Therefore, this feature wasretained. All other correlations were <0.86.

JMIR Serious Games 2022 | vol. 10 | iss. 3 | e38284 | p. 5
(page number not for citation purposes)

JMIR SERIOUS GAMES

Chun et al

Machine Learning Techniques
All machine learning techniques were performed using Python
version 3.7.7. Logistic regression [34], random forest [35], andnaïve Bayes [36] classification were performed using
Scikit-learn version 0.23.2. The random forest model avoids
overfitting, thereby improving model accuracy. Additionally,
we can select relatively important ranks of variables in theclassification model. The naïve Bayes model is a conditional
probability-based classification method that calculates theprobability of features belonging to each class and handles noiseand missing values reasonably.
In addition, when training, it works excellently regardless ofthe size of the data, and it is easy to obtain the estimatedprobabilities for prediction. Although 32 samples of data could
be insufficient, we amplified the patients’ data using SMOTE
methodology after segmentation for each session. We usedmachine learning models that can perform reasonable prediction
with a few data points to minimize performance degradation
because of insufficient data. We calculated the accuracy, F1score (weighted average of precision and recall), and area under
the curve (AUC) for the prediction.
All machine learning models, except for the random forestmodel, were performed using the oversampling technique on
the training data set. This supplemented insufficient data after
the training-test split. We performed 5-fold cross-validation for
the random forest model to improve the prediction score. During

the prediction, sampling of the severe and nonsevere groups
was stratified for training-test splits to ensure equally frequent
severity.

Results
Classified Specific Anxiety Symptoms and VR Sickness
Specific anxiety symptom and VR sickness results werecompared between the severe and nonsevere groups using
k-means clustering. The smallest difference between group sizes
was observed for ISS self-punishment (severe vs nonseveregroup, 53 vs 51). Based on the average score of the specificanxiety symptoms, the most significant difference between thegroups was the ISS score (severe vs nonsevere group, 59.951
vs 33.222). ISS emptiness showed the smallest difference
between groups (severe vs nonsevere group, 12.91 vs 5.75).
After labeling with k-means clustering, the anxiety symptom
with the lowest distortion was ISS mistake anxiety and that withthe highest distortion was total ISS (1.263 vs 7.439).
After 4 VR sessions with 26 participants, 104 sets of VR
sickness data were collected. These were labeled as severe andnonsevere groups using k-means clustering. Distortion of
k-means clustering was 2.57 for total VR sickness, 1.44 fornausea, 1.71 for oculomotor, and 1.16 for disorientation. Table
1 presents the statistical analysis results of the severe andnonsevere groups. The boxplot for the severe and nonseveregroups is illustrated in Multimedia Appendix 3.

Table 1. Severe and nonsevere groups clustered by k-means clustering.
MeasureGroupsa

Minimum Cutoff
value
value

Maximumvalue

Distortionb

Nonseveregroup countSevere group
countNonsevere group Severe group
mean
mean

ISSc

63

41

33.22259.951

3

<48

92

7.439

ISS mistake anxiety

34

70

8.176

12.957

1

<11

16

1.263

ISS self-punishment

51

53

4.961

12.453

1

<9

20

1.835

ISS emptiness

60

44

5.750

12.910

0

<10

20

2.104

ISS inappropriate

58

46

9.155

21.978

2

<16

36

3.344

d

PERS

47

57

32.76648.140

19

<41

64

4.203

PERS positive

36

68

10.05921.000

2

<17

35

3.007

PERS negative

50

54

16.90036.016

3

<27

56

5.163

Total VRe sickness

73

31

3.247

13.710

0

<9

29

2.568

Nausea group

98

6

1.786

10.500

0

<7

15

1.445

Oculomotor group

63

41

2.238

8.341

0

<6

15

1.709

Disorientation group

69

35

0.986

6.000

0

<4

12

1.164

a

After labeling into severe and nonsevere groups through k-means clustering, the numerical characteristics and differences between the groups areshown for each group.
b

Distortion was calculated using the k-means clustering model with k=2.

c

ISS: Internalized Shame Scale.

d

PERS: Post-Event Rumination Scale.

e

VR: virtual reality.

https://games.jmir.org/2022/3/e38284

XSL• FO
RenderX

JMIR Serious Games 2022 | vol. 10 | iss. 3 | e38284 | p. 6
(page number not for citation purposes)

JMIR SERIOUS GAMES

Differences in Physiological Signals According toSpecific Anxiety Symptom Severity
There were significant differences in the GSR min between thesevere and nonsevere groups stratified by the ISS (t102=1.39;
P=.17). In addition, groups stratified by the ISS mistake anxiety
showed a significant difference in the HR peak ratio (t102=2.07;
P=.04) and GSR average change (t102=2.02; P=.046). Groups
stratified by ISS self-punishment showed significantly different
GSR min values (t102=2.34; P=.02). Groups stratified by ISS
emptiness showed a difference in mean HR (t102=−2.44; P=.02).
Groups stratified by ISS inappropriate showed a significant
difference in mean HR (t102=−2.23; P=.03). Groups stratified
by PERS had significantly different mean HRs (t102=−1.99;
P=.050). Furthermore, groups stratified by PERS positivity
showed significantly different mean HRs (t102=2.51; P=.01),
HR min values (t102=2.49; P=.01), and HR peak ratios(t102=2.15; P=.03). Groups stratified by PERS negative hadsignificantly different mean HRs (t102=−2.79; P=.006), HR min
values (t102=−2.49; P=.01), HR peak ratios (t102=−2.15; P=.03),
and GSR max values (t102=−2.00; P=.048). The severe andnonsevere groups by anxiety symptoms are illustrated in
Multimedia Appendix 4.
We performed t tests after labeling the severity of VR sickness,for each VR sickness subdomain. The most significant difference
in total VR sickness was for the HR min value (t102=−1.63;
P=.02). In the nausea group, the HR max value was significantly
different between the groups (t102=−2.47; P=.02). Additionally,
the oculomotor group differed in the HR min value (t102=−2.19;
P=.03). The disorientation group showed a difference in total
HR change (t102=−2.14; P=.04) and total GSR change(t102=−2.09; P=.04). Differences in the severe and nonseveregroups according to VR sickness are illustrated in Multimedia
Appendix 5.

Specific Anxiety Symptom and VR Sickness Prediction
Based on the Physiological Signal Data
After oversampling the minority data using SMOTE, the F1score [37], accuracy, and area under the receiver operating
characteristic curve were calculated after predicting the survey
results using logistic regression analysis, random forest, andhttps://games.jmir.org/2022/3/e38284

XSL• FO
RenderX

Chun et alnaïve Bayesian methodology. Additionally, we calculated thefeature importance of specific anxiety symptoms. The variable
importance can be calculated based on the impurity index [38].
We used the feature importance of a random forest–based
prediction model.

Anxiety Symptom Prediction
The classifications of specific anxiety symptom results wereanalyzed based on 3 models. Logistic regression [34], random
forest [35], and naïve Bayes classifier [36] analyzed the severityof specific anxiety symptoms for participants. The data were
amplified using the SMOTE method to prevent performancedegradation due to unbalanced data.
The classification performance was measured using the F1 score.
The F1 scores of the ISS mistake anxiety subdomain were
0.8421, 0.7368, and 0.7647, as calculated by logistic regression,
random forest, and naïve Bayesian classifier, respectively. These
values were higher than those of the other ISS subdomains. The
classification performance of the PERS negative subdomain
was relatively higher than that of the PERS positive subdomain,with 0.6667, 0.6154, and 0.6452 as measured by logistic
regression, random forest, and naïve Bayes classifier,respectively.
Specifically, we classified the ISS using a logistic regression
model with an F1 score of 0.7619. The logistic regression model
revealed an F1 score of 0.8421 for ISS mistake anxiety, whereas
the naïve Bayes classification model revealed a score of 0.7857
for ISS self-punishment. The logistic regression model revealed
an F1 score of 0.6429 for ISS emptiness and 0.7200 for ISS
inappropriate. The random forest model revealed an F1 scoreof 0.7568 for PERS. Further, the naïve Bayes classification
model showed an F1 score of 0.7619 for PERS positivity.
Finally, the random forest model revealed an F1 score of 0.7097
for PERS negativity. The specific anxiety symptom prediction
results are illustrated in Table 2 and Multimedia Appendix 6.
For ISS, ISS mistake anxiety, ISS self-punishment, ISS
inappropriate, PERS, PERS positive, and PERS negative,
physiological signal data related to GSR had the highest
importance, and for ISS emptiness, physiological signal data
related to HR had the highest importance. The featureimportance of specific anxiety symptoms in the random forestmodel is illustrated in Figure 2.

JMIR Serious Games 2022 | vol. 10 | iss. 3 | e38284 | p. 7
(page number not for citation purposes)

JMIR SERIOUS GAMES

Chun et al

Table 2. Specific anxiety symptom classification model evaluation with F1 score, accuracy, and area under the curve.
Variablea

Logistic regression

Random forestNaïve Bayes classifierF1 score

Accuracy

AUCb

F1 score

Accuracy

AUC

F1 score

Accuracy

AUC

ISSc

0.762

0.808

0.827

0.632

0.731

0.706

0.727

0.769

0.775

ISS mistake anxiety

0.842

0.769

0.733

0.790

0.692

0.608

0.765

0.692

0.660

ISS self-punishment

0.750

0.692

0.673

0.727

0.769

0.769

0.786

0.769

0.769

ISS emptiness

0.643

0.615

0.625

0.571

0.654

0.639

0.600

0.692

0.673

ISS inappropriate

0.720

0.731

0.732

0.571

0.654

0.639

0.714

0.692

0.702

d

PERS

0.667

0.654

0.676

0.757

0.654

0.625

0.667

0.615

0.607

PERS positive

0.615

0.615

0.630

0.500

0.769

0.660

0.762

0.808

0.861

PERS negative

0.667

0.654

0.654

0.710

0.654

0.654

0.645

0.577

0.577

a

After predicting the severity of each specific anxiety symptom using logistic regression, random forest, and naïve Bayes classifier models, the performanceof each model was evaluated.
b

AUC: area under the curve.

c

ISS: Internalized Shame Scale.

d

PERS: Post-Event Rumination Scale.

https://games.jmir.org/2022/3/e38284

XSL• FO
RenderX

JMIR Serious Games 2022 | vol. 10 | iss. 3 | e38284 | p. 8
(page number not for citation purposes)

JMIR SERIOUS GAMES

Chun et al

Figure 2. Feature importance of specific anxiety symptoms (random forest model). After predicting the subdomains of each anxiety symptom using
the random forest model, the feature importance for each model was calculated and sorted in descending order. GSR: galvanic skin response; ISS:
Internalized Shame Scale; PERS: Post-Event Rumination Scale.

VR Sickness Prediction
We classified the severity of total VR sickness with an F1 scoreof 0.7059, that in the nausea group with a score of 0.4000, thatin the oculomotor group with a score of 0.6667, and that in thedisorientation group with a score of 0.6364. The classification
performance for the nausea group subdomain had the highest
AUC of 0.94. The VR sickness results are illustrated in Table
3 and Multimedia Appendix 7.

https://games.jmir.org/2022/3/e38284

XSL• FO
RenderX

When feature importance was calculated for the classificationof the severity of VR sickness for each type, physiological signal
data related to GSR were selected as essential variables for all
subdomains. All subdomains of VR sickness symptoms showed
HR-related features as the second most crucial factor invariation. Figure 3 illustrates the importance of VR sickness
features from the random forest model.

JMIR Serious Games 2022 | vol. 10 | iss. 3 | e38284 | p. 9
(page number not for citation purposes)

JMIR SERIOUS GAMES

Chun et al

Table 3. Virtual reality sickness classification model evaluation with F1 score, accuracy, and area under the curve.
VariableaaRandom forest
F1 score

Accuracy

AUCb

Total VRc sickness

0.70590.80770.7917

Nausea group0.40000.88460.9400Oculomotor group0.66670.65380.7000

Disorientation group0.63640.69230.7124After predicting the severity of each VR sickness scale using the random forest model, the model’s performance was evaluated.

b

AUC: area under the curve.

c

VR: virtual reality.

Figure 3. Feature importance of virtual reality (VR) sickness (random forest model). After predicting the subdomains of each VR sickness scale using
the random forest model, the feature importance for each model was calculated and sorted in descending order. GSR: galvanic skin response.

Discussion
Principal Findings
We developed and tested machine learning models to predict
specific anxiety symptoms and VR sickness for SAD using insitu autonomic physiological signal data measured during
participatory and interactive VR treatment. The severity of
specific anxiety symptoms and the side effects of VR treatment
https://games.jmir.org/2022/3/e38284

XSL• FO
RenderX

are an essential part of digital therapy. Usually, these kinds ofevaluations rely on subjective reporting by patients. However,autonomic physiological responses, such as blushing, sweating,and shivering, could play a central role in assessing symptoms[39]; therefore, they have the potential to be used for monitoring
various symptoms related to anxiety or VR sickness.
Previous studies have assessed the relationship between
physiological cues and SAD. However, real-time analysis is
JMIR Serious Games 2022 | vol. 10 | iss. 3 | e38284 | p. 10(page number not for citation purposes)

JMIR SERIOUS GAMES
limited because of current sensor technology and evaluation
methods [40]. Furthermore, individuals with SAD cannot betreated because of methodological problems [41-43].
VR treatment techniques extend beyond the traditional
psychiatric therapeutic approach [44]. To create a therapeutic
VR system, we must build an interactive system rather thansimply provide a VR environment. Interactive VR may haveseveral benefits for psychiatric disorders in terms of treatment.
First, it enhances ecological validity by immersing individuals
in emotionally engaging virtually constructed therapeutic
environments. Second, it can be flexibly used to present patientsin various contexts, enabling personalized treatment according
to the assessment of individual differences in symptoms during
VR treatment [45]. We expect our findings to be useful ininteractive VR treatment, especially for specific anxiety
symptoms and VR sickness.
We showed that the data extracted from the time series withdifferent in situ autonomic physiological signal data lengths
could be used as the independent variables of a predictive model.
As 32 samples of physiological data were not sufficient for themachine learning model, we increased the data by dividing it
per session using only 32 participants’ intervention data andsupplementing the insufficient data using SMOTE methodology.
This technique can minimize performance degradation due toa lack of data. We generated cutoff values for the classificationgroups for labeling via k-means clustering. This is anunsupervised learning method based on the absence of a label
to evaluate the model performance. We could explain which
variable plays the most crucial role in predicting specific anxiety
symptoms and VR sickness.
Among the anxiety symptoms related to the ISS, the most
predictable was ISS mistake anxiety. Emotion comprised 5 main
subsystems, namely, “cognitive component (appraisal),
neurophysiological component (bodily symptoms), motivational
component (action tendencies), motor expression component(facial and vocal expression), and subjective feeling component(emotional experience)” [46]. Shame is highly correlated withthe body’s physiological responses because of its mechanism[47]. Mistake anxiety is the core symptom of SAD. The highpredictability of these symptoms is related to the reliability of
HR and GSR predictions of SAD symptoms.
Among the scales related to PERS, the most predictable was
PERS positivity. Rumination may negatively affect physicaland mental health [48,49]. However, this is inconsistent withprevious studies reporting that positive rumination either is not
associated with social anxiety or is low [28,30]. In contrast,
other studies have reported that positive and negative rumination
is high in social anxiety. Socially anxious individuals ruminate
broadly, positively, or negatively about all aspects of social
interactions when faced with ambiguity [50]. However, noprevious study has examined the relationship between in situ
autonomic physiological responses, and negative and positive
rumination. Additional studies are required to determine whether
these results are reproducible.
Several factors can cause VR sickness. Humans perceive theirdirection and movement through various sensory organs.
Individuals may experience motion sickness if they repeatedly
https://games.jmir.org/2022/3/e38284

XSL• FO
RenderX

Chun et al
receive sensory information that differs from their prediction[51]. Visual movement may cause motion sickness [52,53].
When visual stimulation is the primary cause of motion sickness,
it is called visually induced motion sickness. Previous studies
have assessed virtual perceptions related to recognizing
self-motion [54]. If the degree of physiological arousal (HR,blood pressure, skin conductance, respiration, skin temperature,and blood volume pulse amplitude) is high, the possibility of
VR motion sickness is high [55].
We used HR and GSR for machine learning–based predictionin our study. These are important indicators of physiologicalarousal. SAD-related psychological symptoms and VR sickness
during VR treatment can be determined in advance through theproposed model. The intervention of medical staff can also
improve patient compliance. However, we must develop
additional real-time measurement factors to better predict otherVR sickness subdomain symptoms.
We predicted specific anxiety symptom severity and VR
sickness severity via in situ physiological signal data from actual
cases. We used supervised and unsupervised learning and data
generation to build and evaluate the SAD and VR sickness
predictive models. Considering these advantages, specificanxiety symptoms and VR sickness could be assessed moreaccurately. The data were labeled using an unsupervised learning
method. After separating the severe and nonsevere groups,
significant differences were found for each classification groupusing a t test. The real-time evaluation of VR motion sickness
can help improve patient compliance with treatment. Moreover,
we can reduce the time required for a survey by predicting
participants’ anxiety using objective data. Predicting survey
results using objective in situ autonomic physiological signal
data makes less subjective intervention possible compared withthe conventional survey-based method. In addition, theevaluation of symptoms through real-time autonomic
physiological signals shows the possibility of evaluating
psychiatric symptoms in real time to increase interactivity, anessential element in the personalized VR treatment process.

Limitations
This study had several limitations. First, our study evaluated insitu autonomic physiological signals and symptom-related data
in a SAD group alone. Subsequently, it included no control
groups. This study used machine learning models to discriminate
between severe and nonsevere groups because real-time
physiological signal data were measured only for participants
above a certain Korean Social Interaction Anxiety Scale(K-SAD) score. In some cases, after labeling the severe andnonsevere groups, there was no significant difference between
groups. These results can be inferred as a limitation of themethodology that divides the group with a K-SAD score above
82 into severe and nonsevere groups according to specificanxiety symptoms. Although an unsupervised learning
methodology distinguished respondents, it was challenging to
explain clear criteria other than to the participants in theexperiment. Second, this study used self-rating score-based
scales, which could be associated with bias. Third, this study
could only classify the severity of SAD (K-SAD ≥82) in thegroup with a particular score or higher. Fourth, there was
JMIR Serious Games 2022 | vol. 10 | iss. 3 | e38284 | p. 11(page number not for citation purposes)

JMIR SERIOUS GAMES
insufficient evaluation of the response to a specific task or
exposure situation during a VR session. Because insufficient
data are obtained using a SMOTE model, there is a limitationin not using the actual data. In addition, it is unknown how
performance changes as an individual progresses during asession. Furthermore, we cannot determine how in situ
autonomic physiological signals change when a specific situation
occurs in each session. Fifth, the number of participants who
participated in the VR treatment session was 32, but the numberof participants used for the analysis was 26 (a small samplesize). The decrease in performance can be inferred to be limited
to the improvement in performance because the number ofindividuals who participated in the VR sessions was small.
Therefore, we need to carefully interpret the results and securea larger number of samples for future studies. However, since
it is challenging to obtain real-time in situ data, we believe that

Chun et al
the strength of this study is that real valuable data fromparticipants were utilized for analysis.

Conclusion
This study showed that using in situ autonomous physiologicalsignal data measured during a VR intervention can predict
specific anxiety symptoms and VR sickness in patients with
SAD. Using real-time physiological data from VR sessions, we
can classify the severity of specific SAD symptoms and utilize
the findings for personalized digital treatment. Machine learningmodels can assist in the decisions of medical staff and theconstruction of interactive VR treatment. Future research should
focus on various predictive methodologies to enhance thetailored interactive function and to maximize the convenienceof VR treatment. Additionally, to improve the clinical prediction
performance and increase accuracy, more abundant andappropriate data will need to be collected.

Acknowledgments
This work was supported by a National Research Foundation (NRF) of Korea grant funded by the Ministry of Science andInformation and Communications Technology (MSIT), the Government of Korea (NRF-2020R1C1C1007463 andNRF-2021R1A5A8032895), and Information and Communications Technology (ICT) and Future Planning for Convergent
Research in the Development Program for R&D Convergence over Science and Technology Liberal Arts
(NRF-2022M3C1B6080866).

Authors' Contributions
Authors CHC (david0203@gmail.com) and TC (tcheong@korea.ac.kr) are co-corresponding authors for this article, responsiblefor data and materials, manuscript submission, peer review, publication process, authorship details, and ethics committee approval.
JWH, DJ, HJL, SL, GK, CYC, SML, SC, TC, and CHC conceived and designed the study. JYC, HJK, TC, and CHC performed
statistical analyses. JYC, HJK, JWH, DJ, TC, and CHC wrote the first draft of this manuscript. JYC, HJK, JWH, DJ, SL, GK,
CYC, SML, HL, SC, TC, and CHC participated in data collection. All authors have edited all versions of the manuscript. All
authors were involved in interpreting the results, and they read, commented on, and approved the final version of the manuscript.

Conflicts of Interest
None declared.

Multimedia Appendix 1
Psychological scale prediction (linear regression model).
[PDF File (Adobe PDF File), 344 KB-Multimedia Appendix 1]

Multimedia Appendix 2
Correlation table for each feature.
[PDF File (Adobe PDF File), 298 KB-Multimedia Appendix 2]

Multimedia Appendix 3
Box plot of specific anxiety symptoms and virtual reality sickness severe and nonsevere group clustering results.
[PDF File (Adobe PDF File), 864 KB-Multimedia Appendix 3]

Multimedia Appendix 4
Results of specific anxiety symptoms.
[PDF File (Adobe PDF File), 385 KB-Multimedia Appendix 4]

Multimedia Appendix 5
Results of virtual reality sickness.
[PDF File (Adobe PDF File), 380 KB-Multimedia Appendix 5]
https://games.jmir.org/2022/3/e38284

XSL• FO
RenderX

JMIR Serious Games 2022 | vol. 10 | iss. 3 | e38284 | p. 12(page number not for citation purposes)

JMIR SERIOUS GAMES

Chun et alMultimedia Appendix 6
Receiver operating characteristic curve of specific anxiety symptom prediction models (logistic regression model, random forestmodel, and naïve Bayesian model).
[PDF File (Adobe PDF File), 853 KB-Multimedia Appendix 6]

Multimedia Appendix 7
Receiver operating characteristic curve of a virtual reality sickness prediction model (random forest model).
[PDF File (Adobe PDF File), 382 KB-Multimedia Appendix 7]

References
1.
2.
3.
4.
5.
6.
7.
8.
9.

10.
11.
12.

13.

14.

15.

16.

17.
18.

19.

Cannistraro P, Rauch S. Neural circuitry of anxiety: evidence from structural and functional neuroimaging studies.
Psychopharmacol Bull 2003;37(4):8-25. [Medline: 15131515]
Garfinkel SN, Eccles JA, Critchley HD. The heart, the brain, and the regulation of emotion. JAMA Psychiatry 2015
Nov;72(11):1071-1072. [doi: 10.1001/jamapsychiatry.2015.1493] [Medline: 26376236]
Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition. Washington, DC: American Psychiatric Association;
2013.
Thurston RC, Rewak M, Kubzansky LD. An anxious heart: anxiety and the onset of cardiovascular diseases. Prog Cardiovasc
Dis 2013 May;55(6):524-537. [doi: 10.1016/j.pcad.2013.03.007] [Medline: 23621962]Friedman BH. An autonomic flexibility-neurovisceral integration model of anxiety and cardiac vagal tone. Biol Psychol
2007 Mar;74(2):185-199. [doi: 10.1016/j.biopsycho.2005.08.009] [Medline: 17069959]
Rapee RM, Heimberg RG. A cognitive-behavioral model of anxiety in social phobia. Behaviour Research and Therapy
1997 Aug;35(8):741-756. [doi: 10.1016/s0005-7967(97)00022-3]
Heimberg RG. Cognitive-behavioral therapy for social anxiety disorder: current status and future directions. BiologicalPsychiatry 2002 Jan;51(1):101-108. [doi: 10.1016/s0006-3223(01)01183-0]
Maples-Keller JL, Bunnell BE, Kim S, Rothbaum BO. The use of virtual reality technology in the treatment of anxiety andother psychiatric disorders. Harv Rev Psychiatry 2017;25(3):103-113. [doi: 10.1097/hrp.0000000000000138]
Geraets CN, Veling W, Witlox M, Staring AB, Matthijssen SJ, Cath D. Virtual reality-based cognitive behavioural therapy
for patients with generalized social anxiety disorder: a pilot study. Behav. Cogn. Psychother 2019 Mar 27;47(6):745-750.
[doi: 10.1017/s1352465819000225]
McCauley ME, Sharkey TJ. Cybersickness: perception of self-motion in virtual environments. Presence: Teleoperators &Virtual Environments 1992 Jan;1(3):311-318. [doi: 10.1162/pres.1992.1.3.311]
LaViola JJ. A discussion of cybersickness in virtual environments. SIGCHI Bull 2000 Jan 01;32(1):47-56. [doi:10.1145/333329.333344]
Kennedy RS, Lane NE, Berbaum KS, Lilienthal MG. Simulator sickness questionnaire: an enhanced method for quantifying
simulator sickness. The International Journal of Aviation Psychology 1993 Jul;3(3):203-220. [doi:10.1207/s15327108ijap0303_3]
Sevinc V, Berkman MI. Psychometric evaluation of Simulator Sickness Questionnaire and its variants as a measure ofcybersickness in consumer virtual environments. Appl Ergon 2020 Jan;82:102958. [doi: 10.1016/j.apergo.2019.102958][Medline: 31563798]
Bouchard S, Dumoulin S, Robillard G, Guitard T, Klinger É, Forget H, et al. Virtual reality compared with exposure in thetreatment of social anxiety disorder: a three-arm randomised controlled trial. Br J Psychiatry 2017 Apr;210(4):276-283.
[doi: 10.1192/bjp.bp.116.184234] [Medline: 27979818]
Kim H, Lee S, Jung D, Hur J, Lee H, Lee S, et al. Effectiveness of a participatory and interactive virtual reality intervention
in patients with social anxiety disorder: longitudinal questionnaire study. J Med Internet Res 2020 Oct 06;22(10):e23024[FREE Full text] [doi: 10.2196/23024] [Medline: 33021481]
Lee H, Choi J, Jung D, Hur J, Cho C. The effects of virtual reality treatment on prefrontal cortex activity in patients withsocial anxiety disorder: participatory and interactive virtual reality treatment study. J Med Internet Res 2021 Dec
17;23(12):e31844 [FREE Full text] [doi: 10.2196/31844] [Medline: 34801979]
Yoo S, Kim Y, Noh J, Oh K, Kim C, NamKoong K, et al. Validity of Korean version of the mini-international neuropsychiatricinterview. Anxiety and Mood 2006;2(1):50-55 [FREE Full text]
Hur J, Shin H, Jung D, Lee H, Lee S, Kim GJ, et al. Virtual reality-based psychotherapy in social anxiety disorder: fMRI
study using a self-referential task. JMIR Ment Health 2021 Apr 14;8(4):e25731 [FREE Full text] [doi: 10.2196/25731][Medline: 33851931]
Powers MB, Briceno NF, Gresham R, Jouriles EN, Emmelkamp PMG, Smits JAJ. Do conversations with virtual avatars
increase feelings of social anxiety? J Anxiety Disord 2013 May;27(4):398-403. [doi: 10.1016/j.janxdis.2013.03.003][Medline: 23746490]

https://games.jmir.org/2022/3/e38284

XSL• FO
RenderX

JMIR Serious Games 2022 | vol. 10 | iss. 3 | e38284 | p. 13(page number not for citation purposes)

JMIR SERIOUS GAMES
20.

21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.

38.
39.
40.

41.

42.

43.

44.

45.

46.

Anderson PL, Price M, Edwards SM, Obasaju MA, Schmertz SK, Zimand E, et al. Virtual reality exposure therapy forsocial anxiety disorder: a randomized controlled trial. J Consult Clin Psychol 2013 Oct;81(5):751-760. [doi:10.1037/a0033559] [Medline: 23796315]
Social Anxiety Disorder VR treatment. YouTube. URL: https://www.youtube.com/channel/UCCpzKK_qEqz8LRfxl_TfGXw/
featured [accessed 2022-08-29]
Beck A, Epstein N, Brown G, Steer R. Beck anxiety inventory. J of Consult Clin Psychol 1993:23-40 [FREE Full text][doi: 10.1037/t02025-000]
Spielberger CD, Gonzalez-Reigosa F, Martinez-Urrutia A, Natalicio LF, Natalicio DS. The state-trait anxiety inventory.
Revista Interamericana De Psicología/Interamerican Journal of Psychology 1971;5:145-158 [FREE Full text]
Davidson J, Potts N, Richichi E, Ford S, Krishnan K, Smith R, et al. The Brief Social Phobia Scale. J Clin Psychiatry 1991
Nov;52 Suppl:48-51. [doi: 10.1037/t07672-000] [Medline: 1757457]
Mattick RP, Clarke JC. Development and validation of measures of social phobia scrutiny fear and social interaction anxiety.
Behav Res Ther 1998 Apr;36(4):455-470. [doi: 10.1016/s0005-7967(97)10031-6] [Medline: 9670605]
Watson D, Friend R. Measurement of social-evaluative anxiety. J Consult Clin Psychol 1969 Aug;33(4):448-457. [doi:10.1037/h0027806] [Medline: 5810590]
Cook DR. Measuring shame. Alcoholism Treatment Quarterly 1988 Mar 07;4(2):197-215. [doi: 10.1300/j020v04n02_12]
Abbott MJ, Rapee RM. Post-event rumination and negative self-appraisal in social phobia before and after treatment. J
Abnorm Psychol 2004 Mar;113(1):136-144. [doi: 10.1037/0021-843X.113.1.136] [Medline: 14992666]
Lee I, Choi H. Assessment of shame and its relationship with maternal attachment, hypersensitive narcissism and loneliness.
Korean Journal of Counseling and Psychotherapy 2005;17(3):651-670 [FREE Full text]
Edwards S, Rapee R, Franklin J. Postevent rumination and recall bias for a social performance event in high and low sociallyanxious individuals. Cognitive Therapy and Research 2003;27:603-617. [doi: 10.1023/A:1026395526858]
Jain AK. Data clustering: 50 years beyond K-means. Pattern Recognition Letters 2010 Jun;31(8):651-666. [doi:10.1016/j.patrec.2009.09.011]
Duarte M. detecta: A Python module to detect events in data (Version v0.0.5). Zenodo. URL: https://zenodo.org/record/
4598962#.YwzRTLRBzDc [accessed 2022-08-29]
Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP. SMOTE: Synthetic Minority Over-sampling Technique. jair 2002
Jun 01;16:321-357. [doi: 10.1613/jair.953]
LaValley MP. Logistic regression. Circulation 2008 May 06;117(18):2395-2399. [doi: 10.1161/circulationaha.106.682658]
Pal M. Random forest classifier for remote sensing classification. International Journal of Remote Sensing 2007 Feb
22;26(1):217-222. [doi: 10.1080/01431160412331269698]
Murphy KP. Naive Bayes classifiers. University of British Columbia. 2006. URL: https://www.ic.unicamp.br/~rocha/
teaching/2011s2/mc906/aulas/naive-bayes.pdf [accessed 2022-08-29]
Sokolova M, Japkowicz N, Szpakowicz S. Beyond Accuracy, F-Score and ROC: A family of discriminant measures for
performance evaluation. In: Sattar A, Kang B, editors. AI 2006: Advances in Artificial Intelligence. AI 2006. Lecture Notes
in Computer Science, vol 4304. Berlin, Heidelberg: Springer; 2006:1015-1021.
Breiman L. Random forests. Machine Learning 2001;45:5-32. [doi: 10.1023/A:1010933404324]
Gerlach AL, Mourlane D, Rist F. Public and private heart rate feedback in social phobia: a manipulation of anxiety visibility.
Cogn Behav Ther 2004 Mar;33(1):36-45. [doi: 10.1080/16506070310014682] [Medline: 15224627]
Alvares GA, Quintana DS, Kemp AH, Van Zwieten A, Balleine BW, Hickie IB, et al. Reduced heart rate variability insocial anxiety disorder: associations with gender and symptom severity. PLoS One 2013;8(7):e70468 [FREE Full text][doi: 10.1371/journal.pone.0070468] [Medline: 23936207]
Rösler L, Göhring S, Strunz M, Gamer M. Social anxiety is associated with heart rate but not gaze behavior in a real social
interaction. J Behav Ther Exp Psychiatry 2021 Mar;70:101600 [FREE Full text] [doi: 10.1016/j.jbtep.2020.101600][Medline: 32882674]
Wen W, Liu G, Mao Z, Huang W, Zhang X, Hu H, et al. Toward constructing a real-time social anxiety evaluation system:
exploring effective heart rate features. IEEE Trans. Affective Comput 2020 Jan 1;11(1):100-110. [doi:10.1109/taffc.2018.2792000]
Zhang X, Wen W, Liu G, Hu H. Recognition of public speaking anxiety on the recurrence quantification analysis of GSR
signals. 2016 Presented at: Sixth International Conference on Information Science and Technology (ICIST); May 06-08,
2016; Dalian, China. [doi: 10.1109/icist.2016.7483472]
Park MJ, Kim DJ, Lee U, Na EJ, Jeon HJ. A literature overview of virtual reality (VR) in treatment of psychiatric disorders:
recent advances and limitations. Front Psychiatry 2019 Jul 19;10:505 [FREE Full text] [doi: 10.3389/fpsyt.2019.00505][Medline: 31379623]
Verhoef REJ, van Dijk A, Verhulp EE, de Castro BO. Interactive virtual reality assessment of aggressive social information
processing in boys with behaviour problems: A pilot study. Clin Psychol Psychother 2021 May 17;28(3):489-499 [FREEFull text] [doi: 10.1002/cpp.2620] [Medline: 34048619]
Scherer KR. What are emotions? And how can they be measured? Social Science Information 2016 Jun 29;44(4):695-729.
[doi: 10.1177/0539018405058216]

https://games.jmir.org/2022/3/e38284

XSL• FO
RenderX

Chun et al

JMIR Serious Games 2022 | vol. 10 | iss. 3 | e38284 | p. 14(page number not for citation purposes)

JMIR SERIOUS GAMES
47.

48.

49.
50.
51.
52.
53.
54.
55.

Chun et al

Hedman E, Ström P, Stünkel A, Mörtberg E. Shame and guilt in social anxiety disorder: effects of cognitive behavior
therapy and association with social anxiety and depressive symptoms. PLoS One 2013;8(4):e61713 [FREE Full text] [doi:10.1371/journal.pone.0061713] [Medline: 23620782]
Brosschot JF, Verkuil B, Thayer JF. Conscious and unconscious perseverative cognition: is a large part of prolonged
physiological activity due to unconscious stress? J Psychosom Res 2010 Oct;69(4):407-416. [doi:10.1016/j.jpsychores.2010.02.002] [Medline: 20846542]
Glynn LM, Christenfeld N, Gerin W. The role of rumination in recovery from reactivity: cardiovascular consequences of
emotional states. Psychosom Med 2002;64(5):714-726. [doi: 10.1097/01.psy.0000031574.42041.23] [Medline: 12271102]
Dannahy L, Stopa L. Post-event processing in social anxiety. Behav Res Ther 2007 Jun;45(6):1207-1219. [doi:10.1016/j.brat.2006.08.017] [Medline: 17112463]
Sherman C. Motion sickness: review of causes and preventive strategies. J Travel Med 2002;9(5):251-256 [FREE Full text][doi: 10.2310/7060.2002.24145] [Medline: 12962599]
Bonato F, Bubka A, Palmisano S, Phillip D, Moreno G. Vection change exacerbates simulator sickness in virtualenvironments. Presence: Teleoperators and Virtual Environments 2008 Jun 01;17(3):283-292. [doi: 10.1162/pres.17.3.283]
Keshavarz B, Riecke BE, Hettinger LJ, Campos JL. Vection and visually induced motion sickness: how are they related?
Front Psychol 2015 Apr 20;6:472 [FREE Full text] [doi: 10.3389/fpsyg.2015.00472] [Medline: 25941509]
Lubeck AJ, Bos JE, Stins JF. Motion in images is essential to cause motion sickness symptoms, but not to increase posturalsway. Displays 2015 Jul;38:55-61. [doi: 10.1016/j.displa.2015.03.001]
Macedonio MF, Parsons TD, DiGiuseppe RA, Weiderhold BA, Rizzo AA. Immersiveness and physiological arousal within
panoramic video-based virtual reality. Cyberpsychol Behav 2007 Aug;10(4):508-515. [doi: 10.1089/cpb.2007.9997][Medline: 17711358]

Abbreviations
AUC: area under the curve
CBT: cognitive behavioral therapy
GSR: galvanic skin response
HR: heart rate
ISS: Internalized Shame Scale
K-SAD: Korean Social Interaction Anxiety Scale
PERS: Post-Event Rumination Scale
SAD: social anxiety disorder
SMOTE: Synthetic Minority Oversampling Technique
SSQ: Simulator Sickness Questionnaire
VR: virtual reality

Edited by N Zary; submitted 27.03.22; peer-reviewed by H Kim, K Lee, K Uludag; comments to author 18.04.22; revised version
received 30.06.22; accepted 21.07.22; published 16.09.22
Please cite as:
Chun JY, Kim HJ, Hur JW, Jung D, Lee HJ, Pack SP, Lee S, Kim G, Cho CY, Lee SM, Lee H, Choi S, Cheong T, Cho CH
Prediction of Specific Anxiety Symptoms and Virtual Reality Sickness Using In Situ Autonomic Physiological Signals During Virtual
Reality Treatment in Patients With Social Anxiety Disorder: Mixed Methods StudyJMIR Serious Games 2022;10(3):e38284
URL: https://games.jmir.org/2022/3/e38284
doi: 10.2196/38284
PMID:©Joo Young Chun, Hyun-Jin Kim, Ji-Won Hur, Dooyoung Jung, Heon-Jeong Lee, Seung Pil Pack, Sungkil Lee, Gerard Kim,
Chung-Yean Cho, Seung-Moo Lee, Hyeri Lee, Seungmoon Choi, Taesu Cheong, Chul-Hyun Cho. Originally published in JMIR
Serious Games (https://games.jmir.org), 16.09.2022. This is an open-access article distributed under the terms of the Creative
Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, andreproduction in any medium, provided the original work, first published in JMIR Serious Games, is properly cited. The complete
bibliographic information, a link to the original publication on https://games.jmir.org, as well as this copyright and license
information must be included.

https://games.jmir.org/2022/3/e38284

XSL• FO
RenderX

JMIR Serious Games 2022 | vol. 10 | iss. 3 | e38284 | p. 15(page number not for citation purposes)