# Two Experimental Virtual Paradigms for Stress Research: Developing Avatar-Based Approaches for Interpersonal and Evaluative Stressors

Oswald D. Kothgassner, Helmut Hlavacs, Leon Beutl, Lisa M. Glenk, Rupert Palme, Anna Felnhofer

► To cite this version:

Oswald D. Kothgassner, Helmut Hlavacs, Leon Beutl, Lisa M. Glenk, Rupert Palme, et al.. Two Experimental Virtual Paradigms for Stress Research: Developing Avatar-Based Approaches for Interpersonal and Evaluative Stressors. 15th International Conference on Entertainment Computing (ICEC), Sep 2016, Wien, Austria. pp.51-62, 10.1007/978-3-319-46100-7_5 . hal-01640284

HAL Id: hal-01640284

https://inria.hal.science/hal-01640284

Submitted on 20 Nov 2017

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
Two experimental virtual paradigms for stress re-
search: Developing avatar-based approaches for inter-
personal and evaluative stressors

Oswald D. Kothgassner<fnref n="1" />, Helmut Hlavacs<fnref n="2" />, Leon Beutl<fnref n="2" />, Lisa M. Glenk<fnref n="3" />,<fnref n="4" />,
Rupert Palme<fnref n="5" />, Anna Felnhofer<fnref n="6" /> (corresponding author)

**Keywords:** virtual reality, social stress, stress induction, stress research, Cyberball-Paradigm, Trier Social Stress Test (TSST)

**Abstract**

In light of the rather limited ecological validity of paradigms traditionally used for social stress research, the current paper set out to introduce virtual analogues of the Cyberball-Game (Williams, 2007) and the Trier Social Stress Test (Kirschbaum, Pirke, & Hellhammer, 1993). Both were tested in samples of healthy adults using salivary cortisol, self-reported stress and presence as dependent measures. Results indicate a significant rise in cortisol levels and subjective stress; presence, however, was not correlated with stress reactivity. In sum, this study clearly supports the use of virtual environments in stress research as they offer both the internal control and ecological validity needed to generalize findings to real-world settings.

**1. Introduction**

Given its role in the development of various diseases, social stress has been a sub-
ject of extensive research in past decades (c.f., Ruiz et al., 2010). The exposure to a
psychosocial stressor is known to result in an acute physiological stress response,
involving the activation of the Sympatho-Adrenal-Medullary (SAM) axis and the
Hypothalamus-Pituitary-Adrenal (HPA) axis. The SAM-system causes an immediate

<footnote n="1">¹ Department of Child and Adolescence Psychiatry, Medical University of Vienna, Vienna General Hospital, Vienna, Austria</footnote>

<footnote n="2">² Research Group Entertainment Computing, University of Vienna, Austria</footnote>

<footnote n="3">³ Comparative Medicine, Messerli Research Institute, University of Veterinary Medi-
cine Vienna, Medical University of Vienna, University of Vienna, Austria</footnote>

<footnote n="4">⁴ Karl Landsteiner Research Institute for Neurochemistry, Neuropharmacology, Neurorehabilitation and Pain Treatment, Landesklinikum Mauer, Austria</footnote>

<footnote n="5">⁵ Unit of Physiology, Pathophysiology and Experimental Endocrinology, University of Veterinary Medicine Vienna, Austria</footnote>

<footnote n="6">⁶ Department of Pediatrics and Adolescent Medicine, Medical University of Vienna, Austria</footnote>
increase in heart rate, whereas the HPA-axis activation entails a rise in blood and salivary cortisol levels (Kudielka & Kirschbaum, 2005).

Generally, two kinds of social stressors may be distinguished: social-evaluative threats versus interpersonal (rejection) stressors (c.f., Dickerson & Kemeny, 2004). The first usually occurs in a performance context such as public speeches or job-interviews. It involves being potentially exposed to negative evaluations and even failure. Studies show that cortisol reactivity is particularly large and recovery rates are low when confronted with social evaluative threats, especially if they are perceived as uncontrollable (Dickerson & Kemeny, 2004). In everyday situations, however, these performance based social evaluations are not as common. More frequently, individuals may be exposed to (quite subtle) forms of social rejection. These may involve disapproval and criticism by peers or even being ignored by others. It has been shown, that social exclusion is particularly threatening for fundamental social needs like belonging and self-esteem (Williams, 2007). Also, a repeated exposure may lead to cardiovascular diseases (Kemp et al., 2012) and depression (McEwen, 2005).

## 1.1 Experimental Paradigms in Social Stress Research

In light of its pathogenic significance, it is not surprising that in recent years research on social stress has gained in numbers (c.f., Ruiz et al. 2010). Traditionally, paradigms such as arithmetic tasks or public speaking scenarios (Dickerson & Kemeny, 2004) have been used to induce stress in laboratory settings.

One of the most widely applied experimental paradigms for social stress induction is the Cyberball-Game (Williams, 2007). It represents a computer-based ball-tossing game, in which the participant is excluded from the game by two computer-animated players. As such, it is a perfect example of an interpersonal rejection stressor. Past studies have demonstrated the paradigm's effectiveness in evoking a number of stress related responses (e.g., Eisenberger, Lieberman & Williams, 2003; Geniole, Carré & McCormick, 2011; Moor, Crone & Van der Molen, 2010; Wesselmann et al., 2012; Williams, 2007; Zwolinski, 2012). However, the task has also been criticized for its lack of mundane realism (Parsons, 2015). The abstract stimulus bears only minimal resemblance to face-to-face social interactions and is, thus, regarded as limited in its ecological validity.

Another commonly used protocol for social stress induction is the Trier Social Stress Test (TSST; Kirschbaum, Pirke, & Hellhammer, 1993). It requires a participant to prepare a five minute speech which s/he then delivers in front of two to three confederate actors; also, the participant performs a five minute arithmetic task. The TSST constitutes a typical social evaluative threat which is mostly perceived as uncontrollable because of the actors' neutral facial expressions and lack of predictability and which results in significant levels of stress (Kudielka et al., 2004). Yet, the drawback of this method is that it requires the actors to hold their reactions constant in each trial. It is a prerequisite which is difficult to achieve across studies and may, thus, limit inter- and intra-individual comparisons (Ruiz et al., 2010).
## 1.2 Virtual Reality based solutions for stress research

A viable solution for these methodological problems is proposed by immersive virtual environments (IVEs). On the one hand, IVEs may provide a standardization of stimuli by introducing computer-controlled characters which reduce human error and circumstantial variations (Bohil, Alicea, & Biocca, 2011). On the other hand, they allow for a real-time, dynamic interaction which approximates real-life social encounters (Parsons, 2015). As such, IVEs constitute a loophole for the intricate problem of combining ecological validity and experimental rigor.

However, the precondition for IVEs to be considered ecologically valid is that they evoke reactions in users which are comparable to real-life situations. A phenomenon which has extensively been discussed to be responsible for the success of IVEs is the so called sense of presence. Presence may be defined as a sense of being there in the IVE. It is seen as a basis for a concordance between an emotion experienced in an IVE and one experienced in a comparable physical environment (Slater, 2003). Yet, the debate regarding the nature of the relationship between presence and emotions is still ongoing (e.g. Diemer et al., 2015).

For both paradigms (Cyberball and TSST), attempts have been made to develop virtual counterparts. So far, three studies introduce a virtual Cyberball-Game and present preliminary results on its usefulness (i.e., Kassner et al., 2012; Kothgassner et al., 2014; Segovia & Bailenson, 2012). However, Kassner et al. (2012) did not use physiological markers to assess stress reactivity, and Kothgassner et al. (2014) only assessed the SAM axis reactivity. Given that cortisol is particularly sensitive to social stressors (Dickerson & Kemeny, 2004), the generalizability of both studies is limited and further empirical support is needed.

Similarly, virtual TSST versions have been introduced by different research groups (e.g. Delahaye et al., 2015; Fich et al., 2014; Kelly et al., 2007; Jönsson et al., 2010; Montero-López et al., 2015; Ruiz et al., 2010). However, the virtual TSSTs used in these studies differ with regards to content. While Fich et al. (2014) and Jönsson et al. (2010) used three virtual interviewers each, other authors used five (Kelly et al., 2007) or even 80 avatars (Delahaye et al., 2015) instead of the formerly proposed two to three interviewers. Also, these studies tend to alter the experimental setup by including an audience that is ‘restless’ rather than neutral (e.g. Ruiz et al., 2010). Hence, it is possible, that these alterations confounded the observed stress reactions.

## 1.3 Objective

In sum, social stress research is in need of ecologically valid stress induction scenarios. As traditional paradigms suffer from a number of disadvantages, IVE based counterparts are increasingly considered as valuable alternatives. This study set out to create virtual analogs for the two most commonly used and most well researched social stress paradigms, the Cyberball-Game and the TSST protocol. The objective was to provide support for their effectiveness using a multimodal assessment of stress responses (including subjective and objective markers). Also, by covering two complementary social stressors (social-evaluative threats vs. rejection stressors), more far-
fetched conclusions about social stress reactivity in IVEs as well as generalizations to
other paradigms were supported.

**2. Methods**

Two experimental psychological studies were conducted to explore the virtual par-
adigms' effectiveness in evoking social stress. Study 1 evaluated a virtual Cyberball-
Game and Study 2 involved a virtual TSST. In accordance with the Declaration of
Helsinki, subjects signed an informed consent and received course credits.

**2.1 Study 1 (Cyberball)**

**Participants.** Eight female (4 each group) German speaking undergraduates with an age ranging from 22 to 25 years participated in the current study.

**Measures.** To assess physiological stress reactions, salivary cortisol samples were taken via cotton swabs (Salivette®, Sarstedt, Wiener Neudorf, Austria). Participants were instructed to put the swabs into their cheek pouch and wait for 80 seconds to ensure sufficient saturation of the cotton with saliva. Prior to further processing, the samples were stored at -20°C. Sample analyses were centrifuged at room temperature (at 3000g for 15 min) and then analyzed according to a sensitive cortisol enzyme immunoassay (see Palme & Möstl, 1997). A total of 10 µl of a 1:10 clear saliva dilution were used and all samples were assayed in duplicates. To assess the virtual experience, a single item from the iGroup Presence Questionnaire (IPQ, Schubert, Friedmann, & Regenbrecht, 2001) was used (“In the computer generated world I had a sense of ‘being there’). Additionally, a single question (“Are you stressed?”) served as an indicator for the subjective stress response. Another single item assessed the participants’ perception of exclusion during the study (“Do you feel rejected and excluded?”) on a visual analogue scale (VAS, 10 cm length).

**Procedure.** Upon their arrival at the VR-lab, participants were randomly assigned to one of two conditions (1) inclusion vs. (2) exclusion. They were asked to fill out a demographic screening and the VAS stress question before being donned the head-mounted-display (Sony HMZ-T1, Sony, Japan) with an external head-tracking unit (TrackIR 5, NaturalPoint, US). A wireless control unit (F710, Logitech, Switzerland) allowed tossing the ball. The left key had to be pressed to toss the ball to the left player and vice versa. Participants in the exclusion condition only received 4 ball tosses (duration: 45-60s) before being excluded from the game with no further tosses until the end of the 5-minute interval. The remaining participants received a pre-programmed amount of ball tosses (30 %). Afterwards, participants were asked to fill out the remaining psychometric measures (IPQ and VAS stress question).

Cortisol samples were collected at 4 time points: (1) 20 min before the IVE exposure, (2) immediately before exposure, (3) 20 mins and (4) 35 min post exposure. The overall procedure lasted approx. 80 min; all experiments were conducted in the afternoon (1:30-3:30 p.m.) to account for diurnal variation in cortisol release.
## 2.2 Study 2 (TSST)

**Participants.** Five German speaking students (3 females, 2 males) with an age ranging from 24 to 26 years were assessed via the VR-TSST.

**Measures.** Salivary cortisol samples also served as an indicator for social stress in this experiment. The extraction method as well as subsequent analyses did not differ from Study 1; the time points for sample collection are described below. Again, the IPQ item “sense of being there” (Schubert, Friedmann, & Regenbrecht, 2001) and the VAS stress question assessed the subjective experience.

**Procedure.** Prior to exposure, participants filled out the pre-experimental questionnaires (demographics, VAS stress). Subsequently, they were informed about the job interview and a preparation time of 5 min was provided. Analogous to study 1, the same HMD and head tracking unit were used. At the start of the simulation, the two virtual interviewers (1 female, 1 male) entered the room, sat down and asked the participant why s/he was the best candidate for the job. Like in the original TSST, participants passed a job-interview in the first half of the task (5 min), for the remaining 5 minutes, they conducted an arithmetic task (subtract 13 from 1022). The avatars retained neutral expressions throughout the interview; their answers included instructions such as “This is incorrect. Please start over from 1022.” After the procedure, participants answered the IPQ item and the single stress-question; 20 min post procedure, subjective stress was measured again. Cortisol samples were collected at 6 time points: (1) 20 min prior to stressor onset, (2) 10 min, (3) 20 min, (4) 25 min, (5) 40 min and (6) 60 min post exposure. The overall procedure lasted 80 min.

## 2.3 Technical implementation

The applications use the open source render engine Ogre3D for real-time rendering of the scenes. The source code is written in C++ using the express version of Visual Studio. OpenAL, the well-known audio library was used for playing the sounds. All objects and animations in the scenes were created using Blender3D.

### Options Cyberball

1. **Mood:** The park can be rendered in 5 different ‘moods’: (1) happy, (2) anger, (3) boredom, (4) sadness, (5) fear. The moods are masked by numbers to prevent the users from seeing the chosen option. Happy will render the park with a clear sky and pleasant surroundings. Anger is the same as happy, but there will be loud construction sound playing and construction vehicles will be visible. Sadness renders the park with a dark cloudy sky and rain. Boredom renders a concrete floor surrounded by concrete walls. Lastly, fear renders the park scenario at night. For this study, the default setting ‘happy’ with a sunny park was used.

2. **People:** If this option is selected, other virtual characters (i.e., agents) will walk around the park, yet interacting with them is not possible for the user.

3. **Ballplayers:** This option is the core of the Cyberball-scenario (figure 1). It will render two additional virtual humans (1 male, 1 female) which will engage the user in a short football game. In the ‘inclusion condition’ the players keep playing
<figure><img src="image_2.png" /><figcaption>Figure 1. The virtual Cyberball-Paradigm (left) and virtual TSST (right)</figcaption></figure>

with the user, in the ‘exclusion condition’, however, the avatars continue to play only
with each other and further exclude the user from the game.

Options TSST

1. **Interviewer:** This setting allows specification of the interviewers' gender.

2. **Speaker:** This option specifies the interviewer who does the talking.

3. **Emotions:** Sets the interviewers' facial expressions to the emotion they are supposed to express. By default neutral is selected, other options are happy or hostile.

4. **Ally:** If this option is selected, another person (an ally) will participate in the interview. Again, the gender can be chosen. The ally is seated directly next to the user and s/he will answer the questions if the user fails to do so.

5. **Helper:** This option will make the ally more supportive, encouraging the user with some predefined phrases (i.e., "You are doing great").

**3. Results**

The results for the stress reactivity in both studies are presented in figure 2.

**3.1 Study 1 (Cyberball)**

The repeated measures ANOVA showed a significant increase of salivary cortisol concentration (F(18,3)=6.262; p<0.004), especially in the excluded group (F(18,3)=4.069; p=0.023). There is a distinct interaction effect, displaying that excluded participants showed significant increases in saliva cortisol (+51 % to baseline), whereas included participants had a lower increase after the Cyberball game (+22 % to baseline). Moreover, there is a significant increase in self-reported stress levels (F(5,2)=12.298; p<0.001), but only excluded participants rated the Cyberball game as stressful (F(5,2)=10.821; p=0.002) and stated, that they felt excluded (F(8,1)=28.612; p=0.002). Table 1 shows a null finding regarding the relationship between the sense of being there and cortisol stress reactivity as well as the self-reported measure after social exclusion and inclusion.
<figure><img src="image_3.png" /><figcaption>Figure 2. Saliva cortisol and self-reported stress response.</figcaption></figure>

## 3.2 Study 2 (TSST)

A repeated measures ANOVA with a subsequent Greenhous-Geisser correction depicts a significant stress response in salivary cortisol (F(7.345,1.836)=5.247; p=0.040). Both the speech and math task seem to evoke a stress response in salivary cortisol concentration of 68 % for the speech task and 83 % for the math task respectively compared to the baseline sample measure. Furthermore, participants had significant stress ratings post exposure (F(4,2)=7.051; p=0.017; η²=0.23). Again, table 1 shows no correlation between presence and cortisol or the subjective stress response.

## 4. Discussion

In light of the rather limited ecological validity of paradigms traditionally used for social stress induction, the current paper set out to introduce virtual analogues of two popular, well researched paradigms, the Cyberball-Game (Williams, 2007) and the
<table><caption>Table 1. Correlations between sense of being there and stress measures*</caption><thead><tr><th rowspan="2"></th><th rowspan="2"></th><th colspan="3">VR-TSST</th><th colspan="3">VR-Cyberball</th></tr><tr><th>1</th><th>2</th><th>3</th><th>1</th><th>2</th><th>3</th></tr></thead><tbody><tr><td>1</td><td>Self-report stress#</td><td>-</td><td>0.518</td><td>-0.263</td><td>-</td><td>0.283</td><td>0.131</td></tr><tr><td>2</td><td>Salivary cortisol#</td><td></td><td>-</td><td>-0.099</td><td></td><td>-</td><td>0.575</td></tr><tr><td>3</td><td>Sense of being there</td><td></td><td></td><td>-</td><td></td><td></td><td>-</td></tr></tbody></table>

\# Measured during stressor; self-report stress is measured by the single-item question immediately after the stressor (VAS); salivary cortisol values point out mean salivary cortisol levels during speech and math task or the saliva sample immediately after the Cyberball stressor

Trier Social Stress Test (TSST, Kirschbaum, Pirke, & Hellhammer, 1993). The present findings suggest that both the virtual Cyberball and the virtual TSST may be used as effective tools for social stress induction. All TSST-participants as well as those excluded from the Cyberball-Game showed significant increases in salivary cortisol during exposure to the stressor and reported to feel stressed. These observations are largely in line with prior research (e.g., Jönsson et al., 2010; Ruiz et al., 2010), and the current effect sizes are comparable to those achieved with the original tasks (Beekman, Stock, & Marcus, 2015; Kudielka et al., 2004).

The findings on the virtual Cyberball-task particularly constitute a valuable addition to the body of data on cortisol responses to social exclusion. To date, there is no consensus on whether the experience of social exclusion under laboratory settings results in an increase in cortisol (c.f. Dickerson, & Zoccola, 2013). Prior studies using the original Cyberball-Game (i.e., Beekman, Stock, & Marcus, 2015; Zöller et al., 2010) produced conflicting findings. A more ecologically valid virtual scenario, however, may lead to different results than the minimalistic original. Thus, the present study offers first evidence that virtual social threats – similar to real life-threats – may result in a considerable HPA axis activity. Certainly, a drawback of this study is the fact that possible gender differences in stress reactivity could not be considered as the sample consisted only of women. In order to substantiate the claim that cortisol increases are larger in women when confronted with social rejection, whereas men react more to achievement stressors (Stroud, Salovey, & Epel, 2002), future VR research is challenged to balance participants' gender.

Another issue that was not considered in the current paper was the question of whether participants believed to interact with an avatar (i.e., a human-controlled character) or an agent (i.e., a computer algorithm). The Threshold Model of Social Influence (Blascovich et al., 2002) assumes that the degree of agency attributed to the virtual other influences the interaction. Other authors, however, have succeeded in demonstrating that it does not matter whether the Cyberball-players are controlled by a human or a computer (e.g., Kothgassner et al., 2014). In contrast to the virtual Cyberball and to the best of our knowledge, agency was not yet evaluated in studies on virtual TSST versions. Hence, additional research is needed to further substantiate the claim that there is no difference in stress responses to avatars or agents for both virtual tasks.

<footnote>* No significance within an Alpha of 5 %</footnote>
A further concern is the validity of the findings. Generally, IVE stress induction paradigms have been found to produce medium to large effect sizes, approximating those of real-life-scenarios (c.f., Parsons, 2015). Even though the present results are in line with those achieved with the original TSST (e.g., Kudielka et al., 2004), a direct comparison of the virtual and real TSST would be the method of choice to establish validity. Also, comparing the virtual Cyberball to a face-to-face version would allow for a more thorough examination of its ecological validity. A first attempt (Kothgassner et al., 2014) has shown that virtual exclusion may result in similar stress responses (heart rate) as real exclusion, but results on cortisol reactivity are still missing.

Finally, presence was not correlated with the salivary cortisol response or subjective stress in the current study. This result corresponds with a virtual TSST study (Ruiz et al., 2010) which found no connection between the experience of presence and cortisol. Similarly, another study (Felnhofer, Kothgassner et al., 2014) found no correlation between presence, subjective anxiety and heart rate. Hence, rather than directly increasing the intensity of the emotional reaction, presence seems to serve as a precondition for an emotion or stress reaction to occur at all in IVEs. This study supports these findings and extends them to yet another indicator of stress –salivary cortisol. Future research should, however, complement questionnaire measures of presence with a multimodal approach (i.e., using behavioral markers) to shed more light on this issue. Also, the assessment of presence could be supplemented by another related yet distinct construct which could proof to be useful especially in avatar based IVEs: social presence (c.f., Biocca, Harms, & Gregg, 2001).

## 4.1 Limitations and Conclusion

The present results once again strongly encourage the use of IVEs for diverse research purposes. Apart from traditional laboratory-based stress research, the introduced virtual counterparts may well be applied especially in fMRI studies. While being scanned, participants are usually confined to an enclosed space with little to no interaction. By creating Internet based shared IVEs, within which participants located kilometers apart may interact with each other, scientist may study neural responses simultaneously and under more ecologically valid circumstances (Bohil, Alicea, & Biocca, 2011). For instance, the Cyberball-Game could consist of three participants actually playing with each other, and the TSST could make use of an extra function – an ally who sits right next to the interviewee and supports him/her.

Everyday social encounters involve a broad array of cues such as verbal and non-verbal reactions, joint attention, theory of mind as well as representation of self and others (Parsons, 2015). Even though IVEs have come a long way to approximating real-life settings, they are still challenged to convincingly implement mechanisms of social interactions. In this study, basic avatar models were used with minimal facial and behavioral variability; spontaneous reactions were limited and the user was not able to freely engage the avatars. Yet, still, our participants responded with considerable stress to the virtual character. The Media Equation Concept (Nass & Moon, 2000) may provide an explanation as to why image fidelity is not a key issue for a social response to computer generated images of others. This theory suggests that humans
are evolutionarily prepared to react socially to even very minimal social cues (e.g. the impression of a face). This is alleviation for IVE programmers, as even low quality IVEs may trigger a substantial emotional and behavioral reaction in users as long as they are present. A study (Kothgassner, Felnhofer et al., 2016) assessing heart rate and cortisol responses to a public speech task in three groups (real audience, virtual audience, controls) further supports this assumption: it shows that stress responses in IVEs may solely be attributed to the confrontation with the social stimulus and not to the VR apparatus or other factors. Hence, the paradigms introduced here may also be used for therapeutic purposes, e.g. learning how to control one's stress level when confronted with social cues which are unpredictable and uncontrollable.

In sum, this study provided strong support for the implementation of virtual paradigms for social stress induction. The virtual analogues may on the one hand overcome the lack of ecological validity stemming from the minimalistic and abstract material; on the other hand, they may ensure an appropriate degree of control in order to enhance internal validity by providing interactions with no inter-subject variations.

## Acknowledgements

We would like to thank Anna-Katherina Heinzle, Ellen Gerlach, Farina Flick and Johanna Rudyk for their support in the acquisition of parts of the data.

## References

Beekman, J. B., Stock, M. L., & Marcus, T. (2016). Need to belong, not rejection sensitivity, moderates cortisol response, self-reported stress, and negative affect following social exclusion. The Journal of social psychology, 156(2), 131-138.

Biocca, F., Harms, C., & Gregg, J. (2001). The networked minds measure of social presence: Pilot test of the factor structure and concurrent validity. In 4th annual International Workshop on Presence, Philadelphia, PA.

Blascovich, J., Loomis, J., Beall, A. C., Swinth, K. R., Hoyt, C. L., & Bailenson, J. N. (2002). Immersive virtual environment technology as a methodological tool for social psychology. Psychological Inquiry, 13(2), 103-124.

Bohil, C. J., Alicea, B., & Biocca, F. A. (2011). Virtual reality in neuroscience research and therapy. Nature reviews neuroscience, 12(12), 752-762.

Delahaye, M., Lemoine, P., Cartwright, S., Deuring, G., Beck, J., Pflueger, M., Graf, M. & Hachtel, H. (2015). Learning aptitude, spatial orientation and cognitive flexibility tested in a virtual labyrinth after virtual stress induction. BMC psychology, 3(1), 22.

Dickerson, S. S., & Kemeny, M. E. (2004). Acute stressors and cortisol responses: a theoretical integration and synthesis of laboratory research. Psychological bulletin, 130(3), 355-389.

Dickerson, S. S., & Zoccola, P. M. (2013). Cortisol responses to social exclusion In C. N. DeWall (Ed.), Oxford handbook of social exclusion. (pp. 143–151). Oxford, UK: Oxford University Press.
Diemer, J., Alpers, G. W., Peperkorn, H. M., Shiban, Y., & Mühlberger, A. (2015). The impact of perception and presence on emotional reactions: a review of research in virtual reality. *Frontiers in psychology, 6*(26).

Eisenberger, N.I., Lieberman, M.D., Williams, K.D. (2003). Does rejection hurt? An FMRI study of social exclusion. *Science* 302, 290-292.

Felnhofer, A., Kothgassner, O. D., Hetterle, T., Beutl, L., Hlavacs, H., & Kryspin-Exner, I. (2014). Afraid to be there? Evaluating the relation between presence, self-reported anxiety, and heart rate in a virtual public speaking task. *Cyberpsychology, Behavior, and Social Networking*, 17(5), 310-316.

Fich, L. B., Jönsson, P., Kirkegaard, P. H., Wallergård, M., Garde, A. H., & Hansen, Å. (2014). Can architectural design alter the physiological reaction to psychosocial stress? A virtual TSST experiment. *Physiology & behavior*, 135, 91-97.

Geniole, S. N., Carré, J. M., & McCormick, C. M. (2011). State, not trait, neuroendocrine function predicts costly reactive aggression in men after social exclusion and inclusion. *Biological Psychology*, 87(1), 137-145.

Jönsson, P., Wallergård, M., Österberg, K., Hansen, Å. M., Johansson, G., & Karlson, B. (2010). Cardiovascular and cortisol reactivity and habituation to a virtual reality version of the Trier Social Stress Test: A pilot study. *Psychoneuroendocrinology*, 35(9), 1397-1403.

Kassner, M. P., Wesselmann, E. D., Law, A. T., & Williams, K. D. (2012). Virtually Ostracized: Studying Ostracism in Immersive Virtual Environments. *Cyberpsychology, Behavior, and Social Networking*, 15(8), 399-403.

Kelly, O., Matheson, K., Martinez, A., Merali, Z., & Anisman, H. (2007). Psychosocial stress evoked by a virtual audience: relation to neuroendocrine activity. *CyberPsychology & Behavior*, 10(5), 655-662.

Kemp, A. H., Quintana, D. S., Felmingham, K. L., Matthews, S., & Jelinek, H. F. (2012). Depression, comorbid anxiety disorders, and heart rate variability in physically healthy, unmedicated patients: implications for cardiovascular risk. *PloS one*, 7(2), e30777.

Kirschbaum, C., Pirke, K. M., & Hellhammer, D. H. (1993). The ‘Trier Social Stress Test’–a tool for investigating psychobiological stress responses in a laboratory setting. *Neuropsychobiology*, 28(1-2), 76-81.

Kothgassner, O.D.*, Felnhofer, A.*, Hlavacs, H., Beutl, L., Palme, R., Kryspin-Exner, I. & Glenk, L. M. (2016). Salivary cortisol and cardiovascular reactivity to a public speaking task in a virtual and real-life environment. *Computers in Human Behavior*, 62, 124-135.

Kothgassner, O.D., Kafka, J., Rudyk, J. Beutl, L., Hlavacs, H. & Felnhofer, A. (2014). Does social exclusion hurt virtually like it hurts in real life? The role of agency and social presence in the perception and experience of social exclusion. *Challenging Presence. Proceedings of the International Society for Presence Research*, 45-56.

Kudielka, B. M., & Kirschbaum, C. (2005). Sex differences in HPA axis responses to stress: a review. *Biological psychology*, 69(1), 113-132.

Kudielka, B. M., Buske-Kirschbaum, A., Hellhammer, D. H., & Kirschbaum, C. (2004). HPA axis responses to laboratory psychosocial stress in healthy elderly
adults, younger adults, and children: impact of age and gender. *Psychoneuroendocrinology*, 29(1), 83-98.

Kudielka, B. M., Schommer, N. C., Hellhammer, D. H., & Kirschbaum, C. (2004). Acute HPA axis responses, heart rate, and mood changes to psychosocial stress (TSST) in humans at different times of day. *Psychoneuroendocrinology*, 29(8), 983-992.

McEwen, B. S. (2005). Glucocorticoids, depression, and mood disorders: structural remodeling in the brain. *Metabolism*, 54(5), 20-23.

Montero-López, E., Santos-Ruiz, A., García-Ríos, M. C., Rodríguez-Blázquez, R., Pérez-García, M., & Peralta-Ramírez, M. I. (2015). A virtual reality approach to the Trier Social Stress Test: Contrasting two distinct protocols. *Behavior research methods*, 1-10.

Moor, B. G., Crone, E. A., & van der Molen, M. W. (2010). The Heartbrake of Social Rejection Heart Rate Deceleration in Response to Unexpected Peer Rejection. *Psychological Science*, 21(9), 1326-1333.

Nass, C., & Moon, Y. (2000). Machines and mindlessness: Social responses to computers. *Journal of Social Issues*, 56(1), 81-103.

Palme, R., & Möstl, E. (1997). Measurement of cortisol metabolites in feces of sheep as a parameter of cortisol concentration in blood. *International Journal of Mammal Biology*, 62(Suppl. 2), 192-197.

Parsons, T. D. (2015). Virtual Reality for Enhanced Ecological Validity and Experimental Control in the Clinical, Affective and Social Neurosciences. *Frontiers in human neuroscience*, 9.

Ruiz, A. S., Peralta-Ramirez, M. I., Garcia-Rios, M. C., Muñoz, M. A., Navarrete-Navarrete, N., & Blazquez-Ortiz, A. (2010). Adaptation of The Trier Social Stress Test To Virtual Reality: psycho-physiological and neuroendocrine modulation. *Journal of CyberTherapy & Rehabilitation*, 3(4), 405-415

Schubert, T., Friedmann, F., & Regenbrecht, H. (2001). The experience of presence: Factor analytic insights. *Presence*, 10(3), 266-281.

Segovia, K. Y., & Bailenson, J. N. (2012). Virtual imposters: Responses to avatars that do not look like their controllers. *Social Influence*, 7(4), 285-303.

Slater, M. (2003). A note on presence terminology. *Presence connect*, 3(3), 1-5.

Stroud, L. R., Salovey, P., & Epel, E. S. (2002). Sex differences in stress responses: social rejection versus achievement stress. *Biological psychiatry*, 52(4), 318-327.

Wesselmann, E. D., Wirth, J. H., Mroczek, D. K., & Williams, K. D. (2012). Dial a feeling: Detecting moderation of affect decline during ostracism. *Personality and individual differences*, 53(5), 580-586

Williams, K.D. (2007). Ostracism. *Annual Review of Psychology* 58, 425-452.

Zöller, C., Maroof, P., Weik, U., & Deinzer, R. (2010). No effect of social exclusion on salivary cortisol secretion in women in a randomized controlled study. *Psychoneuroendocrinology*, 35(9), 1294-1298.

Zwolinski, J. (2012). Psychological and neuroendocrine reactivity to ostracism. *Aggressive Behavior* 38, 108-125.