# Workshop: Challenges for XR in Digital Entertainment

Esteban Clua, Thiago Porcino, Daniela Trevisan, Jorge Cardoso, Thallys Lisboa, Victor Peres, Victor Ferrari, Bruno Marques, Lucas Barbosa, Eder Oliveira

<figure><img src="image_1.png" /></figure>

## To cite this version:

Esteban Clua, Thiago Porcino, Daniela Trevisan, Jorge Cardoso, Thallys Lisboa, et al. Workshop: Challenges for XR in Digital Entertainment. 20th International Conference on Entertainment Computing (ICEC), Nov 2021, Coimbra, Portugal. pp.489-498, 10.1007/978-3-030-89394-1_43. hal-04144356

HAL Id: hal-04144356

https://inria.hal.science/hal-04144356

Submitted on 28 Jun 2023

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_3.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Workshop: Challenges for XR in Digital Entertainment

Esteban Clua<fnref n="1" />, Thiago Porcino<fnref n="1" />,<fnref n="2" />, Daniela Trevisan<fnref n="1" />, Jorge C. S. Cardoso<fnref n="4" />, Thallys Lisboa<fnref n="1" />, Victor Peres<fnref n="1" />, Victor Ferrari<fnref n="1" />, Bruno Marques<fnref n="3" />, Lucas Barbosa<fnref n="1" />, and Eder Oliveira<fnref n="1" />

**Abstract.** Extended Reality as a consolidated game platform was always a dream for both final consumers and game producers. If for one side this technology had enchanted and called the attention due its possibilities, for other side many challenges and difficulties had delayed its proliferation and massification. This workshop intends to rise and discuss aspects and considerations related to these challenges and solutions. We try to bring the most relevant research topics and try to guess how XR games should look in the near future. We divide the challenges into 7 topics: Cybersickness, User Experience, Displays, Rendering, Movements, Body Tracking and External World Information.

**Keywords:** Extended Reality · Virtual Reality · Digital Entertainment · Head-mounted Displays · UX

## 1 Introduction

Extended Reality (XR) platform can be considered as an increment of Virtual Reality in relation to immersion and interaction aspects. While VR platforms are mostly dedicated to visual issues, XR includes more external elements and senses, such as movements, tactile, haptics and the usage of the real environment as the application stage [29]. According to the Milgram Continuum, the virtual immersion is a result that comes not only from accurate visual aspects, but mostly from a precise combination of all human senses, orchestrated in such a way that all of them enhances each other. While many progresses had been achieved in graphics, audio, tracking and interfaces issues, there are still many remaining challenges, mostly related to a correct combination on adaption to recent XR hardware devices.

In this workshop we propose a division of areas for these challenges. We believe that for a real consolidation for entertainment within this platform it is necessary to have robust solutions in each field. We divide the challenges into 7 topics: Cybersickness (CS), user experience and design guidelines, Display

<footnote n="1">¹ Universidade Federal Fluminense, Niterói, Brazil</footnote>

<footnote n="2">² Dalhousie University, Nova Scotia, Canada</footnote>

<footnote n="3">³ Universidade Federal do ABC, São Paulo, Brazil</footnote>

<footnote n="4">⁴ University of Coimbra, CISUC, DEI, Coimbra, Portugal
esteban@ic.uff.br</footnote>
and Fovea, Image quality and rendering, movements and redirect walking, body tracking and finally External world information and acquisition.

## 2 Cybersickness

Motion sickness (MS) is defined as the discomfort felt during a forced visual movement (without body movement), which typically happens in airplane trips, boats, or land vehicles. Such discomfort is also experienced in virtual environments and is called VIMS (Visually Induced Motion Sickness). MS can be split into two subcategories [16]: transportation sickness, which is tied to the real world and simulator sickness, which is associated to the virtual world and includes CS, as shown in Figure 1.

<figure><img src="image_4.png" /><figcaption>**Fig. 1.** Motion sickness and its subcategories according to environments and trigger mechanisms.</figcaption></figure>

XR environments that use head-mounted displays (HMDs) are strongly related to common indications of discomfort [18]. Among the potential causes, CS deserves special attention as it is the most common and is usually associated to long exposures to HMDs. Additionally, more than 60% of HMDs usability problems are considerably related to discomfort [18]. The most persistent symptoms caused by CS are general discomfort, headache, stomach awareness, nausea, vomiting, sweating, fatigue, drowsiness, disorientation, and apathy [7]. These symptoms influence the user experience and impact the profit and coverage of XR game manufacturing. In addition, discomfort symptoms can vary over people, where some individuals are more susceptible than others. Several studies have been conducted using deep learning models to predict and mitigate CS, such as
convolutional neural network (CNNs) and recurrent neural networks (RNNs)[12, 17]. Although deep learning classifiers are the most suitable approach for CS prediction, deep neural networks are black boxes that are very difficult to grasp. In contrast, a recent approach apply techniques to make deep learning models explainable [31]. However, the literature is still not strongly affirmed. Furthermore, symbolic machine learning algorithms enable a straight understanding of decision paths [23]. Another critical problem in CS researching is associated with data labeling. In general, researchers collect verbal, haptic, or brain signal feedback to construct the ground truth of sickness. While verbal feedback is highly subjective and different from each participant, collecting haptic feedback when participants are under discomfort can often be corrupted by the delay associated with participant feedback. A straightforward challenge is related to gender differences tied to XR tasks. Some works [9, 6] pointed out that specific tasks can produce different results of CS for different user-profiles and groups. Overcoming these issues will help designers to produce better XR content and improves the user experience and retain users for longer XR exposures.

## 3 User Experience

Since XR is a new technology and there are many people experiencing it for the first time, it is important that XR designers make their experiences as intuitive and memorable as possible. The game UX accounts for the whole experience players have with a game, from first hearing about it to navigating menus and progressing in the game. The question is: How to make a better game user experience (UX)? Celia Hodent [11] says UX is about understanding the gamer's brain: understanding human capabilities and limitations to anticipate how a game will be perceived, the emotions it will elicit, how players will interact with it, and how engaging the experience will be. As Celia said we believe that UX and cognitive and behavioral psychology can provide very concrete and easy-to-use guidelines to anticipate and even solve design problems. Additionally, techniques from HCI domain such as users interviews, surveys, usability heuristics, analysis of physiological signals, wizardOz [21], among others, need to be properly studied and applied to understand and evaluate the whole UX in the XR context of usage.

## 4 Displays, Foveated Rendering

Computational needs for game rendering in XR systems have grown faster. The advent of wider Field of View and higher resolutions HMDs have amplified shading complexities [2]. These features bring a computing power bottleneck, requiring some sort of optimization for keeping target frame rates. Knowing that the human eye has a non regular distribution of cones and rods, some studies have suggested to create non regular pixel distribution (Foveated Rendering). While it is already being explored by rendering engines, there still many challenges for optimizing and customizing it. Foveated Rendering technique exploits human visual system to render the best resolution possible only where the user looks,
as in Figure 2. According to Swafford et al. [27], since the human eye perceives
more detail at the center of vision, this uniformity in resolution, regardless of
user focus, is a waste of valuable resources such as computing power.

<figure><img src="image_5.png" /><figcaption>Fig. 2. Foveated Rendering Simulation. Source: [1]</figcaption></figure>

Besides finding a correct balance for the foveated areas, there are still many
challenges, such as understanding how this impacts human perception, color
distortions and dynamic factors according with the game scene (games with
constant colors in large areas naturally requires less pixels to be rendered and
enhances the foveated optimizations). It is also important to create robust fac-
tors for measuring and better calibrating rendering parameters. Finally, we be-
lieve that this concept can also be transposed for different refresh rates for each
foveated area, taking into consideration that rods are more dense at the periph-
eral human vision area.

5 Image Quality and Rendering

Traditional rasterized rendering algorithms are limited for achieving photo-realism, which is many situation is desirable for a high immersive environment. Path tracing achieves a higher degree of realism due to the global illumination effects, with a Monte Carlo integration method. Still, it is computationally costly to render a 3D scene at an HMD's resolution. In 2018, with the RTX architecture, access to GPUs capable of optimizing the intersection calculation of a ray with a polygon and thus accelerating realistic rendering became available. With this, some areas of research have been reignited. One of them is the use and optimization of path tracing or hybrid rendering (with rasterization and path tracing) algorithms for
<figure><img src="image_6.png" /><figcaption>**Fig. 3.** Path traced scene rendered in dual screens using foveation, before denoiser.</figcaption></figure>

virtual reality devices such as HMDs, although they remain changeable. By using the properties of vision such as the concentration of cone distribution in the fovea and devices that allow tracking of the user's gaze, we can avoid rendering parts of the screen with such sharp details or rendering at a reduced spatial sampling frequency. Previous studies have experimented with users to define what an optimal distribution would be, with probabilistic selection of which pixels will be selected by rays, thus decreasing the amount of traced rays and optimizing the algorithm [30]. Similarly, other studies use a fixed texture for ray selection.

Not least, reconstruction algorithms called denoisers, which are already commonly used in path tracing rendering, are even more relevant with the fovea distributions, as in Figure 3. The few works that have made approaches in this regard try to adapt reconstruction to be compatible with log-polar space rendering as well.

# 6 Movements and Redirect Walking

Providing the user the capacity to explore a virtual environment is one way to enhance and create a sense of belonging. Therefore, to create a satisfactory experience, there is a necessity to let users interact with their surroundings. Moving is another form people unconsciously and continuously interact with their surroundings, and developers invented several techniques to move in XR. The problem with this kind of interaction is that while developers can create infinite worlds, users play in confined spaces. Hence, the locomotion, made in real life, can't be mimicked in the virtual world. The literature indicates a growing search to solve this issue [3].
Artificial Movement is one example of a solution. It consists of moving the character with a button or joystick and allows the user to freely roam the virtual area. However, it creates severe cybersickness because of the divergence between the seen and felt movement. Another known solution is teleportation. With a self-explanatory name, this technique transfers the player's position to a desired new one. When developers use this solution, they avoid problems with the cybersickness, but it breaks the immersion since people can't do this in real life. With these two examples, we can set the scene and understand what researchers struggle with within this field. One solution that has the purpose of creating, in the user, the feeling of mimicking his movement by misleading his senses is Redirect Walking[14], which is based on tricking the user's perception and make him feel that is walking forward, but in reality, he is walking in a curved path. The main problem with this solution is how to shift the virtual environment without triggering the user's perception, which can cause cybersickness and break the immersion [5]. Instead of only divert the player's movement or turn the whole scenery, researchers are using devices, tools, and methods to improve Redirect Walking. Methods such as pointed by [26] recognizes when is happening the saccade movement of the eye and shifts the scene at the same time. Redirect Walking is the least used method of movement in VR applications mainly because of the necessity of bigger spaces to fully reach its potential of making the user unperceptive by the reorientation of his movement [5]. Matsumoto et al. [20] measured that it needs a circular arc of 22m to avoid perception, but [24] managed to do constrain the movements to an area of 6m x 6m. Even though there is an evolution in this topic, it is needed to develop methods that can use smaller spaces, and it is computed fully in the HMD so this method can become more universally available.

# 7 Body Tracking

XR games must provide users with an immersive experience with a sense of presence and satisfying natural interaction. Body tracking allows reconstruction of the body movement needed to achieve a satisfying natural interaction [15], especially in multiplayer games [13], enabling users to observe other players' movements. A virtual body is crucial for a good level of immersion, and when the user identifies himself with this virtual body, we can see the feeling of presence [25]. Although there are many important works related to the subject [4], most are related for showing only floating hands or VR controllers, due to the lack of movement data. In the application domain, XR in Games, vision-based body tracking remains a challenge because of the sudden change in object motion, cluttered background, partial occlusion, and camera motion [8]. The hands are the most used body parts in XR in Games, as they provide a robust form of interaction. Vision-based hand-tracking is an object of interest of several researchers. Most work on hands-tracking focuses on the use of depth cameras [28] or RGB [21]. Depth-based approaches present results that are superior to RGB-based approaches. A depth camera provides hand geometry in terms of a
2.5D point cloud, and the model-based approaches can reliably fit a hand mesh to the reconstructed point cloud [28]. Using hand tracking input with mobile technology is a problem mainly due to the high energy consumption. Han et al. [10] present a real-time tracking system that uses four egocentric monochrome fisheye cameras to produce 3D hand pose estimates and run not only on PC but also on mobile processors (Fig. 4). The system presents failures in hand-hand and hand-object interactions showing that grasping objects and training data generation are still open issues in mobile hand tracking interactions.

<figure><img src="image_7.png" /><figcaption>**Fig. 4.** A real-time hand-tracking system using four monochrome cameras mounted on a VR headset. Source: [10]</figcaption></figure>

# 8 External World Information

Acquiring and processing the external world information is an essential and challenging aspect of XR applications. The real-world data acquisition for XR applications comes from different sources such as motion sensors, cameras, depth sensors, and other hardware. Aggregating and incorporating this data into meaningful information for XR games is not a trivial task.

Considering the visual features of XR applications, one crucial aspect is the consistent appearance between virtual and real-world objects. One of the main characteristics that drive the consistent appearance is the lighting between virtual and real-world objects. One possible approach to solve this problem is the
<figure><img src="image_8.png" /><figcaption>**Fig. 5.** Left: rendering of XR scene with inconsistent lighting. Right: relighting of XR scene with consistent environment lighting.</figcaption></figure>

relighting of real-world objects into a specific lighting setting [22]. However, relighting the entire scene is still an open challenge. Another possible approach is the relighting of virtual objects (Fig. 5) into a lighting setting that matches the environment lighting [19]. Both of those approaches require the estimation of environmental lighting from the external world. Usually, all the lighting information is estimated from images of the environment, captured in real-time, thus posing as a computer vision problem. With the advances in deep learning, those methods are able to estimate the lighting information and provide a way to improve the XR experience regarding visual fidelity. Extracting and estimating external world information is still considered a difficult task. Developments regarding the representation of the information, including the recent advancements in computer vision methods, can dramatically improve XR environments by allowing new forms of interaction between the virtual and the real world.

## 9 Conclusion

Extended Reality as a game platform has an incredible potential, due its high immersive conditions. However, it is a totally new computational and ubiquitous environment and brings many challenges and problems, some of them not trivial. In this workshop we categorize these issues in 7 different topics, although there can be many others. This classification is not exhaustive and there are many other aspects that could also be included, such as audio, new interfaces devices and collaborative environments.
References

1. Fove's $250,000 kickstarter campaign wants to bring eye-tracking control to virtual reality (Apr 2021), https://thenextweb.com/news/foves-250000-kickstarter-campaign-wants-to-bring-eye-tracking-control-to-virtual-reality, accessed: 2021-07-28

2. Albert, R., Patney, A., Luebke, D., Kim, J.: Latency requirements for foveated rendering in virtual reality. ACM Transactions on Applied Perception (TAP) 14(4), 1–13 (2017)

3. Cannavò, A., Calandra, D., Pratticò, F.G., Gatteschi, V., Lamberti, F.: An evaluation testbed for locomotion in virtual reality. IEEE Transactions on Visualization and Computer Graphics 27(3), 1871–1889 (2021). https://doi.org/10.1109/TVCG.2020.3032440

4. Caserman, P., Garcia-Agundez, A., Göbel, S.: A survey of full-body motion reconstruction in immersive virtual reality applications. IEEE transactions on visualization and computer graphics 26(10), 3089–3108 (2019)

5. Cherni, H., Métayer, N., Souliman, N.: Literature review of locomotion techniques in virtual reality. International Journal of Virtual Reality 20(1), 1–20 (Mar 2020). https://doi.org/10.20870/IJVR.2020.20.1.3183, https://ijvr.eu/article/view/3183

6. Curry, C., Li, R., Peterson, N., Stoffregen, T.A.: Cybersickness in virtual reality head-mounted displays: Examining the influence of sex differences and vehicle control. International Journal of Human-Computer Interaction pp. 1–7 (2020)

7. Dennison, M.S., D'Zmura, M.: Cybersickness without the wobble: experimental results speak against postural instability theory. Applied ergonomics 58, 215–223 (2017)

8. Dutta, A., Mondal, A., Dey, N., Sen, S., Moraru, L., Hassanien, A.E.: Vision tracking: A survey of the state-of-the-art. SN Computer Science 1(1), 1–19 (2020)

9. Grassini, S., Laumann, K.: Are modern head-mounted displays sexist? a systematic review on gender differences in hmd-mediated virtual reality. Frontiers in Psychology 11 (2020)

10. Han, S., Liu, B., Cabezas, R., Twigg, C.D., Zhang, P., Petkau, J., Yu, T.H., Tai, C.J., Akbay, M., Wang, Z., et al.: Megatrack: monochrome egocentric articulated hand-tracking for virtual reality. ACM Trans. Graph. 39(4), 87 (2020)

11. Hodent, C.: The Gamer's Brain: How Neuroscience and UX Can Impact Video Game Design. CRC Press (2017), https://books.google.com.br/books?id=JzyhDwAAQBAJ

12. Jeong, D., Yoo, S., Yun, J.: Cybersickness analysis with EEG using deep learning algorithms. In: 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). pp. 827–835. IEEE (2019)

13. Jiang, F., Yang, X., Feng, L.: Real-time full-body motion reconstruction and recognition for off-the-shelf vr devices. In: Proceedings of the 15th ACM SIGGRAPH Conference on Virtual-Reality Continuum and Its Applications in Industry-Volume 1. pp. 309–318 (2016)

14. Jonathan, E., Roberts, C., Presentation, S., Razzaque, S., Kohn, Z., Whitton, M.: Redirected walking. Proc. of Eurographics (08 2001)

15. Kasahara, S., Konno, K., Owaki, R., Nishi, T., Takeshita, A., Ito, T., Kasuga, S., Ushiba, J.: Malleable embodiment: changing sense of embodiment by spatial-temporal deformation of virtual human body. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. pp. 6438–6448 (2017)
16. Kemeny, A., Chardonnet, J.R., Colombet, F.: Getting Rid of Cybersickness: In Virtual Reality, Augmented Reality, and Simulators. Springer Nature (2020)

17. Kim, J., Kim, W., Oh, H., Lee, S., Lee, S.: A deep cybersickness predictor based on brain signal analysis for virtual reality contents. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 10580–10589 (2019)

18. Kolasinski, E.M.: Simulator sickness in virtual environments. Tech. rep., DTIC Document (1995)

19. Marques, B.A.D., Clua, E.W.G., Vasconcelos, C.N.: Deep spherical harmonics light probe estimator for mixed reality games. Computers & Graphics **76**, 96–106 (2018). https://doi.org/10.1016/j.cag.2018.09.003

20. Matsumoto, K., Ban, Y., Narumi, T., Yanase, Y., Tanikawa, T., Hirose, M.: Unlimited corridor: Redirected walking techniques using visuo-haptic interaction. In: ACM SIGGRAPH 2016 Emerging Technologies. SIGGRAPH '16, Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2929464.2929482; https://doi.org/10.1145/2929464.2929482

21. de Oliveira, E., Clua, E.W.G., Vasconcelos, C.N., Marques, B.A.D., Trevisan, D.G., de Castro Salgado, L.C.: Fpvrgame: Deep learning for hand pose recognition in real-time using low-end hmd. In: Joint International Conference on Entertainment Computing and Serious Games. pp. 70–84. Springer (2019)

22. Pandey, R., Orts-Escolano, S., LeGendre, C., Haene, C., Bouaziz, S., Rhemann, C., Debevec, P., Fanello, S.: Total relighting: Learning to relight portraits for background replacement. vol. 40 (2021). https://doi.org/10.1145/3450626.3459872

23. Porcino, T., Rodrigues, E.O., Silva, A., Clua, E., Trevisan, D.: Using the gameplay and user data to predict and identify causes of cybersickness manifestation in virtual reality games. In: 2020 IEEE 8th International Conference on Serious Games and Applications for Health (SeGAH). pp. 1–8. IEEE (2020)

24. Rietzler, M., Gugenheimer, J., Hirzle, T., Deubzer, M., Langbehn, E., Rukzio, E.: Rethinking redirected walking: On the use of curvature gains beyond perceptual limitations and revisiting bending gains. pp. 115–122 (10 2018). https://doi.org/10.1109/ISMAR.2018.00041

25. Slater, M., Wilbur, S.: A framework for immersive virtual environments (five): Speculations on the role of presence in virtual environments. Presence: Teleoperators & Virtual Environments **6**(6), 603–616 (1997)

26. Sun, Q., Patney, A., Wei, L.Y., Shapira, O., Lu, J., Asente, P., Zhu, S., McGuire, M., Luebke, D., Kaufman, A.: Towards virtual reality infinite walking: dynamic saccadic redirection. ACM Transactions on Graphics (TOG) **37**(4), 67 (2018)

27. Swafford, N.T., Iglesias-Guitian, J.A., Koniaris, C., Moon, B., Cosker, D., Mitchell, K.: User, metric, and computational evaluation of foveated rendering methods. In: Proceedings of the ACM Symposium on Applied Perception. pp. 7–14 (2016)

28. Taylor, J., Tankovich, V., Tang, D., Keskin, C., Kim, D., Davidson, P., Kowdle, A., Izadi, S.: Articulated distance fields for ultra-fast tracking of hands interacting. ACM Transactions on Graphics (TOG) **36**(6), 1–12 (2017)

29. Valente, L., Feijó, B., do Prado Leite, J.C.S., Clua, E.: A method to assess pervasive qualities in mobile games. Personal and Ubiquitous Computing **22**(4), 647–670 (2018)

30. Weier, M., Roth, T., Kruijff, E., Hinkenjann, A., Pérard-Gayot, A., Slusallek, P., Li, Y.: Foveated real-time ray tracing for head-mounted displays. In: Computer Graphics Forum. vol. 35, pp. 289–298. Wiley Online Library (2016)

31. Xie, N., Ras, G., van Gerven, M., Doran, D.: Explainable deep learning: A field guide for the uninitiated. arXiv:2004.14545 (2020)