# Mixed Reality at Work: Health, Social and Ethical Implication

Damien Brun, Stéphane Safin, Charles Gouin-Vallerand, Sébastien George

► To cite this version:

Damien Brun, Stéphane Safin, Charles Gouin-Vallerand, Sébastien George. Mixed Reality at Work: Health, Social and Ethical Implication. CHI'20, Workshop on Exploring Potentially Abusive Ethical, Social and Political Implications of Mixed Reality Research in HCI, Apr 2020, Honolulu, Hawaii, United States. hal-04033129

HAL Id: hal-04033129

https://hal.science/hal-04033129

Submitted on 16 Mar 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

Public Domain
# Mixed Reality at Work: Health, Social and Ethical Implication

**Damien Brun**

Université TÉLUQ
Montréal, QC, Canada
brund@acm.org

**Stéphane Safin**

I3 UMR 9217, Télécom Paris
(Institut Polytechnique)
Paris, France
stephane.safin@telecom-paris.fr

**Charles Gouin-Vallerand**

Université de Sherbrooke
Sherbrooke, QC, Canada
Charles.gouin-vallerand@usherbrooke.ca

**Sébastien George**

LIUM - EA 4023
Le Mans Université
72085 Le Mans, France
sebastien.george@univ-lemans.fr

## Abstract

In this position paper, we address the ethical and social issues regarding the use of mixed reality devices in professional contexts, in particular for industrial settings: health, exclusion and discrimination, accidental risk, over-control, meaning. Finally, we present some potential solutions from a human-computer interaction design perspective such as tangible trusted user control and sensors automatic shutdown.

## Author Keywords

Mixed reality; head-mounted display; accident; over-monitoring; privacy; working environment; industry.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).

CHI 20', April 25-30, 2020, Honolulu, HI, USA.
Proceedings of the Workshop on Exploring Potentially Abusive Ethical, Social and Political Implications of Mixed Reality Research in HCI.
© 2020 Copyright is held by the owner/author(s).
<figure><img src="image_1.png" /><figcaption>**Figure 1:** Health issues regarding devices weight and displays, in particular neck, noise and eyes strains.</figcaption></figure>

<figure><img src="image_2.png" /><figcaption>**Figure 2:** Exacerbated health issues due to the content height.</figcaption></figure>

## Introduction

As with many concepts before, the professional domains are the first to adopt new advanced technologies. Mixed reality is not an exception and it is already applied to many areas (industrial, health, architecture...) and envisioned to much more. The use cases of mixed reality are virtually endless, from the learning and assistance in situ to the remote help and control, the gain and benefits from the executives are easily palpable in terms of productivity but the positive impact must also be reflected to the direct users. Indeed, this new paradigm of adding a virtual layer on top of the reality thanks to multiple sensors directed towards the user and its environment may radically change the nature of work. In this context, we focused our position paper on identifying the potential risks regarding ethics associated with mixed reality in industrial context [8] and discuss about preventing solutions from a human-computer interaction design.

## 1. Health risk inherent from devices

Reasonably we assume that current and even near to medium-term future form factor of mixed reality devices applied to industrial context is between smart glasses and headset. In such case, the ergonomics characteristics such as weight of the device is an important factor of health that may cause (already reported subjectively [21]) nose, neck strain or other musculoskeletal disorders (Figure 1). Moreover, excessive exposition to physically close displays may quickly induce eyes fatigues or headaches [23].

## HCI Research & Design Discussion

From the hardware perspective, we should combine two already implemented solutions. (1) Externalizing most of the components of the mixed reality devices and

thus minimizing the head-worn components (adopted by Magic Leap One<fnref n="1" />, NReal<fnref n="2" /> or Epson Moverio<fnref n="3" />). (2) Integrating the mixed reality device to more comfortable headsets such as hard hats or resting on the forehead, and thus reliving any potential pain to the nose (adopted by Microsoft HoloLens 2⁴).

From the software perspective, we should minimize and limit the number of virtual contents out of users' horizon line such as very high or low (Figure 2) that may exacerbate neck strain due to head movement and posture (observed during authors' industrial experiments). Having virtual content in situ means that we will consume it in a much wider range of postures. We should run field and laboratory-controlled studies and design interactions accordingly to postures that goes beyond the usual sitting, standing or walking by including those regularly encountered in industry such as crouching, lying on the back, cramped working conditions... (Figure 3)

<figure><img src="image_3.png" /><figcaption>**Figure 3:** Effects of regular industrial postures such as lying on the back and crouching that need to be studied.</figcaption></figure>

<footnote n="1">1 https://www.magicleap.com</footnote>

<footnote n="2">2 https://www.nreal.ai</footnote>

<footnote n="3">3 https://moverio.epson.com</footnote>

<footnote n="4">4 https://www.microsoft.com/en-us/hololens</footnote>
<figure><img src="image_4.png" /><figcaption>**Figure 4:** Accidental risk by occlusion (e.g. virtual content over a hole).</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>**Figure 5:** Accidental risk by distraction.</figcaption></figure>

## 2. Accidental risk

Immersive environment offered by mixed reality may increase accidental injuries risk due to collisions or falling caused by occlusion that for instance may prevent seeing a hazardous entity (e.g. a hole or pipe, **Figure 4**). Preventing occlusion is not sufficient, because virtual content may also distract and prevent the user from focusing on the hazardous entities, which would have the same unfortunate accidental results (**Figure 5**), an issue already reported with mobile augmented reality game such as Pokémon Go [2, 9].

### HCI Research & Design Discussion

The occlusion issue has been regarded in previous studies to avoid hiding different kind of real content [11], such as bystanders' head [16]. Less effort has been engaged into the effect of distraction issues [13].

## 3. Work exclusion and discrimination risk

In regard to health risk, some people may be more or less physiologically able to adopt these devices, and thus create a new exclusion from a particular population.

### HCI Research & Design Discussion

We should emphasize the exploration of other devices form factor and way of consuming mixed reality, in particular these with external displays, prone to multi-user and collaborative experience, such as SixthSense [14, 15] that placed a pocket projector on the user.

## 4. Over-monitoring/control risk

At some point, mixed reality needs to collect contextual data in order to exist, the virtual layer is coherent only if the virtual content is well placed, at the right moment. The space and time content cohesion will

increase accordingly with the quantity of collected data. The collected data will be extremely sensitive about the worker and may cause an over-monitoring and control from executives. Even if law will regulate the use of collected data, it will affect the hierarchical trust at work by knowing that what a mixed reality user will see, hear, do and much more may be monitored. Over-monitoring, in particular via cameras, have significant negative effects regarding work stress [1], satisfaction and affective commitment [7]. Other potential effects of the feeling of excessive monitoring:

* Strict respect of the processes to avoid being considered against the prescribed rules, which may harm an essential process at work: regulations-in-action by the operator and thus leading to decrease in performance and system reliability [12].

* More hacks (**Figure 6**) from the worker that cause riskier bypass strategies, such as purposely not looking at some sensitive points of interest, leading to accidental scenarios.

### HCI Research & Design Discussion

From the hardware perspective, the mixed reality devices may directly integrate modular strategy with physical trusted switch to activate and deactivate any sensors (such as cameras, micro, inertial central... **Figure 7**). However, this approach may be a double-edged design, on one hand, reassuring users with extremely trusted solution, one the other hand, it concedes that the technology cannot be trusted [22].

From the software perspective, every mixed reality system should include fine-grained permissions, adding a layer between raw data from continuous sensing
<figure><img src="image_6.png" /><figcaption>**Figure 6:** Famous tangible hack to protect webcam privacy issues.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>**Figure 7:** Integrating tangible sensor switches to the augmented/mixed reality devices.</figcaption></figure>

components and applications to access only specific high-level data [3, 5, 6, 17, 18, 20]. Another design strategy would be to automatically deactivate sensors after some delays, requiring a push from the user to restart, instead of the opposite.

## 5. Work meaning-loss risk

Assistance with mixed reality may be overused and externalizing most of the cognitive task to artificially intelligent helper may increase the robotization of workers and decrease their autonomy, which would lead to a loss of meaning at work and finally deepen mental health problems. Such effect is already palpable in industrial warehouse where workers are told exactly where to find and put objects with auditive or visual cues.

### HCI Research & Design Discussion

Perhaps a potential solution would be to actively integrate mixed reality gamification to disguise work, increase motivation while giving feedback to workers about the benefit of the performed tasks to the company [4, 10].

### Conclusion

In this paper, we started to identify different issues regarding health, social and ethical implication of mixed reality. We are aware that it is far from an exhaustive list and from our examples we focused on industrial working conditions only because it is not only an anticipation for tomorrow, but for the present. However, most of the points may be applied as much to non-working environment, encountered in everyday life.

### Authors

Damien Brun is a Ph.D. student in Cognitive Computing from TELUQ University (Canada) and tends to specialize in mixed reality head-mounted display interactions that include text-entry interfaces and multimodal interaction combining different devices for sensitive industrial context such as nuclear power plants. He is supervised by Charles Gouin-Vallerand from Université de Sherbrooke (Canada) and Sébastien George from Le Mans Université (France). Stéphane Safin is assistant professor in cognitive ergonomics at Télécom Paris, Institut Polytechnique de Paris (France).

### Acknowledgements

The authors gratefully acknowledge the grant from the Natural Sciences and Engineering Research Council (NSERC) of Canada and the company OHRIZON.

### References

[1] Aiello, John R., and Kathryn J. Kolb. "Electronic Performance Monitoring and Social Context: Impact on Productivity and Stress." *Journal of Applied Psychology* 80, no. 3 (1995): 339.

[2] Ayers, John W., Eric C. Leas, Mark Dredze, Jon-Patrick Allem, Jurek G. Grabowski, and Linda Hill. "Pokémon GO—A New Distraction for Drivers and Pedestrians." *JAMA Internal Medicine* 176, no. 12 (December 1, 2016): 1865–66. https://doi.org/10.1001/jamainternmed.2016.6274.

[3] D'Antoni, Loris, Alan Dunn, Suman Jana, Tadayoshi Kohno, Benjamin Livshits, David Molnar, Alexander Moshchuk, Eyal Ofek, Franziska Roesner, and Scott Saponas. "Operating System Support for Augmented Reality Applications." In *Presented as Part of the 14th Workshop on Hot Topics in Operating Systems*, 2013.
[4] Funk, Markus, Thomas Kosch, Romina Kettner, Oliver Korn, and Albrecht Schmidt. "Motioneap: An Overview of 4 Years of Combining Industrial Assembly with Augmented Reality for Industry 4.0." In *Proceedings of the 16th International Conference on Knowledge Technologies and Datadriven Business*, 4, 2016.

[5] Jana, Suman, Arvind Narayanan, and Vitaly Shmatikov. "A Scanner Darkly: Protecting User Privacy from Perceptual Applications." In *2013 IEEE Symposium on Security and Privacy*, 349-363. IEEE, 2013.

[6] Jana, Suman, David Molnar, Alexander Moshchuk, Alan Dunn, Benjamin Livshits, Helen J. Wang, and Eyal Ofek. "Enabling Fine-Grained Permissions for Augmented Reality Applications with Recognizers." In *Presented as Part of the 22nd USENIX Security Symposium Security 13*, 415-430, 2013.

[7] Jeske, Debora, and Alecia M. Santuzzi. "Monitoring What and How: Psychological Implications of Electronic Performance Monitoring." *New Technology, Work and Employment* 30, no. 1 (2015): 62-78.

[8] Jetter, Jérôme, Jörgen Eimecke, and Alexandra Rese. "Augmented Reality Tools for Industrial Applications: What Are Potential Key Performance Indicators and Who Benefits?" *Computers in Human Behavior* 87 (October 1, 2018): 18-33. https://doi.org/10.1016/j.chb.2018.04.054.

[9] Joseph, Bellal, and David G. Armstrong. "Potential Perils of Peri-Pokémon Perambulation: The Dark Reality of Augmented Reality?" *Oxford Medical Case Reports* 2016, no. 10 (2016): omw080.

[10] Korn, Oliver, Markus Funk, and Albrecht Schmidt. "Towards a Gamification of Industrial Production: A Comparative Study in Sheltered Work Environments." In *Proceedings of the 7th ACM SIGCHI Symposium on Engineering Interactive Computing Systems*, 84-93, 2015.

[11] Lebeck, Kiron, Tadayoshi Kohno, and Franziska Roesner. "How to Safely Augment Reality: Challenges and Directions." In *Proceedings of the 17th International Workshop on Mobile Computing Systems and Applications*, 45-50. HotMobile '16. New York, NY, USA: ACM, 2016. https://doi.org/10.1145/2873587.2873595.

[12] Leplat, Jacques. "La Notion de Régulation Dans l'analyse de l'activité." *Perspectives Interdisciplinaires Sur Le Travail et La Santé*, no. 8-1 (2006).

[13] McNamara, Ann, Katherine Boyd, Joanne George, Weston Jones, Somyung Oh, and Annie Suther. "Information Placement in Virtual Reality." In *2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)*, 1765-69, 2019. https://doi.org/10.1109/VR.2019.8797891.

[14] Mistry, Pranav, Pattie Maes, and Liyan Chang. "WUW-Wear Ur World: A Wearable Gestural Interface." In *CHI'09 Extended Abstracts on Human Factors in Computing Systems*, 4111-4116. ACM, 2009.

[15] Mistry, Pranav, and Pattie Maes. "SixthSense: A Wearable Gestural Interface." In *ACM SIGGRAPH ASIA 2009 Sketches*, 11:1-11:1. SIGGRAPH ASIA '09. New York, NY, USA: ACM, 2009. https://doi.org/10.1145/1667146.1667160.

[16] Orlosky, Jason, Kiyoshi Kiyokawa, Takumi Toyama, and Daniel Sonntag. "Halo Content: Context-Aware Viewspace Management for Non-Invasive Augmented Reality." In *Proceedings of the 20th International Conference on Intelligent User Interfaces*, 369-373. IUI '15. New York, NY, USA: ACM, 2015. https://doi.org/10.1145/2678025.2701375.

[17] Raval, Nisarg, Animesh Srivastava, Kiron Lebeck, Landon Cox, and Ashwin Machanavajjhala. "MarkIt: Privacy Markers for Protecting Visual Secrets." In *Proceedings of the 2014 ACM*
*International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication*, 1289-1295. UbiComp '14 Adjunct. Seattle, Washington: Association for Computing Machinery, 2014.
https://doi.org/10.1145/2638728.2641707.

[18] Roesner, Franziska, David Molnar, Alexander Moshchuk, Tadayoshi Kohno, and Helen J. Wang. "World-Driven Access Control for Continuous Sensing." In *Proceedings of the 2014 ACM SIGSAC Conference on Computer and Communications Security*, 1169-1181, 2014.

[19] Roesner, Franziska, Tadayoshi Kohno, and David Molnar. "Security and Privacy for Augmented Reality Systems." *Communications of the ACM* 57, no. 4 (2014): 88-96.

[20] Templeman, Robert, Mohammed Korayem, David J. Crandall, and Apu Kapadia. "PlaceAvoider: Steering First-Person Cameras Away from Sensitive Spaces." In *NDSS*, 23-26. Citeseer, 2014.

[21] Wille, Matthias, Lars Adolph, Britta Grauel, Sascha Wischniewski, Sabine Theis, and Thomas Alexander. "Prolonged Work with Head Mounted Displays." In *Proceedings of the 2014 ACM International Symposium on Wearable Computers: Adjunct Program*, 221-224. ISWC '14 Adjunct. Seattle, Washington: Association for Computing Machinery, 2014.
https://doi.org/10.1145/2641248.2645638.

[22] Wolf, Marty J., Frances Grodzinsky, and Keith Miller. "Augmented Reality All Around Us: Power and Perception at a Crossroads." *SIGCAS Comput. Soc.* 45, no. 3 (January 2016): 126-131.
https://doi.org/10.1145/2874239.2874257.

[23] Wang, Yan, Guangtao Zhai, Sichao Chen, Xiongkuo Min, Zhongpai Gao, and Xuefei Song. "Assessment of Eye Fatigue Caused by Head-Mounted Displays Using Eye-Tracking." *Biomedical Engineering Online* 18, no. 1 (2019): 111.