The Effect of Task Difficulty on Human Workload and Performance in Multitasking Mixed Reality Scenarios

Mixed reality is a form of virtual reality technology that allows people to use both physical and digital worlds at the same time (Choi et al., 2022; Park et al., 2021; Rokhsaritalemi, Sadeghi-Niaraki, & Choi, 2020). Multitasking has been considered an essential element of memory capacity research that helps in providing a global consideration of human existence (Spink, Cole, & Waller, 2008). However, limited research compared multitasking between physical and digital worlds with different task difficulty levels. This study investigates the task difficulty effect on mixed reality multitasking workload, accuracy performance, physical-digital task priority, and dominant hand used. Two different tasks were used in this experiment for the physical and digital worlds: a block matching task and a virtual N-back task, respectively. A block matching task was performed using a wooden-block puzzle board. The board includes 26 uppercase English alphabets, and participants were asked to match pre-assigned blocks on the board within a certain time limit. A virtual N-back task application was developed to evaluate the digital world performance by matching the colors of a virtual cube via a Microsoft HoloLens 2 device. Such tasks allow experimenters to assess users’ individual differences in their working memories (Jaeggi, Buschkuehl, Perrig, & Meier, 2010). Several measures were selected based on the nature of the experiment including NASA-TLX, accuracy, total completion time, task priority, and hand used. We chose NASA-TLX as the main measure because it is considered a multi-dimensional scale that has the ability to assess users’ workload either while performing a task or afterward (Hart, 2006). Participants rated their scores on a percentage scale from 0 to 100 for all NASATLX six dimensions: mental demand, physical demand, temporal demand, performance, effort, and frustration. A total of 288 datasets were collected (8 sessions × 36 participants). Four sessions were for single tasks, and the other four were for multitasks. Each participant was tested in a given random order of an easy real-world (RW) task, an easy virtualworld (VW) task, a hard RW task, a hard VW task, an easy RW task with an easy VW task, an easy RW task with a hard VW task, a hard RW task with an easy VW task, and a hard RW task with a hard VW task. To test the significance of the task difficulty on each of our dependent measures; we employed different statistical tests. Mann-Whitney and Kruskal-Wallis tests were used for the numerical dependent variables (NASA-TLX, total completion time, and accuracy). The Chi-square test was used to test the significance of task difficulty on task priority measure. Correlation analysis was consulted on the hand used variable to determine the relationship between observed and selfreported hand use by each task difficulty level. A significant difference was found in the results of physical demand and effort, where the exact difference was found between comparing easy and hard real-virtual worlds tasks. Accuracy was found to be significantly different with an effect size of 19.06%. However, priority measure was found insignificant. A strong positive correlation was found in the dominant hand use variable with r=0.7. This study is limited to having a similar group of participants and testing them on only two difficulty levels (easy vs. hard). However, given the results of this study, mixed reality task designers should consider the task difficulty factor in predicting their users’ performance to provide a better human engagement with such technologies.