# Investigating the impact of augmented reality in machine operation training: A case study on laser-cutting machines

Meysam Imanipour, Alex Gabriel, Alaa Hassan, Aylen Ricca

<figure><img src="image_1.png" /></figure>

## To cite this version:

Meysam Imanipour, Alex Gabriel, Alaa Hassan, Aylen Ricca. Investigating the impact of augmented reality in machine operation training: A case study on laser-cutting machines. European Journal of Engineering Education, 2025, 50 (6), pp.1346-1372. 10.1080/03043797.2025.2585117. hal-05417957

HAL Id: hal-05417957

https://univ-evry.hal.science/hal-05417957v1

Submitted on 8 Jan 2026

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Investigating the Impact of Augmented Reality in Machine Operation Training: A Case Study on Laser-Cutting Machines

Meysam Imanipurª, Alex Gabrielª,*, Alaa Hassanª, Aylen Riccaᵇ

## Abstract

Numerous studies have demonstrated the positive effects of augmented reality (AR) in educational applications. However, its application in manufacturing training remains limited, particularly in areas beyond assembly and maintenance tasks. This ongoing research investigates AR's impact on learning in machine operation beyond assembly and maintenance. The study aims to explore whether AR positively influences user learning and performance. By using AR to teach the operation of a laser-cutting machine, real-time visual information and spatial synchronization are expected to improve performance and the learning experience. A comparative study between AR-based training and traditional instructor-led methods was conducted. However, results did not fully support the hypothesis. The inexperienced learners who used AR produced fewer errors than instructor-led learners when reproducing the cutting task, but the control group performed faster than the AR group. The experiment also revealed a significant difference in the amount of additional information needed by both groups. These findings highlight the importance of design and challenge the notion that AR consistently enhances learning. However, these results offer valuable insights for future AR-based training system designs, emphasizing the need for more nuanced approaches to optimize AR's educational efficacy.

**Keywords:** augmented reality, human-computer interface, evaluation methodologies, instructional design, manufacturing training

<footnote>ªUniversité de Lorraine, ERPI, F-5400 Nancy, France</footnote>

<footnote>ᵇIBISC Lab, Univ. Evry Paris-Saclay, France</footnote>
# 1. Introduction

In the context of the fourth industrial revolution, factories are undergoing significant changes to become more efficient and safer (Ababsa, 2020). These transformations are driving advancements in production processes, leading to increased productivity and requiring the factory workforce, including operators, to develop new skills and adapt to emerging technologies. Traditional training methods are insufficient to address these needs, making it essential to implement dynamic training systems that can accommodate technological progress (Doolani et al., 2020).

Extended reality (XR) technologies, encompassing virtual reality (VR), augmented reality (AR), mixed reality (MR), and their combinations, have shown promise in education and are being explored as potential solutions to meet this training requirement (Cárdenas-Robledo et al., 2022; Doolani et al., 2020). A systematic review of the literature on XR's impact on learning and user satisfaction published in 2021 reported that most studies (90%) found higher satisfaction among XR users, and improved learning outcomes among 70% of the reviewed studies (Vicente dos Anjos et al., 2021). XR technologies have distinct characteristics: VR immerses users in fully virtual environments, while AR overlays digital content onto real-world environments, including 3D visualizations, and text (Akçayır and Akçayır, 2017; Chiang et al., 2022; Tekedere and Göke, 2016). MR integrates real and digital elements, allowing users to interact with and manipulate digital objects (Doolani et al., 2020).

Among XR technologies, AR has gained popularity in education and demonstrated promising results. Studies have shown its adoption in fields such as surgery, navigation, education, manufacturing, visual tracking, and architecture. The development of AR technology has also reduced costs, making it more accessible across various sectors, including education and manufacturing (Su et al., 2021). Furthermore, research has found that AR enhances memorization and is increasingly being used in industrial applications, with some studies suggesting it surpasses VR in fostering a sense of presence (Scaravetti and François, 2021).

Despite the increasing use of AR in industrial education, most research has focused on assembly and maintenance tasks, with limited studies addressing its application in other manufacturing activities such as machinery operation (Chiang et al., 2022). This gap in the literature on AR for machinery training has also been noted in other reviews of AR-based learning systems (Butaslac et al., 2023). Additionally, while AR is often designed to provide continuous task support, in training contexts, the goal is for trainees to perform tasks independently after the training, without relying on AR assistance (Butaslac et al., 2023). This reveals a critical gap in current AR training applications: the need for systems that support task performance while ensuring trainees can operate autonomously after the training.

To address this issue and improve manufacturing training, this study focuses on applying AR technology specifically for machinery operation training. Given the limitations of previous research in consistently assessing learning outcomes (Vicente dos Anjos et al., 2021), this study aims to evaluate the learning outcomes and performance of learners using AR tools, as well as their perceptions of learning. We developed an AR application for comprehensive training in machinery operation. Based on the scientific literature review presented in the next section, we hypothesize that AR in this training context will significantly improve users' learning outcomes and operational performance. It is also assumed that perceived ability, as well as the utility, clarity and completeness of the information are equivalent. A user study will be conducted to measure the impact of AR on learning outcomes, performance and perception, with the goal of determining whether AR-based training enables users to operate machinery independently and efficiently after training.
This paper begins with a review of the literature on AR applications in manufacturing training and standard evaluation methods for assessing AR-based training. The methodological section details the experimental protocol, metrics, and questionnaires used in the user study. Section 4 presents the preliminary results and analysis. In Section 5, the results are further discussed, and Section 6 outlines the study's limitations, challenges, and potential future research directions.

## 2. Literature Review

### 2.1. Augmented Reality Applied to Manufacturing Training

Previous literature consistently demonstrates the positive impact of AR technology across various educational domains. For example, Akçayır et al. (2016) showed that AR improved university students' laboratory skills, enhancing collaboration and communication during experiments. Karagozlu, (2018) found that seventh-grade students using AR applications exhibited improved problem-solving skills in science courses. Additionally, Eswaran and Raju Bahubalendruni (2023) demonstrated that AR-based object mapping visualization increased assembly task efficiency and reduced errors compared to other guidance modes. Notably, AR headsets provided a more realistic experience, minimized errors, and reduced task completion time compared to handheld visualization devices.

AR technology has also been widely applied in vocational training, which aims to equip the workforce with job-specific skills. Manufacturing training, a subset of vocational training, has seen particular interest (Doolani et al., 2020). Chiang et al. (2022) conducted a systematic review of AR applications in vocational training across various industries, analyzing 703 papers on AR implementations from 2000 to 2021. After filtering and reviewing abstracts, 80 articles were selected for further analysis. The authors classified AR applications into nine industry fields, focusing on their prevalence in each category (Figure 1). The majority of AR applications were related to assembly training, maintenance, automobile part testing, engineering, safety training, transportation, chemical processing, and welding. However, none of the reviewed studies addressed AR applications in machinery operation training.

<figure><img src="image_3.png" /><figcaption>Figure 1. Classification of augmented reality applications in different industry fields, extracted from (Chiang et al., 2022)</figcaption></figure>

Butaslac et al. (2023) conducted a systematic review of AR training systems, analyzing 64 research papers published between 2005 and 2023 in high-impact journals. These papers were categorized by
training method and field, revealing that only 6% of the articles (four papers) focused on industrial applications, with none specifically addressing machinery training.

In another study, Doolani et al. (2020) reviewed XR technologies, including AR, in various domains, including manufacturing training. They identified three task categories in the manufacturing industry: assembly line tasks, job shops, and discrete tasks. These encompass activities such as monitoring assembly lines, sorting, picking, assembling, installing, packing, and operating tools and machinery. The authors divided these tasks into five phases: Introductory, Learning, Operational, Tangent, and End phases, and examined XR technologies used in each phase. Table 1 illustrates the tasks within the phases and the technology used. While AR was utilized in some phases, it was generally less effective than VR and MR, highlighting the need for further AR development in manufacturing training.

<table><caption>Table 1. The utility of XR technologies in different phases of manufacturing tasks, extracted from (Doolani et al., 2020)</caption><thead><tr><td>Manufacturing Phases</td><td>Tasks</td><td>Useful XR technology in training</td></tr></thead><tbody><tr><td>Introductory phase</td><td>Safety training, orientation training, planning and designing of new tasks</td><td>VR, MR</td></tr><tr><td>Learning phase</td><td>Sorting, picking, keeping, assembling, installation</td><td>VR, AR, MR</td></tr><tr><td>Operational Phase</td><td>Inspection, packing, monitoring assembly line, assembly</td><td>MR</td></tr><tr><td>Tangent phase</td><td>Using rare tool/machinery, hand tool, power tool</td><td>AR, MR</td></tr><tr><td>End phase</td><td>Cleaning routine (process, shovel, sweep, clean work areas), inspection</td><td>AR, MR</td></tr></tbody></table>

Scaravetti and François (2021) explored AR's potential in enhancing training for complex topics in higher education, particularly in mechanical engineering. Their study presented AR training scenarios on subjects such as power transmission, kinematics, and mechanism functionality. Case studies included training on gearboxes, gate actuators, and turbofans, and experimental testing demonstrated a positive impact on students' learning achievements and knowledge transfer.

Regarding machinery operation, Monroy Reyes et al. (2016) investigated the use of AR in supporting students with milling and lathe machine setups. This study, however, focused on basic setup tasks, such as opening and closing Chuck's jaws and selecting machine speed, without covering comprehensive machine use or daily manufacturing tasks. The evaluation relied on subjective feedback from students who tested the application and rated its acceptance and perceived performance.

Similarly, Tzimas (2018) developed an AR application to assist operators in workpiece fixation and clamping. The traditional process relies on CAD drawings, which require considerable time to determine proper orientation and clamping tools. Tzimas argued that while setup time is negligible in mass production, it is significant in job shops with limited part production. AR assistance was shown to reduce setup time in these scenarios.

Appendix 1 summarizes other reviewed studies on AR in manufacturing training, most of which focus on maintenance, assembly, and safety tasks. There is a notable lack of comprehensive AR training approaches for machinery operation. Specifically, there is a gap in training programs designed to teach inexperienced users or new employees how to operate machinery and perform fabrication tasks. This article posits that AR can provide a rich learning experience by guiding learners through face-to-face interaction with machines. The objective of this work is to assess the impact of AR-based training in this area, particularly for novice users.

Based on the previous literature reviews (Butaslac et al., 2023; Chiang et al., 2022; Doolani et al., 2020) and the studies referenced in Appendix 1, AR's impact on technical and industrial tasks is relatively
well-explored. However, although AR has been applied in manufacturing assistance, the use of AR for machinery operation training has not been extensively studied. As noted by Doolani et al. (2020), further development is required for AR technology to become effective across all phases of manufacturing tasks.

## 2.2. Standard Evaluation Methods and Metrics

To ensure reliable results and effectively address the research question, a formal evaluation of the proposed AR application is essential. In a study by da Silva et al. (2019), it was reported that only 8% of published AR papers included formal evaluations. Therefore, conducting experiments under well-defined conditions is crucial. This section presents the standard evaluation methods for AR educational applications found in the literature. The review is guided by two key questions: "How are software and experiments evaluated?" (evaluation methods) and "What parameters are used for measurement?" (measurement metrics).

Da Silva et al. (2019) conducted a systematic review of educational AR applications, identifying common evaluation methods and metrics in papers with formal evaluations. The study found that experimental evaluations were the most commonly used method, followed by questionnaires. Other methods, such as observation, were less prevalent. Many papers combined experimental and questionnaire-based methods. The first commonly measured metric was objective performance, including factors such as knowledge retention, learning achievement, precision, accuracy, awareness, reaction time, error rate, task completion, and time taken to complete tasks (da Silva et al., 2019; Doolani et al., 2020). In the context of XR, transfer effectiveness ratio is used to assess the value of training in a simulator by calculating the efficacy of the virtual training session (Kaplan et al., 2021). The second most common metric was usability, which will be discussed in detail later. The third category involved subjective or qualitative measurements, including motivation, user perception, attitude, feelings, engagement, enjoyment, and user feedback, often assessed through questionnaires (da Silva et al., 2019; Doolani et al., 2020). XR training metrics may also include measures of presence, immersion, and the psychological dimension of flow (Kaplan et al., 2021).

Usability is a key metric of User Experience (UX) (Arifin et al., 2018), evaluating an application's ease of use, user acceptance, and interface ergonomics (Akçayır and Akçayır, 2017; Dünser et al., 2008). One common limitation of AR learning applications is usability challenges, where learners experience difficulties interacting with the software (Akçayır and Akçayır, 2017). This issue often arises because AR application development can be relatively simple, leading to designs by non-experts that overlook crucial UX factors such as usability (Arifin et al., 2018). As a result, recent studies have increasingly focused on addressing usability issues in AR applications.

As noted, experimental evaluation is the most common method for assessing AR applications. For this study, a literature review was conducted focusing on AR applications in vocational and manufacturing training. The evaluation methods, metrics, and devices used in these studies are summarized in Appendix 1.

For subjective measurements, most studies assess their AR applications using questionnaires that must be clear and unambiguous (da Silva et al., 2019). Questionnaire methods commonly used to measure UX include AttrakDiff and SUXES according to Dhir and Al-kahtani, (2013). Additionally, Ibáñez et al. (2014) suggested using open-ended questions, allowing participants to provide more detailed responses without being constrained by predefined answers.
# 3. Methods

Addressing the aforementioned gap in AR applications related to machinery handling, this study focused on developing an AR application for manufacturing training in machinery operation handling. The chosen case study involved a laser-cutting machine, aiming to train students on its proper usage and cutting tasks. The research questions driving this investigation are:

* RQ1: What is the effect of AR technology on learners' achievement and performance in machinery operation handling?

* RQ2: What is the effect of AR technology on learners' perceptions of their learning to handle machine?

## 3.1. Experimental Design

To answer these questions, inspired by similar work from the literature (Appendix 1), a quasi-experimental research design was adopted and more precisely a post-test only non-equivalent control-group design (Campbell and Stanley, 1963; Leavy, 2017). It means the experimentation involved an experimental group (AR group) compared to a control group. The independent variable was the type of training provided, which can be either training with AR application or traditional training with an in-situ educator (i.e., verbal and gestural explanations). The objectives dependant variables observed was the task completion, completion time, number of errors, and number of assistance requests, all of which were measured through an analysis grid completed by the experimenter. Subjective dependant variables were composed by participants personal evaluation of the tools regarding the perceived utility, clarity, completeness and sense of competence after training. As in similar research on educational AR applications (Monroy Reyes et al., 2016; Zhang et al., 2021), a Likert scale questionnaire with a 5-point scale was used to assess these variables.

Participants were selected among the visitors of the educational and research FabLab that hosted our experimentation. No compensation was given for participation in this study, which was voluntary. Some experiments took place within a prototyping course context, while others occurred without any specific purpose. The distribution of participants was initially established according to the availability of the head-mounted display (HMD) used for the experiment and, subsequently, the laser cutting machines. Following a recruitment campaign distributed through the university's newsletters, we aimed to balance group sizes as closely as feasible without considering any other factors. However, due to the dynamic nature of the FabLab, which is shared space for both research and instruction, various factors impacted our ability to achieve perfect balance. Some FabLab users affected the experiment by unplanned use of the HMD, forcing the experimentation to be carried out with the instructor. In addition, long experiments with augmented reality were shortened to instructor-led experiments when the HMD's battery was completely drained. Furthermore, some visitors who hadn't initially planned to take part in the study joined the experiment, out of curiosity about FabLab technology and interest in participating in a study. However, the allocation of unannounced participants to one of the two groups was dependent on the availability of the HMD and at least one laser cutter. These factors contributed to variations in group sizes. Given these constraints, we did not use prior experience with augmented reality or machinery handling as a criterion for balancing group composition. Instead, participants completed a questionnaire regarding their previous experiences with augmented reality and machinery handling.

## 3.2. Sample
This study included 90 participants (45 males, 45 females) from the university, with a mean age of 22.1 years (SD = 4.66). The participants formed a diverse group, consisting of students and university staff (professors, technicians, administrative personnel) from various academic backgrounds. Of the 90 participants, 88 responded to the background question. Fifty-eight participants had a background in engineering, primarily industrial engineering, with some also having degrees in mechanical, chemical, and electrical engineering. Twenty participants came from design field. The remainder had backgrounds in linguistics, management, humanities, and cognitive science.

Given the diverse backgrounds, it was expected that some participants might have prior experience with AR technology and machinery handling. Table 2 summarizes the participants' prior experience with AR. The majority of participants (62 participants, or 70%) reported having no prior AR experience. Regarding machinery handling, the groups showed a considerable level of variation. Most participants had no prior experience with machinery (55, or approximately 62.5%), as shown in Table 3. However, a notable proportion (29.5% of the total sample) had minimal or some experience with machinery, including 14.7% who reported having limited experience. A small number of participants had substantial experience with machinery, particularly those familiar with equipment like 3D printers, CNC machines, or milling machines. Among these participants, eighteen reported previous interaction with a laser-cutting machine.

<table><caption>Table 2. Participants with previous AR experience</caption><thead><tr><th rowspan="2">Group</th><th colspan="6">Answers</th><th rowspan="2">Total</th></tr><tr><th>Never</th><th>Rarely</th><th>Sometimes</th><th>Moderately</th><th>Regularly</th><th>Very Regularly</th></tr></thead><tbody><tr><td>AR</td><td>23</td><td>11</td><td>2</td><td>1</td><td>0</td><td>0</td><td>37</td></tr><tr><td>Control</td><td>39</td><td>12</td><td>0</td><td>0</td><td>0</td><td>0</td><td>51</td></tr><tr><td>AR + Control</td><td>62</td><td>23</td><td>2</td><td>1</td><td>0</td><td>0</td><td>88</td></tr></tbody></table>

<table><caption>Table 3. Participants with previous machinery handling experience</caption><thead><tr><th rowspan="2">Group</th><th colspan="6">Answers</th><th rowspan="2">Total</th></tr><tr><th>None</th><th>Very little</th><th>Little</th><th>Moderately familiar</th><th>Familiar</th><th>Very Familiar</th></tr></thead><tbody><tr><td>AR</td><td>25</td><td>4</td><td>4</td><td>2</td><td>1</td><td>1</td><td>37</td></tr><tr><td>Control</td><td>30</td><td>9</td><td>9</td><td>3</td><td>0</td><td>0</td><td>51</td></tr><tr><td>AR + Control</td><td>55</td><td>13</td><td>13</td><td>5</td><td>1</td><td>1</td><td>88</td></tr></tbody></table>

The unequal distribution of total participants between the AR group and control group, as highlighted in Table 2 and Table 3, was anticipated due to operational factors outlined in the experimental design section. However, this imbalance may compromise internal validity by introducing participant selection bias. The moderate difference in group size could be accommodated through inferential statistics, but a more significant concern lies in the unequal distribution of experience levels among participants. Given the relatively small sample size, the presence of more experienced individuals could significantly impact results and potentially introduce bias on the global performance and achievement of each group.

Regarding the impact of AR experience on performance, participants with prior experience in AR technology may exhibit accelerated task completion rates and reduced novelty-induced distraction, as they are already proficient in utilizing this technology. In contrast, none of our participants had prior experience with head-mounted displays used in the study. However, the lack of familiarity with augmented reality could still impact their perception of novelty, usefulness, and clarity of interfaces
and information. This might lead to differences in performance compared to those with more extensive AR experience.

On the other hand, previous machinery handling experience could bias the results through a positive effect on learning with augmented reality through a learning transfer effect. This learning transfer is also applicable to training with the instructor. This effect would introduce mediating variables, such as the degree of mastery and quality of training prior to experimentation which may influence the outcome.

It is worth noting that heterogeneous group compositions are common in real-world settings, which can enhance external validity. Nevertheless, the distribution of experience levels between the two groups remains uneven. Therefore, in order to ensure internal validity, the following results and analyses will focus exclusively on those without prior experience in AR technology or machine handling, representing 18 participants from the AR group and 24 participants from the control group.

### 3.3. Hypothesis

Previous research has consistently demonstrated AR's positive impact on students' performance and learning outcomes, often associating it with increased motivation and active engagement in coursework. It has also been suggested that the lack of "learning by doing" in traditional educational methods may contribute to students' challenges and negative attitudes, particularly in science courses. The use of AR offers an opportunity to move from a passive learning experience to an active learning one, which has been shown in various fields to be beneficial for students (Michael, 2006). According to the principle of active learning, students should engage in some activity that forces them to reflect on their ideas and how they are using those ideas (Michael, 2006). This approach relies on the fact that actions performed by learners are better recalled than when they simply hear or read about the action description (Mulligan and Hornstein, 2003). It is the subject-performed tasks effect (Mulligan and Hornstein, 2003) also known as "learning by doing" (Steffens et al., 2015). Based on this effect, we hypothesize that using AR while directly interacting with the machine will have a positive effect on learning outcomes, as learning is facilitated through doing. If this hypothesis is not supported, we will investigate how information is perceived through AR technology.

In a pedagogical context, if AR proves no more effective than conventional methods, we expect it to be at least equally effective, ensuring that the information is as well-perceived as in traditional training.

Based on these observations, the following working hypotheses were developed for our user study:

* H1.1null - Participants in the AR group, who engage in hands-on learning through the AR application, will show similar task completion time, compared to the Control group, who learn through observing an instructor. As shown by various previous works, usage of AR can lead to significant time reduction in tasks reproduction (Borgen et al., 2021; Eswaran and Raju Bahubalendruni, 2024; Kwiatek et al., 2019). It might be expected of a reduction of the completion time leading to the alternative hypothesis H1.1alt - Participants in the AR group, will show reduced task completion time, compared to the Control group, who learn through observing an instructor.

* H1.2null - Participants in the AR group, who engage in hands-on learning through the AR application, make as many errors as the Control group, who learn through observing an instructor. The literature possesses works that show reduced numbers of errors using AR (Ariansyah et al., 2024; Aschauer et al., 2021; Eswaran and Raju Bahubalendruni, 2024). It can
be expected the same effect, so the hypothesis alternative is H1.2<sub>alt</sub> - Participants in the AR group, will make fewer errors, compared to the Control group.

* H2.1<sub>null</sub> - The perceived utility of the training will be equivalent between the AR group and the control group. The hypothesis alternative is H2.1<sub>alt</sub> - The perceived utility of the training will be different for the AR group than the control group.

* H2.2<sub>null</sub> – The ability perception of participants will be equivalent between the AR group and the control group. The hypothesis alternative is H2.2<sub>alt</sub> - Participants in the AR group will have a different perception of their ability than those in the control group.

* H2.3<sub>null</sub> - The perceived clarity of the information will be equivalent between the AR group and the control group. The hypothesis alternative is H2.3<sub>alt</sub> – The perceived clarity of the information will be different for the AR group than the control group.

* H2.4<sub>null</sub> - The perceived completeness of the information will be equivalent between the AR group and the control group. The hypothesis alternative is H2.4<sub>alt</sub> - The perceived completeness of the information will be different for the AR group than the control group.

### 3.4. Experimental Procedure

The experimental procedure was divided into three consecutive phases: Introductory, Training, and Evaluation (Figure 2). During the Introductory phase, participants received a brief overview of the project's objectives, study design, and experimental procedures, as well as demographic data collection. Then, the training phase involved using an AR device to acquire training information or following a trainer-led training before completing a first questionnaire. Finally, in the Evaluation phase, both groups underwent identical tasks, where they had to reproduce the learned task and complete a final questionnaire.

<figure><img src="image_4.png" /><figcaption>Figure 2. Experimental procedure: Flow chart of the methodology protocol</figcaption></figure>

#### 3.4.1. Introductory Phase

Participants were invited to join the study, either by appointment or by visiting the FabLab. They were informed they would be participating in an experiment on learning laser cutting operation handling without prior knowledge of the details.
Before receiving further information, participants completed an online questionnaire to provide demographic data, including their field of study, age, and prior experience with AR and machine operation.

The participants were then randomly assigned to one of two groups: the AR group or the Control group. Each group received a script read aloud by the experimenter, which described the procedure and their role in the experiment. The scripts emphasized that the experimenter would observe and record data without interacting with the participant.

The AR group participants also received an additional brief to familiarize themselves with the HoloLens2 device, specifically its interaction mode and how to use it effectively. This ensured that all AR group participants started with a common baseline knowledge level. In contrast, the Control group did not require any familiarization phase.

### 3.4.2. Training Phase

The first task of the training phase was the individual training. Participants in the AR group received individualized training using a custom-developed augmented reality application via the HoloLens2 headset. The AR training guided participants step-by-step through the process of performing a cutting task on the laser-cutting machine, with each step requiring participants to confirm completion of the action without receiving feedback on accuracy. This execution and confirmation process was repeated until the entire laser-cutting procedure was completed.

In contrast, participants in the Control group received traditional instructor-led training from an educator. The instructor demonstrated the laser-cutting procedure to each participant individually, following a scripted protocol that included reading aloud the relevant information while performing the task. Participants observed the demonstration but did not perform the task themselves. Table 4 outlines the sequence of steps taught to both groups, while Table 5 provides a description of each task. Some steps in the cutting procedure were repeated multiple times to complete the task.

Once the training was complete, participants completed an online questionnaire using LimeSurvey. As they answered the questions, the experimenter prepared the laser cutting machine for the evaluation phase, making adjustments such as changing the Z offset to ensure clear observation of the adjustment operation.

<table><caption>Table 4. Training steps for AR and Control groups</caption><thead><tr><td>Sequence ID</td><td>Step name</td></tr></thead><tbody><tr><td>1</td><td>Machine on</td></tr><tr><td>2</td><td>Fume extractor on</td></tr><tr><td>3</td><td>Origin button</td></tr><tr><td>4</td><td>Open cover</td></tr><tr><td>5</td><td>Put workpiece</td></tr><tr><td>6</td><td>Esc button</td></tr><tr><td>7</td><td>Arrows</td></tr><tr><td>8</td><td>Z mode</td></tr><tr><td>9</td><td>Z adjustment</td></tr><tr><td>10</td><td>Z validation</td></tr></tbody></table>

<table><caption>Table 5. Explanation of each step required to perform a laser-cutting task</caption><thead><tr><td>Sequence ID</td><td>Step name</td></tr></thead><tbody><tr><td>11</td><td>Test button</td></tr><tr><td>12</td><td>Close cover</td></tr><tr><td>13</td><td>Start button</td></tr><tr><td>14</td><td>Esc button</td></tr><tr><td>15</td><td>Origin button</td></tr><tr><td>16</td><td>Open cover</td></tr><tr><td>17</td><td>Take part</td></tr><tr><td>18</td><td>Close the cover</td></tr><tr><td>19</td><td>Fume extractor off</td></tr><tr><td>20</td><td>Machine off</td></tr></tbody></table>

<table><thead><tr><td>Step name</td><td>Explanation</td></tr></thead><tbody><tr><td>Machine on</td><td>To turn on the machine with the switch key</td></tr><tr><td>Fume extractor on</td><td>To turn on the fume extractor machine</td></tr></tbody></table>
<table><tr><td>Origin button</td><td>To move the laser to its origin. It allows the user to have free space to put or remove the workpiece</td></tr><tr><td>Open cover</td><td>To open the cover of the laser machine</td></tr><tr><td>Put workpiece</td><td>To put the workpiece to be cut in the machine</td></tr><tr><td>Esc button</td><td>To exit the menu mode in the machine</td></tr><tr><td>Arrows</td><td>To move the laser over the workpiece</td></tr><tr><td>Z mode</td><td>To enter into the height adjustment mode to be able to adjust it</td></tr><tr><td>Z adjustment</td><td>To adjust the distance between the lased head and the workpiece with the indicator</td></tr><tr><td>Z validation</td><td>To validate the adjustment height</td></tr><tr><td>Test button</td><td>To verify the contour of the cutting is over the workpiece</td></tr><tr><td>Close cover</td><td>To close the cover of the machine</td></tr><tr><td>Start button</td><td>To start the cutting process</td></tr><tr><td>Fume extractor off</td><td>To turn off the fume extractor</td></tr><tr><td>Machine off</td><td>To turn off the machine</td></tr><tr><td>Take part</td><td>To take the workpiece from the cutting area of the machine.</td></tr></table>

### 3.4.3. Evaluation Phase

The evaluation phase began with an independent session where participants operated the machine and attempted to complete a cutting task without assistance. The experimenter gave the starting signal and observed from a distance, taking notes according to a standardized observation grid.

Following completion of the final task, the participants returned to their computers to complete a follow-up questionnaire or participate in a self-evaluation of their learning experience.

## 3.5. Data Collection and Analysis

Data collection and analysis involved both objective and subjective measurements to compare the two training conditions (AR vs. Control group). To address the research questions and test the hypotheses, we collected user performance data through objective measurements during the Evaluation phase, as described in subsequent subsections. These objective measures were primarily used to examine RQ1 and verify hypothesis H1.1 and H1.2.

In addition, we gathered subjective data through surveys administered after the Training and Evaluation phases using questionnaires based on a 5-point Likert scale. This information was intended to address RQ2 and verify hypotheses H2.1, H2.2, H2.3, and H2.4.

### 3.5.1. Objective Measurements

Objective measurements were collected during the Evaluation phase, where participants performed the cutting task they had learned during the Training phase without assistance. They were instructed to complete the task in the same sequence as it had been taught.

These objective measurements included task completion ability, task completion time, and accuracy of performing the procedure. Specifically, these measurements assessed the participant's ability to successfully perform the cutting task with the machine, replicate the procedure they were taught, and adhere to the prescribed order of steps. The measurements also examined errors, omissions, and requests for help during task execution.

Task completion was defined as a participant's ability to successfully complete the cutting task without significant deviations from the intended procedure. During the Evaluation phase (Figure 2), the instructor observed participants and recorded their performance using a stopwatch to measure task
completion time. Tasks that took more than 10 minutes or required critical intervention from the observer during execution were considered incomplete.

Task completion time was measured from the moment participants turned on the machine until they turned it off. The entire cutting task, when performed by an experienced person, took an average of 105 seconds, including a waiting time of 40 seconds while the machine cut the piece of wood.

Following the correct order of steps was essential during task execution. In the Evaluation phase, the instructor used a checklist to record the order in which the participant completed each step. For example, the "Test button" step was required to be performed after the "Arrows" step. If a participant did not follow the prescribed order, which matched the order taught during training, this was classified as a disorder (i.e., an error in following the procedure).

Table 6 presents an example of the data collected for objective measurements during participant evaluation session. The assessment was based on these data, using specific codes to indicate cases such as errors (E), omissions (F), or requests for help (H).

<table><caption>Table 6. Examples of data collected for two participants during the evaluation step</caption><thead><tr><td>Participant ID</td><td>C1</td><td>Ideal Performance</td></tr></thead><tbody><tr><td>Group</td><td>Control</td><td>Control</td></tr><tr><td>Time</td><td>269 seconds</td><td>105 seconds</td></tr><tr><td>Completed task</td><td>Yes</td><td>Yes</td></tr><tr><td>Machine on</td><td>1</td><td>1</td></tr><tr><td>Fume extractor on</td><td>2</td><td>2</td></tr><tr><td>Origin button</td><td>F</td><td>3</td></tr><tr><td>Open cover</td><td>3</td><td>4</td></tr><tr><td>Put workpiece</td><td>4</td><td>5</td></tr><tr><td>Esc button</td><td>5;10;12</td><td>6</td></tr><tr><td>Arrows</td><td>6</td><td>7</td></tr><tr><td>Z mode</td><td>7</td><td>8</td></tr><tr><td>Z adjustment</td><td>8E</td><td>9</td></tr><tr><td>Z validation</td><td>9</td><td>10</td></tr><tr><td>Test button</td><td>11</td><td>11</td></tr><tr><td>Close cover</td><td>13</td><td>12</td></tr><tr><td>Start button</td><td>14H</td><td>13</td></tr><tr><td>Esc button</td><td>16</td><td>14</td></tr><tr><td>Origin button</td><td>15;17</td><td>15</td></tr><tr><td>Fume extractor off</td><td>18</td><td>16</td></tr><tr><td>Machine off</td><td>19</td><td>17</td></tr></tbody></table>

### 3.5.2. Subjective measurements

In relation to the second research question regarding the perception of the AR learning tool and its impact, subjective measurements were used to verify the four hypotheses mentioned earlier. While the clarity and completeness of the information (H2.3 and H2.4) was evaluated during the Training phase, the utility of the training (H2.1) and the perceived ability resulting from the training (H2.2) were assessed during both the Training and Evaluation phases.

During the Training phase, participants completed a questionnaire specific to their group's training method. The questionnaires shared four common questions, whereas the AR group's questionnaire included two additional questions related to their experience with the application. Table 7 provides details about the questions in the post-training questionnaire for each group.
<table><caption>Table 7. Questions presented after participants finished the training phase</caption><thead><tr><th rowspan="2">ID</th><th colspan="2">Group</th><th rowspan="2">Question</th><th colspan="5">Answering options</th></tr><tr><th>AR</th><th>C</th><th>Useless</th><th>Scarcely useful</th><th>Moderately useful</th><th>Almost useful</th><th>Useful</th></tr></thead><tbody><tr><td>Q1</td><td>X</td><td>X</td><td>To what extent the application/ the instructor is useful to learn to use the machine?</td><td>Useless</td><td>Scarcely useful</td><td>Moderately useful</td><td>Almost useful</td><td>Useful</td></tr><tr><td>Q2</td><td>X</td><td>X</td><td>How much do you feel you are able to perform a cutting task alone?</td><td>Unable</td><td>Barely able</td><td>Moderately able</td><td>Almost able</td><td>Completely Able</td></tr><tr><td>Q3</td><td>X</td><td>X</td><td>How clear the training process is to follow it?</td><td>Not at all clear</td><td>Not much clear</td><td>Understandable</td><td>Very clear</td><td>Limpid</td></tr><tr><td>Q4</td><td>X</td><td>X</td><td>Regarding the training, in what frequency do you feel the need for more explanation via instructor?</td><td>Never</td><td>Rarely</td><td>Sometimes</td><td>Often</td><td>Always</td></tr><tr><td>Q5</td><td>X</td><td></td><td>In what level the application is easy to use?</td><td>Very difficult</td><td>Difficult</td><td>Neutral</td><td>Easy</td><td>Very easy</td></tr><tr><td>Q6</td><td>X</td><td></td><td>In what level the interface of the application is clear and well-structured?</td><td>Very messy</td><td>Messy</td><td>Understandable</td><td>Clear</td><td>Very clear</td></tr></tbody></table>

At the end of the Evaluation phase, participants completed a final questionnaire, which provided self-assessment data on their performance. The questionnaire included three similar questions for both groups, as shown in Table 8. The full procedure and questionnaire details are available online on the OSF platform (Imanipour et al., 2022).

<table><caption>Table 8. Questions presented after participants finished the evaluation phase</caption><thead><tr><th>ID</th><th>Question</th><th colspan="5">Answering options</th></tr></thead><tbody><tr><td>Q7</td><td>How useful do you think the training was in enabling you to carry out the cutting task on your own?</td><td>Useless</td><td>Scarcely useful</td><td>Moderately useful</td><td>Almost useful</td><td>Useful</td></tr><tr><td>Q8</td><td>To what extent do you think the training process was sufficient to perform this task?</td><td>Not at all</td><td>Very little</td><td>More or less</td><td>Rather</td><td>Totally</td></tr><tr><td>Q9</td><td>How would you rate your performance on this task?</td><td>Bad</td><td>Poor</td><td>Average</td><td>Good</td><td>Excellent</td></tr></tbody></table>

The utility of the training, as defined by hypothesis H2.1, was assessed using questions Q1 and Q7. The resulting ability resulting from the training (H2.2) and its change after the Evaluation phase were measured by questions Q2 and Q9. Finally, the clarity of the information (H2.3) and its completeness (H2.4) was evaluated through respectively questions Q3, and Q4.

### 3.5.3. Data Analysis

All data were analyzed using the Python programming language, with packages such as `Pandas`<fnref n="1" />, `NumPy`<fnref n="2" />, `SciPy`<fnref n="3" />, `Pinguoin`<fnref n="4" />, and `plot-likert`<fnref n="5" /> for questionnaire analysis, and `Matplotlib`<fnref n="6" /> for visualization.

<footnote n="1">¹ Pandas python library website: https://pandas.pydata.org/</footnote>

<footnote n="2">² NumPy python library website: https://numpy.org/</footnote>

<footnote n="3">³ SciPy python library website: https://scipy.org/</footnote>

<footnote n="4">⁴ Pingouin python library website: https://pingouin-stats.org/build/html/index.html</footnote>

<footnote n="5">⁵ Plot-likert python library GitHub repository: https://github.com/nmalkin/plot-likert</footnote>

<footnote n="6">⁶ Matplotlib python library website: https:// matplotlib.org/</footnote>
Descriptive statistics were initially computed for variables such as task completion time, error count, and disorder occurrences. Following this, statistical tests were performed to validate the hypothesis. A detailed description of the analysis is available online on OSF (Imanipour et al., 2022).

The independent variables in this study include teaching group and prior experience with AR or machine handling. All other data were treated as dependent variables. Since most dependent variables are ordinal (e.g., Q1 to Q9, based on Likert scale responses), non-parametric tests such as the Mann-Whitney U test were used due to their robustness. The main exceptions are the objective measurements: task completion time is a continuous variable, allowing the use of parametric tests, specifically the t-test. Performance indicators, treated as discrete numerical variables, were also analyzed using parametric methods.

### 3.6. Devices and Development Challenges

The two common display devices used for AR applications are tablets and head-mounted displays (HMDs), each with distinct advantages and limitations. Table 9 compares HMDs with tablets and smartphones. In this study, we utilized the Microsoft HoloLens 2, which offered a hands-free capability, allowing users to perform tasks such as manipulating machine components needing both hands for the task (Ababsa, 2020). This feature was critical for our study, as participants needed both hands for the task.

<table><caption>Table 9. Comparison between common display devices for AR application</caption><thead><tr><th rowspan="2">Parameters</th><th colspan="2">Devices</th></tr><tr><th>Tablets</th><th>HMD</th></tr></thead><tbody><tr><td>Price</td><td>Low</td><td>High</td></tr><tr><td>Hand state</td><td>Busy</td><td>Free</td></tr><tr><td>Direct vision and tangible</td><td>Unable</td><td>Able</td></tr><tr><td>Development</td><td>Easier</td><td>Harder</td></tr></tbody></table>

For the development of XR applications, several game engines are commonly used, with Unity<fnref n="7" /> and Unreal Engine<fnref n="8" /> being the most notable. Unity is a game engine editor that employs C# for scripting and is widely used for AR application development. Unreal Engine, on the other hand, uses C++ for XR application development. In this study, we developed the AR application using the Unity engine and the Mixed Reality Toolkit<fnref n="9" /> (MRTK). MRTK is an open-source project supported by Microsoft that provides components and tools to facilitate cross-platform mixed reality application development in Unity.

During the development process, we encountered two main challenges. The first was related to the tracking of the laser-cutting machine. Given the machine's size, precise tracking with HoloLens 2 was not achievable, which could lead to misalignment of digital content with specific parts of the machine. To resolve this, we implemented a solution using a single QR code attached to the machine as a reference point for positioning the digital content (Figure 3). This approach ensured accurate content alignment during the training phase.

<footnote n="7">⁷ Unity game engine web site: https://unity.com/</footnote>

<footnote n="8">⁸ Unreal Engine web site: https://www.unrealengine.com/en-US</footnote>

<footnote n="9">⁹ MRTK toolkit documentation: https://docs.microsoft.com/en-us/windows/mixed-reality/mrtk-unity/mrtk2/?view=mrtkunity-2022-05</footnote>
<figure><img src="image_5.png" /><figcaption>Figure 3. QR code as origin and coordinate system</figcaption></figure>

The second challenge was the limited field of view of the HMD. Due to the machine's size and the need for users to be close to it while performing tasks, their field of view was constrained. For example, in Figure 4-A, the user's view would not cover the red zone, so any digital content placed there (such as the cyan circle) would be outside their view. The user would only see the triangle. To address this, we implemented an arrow and notification sound to direct the user's attention toward objects outside their immediate view (e.g., the circle in Figure 4-B). Once the object came into view, the arrow disappeared. If the user looked away and the object left their field of view, the arrow and sound would reappear to guide them back to the correct position.

<figure><img src="image_6.png" /><figcaption>Figure 4. Principle of the instruction localization assistant</figcaption></figure>

# 4. Results

## 4.1. Task completion

Out of the 90 participants, 2 did not complete the reproduction phase. In one instance, the experimenter had to activate the emergency button for one participant. In another instance, the observer intervened at nearly every step of the reproduction process. Both participants were in the AR group. The task completion rate was 100% for the control group and 95% for the AR group.

## 4.2. Task completion time

The mean time to complete the laser cutting task was 264 seconds (SD = 78.1) for the AR group and 240 seconds (SD = 81.8) for the control group, as shown in Figure 5. The confidence intervals for the two groups overlap. The initial null hypothesis assumed stating that the mean times for both groups
are equal, and the alternative hypothesis suggests that the mean time for the control group is higher than for the AR group. Visually, it is the case, and this is confirmed by a one-tailed Student's t-test. Given this alternative, the null hypothesis cannot be rejected (p = 0.83, t-value = -0.97, Cohen's d = 0.30).

<figure><img src="image_7.png" /><figcaption>Figure 5. Average reproduction time of inexperienced participants with 95% confidence intervals for the AR and the control group</figcaption></figure>

In the current experiment, reproduction times were equivalent in both groups. Given that only inexperienced individuals were included in the analysis, this suggests that the use of AR had no significant impact on reproduction time, neither positive nor negative. From this perspective, AR may be an interesting alternative to instructor-led training for individual learning.

### 4.3. Performance indicators

Participants' performance during the evaluation phase was assessed using three indicators: the number of help requests, the number of disordered tasks, and the number of errors. Figure 6 presents the mean values for each indicator in both the AR and control groups, completed by Table 10 presenting the averages and standard deviations for each indicator. The unilateral Welch's T-test results are also presented in Table 10, which considered equal performance between the AR and control groups as a null hypothesis ($H1.2_{null}$) and an alternative hypothesis of lower errors in the AR group. The analysis confirmed that the p-value is significantly higher than 0.05, indicating that the null hypothesis is not rejected. This suggests that there is no statistically significant difference between the two groups in terms of performance rates.

However, the unilateral Welch's T-test results on error rates yielded a p-value of 0.98, indicating that there was no significant difference between the AR and control groups in terms of error rates. The extremely low power value (0.00007) suggests that the initial analysis may not be reliable, prompting further investigation. Considering an alternative hypothesis where the AR group had higher error rates than the control group, a new T-test revealed significant but moderate results (p-value = 0.02, t-value = 2.08, Cohen's d = 0.68). Although the p-value suggests that the AR group made more errors, the effect size is moderate (Cohen's d = 0.68) and the statistical power is also moderate (power = 0.70), indicating a significant but moderate difference between the error rates of the two groups, with moderate reliability.
<figure><img src="image_8.png" /><figcaption>Figure 6. Average value of performance indicators for each group of inexperienced participants</figcaption></figure>

<table><caption>Table 10. Comparison of the number of errors between groups of inexperienced participants according to Welch's t-test</caption><thead><tr><td>Performance indicator</td><td>AR group</td><td>Control group</td><td>p-value</td><td>t-value</td><td>power</td></tr></thead><tbody><tr><td>Help</td><td>0.77 ± 1.06</td><td>0.66 ± 0.92</td><td>0.64</td><td>0.36</td><td>0.02</td></tr><tr><td>Disorder</td><td>1.55 ± 1.20</td><td>1.38 ± 1.06</td><td>0.69</td><td>0.50</td><td>0.02</td></tr><tr><td>Error</td><td>1.83 ± 1.54</td><td>0.96 ± 1.04</td><td>0.98</td><td>2.08</td><td>0.00</td></tr></tbody></table>

The actual results of the experiment do not support the hypothesis, which predicted no difference in performance between the AR and control groups. In contrast, previous studies have shown that learners using augmented reality (AR) perform fewer errors than those who do not use AR. Unexpectedly, our finding that a closer examination of the data reveals an opposite effect, where AR-based learners actually exhibited more errors in this study. This suggests that the relationship between AR experience and error production may be influenced by variables other than simply using or not using AR. From this perspective, AR training may not be an alternative as attractive as instructor-led training.

## 4.4. Subjective measurements

Subjective measurements were taken using a questionnaire after the training and evaluation phase. This section will analyse these measurements separately before comparing them.

### 4.4.1. Post training measurement

Figure 7 presents the distribution of responses to the questions asked after the training. For this populations, the trends of the results are relatively similar across both groups, with some differences noted. Table 11 highlights a significant difference notably specifically for question Q4, which measure the need for additional information. A Mann-Whitney U test revealed that $H2.4_{null}$ could be rejected ($p$-value = 0.038, $z$-value = 1.83, effect size = 0.67), indicating that the perceived completeness of the information differs between the AR and control groups as assumed by the alternative hypothesis $H2.4_{alt}$.
For Questions 1, 2, and 3 (Q1, Q2, Q3), which assessed the utility of the training, the self-reported ability to replicate the cutting process, and the clarity of the learning process, both groups provided relatively positive evaluations. However, no significant differences were found between the groups.

<figure><img src="image_9.png" /><figcaption>Figure 7. Post training survey results from inexperienced participants</figcaption></figure>

<table><caption>Table 11. Summary of group comparison for post-training questions using the Mann-Whitney U test for inexperienced participants</caption><thead><tr><td></td><td>AR</td><td>Control</td><td>p-value</td><td>z-value</td><td>effect size</td></tr></thead><tbody><tr><td>Q1</td><td>4.44 ± 0.98</td><td>4.83 ± 0.48</td><td>0.060</td><td>-1.41</td><td>0.37</td></tr><tr><td>Q2</td><td>3.67 ± 0.59</td><td>3.92 ± 0.72</td><td>0.110</td><td>-1.25</td><td>0.39</td></tr><tr><td>Q3</td><td>3.94 ± 0.64</td><td>4.29 ± 0.69</td><td>0.098</td><td>-1.51</td><td>0.36</td></tr><tr><td>Q4</td><td>2.11 ± 0.47</td><td>1.75 ± 0.79</td><td>0.038</td><td>1.83</td><td>0.67</td></tr></tbody></table>

Only considering post-training answers, inexperienced participants in the training perceived it as clear, useful, and judged themselves as able to proceed with a cutting operation independently within their group. Their need for additional information from an instructor was low; however, the responses between the two groups were significantly different. Regarding the answer partition, more AR users required additional information, meaning that the AR training was perceived as less complete than traditional training. As the explanation, description, and process were strictly identical, this difference could be attributed to visual or gestural cues. From this perspective, it is clear that AR training needs to be improved. More specifically, the design of these AR training programs needs to be refined to ensure that they are as effective as instructor-led training.

### 4.4.2. Post evaluation measurements

The survey conducted after the evaluation phase shows similar trends for both groups (Figure 8). While some differences can be noted, none were statistically significant according to the results of a Mann-Whitney U test detailed in Table 12.
<figure><img src="image_10.png" /><figcaption>Figure 8. Post evaluation survey results from inexperienced participants</figcaption></figure>

<table><caption>Table 12. Summary of group comparison for post-evaluation questions using the Mann-Whitney U test for inexperienced participants</caption><thead><tr><th></th><th>AR</th><th>Control</th><th>p-value</th><th>z-value</th><th>Effect size</th></tr></thead><tbody><tr><th>Q7</th><td>4.22 ± 0.94</td><td>4.54 ± 0.72</td><td>0.22</td><td>-1.09</td><td>0.40</td></tr><tr><th>Q8</th><td>4.00 ± 1.03</td><td>3.96 ± 0.91</td><td>0.76</td><td>0.76</td><td>0.53</td></tr><tr><th>Q9</th><td>3.50 ± 0.99</td><td>3.54 ± 0.78</td><td>0.78</td><td>-0.27</td><td>0.48</td></tr></tbody></table>

Once the participants were confronted with the tasks of effectively cutting a piece of wood, they answered to the second questionnaire. Each group evaluated the training similarly, considering it as useful but slightly incomplete in order to properly perform the tasks. They also considered their performance to be between average and good, highlighting that they are not entirely satisfied and are aware of areas for improvement. AR-based training did not make a significant difference compared to instructor-based training, which is actually quite positive for AR training, as it may potentially replace instructor-led training in certain circumstances.

### 4.4.3. Comparing post training and post evaluation measurement

Since the perceived utility of the training was measured before (Q1) and after (Q7) the reproduction task, it is possible to compare the perceived utility within each group over time.

<caption>Figure 9. Evolution of perceived utility before and after the evaluation phase for each group by inexperienced participants</caption>

This figure illustrates this comparison and highlights an evolution in perceived utility as detailed in Table 13. However, according to the results of the Mann-Whitney U test reported in Table 13, none of the before-after scores showed significant differences.
<figure><img src="image_11.png" /><figcaption>Figure 9. Evolution of perceived utility before and after the evaluation phase for each group by inexperienced participants</figcaption></figure>

<table><caption>Table 13. Comparison of utility perception before and after evaluation by inexperienced participants using the Mann-Whitney U method</caption><thead><tr><th></th><th>before</th><th>after</th><th>p-value</th><th>z-value</th><th>Effect size</th></tr></thead><tbody><tr><td>AR</td><td>4.44 ± 0.98</td><td>4.22 ± 0.94</td><td>0.329</td><td>0.89</td><td>0.59</td></tr><tr><td>Control</td><td>4.83 ± 0.48</td><td>4.54 ± 0.72</td><td>0.059</td><td>1.43</td><td>0.62</td></tr></tbody></table>

The results indicate that the training's perceived utility does not undergo significant change after participants interact with and complete a real-world task on the machine. A significant difference would indicate that either the training was not relevant or the machine so simple that the training was not required. It actually means that participant perceived both the difficulty and the interest of the training.

For perceived ability, it is also possible to analyze the change in perception within the groups using questions 2 and 9 (Q2 and Q9). A visualization of the results presented in Figure 10 illustrates a decrease in the perceived ability from both groups after the evaluation session. Assuming an alternative hypothesis that the perceived ability is lower after the reproduction task, the results of a unilateral Mann-Whitney U test provided by Table 14 indicate a significant reduction in the perceived ability within the control group (p-value = 0.018, z-value = 1.79, effect size = 0.65), with a medium effect size.
<figure><img src="image_12.png" /><figcaption>Figure 10. Evolution of the ability perception for each group by inexperienced participants</figcaption></figure>

<table><caption>Table 14. Comparison of ability perception before and after evaluation by inexperienced participants using the Mann-Whitney U method</caption><thead><tr><td></td><td>before</td><td>after</td><td>p-value</td><td>z-value</td><td>Effect size</td></tr></thead><tbody><tr><td>AR</td><td>3.67 ± 0.59</td><td>3.50 ± 0.99</td><td>0.255</td><td>0.62</td><td>0.56</td></tr><tr><td>Control</td><td>3.92 ± 0.72</td><td>3.54 ± 0.78</td><td>0.018</td><td>1.79</td><td>0.65</td></tr></tbody></table>

After having experimented with the machine individually, participants assessed their true abilities. The control group significantly reassessed their abilities, whereas the AR group showed little to no change in their perceptions. It appears that the control group may have overestimated their abilities, potentially because the training modality was a top-down knowledge transfer from the instructor to the learners.

### 4.4.4. AR experience evaluation

To gain a deeper understanding of the results for the AR group, Questions 5 and 6 (Q5, Q6) were asked only of this group. These questions addressed respectively the ease of use of the tool and the clarity of the interface. Both received relatively positive evaluations (Figure 11).
<figure><img src="image_13.png" /><figcaption>Figure 11. Evaluation of the AR experience by AR group</figcaption></figure>

These two questions aimed to investigate whether the performance or subjective measurement
results could be due to the poor design of the AR application. A major limitation of this data is that
participants are inexperienced with this technology, which can lead to two side effects: they may be
susceptible to the novelty effect, potentially impacting their answers, and they lack a comparable
experience to evaluate the ease of use. Given these limitations, evaluations are relatively positive for
the ease of use and clarity of interface. However, it is far from perfect and may influence the perception
of the completeness of the information, which was judged lower for the AR group than the control
group.

**5. Discussion**

This study aimed to address two research questions considering the population with no prior
experience: first, the effect of AR technology on learning outcomes and performance in machinery
handling, and second, the effect of AR on learning perception. Regarding the first research question, it
was hypothesized that the AR group, which interacted directly with the machine, would outperform
the control group in terms of task completion time (H1.1alt) and number of errors (H1.2alt). The results
do not fully support this hypothesis. The inexperienced learner from the AR group did perform better
than the control group in terms of error. However, the control group completed the task faster than
the AR group, contrary to expectations. With the AR group engaged in an active learning context, it
was assumed that interaction with the machine would induce “recall” of actions and thus benefit
learners using XR. However, learners equipped with AR did not achieve better results, so we can
assume the existence of a negative influence factor that impacted completion time. Although the
groups received the same instructions, this difference might be explained by two reasons:

* A difference in terms of contextual and environmental information. As the instructor presented the procedure, he made gestures when interacting with the machine, which provided more information than the AR devices. The participants in the control group had access to these additional cues, including sounds or movements from the machine. Actually, AR training was showing where to act and provided written and auditive guidance, however, the gesture was not represented and the machine feedback was not systematically elicited.
* An automation bias. This bias occurs when learners rely on automation as a heuristic replacement for vigilant information seeking and processing (Mosier and Skitka, 1999). In other words, participants may be less attentive to information transmitted by AR devices, carrying out instructions without actively processing it, like a driver and his GPS (Posey, 2023). In our experiment, it means learners operated the machine without being vigilant about what they were doing and only followed the instruction provided by the AR app.

This finding opens up two potential avenues for future research: compensating for the lack of instructor gestures in AR and reducing automation bias in AR systems.

Although these results contradict some studies in the literature, they are consistent with a more moderate meta-analysis on the effectiveness of XR as a training enhancement tool (Kaplan et al., 2021). These findings suggest that VR, MR, and AR training methods produce performance outcomes comparable to traditional instruction methods.

For the second research question, four hypotheses were proposed based on subjective measurements collected through questionnaires. The first hypothesis posited that the perceived utility of the training (H2.1null) would be equivalent between the two groups. Statistical tests failed to reject the hypothesis. Furthermore, although the perception seemed to evolve over time, the difference between the self-reported utility before and after the evaluation was not significant. This suggests that learner with no prior experience with AR or machinery handling does not overestimate the utility of AR technology.

The second hypothesis suggested that the AR group and control group would have an equivalent perception of their ability (H2.2null). No significant difference was observed in self-reported ability, which means the null hypothesis could not be rejected. However, the perceived ability of the control group decreased significantly after the evaluation phase, with a large effect size. In contrast, the AR group's perception of their ability remained stable over time. This suggests that AR-based learners at least perceived their ability at the same level as those trained by an instructor, with the added benefit of a stable perception over time.

The third hypothesis concerned the clarity of information. Since both groups received the same instructions, there was no expectation of differing perceptions of information clarity between groups (H2.3null). This assumption was largely confirmed, with both groups rating the clarity of information similarly.

However, the fourth hypothesis regarding the completeness of the information revealed complementary information. It was assumed that the perceived completeness of the information will be equivalent between the AR group and the control group (H2.4null). However, the hypothesis was rejected, indicating that there were differences in the need for additional explanation between the AR group and the control group. The results shown by Figure 7 suggests that AR learners required more explanation than the control group. This finding is consistent with the performance results. As previously mentioned, possible reasons for this difference include the lack of non-verbal cues such as gestures or machine feedback, and automation bias.

For the first potential cause, possible improvement would be to integrate virtual hands to visually represent gestures. For the second cause, an experiment aimed at tackling automation bias could involve multi-phase XR training with progressive reduction in information availability, thereby increasing learner vigilance. An alternative explanation for this difference could also be attributed to the novelty effect of AR, which may have distracted from the learning process. However, this is mitigated by the positive evaluation of the AR tool's ease of use and structure reported by the AR learners.
# 6. Conclusion

## 6.1. Findings and implications

The primary goal of this study was to investigate the impact of augmented reality (AR) technology on learning achievement and performance in machinery handling, as well as learners' perceptions of their experience. The results are based on an experimental study in which some learners used an AR training system, while others received instruction from a qualified instructor. Both groups were then asked to perform a common task—operating a laser cutting machine without assistance. To ensure the validity of the experiment, only data from the inexperienced participants in both the AR and machinery handling groups were retained. It was hypothesized that AR technology would enhance learning achievement and performance through a reduction in time and errors during the validation task. However, the results are relatively inconclusive. Inexperienced learners who used AR took as long as learners with the instructor-led training, and made more errors than those who learned with instructor.

Regarding learners' perceptions, four hypotheses were tested. The first hypothesized that AR-based learners and instructor-based learners would have an equivalent perception of the training's utility across both groups, and this hypothesis was not rejected. The second hypothesis posited that learners using AR technology and those receiving instruction from a qualified instructor would perceive their ability equally, but no significant differences in perceptions were measured. Interestingly, the control group's perception of ability decreased significantly, while the AR group's perception remained stable. The third hypothesis assumed that both groups would have equivalent perceptions of information clarity, which was verified. However, the fourth hypothesis, which posited an equivalent perception of the completeness of the information among both groups, was rejected. This suggests that AR users may be missing some crucial information that control group learners possess and could explain the reduced performance of AR learners. It is possible that this lack of information can be balanced by the fact that AR learners directly interact with the machine, leading to a relatively stable evaluation of their ability over time, whereas the control group had more passive training that may have led them to overestimate their abilities. Despite these findings, the overall trends were positive, with high ratings for information clarity and low ratings for the need for additional information.

The study found that AR-based training was equivalent to instructor-based training in various aspects. However, the performance of learners at the end of the training did not meet comparable standards from similar previous work. The results could be attributed to the features, interfaces, and interactions of the AR system used for the experiment, as well as the intrinsic properties of an AR system that trains sequentially. With regard to the design of the AR training, it provided guidance on the elements with which learners should interact, but no guidance on the actions to be performed, whereas the instructor provided this information during the explanation on the machine. Beyond interfaces and interactions limitations, the sequential assisted training approach can introduce "automation bias", which consists of using automation as a heuristic replacement for vigilant information seeking and processing (Mosier and Skitka, 1999).

This finding also aligns with cognitive science research indicating that "learning by doing" can sometimes hinder the learning process (Kirschner et al., 2006), as well as studies warning against excessive cognitive load in AR training when learning materials are not well-designed (Buchner et al., 2021). Recent research comparing information retention in virtual reality simulations versus real-life training has shown similar learning outcomes (Motejlek and Alpay, 2023).

## 6.2. Limitations
It's essential to note several limitations related to this experiment. No specific profile was intentionally targeted, but the experimental context of an academic FabLab mostly brings participants with STEM background. Our sample size was limited by the availability of participants. While the sample size was limited by availability, further restricting our analysis to inexperienced participants has reduced the sample even further. From 88 complete participations, only 32 were kept for the analysis which may affect generalizability. Although only considering inexperienced participants is relevant to assess the benefits of procedure learning through AR and avoid bias from previous training, it means the participants are new to both using AR and machinery such as laser cutting machine which may introduce other variables such as novelty effect or even stress and cognitive workload. It is worth noting that these variables were not explicitly addressed in our initial research design.

## 6.3. Perspectives

These findings open several avenues for future research, particularly on the partial knowledge transfer associated with AR and the potential for automation bias. One research direction could involve the introduction of a virtual hand to mimic operator gestures. Another avenue could explore addressing automation bias by gradually reducing the amount of information provided during training. This could lead to adaptive learning systems in immersive environments, where the system adjusts both the amount of information and the type of cues (audio and text) based on the learner's progress. Additionally, further research could investigate the cognitive load induced by interacting with physical machinery, comparing AR and VR training methods.

## Acknowledgement

The development of the tool that initiated this work was funded by the Grand Est region of France. The authors would like to express their gratitude to Paul Loisil and Antonin Rousseau for their contribution to software development. The authors also extend their appreciation to the Lorraine Fab Living Lab for hosting the experimentation and to Benjamin Ennesser-Serville for providing technical assistance.

We acknowledge the assistance of Language Model (LLM) tools, specifically LLaMA 3.2 via LM Studio, in enhancing the clarity and coherence of this manuscript.

## References

Ababsa, F., 2020. Augmented Reality Application in Manufacturing Industry: Maintenance and Non-destructive Testing (NDT) Use Cases, in: De Paolis, L.T., Bourdot, P. (Eds.), Augmented Reality, Virtual Reality, and Computer Graphics, Lecture Notes in Computer Science. Springer International Publishing, Cham, pp. 333–344. https://doi.org/10.1007/978-3-030-58468-9_24

Akçayır, M., Akçayır, G., 2017. Advantages and challenges associated with augmented reality for education: A systematic review of the literature. Educational Research Review 20, 1–11. https://doi.org/10.1016/j.edurev.2016.11.002

Akçayır, M., Akçayır, G., Pektaş, H.M., Ocak, M.A., 2016. Augmented reality in science laboratories: The effects of augmented reality on university students’ laboratory skills and attitudes toward science laboratories. Computers in Human Behavior 57, 334–342. https://doi.org/10.1016/j.chb.2015.12.054
Ariansyah, D., Pardamean, B., Barbaro, E., Erkoyuncu, J.A., 2024. Augmented reality training for improved learnability. CIRP Journal of Manufacturing Science and Technology 48, 19–27.
https://doi.org/10.1016/j.cirpj.2023.11.003

Arifin, Y., Sastria, T.G., Barlian, E., 2018. User Experience Metric for Augmented Reality Application: A Review. Procedia Computer Science 135, 648–656.
https://doi.org/10.1016/j.procs.2018.08.221

Aschauer, A., Reisner-Kollmann, I., Wolfartsberger, J., 2021. Creating an Open-Source Augmented Reality Remote Support Tool for Industry: Challenges and Learnings. Procedia Computer Science, Proceedings of the 2nd International Conference on Industry 4.0 and Smart Manufacturing (ISM 2020) 180, 269–279.
https://doi.org/10.1016/j.procs.2021.01.164

Borgen, K.B., Ropp, T.D., Weldon, W.T., 2021. Assessment of Augmented Reality Technology's Impact on Speed of Learning and Task Performance in Aeronautical Engineering Technology Education. The International Journal of Aerospace Psychology 31, 219–229.
https://doi.org/10.1080/24721840.2021.1881403

Buchner, J., Buntins, K., Kerres, M., 2021. A systematic map of research characteristics in studies on augmented reality and cognitive load. Computers and Education Open 2, 100036.
https://doi.org/10.1016/j.caeo.2021.100036

Butaslac, I.I.I.I.M., Fujimoto, Y., Sawabe, T., Kanbara, M., Kato, H., 2023. Systematic Review of Augmented Reality Training Systems. IEEE Transactions on Visualization and Computer Graphics 29, 5062–5082.
https://doi.org/10.1109/TVCG.2022.3201120

Campbell, D.T., Stanley, J.C., 1963. Experimental and quasi-experimental designs for research. Wadsworth Publishing Company.

Cárdenas-Robledo, L.A., Hernández-Uribe, Ó., Reta, C., Cantoral-Ceballos, J.A., 2022. Extended reality applications in industry 4.0. – A systematic literature review. Telematics and Informatics 73, 101863.
https://doi.org/10.1016/j.tele.2022.101863

Chardonnet, J.-R., Fromentin, G., Outeiro, J., 2017. Augmented reality as an aid for the use of machine tools. Res. & Sci. Today, 13, 25 7.

Chiang, F.-K., Shang, X., Qiao, L., 2022. Augmented reality in vocational training: A systematic review of research and applications. Computers in Human Behavior 129, 107125.
https://doi.org/10.1016/j.chb.2021.107125

da Silva, M.M.O., Teixeira, J.M.X.N., Cavalcante, P.S., Teichrieb, V., 2019. Perspectives on how to evaluate augmented reality technology tools for education: a systematic review. Journal of the Brazilian Computer Society 25, 3. https://doi.org/10.1186/s13173-019-0084-8

Dhir, A., Al-kahtani, M., 2013. A Case Study on User Experience (UX) Evaluation of Mobile Augmented Reality Prototypes. Journal of Universal Computer Science 19, 1175–1196.
https://doi.org/10.3217/JUCS-019-08-1175

Doolani, S., Wessels, C., Kanal, V., Sevastopoulos, C., Jaiswal, A., Nambiappan, H., Makedon, F., 2020. A Review of Extended Reality (XR) Technologies for Manufacturing Training. Technologies 8, 77.
https://doi.org/10.3390/technologies8040077

Dünser, A., Grasset, R., Billinghurst, M., 2008. A survey of evaluation techniques used in augmented reality studies, in: ACM SIGGRAPH ASIA 2008 Courses on - SIGGRAPH Asia '08. Presented at the ACM SIGGRAPH ASIA 2008 courses, ACM Press, Singapore, pp. 1–27.
https://doi.org/10.1145/1508044.1508049

Eswaran, M., Raju Bahubalendruni, M.V.A., 2024. Augmented reality aided object mapping for worker assistance/training in an industrial assembly context: Exploration of affordance with existing guidance techniques. Comput. Ind. Eng. 185.
https://doi.org/10.1016/j.cie.2023.109663

Eswaran, M., Raju Bahubalendruni, M.V.A., 2023. Augmented reality aided object mapping for worker assistance/training in an industrial assembly context: Exploration of affordance with existing guidance techniques. Computers and Industrial Engineering 185, 109663.
https://doi.org/10.1016/j.cie.2023.109663
Gavish, N., Gutiérrez, T., Webel, S., Rodríguez, J., Peveri, M., Bockholt, U., Tecchia, F., 2015. Evaluating virtual reality and augmented reality training for industrial maintenance and assembly tasks. Interactive Learning Environments 23, 778–798.
https://doi.org/10.1080/10494820.2013.815221

Ibáñez, M.B., Di Serio, Á., Villarán, D., Delgado Kloos, C., 2014. Experimenting with electromagnetism using augmented reality: Impact on flow student experience and educational effectiveness. Computers & Education 71, 1–13. https://doi.org/10.1016/j.compedu.2013.09.004

Imanipour, M., Gabriel, A., Hassan, A., Ricca, A., 2022. Augmented reality application for manufacturing training in machine handling. https://doi.org/10.17605/OSF.IO/7W8VU

Kaplan, A.D., Cruit, J., Endsley, M., Beers, S.M., Sawyer, B.D., Hancock, P.A., 2021. The Effects of Virtual Reality, Augmented Reality, and Mixed Reality as Training Enhancement Methods: A Meta-Analysis. Hum Factors 63, 706–726. https://doi.org/10.1177/0018720820904229

Karagozlu, D., 2018. Determination of the impact of augmented reality application on the success and problem-solving skills of students. Qual Quant 52, 2393–2402.
https://doi.org/10.1007/s11135-017-0674-5

Kirschner, P.A., Sweller, J., Clark, R.E., 2006. Why Minimal Guidance During Instruction Does Not Work: An Analysis of the Failure of Constructivist, Discovery, Problem-Based, Experiential, and Inquiry-Based Teaching. Educational Psychologist 41, 75–86.
https://doi.org/10.1207/s15326985ep4102_1

Kwiatek, C., Sharif, M., Li, S., Haas, C., Walbridge, S., 2019. Impact of augmented reality and spatial cognition on assembly in construction. Automation in Construction 108, 102935.
https://doi.org/10.1016/j.autcon.2019.102935

Leavy, P., 2017. Research design: quantitative, qualitative, mixed methods, arts-based, and community-based participatory research approaches. The Guilford Press, New York London.

Loch, F., Quint, F., Brishtel, I., 2016. Comparing Video and Augmented Reality Assistance in Manual Assembly, in: 2016 12th International Conference on Intelligent Environments (IE). Presented at the 2016 12th International Conference on Intelligent Environments (IE), pp. 147–150.
https://doi.org/10.1109/IE.2016.31

Macchiarella, N.D., Vincenzi, D.A., 2004. Augmented reality in a learning paradigm for flight aerospace maintenance training, in: The 23rd Digital Avionics Systems Conference (IEEE Cat. No.04CH37576). Presented at the The 23rd Digital Avionics Systems Conference (IEEE Cat. No.04CH37576), p. 5.D.1-5.1. https://doi.org/10.1109/DASC.2004.1391342

Michael, J., 2006. Where's the evidence that active learning works? Advances in Physiology Education 30, 159–167. https://doi.org/10.1152/advan.00053.2006

Monroy Reyes, A., Vergara Villegas, O.O., Miranda Bojórquez, E., Cruz Sánchez, V.G., Nandayapa, M., 2016. A mobile augmented reality system to support machinery operations in scholar environments: A MAR SYSTEM TO SUPPORT MACHINERY OPERATIONS IN SCHOLAR ENVIRONMENTS. Comput Appl Eng Educ 24, 967–981. https://doi.org/10.1002/cae.21772

Mosier, K.L., Skitka, L.J., 1999. Automation Use and Automation Bias. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 43, 344–348.
https://doi.org/10.1177/154193129904300346

Motejlek, J., Alpay, E., 2023. The retention of information in virtual reality based engineering simulations. European Journal of Engineering Education 48, 929–948.
https://doi.org/10.1080/03043797.2022.2160968

Mulligan, N.W., Hornstein, S.L., 2003. Memory for actions: Self-performed tasks and the reenactment effect. Memory & Cognition 31, 412–421. https://doi.org/10.3758/BF03194399

Posey, B.M., 2023. The final destination: Incorporating ‘Death by GPS’ into forensic and legal sciences. Science & Justice 63, 421–426. https://doi.org/10.1016/j.scijus.2023.04.005

Rabah, S., Assila, A., Khouri, E., Maier, F., Ababsa, F., bourny, V., Maier, P., Mérienne, F., 2018. Towards improving the future of manufacturing through digital twin and augmented reality technologies. Procedia Manufacturing, 28th International Conference on Flexible Automation and Intelligent Manufacturing (FAIM2018), June 11-14, 2018, Columbus, OH, USAGlobal
Integration of Intelligent Manufacturing and Smart Industry for Good of Humanity 17, 460-
467. https://doi.org/10.1016/j.promfg.2018.10.070

Rios, H., Hincapié, M., Caponio, A., Mercado, E., González Mendivil, E., 2011. Augmented Reality: An
Advantageous Option for Complex Training and Maintenance Operations in Aeronautic
Related Processes, in: Shumaker, R. (Ed.), Virtual and Mixed Reality - New Trends, Lecture
Notes in Computer Science. Springer, Berlin, Heidelberg, pp. 87–96.
https://doi.org/10.1007/978-3-642-22021-0_11

Scaravetti, D., François, R., 2021. Implementation of augmented reality in a mechanical engineering
training context. Computers 10, 1–15. https://doi.org/10.3390/computers10120163

Steffens, M.C., von Stülpnagel, R., Schult, J.C., 2015. Memory Recall After “Learning by Doing” and
“Learning by Viewing”: Boundary Conditions of an Enactment Benefit. Front. Psychol. 6.
https://doi.org/10.3389/fpsyg.2015.01907

Su, W.-H., Chen, K.-Y., Lu, L.Y.Y., Huang, Y.-C., 2021. Identification of Technology Diffusion by Citation and Main Paths Analysis: The Possibility of Measuring Open Innovation. Journal of Open Innovation: Technology, Market, and Complexity 7, 104.
https://doi.org/10.3390/joitmc7010104

Tang, Z., Deng, J., Gao, H., Song, J., Li, Q., Zhang, P., Sun, M., 2019. Using Augmented Reality
Techniques to Simulate Training and Assistance in Electric Power, in: Mao, R., Wang, H., Xie,
X., Lu, Z. (Eds.), Data Science, Communications in Computer and Information Science.
Springer, Singapore, pp. 579–589. https://doi.org/10.1007/978-981-15-0121-0_45

Tekedere, H., Göke, H., 2016. Examining the Effectiveness of Augmented Reality Applications in
Education: A Meta-Analysis. International Journal of Environmental and Science Education
11, 9469–9481.

Torres-Guerrero, F., Neira-Tovar, L., Torres-Treviño, L., 2019. An Introductive Training for Welding
Workshop: A Biometric Evaluation Using Virtual Reality Scenes to Improve Practice, in:
Trojanowska, J., Ciszak, O., Machado, J.M., Pavlenko, I. (Eds.), Advances in Manufacturing II,
Lecture Notes in Mechanical Engineering. Springer International Publishing, Cham, pp. 319–
331. https://doi.org/10.1007/978-3-030-18715-6_27

Tzimas, E., 2018. Machine tool setup instructions in the smart factory using augmented reality: a
system construction perspective. International Journal on Interactive Design and
Manufacturing 16.

Vicente dos Anjos, F.E., Rocha, L.A.O., Oliveira da Silva, D., Pacheco, R., 2021. Impacts of the
Application of Virtual and Augmented Reality on Teaching-Learning Processes in Engineering
Courses. International Journal of Virtual and Personal Learning Environments 12, 1–19.
https://doi.org/10.4018/IJVPLE.291541

Zhang, Z., Li, Z., Han, M., Su, Z., Li, W., Pan, Z., 2021. An augmented reality-based multimedia
environment for experimental education. Multimed Tools Appl 80, 575–590.
https://doi.org/10.1007/s11042-020-09684-x
Appendix 1

<table><thead><tr><td>Authors</td><td>Training field and subject</td><td>Devices used</td><td>Target population</td><td>Evaluation Protocol</td><td>Metrics</td><td>Main conclusion</td></tr></thead><tbody><tr><td>(Macchiarella and Vincenzi, 2004)</td><td>Industrial training – Aerospace maintenance</td><td>Video-based AR</td><td>Undergraduate students</td><td>3 experimental group/1 control group</td><td>immediate-recall test, debrief, long-term-retention-recall test</td><td>This application of AR did not enable learners to recall more information during testing, but it did apparently enable them to recall a greater percentage of what they had learned</td></tr><tr><td>(Rios et al., 2011)</td><td>Industrial training – Aeronautical maintenance</td><td>Camera with pc</td><td>Engineering students</td><td>2 experimental group / 1 control group</td><td>Questionnaire, perf ind. (time, nb errors, nb questions)</td><td>users of the AR application had a better appreciation of spatial and depth concept easier for AR user to understand and perform tasks with small parts AR users perform the steps with a higher grade of confidence</td></tr><tr><td>(Gavish et al., 2015)</td><td>Industrial training – electrical assembly</td><td>Mobile and tablet</td><td>Technicians with 3 years of experience in maintenance tasks</td><td>2 experimental (AR & VR / 2 Control gp (AR & VR))</td><td>Technical capability test, performance time, nb errors, help request, transfer of learning questionnaire, usability questionnaire</td><td>AR training for IMA tasks can reduce the number of unsolved errors</td></tr><tr><td>(Monroy Reyes et al., 2016)</td><td>Manufacturing training – machinery operation</td><td>Mobile and HMD</td><td>mechatronics engineering students, professor and technicians</td><td>Single group, post survey</td><td>Acceptance of application, performance, interface, UX</td><td>good performance, and also a good level of acceptance between mechatronics students, laboratory technicians, and teachers</td></tr><tr><td>(Loch et al., 2016)</td><td>Industrial training – assembly</td><td>Workstation screens</td><td>Undergraduate students (social science & technical disciplines)</td><td>Single group with 2 tasks</td><td>Nb errors, time to complete task, perceived ease of use TAM3, NASA TLX</td><td>AR leads to improvements in accuracy and task performance. reduced the number of errors significantly</td></tr><tr><td>(Chardonnet et al., 2017)</td><td>Assistance for CNC machine</td><td>AR glass and tablets</td><td>Regular CNC users</td><td>Single group with basic milling task</td><td>Interviews about usability, software & hardware manipulation, ergonomics and utility</td><td>easy and practical to use but AR glasses bit heavy and tablet PC depriving users of using their hands. For punctual use.</td></tr><tr><td>(Karagozlu, 2018)</td><td>Science course</td><td>Smartphones</td><td>seventh grade students</td><td>AR group/ control group + pre/post</td><td>Learning achievement and memorization</td><td>achievement scores higher for the AR group at the end of 14 weeks</td></tr><tr><td>(Rabah et al., 2018)</td><td>Electrical wiring training - assembly</td><td>Tablets</td><td>Technicians</td><td>-</td><td>Compilation task, satisfaction</td><td>-</td></tr></tbody></table>
<table><tr><td>(Tzimas, 2018)</td><td>Manufacturing training - machinery setup</td><td>Cameras with sensors</td><td>Operators</td><td>-</td><td>-</td><td>-</td></tr><tr><td>(Kwiatek et al., 2019)</td><td>Industrial training - assembly</td><td>Hand-held AR</td><td>Pipe fitters & engineer</td><td>AR group/ control group</td><td>completion task, error rate, etc.</td><td>Significant shop time reduction &amp; reduce time to interpret and understand the design information</td></tr><tr><td>(Torres-Guerrero et al., 2019)</td><td>Manufacturing training - welding</td><td>VR and AR cabin</td><td>Students</td><td>Experimental/ control group</td><td>Eye tracking and EEG (concentration and stress)</td><td>Experimental group show less stress and more concentration than control group</td></tr><tr><td>(Tang et al., 2019)</td><td>Industrial training - electric power operation</td><td>HoloLens</td><td>Technician</td><td>-</td><td>-</td><td>-</td></tr><tr><td>(Ababsa, 2020)</td><td>Maintenance and assembly</td><td>Tablets</td><td>Aeronautical maintenance technicians</td><td>AR group/ control group</td><td>Time to understand and complete the task, cognitive load (NASA-TLX), UX (SUS)</td><td>Performance improvement with AR, help to focus on action, reduce comprehension time. More beneficial for beginner and confirmed, less for expert</td></tr><tr><td>(Aschauer et al., 2021)</td><td>Maintenance and assembly training</td><td>Tablets and smartphones</td><td>Maintenance technicians</td><td>AR group/ control group</td><td>completion task time, error rate</td><td>Same completion task time but lower error rate for AR user</td></tr><tr><td>(Borgen et al., 2021)</td><td>Industrial training - aviation checklist for safety</td><td>HoloLens</td><td>Aviation students</td><td>AR group/ control group</td><td>completion task time, accuracy</td><td>reduced task times attributed to AR, but reduced pre-task times to control group</td></tr><tr><td>(Scaravetti and François, 2021)</td><td>Mechanical training - kinematics, power transmission, mechanism</td><td>HoloLens and tablets</td><td>Mechanical Engineering students</td><td>AR group/control group</td><td>Knowledge retention and user experience</td><td>Best understanding for AR group</td></tr><tr><td>(Eswaran and Raju Bahubalendruni, 2023)</td><td>Industrial training - assembly</td><td>HoloLens 2</td><td>Engineering students</td><td>AR group/smartphone group/control group</td><td>Completion time, nb of flaws, nb help request, NASA-TLX</td><td>increased assembly task efficiency &amp; minimum number of errors compared to the other guidance modes</td></tr><tr><td>(Ariansyah et al., 2024)</td><td>Electrical wiring training - connection task of a sensor</td><td>HoloLens 2</td><td>Students</td><td>AR group/ enhanced AR group /control group</td><td>Knowledge retention (short and long term)</td><td>higher number of correct answers for AR learners but not significant</td></tr></table>