# A State Validation System for Augmented Reality Based Maintenance Procedures
## 1 Introduction
Moreover, AR applications for maintenance and repair are often completed by tele-presence systems: The user may request assistance to a remote expert that can interactively support the maintainer when AR aids are not sufficient. Usually, one of the main limitations of this kind of system is that users manually validate each step of the procedure. Figure 1: Examples of augmented reality (AR) applications for maintenance. **Left image** from Office of Naval Research, [[https://www.flickr.com/photos/usnavyresearch/23388133082](https://www.flickr.com/photos/usnavyresearch/23388133082)]([https://www.flickr.com/photos/usnavyresearch/23388133082](https://www.flickr.com/photos/usnavyresearch/23388133082)), **right image** from [PERSON], [[https://commons.wikimedia.org/wiki/File:Entrenamiento-industrial-Fyware.jpg](https://commons.wikimedia.org/wiki/File:Entrenamiento-industrial-Fyware.jpg)]([https://commons.wikimedia.org/wiki/File:Entrenamiento-industrial-Fyware.jpg](https://commons.wikimedia.org/wiki/File:Entrenamiento-industrial-Fyware.jpg)). Thus, they might wrongly execute or skip some steps, resulting in a significant waste of time which would greatly affect the cost of these procedures. To the best of the authors' knowledge, the problem of automatically validating user actions has not yet been addressed by other works in the domain of augmented maintenance. Thus, the novelty of the proposed solution is that the system itself can automatically verify if the user has correctly performed the required operations or not. Maintenance procedures commonly require the user to interact with the machinery adding components, removing components or performing other actions that actually change the visibility of a piece of the machinery from a specific point of view, such as displacements. Framing the machinery from a static point of view, it is possible to recognize the key features that uniquely identify the different parts of the machinery, which are involved in the maintenance procedure. As it is known in advance what the user should do at a given step, it is possible to create a virtual model of the machinery that represents how it should look if the user has correctly performed the task. Thus, analyzing the video stream provided by the camera, it is possible to evaluate each change that should occur to the machinery. Then, comparing the framed scene with the virtual model of the machinery (framed from a virtual position corresponding to the real camera location) it is possible to validate the operation performed by the user. The paper is organized as follows: Section 2 briefly outlines the state of the art of AR applications for maintenance procedures. Section 3 presents a detailed description of the proposed system, with a focus on the computer vision algorithms adopted and the technical problems that arose in the process of the development. Section 4 introduces the use cases selected for the assessment phase and presents the tests performed to evaluate the effectiveness of the system, whereas Section 5 provides an evaluation of the experimental data collected through the tests. Finally, open problems and future works are discussed in Section 6.