# A State Validation System for Augmented Reality Based Maintenance Procedures
## 4 Workflow
### Status Analysis  
To properly identify status changes in the tracked model it is necessary to recognize changes in a specific part of the model. At each step of the procedure, the technician will interact with only one element of the target object. Thus, the system will compute the element status prior to the technician interaction, then the final status will be computed when the technician completes the interaction with the target object. In order to obtain two comparable statuses, the technician should guarantee that the position of both the camera and the target object does not change between the two status computation. Otherwise, the system would be unable to correctly compare them. The computation of the feature key points of the whole model could be challenging, depending on its size and on the complexity of the 3D model representing it. For this reason, instead of considering the whole target object, the status comparison only involves the key points pertaining to the position of the element that should change at a given step of the procedure. Thus, it is crucial to associate the feature key points provided by the tracking algorithm to the faces/polygons they are related to, in order to consider only those relevant for a given step. To obtain this result, the tracking algorithm provided by ViSP has been modified as follows: When the object is tracked by the algorithm, all the faces related to the element involved in the current step are projected in 2D, then the minimum and maximum x and y coordinates are computed to define the bounding box area for that element. As a result, it is possible to evaluate if a key point detected in the scene belongs to the element of interest or not. Only the key points whose coordinates are enclosed into the bounding box are taken into account at a given step of the procedure, whereas all the other key points are discarded. Thus, as the tracking algorithm analyzes the scene, all the key points that remain'steady' are used to define the status of the specific component they belong to. A key point is considered'steady' if it maintains the same position for a number of frame equal to the frame rate of the camera. Thus, if the camera frame rate is 24 fps, a key point has to be detected for at least 24 consecutive frames in the same position to be considered'steady'. To evaluate the steadiness, the position of each key point detected at a given frame is compared with the coordinates of the key points detected at the previous frame. If there is a match, then the key point is considered steady and its frame counter is updated. When it reaches a value equal or higher than the frame rate, the key point is marked as steady. Two key points are considered equal if the position of the key point at a given frame is enclosed into a bounding box of 1 pixel with respect to the same key point at the previous frame. This helps to properly account for the errors provided by the camera quality and resolution and the imperceptible movements that may be involved. Figure 5 shows an example of key point detection (left image), positive comparison with a key point from the previous frame (center image) and negative comparison with a key point from the previous frame (right image). Table 1 shows the data structures used to store all the information relevant to identify and to analyze the status of a component. The left table contains the maximum and minimum values for the x and y coordinates of the bounding box defining the component. The right table lists all the feature key points that are identified inside the bounding box at a given time and it is updated at each frame. Each key point with a count value (which is the difference between the first frame and the last frame it was detected in) higher than the camera's frame rate (24 fps) is considered a steady feature (in the example, the first four key points from the top).