# A New Maintainability Evaluation Method Based on Virtual-real Fusion Scene Construction

Authors: Zhexue Ge; Yi Zhang; Yongmin Yang; Xu Luo; Qiang Li; Fang Wang
Journal: Scientific Programming
 (2022)

---

## Abstract

Maintainability is one important attribute of product quality. Maintainability evaluation is important to reduce maintenance cost and improve availability. With this research work, an effective maintainability evaluation method is proposed by mingling virtual environment and real world physical equipment. In order to reduce the cost of maintainability test scenario construction, the surrounding environment of physical equipment is replaced by virtual model for virtual-real fusion. The virtual environment is mainly used to simulate the impact on maintenance operation space. The difficulty of realizing the test concept deals with how to accurately identify the pose of the physical equipment and register the virtual environment around the physical equipment. In the proposed method, initially the ORB feature of the physical product is extracted through binocular vision. Secondly, the ICP method is used to match the physical product feature with the digital prototype feature, so as to identify the relative pose of the physical equipment. The virtual maintenance environment is then accurately superimposed. Thirdly, the experimental evaluation method of virtual reality fusion maintainability qualitative and quantitative indicators is formulated. Finally, taking an engine as an example, a virtual-real fusion maintainability test case study is carried out to verify the effectiveness and feasibility of maintainability evaluation based on virtual-real fusion test scenario. With 5 test groups, evaluation is carried out in three scenes-real environment, virtual-real fusion and without surrounding environment. Results of the evaluation revealed that error rate of the proposed virtual reality fusion maintainability method are somehow closer to maintainability in real environment. The average error rate of the three maintainability qualitative indexes in the virtual-real fusion scene is 9.90%, whereas the average error of the maintenance time is 3.25%. The proposed system offers an absorbing simulation environment where users are privileged to interact with the maintenance objects. The research work bears great significance in maintenance, product designing, visualization and related domains. As maintenance in real environment happens to be costlier and hazardous, the proposed method is suitable to ensure substantial cost reduction and safety in all stages of maintainability. Moreover, outcomes of the evaluation and the generated data of simulation can be generalized to improve maintainability of equipment.


---

## Body

## 1. Introduction

Maintainability evaluation is important to reflect whether product maintenance is convenient, fast and economical [1]. In order to ensure that a product has high availability and low life cycle cost, the product must have good maintainability. This is to reduce the maintenance requirements for manpower, time and resources [2,3]..Therefore, during the development process of industrial products, sufficient maintainability tests must be carried out to verify and evaluate their maintainability and to ensure that they meet the required maintainability requirements.The traditional method of physical maintainability evaluation relies mainly on physical prototype. Such prototypes are not only expensive but are impractical in some scenarios [4]. The method of virtual maintainability simulation evaluation using digital prototypes is difficult to design. Due to the difficulties involved in human-machine force interaction, such methods often fail to accurately evaluate the maintenance force characteristics and maintenance time indicators. However, with virtual-real fusion, real world and virtual world can be represented in one single system at a time. With the fusion, a synthetic scene is augmented by realist objects and information extension is offered for real scenes. In the field of maintenance and assembly, the application of virtual-real fusion has made a considerable progress. Deshpande [5] designed an AR-assisted visual feature system with interactive modes for Ready-to-assemble (RTA) furniture. The application utilizes Microsoft Hololens™ headsets to enable users in timely conceiving the spatial relationship of components. The system also supports assembly tasks that otherwise require high spatial knowledge. For the first time in the literature, a number of users examined the system on RTA furniture. Vicomtech proposed the creation method using an AR workspace. The system offers interaction and visualization modes and provides more effective means for the assembly task of hybrid man-machine production [6]. It is proved that virtual-real fusion based maintainability test is advantageous in terms of accuracy and economy. With the nexus of virtual scene and real objects, a satisfactory reduction in hardware scale is promised. Hence, the virtual-real fusion based maintainability test offers a high application prospect. The key issue in such systems is the integration of physical equipment and virtual environment in accordance with the actual positional relationship. The three-dimensional pose of physical equipment must be accurately identified and need to be properly superimposed in the virtual environment. This paper focuses on accurately identifying positions of physical equipment and appropriate superimposing of real objects in a synthetic environment. Moreover, the paper presents applications of maintainability test in a systematic way. The key issues involved in the maintainability assessment are also discussed to pave the way for an optimize product design.Rest of the paper is organized into 4 sections. In Section2, related work from the literature is discussed. Overall solution of the proposed method is covered in Section 3. Section 4 deals with technological implementation while experimental verification is presented in Section 5. Conclusion and future strategy of the research is discussed in Section 6.


## 2. Literature Review

Maintainability refers to the possibility of restoring a failed item to operative condition within a given period of time [7]. Like proper planning, maintainability is important in product designing in general and in complex products particularly [8]. An appropriate maintainability scheme not only reduces costs but also reduces the chances of incidents and accidents in the product life cycle. It is why researchers are emphasizing to pay due consideration for planning and designing of maintainability testing [9]. Since last decade, the use of Virtual Reality (VR) technology is on the rise in the realm of maintainability. VR is of great significance in product designing and manufacturing sectors [10]. The technology enables users to interact with virtual objects while employing sensual perceptions. A system that utilizes VR for the enhancement of maintainability design is referred to as Virtual maintenance (VM). VM is a sort of virtual simulation-based engineering application that enables engineers to plan, analyze and evaluate the assembly of mechanical systems [11]. VM should satisfy several functional requirements [12]. The desktop based system, Virtual Maintenance Simulation of the Nanyang Technological University [13] is a novel sequence planning technology. The system uses an optimization algorithm for effective and feasible planning. The ergonomic based assessment method of Sanchez [14] aims to assess the ergonomics principals in a workplace. Different Software application is proposed by some researchers [15] for design, analysis and verification of effective maintainability. Integrating the technologies of Virtual and Augmented Reality, a system based on remote handling (RH) is proposed in [16]. It is claimed that the system helps the operator in maintenance of the Tokamak. It is added that the system ensures that all the processes of inspections and maintenance are performed in a safe environment. In [17] a 3D workforce training system is proposed for high-voltage power line maintenance. The study offers a risk-free environment for learning the practice and procedures related to line maintenance. The research also evaluated the tool findings of which suggest that the workforce training system has a positive effect in the overall process of maintenance. As suggested in [18], the mingling of virtual reality technology with the traditional approach of maintainability has a positive impact in product design optimization. However, the use of advance technology in maintainability evaluation is rarely focused, hence this research.


## 3. Overall Solution

In the process of a virtual-real fusion maintainability test, usually a set of digital prototypes of the product is provided. The set of prototypes is treated as the key input to the test. The digital prototypes reflect the relationship between the physical product and the surrounding environment. In order to superimpose the virtual maintenance environment model on the periphery of the physical product object it is necessary to identify the physical product. Moreover, to make it sure that an object is a part of the maintenance environment, the virtual world should be fully aligned with the physical world. Since the current AR glasses have the function of maintaining anchoring after superimposing virtual scenes, the focus of virtual registration is how to achieve initial recognition and registration. Once the registration is successful, they can be anchored so as to adapt to the posture changes of the person and the binocular glasses during the test. In this paper, the binocular camera is used to obtain the video stream of the real maintenance scene. The characteristics of the video image are extracted on the basis of calibrating the internal parameters of the camera. The transformation matrix is solved for pose estimation. Next, the virtual scene is registered to the real scene through coordinate transformation to complete the construction of virtual-real fusion maintainability test scene. The overall process is shown in Figure1.Figure 1
Overall process of maintainability assessment based on virtual-real fusion.


## 4. Technology Implementation

This section is divided into subheadings that provide a concise and precise description of the experimental results, their interpretation, as well as the experimental conclusions that can be drawn.


### 4.1. Image Feature Extraction of Maintainability Test Object Based on ORB

At present, many local features such as SIFT, SURF, ORB, BRISK, FREAK, etc. are widely used in the fields of image matching and object recognition [19]. Since the objects involved in the process of maintainability test are usually of a mechanical product, their surfaces sometimes lack rich texture features. Considering the stability and rapidity based on feature point extraction and matching, the ORB local feature is selected in the proposed approach. ORB local features use FAST as the feature point detector, the improved BRIEF as the feature descriptor, and use the BF pattern matching algorithm for feature descriptor matching [20].FAST feature points are not directional, and the directional parameters are determined by obtaining the center of gravity of the feature point neighborhood. The neighborhood moment is given as:(1)mpq=∑x,yxpyqIx,y,where Ix,y is the gray value at point x,y, x,y∈−r,r, r is the radius of the circle, p and q are non-negative integers. When p is 1 and q is 0, the value Ix of I in the x direction can be obtained. Similarly, if p is 0 and q is 1, the value Iy of I in the y direction can be obtained. C being the image center of gravity can be obtained as:(2)C=m10m00,m01m00.The angle between the feature point and the center of gravity is defined as the direction of the FAST feature point, given as:(3)θ=arctanm01m10=arctan∑x,yyIx,y∑x,yxIx,y.ORB extracts the BRIEF descriptor according to the direction parameters obtained in the above formula. However, due to environmental factors and the introduction of noise, the direction of feature points will change. Furthermore, the correlation of random pixel block pairs will be relatively large, thereby reducing the discrimination of the descriptor. ORB adopts a greedy algorithm to find random pixel block pairs with low correlation. Generally, 256 pixel block pairs with the lowest correlation are selected to form a 256 bit feature descriptor. The two descriptors are given as follows,(4)K1=x0x1⋯x255,K2=y0y1⋯y255.


### 4.2. Matching of Physical Equipment Characteristics and Virtual Environment Registration

The ORB feature set is extracted from the real maintainability test object and the virtual maintenance environment model. The corresponding feature descriptorsK1,K2 are obtained for onward processing. The similarity between two ORB feature descriptors is characterized by the sum of the exclusive ORB Hamming distances:(5)DK1,K2=∑i=0255xi⊕yi.The smaller theDK1,K2, the higher the similarity, and the greater the probability that the two describe the same feature. Conversely, the lower the similarity, the more likely they are not describing the same feature.The BF matcher is used to get all the possible matching feature pairs, assuming that the minimum Hamming distance of feature pairs is MIN_DIST. In order to select the best matching pair and improve the operating efficiency, an appropriate threshold is selected and the matching pair smaller than the threshold is selected for the next camera pose estimation. The threshold value cannot be too small, as it will affect the final effect. It is therefore necessary to select the best threshold value through experiments on the image frame.Given the pointk1i in K1, find the point k2i with the shortest Euclidean distance of k1i from K2, and take k1i and k2i as the corresponding points to obtain the transformation matrix. Through continuous iteration, the following formula is minimized and the iteration is terminated. Finally the most Optimal transformation matrix is obtained to make them coincide.(6)fR,T=1n∑i=1nk1i−Rk2i+T2.In the formula,R indicates the rotary transform matrix and T indicates the translation form of the matrix.The essence of the ICP algorithm is to calculate the transformation matrix between the feature sets, minimize the registration error between the two through rotation and translation and then achieve the best registration effect[21]. Assuming the two feature point sets K1=k1i∈R3,i=1,2,…,n and K2=k2i∈R3,i=1,2,…,n, the registration process using the ICP algorithm is introduced below:(1)
Sample setK1, K10⊂K1, K10 represents a subset of the set K1;(2)
Search in setK2, find the closest point to each point in K10, and get the initial correspondence between K1 and K2;(3)
Remove the wrong corresponding point pairs using algorithms or constraints;(4)
Calculate the transformation relationship between the two according to the corresponding relationship in step (2). Minimize the value of the objective function and apply the calculated transformation matrix toK10 to obtain the changed matrix K10′;(5)
Determine whether the iteration is terminated according tod=1/n∑i=1nK2i−K1i2. If d is greater than the preset threshold, return to step (2) to continue the iteration. If d is less than the preset threshold or reach the set number of iterations, the iteration stops.By obtaining the transformation matrix through the above steps, the pose transformation relationship between the physical equipment and the virtual maintainability test environment can be obtained. Following that, virtual registration can be performed to complete the construction of the virtual-real fusion maintainability test environment.


### 4.3. Maintainability Index Evaluation Based on the Fusion of Virtual and Real Information

In the process of the virtual-real fusion maintainability test, the main maintenance test operation is carried out on physical products and the maintenance environment rendered by the virtual prototype can well simulate the spatial characteristics of the maintenance process. Therefore, it is reasonable to believe that the virtual-real fusion maintainability test scene is basically equivalent to the actual maintenance scene. This in turns prove that the propose test satisfies the standards of accurate maintainability evaluation. The maintainability qualitative indicators to be evaluated usually include visibility, accessibility, operating space and comfort, etc. which are considered to be the standard qualitative indicators [22]. In order to evaluate more finely, it is necessary to develop a reasonable evaluation grade method. The quantitative indicators of maintainability include Mean Time to Repair (MTTR), mean preventive maintenance time, maintenance ratio, maintenance degree and repair rate, etc. Next, we mainly introduce the evaluation methods of some of the main indicators.


#### 4.3.1. Maintenance Visibility

Maintenance visibility refers to the visibility of the maintenance parts during maintenance. The good visibility of the maintenance parts is convenient for maintenance personnel to observe the service condition of the components and to complete maintenance tasks easily by reducing the difficulties involved. On the contrary, if the part to be repaired is invisible or difficult to see, it will greatly increase the difficulty of the maintenance work. Moreover, fatigue in maintenance will increase that will badly affect the completion of maintenance tasks. The values corresponding to the subjective evaluation of visibility index are shown in Table1.Table 1
Evaluation value and standard of visibility index.
The evaluation criteria of visibility indexThe value of visibility indexThe repair part is visible, the vertical viewing angle is within 15 degrees of the normal line of sight and the horizontal viewing angle is within 10 degrees of the center line0.8–1.0The repair part is visible, the vertical viewing angle is within 15 degrees of the normal line of sight or the horizontal viewing angle is within 10 degrees of the center line0.7The repair part is visible, the vertical viewing angle is 15 degrees away from the normal line of sight or the horizontal viewing angle is 10 degrees away from the center line0.5The repair part is visible, the vertical viewing angle is 15 degrees away from the normal line of sight and the horizontal viewing angle is 10 degrees away from the center line0.3The repair part is invisible0.1


#### 4.3.2. Maintenance Accessibility

Maintenance accessibility refers to the degree of difficulty a maintenance personnel faces while accessing the maintenance part during equipment maintenance. The accessibility can reflects the impact of work space, maintenance approach, and component layout on maintenance operations. Accessibility directly affects equipment inspection and maintenance. Therefore, full attention should be paid to maintenance accessibility. The values corresponding to the subjective evaluation of accessibility index are shown in Table2.Table 2
Evaluation value and standard of accessibility index.
The evaluation criteria of accessibility indexThe value of accessibility indexTools/hands are fully accessible to the repair parts and the joint angle does not exceed 10% of the central angle0.8–1.0Tools/hands are fully accessible to the repair parts and the joint angle exceeds the central angle by 10% but not more than 30%0.7Tools/hands are fully accessible to the repair parts and the joint angle exceeds the central angle by 30% but not more than50%0.5Tools/hands are fully accessible to the repair parts and the joint angle exceeds the central angle by 50% but not more than 70%0.3Tools/hands are not allowed to touch the repair parts0.1


#### 4.3.3. Operating Space

The operating space is related to the feasibility and simplicity of the maintenance. An effective operating space is mainly guaranteed by the installation and layout of equipment or parts. The equipment must have a certain space to facilitate maintenance operation. Similarly, the values corresponding to the subjective evaluation of operating space index are shown in Table3.Table 3
Evaluation value and standard of operating space index.
The evaluation criteria of operating space indexThe value of operating space indexThere are no difficulties in maintenance operation and there are no obstacles in the maintenance process0.8–1.0The maintenance operation is less difficult and there are a few obstacles in the maintenance process, which has a small impact on the operation space0.7The maintenance operation is generally difficult and there are some obstacles in the maintenance process, which limits the activity range of maintenance tools/hands to a certain extent0.5The maintenance operation is very difficult and there are many obstacles in the maintenance process, which greatly limits the activity range of maintenance tools/hands0.3The maintenance operation cannot be carried out and the maintenance process is blocked by obstacles0.1


#### 4.3.4. Mean Time to Repair

Mean Time to Repair is the average value of the actual repair time required for troubleshooting. It is a basic parameter of the equipment maintainability, also known as the average repair time. The parameter is usually measured by the cumulative time estimation method [23].In the cumulative model, the basic maintenance operations are synthesized into maintenance activity timeTmnj. The maintenance activity time is synthesized into the average repair time Rnj of each main replaceable unit under each fault detection and isolation output (n represents unit n and j represents output j), represented as:(7)Rnj=∑j=1MnjTmnj,where Mnj is the number of activities for troubleshooting. The maintenance after the nth−RI has a fault and is detected by the jthFD&I including various maintenance activities, i.e. preparation, isolation. It may include operations of other RIs detected in the jth fault isolation result (example, determine the jth, RI with alternating replacement).Tmnj is the average time of the mth troubleshooting and maintenance activities for the nth−RI detected by the jth, FD&I output. The mean value model to solve the average repair time is given as;(8)M¯ct=∑n=1NλnRn∑n=1Nλn.


## 4.1. Image Feature Extraction of Maintainability Test Object Based on ORB

At present, many local features such as SIFT, SURF, ORB, BRISK, FREAK, etc. are widely used in the fields of image matching and object recognition [19]. Since the objects involved in the process of maintainability test are usually of a mechanical product, their surfaces sometimes lack rich texture features. Considering the stability and rapidity based on feature point extraction and matching, the ORB local feature is selected in the proposed approach. ORB local features use FAST as the feature point detector, the improved BRIEF as the feature descriptor, and use the BF pattern matching algorithm for feature descriptor matching [20].FAST feature points are not directional, and the directional parameters are determined by obtaining the center of gravity of the feature point neighborhood. The neighborhood moment is given as:(1)mpq=∑x,yxpyqIx,y,where Ix,y is the gray value at point x,y, x,y∈−r,r, r is the radius of the circle, p and q are non-negative integers. When p is 1 and q is 0, the value Ix of I in the x direction can be obtained. Similarly, if p is 0 and q is 1, the value Iy of I in the y direction can be obtained. C being the image center of gravity can be obtained as:(2)C=m10m00,m01m00.The angle between the feature point and the center of gravity is defined as the direction of the FAST feature point, given as:(3)θ=arctanm01m10=arctan∑x,yyIx,y∑x,yxIx,y.ORB extracts the BRIEF descriptor according to the direction parameters obtained in the above formula. However, due to environmental factors and the introduction of noise, the direction of feature points will change. Furthermore, the correlation of random pixel block pairs will be relatively large, thereby reducing the discrimination of the descriptor. ORB adopts a greedy algorithm to find random pixel block pairs with low correlation. Generally, 256 pixel block pairs with the lowest correlation are selected to form a 256 bit feature descriptor. The two descriptors are given as follows,(4)K1=x0x1⋯x255,K2=y0y1⋯y255.


## 4.2. Matching of Physical Equipment Characteristics and Virtual Environment Registration

The ORB feature set is extracted from the real maintainability test object and the virtual maintenance environment model. The corresponding feature descriptorsK1,K2 are obtained for onward processing. The similarity between two ORB feature descriptors is characterized by the sum of the exclusive ORB Hamming distances:(5)DK1,K2=∑i=0255xi⊕yi.The smaller theDK1,K2, the higher the similarity, and the greater the probability that the two describe the same feature. Conversely, the lower the similarity, the more likely they are not describing the same feature.The BF matcher is used to get all the possible matching feature pairs, assuming that the minimum Hamming distance of feature pairs is MIN_DIST. In order to select the best matching pair and improve the operating efficiency, an appropriate threshold is selected and the matching pair smaller than the threshold is selected for the next camera pose estimation. The threshold value cannot be too small, as it will affect the final effect. It is therefore necessary to select the best threshold value through experiments on the image frame.Given the pointk1i in K1, find the point k2i with the shortest Euclidean distance of k1i from K2, and take k1i and k2i as the corresponding points to obtain the transformation matrix. Through continuous iteration, the following formula is minimized and the iteration is terminated. Finally the most Optimal transformation matrix is obtained to make them coincide.(6)fR,T=1n∑i=1nk1i−Rk2i+T2.In the formula,R indicates the rotary transform matrix and T indicates the translation form of the matrix.The essence of the ICP algorithm is to calculate the transformation matrix between the feature sets, minimize the registration error between the two through rotation and translation and then achieve the best registration effect[21]. Assuming the two feature point sets K1=k1i∈R3,i=1,2,…,n and K2=k2i∈R3,i=1,2,…,n, the registration process using the ICP algorithm is introduced below:(1)
Sample setK1, K10⊂K1, K10 represents a subset of the set K1;(2)
Search in setK2, find the closest point to each point in K10, and get the initial correspondence between K1 and K2;(3)
Remove the wrong corresponding point pairs using algorithms or constraints;(4)
Calculate the transformation relationship between the two according to the corresponding relationship in step (2). Minimize the value of the objective function and apply the calculated transformation matrix toK10 to obtain the changed matrix K10′;(5)
Determine whether the iteration is terminated according tod=1/n∑i=1nK2i−K1i2. If d is greater than the preset threshold, return to step (2) to continue the iteration. If d is less than the preset threshold or reach the set number of iterations, the iteration stops.By obtaining the transformation matrix through the above steps, the pose transformation relationship between the physical equipment and the virtual maintainability test environment can be obtained. Following that, virtual registration can be performed to complete the construction of the virtual-real fusion maintainability test environment.


## 4.3. Maintainability Index Evaluation Based on the Fusion of Virtual and Real Information

In the process of the virtual-real fusion maintainability test, the main maintenance test operation is carried out on physical products and the maintenance environment rendered by the virtual prototype can well simulate the spatial characteristics of the maintenance process. Therefore, it is reasonable to believe that the virtual-real fusion maintainability test scene is basically equivalent to the actual maintenance scene. This in turns prove that the propose test satisfies the standards of accurate maintainability evaluation. The maintainability qualitative indicators to be evaluated usually include visibility, accessibility, operating space and comfort, etc. which are considered to be the standard qualitative indicators [22]. In order to evaluate more finely, it is necessary to develop a reasonable evaluation grade method. The quantitative indicators of maintainability include Mean Time to Repair (MTTR), mean preventive maintenance time, maintenance ratio, maintenance degree and repair rate, etc. Next, we mainly introduce the evaluation methods of some of the main indicators.


### 4.3.1. Maintenance Visibility

Maintenance visibility refers to the visibility of the maintenance parts during maintenance. The good visibility of the maintenance parts is convenient for maintenance personnel to observe the service condition of the components and to complete maintenance tasks easily by reducing the difficulties involved. On the contrary, if the part to be repaired is invisible or difficult to see, it will greatly increase the difficulty of the maintenance work. Moreover, fatigue in maintenance will increase that will badly affect the completion of maintenance tasks. The values corresponding to the subjective evaluation of visibility index are shown in Table1.Table 1
Evaluation value and standard of visibility index.
The evaluation criteria of visibility indexThe value of visibility indexThe repair part is visible, the vertical viewing angle is within 15 degrees of the normal line of sight and the horizontal viewing angle is within 10 degrees of the center line0.8–1.0The repair part is visible, the vertical viewing angle is within 15 degrees of the normal line of sight or the horizontal viewing angle is within 10 degrees of the center line0.7The repair part is visible, the vertical viewing angle is 15 degrees away from the normal line of sight or the horizontal viewing angle is 10 degrees away from the center line0.5The repair part is visible, the vertical viewing angle is 15 degrees away from the normal line of sight and the horizontal viewing angle is 10 degrees away from the center line0.3The repair part is invisible0.1


### 4.3.2. Maintenance Accessibility

Maintenance accessibility refers to the degree of difficulty a maintenance personnel faces while accessing the maintenance part during equipment maintenance. The accessibility can reflects the impact of work space, maintenance approach, and component layout on maintenance operations. Accessibility directly affects equipment inspection and maintenance. Therefore, full attention should be paid to maintenance accessibility. The values corresponding to the subjective evaluation of accessibility index are shown in Table2.Table 2
Evaluation value and standard of accessibility index.
The evaluation criteria of accessibility indexThe value of accessibility indexTools/hands are fully accessible to the repair parts and the joint angle does not exceed 10% of the central angle0.8–1.0Tools/hands are fully accessible to the repair parts and the joint angle exceeds the central angle by 10% but not more than 30%0.7Tools/hands are fully accessible to the repair parts and the joint angle exceeds the central angle by 30% but not more than50%0.5Tools/hands are fully accessible to the repair parts and the joint angle exceeds the central angle by 50% but not more than 70%0.3Tools/hands are not allowed to touch the repair parts0.1


### 4.3.3. Operating Space

The operating space is related to the feasibility and simplicity of the maintenance. An effective operating space is mainly guaranteed by the installation and layout of equipment or parts. The equipment must have a certain space to facilitate maintenance operation. Similarly, the values corresponding to the subjective evaluation of operating space index are shown in Table3.Table 3
Evaluation value and standard of operating space index.
The evaluation criteria of operating space indexThe value of operating space indexThere are no difficulties in maintenance operation and there are no obstacles in the maintenance process0.8–1.0The maintenance operation is less difficult and there are a few obstacles in the maintenance process, which has a small impact on the operation space0.7The maintenance operation is generally difficult and there are some obstacles in the maintenance process, which limits the activity range of maintenance tools/hands to a certain extent0.5The maintenance operation is very difficult and there are many obstacles in the maintenance process, which greatly limits the activity range of maintenance tools/hands0.3The maintenance operation cannot be carried out and the maintenance process is blocked by obstacles0.1


### 4.3.4. Mean Time to Repair

Mean Time to Repair is the average value of the actual repair time required for troubleshooting. It is a basic parameter of the equipment maintainability, also known as the average repair time. The parameter is usually measured by the cumulative time estimation method [23].In the cumulative model, the basic maintenance operations are synthesized into maintenance activity timeTmnj. The maintenance activity time is synthesized into the average repair time Rnj of each main replaceable unit under each fault detection and isolation output (n represents unit n and j represents output j), represented as:(7)Rnj=∑j=1MnjTmnj,where Mnj is the number of activities for troubleshooting. The maintenance after the nth−RI has a fault and is detected by the jthFD&I including various maintenance activities, i.e. preparation, isolation. It may include operations of other RIs detected in the jth fault isolation result (example, determine the jth, RI with alternating replacement).Tmnj is the average time of the mth troubleshooting and maintenance activities for the nth−RI detected by the jth, FD&I output. The mean value model to solve the average repair time is given as;(8)M¯ct=∑n=1NλnRn∑n=1Nλn.


## 4.3.1. Maintenance Visibility

Maintenance visibility refers to the visibility of the maintenance parts during maintenance. The good visibility of the maintenance parts is convenient for maintenance personnel to observe the service condition of the components and to complete maintenance tasks easily by reducing the difficulties involved. On the contrary, if the part to be repaired is invisible or difficult to see, it will greatly increase the difficulty of the maintenance work. Moreover, fatigue in maintenance will increase that will badly affect the completion of maintenance tasks. The values corresponding to the subjective evaluation of visibility index are shown in Table1.Table 1
Evaluation value and standard of visibility index.
The evaluation criteria of visibility indexThe value of visibility indexThe repair part is visible, the vertical viewing angle is within 15 degrees of the normal line of sight and the horizontal viewing angle is within 10 degrees of the center line0.8–1.0The repair part is visible, the vertical viewing angle is within 15 degrees of the normal line of sight or the horizontal viewing angle is within 10 degrees of the center line0.7The repair part is visible, the vertical viewing angle is 15 degrees away from the normal line of sight or the horizontal viewing angle is 10 degrees away from the center line0.5The repair part is visible, the vertical viewing angle is 15 degrees away from the normal line of sight and the horizontal viewing angle is 10 degrees away from the center line0.3The repair part is invisible0.1


## 4.3.2. Maintenance Accessibility

Maintenance accessibility refers to the degree of difficulty a maintenance personnel faces while accessing the maintenance part during equipment maintenance. The accessibility can reflects the impact of work space, maintenance approach, and component layout on maintenance operations. Accessibility directly affects equipment inspection and maintenance. Therefore, full attention should be paid to maintenance accessibility. The values corresponding to the subjective evaluation of accessibility index are shown in Table2.Table 2
Evaluation value and standard of accessibility index.
The evaluation criteria of accessibility indexThe value of accessibility indexTools/hands are fully accessible to the repair parts and the joint angle does not exceed 10% of the central angle0.8–1.0Tools/hands are fully accessible to the repair parts and the joint angle exceeds the central angle by 10% but not more than 30%0.7Tools/hands are fully accessible to the repair parts and the joint angle exceeds the central angle by 30% but not more than50%0.5Tools/hands are fully accessible to the repair parts and the joint angle exceeds the central angle by 50% but not more than 70%0.3Tools/hands are not allowed to touch the repair parts0.1


## 4.3.3. Operating Space

The operating space is related to the feasibility and simplicity of the maintenance. An effective operating space is mainly guaranteed by the installation and layout of equipment or parts. The equipment must have a certain space to facilitate maintenance operation. Similarly, the values corresponding to the subjective evaluation of operating space index are shown in Table3.Table 3
Evaluation value and standard of operating space index.
The evaluation criteria of operating space indexThe value of operating space indexThere are no difficulties in maintenance operation and there are no obstacles in the maintenance process0.8–1.0The maintenance operation is less difficult and there are a few obstacles in the maintenance process, which has a small impact on the operation space0.7The maintenance operation is generally difficult and there are some obstacles in the maintenance process, which limits the activity range of maintenance tools/hands to a certain extent0.5The maintenance operation is very difficult and there are many obstacles in the maintenance process, which greatly limits the activity range of maintenance tools/hands0.3The maintenance operation cannot be carried out and the maintenance process is blocked by obstacles0.1


## 4.3.4. Mean Time to Repair

Mean Time to Repair is the average value of the actual repair time required for troubleshooting. It is a basic parameter of the equipment maintainability, also known as the average repair time. The parameter is usually measured by the cumulative time estimation method [23].In the cumulative model, the basic maintenance operations are synthesized into maintenance activity timeTmnj. The maintenance activity time is synthesized into the average repair time Rnj of each main replaceable unit under each fault detection and isolation output (n represents unit n and j represents output j), represented as:(7)Rnj=∑j=1MnjTmnj,where Mnj is the number of activities for troubleshooting. The maintenance after the nth−RI has a fault and is detected by the jthFD&I including various maintenance activities, i.e. preparation, isolation. It may include operations of other RIs detected in the jth fault isolation result (example, determine the jth, RI with alternating replacement).Tmnj is the average time of the mth troubleshooting and maintenance activities for the nth−RI detected by the jth, FD&I output. The mean value model to solve the average repair time is given as;(8)M¯ct=∑n=1NλnRn∑n=1Nλn.


## 5. Experimental Verification

Taking the auxiliary engine room of a ship as a case study, the test verification is carried out to verify the correctness and applicability of the proposed virtual-real fusion maintainability test evaluation method. The auxiliary engine room is powered by a diesel engine, which is composed of a crank connecting rod mechanism, a gas distribution structure, a fuel system, a lubrication system, a cooling system and a starting system, etc. The engine needs to replace consumable parts such as fuel filter and air filter. The cylinder and starter motor have a certain failure rate. It needs to be well designed for maintenance to ensure rapid maintenance at the crew level.In the ship cabin environment, the equipment maintenance process has a certain level of complexity. The other equipment around the peripheral pipelines and cables are easy to cause insufficient accessibility of the maintenance objects and inadequate operating space. Therefore, in the process of maintainability test of the engine, it is necessary to be able to simulate actual cabin maintenance scenes and maintenance space. Moreover, the impact of various operational obstacles on maintainability needs to be fully considered so as to obtain more accurate maintainability test results.Since for the establishment of a 1 : 1 full-physical maintainability test condition is very costly and has a long cycle, therefore, the proposed virtual-real fusion maintainability test evaluation method is adopted. A small part of the physical equipment and an extended number of virtual scenes are used to realistically simulate a complete test scenario. During the evaluation, the test conditions used include the YN92 physical diesel engine and the complete digital model of the auxiliary engine compartment, as shown in Figures2 and 3. To verify the outcomes, repairing and replacement of the starting motor were taken as examples.Figure 2
YN92 diesel engine. (a) Axonometric drawing. (b) Left view.
(a)(b)Figure 3
Virtual maintenance scene of ship auxiliary engine cabin.


### 5.1. Verification of the Establishment Method of Virtual-Real Fusion Test Scene

(1) In order to build a realistic virtual-real fusion maintenance scene, it is necessary to consider the impact of multiple factors on the registration accuracy of the virtual environment. The feature extraction method is an important factor affecting the registration accuracy. Therefore, firstly, the feature extraction and recognition of diesel engine were carried out. Different feature extraction methods have different feature extraction results. The feature extraction of the same object (diesel engine) is performed using the standard methods of SIFT, SURF, and ORB respectively. The results of the feature extraction of the diesel engine are shown in Figures4–6.Figure 4
SIFT method diesel engine feature extraction results.Figure 5
SURF method diesel engine feature extraction results.Figure 6
ORB method diesel engine feature extraction results.The data results of the three methods for feature extraction are shown in Table4.Table 4
Experimental results of different feature extraction methods.
Physical feature pointsModel feature pointsMatch pointsConsume time (ms)SIFT50252211262.90SURF45442616821.76ORB1023100413613.92Through experimental analysis and comparison, the feature points detected by SIFT, SURF and ORB are 502, 454 and 1023 respectively under the same experimental conditions. The feature points matched by SIFT, SURF and ORB are 112, 168 and 136 respectively. It can be found that although the number of feature points matched by the three methods is roughly the same, the time required for ORB matching is significantly shorter, guaranteeing a higher operation efficiency.(2) To ensure the registration accuracy of the virtual environment, it is also necessary to consider the matching effect when the physical object and its corresponding model are not completely consistent. Therefore, the model of the physical object in the CAD environment is modified. The matching accuracy is calculated and analyzed on the basis of re-extracted feature points. Figure7 is a comparison display of the equipment model after removing some parts.Figure 7
Comparative display of different models. (a) Model 1. (b) Model 2. (c) Model 3. (d) Model 4. (e) Model 5. (f) Model 6.
(a)(b)(c)(d)(e)(f)The results of matching after removing some parts from the equipment model are shown in Table5.Table 5
Matching results of different models.
ModelMatching accuracy of ICP method (%)Model 194.42Model 278.80Model 378.33Model 475.93Model 573.13Model 670.86It can be seen from the above table that the more similar the model and the real object are, the higher the matching accuracy is. When registering the virtual environment, we should not only ensure the registration accuracy, but should also prevent the model and the physical object from any such inconsistency that may lead to registration failure. Therefore, an appropriate matching threshold must be selected. Through comprehensive analysis, it can be seen that the threshold 2 is more appropriate, which can guarantee a certain registration accuracy and avoid registration failure.Incorporating the above two algorithms into AR glasses, an immersive three-dimensional visual information of the physical equipment through the binocular lens of the glasses is obtained. Feature extraction is then performed and matching is performed the virtual model one by one. The resulting virtual-real fusion ship cabin repair scene is shown in Figure8.Figure 8
The obtained virtual-real fusion ship engine maintenance scene.


### 5.2. Maintainability Test Operation and Result Analysis

Next, according to the established virtual-real fusion maintainability test scene of YN92 physical diesel engine, the maintainability operation test of the replacement of the starting motor is carried out. The tester wears the AR glasses to carry out maintainability test operation and to obtain basic test data. The operation steps of the maintenance process are shown in Figure9.Figure 9
Operation steps for the starting motor repair process.A total of 5 groups of tests are carried out, and each group of tests is carried out in three scenes of real environment, virtual-real fusion and without surrounding environment respectively, as shown in Figure10–12. The Maintainability (visibility, accessibility, operation space and operation time) results of the maintenance tests are shown in Table 6.Figure 10
Maintenance operation in simulated real environment.Figure 11
Maintenance operation in virtual-real fusion environment.Figure 12
Maintenance operation without surrounding environment.Table 6
Statistics of partial maintainability test data of the starting motor.
ResultsVisibilityAccessibilityOperating spaceRepair time/sEvaluation valueRelative errorEvaluation valueRelative errorEvaluation valueRelative errorEvaluation valueRelative errorGroup 1Real environment0.700.700.50840Virtual reality fusion0.700.700.740%795.95%Without surrounding environment0.700.700.860%7016.67%Group 2Real environment0.500.700.50810Virtual reality fusion0.500.700.740%801.23%Without surrounding environment0.740%0.700.740%7013.58%Group 3Real environment0.700.800.50780Virtual reality fusion0.700.800.50727.69%Without surrounding environment0.814.29%0.800.740%718.97%Group 4Real environment0.500.700.50770Virtual reality fusion0.500.700.740%770Without surrounding environment0.740%0.700.860%743.90%Group 5Real environment0.700.700.50730Virtual reality fusion0.528.57%0.700.50741.37%Without surrounding environment0.700.700.740%704.11%The computed average error of the three maintainability qualitative indexes in the virtual-real fusion scene is 9.90%, and the average error of the maintenance time is 3.25%. The error of the three maintainability qualitative indexes in the scene without surrounding environment is 22.29%, and the average error of the maintenance time is 9.45%. It can be found that the relative error of virtual reality fusion maintainability evaluation results is significantly reduced as compared to the results of the tests without surrounding environment. In the virtual-real integration maintenance test, the maintenance personnel perceive the existence of the surrounding cabin equipment through vision. During maintenance, in order to avoid collisions with the virtual cabin equipment, the bending angle and the movement range of arms were kept smaller. The posture of the maintenance personnel was also adjusted accordingly to be closer to the real maintenance situation. This was to ensure a reduced maintainability evaluation error. The main reason for the errors in the virtual-real fusion maintenance test is the psychological factors of the maintenance tester. It is also possible that the maintenance personnel passed through the virtual environment and did not correct the action in time. From the stated outcomes it is easy to deduce that the virtual-real fusion maintainability evaluation method takes into account the influence of the surrounding environment, and obviously enhances the accuracy and credibility of the test evaluation results.


## 5.1. Verification of the Establishment Method of Virtual-Real Fusion Test Scene

(1) In order to build a realistic virtual-real fusion maintenance scene, it is necessary to consider the impact of multiple factors on the registration accuracy of the virtual environment. The feature extraction method is an important factor affecting the registration accuracy. Therefore, firstly, the feature extraction and recognition of diesel engine were carried out. Different feature extraction methods have different feature extraction results. The feature extraction of the same object (diesel engine) is performed using the standard methods of SIFT, SURF, and ORB respectively. The results of the feature extraction of the diesel engine are shown in Figures4–6.Figure 4
SIFT method diesel engine feature extraction results.Figure 5
SURF method diesel engine feature extraction results.Figure 6
ORB method diesel engine feature extraction results.The data results of the three methods for feature extraction are shown in Table4.Table 4
Experimental results of different feature extraction methods.
Physical feature pointsModel feature pointsMatch pointsConsume time (ms)SIFT50252211262.90SURF45442616821.76ORB1023100413613.92Through experimental analysis and comparison, the feature points detected by SIFT, SURF and ORB are 502, 454 and 1023 respectively under the same experimental conditions. The feature points matched by SIFT, SURF and ORB are 112, 168 and 136 respectively. It can be found that although the number of feature points matched by the three methods is roughly the same, the time required for ORB matching is significantly shorter, guaranteeing a higher operation efficiency.(2) To ensure the registration accuracy of the virtual environment, it is also necessary to consider the matching effect when the physical object and its corresponding model are not completely consistent. Therefore, the model of the physical object in the CAD environment is modified. The matching accuracy is calculated and analyzed on the basis of re-extracted feature points. Figure7 is a comparison display of the equipment model after removing some parts.Figure 7
Comparative display of different models. (a) Model 1. (b) Model 2. (c) Model 3. (d) Model 4. (e) Model 5. (f) Model 6.
(a)(b)(c)(d)(e)(f)The results of matching after removing some parts from the equipment model are shown in Table5.Table 5
Matching results of different models.
ModelMatching accuracy of ICP method (%)Model 194.42Model 278.80Model 378.33Model 475.93Model 573.13Model 670.86It can be seen from the above table that the more similar the model and the real object are, the higher the matching accuracy is. When registering the virtual environment, we should not only ensure the registration accuracy, but should also prevent the model and the physical object from any such inconsistency that may lead to registration failure. Therefore, an appropriate matching threshold must be selected. Through comprehensive analysis, it can be seen that the threshold 2 is more appropriate, which can guarantee a certain registration accuracy and avoid registration failure.Incorporating the above two algorithms into AR glasses, an immersive three-dimensional visual information of the physical equipment through the binocular lens of the glasses is obtained. Feature extraction is then performed and matching is performed the virtual model one by one. The resulting virtual-real fusion ship cabin repair scene is shown in Figure8.Figure 8
The obtained virtual-real fusion ship engine maintenance scene.


## 5.2. Maintainability Test Operation and Result Analysis

Next, according to the established virtual-real fusion maintainability test scene of YN92 physical diesel engine, the maintainability operation test of the replacement of the starting motor is carried out. The tester wears the AR glasses to carry out maintainability test operation and to obtain basic test data. The operation steps of the maintenance process are shown in Figure9.Figure 9
Operation steps for the starting motor repair process.A total of 5 groups of tests are carried out, and each group of tests is carried out in three scenes of real environment, virtual-real fusion and without surrounding environment respectively, as shown in Figure10–12. The Maintainability (visibility, accessibility, operation space and operation time) results of the maintenance tests are shown in Table 6.Figure 10
Maintenance operation in simulated real environment.Figure 11
Maintenance operation in virtual-real fusion environment.Figure 12
Maintenance operation without surrounding environment.Table 6
Statistics of partial maintainability test data of the starting motor.
ResultsVisibilityAccessibilityOperating spaceRepair time/sEvaluation valueRelative errorEvaluation valueRelative errorEvaluation valueRelative errorEvaluation valueRelative errorGroup 1Real environment0.700.700.50840Virtual reality fusion0.700.700.740%795.95%Without surrounding environment0.700.700.860%7016.67%Group 2Real environment0.500.700.50810Virtual reality fusion0.500.700.740%801.23%Without surrounding environment0.740%0.700.740%7013.58%Group 3Real environment0.700.800.50780Virtual reality fusion0.700.800.50727.69%Without surrounding environment0.814.29%0.800.740%718.97%Group 4Real environment0.500.700.50770Virtual reality fusion0.500.700.740%770Without surrounding environment0.740%0.700.860%743.90%Group 5Real environment0.700.700.50730Virtual reality fusion0.528.57%0.700.50741.37%Without surrounding environment0.700.700.740%704.11%The computed average error of the three maintainability qualitative indexes in the virtual-real fusion scene is 9.90%, and the average error of the maintenance time is 3.25%. The error of the three maintainability qualitative indexes in the scene without surrounding environment is 22.29%, and the average error of the maintenance time is 9.45%. It can be found that the relative error of virtual reality fusion maintainability evaluation results is significantly reduced as compared to the results of the tests without surrounding environment. In the virtual-real integration maintenance test, the maintenance personnel perceive the existence of the surrounding cabin equipment through vision. During maintenance, in order to avoid collisions with the virtual cabin equipment, the bending angle and the movement range of arms were kept smaller. The posture of the maintenance personnel was also adjusted accordingly to be closer to the real maintenance situation. This was to ensure a reduced maintainability evaluation error. The main reason for the errors in the virtual-real fusion maintenance test is the psychological factors of the maintenance tester. It is also possible that the maintenance personnel passed through the virtual environment and did not correct the action in time. From the stated outcomes it is easy to deduce that the virtual-real fusion maintainability evaluation method takes into account the influence of the surrounding environment, and obviously enhances the accuracy and credibility of the test evaluation results.


## 6. Conclusions

This paper proposes a method of constructing a maintainability test scene based on the fusion of virtuality and reality for maintainability evaluation. The ORB feature of the equipment is extracted based on binocular vision. The ICP method is used for feature matching and recognition according to the feature extraction results, while the virtual environment is registered to complete the construction of virtual reality fusion maintainability test scene. Experiments show that the use of ORB features effectively extract equipment features with high speed and promising precision. The ICP method can be used to realize the registration of the physical object and the virtual environment, thereby completing the registration of the virtual environment. The maintainability test was carried out and evaluated in the built virtual-real fusion test scene. The results show that the surrounding virtual environment has a certain impact on the maintenance process, and the maintainability verification is closer to the maintenance process in the real maintenance environment.The virtual-real fusion maintainability test method studied in this paper provides a novel and efficient method for simulating the real maintenance performance of the products under complex maintenance conditions. The method can be carried out for any operations on real objects and can effortlessly simulate the spatial characteristics at low cost. Moreover, the approach is suitable to perform accurate and timely index evaluation of visibility, accessibility and maintenance time. During the experimentation it was observed that psychological factors were involved in the maintenance process. Some of the reported errors were mainly due to the psychological factors. As our future strategy, we are planning to extend the research so that to trace out the psychological factors contributing in high error rate in the maintenance operations. This will help in avoiding the errors and tolerating the psychological.



---

*Source: 6547225-2022-03-09.xml*