Mitsubishi Heavy Industries Technical Review Vol. 60 No. 1 (March 2023)
1Simple 3D Inspection Recording System Using
Photographic Projection to Create 3D Digital Records

SHOTA ISHII*1

TATSUYA SAKAI*2

The method of recording damage information on a two-dimensional (2D) exploded view, ingeneral, has the advantage of allowing an inspection record to be easily prepared. On the otherhand, it has issues in terms of the fidelity of damage positions recorded on the exploded view andthe precision of ex-post analyses of damaged regions. As such, technology for saving inspectionrecords as three-dimensional (3D) digital information is sought after. Mitsubishi Heavy Industries,Ltd. (MHI) has developed technology for obtaining and recording the areas and photographtextures of damaged regions by reproducing the position/orientation of the photo (camera) of an
object to be inspected on a 3D CG (Computer Graphics) space and projecting the damaged
regions indicated on the photo and the photo on the target 3D CAD (Computer Aided Design)model. This technology allows preparation of a 3D digital inspection record with the same
workload and equipment as those required for preparation of a 2D inspection record, thereby
improving quality control.

|1. Introduction
In MHI, we prepare the inspection records of the components (such as power generationturbines) which are used in high-temperature and high-pressure environments by rendering theinformation of damage such as erosion and cracks on 2D exploded views. However, the method ofrecording damage information on 2D exploded views generally has issues with the fidelity ofdamage positions on the exploded views and the precision of ex-post analyses of damaged regions
and traceability of damage. Therefore, technology for saving inspection records as 3D digital
information has been needed. There have also been requests from the inspection site about newtechnology that the working time should be approximately the same as that with the conventional
method of recording damage information on a 2D exploded view and no expensive and dedicateddevices such as 3D smart cameras and handy 3D scanners should be used in view of versatility on
the assumption that it will be used in various environments and by users in Japan and overseas, inthe open air, etc..
Against the background and issues described above, another company developed atechnology for calculating pseudo-3D information (areas and lengths of damaged regions) with theuse of a pseudo-image linked to the 3D CAD information corresponding to actual 2D image pixels.
But the technology has a problem of lack of flexibility because the photographing position andangle of view need to completely match with those of the pseudo-image. Many other companies
obtain 3D information by using large-scale devices and techniques (such as 3D smart cameras,
handy 3D scanners and photogrammetry technique), but these methods lack versatility and require
cost and time.
The purpose of this report is to develop a versatile 3D inspection recording
system/technology for a wide range of products, which allows the areas and photograph textures of*1

Service Engineering Department, Research & Innovation Center, Mitsubishi Heavy industries, Ltd.

*2

Hitachi Quality Assurance Department, GTCC Business Division, Energy Transition & Power Headquarters, Energy Systems,
Mitsubishi Heavy industries, Ltd.

Mitsubishi Heavy Industries Technical Review Vol. 60 No. 1 (March 2023)
2

damaged regions to be obtained and recorded without depending on the photographing position andangle of view and without using any large-scale devices that require cost and time.

|2. Characteristics of the developed technology
In order to meet the requests regarding inspection records, MHI has developed atechnology/system for calculating the area of a damaged region from only the photo and further
preparing a 3D digital inspection record from the obtained photo texture without depending on thephotographing position/angle of view. Table 1 shows the overview of the functions and uses of thedeveloped technology/system.
Table 1

Overview of the functions and uses of the developed technology/system

No.

Developed function

Use・ Setting theposition/orientation of thecamera in a 3D CG space
・ Checking for deformationon the subject

1

Estimation of theposition/orientation of thephoto (camera)

2

Drawing of damaged regions(handwriting or automatic)

3

Projection of the photo and the ・ Preparing a 3D inspection
damaged regions
record・ Indicating damaged regions
on the inspection record

Technology or information used・ 3D superimposition
・ GUI (Graphical User Interface),
・ CNN (Convolutional NeuralNetwork)・ 3D CG, geometric transformation
・ RPA (Robotic ProcessAutomation)

Calculation of the areas and・ Recording and assessing the ・ Geometry/scale of 3D CAD
lengths of damaged regions
state of the damage
model
only from the photo
Browsing of the 3D inspection ・ Sharing the information on・ glTF (GL Transmission Format)
inspection records
records by a web browser

45|3. Software and equipment configuration
The developed system/technology is configured to link the application (named as Simple 3D
Inspection Recording System) developed by MHI and commercial 3D CG software. Free BlenderVer. 3.3.0(1) is used as 3D CG software and free libraries are used in processing of each program
developed by MHI. The user may add camera equipment that can be used in the application andthere is no limit to the kind of camera. Thus, compared to the software/devices requiring license
fees, the developed system/technology requires lower initial costs and maintenance costs.

|4. Operating procedure and processing process of the application
4.1

Overview of the operating procedure and the processing process

Table 2 shows the overview of the user's operating procedure and the processing process of
this system. In the Simple 3D Inspection Recording System, a 3D digital inspection record isprepared according to the operating procedure and the processing process shown in Table 2. The
details of each procedure and processing process are described in the following sections.
Table 2No.
(1)
(2)
(3)
(4)
(5)
(6)
(7)
(8)

Operating procedure and processing process

Operating procedure and processing process
Loading of a photo taken at an arbitrary camera position into
the system
Indication of feature points and drawing of damaged regions
Automatic estimation of the position/orientation of the photo(camera) by use of the feature points
Setting of the position/orientation of the camera in the 3D CGsoftware
Rendering based on the position/orientation of the camera in
the 3D CG software
Display of the rendered image on the Simple 3D Inspection
Recording System(Confirmation of the estimated position/orientation of thecamera)
Projection of the photo and the damaged regions to the 3D
CAD model in the 3D CG software
Calculation of the areas/lengths of the damaged regions anddisplay them on the Simple 3D Inspection Recording System

Manualoperation

Automaticprocessing✓
✓

✓
✓
✓
✓
✓
✓
✓

Mitsubishi Heavy Industries Technical Review Vol. 60 No. 1 (March 2023)
34.2

Indication of feature points and drawing of damaged regions

The indication of feature points and the drawing of damaged regions on the photo loaded into
this system in (1) and (2) of Table 2 are described below. Figure 1 shows the indication of featurepoints and the drawing of damaged regions. The photo loaded into the Simple 3D Inspection
Recording System in Figure 1 is of a turbine blade which is a component of a gas turbine.
The yellow cross marks on the screen are the feature points indicated by the user and several
feature points are indicated on the screen of the Simple 3D Inspection Recording System forestimation of the camera position/orientation. Furthermore, it is desirable that these feature points
are set in the position where the user can specify accurately on the photograph, such as the shapeand pattern of the object to be inspected. The positions of the indicated feature points correspond to
the positions on the 3D CAD model in the 3D CG spatial coordinate system. Feature points may be
automatically identified by the machine learning function, instead of being indicated by the user.
On a turbine blade used in a high-temperature and high-pressure environment, various formsof damage (such as wear, erosion and cracks) occur, and the regions and forms of such damage are
saved as inspection records. Damage can be drawn with color-coded frames (regions) and lines,
and the forms of damage can be set for each product. For a turbine blade, frames are used toindicate wear and erosion and solid lines are used to indicate cracks. However, the regions andforms of damage indicated in this report are not of actual damage but of hypothetical damage for
explanation purposes. After specifying the type of damage, the user indicates the damage
by drawing the damaged region on the screen. For damage that can be identified on
photographs, it is possible to automatically identify the kind and region of damage using machinelearning functions, instead of being drawn by the user. It is also possible to incorporate functions
that utilize conventional inspection recording and systems, such as automatically drawing damaged
regions, by reading conventional inspection recording data into a simplified 3D inspection
recording system.

Figure 1

4.3

Indication of feature points and drawing of damaged regions

Estimation of the position/orientation of the camera

The estimation and setting of the position/orientation of the camera in the 3D CG in (3) and(4) of Table 2 are described below. Figure 2 shows the initial position of the camera in the 3D CG
software. In this 3D CG software, a 3D CAD model of the turbine blade corresponding to the photoof Figure 1 is placed. The orange-colored quadrangular pyramid represents a camera in the 3D CG
software and the subject photographed by this camera is shown in the white frame. The Simple 3DInspection Recording System reproduces the position/orientation of the photo arbitrarily taken by
the user (of the camera used) to the camera in the 3D CG software regardless of the initial positionof the camera in the 3D CG software. As described in section 4.2, the user only indicates several
feature points to estimate the position/orientation of the camera by which the user arbitrarily took
the photo without using any fixing apparatus such as jigs. Thus, the system requires no complicatedsetting, is easy to use and can be applied to a broad range of photos (positions, orientations andangles of view of the camera).

Mitsubishi Heavy Industries Technical Review Vol. 60 No. 1 (March 2023)
4Figure 2

4.4

Initial position of the camera in the 3D CG software

Confirmation of the estimated position/orientation of the camera

The confirmation of the estimated position/orientation of the camera in (5) and (6) of Table 2
is described below. In the developed Simple 3D Inspection Recording System, it can be confirmedon the screen that the position/orientation of the camera in the 3D CG software has been
automatically adjusted. In Figure 3, the image rendered by automatic processing is superimposedand displayed on the photo. Rendering is the process of converting objects in 3D space into 2D
images in 3D CG production. In Figure 3, the rendered image is displayed with the transparency of60% (semi-transparency), and it should be noted that the position/orientation of the 3D CAD modelin the 3D CG software is completely in agreement with that of the subject of the photo. Bysuperimposing the image on the photo like this, we can not only confirm that theposition/orientation of the camera in the 3D CG software has been adjusted but also check whether
there is any deformation on the subject.

Figure 3 State of the image rendered by automatic processing being
superimposed and displayed on the photo

4.5

Projection of the photo and the damaged regions

The projection of the photo and the damaged regions in (7) of Table 2 is described below.
After the position/orientation of the camera in the 3D CG software is adjusted, the photo and thedamaged regions are automatically projected to the 3D CAD model in the 3D CG software. The
damaged regions were drawn on the screen by the user or automatically identified by use of themachine learning function. Figure 4 shows the photo and damaged regions projected to the 3D
CAD model by automatic processing. The photo and damaged regions are projected according tothe geometry of the 3D CAD model and saved as a 3D inspection record (Blender file format). The
above-described CG processing is automatically executed by program control. Also, multiple
photos taken from different positions/directions can be projected to the same 3D CAD model.

Mitsubishi Heavy Industries Technical Review Vol. 60 No. 1 (March 2023)
5Figure 4 Photo and damaged regions projected to the 3D CAD modelby automatic processing

4.6

Calculation of the areas/lengths of the damaged regions

The calculation and display of the areas/lengths of the damaged regions in (8) of Table 2 aredescribed below. The areas (mm2) or lengths (mm) of the damaged regions projected to the 3D
CAD model are calculated based on the geometry and scale of the 3D CAD model by automatic
processing. The calculated areas/lengths are displayed on the Simple 3D Inspection Recording
System and saved as digital information. In Figure 5, the areas and lengths of the damaged regions
are displayed on the Simple 3D Inspection Recording System. The areas or lengths of the damaged
regions are displayed near the regions drawn on the screen by the user or near the regions
automatically identified by use of the machine learning function in the black outlined rectangles. In
this report, however, each dimension value, which is product information, is hidden. The above
processing allows the areas or length of damaged regions to be calculated only with the photo andeliminates the need for measuring damaged regions on an actual product.

Figure 5

Display of the areas/lengths of the damaged regions|5. 3D digital inspection recordThe information obtained by the operation and processing in chapter 4 is used to prepare a3D digital inspection record. Table 3 shows the file formats that can be used for a 3D digitalinspection record in which the photo and the damaged regions are projected to a 3D CAD model.
Either one of the two typical file formats which store photo texture information is selected toprepare a 3D digital inspection record.

Mitsubishi Heavy Industries Technical Review Vol. 60 No. 1 (March 2023)
6

Table 3 Output file formats for 3D digital inspection record
File format

Description of file format

Texture informationTexture information is contained
inside.

.blender

Dedicated file format for Blender.obj.mtl

obj is a 3D model format developed by Wavefront
Technologies.
mtl is file which contains the color and texture
information of models.

Image files are required for displayof texture.

.glb(glTF 2.0)

File format for 3D models running on a web browser

Texture information is contained
inside.

Figure 6 shows an example of one of the projected turbine blade photos displayed as a 3D
digital inspection record. In Figure 6, the 3D digital inspection record in glb format is displayed byuse of the 3D Viewer installed on the Microsoft Windows OS. The right and left views show the3D digital inspection record displayed in different orientations (viewpoints). This 3D digitalinspection record is designed to display the photo (texture) only on the surface of the 3D CAD
model having the regions to be projected and the surfaces where no photo is projected are blank.
The frames and lines for the damaged regions indicated on the Simple 3D Inspection Recording
System can be checked on the 3D digital inspection record. Thus, compared to the method ofrecording the damage information to 2D exploded views, 3D digital inspection records allow
improvement of quality control in terms of the accuracy of the information of damage positions andregions.

Figure 6

3D digital inspection record using the projected photo|6. Conclusion
With this technology/system, the position/orientation of the photo arbitrarily taken by camera
equipment (the position/orientation of the camera) is reproduced on a 3D CG space and thedamaged regions indicated on the photo and the photo are projected on the target 3D CAD model.
By this, it became possible to obtain the texture information and automatically calculate the areasor lengths of the damaged regions based on the geometry of the 3D CAD model. One of thefeatures of this technology/system is that it can prepare a 3D digital inspection record easily andinexpensively by using a general photographing device (camera), not requiring a less versatile andexpensive device, and in about the same working time as the conventional method for recording
damage information on a 2D exploded view. Furthermore, since damaged regions are drawn on thephoto displayed on the system screen, the recording accuracy and reproducibility for damage
positions and regions can be improved. In addition, use of 3D digital inspection records is effectivein improving the precision of ex-post analysis of damaged regions and traceability of damage.
Moving forward, we will develop functions required for each inspection record or assessment
method to which this system is applied, thereby extending the range of application.

References(1)

Blender official website, (2022), https://www.blender.org/