THREE-DIMENSIONAL DATA GENERATION SYSTEM, THREE-DIMENSIONAL DATA GENERATION METHOD, AND RECORDING MEDIUM

A processor of a three-dimensional data generation system is configured to determine whether two or more images include a graphics region in which graphics information is superimposed. The processor is configured to execute first processing when the two or more images do not include the graphics region. The first processing includes generation processing of generating the three-dimensional data by using the two or more images. The processor is configured to execute second processing when at least one image of the two or more images includes the graphics region. The second processing includes processing of preventing the graphics region in the at least one image from contributing to generation of the three-dimensional data.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to a three-dimensional data generation system, a three-dimensional data generation method, and a program.

Priority is claimed on Japanese Patent Application No. 2023-160040, filed on Sep. 25, 2023, the content of which is incorporated herein by reference.

Description of Related Art

Industrial endoscope devices have been used for inspection of abnormalities (damage, corrosion, and the like) occurring inside industrial equipment such as boilers, turbines, engines, pipes, and the like. Various subjects are targets for inspection using industrial endoscope devices.

In general, an industrial endoscope device uses a monocular optical adaptor and a measurement-dedicated optical adaptor. The monocular optical adaptor is used for normal observation of a subject. The measurement-dedicated optical adaptor is used for reconstruction of three-dimensional (3D) information of a subject. For example, the measurement-dedicated optical adaptor is a stereo optical adaptor having two visual fields. The industrial endoscope device can measure the size of a detected abnormality by using the 3D information. A user can check the shape (unevenness or the like) of the subject reconstructed by using the 3D information. As described above, the 3D information contributes to improving the quality of inspection and streamlining inspection.

In recent years, a technique of acquiring two or more images of a subject by using a monocular optical adaptor and reconstructing 3D information of the subject by using the images has been developed. Such a technique estimates a change of relative movement of a distal end portion of an endoscope to the subject, executes 3D reconstruction processing based on the results of the estimation, and reconstruct the 3D information. According to such a technique, it is possible to acquire the 3D information without the replacement from a monocular optical adaptor to a stereo optical adaptor. Therefore, the inspection efficiency is improved.

An industrial endoscope device can superimpose a character, an icon, or the like called on-screen display (OSD) information on a still image and a video. Hereinafter, the OSD information is called graphics information. The graphics information includes the type of optical adaptor, the brightness of an image, a date, a logo, a zoom state, and the like.

The position of an image region in which graphics information is superimposed varies in accordance with the type of endoscope device. In addition, the type of graphics information varies in accordance with the type of endoscope device. For example, information of an insertion length indicating an observation position is important in an endoscope device including a long insertion unit having the length of 30 m, and thus the insertion length is superimposed and displayed on an image. The insertion length is not displayed in an endoscope device including a short insertion unit. A user can set whether each item of the graphics information is to be superimposed on an image by operating a setting screen displayed on a monitor.

The use can easily check conditions under which an image is acquired by observing the image on which the graphics information is superimposed. Therefore, the graphics information is useful for reporting inspection results or managing data.

Japanese Unexamined Patent Application, First Publication No. 2020-134242 discloses a device that executes the 3D reconstruction processing by using the following method. The device uses two or more images acquired in a state in which a monocular optical adaptor is used. In addition, the device detects a distinctive small region (feature region) in the two or more images and analyzes movement of the small region. The small region corresponds to the same position of a subject seen in each of the images. The device executes the 3D reconstruction processing in accordance with the movement of the small region.

SUMMARY OF THE INVENTION

A three-dimensional data generation system according to an aspect of the present invention is configured to generate three-dimensional data indicating a three-dimensional shape inside an object. The three-dimensional data generation system includes an imaging apparatus, an image-processing device, and a processor. The imaging apparatus includes a tubular insertion unit configured to acquire an optical image inside the object and is configured to generate two or more images of the object in a state in which the insertion unit is inserted inside the object. The image-processing device is configured to superimpose graphics information on at least one image of the two or more images. The processor is configured to: acquire the two or more images; determine whether the two or more images include a graphics region in which the graphics information is superimposed; and execute first processing when it is determined that the two or more images do not include the graphics region. The first processing includes generation processing of generating the three-dimensional data by using the two or more images. The processor is configured to execute second processing when it is determined that at least one image of the two or more images includes the graphics region. The second processing includes processing of preventing the graphics region in the at least one image from contributing to generation of the three-dimensional data.

In the three-dimensional data generation system according to an aspect of the present invention, the second processing may include processing of setting the graphics region in the at least one image as an ineffective region.

In the three-dimensional data generation system according to an aspect of the present invention, the second processing may include the generation processing in which a region other than the ineffective region in the at least one image is used.

In the three-dimensional data generation system according to an aspect of the present invention, the second processing may include processing of setting the graphics region as the ineffective region based on setting information and position information. The setting information indicates an item corresponding to the graphics information superimposed on the at least one image. The position information indicates the position of the graphics region in which the graphics information corresponding to the item is superimposed.

In the three-dimensional data generation system according to an aspect of the present invention, the second processing may include processing of setting the graphics region as the ineffective region based on type information, setting information, and position information. The type information indicates the type of imaging apparatus. The setting information is prepared for each type of imaging apparatus and indicates an item corresponding to the graphics information superimposed on the at least one image. The position information indicates the position of the graphics region in which the graphics information corresponding to the item is superimposed.

In the three-dimensional data generation system according to an aspect of the present invention, the processor may be configured to receive the type information and the setting information input through an input device.

In the three-dimensional data generation system according to an aspect of the present invention, the generation processing may include detection processing of detecting a feature point from the two or more images. The second processing may include: processing of changing an image in the ineffective region such that a feature point is not detected from the ineffective region; and the generation processing in which the two or more images are used.

In the three-dimensional data generation system according to an aspect of the present invention, the generation processing may include detection processing of detecting a feature point from the two or more images. The second processing may include processing of excluding the ineffective region from a region in which the detection processing is executed.

In the three-dimensional data generation system according to an aspect of the present invention, the second processing may include processing of canceling execution of the generation processing.

In the three-dimensional data generation system according to an aspect of the present invention, the generation processing may include detection processing of detecting a feature point from the two or more images. The second processing may include processing of deleting a feature point detected from the ineffective region.

In the three-dimensional data generation system according to an aspect of the present invention, a storage medium may store two or more images, generated by the imaging apparatus, on which the graphics information is not superimposed by the image-processing device. The second processing may include the generation processing in which the two or more images stored on the storage medium are used.

In the three-dimensional data generation system according to an aspect of the present invention, the processor may be configured to determine whether the two or more images include the graphics region based on setting information indicating whether the graphics information is superimposed on each of the two or more images.

In the three-dimensional data generation system according to an aspect of the present invention, the processor may be configured to: perform, on the two or more images, image processing of detecting the graphics region; and determine whether the two or more images include the graphics region based on a result of the image processing.

In the three-dimensional data generation system according to an aspect of the present invention, the graphics information may include at least one of type of optical adaptor attached to a distal end of the insertion unit, brightness of an image generated by the imaging apparatus, a date, a mark, and a magnification of an image generated by the imaging apparatus.

In the three-dimensional data generation system according to an aspect of the present invention, the processor may be configured to acquire the two or more images from a storage medium after the two or more images are stored on the storage medium.

In the three-dimensional data generation system according to an aspect of the present invention, the insertion unit may include a lens and an image sensor.

In the three-dimensional data generation system according to an aspect of the present invention, the lens and the image sensor may be built in a distal end of the insertion unit. The positions of the distal end when the two or more images are generated may be different from each other. The orientations of the distal end when the two or more images are generated may be different from each other.

In the three-dimensional data generation system according to an aspect of the present invention, the imaging apparatus, the image-processing device, and the processor may be included in an endoscope device.

In the three-dimensional data generation system according to an aspect of the present invention, the imaging apparatus and the image-processing device may be included in an endoscope device. The processor may be included in an external device that is separate from the endoscope device.

In the three-dimensional data generation system according to an aspect of the present invention, the imaging apparatus may be configured to generate the two or more images based on an optical image formed through a monocular optical adaptor attached to a distal end of the insertion unit.

In the three-dimensional data generation system according to an aspect of the present invention, the same region of a component of the object may be seen in at least two images included in the two or more images.

According to an aspect of the present invention, a three-dimensional data generation method of generating three-dimensional data indicating a three-dimensional shape inside an object is provided. The three-dimensional data generation method includes acquiring two or more images of the object by using a processor. The two or more images are generated by an imaging apparatus that includes a tubular insertion unit configured to acquire an optical image inside the object and is configured to generate the two or more images in a state in which the insertion unit is inserted inside the object. The three-dimensional data generation method includes: determining by using the processor whether the two or more images include a graphics region in which graphics information is superimposed by an image-processing device; and executing first processing by using the processor when it is determined that the two or more images do not include the graphics region. The first processing includes generation processing of generating the three-dimensional data by using the two or more images. The three-dimensional data generation method includes executing second processing by using the processor when it is determined that at least one image of the two or more images includes the graphics region. The second processing includes processing of preventing the graphics region in the at least one image from contributing to generation of the three-dimensional data.

According to an aspect of the present invention, a non-transitory computer-readable recording medium stores a program causing a computer to execute processing of generating three-dimensional data indicating a three-dimensional shape inside an object. The processing includes acquiring two or more images of the object. The two or more images are generated by an imaging apparatus that includes a tubular insertion unit configured to acquire an optical image inside the object and is configured to generate the two or more images in a state in which the insertion unit is inserted inside the object. The processing includes: determining whether the two or more images include a graphics region in which graphics information is superimposed by an image-processing device; and executing first processing when it is determined that the two or more images do not include the graphics region. The first processing includes generation processing of generating the three-dimensional data by using the two or more images. The processing includes executing second processing when it is determined that at least one image of the two or more images includes the graphics region. The second processing includes processing of preventing the graphics region in the at least one image from contributing to generation of the three-dimensional data.

DETAILED DESCRIPTION OF THE INVENTION

First Embodiment

A first embodiment of the present invention will be described. In the first embodiment, an example in which a three-dimensional (3D) data generation device is included in an endoscope device will be described. Hereinafter, an example in which an inspection target is a turbine will be described. The following example can also be applied to a case in which the inspection target is a boiler, a turbine, an engine, a pipe, or the like.

A configuration of an endoscope device1in the first embodiment will be described by usingFIG.1andFIG.2.FIG.1shows an external appearance of the endoscope device1.FIG.2shows an internal configuration of the endoscope device1.

The endoscope device1shown inFIG.1includes an insertion unit2, a main body unit3, an operation unit4, and a display unit5. The endoscope device1images a subject inside an inspection target and generates an image. The subject is an industrial product. In order to inspect various inspection targets, a user can replace an optical adaptor mounted at a distal end20of the insertion unit2, select a built-in video-processing program, and add a video-processing program.

The insertion unit2is inserted inside an inspection target. The insertion unit2has a long and thin bendable tubular shape from the distal end20to a base end portion. The insertion unit2images a subject and outputs an imaging signal to the main body unit3. An optical adaptor is mounted on the distal end20of the insertion unit2. For example, a monocular optical adaptor is mounted on the distal end20. The main body unit3is a control device including a housing unit that houses the insertion unit2. The operation unit4receives an operation for the endoscope device1from a user. The display unit5includes a display screen and displays an image of a subject acquired by the insertion unit2, an operation menu, and the like on the display screen.

The operation unit4is a user interface. The display unit5is a monitor (display) such as a liquid crystal display (LCD). The display unit5may be a touch panel. In such a case, the operation unit4and the display unit5are integrated.

The main body unit3shown inFIG.2includes an endoscope unit8, a camera control unit (CCU)9, and a control device10.

The endoscope unit8includes a light source device and a bending device that are not shown in the drawing. The light source device provides the distal end20with illumination light that is necessary for observation. The bending device bends a bending mechanism that is built in the insertion unit2.

A lens21and an imaging device28are built in the distal end20of the insertion unit2. The lens21is an observation optical system. The lens21captures an optical image of a subject formed by an optical adaptor. The imaging device28is an image sensor. The imaging device28photo-electrically converts the optical image of the subject and generates an imaging signal. The lens21and the imaging device28constitute a monocular camera having a single viewpoint.

The CCU9drives the imaging device28. An imaging signal output from the imaging device28is input into the CCU9. The CCU9performs pre-processing including amplification, noise elimination, and the like on the imaging signal acquired by the imaging device28. The CCU9converts the imaging signal on which the pre-processing has been executed into a video signal such as an NTSC signal.

The control device10includes a video-signal-processing circuit12, a read-only memory (ROM)13, a random-access memory (RAM)14, a card interface15, an external device interface16, a control interface17, and a central processing unit (CPU)18.

The video-signal-processing circuit12performs predetermined video processing on the video signal output from the CCU9. For example, the video-signal-processing circuit12performs video processing related to improvement of visibility. For example, the video processing is color reproduction, gray scale correction, noise suppression, contour enhancement, and the like. For example, the video-signal-processing circuit12combines the video signal output from the CCU9and information generated by the CPU18. The video-signal-processing circuit12outputs a combined video signal to the display unit5.

In a case in which graphics information is output from the CPU18, the video-signal-processing circuit12superimposes the graphics information on the video signal. The video-signal-processing circuit12outputs the video signal on which the graphics information is superimposed to the display unit5. The video-signal-processing circuit12may be constituted by at least one of a processor and a logic circuit described later.

The ROM13is a nonvolatile recording medium on which a program for causing the CPU18to control the operation of the endoscope device1is recorded. The RAM14is a volatile recording medium that temporarily stores information used by the CPU18for controlling the endoscope device1. The CPU18controls the operation of the endoscope device1based on the program recorded on the ROM13.

A memory card42is connected to the card interface15. The memory card42is a recording medium that is attachable to and detachable from the endoscope device1. The card interface15inputs control-processing information, image information, and the like stored on the memory card42into the control device10. In addition, the card interface15records the control-processing information, the image information, and the like generated by the endoscope device1on the memory card42.

An external device such as a USB device is connected to the external device interface16. For example, a personal computer (PC)41is connected to the external device interface16. The external device interface16transmits information to the PC41and receives information from the PC41. By doing this, the PC41can display information. In addition, by inputting an instruction into the PC41, a user can perform an operation related to control of the endoscope device1.

The control interface17performs communication with the operation unit4, the endoscope unit8, and the CCU9for operation control. The control interface17notifies the CPU18of information input into the operation unit4by the user. The control interface17outputs control signals used for controlling the light source device and the bending device to the endoscope unit8. The control interface17outputs a control signal used for controlling the imaging device28to the CCU9.

A program executed by the CPU18may be recorded on a computer-readable recording medium. The program recorded on this recording medium may be read and executed by a computer other than the endoscope device1. For example, the program may be read and executed by the PC41. The PC41may control the endoscope device1by transmitting control information used for controlling the endoscope device1to the endoscope device1in accordance with the program. Alternatively, the PC41may acquire a video signal from the endoscope device1and may process the acquired video signal.

The insertion unit2constitutes an imaging apparatus (camera). The imaging device28may be disposed in the main body unit3, and an optical fiber may be disposed in the insertion unit2. Light incident on the lens21may reach the imaging device28via the optical fiber. A borescope may be used as a camera.

Turbines are used for aircraft engines or power generators. There are gas turbines, steam turbines, or the like. Hereinafter, a structure of a gas turbine will be described. Hereinafter, the gas turbine will be called a turbine.

A turbine includes a compressor section, a combustion chamber, and a turbine section. Air is compressed in the compressor section. The compressed air is sent to the combustion chamber. Fuel continuously burns in the combustion chamber, and gas of high pressure and high temperature is generated. The gas expands in the turbine section and generates energy. A compressor rotates by using the energy, and the rest of the energy is extracted. In the compressor section and the turbine section, a rotor fixed to a rotation axis of an engine and a stator fixed to a casing are alternately disposed.

The turbine includes a component disposed in a space inside the turbine. The component is a moving object capable of moving inside the turbine or is a stationary object that stands still inside the turbine. The moving object is a rotor. The stationary object is a stator or a shroud.

FIG.3schematically shows the disposition of rotors and stators in a compressor section of a turbine TB10.FIG.3shows part of a cross-section of the turbine TB10passing through a rotation axis RA10of an engine. The turbine TB10includes a rotor RT10, a stator ST10, a rotor RT11, a stator ST11, a rotor RT12, a stator ST12, a rotor RT13, and a stator ST13in the compressor section. These rotors rotate in a direction DR12around the rotation axis RA10.

Air introduced into the turbine TB10flows in a direction DR11. The rotor RT10is disposed in a low-pressure section that introduces air. The rotor RT13is disposed in a high-pressure section that expels air.

An access port AP10is formed to enable internal inspection of the turbine TB10without disassembling the turbine TB10. The turbine TB10includes two or more access ports, and one of the two or more access ports is shown as the access port AP10inFIG.3. The access port AP10is a hole formed in the turbine TB10.

The insertion unit2constitutes an endoscope. The insertion unit2is inserted into the turbine TB10through the access port AP10. When the insertion unit2is inserted into the turbine TB10, the insertion unit2moves in a direction DR10. When the insertion unit2is pulled out of the turbine TB10, the insertion unit2moves in an opposite direction to the direction DR10. The direction DR10is different from the direction DR12. Illumination light LT10is emitted from the distal end20of the insertion unit2.

FIG.4schematically shows the disposition of two or more rotors RT10seen in a parallel direction to a rotation axis RA10. InFIG.4, eight rotors RT10are disposed in a disk DS10. The rotation axis RA10passes through the center of the disk DS10. The disk DS10rotates in a direction DR12. Therefore, the eight rotors RT10rotate in the direction DR12.

Several tens of rotors or more than 100 rotors are actually disposed in one disk. The number of rotors in one disk depends on the type of engine and also depends on the position of the disk in a region ranging from a low-pressure section to a high-pressure section.

When rotors of a turbine are inspected, a user manually rotates a disk, or a device called a turning tool rotates the disk. The insertion unit2is inserted into the turbine TB10through the access port AP10, and the distal end20is fixed. When the disk is rotating, the user performs inspection of two or more rotors and determines whether there is an abnormality in each rotor. This inspection is one of major inspection items in inspection of a turbine.

The imaging device28generates two or more images. Each of the two or more images is temporally associated with the other images included in the two or more images. For example, each of the two or more images is a still image. A video may be used instead of the still image. Two or more images (frames) included in the video are associated with each other by timestamps (timecodes).

The RAM14stores the two or more images generated by the imaging device28. In addition, the RAM14stores necessary parameters for 3D reconstruction processing. The parameters include an internal parameter of a camera, a distortion correction parameter of the camera, a setting value, scale information, and the like. The setting value is used for various kinds of processing of generating 3D data indicating a 3D shape of a subject. The scale information is used for converting the scale of the 3D data into an actual scale of the subject.

The memory card42may store the two or more images and the above-described parameters. The endoscope device1may read the two or more images and the parameters from the memory card42and may store the two or more images and the parameters on the RAM14.

The endoscope device1may perform wireless or wired communication with an external device via the external device interface16. The external device is the PC41, a cloud server, or the like. The endoscope device1may transmit the two or more images generated by the imaging device28to the external device. The external device may store the two or more images and the above-described parameters. The endoscope device1may receive the two or more images and the parameters from the external device and store the two or more images and the parameters on the RAM14.

As described above, the endoscope device1includes the imaging device28and the CPU18. The imaging device28images a subject and generates an imaging signal. Accordingly, the imaging device28acquires an image of the subject generated by imaging the subject. The image is a two-dimensional (2D) image. The image acquired by the imaging device28is input into the CPU18via the video-signal-processing circuit12.

FIG.5shows a functional configuration of the CPU18. The CPU18functions as a control unit180, an image acquisition unit181, a graphics-processing unit182, a 3D data generation unit183, a display control unit184, and an operation-processing unit185. At least one of the blocks shown inFIG.5may be constituted by a different circuit from the CPU18.

The control unit180controls processing executed by each unit shown inFIG.5.

The image acquisition unit181acquires the two or more images and the above-described parameters from the RAM14. The image acquisition unit181may acquire the two or more images and the above-described parameters from the memory card42or the external device via the external device interface16.

The graphics-processing unit182determines whether the two or more images acquired by the image acquisition unit181include a graphics region in which the graphics information is superimposed. When it is determined that an image includes the graphics region, the graphics-processing unit182prevents the graphics region in the image from contributing to generation of the 3D data.

The 3D data generation unit183executes the 3D reconstruction processing by using the two or more images acquired by the image acquisition unit181and generates 3D data.

The 3D data include 3D coordinates of two or more points (3D point cloud) of a subject and also include a camera coordinate and orientation information. The 3D data may include meshes, each of which is a plane having a 3D point cloud at vertices, and may include mesh polygon data that are a set of texture information associated with the meshes. The camera coordinate indicates 3D coordinates of a camera that has acquired each of the two or more images and are associated with each of the two or more images. The camera coordinate indicates 3D coordinates of a viewpoint when each image is acquired and indicates the position of a camera. For example, the camera coordinate indicates 3D coordinates of an observation optical system included in the camera. The orientation information indicates the orientation (posture) of the camera that has acquired each of the two or more images and is associated with each of the two or more images. For example, the orientation information indicates the orientation of the observation optical system included in the camera.

The display control unit184controls processing executed by the video-signal-processing circuit12. The CCU9outputs a video signal. The video signal includes color data of each pixel of an image generated by the imaging device28. The display control unit184causes the video-signal-processing circuit12to output the video signal output from the CCU9to the display unit5. The video-signal-processing circuit12outputs the video signal to the display unit5. The display unit5displays an image based on the video signal output from the video-signal-processing circuit12. By doing this, the display control unit184displays the image generated by the imaging device28on the display unit5.

The display control unit184displays various kinds of information on the display unit5. In other words, the display control unit184displays various kinds of information on an image.

For example, the display control unit184generates various kinds of information. The various kinds of information are an image of an operation screen, graphics information, and the like. The display control unit184outputs the generated information to the video-signal-processing circuit12. The video-signal-processing circuit12combines the video signal output from the CCU9and the information output from the CPU18. Due to this, the various kinds of information are superimposed on an image. The video-signal-processing circuit12outputs the combined video signal to the display unit5. The display unit5displays an image on which the various kinds of information are superimposed.

In addition, the display control unit184generates image information of the 3D data. The display control unit184outputs the image information to the video-signal-processing circuit12. Similar processing to that described above is executed, and the display unit5displays an image of the 3D data. By doing this, the display control unit184displays the image of the 3D data on the display unit5.

A user inputs various kinds of information into the endoscope device1by operating the operation unit4. The operation unit4outputs the information input by the user. The information is input to the control interface17that is an input unit. The information is output from the control interface17to the CPU18. The operation-processing unit185receives the information input into the endoscope device1via the operation unit4.

Each unit shown inFIG.5may be constituted by at least one of a processor and a logic circuit. For example, the processor is at least one of a central processing unit (CPU), a digital signal processor (DSP), and a graphics-processing unit (GPU). For example, the logic circuit is at least one of an application-specific integrated circuit (ASIC) and a field-programmable gate array (FPGA). Each unit shown inFIG.5may include one or a plurality of processors. Each unit shown inFIG.5may include one or a plurality of logic circuits.

A computer of the endoscope device1may read a program and may execute the read program. The program includes commands defining the operations of each unit shown inFIG.5. In other words, the functions of each unit shown inFIG.5may be realized by software.

The program described above, for example, may be provided by using a “computer-readable storage medium” such as a flash memory. The program may be transmitted from the computer storing the program to the endoscope device1through a transmission medium or transmission waves in a transmission medium. The “transmission medium” transmitting the program is a medium having a function of transmitting information. The medium having the function of transmitting information includes a network (communication network) such as the Internet and a communication circuit line (communication line) such as a telephone line. The program described above may realize some of the functions described above. In addition, the program described above may be a differential file (differential program). The functions described above may be realized by a combination of a program that has already been recorded in a computer and a differential program.

The graphics information is constituted by a character, a numeral, a symbol, a mark, or the like generated by a computer. In the following examples, the graphics information includes sub-graphics information of two or more items. The two or more items are the optical adaptor type, the brightness, the date, a company logo, or a zoom state. The optical adaptor type indicates the type of optical adaptor attached to the distal end20. The brightness indicates the brightness of an image on which the graphics information is superimposed. The company logo is a predetermined mark of a company that produces or sells the endoscope device1. The date indicates a date on which the image is generated. The zoom state indicates the magnification of the image. The sub-graphics information of each item is superimposed in a graphics region of each item.

The graphics information is included in a header or a footer as meta-data of a video file. Alternatively, the graphics information is included in exchangeable image file format (EXIF) information attached to a still image file.

The position of an image region in which the graphics information is superimposed varies in accordance with the type of inspection equipment that generates an image. The meta-data of the video file or the EXIF information of the still image file includes type information indicating the type of inspection equipment.

The diameter or the length of the insertion unit2may vary in accordance with the type of inspection equipment. Alternatively, the optical adaptor may vary in accordance with the type of inspection equipment. The diameter of the insertion unit2, the length of the insertion unit2, or the information of the optical adaptor may be used as the type information.

The meta-data of the video file or the EXIF information of the still image file includes setting information indicating the setting of the graphics information. The setting information includes information indicating whether the graphics information is superimposed on each image. In other words, the setting information includes information indicating whether each image includes a graphics region. In a case in which the setting information includes the information indicating whether the graphics information is superimposed on each image, the setting information includes information indicating whether the sub-graphics information of each item is superimposed on each image.

Hereinafter, distinctive processing of the first embodiment will be described. In the following descriptions, it is assumed that 3D data are generated by using two or more images acquired by endoscope equipment. Inspection equipment that acquires two or more images is not limited to the endoscope equipment. As long as an image of a component inside a turbine is acquired by using equipment including a camera, any equipment may be used.

The control device10functions as a 3D data generation device. A 3D data generation device according to each aspect of the present invention may be a computer system such as a PC that is separate from endoscope equipment. The 3D data generation device may be any one of a desktop PC, a laptop PC, and a tablet terminal. The 3D data generation device may be a computer system that operates on a cloud.

Processing executed by the endoscope device1will be described by usingFIG.6.FIG.6shows a procedure of 3D data generation processing executed by the endoscope device1.

Hereinafter, an example in which a video of which the header stores meta-data is used will be described. Two or more still images may be used instead of the video.

The imaging device28sequentially generates an imaging signal. In other words, the imaging device28generates an imaging signal of each frame corresponding to the video. The video includes two or more frames. Each of the frames is constituted by an image acquired by the imaging device28. When the imaging device28has completed imaging, a video file including the video is recorded on the PC41or the memory card42.

Hereinafter, an example in which the video file recorded on the PC41or the memory card42is used in the 3D reconstruction processing will be described. The CPU18may execute the 3D reconstruction processing in real time at the same time as the imaging device28generates a video.

When the processing shown inFIG.6has been started, the video file is input from the PC41into the RAM14via the external device interface16. Alternatively, the video file is input from the memory card42into the RAM14via the card interface15. The image acquisition unit181acquires a video included in the video file from the RAM14(Step S100).

After Step S100, the graphics-processing unit182acquires the type information indicating the type of inspection equipment from the header of the video file (Step S101).

After Step S101, the graphics-processing unit182acquires the setting information indicating whether the graphics information is superimposed in the video from the header of the video file (Step S102). The setting information includes information indicating whether the graphics information is superimposed in the video.

After Step S102, the graphics-processing unit182refers to the setting information acquired in Step S102and determines whether the graphics information is superimposed in the video. By doing this, the graphics-processing unit182determines whether the video includes a graphics region (Step S103). When the graphics-processing unit182has determined in Step S103that no graphics information is superimposed in the video, Step S108described later is executed.

When the graphics-processing unit182has determined in Step S103that the graphics information is superimposed in the video, the graphics-processing unit182acquires, from the setting information, information indicating whether the sub-graphics information of each item is superimposed on each frame (Step S104). As described above, in a case in which the setting information includes the information indicating whether the graphics information is superimposed on each image, the setting information includes the information indicating whether the sub-graphics information of each item is superimposed on each frame.

After Step S104, the graphics-processing unit182refers to the information acquired in Step S104and determines whether the sub-graphics information of each item is superimposed on each frame. The graphics-processing unit182generates a superimposition state table indicating whether the sub-graphics information of each item is superimposed on each frame (Step S105).

FIG.7shows an example of the superimposition state table. The superimposition state table shown inFIG.7includes type information IF1and state information SI1to SI6.

The type information IF1indicates the type of inspection equipment. The state information SI1to SI6constitutes the setting information. The state information SI1indicates whether the graphics information is superimposed in a video. The state information SI2to SI6indicates whether the sub-graphics information of each item is superimposed on each frame. When the state information SI1indicates that the graphics information is superimposed in the video, at least one piece of the state information SI2to SI6indicates that the sub-graphics information of each item is superimposed on each frame.

The state information SI2indicates whether the sub-graphics information indicating the optical adaptor type is superimposed on each frame. The state information SI3indicates whether the sub-graphics information indicating the brightness of each frame is superimposed on each frame. The state information SI4indicates whether the sub-graphics information indicating the date is superimposed on each frame. The state information SI5indicates whether the sub-graphics information indicating the company logo is superimposed on each frame. The state information SI6indicates whether the sub-graphics information indicating the zoom state of each frame is superimposed on each frame.

In the example shown inFIG.7, the state information SI1indicates that the graphics information is superimposed in the video. The state information SI2and SI3indicates that the sub-graphics information of each item is not superimposed on each frame. The state information SI4to SI6indicates that the sub-graphics information of each item is superimposed on each frame.

After Step S105, the graphics-processing unit182identifies one or more graphics regions in which the sub-graphics information is superimposed (Step S106).

The graphics-processing unit182executes the following processing in Step S106. The graphics-processing unit182acquires a coordinate table indicating an image coordinate of a region in which the sub-graphics information of each item is superimposed. For example, the coordinate table is stored on the ROM13.

FIG.8shows an example of the coordinate table. The coordinate table includes type information IF2and coordinate information C1to C5associated with each other. The type information IF2corresponds to the type information IF1shown inFIG.7and indicates the type of inspection equipment. The coordinate information C1to C5indicates an image coordinate of a region in which the sub-graphics information of each item is superimposed. The coordinate information C1indicates an image coordinate of a region in which the sub-graphics information indicating the optical adaptor type is superimposed. The coordinate information C2indicates an image coordinate of a region in which the sub-graphics information indicating the brightness of each frame is superimposed. The coordinate information C3indicates an image coordinate of a region in which the sub-graphics information indicating the date is superimposed. The coordinate information C4indicates an image coordinate of a region in which the sub-graphics information indicating the company logo is superimposed. The coordinate information C5indicates an image coordinate of a region in which the sub-graphics information indicating the zoom state of each frame is superimposed. For example, the coordinate information C1to C5includes the image coordinate of the upper left point of the graphics region of each item, the horizontal width of the graphics region, and the vertical width of the graphics region.

FIG.9shows an example of the position of a graphics region in an image IMG1generated by the imaging device28. The image IMG1includes graphics regions G1to G5. Sub-graphics information indicating the optical adaptor type is superimposed in the graphics region G1. Sub-graphics information indicating the brightness of the image IMG1is superimposed in the graphics region G2. Sub-graphics information indicating the date is superimposed in the graphics region G3. Sub-graphics information indicating the company logo is superimposed in the graphics region G4. Sub-graphics information indicating the zoom state of the image IMG1is superimposed in the graphics region G5.

The graphics-processing unit182acquires the coordinate information corresponding to both the type information in the superimposition state table and the state information in the superimposition state table from the coordinate table. For example, the graphics-processing unit182searches the coordinate table for the same type information IF2as the type information IF1shown inFIG.7. The graphics-processing unit182acquires at least one piece of coordinate information C1to C5associated with the found type information IF2. Specifically, the graphics-processing unit182acquires the coordinate information corresponding to the state information indicating that the sub-graphics information is superimposed on each frame.

In the superimposition state table shown inFIG.7, the state information SI4to SI6indicates that the sub-graphics information of each item is superimposed on each frame. The graphics-processing unit182acquires the coordinate information C3of the date indicated by the state information SI4, acquires the coordinate information C4of the company logo indicated by the state information SI5, and acquires the coordinate information C5of the zoom state indicated by the state information SI6. The graphics-processing unit182identifies graphics regions indicated by the coordinate information C3to C5. For example, the graphics-processing unit182identifies the graphics regions G3to G5shown inFIG.9.

After Step S106, the graphics-processing unit182sets the one or more graphics regions identified in Step S106as an ineffective region (Step S107). The regions other than the ineffective region in an image generated by the imaging device28is an effective region used in the 3D reconstruction processing.

As described later, the 3D reconstruction processing includes processing of detecting a feature point from at least two images. The graphics-processing unit182changes an image in the ineffective region such that a feature point is not detected from the ineffective region. For example, the graphics-processing unit182changes the image in the ineffective region to a black image. At this time, the graphics-processing unit182changes the pixel values of the image in the ineffective region to values corresponding to a black level. The graphics-processing unit182may exclude the ineffective region from a region in which the processing of detecting a feature point is executed. The graphics-processing unit182may detect a feature point regardless of the effective region or the ineffective region and then may delete a feature point present in the ineffective region.

After Step S107, the 3D data generation unit183executes the 3D reconstruction processing by using two or more frames included in the video and generates 3D data (Step S108). The 3D data generation unit183reads necessary parameters for the 3D reconstruction processing from the RAM14and uses the parameters in the 3D reconstruction processing.

When the graphics-processing unit182has determined in Step S103that no graphics information is superimposed in the video, the entire region of each of the two or more frames is set as an effective region. When the graphics-processing unit182has set a graphics region as an ineffective region in Step S107, the region other than the ineffective region in each of the two or more frames is set as an effective region. The 3D data generation unit183uses the effective region in the 3D reconstruction processing.

After Step S108, the display control unit184displays an image of the 3D data on the display unit5(Step S109). When Step S109has been executed, the 3D data generation processing shown inFIG.6is completed.

The graphics-processing unit182may execute the following processing instead of executing Step S101and Step S102. The graphics-processing unit182acquires n frames in the video. For example, n is 10. The graphics-processing unit182divides a region of each frame into small lattice-like regions at intervals of p pixels. For example, p is 8.

The graphics-processing unit182compares pixel values of pixels having the same coordinates in n frames with each other. For example, the graphics-processing unit182compares the RGB value of a pixel having specific coordinates in a first frame with the RGB value of a pixel having the specific coordinates in a second frame following the first frame. The graphics-processing unit182compares the RGB value of the pixel having the specific coordinates in the second frame with the RGB value of a pixel having the specific coordinates in a third frame following the second frame. The graphics-processing unit182repeats this and detects a small region including pixels of which the RGB values do not change in n frames. The graphics-processing unit182determines that the detected small region is a graphics region.

FIG.10shows an example of the screen of the display unit5. The display unit5displays a screen SC1shown inFIG.10. The screen SC1includes a frame FR1, a button BT1, a button BT2, and a seek-bar SB1.

A user operates the button BT1and the button BT2by operating the operation unit4. In a case in which the display unit5is constituted as a touch panel, the user operates the button BT1and the button BT2by touching the screen of the display unit5.

A user presses the button BT1in order to reproduce a video. After the button BT1is pressed, a frame FR1of the video is displayed.FIG.10shows the screen SC1displayed on the display unit5after the button BT1is pressed.

The user may perform a predetermined operation on the screen SC1by operating the operation unit4or touching the screen of the display unit5. When the predetermined operation has been performed, an instruction to reproduce or pause the video may be input into the endoscope device1. The screen SC1may include a button used for inputting the instruction to reproduce or pause the video.

The seek-bar SB1indicates the position of the frame FR1. The user can change the position of a frame in the seek-bar SB1by operating the operation unit4or touching the screen of the display unit5. In addition, the user can designate a frame for which the 3D reconstruction processing is started and a frame for which the 3D reconstruction processing is completed by operating the operation unit4or touching the screen of the display unit5.

In the above-described example, the user designates a start frame for which the 3D reconstruction processing is started and a completion frame for which the 3D reconstruction processing is completed. The control unit180may automatically designate the start frame and the completion frame. For example, the control unit180may detect a section of the video in which a subject is moving. Alternatively, the control unit180may detect a section of the video in which an abnormality such as damage is seen. The section includes two or more frames of the video. The control unit180may designate the initial frame of the section as the start frame and may designate the last frame of the section as the completion frame.

Only one of the start frame and the completion frame may be designated by the user. Alternatively, only one of the start frame and the completion frame may be automatically designated. A method of setting a section including a frame to be used in the 3D reconstruction processing is not limited to the above-described examples.

The user presses the button BT2in order to start the 3D reconstruction processing. After the button BT2is pressed, the 3D data generation processing shown inFIG.6is started.

Brightness GI1, a date GI2, and a company logo GI3are superimposed on the frame FR1as graphics information. The graphics region in which these are superimposed is set as an ineffective region.

FIG.11shows an example of a graphics region in an image generated by the imaging device28. Graphics regions R1to R3are superimposed on an image IMG2. The image IMG2corresponds to the frame FR1shown inFIG.10. The graphics region R1is a region in which the brightness GI1shown inFIG.10is superimposed. The graphics region R2is a region in which the date GI2shown inFIG.10is superimposed. The graphics region R3is a region in which the company logo GI3shown inFIG.10is superimposed.

The graphics regions R1to R3in all the frames included in the video are set as ineffective regions. The position of each graphics region is the same in all the frames. In addition, a region excluding the graphics regions R1to R3from the entire region in all the frames included in the video is set as an effective region.

The 3D data generation unit183executes the following processing in Step S108.FIG.12schematically shows a situation of image acquisition in a case in which two images are acquired.

As shown inFIG.12, first, an image I1is acquired in an imaging state c1of the camera. Next, an image I2is acquired in an imaging state c2of the camera. At least one of an imaging position and an imaging orientation is different between the imaging state c1and the imaging state c2. InFIG.12, both the imaging position and the imaging orientation are different between the imaging state c1and the imaging state c2.

In each embodiment of the present invention, it is assumed that the image I1and the image I2are acquired by the same endoscope. In addition, in each embodiment of the present invention, it is assumed that parameters of an objective optical system of the endoscope do not change. The parameters of the objective optical system are a focal distance, a distortion aberration, a pixel size of an image sensor, and the like. Hereinafter, for the convenience of description, the parameters of the objective optical system will be abbreviated as internal parameters. When such conditions are assumed, the internal parameters specifying characteristics of the optical system of the endoscope can be used in common regardless of the position and the orientation of the camera (observation optical system). In each embodiment of the present invention, it is assumed that the internal parameters are acquired at the time of factory shipment or at the time of delivery of a product. The internal parameters may be acquired before inspection is started in an inspection cite. In addition, in each embodiment of the present invention, it is assumed that the internal parameters are known at the time of acquiring an image.

In each embodiment of the present invention, it is assumed that the image I1and the image I2are acquired by one endoscope. However, the present invention is not limited to this. For example, the present invention may be also applied to a case in which 3D data are generated by using a plurality of videos acquired by a plurality of endoscopes. In this case, the image I1and the image I2have only to be acquired by using different endoscope devices, and each internal parameter has only to be stored for each endoscope. Even if the internal parameters are unknown when the 3D data are generated, it is possible to perform calculation by using the internal parameters as variables of a target of estimation. Therefore, the subsequent procedure does not greatly change in accordance with whether the internal parameters are known.

The details of the 3D reconstruction processing in Step S108will be described by usingFIG.13.FIG.13shows a procedure of the 3D reconstruction processing.

First, the 3D data generation unit183executes feature point detection processing (Step S108a). The 3D data generation unit183detects a feature point of each of two images in the feature point detection processing. The feature point indicates a corner, an edge, and the like in which an image luminance gradient is large in information of a subject seen in an image. The feature point may be constituted by one pixel, which is a minimum unit of an image. Alternatively, the feature point may be constituted by two or more pixels. The 3D data generation unit183detects a feature point by using a descriptor of image features such as scale-invariant feature transform (SIFT) or features from accelerated segment test (FAST).

FIG.12shows an example in which a feature point P11is detected from the image I1and a feature point P12is detected from the image I2. The feature point P11and the feature point P12correspond to coordinates P1on a subject Although only one feature point of each image is shown inFIG.12, in fact, a plurality of feature points are detected in each image. There is a possibility that the number of feature points detected in each image is different between images. Each feature point detected from each image is converted into data called a feature quantity by using the descriptor. The feature quantity is data indicating characteristics of a feature point.

After Step S108a, the 3D data generation unit183executes feature point association processing (Step S108b). In the feature point association processing, the 3D data generation unit183compares correlations of feature quantities between images for each feature point detected through the feature point detection processing (Step S108a). In a case in which the correlations of the feature quantities are compared and a feature point of which feature quantities are close to those of a feature point of another image is found in each image, the 3D data generation unit183stores information of the feature point on the RAM14. By doing this, the 3D data generation unit183associates feature points of respective images together. On the other hand, in a case in which a feature point of which feature quantities are close to those of a feature point of another image is not found, the 3D data generation unit183discards information of the feature point.

After Step S108b, the 3D data generation unit183reads coordinates of feature points (a pair of feature points) of two images associated with each other from the RAM14. The coordinates are a pair of coordinates of feature points in each image. The 3D data generation unit183executes position-and-orientation calculation processing based on the read coordinates (Step S108c). In the position-and-orientation calculation processing, the 3D data generation unit183calculates a relative position and a relative orientation between the imaging state c1of the camera that acquires the image I1and the imaging state c2of the camera that acquires the image I2. More specifically, the 3D data generation unit183calculates a matrix E by solving the following Equation (1) using an epipolar restriction.

The matrix E is called a basic matrix. The basic matrix E is a matrix storing a relative position and a relative orientation between the imaging state c1of the camera that acquires the image I1and the imaging state c2of the camera that acquires the image I2. In Equation (1), a matrix p1is a matrix including coordinates of a feature point detected from the image I1. A matrix p2is a matrix including coordinates of a feature point detected from the image I2. The basic matrix E includes information related to a relative position and a relative orientation of the camera and thus corresponds to external parameters of the camera. The 3D data generation unit183can solve the basic matrix E by using a known algorithm.

As shown inFIG.12, Expression (2) and Expression (3) are satisfied in a case in which the amount of position (relative position) change of the camera is t and the amount of orientation (relative orientation) change of the camera is R.

In Expression (2), a moving amount in an x-axis direction is expressed as tx, a moving amount in a y-axis direction is expressed as ty, and a moving amount in a z-axis direction is expressed as tz. In Expression (3), a rotation amount α around the x-axis is expressed as Rx(α), a rotation amount β around the y axis is expressed as Ry(β), and a rotation amount γ around the z axis is expressed as Rz(γ). After the basic matrix E is calculated, optimization processing called bundle adjustment may be executed in order to improve restoration accuracy of 3D coordinates.

The 3D data generation unit183calculates 3D coordinates (camera coordinate) in a coordinate system of 3D data by using the calculated amount of position change of the camera. For example, the 3D data generation unit183defines the 3D coordinates of the camera that acquires the image I1. The 3D data generation unit183calculates the 3D coordinates of the camera that acquires the image I2based on both the 3D coordinates of the camera that acquires the image I1and the amount of position change of the camera that acquires the image I2.

The 3D data generation unit183calculates orientation information in the coordinate system of the 3D data by using the calculated amount of orientation change of the camera. For example, the 3D data generation unit183defines orientation information of the camera that acquires the image I1. The 3D data generation unit183generates orientation information of the camera that acquires the image I2based on both the orientation information of the camera that acquires the image I1and the amount of orientation change of the camera that acquires the image I2.

The 3D data generation unit183generates 3D shape data by executing the position-and-orientation calculation processing (Step S108c). The 3D shape data include 3D coordinates (camera coordinate) at the position of the camera and include orientation information indicating the orientation of the camera. In addition, in a case in which a method such as structure from motion, visual-SLAM, or the like is applied to the position-and-orientation calculation processing (Step S108c), the 3D data generation unit183further calculates 3D coordinates of each feature point in Step S108c. The 3D shape data generated in Step S108cdo not include 3D coordinates of a point on the subject other than the feature point. Therefore, the 3D shape data indicate a sparse 3D shape of the subject.

The 3D shape data include the 3D coordinates of each feature point, the above-described camera coordinate, and the above-described orientation information. The 3D coordinates of each feature point are defined in the coordinate system of the 3D data. The 3D coordinates of each feature point are associated with two-dimensional coordinates (2D coordinates) of each feature point. The 2D coordinates of each feature point are defined in a coordinate system of an image including each feature point. The 2D coordinates and the 3D coordinates of each feature point are associated with an image including each feature point.

After Step S108c, the 3D data generation unit183executes 3D shape reconstruction processing based on the relative position and the relative orientation of the camera (the amount t of position change and the amount R of orientation change) calculated in Step S108c(Step S108d). The 3D data generation unit183generates 3D data of the subject in the 3D shape reconstruction processing. As a technique for restoring a 3D shape of the subject, there are patch-based multi-view stereo (PMVS), matching-processing that uses parallelization stereo, and the like. However, a means therefor is not particularly limited.

The 3D data generation unit183calculates 3D coordinates of points on the subject other than feature points in Step S108d. The 3D coordinates of each point other than the feature points are defined in the coordinate system of the 3D data. The 3D coordinates of each point are associated with the 2D coordinates of each point. The 2D coordinates of each point are defined in a coordinate system of a 2D image including each point. The 2D coordinates and the 3D coordinates of each point are associated with a 2D image including each point. The 3D data generation unit183updates the 3D shape data. The updated 3D shape data include the 3D coordinates of each feature point, the 3D coordinates of each point other than the feature points, the camera coordinate, and the orientation information. The 3D shape data updated in Step S108dinclude the 3D coordinates of a point on the subject other than the feature points in addition to the 3D coordinates of the feature points. Therefore, the 3D shape data indicate a dense 3D shape of the subject.

After Step S108d, the 3D data generation unit183executes scale conversion processing based on both the 3D shape data processed in the 3D shape reconstruction processing (Step S108d) and the scale information read from the RAM14(Step S108e). The 3D data generation unit183transforms the 3D shape data of the subject into 3D coordinate data (3D data) having a dimension of length in the scale conversion processing. When Step S108eis executed, the 3D reconstruction processing shown inFIG.13is completed.

In order to shorten a processing time, Step S108dmay be omitted. In this case, after Step S108cis executed, Step S108eis executed without Step S108dbeing executed.

Step S108emay be omitted. In this case, after Step S108dis executed, the 3D reconstruction processing shown inFIG.13is completed without Step S108ebeing executed. In this case, the 3D data indicate a relative shape of the subject not having a dimension of length.

It is necessary that at least part of a region of one of the images and at least part of a region of at least one of the other images overlap each other in order to generate 3D data in accordance with the principle shown inFIG.12. In other words, a region of a first image and a region of a second image different from the first image include a common region. The other region in the first image excluding the common region and the other region in the second image excluding the common region are different from each other. Two or more images used for generating the 3D data include at least two images in which the same region of the subject is seen.

The image I1and the image I2are not necessarily two temporally consecutive frames in a video. There may be one or more frames between the image I1and the image I2in the video.

As described above, the graphics-processing unit182sets one or more graphics regions as ineffective regions in Step S107shown inFIG.6. Specifically, the graphics-processing unit182changes an image in an ineffective region such that a feature point is not detected from a graphics region. For example, the graphics-processing unit182changes the image in the ineffective region to a black image. Alternatively, the graphics-processing unit182excludes the ineffective region from a region in which the processing of detecting a feature point is executed. Alternatively, the graphics-processing unit182detects a feature point regardless of the effective region or the ineffective region and then deletes a feature point present in the ineffective region.

In the feature point detection processing (Step S108a) shown inFIG.13, no feature points in the black image are detected. Alternatively, the feature point detection processing (Step S108a) is not executed in the ineffective region. Alternatively, in the feature point detection processing (Step S108a), a feature point present in the ineffective region is deleted after a feature point is detected regardless of the effective region or the ineffective region. In other words, no feature points in the ineffective region are detected. Therefore, the ineffective region is not used in the feature point association processing (Step S108b), the position-and-orientation calculation processing (Step S108c), and the 3D shape reconstruction processing (Step S108d) shown inFIG.13. In other words, the ineffective region does not contribute to generation of 3D data. The 3D data generation unit183generates 3D data of a subject without using the ineffective region.

The 3D data generation unit183generates the 3D data of the subject by using only the effective region other than the ineffective region in an image on which graphics information is superimposed. Therefore, the endoscope device1can generate the 3D data with high accuracy.

When the graphics-processing unit182has determined in Step S103shown inFIG.6that the graphics information is superimposed in the video, the graphics-processing unit182may cancel execution of the 3D reconstruction processing. In this case, the graphics-processing unit182does not execute the 3D reconstruction processing. Therefore, the endoscope device1can avoid failure of the 3D reconstruction processing or can avoid deterioration of accuracy of 3D data.

The video-signal-processing circuit12may output a first image and a second image to the CPU18. No graphics information is superimposed on the first image. The second image is generated by superimposing graphics information on the first image. The CPU18may acquire the first image and the second image from the video-signal-processing circuit12. The CPU18may record two or more first images and two or more second images on a storage medium. For example, the storage medium is a storage medium included in the PC41or the memory card42.

The CPU18may execute the 3D data generation processing shown inFIG.6by using the two or more second images. When the graphics-processing unit182has determined in Step S103that the graphics information is superimposed on the two or more second images, the graphics-processing unit182may execute the 3D data generation processing by using the two or more first images. The endoscope device1can avoid failure of the 3D reconstruction processing or can avoid deterioration of accuracy of 3D data by using the two or more first images in the 3D data generation processing.

At least one image of two or more images used in the 3D reconstruction processing may include graphics information. When the graphics-processing unit182has determined that no graphics information is superimposed on any of the two or more images, the graphics-processing unit182may execute the 3D data generation processing by using the two or more images. When the graphics-processing unit182has determined that graphics information is superimposed on at least one image of the two or more images, the graphics-processing unit182may set a graphics region in the at least one image as an ineffective region.

The endoscope device1(3D data generation system) according to each aspect of the present invention includes an imaging apparatus, the video-signal-processing circuit12(image-processing device), and the CPU18. The endoscope device1generates 3D data indicating a 3D shape inside an object. The imaging apparatus includes the tubular insertion unit2that acquires an optical image inside the object. The imaging apparatus generates two or more images of the object in a state in which the insertion unit2is inserted inside the object. The video-signal-processing circuit12superimposes graphics information on at least one image of the two or more images.

The CPU18acquires the two or more images and determines whether the two or more images include a graphics region in which the graphics information is superimposed. The CPU18executes first processing when it is determined that the two or more images do not include the graphics region. The first processing includes the 3D reconstruction processing (generation processing) of generating the 3D data by using the two or more images. The CPU18executes second processing when it is determined that at least one image of the two or more images includes the graphics region. The second processing includes processing of preventing the graphics region in the at least one image from contributing to generation of the 3D data.

A 3D data generation method according to each aspect of the present invention generates 3D data indicating a 3D shape inside an object. The 3D data generation method includes an image acquisition step, a determination step, a first processing step, and a second processing step.

The CPU18acquires two or more images of an object in the image acquisition step (Step S100). The CPU18determines in the determination step (Step S103) whether the two or more images include a graphics region in which the graphics information is superimposed by the video-signal-processing circuit12. The CPU18executes first processing in the first processing step (Step S108) when it is determined that the two or more images do not include the graphics region. The first processing includes the 3D reconstruction processing of generating the 3D data by using the two or more images. The CPU18executes second processing in the second processing step (Step S107) when it is determined that at least one image of the two or more images includes the graphics region. The second processing includes processing of preventing the graphics region in the at least one image from contributing to generation of the 3D data.

Each aspect of the present invention may include the following modified example. The second processing includes processing of setting the graphics region in the at least one image determined to include the graphics region as an ineffective region.

Each aspect of the present invention may include the following modified example. The second processing includes the 3D reconstruction processing in which a region other than the ineffective region in the two or more images is used.

Each aspect of the present invention may include the following modified example. The second processing includes processing of setting the graphics region as the ineffective region based on setting information and coordinate information (position information). The setting information indicates an item corresponding to the graphics information superimposed on the at least one image determined to include the graphics region. The coordinate information indicates the position of the graphics region in which the graphics information corresponding to the item is superimposed.

Each aspect of the present invention may include the following modified example. The second processing includes processing of setting the graphics region as the ineffective region based on type information, setting information, and coordinate information (display position information). The type information indicates the type of imaging apparatus. The setting information is prepared for each type of imaging apparatus and indicates an item corresponding to the graphics information superimposed on the at least one image determined to include the graphics region. The coordinate information indicates the position of the graphics region in which the graphics information corresponding to the item is superimposed.

Each aspect of the present invention may include the following modified example. The 3D reconstruction processing includes feature point detection processing of detecting a feature point from the two or more images. The second processing includes both processing of changing an image in the ineffective region such that a feature point is not detected from the ineffective region and processing of the 3D reconstruction processing in which the two or more images are used.

Each aspect of the present invention may include the following modified example. The 3D reconstruction processing includes feature point detection processing of detecting a feature point from the two or more images. The second processing includes processing of excluding the ineffective region from a region in which the feature point detection processing is executed.

Each aspect of the present invention may include the following modified example. The 3D reconstruction processing includes feature point detection processing of detecting a feature point from the two or more images. The second processing includes processing of deleting a feature point detected from the ineffective region.

Each aspect of the present invention may include the following modified example. The second processing includes processing of canceling execution of the 3D reconstruction processing.

Each aspect of the present invention may include the following modified example. A storage medium stores two or more images, generated by the imaging apparatus, on which graphics information is not superimposed by the video-signal-processing circuit12. The second processing includes the 3D reconstruction processing in which the two or more images stored on the storage medium are used.

Each aspect of the present invention may include the following modified example. The CPU18determines whether the two or more images include the graphics region based on setting information indicating whether graphics information is superimposed on each of the two or more images.

Each aspect of the present invention may include the following modified example. The CPU18performs, on the two or more images, image processing of detecting a graphics region. The CPU18determines whether the two or more images include the graphics region based on a result of the image processing.

Each aspect of the present invention may include the following modified example. The graphics information includes at least one of the type of optical adaptor attached to the distal end20of the insertion unit2, the brightness of an image generated by the imaging apparatus, the date, a mark, and the magnification of the image generated by the imaging apparatus. In addition, items of the graphics information may be different in accordance with the type of imaging apparatus.

Each aspect of the present invention may include the following modified example. The CPU18acquires the two or more images from a storage medium after the two or more images are stored on the storage medium. In other words, the CPU18executes the 3D data generation processing by using the two or more images stored on the storage medium instead of executing the 3D data generation processing at the same time as the imaging apparatus generates the two or more images.

Each aspect of the present invention may include the following modified example. The insertion unit2includes the lens21and the imaging device28(image sensor).

Each aspect of the present invention may include the following modified example. The lens21and the imaging device28are built in the distal end20of the insertion unit2. The positions of the distal end20when the two or more images are generated are different from each other. The orientations of the distal end20when the two or more images are generated are different from each other. For example, the two or more images include a first image and a second image. The position of the distal end20when the second image is generated is different from that of the distal end20when the first image is generated. The orientation of the distal end20when the second image is generated is different from that of the distal end20when the first image is generated.

Each aspect of the present invention may include the following modified example. The imaging apparatus, the video-signal-processing circuit12, and the CPU18are included in the endoscope device1.

Each aspect of the present invention may include the following modified example. The imaging apparatus generates the two or more images based on an optical image formed through a monocular optical adaptor attached to the distal end20of the insertion unit2.

Each aspect of the present invention may include the following modified example. The same region of a component of an object is seen in at least two images included in the two or more images.

In the first embodiment, in a case in which at least one image of the two or more images includes a graphics region, the CPU18sets the graphics region as an ineffective region. Therefore, the endoscope device1can avoid failure of processing of generating 3D data or can avoid deterioration of accuracy of the 3D data.

Furthermore, the endoscope device1can generate the 3D data with graphics information, which is useful for reporting inspection results or managing data, remaining in images. Therefore, the endoscope device1can avoid deterioration of inspection efficiency for generating the 3D data.

In a case in which at least one image of the two or more images includes a graphics region, the CPU18generates the 3D data by using a region other than the graphics region in each of the two or more images. Therefore, the endoscope device1can generate the 3D data with high accuracy.

Second Embodiment

A second embodiment of the present invention will be described. In the first embodiment described above, the graphics-processing unit182acquires, from the header of a video file, type information indicating the type of inspection equipment and setting information indicating whether graphics information is superimposed in the video. On the other hand, in the second embodiment, a user inputs the type information and the setting information into the endoscope device1, and the graphics-processing unit182receives the type information and the setting information. In the second embodiment, meta-data stored in the header or the footer of the video file need not include the type information or the setting information.

In the second embodiment, the 3D data generation processing shown inFIG.6is change as described below. A user understands the type of endoscope device1serving as inspection equipment in advance. After the video is displayed on the display unit5, the user inputs the type information into the endoscope device1by operating the operation unit4or the touch panel. The graphics-processing unit182receives the type information input through the operation unit4or the touch panel in Step S101.

The user checks a frame of the video displayed on the display unit5and determines whether graphics information is superimposed on the frame. The user inputs the setting information into the endoscope device1by operating the operation unit4or the touch panel. The graphics-processing unit182receives the setting information input through the operation unit4or the touch panel in Step S102.

FIG.14shows an example of the screen of the display unit5. The display unit5displays a screen SC2shown inFIG.14. The screen SC2includes a frame FR1, a button BT1, a button BT2, a button BT3, and a seek-bar SB1. The same parts as those ofFIG.10will not be described.

A user presses the button BT3in order to input type information and setting information into the endoscope device1. After the button BT3is pressed, the display unit5displays a screen SC3shown inFIG.15. When the image acquisition unit181has acquired the video from the RAM14, the display unit5may display the screen SC3without displaying the screen SC2. The screen SC3includes a pull-down menu PM1and check boxes CB1to CB5. The user operates the pull-down menu PM1and the check boxes CB1to CB5by operating the operation unit4or touching the screen of the display unit5.

The user operates the pull-down menu PM1and selects the type of inspection equipment corresponding to the endoscope device1in use. The graphics-processing unit182receives the type information indicating the type selected by the user.

The user operates the check boxes CB1to CB5and selects an item of sub-graphics information superimposed on each frame. The check box CB1indicates the setting of sub-graphics information indicating the brightness of each frame. The check box CB2indicates the setting of sub-graphics information indicating a company logo. The check box CB3indicates the setting of sub-graphics information indicating a date. The check box CB4indicates the setting of sub-graphics information indicating the optical adaptor type. The check box CB5indicates the setting of sub-graphics information indicating the zoom state of each frame.

Brightness GI1, a date GI2, and a company logo GI3are superimposed on the frame FR1shown inFIG.14. Therefore, the user operates the check boxes CB1to CB3. The graphics-processing unit182receives setting information indicating that the sub-graphics information of the item corresponding to each of the check boxes CB1to CB3is superimposed on each frame.

The graphics-processing unit182refers to the setting information and determines whether the graphics information is superimposed in the video in Step S103. When the setting information indicates that the sub-graphics information of the item corresponding to at least one of the check boxes CB1to CB5is superimposed on each frame, the graphics-processing unit182determines that the graphics information is superimposed in the video. When the setting information indicates that no sub-graphics information of the item corresponding to any of the check boxes CB1to CB5is superimposed on each frame, the graphics-processing unit182determines that no graphics information is superimposed in the video.

When the graphics-processing unit182has determined in Step S103that the graphics information is superimposed in the video, the graphics-processing unit182refers to the setting information and determines whether the sub-graphics information of each item is superimposed on each frame in Step S105. The graphics-processing unit182generates a superimposition state table indicating whether the sub-graphics information of each item is superimposed on each frame in Step S105.

Each aspect of the present invention may include the following modified example. The CPU18receives type information and setting information input through an input device. For example, the input device is the operation unit4or the touch panel.

In the second embodiment, even when the type information and setting information are not associated with the video in advance, the CPU18sets a graphics region as an ineffective region based on the type information and setting information input by a user via the input device. Therefore, the endoscope device1can avoid failure of processing of generating 3D data or can avoid deterioration of accuracy of the 3D data.

Third Embodiment

A third embodiment of the present invention will be described. In the third embodiment, a device that acquires an image of a subject and a device that generates 3D data are different.FIG.16shows a configuration of an endoscope system100(3D data generation system) according to the third embodiment. The endoscope system100shown inFIG.16includes an endoscope device1and an external device6. The endoscope device1acquires two or more images of the subject and transmits the two or more images to the external device6. The external device6receives the two or more images from the endoscope device1and executes the 3D reconstruction processing by using the two or more images.

The configuration of the endoscope device1is similar to that shown inFIG.2. The external device interface16performs communication with the external device6. Specifically, the external device interface16transmits the two or more images of the subject to the external device6.

For example, the external device interface16is a wireless module and performs wireless communication with the external device6. The endoscope device1and the external device6may be connected to each other via a cable such as a local area network (LAN) cable, and the external device interface16may perform communication with the external device6via the cable.

For example, the external device6is a mobile terminal such as a tablet terminal. The external device6may be a fixed terminal. The form of the external device6is not limited thereto.

FIG.17shows a functional configuration of the external device6. The external device6shown inFIG.17includes a data communication unit60, a CPU61, a display unit62, and a RAM63.

The data communication unit60receives two or more images from the endoscope device1. For example, the data communication unit60is a wireless module and performs wireless communication with the endoscope device1. The data communication unit60may perform communication with the endoscope device1via a cable.

CPU61is configured similarly to the CPU18shown inFIG.5. In addition, the CPU61controls communication executed by the data communication unit60with the endoscope device1. In other words, the CPU61causes the data communication unit60to receive the two or more images from the endoscope device1.

The CPU61may read a program including commands defining the operations of the CPU61and may execute the read program. In other words, the functions of the CPU61may be realized by software. A method of implementing this program is similar to that of implementing a program realizing the functions of the endoscope device1.

The display unit62includes a display screen and displays an image, an operation menu and the like on the display screen. The display unit62is a monitor (display) such as an LCD.

The RAM63temporarily stores information used for causing the CPU61to control the external device6.

Processing executed by the endoscope device1will be described by usingFIG.18.FIG.18shows a procedure of the processing executed by the endoscope device1.

A user understands the type of endoscope device1serving as inspection equipment in advance. The user inputs the type information into the endoscope device1by operating the operation unit4or the touch panel. The CPU18receives the type information input through the operation unit4or the touch panel1(Step S110).

The user inputs information indicating whether graphics information is superimposed in a video into the endoscope device1by operating the operation unit4or the touch panel. After Step S110, the CPU18receives the information input through the operation unit4or the touch panel (Step S111).

The user inputs information indicating whether sub-graphics information of each item is superimposed on each frame into the endoscope device1by operating the operation unit4or the touch panel. After Step S111, the CPU18receives the information input through the operation unit4or the touch panel (Step S112).

After Step S112, the imaging device28starts imaging and generates a video (Step S113). When the imaging device28has completed the imaging, a video file including the video is recorded on the PC41or the memory card42.

After Step S113, the CPU18generates type information including the information received in Step S110. In addition, the CPU18generates setting information including the information received in Step S111and the information received in Step S112. The CPU18records the type information and the setting information in the header of the video file (Step S114). Due to this, the type information and the setting information are associated with the video.

After Step S114, the CPU18reads the video file from the PC41via the external device interface16. Alternatively, the CPU18reads the video file from the memory card42via the card interface15. The CPU18transmits the video file to the external device6via the external device interface16(Step S115). When Step S115has been executed, the processing shown inFIG.18is completed.

Processing executed by the external device6will be described by usingFIG.19.FIG.19shows a procedure of the processing executed by the external device6.

The CPU61causes the data communication unit60to receive the video file from the endoscope device1. By doing this, the data communication unit60receives the video file from the endoscope device1(Step S200).

After Step S200, the CPU61stores the video file received from the endoscope device1on the RAM63(Step S201).

The endoscope device1may transmit the video to the external device6at the same time as the imaging device28generates the video. The CPU61may execute the 3D reconstruction processing in real time at the same time of receiving the video from the endoscope device1.

Various modifications that can be applied to the endoscope device1in the first embodiment and the second embodiment can also be applied to the endoscope system100in the third embodiment alike.

Each aspect of the present invention may include the following modified example. The imaging apparatus and the video-signal-processing circuit12(image-processing device) are included in the endoscope device1. The CPU61is included in the external device6that is separate from the endoscope device1.

In the third embodiment, the endoscope system100can avoid failure of processing of generating 3D data or can avoid deterioration of accuracy of the 3D data.