Image processing apparatus and image processing method

A three-dimensional shape of a subject is analyzed by inputting captured images of a depth camera and a visible light camera. There is provided an image processing unit configured to input captured images of the depth camera and the visible light camera, to analyze a three-dimensional shape of the subject. The image processing unit generates a depth map based TSDF space (TSDF Volume) by using a depth map acquired from a captured image of the depth camera, and generates a visible light image based TSDF space by using a captured image of the visible light camera. Moreover, an integrated TSDF space is generated by integration processing on the depth map based TSDF space and the visible light image based TSDF space, and three-dimensional shape analysis processing on the subject is executed using the integrated TSDF space.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a U.S. National Phase of International Patent Application No. PCT/JP2020/009154 filed on Mar. 4, 2020, which claims priority benefit of Japanese Patent Application No. JP 2019-073298 filed in the Japan Patent Office on Apr. 8, 2019. Each of the above-referenced applications is hereby incorporated herein by reference in its entirety.

TECHNICAL FIELD

The present disclosure relates to an image processing apparatus, an image processing method, and a program. More specifically, the present disclosure relates to an image processing apparatus, an image processing system, and a program that execute three-dimensional (3D) shape analysis processing.

BACKGROUND ART

In recent years, three-dimensional image generation technology using computer graphics (CG) technology has been used in various fields such as movies and games.

For example, Patent Document 1 (Japanese Patent No. 5906258) discloses a technique for reconfiguring a highly accurate three-dimensional shape of a subject by using a depth map generated from a captured image of a depth camera capable of measuring a distance (depth) from the camera. The reconfiguration processing of the three-dimensional shape by using the depth map is called Kinect Fusion.

In Kinect Fusion, by executing processing similar to simultaneous localization and mapping (SLAM) that sets a target of three-dimensional shape reconfiguration as a still scene and simultaneously generates self-location estimation and an environment map, map generation and self-location estimation by tracking are simultaneously performed. As a result, the three-dimensional shape of a subject is reconfigured with high quality and high accuracy.

In Kinect Fusion, as a map expressing a structure of a scene (subject), there is used a map that expresses the scene as a volume that is a stereoscopic space, and stores, with a sign, a distance from a depth value acquired by a depth camera in each voxel of a cube, which is a volume component. This map is referred to as a TSDF space (volumetric truncated signed distance function=TSDF Volume).

When a location and a position of the camera are known, a more accurate three-dimensional structure can be restored by taking a weighted average of individual voxels constituting the TSDF space (TSDF Volume) generated on the basis of depth information acquired from a plurality of viewpoints.

After the release of Kinect Fusion, several technologies to which Kinect Fusion is applied have been released. For example, a method of processing a large-scale space exceeding a memory capacity of a three-dimensional shape analysis device, a method of performing expansion on a dynamic subject, and the like are released, showing that a three-dimensional shape can be reconstructed with high quality.

Whereas, in all of these approaches, since the shape is restored only from the depth map obtained from the depth camera, there is a problem that it is difficult to perform reconfiguration for a material or the like for which it is difficult to acquire a depth.

Note that many depth cameras that are currently available at relatively low cost are cameras using a method called active sensing using infrared light (IR light). However, this depth camera using an infrared light (IR light) source has a problem that highly accurate depth measurement becomes difficult or depth can no longer be measured for a material having weak reflection for infrared light.

Examples of a material that absorbs infrared light (IR light) include a black material, particularly black hair of oriental people as a remarkable example. Therefore, for example, the depth measurement accuracy decreases in a black hair region, and the three-dimensional shape can no longer be calculated.

As described above, since the depth map is used as an input in the three-dimensional shape reconfiguration processing using the TSDF space (TSDF Volume), there is a problem that a correct three-dimensional shape cannot be estimated for a region where the depth cannot be accurately acquired.

CITATION LIST

Patent Document

SUMMARY OF THE INVENTION

Problems to be Solved by the Invention

The present disclosure has been made in view of the above-described problems, for example, and provides an image processing apparatus, an image processing method, and a program capable of analyzing a highly accurate three-dimensional shape even in a region where it is difficult to acquire an accurate depth by a depth camera.

Solutions to Problems

A first aspect of the present disclosure is an image processing apparatus having:an image processing unit configured to input captured images of a depth camera and a visible light camera, to analyze a three-dimensional shape of a subject, in whichthe image processing unitgenerates a depth map based TSDF space (TSDF Volume) by using a depth map acquired from a captured image of the depth camera,generates a visible light image based TSDF space by using a captured image of the visible light camera,generates an integrated TSDF space by integration processing on the depth map based TSDF space and the visible light image based TSDF space, andexecutes three-dimensional shape analysis processing on the subject by using the integrated TSDF space.

Moreover, a second aspect of the present disclosure isan image processing method executed in an image processing apparatus, in whichthe image processing apparatus includes:an image processing unit configured to input captured images of a depth camera and a visible light camera, to analyze a three-dimensional shape of a subject, andthe image processing unitgenerates a depth map based TSDF space (TSDF Volume) by using a depth map acquired from a captured image of the depth camera,generates a visible light image based TSDF space by using a captured image of the visible light camera,generates an integrated TSDF space by integration processing on the depth map based TSDF space and the visible light image based TSDF space, andexecutes three-dimensional shape analysis processing on the subject by using the integrated TSDF space.

Moreover, a third aspect of the present disclosure isa program for causing an image processing apparatus to execute image processing, in whichthe image processing apparatus includes:an image processing unit configured to input captured images of a depth camera and a visible light camera, to analyze a three-dimensional shape of a subject, andthe program causes the image processing unit to execute:a process of generating a depth map based TSDF space (TSDF Volume) by using a depth map acquired from a captured image of the depth camera;a process of generating a visible light image based TSDF space by using a captured image of the visible light camera;a process of generating an integrated TSDF space by integration processing on the depth map based TSDF space and the visible light image based TSDF space; anda process of executing three-dimensional shape analysis processing on the subject by using the integrated TSDF space.

Note that the program of the present disclosure is, for example, a program that can be provided by a storage medium or a communication medium that provides a variety of program codes in a computer-readable format, to an image processing apparatus or a computer system capable of executing the program codes. By providing such a program in a computer-readable format, processing corresponding to the program is realized on the image processing apparatus or the computer system.

Still other objects, features, and advantages of the present disclosure will become apparent from the more detailed description based on the embodiment of the present disclosure as described later and the attached drawings. Note that a system in this specification is a logical set configuration of a plurality of devices, and is not limited to one in which a device of each configuration is in a same casing.

According to a configuration of an embodiment of the present disclosure, a configuration is realized in which captured images of a depth camera and a visible light camera are inputted, and a highly accurate three-dimensional shape of a subject is analyzed and restored.

Specifically, for example, there is provided an image processing unit that inputs captured images of the depth camera and the visible light camera to analyze a three-dimensional shape of a subject. The image processing unit generates a depth map based TSDF space (TSDF Volume) by using a depth map acquired from a captured image of the depth camera, and generates a visible light image based TSDF space by using a captured image of the visible light camera. Moreover, an integrated TSDF space is generated by integration processing on the depth map based TSDF space and the visible light image based TSDF space, and three-dimensional shape analysis processing on the subject is executed using the integrated TSDF space.

This configuration realizes a configuration in which captured images of the depth camera and the visible light camera are inputted and a highly accurate three-dimensional shape of a subject is analyzed and restored.

Note that the effects described in this specification are merely examples and are not limited, and additional effects may be present.

MODE FOR CARRYING OUT THE INVENTION

Hereinafter, with reference to the drawings, an image processing apparatus, an image processing method, and a program of the present disclosure will be described. Note that the description will be made in accordance with the following items.

1. About outline of processing executed by image processing apparatus of present disclosure

2. About configuration and processing of image processing unit

2-1. About details of processing executed by TSDF space (TSDF Volume) generation unit

2-2. About processing executed by visual volume intersection method (Visual Hull) based three-dimensional shape (polygon mesh) generation unit

2-3. About processing executed by polygon mesh based TSDF space (TSDF Volume) generation unit

2-4. About processing executed by TSDF space (TSDF Volume) integration unit

2-5. About processing executed by three-dimensional shape generation unit

3. About processing sequence executed by image processing apparatus of present disclosure

4. About specific example of three-dimensional structure data of subject obtained by applying processing of present disclosure

5. About other embodiments

6. About hardware configuration example of image processing apparatus

7. Summary of configuration of present disclosure

1. About Configuration of Image Processing Apparatus of Present Disclosure and Processing to be Executed

A configuration of an image processing apparatus of the present disclosure and processing to be executed will be described with reference toFIG.1and subsequent drawings.

The image processing apparatus according to the present disclosure is an apparatus capable of analyzing a highly accurate three-dimensional shape even in a region where a depth cannot be acquired by a depth camera, by using captured images of a depth camera and an RGB camera that capture images of a subject (scene) as an analysis target of the three-dimensional shape.

Specifically, the following processes are executed.

(Process 1) A “truncated signed distance function space”=TSDF space (volumetric truncated signed distance function=TSDF Volume) is generated, from a depth map of a subject (scene) acquired by a depth camera.

(Process 2) Images of the same subject (scene) are captured by RGB cameras from a plurality of different viewpoints, and silhouettes (RGB images) of the subject (scene) from the plurality of different viewpoints are acquired.

Note that Process 1 and Process 2 can be executed simultaneously.

(Process 3) A subject three-dimensional shape (polygon mesh) is generated by a visual volume intersection method (Visual Hull) that detects and executes an intersection region (And region) of the silhouettes of the subject (scene) captured from the plurality of different viewpoints by the plurality of RGB cameras.

Note that the visual volume intersection method (Visual Hull) will be described later in detail.

(Process 4) A TSDF space (TSDF Volume) is generated on the basis of the subject three-dimensional shape (polygon mesh) generated by the visual volume intersection method (Visual Hull) in the (Process 3) described above.

(Process 5) The TSDF space (TSDF Volume) based on the subject three-dimensional shape (polygon mesh) generated in the (Process 4) described above by the visual volume intersection method (Visual Hull) and the TSDF space (TSDF Volume) acquired from the depth map in the (Process 1) described above are integrated to reconfigure the three-dimensional shape of the entire subject (scene).

These processes enable a three-dimensional structure of a subject region that cannot be acquired from the TSDF space (TSDF Volume) acquired from the depth map in (Process 1) or that has low accuracy, to be complemented with the TSDF space (TSDF Volume) calculated in (Process 4) and generated from the subject three-dimensional shape (polygon mesh) generated by the visual volume intersection method (Visual Hull) based on the RGB image, which makes it possible to reconfigure the three-dimensional shape of the entire subject (scene) with high accuracy.

A specific configuration and processing of the image processing apparatus of the present disclosure will be described with reference toFIG.1and subsequent drawings.

FIG.1is a diagram for explaining a system configuration example applicable to three-dimensional shape analysis processing on a subject (scene) executed in an image processing apparatus100of the present disclosure.

InFIG.1, a target of the three-dimensional shape analysis processing is a subject1. Around the subject1, a plurality of RGB-D cameras10that captures images of the subject from different viewpoint locations is arranged.

The RGB-D camera10is a camera capable of simultaneously capturing a depth map (depth image (=distance image)) and an RGB image.

For example, as illustrated inFIGS.2A and2B, the following two pieces of image data can be simultaneously captured.

The RGB image is a color image that can be captured by a normal camera.

The depth map is an image in which a density value corresponding to a distance from the camera is set for each pixel.

Note that many depth cameras measure a distance by detecting reflected light from a subject by using infrared light (IR light). For example, a depth camera of a time of flight (ToF) system uses a time during which infrared light reciprocates between the camera and the subject, and a depth camera of a structured light system measures a distance by observing a pattern of emitted infrared light with the camera.

However, as described above, there is a problem that highly accurate depth measurement becomes difficult or depth can no longer be measured for a material that absorbs infrared light (IR light), that is, a material that has weak reflection for infrared light.

Examples of a material that absorbs infrared light (IR light) include a black material, particularly black hair of oriental people. That is, for example, the depth measurement accuracy by the depth camera decreases for a black hair region.

Therefore, as illustrated inFIG.3B, in a depth map actually captured by the RGB-D camera10illustrated inFIG.1, a pixel value corresponding to a depth (distance) of a partial region such as a person's black hair is set to an incorrect value, that is, depth information with low accuracy. Even if such depth information with low accuracy is used, a highly accurate three-dimensional shape cannot be restored.

The image processing apparatus of the present disclosure restores a three-dimensional shape by using an image captured by the RGB camera for such a region.

Note that, in the example illustrated inFIG.1, an example is illustrated in which a camera capable of simultaneously capturing a depth map and an RGB image is used as a camera that captures an image of the subject1, but a configuration may be adopted in which the depth camera and the RGB camera are separately prepared, and each camera is arranged around the subject.

Note that, in the present embodiment, an example of capturing a visible-light color image of a subject by using the RGB camera will be described, but a monochrome camera may be used instead of the RGB camera. Various visible light cameras that capture images with normal visible light can be used.

Furthermore, the camera arrangement illustrated inFIG.1is an example, and the number of cameras may be further increased.

For example, as illustrated inFIG.4, a configuration may be adopted in which a large number of cameras are arranged.

Moreover, a camera may be arranged so as to capture an image from above or an image from below with respect to the subject1.

As illustrated inFIG.1, the image processing apparatus20includes an input unit50and an image processing unit100.

The input unit50inputs an RGB image that is a captured image, a depth map (depth image (=distance image)), and internal parameters and external parameters of the camera from each of the RGB-D cameras10.

The internal parameter is, for example, data such as a focal length and an optical center of the camera, and the external parameter is a parameter such as a location and an image-capturing direction of the camera.

These parameters are set as attribute information of a captured image of each camera, and are inputted to the image processing apparatus20together with the image.

The input unit50of the image processing apparatus20outputs, to the image processing unit100, the RGB image, the depth map, the internal parameter, and the external parameter inputted from each of the RGB-D cameras10.

The image processing unit100uses these pieces of input data to analyze the three-dimensional shape of the subject.

Hereinafter, a specific configuration of the image processing unit100and processing to be executed will be described with reference toFIG.5.

2. About Configuration and Processing of Image Processing Unit

FIG.5is a diagram illustrating a specific configuration example of the image processing unit100.

As illustrated inFIG.5, the image processing unit100has: a plurality of TSDF space (TSDF Volume) generation units111-1to N; a visual volume intersection method (Visual Hull) based three-dimensional shape (polygon mesh) generation unit121; a polygon mesh based TSDF space (TSDF Volume) generation unit122; a TSDF space (TSDF Volume) integration unit131; and a three-dimensional shape generation unit132.

As illustrated inFIG.5, to the image processing unit100, input data from each camera arranged around the subject is inputted via the input unit50.

The input data is depth camera data1to N and50-1to N and RGB camera data1to N and60-1to N illustrated inFIG.5.

These pieces of input data are input data from individual cameras arranged around the subject.

A specific example of this input data will be described with reference toFIG.6.

As illustrated inFIG.6, in depth camera data50,(a) a depth map,(b) camera internal parameters, and(c) camera external parameters,these pieces of data are included.

The (a) depth map is the depth map described above with reference toFIGS.2A and2B, and is an image in which a density value corresponding to a distance from the camera is set to each pixel.

The (b) camera internal parameters are parameters required for the TSDF space (TSDF Volume) generation processing based on the depth map, and are, for example, camera internal parameters such as a focal length and an optical center of the camera.

The (c) camera external parameters are also parameters required for the TSDF space (TSDF Volume) generation processing based on the depth map, and are, for example, camera external parameters such as location information and image-capturing direction information of the camera.

Note that parameters such as a location and a position of each camera can be acquired by calibration in advance.

Furthermore, as illustrated inFIG.6, in RGB image camera data60,(a) an RGB image,(b) camera internal parameters, and(c) camera external parameters,these pieces of data are included.

The (a) RGB image is the RGB image described above with reference toFIGS.2A and2B, and is an RGB color image that can be captured by a normal camera.

The (b) camera internal parameters are parameters required for calculation processing of the subject three-dimensional shape based on the visual volume intersection method (Visual Hull) based on the RGB image, and are, for example, camera internal parameters such as a focal length and an optical center of the camera.

The (c) camera external parameters are also parameters required for calculation processing of the subject three-dimensional shape based on the visual volume intersection method (Visual Hull) based on the RGB image, and are, for example, camera external parameters such as location information and image-capturing direction information of the camera.

These depth camera data and RGB camera data are inputted from individual cameras around the subject.

2-1. About Details of Processing Executed by TSDF Space (TSDF Volume) Generation Unit

Next, details of processing executed by the TSDF space (TSDF Volume) generation units111-1to N will be described.

The depth camera data1to N and50-1to N of the individual RGB-D cameras10illustrated inFIG.1are inputted to the individual TSDF space (TSDF Volume) generation units111-1to N as illustrated inFIG.5.

Each of the TSDF space (TSDF Volume) generation units111-1to N inputs depth camera data of one RGB-D camera10, and generates one “truncated signed distance function space”=TSDF space (TSDF Volume) based on the depth map acquired by one depth camera.

A processing sequence executed by each of the TSDF space (TSDF Volume) generation units111-1to N will be described with reference to the flowchart illustrated inFIG.7.

Note that the processing according to the flow illustrated inFIG.7can be executed under control of a CPU or the like having a program execution function, in accordance with a program stored in a storage unit of the image processing apparatus20, for example.

Processing of each step in the flow shown inFIG.7will be sequentially described.

Note that the processing of step S101and subsequent steps of the flow illustrated inFIG.7is started by the one TSDF space (TSDF Volume) generation unit111inputting the depth camera data described above with reference toFIG.6from one depth camera.

That is, after the following pieces of the depth camera data are inputted,(a) a depth map,(b) camera internal parameters, and(c) camera external parameters,the processing is executed.
(Step S101)

First, in step S101, the TSDF space (TSDF Volume) generation unit111sets an initial value of a voxel identifier: i.

Specifically, voxel identifier initial value setting that satisfies

is performed.

A specific example of the voxel will be described with reference toFIG.8.

The voxel is a region corresponding to a cube as a component of a three-dimensional space. As illustrated inFIG.8, one cube obtained by subdividing a TSDF space (TSDF Volume) defined as a location region in the three-dimensional space is the voxel.

A specific example of processing executed in accordance with the flow illustrated inFIG.7will be described with reference toFIG.9.

InFIG.9, a three-dimensional space including the subject1to be the analysis target in the three-dimensional structure is set as a TSDF space (TSDF Volume), and one voxel201in the TSDF space (TSDF Volume) is shown.

The voxel201exists between the subject1and a depth camera220that has captured a depth map. An optical axis direction in front of the depth camera220is defined as a Z axis (depth), and a vertical upward direction is defined as a y axis. Note that these coordinate axes are camera coordinates.

In the TSDF space (TSDF Volume) generation processing executed in accordance with the flow illustrated inFIG.7, a process of storing a signed distance value F(v) (TSDF value) in each voxel constituting the TSDF space (TSDF Volume) is executed.

As illustrated inFIG.9, the signed distance value F(v) (TSDF value) stores a distance between the voxel201and the subject with a plus or minus sign. The plus or minus sign is determined depending on whether the voxel is outside or inside the subject.

When a value of Z of the voxel201as seen from the depth camera is smaller than a value of Z of a subject surface, a plus sign is given since the voxel201is in front of the subject as seen from the camera and outside the subject. Conversely, when the value of Z of the voxel201is larger than the value of Z of the subject surface, a minus sign is given.

As described above, in each of the voxels constituting the TSDF space (TSDF Volume), the distance between the subject and the voxel is stored as the signed distance value F(v) (TSDF value).

Note that a relationship between the plus and minus signs and the inside and outside of the subject is only required to be unified in the entire system, and the outside of the subject may be expressed as minus and the inside of the subject as plus.

Note that, the voxel further stores, in addition to this signed distance value F(v) (TSDF value), a value (reliability value) indicating reliability of the signed distance value F(v) (TSDF value) in association.

As the reliability, for example, a value corresponding to received light intensity of infrared light (IR light) in each pixel unit by the depth camera is set. High reliability is obtained when the received light intensity of infrared light (IR light) is high, and low reliability is obtained when the received light intensity of infrared light (IR light) is low.

For example, a value of the reliability value is set in a range of 1.0 to 0.0 with the highest reliability=1.0 and the lowest reliability=0.

As described above, each of the voxels constituting the TSDF space (TSDF Volume) stores the signed distance value F(v) (TSDF value) between the voxel and the subject and its reliability value (for example, 0 to 1).

Returning to the flow illustrated inFIG.7, the description of the processing executed by the TSDF space (TSDF Volume) generation unit111will be continued.

In step S101, when the initial value setting of the voxel identifier: i, that is, the voxel identifier initial value setting that satisfies

is completed, the process proceeds to step S102.

In step S102, coordinates of an i-th voxel are acquired.

Here, coordinates of the i-th voxel on voxel coordinate axes are acquired by using the voxel coordinate axes.

The voxel coordinate axes are, for example, coordinate axes configured by coordinate axes x, y, and z of the TSDF space (TSDF Volume) illustrated inFIG.8.

In step S102, coordinates (x, y, z) of the selected voxel (voxel identifier=i) on the voxel coordinate axes are acquired.

Next, in step S103, coordinates (x, y, z) according to the voxel coordinate axes of the selected voxel (voxel identifier=i) acquired in step S102are converted into world coordinates.

The world coordinates are coordinates having coordinate axes corresponding to a real region where the subject and the camera are present.

From a voxel location (X, Y, Z) according to the world coordinates obtained by this conversion processing, a relative location with respect to each location of the subject and the camera on the same world coordinates can also be calculated.

Next, the voxel location (X, Y, Z) according to the world coordinates calculated in step S103is converted into camera coordinates.

The camera coordinates are coordinates according to coordinate axes with, as an origin, a location of a depth camera that has captured a depth map to be processed by the TSDF space (TSDF Volume) generation unit111.

For example, the camera coordinates are coordinates according to the coordinate axes (x, y, z) shown on the depth camera220illustrated inFIG.9.

By using the camera coordinates, the calculation processing of the signed distance value F(v) (TSDF value) stored in the voxel described above with reference toFIG.9is to be executed.

Next, in step S105, the signed distance value: F(v) is calculated on the basis of the voxel location in the camera coordinates of the voxel (identifier=i) selected in step S102and the depth value (depth (distance)) obtained from the depth map.

This processing is the processing described above with reference toFIG.9.

As described above with reference toFIG.9, the signed distance value F(v) (TSDF value) is a value in which the distance between the voxel and the subject is stored with a sign.

In step S105, the signed distance value F(v) (TSDF value) corresponding to the voxel (identifier=i) selected in step S102is calculated.

Next, in step S106, a distance absolute value included in the signed distance value F(v) (TSDF value) that is calculated in step S105and corresponds to the voxel (identifier=i) is compared with a predefined threshold value, and it is determined whether or not the distance absolute value is larger than the threshold value.

When it is determined that the distance absolute value is larger than the threshold value, the process proceeds to step S108without performing the processing of step S107.

When it is determined that the distance absolute value is not larger than the threshold value, the processing of step S107is executed.

That is, when it is determined that the distance absolute value is larger than the threshold value, the signed distance value F(v) (TSDF value) calculated in step S105is determined to be an invalid value, and storage processing for the voxel of F(v) in step S107is not performed.

This processing is, for example, processing for not storing a value unnecessary for restoring the three-dimensional shape of the subject1into the voxel. One specific example thereof is, for example, a case as illustrated inFIG.10. As illustrated inFIG.10, the signed distance value F(v) (TSDF value) calculated for a voxel202illustrated inFIG.10is to have a distance value corresponding to a distance between a back wall of the subject1and the voxel202. This distance value is a value unnecessary for restoring the three-dimensional shape of the subject1. In such a case illustrated inFIG.10, the signed distance value F(v) (TSDF value) having a large distance value is determined to be an invalid value, and is not to be stored in the voxel.

However, the example illustrated inFIG.10is an example, and necessity determination for voxel storage processing of F(v) can be performed under various conditions other than this.

Note that the threshold value is defined in advance on the basis of a subject location and a location of the TSDF space in which the voxels are arranged.

When it is determined in step S106that the distance absolute value included in the signed distance value F(v) (TSDF value) corresponding to the voxel (identifier=i) is not larger than the threshold value, the processing of step S107is executed.

In step S107, the signed distance value F(v) (TSDF value) calculated in step S105and corresponding to the voxel (identifier=i) is stored in the voxel (identifier=i).

Note that, as described above, additionally, the voxel further stores the value (reliability value) indicating reliability as well, in association with the signed distance value F(v) (TSDF value).

As the reliability, for example, a value corresponding to received light intensity of infrared light (IR light) in each pixel unit by the depth camera is set. High reliability is obtained when the received light intensity of infrared light (IR light) is high, and low reliability is obtained when the received light intensity of infrared light (IR light) is low.

For example, a value of the reliability value is set in a range of 1.0 to 0.0 with the highest reliability=1.0 and the lowest reliability=0.

In step S107, the signed distance value F(v) (TSDF value) and its reliability value (for example, 0 to 1) are stored in the voxel (identifier=i).

Step S108is a step of determining the presence or absence of an unprocessed voxel. It is determined whether or not the processing on all the voxels constituting the TSDF space (TSDF Volume) has been completed.

When there is an unprocessed voxel, the value of the voxel identifier: i is updated in step S109, and the processing in and after step S102is executed for the voxel having the updated voxel identifier.

When it is determined in step S108that the processing on all the voxels constituting the TSDF space (TSDF Volume) has been completed, the process ends.

By executing the flow illustrated inFIG.7, the signed distance value F(v) (TSDF value) between the subject and the voxel and the value (reliability value) indicating the reliability thereof are stored in each of the voxels constituting the TSDF space (TSDF Volume).

Each of the TSDF space (TSDF Volume) generation units111-1to N in the image processing unit100illustrated inFIG.5inputs depth camera data of one RGB-D camera10, and executes processing according to the flow illustrated inFIG.7.

Note that the TSDF space (TSDF Volume) used by each of the TSDF space (TSDF Volume) generation units111-1to N is one common space configured at the same location on the world coordinates.

Each of the TSDF space (TSDF Volume) generation units111-1to N stores, in a large number of voxels configured in one TSDF space (TSDF Volume), the signed distance value F(v) (TSDF value) calculated on the basis of captured images (depth maps) of cameras at different locations and the value (reliability value) indicating the reliability thereof.

N pieces of TSDF space (TSDF Volume) generated by the TSDF space (TSDF Volume) generation units111-1to N are inputted to the TSDF space (TSDF Volume) integration unit131.

2-2. About Processing Executed by Visual Volume Intersection Method (Visual Hull) Based Three-Dimensional Shape (Polygon Mesh) Generation Unit

Next, processing executed by the visual volume intersection method (Visual Hull) based three-dimensional shape (polygon mesh) generation unit121illustrated inFIG.5will be described.

The visual volume intersection method (Visual Hull) based three-dimensional shape (polygon mesh) generation unit121inputs the RGB camera data60-1to N including RGB images captured by the RGB-D cameras10that capture images of the subject1from a plurality of different locations illustrated inFIG.1.

Note that, as described above with reference toFIG.6, in each of the RGB camera data60-1to N,(a) an RGB image,(b) camera internal parameters, and(c) camera external parameters,these pieces of data are individually included.

The (a) RGB image is the RGB image described above with reference toFIGS.2A and2B, and is an RGB color image that can be captured by a normal camera.

The (b) camera internal parameters are parameters required for calculation processing of the subject three-dimensional shape based on the visual volume intersection method (Visual Hull) based on the RGB image, and are, for example, camera internal parameters such as a focal length and an optical center of the camera.

The (c) camera external parameters are also parameters required for calculation processing of the subject three-dimensional shape based on the visual volume intersection method (Visual Hull) based on the RGB image, and are, for example, camera external parameters such as location information and image-capturing direction information of the camera.

The RGB camera data including these pieces of data is inputted from individual cameras around the subject to the visual volume intersection method (Visual Hull) based three-dimensional shape (polygon mesh) generation unit121.

The visual volume intersection method (Visual Hull) based three-dimensional shape (polygon mesh) generation unit121executes the visual volume intersection method (Visual Hull) on the basis of these pieces of input data, to generate a subject three-dimensional shape (polygon mesh).

The visual volume intersection method (Visual Hull) will be described with reference toFIG.11and subsequent drawings.

FIG.11illustrates four RGB cameras around the subject1. Individual cameras capture images of the subject1from different directions. In this example, the captured image is an RGB color image, but may be a monochrome image.

Each camera captures an image of the subject1from a specific direction, and a silhouette obtained by observing the subject1from the specific direction can be acquired from the captured image.

FIG.12illustrates an example of the silhouette acquired from the captured image of each camera.FIG.12illustrates silhouettes based on captured images of cameras that capture images of the subject1from individually different directions. The silhouettes are silhouettes1to4illustrated inFIG.12. The visual volume intersection method (Visual Hull) is a method of acquiring a three-dimensional shape of a subject, by extracting and storing an AND region of silhouettes of the subject in individual RGB cameras that capture images of the subject.

Specifically, for example, the following processing is executed.

(Process 1) A target space (volume) including a subject captured by the RGB camera is defined, coordinates of each voxel in the space (volume) are projected to each camera, and whether the voxel is inside or outside the subject silhouette is determined.

(Process 2) A voxel determined to be inside the subject in all the cameras is determined to be in an internal region of a subject shape, the other voxels are determined to be external, and a label corresponding to the determination result is set for each voxel.

(Process 3) A three-dimensional shape of a subject is acquired by applying marching cubes to a space (volume) having a plurality of labeled voxels.

Note that the marching cubes is an algorithm that converts data including a plurality of voxels into a three-dimensional shape.

As described above, the visual volume intersection method (Visual Hull) based three-dimensional shape (polygon mesh) generation unit121in the image processing unit100illustrated inFIG.5executes the visual volume intersection method (Visual Hull) by applying RGB images obtained by capturing images of the subject1from different directions, to generate a subject three-dimensional shape (polygon mesh) of the subject1.

The subject three-dimensional shape (polygon mesh) of the subject1generated by the visual volume intersection method (Visual Hull) based three-dimensional shape (polygon mesh) generation unit121is inputted to the polygon mesh based TSDF space (TSDF Volume) generation unit122.

2-3. About Processing Executed by Polygon Mesh Based TSDF Space (TSDF Volume) Generation Unit

Next, processing executed by the polygon mesh based TSDF space (TSDF Volume) generation unit122will be described.

The polygon mesh based TSDF space (TSDF Volume) generation unit122converts, into a TSDF space (TSDF Volume), the subject three-dimensional shape (polygon mesh) of the subject1generated by the visual volume intersection method (Visual Hull) based three-dimensional shape (polygon mesh) generation unit121.

As described above, the TSDF space (TSDF Volume) is a data structure in which the signed distance value F(v) (TSDF value) is stored in a large number of voxels inside the space.

This data structure can be generated from the three-dimensional structure (polygon mesh) of the subject.

As described above with reference toFIG.9and the like, the signed distance value F(v) (TSDF value) between the subject and the voxel is stored in each voxel in the TSDF space (TSDF Volume).

If a subject shape is known, and a location of each voxel in the TSDF space (TSDF Volume) including the subject is also known, a distance to the subject and whether it is inside or outside the subject can be analyzed for each voxel. That is, the signed distance value F(v) (TSDF value) can be calculated.

In this way, the polygon mesh based TSDF space (TSDF Volume) generation unit122converts, into a TSDF space (TSDF Volume), the subject three-dimensional shape (polygon mesh) of the subject1generated by the visual volume intersection method (Visual Hull) based three-dimensional shape (polygon mesh) generation unit121.

Note that a location (location on the world coordinates) of the TSDF space (TSDF Volume) generated by the polygon mesh based TSDF space (TSDF Volume) generation unit122is set to the same location as the locations of the N pieces of TSDF space (TSDF Volume) generated by the TSDF space (TSDF Volume) generation units111-1to N described above. That is, a common TSDF space (TSDF Volume) is used.

As a result, all locations of voxels in the TSDF space (TSDF Volume) generated by the polygon mesh based TSDF space (TSDF Volume) generation unit122and the TSDF space (TSDF Volume) are to be the same location in one common TSDF space (TSDF Volume).

Note that, for each voxel of the TSDF space (TSDF Volume) generated by the polygon mesh based TSDF space (TSDF Volume) generation unit122, the reliability value=0.5, which is an intermediate value, is stored in addition to the signed distance value F(v) (TSDF value), when a predetermined reliability value, for example, reliability=0 to 1.0 is set.

The TSDF space (TSDF Volume) generated by the polygon mesh based TSDF space (TSDF Volume) generation unit122is inputted to the TSDF space (TSDF Volume) integration unit131.

2-4. About Processing Executed by TSDF Space (TSDF Volume) Integration Unit

Next, processing executed by the TSDF space (TSDF Volume) integration unit131will be described.

The TSDF space (TSDF Volume) integration unit131inputs the following data.(A) The N pieces of TSDF space (TSDF Volume) generated by the TSDF space (TSDF Volume) generation units111-1to N(B) The TSDF space (TSDF Volume) generated by the polygon mesh based TSDF space (TSDF Volume) generation unit122

The (A) described above is the N pieces of TSDF space (TSDF Volume) generated on the basis of N pieces of depth map obtained by capturing images of the subject from N different locations by N pieces of RGB-D camera10.

Furthermore, (B) is one TSDF space (TSDF Volume) generated on the basis of a subject three-dimensional shape (polygon mesh) generated by the visual volume intersection method (Visual Hull) to which N pieces of RGB image obtained by capturing images of the subject from N different locations by N pieces of RGB-D camera10are applied.

Note that, as described above, the TSDF space (TSDF Volume) generated by the polygon mesh based TSDF space (TSDF Volume) generation unit122and the TSDF space (TSDF Volume) are at the same location on the world coordinates, and all locations of voxels in these N+1 pieces of TSDF space (TSDF Volume) are also at the same location.

In the voxels in the N+1 pieces of TSDF space (TSDF Volume), individually different signed distance values F(v) (TSDF values) and their reliability values (0 to 1) are stored.

When a plurality of signed distance values F(v) (TSDF values) is detected in voxels at the same location in the N+1 pieces of TSDF space (TSDF Volume), the TSDF space (TSDF Volume) integration unit131calculates a final one signed distance value F′(v) on the basis of the plurality of signed distance values F(v).

For example, the one final signed distance value F′(v) is calculated by executing weighting addition to which a reliability value recorded in association with each signed distance value F(v) is applied.

A specific calculation formula of the final one signed distance value F′(v) is, for example, the following (Equation 1).

i is an identifier of a TSDF space (TSDF Volume),

Fi(v) is a signed distance value in a voxel v of the TSDF space (TSDF Volume) i, and

wi (v) is a weight value in the voxel v of the TSDF space (TSDF Volume) i.

As a weight value, reliability (0 to 1) of a TSDF value for each voxel can be applied.

Note that, as the reliability, for example, a luminance value of infrared light (IR light) in each pixel described above can be used. Furthermore, in addition to this, the reliability may be calculated by using the following camera arrangement information,(a) an angle between a normal direction of a subject surface and a camera location, and(b) a distance from the camera to the subject surface,

and used as a weight value of (Equation 1) described above.

As described above, the TSDF space (TSDF Volume) integration unit131uses the signed distance values F(v) stored in the corresponding voxels of the N+1 pieces of TSDF space (TSDF Volume), to generate one integrated TSDF space (TSDF Volume) in which one signed distance value F(v) is stored in each voxel, at the end.

The one integrated TSDF space (TSDF Volume) generated by the TSDF space (TSDF Volume) integration unit131is outputted to the three-dimensional shape generation unit132.

2-5. About Processing Executed by Three-Dimensional Shape Generation Unit

Next, processing executed by the three-dimensional shape generation unit132will be described.

The three-dimensional shape generation unit132inputs the one integrated TSDF space (TSDF Volume) generated by the TSDF space (TSDF Volume) integration unit131, from the TSDF space (TSDF Volume) integration unit131.

The three-dimensional shape generation unit132restores the three-dimensional shape of the subject by using this one integrated TSDF space (TSDF Volume).

One signed distance value F(v) is stored in a voxel of the one integrated TSDF space (TSDF Volume), and the three-dimensional structure of the subject can be restored on the basis of this value.

For example, by extracting a location of zero-cross by the marching cubes, a mesh that is a highly accurate surface shape of the subject can be acquired, and a three-dimensional structure of the subject including the mesh can be generated.

Note that, as described above, in the depth map, accuracy of depth information of a region such as black hair is low, and a low reliability signed distance value F(v) is stored in the voxel of the TSDF space (TSDF Volume) generated on the basis of the depth map. Therefore, even if an attempt is made to restore the three-dimensional shape of the subject by using this value, a highly accurate three-dimensional shape cannot be restored.

In a technique of the present disclosure, for such a region, it is possible to perform three-dimensional shape restoration to which the signed distance value F(v) stored in the voxel of the TSDF space (TSDF Volume) generated on the basis of the RGB image is applied, and highly accurate three-dimensional shape restoration of the entire subject is achieved.

3. About Processing Sequence Executed by Image Processing Apparatus of Present Disclosure

Next, a processing sequence executed by the image processing apparatus20of the present disclosure will be described.

The flowchart illustrated inFIG.13is a flowchart for explaining a processing sequence executed by the image processing apparatus20of the present disclosure. Note that the processing according to the flow illustrated inFIG.13can be executed under control of a CPU or the like having a program execution function, in accordance with a program stored in a storage unit of the image processing apparatus20, for example.

Processing of each step in the flow shown inFIG.13will be sequentially described.

Note that the processing of steps S211to S213of the flow illustrated inFIG.13is processing executed by the TSDF space (TSDF Volume) generation unit111illustrated inFIG.5, and the processing of steps S221to S223is processing executed by the visual volume intersection method (Visual Hull) based three-dimensional shape (polygon mesh) generation unit121and the polygon mesh based TSDF space (TSDF Volume) generation unit122illustrated inFIG.5.

Note that, the processing of steps S211to S213and the processing of steps S221to S223are processing that can be executed independently in parallel.

First, the processing of steps S211to S213will be described.

The processing of steps S211to S213is processing executed in the TSDF space (TSDF Volume) generation unit111illustrated inFIG.5.

First, in step S211, the TSDF space (TSDF Volume) generation unit111inputs depth camera data from each of the RGB-D cameras10illustrated inFIG.1.

The depth camera data contains the following data described above with reference toFIG.6:(a) depth map,(b) camera internal parameters, and(c) camera external parameters.

The (a) depth map is the depth map described above with reference toFIGS.2A and2B, and is an image in which a density value corresponding to a distance from the camera is set to each pixel.

The (b) camera internal parameters are parameters required for the TSDF space (TSDF Volume) generation processing based on the depth map, and are, for example, camera internal parameters such as a focal length and an optical center of the camera.

The (c) camera external parameters are also parameters required for the TSDF space (TSDF Volume) generation processing based on the depth map, and are, for example, camera external parameters such as location information and image-capturing direction information of the camera.

Next, in step S212, the TSDF space (TSDF Volume) generation unit111inputs the depth camera data, and generates a TSDF space (TSDF Volume).

This processing corresponds to the flow illustrated inFIG.7and the processing described above with reference toFIGS.8and9. Specifically, a process of storing the signed distance value F(v) (TSDF value) in each voxel constituting the TSDF space (TSDF Volume) is executed.

As illustrated inFIG.9, the signed distance value F(v) (TSDF value) is a value including a distance between the voxel201and the subject and including a plus or minus sign indicating whether the subject is inside or outside.

Note that, additionally, the voxel also stores a value (reliability value) indicating reliability, in association with the signed distance value F(v) (TSDF value).

Step S213next is determination processing as to whether or not generation of the TSDF space (TSDF Volume) based on the depth maps acquired by all the depth cameras has been completed.

When there is an unprocessed depth map, the processing of steps S211to S212is executed for the unprocessed depth map.

When it is determined that the generation of the TSDF space (TSDF Volume) based on the depth maps acquired by all the depth cameras has been completed, the process proceeds to step S231.

Note that, at this point, for example, when there are N pieces of depth camera, it means that N pieces of TSDF space (TSDF Volume) are generated.

In voxels of the N pieces of TSDF space (TSDF Volume), the signed distance value F(v) (TSDF value) and its reliability value are stored.

The N pieces of TSDF space (TSDF Volume) generated by the TSDF space (TSDF Volume) generation unit111through the processing of steps S211to S213are inputted to the TSDF space (TSDF Volume) integration unit131.

Next, a description is given to the processing of steps S221to S223executed in the visual volume intersection method (Visual Hull) based three-dimensional shape (polygon mesh) generation unit121and the polygon mesh based TSDF space (TSDF Volume) generation unit122illustrated inFIG.5.

First, in step S221, the visual volume intersection method (Visual Hull) based three-dimensional shape (polygon mesh) generation unit121inputs the RGB camera data60-1to N including RGB images captured by the RGB-D cameras10that capture images of the subject1from a plurality of different locations illustrated inFIG.1.

As described above with reference toFIG.6, in each of the RGB camera data60-1to N,(a) an RGB image,(b) camera internal parameters, and(c) camera external parameters,these pieces of data are individually included.

The (a) RGB image is the RGB image described above with reference toFIGS.2A and2B, and is an RGB color image that can be captured by a normal camera.

The (b) camera internal parameters are parameters required for calculation processing of the subject three-dimensional shape based on the visual volume intersection method (Visual Hull) based on the RGB image, and are, for example, camera internal parameters such as a focal length and an optical center of the camera.

The (c) camera external parameters are also parameters required for calculation processing of the subject three-dimensional shape based on the visual volume intersection method (Visual Hull) based on the RGB image, and are, for example, camera external parameters such as location information and image-capturing direction information of the camera.

Next, in step S222, the visual volume intersection method (Visual Hull) based three-dimensional shape (polygon mesh) generation unit121executes the visual volume intersection method (Visual Hull), to generate a subject three-dimensional shape (polygon mesh).

This processing is the processing described above with reference toFIGS.11and12.

Specifically, for example, the following processing is executed.

(Process 1) A target space (volume) including a subject captured by the RGB camera is defined, coordinates of each voxel in the space (volume) are projected to each camera, and whether the voxel is inside or outside the subject silhouette is determined.

(Process 2) A voxel determined to be inside the subject in all the cameras is determined to be in an internal region of a subject shape, the other voxels are determined to be external, and a label corresponding to the determination result is set for each voxel.

(Process 3) A three-dimensional shape of a subject is acquired by applying marching cubes to a space (volume) having a plurality of labeled voxels.

Next, in the polygon mesh based TSDF space (TSDF Volume) generation unit122, in step S223, the polygon mesh based TSDF space (TSDF Volume) generation unit122converts, into a TSDF space (TSDF Volume), the subject three-dimensional shape (polygon mesh) of the subject1generated by the visual volume intersection method (Visual Hull) based three-dimensional shape (polygon mesh) generation unit121.

Note that, for each voxel of the TSDF space (TSDF Volume) generated by the polygon mesh based TSDF space (TSDF Volume) generation unit122, the reliability value=0.5, which is an intermediate value, is stored in addition to the signed distance value F(v) (TSDF value), when a predetermined reliability value, for example, reliability=0 to 1.0 is set.

Note that the value of the reliability value can be variously set.

The TSDF space (TSDF Volume) generated by the polygon mesh based TSDF space (TSDF Volume) generation unit122is inputted to the TSDF space (TSDF Volume) integration unit131.

The processing of step S231is processing executed by the TSDF space (TSDF Volume) integration unit131.

The TSDF space (TSDF Volume) integration unit131inputs the following data.(A) The N pieces of TSDF space (TSDF Volume) generated by the TSDF space (TSDF Volume) generation units111-1to N(B) The TSDF space (TSDF Volume) generated by the polygon mesh based TSDF space (TSDF Volume) generation unit122

In voxels in the N+1 pieces of TSDF space (TSDF Volume) in total of (A) and (B) described above, there are voxels storing individually different signed distance values F(v) (TSDF values) and their reliability values (0 to 1).

When a plurality of signed distance values F(v) (TSDF values) is detected in voxels at the same location in the N+1 pieces of TSDF space (TSDF Volume), on the basis of the plurality of signed distance values F(v), the TSDF space (TSDF Volume) integration unit131calculates a final one signed distance value F′(v) in accordance with (Equation 1) described above.

As described above, the TSDF space (TSDF Volume) integration unit131uses the signed distance values F(v) stored in the corresponding voxels of the N+1 pieces of TSDF space (TSDF Volume), to generate one integrated TSDF space (TSDF Volume) in which one signed distance value F(v) is stored in each voxel, at the end.

The one integrated TSDF space (TSDF Volume) generated by the TSDF space (TSDF Volume) integration unit131is outputted to the three-dimensional shape generation unit132.

The processing in the final step S232is processing executed by the three-dimensional shape generation unit132of the image processing unit100illustrated inFIG.5.

The three-dimensional shape generation unit132inputs the one integrated TSDF space (TSDF Volume) generated by the TSDF space (TSDF Volume) integration unit131, from the TSDF space (TSDF Volume) integration unit131.

The one signed distance value F(v) is stored in the voxel of the integrated TSDF space (TSDF Volume).

The three-dimensional shape generation unit132can restore the three-dimensional structure of the subject by analyzing the signed distance values F(v) of all the voxels on the basis of this voxel storage value.

For example, by extracting a location of zero-cross by the marching cubes, a mesh that is a highly accurate surface shape of the subject can be acquired, and a three-dimensional structure of the subject including the mesh can be generated.

As described above, the image processing apparatus of the present disclosure generates an integrated TSDF space (TSDF Volume) obtained by integrating a TSDF space (TSDF Volume) generated on the basis of a depth map and an RGB image-based TSDF space (TSDF Volume), and analyzes a three-dimensional shape of the subject on the basis of this integrated TSDF space (TSDF Volume), to generate highly accurate three-dimensional structure data of the subject.

In the depth map, accuracy of depth information of a region such as black hair is low, and a low reliability signed distance value F(v) is stored in the voxel of the TSDF space (TSDF Volume) generated on the basis of the depth map. Therefore, even if an attempt is made to restore the three-dimensional shape of the subject by using this value, a highly accurate shape cannot be restored.

However, in a technique of the present disclosure, for such a region, it is possible to perform three-dimensional shape restoration to which the signed distance value F(v) stored in the voxel of the TSDF space (TSDF Volume) generated on the basis of the RGB image is applied, and highly accurate three-dimensional shape restoration of the entire subject is achieved.

4. About Specific Example of Three-Dimensional Structure Data of Subject Obtained by Applying Processing of Present Disclosure

Next, a specific example of three-dimensional structure data of the subject obtained by applying the processing of the present disclosure will be described.

FIG.14is a diagram for explaining a specific example of data generated in accordance with processing executed by the image processing apparatus of the present disclosure.

First, a depth map270and an RGB image280illustrated inFIG.14are acquired from the RGB-D camera that captures an image of the subject from various directions.

Note that the depth map270and the RGB image280illustrated inFIG.14are N pieces of depth map and N pieces of RGB image obtained by capturing images of the subject from various directions.

The N pieces of depth map270are inputted to the TSDF space (TSDF Volume) generation unit111, and N pieces of TSDF space (TSDF Volume)271corresponding to the individual depth maps are generated.

An example of the subject three-dimensional structure generated on the basis of the N pieces of TSDF space (TSDF Volume)211is three-dimensional structure data272illustrated inFIG.14.

Since the three-dimensional structure data272is generated using only the depth map, a region where a low accuracy depth value is set, for example, a black hair region is to be incomplete three-dimensional structure data in which the three-dimensional structure cannot be restored.

The three-dimensional structure data272illustrated inFIG.14is an example of data when the processing of the present disclosure is not applied.

Next, processing when the processing of the present disclosure is applied will be described.

The RGB image281is inputted to the visual volume intersection method (Visual Hull) based three-dimensional shape (polygon mesh) generation unit121, and first, a silhouette281illustrated inFIG.14is generated.

This is the processing described above with reference toFIGS.11and12.

Moreover, on the basis of the silhouette281, the subject three-dimensional shape (polygon mesh)282is generated by the visual volume intersection method (Visual Hull).

Moreover, the polygon mesh based TSDF space (TSDF Volume) generation unit122generates a polygon mesh based TSDF space (TSDF Volume)283on the basis of the subject three-dimensional shape (polygon mesh)282.

Next, the TSDF space (TSDF Volume) integration unit131inputs the following data.(A) The N pieces of TSDF space (TSDF Volume) corresponding to the individual depth maps271(B) The polygon mesh based TSDF space (TSDF Volume)283

The TSDF space (TSDF Volume) integration unit131synthesizes the plurality of TSDF spaces (TSDF Volumes), to generate one integrated TSDF space (TSDF Volume)291.

Finally, on the basis of the one integrated TSDF space (TSDF Volume)291, the three-dimensional shape generation unit132generates three-dimensional structure data292of the subject.

The three-dimensional structure data292of the subject illustrated inFIG.14is three-dimensional structure data generated on the basis of the integrated TSDF space (TSDF Volume) obtained by integrating the TSDF space (TSDF Volume) generated on the basis of the depth map and the RGB image-based TSDF space (TSDF Volume).

Although voxels in the TSDF space (TSDF Volume) generated on the basis of the depth map include voxels storing less accurate signed distance values F(v), it becomes possible to utilize a signed distance value F(v) of the RGB image-based TSDF space (TSDF Volume) for such voxels by applying the processing of the present disclosure.

As a result, highly accurate three-dimensional structure data of the subject can be generated.

5. About Other Embodiments

Next, other embodiments (modified examples) of the image processing apparatus of the present disclosure will be described.

In the above-described embodiment, as described above with reference toFIG.1, a configuration has been adopted in which a plurality of cameras is arranged around a subject, and processing is performed using images captured by the plurality of cameras.

Rather than using the plurality of cameras in this way, a configuration may be adopted in which only one camera is used. For example, as illustrated inFIG.15, a configuration may be adopted in which one RGB-D camera10sequentially captures images (RGB image, depth map) while moving around the subject1.

However, in a case of this method, it is necessary to acquire a location and a position of the camera corresponding to an image-capturing frame at each time. For camera location and position acquisition processing at each time point, an existing method, for example, a method such as KinectFusion or Visual SLAM, or a method such as Structure From Motion can be used.

Moreover, as illustrated inFIG.16, a configuration may be adopted in which, by continuously capturing an image of the subject while rotationally moving the subject1in front of one fixed RGB-D camera10, subject images (RGB image, depth map) are sequentially captured from different locations.

For example, the subject1is rotationally moved on a turntable. In this case, similarly to Modified example 1 described above, a method represented by visual SLAM and the like can be used for the camera location and position acquisition processing.

Even when the motion of the subject1is a non-rigid motion, by applying an existing method, for example, a method such as “DynamicFusion”, the camera location and position can be estimated by combining estimation of Warp-Field that parametrically expresses the non-rigid motion in each video frame.

Moreover, as a modified example of the image processing apparatus of the present disclosure, it is possible to adopt a configuration of reducing a calculation amount of the generation processing of the TSDF space (TSDF Volume) to which the visual volume intersection method (Visual Hull) described above is applied.

That is, a configuration may be adopted in which, without calculating the signed distance values F(v) (TSDF values) of all voxels for the TSDF space (TSDF Volume) to which the visual volume intersection method (Visual Hull) described above is applied, the signed distance value F(v) (TSDF value) is calculated by selecting only voxels corresponding to a region where the reliability of the depth value acquired from the depth map is low.

As described above, the reliability of the depth value of the depth map can be determined by high or low of a luminance value of the infrared light (IR light). A depth value of a pixel having a low luminance value of infrared light (IR light) is determined to have low reliability.

A configuration may be adopted in which only voxels corresponding to such a region where the reliability of the depth value is low are selected from the TSDF space (TSDF Volume) to which the visual volume intersection method (Visual Hull) is applied, and the signed distance value F(v) (TSDF value) is calculated only for these selected voxels.

Furthermore, when a subject region with low reliability of the depth value of the depth map can be estimated in advance, only voxels corresponding to the subject region may be set as the voxels to be subjected to the signed distance value F(v) (TSDF value) calculation of the TSDF space (TSDF Volume) to which the visual volume intersection method (Visual Hull) is applied.

For example, when the three-dimensional shape restoration target is a human, a region of a head part (hair) can be estimated as a region where the reliability of the depth value of the depth map is low. In this case, only voxels corresponding to the hair region of the subject are set as voxels to be subjected to the signed distance value F(v) (TSDF value) calculation of the TSDF space (TSDF Volume) to which the visual volume intersection method (Visual Hull) is applied.

In the embodiment described above, the TSDF space (TSDF Volume) integration unit131has executed the following processing.

That is, with(A) the N pieces of TSDF space (TSDF Volume) generated by the TSDF space (TSDF Volume) generation units111-1to N, and(B) the TSDF space (TSDF Volume) generated by the polygon mesh based TSDF space (TSDF Volume) generation unit122,

when a plurality of signed distance values F(v) (TSDF values) has been detected in voxels at the same location in these N+1 pieces of TSDF space (TSDF Volume), a final one signed distance value F′(v) has been calculated in accordance with (Equation 1) described above, on the basis of the plurality of signed distance values F(v).

Moreover, a configuration has been adopted in which one integrated TSDF space (TSDF Volume) having the signed distance value F′(v) is generated, and the three-dimensional shape is restored on the basis of the one integrated TSDF space (TSDF Volume).

However, with(A) the N pieces of TSDF space (TSDF Volume) generated by the TSDF space (TSDF Volume) generation units111-1to N, and(B) the TSDF space (TSDF Volume) generated by the polygon mesh based TSDF space (TSDF Volume) generation unit122,an optimum synthesis ratio of these (A) and (B) is expected to vary depending on the subject.

A modified example described below has a configuration in which a user can input and change a weight coefficient, and has a configuration in which three-dimensional structure data generated in accordance with a setting value of the user is displayed on a display unit.

The user can perform such a process of: observing three-dimensional structure data according to various synthesis ratios displayed on the display unit; selecting optimum three-dimensional structure data; determining, as an optimum value, a synthesis ratio at a point of time when the display data is displayed; and storing, for example, the three-dimensional structure data in a storage unit together with the value of the synthesis ratio.

FIG.17illustrates a configuration example of the image processing apparatus of the present embodiment.

FIG.17illustrates a configuration in which an internal configuration of the image processing unit100described above with reference toFIG.5is partially changed, and a weight coefficient input unit401and a display unit450are further added.

The internal configuration of the image processing unit100is changed as follows.

(1) Addition of a depth map based TSDF space (TSDF Volume) integration unit311.

(2) A change of the TSDF space (TSDF Volume) integration unit312to have a configuration of inputting:

one depth map based TSDF space (TSDF Volume) generated by the depth map based TSDF space (TSDF Volume) integration unit311; and a TSDF space (TSDF Volume) generated by the polygon mesh based TSDF space (TSDF Volume) generation unit122.

(3) A point where a weight coefficient α421inputted by the user from the weight coefficient input unit401can be inputted to the TSDF space (TSDF Volume) integration unit312.

(4) A point where a configuration is adopted in which three-dimensional structure data of the subject generated by the three-dimensional shape generation unit132is outputted to the display unit450.

First, the depth map based TSDF space (TSDF Volume) integration unit311executes integration processing on the N pieces of TSDF space (TSDF Volume) generated by the TSDF space (TSDF Volume) generation units111-1to N, and generates one depth map based TSDF space (TSDF Volume).

The generation processing of the one depth map based TSDF space (TSDF Volume) can be executed by processing similar to the integration processing executed in the TSDF space (TSDF Volume) integration unit131described above in the embodiment having the configuration illustrated inFIG.5.

That is, the one depth map based TSDF space (TSDF Volume) in which one signed distance value F(v) (TSDF value) is set for each voxel is generated by weighting addition based on a reliability value of the signed distance value F(v) (TSDF value) of each corresponding voxel.

However, here, only the depth map based TSDF space (TSDF Volume) is to be the integration processing target.

As described above, the depth map based TSDF space (TSDF Volume) integration unit311executes integration processing on the N pieces of TSDF space (TSDF Volume) generated by the TSDF space (TSDF Volume) generation units111-1to N, and generates the one depth map based TSDF space (TSDF Volume).

The generated one depth map based TSDF space (TSDF Volume) is inputted to the TSDF space (TSDF Volume) integration unit312.

The TSDF space (TSDF Volume) integration unit312inputs the following two TSDF spaces (TSDF Volumes):(A) the one depth map based TSDF space (TSDF Volume) generated by the depth map based TSDF space (TSDF Volume) integration unit311; and(B) the TSDF space (TSDF Volume) generated by the polygon mesh based TSDF space (TSDF Volume) generation unit122, and synthesizes these two TSDF spaces (TSDF Volumes).

The weight coefficient (a) corresponding to a synthesis ratio of the two TSDF spaces (TSDF Volumes) is inputted by the user via the weight coefficient input unit401, and can be changed at any time.

As illustrated inFIG.17, the user can input the weight coefficient (a)421to the TSDF space (TSDF Volume) integration unit312via the weight coefficient input unit401.

In accordance with the weight coefficient (a)421inputted by the user, the TSDF space (TSDF Volume) integration unit312executes synthesis processing of the following two TSDF spaces (TSDF Volumes):(A) the one depth map based TSDF space (TSDF Volume) generated by the depth map based TSDF space (TSDF Volume) integration unit311; and(B) the TSDF space (TSDF Volume) generated by the polygon mesh based TSDF space (TSDF Volume) generation unit122,

to generate one integrated TSDF space (TSDF Volume).

Specifically, for example, the signed distance value F′(v) to be stored in each voxel of the integrated TSDF space (TSDF Volume) is calculated in accordance with (Equation 2) shown below.
[Formula 2]
F′(v)=αFvisualHull(v)+(1−α)FDepth(v)   (EQUATION 2)

FvisualHull(V) is a signed distance value stored in a voxel of the TSDF space (TSDF Volume) generated by the polygon mesh based TSDF space (TSDF Volume) generation unit122,

FDepth(v) is a signed distance value stored in a voxel of the one depth map based TSDF space (TSDF Volume) generated by the depth map based TSDF space (TSDF Volume) integration unit311, and a is the weight coefficient α and corresponds to a ratio of a polygon mesh-based TSDF space (TSDF Volume) included in the one integrated TSDF space (TSDF Volume) generated by the TSDF space (TSDF Volume) integration unit312.

The one integrated TSDF space (TSDF Volume) generated by the TSDF space (TSDF Volume) integration unit312is inputted to the three-dimensional shape generation unit132.

The three-dimensional shape generation unit132generates three-dimensional structure data of the subject on the basis of the one integrated TSDF space (TSDF Volume) generated by the TSDF space (TSDF Volume) integration unit312, and outputs and displays the generated three-dimensional structure data of the subject on the display unit450.

The user can change the weight coefficient (α) while observing the three-dimensional structure data of the subject displayed on the display unit450, and can stop the change of the weight coefficient (α) at a time when the high-quality three-dimensional structure data is displayed, and store the three-dimensional structure data in a storage unit together with the weight coefficient (α).

FIGS.18A,18B, and18Cillustrate a specific example of the three-dimensional structure data displayed on the display unit450in accordance with the change of the weight coefficient (α).

FIGS.18A,18B, and18Cillustrate an example of these three types of display data of:

FIG.18Athree-dimensional structure data with setting of the weight coefficient α=0.0,

FIG.18Bthree-dimensional structure data with setting of the weight coefficient α=0.5, and

FIG.18Cthree-dimensional structure data with setting of the weight coefficient α=1.0.

Note that the weight coefficient α corresponds to a ratio of a polygon mesh-based TSDF space (TSDF Volume) included in the one integrated TSDF space (TSDF Volume) generated by the TSDF space (TSDF Volume) integration unit312.

TheFIG.18Athree-dimensional structure data with setting of the weight coefficient α=0.0 is three-dimensional structure data generated only on the basis of the depth map based TSDF space (TSDF Volume).

TheFIG.18Cthree-dimensional structure data with setting of the weight coefficient α=1.0 is three-dimensional structure data generated only on the basis of the polygon mesh-based TSDF space (TSDF Volume) based on the RGB image.

TheFIG.18Bthree-dimensional structure data with setting of the weight coefficient α=0.5 is three-dimensional structure data generated only on the basis of a TSDF space (TSDF Volume) generated with a synthesis ratio of 1:1 between the depth map based TSDF space (TSDF Volume) and the polygon mesh-based TSDF space (TSDF Volume) based on the RGB image.

In this way, the user can perform such a process of: observing the three-dimensional structure data corresponding to various values of the weight coefficient α displayed on the display unit450; selecting optimum three-dimensional structure data; determining, as an optimum value, the weight coefficient α at a point of time when the display data is displayed; and storing, for example, the three-dimensional structure data in a storage unit together with the value of the weight coefficient (α).

6. About Hardware Configuration Example of Image Processing Apparatus

Next, a hardware configuration example of the image processing apparatus of the present disclosure will be described with reference toFIG.19.

For example, specifically, the image processing apparatus of the present disclosure can be configured by an information processing apparatus such as a personal computer (PC).

As illustrated inFIG.19, an image processing apparatus500includes a controller501, a read only memory (ROM)502, a random access memory (RAM)503, an input/output interface505, and a bus504that connects these components to each other.

The controller501appropriately accesses the RAM503and the like as necessary, and integrally controls each of the entire blocks of the image processing apparatus500while performing various kinds of arithmetic processing. The controller501may be a central processing unit (CPU), a graphics processing unit (GPU), or the like. The ROM502is a nonvolatile memory that fixedly stores firmware to be executed by the CPU501, such as an OS, programs, and various parameters. The RAM503is used as a work region or the like of the CPU501, and temporarily holds the OS, various applications being executed, and various data being processed.

To the input/output interface505, an input unit506, an output unit507, a storage unit508, a communication unit509, a drive510to which a removable medium511can be attached, and the like are connected.

Note that the input/output interface505may be configured to be connectable to an external peripheral device via a universal serial bus (USB) terminal, an IEEE terminal, or the like, in addition to these individual elements.

The input unit506is, for example, an input unit that can input a captured image (RGB image, depth map) of the RGB-D camera10illustrated inFIG.1, and can further input user's operation information and the like. A configuration may be adopted in which the input unit506includes an imaging unit.

The input unit506also includes, for example, a pointing device such as a mouse, a keyboard, a touch panel, and other input devices.

The output unit507outputs data such as an image and sound. A configuration may be adopted in which the output unit507includes a display unit.

The storage unit508is, for example, a nonvolatile memory such as a hard disk drive (HDD), a flash memory (solid state drive; SSD), or other solid state memory. The storage unit508stores an OS, various applications, and various data. The storage unit508is also used as a storage region for an input image, image information, a generated output image group, and the like.

The communication unit509is, for example, a network interface card (NIC) for Ethernet (registered trademark), and is responsible for communication processing via a network.

The drive510is used for data recording and reproduction processing using the removable medium511.

The removable medium511includes, for example, a BD, a DVD, a CD, an HDD, a flash memory, or the like.

7. Summary of Configuration of Present Disclosure

The embodiment of the present disclosure has been described in detail with reference to the specific embodiment. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiment without departing from the scope of the present disclosure. In other words, the present invention has been disclosed in the form of exemplification, and should not be construed as limiting. In order to determine the scope of the present disclosure, the section of the claims should be taken into consideration.

Note that the technology disclosed in this specification can have the following configurations.

(1) An image processing apparatus including:an image processing unit configured to input captured images of a depth camera and a visible light camera, to analyze a three-dimensional shape of a subject, in whichthe image processing unitgenerates a depth map based TSDF space (TSDF Volume) by using a depth map acquired from a captured image of the depth camera,generates a visible light image based TSDF space by using a captured image of the visible light camera,generates an integrated TSDF space by integration processing on the depth map based TSDF space and the visible light image based TSDF space, andexecutes three-dimensional shape analysis processing on the subject by using the integrated TSDF space.

(2) The image processing apparatus according to (1), in whichthe image processing unitgenerates a three-dimensional shape (polygon mesh) of the subject from a captured image of the visible light camera by applying a visual volume intersection method (Visual Hull), and generates the visible light image based TSDF space by using the generated three-dimensional shape (polygon mesh).

(3) The image processing apparatus according to (2), in whichthe image processing unitdetects an intersection region of a plurality of silhouettes acquired from a plurality of visible light camera-captured images obtained by capturing images of the subject from a plurality of different directions, to generate a three-dimensional shape (polygon mesh) of the subject.

(4) The image processing apparatus according to any one of (1) to (3), in whichthe image processing unitgenerates a plurality of depth map based TSDF spaces by using a plurality of depth maps acquired from captured images obtained by capturing images of the subject with the depth camera from a plurality of different directions, andgenerates the integrated TSDF space by integration processing on the plurality of depth map based TSDF spaces and the visible light image based TSDF space.

(5) The image processing apparatus according to any one of (1) to (4), in whichthe image processing unitgenerates a TSDF space in which a signed distance value F(v) (TSDF value) including distance information between the subject and a voxel is stored in a voxel constituting each TSDF space of the depth map based TSDF space and the visible light image based TSDF space.

(6) The image processing apparatus according to (5), in whichthe image processing unitstores, in a voxel of the integrated TSDF space, a value calculated by executing weighting addition of a signed distance value F(v) (TSDF value) in a voxel of each of the depth map based TSDF space and the visible light image based TSDF space.

(7) The image processing apparatus according to (6), in whichthe image processing unit applies,as a weight value to be applied to the weighting addition,a reliability value recorded in association with a signed distance value F(v) (TSDF value) in a voxel of the depth map based TSDF space.

(8) The image processing apparatus according to (7), in whichthe depth camera is configured to perform depth map generation processing using infrared light, andthe reliability value is a value corresponding to a pixel value of a captured image of the depth camera.

(9) The image processing apparatus according to any one of (1) to (8), in whichthe image processing unitinputs subject images from different directions captured by a plurality of cameras arranged around the subject, and executes three-dimensional shape analysis processing on the subject.

(10) The image processing apparatus according to any one of (1) to (8), in whichthe image processing unitinputs subject images from different directions captured by a camera that moves around the subject, and executes three-dimensional shape analysis processing on the subject.

(11) The image processing apparatus according to any one of (1) to (8), in whichthe image processing unitinputs subject images from different directions captured by a fixed camera that captures an image of the subject that rotationally moves, and executes three-dimensional shape analysis processing on the subject.

(12) The image processing apparatus according to any one of (1) to (11), further including:an input unit configured to input a weight coefficient to be applied to integration processing on the depth map based TSDF space and the visible light image based TSDF space, in whichthe image processing unitintegrates the depth map based TSDF space and the visible light image based TSDF space to generate the integrated TSDF space, in accordance with a weight coefficient inputted via the input unit.

(13) The image processing apparatus according to (12), further including:a display unit configured to display three-dimensional structure data of the subject generated by the image processing unit, in whichthe image processing unitdisplays, on the display unit, three-dimensional structure data changed in accordance with the weight coefficient inputted via the input unit.

(14) An image processing method executed in an image processing apparatus, in whichthe image processing apparatus includes:an image processing unit configured to input captured images of a depth camera and a visible light camera, to analyze a three-dimensional shape of a subject, andthe image processing unitgenerates a depth map based TSDF space (TSDF Volume) by using a depth map acquired from a captured image of the depth camera,generates a visible light image based TSDF space by using a captured image of the visible light camera,generates an integrated TSDF space by integration processing on the depth map based TSDF space and the visible light image based TSDF space, andexecutes three-dimensional shape analysis processing on the subject by using the integrated TSDF space.

(15) A program for causing an image processing apparatus to execute image processing, in whichthe image processing apparatus includes:an image processing unit configured to input captured images of a depth camera and a visible light camera, to analyze a three-dimensional shape of a subject, andthe program causes the image processing unit to execute:a process of generating a depth map based TSDF space (TSDF Volume) by using a depth map acquired from a captured image of the depth camera;a process of generating a visible light image based TSDF space by using a captured image of the visible light camera;a process of generating an integrated TSDF space by integration processing on the depth map based TSDF space and the visible light image based TSDF space; anda process of executing three-dimensional shape analysis processing on the subject by using the integrated TSDF space.

Note that, the series of processes described in the specification can be executed by hardware, software, or a combined configuration of both. In a case of executing processing by software, a program recording a processing sequence can be installed and executed in a memory in a computer incorporated in dedicated hardware, or a program can be installed and executed in a general-purpose computer capable of executing various processes. For example, the program can be recorded in advance on a recording medium. The program can be installed from a recording medium to a computer, or can be received via a network such as a local area network (LAN) or the Internet, and installed in a recording medium such as an incorporated hard disk.

Furthermore, the various processes described in the specification may be executed not only in a chronological order in accordance with the description, but may also be executed in parallel or individually depending on processing capability of a device that executes the processing or depending on the necessity. Furthermore, a system in this specification is a logical set configuration of a plurality of devices, and is not limited to one in which a device of each configuration is in a same casing.

INDUSTRIAL APPLICABILITY

As described above, according to a configuration of one embodiment of the present disclosure, a configuration is realized in which captured images of the depth camera and the visible light camera are inputted, and a highly accurate three-dimensional shape of a subject is analyzed and restored.

Specifically, for example, there is provided an image processing unit that inputs captured images of the depth camera and the visible light camera to analyze a three-dimensional shape of a subject. The image processing unit generates a depth map based TSDF space (TSDF Volume) by using a depth map acquired from a captured image of the depth camera, and generates a visible light image based TSDF space by using a captured image of the visible light camera. Moreover, an integrated TSDF space is generated by integration processing on the depth map based TSDF space and the visible light image based TSDF space, and three-dimensional shape analysis processing on the subject is executed using the integrated TSDF space.

This configuration realizes a configuration in which captured images of the depth camera and the visible light camera are inputted and a highly accurate three-dimensional shape of a subject is analyzed and restored.

REFERENCE SIGNS LIST