Patent Description:
There is a technology in which a three-dimensional model (3D model) that is a model having three-dimensional information of a subject is generated from a moving image captured from multiple viewpoints, and a free viewpoint moving image that is a moving image according to an arbitrary viewpoint position is generated on the basis of the 3D model. Such a technology is also called a volumetric capture technology or the like.

For example, there has been proposed a technology of generating a 3D model using a method such as Visual Hull in which a three-dimensional shape of a subject is cut out on the basis of a plurality of captured images obtained by imaging from different directions (see, for example, <CIT>).

In imaging of a subject in such a volumetric capture technology, light is generally emitted to the subject and the periphery of the subject using an illumination device in order to secure luminance. <CIT> discloses an imaging system comprising a plurality of light emitting devices each with a polariser and a plurality of imaging devices each with a polariser which are configured to generate a three-dimensional model of an object. <CIT> discloses an imaging system for capturing high resolution geometry and reflectance data of a portion of a human subject. A plurality of imaging devices are arranged to view an object and generate images by illuminating the object with polarised light and capturing the images through a polarised filter.

However, in a case of the volumetric capture technology, in order to capture an image of the subject from more various directions, imaging is performed by a plurality of imaging devices arranged so as to surround the subject, and thus there are few blind spots and the illumination device easily fits within the angle of view. When the high-luminance illumination device is included within the angle of view, optical phenomena such as what is called flare, ghost, halation, and the like are likely to occur. When such an optical phenomenon occurs, accuracy of the 3D model generated from the captured image may be reduced.

The present disclosure has been made in view of such a situation, and an object thereof is to generate a more accurate 3D model.

Various aspects and features of the present invention are defined in the appended claims.

An imaging processing system according to one aspect of the present technology is an imaging processing system that generates a three-dimensional (3D) model of an object by using a plurality of captured images obtained by imaging the object, the imaging processing system including a plurality of polarization illumination devices that includes a polarizer and irradiates the object with polarized light obtained by transmitting light emitted from a light emitting unit through the polarizer from positions different from each other, and a plurality of polarization imaging devices that includes a polarizer and generates the captured images by using polarized light obtained by transmitting light from outside through the polarizer at positions different from each other where the object and at least one of the polarization illumination devices are within an angle of view, in which a polarization direction of the polarizer of the polarization imaging device is different from a polarization direction of the polarizer of the polarization illumination device.

A three-dimensional (3D) model generation method according to another aspect of the present technology is a 3D model generation method including generating a captured image of an object by using polarized light in a polarization direction different from a polarization direction of polarized light emitted from a polarization illumination device within an angle of view at positions different from each other, and generating a 3D model of the object by using a plurality of the captured images obtained at the positions different from each other.

An imaging processing system according to still another aspect of the present technology is an imaging processing system including a plurality of polarization illumination devices that includes a polarizer and irradiates an object with polarized light obtained by transmitting light emitted from a light emitting unit through the polarizer from positions different from each other, and a plurality of polarization imaging devices that includes a polarizer and generates a captured image of the object by using polarized light obtained by transmitting light from outside through the polarizer at positions different from each other where the object and at least one of the polarization illumination devices are within an angle of view, in which a polarization direction of the polarizer of the polarization imaging device is different from a polarization direction of the polarizer of the polarization illumination device.

In the imaging processing system that generates a 3D model of an object by using a plurality of captured images obtained by imaging the object according to the one aspect of the present technology, the object is irradiated with polarized light obtained by transmitting light emitted from a light emitting unit through a polarizer from positions different from each other by a plurality of polarization illumination devices that includes the polarizer, and polarized light obtained by transmitting light from the outside through a polarizer is used to generate the captured images by a plurality of polarization imaging devices that includes the polarizer having a polarization direction different from that of the polarizer of the polarization illumination device at positions different from each other where the object and at least one of the polarization illumination devices are within an angle of view.

In the 3D model generation method according to the another aspect of the present technology, a captured image of an object is generated by using polarized light in a polarization direction different from a polarization direction of polarized light emitted from a polarization illumination device within an angle of view at positions different from each other, and a 3D model of the object is generated by using a plurality of the captured images obtained at the positions different from each other.

The imaging processing system according to the still another aspect of the present technology includes a plurality of polarization illumination devices that includes a polarizer and irradiates an object with polarized light obtained by transmitting light emitted from a light emitting unit through the polarizer from positions different from each other, and a plurality of polarization imaging devices that includes a polarizer whose polarization direction is different from that of the polarizer of the polarized illumination device and generates a captured image of the object by using polarized light obtained by transmitting light from the outside through the polarizer at positions different from each other where the object and at least one of the polarization illumination devices are within an angle of view.

Hereinafter, modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described. Note that the description will be made in the following order.

There is a volumetric capture technology in which a three-dimensional model (3D model) that is a model having three-dimensional information of a subject is generated from a moving image captured from multiple viewpoints, and a free viewpoint moving image that is a moving image according to an arbitrary viewpoint position is generated on the basis of the 3D model. An information processing system <NUM> in <FIG> is a system that images a subject in multiple viewpoints by such a volumetric capture technology, generates a 3D model of the subject from the captured image, and generates a free viewpoint image according to an arbitrary viewpoint position on the basis of the 3D model.

As illustrated in <FIG>, the information processing system <NUM> includes a data acquisition unit <NUM>, a 3D model generation unit <NUM>, a formatting unit <NUM>, a transmission unit <NUM>, a reception unit <NUM>, a rendering unit <NUM>, and a display unit <NUM>.

The data acquisition unit <NUM> acquires image data for generating a 3D model of the subject. For example, the data acquisition unit <NUM> acquires, as image data, a plurality of viewpoint images captured by a plurality of imaging devices arranged to surround the subject. In this case, preferably, the plurality of viewpoint images is images obtained by the plurality of imaging devices performing imaging in synchronization.

Note that the data acquisition unit <NUM> may perform calibration on the basis of the image data and acquire internal parameters and external parameters of each imaging device. Furthermore, the data acquisition unit <NUM> may acquire, for example, a plurality of pieces of depth information indicating distances from viewpoints at multiple positions to the subject.

The data acquisition unit <NUM> supplies the acquired image data to the 3D model generation unit <NUM>.

The 3D model generation unit <NUM> generates a 3D model, which is a model having three-dimensional information of the subject, on the basis of the image data supplied from the data acquisition unit <NUM>. The 3D model generation unit <NUM> generates the 3D model of the subject by, for example, scraping the three-dimensional shape of the subject using images from a plurality of viewpoints (for example, silhouette images from the plurality of viewpoints) using what is called a visual hull.

Here, the silhouette image is an image representing only an outline (outer shape) of the subject, and a region inside the outline is represented by being filled with a single color like a shadow picture, for example. That is, the 3D model generation unit <NUM> generates such a silhouette image from the image data (captured image) supplied from the data acquisition unit <NUM>. Note that the image data of the silhouette image may be supplied from the data acquisition unit <NUM> to the 3D model generation unit <NUM>.

The 3D model generation unit <NUM> can further deform the 3D model generated using the visual hull with high accuracy using the plurality of pieces of the depth information indicating distances from viewpoints at multiple positions to the subject.

The 3D model generated by the 3D model generation unit can also be referred to as a moving image of the 3D model by generating the 3D model in time series frame units. Furthermore, since the 3D model is generated using images captured by the imaging devices of the data acquisition unit <NUM>, the 3D model can also be said as a 3D model of a real picture. The 3D model can represent shape information representing a surface shape of the subject in the form of, for example, mesh data representing the 3D model by connection of vertices, which is called a polygon mesh. The method of representing the 3D model is not limited thereto, and the 3D model may be described by what is called a point cloud representation method that represents the 3D model by position information of points.

Data of color information is also generated as a texture in association with the 3D shape data. For example, there are a case of a view independent texture in which colors are constant when viewed from any direction and a case of a view dependent texture in which colors change depending on a viewing direction.

The 3D model generation unit <NUM> supplies data of the generated 3D model to the formatting unit <NUM>.

The formatting unit <NUM> converts the 3D model data supplied from the 3D model generation unit <NUM> into a format suitable for transmission and accumulation. For example, the formatting unit <NUM> may convert the 3D model generated by the 3D model generation unit <NUM> into a plurality of two-dimensional images by performing perspective projection from a plurality of directions. Moreover, the formatting unit <NUM> may generate depth information, which is a two-dimensional depth image from a plurality of viewpoints, using the 3D model. In this case, the formatting unit <NUM> may encode (compress) the depth information and the color information in a state of this two-dimensional image. In this case, the formatting unit <NUM> may encode the depth information and the color information side by side as one image or as two separate images. Furthermore, since the depth information and the color information are in the form of two-dimensional image data, the formatting unit <NUM> may encode (compress) the depth information and the color information using a two-dimensional compression technique such as advanced video coding (AVC).

In the above case, the formatting unit <NUM> supplies the 3D model data to the transmission unit <NUM> as transmission data including 2D data (or coded data thereof).

Furthermore, for example, the formatting unit <NUM> may convert the 3D data of the mesh data into a point cloud format and supply the data to the transmission unit <NUM> as transmission data including the 3D data. In this case, the formatting unit <NUM> may encode (compress) the 3D data using, for example, a three-dimensional compression technology of Geometry-based Approach discussed in MPEG.

The transmission unit <NUM> transmits the transmission data formed by the formatting unit <NUM> to the reception unit <NUM>. The transmission unit <NUM> performs a series of processing of the data acquisition unit <NUM>, the 3D model generation unit <NUM>, and the formatting unit <NUM> offline, and then transmits the transmission data to the reception unit <NUM>. Furthermore, the transmission unit <NUM> may transmit the transmission data generated from the series of processes described above to the reception unit <NUM> in real time.

The reception unit <NUM> receives the transmission data transmitted from the transmission unit <NUM> and supplies the transmission data to the rendering unit <NUM>.

The rendering unit <NUM> performs rendering using the transmission data received by the reception unit <NUM>. For example, the rendering unit <NUM> projects a mesh of a 3D model from a viewpoint of a camera that draws the mesh of the 3D model, and performs texture mapping to paste a texture representing a color or a pattern. The drawing at this time can be arbitrarily set and viewed from a free viewpoint regardless of the camera position at the time of imaging.

For example, the rendering unit <NUM> performs texture mapping to paste a texture representing the color, pattern, or texture of the mesh according to the position of the mesh of the 3D model. The texture mapping includes what is called a view dependent method in which the viewing viewpoint of the user is considered and a view independent method in which the viewing viewpoint of the user is not considered. Since the view dependent method changes the texture to be pasted on the 3D model according to the position of the viewing viewpoint, there is an advantage that rendering of higher quality can be achieved than by the View Independent method. On the other hand, the view independent method does not consider the position of the viewing viewpoint, and thus there is an advantage that the processing amount is reduced as compared with the view dependent method. Note that data of the viewing viewpoint is input from the display unit <NUM> to the rendering unit <NUM> after the display unit <NUM> detects a viewing point (region of interest) of the user. Furthermore, the rendering unit <NUM> may employ, for example, billboard rendering for rendering an object so that the object maintains a vertical posture with respect to the viewing viewpoint. For example, when rendering a plurality of objects, the rendering unit <NUM> can render objects of low interest to the viewer by billboard and render other objects by another rendering method.

The rendering unit <NUM> supplies data of a rendering result to the display unit <NUM>.

The display unit <NUM> displays a result of rendering by the rendering unit <NUM> on the display unit of a display device. The display device may be, for example, a 2D monitor or a 3D monitor, such as a head mounted display, a spatial display, a mobile phone, a television, or a personal computer (PC).

An example of a flow of system processing executed by the information processing system <NUM> will be described with reference to a flowchart of <FIG>.

When the processing is started, in step S101, the data acquisition unit <NUM> acquires image data for generating the 3D model of the subject.

In step S102, the 3D model generation unit <NUM> generates a 3D model that is a model having three-dimensional information of the subject on the basis of the image data acquired in step S101.

In step S103, the formatting unit <NUM> encodes the shape and texture data of the 3D model generated in step S102 into a format suitable for transmission and accumulation.

In step S104, the transmission unit <NUM> transmits encoded data generated in step S103.

In step S105, the reception unit <NUM> receives the data transmitted in step S104.

In step S106, the rendering unit <NUM> performs a decoding process and converts the data into data of a shape and a texture necessary for display. Furthermore, the rendering unit <NUM> performs rendering using the data of the shape and texture.

In step S107, the display unit <NUM> displays the rendering result.

When the processing of step S107 ends, the system processing ends.

By performing each processing as described above, the information processing system <NUM> can generate a 3D model of the subject and generate and display an image of the subject viewed from a free viewpoint. Thus, the user who is the viewer can view the subject from the free viewpoint.

In the above, the information processing system <NUM> has described a series of flow from the data acquisition unit <NUM> that acquires the captured image, which is a material for generating content, to the display unit <NUM> that displays the image viewed by the user. However, it is not meant that all functional blocks are required for implementation of the present invention, the present invention may be implemented for each functional block or a combination of a plurality of functional blocks. For example, in <FIG>, the transmission unit <NUM> and the reception unit <NUM> are provided in order to illustrate a series of flow from a side of creating content to a side of viewing the content through distribution of content data, but the process from creation to viewing of the content can also be performed by the same information processing device (for example, a personal computer). In that case, the formatting unit <NUM>, the transmission unit <NUM>, and the reception unit <NUM> can be omitted.

Furthermore, in a case where the information processing system <NUM> is implemented, the same implementer can implement all of the functional blocks, or different implementers can implement the functional blocks. For example, a business operator A may implement the data acquisition unit <NUM>, the 3D model generation unit <NUM>, and the formatting unit <NUM> to generate 3D content, a business operator B may implement the transmission unit <NUM> (platform) to distribute the 3D content, and a business operator C may implement the reception unit <NUM>, the rendering unit <NUM>, and the display unit <NUM> to receive, render, control display, and the like of the 3D content.

Furthermore, each functional block can be implemented on a cloud. For example, the rendering unit <NUM> may be implemented in the display device or may be implemented in a server. In this case, information is exchanged between the display device and the server.

In <FIG>, the data acquisition unit <NUM>, the 3D model generation unit <NUM>, the formatting unit <NUM>, the transmission unit <NUM>, the reception unit <NUM>, the rendering unit <NUM>, and the display unit <NUM> are collectively described as the information processing system <NUM>. However, the configuration of the information processing system <NUM> is not limited to this example, and is only required to include at least the data acquisition unit <NUM>. For example, in the configuration illustrated in <FIG>, any one or more of the 3D model generation unit <NUM> to the display unit <NUM> may be omitted. Furthermore, the information processing system <NUM> may have a configuration (functional block) other than the above-described configuration.

Furthermore, each functional block (the data acquisition unit <NUM> to the display unit <NUM>) described above is achieved by an arbitrary configuration. For example, each functional block may be achieved by one or more devices (apparatuses). Furthermore, a plurality of functional blocks may be achieved by one device (apparatus).

<FIG> is a block diagram illustrating a main configuration example of the data acquisition unit <NUM> in <FIG>. The data acquisition unit <NUM>, which is an embodiment of the imaging processing system to which the present technology is applied, includes an imaging illumination unit <NUM> and a transmission unit <NUM> as illustrated in <FIG>.

The imaging illumination unit <NUM> images a subject and illuminates the subject. The imaging illumination unit <NUM> includes imaging units <NUM>-<NUM> to <NUM>-M and illumination units <NUM>-<NUM> to <NUM>-N (M and N are integers of <NUM> or more). The imaging unit <NUM>-<NUM> to the imaging unit <NUM>-M will be referred to as imaging units <NUM> in a case where it is not necessary to distinguish the imaging units from each other for explanation. The illumination units <NUM>-<NUM> to <NUM>-N will be referred to as illumination units <NUM> in a case where it is not necessary to distinguish the illumination units from each other for explanation.

That is, the imaging illumination unit <NUM> includes a plurality of imaging units <NUM> and a plurality of illumination units <NUM>. Note that the number of the imaging units <NUM> and the number of the illumination units <NUM> included in the imaging illumination unit <NUM> may be the same (that is, M = N) as each other or may be different from each other.

The imaging unit <NUM> includes one or a plurality of image pickup devices, and images a subject to generate a captured image for 3D model generation. That is, the imaging unit <NUM> generates a captured image used to extract the silhouette and texture of the subject. The imaging unit <NUM> supplies data of the generated captured image to the transmission unit <NUM>.

A wavelength band of light received by the image pickup device of the imaging unit <NUM> is arbitrary, and may be either visible light or invisible light. For example, the imaging unit <NUM> may receive visible rays (RGB rays) and generate a captured image of visible light, or may receive infrared rays (infrared (IR) rays) and generate a captured image of infrared light.

The illumination unit <NUM> includes one or a plurality of illumination devices, and illuminates a subject imaged by the imaging unit <NUM>. A wavelength band of light emitted by the illumination device of the illumination unit <NUM> is arbitrary, and may be either visible light or invisible light. For example, the illumination unit <NUM> may illuminate the subject with visible rays (RGB rays) or may illuminate the subject with infrared rays (IR rays).

The transmission unit <NUM> transmits data of the captured image supplied from the imaging unit <NUM> to the 3D model generation unit <NUM>. At that time, the transmission unit <NUM> may supply the data of the captured image to the 3D model generation unit <NUM> without encoding the data, or may encode the data of the captured image and supply coded data to the 3D model generation unit <NUM>. Furthermore, the transmission unit <NUM> may perform arbitrary image processing on the captured image. For example, the transmission unit <NUM> may extract a silhouette or texture from the captured image and supply data of the extracted silhouette or texture to the 3D model generation unit <NUM>.

Generally, when the luminance is too low (too dark), it is difficult to image the subject. Thus, by the illumination unit <NUM> illuminating the subject, the imaging unit <NUM> can capture an image of the subject with sufficient brightness, and can obtain a captured image in which luminance is sufficiently secured.

However, in a case of the data acquisition unit <NUM> that acquires the captured image for 3D model generation, the plurality of imaging units <NUM> is arranged around the subject so as to reduce blind spots. Therefore, there is a very high possibility that the illumination unit <NUM> is included within the angle of view of the imaging unit <NUM>. In other words, it is difficult to arrange the illumination unit <NUM> so as not to be included within the angle of view of the imaging unit <NUM>.

Since (the illumination device of) the illumination unit <NUM> is a light source with high luminance, when the illumination unit <NUM> is included within the angle of view, a phenomenon such as what is called flare, ghost, halation, and the like, in which the light leaks to a dark portion, is likely to occur. When such a phenomenon occurs, it may be difficult to extract an accurate silhouette of the subject from the captured image. Furthermore, it may also be difficult to extract the texture of the subject. Thus, accuracy of the 3D model generated from the captured image may be reduced.

Accordingly, a polarizer that generates straight polarized light from natural light (non-polarized light) or circularly polarized light is provided in the imaging unit <NUM> (image pickup device) and the illumination unit <NUM> (illumination device), the illumination unit <NUM> illuminates the subject with polarized light, and the imaging unit <NUM> receives polarized light to generate a captured image. Then, the polarization direction (that is, the polarization direction of the polarizer included in the illumination unit <NUM>) of the polarized light emitted by the illumination unit <NUM> and the polarization direction (that is, the polarization direction of the polarizer included in the imaging unit <NUM>) of the polarized light received by the imaging unit <NUM> are made different from each other.

Note that, in the present description, a beam mainly including a vibration component in a predetermined direction will be referred to as polarized light, and a main vibration direction of the polarized light will be referred to as a polarization direction (or a polarization angle). Furthermore, the polarizer generates polarized light in a predetermined polarization direction, and the polarization direction will also be referred to as a polarization direction (or polarization angle) of the polarizer.

For example, an imaging processing system that generates a 3D model of an object by using a plurality of captured images obtained by imaging the object includes a plurality of polarization illumination devices (for example, the illumination unit <NUM>) that includes a polarizer and irradiates the object with polarized light obtained by transmitting light emitted from a light emitting unit through the polarizer from positions different from each other, and a plurality of polarization imaging devices (for example, the imaging unit <NUM>) that includes a polarizer and generates the captured images by using polarized light obtained by transmitting light from the outside through the polarizer at positions different from each other where the object and at least one of the polarization illumination devices are within an angle of view, in which a polarization direction of the polarizer of the polarization imaging device is different from a polarization direction of the polarizer of the polarization illumination device.

For example, a captured image of an object is generated by using polarized light in a polarization direction different from that of polarized light emitted from a polarization illumination device (for example, the illumination unit <NUM>) within an angle of view at positions different from each other, and a 3D model of the object is generated by using a plurality of captured images obtained at the positions different from each other.

For example, an imaging processing system includes a plurality of polarization illumination devices (for example, the illumination unit <NUM>) that includes a polarizer and irradiates an object with polarized light obtained by transmitting light emitted from a light emitting unit through the polarizer from positions different from each other, and a plurality of polarization imaging devices (for example, the imaging unit <NUM>) that includes a polarizer and generates captured images of the object by using polarized light obtained by transmitting light from the outside through the polarizer at positions different from each other where the object and at least one of the polarization illumination devices are within an angle of view, in which a polarization direction of the polarizer of the polarization imaging device is different from a polarization direction of the polarizer of the polarization illumination device.

When the polarization direction of the polarizer of the imaging unit <NUM> and the polarization direction of the polarizer of the illumination unit <NUM> are different from each other, the amount of direct light from the illumination unit <NUM> that passes through the polarizer of the imaging unit <NUM> and enters the sensor is reduced. Therefore, in the captured image generated by the imaging unit <NUM>, a luminance value of a portion of the illumination unit <NUM> that is included within the angle of view can be reduced, and thus the occurrence of what is called flare, ghost, halation, and the like can be suppressed. Accordingly, the silhouette and the texture can be more accurately extracted from the captured image, and thus the 3D model generation unit <NUM> can generate a more accurate 3D model (reduction in accuracy of the 3D model can be suppressed).

Note that the degree of reduction in the amount of direct light from the illumination unit <NUM> that is transmitted through the polarizer of the imaging unit <NUM> and enters the sensor in this manner depends on the relationship (angle) between the polarization direction of the polarizer of the imaging unit <NUM> and the polarization direction of the polarizer of the illumination unit <NUM>. In general, the closer the angle between them is to <NUM> degrees, the more the amount of light decreases. That is, as the angle between the polarization direction of the polarizer of the imaging unit <NUM> and the polarization direction of the polarizer of the illumination unit <NUM> approaches <NUM> degrees, the occurrence of what is called flare, ghost, halation, and the like can be more strongly suppressed.

<FIG> is a block diagram illustrating a main configuration example of the illumination unit <NUM>. As illustrated in <FIG>, the illumination unit <NUM> includes a polarizing filter <NUM> and a light emitting unit <NUM>.

The polarizing filter <NUM> is an example of a polarizer, and generates polarized light by transmitting light of a component vibrating in a predetermined direction. The light emitting unit <NUM> is a light source and emits a beam (non-polarized light) having a predetermined wavelength in a predetermined direction.

As illustrated in <FIG>, the polarizing filter <NUM> is arranged in front of the light emitting unit <NUM> in a beam emission direction (irradiation direction). Non-polarized light <NUM> emitted from the light emitting unit <NUM> is directed to the polarizing filter <NUM>. The polarizing filter <NUM> transmits a vibration component of the non-polarized light <NUM> in a predetermined direction. That is, polarized light <NUM> having the predetermined direction as the polarization direction is generated by the polarizing filter <NUM>. This polarized light <NUM> is emitted from the illumination unit <NUM>. That is, the illumination unit <NUM> is a polarization illumination device that includes a polarizer and emits polarized light generated by the polarizer using the light from the light source.

The illumination unit <NUM> is installed at a position and in a posture to illuminate an object to be a subject of the imaging unit <NUM>, and thus at least a part of the polarized light <NUM> is emitted to the object. Then, at least the part of the emitted polarized light <NUM> is reflected by the object or the like to become non-polarized light, and travels toward the imaging unit <NUM>. That is, by the illumination unit <NUM> performing illumination in this manner, luminance of the captured image can be increased.

Note that a wavelength band of the polarized light <NUM> emitted by the illumination unit <NUM> is arbitrary. For example, the polarized light <NUM> may be visible light, invisible light, or both. For example, the polarized light <NUM> may be infrared rays (IR rays). Furthermore, the illumination unit <NUM> may include a plurality of light emitting units <NUM> (light sources) that emits beams in different wavelength regions from each other, and the imaging illumination unit <NUM> may include a plurality of illumination units <NUM> that emits polarized light <NUM> in different wavelength regions from each other.

Furthermore, the polarization direction (that is, the polarization direction of the polarized light <NUM>) of the polarizing filter <NUM> may be determined in advance (may be fixed) or variable. For example, a polarization direction control mechanism (movable ring or the like) that controls the polarization direction of the polarizing filter <NUM> may be provided, and the polarization direction of the polarizing filter <NUM> may be variable by the polarization direction control mechanism.

<FIG> is a block diagram illustrating a main configuration example of the imaging unit <NUM>. For example, as illustrated in A of <FIG>, the imaging unit <NUM> includes a polarizing filter <NUM> and an image sensor <NUM>.

The polarizing filter <NUM> is an example of a polarizer, and generates polarized light by transmitting light of a component vibrating in a predetermined direction. The image sensor <NUM> includes a plurality of pixels, photoelectrically converts incident light in each pixel, and generates a captured image. The image sensor <NUM> supplies data of the generated captured image to the transmission unit <NUM>.

As illustrated in A of <FIG>, the polarizing filter <NUM> is arranged on a beam incident side of the image sensor <NUM>. Non-polarized light <NUM> incident on the imaging unit <NUM> is directed to the polarizing filter <NUM>. The polarizing filter <NUM> transmits a vibration component of the non-polarized light <NUM> in a predetermined direction. That is, polarized light <NUM> having the predetermined direction as the polarization direction is generated by the polarizing filter <NUM>. The polarized light <NUM> enters the image sensor <NUM> and is photoelectrically converted. That is, the image sensor <NUM> generates a captured image corresponding to the polarized light <NUM>. That is, the imaging unit <NUM> is a polarization imaging device that includes a polarizer and generates a captured image using polarized light generated by the polarizer.

Note that, in a case where the illumination unit <NUM> is located within the angle of view of the imaging unit <NUM>, there is a case where direct light from the illumination unit <NUM> enters the imaging unit <NUM>. That is, the polarized light <NUM> emitted from the illumination unit <NUM> may be directed to the polarizing filter <NUM>. Here, the polarization direction of the polarizing filter <NUM> is set to a direction different from the polarization direction of the polarizing filter <NUM>. That is, the polarizing filter <NUM> and the polarizing filter <NUM> have different polarization directions from each other. Thus, at least a part of the polarized light <NUM> is blocked by the polarizing filter <NUM>. That is, the amount of polarized light <NUM> incident on the image sensor <NUM> is reduced.

That is, in the captured image generated by the imaging unit <NUM>, the luminance value of the portion of the illumination unit <NUM> that is included within the angle of view can be reduced, and thus the occurrence of what is called flare, ghost, halation, and the like can be suppressed. Accordingly, the silhouette and the texture can be more accurately extracted from the captured image, and thus the 3D model generation unit <NUM> can generate a more accurate 3D model (reduction in the accuracy of the 3D model can be suppressed).

Note that a wavelength band of light received and photoelectrically converted by the image sensor <NUM> (that is, a wavelength band of the polarized light <NUM>) is arbitrary. For example, the image sensor <NUM> may photoelectrically convert visible light, may photoelectrically convert invisible light, or may photoelectrically convert both. That is, the imaging unit <NUM> may generate data of a captured image of visible light, may generate data of a captured image of invisible light, or may generate both the captured images. For example, the image sensor <NUM> may photoelectrically convert infrared rays (IR rays). That is, the imaging unit <NUM> may generate a captured image of infrared rays (IR rays). Furthermore, the imaging unit <NUM> may include a plurality of image sensors <NUM> that photoelectrically converts beams in different wavelength regions from each other, and the imaging illumination unit <NUM> may include a plurality of imaging units <NUM> that generates captured images of beams in different wavelength regions from each other.

The captured image generated by the imaging unit <NUM> may be used to extract a silhouette of an object that is a subject. In other words, the imaging unit <NUM> may generate a captured image for extracting the silhouette of the object. By providing a polarizer (for example, the polarizing filter <NUM>) in the imaging unit <NUM> that generates such a captured image, a silhouette can be more accurately extracted from the captured image.

Furthermore, the captured image generated by the imaging unit <NUM> may be used to extract a texture of an object that is a subject. In other words, the imaging unit <NUM> may generate a captured image for extracting the texture of the object. By providing a polarizer (for example, the polarizing filter <NUM>) in the imaging unit <NUM> that generates such a captured image, the texture can be more accurately extracted from the captured image.

Of course, the captured image generated by the imaging unit <NUM> may be used to extract both the silhouette and the texture of the object that is the subject. In other words, the imaging unit <NUM> may generate a captured image for extracting the silhouette and texture of the object. Furthermore, the imaging unit <NUM> may generate each of the captured image for extracting the silhouette of the object and the captured image for extracting the texture of the object.

Furthermore, the imaging illumination unit <NUM> may include the imaging unit <NUM> that generates a captured image used to extract a silhouette of an object, and the imaging unit <NUM> that generates a captured image used to extract a texture of an object. In that case, the polarizer (for example, the polarizing filter <NUM>) may be provided in the imaging unit <NUM> that generates a captured image used to extract a silhouette of an object, may be provided in the imaging unit <NUM> that generates a captured image used to extract a texture of an object, or may be provided in both the imaging units <NUM>.

Furthermore, the polarization direction (that is, a vibration direction of the polarized light <NUM>) of the polarizing filter <NUM> may be determined in advance (may be fixed) or variable. For example, a polarization direction control mechanism (movable ring or the like) that controls the polarization direction of the polarizing filter <NUM> may be provided, and the polarization direction of the polarizing filter <NUM> may be variable by the polarization direction control mechanism.

Furthermore, as illustrated in B of <FIG>, the imaging unit <NUM> may include a polarization sensor <NUM>. The polarization sensor <NUM> is an image sensor that photoelectrically converts polarized light to generate a captured image. The polarization sensor <NUM> includes a plurality of pixels, each pixel is provided with a polarizer that generates polarized light from incident light, and a light receiving unit provided in each pixel receives the polarized light generated by the polarizer and performs photoelectric conversion. That is, the polarization sensor <NUM> polarizes and photoelectrically converts the incident non-polarized light <NUM> to generate a captured image thereof. Note that the polarization direction of the polarizer provided in each pixel of the polarization sensor <NUM> is designed to be different from the polarization direction of the polarizing filter <NUM>. That is, the polarizer provided in each pixel of the polarization sensor <NUM> and the polarizing filter <NUM> have different polarization directions. Thus, since at least a part of the polarized light <NUM> is blocked by the polarizer, the light amount (luminance in the captured image) of the polarized light <NUM> to be photoelectrically converted is reduced.

Therefore, as in a case of the polarizing filter <NUM>, in the captured image generated by the imaging unit <NUM> (polarization sensor <NUM>), the luminance value of the portion of the illumination unit <NUM> that is included within the angle of view can be reduced, and the occurrence of what is called flare, ghost, halation, and the like can be suppressed. Accordingly, the silhouette and the texture can be more accurately extracted from the captured image, and thus the 3D model generation unit <NUM> can generate a more accurate 3D model (reduction in the accuracy of the 3D model can be suppressed).

Note that the imaging unit <NUM> and the illumination unit <NUM> may be configured as a tim-of-flight (ToF) sensor. That is, the imaging unit <NUM> and the illumination unit <NUM> may be configured as a distance measurement sensor in which the illumination unit <NUM> illuminates the subject, the imaging unit <NUM> receives the reflected light, and the distance to the subject is measured on the basis of a light reception timing thereof. In other words, the present technology can also be applied to an optical distance measurement sensor such as a ToF sensor.

The imaging unit <NUM> and the illumination unit <NUM> may be arranged at positions close to each other. Moreover, the imaging unit <NUM> and the illumination unit <NUM> may be arranged so that a light irradiation direction by the illumination unit <NUM> and an imaging direction (for example, a direction of the center of an angle of view) of the imaging unit <NUM> are the same as each other. In other words, each of the illumination units <NUM> may be located close to any one of the imaging units <NUM> and may take a posture in which the irradiation direction of polarized light is the same as the imaging direction of the imaging unit <NUM> in the vicinity thereof. With this configuration, the illumination unit <NUM> can perform illumination from the front of the object that is the subject as viewed from the imaging unit <NUM>. Therefore, the imaging unit <NUM> can generate a captured image in which there are few unnecessary shadows and shades on the subject and the subject has sufficient luminance.

For example, an imaging and illumination unit may be formed by the imaging unit <NUM> and the illumination unit <NUM> arranged close to each other. <FIG> is a diagram illustrating an example of the imaging and illumination unit.

In the example of <FIG>, an imaging and illumination unit <NUM> includes an RGB camera <NUM>, an IR camera <NUM>, and an IR light <NUM>.

The RGB camera <NUM> is an imaging unit <NUM> that receives visible rays and generates a captured image in a wavelength region of visible light. The IR camera <NUM> is an imaging unit <NUM> that receives infrared rays and generates a captured image in a wavelength region of infrared light. The IR light <NUM> is an illumination unit <NUM> that emits infrared rays.

For example, it is highly possible that the light source of visible light changes drastically outdoors or at a live concert venue. For example, it is conceivable that a subject is irradiated with a spotlight or a laser beam at a live concert venue. In a case where imaging is performed by the imaging processing system as described above under such an environment, a captured image in the wavelength region of visible light is likely to be affected by such illumination, and optical phenomena such as what is called flare, ghost, halation, and the like are likely to occur. Thus, it may be difficult to accurately extract the silhouette of the subject using such a captured image.

Accordingly, the imaging and illumination unit <NUM> generates a captured image in the wavelength region of infrared light as a captured image for extracting a silhouette of the subject using the IR camera <NUM>. That is, the silhouette of the subject is extracted using the captured image in the wavelength region of infrared light. Then, for imaging by the IR camera <NUM> (to ensure sufficient luminance in the wavelength region of infrared light), the IR light <NUM> illuminates the subject using infrared rays.

Note that since the IR camera <NUM> and the IR light <NUM> are installed at positions close to each other and facing the same subject as each other, the IR light <NUM> can be illuminated from the front of the subject as viewed from the IR camera <NUM>. Therefore, the IR camera <NUM> can generate a captured image in which there are few unnecessary shadows and shades on the subject and the subject has sufficient luminance. That is, the IR camera <NUM> can generate a captured image from which a more accurate silhouette can be extracted. In other words, by using the captured image generated by the IR camera <NUM>, the silhouette of the subject can be more accurately extracted.

Furthermore, the IR camera <NUM> has a polarizer as in the example of <FIG>. Similarly, the IR light <NUM> has a polarizer as in the example of <FIG>. Then, the polarization direction of the polarizer of the IR camera <NUM> and the polarization direction of the polarizer of the IR light <NUM> are different from each other. Therefore, as described above, in the captured image generated by the IR camera <NUM>, it is possible to suppress the occurrence of what is called flare, ghost, halation, and the like due to infrared light emitted by the IR light <NUM> that is included within the angle of view. Accordingly, the silhouette can be more accurately extracted from the captured image, and thus the 3D model generation unit <NUM> can generate a more accurate 3D model (reduction in the accuracy of the 3D model can be suppressed).

Since the RGB camera <NUM> can generate a captured image in the wavelength region of visible light, it is possible to generate a captured image used to extract the texture of the subject. The RGB camera <NUM> and the IR camera <NUM> are installed at positions close to each other toward the same subject as each other. That is, the angles of view of the RGB camera <NUM> and the IR camera <NUM> are the same or approximate. Therefore, the texture corresponding to the silhouette of the subject extracted using the captured image generated by the IR camera <NUM> can be extracted using the captured image generated by the RGB camera <NUM>.

An arrangement example of the imaging unit <NUM> and the illumination unit <NUM> will be described in units of the imaging and illumination unit <NUM>. As illustrated in <FIG>, the plurality of imaging and illumination units <NUM> (that is, the imaging unit <NUM> and the illumination unit <NUM>) may be installed around (so as to surround) an object <NUM> that is the subject. For example, each imaging and illumination unit <NUM> may be arranged so that an object to be a subject is positioned in a region (plane or space) having a line connecting adjacent imaging and illumination units <NUM> with each other as an outer frame.

In that case, the object <NUM> and at least one of the illumination units <NUM> may be arranged to be included within the angle of view of each imaging unit <NUM>. Moreover, another imaging unit <NUM> may be arranged to be included within the angle of view.

For example, as illustrated in <FIG>, the two imaging and illumination units <NUM> (imaging and illumination unit <NUM>-<NUM> and imaging and illumination unit <NUM>-<NUM>) may be arranged to face each other. In the case of the example of <FIG>, the imaging and illumination unit <NUM>-<NUM> and the imaging and illumination unit <NUM>-<NUM> are installed on opposite sides of the object <NUM> on a straight line <NUM> passing through the object <NUM> toward the object <NUM>. That is, the imaging direction and the illumination direction of the imaging and illumination unit <NUM>-<NUM> and the imaging and illumination unit <NUM>-<NUM> are opposite to each other.

By installing the two imaging and illumination units <NUM> (imaging and illumination unit <NUM>-<NUM> and imaging and illumination unit <NUM>-<NUM>) in this manner, a wider range of the object <NUM> can be imaged (the blind spot can be reduced).

In a case of such an arrangement, although the IR light <NUM> is included within the angle of view of the IR camera <NUM>, as described above, the incidence of direct light from the IR light <NUM> can be suppressed by using the polarizer, and thus it is possible to suppress the occurrence of what is called flare, ghost, halation, and the like due to infrared light emitted by the IR light <NUM>. Therefore, the silhouette of the object <NUM> can be more accurately extracted using the captured image generated by the IR camera <NUM>.

Note that the number of imaging and illumination units <NUM> to be installed is arbitrary as long as it is plural. For example, eight imaging and illumination units <NUM> may be installed. In a case where a large number of imaging and illumination units <NUM> are installed as described above, a plurality of other imaging units <NUM> or illumination units <NUM> may be included within the angle of view of the imaging unit <NUM>.

That is, the imaging and illumination unit <NUM> (the imaging unit <NUM> and the illumination unit <NUM>) may be installed so that the plurality of illumination units <NUM> is included within the angle of view of the imaging unit <NUM>. In that case, the polarization directions of the polarizers of the plurality of illumination units <NUM> may be the same as each other. With this configuration, it is possible to similarly suppress the incidence of direct light from each of the illumination units <NUM> to the imaging unit <NUM> included within the angle of view. That is, it is possible to further suppress the occurrence of what is called flare, ghost, halation, and the like.

Furthermore, the imaging and illumination unit <NUM> may be installed so that another imaging unit <NUM> is included within the angle of view of the imaging unit <NUM>. Moreover, the imaging and illumination unit <NUM> may be installed so that a plurality of other imaging units <NUM> is included within the angle of view of the imaging unit <NUM>.

Furthermore, the plurality of polarization illumination devices (for example, the illumination unit <NUM>) may include a first polarization illumination device and a second polarization illumination device, the plurality of polarization imaging devices (for example, the imaging unit <NUM>) may include a first polarization imaging device in which the object and the first polarization illumination device are at positions within the angle of view and a second polarization imaging device in which the object and the second polarization illumination device are at positions within the angle of view, the polarization direction of the polarizer of the first polarization imaging device may be different from the polarization direction of the polarizer of the first polarization illumination device, and the polarization direction of the polarizer of the second polarization imaging device may be different from the polarization direction of the polarizer of the second polarization illumination device. That is, there may be a plurality of imaging units <NUM> in which a single illumination unit <NUM> is included within the angle of view.

In such a case, the polarization directions of the polarizers of the plurality of illumination units <NUM> that is included within the angles of view of the imaging units <NUM> different from each other may be the same or may not be the same as each other. That is, the polarization direction of the polarizer of the second polarization imaging device may be different from the polarization direction of the polarizer of the second polarization illumination device.

That is, the polarization directions of the polarizers of the plurality of illumination units <NUM> may not be the same as each other. Similarly, the polarization directions of the polarizers of the plurality of imaging units <NUM> may not be the same as each other. For example, if the polarization direction of the polarizer of one imaging unit <NUM> among the plurality of imaging units <NUM> is different from the polarization direction of the polarizer of the illumination unit <NUM> that is included within the angle of view of the imaging unit <NUM>, the effect of the present embodiment can be obtained.

For example, as illustrated in A of <FIG>, the plurality of imaging and illumination units <NUM> (the imaging units <NUM> and the illumination units <NUM>) may be arranged in a circular shape centered on the object <NUM>. In the example of A of <FIG>, eight imaging and illumination units <NUM> are arranged on a circle <NUM> centered on the object <NUM>. As illustrated in B of <FIG>, each of the imaging and illumination units <NUM> (the imaging and illumination unit <NUM>-<NUM> to the imaging and illumination unit <NUM>-<NUM>) is installed toward the object <NUM>. More specifically, the imaging and illumination unit <NUM>-<NUM> and the imaging and illumination unit <NUM>-<NUM> are arranged to face each other on a straight line <NUM> passing through the object <NUM>. The imaging and illumination unit <NUM>-<NUM> and the imaging and illumination unit <NUM>-<NUM> are arranged to face each other on a straight line <NUM> passing through the object <NUM>. The imaging and illumination unit <NUM>-<NUM> and the imaging and illumination unit <NUM>-<NUM> are arranged to face each other on a straight line <NUM> passing through the object <NUM>. The imaging and illumination unit <NUM>-<NUM> and the imaging and illumination unit <NUM>-<NUM> are arranged to face each other on a straight line <NUM> passing through the object <NUM>.

Even in such a case, the occurrence of what is called flare, ghost, halation, and the like can be suppressed by the polarizers as described above by applying the present technology.

For example, as illustrated in <FIG>, the plurality of imaging and illumination units <NUM> (the imaging units <NUM> and the illumination units <NUM>) may be arranged in a cylindrical shape with a vertical line <NUM> passing through the object <NUM> as a central axis. Even in such a case, the occurrence of what is called flare, ghost, halation, and the like can be suppressed by the polarizers as described above by applying the present technology.

For example, as illustrated in <FIG>, the plurality of imaging and illumination units <NUM> (the imaging units <NUM> and the illumination units <NUM>) may be arranged in a spherical (or hemispherical) shape centered on the object <NUM>. Even in such a case, the occurrence of what is called flare, ghost, halation, and the like can be suppressed by the polarizers as described above by applying the present technology.

For example, as illustrated in A of <FIG>, in a captured image <NUM> generated by the RGB camera <NUM>, an object <NUM> as a subject and an IR light <NUM> of another imaging and illumination unit <NUM> appear together (are included within the angle of view). In this case, since the captured image <NUM> is a captured image in the wavelength region of visible light, flare as indicated by an ellipse <NUM> or an ellipse <NUM> due to direct light (infrared light) from the IR light <NUM> does not occur.

On the other hand, the IR camera <NUM> generates a captured image <NUM> of the wavelength region of infrared light as illustrated in B of <FIG>. The angles of view of the RGB camera <NUM> and the IR camera <NUM> are substantially the same. Thus, the IR light <NUM> of another imaging and illumination unit <NUM> also appears in the captured image <NUM> (is included within the angle of view). Thus, in a case where the present technology is not applied, in the captured image <NUM>, flare (ellipse <NUM> or ellipse <NUM>) occurs due to direct light (infrared light) from the IR light <NUM>. Thus, it is difficult to accurately extract the silhouette of the object <NUM>.

By applying the present technology, the IR camera <NUM> can suppress the direct light from the IR light <NUM> by the polarizer in a polarization direction different from the polarization direction of the polarizer of the IR light <NUM>, and can generate the captured image <NUM> as illustrated in C of <FIG>. That is, it is possible to suppress the occurrence of what is called flare, ghost, halation, and the like. Accordingly, the silhouette can be more accurately extracted from the captured image <NUM>, and thus the 3D model generation unit <NUM> can generate a more accurate 3D model (reduction in the accuracy of the 3D model can be suppressed).

The polarization directions of the polarizers of the imaging unit <NUM> and the illumination unit <NUM> may be calibrated (adjusted). As described above, the amount of light suppressed by the polarizer of the imaging unit <NUM> varies depending on the relative angle between the polarization directions of the polarizers of the imaging unit <NUM> and the illumination unit <NUM>. That is, the degree of suppression of the occurrence of what is called flare, ghost, halation, and the like changes. Therefore, for example, the polarization direction of each polarizer may be calibrated so that the relative angle becomes an appropriate angle (occurrence of what is called flare, ghost, halation, and the like can be further suppressed) according to the position or attitude in which the imaging unit <NUM> or the illumination unit <NUM> is installed.

<FIG> is a block diagram illustrating a main configuration example of the data acquisition unit <NUM> in that case. As illustrated in <FIG>, the data acquisition unit <NUM> includes a calibration processing unit <NUM> and a display unit <NUM> in addition to the configuration of <FIG>.

The calibration processing unit <NUM> is an example of a calibration device that calibrates the polarization direction of the polarizing filter, acquires a captured image generated by the imaging unit <NUM>, and derives a more suitable polarization direction (polarization direction capable of further suppressing occurrence of what is called flare, ghost, halation, and the like) on the basis of the captured image. Furthermore, the calibration processing unit <NUM> generates a display image indicating the derived polarization direction and supplies the display image to the display unit <NUM>.

The display unit <NUM> displays the display image supplied from the calibration processing unit <NUM>. The user refers to the display image displayed on the display unit <NUM> to grasp the polarization direction in which the occurrence of what is called flare, ghost, halation, and the like can be further suppressed. The polarization directions of the polarizers of the imaging unit <NUM> and the illumination unit <NUM> are variable, and the imaging unit <NUM> and the illumination unit <NUM> have a polarization direction control mechanism (movable ring or the like) that controls the polarization direction of the polarizer. The user operates the polarization direction control mechanism to calibrate the polarization direction to a desired direction.

An example of flow of the calibration processing executed by such a calibration processing unit <NUM> will be described with reference to a flowchart of <FIG>.

When the calibration processing is started, the calibration processing unit <NUM> acquires a captured image in step S201.

In step S202, the user sets the polarization direction (polarization angle) of the polarizers of the imaging unit <NUM> and the illumination unit <NUM> to a predetermined direction (angle) different from the previous direction.

In step S203, the calibration processing unit <NUM> calculates a luminance value of the acquired captured image.

In step S204, the calibration processing unit <NUM> determines whether or not the polarization direction (polarization angle) has been set to all possible directions (angles). That is, the calibration processing unit <NUM> acquires the captured image in all the possible polarization directions (polarization angles) and determines whether or not the luminance value has been calculated.

In a case where it is determined that there is an unprocessed direction (angle), the processing returns to step S202. That is, imaging is performed in a new polarization direction (polarization angle), and a luminance value of the captured image is calculated. As described above, in a case where it is determined that the processing has been performed for all the possible directions (angles), the processing proceeds to step S205.

In step S205, the calibration processing unit <NUM> determines a polarization direction (polarization angle) in which the luminance value is minimized from among the polarization directions (polarization angles) in which the luminance value of the captured image is calculated, and generates a display image indicating the polarization direction (polarization angle). The display unit <NUM> displays the display image.

Thus, the user can calibrate the polarization directions of the polarizers of the imaging unit <NUM> and the illumination unit <NUM> to more appropriate directions on the basis of the display. Therefore, it is possible to further suppress the occurrence of what is called flare, ghost, halation, and the like.

Note that a polarization direction control unit (actuator) that updates the polarization directions of the polarizers of the imaging unit <NUM> and the illumination unit <NUM> to the polarization directions derived by the calibration processing unit <NUM> may be provided.

Note that a device that detects the inclination of the camera by camera calibration may be further included. The position of the calibrated camera is expressed in rotation and translation with respect to a certain origin. Furthermore, a device that changes the angle of the polarizing filter in a case where the rotation of the camera is detected may be provided.

Furthermore, in a case where a polarization sensor is used as the imaging unit <NUM> as in the example of B of <FIG>, a sensor that images a plurality of deflection directions (for example, four directions of <NUM>°, <NUM>°, <NUM>°, and <NUM>°, and the like) may be used. For example, rotation information of an automatically controlled light source may be acquired, and the polarization sensor may select a pixel in an optimum polarization direction on the basis of the opening information and generate a captured image corresponding to polarized light in the polarization direction.

Furthermore, a device that controls a deflection angle of the light source according to the camera position may be provided. For example, in a case where an imaging unit such as a drone or a crane camera whose imaging position can be changed is introduced into the surrounding Volumetric imaging environment, there is a possibility that the light enters the opposite position due to the movement. Accordingly, the polarization direction (polarization angle) on the light side may be controlled using the position and rotation information of the camera.

The technology according to the present disclosure can be applied to various products and services.

For example, new video content may be produced by combining the 3D model of the subject generated in the present embodiment with 3D data managed by another server. Furthermore, for example, in a case where there is background data acquired by an imaging device such as Lidar, content as if the subject is at a place indicated by the background data can be produced by combining the 3D model of the subject generated in the present embodiment and the background data. Note that the video content may be three-dimensional video content or two-dimensional video content converted into two dimensions. Note that examples of the 3D model of the subject generated in the present embodiment include a 3D model generated by the 3D model generation unit and a 3D model reconstructed by the rendering unit, and the like.

For example, the subject (for example, a performer) generated in the present embodiment can be arranged in a virtual space that is a place where the user communicates as an avatar. In this case, the user has an avatar and can view a subject of a live image in the virtual space.

For example, by transmitting the 3D model of the subject generated by the 3D model generation unit <NUM> from the transmission unit <NUM> to a remote location, a user at the remote location can view the 3D model of the subject through a reproduction device at the remote location. For example, by transmitting the 3D model of the subject in real time, the subject and the user at the remote location can communicate with each other in real time. For example, a case where the subject is a teacher and the user is a student, or a case where the subject is a physician and the user is a patient can be assumed.

For example, a free viewpoint video of a sport or the like can be generated on the basis of the 3D models of the plurality of subjects generated in the present embodiment, or an individual can distribute himself/herself, which is a 3D model generated in the present embodiment, to a distribution platform. As described above, the contents in the embodiments described in the present description can be applied to various technologies and services.

The series of processes described above can be executed by hardware or can be executed by software. In a case where the series of processes is executed by software, a program constituting the software is installed in a computer. Here, the computer includes a computer incorporated in dedicated hardware, a general-purpose personal computer for example that can execute various functions by installing various programs, and the like.

<FIG> is a block diagram illustrating a configuration example of hardware of a computer that executes the above-described series of processes by a program.

In a computer <NUM> illustrated in <FIG>, a central processing unit (CPU) <NUM>, a read only memory (ROM) <NUM>, and a random access memory (RAM) <NUM> are interconnected via a bus <NUM>.

An input-output interface <NUM> is also connected to the bus <NUM>. An input unit <NUM>, an output unit <NUM>, a storage unit <NUM>, a communication unit <NUM>, and a drive <NUM> are connected to the input-output interface <NUM>.

The input unit <NUM> includes, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, and the like. The output unit <NUM> includes, for example, a display, a speaker, an output terminal, and the like. The storage unit <NUM> includes, for example, a hard disk, a RAM disk, a nonvolatile memory, and the like. The communication unit <NUM> includes, for example, a network interface. The drive <NUM> drives a removable medium <NUM> such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

In the computer configured as described above, the CPU <NUM> loads, for example, a program stored in the storage unit <NUM> into the RAM <NUM> via the input-output interface <NUM> and the bus <NUM> and executes the program, so as to perform the above-described series of processes. The RAM <NUM> also appropriately stores data and the like necessary for the CPU <NUM> to execute various processes.

The program executed by the computer can be applied by being recorded in the removable medium <NUM> as a package medium or the like, for example. In this case, the program can be installed in the storage unit <NUM> via the input-output interface <NUM> by attaching the removable medium <NUM> to the drive <NUM>.

Furthermore, this program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting. In this case, the program can be received by the communication unit <NUM> and installed in the storage unit <NUM>.

In addition, this program can be installed in the ROM <NUM> or the storage unit <NUM> in advance.

Furthermore, although the information processing system and the like have been described above as application examples of the present technology, the present technology can be applied to any configuration.

For example, the present technology can be applied to various electronic devices such as a transmitter and a receiver (for example, a television receiver and a mobile phone) in satellite broadcasting, cable broadcasting such as cable TV, distribution on the Internet, and distribution to a terminal by cellular communication, or the like, or a device (for example, a hard disk recorder and a camera) that records an image on a medium such as an optical disk, a magnetic disk, and a flash memory, or reproduces an image from the storage medium.

Furthermore, for example, the present technology can also be implemented as a configuration of a part of the device, such as a processor (for example, a video processor) as a system large scale integration (LSI) or the like, a module (for example, a video module) using a plurality of processors or the like, a unit (for example, a video unit) using a plurality of modules or the like, or a set (for example, a video set) obtained by further adding other functions to a unit.

Furthermore, for example, the present technology can also be applied to a network system including a plurality of devices. For example, the present technology may be implemented as cloud computing shared and processed in cooperation by a plurality of devices via a network. For example, the present technology may be implemented in a cloud service that provides a service related to an image (moving image) to any terminal such as a computer, an audio visual (AV) device, a portable information processing terminal, or an Internet of Things (IoT) device.

Note that in the present description, the system means a set of a plurality of components (devices, modules (parts), and the like), and it does not matter whether or not all the components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network, and one device in which a plurality of modules is housed in one housing are all systems.

Note that the system, device, processing unit, and the like to which the present technology is applied can be used in any fields, for example, traffic, medical care, crime prevention, agriculture, livestock industry, mining, beauty, factory, household appliance, weather, nature monitoring, and the like. Furthermore, its use is arbitrary.

The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible.

For example, a configuration described as one device (or processing unit) may be divided and configured as a plurality of devices (or processing units). Conversely, configurations described above as a plurality of devices (or processing units) may be combined and configured as one device (or processing unit). Furthermore, a configuration other than those described above may of course be added to the configuration of each device (or each processing unit). Moreover, if the configuration and operation of the entire system are substantially the same, a part of the configuration of a certain device (or processing unit) may be included in the configuration of another device (or another processing unit).

Furthermore, for example, the above-described program may be executed in any device. In that case, it is sufficient if the device has necessary functions (functional blocks and the like) and can acquire necessary information.

Furthermore, for example, each step of one flowchart may be executed by one device, or may be shared and executed by a plurality of devices. Moreover, in a case where a plurality of processes is included in one step, the plurality of processes may be executed by one device, or may be shared and executed by a plurality of devices. In other words, a plurality of processes included in one step can be executed as processes of a plurality of steps. Conversely, a process described as a plurality of steps can be collectively executed as one step.

Furthermore, for example, in the program executed by the computer, processes in steps for describing the program may be executed in time series in the order described in the present description, or may be executed in parallel or individually at necessary timing such as when a call is made. That is, as long as no contradiction occurs, the processes in the respective steps may be executed in an order different from the above-described orders. Moreover, the processes in steps for describing this program may be executed in parallel with processes in another program, or may be executed in combination with processes in another program.

Claim 1:
An imaging processing system (<NUM>) that generates a three-dimensional (3D) model of an object by using a plurality of captured images obtained by imaging the object, the imaging processing system comprising:
a plurality of polarization illumination devices (<NUM>) each including a polarizer and each irradiating the object with polarized light obtained by transmitting light emitted from a light emitting unit through the polarizer from positions different from each other; and
a plurality of polarization imaging devices (<NUM>) each including a polarizer and each generating the captured images by using polarized light obtained by transmitting light from outside through the polarizer at positions different from each other where the object and at least one of the polarization illumination devices (<NUM>) are within an angle of view, wherein
a first of the plurality of the polarization imaging devices (<NUM>) is at a position where the object and a first of the plurality of polarization illumination devices (<NUM>) are within an angle of view of the first of the plurality of the polarization imaging devices, and a second of the polarization imaging devices (<NUM>) is at a position where the object and a second of the plurality of polarization illumination devices (<NUM>) are within an angle of view of the second of the plurality of the polarization devices,
a polarization direction of a polarizer of the first polarization imaging device (<NUM>) is different from a polarization direction of a polarizer of the first polarization illumination device (<NUM>),
a polarization direction of a polarizer of the second polarization imaging device (<NUM>) is different from a polarization direction of a polarizer of the second polarization illumination device (<NUM>), and
the polarization direction of the polarizer of the first polarization illumination device (<NUM>) is different from the polarization direction of the polarizer of the second polarization illumination device (<NUM>).