An image processing apparatus 200 includes: a first saturation information obtaining unit 306 configured to obtain saturation of a frame image; a second saturation information obtaining unit 309 configured to obtain saturation of a background image; and a threshold value determining unit 311 configured to determine a threshold value, which is used for checking a difference in color information, for each pixel, based on the saturation obtained by each of the first saturation information obtaining unit 306 and the second saturation information obtaining unit 309. A hue foreground area estimating unit 312 is configured to estimate a foreground area by use of the threshold value determined by the threshold value determining unit 311 as well as color information of the frame image and color information of the background image.

BACKGROUND

Field

The present disclosure relates to a technology for determining a foreground area from a captured image.

Description of the Related Art

Processing for determining a foreground area, such as a human figure, from a captured image is performed. Patent Laid-Open No. 2014-23057 (hereinafter referred to as Document 1) discloses a method for detecting a subject area based on a color characteristic amount such as color difference and luminance. In Document 1, there is a description about: obtaining an evaluation value, which indicates an achromatic color degree of an image; determining in accordance with the evaluation value whether to use a mask, which has been created based on each characteristic amount such as color difference and luminance; and detecting a subject area by use of a mask, which is targeted for use.

However, the technology of Document 1 is for determining whether to use a mask based on an evaluation value, which indicates an achromatic color degree of an image, and is not for adaptively changing conditions for determining whether to be a foreground in accordance with an achromatic color degree. Therefore, there is a case where a foreground area cannot be determined appropriately.

SUMMARY OF THE INVENTION

An image processing apparatus includes: a first obtaining unit configured to obtain a captured image and a background image generated based on a captured image; a second obtaining unit configured to obtain a difference in hue or a difference in chromaticity between the captured image and the background image obtained by the first obtaining unit; a third obtaining unit configured to obtain at least one of saturation of the captured image obtained by the first obtaining unit and saturation of the background image obtained by the first obtaining unit; and an estimating unit configured to estimate a foreground area by comparing a threshold value set according to the at least one of saturation obtained by the third obtaining unit with the difference in hue or the difference in chromaticity obtained by the second obtaining unit.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, a detailed explanation based on embodiments is given with reference to the accompanying drawings. The configurations shown in the following embodiments are merely examples, and the present disclosure is not limited to the illustrated configurations.

First Embodiment

In the present embodiment, an explanation is given of a mode in which a foreground area, which is used for generating a virtual viewpoint image, is determined. First, for ease of understanding of the embodiment, a brief explanation is given of an overview of a virtual viewpoint image. There is a technology of generating a virtual viewpoint image of a given virtual viewpoint by use of images of multiple viewpoints, which are captured from multiple viewpoints. For example, by use of virtual viewpoint images, highlight scenes of a soccer or basketball game can be viewed/browsed from various angles, and, therefore, it is possible to give a user a highly realistic feeling, compared to normal images.

Generation of such a virtual viewpoint image based on multi-viewpoint images is performed by collecting images captured by multiple cameras into an image processing unit of a server, or the like, and performing such a process as rendering in the image processing unit. Further, a generated virtual viewpoint image is transmitted to a user terminal so as to be browsed on the user terminal.

For generating a virtual viewpoint image, a process of modeling a foreground, which is a main subject (object), apart from a background part and then rendering the foreground is performed. Modeling of a foreground requires information of a foreground mask, which corresponds to the silhouette of the foreground seen from multiple cameras, and texture information of the foreground (such as RGB color information of each pixel of the foreground).

The process of separating a foreground from a background part is referred to as a foreground background separation process. The foreground background separation process is a process of estimating and determining a foreground area, and the process is generally performed by a background subtraction method. The background subtraction method is a method in which a difference between a background image and an input image, which includes a foreground, is obtained and the area made by a collection of pixels of which difference values have been determined to be equal to or greater than a predetermined threshold value is regarded as a foreground area.

In the estimation processing of a foreground area, it is common to obtain a difference by use of a characteristic amount of an image, such as luminance, color, or texture. Here, in the mode using a difference between characteristic amounts of color, in a case where the saturation is low, the hue accuracy decreases. This may increase erroneous determination in the estimation of a foreground area. In the embodiment explained below, an explanation is given of an example of processing of determining an appropriate foreground area, such that, in a case of estimating a foreground area by use of a difference between characteristic amounts of color, a threshold value is changed according to saturation so as to improve the accuracy of the estimation of a foreground area.

FIG. 1is a diagram for explaining a schematic configuration of a system100of the present embodiment. There are multiple cameras102arranged side by side around the stadium101, so that images of the stadium101are captured from multiple viewpoints. It is assumed that a competition such as a soccer game is held in the stadium101and that a humanfigure 103, which is to be a foreground object, is present in the stadium101. An object is a specific human figure such as a player, a manager, or a referee. The object may be an object having a predetermined image pattern, such as a ball or a goal.

Each camera102includes input/output hardware for data transmission. The cameras102are connected to each other in a ring-shaped network by use of a network cable, or the like, and are configured to sequentially transmit image data to adjacent cameras via the network. That is, a camera102is configured to transmit received image data together with image data obtained by capturing by itself to an adjacent camera. One of the cameras102is connected to an image processing apparatus200, and image data of each camera102is transmitted to the image processing apparatus200. In the image processing apparatus200, processing for generating a virtual viewpoint image is performed by use of each received image data.

<Configuration and Processing of Image Processing Apparatus>

FIG. 2is a diagram illustrating an example of a hardware configuration of the image processing apparatus200. The image processing apparatus200includes a CPU201, a RAM202, a ROM203, an HDD204, and an input/output I/F205. The components of the image processing apparatus200are mutually connected via a system bus206. Further, the image processing apparatus200is connected to a camera102and a display operation unit210via the input/output I/F205.

The CPU201executes a program stored in the ROM203, using the RAM202as a work memory and comprehensively controls each component of the image processing apparatus200via the system bus206. In this way, the various processing described below are implemented. The HDD204is a high-capacity storage device that stores various kinds of data to be handled by the image processing apparatus200, and the HDD204may be an SSD, for example. The CPU201is capable of writing data on the HDD204and reading data stored in the HDD204via the system bus206.

The input/output I/F205is, for example, a serial bus I/F such as USB or IEEE1394, and input or output of various kinds of data, commands, etc., between an external device and the image processing apparatus200is performed via the input/output I/F205. The display operation unit210is configured with, for example, a liquid crystal display having a touch-sensitive panel function. Further, the display operation unit210displays necessary information for a user or obtains an instruction from a user via a UI screen. Although the image processing apparatus200may include more components other than those described above, explanation thereof is omitted.

FIG. 3is a block diagram illustrating a functional configuration of the image processing apparatus200. The image processing apparatus200includes an image receiving unit301, a recording unit302, a background generating unit303, a first luminance information obtaining unit304, a first hue information obtaining unit305, a first saturation information obtaining unit306, a second luminance information obtaining unit307, a second hue information obtaining unit308, a second saturation information obtaining unit309, a luminance foreground area estimating unit310, a threshold value determining unit311, a hue foreground area estimating unit312, a foreground area determining unit313, and a virtual viewpoint image generating unit314. The recording unit302inFIG. 3is realized by the HDD204or the RAM202, for example. Regarding the other units ofFIG. 3, the CPU201reads out a program stored in the ROM203, or the like, to the RAM202and executes the program, so that the CPU201functions as each unit illustrated inFIG. 3. That is, the image processing apparatus200is capable of realizing each module illustrated inFIG. 3as a software module. Alternatively, the image processing apparatus200may include built-in hardware such as an ASIC (application specific integrated circuit) and an FPGA (field programmable gate array), which are not illustrated inFIG. 3. Furthermore, each unit illustrated inFIG. 3may be processed by hardware such as an ASIC and an FPGA, and each unit illustrated inFIG. 3may be mounted inside an ASIC or an FPGA as hardware. Further, a part ofFIG. 3may be realized by software, and the rest may be realized by hardware.

FIG. 4is a diagram illustrating an example of a flowchart performed in the image processing apparatus200. The series of processes illustrated in the flowchart ofFIG. 4is performed by the CPU201retrieving a program code stored in the ROM203or the HDD204into the RAM202and executing the program code. Alternatively, a part or all of the functions in the steps ofFIG. 4may be implemented by hardware such as an ASIC, an FPGA, or an electronic circuit. The symbol “S” in the explanation of each process means that it is a step in the flowchart. Hereinafter, an explanation is given of an overview of the configuration and the processing of the image processing apparatus200with reference toFIG. 2,FIG. 3, andFIG. 4.

In S401, the image processing apparatus200performs preprocessing. The preprocessing is processing performed prior to the foreground area determination processing performed in the present embodiment. For example, the image receiving unit301receives image data of each camera102via a network. Hereinafter, an image represented by image data received by the image receiving unit301is referred to as an input image. Input images are images input continuously as time passes and may include a moving image and a still image. In the present embodiment, an input image is explained as a frame image of a moving image. In S401, the image receiving unit301performs preprocessing, such as distortion correction of an image having a lens distortion and color and gamma adjustment, to a received input image. The image receiving unit301stores the preprocessed image data in the recording unit302.

In S402, the image processing apparatus200performs generation processing of a background image. For example, the image processing apparatus200obtains information of time for generating a virtual viewpoint image through the display operation unit210. The image receiving unit301sequentially reads out frame images of respective cameras corresponding to the time for generating a virtual viewpoint image from the recording unit302. Further, the image receiving unit301outputs the frame images to the background generating unit303, the first luminance information obtaining unit304, the first hue information obtaining unit305, and the first saturation information obtaining unit306, respectively.

An explanation is given of the background generating unit303. The background generating unit303performs processing of estimating a background image of each camera based on multiple frame images (including frame images received in the past) captured by respective cameras and storing a result thereof in a memory such as the HDD204. For the background generation processing based on multiple frame images, a background generation method using Gaussian Mixture Model can be used, for example. Gaussian Mixture Model is a generally well-known method, and, therefore, a detail explanation thereof is omitted.

In S402, upon receiving a frame image from the image receiving unit301, the background generating unit303reads out a background image stored in a memory for the corresponding camera, updates the background image by use of the received frame image, and stores the background image in the memory. In this way, a background image corresponding to a frame image is generated. The background generating unit303outputs the background image data of the camera corresponding to the frame image received from the image receiving unit301to the second luminance information obtaining unit307, the second hue information obtaining unit308, and the second saturation information obtaining unit309. Further, the background generating unit303outputs the background image data to the virtual viewpoint image generating unit314.

In S403, processing of obtaining luminance information is performed by the first luminance information obtaining unit304and the second luminance information obtaining unit307. In the present embodiment, a frame image is an image of which each pixel is configured with RGB pixel values, and the processing of obtaining luminance information is performed by calculating luminance information from the RGB pixel values. In a case where an image to which color conversion processing has already been performed is received by the image processing apparatus200, the luminance information of the received image may be obtained without performing the calculation process. The same applies to saturation and hue, which are described later.

First, an explanation is given of processing by the first luminance information obtaining unit304. The first luminance information obtaining unit304determines luminance information of each pixel of a frame image that is output from the image receiving unit301. In the present embodiment, processing by use of the HSV color space is performed to a frame image, which is an RGB image. The first luminance information obtaining unit304determines a V component, which is luminance information, by Formula (1).
V=max(R,G,B)  Formula(1)

Here, “max” is a function for selecting the maximum value among arguments. The second luminance information obtaining unit307determines a V component, which is luminance information of each pixel of a background image that is output from the background generating unit303, by Formula (1), as with the first luminance information obtaining unit304. The luminance information obtained by the first luminance information obtaining unit304and the second luminance information obtaining unit307are output to the luminance foreground area estimating unit310.

In S404, processing of obtaining saturation information is performed by the first saturation information obtaining unit306and the second saturation information obtaining unit309. First, an explanation is given of processing by the first saturation information obtaining unit306. The first saturation information obtaining unit306obtains saturation information of each pixel of a frame image that is output from the image receiving unit301. The first saturation information obtaining unit306calculates an S component, which is saturation information, by Formula (2) as color information of each pixel.
S=max(R,G,B)−min(R,G,B)  Formula (2)

Here, “min” is a function for selecting the minimum value among arguments. The second saturation information obtaining unit309calculates an S component, which is saturation information of each pixel of a background image that is output from the background generating unit303, by Formula (2), as with the first saturation information obtaining unit306. The saturation information obtained by the first saturation information obtaining unit306and the second saturation information obtaining unit309are output to the threshold value determining unit311.

In S405, processing of obtaining hue information is performed by the first hue information obtaining unit305and the second hue information obtaining unit308. First, an explanation is given of processing by the first hue information obtaining unit305. The first hue information obtaining unit305obtains hue information of each pixel of a frame image that is output from the image receiving unit301. The first hue information obtaining unit305calculates an H component, which is hue information, by Formulas (3) through (5) as color information of each pixel. The value calculated by Formula (2) is used for “S”.

In a case where min(R, G, B)=B,
H=(G−R)/S×60+60  Formula (3)
In a case where min(R, G, B)=R,
H=(B−G)/S×60+180  Formula (4)
In a case where min(R, G, B)=G,
H=(R−B)/S×60+300  Formula (5)

In a case where max(R, G, B)=min(R, G, B), the value of “H” is invalid. The second hue information obtaining unit308calculates an H component, which is hue information of each pixel of a background image that is output from the background generating unit303, by Formulas (3) through (5), as with the first hue information obtaining unit305. The value calculated by Formula (2) is used for “S”. Further, in a case where max(R, G, B)=min(R, G, B), the value of “H” is invalid. The hue information obtained by the first hue information obtaining unit305and the second hue information obtaining unit308are output to the hue foreground area estimating unit312.

In S406, the luminance foreground area estimating unit310estimates the foreground area of the frame image by use of the luminance information of the frame image and the luminance information of the background image obtained in S403. For example, the luminance foreground area estimating unit310checks the difference between the V component of the frame image and the V component of the background image, and, in a case where the value thereof is greater than a predetermined threshold value, the luminance foreground area estimating unit310estimates that the pixel is (corresponds to) the foreground. In the present embodiment, the threshold value used by the luminance foreground area estimating unit310is a threshold value that is the same for every pixel. In the present embodiment, a description has been given of the example in which a V component in the HSV color space is used as luminance information. However, the present embodiment is not limited to the example Regarding luminance information, processing by use of another color space may be performed. For example, processing by use of the YUV color space may be performed. Generally, in the YUV color space, a method of calculating luminance “Y” by Formula (6) is known.
Y=0.299×R+0.587×G+0.114×BFormula (6)

Regarding calculation of luminance information, various calculation methods are known other than the above calculation method, and any calculation method may be used.

FIGS. 5A through 5Care diagrams illustrating an example of a foreground area. InFIGS. 5A through 5C, the black part indicates a foreground.FIG. 5Ais a diagram illustrating an example of a processing result of the luminance foreground area estimating unit310. InFIG. 5A, there is a defect in a part of the human figure. In a case where there is a portion of which the difference in luminance between the frame image and the background image is small, such a phenomenon with a defective portion as illustrated inFIG. 5Aoccurs. The image indicating the foreground area estimated by the luminance foreground area estimating unit310is output to the foreground area determining unit313.

In S407, the threshold value determining unit311determines a threshold value for checking a difference in hue based on saturation information output from the first saturation information obtaining unit306and the second saturation information obtaining unit309. Then, the threshold value determining unit311outputs the determined threshold value to the hue foreground area estimating unit312. In the present embodiment, the threshold value determining unit311retains a table for determining the threshold value corresponding to saturation. The threshold value determining unit311determines the threshold value by referring to the table. For example, firstly, the threshold value determining unit311determines the threshold value by referring to the table, based on the smaller value of the S components of the frame image and the background image. This table is preset so that the threshold value becomes greater as the value of the S component is smaller. The lower the saturation (S component) is, the lower the accuracy of the hue (H component) becomes. Therefore, by making the threshold value for determining a foreground greater as the saturation (S component) is lower, it is made difficult to estimate a foreground area in a case where the hue accuracy decreases due to low saturation. Thereby, it is possible to prevent occurrence of erroneous determination. Contrarily, the greater the saturation (S component) is, the higher the accuracy of the hue (H component) becomes. Therefore, by making the threshold value for determining a foreground smaller as the saturation (S component) is higher, it is made easy to determine a foreground by use of a difference in hue in a case of high saturation.

In S408, the hue foreground area estimating unit312performs processing of estimating a foreground area by use of hue information output from the first hue information obtaining unit305and the second hue information obtaining unit308as well as the threshold value output from the threshold value determining unit311. The hue foreground area estimating unit312calculates a difference value between an H component of the frame image and an H component of the background image and compares the difference value with the threshold value. In a case where the difference value is greater than the threshold value, the hue foreground area estimating unit312estimates that the pixel is the foreground. Here, the hue foreground area estimating unit312may calculate a difference between an S component of the frame image and an S component of the background image and, in a case where the difference is greater than a second threshold value, the hue foreground area estimating unit312may estimate that the pixel is the foreground. That is, in a case where a difference value in hue is greater than the predetermined threshold value and a difference in saturation is greater than the second threshold value, the pixel may be determined as the foreground. Alternatively, even though a difference value in hue is greater than the predetermined threshold value, in a case where the value of the S component of the frame image and the value of the S component of the background image are both smaller than a third threshold value, processing of not estimating to be the foreground may be performed.

In a case where above processing is performed with all pixels as a processing target pixel, an image representing the foreground area estimated based on hue information is completed. This processing is executed separately for each pixel as described above. For this reason, fine pixel defects may occur inside the area that should be the foreground, or fine noise may occur in the area that should be the background. Therefore, for example, a process of removing defects or noise by performing a smoothing process to an image indicating a foreground area may be performed. The image indicating the foreground area estimated by the hue foreground area estimating unit312is output to the foreground area determining unit313.

FIG. 5Bis a diagram illustrating an example of a processing result of the hue foreground area estimating unit312. In a result of the estimation processing of a foreground area based on hue information, there is often a tendency that fine noise occurs inside the foreground area or the accuracy of the foreground boundary becomes lower.

In S409, the foreground area determining unit313combines an image indicating a foreground area, which are output from the luminance foreground area estimating unit310with an image indicating a foreground area, which are output from the hue foreground area estimating unit312, by performing logical sum (OR) operation for each pixel, so as to determine the foreground area. Not limited to OR operation for both of the images, the foreground area determining unit313may perform other logical operations and various other processes.

FIG. 5Cis a diagram illustrating an example of an operation result corresponding to the processing result of the luminance foreground area estimating unit310(c.f.,FIG. 5A) and the processing result of the hue foreground area estimating unit312(c.f.,FIG. 5B). It can be seen that the foreground area is improved as the defective portion of the human figure generated by estimation by use of luminance is complemented with estimation by use of hue. The image data indicating the foreground area determined by the foreground area determining unit313is output to the virtual viewpoint image generating unit314.

In S410, the virtual viewpoint image generating unit314generates a virtual viewpoint image according to a virtual viewpoint position, which is input by user designation or the like. As a method for generating a virtual viewpoint image, a method such as Visual Hull is known, for example. The virtual viewpoint image generating unit314firstly generates a background of the virtual viewpoint by performing re-projection onto the two dimensions by use of the background image data, which is output from the background generating unit303. Further, the virtual viewpoint image generating unit314restores the three-dimensional shape of each foreground based on image data indicating a foreground area of each camera, which is output from the foreground area determining unit313. The virtual viewpoint image generating unit314generates a virtual viewpoint image by associating an image corresponding to the foreground with a three-dimensional model and then performing re-projection onto the two dimensions. Since Visual Hull is a well-known method, a detailed explanation thereof is omitted.

As described above, in the present embodiment, processing of estimating a foreground based on color information and processing of estimating a foreground based on luminance information are performed in the HSV color space. That is, processing of estimating a foreground is performed in two stages. Then, a foreground area is determined by performing a predetermined operation to the result of each processing of estimating a foreground. In the present embodiment, in the processing of estimating a foreground based on color information, a threshold value is provided for each pixel instead of providing a threshold value for the entire image. Further, the processing of changing the threshold value provided for each pixel according to the saturation of each pixel is performed. According to such processing, the threshold value is adaptively changed according to the saturation of each pixel. Therefore, for example, even though the hue accuracy becomes lower in a case of low saturation, it is possible to prevent occurrence of erroneous determination.

Although, in the processing of estimating a foreground based on luminance information, an description is given of an example in which the same threshold value is used for each pixel without changing the threshold value for each pixel, a threshold value may be provided for each pixel in the processing of estimating a foreground based on luminance information as well.

Furthermore, in the flowchart ofFIG. 4, the example in which the processing of obtaining luminance information is performed prior to the processing of obtaining saturation information and the processing of obtaining hue information. However, the order is not limited thereto. Moreover, although the example in which the estimation processing of a foreground based on hue is performed after the estimation processing of a foreground based on luminance has been described, the processing may be performed in the reversed order.

Second Embodiment

In the first embodiment, the processing of estimating a foreground area in the HSV color space has been explained as processing of estimating a foreground based on color information. In the present embodiment, an example of estimating a foreground area in the Lab color space is explained as the processing of estimating a foreground based on color information.

Since the configuration of the image processing system in the present embodiment is the same as that in the first embodiment, explanation thereof is omitted.

FIG. 6is a block diagram illustrating a functional configuration of the image processing apparatus200of the present embodiment. The image processing apparatus200in the present embodiment includes an image receiving unit601, a recording unit602, a background generating unit603, a first luminance information obtaining unit604, a first chromaticity information obtaining unit605, a second luminance information obtaining unit607, a second chromaticity information obtaining unit608, a luminance foreground area estimating unit610, a threshold value determining unit611, a chromaticity foreground area estimating unit612, a foreground area determining unit613, and a virtual viewpoint image generating unit614.

The recording unit602inFIG. 6is realized by the HDD204or the RAM202, for example. Regarding the other units ofFIG. 6, there may be a mode in which the CPU201reads out a program stored in the ROM203, or the like, to the RAM202and executes the program, so that the CPU201functions as each unit illustrated inFIG. 6. Alternatively, the image processing apparatus200may include built-in hardware such as an ASIC or an FPGA, which is not illustrated inFIG. 6. Furthermore, each unit illustrated inFIG. 6may be processed by hardware such as an ASIC and an FPGA, and each unit illustrated inFIG. 6may be mounted inside an ASIC or an FPGA as hardware. Further, a part ofFIG. 6may be realized by software, and the rest may be realized by hardware.

FIG. 7is a diagram illustrating an example of a flowchart performed in the image processing apparatus200. In S701, the image processing apparatus200performs preprocessing. Since the preprocessing is the same as the processing explained in the first embodiment, the explanation thereof is omitted.

In S702, the image processing apparatus200performs generation processing of a background image. Since the generation processing of a background image is the same as the processing explained in the first embodiment, the explanation thereof is omitted. The image receiving unit601outputs a frame image to the first luminance information obtaining unit604, the first chromaticity information obtaining unit605, and the background generating unit603. Further, the background image data generated by the background generating unit603is output to the second luminance information obtaining unit607, the second chromaticity information obtaining unit608, and the virtual viewpoint image generating unit614.

In S703, processing of obtaining luminance information is performed by the first luminance information obtaining unit604and the second luminance information obtaining unit607. In the present embodiment, a frame image is an image of which each pixel is configured with RGB pixel values, and the processing of obtaining luminance information is performed by calculating luminance information from the RGB pixel values. In a case where an image to which color conversion processing has already been performed is received by the image processing apparatus200, the luminance information of the received image may be obtained without performing the calculation process. The same applies to chromaticity, which is described later.

First, an explanation is given of processing by the first luminance information obtaining unit604. The first luminance information obtaining unit604determines luminance information of each pixel of a frame image that is output from the image receiving unit601. In the present embodiment, processing by use of the Lab color space is performed to a frame image, which is an RGB image. Since calculation formulas for an “L component”, an “a component”, and a “b component” are generally known, a detailed explanation thereof are omitted. The first luminance information obtaining unit604calculates an L component, which is luminance information, from a frame image. Similarly, the second luminance information obtaining unit607calculates an L component, which is luminance information, from a background image.

In S704, processing of obtaining chromaticity information is performed by the first chromaticity information obtaining unit605and the second chromaticity information obtaining unit608. The first chromaticity information obtaining unit605calculates an “a component” and a “b component”, which indicate chromaticity corresponding to saturation information and hue information of each pixel of the frame image, which is output from the image receiving unit601. Similarly, the second chromaticity information obtaining unit608calculates an “a component” and a “b component” of each pixel of the background image, which is output from the background generating unit603.

In S705, the luminance foreground area estimating unit610estimates a foreground area of the frame image by use of the luminance information of the frame image and the luminance information of the background image, which are obtained in S703. For example, the luminance foreground area estimating unit610checks the difference between the L components of the frame image and the background image, and, in a case where the value thereof is greater than a predetermined threshold value, the luminance foreground area estimating unit610estimates that the pixel is the foreground.

In S706, the threshold value determining unit611determines a threshold value for checking difference in chromaticity based on chromaticity information output from the first chromaticity information obtaining unit605and the second chromaticity information obtaining unit608. Then, the threshold value determining unit611outputs the determined threshold value to the chromaticity foreground area estimating unit612. First, the threshold value determining unit611calculates saturation “S” of the frame image and the background image by Formula (7).
S=sqrt(a{circumflex over ( )}2+b{circumflex over ( )}e2)  Formula (7)

Here, “sqrt” is a function for obtaining a square root, and “{circumflex over ( )}” is a symbol representing a power. The threshold value determining unit611determines the threshold value by referring to a table, based on the smaller value of S components of the frame image and the background image. This table is preset so that the threshold value becomes greater as the value of the S component is smaller. Therefore, by making the threshold value for determining a foreground greater to attend to the chromaticity accuracy which decreases as the saturation (S component) is lower, it is possible to prevent occurrence of erroneous determination in a case where the chromaticity accuracy decreases due to low saturation. The threshold value determining unit611determines a threshold value for each pixel and outputs the determined threshold value to the chromaticity foreground area estimating unit612.

In S707, the chromaticity foreground area estimating unit612performs processing of estimating a foreground area by use of chromaticity information output from the first chromaticity information obtaining unit605and the second chromaticity information obtaining unit608as well as the threshold value output from the threshold value determining unit611. The chromaticity foreground area estimating unit612calculates a difference value between chromaticity of the frame image and chromaticity of the background image and compares the difference value with the threshold value. In a case where the difference value is greater than the threshold value, the chromaticity foreground area estimating unit612estimates that the pixel is a foreground. In a case of performing the above processing to all pixels, an image representing the foreground area estimated based on chromaticity information is completed. The image indicating the foreground area estimated by the chromaticity foreground area estimating unit612is output to the foreground area determining unit613.

In S708, the foreground area determining unit613combines an image indicating a foreground area, which are output from the luminance foreground area estimating unit610with an image indicating a foreground area, which are output from the chromaticity foreground area estimating unit612, by performing OR operation for each pixel, so as to determine the foreground area. In this way, as in the first embodiment, a foreground shape is improved by complementing a defective portion, etc., of a human figure, which is estimated as the foreground based on luminance, by use of the result of estimation of the foreground based on chromaticity. The image data indicating the foreground area determined by the foreground area determining unit613is output to the virtual viewpoint image generating unit614.

In S709, the virtual viewpoint image generating unit614generates a virtual viewpoint image. Since the generation processing of a virtual viewpoint image is the same as the processing explained in the first embodiment, the explanation thereof is omitted.

As explained above, in the present embodiment, processing of estimating a foreground based on color information and processing of estimating a foreground based on luminance information are performed in the Lab color space. Then, a foreground area is determined by performing a predetermined operation to the result of each processing of estimating a foreground. In the present embodiment, as with the first embodiment, in the processing of estimating a foreground based on color information, a threshold value is provided for each pixel instead of providing a threshold value for the entire image. Further, the processing of changing the threshold value provided for each pixel according to the saturation of each pixel is performed. According to such processing, the threshold value is adaptively changed according to the saturation of each pixel. Therefore, for example, even though the chromaticity accuracy decreases in a case of low saturation, it is possible to prevent occurrence of erroneous determination.

As a difference between characteristic amounts of color, a method using color difference ΔE in the Lab color space may be performed, for example. However, color difference ΔE is an evaluation value including luminance and color. For this reason, it is not possible to perform adaptive control to attend to the change in the accuracy of color information depending on saturation. In the present embodiment, it is possible to prevent erroneous determination by changing a threshold value for comparing a chromaticity difference in the Lab color space according to saturation.

Other Embodiments

In the above-described embodiments, an explanation has been given with an example in which the processing of estimating a foreground based on color information and the processing of estimating a foreground based on luminance information are respectively performed, and, based on the results thereof, a result of the processing of estimating a foreground is determined. However, the embodiments are not limited to this example. There may be a mode in which, as processing of estimating a foreground, the processing of estimating a foreground based on color information and the processing of estimating a foreground based on luminance information are collectively performed.

Further, in the above-described embodiments, a mode in which a threshold value is changed for each pixel according to saturation is taken as an example for the explanation. However, the embodiments are not limited to this example. Instead of changing the threshold value for each pixel, it is possible to correct hue or chromaticity, which is to be compared with the threshold value, by weighting according to saturation, so as to compare the corrected value with the threshold value for each pixel. The corrected value is a value that is temporarily used for processing of estimating a foreground. With such processing, it is possible to obtain the same effect as in the above-described embodiments as well.

Further, in the above-described embodiments, a mode in which a threshold value corresponding to the smaller value of S components of a frame image and a background image is determined by referring to a table is taken as an example for explanation. However, the embodiments are not limited thereto. For example, in a case where the difference between saturation Sf of a frame image and saturation Sb of a background image is within a predetermined range, it is possible to determine a threshold value corresponding to a given value between saturation Sf and saturation Sb, such as the average value of saturation Sf and saturation Sb. That is, even with a threshold value corresponding to a value more or less shifted from S components of a frame image and a background image, the same effect can be obtained.

Further, in the above-described embodiments, for convenience of explanation, an explanation is given of an example in which the luminance information obtaining unit, the hue information obtaining unit, the saturation information obtaining unit, and the chromaticity information obtaining unit are configured for a frame image and for a background image, respectively. However, the embodiments are not limited thereto. Each information obtaining unit may be configured as the same obtaining unit for a frame image and a background image.

Further, in the above-described embodiments, an explanation has been given of a mode in which information such as luminance, saturation, hue, and chromaticity is used as a value for a color attribute of each pixel of an image. However, the embodiments are not limited thereto. Any mode may be possible as long as a value indicating a color attribute of each pixel of an image is used. Further, it is possible to estimate a first foreground area by comparing a difference value of values of a first attribute with a threshold value that is changed according to a value of a second attribute for each pixel and estimate a second foreground area by comparing a difference value of a third attribute with a threshold value that is the same for every pixel. Then, it is possible to determine a foreground area based on the first foreground area and the second foreground area.

Further, in the above-described embodiments, the processing of determining a foreground area in order to generate a virtual viewpoint image is taken as an example for explanation. However, the embodiments are not limited thereto. For example, it is possible to apply the embodiments to processing of determining a foreground area from a captured image obtained by a single camera such as a monitoring camera.

Although examples of an embodiment have been described in detail above, it is possible that the present disclosure is provided in such a form of implementation as a system, an apparatus, a method, a program, or a recording medium (storage medium). Specifically, the embodiments may be applied to a system configured with multiple devices (for example, a host computer, an interface device, an image capturing device, a web application, etc.) or may be applied to an apparatus composed of a single device.

Further, in the above-described embodiments, an explanation is given of a mode in which one image processing apparatus200obtains image data from multiple cameras, so as to generate a background image for each camera and determine a foreground area. However, the embodiments are not limited to thereto. For example, there may be a mode in which hardware of each camera or an image processing apparatus associated with each camera has functions except for the function of the virtual viewpoint image generating unit. Further, there may be a mode in which images representing a background image and a foreground area may be generated on each camera side, and each of the generated data may be transmitted to a device that generates a virtual viewpoint image.

Further, in the first embodiment, an explanation is given of the processing of estimating a foreground area using hue, and, in the second embodiment, an explanation is given of the processing of estimating a foreground area using chromaticity. It is possible to adopt an embodiment in which the above processing are combined. That is, both of the processing of estimating a foreground area using hue and the processing of estimating a foreground area using chromaticity may be performed. Then, a predetermined operation may be performed to each of the estimated foreground areas, so as to determine a foreground area estimated by use of the color information. That is, it is possible to adopt a mode in which at least one of the processing of estimating a foreground area using hue and the processing of estimating a foreground area using chromaticity.

According to the present disclosure, it is possible to appropriately determine a foreground area from a captured image.

This application claims the benefit of Japanese Patent Application No. 2018-209731, filed Nov. 7, 2018, which is hereby incorporated by reference wherein in its entirety.