Patent Description:
In recent years, 3D contents are actively made use of mainly by the movie world. The development of the multi-viewpoint image capturing technique and the multi-viewpoint display technique is in progress for seeking an enhanced sense of realism.

For multi-viewpoint image capturing, an image capturing apparatus, such as a camera array, a plenoptic camera, and a camera array system, has been developed. With a multi-viewpoint image capturing apparatus, such as a camera array and a plenoptic camera, it is possible to acquire information called a light field representing the position of a light ray and angle information. By using the light field, it is possible to adjust the focus position after image capturing, to change the viewpoint position after image capturing, and to acquire the distance to a subject. The technique such as this is being actively studied in the field called computational photography.

Image data or additional data (e.g., distance data) obtained by performing image capturing using a camera array or a plenoptic camera is encoded and compressed to an appropriate amount of information. Further, the encoded image data or additional data is saved in accordance with a predetermined file format (hereinafter, simply referred to as format).

As a format to record a plurality of images, for example, there is Multi-Picture Format. The Multi-Picture Format is a format to record a plurality of still images in the same file and was established by the CIPA in <NUM>. The Multi-Picture Format is also made use of as an image capturing format of a 3D digital camera (stereo camera). In the case where the Multi-Picture Format is made use of, it is possible to store a plurality of pieces of image data in one file. In the Multi-Picture Format, each piece of image data is encoded by the JPEG. The kinds of image compatible with the Multi-Picture Format include a panorama image, a stereoscopic image, a multiangle image, etc..

Besides the above, a format to record an extended image file to store a plurality of pieces of image data obtained by performing image capturing from different viewpoints and a basic file to store image data obtained by processing the representative image data selected from among the plurality of pieces of image data in association with one another has been proposed (see <CIT>).

The article entitled "<NPL>, presents a method for synthesizing a novel view from two sets of differently focused images taken by an aperture camera array for a scene consisting of two approximately constant depths, wherein this method can effectively create a dense array of pin-hole cameras (i.e. all-in-focus images). Further background art is known from the document <CIT> which discloses an information processing apparatus for acquiring distance information from image data on subject, wherein when a plurality of images are obtained by using a plenoptic camera to calculate distance information, a method for calculating the distance information is selected; and the document <CIT> which discloses a camera for obtaining a plurality of images from multi-viewpoints to generate an image that is focused on an object of interest.

In order to make it possible to make use of a plurality of pieces of image data obtained by performing image capturing using a camera array or a plenoptic camera for more purposes of use, it is important to record the image data in an appropriate format. Due to this, it is made possible to make use of the image data for various purposes of use, such as a change of viewpoint position, refocus, and adjustment of depth of field. Consequently, an object of the present invention is to make it possible to make use of a plurality of pieces of image data of different focus positions or different viewpoints obtained by performing image capturing for more purposes of use.

The present invention in its first aspect provides a data recording apparatus as specified in claims <NUM> to <NUM>.

The present invention in its second aspect provides an image capturing apparatus as specified in claim <NUM>.

The present invention in its third aspect provides a data recording method as specified in claim <NUM>.

The present invention in its fourth aspect provides a program as specified in claim <NUM>.

According to the present invention, it is possible to make use of a plurality of pieces of image data of different focus positions or different viewpoints obtained by performing image capturing for more purposes of use.

In the following, with reference to the drawings, aspects for embodying the present invention are explained.

<FIG> are diagrams each showing an outline of an image capturing apparatus according to a first embodiment. <FIG> each show an example of an outline of a camera array image capturing apparatus including a plurality of image capturing units in the case where it is viewed from the front, and an example of an outline of a plenoptic image capturing apparatus in the case where it is viewed from the front.

First, the camera array image capturing apparatus including a plurality of image capturing units shown in <FIG> is explained.

As shown in <FIG>, an image capturing apparatus <NUM> includes a photographing button <NUM> on the top of a casing (upper side in <FIG>). Further, the image capturing apparatus <NUM> includes four image capturing units <NUM> to <NUM> configured to acquire image data on the front of the casing (front side in <FIG>). The four image capturing units all have the same focus length and are arranged in the form of a square grid. By a user pressing down the photographing button <NUM>, image capturing processing is started.

First, a first focus position is set and the image capturing units <NUM> to <NUM> receive light information on a subject by a sensor (image capturing element). The received signal is subjected to A/D conversion and a plurality of pieces of image data is acquired at the same time. By the camera array such as this, it is possible to obtain an image data group (multi-viewpoint image data) obtained by performing image capturing of the same subject from a plurality of viewpoint positions.

<FIG> are each a diagram showing an example of multi-viewpoint image data. In <FIG>, an image of a top-left viewpoint (viewpoint <NUM>) captured by the image capturing unit <NUM> is shown. In <FIG>, an image of a top-right viewpoint (viewpoint <NUM>) captured by the image capturing unit <NUM> is shown. In <FIG>, an image of a bottom-left viewpoint (viewpoint <NUM>) captured by the image capturing unit <NUM> is shown. In <FIG>, an image of a bottom-right viewpoint (viewpoint <NUM>) captured by the image capturing unit <NUM> is shown. From a comparison between the <FIG>, it is known that the positions of the objects in each image are different depending on the arrangement of the image capturing unit. Here, it is assumed that an object <NUM> is in focus.

Next, a second focus position different from the first focus position is set and multi-viewpoint image data is acquired again. At this time also, the image capturing units <NUM> to <NUM> similarly receive light information on a subject by a sensor. The received signal is subjected to A/D conversion and a plurality of pieces of image data is acquired at the same time. Here, it is assumed that an object <NUM> is in focus.

As described above, multi-viewpoint image data of different focus positions is acquired by one-time image capturing instructions.

By using <FIG>, image data obtained by performing image capturing at a plurality of focus positions and from a plurality of viewpoints is explained. <FIG> is a diagram schematically showing a relationship between image data obtained by performing image capturing at a plurality of focus positions and from a plurality of viewpoints. The horizontal axis represents the viewpoint. Consequently, the example shown in <FIG> shows that the viewpoints of respective pieces of captured image data (captured image data <NUM>, <NUM>, <NUM>, <NUM>) of multi-viewpoint image data <NUM> are different from one another. Further, the example shows that the viewpoints of respective pieces of captured image data (captured image data <NUM>, <NUM>, <NUM>, <NUM>) of multi-viewpoint image data <NUM> are different from one another. The example also shows that the captured image data <NUM> and <NUM> are image data obtained by performing image capturing from the same viewpoint. The example also shows that the captured image data <NUM> and <NUM> are image data obtained by performing image capturing from the same viewpoint. The example also shows that the captured image data <NUM> and <NUM> are image data obtained by performing image capturing from the same viewpoint. The example also shows that the captured image data <NUM> and <NUM> are image data obtained by performing image capturing from the same viewpoint. Further, the example shows that the focus positions of the multi-viewpoint image data <NUM> and <NUM> shown one on top of the other in the vertical direction (in the vertical direction in <FIG>) are different from each other. By causing <FIG> to correspond to the examples shown in <FIG>, the multi-viewpoint image <NUM> corresponds to the multi-viewpoint image data in which the object <NUM> is in focus (captured image data of viewpoints <NUM>, <NUM>, <NUM>, <NUM>). Further, the multi-viewpoint image <NUM> corresponds to the multi-viewpoint image data in which the object <NUM> is in focus (captured image data of viewpoints <NUM>, <NUM>, <NUM>, <NUM>).

Here, the number of image capturing units is four, but the number of image capturing units is not limited to four. It is possible to apply the present embodiment as long as an image capturing apparatus includes a plurality of image capturing units. Further, the example in which the four image capturing units are arranged in the form of a square grid is explained here, but the arrangement of the image capturing units is arbitrary. For example, each image capturing unit may be arranged in the form of a straight line or may be arranged completely randomly. In the following, there is a case where the captured image data <NUM> to <NUM> are referred to simply as image data <NUM> to <NUM>.

Next, the plenoptic image capturing apparatus shown in <FIG> is explained.

As shown in <FIG>, the image capturing apparatus <NUM> includes the photographing button <NUM> on the top of the casing (upper side in <FIG>). Further, the image capturing apparatus <NUM> includes an image capturing unit <NUM> configured to acquire image data on the front of the casing (front side in <FIG>). By a user pressing down the photographing button <NUM>, image capturing processing is started.

First, the first focus position is set and the image capturing unit <NUM> receives light information on a subject by a sensor.

<FIG> shows an internal configuration of the plenoptic image capturing unit <NUM>. In the plenoptic camera, between a main lens <NUM> and a sensor plane <NUM>, a microlens array <NUM> is arranged. The light emitted from an object <NUM> arranged on a focus plane <NUM> of the main lens is collected by the main lens <NUM> and separated in the microlens array <NUM>, and received by the sensor plane <NUM>. By the signal received by the sensor plane <NUM> being subjected to A/D conversion, plenoptic image data is acquired. At the bottom-right in <FIG>, an enlarged view of a sensor <NUM>, which is part of the sensor plane <NUM>, is shown. The sensor <NUM> is a sensor of <NUM> × <NUM> (vertical × horizontal) pixels and receives light in the form of a circle.

In <FIG>, an example of image data acquired by the plenoptic image capturing unit <NUM> is shown. In <FIG>, the area divided into the shape of a grid corresponds to a pixel. For example, there are pixels <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>. The pixels <NUM>, <NUM>, <NUM>, and <NUM> are each a pixel obtained by angle-resolving the light received in the form of a circle by the sensor <NUM>. A circle <NUM> shown in <FIG> represents the light received in the form of a circle by the sensor <NUM>. In the example shown in <FIG>, the light received in the form of a circle is separated into <NUM> × <NUM> (vertical × horizontal) pixels.

A method of generating multi-viewpoint image data from plenoptic image data is explained. By selecting and putting side by side the top-left pixels (pixels shown with slashes in <FIG>) for each circle shown in <FIG> in the order of the pixels <NUM>, <NUM>, <NUM>, and <NUM>, image data of the top-left viewpoint as shown in <FIG> is generated. By performing the same processing for the top-right pixels, the bottom-left pixels, and the bottom-right pixels, image data of the top-right viewpoint, that of the bottom-left viewpoint, and that of the bottom-right viewpoint are generated. In this manner, the multi-viewpoint image data as shown in <FIG> is generated from plenoptic image data. Here, it is assumed that the object <NUM> is in focus. The demosaicking processing of an image is not the main point of the present embodiment, and therefore, explanation thereof is omitted.

Next, the second focus position different from the first focus position is set and multi-viewpoint image data is acquired. Similarly, at this time, the light emitted from an object arranged on the focus plane (another focus plane different from the focus plane <NUM>) of the main lens is collected by the main lens <NUM> and the light is separated in the microlens array <NUM>, and received by the sensor plane <NUM>. By the signal received by the sensor plane <NUM> being subjected to A/D conversion, plenoptic image data is acquired. After this, multi-viewpoint image data is generated from the plenoptic image data. Here, it is assumed that the object <NUM> is in focus. As described above, multi-viewpoint image data of different focus positions is acquired by one-time image capturing instructions.

Here, the number of times of division of the sensor <NUM> is set to four, i.e., <NUM> × <NUM> pixels, but the sensor <NUM> is not limited to <NUM> × <NUM> pixels. That is, it is possible to apply the present embodiment as long as light is divided on the sensor plane via a microlens.

As described above, it is possible for the image capturing apparatus <NUM> of the present embodiment to acquire multi-viewpoint image data of different focus positions as shown in <FIG> by both the camera array and the plenoptic camera by one-time image capturing instructions. In the above-described explanation, the image capturing apparatus that acquires two pieces of multi-viewpoint image data by one-time image capturing instructions is shown, but it is also possible to apply the present embodiment to an image capturing apparatus that acquires three or more pieces of multi-viewpoint image data by one-time image capturing instructions. In the following, how to handle image data of different viewpoint positions and different focus positions is explained, but the method of handling image data is the same for both the camera array and the plenoptic camera and the present embodiment can be applied to both the image capturing apparatuses.

<FIG> is a block diagram showing an example of an internal configuration of the image capturing apparatus <NUM>.

The image capturing apparatus <NUM> includes a light field image capturing unit <NUM>, a distance data acquisition unit <NUM>, a bus <NUM>, a central processing unit (CPU) <NUM>, a RAM <NUM>, a ROM <NUM>, an operation unit <NUM>, a display control unit <NUM>, a display unit <NUM>, a light field image capturing control unit <NUM>, a distance data acquisition control unit <NUM>, an external memory control unit <NUM>, an encoding unit <NUM>, a free focus point image generation unit <NUM>, a free viewpoint image generation unit <NUM>, and an additional information generation unit <NUM>.

The light field image capturing unit <NUM> obtains a plurality of pieces of multi-viewpoint image data whose focus positions are different from one another by image capturing. In the case where the image capturing apparatus <NUM> is a camera array, the light field image capturing unit <NUM> corresponds to the image capturing units <NUM> to <NUM> shown in <FIG>. In the case where the image capturing apparatus <NUM> is a plenoptic camera, the light field image capturing unit <NUM> corresponds to the image capturing unit <NUM> shown in <FIG>. In the case where the image capturing apparatus <NUM> is a plenoptic camera, the light field image capturing unit <NUM> generates the multi-viewpoint image data shown in <FIG> from the plenoptic image data shown in <FIG>. It may also be possible to improve image quality by performing image processing, such as by removing noise by using an image data group whose viewpoint is the same but whose focus positions are different, in the light field image capturing unit <NUM>.

The distance data acquisition unit <NUM> acquires distance data by using a sensor other than an image sensor, such as a TOF (Time-of-Flight) distance sensor. The method of acquiring distance data does not need to be the TOF method as long as distance data can be acquired, and another method, such as a method in which a laser pattern is irradiated, may be accepted. Further, it may also be possible for the additional information generation unit <NUM> to generate distance data from the image data acquired by the image sensor. According to the aspect such as this, it is no longer necessary for the image capturing apparatus <NUM> to include the distance data acquisition unit <NUM>.

The bus <NUM> is a transfer path of various kinds of data. For example, via the bus <NUM>, the image data obtained by the light field image capturing unit <NUM> by image capturing and the image data acquired by the distance data acquisition unit <NUM> are sent to a predetermined processing unit.

The CPU <NUM> centralizedly controls each unit.

The RAM <NUM> functions as a main memory, a work area, etc., of the CPU <NUM>.

The ROM <NUM> stores control programs or the like executed by the CPU <NUM>.

The operation unit <NUM> includes a button, a mode dial, etc. Via the operation unit <NUM>, user instructions are input.

The display unit <NUM> displays a photographed image and a character. The display unit <NUM> is, for example, a liquid crystal display. It may also be possible for the display unit <NUM> to have a touch screen function. In this case, it may also be possible to input user instructions via a touch screen in place of the operation unit <NUM>.

The display control unit <NUM> performs display control of an image and a character that are displayed on the display unit <NUM>.

The light field image capturing control unit <NUM> performs control of the image capturing system based on instructions from the CPU <NUM>. For example, the light field image capturing control unit <NUM> performs focusing, opens/closes a shutter, adjusts an aperture, performs continuous photographing, and so on, based on instructions from the CPU <NUM>. Due to this, in the light field image capturing unit <NUM>, a plurality of pieces of multi-viewpoint image data whose focus positions are different from one another is acquired.

The distance data acquisition control unit <NUM> controls the distance data acquisition unit <NUM> based on instructions from the CPU <NUM>. In the present embodiment, the distance data acquisition control unit <NUM> controls starting and terminating the acquisition of distance data by the distance data acquisition unit <NUM>.

The external memory control unit <NUM> is an interface for connecting a personal computer (PC) and other media (e.g., hard disc, memory card, CF card, SD card, USB memory) and the bus <NUM>.

The encoding unit <NUM> encodes digital data. Further, the encoding unit <NUM> stores encoded digital data (hereinafter, called encoded data) in a predetermined format. Furthermore, the encoding unit <NUM> generates management information, to be described later, and stores the management information in the above-described predetermined format along with the encoded data.

The free focus point image generation unit <NUM> generates image data whose focus position is different from that of the image data obtained by the light field image capturing unit <NUM> by image capturing.

The free viewpoint image generation unit <NUM> generates image data whose viewpoint position is different from that of the image data obtained by the light field image capturing unit <NUM> by image capturing.

The additional information generation unit <NUM> extracts structural information on an image. For example, the additional information generation unit <NUM> generates distance data from multi-viewpoint image data. Further, for example, the additional information generation unit <NUM> generates area division data by performing area division for each object based on the multi-viewpoint image data and the distance data.

Details of the encoding unit <NUM>, the free focus point image generation unit <NUM>, the free viewpoint image generation unit <NUM>, and the additional information generation unit <NUM> will be described later. It may also be possible for the image capturing apparatus <NUM> to include components other than those described above.

The encoding unit <NUM> is explained. The encoding unit <NUM> is capable of inputting the following digital data.

The digital data input to the encoding unit <NUM> is encoded and stored in a predetermined format. The wording such as "data is stored in a predetermined format" is used, but specifically, this means that data is stored in a storage medium or the like in accordance with a predetermined format. Digital data that is stored in a format can be added and deleted. The multi-viewpoint image data of different focus positions obtained by the light field image capturing unit <NUM> by image capturing and the distance data acquired by the distance data acquisition unit <NUM> are input to the encoding unit <NUM> via the bus <NUM>. The image data generated by the free focus point image generation unit <NUM>, the image data generated by the free viewpoint image generation unit <NUM>, and the distance data and the area division data generated by the additional information generation unit <NUM> are input to the encoding unit <NUM> via the bus <NUM>. The camera external parameters and the camera internal parameters are input to the encoding unit <NUM> from the light field image capturing control unit <NUM> via the bus <NUM>.

Next, a method of encoding multi-viewpoint image data, image data, distance data, and area division data is explained. The multi-viewpoint image data is a collection of image data whose focus position is the same and whose viewpoint positions are different.

For image data, the encoding unit <NUM> encodes the image data by using an encoding scheme of a single-viewpoint image, such as the JPEG and the PNG.

For multi-viewpoint image data, the encoding unit <NUM> may encode each piece of the image data by using an encoding scheme of a single-viewpoint image, such as the JPEG and the PNG, or by using an encoding scheme of a multi-viewpoint image, such as the MVC (Multiview Video Coding).

For distance data, the encoding unit <NUM> represents the distance data as image data and encodes the image data by using an encoding scheme of a single-viewpoint image, such as the JPEG and the PNG. For example, distance data is represented as an <NUM>-bit gray image. The pixel value of each pixel in the gray image corresponds to the distance value in a one-to-one manner. Conversion from the distance value to an <NUM>-bit pixel value may be performed by equally dividing the distance value between the minimum distance value and the maximum distance value in eight bits or by performing nonlinear division so that the resolution at a nearer distance has a higher resolution. Alternatively, another method, such as a method of causing the pixel value and the distance value to correspond to each other by using a lookup table, may be accepted. The representation of image data is not limited to an <NUM>-bit gray image and it may also be possible to use another representation method, such as a method of holding the distance value of each pixel as binary data. <FIG> are diagrams each showing an example of an image represented by distance data and an example of an image represented by area division data. In <FIG>, an example of an image represented by distance data is shown. To the object <NUM> and the object <NUM>, different pixel values are allocated. Further, also within the object <NUM>, different pixel values are allocated to portions whose distances from the image capturing unit are different.

For area division data, the encoding unit <NUM> represents the area division data as image data and encodes the image data by using an encoding scheme of a single-viewpoint image, such as the JPEG and the PNG. The area division data is also represented as an <NUM>-bit gray image like distance data. The pixel value of each pixel in the gray image corresponds to the area number. For example, in the case of black (pixel value: <NUM>), the area number is <NUM> and in the case of white (pixel value: <NUM>), the area number is <NUM>. Of course, as long as the area number and the pixel value correspond to each other, it may also be possible to use another representation method, such as a method of representing area division data as an RGB color image and a method of holding the area number as binary data. In <FIG>, an example of an image represented by area division data is shown. In <FIG>, different pixel values are allocated to the object <NUM> and the object <NUM>. The pixel values within the object <NUM> are the same because of being the same object.

Next, the format in which encoded data is stored is explained.

In the format, the previously described encoded data and management information that associates each piece of data with one another are stored. In <FIG>, an example of management information is shown. The management information is information in which a relationship between each piece of the encoded data and a pointer are described. In the present embodiment, as shown in <FIG>, the management information is described in a hierarchical structure. Further, as shown in <FIG>, the management information includes multi-viewpoint data <NUM> that centralizedly controls data of all viewpoints, viewpoint data <NUM> (<NUM>-<NUM> to <NUM>-N) that centralizedly controls data of each viewpoint, and focus point data <NUM> (<NUM>-<NUM> to <NUM>-M) that centralizedly controls data of each focus point. In <FIG> the management information in the case where the number of viewpoints is N and the number of focus points is M is shown.

Here, information that is stored in the multi-viewpoint data <NUM>, the viewpoint data <NUM>, and the focus point data <NUM> is explained.

In the multi-viewpoint data <NUM>, information that centralizedly controls data of all viewpoints, such as the number of viewpoints and the number of the representative viewpoint, is described. The number of viewpoints corresponds to the number of image capturing units in the case of the camera array image capturing apparatus as shown in <FIG>. The representative viewpoint a viewpoint to which priority is given in the case where a thumbnail of images or the like is displayed. The number of the representative viewpoint is a number capable of identifying the representative viewpoint. In addition to the above, in the case where distance data is represented by a lookup table, information on the lookup table is described. Further, in the case where the image size is the same for all the viewpoints, information on the image size is described. As long as the information is information that centralizedly controls data of all viewpoints, the contents that are described are not limited to those.

In the viewpoint data <NUM>, the camera external parameters, the number of focus point images, the number of the representative focus point image, the distance data reference information, the representation method of distance data, the minimum value and the maximum value of the distance, area division data reference information, etc., are described. The camera external parameters are information indicating a viewpoint (specifically, viewpoint position, viewpoint direction) or the like. In the present embodiment, the coordinates of the viewpoint position are described in the viewpoint data <NUM> as the camera external parameter. The representative focus point image is an image corresponding to a focus point to which priority is given in the case where a thumbnail of images is displayed. The number of the representative focus point image is a number capable of identifying the representative focus point image. The distance data reference information is information for accessing the distance data (e.g., a pointer to the distance data). The area division data reference information is information for accessing the area division data (e.g., a pointer to the area division data). As long as the information is information that is made use of for each viewpoint, the contents that are described are not limited to those.

In the focus point data <NUM>, the camera internal parameters or the like are described. The camera internal parameters indicate the focal length, the f-stop, the AF (auto focus) information at the time of being brought into focus, the distortion of a lens, etc. As long as the information is information that is made use of for each image, the contents that are described are not limited to those. In the focus point data <NUM>, image data reference information is further described. The image data reference information is information for accessing the image data (e.g., a pointer to the image data). Due to this, the image data reference information is associated with the viewpoint information (e.g., the coordinates of the viewpoint position described in the viewpoint data <NUM>) indicating the viewpoint of the image data and the focus point information (e.g., the AF information described in the focus point data <NUM>) indicating the focus position of the image data.

By describing the above-described multi-viewpoint data <NUM>, the viewpoint data <NUM>, and the focus point data <NUM> as management information, it is possible to associate the multi-viewpoint image data, the distance data, and the area division data with one another. Further, by describing the management information in the XML format, it is also made possible to read the management information by a standard XML parser. The structure of the management information is not limited to the structure shown in <FIG>. As long as the image data reference information for accessing the image data is associated with the viewpoint information indicating the viewpoint of the image data and the focus point information indicating the focus position of the image data, the management information may have another structure.

As to the file format in which management information, multi-viewpoint image data, distance data, and area division data are stored, two formats are shown below. In <FIG>, an example of the format is shown, respectively.

The first format is a format that saves a management file <NUM> in which management information is described and each pieces of data in a folder <NUM>. Each piece data is the image data <NUM> to <NUM>, distance data <NUM>, area division data <NUM>, image data <NUM> that is generated by the free-viewpoint image generation unit <NUM> (hereinafter, called free viewpoint image data), and image data <NUM> that is generated by the free focus point image generation unit <NUM> (hereinafter, called free focus point image data).

The second format is a format that describes management information in a header <NUM> of a file <NUM> and saves each piece of data in the file <NUM>. Each piece of data is the image data <NUM> to <NUM>, the distance data <NUM>, the area division data <NUM>, the free viewpoint image data <NUM>, and the free focus point image data <NUM>.

As described above, the multi-viewpoint image data, the image data, the distance data, and the area division data are encoded and stored in the above-described format along with the management information indicating the relationship between each piece of data. Hereinafter, the above-described format is called a "multidimensional information format".

The encoding unit <NUM> saves the multidimensional information format in the storage unit (storage medium), not shown schematically, which the encoding unit <NUM> itself has. It may also be possible for the encoding unit <NUM> to store the multidimensional information format in an external memory (storage medium such as an SD card) via the external memory control unit <NUM>.

The additional information generation unit <NUM> is explained. The additional information generation unit <NUM> inputs the multidimensional information format from the encoding unit <NUM> via the bus <NUM>. In the case where the multidimensional information format is stored in an external memory, it may be possible for the additional information generation unit <NUM> to read the multidimensional information format from the external memory via the external memory control unit <NUM>.

In <FIG>, data that is input to the additional information generation unit <NUM> and data that is output from the additional information generation unit <NUM> are shown schematically. The additional information generation unit <NUM> acquires multi-viewpoint image data and distance data from the input multidimensional information format. Here, the additional information generation unit <NUM> acquires the multi-viewpoint image data <NUM> (image data <NUM>, <NUM>, <NUM>, <NUM>) and the distance data <NUM> of the viewpoint for which additional information is generated shown in <FIG>. In the case where the distance data is not stored in the above-described format, only the multi-viewpoint image data <NUM> (image data <NUM>, <NUM>, <NUM>, <NUM>) is acquired. The viewpoint for which additional information is generated is specified by a user or the like via the operation unit <NUM> and the display unit <NUM>. Here, it is assumed that viewpoint <NUM> is specified.

In the case where the distance data is stored in the multidimensional information format, the additional information generation unit <NUM> generates and outputs the area division data <NUM>. In the case where the distance data is not stored in the multidimensional information format, the additional information generation unit <NUM> generates and outputs the distance data <NUM> and the area division data <NUM>. The area division data is data that is made use of in refocus processing of a second embodiment, to be described later. Consequently, in the present embodiment, it may also be possible for the additional information generation unit <NUM> to generate and output only the distance data <NUM>. The output digital data is stored in the multidimensional information format in the encoding unit <NUM> via the bus <NUM>. At this time, the encoding unit <NUM> adds information (a pointer of the distance data <NUM> or the like) relating to the distance data <NUM> to the viewpoint data corresponding to viewpoint <NUM> of the management information within the multidimensional information format. In the case where the multidimensional information format is stored in an external memory, it is sufficient for the additional information generation unit <NUM> to update the multidimensional information format stored in the external memory by using the generated additional information.

<FIG> is a block diagram showing an example of an internal configuration of the additional information generation unit <NUM>. The additional information generation unit <NUM> includes a distance data generation unit <NUM> and an area division data generation unit <NUM>. In the present embodiment, explanation is given on the assumption that the additional information generation unit <NUM> is one component within the image capturing apparatus <NUM>, but the function of the additional information generation unit <NUM> may be implemented by an external apparatus, such as a PC. That is, it is possible to implement the additional information generation unit <NUM> in the present embodiment also as one function of the image capturing apparatus or as an independent image processing apparatus.

In the following, each component of the additional information generation unit <NUM> is explained.

In the case where only the multi-viewpoint image data is input to the additional information generation unit <NUM>, the distance data generation unit <NUM> generates distance data from the multi-viewpoint image data and outputs the generated distance data to the area division data generation unit <NUM> and the bus <NUM>. The area division data generation unit <NUM> generates area division data from the multi-viewpoint image data and the distance data input from the distance data generation unit <NUM> and outputs the area division data to the bus <NUM>. In the case where the additional information generation unit <NUM> outputs only the distance data as output data, the processing by the area division data generation unit <NUM> is not performed.

In the case where the multi-viewpoint image data and the distance data acquired by the distance data acquisition unit <NUM> are input to the additional information generation unit <NUM>, the area division data generation unit <NUM> generates area division data from both pieces of input data and outputs the area division data to the bus <NUM>. At this time, the processing by the distance data generation unit <NUM> is not performed.

The distance data generation unit <NUM> is explained. <FIG> is a flowchart showing processing of the distance data generation unit <NUM>.

At step S1301, the distance data generation unit <NUM> inputs multi-viewpoint image data. Here, the case where the multi-viewpoint image data is image data corresponding to the images of four viewpoints shown in <FIG> is taken as an example.

At step S1302, the distance data generation unit <NUM> selects a base image of a viewpoint for which distance data is generated and a reference image that is referred to for generating distance data. Here, it is assumed that the image of viewpoint <NUM> shown in <FIG> is a base image and the image of viewpoint <NUM> shown in <FIG> is a reference image. The reference image may be an image of a plurality of viewpoints, but in the present embodiment, in order to make explanation easy-to-understand, it is assumed that the reference image is an image of one viewpoint.

At step S1303, the distance data generation unit <NUM> calculates a disparity from the reference image with the base image as a base. This is called a base disparity. <FIG> are diagrams for explaining a calculation method of a disparity.

First, a calculation method of a base disparity is explained by using <FIG> and <FIG>.

<FIG> is a base image (image of viewpoint <NUM>) and <FIG> is a reference image (image of viewpoint <NUM>). The viewpoint position of the base image is different from that of the reference image, and therefore, the positions of an object whose image is captured are different. The amount of deviation (disparity) of an object whose image is captured depends on the distance from the image capturing apparatus <NUM> to the object, and therefore, it is possible to calculate distance data from the disparity.

In the case where an X-coordinate (coordinate in the horizontal direction in <FIG>) <NUM> of the right eye of the object <NUM> in the base image is represented by the same coordinate in the reference image, an X-coordinate <NUM> is obtained. From the X-coordinate <NUM>, the X-coordinate of the right eye of the object <NUM> in the reference image is searched for and a corresponding point is found. The difference between the X-coordinate <NUM> and the X-coordinate of the corresponding point is a disparity <NUM>. For all the pixels in the base image, corresponding points are searched for and a base disparity is calculated.

There are various methods of searching for a corresponding point and any method may be used. For example, there is a method in which search is made for each area and a disparity that minimizes the cost value (color difference) is taken to be a corresponding point. Further, for example, there is a method in which search is made for each pixel and the cost value (color difference) is calculated, and smoothing is performed on the calculated cost value with an edge holding type filter and a disparity that minimizes the cost value is taken to be a corresponding point.

At step S1304, the distance data generation unit <NUM> calculates a disparity from the base image with the reference image as a base. This is called a reference disparity.

Next, a method of calculating a reference disparity is explained by using <FIG>.

<FIG> is a base image (image of viewpoint <NUM>) and <FIG> is a reference image (image of viewpoint <NUM>). In the case where an X-coordinate <NUM> of the right eye of the object <NUM> in the reference image is represented by the same coordinate in the base image, an X-coordinate <NUM> is obtained. From the X-coordinate <NUM>, the X-coordinate of the right eye of the object <NUM> in the base image is searched for and a corresponding point is found. The difference between the X-coordinate <NUM> and the X-coordinate of the corresponding point is a disparity <NUM>. For all the pixels in the reference image, corresponding points are searched for and a reference disparity is calculated.

At S1305, the distance data generation unit <NUM> calculates a corresponding area between the base disparity calculated at step S1303 and the reference disparity calculated at step S1304. The base disparity and the reference disparity are compared for each pixel and in the case where the difference between the base disparity and the reference disparity is less than or equal to a threshold value, the comparison-target pixel is classified as a corresponding area and in the case where the difference is greater than the threshold value, the comparison-target pixel is classified as a non-corresponding area. That is, the corresponding area is an area in which the coincidence between the base disparity and the reference disparity is high and the reliability of the disparity is high. The non-corresponding area is an area in which the coincidence between the base disparity and the reference disparity is low and the reliability of the disparity is low.

At step S1306, the distance data generation unit <NUM> corrects the disparity in the non-corresponding area classified at step S1304. As described previously, the reliability of the disparity is low in the non-corresponding area, and therefore, the disparity is supplemented by the base disparities in the peripheral corresponding areas in which the reliability is high and the base disparity in the non-corresponding area is determined.

At step S1307, the distance data generation unit <NUM> converts the base disparity into distance data and outputs the distance data.

The generation method of distance data in the distance data generation unit <NUM> is not limited to the above-described method. For the generation processing of distance data, another method may be used, such as a method that uses a reference image of a plurality of viewpoints, as long as the method generates distance data from multi-viewpoint image data. Further, in the case where the distance data generation unit <NUM> inputs a plurality of pieces of multi-viewpoint image of different focus positions at step S1301, it is sufficient to output distance data generated for each piece of multi-viewpoint image data. Then, it is sufficient for the encoding unit <NUM> to store the distance data in the multidimensional information format after integrating each piece of distance data by weighted averaging or the like. With the aspect such as this, it is made possible to acquire more accurate distance data. It may also be possible for the distance data generation unit <NUM> to output the distance data after integrating each piece of distance data.

Next, the area division data generation unit <NUM> is explained. <FIG> is a flowchart showing processing of the area division data generation unit <NUM>.

At step S1501, the area division data generation unit <NUM> inputs image data of the viewpoint for which area division data is generated and distance data. Here, the image data corresponding to the image of viewpoint <NUM> shown in <FIG> and the distance data of viewpoint <NUM> shown in <FIG> are input.

At step S1502, the area division data generation unit <NUM> selects a rectangular area that surrounds an object to be cut out based on a user operation that is input via the operation unit <NUM>. <FIG> and <FIG> are diagrams showing the way a rectangular area that surrounds an object is selected. It may also be possible to specify a rectangular area that surrounds an object to be cut out by using recognition processing, such as processing to detect a human body, without a user operation. In the example shown in <FIG>, a rectangular area <NUM> that surrounds the object <NUM> is selected.

At step S1503, the area division data generation unit <NUM> performs processing to cut out an object from the selected rectangular area. The area division data generation unit <NUM> extracts a main object within the rectangular area in the image data by performing clustering processing on the distance data within the rectangular area that surrounds the object. It may also be possible to extract a main object within the rectangular area in the image data by adding the distance data as a parameter of a cost function and by performing global optimization processing whose typical example is Graph Cut.

At step S1504, the area division data generation unit <NUM> sets an area number to the cut-out object. In the present embodiment, the area number is represented by an <NUM>-bit numerical value (<NUM> to <NUM>). Any number may be set as the area number as long as the number can be represented with eight bits (<NUM> to <NUM>). Due to this, for example, in the example shown in <FIG>, area number <NUM> is allocated to the object <NUM>.

At step S1505, the area division data generation unit <NUM> checks whether to terminate the area division processing.

In the case where an object to be cut out is left (NO at step S1505), the area division data generation unit <NUM> returns to the processing at step S1502. After returning to the processing at step S1502, the area division data generation unit <NUM> selects a rectangular area <NUM> that surrounds the object <NUM> as shown in <FIG> and performs the processing at steps S1503 and S1504. Due to this, for example, in the example shown in <FIG>, area number <NUM> is allocated to the object <NUM>.

In the case where there is no object to be cut out left (YES at step S1505), the area division data generation unit <NUM> terminates the area division.

At step S1506, the area division data generation unit <NUM> outputs area division data.

The generation processing of area division data by the area division data generation unit <NUM> is not limited to the above-described method. For the generation processing of area division data, another method may be used, such as a method of selecting part of an object in place of a rectangular area, as long as the method generates area division data from image data and distance data.

The free viewpoint image generation unit <NUM> is explained. The free viewpoint image generation unit <NUM> inputs the multidimensional information format from the encoding unit <NUM> via the bus <NUM>. Here, in the case where the multidimensional information format is stored in an external memory, it is sufficient for the free viewpoint image generation unit <NUM> to read the multidimensional information format from the external memory via the external memory control unit <NUM>. The data that is input to the free viewpoint image generation unit <NUM> and the data that is output from the free viewpoint image generation unit <NUM> are explained by using <FIG>.

The free viewpoint image generation unit <NUM> acquires multi-viewpoint image data and distance data corresponding to each viewpoint from the input multidimensional information format. Here, the free viewpoint image generation unit <NUM> acquires the multi-viewpoint image data <NUM> (image data <NUM>, <NUM>, <NUM>, <NUM>) and the distance data <NUM>, <NUM>, <NUM>, and <NUM> corresponding to each viewpoint.

The free viewpoint image generation unit <NUM> generates and outputs the image data (free viewpoint image data) <NUM> of the viewpoint different from that of the input multi-viewpoint image data. The output digital data is stored in the multidimensional information format in the encoding unit <NUM> via the bus <NUM>. At this time, the encoding unit <NUM> adds the viewpoint data corresponding to the free viewpoint image data <NUM> to the management information within the multidimensional information format, and further adds the focus point data corresponding to the free viewpoint image data <NUM> in association with the viewpoint data. In the case where the multidimensional information format is stored in an external memory, it is sufficient for the free viewpoint image generation unit <NUM> to update the multidimensional information format stored in the external memory by using the generated free viewpoint image data <NUM>.

<FIG> is a block diagram showing an example of an internal configuration of the free viewpoint image generation unit <NUM>. The free viewpoint image generation unit <NUM> includes a separation information generation unit <NUM> and a free viewpoint image combination unit <NUM>. In the present embodiment, explanation is given on the assumption that the free viewpoint image generation unit <NUM> is one component within the image capturing apparatus <NUM>, but the function of the free viewpoint image generation unit <NUM> may be implemented by an external apparatus, such as a PC. That is, it is possible to implement the free viewpoint image generation unit <NUM> in the present embodiment also as one function of an image capturing apparatus or as an independent image processing apparatus.

In the following, each component of the free viewpoint image generation unit <NUM> is explained.

In the case where multi-viewpoint image data and distance data corresponding to each viewpoint are input to the free viewpoint image generation unit <NUM>, first, both pieces of data are sent to the separation information generation unit <NUM>. Hereinafter, an image represented by image data of each viewpoint is called a viewpoint image.

The separation information generation unit <NUM> generates information (separation information) that serves as a foundation for separating each viewpoint image corresponding to the input multi-viewpoint image data into two layers (a boundary layer that is a boundary of a subject, a main layer that is not the boundary of the subject). Specifically, the separation information generation unit <NUM> classifies each pixel within each viewpoint image into two kinds of pixel: a boundary pixel adjacent to the boundary of a subject (hereinafter, called "object boundary") and a normal pixel other than the boundary pixel. Then, the separation information generation unit <NUM> generates information capable of specifying the kind to which each pixel corresponds.

<FIG> is a flowchart showing processing of the separation information generation unit <NUM>.

At step S1901, the separation information generation unit <NUM> inputs multi-viewpoint image data and distance data corresponding to each viewpoint.

At step S1902, the separation information generation unit <NUM> extracts the object boundary of a viewpoint image. In the present embodiment, the portion at which the difference between the distance data of the target pixel and the distance data of an adjacent pixel (hereinafter, called "difference in distance data") is greater than or equal to a threshold value is specified as the object boundary. Specifically, the processing is as follows.

First, the separation information generation unit <NUM> scans the viewpoint image in the longitudinal direction, compares the difference in distance data with the threshold value, and specifies the pixel whose difference in distance data is greater than or equal to the threshold value. Next, the separation information generation unit <NUM> scans the viewpoint image in the transverse direction, similarly compares the difference in distance data with the threshold value, and specifies the pixel whose difference in distance data is greater than or equal to the threshold value. Then, the separation information generation unit <NUM> specifies the sum-set of the pixels specified in the longitudinal direction and in the transverse direction, respectively, as the object boundary. As the threshold value, for example, a value, such as "<NUM>", is set in the case where the distance data is quantized with eight bits (<NUM> to <NUM>).

At step S1903, the separation information generation unit <NUM> classifies each pixel within each viewpoint image into the two kinds of pixel: the boundary pixel and the normal pixel. Specifically, the separation information generation unit <NUM> refers to the distance data acquired at step S1901 and determines a pixel adjacent to the object boundary specified at step S1902 as the boundary pixel.

<FIG> is a diagram showing the way each pixel within a viewpoint image is classified into two kinds of pixel: the boundary pixel and the normal pixel. Adjacent pixels that stride an object boundary <NUM> are classified as boundary pixels <NUM> and2003, and the rest of the pixels are classified as normal pixels <NUM>, respectively. In <FIG>, the boundary pixel is represented by a black circle and the normal pixel is represented by a white circle. In the following diagrams also, the boundary pixel is represented by a black circle and the normal pixel by a white circle. Here, only the pixels adjacent to the object boundary are classified as the boundary pixels, but another separation method may be used, such as method of classifying pixels within the width of two pixels from the object boundary as the boundary pixels.

At step S1904, the separation information generation unit <NUM> determines whether the classification of the pixel has been completed for all the viewpoint images corresponding to the input multi-viewpoint image data.

In the case where there is a viewpoint image for which the processing has not been performed yet (YES at step S1904), the separation information generation unit <NUM> returns to the processing at step S1902 and performs the processing at step S1902 and step S1903 for the next viewpoint image. On the other hand, in the case where the classification of the pixel has been completed for all the viewpoint images (NO at step S1904), the separation information generation unit <NUM> proceeds to the processing at step S1905.

At step S1905, the separation information generation unit <NUM> sends separation information capable of specifying the boundary pixel and the normal pixel to the free viewpoint image combination unit <NUM>. Once the boundary pixels are specified, it turns out that the rest of the pixels are the normal pixels, and therefore, the separation information may be any information capable of specifying the boundary pixel. Consequently, for example, as the separation information, a method or the like is considered, in which a flag is attached to the pixel in such a manner that " <NUM>" is attached to the pixel determined to be the boundary pixel and "<NUM>" is attached to the pixel determined to be the normal pixel. The free viewpoint image combination unit <NUM> separates a predetermined viewpoint image into two layers (i.e., a boundary layer made up of the boundary pixels and a main layer made up of the normal pixels) by using the separation information such as this.

<FIG> are diagrams for explaining the processing of the separation information generation unit <NUM>. In the examples shown in <FIG>, object insides <NUM> and <NUM> in the image of viewpoint <NUM> and object insides <NUM> and <NUM> in the image of viewpoint <NUM> are represented as the main layer. Further, object boundary portions <NUM> and <NUM> in the image of viewpoint <NUM> and object boundary portions <NUM> and <NUM> in the image of viewpoint <NUM> are represented as the boundary layer. In <FIG> and <FIG>, in order to simplify explanation, only the layer made up of the normal pixels inside the object boundary portion is represented as the main layer.

The free viewpoint image combination unit <NUM> sets a reference image group that is made use of for free viewpoint image combination, and performs rendering of the main layer of the reference image group first, and next, performs rendering of the boundary layer of the reference image group. Then, the free viewpoint image combination unit <NUM> generates image data (free viewpoint image data) at an arbitrary viewpoint position by combining each rendered image. <FIG> is a flowchart showing the processing of the free viewpoint image combination unit <NUM>.

At step S2101, the free viewpoint image combination unit <NUM> acquires position information on an arbitrary viewpoint (hereinafter, called "free viewpoint") specified by a user. In the present embodiment, the position information on a free viewpoint is coordinate information indicating the position of a free viewpoint in the case where the position of viewpoint <NUM> shown in <FIG> is taken as a base. In the case where the coordinates of viewpoint <NUM> taken as a base are supposed to be (<NUM>, <NUM>), viewpoint <NUM> shown in <FIG> is represented by the coordinates (<NUM>, <NUM>), viewpoint <NUM> shown in <FIG> by (<NUM>, <NUM>), and viewpoint <NUM> shown in <FIG> by (<NUM>, <NUM>), respectively. Here, for example, in the case where a user desires to combine an image whose free viewpoint is the middle position of viewpoints <NUM> to <NUM>, the user inputs the coordinates (<NUM>, <NUM>). The method of defining the coordinates is not limited to the above-described method and it may also be possible to take a position other than viewpoint <NUM> as a base. Further, the method of inputting position information on a free viewpoint is not limited to the above-described method of directly inputting the coordinates. For example, in the case where the image capturing units are arranged as shown in <FIG>, it may also be possible to display a UI screen (not shown) indicating the arrangement of the image capturing units <NUM> to <NUM> on the display unit <NUM>, thereby enabling a user to specify a desired free viewpoint by a touch operation or the like.

At step S2102, the free viewpoint image combination unit <NUM> sets a plurality of viewpoint images (hereinafter, called "reference image group") that is referred to in generating free viewpoint image data. In the present embodiment, the free viewpoint image combination unit <NUM> sets four viewpoint images close to the position of the specified free viewpoint as a reference image group. As described above, the reference image group in the case where the coordinates (<NUM>, <NUM>) are specified as the position of the free viewpoint, the reference image group is made up of the viewpoint images of viewpoints <NUM> to <NUM> shown in <FIG>. The number of viewpoint images making up the reference image group is not limited to four and the reference image group may be made up of three viewpoint images around the specified free viewpoint. Further, the reference image group is only required to be a group of images corresponding to the viewpoints surrounding the position of the specified free viewpoint, and therefore, for example, it may also be possible to set viewpoint images captured at four viewpoint positions not nearest to the position of the specified free viewpoint as a reference image group.

At step S2103, the free viewpoint image combination unit <NUM> performs processing to generate a three-dimensional model of the main layer of the reference image. The three-dimensional model of the main layer is generated by constructing a quadrilateral mesh by mutually connecting four pixels including the normal pixels that are not related to the object boundary. In <FIG>, the way the three-dimensional model of the main layer is generated is shown. As shown in <FIG>, for example, a quadrilateral mesh <NUM> is constructed by connecting pixels (the one normal pixel <NUM>, one normal pixel <NUM>, and two boundary pixels <NUM> and <NUM>) that are four pixels including the normal pixels and none of which is related to the object boundary <NUM>. By repeatedly performing the processing such as this, all the quadrilateral meshes, each of which forms the three-dimensional model of the main layer, are constructed. The minimum size of the quadrilateral mesh at this time is one pixel × one pixel. In the present embodiment, all the main layers are constructed by the quadrilateral mesh having the size of one pixel × one pixel, but it may also be possible to construct the main layer by a larger quadrilateral mesh. Further, it may also be possible to construct the main layer by a mesh having a shape other than the quadrilateral, for example, such as a triangular mesh.

The X coordinate and the Y coordinate of the quadrilateral mesh made up of one pixel, which is constructed as described above, correspond to the global coordinates calculated from the camera parameters of the viewpoint image and the Z coordinate corresponds to the distance of each pixel to the subject, which is obtained from the distance information. Then, the free viewpoint image combination unit <NUM> generates the three-dimensional model of the main layer by texture-mapping the color information on each pixel to the quadrilateral mesh.

Explanation is returned to the flowchart in <FIG>.

At step S2104, the free viewpoint image combination unit <NUM> performs rendering of the main layer of the reference image at the free viewpoint position. Specifically, the free viewpoint image combination unit <NUM> performs rendering of the three-dimensional model of the main layer of the reference image generated at step S2103 at the free viewpoint position acquired at step S2101.

The processing at steps S2103 and S2104 is performed for each reference image of the reference image group.

In <FIG>, the way rendering of the main layer is performed is shown. In <FIG>, the horizontal axis represents the X-coordinate and the vertical axis represents the Z-coordinate. Further, in <FIG>, it is assumed that the object boundary (not shown schematically) exists between boundary pixels <NUM> and <NUM>. Furthermore, in <FIG>, segments <NUM> and <NUM> (hereinafter, also represented sometimes as boundary layers <NUM> and <NUM> or quadrilateral meshes <NUM> and <NUM>) each indicate the quadrilateral mesh of the main layer in the case where the three-dimensional model is generated from a viewpoint (hereinafter, called "reference viewpoint") <NUM> of the reference image indicated by a white inverted triangle. That is, in the example shown in <FIG>, the quadrilateral mesh <NUM> connecting a normal pixel <NUM> and the boundary pixel <NUM> and the quadrilateral mesh <NUM> connecting a normal image <NUM> and the boundary image <NUM> are generated as the three-dimensional model of the main layer. The image obtained by performing rendering of the quadrilateral meshes <NUM> and <NUM> at a free viewpoint <NUM> indicated by a black inverted triangle is a rendered image. In the rendering processing, the pixel portion at which no color exists is left as a hole. In <FIG>, arrows <NUM> and <NUM> indicate at which position the quadrilateral mesh <NUM> is viewed at the reference viewpoint <NUM> and the free viewpoint <NUM>, respectively. At the free viewpoint <NUM> located on the left side of the reference viewpoint <NUM>, the quadrilateral mesh <NUM> is located on the right side of the reference viewpoint <NUM>. Similarly, arrows <NUM> and <NUM> indicate at which position the quadrilateral mesh <NUM> is viewed at the reference viewpoint <NUM> and the free viewpoint <NUM>, respectively.

At step S2105, the free viewpoint image combination unit <NUM> obtains the integrated image data of the main layer by integrating the rendering results of the main layer at the specified free viewpoint position. In the present embodiment, the (four) rendered images generated from the main layer of the reference image are integrated. The integration processing is performed for each pixel and the color after the integration is calculated by using a weighted average of each rendered image, specifically, a weighted average based on the position of the specified free viewpoint and the distance from the reference image. For example, in the case where the position of the specified free viewpoint is equidistant from the four viewpoint positions corresponding to each reference image, the weight corresponding to each rendered image is the same and is <NUM>. On the other hand, in the case where the position of the specified free viewpoint is close to the viewpoint position of any of the reference images, the smaller the distance, the larger the weight is. The method of finding the average color is not limited to this. Further, the portion of a hole (the pixel at which the quadrilateral mesh is not constructed) of each rendered image is not taken to be the target of the color calculation at the time of integration. That is, for the portion of a hole in any of the rendered images, the color after the integration is calculated by using the weighted average that targets the rendered image with no hole at the portion. The portion where there is a hole in all the rendered images is left as a hole.

At step S2106, the free viewpoint image combination unit <NUM> generates a three-dimensional model of a boundary layer of a reference image. In the boundary layer in contact with the object boundary, connection with an adjacent pixel is not performed at the time of generation of a mesh. Specifically, the free viewpoint image combination unit <NUM> generates the three-dimensional model of the boundary layer by constructing one quadrilateral mesh for one pixel. In <FIG>, the way the three-dimensional model of the boundary layer is generated is shown. The free viewpoint image combination unit <NUM> constructs a quadrilateral mesh <NUM> whose size is one pixel × one pixel for a boundary pixel <NUM>. The free viewpoint image combination unit <NUM> repeatedly performs the processing such as this for the boundary pixel and constructs all the quadrilateral meshes, each of which forms the three-dimensional model of the boundary layer. The X-coordinate and the Y-coordinate of the quadrilateral mesh made up of one pixel, which is constructed as described above, correspond to the global coordinates calculated from the camera parameters of the viewpoint image and the Z-coordinate corresponds to the distance of each boundary pixel to the subject, which is obtained from the distance information. Then, the free viewpoint image combination unit <NUM> generates the three-dimensional model of the boundary layer by using the color information on each boundary pixel as the color of the quadrilateral mesh. The processing at step S2106 is performed for each reference image of the reference image group.

At step S2107, the free viewpoint image combination unit <NUM> performs rendering of the boundary layer of the reference image. <FIG> is a diagram showing the way rendering of the boundary layer is performed. As in <FIG>, in <FIG>, the horizontal axis represents the X-coordinate and the vertical axis represents the Z-coordinate. Further, in <FIG>, it is assumed that the object boundary (not shown schematically) exists between the boundary pixel <NUM> and the boundary pixel <NUM>. Furthermore, in <FIG>, segments <NUM> and <NUM> (hereinafter, also represented sometimes as boundary layers <NUM> and <NUM> or quadrilateral meshes <NUM> and <NUM>) each indicate the quadrilateral mesh of the boundary layer in the case where the three-dimensional model is generated from the reference viewpoint <NUM> indicated by a white inverted triangle. The boundary layers <NUM> and <NUM> are each a quadrilateral mesh having distance information and color information on the boundary pixels <NUM> and <NUM> and made up of one pixel. The image obtained by performing rendering of the quadrilateral meshes <NUM> and <NUM> each made up of one pixel at the position of the free viewpoint (free viewpoint <NUM> indicated by a black inverted triangle in <FIG>) specified at step S2101 is a rendered image of the boundary layer. In the rendering processing of the boundary layer also, the portion at which no color exists (pixel for which the quadrilateral mesh is not constructed) is left as a hole. Then, the free viewpoint image combination unit <NUM> performs the rendering processing as described above for all the reference images of the reference image group and obtains the rendered image group of the boundary layer. In <FIG>, arrows <NUM> and <NUM> indicate at which position the quadrilateral mesh <NUM> is viewed at the reference viewpoint <NUM> and the free viewpoint <NUM>, respectively. At the free viewpoint <NUM> located on the left side of the reference viewpoint <NUM>, the quadrilateral mesh <NUM> is located on the right side of the reference viewpoint <NUM>.

At step S2108, the free viewpoint image combination unit <NUM> obtains the integrated image data of the boundary layer by integrating the rendered image group of the boundary layer. At this time, by the same integration processing as that at step S2105, the (four) rendered images of the boundary layer generated from the four viewpoint images are integrated.

At step S2109, the free viewpoint image combination unit <NUM> obtains two-layer integrated image data by integrating the integrated image data of the main layer obtained at step S2105 and the integrated image data of the boundary layer obtained at step S2108. The integration processing here is also performed for each pixel. At this time, an image with higher accuracy is obtained stably from the integrated image of the main layer than from the integrated image of the boundary layer, and therefore, the integrated image of the main layer is preferentially made use of. Consequently, only in the case where there is a hole in the integrated image of the main layer and there is no hole in the integrated image of the boundary layer, supplementation is performed by using the color of the boundary layer. In the case where there is a hole both in the integrated image of the main layer and in the integrated image of the boundary layer, the portion is left as a hole. By the above processing, the free viewpoint image combination unit <NUM> obtains two-layer integrated image data.

The reason the processing is performed in the order of the rendering of the main layer and the rendering of the boundary layer in the present embodiment is to suppress the image quality in the vicinity of the object boundary from deteriorating.

At step S2110, the free viewpoint image combination unit <NUM> performs hole filling processing. Specifically, the free viewpoint image combination unit <NUM> supplements the portion left as a hole in the two-layer integrated image data obtained at step S2109 by using the peripheral color. In the present embodiment, the hole filling processing is performed by selecting a pixel whose distance data exhibits a larger value from among the peripheral pixels of the hole filling target pixels. For the hole filling processing, another method may be used.

At step S2111, the free viewpoint image combination unit <NUM> outputs the free viewpoint image data for which the hole filling processing has been completed.

The free focus point image generation unit <NUM> is explained. The free focus point image generation unit <NUM> inputs the multidimensional information format from the encoding unit <NUM> via the bus <NUM>. Here, in the case where the multidimensional information format is stored in an external memory, it is sufficient for the free focus point image generation unit <NUM> to read the multidimensional information format from the external memory via the external memory control unit <NUM>. The data that is input to the free focus point image generation unit <NUM> and the data that is output from the free focus point image generation unit <NUM> are explained by using <FIG>, <FIG>.

The free focus point image generation unit <NUM> of the present embodiment acquires the multi-viewpoint image data of different focus positions and the distance data of the viewpoint for which a free focus point image is generated from the input multidimensional information format. The viewpoint for which a free focus point image is generated is specified by a user operation that is input to the operation unit <NUM>.

The free focus point image generation unit <NUM> generates and outputs the image data (free focus point image data) <NUM> whose focus position is different from that of the input multi-viewpoint image data. The output digital data is stored in the multidimensional information format in the encoding unit <NUM> via the bus <NUM>. At this time, the encoding unit <NUM> adds the viewpoint data corresponding to the free focus point image data <NUM> to the management information within the multidimensional information format and further, adds the focus point data corresponding to the free focus point image data <NUM> in association with the viewpoint data. In the case where the multidimensional information format is stored in an external memory, it is sufficient for the free focus point image generation unit <NUM> to update the multidimensional information format stored in the external memory by using the generated free focus point image data <NUM>.

<FIG> is a flowchart showing processing of the free focus point image generation unit <NUM> according to the first embodiment. In the present embodiment, explanation is given on the assumption that the free focus point image generation unit <NUM> is one component within the image capturing apparatus <NUM>, but the function of the free focus point image generation unit <NUM> may be implemented by an external apparatus, such as a PC. That is, it is possible to implement the free focus point image generation unit <NUM> in the present embodiment also as one function of the image capturing apparatus or as an independent image processing apparatus.

At step S2601, the free focus point image generation unit <NUM> acquires multi-viewpoint image data of different focus positions and distance data. Here, the multi-viewpoint image data <NUM> (image data <NUM>, <NUM>, <NUM>, <NUM>), the multi-viewpoint image data <NUM> (image data <NUM>, <NUM>, <NUM>, <NUM>) of the focus position different from that of the multi-viewpoint image data <NUM>, and the distance data <NUM> of the viewpoint for which a free focus point image is generated shown in <FIG>. Hereinafter, the focus position of the multi-viewpoint image data <NUM> is called "focus point <NUM>" and the focus position of the multi-viewpoint image data <NUM> is called "focus point <NUM>". Further, the viewpoint positions of the image data <NUM>, <NUM>, <NUM>, and <NUM> of the multi-viewpoint image data <NUM> are called "viewpoint <NUM>", "viewpoint <NUM>", "viewpoint <NUM>", and "viewpoint <NUM>", respectively. Similarly, the viewpoint positions of the image data <NUM>, <NUM>, <NUM>, and <NUM> of the multi-viewpoint image data <NUM> are called "viewpoint <NUM>", "viewpoint <NUM>", "viewpoint <NUM>", and "viewpoint <NUM>", respectively. Further, the image corresponding to viewpoint n and focus point m is represented as "image (viewpoint n, focus point m)".

<FIG> are diagrams for explaining the generation processing of free focus point image data. Here, the case where a user selects "viewpoint <NUM>" as the viewpoint for which a free focus point image is generated is taken as an example. <FIG> shows the image represented by the image data <NUM>, i.e., the image (viewpoint <NUM>, focus point <NUM>). <FIG> shows the image represented by the image data <NUM>, i.e., the image (viewpoint <NUM>, focus point <NUM>). In <FIG>, objects <NUM> and <NUM> are in focus and objects <NUM> and <NUM> are out of focus. On the contrary, in <FIG>, the objects <NUM> and <NUM> are in focus and the objects <NUM> and <NUM> are out of focus. <FIG> and <FIG> each show a free focus point image that is generated by processing at step S2604, to be described later.

In <FIG>, depths of field of the multi-viewpoint image data and the generated free focus point image are shown. The vertical axis represents the Z-direction (distance). A depth of field <NUM> is a depth of field of the multi-viewpoint image data <NUM> including the image data <NUM>. The position and the length in the vertical direction of a bidirectional arrow indicate the position and the range of the depth of field. A depth of field <NUM> is a depth of field of the multi-viewpoint image data <NUM> including the image data <NUM>. A depth of field <NUM> is a depth of field of the free focus point image in which the object <NUM> is in focus shown in <FIG>. A depth of field <NUM> is a depth of field of the free focus point image in which the object <NUM> is in focus shown in <FIG>.

At step S2602, the free focus point image generation unit <NUM> selects a subject to be brought into focus and acquires the distance to the subject. In the present embodiment, selection of a subject to be brought into focus is made by a user operation that is input to the operation unit <NUM>. For example, it may also be possible to display a thumbnail of the images shown in <FIG>, and for a user to select a subject to be brought into focus from among the thumbnail by a touch operation or the like. It may also be possible to specify a subject recognized by face detection or the like as a subject to be brought into focus without a user operation. For example, in the case where a pixel within the object <NUM> is specified, the free focus point image generation unit <NUM> acquires a representative distance value of the object <NUM> from the distance data <NUM> of the corresponding viewpoint. The representative distance value is the median of the peripheral blocks (e.g., <NUM> × <NUM> blocks) of the specified pixel position. The method of finding a representative distance value is not limited to this and another method may be used, such as a method in which the average value of the peripheral blocks is used and a method in which the distance value of the specified pixel position is used.

At step S2603, the free focus point image generation unit <NUM> selects multi-viewpoint image data based on the distance to the subject acquired at step S2602.

Here, details of the processing at step S2603 are explained. Here, the case where multi-viewpoint image data is stored in a storage medium in accordance with the multidimensional information format (folder <NUM>) shown in <FIG> is taken as an example.

First, the free focus point image generation unit <NUM> refers to the management information (specifically, the multi-viewpoint data <NUM>) described in the management file <NUM> within the folder <NUM> and acquires the number of viewpoints.

Further, the free focus point image generation unit <NUM> acquires viewpoint data corresponding to the acquired number of viewpoints. For example, in the case where the number of viewpoints is four, the viewpoint data <NUM>-<NUM> to <NUM>-<NUM> is acquired.

Furthermore, the free focus point image generation unit <NUM> acquires the focus point data corresponding to the distance to the subject from the focus point data associated with each viewpoint data. In the present embodiment, the free focus point image generation unit <NUM> refers to the camera internal parameters (e.g., f-stop, AF information at the time of being brought into focus) described in the focus point data and determines whether or not the subject is included within the depth of field indicated by the camera internal parameters. Then, in the case of determining that the subject is included, the free focus point image generation unit <NUM> acquires the focus point data as focus point data corresponding to the distance to the subject.

Finally, the free focus point image generation unit <NUM> refers to the pointer to the image data described in the acquired focus point data and reads the image data from the folder <NUM>.

By the processing such as this, in the case where the object <NUM> is specified at step S2602, the multi-viewpoint image data having the depth of field including the object <NUM> is selected. Specifically, the multi-viewpoint image data <NUM> having the depth of field <NUM> including the depth of field <NUM> is selected. Further, for example, in the case where the object <NUM> is specified at step S2602, the multi-viewpoint image data having the depth of field including the object <NUM> is selected. Specifically, the multi-viewpoint image data <NUM> having the depth of field <NUM> including the depth of field <NUM> is selected. The selected multi-viewpoint image data is made use of in refocus processing (change processing of focus position) at step S2604.

At step S2604, the free focus point image generation unit <NUM> performs refocus processing by using the multi-viewpoint image data selected at step S2603. In the refocus processing of the present embodiment, the multi-viewpoint image is shifted and the free focus point image in which the subject selected by a user is in focus is acquired. Specifically, the refocus processing is performed by performing shift addition of the multi-viewpoint image data. The amount of shift is determined based on the distance value acquired at step S2602.

The shift addition is explained by using <FIG> are diagrams for explaining the shift addition. It is assumed that the image shown in <FIG> is the image represented by the image data <NUM> (image of viewpoint <NUM> (left-eye viewpoint)) and is the image that is taken to be a base in the shift addition. It is also assumed that the image shown in <FIG> is the image represented by the image data <NUM> (image of viewpoint <NUM> (right-eye viewpoint)) and is the image that is shifted in the shift addition. The distance value acquired at step S2602 corresponds to a disparity <NUM> of the object <NUM>. In the case where it is desired to bring the object <NUM> into focus, the free focus point image generation unit <NUM> shifts the image of viewpoint <NUM> (right-eye viewpoint) in the rightward direction (rightward direction in <FIG>) by the disparity <NUM> and adds the shifted image of viewpoint <NUM> (right-eye viewpoint) to the image of viewpoint <NUM>, which is a base. The wording "add an image" or the like is used, but specifically, this means to add the image data (pixel value) representing the image. In the case where the disparity <NUM> is in the opposite direction, it is sufficient to shift the image of viewpoint <NUM> (right-eye point) in the leftward direction (leftward direction in <FIG>) by the disparity <NUM>. The free focus point image generation unit <NUM> performs the same processing by taking the image of viewpoint <NUM> to be a base and the image of another viewpoint (viewpoint <NUM>, viewpoint <NUM>) to be the image that is shifted. By integrating the image data <NUM> that is taken to be a base and the three shifted pieces of image data <NUM>, <NUM>, and <NUM> in this manner, the free focus point image data <NUM> in which the object <NUM> is in focus is generated.

In the case where the object <NUM> is specified at step S2602, the image (shown in <FIG>) represented by the free focus point image data <NUM> generated at step S2604 is the image in which only the object <NUM> is in focus and the objects <NUM>, <NUM>, and <NUM> are out of focus. Further, the range of the depth of field (the depth of field <NUM> shown in <FIG>) of the image is narrower than the range of the depth of field <NUM> of the multi-viewpoint image data <NUM> as shown in <FIG>. That is, the image is an image whose depth of field is shallower than that of the image represented by the multi-viewpoint image data <NUM>.

Further, in the case where the object <NUM> is specified at step S2602, the image (image shown in <FIG>) represented by the free focus point image data <NUM> generated at step S2604 is the image in which only the object <NUM> is in focus and the objects <NUM>, <NUM>, and <NUM> are out of focus. The range of the depth of field (the depth of field <NUM> shown in <FIG>) of the image is narrower than the range of the depth of field <NUM> of the multi-viewpoint image data <NUM> as shown in <FIG>. That is, the image is an image whose depth of field is shallower than that of the image represented by the multi-viewpoint image data <NUM>.

At step S2605, the free focus point image generation unit <NUM> outputs the generated free focus point image data <NUM>.

As explained above, in the present embodiment, the data recording apparatus (corresponding to the encoding unit <NUM>, the additional information generation unit <NUM>, the free viewpoint image generation unit <NUM>, and the free focus point image generation unit <NUM> shown in <FIG>) associates the image data of a plurality of focus positions, the image data of a plurality of viewpoint positions, the distance data, and the area division data with one another and stores them in the storage medium in accordance with the predetermined format. Due to this, it is made possible to make use of the image data obtained by performing image capturing using a camera array or a plenoptic camera for more purposes of use. For example, it is possible to make use of the image data in image processing, such as processing to change the viewpoint position after image capturing. Further, it is possible to make use of the image data in refocus processing, such as processing to adjust the focus position after image capturing and processing to control the depth of field.

Furthermore, in the present embodiment, the image data generated by the free focus point image generation unit <NUM> and the free viewpoint image generation unit <NUM> is stored in the multidimensional information format. Due to this, by making use of the multidimensional information format according to the present embodiment, it is made possible to use not only the captured image data but also the image data generated from the captured image data in image processing.

Furthermore, by using the format in the present embodiment, the image data, the distance data, and the area division data are recorded in association with one another, and therefore, it is made easy to access the data. That is, it is made possible to quickly perform image processing that makes use of the data.

In the first embodiment, the distance data of the base viewpoint is made use of for free focus point image generation. That is, in the first embodiment, the distance data is made use of as the amount of shift of the refocus processing. In the present embodiment, the distance data of the base viewpoint and the area division data are made use of for free focus point image generation. Due to this, the refocus processing to bring each object into focus is implemented. In the following, explanation of the portions in common to those of the first embodiment is omitted and the processing in the free focus point image generation unit <NUM>, which is a different point, is explained mainly.

The free focus point image generation unit <NUM> of the present embodiment further inputs the area division data in addition to the multi-viewpoint image data of different focus positions and the distance data of the viewpoint for which a free focus point image is generated.

The free focus point image generation unit <NUM> of the present embodiment is the same as that of the first embodiment and generates and outputs the free focus point image data <NUM> of the focus position different from that of the input multi-viewpoint image data. The output digital data is stored in the multidimensional information format in the encoding unit <NUM> via the bus <NUM>. <FIG> is a diagram schematically showing data that is input and output by the free focus point image generation unit <NUM> according to the second embodiment.

<FIG> is a flowchart showing processing of the free focus point image generation unit <NUM> according to the second embodiment. Here, the case where a user selects "viewpoint <NUM>" as a viewpoint for which a free focus point image is generated is taken as an example.

At step S3101, the free focus point image generation unit <NUM> acquires the multi-viewpoint image data of the different focus positions, the distance data, and the area division data. As shown in <FIG>, here, in addition to the multi-viewpoint image data <NUM>, the multi-viewpoint image data <NUM> of the focus position different from that of the multi-viewpoint image data <NUM>, and the distance data <NUM> of the viewpoint for which a free focus point image is generated shown in <FIG>, the area division data <NUM> is input.

At step S3102, the free focus point image generation unit <NUM> acquires an area (target area) to be brought into focus and the distance to a subject in the target area. In the present embodiment, it is assumed that selection of the target area to be brought into focus is made by a user operation that is input to the operation unit <NUM>. For example, it may also be possible to display a thumbnail of the images shown in <FIG> on the display unit <NUM>, and for a user to select a target area to be brought into focus from among the thumbnail by a touch operation or the like. It may also be possible to specify a target area recognized by face detection or the like as an area to be brought into focus. For example, in the case where the object <NUM> shown in <FIG> is selected as the target area, the free focus point image generation unit <NUM> acquires the distance value of each pixel in the target area from the distance data <NUM>. Further, in the case where the object <NUM> is selected as the target area, the free focus point image generation unit <NUM> acquires the distance value of each pixel in the target area from the distance data <NUM>.

At step S3103, the free focus point image generation unit <NUM> selects multi-viewpoint image data based on the distance to the subject in the target area selected at step S3102. The selection processing of multi-viewpoint image data at step S3103 is the same as the processing at step S2603, and therefore, detailed explanation is omitted. The selected multi-viewpoint image data is made use of in refocus processing (change processing of the focus position) at step S3104. In the case where the object <NUM> is specified at step S3102, the multi-viewpoint image data <NUM> having the depth of field (in the example shown in <FIG>, the depth of field <NUM>) including the object <NUM> is selected. In the case where the object <NUM> is specified at step S3102, the multi-viewpoint image data <NUM> having the depth of field (in the example shown in <FIG>, the depth of field <NUM>) including the object <NUM> is selected.

At step S3104, the free focus point image generation unit <NUM> performs refocus processing by using the multi-viewpoint image data selected at step S3103. In the refocus processing of the present embodiment, the multi-viewpoint image data is shifted and the free focus point image in which the target area (object) selected by a user is in focus is acquired. Specifically, by performing shift addition of the multi-viewpoint image data, the refocus processing is performed. The amount of shift at this time is determined based on the distance value of the target area acquired at step S3102.

The shift addition is explained by using <FIG>, and <FIG> and <FIG>. <FIG> and <FIG> are diagrams for explaining the shift addition according to the second embodiment. Similar to the image shown in <FIG>, it is assumed that the image shown in <FIG> is the image (image of viewpoint <NUM> (left-eye viewpoint)) represented by the image data <NUM> and is the image that is taken to be a base in the shift addition. Similar to the image shown in <FIG>, it is assumed that the image shown in <FIG> is the image (image of viewpoint <NUM> (right-eye viewpoint)) represented by the image data <NUM> and is the image that is shifted in the shift addition. Of the distance values acquired at step S3102, the distance value in the area of the nose of the object <NUM> corresponds to a disparity <NUM>.

Here, in the case where the disparity <NUM> and the disparity <NUM> have different amounts of disparity, by the refocus processing method in the first embodiment, in the case where the shift addition is performed based on the disparity <NUM>, the eyes of the object <NUM> are brought into focus and the nose of the object <NUM> is not brought into focus. In the case where the shift addition is performed based on the disparity <NUM>, the nose of the object <NUM> is brought into focus and the eyes of the object <NUM> are not brought into focus.

That is, in the case where it is desired to bring the whole of the object <NUM> into focus, it is necessary to perform the addition by changing the amount of shift. In the case where it is desired to bring the object <NUM> into focus, the image of viewpoint <NUM> (right viewpoint) is shifted in the rightward direction (rightward direction in <FIG>) in such a manner that the area of the eyes is shifted by the disparity <NUM> and the area of the nose is shifted by the disparity <NUM>, and then, the shifted image is added to the image of viewpoint <NUM>, which is a base. In the case where the disparity is in the opposite direction, it is sufficient to shift the image of viewpoint <NUM> (right viewpoint) in the leftward direction (leftward direction in <FIG>) in such a manner that the area of the eyes is shifted by the disparity <NUM> and the area of the nose is shifted by the disparity <NUM>. As to the area other than the object <NUM>, the area is shifted by a representative disparity for example, the area is shifted by the disparity <NUM> of the area of the eyes.

The free focus point image generation unit <NUM> performs the same processing by taking the image of viewpoint <NUM> to be a base and the image of another viewpoint (viewpoint <NUM>, viewpoint <NUM>) to be the image that is shifted. By integrating the image data <NUM> that is a base and the three shifted pieces of the image data <NUM>, <NUM>, and <NUM> in this manner, the free focus point image data <NUM> in which the object <NUM> is in focus is generated.

The subsequent processing is the same as that of the first embodiment.

In the present embodiment, in order to simplify explanation, the amount of shift of the area of the eyes is made to differ from that of the area of the nose within the object <NUM> in the refocus processing, but actually, as to the area other than those of the eyes and the nose within the object <NUM>, the refocus processing is performed by making the amounts of shift to differ from one another.

As above, in the present embodiment, in the free focus point image generation after image capturing, the refocus processing is performed by using not only the distance data of the base viewpoint but also the area division data of the base viewpoint. Due to this, not only the same effect as that of the first embodiment is obtained but also it is made possible to appropriately bring the object specified by a user into focus for the image data obtained by performing image capturing using a camera array or a plenoptic camera.

Claim 1:
A data recording apparatus comprising:
an input unit (<NUM>) configured to input an image data group including at least:
image data obtained by performing image capturing at a first focus position from a first viewpoint;
image data obtained by performing image capturing at a second focus position different from the first focus position from the first viewpoint; and
image data obtained by performing image capturing at the first focus position from a second viewpoint different from the first viewpoint; and
a recording unit (<NUM>) configured to record management information and the image data group,
characterized in that
the recording unit (<NUM>) is configured to generate the management information, which associates each piece of image data of the image data group that is input by the input unit, and which includes information indicating a viewpoint which is related with information indicating a focus position and the information indicating the focus position which includes a pointer for accessing the image data, and to record the generated management information and the image data group in a storage medium in accordance with a predetermined format, wherein
the management information is information storing image data reference information for accessing image data, viewpoint information indicating a viewpoint of the image data, and focus point information indicating a focus position of the image data in association with one another for each piece of the image data, and
the input unit (<NUM>) is configured to input area division data corresponding to the first viewpoint, which is generated by dividing the image data of the first viewpoint for each object, and
the recording unit (<NUM>) is configured to store area division data reference information for accessing the area division data that is input by the input unit in the management information in association with the viewpoint information corresponding to the first viewpoint, and
the input unit (<NUM>) is further configured to input distance data between the first viewpoint and each object, and
the recording unit (<NUM>) is further configured to store distance data reference information for accessing the distance data that is input by the input unit in the management information in association with the viewpoint information corresponding to the first viewpoint.