Image processing apparatus, method, and storage medium

An image processing apparatus according to an embodiment of the present invention specifies a partial area of a three-dimensional shape model that is generated based on a plurality of captured images obtained by a plurality of cameras. The image processing apparatus includes: a display control unit configured to display a display image based on a captured image of at least one camera of the plurality of cameras on a display unit; a designation unit configured to designate an area on the display image displayed by the display control unit; and a specification unit configured to specify an area on a three-dimensional shape model, which corresponds to an area designated by the designation unit on the display image, based on captured images of two or more cameras of the plurality of cameras.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to an image processing apparatus, a method, and a storage medium.

Description of the Related Art

In recent years, a technique to acquire three-dimensional shape data of an object has been developing. It is possible to acquire three-dimensional shape data of an object by, for example, a method using a three-dimensional scanner with a laser or a method of estimating three-dimensional shape data from a plurality of images obtained by capturing the same object from different viewpoints. In the case where acquired three-dimensional shape data is edited, it is necessary for a user to designate an editing range.

Japanese Patent Laid-Open No. 2003-067778 has disclosed a technique to specify a polygon (mesh) of a three-dimensional shape of a target by finding a corresponding point of a point designated by a user on a two-dimensional texture within a three-dimensional space.

However, with the conventional technique, it is not possible for a user to designate a range of a three-dimensional point group with accuracy. With the technique described in Japanese Patent Laid-Open No. 2003-067778, the accuracy of a specified three-dimensional shape is reduced because a polygon (mesh) is used. Further, in the case where a two-dimensional texture is generated from a plurality of images, inconsistency occurs at a seam, and in the case where a two-dimensional texture is generated from one image, it is not possible to specify a portion that cannot be recognized on the two-dimensional texture (so-called occlusion portion).

SUMMARY OF THE INVENTION

An image processing apparatus according to an embodiment of the present invention specifies a partial area of a three-dimensional shape model that is generated based on a plurality of captured images obtained by a plurality of cameras. The image processing apparatus comprises: a display control unit configured to display a display image based on a captured image of at least one camera of the plurality of cameras on a display unit; a designation unit configured to designate an area on the display image displayed by the display control unit; and a specification unit configured to specify an area on a three-dimensional shape model, which corresponds to an area designated by the designation unit on the display image, based on captured images of two or more cameras of the plurality of cameras.

DESCRIPTION OF THE EMBODIMENTS

In the following, embodiments of the present invention are explained in detail with reference to the attached drawings.

First Embodiment

In the present embodiment, by using a plurality of pieces of image data corresponding to a three-dimensional point group of an object and image feature points thereof (hereinafter, referred to as a two-dimensional point group), range selection of a three-dimensional point group is performed by an intuitive, simple method as in the case of range selection in a two-dimensional image. The three-dimensional point group in the present embodiment is one estimated by a method, such as SfM (Structure from Motion), from a two-dimensional point group of a plurality of images. The three-dimensional point group is not limited to this and may also be one in which the value of a depth sensor and an image are caused to correspond to each other as long as there is a correspondence between an image and a three-dimensional point group.

In the following, a case is explained where four pieces of image data obtained by capturing an object from viewpoints different from one another, but the present embodiment is not limited to four pieces of image data and can be applied to two or more pieces of image data. Further, the image data of the present embodiment is not limited to that obtained by capturing an object. That is, it is possible to apply the present embodiment to a plurality of pieces of image data representing the same object from different viewpoints.

FIG. 1shows a configuration of an image processing apparatus100according to the present embodiment. The image processing apparatus100is made up by including a CPU101, a RAM102, a ROM103, a storage unit104, an input interface105, an output interface106, and a system bus107. To the input interface105, an external memory108is connected and to the output interface106, a display device109is connected.

The CPU101is a processor that centralizedly controls each component of the image processing apparatus100. The RAM102is a memory that functions as a main memory and a work area of the CPU101. The ROM103is a memory that stores programs and the like used for processing within the image processing apparatus100. The CPU101performs various kinds of processing, to be described later, by using the RAM102as a work area and executing programs stored in the ROM103.

The storage unit104is a storage device that stores image data used for processing in the image processing apparatus100, parameters (that is, setting values) for processing, and so on. As the storage unit104, it is possible to use an HDD, an optical disk drive, a flash memory, and so on.

The input interface105is, for example, a serial bus interface, such as USB and IEEE 1394. It is possible for the image processing apparatus100to acquire processing-target image data and the like from the external memory108(for example, hard disk, memory card, CF card, SD card, USB memory) via the input interface105. The output interface106is, for example, a video output terminal, such as DVI and HDMI (registered trademark). It is possible for the image processing apparatus100to output image data processed by the image processing apparatus100to the display device109(image display device, such as a liquid crystal display) via the output interface106. The image processing apparatus100may include components other than those described above, but explanation is omitted here.

In the following, image processing in the image processing apparatus100is explained with reference toFIG. 2andFIG. 3.FIG. 2is a function block diagram of the image processing apparatus100according to the present embodiment.FIG. 3is a processing flowchart of an image processing method according to the present embodiment. In the present embodiment, by executing programs stored in the ROM103, the CPU101functions as each block described inFIG. 2and performs the processing flow inFIG. 3. The CPU101does not necessarily need to perform all the functions and it may also be possible to provide a processing circuit corresponding to each function within the image processing apparatus100.

At step301, an image data acquisition unit201acquires a plurality of pieces of image data from the external memory108via the input interface105and stores the image data in the RAM102along with an image ID (image identifier) to identify each piece of image data. Each of the plurality of pieces of image data is image data obtained by capturing the same object from viewpoints different from one another, that is, image data representing the same object from different viewpoints.

At step302, a point group information acquisition unit202acquires two-dimensional point group coordinates included in each of the plurality of pieces of image data, three-dimensional point group coordinates of the object, and correspondence information on the two-dimensional point group and the three-dimensional point group from the external memory108via the input interface105. The acquired two-dimensional point group, the three-dimensional point group, and the correspondence information are stored in the RAM102.

At step303, a selected image display unit203acquires an image ID of one piece of image data selected by a user from the plurality of pieces of image data and acquires image data corresponding to the image ID from the RAM102. That is, the selected image display unit203functions as a determination unit configured to determine image data selected by a user from a plurality of pieces of image data as image data that should be displayed. The selected image display unit203displays the acquired image data on the display device109via the output interface106.

At step304, a selected range acquisition unit204acquires a two-dimensional point group in a selected range selected by a user on the image displayed on the display device109and stores the two-dimensional point group in the RAM102. A specific acquisition method of a selected range includes, as a simple method, a method of taking an area as a selected range, in which the difference in the pixel value between a pixel within a designated area and a pixel in the designated area, which is one of neighboring pixels, is smaller than or equal to a threshold value. Further, in recent years, as a method of selecting a range with high accuracy, an area division method, called a graph cut, is proposed.

FIG. 4is a schematic diagram explaining a graph cut method. In the graph cut method, first, as shown inFIG. 4(a), a graph corresponding to an image is generated and a within-range seed pixel401and an outside-range seed pixel402are given. Next, as shown inFIG. 4(b), similarity between pixels is calculated and pixels between which similarity is high are linked by an edge403with a heavy weight and pixels between which similarity is low are linked by an edge404with a light weight. Further, a pixel similar to the within-range seed pixel401is linked to a within-range node S405with a heavy weight and a pixel similar to the outside-range seed pixel402is linked to an outside-range node T406with a heavy weight. Schematic representation of a link with a weight between pixels not similar to each other is omitted here. Finally, as shown inFIG. 4(c), the graph is cut so that the sum of nodes to be cut becomes a minimum and the side of the within-range node S405is taken to be a within-range area and the side of the outside-range node T406is taken to be an outside-range area. Specifically, the pixel in the range designated by a user with a mouse and the like is a within-range seed pixel or an outside-range seed pixel. The index of similarity between pixels includes the similarity of color between pixels, the intensity of an edge derived from the magnitude of a difference in the feature amount of an image, the distribution of color in an adjacent area, the distance between pixels, and so on.

At step305, a first three-dimensional point group range estimation unit205extracts a three-dimensional point group corresponding to a two-dimensional point group within the selected range, which is stored in the RAM102at step304, from the three-dimensional point groups acquired at step302based on the correspondence information. The first three-dimensional point group range estimation unit205stores the extracted three-dimensional point group in the RAM102as a first within-range three-dimensional point group. That is, the first three-dimensional point group range estimation unit205functions as a first three-dimensional point group extraction unit. Specifically, it may also be possible to add an attribute, such as “within range”, to the three-dimensional point group stored in the RAM102. Further, it may also be possible for the first three-dimensional point group range estimation unit205to extract a three-dimensional point group corresponding to the two-dimensional point group determined to be outside the selected range at step304based on the correspondence information and to store the three-dimensional point group in the RAM102as a first outside-range three-dimensional point group. Furthermore, it may also be possible to extract a three-dimensional point group (not seen in the display image) other than those described above based on the correspondence information and to store the three-dimensional point group in the RAM102as an unclassified three-dimensional point group. Specifically, it may also be possible to add an attribute, such as “outside range”, to the first outside-range three-dimensional point group and an attribute, such as “unclassified”, to the unclassified three-dimensional point group.

Next, at step306and step307, the processing is performed for each of the other pieces of image data other than the image data displayed at step303of the plurality of pieces of image data acquired at step301.

At step306, an image range estimation unit206extracts a two-dimensional point group corresponding to the first within-range three-dimensional point group extracted at step305based on the correspondence information for one piece of image data other than the display image data. The image range estimation unit206stores the extracted two-dimensional point group in the RAM102as a first within-range two-dimensional point group. Specifically, it may also be possible to add an attribute, such as “within range”, to the two-dimensional point group stored in the RAM102. Further, it may also be possible for the image range estimation unit206to extract two-dimensional point groups corresponding to the first outside-range three-dimensional point group and the unclassified three-dimensional point group extracted at step305based on the correspondence information and to store the two-dimensional point groups in the RAM102as a first outside-range two-dimensional point group and a first unknown two-dimensional point group.

At step307, the image range estimation unit206performs area division of an image based on the first within-range two-dimensional point group stored at step306and determines whether a point group other than the first within-range two-dimensional point group is included in the same area of the first within-range two-dimensional point group. The image range estimation unit206updates the point group other than the first within-range two-dimensional point group, which is included in the same area, and takes the point group as the first within-range two-dimensional point group (that is, the point group is added to the first within-range two-dimensional point group). Further, in the case where the first outside-range two-dimensional point group and the first unknown two-dimensional point group are used, it is sufficient to determine whether the first unknown two-dimensional point group is nearer to the first within-range two-dimensional point group or to the first outside-range two-dimensional point group. In this case, it is sufficient to add the point group of the first unknown two-dimensional point groups, which is determined to be nearer to the first within-range two-dimensional point group, to the first within-range two-dimensional point group. At this time, it may be possible to use the method explained at step304as a range estimation method of an image. For example, in the case where the graph cut method is used, it may be possible to use the first within-range two-dimensional point group before updating as the within-range seed401. Further, in the case where there is an outside-range point group, it may be possible to use the first outside-range two-dimensional point group before updating as the outside-range seed402. As described above, at steps306and307, the image range estimation unit206functions as a first two-dimensional point group acquisition unit configured to acquire the first within-range two-dimensional point group existing in the same area in which the two-dimensional point group corresponding to the first within-range three-dimensional point group is included.

At step308, the image range estimation unit206determines whether the processing has been completed for all the processing-target image data and in the case where there is unprocessed image data, the processing returns to step306and the processing is repeated. On the other hand, in the case where the processing of all the image data has been completed, the processing advances to step309. Here, it may be possible to take all the image data in which the first within-range three-dimensional point group is seen (that is, all the image data in which the two-dimensional point group corresponding to the first within-range three-dimensional point group is displayed) to be the processing-target image data.

At step309, a second three-dimensional point group range estimation unit207extracts a three-dimensional point group of the three-dimensional point groups acquired at step302, whose ratio of a point being the first within-range two-dimensional point group becomes larger than or equal to a threshold value in each piece of image data. The second three-dimensional point group range estimation unit207stores the extracted three-dimensional point group in the RAM102as a second within-range three-dimensional point group. Here, a within-range determination threshold value, which is the predetermined threshold value, is acquired from the outside via the input interface105. As a specific threshold value determination method, for example, it may be possible to perform determination by a ratio of the number of pieces of image data including the point group determined as the first within-range two-dimensional point group to the number of all the pieces of image data in which the first within-range three-dimensional point group is seen. It may also be possible to extract the second within-range three-dimensional point group by linking points in the three-dimensional point group, whose three-dimensional coordinates are near and whose similarity therebetween is high, by an edge and by finding the minimum cut by using the graph cut method explained with reference toFIG. 4. That is, the second three-dimensional point group range estimation unit207functions as a second three-dimensional point group estimation unit configured to estimate a second within-range three-dimensional point group corresponding to the selected range selected by a user on the two-dimensional image.

FIG. 5is a schematic diagram explaining image processing according to the present embodiment. In the following, with reference to the schematic diagram, the image processing according to the present embodiment is explained.

InFIG. 5, a three-dimensional point group500representing a three-dimensional shape of an object is a point group including nine points1to9and four pieces of image data501,502,503, and504obtained by capturing the object from different viewpoints are shown. It is assumed that at step303, the image data501is displayed and the range surrounded by a dotted line of the image data501(selected image) is taken to be the selected range acquired at step304. In the selected range, four points1to4(two-dimensional point group) are included and at step305, four points1to4of the three-dimensional point group500are extracted as a first within-range three-dimensional point group. In the present embodiment, an example in the case where the selected image shown inFIG. 5is the captured image itself by the camera selected from the plurality of cameras is explained mainly. However, the example is not limited to this. For example, an image obtained by performing predetermined image processing (for example, pixel thinning processing) for the captured image of the selected camera may be displayed as a selected image or a virtual viewpoint image based on the viewpoint (virtual viewpoint) designated by a user may be displayed as a selected image. Further, in the present embodiment, an example in the case where the number of selected images is one is explained mainly, but the example is not limited to this. For example, captured images (for example, the image data501and the image data503inFIG. 5) of the two or more cameras of the plurality of cameras may be displayed as selected images.

In the image data502of viewpoint1, in the case where area division is performed according to image similarity, points1to4,7, and8of the two-dimensional point groups are included in one area. Further, points1to4are the first within-range two-dimensional point group corresponding to the first within-range three-dimensional point group, and therefore, points7and8included in the same area of points1to4are also the first within-range two-dimensional point group of the image data502(steps306and307).

In the image data503of another viewpoint2, in the case where area division is performed according to image similarity, the area is divided into three areas of the two-dimensional point group: an area of points1,2,7, and8, an area of point9, and an area of point6. Points1and2are the first within-range two-dimensional point group corresponding to the first within-range three-dimensional point group, and therefore, points7and8included in the same area of points1and2are also the first within-range two-dimensional point group of the image data503(steps306and307).

In the image data504of another viewpoint3, in the case where area division is performed according to image similarity, the area is divided into four areas of the two-dimensional point group: an area of points1and4, an area of points5and6, an area of point9, and an area of point8. Points1and4are the first within-range two-dimensional point group corresponding to the first within-range three-dimensional point group and another point group is not included in the same area, and therefore, points1and4are the first within-range two-dimensional point group of the image data504(steps306and307).

The above results are put together in a table505. Of the two-dimensional point groups of each piece of image data, the point that is the first within-range two-dimensional point group is indicated by ∘ and the point that is not the first within-range two-dimensional point group is indicated by x. The empty cell in the table505indicates a point group (occlusion) of the three-dimensional point group of the object, in which the two-dimensional point group corresponding to each piece of image data is not included.

From the table505, the ratio of the point being the first within-range two-dimensional point group is 100% for points1to4and7, 66.7% for point8, and 0% for points5,6, and9. In the case where the threshold value (within-range determination threshold value) to determine as the second within-range three-dimensional point group is taken to be 50%, points1to4,7, and8are extracted as the second within-range three-dimensional point group (step309).

In the present embodiment, the ratio of the point group being the first within-range two-dimensional point group is determined, but it may also be possible to calculate an evaluation value by weighting the image similarity and the distance between a pixel in the first within-range two-dimensional point group and a pixel in the first outside-range two-dimensional point group to find an average value and to perform threshold value processing. Further, it may also be possible to perform setting so as to reduce the weight (degree of reliability) at the time of area division of a plurality of images by taking a point of the first within-range three-dimensional point group, which is located near the boundary between the inside of the selected range and the outside of the selected range in the selected image to be a boundary point.

In the present embodiment, it is possible to automatically perform range selection of a three-dimensional point group within a range neighboring and similar to the selected range based on the selected range on one selected image in this manner.

As explained above, according to the present embodiment, it is possible to estimate the range of a three-dimensional point group with high accuracy, which corresponds to the range selected by a user on the two-dimensional image. In the above-described embodiment, the four cameras are used, but the number of cameras is not limited to four. Further, in the present embodiment, the example is explained in which range selection of a three-dimensional point group is performed by using all the captured images of the plurality of cameras, but the example is not limited to this. For example, in the case where the range of points1to4of the image data501inFIG. 5is designated, it may also be possible to perform range selection of a three-dimensional point group by using the image data502and503without using the image data504. The larger the number of pieces of image data to be used for range selection of a three-dimensional point group, the heavier the processing load becomes, but the higher the accuracy of range selection becomes. Consequently, it may also be possible to determine the number of images to be used for range selection in accordance with the required accuracy.

Second Embodiment

In a second embodiment, by using the second within-range three-dimensional point group that is output in the above-described first embodiment, range selection is performed in a plurality of pieces of image data representing the same object from different viewpoints.

In the above-described first embodiment, it is premised that the two-dimensional point group corresponding to the three-dimensional point group is set correctly in each piece of image data, but there is a case where the two-dimensional point group corresponding to the three-dimensional point group is not set correctly. For example, there is a case where three-dimensionally the same area is not seen in the same way on an image because the object has gloss anisotropy or a case where the tone level that is seen in an image is not seen in another image because of a difference in dynamic range. In such a case, the correspondence between the three-dimensional point group and the two-dimensional point group is not set correctly. Consequently, in the present embodiment, even in the case where the two-dimensional point group corresponding to the three-dimensional point group is not set correctly, it is made possible to perform range selection in a plurality of pieces of image data using the second within-range three-dimensional point group with high accuracy.

In the following, with reference toFIG. 6andFIG. 7, image processing in the image processing apparatus100according to the present embodiment is explained.FIG. 6is a function block diagram of the image processing apparatus100according to the present embodiment.FIG. 7is a processing flowchart of an image processing method according to the present embodiment.

Compared to the function block diagram inFIG. 2, the function block diagram inFIG. 6further includes an occlusion determination unit608and a second image range estimation unit609. In the present embodiment, mainly by the occlusion determination unit608and the second image range estimation unit609, range selection processing in a plurality of pieces of image data using the second within-range three-dimensional point group that is output in the first embodiment is performed.

Blocks601to607inFIG. 6are the same as the blocks201to207inFIG. 2and a point group information acquisition unit602in the present embodiment further acquires camera position attitude information (hereinafter, also described as a camera position attitude) on the camera that has captured each of the plurality of pieces of image data. The camera position attitude information includes position information on the camera and the direction of the camera. The camera position attitude is only required to be a viewpoint position attitude including position information on and the direction of the viewpoint for the object represented by each piece of image data. Specifically, compared to the point group information acquisition unit202inFIG. 2, the point group information acquisition unit602further acquires the camera position attitude information corresponding to each of the plurality of pieces of image data from the external memory108via the input interface105and stores the camera position attitude information in the RAM102.

Further, in the present embodiment also, by executing programs stored in the ROM103, the CPU101functions as each block described inFIG. 6and performs the processing flow inFIG. 7. Further, the CPU101does not necessarily need to perform all the functions and it may also be possible to provide a processing circuit corresponding to each function within the image processing apparatus100.

At step701, the occlusion determination unit608acquires three-dimensional point group coordinates, two-dimensional point group coordinates, correspondence information on those, and the camera position attitude (viewpoint position attitude) corresponding to each of the plurality of pieces of image data stored in the RAM102.

At step702, the occlusion determination unit608acquires the second within-range three-dimensional point group stored in the RAM102. At this time, it may also be possible to further acquire the outside-range three-dimensional point group other than the second within-range three-dimensional point group from the three-dimensional point groups stored in the RAM102.

After this, processing at steps703to707is performed for each of the plurality of pieces of image data. Further, an image ID to identify the processing-target image data is described as a processing image ID.

At step703, the occlusion determination unit608extracts a point group of the two-dimensional point groups of the image data with a processing image ID, which has a correspondence with the second within-range three-dimensional point group, as a second within-range two-dimensional point group of the image data with a processing image ID. That is, the occlusion determination unit608functions as a second two-dimensional point group extraction unit. The occlusion determination unit608stores the extracted second within-range two-dimensional point group in the RAM102.

At this time, it may also be possible to further extract a point group of the two-dimensional point groups of the image data with a processing image ID, which has a correspondence with the outside-range three-dimensional point group, as a second outside-range two-dimensional point group in the image data with a processing image ID and to store the point group in the RAM102.

At step704, the occlusion determination unit608extracts a point group of the second within-range three-dimensional point groups, which does not have a correspondence with the two-dimensional point group of the image data with a processing image ID, as a non-corresponding within-range three-dimensional point group. That is, the occlusion determination unit608functions as a non-corresponding three-dimensional point group extraction unit. The occlusion determination unit608stores the extracted non-corresponding within-range three-dimensional point group in the RAM102. The non-corresponding within-range three-dimensional point group includes, for example, a point group that is not set correctly so as to correspond to the two-dimensional point group because of gloss anisotropy and the like in the image data with a processing image ID.

At step705, the occlusion determination unit608and the second image range estimation unit609acquire the image data with a processing image ID stored in the RAM102.

At step706, the occlusion determination unit608projects the non-corresponding within-range three-dimensional point group onto the two-dimensional image with the camera position attitude corresponding to the image data with a processing image ID and determines whether the projected point group is an occlusion. The occlusion determination unit608adds a point group that is not an occlusion to the second within-range two-dimensional point group. That is, the occlusion determination unit608functions as a second two-dimensional point group addition unit. Specifically, it is sufficient to determine that the projected point group is not an occlusion in the case where the color adjacent to the two-dimensional point group in another piece of image data, which corresponds to the non-corresponding within-range three-dimensional point group, and the color adjacent to the projected point group are similar, and that the projected point group is an occlusion in the case where the colors are different.

As described above, the occlusion determination unit608functions as a second two-dimensional point group acquisition unit configured to acquire the second within-range two-dimensional point group including the two-dimensional point group corresponding to the second within-range three-dimensional point group.

At step707, the second image range estimation unit609estimates and outputs an image range corresponding to the second within-range two-dimensional point group based on the similarity between the second within-range two-dimensional point group and the image. At this time, it may be possible to use the method explained at step304as an estimation method of an image range. For example, in the case where the graph cut method is used, it may be possible to use the second within-range two-dimensional point group as the within-range seed401. Further, in the case where there is a second outside-range two-dimensional point group, it may be possible to use the second outside-range two-dimensional point group as the outside-range seed402. As described above, the second image range estimation unit609functions as an image range estimation unit configured to estimate an image range corresponding to the second within-range two-dimensional point group.

At step708, the second image range estimation unit609determines whether the processing has been completed for all the image data and in the case where the processing has not been completed for all the image data, the processing returns to step703and the processing is repeated. On the other hand, in the case where the processing of all the image data has been completed, the processing flow is terminated.

As explained above, in the present embodiment, by using the second within-range three-dimensional point group that is output in the first embodiment, an image range in a plurality of pieces of image data representing the same object from different viewpoints is estimated and output.

FIGS. 8A and 8Bare schematic diagrams explaining image processing according to the present embodiment. In the following, with reference toFIGS. 8A and 8B, the image processing according to the present embodiment is explained.

InFIG. 8A, a three-dimensional point group800representing a three-dimensional shape of an object is a point group including nine points1to9and five pieces of image data8A01,8A02,8A03,8A04, and8A05obtained by capturing the object from different viewpoints are shown. It is assumed that at step303, the image data8A01is displayed and the range surrounded by a dotted line in the image data8A01is taken to be the range selected at step304. In the selected range, three points1,3, and4(two-dimensional point group) are included and at step305, from the three-dimensional point group800, three points1,3, and4are extracted as a first within-range three-dimensional point group.

Further, it is assumed that inFIG. 8A, point3is not recognized because of gloss anisotropy or a difference in dynamic range in the image data8A03of another viewpoint2. That is, point3is a point that is not recognized despite that it is not an occlusion (that is, point3is a point that is not set correctly).

In the image data8A02of another viewpoint1, in the case where area division is performed according to image similarity, the area is divided into two areas of the two-dimensional point group: an area of points1,3, and4and an area of points2,7, and8. Points1,3, and4are the first within-range two-dimensional point group corresponding to the first within-range three-dimensional point group and another point group is not included in the same area, and therefore, points1,3, and4are the first within-range two-dimensional point group of the image data8A02.

In the image data8A03of another viewpoint2, in the case where area division is performed according to image similarity, the area is divided into two areas of the two-dimensional point group: an area of points1and4and an area of points2,7, and8. Points1and4are the first within-range two-dimensional point group corresponding to the first within-range three-dimensional point group and another point group is not included in the same area, and therefore, points1and4are the first within-range two-dimensional point group of the image data8A03.

In the image data8A04of another viewpoint3, in the case where area division is performed according to image similarity, the area is divided into three areas of the two-dimensional point group: an area of point1, an area of points2and7to9, and an area of point6. Point1is the first within-range two-dimensional point group corresponding to the first within-range three-dimensional point group and another point group is not included in the same area, and therefore, point1is the first within-range two-dimensional point group of the image data8A04.

In the image data8A05of another viewpoint4, in the case where area division is performed according to image similarity, the area is divided into three areas of the two-dimensional point group: an area of points1and4, an area of points5and6, and an area of points8and9. Points1and4are the first within-range two-dimensional point group corresponding to the first within-range three-dimensional point group and another point group is not included in the same area, and therefore, points1and4are the first within-range two-dimensional point group of the image data8A05.

The above results are put together in a table8A06. Of the two-dimensional point groups of each piece of image data, the point that is the first within-range two-dimensional point group is indicated by ∘ and the point that is not the first within-range two-dimensional point group is indicated by x. The empty cell in the table8A06indicates a point group (occlusion) of the three-dimensional point group of the object, in which the two-dimensional point group corresponding to each piece of image data is not included.

From the table8A06, the ratio of the point being the first within-range two-dimensional point group is 100% for points1,3, and4and 0% for points2and5to9. In the case where the threshold value (within-range determination threshold value) to determine as the second three-dimensional point group is taken to be 50%, points1,3, and4are extracted as the second within-range three-dimensional point group. In the table8A06, point3of another viewpoint3, for which a correspondence is not taken correctly despite that it is not an occlusion, is represented by a cell to which slashes are attached.

Here, the portion surrounded by a dotted line in each piece of image data inFIG. 8Ais taken to be a first image range.

Next, the occlusion determination unit608performs occlusion determination of a non-corresponding within-range three-dimensional point group not having a correspondence with a two-dimensional point group in each piece of image data. As a result of this, in the image data8A03of another viewpoint2, point3is determined not to be an occlusion (step706).

Next, the second image range estimation unit609performs area division again based on the image similarity and the information on the within-range three-dimensional point group for the image in which a point group determined not to be an occlusion exists, in this example, the image data8A03of another viewpoint2(step707). As a result of this, as shown inFIG. 8B, an adjacent area of points1,3, and4shown schematically by a dotted line in the image data8A03of another viewpoint2becomes a new second image range.

In the case where there is a point for which consistency between the second within-range three-dimensional point group and the first within-range two-dimensional point group is not achieved, although such a point does not exist in this example, it is desirable to update the first image range and to take the second image range.

Specifically, in the case where there is a point that does not correspond to the second within-range three-dimensional point group but is the first within-range two-dimensional point group, or conversely, there is a point the corresponds to the second within-range three-dimensional point group but is not the first within-range two-dimensional point group, it is sufficient to find the second image range by updating information on the second within-range two-dimensional point group.

In this manner, even in the case where the two-dimensional point group corresponding to the three-dimensional point group is not set correctly, it is possible to automatically perform range selection of a plurality of pieces of image data within a range neighboring and similar to the selected range based on the selected range on one selected image.

As explained above, according to the present embodiment, by using the second within-range three-dimensional point group that is output in the above-described first embodiment, it is possible to perform range selection in a plurality of pieces of image data representing the same object from different viewpoints. Further, even in the case where the two-dimensional point group corresponding to the three-dimensional point group is not set correctly, it is possible to perform range selection with high accuracy in a plurality of pieces of image data.

Third Embodiment

In a third embodiment, processing at the time of a user selecting an image that is displayed at step303of the above-described first embodiment is explained. At step303, a user selects image data that is used for range selection from a plurality of pieces of image data, but in the case where the number of pieces of image data is large, it is difficult to select image data including the range of a desired three-dimensional point group. Because of this, in the present embodiment, a virtual viewpoint from which the range of a desired three-dimensional point group is easy to see is set and one or a plurality of pieces of image data of a viewpoint near to the virtual viewpoint is automatically selected in accordance with adjacency setting for the virtual viewpoint and presented to a user.

In the following, with reference toFIG. 9andFIG. 10, image processing in the image processing apparatus100according to the present embodiment is explained.FIG. 9is a function block diagram of the image processing apparatus100according to the present embodiment.FIG. 10is a processing flowchart of an image processing method according to the present embodiment.

Compared to the function block diagram inFIG. 2, the function block diagram inFIG. 9further includes a point group display unit903and a display viewpoint adjacent image selection unit904. Blocks901,902, and905to908are the same as the blocks201,202, and204to207inFIG. 2and the point group information acquisition unit902in the present embodiment further acquires a camera position attitude (viewpoint position attitude) of the camera that has captured each of the plurality of pieces of image data. Specifically, compared to the point group information acquisition unit202inFIG. 2, the point group information acquisition unit902further acquires the camera position attitude corresponding to each of the plurality of pieces of image data from the external memory108via the input interface105and stores the camera position attitude in the RAM102.

In the present embodiment also, by executing programs stored in the ROM103, the CPU101functions as each block described inFIG. 9and performs the processing flow inFIG. 10. Further, the CPU101does not necessarily need to perform all the functions and it may also be possible to provide a processing circuit corresponding to each function within the image processing apparatus100.

At step1001, the point group information acquisition unit902acquires the camera position attitude (viewpoint position attitude) corresponding to each of the plurality of pieces of image data from the external memory108via the input interface105and the stores the camera position attitude in the RAM102.

At step1002, the point group display unit903acquires virtual viewpoint position attitude information and adjacency setting information from the external memory108via the input interface105and stores both the information in the RAM102. The virtual viewpoint position attitude information includes position information on and the direction of the virtual viewpoint. The adjacency setting information is threshold value information about the value of the inner product of the camera position attitude and the virtual viewpoint position attitude. Further, the point group display unit903generates a display image by projecting the three-dimensional point group acquired at step302for the virtual viewpoint position attitude and displays the display image on the display device109via the output interface106.

After this, the processing at steps1003to1005is performed for each of the plurality of pieces of image data.

At step1003, the display viewpoint adjacent image selection unit904calculates the inner product of the camera position attitude acquired at step1001and the virtual viewpoint position attitude acquired at step1002and calculates the value of the inner product for the processing-target image data.

At step1004, the display viewpoint adjacent image selection unit904determines whether the value of the inner product calculated at step1003is larger than or equal to the value of adjacency setting acquired at step1002. In the case where the value of the inner product is larger than or equal to the value of adjacency setting, the processing advances to step1005and in the case where the value of the inner product is smaller than the value of adjacency setting, the processing at step1005is not performed and the processing advances to step1006.

At step1005, the display viewpoint adjacent image selection unit904stores the value of the inner product and the image ID of the processing-target image data in association with each other in the RAM102as an adjacent image list.

At step1006, the display viewpoint adjacent image selection unit904determines whether the values of the inner product have been calculated with the camera position attitudes of all the image data. In the case where the values of the inner product have been calculated with the camera position attitudes of all the image data, the processing advances to step1007and in the other case, the processing returns to step1003and the processing is repeated.

At step1007, the display viewpoint adjacent image selection unit904sorts the values of the inner product in the adjacent image list in the order from the largest value and displays the image data corresponding to the image ID on the display device109via the output interface106in this order. As described above, the display viewpoint adjacent image selection unit904functions as an image display unit configured to select and display one or a plurality of pieces of image data corresponding to a viewpoint or viewpoints adjacent to the virtual viewpoint.

FIG. 11is a schematic diagram explaining image processing according to the present embodiment. In the following, with reference toFIG. 11, the image processing according to the present embodiment is explained.

InFIG. 11, a three-dimensional point group1100representing a three-dimensional shape of an object is a point group including nine points1to9and four pieces of image data1106,1107,1108, and1109obtained by capturing the object from different viewpoints are shown.

Here, it is assumed that the value of adjacency setting for the value of the inner product of a virtual viewpoint position attitude1101acquired by the point group display unit903and each camera position attitude is 0. In this case, the camera position attitude whose inner product with the virtual viewpoint position attitude is larger than or equal to the value of adjacency setting (that is, the angle formed therebetween is 90 degrees or less) is a camera position attitude1102(formed angle1104) and a camera position attitude1103(formed angle1105). Consequently, in the adjacent image list, the image ID of the adjacent image1106with the camera position attitude1102, the image ID of the adjacent image1107with the camera position attitude1103, and the values of the inner product thereof are included in association with one another.

As a result of this, by the display viewpoint adjacent image selection unit904displaying the adjacent image1106and the adjacent image1107, it is possible for a user to select a desired image from among adjacent images with the virtual viewpoint position attitude without the need to check an image distant from the virtual viewpoint position attitude.

As explained above, according to the present embodiment, it is possible for a user to easily select image data including the range of a desired three-dimensional point group even in the case where the number of pieces of image data is large.

Fourth Embodiment

In a fourth embodiment, processing of the image processing apparatus100to generate an image that is displayed at step303of the above-described first embodiment is explained. In the above-described third embodiment, an image adjacent to the virtual viewpoint position attitude of a plurality of pieces of image data is presented to a user, but in the present embodiment, the image processing apparatus100generates an image whose viewpoint is the same as that of the virtual viewpoint position attitude and presents the image to a user.

In the following, with reference toFIG. 12andFIG. 13, image processing in the image processing apparatus100according to the present embodiment is explained.FIG. 12is a function block diagram of the image processing apparatus100according to the present embodiment.FIG. 13is a processing flowchart of an image processing method according to the present embodiment.

Compared to the function block diagram inFIG. 2, the function block diagram inFIG. 12further includes a point group display unit1203and a display viewpoint image generation/display unit1204. Blocks1201,1202, and1205to1208inFIG. 12are the same as the blocks201,202, and204to207inFIG. 2. The point group information acquisition unit1202in the present embodiment further acquires a camera position attitude (viewpoint position attitude) corresponding to each of a plurality of pieces of image data. Specifically, compared to the point group information acquisition unit202inFIG. 2, the point group information acquisition unit1202further acquires the camera position attitude corresponding to each of the plurality of pieces of image data from the external memory108via the input interface105and stores the camera position attitude in the RAM102.

In the processing flow inFIG. 13, explanation of the same processing step as the processing step inFIG. 10in the above-described third embodiment is omitted.

In the present embodiment also, by executing programs stored in the ROM103, the CPU101functions as each block described inFIG. 12and performs the processing flow inFIG. 13. Further, the CPU101does not necessarily need to perform all the functions and it may also be possible to provide a processing circuit corresponding to each function within the image processing apparatus100.

At step1303, the display viewpoint image generation/display unit1204acquires display setting from the external memory108via the input interface105and stores the display setting in the RAM102. The display setting represents a predetermined number of camera position attitudes (that is, viewpoints) from which the target point is seen. Details will be described later.

At step1304, the display viewpoint image generation/display unit1204acquires correspondence information on the three-dimensional point group and the two-dimensional point group stored in the RAM102.

After this, at steps1305to1308, the same processing as that at steps1003to1006inFIG. 10is performed and an adjacent image list is generated. The processing at steps1305to1308is the same as that at steps1003to1006inFIG. 10, and therefore, explanation is omitted.

At step1309, the display viewpoint image generation/display unit1204acquires a two-dimensional point group included in image data whose number is larger than or equal to the value of display setting from the image data corresponding to the image ID in the adjacent image list and extracts a three-dimensional point group corresponding to the acquired two-dimensional point group based on the correspondence information. The display viewpoint image generation/display unit1204sets the extracted three-dimensional point group as a display three-dimensional point group. For example, in the case of image data whose value of display setting is two or more viewpoints, the three-dimensional point group that is seen from image data with two or more viewpoints as image data whose value of display setting is larger than or equal to a predetermined number is set as a display three-dimensional point group.

At step1310, the display viewpoint image generation/display unit1204generates an image (display viewpoint image) of the three-dimensional point group projected for the virtual viewpoint position attitude and displays the display three-dimensional point group so that the display three-dimensional point group can be identified. In the display viewpoint image, the display three-dimensional point group and the other three-dimensional point groups are displayed in different display methods. For example, it is may also be possible to display the display three-dimensional point group in a dark color and the other three-dimensional point groups in a pale color, or to display only the display three-dimensional point group. As described above, the display viewpoint image generation/display unit1204functions as an image generation/display unit of a display viewpoint image.

FIG. 14is a schematic diagram explaining an image processing method according to the present embodiment. In the following, with reference toFIG. 14, the image processing method according to the present embodiment is explained.

InFIG. 14, a three-dimensional point group1400representing a three-dimensional shape of an object is a point group including nine points1to9and four pieces of image data1406,1407,1408, and1409obtained by capturing the object from different viewpoints are shown.

Here, it is assumed that the value of adjacency setting for the value of the inner product of a virtual viewpoint position attitude1401acquired by the point group display unit1203and each camera position attitude is 0. In this case, the camera position attitude whose inner product with the virtual viewpoint position attitude is larger than or equal to the value of adjacency setting (that is, the angle formed therebetween is 90 degrees or less) is a camera position attitude1402(formed angle1404) and a camera position attitude1403(formed angle1405). Consequently, in the adjacent image list, the image ID of the adjacent image1406with the camera position attitude1402, the image ID of the adjacent image1407with the camera position attitude1403, and the values of the inner product thereof are included in association with one another.

Here, in the case where the value of display setting is taken to be the camera position attitude with two or more viewpoints, the display three-dimensional point group (points displayed on two or more pieces of image data) is a point group of the three-dimensional groups, which includes points1,4,5, and6.

As a result of this, in the case where the display viewpoint image generation/display unit1204displays only the display three-dimensional point group, a display viewpoint image1410including points1,4,5, and6is displayed. Consequently, points2,3,7,8, and9that ought not to be seen in the case where the object is seen from the virtual viewpoint position attitude1401are not displayed, and therefore, selection of a point group by a user is made simple.

As explained above, according to the present embodiment, by generating and displaying an image viewed from the same viewpoint as that of the virtual viewpoint position attitude, it is possible for a user to easily perform range selection at the same viewpoint as that of the virtual viewpoint position attitude.

OTHER EMBODIMENTS

This application claims the benefit of Japanese Patent Application No. 2017-075389, filed Apr. 5, 2017, which is hereby incorporated by reference wherein in its entirety.