Patent ID: 12260604

EXAMPLE EMBODIMENTS

Example Embodiment

Hereinafter, an example embodiment will be described with reference toFIGS.1to12.

[Apparatus Configuration]

Initially, the configuration of a matching support apparatus10in this example embodiment will be described usingFIGS.1and2.FIG.1is a diagram for describing an example of the matching support apparatus.FIG.2is a diagram for describing a user interface that is used in matching support.

The matching support apparatus10shown inFIG.1is an apparatus with which a feature visible on the skin surface of the face of a person targeted for matching can be designated, according to the orientation of the face in a captured image. Also, as shown inFIG.1, the matching support apparatus10has a first display information generation unit11, a second display information generation unit12and a user interface display information generation unit13. Furthermore, as shown inFIG.1, the matching support apparatus10is connected to an image capturing apparatus20, a storage device21, an input device22and a display device30.

Of these, the first display information generation unit11generates first display information for displaying, on the screen of the display device30, a first display region for displaying a matching image that includes the face of the person targeted for matching captured using the image capturing apparatus20.

The first display region is an area for displaying a matching image41on a user interface40shown inFIG.2, for example. The matching image41is, for instance, a frame image of a still image or moving image. Note that, in the example inFIG.2, the matching image41is displayed in a window42. The configuration of the display screen is, however, not limited to that inFIG.2.

The second display information generation unit12generates second display information for displaying, on the screen of the display device30, a second display region for displaying a reference face development image generated based on three-dimensional data of a head serving as a reference.

The reference head is, for instance, a head created by CG (Computer Graphics) based on data of one or more heads measured or captured in the past. The reference head may also be created based on the head of a specific person measured or captured in the past.

The second display region is an area for displaying a reference face development image43on the user interface40shown inFIG.2, for example. The reference face development image43is a development image of a face cylindrically projected by executing UV development processing, for example, using the three-dimensional data of the reference head stored in the storage device21in advance. The creation of the development image of a face is, however, not limited to the above-described cylindrical projection. Note that, in the example inFIG.2, the reference face development image43is displayed in a window44. The configuration of the display screen is, however, not limited to that inFIG.2.

The user interface display information generation unit13generates first user interface display information for displaying, on the screen of the display device30, a first user interface to be used in an operation for enabling the user to designate, in the second display region, a feature region indicating a facial feature of the person targeted for matching visible on the skin surface of the person targeted for matching.

Features on a person's face are sites indicating features of the person that are visible on the skin surface such as moles, freckles, tattoos, birthmarks, wrinkles, dimples, scars, warts, lumps, rough skin and discolored skin patches, for example. In the example inFIG.2, there is a mole51on the left cheek of the person captured in the matching image41, and thus this mole51is a feature.

The feature region is a region corresponding to a feature on the person's face recognized by the user, with a marker that the user attaches to the reference face development image after having recognized the feature. In the example inFIG.2, the region corresponding to the marker (x) on the reference face development image43is a feature region52. Note that a region of a person's face in which there are no features may also be taken as a feature region.

In the example inFIG.2, the first user interface corresponds to the user interface40. For example, in the case where the feature region52is drawn (designated) using the user interface40and the input device22, which is a physical user interface such as a mouse, touch panel or keyboard, the drawn feature region52is added to the reference face development image43when an “add” button45displayed on the user interface40is selected.

In this way, in this example embodiment, by the user viewing the matching image41and designating the feature region52on the reference face development image43, matching support leading to specification of the person targeted for matching can be provided using the designated feature region52, even if the person in the matching image41is not facing forward. Matching support is processing for selecting a person having a feature in the same position as the person targeted for matching, using the feature region52designated on the reference face development image43.

Also, by the feature region52being designated utilizing the reference face development image43, the influence of the undulations of the face arising from a change in the face orientation can be reduced, thus enabling the user to easily designate the feature region52corresponding to the mole51on the reference face development image43, even if the apparent position of the mole51changes.

[System Configuration]

Next, the configuration of the matching support apparatus10in this example embodiment will be described more specifically usingFIG.3.FIG.3is a diagram for describing an example of a system having the matching support apparatus.

As shown inFIG.3, the system in this example embodiment has the matching support apparatus10, the image capturing apparatus20, the storage device21, the input device22and the display device30. For example, the system is conceivably a monitoring system or an authentication system. Also, the matching support apparatus10inFIG.3has a candidate face image display information generation unit14, a generation unit61, a matching unit62, a selection unit63, a detection unit64and an association unit65, in addition to the first display information generation unit11, the second display information generation unit12and the user interface display information generation unit13.

The system will now be described.

The matching support apparatus10is an information processing apparatus such as a server computer, personal computer or mobile terminal equipped with a CPU (Central Processing Unit), an FPGA (Field-Programmable Gate Array) or both thereof, for example.

The image capturing apparatus20is an apparatus for capturing an image of the face of the person targeted for matching. Specifically, the image capturing apparatus20transmits the captured image to the matching support apparatus10via a communication network. The image capturing apparatus20is an image capturing apparatus such as a camera, for example.

The storage device21stores the three-dimensional data of the reference head described above and matching information. The matching information is information in which a face development image for use in matching is associated with a feature region for use in matching. The matching-use face development image is a face development image generated based on the three-dimensional data of the head of each of a plurality of persons registered in advance. The matching-use feature region is a feature region indicating a feature visible on the skin surface of the head of each of the plurality of persons registered in advance. Note that the storage device21may also store a reference face development image, a reference face three-dimensional image or both thereof in advance.

Specifically, in the case of displaying the reference face development image in the second display region, the storage device21transmits the three-dimensional data of the reference head to the matching support apparatus10via the communication network. Note that, in the case where a reference face development image is stored, the storage device21transmits the reference face development image to the matching support apparatus10.

Also, in the case where a person to serve as a candidate is selected, the storage device21transmits the matching information in which the matching-use face development image and matching-use feature region corresponding to the selected person are associated with each other to the matching support apparatus10via the communication network.

Note that the storage device21is a storage device such as a database, for example. Also, information such as the three-dimensional data of the reference head and matching information described above may be stored separately in a plurality of storage devices. Also, the storage device21may be provided inside the matching support apparatus10or may be provided externally thereto.

The input device22is a physical user interface such as a mouse, a touch panel or a keyboard, for example. Specifically, the input device22is used by the user when providing matching support using a user interface displayed on the display device30.

The display device30acquires various display information and displays generated images and the like on the screen, based on the acquired display information. The display device30is a device that uses liquid crystals, organic EL (Electroluminescence) or CRTs (Cathode Ray Tubes), for example. Furthermore, the display device30may also include an audio output device such as a speaker. Note that the display device30may also be a printing device such as a printer.

The matching support apparatus will now be described.

The first display information generation unit11generates first display information for displaying, on the screen of the display device30, a first display region for displaying a matching image captured using the image capturing apparatus20. Specifically, the first display information generation unit11acquires an image of a person captured by the image capturing apparatus20. Then, the first display information generation unit11generates first display information for displaying, on the screen of the display device30, a matching image41such as shown inFIG.2, based on the acquired image. Thereafter, the first display information generation unit11transmits the first display information to the display device30.

Note that a configuration may be adopted in which only a frame image in which a feature is readily visible is used as the matching image. Also, a frame image in which a feature that was not visible until the orientation of the face changed is detected may be used as the matching image.

The second display information generation unit12generates second display information for displaying, on the screen of the display device30, a second display region for displaying a reference face development image, based on the three-dimensional data of the reference head. Specifically, the second display information generation unit12acquires the three-dimensional data of the reference head from the storage device21.

Then, the second display information generation unit12generates a reference face development image using the three-dimensional data of the reference head. Then, the second display information generation unit12generates second display information for displaying, on the screen of the display device30, a reference face development image43such as shown inFIG.2, based on the generated reference face development image. Thereafter, the second display information generation unit12transmits the second display information to the display device30. Note that, in the case where a reference face development image is stored in the storage device21, the second display information generation unit12may acquire the reference face development image directly from the storage device21.

The user interface display information generation unit13generates first user interface display information for displaying, on the screen of the display device30, a first user interface for enabling the user to designate a feature region in the second display region with reference to the first display region. Specifically, the user interface display information generation unit13displays a user interface40such as shown inFIG.2as the first user interface to enable the user to designate a feature region on the reference face development image.

Note that the user may designate a plurality of feature regions on the reference face development image, while viewing one matching image. Also, the user may designate one or more feature regions on the reference face development image, while viewing a plurality of matching images (frame images) in which the face is oriented differently. For example, the user may designate one or more feature regions on the reference face development image, while viewing a plurality of matching images in which the face is oriented differently. This results in feature regions being positioned accurately and enables the number of feature regions to be increased, thus improving matching accuracy.

Also, as shown inFIG.2, it is conceivable to provide an “add” button45, a “delete” button46, a “save” button47, an “extract feature” button48and an “enlarge/reduce” button49, for example. When the “add” button45is selected, the feature region52drawn on the reference face development image43can be added. When the “delete” button46is selected, the feature region52drawn on the reference face development image43can be deleted. When the “save” button47is selected, feature information (e.g., texture information, position information, size information, shape information, feature type information indicating the type of feature, etc.) relating to the designated feature region52is stored in a storage unit. When the “extract feature” button48is selected, a feature is automatically extracted from the matching image41. When the “enlarge/reduce” button49is selected, display of the matching image41or reference face development image43that is selected is enlarged or reduced. The editing functions are, however, not limited to the above-described functions.

In the case where a feature region indicating a facial feature of the person targeted for matching visible on the skin surface of the person targeted for matching is designated, using the second display region, displayed on the screen of the display device30, that is for displaying the reference face development image generated based on the three-dimensional data of the reference head, the generation unit61generates feature information relating to that feature region.

Specifically, in the case where one or more feature regions are designated on the reference face development image, using the first user interface, the generation unit61, first, generates feature information for each designated feature region. Thereafter, the generation unit61outputs the feature information to the matching unit62.

In the example inFIG.2, in the case where the feature region52is designated on the reference face development image43using the user interface40, the generation unit61generates feature information of the designated feature region52.

The matching unit62matches the feature information against matching information in which a matching-use face development image and matching-use feature region for each person registered in advance are associated with each other. Specifically, the matching unit62, first, acquires feature information from the generation unit61. Then, the matching unit62executes matching processing, with reference to the respective matching information stored in the storage device21using the acquired feature information, and calculates a matching result. Thereafter, the matching unit62associates the matching information with the calculated matching result.

The matching processing involves calculating a matching index (score) as a matching result, using a value indicating the approximateness of the position of the designated feature region to the position of the matching-use feature region, or a deviation (distance) between the position of the designated feature region and the position of the matching-use feature region, or a combination thereof, for example. Furthermore, in the case where a plurality of feature regions are designated, the matching index may be calculated using the interpositional relationship between the plurality of feature regions and the interpositional relationship between the plurality of matching-use feature regions.

The Euclidean distance between the two position coordinates of the designated feature region and the matching-use feature region, the similarity obtained through normalized correlation of texture information of the two feature regions, or the overlapping area of the two feature regions, for example, can be used as the matching index.

The selection unit63selects a person to serve as a candidate based on a matching result. Specifically, the selection unit63selects matching information whose matching index is greater than or equal to a threshold value set in advance. Then, the selection unit63outputs the selected matching information to the candidate face image display information generation unit14.

In the case where the feature region designated using the first user interface is matched against matching information in which a matching-use face development image and matching-use feature region for each person registered in advance are associated with each other, and a candidate person is selected based on a matching result, the candidate face image display information generation unit14generates candidate face image display information for displaying, on the screen of the display device30, the matching-use face development image corresponding to the selected candidate person together with the matching-use feature region.

A specific description will now be given usingFIG.4.FIG.4is a diagram for describing an example of candidate face image display.

In the case where a candidate person is selected by the selection unit63based on a matching result of the matching unit62, the candidate face image display information generation unit14, first, acquires, from the storage device21, the matching information in which the matching-use face development image and matching-use feature region corresponding to the selected candidate person are associated with each other.

Then, the candidate face image display information generation unit14generates candidate face image display information for displaying, on the screen of the display device30, a matching-use face development image71and matching-use feature region73such as shown in a window72inFIG.4, based on the acquired matching information. Thereafter, the candidate face image display information generation unit14transmits the candidate face image display information to the display device30.

Note that, in the case where there are a plurality of candidate persons, the matching-use face development images are displayed in order, according to the matching results. For example, the matching-use face development images are displayed in descending order of matching indices (scores) indicated by the matching results.

The detection unit64automatically detects a feature region from an image, displayed on the screen of the display device30, that includes the face of the person targeted for matching. Specifically, the detection unit64automatically detects a feature region corresponding to a feature on the person's face (e.g., feature of the person visible on the skin surface such as a mole, freckles, tattoo, birthmark, wrinkles, dimple, scar, wart, lump, skin roughness, discolored skin patch, etc.), using a matching image. Use of a technique such as segmentation processing is conceivable for detecting a feature region.

The association unit65may automatically associate the position of the detected feature region with a corresponding position on the reference face development image. Association may involve automatically associating the position of the detected feature region with a corresponding position on the reference face development image on the basis of the relative positional relationship between parts of the face such as the eyes, nose and mouth, for example.

[Apparatus Operations]

Next, operations of the matching support apparatus in the example embodiment will be described usingFIG.5.FIG.5is a diagram for describing an example of operations of the matching support apparatus. In the following description,FIGS.1to4will be referred to as appropriate. Also, in this example embodiment, a matching support method is implemented by operating the matching support apparatus. Therefore, a description of the matching support method in this example embodiment is replaced by the following description of operations of the matching support apparatus.

As shown inFIG.5, initially, the first display information generation unit11generates first display information for displaying, on the screen of the display device30, a first display region for displaying a matching image captured using the image capturing apparatus20(step A1). Specifically, in step A1, the first display information generation unit11acquires an image of a person captured by the image capturing apparatus20.

Then, in step A1, the first display information generation unit11generates first display information for displaying, on the screen of the display device30, a matching image41such as shown inFIG.2, based on the acquired image. Thereafter, in step A1, the first display information generation unit11transmits the first display information to the display device30.

Note that a configuration may be adopted in which only a frame image in which a feature is readily visible is used as the matching image. Also, a frame image in which a feature that was not visible until the orientation of the face changed is detected may be used as the matching image.

Also, the second display information generation unit12generates second display information for displaying, on the screen of the display device30, a second display region for displaying a reference face development image, based on the three-dimensional data of the reference head (step A2). Specifically, in step A2, the second display information generation unit12acquires the three-dimensional data of the reference head from the storage device21.

Then, in step A2, the second display information generation unit12generates a reference face development image using the three-dimensional data of the reference head. Then, in step A2, the second display information generation unit12generates second display information for displaying, on the screen of the display device30, a reference face development image43such as shown inFIG.2, based on the generated reference face development image. Thereafter, in step A2, the second display information generation unit12transmits the second display information to the display device30.

Note that, in the case where a reference face development image is stored in the storage device21, the second display information generation unit12may acquire the reference face development image directly from the storage device21.

The order of the above-described processing of A1and processing of step A2may be reversed or the respective processing may be executed in parallel.

Next, the user interface display information generation unit13generates first user interface display information for displaying, on the screen of the display device30, a first user interface for enabling the user to designated a feature region in the second display region with reference to the first display region (step A3).

Specifically, in step A3, the user interface display information generation unit13displays a user interface40such as shown inFIG.2as the first user interface to enable the user to designate a feature region on the reference face development image.

Note that the user may designate a plurality of feature regions on the reference face development image, while viewing one matching image. Also, the user may designate one or more feature regions on the reference face development image, while viewing a plurality of matching images (frame images) in which the face is oriented differently. For example, the user may designate one or more feature regions on the reference face development image, while viewing a plurality of matching images in which the face is oriented differently. This results in feature regions being positioned accurately, and enables the number of feature regions to be increased.

Furthermore, designation of a feature region may be performed automatically. In that case, the detection unit64automatically detects a feature region from an image, displayed on the screen of the display device30, that includes the face of the person targeted for matching, and the association unit65automatically associates the position of the detected feature region with a corresponding position on the reference face development image.

Next, in the case where a feature region indicating a facial feature of the person targeted for matching visible on the skin surface of the person targeted for matching is designated, using the second display region, displayed on the screen of the display device30, that is for displaying the reference face development image generated based on the three-dimensional data of the reference head, the generation unit61generates feature information relating to that feature region (step A4).

Specifically, in the case where one or more feature regions are designated on the reference face development image using the first user interface, the generation unit61, in step A4, first, generates feature information for each designated feature region. Thereafter, in step A4, the generation unit61outputs the feature information to the matching unit62.

In the example inFIG.2, in the case where the feature region52is designated on the reference face development image43using the user interface40, the generation unit61generates feature information of the designated feature region52.

The matching unit62matches the feature information against matching information in which a matching-use face development image and matching-use feature region for each person registered in advance are associated with each other (step A5). Specifically, in step A5, the matching unit62, first, acquires feature information from the generation unit61. Then, in step A5, the matching unit62executes matching processing with reference to respective matching information stored in the storage device21using the acquired feature information, and calculates a matching result. Thereafter, in step A5, the matching unit62associates the matching information with the calculated matching result.

The selection unit63selects a person to serve as a candidate based on a matching result (step A6). Specifically, in step A6, the selection unit63selects matching information whose matching index is greater than or equal to a threshold value set in advance. Then, in step A6, the selection unit63outputs the selected matching information to the candidate face image display information generation unit14.

In the case where the feature region designated using the first user interface is matched against matching information in which a matching-use face development image and matching-use feature region for each person registered in advance are associated with each other, and a candidate person is selected based on a matching result, the candidate face image display information generation unit14generates candidate face image display information for displaying, on the screen of the display device30, the matching-use face development image corresponding to the selected candidate person together with the matching-use feature region (step A7).

Specifically, in the case where a candidate person is selected by the selection unit63based on a matching result of the matching unit62, the candidate face image display information generation unit14, in step A7, first, acquires, from the storage device21, the matching information in which the matching-use face development image and matching-use feature region corresponding to the selected candidate person are associated with each other.

Then, in step A7, the candidate face image display information generation unit14generates candidate face image display information for displaying, on the screen of the display device30, a matching-use face development image71and matching-use feature region73such as shown in window72inFIG.4, based on the acquired matching information. Thereafter, the candidate face image display information generation unit14transmits the candidate face image display information to the display device30.

Note that, in the case where there are a plurality of candidate persons, the matching-use face development images are displayed in order, according to the matching results. For example, the matching-use face development images are displayed in descending order of matching indices (scores) indicated by the matching results.

Next, the display device30acquires the candidate face image display information and displays a matching-use face development image71and matching-use feature region73such as shown in window72inFIG.4on the screen (step A8).

Effects of Example Embodiment

According to this example embodiment as described above, by the user viewing a matching image and designating a feature region on a reference face development image, matching support leading to specification of the person targeted for matching can be provided using the designated feature region, even if the person in the matching image is not facing forward.

Also, by the feature region being designated utilizing the reference face development image, the influence of the undulations of the face arising from a change in the face orientation can be reduced, thus enabling the feature region corresponding to the feature to be easily designated on the reference face development image by the user, even if the apparent position of the feature changes.

Also, a matching apparatus that extracts an image corresponding to a person from a frame image captured by an image capturing apparatus and executes matching using the extracted image may be coordinated with the matching support apparatus. In that case, if an image corresponding to the feature is detected from any of the frame images, the processing may be switched from a matching processing mode for performing matching using the matching apparatus to a matching support processing mode for providing matching support using the matching support apparatus.

Also, if it is judged that the person in the captured image is the same as a person registered in advance, the matching-use face development image and matching-use feature region corresponding to the person in the captured image may be edited, based on the feature of the person.

Also, if it is judged that the person in the captured image is not the same as a person registered in advance, the matching-use face development image and matching-use feature region corresponding to the person in the captured image may be edited, based on the feature showing the difference between the person in the captured image and the person registered in advance.

[Program]

A program in the example embodiment of the invention need only be a program for causing a computer to execute the processing from step A1shown inFIG.5. The matching support apparatus and matching support method of this example embodiment can be realized, by this program being installed on a computer and executed. In this case, a processor of the computer functions and performs processing as the first display information generation unit11, the second display information generation unit12, the user interface display information generation unit13, the candidate face image display information generation unit14, the generation unit61, the matching unit62, the selection unit63, the detection unit64and the association unit65.

Also, the program in this example embodiment may be executed by a computer system constructed from a plurality of computers. In this case, for example, the computers may each function as one of the first display information generation unit11, the second display information generation unit12, the user interface display information generation unit13, the candidate face image display information generation unit14, the generation unit61, the matching unit62, the selection unit63, the detection unit64and the association unit65.

Example Variation

Hereinafter, an example variation will be described with reference toFIGS.6to11.

[System Configuration]

The example variation will be described usingFIG.6.FIG.6is a diagram for describing an example of a system having the matching support apparatus.

As shown inFIG.6, the matching support apparatus10in the example variation has a third display information generation unit15and a conversion unit66, in addition to the first display information generation unit11, the second display information generation unit12, the user interface display information generation unit13, the candidate face image display information generation unit14, the generation unit61, the matching unit62, the selection unit63, the detection unit64and the association unit65.

In the example variation, in the case of further displaying a reference face three-dimensional image in a third display region, the storage device21transmits the three-dimensional data of the reference head to the matching support apparatus10, via the communication network. The third display region will be described later. Note that, in the case where a reference face three-dimensional image is stored, the storage device21transmits the the reference face three-dimensional image to the matching support apparatus10.

The third display information generation unit15generates third display information for displaying, on the screen of the display device30, a third display region in which the face in a reference face three-dimensional image generated based on the three-dimensional data of the reference head is displayed to be oriented in alignment with the orientation of the face of the person targeted for matching that is displayed in the first display region. Specifically, the third display information generation unit15acquires the three-dimensional data of the reference head from the storage device21.

Then, the third display information generation unit15generates a reference face three-dimensional image using the three-dimensional data of the reference head. Then, the third display information generation unit15generates third display information for displaying, on the screen of the display device30, a reference face three-dimensional image81such as shown inFIG.7, based on the generated reference face three-dimensional image. Thereafter, the third display information generation unit15transmits the third display information to the display device30. Note that, in the case where a reference face three-dimensional image is stored in the storage device21, the third display information generation unit15may acquire the reference face three-dimensional image directly from the storage device21.

Note that the orientation of the face may be manually aligned by the user using the user interface, or may be automatically aligned.

In the example variation, the user interface display information generation unit13generates second user interface display information for displaying, on the screen of the display device30, a second user interface to be used in an operation for enabling the user to designate a feature region in the third display region.

A specific description will now be given usingFIG.7.FIG.7is a diagram for describing a user interface that is used in matching support.

The user interface display information generation unit13displays a user interface80such as shown inFIG.7as the second user interface to enable the user to designate a feature region on the reference face three-dimensional image. Note that, in the example inFIG.7, the reference face three-dimensional image81is displayed in a window82. The configuration of the display screen is, however, not limited to that inFIG.7.

Note that the user may designate a plurality of feature regions on the reference face three-dimensional image, while viewing one matching image. Also, the user may designate one or more feature regions on the reference face three-dimensional image, while viewing a plurality of matching images (frame images) in which the face is oriented differently. For example, the user may designate one or more feature regions on the reference face three-dimensional image, while viewing a plurality of matching images in which the face is oriented differently. This results in feature regions being positioned accurately, and enables the number of feature regions to be increased.

Also, in the example variation, for example, when the “add” button45is selected after the user has selected the reference face three-dimensional image81shown inFIG.7, a feature region83drawn on the reference face three-dimensional image81can be added. When the “delete” button46is selected, the feature region83drawn on the reference face three-dimensional image81can be deleted. When the “save” button47is selected, feature information (e.g., texture information, position information, feature type information, etc.) relating to the designated feature region83is stored in a storage unit. When the “extract feature” button48is selected, a feature is automatically extracted from the matching image41. When the “enlarge/reduce” button49is selected, display of the matching image41or reference face development image43or reference face three-dimensional image81that is selected is enlarged or reduced. The editing functions are, however, not limited to the above-described functions.

In the example variation, in the case where a feature region indicating a facial feature of the person targeted for matching visible on the skin surface of the person targeted for matching is designated, using a third display region, displayed on the screen of the display device30, that is for displaying a reference face three-dimensional image generated based on the three-dimensional data of the reference head, the generation unit61generates feature information relating to that feature region.

Specifically, in the case where one or more feature regions are designated on the reference face three-dimensional image using the second user interface, the generation unit61, first, generates feature information for each designated feature region. Thereafter, the generation unit61outputs the feature information to the matching unit62. In the example inFIG.7, in the case where the feature region83is designated on the reference face three-dimensional image81using the user interface80, the generation unit61generates feature information of the designated feature region83.

In the case where a feature region is designated using the reference face three-dimensional image display region in which the face in the reference face three-dimensional image generated based on the three-dimensional data of the reference head is displayed to be oriented in alignment with the orientation of the face of the person targeted for matching, the conversion unit66converts the feature information relating to the feature region designated on the reference face three-dimensional image to feature information to be used with the reference face development image.

A description of the matching unit62and the selection unit63is given above and will thus be omitted here.

A specific description will now be given usingFIG.8.FIG.8is a diagram for describing an example of display in which a feature region on a reference face three-dimensional image is converted to a feature region on a face image in a reference face development image.

InFIG.8, when the feature region83is added, a feature region86corresponding to the feature region83is also added to a reference face development image87. Note that, in the example inFIG.8, the reference face development image87is displayed in a window85. The configuration of the display screen is, however, not limited to that inFIG.8.

In the example variation, in the case where the feature region designated using the second user interface is matched against matching information in which a matching-use face development image and matching-use feature region for each person registered in advance are associated with each other, and a candidate person is selected based on a matching result, the candidate face image display information generation unit14generates candidate face image display information for displaying, on the screen of the display device30, the matching-use face development image corresponding to the selected candidate together with the feature region.

A specific description will now be given usingFIG.9.FIG.9is a diagram for describing an example of candidate face image display.

In the case where a candidate person is selected by the selection unit63based on a matching result of the matching unit62, the candidate face image display information generation unit14, first, acquires, from the storage device21, the matching information in which the matching-use face development image and matching-use feature region corresponding to the selected candidate person are associated with each other.

Then, the candidate face image display information generation unit14generates candidate face image display information for displaying, on the screen of the display device30, a matching-use face development image91and matching-use feature region93such as shown in a window92inFIG.9, based on the acquired matching information. Thereafter, the candidate face image display information generation unit14transmits the candidate face image display information to the display device30.

Note that, in the case where there are a plurality of candidate persons, the matching-use face development images are displayed in order, according to the matching results. For example, the matching-use face development images are displayed in descending order of matching indices (scores) indicated by the matching results.

Also, in the example variation, the candidate face image display information generation unit14generates candidate face three-dimensional image display information for displaying, on the screen of the display device30, the matching-use face three-dimensional image corresponding to the candidate person, based on the three-dimensional data of the head of the selected candidate person.

A specific description will now be given usingFIG.10.FIG.10is a diagram for describing an example of candidate face image display.

In the case where a candidate person is selected by the selection unit63based on a matching result of the matching unit62, the candidate face image display information generation unit14, first, acquires, from the storage device21, the matching information in which the matching-use face three-dimensional image and matching-use feature region corresponding to the selected candidate person are associated with each other.

Then, the candidate face image display information generation unit14generates candidate face image display information for displaying, on the screen of the display device30, a matching-use face three-dimensional image91′ and matching-use feature region93′ such as shown in a window92′ inFIG.10, based on the acquired matching information. Thereafter, the candidate face image display information generation unit14transmits the candidate face image display information to the display device30.

Note that one or more of the above-described matching image, reference face development image, reference face three-dimensional image, matching-use face development image including a matching-use feature region and matching-use face three-dimensional image including a matching-use feature region may be combined for display on the user interface.

[Apparatus Operations]

Operations of the example variation will be described usingFIG.11.FIG.11is a diagram for describing an example of operations of the matching support apparatus. In the following description,FIGS.1to10will be referred to as appropriate. Also, in the example variation, a matching support method is implemented by operating the matching support apparatus. Therefore, the description of the matching support method in the example variation is replaced by the following description of the operations of the matching support apparatus.

As shown inFIG.11, initially, the processing of step A1described above is executed.

The third display information generation unit15generates third display information for displaying, on the screen of the display device30, a third display region in which the face in a reference face three-dimensional image generated based on the three-dimensional data of the reference head is displayed to be oriented in alignment with the orientation of the face of the person targeted for matching that is displayed in the first display region (step B1). Specifically, the third display information generation unit15acquires the three-dimensional data of the reference head from the storage device21.

Then, in step B1, the third display information generation unit15generates a reference face three-dimensional image using the three-dimensional data of the reference head. Then, in step B1, the third display information generation unit15generates third display information for displaying, on the screen of the display device30, a reference face three-dimensional image81such as shown inFIG.7, based on the generated reference face three-dimensional image. Thereafter, in step B1, the third display information generation unit15transmits the third display information to the display device30.

Note that, in the case where a reference face three-dimensional image is stored in the storage device21, the third display information generation unit15may acquire the reference face three-dimensional image directly from the storage device21.

Note that the orientation of the face may be manually aligned by the user, or may be automatically aligned.

The order of the processing of step A1and the processing of step B1described above may be reversed, or the respective processing may be executed in parallel.

Next, the user interface display information generation unit13generates second user interface display information for displaying, on the screen of the display device30, a second user interface to be used in an operation for enabling the user to designate a feature region in the third display region (step B2).

Specifically, in step B2, the user interface display information generation unit13displays a user interface80such as shown inFIG.7as the second user interface to enable the user to designate a feature region on the reference face three-dimensional image. Note that, in the example inFIG.7, the reference face three-dimensional image81is displayed in the window82. The configuration of the display screen is, however, not limited to that inFIG.7.

Note that the user may designate a plurality of feature regions on the reference face three-dimensional image, while viewing one matching image. Also, the user may designate one or more feature regions on the reference face three-dimensional image, while viewing a plurality of matching images (frame images) in which the face is oriented differently. For example, the user may designate one or more feature regions on the reference face three-dimensional image, while viewing a plurality of matching images in which the face is oriented differently. This results in feature regions being positioned accurately and enables the number of feature regions to be increased, thus improving matching accuracy.

Furthermore, designation of a feature region may be performed automatically. In that case, the detection unit64automatically detects a feature region from an image, displayed on the screen of the display device30, that includes the face of the person targeted for matching, and the association unit65automatically associates the position of the detected feature region with a corresponding position on the reference face three-dimensional image.

Next, in the case where a feature region indicating a facial feature of the person targeted for matching visible on the skin surface of the person targeted for matching is designated, using a third display region, displayed on the screen of the display device30, that is for displaying the reference face three-dimensional image generated based on the three-dimensional data of the reference head, the generation unit61generates feature information relating to that feature region (step B3).

Specifically, in the case where one or more feature regions are designated on the reference face three-dimensional image, using the second user interface, the generation unit61, in step B3, first, generates feature information for each designated feature region. Thereafter, in step B3, the generation unit61outputs the feature information to the matching unit62. In the example inFIG.7, in the case where the feature region83is designated on the reference face three-dimensional image81using the user interface80, the generation unit61generates feature information of the designated feature region83.

Next, in the case where a feature region is designated using the reference face three-dimensional image display region in which the face in the reference face three-dimensional image generated based on the three-dimensional data of the reference head is displayed to be oriented in alignment with the orientation of the face of the person targeted for matching, the conversion unit66converts the feature information relating to the feature region designated on the reference face three-dimensional image to feature information to be used with the reference face development image (step B4).

Next, the processing of steps A5and A6described above is executed.

Next, in the case where the feature region designated using the second user interface is matched against matching information in which a matching-use face development image and matching-use feature region for each person registered in advance are associated with each other, and a candidate person is selected based on a matching result, the candidate face image display information generation unit14generates candidate face image display information for displaying, on the screen of the display device30, the matching-use face development image corresponding to the selected candidate together with the feature region (step B5).

In the case where a candidate person is selected by the selection unit63based on a matching result of the matching unit62, the candidate face image display information generation unit14, in step B5, first, acquires, from the storage device21, the matching information in which the matching-use face development image and matching-use feature region corresponding to the selected candidate person are associated with each other.

Then, in step B5, the candidate face image display information generation unit14generates candidate face image display information for displaying, on the screen of the display device30, a matching-use face development image91and matching-use feature region93such as shown in the window92inFIG.9, based on the acquired matching information. Thereafter, the candidate face image display information generation unit14transmits the candidate face image display information to the display device30.

Note that, in the case where there are a plurality of candidate persons, the matching-use face development images are displayed in order, according to the matching results. For example, the matching-use face development images are displayed in descending order of matching indices (scores) indicated by the matching results.

Also, in step B5, the candidate face image display information generation unit14may generate candidate face three-dimensional image display information for displaying, on the screen of the display device30, the matching-use face three-dimensional image corresponding to the candidate person, based on the three-dimensional data of the head of the selected candidate person.

In the case where a candidate person is selected by the selection unit63based on a matching result of the matching unit62, the candidate face image display information generation unit14, in step B5, first, acquires, from the storage device21, the matching information in which the matching-use face three-dimensional image and matching-use feature region corresponding to the selected candidate person are associated with each other.

Then, in step B5, the candidate face image display information generation unit14generates candidate face image display information for displaying, on the screen of the display device30, a matching-use face three-dimensional image91′ and matching-use feature region93′ such as shown in the window92′ inFIG.10, based on the acquired matching information. Thereafter, the candidate face image display information generation unit14transmits the candidate face image display information to the display device30.

Note that, in the case where there are a plurality of candidate persons, the matching-use face development images are displayed in order, according to the matching results. For example, the matching-use face development images are displayed in descending order of matching indices (scores) indicated by the matching results.

Next, the display device30acquires the candidate face image display information and displays a matching-use face development image91and matching-use feature region93such as shown in the window92inFIG.9on the screen (step B6). Alternatively, in step B6, the display device30may display a matching-use face development image91′ and matching-use feature region93′ such as shown in the window92′ inFIG.10on the screen.

Effects of Variation

According to the example variation as described above, by the user viewing a matching image and designating a feature region on a reference face three-dimensional image, matching support leading to specification of the person targeted for matching can be provided using the designated feature region, even if the person in the matching image is not facing forward.

Also, by the feature region being designated utilizing the reference face three-dimensional image, the influence of the undulations of the face arising from a change in the face orientation can be reduced, thus enabling the feature region corresponding to the feature to be easily designated on the reference face three-dimensional image by the user, even if the apparent position of the feature changes.

Also, a matching apparatus that extracts an image corresponding to a person from a frame image captured by an image capturing apparatus and executes matching using the extracted image may be coordinated with the matching support apparatus. In that case, if an image corresponding to the feature is detected from any of the frame images, the processing may be switched from a matching processing mode for performing matching using the matching apparatus that is currently set to a matching support processing mode for providing matching support using the matching support apparatus.

Also, if it is judged that the person in the captured image is the same as a person registered in advance, the matching-use face development image and matching-use feature region corresponding to the person in the captured image may be edited, based on the feature of the person.

Also, if it is judged that the person in the captured image is not the same as a person registered in advance, the matching-use face development image and matching-use feature region corresponding to the person in the captured image may be edited, based on the feature showing the difference between the person in the captured image and the person registered in advance.

[Program]

A program in the example variation need only be a program for causing a computer to execute the processing from step A1shown inFIG.11. The matching support apparatus and matching support method of this example variation can be realized, by this program being installed on a computer and executed. In this case, a processor of the computer functions and performs processing as the first display information generation unit11, the second display information generation unit12, the user interface display information generation unit13, the candidate face image display information generation unit14, the third display information generation unit15, the generation unit61, the matching unit62, the selection unit63, the detection unit64, the association unit65and the conversion unit66.

Also, the program in this example variation may be executed by a computer system constructed from a plurality of computers. In this case, for example, the computers may each function as one of the first display information generation unit11, the second display information generation unit12, the user interface display information generation unit13, the candidate face image display information generation unit14, the third display information generation unit15, the generation unit61, the matching unit62, the selection unit63, the detection unit64, the association unit65and the conversion unit66.

[Physical Configuration]

Here, a computer that realizes the matching support apparatus by executing programs of the example embodiment and variation will be described usingFIG.12.FIG.12is a block diagram showing an example of a computer that realizes the matching support apparatus of the example embodiment and variation.

As shown inFIG.12, a computer110includes a CPU111, a main memory112, a storage device113, an input interface114, a display controller115, a data reader/writer116and a communication interface117. These constituent elements are connected to each other in a data communicable manner via a bus121. Note that the computer110may also include a GPU (Graphics Processing Unit) or FPGA, in addition to the CPU111or instead of the CPU111.

The CPU111carries out various computational operations by extracting programs (code) of the example embodiment that are stored in the storage device113to the main memory112and executing these programs in predetermined order. The main memory112is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory). Also, programs of the example embodiment are provided in a state of being stored in a computer-readable recording medium120. Note that programs of the example embodiment may also be distributed over the Internet connected via the communication interface117. Note that the recording medium120is a nonvolatile storage medium.

Also, specific examples of the storage device113include a semiconductor storage device such as a flash memory, in addition to a hard disk drive. The input interface114mediates data transmission between the CPU111and an input device118such as a keyboard and a mouse. The display controller115is connected to a display device119and controls display on the display device119.

The data reader/writer116mediates data transmission between the CPU111and the recording medium120, and executes readout of programs from the recording medium120and writing of processing results of the computer110to the recording medium120. The communication interface117mediates data transmission between the CPU111and other computers.

Also, specific examples of the recording medium120include a general-purpose semiconductor storage device such as a CF (Compact Flash (registered trademark) card or SD (Secure Digital) card, a magnetic recording medium such as a flexible disk, and an optical recording medium such as a CD-ROM (Compact Disk Read Only Memory).

Note that the matching support apparatus in the example embodiment can also be realized by using hardware corresponding to the respective constituent elements, rather than by a computer on which programs are installed. Furthermore, the matching support apparatus may be partially realized by programs and the remaining portion thereof may be realized by hardware.

The following supplementary notes will be further disclosed in relation to the above example embodiment. The example embodiment described above can be partially or wholly realized by supplementary notes 1 to 18 described below, but the invention is not limited to the following description.

Supplementary Note 1

A matching support apparatus including:a first display information generation unit configured to generate first display information for displaying, on a screen of a display device, a first display region for displaying an image including a face of a person targeted for matching captured using an image capturing apparatus;a second display information generation unit configured to generate second display information for displaying, on the screen of the display device, a second display region for displaying a reference face development image generated based on three-dimensional data of a head serving as a reference; anda user interface display information generation unit configured to generate first user interface display information for displaying, on the screen of the display device, a first user interface to be used in an operation for enabling a user to designate, in the second display region, a feature region indicating a facial feature of the person targeted for matching visible on a skin surface of the person targeted for matching.

Supplementary Note 2

The matching support apparatus according to supplementary note 1, including:a candidate face image display information generation unit configured to, in a case where the feature region designated using the first user interface is matched against matching information in which a matching-use face development image and matching-use feature region for each person registered in advance are associated with each other, and a person to serve as a candidate is selected based on a matching result, generate candidate face image display information for displaying, on the screen of the display device, the matching-use face development image corresponding to the selected candidate person together with the matching-use feature region.

Supplementary Note 3

The matching support apparatus according to supplementary note 2, including:a third display information generation unit configured to generate third display information for displaying, on the screen of the display device, a third display region in which a face in a reference face three-dimensional image generated based on the three-dimensional data of the reference head is displayed to be oriented in alignment with an orientation of the face of the person targeted for matching displayed in the first display region.

Supplementary Note 4

The matching support apparatus according to supplementary note 3,whereby the user interface display information generation unit generates second user interface display information for displaying, on the screen of the display device, a second user interface to be used in an operation for enabling the user to designate the feature region in the third display region.

Supplementary Note 5

The matching support apparatus according to supplementary note 4,whereby the candidate face image display information generation unit, in a case where the feature region designated using the second user interface is matched against matching information in which a matching-use face development image and matching-use feature region for each person registered in advance are associated with each other, and a person to serve as a candidate is selected based on a matching result, generates candidate face image display information for displaying, on the screen of the display device, the matching-use face development image corresponding to the selected candidate person together with the feature region.

Supplementary Note 6

The matching support apparatus according to any one of supplementary notes 2 to 5, whereby the candidate face image display information generation unit generates candidate face three-dimensional image display information for displaying, on the screen of the display device, a matching-use face three-dimensional image corresponding to the selected candidate person, based on three-dimensional data of a head of the selected candidate person.

Supplementary Note 7

A matching support method including:(a) a step of generating first display information for displaying, on a screen of a display device, a first display region for displaying an image including a face of a person targeted for matching captured using an image capturing apparatus;(b) a step of generating second display information for displaying, on the screen of the display device, a second display region for displaying a reference face development image generated based on three-dimensional data of a head serving as a reference; and(c) a step of generating first user interface display information for displaying, on the screen of the display device, a first user interface to be used in an operation for enabling a user to designate, in the second display region, a feature region indicating a facial feature of the person targeted for matching visible on a skin surface of the person targeted for matching.

Supplementary Note 8

The matching support method according to supplementary note 7, including:(d) a step of, in a case where the feature region designated using the first user interface is matched against matching information in which a matching-use face development image and matching-use feature region for each person registered in advance are associated with each other, and a person to serve as a candidate is selected based on a matching result, generating candidate face image display information for displaying, on the screen of the display device, the matching-use face development image corresponding to the selected candidate person together with the matching-use feature region.

Supplementary Note 9

The matching support method according to supplementary note 8, including:(e) a step of generating third display information for displaying, on the screen of the display device, a third display region in which a face in a reference face three-dimensional image generated based on the three-dimensional data of the reference head is displayed to be oriented in alignment with an orientation of the face of the person targeted for matching displayed in the first display region.

Supplementary Note 10

The matching support method according to supplementary note 9, including:(f) a step of generating second user interface display information for displaying, on the screen of the display device, a second user interface to be used in an operation for enabling the user to designate the feature region in the third display region.

Supplementary Note 11

The matching support method according to supplementary note 10, including:(g) a step of, in a case where the feature region designated using the second user interface is matched against matching information in which a matching-use face development image and matching-use feature region for each person registered in advance are associated with each other, and a person to serve as a candidate is selected based on a matching result, generating candidate face image display information for displaying, on the screen of the display device, the matching-use face development image corresponding to the selected candidate person together with the feature region.

Supplementary Note 12

The matching support method according to any one of supplementary notes 8 to 11, including:(h) a step of generating candidate face three-dimensional image display information for displaying, on the screen of the display device, a matching-use face three-dimensional image corresponding to the selected candidate person, based on three-dimensional data of a head of the selected candidate person.

Supplementary Note 13

A computer-readable recording medium that includes a program recorded thereon, the program including instructions that cause a computer to carry out:(a) a step of generating first display information for displaying, on a screen of a display device, a first display region for displaying an image including a face of a person targeted for matching captured using an image capturing apparatus;(b) a step of generating second display information for displaying, on the screen of the display device, a second display region for displaying a reference face development image generated based on three-dimensional data of a head serving as a reference; and(c) a step of generating first user interface display information for displaying, on the screen of the display device, a first user interface to be used in an operation for enabling a user to designate, in the second display region, a feature region indicating a facial feature of the person targeted for matching visible on a skin surface of the person targeted for matching.

Supplementary Note 14

The computer-readable recording medium according to supplementary note 13, the program further including instructions that cause the computer to carry out:(d) a step of, in a case where the feature region designated using the first user interface is matched against matching information in which a matching-use face development image and matching-use feature region for each person registered in advance are associated with each other, and a person to serve as a candidate is selected based on a matching result, generating candidate face image display information for displaying, on the screen of the display device, the matching-use face development image corresponding to the selected candidate person together with the matching-use feature region.

Supplementary Note 15

The computer-readable recording medium according to supplementary note 14, the program further including instructions that cause the computer to carry out:(e) a step of generating third display information for displaying, on the screen of the display device, a third display region in which a face in a reference face three-dimensional image generated based on the three-dimensional data of the reference head is displayed to be oriented in alignment with an orientation of the face of the person targeted for matching displayed in the first display region.

Supplementary Note 16

The computer-readable recording medium according to supplementary note 15, the program further including instructions that cause the computer to carry out:(f) a step of generating second user interface display information for displaying, on the screen of the display device, a second user interface to be used in an operation for enabling the user to designate the feature region in the third display region.

Supplementary Note 17

The computer-readable recording medium according to supplementary note 16, the program further including instructions that cause the computer to carry out:(g) a step of, in a case where the feature region designated using the second user interface is matched against matching information in which a matching-use face development image and matching-use feature region for each person registered in advance are associated with each other, and a person to serve as a candidate is selected based on a matching result, generating candidate face image display information for displaying, on the screen of the display device, the matching-use face development image corresponding to the selected candidate person together with the feature region.

Supplementary Note 18

The computer-readable recording medium according to any one of supplementary notes 14 to 17, the program further including instructions that cause the computer to carry out:(h) a step of generating candidate face three-dimensional image display information for displaying, on the screen of the display device, a matching-use face three-dimensional image corresponding to the selected candidate person, based on three-dimensional data of a head of the selected candidate person.

Although the application has been described above with reference to an example embodiment, the invention is not limited to the foregoing example embodiment. Various modifications that will be appreciated by those skilled in the art can be made to the configurations and details of the application within the scope of the application.

INDUSTRIAL APPLICABILITY

According to the application as described above, a feature visible on the skin surface of the face of a person targeted for matching can be designated, according to the orientation of the face in a captured image. The application is useful in fields that require matching such as monitoring systems and authentication systems.

LIST OF REFERENCE SIGNS

10Matching support apparatus11First display information generation unit12Second display information generation unit13User interface display information generation unit14Candidate face image display information generation unit15Third display information generation unit20Image capturing apparatus21Storage device22Input device30Display device61Generation unit62Matching unit63Selection unit64Detection unit65Association unit66Conversion unit110Computer111CPU112Main memory113Storage device114Input interface115Display controller116Data reader/writer117Communication interface118Input device119Display device120Recording medium121Bus