Image processing device and recording medium storing image processing program

An image processing device includes a storage unit that stores dictionary information; a generating unit that extracts, from an input image, a plurality of characteristic point candidates, and generates a plurality of combinations that each include a plurality of characteristic point candidates; a removing unit that removes, for each of the combinations, at least one characteristic point candidate based on at least one of the dictionary information and information obtained by analyzing the input image; and a determining unit that acquires, for each of the combinations, results of matching the dictionary information with the combination of which the at least one characteristic point candidate has been removed, selects a combination of characteristic point candidates based on the acquired matching results, and determines, as the characteristic points, the plurality of characteristic point candidates included in the selected combination.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims the benefit of priority of the prior Japanese Patent Application No. 2010-270538, filed on Dec. 3, 2010, the entire contents of which are incorporated herein by reference.

FIELD

Various embodiments described herein relate to an image processing device and a medium for recording an image processing program.

BACKGROUND

There is a technique for extracting a plurality of characteristic point candidates from an input image and detecting characteristic points from the extracted characteristic point candidates in order to use the characteristic points for a process to be performed by an information processing device. Japanese Laid-Open Patent Publication No. 2007-233997 discusses that when a pupil is to be detected as a characteristic point from an input image obtained by imaging a person's face, positional relationships between the pupil and other characteristic points such as a mouth corner and a naris are used. For example, in a process using this technique, a plurality of characteristic point candidates that can be characteristic points such as pupils, mouth corners and nares are extracted from an input image obtained by imaging a person's face. In the process, a plurality of combinations of the characteristic point candidates such as the pupils, the mouth corners and the nares are then generated and compared with a model in which positional relationships among characteristic points such as pupils, mouth corners and nares are defined. In the process, as a result of the comparison, characteristic point candidates that are included in a combination that best matches the model are determined as characteristic points such as the pupils, the mouth corners and the nares.

The positions of the pupils that are detected as the characteristic points from the input image are used for the following processes. For example, the positions of the pupils are used for the process in which an information processing device such as a mobile phone or a personal computer tracks movements of the pupils of a user, captures a line of sight of the user, and scrolls a screen displayed on a display or the like on the basis of a movement of the line of sight. In addition, the positions of the pupils are used for the process in which an electronic device that is installed in a vehicle tracks movements of the pupils of a driver, captures a line of sight of the driver, and provides an alarm when the device detects that the driver looks away from a road.

SUMMARY

According to an aspect of the invention, an image processing device includes a storage unit that stores dictionary information; a generating unit that extracts, from an input image, a plurality of characteristic point candidates, and generates a plurality of combinations that each include a plurality of characteristic point candidates; a removing unit that removes, for each of the combinations, at least one characteristic point candidate based on at least one of the dictionary information and information obtained by analyzing the input image; and a determining unit that acquires, for each of the combinations, results of matching the dictionary information with the combination of which the at least one characteristic point candidate has been removed, selects a combination of characteristic point candidates based on the acquired matching results, and determines, as the characteristic points, the plurality of characteristic point candidates included in the selected combination.

The object and advantages of the invention will be realized and attained by at least the features, elements, and combinations particularly pointed out in the claims.

DESCRIPTION OF EMBODIMENTS

Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout. The embodiments are described below to explain the present invention by referring to the figures. In the figures, dimensions and/or proportions may be exaggerated for clarity of illustration. It will also be understood that when an element is referred to as being “connected to” another element, it may be directly connected or indirectly connected, i.e., intervening elements may also be present. Further, it will be understood that when an element is referred to as being “between” two elements, it may be the only element between the two elements, or one or more intervening elements may also be present.

In the process using the aforementioned conventional technique, the combinations of the characteristic point candidates are compared with the model, and the characteristic point candidates that form the combination that best matches the model are determined as the characteristic points. Thus, if a characteristic point candidate that corresponds to an authentic characteristic point is included in the combination of the characteristic point candidates, and another characteristic point candidate is located significantly far from an authentic characteristic point, there is a possibility that the combination may not best match the model.

For example, from an input image that is obtained by imaging a person's face in an environment in which high-intensity light is irradiated with the right half of the person's face, characteristic point candidates that correspond to characteristic points such as a right eye, a right naris and a right mouth corner, which are located in the right half of the person's face, may not be extracted in an excellent manner due to so-called overexposure of the image in some cases. In addition, characteristic points of the right eye, the right naris and the right mouth corner are not included in the input image in some cases, depending on the orientation of the face when the face is imaged. Even if other characteristic point candidates (such as a left eye and a left naris) that are included in a combination including the characteristic point candidates (corresponding to the right eye, the right naris and the right mouth corner) extracted in the aforementioned environment are authentic, there is a possibility that the degree of matching of the combination with the model may be reduced. Specifically, the authentic characteristic points may be hidden and may not be finally detected, depending on the state of the input image. As a result, in the process using the aforementioned technique, the accuracy of detecting characteristic points may be reduced.

The inventors propose an image processing program and an image processing device, which enable the accuracy of detecting characteristic points to be increased.

Embodiments of the image processing program disclosed herein and the image processing device disclosed herein are described in detail with reference to the accompanying drawings. The embodiments described below are embodiments, and do not limit the image processing program disclosed herein and the image processing device disclosed herein.

First Embodiment

Configuration of Image Processing Device (First Embodiment)

FIG. 1is a functional block diagram illustrating a configuration of an image processing device according to a first embodiment. As illustrated inFIG. 1, the image processing device200according to the first embodiment is connected to an imaging device100. The imaging device100images an object from which characteristic points are extracted, for example. For example, the imaging device100images an object such as a person's face, a vehicle or the like. In the embodiment described below, the imaging device100images a person's face as an object to be imaged as an example.

In addition, as illustrated inFIG. 1, the image processing device200includes an input unit210, an output unit220, a storage unit230and a controller240.

The input unit210acquires an image obtained by the imaging device100. The image that is acquired by the input unit210is stored in an image storage unit231of the storage unit230. The output unit220outputs a result of a process performed by the controller240from the image processing device200to an external device. For example, the output unit220outputs the result of the process performed by the controller240to an external device that detects a line of sight and recognizes a facial expression.

The storage unit230includes the image storage unit231and a model storage unit232as illustrated inFIG. 1. Examples of the storage unit230are semiconductor elements such as a random access memory and a flash memory. The image storage unit231stores the image acquired by the input unit210.

The model storage unit232stores dictionary information on a combination of a plurality of authentic characteristic points included in the person's face. An example of the dictionary information stored in the model storage unit232is described with reference toFIG. 2.FIG. 2is a diagram explaining the model storage unit according to the first embodiment.

The model storage unit232stores characteristic points to be detected from an image obtained by imaging the person's face. For example, the model storage unit232stores information of distances between the characteristic points such as the pupils, the tip of the nose, the nares and the mouth corners. In the following description, the image that is obtained by imaging the person's face is called a face image in some cases. In addition, information of each of the distances between the characteristic points is called a link length in some cases. For example, the model storage unit232stores information of a distance between the pupils as a reference, and stores values obtained by normalizing distances between the other characteristic points. For example, as illustrated inFIG. 2, when the distance between the right pupil and the left pupil is “1.0”, the model storage unit232stores a value of “0.6” as a distance between the right pupil or the left pupil and the tip of the nose.

In addition, the model storage unit232can combine pairs of the characteristic points prepared as the dictionary information and use various types or an arbitrary number of pairs that are among the pairs of the characteristic points on the basis of positional relationships among characteristic points to be detected. For each of the aforementioned distances between the characteristic points, the number of pixels between the characteristic points in the face image can be used. For example, the model storage unit232can calculate the number of pixels between characteristic points for each of multiple face images for learning, and treat the average of the calculated numbers of the pixels as the distance between the characteristic points. The model storage unit232may not use the distance between the pupils as the reference and store the values obtained by normalizing the distances between the other characteristic points. The model storage unit232may store the numbers of the pixels without the normalization.

Returning toFIG. 1, the controller240includes a face detecting unit241, an extracting unit242, a generating unit243, a removing unit244and a determining unit245. Examples of the controller240may be an electronic circuit and an integrated circuit. The electronic circuit may be a central processing unit (CPU), a micro processing unit (MPU) or the like, for example. The integrated circuit may be an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or the like, for example.

The face detecting unit241acquires the image from the image storage unit231and detects the face from the acquired image. For example, the face detecting unit241uses at least one of existing techniques such as an edge detection, pattern matching, an inter-frame difference and a neutral network and tries to detect the face from the image.

When the face detecting unit241detects the face from the image, the extracting unit242extracts, from the image, a plurality of characteristic point candidates that can be characteristic points such as the pupils, the tip of the nose, the nares and the mouth corners. For example, the extracting unit242performs a process using a separability filter or performs a process using pattern matching and thereby extracts the characteristic point candidates.

The generating unit243generates a plurality of combinations of the characteristic point candidates extracted by the extracting unit242. For example, the generating unit243generates a plurality of exclusive combinations that each include a single characteristic point candidate for the right pupil, a single characteristic point candidate for the left pupil, a single characteristic point candidate for the tip of the nose, a single characteristic point candidate for the right naris, a single characteristic point candidate for the left naris, a single characteristic point candidate for the right mouth corner and a single characteristic point candidate for the left mouth corner.

The removing unit244removes, from each of the combinations generated by the generating unit243, a characteristic point candidate that causes the largest deviation from the dictionary information among the characteristic point candidates included in the combination. The removing unit244is described below with reference toFIGS. 3 to 12.FIGS. 3 to 7are diagrams explaining the removing unit according to the first embodiment.

Reference numeral10illustrated inFIG. 3indicates a conceptual model that represents positional relationships among the characteristic points on the basis of the dictionary information. Symbols E1and E2illustrated inFIG. 3indicate models of the pupils indicated in the dictionary information. A symbol N1illustrated inFIG. 3indicates a model of the tip of the nose, while the tip of the nose is indicated in the dictionary information. Symbols N2and N3illustrated inFIG. 3indicate models of the nares indicated in the dictionary information. Symbols M1and M2illustrated inFIG. 3indicate models of the mouth corners indicated in the dictionary information. Reference numeral20illustrated inFIGS. 3 to 7indicates a conceptual model that represents positional relationships among characteristic point candidates included in a combination generated by the generating unit243or represents the positional relationships among the characteristic point candidates on the image acquired from the image storage unit231. Symbols E1′ and E2′ illustrated inFIG. 3indicate models of candidates that are candidates for the pupils and included in the combination. A symbol N1′ illustrated inFIG. 3indicates a model of a candidate that is a candidate for the tip of the nose and included in the combination. Symbols N2′ and N3′ illustrated inFIG. 3indicate models of candidates that are candidates for the nares and included in the combination. Symbols M1′ and M2′ illustrated inFIG. 3indicate models of candidates that are candidates for the mouth corners and included in the combination.

First, the removing unit244acquires a single combination from the plurality of combinations generated by the generating unit243. Then, the removing unit244acquires the dictionary information from the model storage unit232and calculates deviations of the distances (link lengths) between the characteristic point candidates included in the combination generated by the generating unit243from the distances (link lengths) between the characteristic points included in the dictionary information. Conceptually speaking, the removing unit244compares the conceptual model10based on the dictionary information with the conceptual model20of the combination generated by the generating unit243as illustrated inFIG. 3, and calculates the deviations of the conceptual model20from the conceptual model10.

Next, the removing unit244specifies a link that causes the maximum deviation among the deviations of the distances (link lengths) between the characteristic point candidates included in the combination generated by the generating unit243from the distances (link lengths) between the characteristic points included in the dictionary information. Then, the removing unit244detects a characteristic point candidate that is among a plurality of characteristic point candidates connected to the specified link causing the maximum deviation from the dictionary information and causes a larger deviation than the other characteristic point candidate connected to the specified link. The removing unit244removes the detected characteristic point candidate from the combination of the characteristic point candidates.

For example, it is assumed that the removing unit244specifies, as the link causing the maximum deviation of the link length, a link that connects the model N1′ of the candidate for the tip of the nose to the model N2′ of the candidate for the right naris. In this assumption, the removing unit244detects a characteristic point candidate that is among the model N1′ of the candidate for the tip of the nose and the model N2′ of the candidate for the right naris and causes a larger deviation than the other characteristic point candidate. As illustrated by dotted lines ofFIG. 4, the removing unit244calculates the sum of deviations of link lengths of links connected to the model N1′ of the candidate for the tip of the nose. Similarly, as illustrated by dotted lines ofFIG. 5, the removing unit244calculates the sum of deviations of link lengths of links connected to the model N2′ of the candidate for the right naris. Then, the removing unit244compares the calculated sum of the deviations relating to the model N1′ of the candidate for the tip of the nose with the calculated sum of the deviations relating to the model N2′ of the candidate for the right naris, and detects, as the characteristic point candidate causing the larger deviation than the other characteristic point candidate, a model that is among the models N1′ and N2′ and corresponds to the larger calculated sum.

If the removing unit244detects, as the characteristic point candidate causing the larger deviation than the other characteristic point candidate, the model N2′ (illustrated inFIG. 5) of the candidate for the right naris as a result of the aforementioned process, the removing unit244removes the model N2′ of the candidate for the right naris from the selected combination. Conceptually speaking, as illustrated inFIG. 6, the conceptual model20is in a state after the removing unit244removes the model N2′ of the candidate for the right naris from the conceptual model20selected from among the combinations generated by the generating unit243, for example.

Subsequently, the removing unit244recalculates the deviations of the distances (link lengths) between the characteristic point candidates remaining after the removal of the model N2′ of the candidate for the right naris from the distances (link lengths) between the characteristic points indicated in the dictionary information. Conceptually speaking, the removing unit244compares the conceptual model10based on the dictionary information with the conceptual model20configured with the characteristic point candidates remaining after the removal of the model N2′ of the candidate for the right naris, and calculates the deviations of the conceptual model20from the conceptual model10.

Then, the removing unit244acquires the maximum value among the deviations of the distances between the characteristic point candidates remaining after the removal of the model N2′ of the candidate for the right naris from the distances between the characteristic points indicated in the dictionary information. The removing unit244associates the acquired maximum value with the combination of the characteristic point candidates remaining after the removal of the model N2′ of the candidate for the right naris, and stores the acquired maximum value and the combination of the characteristic point candidates remaining after the removal of the model N2′ of the candidate for the right naris. For example, when a deviation of the link length of a link a (illustrated inFIG. 7) that connects the characteristic point candidates to each other is the maximum value among deviations of the characteristic point candidates included in the combination illustrated inFIG. 6, the removing unit244associates the maximum value with the combination of the characteristic point candidates illustrated inFIG. 7, and stores the maximum value and the combination of the characteristic point candidates illustrated inFIG. 7. Specifically, the removing unit244associates the deviation of the link length with the positions of the characteristic point candidates (illustrated inFIG. 7) on the image, and stores the deviation of the link length and the positions of the characteristic point candidates (illustrated inFIG. 7) on the image.

Then, the removing unit244determines whether or not the removing unit244has completed the aforementioned process on all the combinations generated by the generating unit243. When the aforementioned process has yet to be performed on at least one of the combinations generated by the generating unit243, the removing unit244selects the next combination and performs the aforementioned process on the selected combination. On the other hand, when the removing unit244has completed the aforementioned process on all the combinations generated by the generating unit243, the removing unit244activates a process to be performed by the determining unit245(described later).

When the process has been completed by the removing unit244, the determining unit245searches the minimum value among the maximum values that are among the deviations and have been stored by the removing unit244. Then, the determining unit245acquires a combination associated with the searched minimum value, and determines, as final characteristic points, characteristic point candidates included in the acquired combination.

Process to be Performed by Imaging Processing Device (First Embodiment)

The flow of a process that is performed by the image processing device200according to the first embodiment is described with reference toFIG. 8.FIG. 8is a diagram illustrating the overall flow of the process that is performed by the image processing device according to the first embodiment. The process illustrated inFIG. 8is activated at a time set in the image processing device200or at the time when the image processing device200acquires an image obtained by the imaging device100. In the following description, a “model” indicates distances (link lengths) between the characteristic points that are stored as the dictionary information in the model storage unit232.

As illustrated inFIG. 8, the face detecting unit241acquires the image from the image storage unit231(in operation S101). When the face detecting unit241detects the face from the image, the face detecting unit241activates the process to be performed by the extracting unit242. The extracting unit242extracts, from the image, a plurality of characteristic point candidates that can be the characteristic points such as the pupils, the tip of the nose, the nares, and the mouth corners (in operation S102). Subsequently, the generating unit243generates a plurality of combinations of the characteristic point candidates extracted by the extracting unit242(in operation S103).

Next, the removing unit244selects a single combination from among the combinations generated in operation S103(in operation S104), and compares the selected combination with the model (in operation S105). Then, the removing unit244calculates deviations of link lengths of the combination from the model, or calculates the deviations of the link lengths between characteristic point candidates included in the combination selected in operation S104from the model (in operation S106).

Next, the removing unit244specifies a link that causes the maximum deviation among the results of the process of operation S106(in operation S107). Subsequently, the removing unit244detects a characteristic point candidate that is among characteristic point candidates relating to the link specified in operation S107and causes a larger deviation than the other characteristic point candidate relating to the link specified in operation S107(in operation S108). Then, the removing unit244removes the characteristic point candidate detected in operation S108from the combination selected in operation S104(in operation S109).

Next, the removing unit244compares the combination after the process of operation S109with the model in substantially the same manner as the aforementioned operation S105(in operation S110). Subsequently, the removing unit244recalculates the deviations of the link lengths from the model in substantially the same manner as the aforementioned operation S106(in operation S111). Then, the removing unit244associates the maximum value among the deviations calculated in operation S111with the interested combination that is currently processed, and the removing unit244stores the maximum value among the deviations calculated in operation S111and the interested combination that is currently processed (in operation S112). The removing unit244determines whether or not the removing unit244has completed the process on all the combinations generated in operation S103(in operation S113).

When the process has yet to be processed on at least one of all the combinations as a result of the determination (No in operation S113), the removing unit244causes the process illustrated inFIG. 8to return to the aforementioned process of operation S104and acquires the next combination. When the process has been completed on all the combinations (Yes in operation S113), the removing unit244activates the process to be performed by the determining unit245. The determining unit245searches the minimum value among the maximum values that are among the deviations and have been stored in operation S112(in operation S114). Then, the determining unit245acquires a combination associated with the minimum value searched in operation S114, and determines, as the final characteristic points, characteristic point candidates included in the acquired combination (in operation S115). Then, determining unit245terminates the process.

Effects Obtained in First Embodiment

As described above, the image processing device200performs matching of the model with a combination from which a characteristic point candidate that is likely to cause a deviation has been removed. The image processing device200determines, as the final characteristic points, a plurality of characteristic point candidates included in the selected combination on the basis of the matching results. Thus, according to the first embodiment, the accuracy of detecting characteristic points can be improved.

In addition, the image processing device200removes, from the combination, the characteristic point candidate that is likely to cause the deviation, and the image processing device200compares the model with the combination from which the characteristic point candidate that is likely to cause the deviation has been removed. Thus, before the image processing device200compares the combination with the model, the image processing device200can remove an effect of the characteristic point candidate that is likely to cause the deviation. Thus, according to the first embodiment, the accuracy of detecting characteristic points can be improved.

Second Embodiment

The first embodiment describes that the deviations of the combinations of the characteristic point candidates from the model of the dictionary information stored in the model storage unit232are calculated, and that a characteristic point candidate that is likely to cause a deviation is removed on the basis of the calculated deviations. However, the technique disclosed herein is not limited to this. For example, a characteristic point candidate that is likely to cause a deviation can be removed on the basis of likelihoods that indicate likelihoods of the characteristic point candidates for the characteristic points.

Process to be Performed by Image Processing Device (Second Embodiment)

The image processing device according to a second embodiment has the same constituent blocks as the image processing device according to the first embodiment, or has substantially the same constituent blocks as the image processing device200illustrated inFIG. 1. The flows of processes that are performed by the image processing device according to the second embodiment are described with reference toFIGS. 9 and 10.FIGS. 9 and 10are diagrams illustrating the flows of the processes that are performed by the image processing device according to the second embodiment.

As illustrated inFIG. 9, the face detecting unit241acquires the image from the image storage unit231(in operation S201). When the face detecting unit241detects the face from the image, the face detecting unit241activates the process to be performed by the extracting unit242. The extracting unit242extracts, from the image, a plurality of characteristic point candidates that can be the characteristic points such as the pupils, the tip of the nose, the nares, and the mouth corners (in operation S202). Subsequently, the generating unit243generates a plurality of combinations of the characteristic point candidates extracted by the extracting unit242(in operation S203). In other words, processes in operation S201to S203are substantially the same as the processes in operation S101to S103of the first embodiment.

The processes that are performed by the image processing device according to the second embodiment are different in the following features from the process that is performed by the image processing device according to the first embodiment. Specifically, as illustrated inFIG. 9, the removing unit244selects a single combination from among combinations generated in operation S203(in operation S204) and removes, from the selected combination, a characteristic point candidate whose likelihood is lower than a threshold (in operation S205).

Next, the removing unit244compares the combination after the process of operation S205with the model (in operation S206). Subsequently, the removing unit244calculates deviations of link lengths of the combination from the model, or calculates the deviations of the link lengths between characteristic point candidates included in the combination after the process of operation S205from the model (in operation S207). Then, the removing unit244associates the maximum value among the deviations calculated in operation S207with the interested combination that is currently processed, and the removing unit244stores the maximum value among the deviations calculated in operation S207and the interested combination that is currently processed (in operation S208). Processes in operation S209to S211are substantially the same as the processes in operation S113to S115of the first embodiment.

Specifically, the removing unit244determines whether or not the removing unit244has completed the process on all the combinations generated in operation S203(in operation S209). When the process has yet to be performed on at least one of all the combinations as a result of the determination (No in operation S209), the removing unit244causes the process illustrated inFIG. 9to return to the aforementioned process of operation S204and acquires the next combination. On the other hand, when the process has been completed on all the combinations (Yes in operation S209), the removing unit244activates the process to be performed by the determining unit245. The determining unit245searches the minimum value among the maximum values that are among the deviations and have been stored in operation S208(in operation S210). Then, the determining unit245acquires a combination associated with the minimum value searched in operation S210, and determines, as the final characteristic points, characteristic point candidates included in the acquired combination (in operation S211). Then, the determining unit245terminates the process.

The extracting unit242can use, as the likelihoods used in the process of the aforementioned operation S205, response values that are provided from the separability filter in order to extract the characteristic point candidates such as the pupils and the nares from the image.

As illustrated inFIG. 10, both removal of a characteristic point candidate on the basis of the deviations of the selected combination from the model described in the first embodiment and removal of a characteristic point candidate on the basis of the likelihoods can be performed.

As illustrated inFIG. 10, the face detecting unit241acquires the image from the image storage unit231(in operation S301). When the face detecting unit241detects the face from the image, the face detecting unit241activates the process to be performed by the extracting unit242. The extracting unit242extracts, from the image, a plurality of characteristic point candidates that can be the characteristic points such as the pupils, the tip of the nose, the nares, and the mouth corners (in operation S302). Subsequently, the generating unit243generates a plurality of combinations of the characteristic point candidates extracted by the extracting unit242(in operation S303). In other words, processes in operation S301to S303are substantially the same as the processes in operation S101to S103of the first embodiment.

Next, the removing unit244selects a single combination from among combinations generated in operation S303(in operation S304) and removes, from the selected combination, a characteristic point candidate whose likelihood is lower than the threshold (in operation S305).

Next, the removing unit244compares the combination after the process of operation S305with the model (in operation S306). Subsequently, the removing unit244calculates deviations of link lengths of the combinations from the model, or calculates the deviations of the link lengths between characteristic point candidates included in the combination after the process of operation S305from the model (in operation S307).

Next, the removing unit244specifies a link that causes the maximum deviation among the results of the process of operation S307(in operation S308). Subsequently, the removing unit244detects a characteristic point candidate that is among characteristic point candidates relating to the link specified in operation S308and causes a larger deviation than the other characteristic point candidate relating to the link specified in operation S308(in operation S309). Then, the removing unit244removes the characteristic point candidate detected in operation S309from the combination after the process of operation S305(in operation S310).

Next, the removing unit244compares the combination after the process of operation S310with the model in substantially the same manner as the aforementioned operation S306(in operation S311). Subsequently, the removing unit244recalculates the deviations of the link lengths from the model in substantially the same manner as the aforementioned operation S307(in operation S312). Then, the removing unit244associates the maximum value among the deviations calculated in operation S312with the interested combination that is currently processed, and the removing unit244stores the maximum value among the deviations calculated in operation S312and the interested combination that is currently processed (in operation S313). Processes other than the aforementioned processes are substantially the same as the processes illustrated inFIG. 8in the first embodiment and the processes illustrated inFIG. 9.

Specifically, the removing unit244determines whether or not the removing unit244has completed the process on all the combinations generated in operation S303(in operation S314). When the process has yet to be performed on at least one of all the combinations as a result of the determination (No in operation S314), the removing unit244causes the process illustrated inFIG. 10to return to the aforementioned process of operation S304and acquires the next combination. On the other hand, when the process has been completed on all the combinations (Yes in operation S314), the removing unit activates the process to be performed by the determining unit245. The determining unit245searches the minimum value among the maximum values that are the deviations and have been stored in operation S313(in operation S315). Then, the determining unit245acquires a combination associated with the minimum value searched in operation S315, and determines, as the final characteristic points, characteristic point candidates included in the acquired combination (in operation S316). Then, the determining unit245terminates the process.

Effects Obtained in Second Embodiment

As described above, in the second embodiment, a characteristic point candidate that is likely to cause a deviation is removed on the basis of the likelihoods that indicate the likelihoods of the characteristic point candidates for the characteristic points. Thus, according to the second embodiment, an effect of the characteristic point candidate that is likely to cause the deviation can be removed by the simple process, and the accuracy of detecting characteristic points can be improved.

In the second embodiment, both removal of a characteristic point candidate on the basis of the deviations of the link lengths from the model and removal of a characteristic point candidate on the basis of the likelihoods are performed. Thus, according to the second embodiment, effects of characteristic point candidates that are likely to cause deviations can be removed on the basis of the various indexes, and the accuracy of detecting characteristic points can be improved.

The second embodiment describes that the characteristic point candidate that is likely to cause the deviation is removed on the basis of the likelihoods of the characteristic point candidates for the characteristic points. The second embodiment is not limited to this.

For example, the removing unit244may acquire, from an image analyzing unit246, information of the position of a characteristic point candidate removed by a previous process. Then, the removing unit244may remove, from a combination of characteristic point candidates, a characteristic point candidate that corresponds to the characteristic point candidate removed by the previous process on the basis of the acquired information of the position of the characteristic point candidate. In this manner, the characteristic point candidate that is likely to cause a deviation and corresponds to the characteristic point candidate removed by the previous process may be removed from the combination of the characteristic point candidates in the second embodiment. In addition, in the second embodiment, both removal of a characteristic point candidate from the combination of the characteristic point candidates on the basis of deviations of link lengths from the model and removal of a characteristic point candidate from the combination of the characteristic point candidates on the basis of the result of the previous process may be performed.

In this case, it is possible to effectively use the result of the previous process and remove, in advance, an effect of the characteristic point candidate that is likely to cause the deviation.

In addition, the second embodiment describes that a characteristic point candidate is removed on the basis of the likelihoods of the characteristic point candidates. However, the following process may be performed. For example, deviations of the link lengths of links connecting characteristic point candidates to each other from the model are corrected on the basis of the likelihoods of the characteristic point candidates that form the links. For example, the deviations are corrected so that as the likelihoods of the characteristic point candidates that form each of the links are larger, the deviation of the link length of the link from the model is smaller. Then, similarly to the first embodiment, a link that causes the maximum deviation of the link length of the link from the model is specified, and a characteristic point candidate that is among characteristic point candidates relating to the specified link and causes a larger deviation than the other characteristic point candidate relating to the specified link is removed from a combination of the characteristic point candidates, for example. In this process, an effect of a characteristic point candidate that is likely to cause a deviation can be removed in advance, and the accuracy of detecting characteristic points can be improved.

Third Embodiment

In the second embodiment, the likelihoods of the characteristic point candidates may be adjusted on the basis of a facial orientation that is acquired from information obtained by analyzing the image, and a characteristic point candidate may be removed on the basis of the adjusted likelihoods.

FIG. 11is a diagram explaining the image process device according to a third embodiment. The image processing device according to the third embodiment receives an image as indicated by reference numeral1illustrated inFIG. 11. Next, the image processing device according to the third embodiment extracts, from the received image, a plurality of characteristic point candidates represented by circles as indicated by reference numeral2illustrated inFIG. 11. Subsequently, the image processing device according to the third embodiment generates a combination of the characteristic point candidates as indicated by reference numeral3illustrated inFIG. 11. In this case, the image processing device according to the third embodiment analyzes the received image, acquires a facial orientation from the analyzed image, and adjusts likelihoods of the characteristic point candidates on the basis of the facial orientation. Then, the image processing device according to the third embodiment removes a characteristic point candidate on the basis of the adjusted likelihoods as indicated by reference numeral4illustrated inFIG. 11.

Configuration of Image Processing Device (Third Embodiment)

FIG. 12is a functional block diagram illustrating a configuration of the image processing device according to the third embodiment. The image processing device according to the third embodiment basically has substantially the same configuration as the image processing device according to the first embodiment, but is different in the following features from the image processing device according to the first embodiment.

When the face detecting unit241detects the face from the image, the image analyzing unit246acquires, from the image storage units231, a plurality of input frames of the image, and acquires information of the facial orientation by analyzing the acquired frames. In this case, the plurality of input frames are a current input frame and a previous input frame. For example, the image analyzing unit246extracts and acquires the outlines of the eyes from the previous frame, and calculates a horizontal distance between the outer corners of the eyes. Then, the image analyzing unit246acquires information of the facial orientation by tracking a change in horizontal distance between the outer corners of the eyes over time from the previous input frame to the current input frame.

The removing unit244adjusts, on the basis of the information that indicates the facial orientation and has been acquired by the image analyzing unit246, the likelihoods of the characteristic point candidates included in each of the combinations generated by the generating unit243. For example, when the removing unit244acquires, from the image analyzing unit246, information that indicates that the face is oriented toward the right side as illustrated inFIG. 11, the removing unit244adjusts the likelihoods of the right pupil, the right naris and the right mouth corner, which are among the characteristic point candidates and located on the right side of the face, so that the likelihoods of the right pupil, the right naris and the right mouth corner are reduced at a rate set in advance. After the adjustment of the likelihoods, the removing unit244removes, from the characteristic point candidates included in the combination, a characteristic point candidate whose likelihood is lower than the threshold.

In addition, the image analyzing unit246acquires, from the removing unit244, information of a characteristic point candidate that has been removed by the previous process from the previous input frame on the basis of the likelihoods. Then, the image analyzing unit246performs matching of the current input frame with the previous input frame, and acquires information of the position of the characteristic point candidate that has been removed by the previous process from the previous input frame.

The removing unit244acquires, from the image analyzing unit246, the information of the position of the characteristic point candidate removed by previous process. Then, the removing unit244removes, from the characteristic point candidates included in the combination on the basis of the position of the characteristic point candidate removed by the previous process, a characteristic point candidate that corresponds to the characteristic point candidate removed by previous process.

Process to be Performed by Image Processing Device (Third Embodiment)

The flows of processes that are performed by the image processing device according to the third embodiment are described with reference toFIGS. 13 and 14.FIGS. 13 and 14are diagrams illustrating the flows of the processes that are performed by the image processing device according to the third embodiment.

First, the flow of the process, in which likelihoods are adjusted and a characteristic point candidate is removed on the basis of the adjusted likelihoods, is described with reference toFIG. 13. As illustrated inFIG. 13, the face detecting unit241acquires the image from the image storage unit231(in operation S401). When the face detecting unit241detects the face from the image, the face detecting unit241activates the process to be performed by the extracting unit242. The extracting unit242extracts, from the image, a plurality of characteristic point candidates that can be the characteristic points such as the pupils, the tip of the nose, the nares, and the mouth corners (in operation S402). Subsequently, the generating unit243generates a plurality of combinations of the characteristic point candidates extracted by the extracting unit242(in operation S403). In other words, processes in operation S401to S403are substantially the same as the processes in operation S301to S303of the second embodiment.

The process that is performed by the image processing device according to the third embodiment is different in the following features from the process that is performed by the image processing device according to the second embodiment. Specifically, as illustrated inFIG. 13, the removing unit244selects a single combination from among combinations generated in operation S403(in operation S404), and adjusts likelihoods of characteristic point candidates included in the selected combination on the basis of the facial orientation (in operation S405). After the process of operation S405, the removing unit244removes, from the combination selected in operation S404, a characteristic point candidate whose likelihood is lower than the threshold (in operation S406). Processes other than the aforementioned processes are substantially the same as the second embodiment.

Specifically, the removing unit244compares the combination after the process of operation S406with the model (in operation S407). Then, the removing unit244calculates deviations of link lengths of the combination from the model, or calculates the deviations of the link lengths between the characteristic point candidates included in the combination after the process of operation S406from the model (in operation S408). Subsequently, the removing unit244associates the maximum value among the deviations calculated in operation S408with the interested combination that is currently processed, and the removing unit244stores the maximum value among the deviations calculated in operation S408and the interested combination that is currently processed (in operation S409). Then, the removing unit244determines whether or not the removing unit244has completed the process on all the combinations generated in operation S403(in operation S410). When the process has yet to be performed on at least one of all the combinations as a result of the determination (No in operation S410), the removing unit244causes the process illustrated inFIG. 13to return to the aforementioned process of operation S404and acquires the next combination. On the other hand, when the process has been completed on all the combinations (Yes in operation S410), the removing unit244activates the process to be performed by the determining unit245. The determining unit245searches the minimum value among the maximum values that are among the deviations and have been stored in operation S409(in operation S411). Then, the determining unit245acquires a combination associated with the minimum value searched in operation S411, and determines, as the final characteristic points, characteristic point candidates included in the acquired combination (in operation S412). Then, the determining unit245terminates the process.

After the description of the process illustrated inFIG. 13, the flow of the process, in which a characteristic point candidate that corresponds to a characteristic point candidate removed by a previous process is removed, is described with reference toFIG. 14. As illustrated inFIG. 14, the face detecting unit241acquires the image from the image storage unit231(in operation S501). When the face detecting unit241detects the face from the image, the face detecting unit241activates the process to be performed by the extracting unit242. The extracting unit242extracts, from the image, a plurality of characteristic point candidates that can be the characteristic points such as the pupils, the tip of the nose, the nares, and the mouth corners (in operation S502). Subsequently, the generating unit243generates a plurality of combinations of the characteristic point candidates extracted by the extracting unit242(in operation S503). In other words, processes in operation S501to S503are substantially the same as the processes in operation S401to S403of theFIG. 13.

Next, the removing unit244selects a single combination from among combinations generated in operation S503(in operation S504), and adjusts likelihoods of characteristic point candidates included in the selected combination on the basis of the facial orientation (in operation S505). After the process of operation S505, the removing unit244removes, from the combination selected in operation S504, a characteristic point candidate whose likelihood is lower than the threshold (in operation S506). Then, the removing unit244removes, from the combination selected in operation S504, a characteristic point candidate that corresponds to the characteristic point candidate removed by the previous process (in operation S507). Processes other than the aforementioned processes are substantially the same as the processes illustrated inFIG. 13.

Specifically, the removing unit244compares the combination after the process of operation S507with the model (in operation S508). Next, the removing unit244calculates deviations of link lengths from the model, or calculates the deviations of the link lengths between characteristic point candidates included in the combination after the process of operation S507from the model (in operation S509). Subsequently, the removing unit244associates the maximum value among the deviations calculated in operation S509with the interested combination that is currently processed, and the removing unit244stores the maximum value among the deviations calculated in operation S509and the interested combination that is currently processed (in operation S510). Then, the removing unit244determines whether or not the removing unit244has completed the process on all the combinations generated in operation S503(in operation S511). When the process has yet to be performed on at least one of all the combinations as a result of the determination (No in operation S511), the removing unit244causes the process illustrated inFIG. 14to return to the aforementioned process of operation S504and acquires the next combination. On the other hand, when the process has been completed on all the combinations (Yes in operation S511), the removing unit244activates the process to be performed by the determining unit245. Then, the determining unit245searches the minimum value among the maximum values that are among the deviations and have been stored in operation S510(in operation S512). Then, the determining unit245acquires a combination associated with the minimum value searched in operation S512, and determines, as the final characteristic points, characteristic point candidates included in the acquired combination (in operation S513). Then, the determining unit245terminates the process.

In the above description, the image analyzing unit246acquires the information of the facial orientation. However, the present embodiment is not limited to this. For example, the image analyzing unit246removes, from the combination, a characteristic point candidate whose likelihood is lower than the threshold in an environment in which the face is irradiated with high-intensity light. Specifically, when the face detecting unit241detects the face, the image analyzing unit246divides an image region of the face into a plurality of regions. Then, the image analyzing unit246calculates brightness for each of the plurality of regions. For example, the average of luminescence values of pixels included in each of the regions is treated as the brightness. Then, the removing unit244adjusts a likelihood of a characteristic point candidate included in a region whose brightness is equal to or higher than a threshold so that the likelihood of the characteristic point candidate is reduced. There is a possibility that a characteristic point that is finally necessary may not be detected from an image that is acquired in the environment in which the face is irradiated with the high-intensity light. Thus, an effect of a characteristic point candidate that is likely to cause a deviation can be removed in advance by reducing the likelihood of the characteristic point candidate included in the region whose brightness is equal to or higher than the threshold.

Effects Obtained in Third Embodiment

As described above, in the third embodiment, after the likelihoods of the characteristic point candidates are adjusted on the basis of the facial orientation, a characteristic point candidate that is included in the combination is removed. According to the third embodiment, it is possible to remove a characteristic point candidate in accordance with reliabilities of the characteristic point candidates, while the reliabilities of the characteristic point candidates are based on an environment in which the object is imaged. According to the third embodiment, when an effect of a characteristic point candidate that is likely to cause a deviation is to be removed in advance, the accuracy of the removal can be improved.

Fourth Embodiment

The method (described in the first embodiment) for removing a characteristic point candidate and the method (described in the third embodiment) for removing a characteristic point candidate can be combined.

Configuration of Image Processing Device (Fourth Embodiment)

The image processing device according to a fourth embodiment has substantially the same constituent blocks as the image processing device according to the third embodiment, or has substantially the same constituent blocks as the image processing device200illustrated inFIG. 12. A part of processing functions of the functional blocks of the image processing device according to the fourth embodiment is different from the functional blocks illustrated inFIG. 12as described below.

Specifically, the removing unit244performs a process that is achieved by combining the method (described in the first embodiment) for removing a characteristic point candidate with the method (described in the third embodiment) for removing a characteristic point candidate. For example, the removing unit244combines the process illustrated inFIG. 8with the process illustrated inFIG. 13and performs the following process. First, the removing unit244adjusts the likelihoods of the characteristic point candidates included in each of the combinations generated by the generating unit243. After the adjustment of the likelihoods, the removing unit244removes, from the characteristic point candidates included in the combination, a characteristic point candidate whose likelihood is lower than the threshold.

Next, the removing unit244compares the model with the combination from which the characteristic point candidate whose likelihood is lower than the threshold has been removed. Then, the removing unit244calculates deviations of the link lengths between the characteristic point candidates included in the combination from the model. Subsequently, the removing unit244specifies a link that causes the maximum deviation among the calculated deviations. Then, the removing unit244detects a characteristic point candidate that is among the characteristic point candidates relating to the specified link and causes a larger deviation than the other characteristic point candidate relating to the specified link. Then, the removing unit244removes the detected characteristic point candidate from the combination, and compares the combination after the removal process with the model. Next, the removing unit244recalculates the deviations of the link lengths from the model. The removing unit244associates the maximum value among the calculated deviations with the interested combination that is currently processed, and the removing unit244stores the maximum value among the calculated deviations and the interested combination that is currently processed. The removing unit244performs the aforementioned process on all the combinations generated by the generating unit243.

In addition, the removing unit244can combine the process illustrated inFIG. 8with the process illustrated inFIG. 14and perform the following process, for example. The removing unit244removes, from the characteristic point candidates included in the combination, a characteristic point candidate whose likelihood is lower than the threshold. After the removal, the removing unit244removes a characteristic point candidate that corresponds to a characteristic point candidate removed by a previous process. Then, as described above, the removing unit244compares the model with the combination from which the characteristic point candidate whose likelihood is lower than the threshold has been removed and the characteristic point candidate that corresponds to the characteristic point candidate removed by the previous process has been removed, and the removing unit244then calculates deviations of the link lengths between the characteristic point candidates included in the combination from the model. Subsequently, the removing unit244specifies a link causing the maximum deviation among the calculated deviations and detects a characteristic point candidate that is among characteristic point candidates relating to the specified link and causes a larger deviation than the other characteristic point candidate relating to the specified link. Then, the removing unit244removes the detected characteristic point candidate from the combination and compares with the combination after the removal process with the model. Next, the removing unit244recalculates the deviations of the link lengths from the model. The removing unit244associates the maximum value among the calculated deviations with the interested combination that is currently processed, and the removing unit244stores the maximum value among the calculated deviations and the interested combination that is currently processed. The removing unit244performs the aforementioned process on all the combinations generated by the generating unit243.

Process to be Performed by Image Processing Device (Fourth Embodiment)

The flows of processes that are performed by the image processing device according to the fourth embodiment are described with reference toFIGS. 15 and 16.FIGS. 15 and 16are diagrams illustrating the flows of the processes that are performed by the image processing device according to the fourth embodiment.

As illustrated inFIG. 15, the face detecting unit241acquires the image from the image storage unit231(in operation S601). When the face detecting unit241detects the face from the image, the face detecting unit241activates the process to be performed by the extracting unit242. The extracting unit242extracts, from the image, a plurality of characteristic point candidates that can be the characteristic points such as the pupils, the tip of the nose, the nares, and the mouth corners (in operation S602). Subsequently, the generating unit243generates a plurality of combinations of the characteristic point candidates extracted by the extracting unit242(in operation S603). In other words, processes in operation S601to S603are substantially the same as the processes in operation S101to S103of the first embodiment, S201to S203of the second embodiment, and S401to S403of the third embodiment.

As illustrated inFIG. 15, the process that is performed by the image processing device according to the fourth embodiment is different in the following features from the processes that are performed by the image processing device according to the aforementioned embodiments. Specifically, the removing unit244selects a single combination from among combinations generated in operation S603(in operation S604), and adjusts, on the basis of the facial orientation, likelihoods of characteristic point candidates included in the selected combination (in operation S605). After the process of operation S605, the removing unit244removes, from the combination selected in operation S604, a characteristic point candidate whose likelihood is lower than the threshold (in operation S606).

Next, the removing unit244compares the combination after the process of operation S606with the model (in operation S607). Subsequently, the removing unit244calculates deviations of link lengths of the combination from the model, or calculates the deviations of the link lengths between the characteristic point candidates included in the combination after the process of operation S606from the model (in operation S608). Then, the removing unit244specifies a link that causes the maximum deviation among the results of the process of operation S608(in operation S609). Subsequently, the removing unit244detects a characteristic point candidate that is among characteristic point candidates relating to the link specified in operation S609and causes a larger deviation than the other characteristic point candidate relating to the link specified in operation S609(in operation S610). Then, the removing unit244removes, from the combination after the process of operation S606, the characteristic point candidate detected in operation S610(in operation S611).

Next, the removing unit244compares the combination after the process of operation S611with the model in substantially the same manner as the aforementioned operation S607(in operation S612). Then, the removing unit244recalculates the deviations of the link lengths from the model in substantially the same manner as the aforementioned operation S608(in operation S613). Then, the removing unit244associates the maximum value among the deviations calculated in operation S613with the interested combination that is currently processed, and the removing unit244stores the maximum value among the deviations calculated in operation S613and the interested combination that is currently processed (in operation S614). Processes other than the aforementioned processes are substantially the same as the aforementioned embodiments.

Specifically, the removing unit244determines whether or not the removing unit244has completed the process on all the combinations generated in operation S603(in operation S615). When the process has yet to be performed on at least one of all the combinations as a result of the determination (No in operation S615), the removing unit244causes the process illustrated inFIG. 15to return to the aforementioned process of operation S604and acquires the next combination. On the other hand, when the process has been completed on all the combinations (Yes in operation S615), the removing unit244activates the process to be performed by the determining unit245. The determining unit245searches the minimum value among the maximum values that are among the deviations and have been stored in operation S614(in operation S616). Then, the determining unit245acquires a combination associated with the minimum value searched in operation S616, and determines, as the final characteristic points, characteristic point candidates included in the acquired combination (in operation S617). Then, the determining unit245terminates the process.

In addition, as illustrated inFIG. 16, the face detecting unit241acquires the image from the image storage unit231(in operation S701). When the face detecting unit241detects the face from the image, the face detecting unit241activates the process to be performed by the extracting unit242. The extracting unit242extracts, from the image, a plurality of characteristic point candidates that can be the characteristic points such as the pupils, the tip of the nose, the nares, and the mouth corners (in operation S702). Subsequently, the generating unit243generates a plurality of combinations of the characteristic point candidates extracted by the extracting unit242(in operation S703).

As illustrated inFIG. 16, the process that is performed by the image processing device according to the fourth embodiment is different in the following features from the processes that are performed by the image processing device according to the aforementioned embodiments. Specifically, the removing unit244selects a single combination from among combinations generated in operation S703(in operation S704), and adjusts, on the basis of the facial orientation, likelihoods of characteristic point candidates included in the selected combination (in operation S705). After the process of operation S705, the removing unit244removes, from the combination selected in operation S704, a characteristic point candidate whose likelihood is lower than the threshold (in operation S706).

Next, the removing unit244removes, from the combination after the process of operation S706, a characteristic point candidate that corresponds to a characteristic point candidate removed by a previous process (in operation S707). Subsequently, the removing unit244compares the combination after the process of operation S707with the model (in operation S708). Then, the removing unit244calculates deviations of link lengths of the combination from the model, or calculates the deviations of the link lengths between the characteristic point candidates included in the combination after the process of operation S707from the model (in operation S709).

Next, the removing unit244specifies a link that causes the maximum deviation among the results of the process of operation S709(in operation S710). Subsequently, the removing unit244detects a characteristic point candidate that is among characteristic point candidates relating to the link specified in operation S710and causes a larger deviation than the other characteristic point candidate relating to the link specified in operation S710(in operation S711). Then, the removing unit244removes, from the combination after the process of operation S707, the characteristic point candidate detected in operation S711(in operation S712).

Next, the removing unit244compares the combination after the process of operation S712with the model in substantially the same manner as the aforementioned operation S708(in operation S713). Subsequently, the removing unit244recalculates the deviations of the link lengths from the model in substantially the same manner as the aforementioned operation S709(in operation S714). Then, the removing unit244associates the maximum value among the deviations calculated in operation S714with the interested combination that is currently processed, and the removing unit244stores the maximum value among the deviations calculated in operation S714and the interested combination that is currently processed (in operation S715). Processes other than the aforementioned processes are substantially the same as the aforementioned embodiments.

Specifically, the removing unit244determines whether or not the removing unit244has completed the process on all the combinations generated in operation S703(in operation S716). When the process has yet to be performed on at least one of all the combinations as a result of the determination (No in operation S716), the removing unit244causes the process illustrated inFIG. 16to return to the aforementioned process of operation S704and acquires the next combination. On the other hand, when the process has been completed on all the combinations (Yes in operation S716), the removing unit244activates the process to be performed by the determining unit245. The determining unit245searches the minimum value among the maximum values that are among the deviations and have been stored in operation S715(in operation S717). Then, the determining unit245acquires a combination associated with the minimum value searched in operation S717, and determines, as the final characteristic points, characteristic point candidates included in the acquired combination (in operation S718). Then, the determining unit245terminates the process.

Effects Obtained in Fourth Embodiment

As described above, in the fourth embodiment, a characteristic point candidate that is included in a combination is removed using a plurality of parameters among likelihoods adjusted on the basis of the facial orientation, a result of the previous process, or deviations of link lengths from the model. Thus, according to the fourth embodiment, when an effect of a characteristic point candidate that is likely to cause a deviation is to be removed in advance, the accuracy of the removal can be improved compared to the aforementioned embodiments.

Fifth Embodiment

Other embodiments of the image processing program disclosed herein and the image processing device disclosed herein are described below.

(1) Detection of Characteristic Points of Vehicle

The processing functions of the image processing device200described above can be applied to the case where the object is a vehicle, for example.FIG. 17is a diagram explaining the image processing device according to a fifth embodiment. As illustrated inFIG. 17, the image processing device200stores a model30of dictionary information on a combination of a plurality of authentic characteristic points included in a vehicle. The image processing device200acquires characteristic points from an image obtained by imaging the vehicle in substantially the same manner as the aforementioned embodiments.

For example, the image processing device200performs a process using an existing technique such as pattern matching and thereby extracts characteristic point candidates P1, P2and P3(illustrated inFIG. 17) from the image obtained by imaging the vehicle that is the object. Then, the image processing device200generates a plurality of combinations of characteristic point candidates as indicated by symbols C1and C2illustrated inFIG. 17, and compares one of the generated combinations with the model30. Subsequently, the image processing device200specifies a link that is among links connecting the characteristic point candidates of the combination and causes the maximum deviation among deviations of the link lengths of the links from the model. Then, the image processing device200detects a characteristic point candidate that is among characteristic point candidates relating to the specified link and causes a larger deviation than the other characteristic point candidate relating to the specified link. For example, the image processing device200detects a characteristic point candidate P4illustrated inFIG. 17. Then, the image processing device200compares the model30with the combination from which the characteristic point candidate P4has been removed again. The image processing device200calculates the deviations of the combination (from which the characteristic point candidate P4has been removed) from the model30. Then, the image processing device200associates the combination of the characteristic point candidates with the maximum value among the deviations of the link lengths of the links connecting the characteristic point candidates from the model30, and stores the combination of the characteristic point candidates and the maximum value among the deviations of the link lengths of the links connecting the characteristic point candidates from the model30. The image processing device200performs the aforementioned process on all the generated combinations, and finally determines, as characteristic points, characteristic point candidates included in a combination associated with the minimum value among the maximum values that are among the deviations of the link lengths from the model.

(2) Configuration of Image Processing Device and the Like

For example, the configuration of the functional blocks of the image processing device200illustrated inFIG. 1is a conceptual configuration. Thus, the functional blocks of the image processing device200do not need to be physically configured as illustrated inFIG. 1. For example, the extracting unit242and the generating unit243, which are illustrated inFIG. 1, may be functionally or physically integrated with each other. In addition, for example, the removing unit244and the determining unit245, which are illustrated inFIG. 1, may be functionally or physically integrated with each other. For example, all or a part of the functional blocks of the image processing device200illustrated inFIG. 1can be functionally or physically separated or integrated on an arbitrary basis, depending on loads of the blocks and usage states of the blocks.

(3) Image Processing Program

The processes that are performed by the image processing device200described in the aforementioned embodiments can be each achieved by causing an electronic device such as an electronic circuit installed in an information processing device or an integrated circuit installed in the information processing device to execute a predetermined program, for example.

An example of a computer that executes the image processing program is described below with reference toFIG. 18, while the image processing program achieves substantially the same functions as the processes that are performed by the image processing device200described in the aforementioned embodiments.FIG. 18is a diagram illustrating an example of an electronic device that executes the image processing program.

As illustrated inFIG. 18, an electronic device300achieves the various types of the processes that are performed by the image processing device200. The electronic device300includes a central processing unit (CPU)310that executes various types of computational processing. As illustrated inFIG. 18, the electronic device300includes a camera interface320for acquiring an image obtained by a camera and a display interface330for transmitting and receiving various types of data between the electronic device300and a display. As illustrated inFIG. 18, the electronic device300includes a graphic engine340that functions as a hardware accelerator.

In addition, as illustrated inFIG. 18, the electronic device300includes a hard disk device350and a memory360. The hard disk device350stores data, a program and the like, while the program enables the CPU310to execute the various types of the processing. The memory360may be a random access memory (RAM) or the like, and temporarily stores various types of information. The devices310to360are connected to each other through a bus370.

An electronic circuit such as a mirco processing unit (MPU), or an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA), can be used instead of the CPU310. A semiconductor memory element such as a flash memory can be used instead of the memory360.

An image processing program351and image processing data352are stored in the hard disk device350. The image processing program351can achieve substantially the same functions as the functions of the image processing device200. The image processing program351can be distributed through a network to a storage unit of another computer and stored in the storage unit of the other computer, while the other computer is connected to the electronic device300through the network so that the electronic device300can communicate with the other computer.

The CPU310reads the image processing program351from the hard disk device350and loads the read image processing program351into the memory360, and whereby the image processing program351functions as an image processing process361as illustrated inFIG. 18. The image processing process361causes various types of data such as the image processing data352read from the hard disk device350to be loaded into regions that are arranged on the memory360and to which the data has been assigned. The image processing process361causes the various types of the processes to be performed on the basis of the various types of the loaded data.

The image processing process361includes the processes that are performed by the controller240of the image processing device200. For example, the image processing process361includes the processes described with reference toFIGS. 8,9,10,13,14,15and16.

The hard disk device350does not need to have the image processing program351stored therein. For example, the programs may be stored in a portable physical medium such as a flexible disk (FD), a CD-ROM, a DVD disc, a magneto-optical disc or an IC card, while a drive that supports the portable physical medium can be connected to an ECU that has the electronic device300installed therein. The electronic device300may read the programs from the portable physical medium and execute the programs.

In addition, the programs may be stored in another computer (or a server) that is connected through a public line, the Internet, a LAN, a WAN or the like to the ECU having the electronic device300installed therein, for example. The electronic device300may read the programs from the other computer (or the server) and execute the programs.

The embodiments can be implemented in computing hardware (computing apparatus) and/or software, such as (in a non-limiting example) any computer that can store, retrieve, process and/or output data and/or communicate with other computers. The results produced can be displayed on a display of the computing hardware. A program/software implementing the embodiments may be recorded on computer-readable media comprising computer-readable recording media. The program/software implementing the embodiments may also be transmitted over transmission communication media. Examples of the computer-readable recording media include a magnetic recording apparatus, an optical disk, a magneto-optical disk, and/or a semiconductor memory (for example, RAM, ROM, etc.). Examples of the magnetic recording apparatus include a hard disk device (HDD), a flexible disk (FD), and a magnetic tape (MT). Examples of the optical disk include a DVD (Digital Versatile Disc), a DVD-RAM, a CD-ROM (Compact Disc-Read Only Memory), and a CD-R (Recordable)/RW (ReWritable). An example of communication media includes a carrier-wave signal. The media described above does not include a transitory medium such as a propagation signal.

According to an aspect of the embodiments of the invention, any combinations of one or more of the described features, functions, operations, and/or benefits can be provided. A combination can be one or a plurality. In addition, an apparatus can include one or more apparatuses in computer network communication with each other or other apparatuses. In addition, a computer processor can include one or more computer processors in one or more apparatuses or any combinations of one or more computer processors and/or apparatuses. An aspect of an embodiment relates to causing one or more apparatuses and/or computer processors to execute the described operations.