Image identification method and image identification device based on fingertip tracking

An image identification method and an image identification device are provided. The method comprises acquiring a hand feature region within a sight from a first view by skin color detection; acquiring a feature and a position of a tip of a finger from the hand feature region by performing a pattern recognition for a morphological feature of a stretched hand; recording an interframe displacement of a feature point of the tip of the finger when the tip of the finger delimits a periphery of a target object to obtain a delimiting trajectory from the interframe displacement, closing the delimiting trajectory to form a full-perimeter geometry; projecting the full-perimeter geometry on a plane where a direction of the sight is perpendicular to a plane where the target object is located to obtain a projection region, performing an image identification using the projection region as an identification region of the target object.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to and benefits of Chinese Patent Application Serial No. 201410594523.3, filed with the State Intellectual Property Office of P. R. China on Oct. 29, 2014, the entire content of which is incorporated herein by reference.

FIELD

The present disclosure relates to the field of Internet, and more particularly to an image identification method and an image identification device.

BACKGROUND

In the related art, the input for image identification and machine vision recognition is performed by first taking pictures and then determining a target object. Specifically, pictures are first taken, foreground environment images and background environment images of the target object are saved, and then the target object is selected by delimiting the target object on a screen by a finger and segmented so as to perform the image identification. In this way, the pictures shot need to subject to human intervention, for example, the pictures need to be delimited manually, such that the operation step is complicated and the experience of a user is not smooth. In addition, the above process is only applicable to smart terminals with touch screens, and thus the applicability is poor.

SUMMARY

Embodiments of the present disclosure seek to solve at least one of the problems existing in the related art to at least some extent.

According to a first aspect of the present disclosure, an image identification method is provided. The image identification method comprises acquiring, at one or more computing devices, a hand feature region within a sight from a first view by a skin color detection, and capturing and tracking the hand feature region in real time; acquiring, at the one or more computing devices, a feature and a position of a tip of a finger from the hand feature region by performing a pattern recognition for a morphological feature of a stretched hand, and capturing and tracking the feature and the position of the tip of the finger in real time; recording, using the one or more computing devices, an interframe displacement of a feature point of the tip of the finger when the tip of the finger delimits a periphery of a target object to obtain a delimiting trajectory from the interframe displacement, and closing the delimiting trajectory to form a full-perimeter geometry; projecting, using the one or more computing devices, the full-perimeter geometry on a plane where a direction of the sight is perpendicular to a plane where the target object is located so as to obtain a projection region; and performing, using the one or more computing devices, an image identification using the projection region as an identification region of the target object.

With the image identification method according to embodiments of the present disclosure, the hand feature region within the sight from the first view of a user is acquired by the skin color detection, and the feature and the position of the tip of the finger are acquired from the hand feature region by performing the pattern recognition for the morphological feature of the stretched hand. The interframe displacement of the feature point of the tip of the finger is recorded when the tip of the finger delimits the periphery of the target object to obtain the delimiting trajectory from the interframe displacement, and the delimiting trajectory is closed to form the full-perimeter geometry. Then, the full-perimeter geometry is projected on the plane where the direction of the sight of the user is perpendicular to the plane where the target object is located so as to obtain the projection region, and the image identification is performed using the projection region as the identification region of the target object. Thus, when a user performs the image identification on the target object, he/she only needs to stretch out a finger and delimit the periphery of the target object with the tip of the finger directed at the periphery of the target object so as to perform the delimiting and image segmentation on the target object quickly, which may be used as an identification input for the image identification of the target object, such that the operation is simple, the experience of the user is better, and the applicability is high.

According to a second aspect of the present disclosure, an image identification device is provided. The image identification device comprises one or more computing devices configured to execute one or more software modules, the one or more software modules including: a first acquiring module configured to acquire a hand feature region within a sight from a first view by a skin color detection; a first tracking module configured to capture and track the hand feature region in real time; a second acquiring module configured to acquire a feature and a position of a tip of a finger from the hand feature region by performing a pattern recognition for a morphological feature of a stretched hand; a second tracking module configured to capture and track the feature and the position of the tip of the finger in real time; a recording module configured to record an interframe displacement of a feature point of the tip of the finger when the tip of the finger delimits a periphery of a target object to obtain a delimiting trajectory from the interframe displacement, and to close the delimiting trajectory to form a full-perimeter geometry; and a projecting module configured to project the full-perimeter geometry on a plane where a direction of the sight is perpendicular to a plane where the target object is located so as to obtain a projection region, and to perform an image identification using the projection region as an identification region of the target object.

With the image identification device according to embodiments of the present disclosure, the first acquiring module acquires the hand feature region within the sight of a user from the first view by the skin color detection, and the second acquiring module acquires the feature and the position of the tip of the finger from the hand feature region captured and tracked by the first tracking module by performing the pattern recognition for the morphological feature of the stretched hand. The recording module records the interframe displacement of the feature point of the tip of the finger when the tip of the finger delimits the periphery of the target object to obtain the delimiting trajectory from the interframe displacement, and closes the delimiting trajectory to form the full-perimeter geometry. Then, the projecting module projects the full-perimeter geometry on the plane where the direction of the sight is perpendicular to the plane where the target object is located so as to obtain the projection region, and performs the image identification using the projection region as the identification region of the target object. Thus, when a user performs the image identification on the target object, he/she only needs to stretch out a finger and delimit the periphery of the target object with the tip of the finger directed at the periphery of the target object so as to perform the delimiting and image segmentation on the target object quickly, which may be used as an identification input for the image identification of the target object, such that the operation is simple, the experience of the user is better, and the applicability is high.

According to a third aspect of the present disclosure, a non-transitory computer readable storage medium is provided. The computer readable storage medium comprises a computer program for executing the image identification method according to the first aspect of the present disclosure, when running on a computer.

DETAILED DESCRIPTION

FIG. 1is a flow chart of an image identification method according to an embodiment of the present disclosure.

As shown inFIG. 1, the image identification method comprises the following steps that may be performed by one or more computing devices.

In step101, a hand feature region within a sight from a first view is acquired by a skin color detection, and the hand feature region is captured and tracked in real time.

Further, before the hand feature region within the sight from the first view is acquired by the skin color detection, an image identification request input via speech, a touch screen or a dynamic sensor is received so as to trigger an operation of acquiring the hand feature region within the sight from the first view by the skin color detection.

In step102, a feature and a position of a tip of a finger are acquired from the hand feature region by performing a pattern recognition for a morphological feature of a stretched hand, and the feature and the position of the tip of the finger are captured and tracked in real time.

In step103, an interframe displacement of a feature point of the tip of the finger is recorded when the tip of the finger delimits a periphery of a target object to obtain a delimiting trajectory from the interframe displacement, and the delimiting trajectory is closed to form a full-perimeter geometry.

Specifically, closing the delimiting trajectory to form the full-perimeter geometry comprises: closing the delimiting trajectory to form the full-perimeter geometry by a closed curve optimization algorithm.

In step104, the full-perimeter geometry is projected on a plane where a direction of the sight is perpendicular to a plane where the target object is located so as to obtain a projection region, and an image identification is performed using the projection region as an identification region of the target object.

With the image identification method according to embodiments of the present disclosure, the hand feature region within the sight from the first view of a user is acquired by the skin color detection, and the feature and the position of the tip of the finger are acquired from the hand feature region by performing the pattern recognition for the morphological feature of the stretched hand. The interframe displacement of the feature point of the tip of the finger is recorded when the tip of the finger delimits the periphery of the target object to obtain the delimiting trajectory from the interframe displacement, and the delimiting trajectory is closed to form the full-perimeter geometry. Then, the full-perimeter geometry is projected on the plane where the direction of the sight of the user is perpendicular to the plane where the target object is located so as to obtain the projection region, and the image identification is performed using the projection region as the identification region of the target object. Thus, when a user performs the image identification on the target object, he/she only needs to stretch out a finger and delimit the periphery of the target object with the tip of the finger directed at the periphery of the target object so as to perform the delimiting and image segmentation on the target object quickly, which may be used as an identification input for the image identification of the target object, such that the operation is simple, the experience of the user is better, and the applicability is high.

FIG. 2is a schematic diagram of an image identification method according to another embodiment of the present disclosure.

As shown inFIG. 2, in this embodiment, after a smart device with a camera receives an image identification request input by a user via speech, a touch screen or a dynamic sensor, the camera on the smart device is turned on. A hand feature region within a sight of the user from a first view is acquired by a skin color detection, and the hand feature region is captured and tracked in real time. The user is guided by an interactive means such as speech to identify and capture a feature point of a tip of a finger, and then guided to complete a gesture of delimiting a periphery of a target object. When the tip of the finger delimits the periphery of the target object, a delimiting path of the tip of the finger is tracked, an interframe displacement of the feature point of the tip of the finger is recorded to obtain a delimiting trajectory from the interframe displacement, and the delimiting trajectory is closed to form a full-perimeter geometry. The full-perimeter geometry is projected on a plane where a direction of the sight of the user is perpendicular to a plane where the target object is located so as to obtain a projection region, and an image identification is performed using the projection region as an identification region of the target object. Thus, an image segmentation and identification are performed in a natural delimiting behavior, and then the identification result is output to the user via an audio output channel or a screen image output channel of the smart device so as to complete the identification of the target object.

In some embodiments, the target object is an object at a far distance from the user, an object the user cannot touch, a large object, or a fixed object which cannot be moved. However, the present disclosure is not limited thereby, and the particular morphology of the target object is not limited in the present disclosure.

In some embodiments, the smart device with the camera may be a smart mobile terminal with a camera, a headset wearable device (e.g., Google Glass or BaiduEye) with a camera, a smart watch with a camera, or a robot with a camera visual input. However, the present disclosure is not limited thereby.

FIG. 3is a block diagram of an image identification device according to an embodiment of the present disclosure. The image identification device in this embodiment may be used to realize the image identification method as shown inFIG. 1.

As shown inFIG. 3, the image identification device comprises a first acquiring module31, a first tracking module32, a second acquiring module33, a second tracking module34, a recording module35and a projecting module36. The modules31-36may be configured to be executed by one or more computing devices.

The first acquiring module31is configured to acquire a hand feature region within a sight from a first view by a skin color detection.

The first tracking module32is configured to capture and track the hand feature region acquired by the first acquiring module31in real time.

The second acquiring module33is configured to acquire a feature and a position of a tip of a finger from the hand feature region captured and tracked by the first tracking module32by performing a pattern recognition for a morphological feature of a stretched hand.

The second tracking module34is configured to capture and track the feature and the position of the tip of the finger in real time.

The recording module35is configured to record an interframe displacement of a feature point of the tip of the finger when the tip of the finger delimits a periphery of a target object to obtain a delimiting trajectory from the interframe displacement, and to close the delimiting trajectory to form a full-perimeter geometry. In some embodiments, the recording module35closes the delimiting trajectory to form the full-perimeter geometry by a closed curve optimization algorithm.

The projecting module36is configured to project the full-perimeter geometry on a plane where a direction of the sight is perpendicular to a plane where the target object is located so as to obtain a projection region, and to perform an image identification using the projection region as an identification region of the target object.

In some embodiments, the image identification device may be a smart device with a camera, or a part of the smart device with the camera, in which a part or all of the functions of the first acquiring module31, the first tracking module32, the second acquiring module33, the second tracking module34, and/or the recording module35may be performed by the camera of the smart device.

In some embodiments, the smart device with the camera may be a smart mobile terminal with a camera, a headset wearable device (e.g., Google Glass or BaiduEye) with a camera, a smart watch with a camera, or a robot with a camera visual input. However, the present disclosure is not limited thereby.

With the image identification device according to embodiments of the present disclosure, the first acquiring module31acquires the hand feature region within the sight of a user from the first view by the skin color detection, and the second acquiring module33acquires the feature and the position of the tip of the finger from the hand feature region captured and tracked by the first tracking module32by performing the pattern recognition for the morphological feature of the stretched hand. The recording module35records the interframe displacement of the feature point of the tip of the finger when the tip of the finger delimits the periphery of the target object to obtain the delimiting trajectory from the interframe displacement, and closes the delimiting trajectory to form the full-perimeter geometry. Then, the projecting module36projects the full-perimeter geometry on the plane where the direction of the sight is perpendicular to the plane where the target object is located so as to obtain the projection region, and performs the image identification using the projection region as the identification region of the target object. Thus, when a user performs the image identification on the target object, he/she only needs to stretch out a finger and delimit the periphery of the target object with the tip of the finger directed at the periphery of the target object so as to perform the delimiting and image segmentation on the target object quickly, which may be used as an identification input for the image identification of the target object, such that the operation is simple, the experience of the user is better, and the applicability is high.

FIG. 4is a block diagram of an image identification device according to another embodiment of the present disclosure.

The image identification device in this embodiment differs from that shown inFIG. 3in that the image identification device shown inFIG. 4further comprises a receiving module37.

The receiving module37is configured to receive an image identification request input via speech, a touch screen or a dynamic sensor so as to trigger an operation of acquiring the hand feature region within the sight from the first view by the skin color detection.

In other words, in this embodiment, after the receiving module37receives the image identification request input by a user via speech, the touch screen or the dynamic sensor, a camera is turned on. The first acquiring module31acquires a hand feature region within a sight of the user from a first view by a skin color detection, and the first tracking module32captures and tracks the hand feature region in real time. The user is guided by an interactive means such as speech to identify and capture a feature point of a tip of a finger, and then guided to complete a gesture of delimiting a periphery of a target object. When the tip of the finger delimits the periphery of the target object, the second tracking module34tracks a delimiting path of the tip of the finger, the recording module35records an interframe displacement of the feature point of the tip of the finger to obtain a delimiting trajectory from the interframe displacement, and closes the delimiting trajectory to form a full-perimeter geometry. The projecting module36projects the full-perimeter geometry on a plane where a direction of the sight of the user is perpendicular to a plane where the target object is located so as to obtain a projection region, and performs an image identification using the projection region as an identification region of the target object. Thus, an image segmentation and identification are performed in a natural delimiting behavior, and then the identification result is output to the user via an audio output channel or a screen image output channel of the smart device so as to complete the identification of the target object.

With the image identification device according to embodiments of the present disclosure, when a user performs the image identification on the target object, he/she only needs to stretch out a finger and delimit the periphery of the target object with the tip of the finger directed at the periphery of the target object so as to perform the delimiting and image segmentation on the target object quickly, which may be used as an identification input for the image identification of the target object, such that the operation is simple, the experience of the user is better, and the applicability is high.

Image identification is the most important breakthrough in artificial intelligence, and has quite a wide range of valuable use scenes, which is the top priority in the future search field. The present disclosure employs a new human-computer interaction and a natural user interface, which is more natural and more convenient compared with the conventional image identification by first taking pictures and then delimiting the target object. Natural interaction is particularly important for the image identification function of a smart wearable device, and in the present disclosure, the user may directly delimit a remote object in front of he/she to obtain a search result, which may realize a more natural interaction and a better experience of the user.

The image identification method according to embodiments of the present disclosure may be applied to various smart devices performing visual image information capturing and processing from the first view, and may provide a basic technical support for the gesture recognition of smart wearable vision products.

A non-transitory computer readable storage medium according to embodiments of the present disclosure is also provided. The computer readable storage medium comprises a computer program for executing the image identification method according to the above embodiments of the present disclosure, when running on a computer.

It will be understood that, the flow chart or any process or method described herein in other manners may represent a module, segment, or portion of code that comprises one or more executable instructions to implement the specified logic function(s) or that comprises one or more executable instructions of the steps of the progress. Although the flow chart shows a specific order of execution, it is understood that the order of execution may differ from that which is depicted. For example, the order of execution of two or more boxes may be scrambled relative to the order shown.