Patent Description:
Human action recognition has a wide range of application prospects. For example, human action recognition may be applied to consumer behavior analysis, health care, smart factories and other scenarios.

In existing techniques, common action recognition methods include: performing human action recognition based on image frames captured by a camera, such as classifying image frames by using a classifier based on a deep learning network, so as to recognize human body actions in an image frame; or, extracting key points of the human body from the image frames captured by the camera, and recognizing human body actions according to the key points.

The following prior art documents disclosing action recognition are relevant to the subject matter of the present invention:.

According to the invention there is provided a device and a method for action recognition as set out in the appended claims.

With reference to the following description and drawings, the particular embodiments of this disclosure are disclosed in detail, and the principle of this disclosure and the manners of use are indicated. It should be understood that the scope of the embodiments of this disclosure is not limited thereto. The embodiments of this disclosure contain many alternations, modifications and equivalents within the scope of the terms of the appended claims.

It should be emphasized that the term "include/comprise" when used in this specification is taken to specify the presence of stated features, integers, steps or components but does not preclude the presence or addition of one or more other features, integers, steps, components or groups thereof.

The drawings are included to provide further understanding of this disclosure, which constitute a part of the specification and illustrate the preferred embodiments of this disclosure, and are used for setting forth the principles of this disclosure together with the description. It is obvious that the accompanying drawings in the following description are some embodiments of this disclosure, and for those of ordinary skills in the art, other accompanying drawings may be obtained according to these accompanying drawings without making an inventive effort. In the drawings:.

These and further aspects and features of this disclosure will be apparent with reference to the following description and attached drawings. In the description and drawings, particular embodiments of this disclosure have been disclosed in detail as being indicative of some of the ways in which the principles of this disclosure may be employed, but it is understood that this disclosure is not limited correspondingly in scope. Rather, this disclosure includes all changes, modifications and equivalents coming within the terms of the appended claims.

The first aspect of the embodiments provides an action recognition apparatus.

<FIG> is a schematic diagram of the action recognition apparatus of the first aspect of the embodiments of this disclosure. As shown in <FIG>, the action recognition apparatus <NUM> includes: a key-point feature extracting unit <NUM> and a first recognizing unit <NUM>.

The key-point feature extracting unit <NUM> is configured to process information on key points in a target human body in an image frame, to calculate key-point features of the target human body, the key-point features including a position of a predetermined key point and an angle of a predetermined joint in the target human body; and the first recognizing unit <NUM> is configured to recognize an action of the target human body according to the key-point features calculated by the key-point feature extracting unit <NUM>, and output a first recognition result.

According to the first aspect of the embodiments of this disclosure, the action recognition apparatus <NUM> recognizes an action of the human body according to the position of the key point and the angle of the joint in the human body in the image frame. The position of the key point and the angle of the joint are a part of the information in the image frame, hence, the first recognizing unit <NUM> may accurately classify with a relatively small amount of calculation without needing to use a deep learning network. And in comparison with a scheme where action recognition is performed based only on a position of a key point, the action recognition apparatus <NUM> of this disclosure performs action recognition according also to the angle of the joint of the human body, hence, an accuracy of recognition will be higher. And furthermore, if an action needing to be recognized needs to be added or changed, it may be realized by adjusting the predetermined key point and/or the predetermined joint, without needing a large amount of training. Hence, the action recognition apparatus <NUM> of this disclosure is relatively better in scalability and relatively higher in flexibility.

In at least one embodiment, the image frame may be, for example, an image frame in a video captured by a camera. Image frames may be in an order in a time sequence. For example, an n-th image frame corresponds to a moment Tn in the time sequence; where, n is a natural number.

In the following description of this disclosure, in a case of recognizing an action of the target human body in the n-th image frame, the n-th image frame is referred to as a current image frame, and an image preceding the current image frame in the time sequence is referred to as a historical image frame; for example, the historical image frame corresponds to a moment T(n-k) in the time sequence.

In at least one embodiment, the information on the key point in the target human body in the current image frame may be input to the key-point feature extracting unit <NUM>.

The information on the key point in the target human body in the current image frame may be, for example, coordinate values of the key points on the target human body in the current image frame, which may be denoted by the number of pixels in two mutually perpendicular directions; where, coordinate values of an upper leftmost pixel in the current image frame may be set to be (<NUM>, <NUM>), and the two mutually perpendicular directions may be a width direction and a height direction of the current image frame.

<FIG> is a schematic diagram of the key points in the target human body in the current image frame. As shown in <FIG>, the key points may be, for example, the nose <NUM>, the left shoulder <NUM>, the right shoulder <NUM>, the left hip <NUM>, the right hip <NUM>, the left knee <NUM>, the right knee <NUM>, the left ankle <NUM>, the right ankle <NUM>, the left elbow <NUM>, the right elbow <NUM>, the left wrist <NUM>, the right wrist <NUM>, the left eye <NUM>, the right eye <NUM>, the left ear <NUM>, and the right ear <NUM>, of the target human body. Distribution of the key points shown in <FIG> is an example only, and this disclosure is not limited thereto.

In at least one embodiment, the information on the key points in <FIG> may be obtained by performing target detection on the current image frame. For example, the target human body in the current image frame and body parts of the target human body are detected by using a template method, etc., a key point is set in each detected part to represent the part, and coordinate values of key points are outputted as the information on the key points in the target human body in the current image frame. For another example, a sensor may be set on a human body taken as a subject of a camera, and the information on the key points in the target human body in the current image frame is determined according to information sensed by the sensor.

In at least one embodiment, the key-point feature extracting unit <NUM> may select at least a part of the key points in the information on the target human body in the current image frame inputted therein as the predetermined key points, and process information on the predetermined key points.

<FIG> is a schematic diagram of the predetermined key points. As shown in <FIG>, the predetermined key points may be, for example, the nose <NUM>, the left shoulder <NUM>, the right shoulder <NUM>, the left hip <NUM>, the right hip <NUM>, the left knee <NUM>, the right knee <NUM>, the left ankle <NUM>, and the right ankle <NUM>, of the target human body. In addition, as shown in <FIG>, a joint <NUM> at the left hip <NUM>, a joint <NUM> at the right hip <NUM>, a joint <NUM> at the left knee <NUM>, and a joint <NUM> at the right knee <NUM>, may be predetermined joints.

What are shown in <FIG> are only examples of the predetermined key points and the predetermined joints. However, this disclosure is not limited thereto, and the predetermined key points and predetermined joints may be set according to actions needing to be recognized. For example, in a case where actions related to a torso and a leg of the human body need to be recognized, nine predetermined key points and four predetermined joints may be set as shown in <FIG>, and the nine key points may not be located in such parts as arms; and for example, in a case where other actions of the human body need to be recognized, such as when lifting or opening of arms needs to be recognized, the key points in the arm (such as the left elbow <NUM>, the right elbow <NUM>, the left wrist <NUM>, and the right wrist <NUM>, in <FIG>) may be set as predetermined key points, and joints at the left elbow <NUM> and the right elbow <NUM> may be set as predetermined joints.

In at least one embodiment, the key-point features of the target human body in the current image frame extracted by the key-point feature extracting unit <NUM> include the position of the predetermined key point and the angle of the predetermined joint in the target human body in the current image frame.

<FIG> is a schematic diagram of a method for calculating a key-point feature by the key-point feature extracting unit <NUM>. As shown in <FIG>, the method includes:.

In operation <NUM>, the height of the target human body in the current image frame may be H shown in <FIG>, that is, a difference between a coordinate value in a height direction of a key point where an eye of the target human body (such as the left eye <NUM> or the right eye <NUM> in <FIG>) in the image frame is located and a coordinate value in a height direction of a key point where an ankle of the target human body (such as the left ankle <NUM> or the right ankle <NUM> in <FIG>) in the image frame is located.

In operation <NUM>, a ratio of the height H of the target human body in the current image frame to a preset standard height H0 may be calculated, and the adjusted coordinate values may be obtained by dividing coordinate values of predetermined pixels of the target human body in the current image frame (coordinate values in the height direction and coordinate values in the width direction) by the ratio.

With operation <NUM>, the coordinate values of the predetermined key points of the target human body in the current image frame may be normalized, so as to avoid changes of the coordinate values of the predetermined key points due to a distance between the subject and the camera or a difference between field angles, thereby improving accuracy of recognition.

And this disclosure may not be limited thereto. In operation <NUM>, other methods may also be used to adjust the coordinate values of the predetermined key points. Furthermore, the coordinate values of the predetermined key points may also be adjusted according to other dimensions of the target human body.

In operation <NUM>, angles of a joint <NUM>, a joint <NUM>, a joint <NUM>, and a joint <NUM> may be calculated according to the adjusted coordinate values. The angle of the joint <NUM> is, for example, an included angle between a connecting line L1 of the left shoulder <NUM> and the left hip <NUM> and a connecting line L2 of the left hip <NUM> and the left knee <NUM>, in <FIG>, and angles of other joints are calculated similarly.

In at least one embodiment, the key-point features of the target human body in the current image frame extracted by the key-point feature extracting unit <NUM> are inputted into the first recognizing unit <NUM>, and the first recognizing unit <NUM> recognizes the action of the target human body according to the information on the key-point features, and outputs the first recognition result. The first recognizing unit <NUM> may perform the recognition based on a shallow learning network. Hence, accurate recognition may be performed with a relatively small amount of calculation.

For example, the first recognizing unit <NUM> may perform the recognition based on a multilayer perception machine model, which may include an input layer, two hidden layers and an output layer; the input layer may be inputted with a <NUM>*<NUM> vector, which contains, for example, the coordinate values in the height direction and the width direction of the nine predetermined key points and the angles of the four predetermined joints shown in <FIG>; the hidden layers may, for example, include <NUM> nodes, respectively; and the output layer may output probabilities of six types of actions, such as walking, standing, sitting, squatting, lying, and bending. An action with a highest probability is the first recognition result of the first recognizing unit <NUM>.

In this application, the multilayer perception machine model is an example only, and the first recognizing unit <NUM> may also perform recognition based on other models, such as a support vector machine (SVM) model.

The action recognition apparatus <NUM> further includes: a second recognizing unit <NUM>.

According to a parameter of motion of the predetermined key point in the target human body, and a second recognition result outputted by the second recognizing unit <NUM> for an action of the target human body in a historical image frame preceding the image frame (i.e. a historical second recognition result), the second recognizing unit <NUM> corrects the first recognition result of the first recognizing unit <NUM> for the action of the target human body in the current image frame, and outputs the second recognition result for the action of the target human body in the image frame.

In at least one embodiment, the motion parameter of the predetermined key point on the target human body may be obtained according to the adjusted position of the predetermined key point in the target human body in the current image frame and the adjusted predetermined key point in the target human body in the historical image frame. For example, as shown in <FIG>, the action recognition apparatus <NUM> may further include a key-point motion parameter calculating unit <NUM> configured to calculate motion vectors between each predetermined key point in the target human body in the image frame and a predetermined key point in the target human body in the historical image frame, and calculate the parameters of motion of the predetermined key points in the target human body according to the motion vectors of the predetermined key points.

The key-point feature extracting unit <NUM> may perform the processing as described in operation <NUM> of <FIG> on information that is the key points in the target human body in the historical image frame, to obtain the adjusted positions of the predetermined key points on the target human body in the historical image frame.

In at least one embodiment, the parameters of motion of the predetermined key points in the target human body include: a mean value of a difference between the coordinate values (i.e., the motion vectors) of the predetermined key points, and/or a standard variance of a difference between the coordinate values (i.e., the motion vectors) of the predetermined key points, and/or moving directions of the predetermined key points, and/or the number of the predetermined key points moving in a predetermined direction. However, this embodiment may not be limited thereto, and the motion parameters of the predetermined key points in the target human body may also be other parameters.

It should be noted that in the second recognizing unit <NUM> and the key-point motion parameter calculating unit <NUM>, the target human body in the current image frame and the target human body in the historical image frame correspond to the same photographed person, and the target human body corresponding to the same photographed person in the current image frame and the historical image frame may be determined based on a target tracking technique.

In addition, the second recognition result outputted by the second recognizing unit <NUM> for the current image frame may be inputted into the second recognizing unit <NUM> for performing action recognition of the target human body in a subsequent image frame after the current image frame.

<FIG> is a schematic diagram of adjusting the first recognition result by the second recognizing unit <NUM>. As shown in <FIG>, <NUM> denotes the first recognition result, <NUM> denotes the second recognition result, and <NUM> denotes bases for generating the second recognition result <NUM> by the second recognizing unit <NUM>.

As shown in <FIG>, the second recognition result may be, for example, walking, standing, sitting, squatting, lying, bending, running, crawling, getting up, falling down, and jumping, etc..

As shown in <FIG>, the bases <NUM>~<NUM> may be:.

The first recognition result <NUM>, the second recognition result <NUM> and the bases <NUM>~<NUM> shown in <FIG> are examples only, and the embodiments of this disclosure are not limited thereto.

With the second recognizing unit <NUM>, the recognition result of action may be adjusted based on a relationship between actions of the human body and time, hence, the action of the human body may be recognized more accurately.

In at least one embodiment, as shown in <FIG>, the action recognition apparatus <NUM> may further include an image frame interval determining unit <NUM> configured to determine an amount of image frames spacing the historical frame and the image frame apart according to an image frame acquisition rate (such as a frame rate of image frames taken by the camera, i.e. frames per second (FPS)).

In Table <NUM>, when the current image frame is an n-th frame, if the frame rate is <NUM>~<NUM>, the historical image frame is an (n-<NUM>)-th frame; if the frame rate is <NUM>~<NUM>, the historical image frame is an (n-<NUM>)-th frame; if the frame rate is <NUM>~<NUM>, the historical image frame is an (n-<NUM>)-th frame; if the frame rate is <NUM>~<NUM>, the historical image frame is an (n-<NUM>)-th frame; and if the frame rate is <NUM>~<NUM>, the historical image frame is an (n-<NUM>)-th frame.

Hence, in a case where the frame rate of the image frame changes, the amount of image frames spacing the historical frame and the image frame apart may be selected by the image frame interval determining unit <NUM>, so that the action recognition apparatus <NUM> is used while the above set thresholds (such as the first to the fourth thresholds concerned in <FIG>) are kept unchanged, thereby expanding an application scope of the action recognition apparatus <NUM> and improving its scalability.

According to the first aspect of the embodiments of this disclosure, the action recognition apparatus <NUM> recognizes the action of the human body based on the positions of the key points and the angles of joints in the human body in the image frame. As the positions of the key points and the angles of the joints are a part of the information in the image frame, the first recognizing unit <NUM> may perform accurate classification with a relatively small amount of calculation without needing to use a deep learning network; and in comparison with a scheme of performing action recognition based only on the positions of key points, the action recognition apparatus <NUM> of this disclosure may further perform action recognition based the angles of the joints in the human body, hence, an accuracy of recognition will be higher. And furthermore, if an action needing to be recognized needs to be added or changed, it is realized by adjusting the predetermined key point and/or the predetermined joint, without needing a large amount of training. Hence, the action recognition apparatus <NUM> of this disclosure is relatively good in scalability and relatively high in flexibility. And furthermore, as the action recognition apparatus <NUM> of this disclosure is able to adjust the action recognition results based on the relationship between actions of the human body and time, the action of the human body may be recognized more accurately.

The second aspect of the embodiments of this disclosure provides an action recognition method, corresponding to the action recognition apparatus described in the first aspect of the embodiments of this disclosure.

<FIG> is a schematic diagram of the action recognition method of the second aspect of the embodiments of this disclosure. As shown in <FIG>, the method includes:.

In at least one embodiment, operation <NUM> is carried out, according to options <NUM> and <NUM> in <FIG>.

As shown in <FIG>, the method further includes:
operation <NUM>: according to parameters of motion of the predetermined key points in the target human body, and an outputted second recognition result for an action of the target human body in a historical image frame preceding the image frame, the first recognition result is corrected, and a second recognition result for the action of the target human body in the image frame is outputted.

As shown in <FIG>, the method further includes:
operation <NUM>: motion vectors between each predetermined key point in the target human body in the image frame and a corresponding predetermined key point in the target human body in the historical image frame are calculated, and the parameters of motion are calculated according to the motion vectors of the predetermined key points.

In operation <NUM>, the parameters of motion include: a mean value of the motion vectors of the predetermined key points, and/or a standard variance between the motion vectors of the predetermined key points, and/or moving directions of the predetermined key points, and/or the number of the predetermined key points moving in a predetermined direction.

As shown in <FIG>, the method further includes:
operation <NUM>: an amount of image frames spacing the historical frame and the image frame apart is determined according to an acquisition rate of the image frame.

According to the second aspect of the embodiments of this disclosure, the action recognition method recognizes the action of the human body based on the positions of the key points and the angles of joints in the human body in the image frame. As the positions of the key points and the angles of the joints are a part of the information in the image frame, accurate classification may be performed with a relatively small amount of calculation without needing to use a deep learning network; and in comparison with a scheme of performing action recognition based only on the positions of key points, the action recognition method of this disclosure may further perform action recognition based the angles of the joints in the human body, hence, an accuracy of recognition will be higher. And furthermore, if an action needing to be recognized needs to be added or changed, it may be realized by adjusting the predetermined key point and/or the predetermined joint, without needing a large amount of training. Hence, the action recognition method of this disclosure is relatively good in scalability and relatively high in flexibility. And furthermore, as the action recognition method of this disclosure is able to adjust the action recognition results based on the relationship between actions of the human body and time, the action of the human body may be recognized more accurately.

The third aspect of the embodiments of this disclosure provides an electronic equipment, including the action recognition apparatus described in the first aspect of the embodiments of this disclosure.

<FIG> is a schematic diagram of a structure of the electronic equipment of the third aspect of the embodiments. As shown in <FIG>, an electronic equipment <NUM> may include a central processing unit (CPU) <NUM> and a memory <NUM>, the memory <NUM> being coupled to the central processing unit <NUM>. The memory <NUM> may store various data, and furthermore, it may store a program for control, and execute the program under control of the central processing unit <NUM>.

In one implementation, the functions of the action recognition apparatus <NUM> are integrated into the central processing unit <NUM>.

The central processing unit <NUM> is configured to carry out the action recognition method described in the second aspect of the embodiments.

Furthermore, as shown in <FIG>, the electronic equipment <NUM> may include an input unit <NUM>, and a display unit <NUM>, etc.; wherein, functions of the above components are similar to those in the related art, and shall not be described herein any further. It should be noted that the electronic equipment <NUM> does not necessarily include all the parts shown in <FIG>. Furthermore, the electronic equipment <NUM> may include parts not shown in <FIG>, and the related art may be referred to.

The action recognition apparatus described with reference to the embodiments of this disclosure may be directly embodied as hardware, software modules executed by a processor, or a combination thereof. For example, one or more functional block diagrams and/or one or more combinations of the functional block diagrams shown in the drawing may either correspond to software modules of procedures of a computer program, or correspond to hardware modules. Such software modules may respectively correspond to the steps shown in the first aspect of the embodiments. And the hardware module, for example, may be carried out by firming the soft modules by using a field programmable gate array (FPGA).

One or more functional blocks and/or one or more combinations of the functional blocks in the drawing may be realized as a universal processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware component or any appropriate combinations thereof carrying out the functions described in this application. And the one or more functional block diagrams and/or one or more combinations of the functional block diagrams in the drawing may also be realized as a combination of computing equipment, such as a combination of a DSP and a microprocessor, multiple processors, one or more microprocessors in communication combination with a DSP, or any other such configuration.

Claim 1:
An action recognition apparatus (<NUM>), characterized in that the action recognition apparatus comprises:
a key-point feature extracting unit (<NUM>) configured to process information on key points in a target human body in an image frame, to calculate key-point features of the target human body, the key-point features comprising a position of a predetermined key point and an angle of a predetermined joint in the target human body; and
a first recognizing unit (<NUM>) configured to recognize an action of the target human body according to the key-point features, and output a first recognition result;
wherein processing information on key points in a target human body in an image frame by the key-point feature extracting unit (<NUM>) comprises:
adjusting coordinate values of the predetermined key point in the target human body in the image frame according to a height of the target human body in the image frame; and
calculating the angle of the predetermined joint according to adjusted coordinate values of the predetermined key point in the target human body in the image frame,
wherein,
the height of the target human body in the image frame is a difference between a coordinate value in a height direction of a key point where an eye of the target human body in the image frame is located and a coordinate value in a height direction of a key point where an ankle of the target human body in the image frame is located,
the action recognition apparatus further comprises:
a second recognizing unit configured to, according to a parameter of motion of the predetermined key point in the target human body, and/or a second recognition result outputted by the second recognizing unit for an action of the target human body in a historical image frame preceding the image frame, correct the first recognition result, and output a second recognition result for the action of the target human body in the image frame.