Patent Description:
Conventionally, techniques for tracking an object or a human body detected from an image captured by a camera have been provided. For example, a technique for detecting an object from motion vectors, estimating a search point in the next frame, and performing object tracking by template matching is discussed in, for example, <CIT>. Also, a technique for performing detection of a face, and performing face tracking using movement information detected from a correlation between a current frame and a past frame is discussed in, for example, <CIT>.

However, in the image captured by the camera, the face and the human body of the object are not always captured in a direction features of the face and the human body are easily detected. Due to the problem, it is difficult to keep detecting the face and the human body. With the problem, if the face detection or the human body detection is performed, an appropriate tracking result may not be obtained. Moreover, in the processing of detecting and tracking an object, the target of the tracking processing is not specified to a specific object. Accordingly, the tracking technique is not suitable for analyzing the movement of the specific object.

Document by <NPL>, and related document <NPL>, disclose human tracking in surveillance videos using both moving blob detection and shape based human detection. Furthermore, patent document <CIT> discloses human hand detection employing motion and skin color/shape detection.

The present disclosure is directed to accurately tracking a specific object in an image, such as the face, in which detection of features as the specific object is difficult.

The present invention in its first aspect provides an image processing apparatus as specified in claim <NUM>.

The present invention in its second aspect provides a computer-implemented image processing method as specified in claim <NUM>.

The present invention in its third aspect provides a computer program as specified in claim <NUM>.

Further features and aspects of the present invention will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings.

The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the invention and, together with the description, serve to explain the principles of the invention.

Various exemplary embodiments, features, and aspects of the invention will be described in detail below with reference to the drawings.

Configurations described in the following exemplary embodiments are only examples, and the present invention is not limited to the configurations illustrated in the drawings, but to the appended claims.

A first exemplary system is described. <FIG> illustrates a system configuration of an image processing apparatus <NUM> that serves as a tracking apparatus. The image processing apparatus <NUM> that serves as the tracking apparatus includes an image acquisition unit <NUM>, an object detection unit <NUM>, an object tracking unit <NUM>, a human body detection unit <NUM>, an object association unit <NUM>, and a trajectory management unit <NUM>. The image processing apparatus <NUM> that serves as the tracking apparatus further includes a determination parameter setting unit <NUM>, a trajectory information determination unit <NUM>, and an external output unit <NUM>.

The image acquisition unit <NUM> time-sequentially acquires images to be processed from an external device. The external device is, for example, a camera or a server. Alternatively, an image stored in an external memory, or the like can be acquired.

The object detection unit <NUM> detects an object from the images acquired by the image acquisition unit <NUM> using a background subtraction technique. The object detected in the processing is, for example, a moving object. The object can be a foreground detected by the background subtraction technique, or a part that is determined not to be a background. The information of the detected object includes the position on the image, a bounding box, and the size of the object. In the object detection unit <NUM>, the processing for detecting the object from the image is performed, and, in an example not covered by the claims, the processing method is not limited to the background subtraction technique.

The object tracking unit <NUM> associates the object detected by the object detection unit <NUM> with an object detected in a previous image, and tracks the detected object. To a newly appeared object, a new object identification (ID) (object identifier) is given. To the object that is associated with the previous image, an obj ect ID that has been given in the previous image is given.

The human body detection unit <NUM> detects a human body from the region where the object detection unit <NUM> has detected the object using pattern matching processing. The processing in the human body detection unit <NUM> is for detecting a human body from an image, and the processing is not limited to the pattern matching processing. Moreover, it is not always necessary to detect a human body from the region detected by the object detection unit <NUM>. The human body detection processing can be performed to the whole image. In examples not covered by the claims, the detection target is not limited to the human body, but a specific object (an object that has a specific character or an object that is determined to have a specific pattern) can be detected. For example, the detection can be performed to determine that the object is an automobile or an animal.

The object association unit <NUM> establishes an association between the object detected by the object detection unit <NUM> and the human body detected by the human body detection unit <NUM>.

<FIG> illustrate examples of the association between the detected object and the detected human body. <FIG> illustrates a case where a bounding box <NUM> of the detected object does not include a bounding box <NUM> of the detected human body. The bounding box <NUM> is a bounding box of the object detected by the object detection unit <NUM>. The bounding box <NUM> of the human body is a bounding box of the human body detected by the human body detection unit <NUM>. If an overlap ratio of the bounding box <NUM> of the human body to the bounding box <NUM> of the object exceeds a predetermined threshold, it is determined that the detected object and the detected human body are associated with each other. The overlap ratio is a ratio of an area of an overlapping part of the bounding box <NUM> of the object to the bounding box <NUM> of the human body to the area of the bounding box <NUM> of the human body.

The human body detection unit <NUM> extracts a feature amount of a part expected to be a human body in an image, and determines whether the part is a human body. If the human body detection unit <NUM> can adequately determine that the part is a human body from only the feature amount of the right side of the body, the human body detection unit <NUM> does not extract the shape of the left side of the body, determines the shape of the human body by estimation, and outputs the bounding box of the human body. By the operation, in the case where the feature amount as the human body can be adequately extracted in the bounding box of the moving body, and the feature amount of only the right side of the human body is included in the circumscribed rectangle of the moving body, the human body detection unit <NUM> estimates the shape of the left side of the human body and outputs the human body bounding box. Accordingly, in such a case, the left side of the human body is not included in the bounding box of the moving body.

<FIG> illustrates a case where a plurality of human bodies are detected from a bounding box <NUM> of a detected object. In this case, if the individual overlap ratios of a bounding box <NUM> of a human body and a bounding box <NUM> of a human body to the bounding box <NUM> of the object exceed a predetermined threshold respectively, it is determined that the detected obj ect and the detected human bodies are associated with each other respectively. In such a case, the bounding boxes <NUM> and <NUM> of the human bodies are associated with the bounding box <NUM> of the object, and it is determined that the object of the bounding box <NUM> includes the two human bodies.

The trajectory management unit <NUM> manages movement information of an object tracked by the object tracking unit <NUM>. Further, the trajectory management unit <NUM> manages information of a human body associated with the object by the object association unit <NUM> as an attribute of the object. In other words, the trajectory management unit <NUM> manages the association of the object being tracked with the human body (specific object) by the object association unit <NUM>.

<FIG> illustrates an example of the movement information management of an obj ect by the trajectory management unit <NUM>. Movement information (tracking information) <NUM> of the object is managed as object information <NUM> for each object ID. The object information <NUM> includes the time the object is detected first, and a group of position information <NUM> including the time the object is detected, the position on the screen, the bounding box, the size, the attribute of the object, and the attribute count. The attribute of the object is "no-attribute" or "human body". The attribute count is the number of human bodies associated with an object by the object association unit <NUM>. In the example in <FIG>, the number of human bodies is "<NUM>". For example, in the object information <NUM> of an object ID "<NUM>", the time the object is detected first is <NUM>/<NUM>/<NUM>/<NUM>:<NUM>:<NUM>. When the object is detected at <NUM>/<NUM>/<NUM>/<NUM>:<NUM>:<NUM>, the center position of the bounding box is (<NUM>, <NUM>), and the bounding box is expressed by (<NUM>, <NUM>) and (<NUM>, <NUM>). The size of the object is <NUM>. The attribute is "human body", and the number of human bodies is "<NUM>".

By detecting a human body by the human body detection unit <NUM> and using a result of the object association unit <NUM>, the corresponding attribute and the attribute count of the position information of the object are updated. Further, the attribute and the attribute count of the past position information is updated using the result. Further, to an attribute and the attribute count of future position information, the result is set.

In the above example, the position information includes the attribute and the attribute count, however, the information of the attribute or the attribute count can be directly associated with the object information <NUM>. For example, at a time point the attribute of the object is detected to be "human body", the object ID of the object information <NUM> indicating the object can be associated with the attribute of "human body". By associating the object ID with the attribute of "human body", the current, past, and future position information included in the object ID and the attribute of "human body" can be associated with each other. In addition to the attribute, similarly, the attribute count can be directly associated with the object information <NUM>.

When the human body detection unit <NUM> detects the object being tracked by the object tracking unit <NUM> to be the human body by the above-described processing, the trajectory management unit <NUM> can manage the trajectory of the object tracked at a past time point before the time point when the object being tracked is detected to be a human body as the trajectory of the human body.

The case where the human body detection unit <NUM> detects the object being tracked by the object tracking unit <NUM> to be the human body includes the case described with reference <FIG> in which the bounding box <NUM> of the detected object does not include the bounding box <NUM> of the detected human body. In other words, the object being tracked by the object tracking unit <NUM> is simply to be associated with the human body by the object association unit <NUM>, and the object being tracked can be a part of the human body.

The case where the human body detection unit <NUM> detects the object being tracked by the object tracking unit <NUM> to be the human body also includes the case described with reference <FIG> in which the object of the bounding box <NUM> is associated with the human bodies of two people. In other words, the case where the object being tracked by the object tracking unit <NUM> is associated with the plurality of human bodies by the object association unit <NUM> is included.

Further, the case where the trajectory of the object being tracked is managed as the trajectory of the human body includes the case where only a part of the trajectory of the object being tracked is associated as the human body.

<FIG> illustrates an example of the position information update. For example, at a point <NUM>, if a human body is associated with an object, by setting an attribute and the attribute count associated at the point <NUM> to the following position information, a trajectory <NUM> is managed as the object having the human body attribute. Further by setting the attribute and the attribute count associated at the point <NUM> to past position information, a trajectory <NUM> can be managed as the object having the human body attribute. The trajectories <NUM> , <NUM> and the point <NUM> correspond to the center positions of bounding boxes of the object detected at a predetermined time interval. When a human body is associated with the object at the point <NUM>, the information of the attribute of "human body" is stored in the position information indicating the point <NUM>. Further, the information of the attribute of "human body" is stored in the position information indicating the trajectory <NUM>. In <FIG>, reference numeral <NUM> denotes a screen, and reference numeral <NUM> denotes the object. As described above, in the case where the object detected by the object detection unit <NUM> and the human body detected by the human body detection unit <NUM> are associated with each other, the trajectory management unit <NUM> manages the information of the positions where the obj ect existed by adding the attribute of "human body". In this example, the information of the positions the object existed at the time points before the time point (the time point the object existed at the point <NUM>) when the association of the object and the human body is performed is associated with the human body and managed. The past time points are time points the object moved on the trajectory <NUM>.

As described above, in the exemplary embodiment, the attribute of the object is determined by the human body detection unit <NUM>, however, in examples not covered by the claims it is not limited to the above-described case. As the attribute, any attributes such as a face, an automobile, and an animal that can be used for identifying and determining the object can be employed. Further, if a plurality of types of specific object detection units are provided and an object includes a plurality of attributes simultaneously, the object can have the plurality of types and the attribute count. In other words, as the attribute of the object of the position information <NUM>, attributes of "human body", "automobile", "animal" and the like can be stored. Further, as the attribute count in the position information <NUM>, the number of detected objects can be stored respectively for each of the attributes of "human body", "automobile" and "animal". For example, when one object detected by the object detection unit <NUM> includes two human bodies and one automobile, the information that the attributes of the object whose trajectory is managed are "human body" and "automobile", the number of the human bodies is two, and the number of the automobiles is one is managed.

The determination parameter setting unit <NUM> externally sets determination parameters for determining traj ectory information of an object to the traj ectory information determination unit <NUM>. The determination parameters includes a pair of coordinates indicating a position for performing the determination, a region type for determining whether the pair of coordinates belongs to "region" or "line", a range of an object size, an object attribute, the attribute count, a range of the life time in the screen, and a detection event. <FIG> illustrates an example of the determination parameters. In the example, a passage of an object through the line connecting the coordinate (<NUM>, <NUM>) and the coordinate (<NUM>, <NUM>) is detected, in which, the object has a size from <NUM> to <NUM>, an attribute of "human body", the number of attributes is one, and the life time is one second or more.

<FIG> illustrates an example of the line passage determination. The determination of the passage is performed by determining whether a line segment <NUM> connecting the center point of a bounding box <NUM> of an object in a previous image and the center point of a bounding box <NUM> of the object in a current image intersects a determination line segment <NUM>. The line segment <NUM> is a moving vectorof the object. In the example, the passage is determined using the center points of the bounding boxes. Alternatively, the passage can be determined by detecting one of the bounding boxes passes the determination line segment <NUM>. If the determination parameter is set to "region", whether the object enters the region or the object comes out of the region is determined. The specification for detecting whether the object enters the region or the object comes out of the region can be made by a detection event in the determination parameters.

The trajectory information determination unit <NUM> detects a target object by determining the trajectory information of the object managed by the trajectory management unit <NUM> using the determination parameters set in the determination parameter setting unit <NUM>. The determination processing of the trajectory information is performed to all of the set determination parameters each time the trajectory information is updated in the trajectory management unit <NUM>. For example, the trajectory information determination unit <NUM> determines a passage of an object being tracked across a specific place.

By the processing, in the case the determination is performed using the determination parameters illustrated in <FIG>, the following determination can be performed. In other words, at a time point the object crosses the determination line segment <NUM>, if the attribute of "human body" is not associated with the object, after the time point, at a time point the attribute of "human body" is associated with the object, the passage of the object to which the attribute of "human body" is associated through the line segment can be detected. The association of the attribute of "human body" is performed by the object association unit <NUM>.

The external output unit <NUM> outputs the trajectory information of the object managed by the trajectory management unit <NUM> and the determination result of the trajectory information determination unit <NUM> to the outside.

<FIG> is a flowchart of the processing. In step f1, whether the processing is to be continued or finished is determined. When the processing is to be continued (YES in step f1), in step f2, the image acquisition unit <NUM> acquires an image from the outside. In step f3, the object detection unit <NUM> performs the object detection processing to the acquired image. In step f3, for example, the object detection unit <NUM> compares two images with each other, and determines that an object exists in different parts. In step f4, determination of the result in the object detection processing is performed. If an object is detected (YES in step f4), the processing proceeds to step f5, and if the object is not detected (NO in step f4), the processing returns to step f1.

If the object is detected(YES in step f4), in step f5, the object tracking unit <NUM> performs the tracking processing of the object. In step f6, using the tracking processing result, the trajectory management unit <NUM> updates the movement information (tracking information). In step f5, for example, in a case where the object detection unit <NUM> detects an object in a first image and detects an object in a second image, if the object tracking unit <NUM> determines that the distance between those two objects is closer than a predetermined distance, the object tracking unit <NUM> determines that those two objects are the same object. The trajectory management unit <NUM> manages the trajectory of the object being tracked by storing the individual positions of the object detected by the object detection unit <NUM> at each time point. The movement information managed by the traj ectory management unit <NUM> is described above with reference to <FIG>.

In step f7, the human body detection unit <NUM> performs the human body detection processing for detecting a human body from the region the object detection unit <NUM> detected the obj ect. In step f8, the human body detection unit <NUM> determines the result. If a human body is detected (YES in step f8), the processing proceeds to step f9. If a human body is not detected (YES in step f8), the processing proceeds to step f11. The human body detection performed in step f8 is an example of the detection of a specific object. The specific object to be detected in step f8 is not limited to a human body. For example, an animal or an automobile can be detected.

If the human body is detected (YES in step f8), in step f9, the object association unit <NUM> performs the processing to associate the object with the human body. In step f10, using the result, the trajectory management unit <NUM> updates the movement information (tracking information). If the object detected by the object detection unit <NUM> overlaps with the human body detected by the human body detection unit <NUM> at a predetermined ratio or more, the object association unit <NUM> determines that the object detected by the object detection unit <NUM> is a human body, and adds an attribute of "human body" to the object detected by the object detection unit <NUM>. Instead of the human body detection, if the object is determined to be an automobile or an animal, a corresponding attribute is added to the object detected by the object detection unit <NUM>. The trajectory management unit <NUM> stores the information that the object being tracked by the object tracking unit <NUM> is determined to be the human body by the human body detection unit <NUM>, and the attribute is associated with the object by the object association unit <NUM> as the movement information, and manages the trajectory and the attribute of the object. The attribute indicates that the object is a specific object.

As described above, if the object detected by the object detection unit <NUM> is associated with the human body detected by the human body detection unit <NUM>, the trajectory management unit <NUM> manages the object as described below. In other words, the trajectory management unit <NUM> associates the information of the positions existed at time points before the time point (the time point corresponding to the point <NUM> in <FIG>) when the object is associated with the human body with the human body and manages the information. The associated time points before the time point (the time point corresponding to the point <NUM> in <FIG>) are time points corresponding to the trajectory <NUM> in <FIG>. Accordingly, the trajectory of the movement of the specific object such as the human body can be accurately managed.

Further, in step f11, the trajectory information determination unit <NUM> performs the determination processing of the trajectory information. In step f12, the result of the determination and the trajectory information of the object managed by the trajectory management unit <NUM> is output to the outside. The trajectory information determination unit <NUM>, for example, determines that the object being tracked passes through a specific place. The trajectory management unit <NUM> can include the function of the trajectory information determination unit <NUM>, and perform the determination of the trajectory information.

As described above, in the face of an object or an image which is hard to be detected as a feature of a human body, by performing the human body tracking and further updating the tracking information, accurate counting of the human body can be performed.

Other exemplary embodiments are described. In implementing the above-described example, the trajectory of an object being tracked by the object tracking unit <NUM> can be displayed on a captured image. In such a case, if the trajectory management unit <NUM> associates the object being tracked with a specific object and manages the information, the display of the trajectory of the object can be displayed differently from the case where the association is not performed. For example, the color of the trajectory of the specific object can be different from colors of the trajectory of the other objects.

Further, if the association of the object being tracked with the specific object is performed in the object association unit <NUM>, a trajectory of the object being tracked at a time point before the time point, when the association is performed, can be displayed as the trajectory of the specific object.

In other words, a part or the whole of the image processing apparatus (tracking apparatus) <NUM> in <FIG> can be implemented by a computer. In the example for implementing the whole of the image processing apparatus (tracking apparatus) <NUM> in <FIG> by the computer, the flowchart in <FIG> illustrates a flow of a program in the computer.

Both of the object detection processing in step f3 in <FIG> and the human body detection processing in step f7 or one of the processings can be implemented using hardware, and the other processings can be implemented by software processing by the computer. The exemplary embodiment is implemented by executing the following procedure with a computer-executable program. In other words, in the exemplary embodiment, the object determination procedure f4 for determining whether an object is detected from an image is performed, the tracking procedure f5 for tracking the detected object is performed, and further the trajectory management procedure f6 for managing the trajectory of the object being tracked is performed by updating the movement information (tracking information). The movement information is the information illustrated in <FIG>. Further, the specific object determination procedure f8 for determining whether a specific object (human body) is detected from the image is performed. After the processing, the association procedure f9 for associating the object that is determined to be detected in the object determination procedure f4 with the specific object that is determined to be detected in the specific object determination procedure f8 is performed, and the movement information (trajectory information) is updated. By the updating processing, the specific object management procedure f10 for managing the association of the object being tracked with the specific object is performed.

In the specific object management procedure f10, the information of time points before the time point, when the association of the object with the specific object (human body) is performed, is associated withthe specific object (human body), and managed. Further, in f11, the passage of the object being tracked across a specific place is determined.

The exemplary embodiments have been described in detail. The present invention can be applied to a system including a plurality of devices (for example, a host computer, an interface device, an imaging device, and web application), or an apparatus consists of a single device.

Aspects of the present invention can also be realized by a computer of a system or apparatus (or devices such as a CPU or MPU) that reads out and executes a program recorded on a memory device to perform the functions of the above-described embodiments, and by a method, the steps of which are performed by a computer of a system or apparatus by, for example, reading out and executing a program recorded on a memory device to perform the functions of the above-described embodiments. For this purpose, the program is provided to the computer for example via a network or from a recording medium of various types serving as the memory device (e.g., computer-readable medium). In such a case, the system or apparatus, and the recording medium where the program is stored, are included as being within the scope of the present invention.

Claim 1:
An image processing apparatus comprising:
an object detection unit configured to detect an object in an image by using a first method which is a method using a background subtraction process related to the image;
a human detection unit configured to detect a human in the image by using a second method different from the first method, the second method being a method using a feature amount or a pattern corresponding to a human;
a determination unit configured to determine whether an overlap ratio exceeds a predetermined threshold, the overlap ratio being a ratio of an area of a part which overlaps a bounding box corresponding to the object and a bounding box corresponding to the human to an area of the bounding box corresponding to the human;
a storing unit configured to store information indicating whether the object is a human; and
an updating unit configured to update, if the overlap ratio exceeds the predetermined threshold, the information for the object stored by the storing unit to information indicating that the object is a human.