Patent Description:
A moving object tracking device that tracks a moving object, detected from a plurality of camera videos, across cameras has been known. Among conventional moving object tracking devices, a configuration in which whether or not moving objects reflected in each image are the same is determined based on appearance of the moving object in the image to obtain each tracking result of the moving objects has been known.

In the conventional moving object tracking devices, however, the moving objects reflected in the respective images are associated with each other (tracking information is generated) based on the appearance of the moving object, and thus, it is difficult to cope with a change in clothing when the moving object is a pedestrian, for example. In addition, in the case of limiting to pedestrians, it is possible to associate the moving objects reflected in the respective images with each other using results obtained by recognizing a face even if the appearance varies due to different kinds of clothing and environmental change, but such association is possible only for sections where the face can be seen. It is difficult to track the moving object over a long period of time in the above conventional tracking device.

<CIT> relates to combining and fusing the tracking of people and objects with image processing and providing identification of the people and objects being tracked. Also, status and condition of a person, object, area or facility are detected, determined, evaluated and monitored.

<CIT> relates to a device and method for improving, when performing tracking of moving objects by using captured images taken by multiple cameras, the accuracy of the tracking process across the cameras. The tracking processing device includes: a storage unit that stores, for each camera, a plurality of pieces of intra-camera tracking information including image information of persons obtained from the captured images; a refining unit that performs refining of the plurality of pieces of intra-camera tracking information and thereby extracts pieces of inter-camera tracking information to be used in a process of tracking the persons across the multiple cameras; and an associating unit that, on the basis of the pieces of inter-camera tracking information, associates the persons in the captured images across the multiple cameras.

The invention is defined by the appended independent claims and will be more readily understood after reading the following detailed description in conjunction with the accompanying drawings.

<FIG> is a diagram illustrating an example of a schematic configuration of a moving object tracking system <NUM> according to the present arrangement. As illustrated in <FIG>, the moving object tracking system <NUM> includes two cameras 10A and 10B and a moving object tracking device <NUM>. In the following description, when the cameras 10A and 10B will be simply referred to as a "camera <NUM>" when not distinguished from each other. Although the number of cameras <NUM> included in the moving object tracking system <NUM> is two in the example of <FIG>, the arrangement is not limited thereto, but may be provided in a form of including three or more cameras <NUM> or a form of including a single camera <NUM>. Each of the cameras <NUM> and the moving object tracking device <NUM> are connected to each other in a wireless or wired manner.

<FIG> is a diagram illustrating an example of a hardware configuration of the moving object tracking device <NUM>. As illustrated in <FIG>, the moving object tracking device <NUM> includes a CPU <NUM>, a ROM <NUM>, a RAM <NUM>, an I/F unit <NUM>, an operation unit <NUM>, and a display <NUM>.

The CPU <NUM> comprehensively controls an operation of the moving object tracking device <NUM>. The CPU <NUM> executes various control programs recorded in the ROM <NUM> using a predetermined area of the RAM <NUM> as a work area to implement various functions provided in the moving object tracking device <NUM>. Detailed content of the functions provided in the moving object tracking device <NUM> will be described later.

The ROM <NUM> is a non-volatile memory (non-rewritable memory) that stores programs, various kinds of setting information, and the like which relate to the moving object tracking device <NUM>.

The RAM <NUM> is a storage, for example, an SDRAM or the like, functions as the work area of the CPU <NUM> and serves a role as a buffer or the like.

The I/F unit <NUM> is an interface configured for connection with an external device such as the camera <NUM>. The operation unit <NUM> is a device configured to receive an operation of a user, and is configured using a mouse, a keyboard, or the like. The display <NUM> is a device that displays various kinds of information relating to the moving object tracking device <NUM>, and is configured using, for example, a liquid crystal display or the like. The operation unit <NUM> and the display <NUM> may be a form of being configured in an integrated manner (for example, configured as a touch panel).

<FIG> is a diagram illustrating an example of the functions provided in the moving object tracking device <NUM>. As illustrated in <FIG>, the moving object tracking device <NUM> includes an acquirer <NUM>, a tracker <NUM>, a first associator <NUM>, a second associator <NUM>, and an output unit <NUM>. The functions relating to the present arrangement are mainly illustrated in the example of <FIG>, but the functions provided in the moving object tracking device <NUM> are not limited thereto. In the present arrangement, the respective functions of the acquirer <NUM>, the tracker <NUM>, the first associator <NUM>, the second associator <NUM>, and the output unit <NUM> are implemented by the CPU <NUM> when executing the programs stored in the ROM <NUM> or the like, but the arrangement is not limited thereto. For example, at least a part of the functions may be implemented by a dedicated hardware circuit (a semiconductor integrated circuit or the like). In addition, the moving object tracking device <NUM> may be provided in a form of being partially or entirely present in a server.

The acquirer <NUM> acquires a plurality of images. More specifically, the acquirer <NUM> acquires a plurality of images (a plurality of temporally consecutive images) captured by the camera <NUM> from each of the cameras <NUM>. For example, the acquirer <NUM> may be provided for each of the plurality of cameras <NUM> (the plurality of acquirers <NUM> corresponding to the plurality of cameras <NUM> on a one-to-one basis may be provided), or the single acquirer <NUM> that acquires the images from the respective cameras <NUM> may be provided.

The tracker <NUM> generates a plurality of pieces of tracking information indicating information obtained by tracking a moving object (reflected) included in the image acquired by the acquirer <NUM>. The tracker <NUM> detects and tracks the moving objects reflected in the respective images with respect to an image group acquired by the acquirer <NUM>, and generates the tracking information. Here, the tracker <NUM> generates the tracking information corresponding to each of the image groups captured by the respective cameras <NUM>. In other words, it may be considered that one or more pieces of tracking information are generated for each of the cameras <NUM>. In addition, the moving object is assumed to be a person (pedestrian) in this example, but may be, for example, a vehicle or the like without being limited thereto. Various known techniques can be used as a method of detecting and tracking the moving object among the images, and for example, a technique disclosed in <CIT> may be used. The tracking information may be considered as information which indicates a position of the moving object in each of the plurality of images (the plurality of temporally consecutive images) from which the same moving object is detected and which have a mutual time difference (a difference in imaged time) equal to or smaller than a predetermined value, and the imaged time. The moving object is no longer detected from the image when the moving object is out of an imaging range, for example, so that the tracking information is interrupted at this point in time.

The first associator <NUM> generates first associated tracking information obtained by association targeting a plurality of pieces of tracking information having the mutual time difference equal to or smaller than the threshold. Specifically, the first associator <NUM> collects the plurality of pieces of tracking information corresponding to the same moving object into one piece of information, and this collected result is referred to as the "first associated tracking information". The first associator <NUM> generates one or more pieces of the first associated tracking information. Here, when two pieces of tracking information, as targets, has a difference in imaged time (difference in imaged timing) equal to or larger than a certain value, appearance is highly likely to greatly vary therebetween, and thus, such information is excluded from targets of association by the first associator <NUM>.

For example, as illustrated in <FIG>, assumed is a case where tracking information 100A, 100B, and 100C are generated as tracking information corresponding to the camera 10A, and tracking information 100D and 100E are generated as tracking information corresponding to the camera 10B. In this example, a time difference t1 between the tracking information 100A and 100B is equal to or smaller than the threshold. Thus, these two pieces of the tracking information <NUM> become targets of association by the first associator <NUM>, and are associated with each other when a degree of similarity therebetween is equal to or larger than a reference value. Here, a description will be given by exemplifying a case where a difference between average values of the respective times included in the tracking information is taken as the time difference between the pieces of tracking information, but the arrangement is not limited thereto. For example, a difference between time of an end point of one tracking information and time of a start point of the other tracking time may be used.

In addition, when focusing on the tracking information 100A and the tracking information 100D, a time difference t2 between the tracking information 100A and 100D is equal to or smaller than the threshold in this example. Thus, these two pieces of the tracking information <NUM> become targets of association by the first associator <NUM>, and are associated with each other when similarity therebetween is equal to or larger than a reference value. Similarly, the tracking information 100B and 100D, the tracking information 100B and 100E, and the tracking information 100D and 100E become targets of association in this example. Meanwhile, a time difference between the tracking information 100C and each of the other tracking information 100A, 100B, 100D and 100E exceeds the threshold, and thus, any combination of the tracking information 100C and the other tracking information 100A, 100B, 100D or 100E does not become a target of association by the first associator <NUM>.

In addition, it is conceivable that clothing changes such as putting on or off of an outer garment in a group of the cameras <NUM> under different environments. Thus, such a group is excluded, in advance, from targets of association by the first associator <NUM>. In short, the first associator <NUM> generates the first associated tracking information for each group of the cameras <NUM> set in advance.

The first associator <NUM> associates the trace information and the trace information having the similarity equal to or higher than a threshold. Here, the similarity between the pieces of tracking information as targets of association by the first associator <NUM> is defined using similarity in appearance, a time difference, a distance between the cameras <NUM>, and the like, and an authentication result using biometric information is also added when the biometric information such as a facial image or a fingerprint image is acquired from both the tracking information. In addition, when personal information of a person (pedestrian) such as an ID card is acquired in conjunction with a camera video at a passage gate or the like, these pieces of information can be also used.

In addition, the similarity in appearance of the pedestrian is calculated by, for example, multi-channelizing a person image (an area in which a person is reflected in an image) using a plurality of color spaces having tolerance to variation in environmental light, such as a YCbCr color space, an HSV color space, and a Color Names color space, performing Multi-Layer Block division of dividing each channel image into blocks with different number of divisions in a Y direction, calculating each of a histogram of values and a histogram of binary codes based on a difference value between adjacent pixels from each block, and performing extraction of a feature normalizing and concatenating all the blocks. Then, the similarity may be calculated using similarity in an identification space in which the same person learned by metric learning in advance is mapped to be close and another person is mapped to be distant.

Here, a person image acquired by three channels of RGB is converted into a plurality of color spaces so as to be multi-channelized. For example, the channel is expanded into seventeen channels in total including three channels in the YCbCr color space, three channels in the HSV color space, and eleven channels in the Color Names descriptor*. This expansion aims to absorb individual differences of cameras and variation in color information and luminance information of the acquired images caused by environmental variation, and it is known that variation on the color space are reduced with respect to the above-described variation (see Non-Patent Document "<NPL>)".

Next, block division is performed with a plurality of layers for each channel image. Although the histogram information is acquired from each block at the subsequent stage, it is possible to use position information when the division is performed in detail while influence caused by a positional deviation of a person rectangle is great, thereby increasing identifiability. Thus, division is performed step by step from detailed division to broad division. The division can be performed to be divided in the y-axis direction and the x-axis direction, but may be divided only in any one direction therebetween. Specifically, a plurality of areas called blocks are defined from the respective channel images for each person image. <FIG> is an image view of hierarchical division, and defines the plurality of areas called blocks, step by step, in four stages for one channel image.

Next, the histogram information is extracted from the respective defined areas (blocks). The histogram to be extracted normalizes and concatenates a histogram feature, which indicates a frequency of occurrence of each value, and a feature which makes a histogram using binary codes, allocated based on values of a pixel of interest and a peripheral pixel, that is called a local binary pattern to obtain the histogram of a frequency of occurrence in a gradient direction with a peripheral value. As illustrated in <FIG>, a luminance value of the pixel of interest and an average luminance value of pixels present in a direction are compared for each direction from the pixel of interest (center point), and coding is performed upon performing weighting depending on a result. For example, a luminance pattern illustrated in (A) of <FIG> is converted into a bit string in the direction indicated in (B) of <FIG>. In this example, a luminance value of a peripheral pixel is converted to "<NUM>" when the luminance value of the peripheral pixel is lower than a luminance value of a center pixel, and a luminance value of a peripheral pixel is converted to "<NUM>" when the luminance value of the peripheral pixel is higher than a luminance value of a center pixel, but the arrangement is not limited thereto (the conversion may be performed in the opposite manner). The features in the number of dimensions, obtained by combining a bin number of the histogram of values and a bin number of the histogram of the LBP, is obtained from the respective blocks (areas).

Next, features corresponding to the number of channels × the number of blocks, extracted from the respective blocks, are normalized and concatenated to extract a high-order feature vector. Metric learning is performed to perform conversion, in advance, into a feature space in which the same moving object becomes close and a different moving object becomes distant, and the acquired conversion matrix is used to take a feature vector (d-dimension) obtained by converting the high-order feature vector (D-dimension) as a feature vector of each person image. Several methods are proposed for the metric learning, and in general, only a conversion matrix in a dimension corresponding to the number of categories, which are desirably obtained by division, is calculated in most methods. However, it is possible to calculate a conversion matrix in an arbitrary number of dimensions when REMetric* is used, for example (see "<NPL>").

The similarity between the respective person images can be defined by normalized cross-correlation between the above-described two feature vectors. In general, the tracking information has a plurality of person images, and thus, it is possible to define the similarity between a plurality of images and a plurality of images. A maximum value among all combinations may be used as the similarity between the pieces of tracking information or an average value thereof may be used as the similarity between the pieces of tracking information. Alternatively, principal component analysis is performed on a plurality of feature vector sequences obtained relating to each piece of tracking information to calculate a subspace. This subspace is a space that expresses each moving object expressing variation in appearance, and thus, similarity between the respective subspaces among the pieces of tracking information is calculated by a mutual subspace method and taken as the similarity between the tracking information. Further, the first associator <NUM> associates the tracking information and the tracking information having the similarity equal to or higher than the reference value, and generates one or more pieces of the first associated tracking information.

The second associator <NUM> generates second associated tracking information obtained by association targeting the first associated tracking information and tracking information, which has not been associated by the first associator <NUM>, based on authentication information for identifying the moving object. The authentication information includes biometric information. More specifically, the biometric information includes a facial image or a fingerprint image. In addition, the authentication information may include personal information of a person (pedestrian) such as an ID card.

In this example, the second associator <NUM> extracts facial images, respectively, for the first associated tracking information and the tracking information, which are the targets of association, and generates the second associated tracking information by associating the first associated tracking information and the first associated tracking information, the first associated tracking information and the tracking information, or the tracking information and the tracking information which have the extracted faces images matching each other. Accordingly, the tracking information 100C that has not been associated by the first associator <NUM> is also associated (collected as one piece of tracking information) with the first associated tracking information (or tracking information that has not been associated by the first associator <NUM>) including the facial image extracted from the tracking information 100C in <FIG>, for example. The invention is not limited thereto, and may be provided in a form of performing association by performing the similar authentication using a fingerprint image. In addition, when personal information of a person such as an ID card is acquired in conjunction with an image imaged by the camera <NUM> at a pass gate or the like, the second associator <NUM> may be provided in a form of performing association using these kinds of information. In the above-described manner, the second associated tracking information is obtained for each moving object reflected in the image. Here, it is known recognition performance for facial recognition is improved by performing authentication using a set of facial images acquired from the same person at a plurality of angles and under a plurality of illumination environments. While a facial image obtained from the tracking information, which is an analysis result from a single camera video, has little variation, the facial image obtained from the above-described first associated tracking information, which is an analysis result from a plurality of camera videos, is a facial image that has greater variation. Thus, the facial recognition performance is further improved when the first associated tracking information is used. As for calculation of similarity in the facial recognition, similarity between two facial images may be defined using a predetermined feature vector and a maximum value among all combinations may be defined as similarity between sets, or similarity may be calculated using mutual subspaces, which is similar to the above-described calculation of similarity between the person images.

The output unit <NUM> outputs the second associated tracking information. For example, as illustrated in <FIG>, the output unit <NUM> can also perform control to display a movement trajectory <NUM> corresponding to the second associated tracking information on the display <NUM>.

As described above, one or more pieces of the first associated tracking information, obtained by association targeting a plurality of pieces of tracking information having the mutual time difference equal to or smaller than the threshold, are generated, and then, the second associated tracking information, obtained by association targeting the one or more pieces of first associated tracking information and tracking information, which has not been associated as the first associated tracking information, based on the authentication information for identifying the moving object, is generated and output in the present arrangement. In other words, it is possible to obtain tracking results over a longer period of time by creating one or more first associated tracking information, obtained by associating short-term tracking information and short-term tracking information on the same moving object, and generating the second associated tracking information, obtained by association based on the authentication information, relating to the first associated tracking information and the first associated tracking information which have not been associated due to the time difference therebetween, and the tracking information (tracking information missed from association by the first associator <NUM>) which have not been associated as the first associated tracking information. Therefore, it is possible to achieve an advantageous effect that tracking of a moving object over a long period of time becomes possible according to the present arrangement.

The function of the tracker <NUM> may be a form of being provided in the camera <NUM>, for example. In this case, the moving object tracking device <NUM> is provided in a form of having a function of acquiring a tracking information group instead of the acquirer <NUM> and the tracker <NUM>. In short, the moving object tracking system <NUM> may be provided in a form of including the acquirer <NUM>, the tracker <NUM>, the first associator <NUM>, the second associator <NUM>, and the output unit <NUM>.

Next, a second arrangement will be described. A description on common parts as the above-described first arrangement will be suitably omitted. In the present arrangement, <FIG> is a diagram illustrating an example of functions provided in the moving object tracking device <NUM> according to the second arrangement. As illustrated in <FIG>, the moving object tracking device <NUM> further includes a selector <NUM>, which is different from the above-described first arrangement. The selector <NUM> selects any moving object included in a plurality of temporally consecutive images. A selection method is arbitrary. For example, the selector <NUM> can select a pedestrian reflected in an image in accordance with a touch operation or a mouse operation with respect to the image acquired by the acquirer <NUM>, or may select any pedestrian in accordance with a touch operation or a mouse operation among pedestrians (people), reflected in images, retrieved with a facial image as a query, and then, displayed in the descending order of similarity. In addition, for example, person images (pedestrians) cut out from the images may be displayed in an aligned manner, and any person image (pedestrian) may be selected in accordance with a touch operation or a mouse operation.

The output unit <NUM> according to the present arrangement performs control to display a movement trajectory of a pedestrian (an example of a moving object), selected by the selector <NUM> (a movement trajectory corresponding to the second associated tracking information of the pedestrian), and one or more images, which correspond to one or more points (may be fixed or may be arbitrarily changeable in accordance with designation by a user) on the movement trajectory, in an aligned manner. For example, as illustrated in <FIG>, it is possible to display the movement trajectory <NUM>, an image <NUM>, which corresponds to a point corresponding to the latest time in the movement trajectory <NUM>, and an image <NUM>, which corresponds to a point corresponding to an arbitrary time in the past, in an aligned manner. In the example of <FIG>, icons of pedestrians are displayed together with the images. Here, the output unit <NUM> functions as a "display controller". Further, the moving object tracking device <NUM> corresponds to a "display device".

The function of the output unit <NUM> and the function of the selector <NUM> among the functions illustrated in <FIG>, for example, may be extracted and mounted to another device. In this case, the other device functions as the "display device".

<FIG> is a diagram illustrating an example of functions provided in the moving object tracking device (display device) <NUM> according to a modified example of the second arrangement. As illustrated in <FIG>, the moving object tracking device <NUM> further includes a designator <NUM>. The designator <NUM> designates a period. A method of designating the period is arbitrary. For example, when a calendar image <NUM> illustrated in <FIG> is displayed and a touch operation or a mouse operation with respect to any date inside the calendar image <NUM> is received, the designator <NUM> can designate a period (one day) corresponding to the date.

The output unit <NUM> performs control to display a movement trajectory, which corresponds to a period designated by the designator <NUM>, of a pedestrian selected by the selector <NUM> and one or more images corresponding to one or more points on the movement trajectory in an aligned manner. For example, when a date of "October <NUM>, <NUM>" in the calendar image <NUM> of <FIG> is designated, the output unit <NUM> can also display the movement trajectory <NUM>, which corresponds to the designated date, of the pedestrian selected by the selector <NUM>, an image <NUM>, which corresponds to a point corresponding to the latest time in the movement trajectory <NUM>, and an image <NUM>, which corresponds to a point corresponding to an arbitrary time in the past, in an aligned manner as illustrated in <FIG>. In the example of <FIG>, icons of pedestrians are displayed together with the images.

In addition, the designator <NUM> can also designate a plurality of periods. In this case, the output unit <NUM> performs control to display a movement trajectory of a pedestrian selected by the selector <NUM> and one or more images corresponding to one or more points on the movement trajectory, in an aligned manner, for each of the plurality of periods designated by the designator <NUM>. For example, assumed is a case where the date of "October <NUM>, <NUM>" and a date "October <NUM>, <NUM>" in the calendar image <NUM> of <FIG> are designated. In this case, the output unit <NUM> can display a movement trajectory <NUM>, which corresponds to "October <NUM>, <NUM>", of the pedestrian selected by the selector <NUM>, an image <NUM>, which corresponds to a point corresponding to the latest time in the movement trajectory <NUM>, and an image <NUM>, which corresponds to a point corresponding to an arbitrary time in the past, in an aligned manner, and display a movement trajectory <NUM>, which corresponds to "October <NUM>, <NUM>", an image <NUM>, which corresponds to a point corresponding to the latest time in the movement trajectory <NUM>, and an image <NUM>, which corresponds to a point corresponding to an arbitrary time in the past, in an aligned manner as illustrated in <FIG>.

In addition, it may be configured such that the program executed by the moving object tracking device <NUM> according to the respective arrangements and modified examples described above is stored in a computer connected to the network such as the Internet and is provided through download via the network. In addition, it may be configured such that the program executed by the moving object tracking device <NUM> according to the respective arrangements and modified examples described above is provided or distributed via the network such as the Internet. In addition, it may be configured such that the program executed by the moving object tracking device <NUM> according to the respective arrangements and modified examples described above is provided in the state of being incorporated in a non-volatile recording medium such as a ROM in advance.

Claim 1:
A moving object tracking device (<NUM>) comprising:
an acquirer (<NUM>) configured to acquire a plurality of images;
a tracker (<NUM>) configured to generate a plurality of pieces of tracking information indicating information obtained by tracking one or more moving objects included in the images, wherein each of the plurality of pieces of tracking information indicates:
for each image of a respective plurality of images, a position of a moving object of the one or more moving objects in the image and an imaged time of the image, where in the respective plurality of images are those images of plurality of images from which the moving object is detected and having a difference in imaged time equal to or smaller than a predetermined value;
a first associator (<NUM>) configured to generate first associated tracking information by collecting pieces of tracking information having a mutual time difference equal to or smaller than a threshold among the plurality of pieces of tracking information as pieces of tracking information corresponding to a same moving object of the one or more moving objects;
a second associator (<NUM>) configured to:
extract authentication information, respectively, for the first associated tracking information and one or more pieces of the plurality of pieces of tracking information which have not been associated by the first associator; and
generate second associated tracking information by associating the first associated tracking information and the one or more pieces of the plurality of pieces of tracking information having authentication information matching each other; and
an output unit (<NUM>) configured to output the second associated tracking information.