IMAGE PROCESSING APPARATUS

A registered object image storage unit stores at least one image of an object as a registered object image in association with an object identification number that uniquely identifies the object. The unknown object image acquisition unit acquires an unknown object image that is an image of an unknown object. An object re-identification unit re-identifies the acquired unknown object image by using the registered object image stored in the registered object image storage unit. When a condition that all registered object images stored in the registered object image storage unit in association with the object identification number of the object in the unknown object image whose re-identification succeeded differ from the unknown object image, a registered object image update unit adds the unknown object image to the registered object image storage unit as a new registered object image in association with the object identification number.

TECHNICAL FIELD

The present invention relates to an image processing apparatus, an image processing method, and a storage medium.

BACKGROUND ART

A technology of finding the same object from among a plurality of input images of objects captured at different points of time is called re-identification. Here, an object is a person or an article. Objects include belongings of a person such as a backpack and a bag, mobile bodies such as a car, a motorcycle, a ship, and an airplane, animals other than persons such as a dog and a cat, and objects other than persons. As a literature describing re-identification, Patent Literature1has been known, for example.

CITATION LIST

Patent Literature

Patent Literature 1: JP 2018-506788 A

SUMMARY OF INVENTION

Technical Problem

Re-identification of an unknown object is performed by using an image of a known object that is registered previously (registered object image). For example, feature values are extracted from both an image of an unknown object and an image of a known object and are compared with each other, and when they match, the unknown object is determined to the be the same object (same individual) as the known object. At that time, to the unknown object, an object identification number (object ID) having been assigned to the known object is assigned. In the case where only one registered object image is stored for one known object, re-identification cannot be made unless the image of the unknown object matches the only one registered object image. On the other hand, when various registered object images are stored for one known object, since it is only necessary that the image of the unknown object matches any one of the registered object images, the accuracy of re-identification is improved. However, when a plurality of registered object images are randomly stored for a known object, variations of the registered object images are not increased for an increase in the number of registered object images, and the accuracy of re-identification is not improved so much despite the fact that the processing speed for re-identification is lowered due to an increase in the number of times of re-identification.

The present invention provides an image processing apparatus that solves the above-described problem.

Solution to Problem

An image processing apparatus, according to one aspect of the present invention, is configured to include

a registered object image storage unit that stores at least one image of an object as a registered object image in association with an object identification number that uniquely identifies the object;

an unknown object image acquisition unit that acquires an unknown object image that is an image of an unknown object;

an object re-identification unit that re-identifies the acquired unknown object image by using the registered object image stored in the registered object image storage unit; and

a registered object image update unit that, when a condition is satisfied, the condition being that all registered object images stored in the registered object image storage unit in association with the object identification number of the object in the unknown object image whose re-identification by the object re-identification unit succeeded differ from the unknown object image, adds the unknown object image to the registered object image storage unit as a new registered object image in association with the object identification number.

Further, an image processing method, according to one aspect of the present invention, is configure to include

acquiring an unknown object image that is an image of an unknown object;

re-identifying the acquired unknown object image by using a registered object image stored in a registered object image storage unit, the registered object image storage unit storing at least one image of an object as the registered object image in association with an object identification number that uniquely identifies the object; and

when a condition is satisfied, the condition being that all registered object images stored in the registered object image storage unit in association with the object identification number of the object in the unknown object image whose re-identification succeeded differ from the unknown object image, adding the unknown object image to the registered object image storage unit as a new registered object image in association with the object identification number.

Further, a computer-readable medium, according to one aspect of the present invention, is configured to store thereon a program for causing a computer to execute processing to:

acquire an unknown object image that is an image of an unknown object;

re-identify the acquired unknown object image by using a registered object image stored in a registered object image storage unit, the registered object image storage unit storing at least one image of an object as the registered object image in association with an object identification number that uniquely identifies the object; and

when a condition is satisfied, the condition being that all registered object images stored in the registered object image storage unit in association with the object identification number of the object in the unknown object image whose re-identification succeeded differ from the unknown object image, add the unknown object image to the registered object image storage unit as a new registered object image in association with the object identification number.

ADVANTAGEOUS EFFECTS OF INVENTION

With the configurations described above, the present invention can enhance the accuracy of re-identification by increasing variations of registered object images without uselessly increasing the number of registered object images, that is, without lowering the processing speed of re-identification.

DESCRIPTION OF EMBODIMENTS

First Exemplary Embodiment

FIG.1is a block diagram illustrating an image processing apparatus100according to a first exemplary embodiment of the present invention. The image processing apparatus100is configured to re-identify an object. Here, an object may be a person. When an object is a person, the image processing apparatus100has a function of re-identifying a person. An object may be an article other than a person. When an object is an article other than a person, the image processing apparatus100has a function of re-identifying an article other than a person. Moreover, an object may include both a person and an article other than a person. When an object includes both a person and an article other than a person, the image processing apparatus100has both a function of re-identifying a person and a function of re-identifying an article other than a person. Referring toFIG.1, the image processing apparatus100is configured to include a camera interface (I/F) unit110, a communication I/F unit120, an operation input unit130, a screen display unit140, a storage unit150, and an arithmetic processing unit160.

The camera I/F unit110is connected to an image server170in a wired or wireless manner, and is configured to transmit and receive data between the image server170and the arithmetic processing unit160. The image server170is connected to a camera175in a wired or wireless manner, and is configured to accumulate a plurality of images captured by the camera175at different time in a past certain period. The camera175may be a color camera equipped with a charge-coupled device (CCD) image sensor or a complementary MOS (CMOS) image sensor having a pixel capacity of about several millions pixels. The camera175may be a camera installed on a street where a plurality of persons and goods come and go, in a room, or the like for the purpose of crime prevention and monitoring. The camera175may be a camera that captures the same or different imaging areas from a fixed place in a fixed capturing direction. Alternatively, the camera175may be a camera that is mounted on a moving object such as a car and captures the same or different imaging areas while moving. The number of cameras175is not limited to one. A plurality of cameras that capture different imaging areas from different places may be used.

The communication I/F unit120is configured of a data communication circuit, and is configured to perform data communication with an external device, not illustrated, in a wired or wireless manner. The operation input unit130is configured of operation input devices such as a keyboard and a mouse, and is configured to detect operation by an operator and output it to the arithmetic processing unit160. The screen display unit140is configured of a screen display device such as a liquid crystal display (LCD), and is configured to display, on a screen, various types of information according to an instruction from the arithmetic processing unit160.

The storage unit150is configured of storage devices such as a hard disk and a memory, and is configured to store therein processing information and a program1501necessary for various types of processing in the arithmetic processing unit160. The program1501is a program for implementing various processing units by being read and executed by the arithmetic processing unit160, and is read in advance from an external device or a storage medium, not illustrated, via a data input-output function of the communication I/F unit120and is stored in the storage unit150. The main processing information stored in the storage unit150includes a registered object image storage unit1502and an object re-identification information1503.

The registered object image storage unit1502is configured to store therein at least one registered object image that is an image of an object, in association with an object identification number that uniquely identifying the object (individual).

The object re-identification information1503is information representing the result of object re-identification. The object re-identification information1503is configured to include, for example, information specifying a frame image (for example, camera ID and captured time), an object image of each object detected from the frame image and the position information thereof, and an object identification number of each re-identified object. Note that the object re-identification information1503is also referred to as an object re-identification information storage unit.

The arithmetic processing unit160has a processor such as an MPU and its peripheral circuits, and is configured to read and execute the program1501from the storage unit150to allow the hardware and the program1501to cooperate with each other to implement the various processing units. The main processing units implemented by the arithmetic processing unit160include an unknown object image acquisition unit1601, an object re-identification unit1602, and a registered object image update unit1603.

The unknown object image acquisition unit1601is configured to acquire, from the image server170via the camera I/F unit110, frame images constituting a moving image captured by the camera175or frame images obtained by down-sampling them. To each of the acquired frame images, the camera ID and the captured time are added. The captured time of a frame image differs for each frame. The unknown object image acquisition unit1601is also configured to detect an unknown object image from the acquired frame image. An unknown object image is an image inside the bounding rectangle of an image of the whole object existing in the frame image, for example. In the case of a person, for example, an unknown object image is an image inside the bounding rectangle of an image of the whole body of the person. In the case of an article, an unknown object image is an image inside the bounding rectangle of an image of the whole article.

The unknown object image acquisition unit1601is configured to, for example, by inputting a frame image to a learning model having been learned through machine learning for estimating an object image from the frame image, acquire an object image existing in the frame image as an unknown object image from the learning model. The learning model can be generated in advance through machine learning using a machine learning algorism such as a neural network by using various frame images and various object images in the frame images as teacher data. However, the method of detecting an unknown object image from a frame image is not limited to that described above. Another method such as pattern matching may be used.

The unknown object image acquisition unit1601is configured to add, to the detected unknown object image, a provisional object ID, a camera ID, captured time, and position information of the unknown object image on the frame image, and transmit them to the object re-identification unit1602. The position information of the unknown object image may be coordinate information specifying the bounding rectangle of the unknown object image, for example.

The object re-identification unit1602is configured to re-identify an unknown object image transmitted from the unknown object image acquisition unit1601by using registered object images stored in the registered object image storage unit1502in the storage unit150.

The object re-identification unit1602is configured to, for example, input the unknown object image and one of the registered object images to a first learning model having been learned by machine learning for estimating whether or not the two object images are object images of the same object, and acquire an estimation result with a probability of whether or not they are object images of the same object, from the first learning model. The first learning model can be generated in advance by machine learning using a machine learning algorism such as a neural network by using various pairs of object images of the same object and various pairs of object images of different objects as teacher data. The object re-identification unit1602determines, for example, that the unknown object image is the same object as the registered object image when the probability is equal to or higher than a predetermined threshold probability. The object re-identification unit1602determines that the unknown object image is not the same object as the registered object image when the probability is lower than the predetermined threshold probability. In that case, when another registered object image is stored in the registered object image storage unit1502, the object re-identification unit1602repeats re-identification of the unknown object image by using the other registered object image. Then, in the case where an estimation result that is equal to or higher than the threshold probability is not acquired even performing re-identification of the unknown object image by using all registered object images or those within a predetermined range, the object re-identification unit1602determines that the unknown object image shows an newly appearing object, and adopts and assigns a new object identification number to the object. Further, the object re-identification unit1602stores the unknown object image as a newly appearing registered object image in the registered object image storage unit1502in association with the object identification number.

In the above description, object re-identification is performed by directly inputting an image to the learning model. However, it is also possible to extract a feature value appropriate for object re-identification from the image, and perform re-identification by inputting the feature value to the learning model. Moreover, the method of object re-identification is not limited to that described above. It is possible to use another method such as a method of determining whether or not a distance between feature vectors extracted from two object images is a predetermined distance or shorter. Furthermore, the object re-identification unit1602may calculate, for each object identification number, a statistical value (for example, average value, maximum value) by obtaining the probability between each of one or more registered object images stored in association with the object identification number and the unknown object image, and determine that the unknown object image is the same object as the registered object image when the statistical value is equal to or larger than a threshold.

The object re-identification unit1602creates the object re-identification information1503representing the result of object re-identification performed on the unknown object image, and stores it in the storage unit150. The object re-identification information1503stored in the storage unit150may be used for selecting a registered object image to be used for re-identification when the object re-identification unit1602performs re-identification of an object image in a new frame image. Alternatively, the object re-identification information1503may be displayed on the screen display unit140, and/or transmitted to an external device via the communication I/F unit120. Alternatively, the object re-identification information1503may be used as input data to an application program for monitoring the behavior of persons and articles. As an example of monitoring the behavior of a person and an object, monitoring suspicious persons and suspicious articles (for example, desertion, carrying-away, switching, shoplifting) may be conceived.

The object re-identification unit1602is also configured to, upon success of re-identification of the unknown object image, transmit re-identification success information including the object identification number of the re-identified object and the unknown object image to the registered object image update unit1603. The object re-identification unit1602is also configured to, upon failure of re-identification of the unknown object image, that is, when there is no registered object image matching the unknown object image, transmit re-identification failure information including the unknown object image that is not re-identified to the registered object image update unit1603.

The registered object image update unit1603updates the registered object image storage unit1502on the basis of the re-identification failure information or the re-identification success information received from the object re-identification unit1602. Specifically, when receiving re-identification failure information from the object re-identification unit1602, the registered object image update unit1603registers the unknown object image in the registered object image storage unit1502as a registered object image, in association with the object identification number adopted for the object of the unknown object image not re-identified.

On the contrary, when receiving re-identification success information from the object re-identification unit1602, the registered object image update unit1603first reads out all registered object images stored in the registered object image storage unit1502in association with the object identification number of the object of the unknown object image in which re-identification succeeded. Then, for each pair of the unknown object image and a registered object image, the registered object image update unit1603checks whether or not they match.

For example, the registered object image update unit1603calculates the similarity between the unknown object image and the registered object image, and when the calculated similarity is equal to or smaller than a predetermined threshold, the registered object image update unit1603determines that they match, but otherwise determines that they do not match. Then, when the registered object image update unit1603determines that the unknown object image and the registered object image do not match in all pairs, the registered object image update unit1603adds the unknown object image to the registered object image storage unit1502as a new registered object image, in association with the object identification number of the object of the unknown object image. At that time, the registered object images having been registered in the registered object image storage unit1502will not be deleted but kept. As a result, variations of the registered object image that can be used for re-identification of the object are increased by one.

Meanwhile, when at least one pair of the unknown object image and a registered object image match, the registered object image update unit1603performs a process (A) or a process (B) described below.

(A) The registered object image update unit1603deletes one registered object image that matches the unknown object image from the registered object image storage unit1502and instead, adds the unknown object image as a new registered object image to the registered object image storage unit1502. As a result, the registered object image is replaced with the latest one although the variations of the registered object images that can be used for re-identification of the object are not changed.

(B) The registered object image update unit1603performs neither addition of the unknown object image to the registered object image storage unit1502nor deletion of the registered object image from the registered object image storage unit1502. As a result, it is possible to prevent the number of registered object images from being increased uselessly like the case of randomly adding the unknown object image to the registered object image storage unit1502.

Next, operation of the image processing apparatus100will be described.FIG.2is a flowchart illustrating an exemplary operation of the image processing apparatus100.

Referring toFIG.2, the unknown object image acquisition unit1601first acquires, from the image server170via the camera I/F unit110, a frame image captured by the camera175, and acquires an unknown object image from the frame image (step S1).

Then, the object re-identification unit1602re-identifies the acquired unknown object image by using one or more registered object images stored in the registered object image storage unit1502(step S2).

Then, the registered object image update unit1603updates the registered object image storage unit1502on the basis of a result of re-identification of the unknown object image by the object re-identification unit1602(step S3). Specifically, when re-identification failed, the registered object image update unit1603registers the unknown object image in the registered object image storage unit1502as a registered object image, in association with the object identification number adopted for the object of the unknown object image not re-identified. Meanwhile, when re-identification of the object succeeded, the registered object image update unit1603checks, for each pair of the unknown object image and each of all registered object images stored in the registered object image storage unit1502in association with the object identification number of the object in the unknown object image whose re-identification succeeded, whether or not the two match, and when all pairs do not match, adds the unknown object image to the registered object image storage unit1502as a new registered object image, in association with the object identification number of the object in the unknown object image. On the contrary, when at least one pair of the unknown object image and a registered object image match, the registered object image update unit1603performs the process (A) or the process (B).

Upon completion of the process of step S3, the image processing apparatus100returns to step S1and repeats the same process as that described above. Thereby, when there is another unknown object image in the frame image captured by the camera175, for each unknown object image, acquisition of an unknown object image (step S1), re-identification of the unknown object image (step S2), and update of the registered object image storage unit1502based on the re-identification result (step S3) are repeated. When the processes of steps S1to S3are completed for all unknown object images existing in one frame image, the next frame image is acquired, and the same processes as those performed on the previous frame image are repeated.

As described above, according to the image processing apparatus100of the present embodiment, it is possible to enhance the accuracy of re-identification by increasing the variations of registered object images without uselessly increasing the number of registered object images, that is, without lowering the processing speed of re-identification. This is because upon success of object re-identification, the registered object image update unit1603checks, for each pair of the unknown object image and each of all registered object images stored in the registered object image storage unit1502in association with the object identification number of the object in the unknown object image whose re-identification succeeded, whether or not the two match, and when all pairs do not match, adds the unknown object image to the registered object image storage unit1502as a new registered object image, in association with the object identification number of the object in the unknown object image.

Next, the registered object image update unit1603will be described in more detail.

As a method of registering a plurality of different images for the same object, two methods will be considered. A first method is a method including previously determining conditions for a plurality of different images, and determining whether a newly acquired image satisfies one of the conditions not satisfied by the registered images. A second method is a method including unconditionally registering the first acquired image, and for an image acquired secondly or later, determining whether or not it is an exceptional image not belonging to any of one or more clusters obtained by clustering one or more registered images by means of one-class identification plane, and determining propriety of registration according to the determination result.

In the case where the registered object image update unit1603uses the first method, various conditions can be conceived. For example, as the conditions, different postures of an object may be used. Alternatively, as the conditions, presence or absence of accessories (belongings) of an object or the type thereof may be used. For example, in the case of a person, a person not having any belongings, a person having a backpack as belongings, a person having a handbag as belongings, may be used as conditions for different images.

In the case where the registered object image update unit1603uses the second method, it is considered to extract a predetermined feature value from an image, and perform clustering according to the degree of similarity of the extracted feature value. Various types of feature values may be considered. For example, a feature value representing the posture of an object or a feature value representing presence or absence of an accessory of an object and the type thereof may be used.

Next, a specific example of the registered object image update unit1603will be described.

Example 1 of Registered Object Image Update Unit1603

FIG.3illustrates an exemplary format of information stored in the registered object image storage unit1502in association with the registered object number that uniquely identifies an object. Referring toFIG.3, the registered object image storage unit1502includes as many pairs of the object identification number171and the registered object image information172as the registered objects. In the field of the object identification number171, a registered object number that uniquely identifies a registered object is set. In the field of the registered object image information172, information of the registered object image to be used for re-identification of the registered object specified by the registered object number171is set. In the example shown inFIG.3, the registered object image information172includes “m” sets of a posture number1721, a registered object image1722, and a property1723.

In the field of the posture number1721, a number defining one posture, when the postures of an object are classified into m types, is set in advance. In the field of the registered object image1722, a registered object image matching the posture defined by the posture number set in the posture number1721is set. In the field of the property1723, a camera ID, capturing time, and the like, added to the frame image in which the registered object image set in the registered object image1722is included, is set. When there is no registered object image corresponding to the posture number1721, NULL value is shown in the fields of the registered object image1722and the property1723, for example. In the present example, one registered object image is registered with one posture number. However, as another embodiment, two or more registered object images may be registered with one posture number.

Various methods are conceivable for detecting the posture of an object from an object image. In the below description, an example of detecting the posture of an object by using skeletal information of the object will be described. This method is applicable when an object is a person. Even in the case where an object is other than a person, this method is applicable when the object is a living thing having a skeleton such as a dog or a cat.

First, the registered object image update unit1603detects skeletal information of an object from an object image. For example, human skeletal information includes information representing positions of joints constituting the human body. Joints may include not only joints of a neck or shoulders but also parts of a face such as eyes and a nose. The skeletal information may include reliability (likelihood of feature recognition) for each joint, in addition to the position of each joint. For example, the registered object image update unit1603may realize skeletal information of an object in an image by using a system that estimates it by deep learning. Such a system includes OpenPose, HRNet, AlphaPose, or the like, for example. For example, OpenPose extracts positions (x, y coordinates) and reliabilities of eighteen joints such as nose, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, right hip, light knee, right ankle, left hip, left knee, left ankle, right eye, right ear, left eye, and left ear. However, a method of detecting skeletal information from an object image is not limited to that described above, and any method can be used.

Next, the registered object image update unit1603determines the posture of the object from the detected skeletal information. As a method of determining the posture of the object from the skeletal information, a method described below may be used for example.

First, the registered object image update unit1603acquires position information of a predetermined pair of joints that are present symmetrically at left and right, from the skeletal information. There are some pairs of joints that are present symmetrically at leaf and right. In the case of a person, it is desirable to acquire position information of a joint pair of the left shoulder and the right shoulder that largely affects the posture. Then, the registered object image update unit1603calculates a normal direction of a vector from one (for example, a joint of the left shoulder) to the other (for example, a joint of the right shoulder) of the pair. Since there are two directions vertical to the vector, a predetermined one direction is defined as the normal direction. Then, the registered object image update unit1603determines which of the reference directions corresponding to the predetermined “m” types of posture numbers the normal direction matches. For example, in the case of m=8, the reference directions corresponding to the “m” types of posture numbers may be 0° , 45° , 90° , 135° , 180° , 225° , 270° , and 315° . Moreover, the registered object image update unit1603calculates the angle of a difference between the determined normal direction and each reference direction, and determines the posture number corresponding to the reference direction in which the absolute value of the angle of the difference becomes minimum to be the posture of the object. For example, when the normal direction is 5°, the registered object image update unit1603determines that it is the posture of the posture number corresponding to the reference direction 0°. However, the method of determining the posture of the object from the skeletal information and the number of types of postures are not limited to those described above.

FIG.4is a flowchart illustrating exemplary processing by the registered object image update unit1603. Referring toFIG.4, the registered object image update unit1603first determines the position of an object that is an unknown object from an unknown object image included in re-identification success information or re-identification failure information transmitted from the object re-identification unit1602(step S11). Then, the registered object image update unit1603determines whether the object re-identification succeeded or failed (step S12). Then, when the object re-identification failed, the registered object image update unit1603creates a set of a new object identification number171and registered object image information172for the current object that is an unknown object, in the registered object image storage unit1502(step S13). Specifically, the registered object image update unit1603adopts a new object identification number for the current object, and sets the adopted object identification number in the field of the object identification number171that is newly secured in the registered object image storage unit1502. Further, the registered object image update unit1603sets the current unknown object image and its property in the fields of the registered object image1722and the property1723of the same set as the posture number1721that matches the posture detected from the object image in the newly secured registered object image information172.

Then, the registered object image update unit1603ends the processing ofFIG.4.

On the other hand, when the object re-identification succeeded, the registered object image update unit1603focuses on one of the registered object images1722registered in the registered object image storage unit1502in association with the re-identified object identification number (step S14). Then, the registered object image update unit1603determines the posture of the registered object from the focused registered object image1722(step S15), and compares it with the posture of the unknown object image determined at step S11(step S16).

Then, when the two postures match (YES at step S17), the registered object image update unit1603overwrites the focused registered object image1722stored in the registered object image storage unit1502with the unknown object image (step S18). At that time, the corresponding property1723is overwritten with that of the unknown object image. Then, the registered object image update unit1603ends the processing ofFIG.4.

Meanwhile, when the two postures do not match (NO at step S17), the registered object image update unit1603moves the focus to the next registered object image1722associated with the re-identified object identification number (step S19), returns to step S15via step S20, and with respect to the newly focused registered object image1722, repeats the same process as that described above. Then, upon completion of the process with respect to all registered object images1722stored in the registered object image storage unit1502associated with the re-identified object identification number (that is, when the posture does not match any of the registered object image) (YES at step S20), the registered object image update unit1603executes step S21. At step S21, the registered object image update unit1603registers the set of the unknown object image and its property in the registered object image storage unit1502as a set of the registered object image1722and the property1723of the posture number corresponding to the posture. Then, the registered object image update unit1603ends the processing ofFIG.4.

Example 2 of Registered Object Image Update Unit1603

FIG.5illustrates another exemplary format of information stored in the registered object image storage unit1502in association with a registered object number that uniquely identifies an object. Referring toFIG.5, the registered object image storage unit1502includes as many pairs of the object identification number173and the registered object image information174as the registered objects. In the field of the object identification number173, a registered object number that uniquely identifies a registered object is set. In the field of the registered object image information174, information of the registered object image to be used for re-identification of the registered object specified by the registered object number173is set. In the example shown inFIG.5, the registered object image information174includes one or more sets of a registered object image1741and a property1742. Here, the upper limit of the number of registered object images registered for one object identification number may be set or may not be set.

In the field of the registered object image1741, an object image of the registered object is set. In the field of the property1742, a camera ID, capturing time, and the like, added to the frame image in which the registered object image set in the registered object image1741is included, are set.

FIG.6is a flowchart illustrating exemplary processing by the registered object image update unit1603. Referring toFIG.6, the registered object image update unit1603first detects skeletal information of an object that is an unknown object from an unknown object image included in re-identification success information or re-identification failure information transmitted from the object re-identification unit1602(step S31). Then, the registered object image update unit1603determines whether the object re-identification succeeded or failed (step S32). Then, when the object re-identification failed, the registered object image update unit1603creates a set of the new object identification number173and the registered object image information174for the current object that is an unknown object, in the registered object image storage unit1502(step S33). Specifically, the registered object image update unit1603adopts a new object identification number for the current object, and sets the adopted object identification number in the field of the object identification number173that is newly secured in the registered object image storage unit1502. Moreover, the registered object image update unit1603sets, in the newly secured registered object image information174, a set of the registered object image1741in which the current unknown object image is set and the property1742. Then, the registered object image update unit1603ends the processing ofFIG.6.

On the other hand, when the object re-identification succeeded, the registered object image update unit1603performs clustering on all registered object images1741registered in the registered object image storage unit1502in association with the re-identified object identification number, on the basis of the skeletal information extracted therefrom (step S34).

Then, the registered object image update unit1603focuses on one cluster obtained by the clustering (step S35), and creates one-class identification plane of the focused cluster (step S36).

Then, the registered object image update unit1603determines whether or not the skeletal information of the unknown object image is in the same class as that of the focused cluster, by using the created one-class identification plane (step S37). Then, when they are determined to be in the same class (YES at step S38), the registered object image update unit1603overwrites one of the registered object images1741in the focused cluster stored in the registered object image storage unit1502with the unknown object image (step S39). At that time, the corresponding property1742is overwritten with that of the unknown object image. Then, the registered object image update unit1603ends the processing ofFIG.6.

Meanwhile, when they are determined not to be in the same class (NO at step S38), the registered object image update unit1603moves the focus to another cluster generated by the clustering (step S40), returns to step S36via step S41, and with respect to the newly focused cluster, repeats the same process as that described above. Then, upon completion of the process with respect to all created clusters (that is, when the joint information of the unknown object image does not belong to any of the clusters) (YES at step S41), the registered object image update unit1603executes step S42. At step S42, the registered object image update unit1603registers the set of the unknown object image and its property in the registered object image storage unit1502, as a set of the registered object image1741and the property1742corresponding to the re-identified object identification number173. Then, the registered object image update unit1603ends the processing ofFIG.6.

Next, the object re-identification unit1602will be described in more detail.

FIG.7is a schematic diagram for explaining an example of a method of selecting a registered object image to be used for re-identification by the object re-identification unit1602. InFIG.7, a frame image181is a frame image to be re-identified by the object re-identification unit1602, and includes three unknown object images X1to X3detected by the unknown object image acquisition unit1601. On the other hand, frame images182,183,184, and185are frame images that are one frame, two frames, three frames, and four frames before the frame image181, respectively. In each of the frames182to185, three object images are included, and are re-identified by the object re-identification unit1602. That is, in the three object images, one is re-identified as a person A, another one is re-identified as a person B, and the other one is re-identified as a person C. The re-identified information thereof is included in the object re-identification information1503. InFIG.7, images of the person A in the respective frames are distinguished as A1, A2, A3, and A4Similarly, images of the person B in the respective frames are distinguished as B1, B2, B3, and B4, and images of the person C in the respective frames are distinguished as C1, C2, C3, and C4.

Further, inFIG.7, the angle added to each image represents the posture of the person in the image. The posture of the person A in the image A1is 0°, and the postures in the images A2, A3, and A4are the same at 90°. Therefore, in the registered object image storage unit1502, the images A4and A1are registered in association with the object identification number of the person A as illustrated inFIG.8, and the images A2and A3are not registered. Similarly, in the registered object image storage unit1502, the images B4, B2, and B1and the images C4and C2are registered in association with the object identification numbers of the person B and the person C as illustrated inFIG.8, and the images B3, C1, and C3are not registered.

When the object re-identification unit1602performs re-identification of an unknown object image X1in the latest frame image181, the object re-identification unit1602re-identifies the unknown object image X1by using the registered object images, illustrated inFIG.8, that are stored in the registered object image storage unit1502in association with the object identification numbers of the persons A, B, and C whose re-identification succeeded in one or more most recent frame images182to185. That is, the object re-identification unit1602calculates the probability that the unknown object image X1shows the same person as that of the image A4by using the learning model having been learned for example, and when the probability is equal to or larger than a threshold, the object re-identification unit1602determines that the person of the unknown object image X1is the person A. Meanwhile, when the probability is less than the threshold, the object re-identification unit1602calculates the probability that the person of the unknown object image X1is the same as that of the image A1, and when the probability is equal to or larger than the threshold, the object re-identification unit1602determines that the person of the unknown object image X1is the person A. When the probability is still less than the threshold, the object re-identification unit1602attempts to re-identify the unknown object image X1by sequentially using the remaining registered images, namely, the images B4, B2, B1, C4, and C2. As described above, in the case of re-identifying one unknown object image X1in the latest frame image181as a registered object image of a person re-identified in the most recent four frames, in the present embodiment, it is possible to limit to seven pieces of registered object images in total (A1, A4, B1, B2, B4, C2, and C4). In contrast, in the configuration in which images of the person re-identified in the most recent four frames are directly used as registered object images, re-identification is performed on twelve pieces of registered object images in total with respect to the unknown object image X1. Therefore, the processing speed of re-identification is lowered.

Further, the object re-identification unit1602may re-identify the unknown object image by preferentially using, among one or more registered object images stored in the registered object image storage unit1502, a registered object image having the posture matching the posture of the unknown object detected from the unknown object image. That is, in the example of the unknown object image X1, the object re-identification unit1602may detect the posture 0° from the unknown object image X1, select the images A1, B4, and C2having the same posture 0° from the registered object image storage unit1502, and firstly use these three images in sequence to re-identify the unknown object image X1. As a result, reduction in the number of times of re-identification required until success of re-identification of the unknown object image X1can be expected, so that the processing speed of re-identification can be much higher.

Second Exemplary Embodiment

Next, a second exemplary embodiment of the present invention will be described with reference to the drawings.FIG.9is a block diagram of an image processing apparatus according to the present embodiment. The present embodiment describes an outline of the image processing apparatus of the present invention.

Referring toFIG.9, an image processing apparatus200according to the present embodiment is configured to include a registered object image storage unit201, an unknown object image acquisition unit202, an object re-identification unit203, and a registered object image update unit204.

The registered object image storage unit201is configured to store therein at least one image of an object as a registered object image in association with an object identification number that uniquely identifies the object. Here, an object may be a person. An object may be an article other than a person. The registered object image storage unit201may have the same configuration as that of the registered object image storage unit1502ofFIG.1for example, but is not limited thereto.

The unknown object image acquisition unit202is configured to acquire an unknown object image that is an image of an unknown object. The source from which an unknown object image is acquired may be a camera image, for example. The unknown object image acquisition unit202may have the same configuration as that of the unknown object image acquisition unit1601ofFIG.1, but it is not limited thereto.

The object re-identification unit203is configured to re-identify an unknown object image acquired by the unknown object image acquisition unit202, by using the registered object image stored in the registered object image storage unit201. The object re-identification unit203may have the same configuration as that of the object re-identification unit1602ofFIG.1for example, but is not limited thereto.

The registered object image update unit204is configured to, when a condition is satisfied, the condition being that all registered object images stored in the registered object image storage unit201in association with the object identification number of the object in an unknown object image whose re-identification by the object re-identification unit203succeeded differ from the unknown object image, add the unknown object image to the registered object image storage unit201as a new registered object image, in association with the object identification number. The registered object image update unit204may have the same configuration as that of the registered object image update unit1603ofFIG.1for example, but is not limited thereto.

The image processing apparatus200configured as described above operates as described below. The registered object image storage unit201stores therein at least one image of an object as a registered object image in association with an object identification number that uniquely identifies the object. Under this state, the unknown object image acquisition unit202acquires an unknown object image that is an image of an unknown object. Then, the object re-identification unit203re-identifies the unknown object image acquired by the unknown object image acquisition unit202by using the registered object image stored in the registered object image storage unit201. Then, when a condition is satisfied, the condition being that all registered object images stored in the registered object image storage unit201in association with the object identification number of the object in the unknown object image whose re-identification by the object re-identification unit203succeeded differ from the unknown object image, the registered object image update unit204adds the unknown object image to the registered object image storage unit201as a new registered object image, in association with the object identification number.

According to the image processing apparatus200that is configured and operates as described above, it is possible to improve the accuracy of re-identification by increasing variations of the registered object images without uselessly increasing the number of the registered object images, that is, without lowering the processing speed of re-identification. This is because when a condition is satisfied, the condition being that all registered object images stored in the registered object image storage unit201in association with the object identification number of the object in the unknown object image whose re-identification by the object re-identification unit203succeeded differ from the unknown object image, the registered object image update unit204adds the unknown object image to the registered object image storage unit201as a new registered object image, in association with the object identification number.

While the present invention has been described with reference to the exemplary embodiments described above, the present invention is not limited to the above-described embodiments. The form and details of the present invention can be changed within the scope of the present invention in various manners that can be understood by those skilled in the art.

INDUSTRIAL APPLICABILITY

The present invention is applicable to re-identification of a person and/or re-identification of an article such as belongings of a person. Moreover, the present invention can be applicable to a technology of detecting events such as desertion, carrying-away, and switching by a person.

Supplementary Note 1

An image processing apparatus comprising:

a registered object image storage unit that stores at least one image of an object as a registered object image in association with an object identification number that uniquely identifies the object;

an unknown object image acquisition unit that acquires an unknown object image that is an image of an unknown object;

an object re-identification unit that re-identifies the acquired unknown object image by using the registered object image stored in the registered object image storage unit; and

a registered object image update unit that, when a condition is satisfied, the condition being that all registered object images stored in the registered object image storage unit in association with the object identification number of the object in the unknown object image whose re-identification by the object re-identification unit succeeded differ from the unknown object image, adds the unknown object image to the registered object image storage unit as a new registered object image in association with the object identification number.

Supplementary Note 2

The image processing apparatus according to supplementary note 1, wherein when the condition is not satisfied, the registered object image update unit deletes the registered object image matching the unknown object image from the registered object image storage unit, and adds the unknown object image to the registered object image storage unit as a new registered object image in association with the object identification number.

Supplementary Note 3

The image processing apparatus according to supplementary note 1 or 2, wherein

when determining whether or not the condition is satisfied, the registered object image update unit compares posture of a registered object detected from each of the one or more registered object images with posture of the unknown object detected from the unknown object image, and when there is no posture of the registered object that matches the posture of the unknown object, the registered object image update unit determines that the condition is satisfied.

Supplementary Note 4

The image processing apparatus according to supplementary note 3, wherein

the registered object image update unit detects the posture of the registered object on a basis of skeletal information detected from the registered object image, and detects the posture of the unknown object on a basis of skeletal information detected from the unknown object image.

Supplementary Note 5

The image processing apparatus according to supplementary note 3 or 4, wherein

the object re-identification unit re-identifies the acquired unknown object image by preferentially using, among the one or more registered object images stored in the registered object image storage unit, a registered object image having the posture matching the posture of the unknown object detected from the unknown object image.

Supplementary Note 6

The image processing apparatus according to supplementary note 1 or 2, wherein

when determining whether or not the condition is satisfied, the registered object image update unit performs clustering on the one or more registered object images on a basis of a given feature value, creates a one-class identification plane for each of clusters obtained by the clustering, determines whether or not the unknown object image is in a class that is same as a class of the cluster by using the created one-class identification plane, and when determining that the unknown object image is in a class that is not same as any of classes of the clusters, the registered object image update unit determines that the condition is satisfied.

Supplementary Note 7

The image processing apparatus according to supplementary note 6, wherein

the registered object image update unit performs the clustering on a basis of skeletal information detected from the registered object image.

Supplementary Note 8

The image processing apparatus according to any of supplementary notes 1 to 7, further comprising

an object re-identification information storage unit that stores an object identification number of an object in the unknown object image in a frame image whose re-identification by the object re-identification unit succeeded, in association with capturing time of the frame image, wherein

when the object re-identification unit re-identifies the unknown object image in a latest frame image, the object re-identification unit re-identifies the unknown object image in the latest frame image by using the registered object image stored in the registered object image storage unit in association with an object identification number of an object whose re-identification succeeded in a most recent frame image stored in the object re-identification information storage unit.

Supplementary Note 9

An image processing method comprising:

acquiring an unknown object image that is an image of an unknown object;

re-identifying the acquired unknown object image by using a registered object image stored in a registered object image storage unit, the registered object image storage unit storing at least one image of an object as the registered object image in association with an object identification number that uniquely identifies the object; and

when a condition is satisfied, the condition being that all registered object images stored in the registered object image storage unit in association with the object identification number of the object in the unknown object image whose re-identification succeeded differ from the unknown object image, adding the unknown object image to the registered object image storage unit as a new registered object image in association with the object identification number.

Supplementary Note 10

A computer-readable medium storing thereon a program for causing a computer to execute processing to:

acquire an unknown object image that is an image of an unknown object;

re-identify the acquired unknown object image by using a registered object image stored in a registered object image storage unit, the registered object image storage unit storing at least one image of an object as the registered object image in association with an object identification number that uniquely identifies the object; and

when a condition is satisfied, the condition being that all registered object images stored in the registered object image storage unit in association with the object identification number of the object in the unknown object image whose re-identification succeeded differ from the unknown object image, add the unknown object image to the registered object image storage unit as a new registered object image in association with the object identification number.

REFERENCE SIGNS LIST

100,200image processing apparatus

130operation input unit

140screen display unit

160arithmetic processing unit

171,173object identification number

172,174registered object image information

201registered object image storage unit

202unknown object image acquisition unit

204registered object image update unit

1502registered object image storage unit

1601unknown object image acquisition unit

1603registered object image update unit

1722,1741registered object image