Patent ID: 12190590

DETAILED DESCRIPTION

FIG.1illustrates an image processing device1which is configured to execute a method of determining images plausible to have a false negative object detection. The image processing device1comprises an image sensor10, an image processing pipeline (IPP) section11, a tracker12, an object detector14, a memory15, a processor16, an encoder17, and a transmitter18. These are all known components to include in an image processing device. The image processing device1is also configured to transmit data, via the transmitter18, to a server19. The server19may have a connection to the object detector14which will be explained in detail later.

First, the general functions of the image processing device1will be discussed. The main function of the image processing device1is to acquire images, and in particular image sequences that correspond to video. The purpose of acquiring video may be surveillance monitoring. The image sensor10produces raw image data that is processed by the IPP11. The IPP11may process the raw image data by adjusting, e.g., white balance and noise parameters. Next, processed image data corresponding to an image sequence is analyzed by the tracker12and the object detector14. The purpose of the tracker12is to determined tracks of the image sequence. By track is meant a movement of an image area having a certain pixel pattern through the image sequence. The purpose of the object detector14is to find objects in the images of the image sequence.

The object detector14and the tracker12may execute neural networks for performing tracking and object detection. The neural networks may be run locally in the image processing device1or be run remotely and accessed by the image processing device1when needed. A combination of local and remote execution may also be possible, for example by executing one neural network in the image processing device1and one neural network remotely.

Each of the object detector14and the tracker12is known components for finding tracks and objects in image sequences. It is also known to implement these components in an image processing device such as the one illustrated inFIG.1.

Tracks determined by the tracker12and object detections determined by the object detector14are forwarded to the processor16. The processor16is configured to evaluate the object detections and the tracks to determine if there are matches therebetween. In other words, the processor16tries to find matching pairs between object detections and tracks. A condition for successful pairing is that the object detection and the track are caused by the same object in the depicted scene. For example, a person walking in the scene will, in good circumstances, cause a track determined by the tracker12and an object detection determined by the object detector14. The person's track and object detection will be matched by the processor16. The processor16uses well-known methods for evaluating tracks and object detections for finding matching pairs. The evaluation may include calculating an Intersection over Union (IoU) value between the track and object detection. A minimum IoU value may be set as condition for a matching pair. Information about the determined matching pairs can be sent as metadata together with the image sequence or be used for other purposes such as adding overlays to images.

The encoder17encodes the processed image data in a conventional manner. Various video compression standards may be used, including AVC and HEVC.

Going more into details, a method of determining images plausible to have a false negative object detection will now be disclosed with further reference toFIG.3. The method begins with the already discussed steps of tracking S01aand object detection S01bbased on acquired images of an image sequence. The processor16evaluates the determined tracks and object detections to find matching pairs which is represented by step S02inFIG.3. The method of finding images plausible to have false negative object detections is based on first finding tracks that have no matching object detections and then evaluating if it is plausible, based on historic matches, that a matching object detection has been missed by the object detector14for some reason. For this purpose, an unmatched track, i.e., a track that has not been matched with an object detection in step S02, is compared in a step S04with reference tracks that each represents one or more historic tracks. If a match is found, it is plausible that the unmatched track was supposed to be matched to an object detection of the object class associated with that reference track.

The method thus needs access to reference tracks representing historic tracks32. These will now be explained in more detail with further reference toFIG.2that illustrates relationships between historic tracks32, object classes, and historic trajectories34. In this figure, three historic trajectories A, B, and C are formed and comprises historic tracks1-6and associated object classes OC A-C. The historic tracks1,4, and6are all associated with the object class OC A and comprised in the historic trajectory A. The historic track3is associated with the object class OC B and comprised in the historic trajectory B. The historic tracks2and5are associated with the object class OC C and comprised in the historic trajectory C. Thus, each of the historic tracks1-6has been matched with an object detection. The object class OC of the matching object detection is the same for all historic tracks32within the same historic trajectory. A reference track represents historic tracks32of a single object class. Different reference tracks typically represent historic tracks32of different object classes. Thus, each reference track can be said to be associated with an object class and the reference track together with the associated object class form a historic trajectory34.

The historic trajectories34may be stored as a database or a list, or in any other suitable format, in the memory15of the image processing device1. Alternatively, the group of historic trajectories34may be located remotely and accessed by the processor16when needed. The historic tracks32may also be stored in the same manner, if needed for the method. For example, in an embodiment where the reference tracks are updated based on added or removed historic tracks32, the historic tracks32may need to be stored as well. As input to the method, at least the group of historic trajectories34are provided for the evaluation of unmatched tracks. As indicated, the group of historic trajectories34may be regularly updated. For example, tracks that match an object detection may be added to the historic trajectory34having the object class of matching object detection. The reference track of that historic trajectory34is updated, as indicated by step S03, based on the newly added historic trajectory34such that the reference track represents this historic track32as well. In another example, the historic trajectories34are updated on a regular basis to discard historic tracks32above a certain age. At the same time, the reference tracks are updated or regenerated to reflect the updated historic trajectories34. Thus, the reference tracks adapt to any changes of object travel path in a scene, for example if a sidewalk is rebuilt into a combined sidewalk and bicycle lane or a road is temporally relocated due to road constructions.

The method is thus based on a realization that images plausible to have a false negative object detection can be identified by finding images with unmatched tracks that are similar to historic tracks32with matching object detections. Based on this approach, unmatched tracks that are plausible to have a missed object detection can be filtered out from all unmatched tracks by finding a matching reference track. Further, since the reference track is associated with an object class, it is also possible to have a qualitative guess of which object class the missed object detection would have if caught.

The outcome from the comparison of the step S04between the unmatched track and reference tracks is either that a match is found or that a match is not found. In case a match is found, the method continues with a step S06of defining images that the unmatched track belongs to as being plausible to have a false negative object detection for the object class of the historic trajectory34that includes the matching reference track. In other words, the images that the tracker have determined the unmatched track from are defined as being plausible to have a false negative object detection if the unmatched track matches a reference track. These images may be added to a group of images that is stored in the memory15, as indicated by step S08. The group of images may be transmitted S10, by the transmitter18, to the server19. The transmission may be made on condition that the group of images has reached a certain number of images to avoid sending groups of images that are too small to be useful for an intended application. For example, the group of images may be used for forming training data for an object detector. For the training data to be useful, the number of images therein should exceed a certain threshold.

With or without the step S10of transmitting the group of images, the method may comprise a step S12of verifying the images that have been filtered out to be plausible to have a false negative object detection. The verification may be performed in an automated or manual manner. For example, the images may be verified by manually going through the images to confirm whether there is an object corresponding to the track. Alternatively, the images may be analyzed by a high precision object detection algorithm. Such an object detection algorithm has a high processing power demand. The verification may be performed on a remote server having high processing capacity. In one embodiment, multiple object detection algorithms may be set to analyze the filtered images. A condition for the verification may in that case be set as a minimum number of positive outcomes from the algorithms with respect to the suspected missing object.

As mentioned, the group of images, with or without verification, may be used for training an object detector, and in particular the object detector14of the image processing device1. For this purpose, training data comprising the image plausible to have a false negative object detection associated with their presumed object classes may be formed. By performing an additional training of the object detector14, its performance may be increased. The additional training may be performed by the server19that has access to the object detector14. However, the actual training may be performed in the server19. For example, if the object detector is in the form of a neural network, a copy of the neural network may be downloaded to the server19and additionally trained with the training data before uploading it again to the object detector14. The additional training and updates of the object detector14may be performed on a regular basis or be triggered by receiving a group of images from the image processing device1. The training data could also be used for training object detectors of other image processing devices. In particular, image processing devices having a similar field of view may benefit from the training. For example, if several cameras in a camera system monitor the same scene and have overlapping fields of view, at least to a great extent, all these cameras may share the same training data. However, all these cameras do not necessarily take part in forming the training data, instead it might be that only one or a couple of these cameras contribute images to the training data.

Going back toFIG.3, the other outcome from the comparing step S04is that no match is found between the unmatched track and the reference tracks. In that case, and as indicated by step S05, the unmatched track is defined as a false positive track, i.e., as a track that has been determined by the tracker12without there being an object in the scene causing the track.

Thus, by the step of comparing S04unmatched track with reference tracks, true positive tracks for which an object detection is missing are separated from false positive tracks. Thus, false positive tracks can be discarded and not cause any unnecessary verification. Alternatively, for applications where it may be interesting to find false positive tracks, for example in order to improve the tracker's12performance, the method is beneficial in that those tracks are filtered out from the other tracks.

Going more into detail of the comparison between an unmatched track and reference tracks, exemplifying embodiments will now be disclosed with further reference toFIGS.4A-4CandFIGS.5A-5C.

FIG.4Aillustrates a scene as viewed by a camera comprising an image processing device1ofFIG.1. The scene includes static objects including buildings41and moving objects including pedestrians42, a cyclist43, and a car44. The camera is continuously performing tracking and object detection on acquired image sequences depicting the scene. The resulting tracks and object detections are evaluated to find matches as discussed above.FIG.4Billustrates what tracks with matching object detections can look like. Different line dash formats represent different pluralities of tracks. Tracks that are matched with object detections of the object class pedestrians walking on a sidewalk are illustrated as solid lines47in the upper part of the view. Cars moving through the scene on a road have caused tracks illustrated with larger dashed lines46in the middle of the view. In the lower right part of the image, cyclists moving on the bicycle lane have caused tracks illustrated with smaller dashed lines47. The illustrated tracks form historic tracks32for the scene. Together with the object class of the matched object detection, the historic tracks32are part of different historic trajectories34based on their object class. It is noted that multiple historic trajectories34may exist with the same object class. For example, the dashed lines46may represent two pluralities of tracks that represent cars that have followed the straight part of the road in a first plurality of tracks and cars that have followed the curved part of the road in a second plurality of tracks. This plurality of tracks can each form a historic trajectory34with the object class car. In a similar example, if a scene depicts two separate sidewalks, there may exist one plurality of historic tracks32for each sidewalk area for the object class pedestrians.

If a matching object detection is not found for a track, this unmatched track is compared with reference tracks for determining whether the unmatched track is a false positive track, or a positive track caused by an object that was missed by the object detector. As described above, this is done by comparing the unmatched track with reference tracks450,460, and470illustrated inFIG.4C. In this example, the dashed lines46form one plurality of historic tracks32and thus there exist one reference track460for these tracks. The reference tracks450,460,470may be formed by averaging the plurality of historic tracks32of each historic trajectory34. The reference tracks450,460,470may be seen as a track that represents all historic tracks32in a plurality of historic tracks32for an object class along an area in the scene. A reference track may be compared to a forest path which has been formed by people walking along a forest area. However, similar to the forest path, the reference track does not necessarily correspond exactly to any one of the historic tracks32.

FIG.4Calso indicates tolerance intervals along points of each reference track450,460,470. These tolerance intervals are used for an embodiment of the comparison between unmatched tracks and the reference tracks450,460,470. In order for an unmatched track to match any of the reference tracks450,460,470, the unmatched track should be within the tolerance interval at every point, or at least at a minimum number of points, for a certain reference track. The tolerance intervals may be determined based on how much historic tracks32are spread out at the different points. If the spread is lower, a lower tolerance interval may be set compared to if the spread is higher.

According to another embodiment, in order for an unmatched track to match any of the reference tracks450,460,470, the total deviation distance between the unmatched track and a certain reference track should be below a predetermined threshold. The deviation distance is the difference, as can be determined by the images, between the tracks at a certain point. The total deviation distance is the sum of all deviation distances for a certain reference track. The comparison can be implemented as determining the smallest possible difference between the reference track and the unmatched track at a number of points along the reference track and summing the determined differences into a total deviation distance for comparison with a threshold.

FIG.5Aillustrates another scene that is monitored by a camera with the same configuration as the previously discussed cameras. The scene includes a road with moving vehicles51and a sidewalk with pedestrians52. Tracking and object detections of the camera provides the historic tracks32as illustrated inFIG.5B. As in previous embodiment, these tracks have been matched with objects detections. The dashed lines53indicate tracks that have been matched with object detections comprising the object class car, and the solid lines54indicate tracks that have been matched with object detections comprising the object class pedestrian.

The tracks53,54are represented by reference tracks530,540that are illustrated inFIG.5C. In this embodiment, the comparison of an unmatched track against the reference tracks530,540is performed by checking if the starting points55a,56aor end points55b,56bof the unmatched track and a certain reference track is within a predetermined distance from each other. In other words, the unmatched track is defined as matching a reference track if the distance between the starting points55a,56aor the end points55b,56bof the tracks are sufficiently close to each other. For increased accuracy, it may be desired to configure a condition comprising that a starting point of the reference track is within a first threshold distance from a starting or end point of the unmatched track and, in addition, that an end point of the reference track is within a second, same or different from the first, threshold distance from the other end of the unmatched track. In other words, an unmatched track must begin and end at similar positions in the image as a certain reference track in order to match that reference track. In a similar embodiment, it may suffice that the end points of an unmatched track are sufficiently close to end points of a reference track. In other words, the direction of the tracks that are compared are not taken into account in the comparison.

A comparison of the unmatched track with reference tracks which is based on comparing starting or end points is a quick evaluation method which requires little processing. It is thus advantageous to use, provided that the scene is of a suitable structure. In the example ofFIGS.5A-5C, the scene comprises distinct areas where object enter or leave the scene which makes it a suitable candidate for an approach of comparing starting/end points.

It is noted that the illustrated embodiments provide simplified versions of scenarios in that the number of historic tracks32is low and that the pluralities of historic tracks32are located separated from each other. In a real implementation, historic tracks32of different object classes may be located in overlapping image areas. An unmatched track may thus be determined to be similar to a plurality of reference tracks of different object classes. In that case, all object classes of sufficiently similar reference tracks may be provided as object classes for a potential missed object detection. In a verification process of the images that are plausible to have a false negative object detection, it can be determined if there was a missed object detection and also of which object class of the object class candidates.

According to one embodiment, a system of cameras is provided where each camera implements the method according toFIG.3. The system of cameras may be configured to monitor the same scene with overlapping fields of view, or be configured to monitor the same type of scene, for example different segments of a road. The historic trajectories34are shared between the cameras of the system. In case the cameras are monitoring the same scene, the historic trajectories34may be transformed before they are applied to images of any of the cameras in the system. The transforms are camera specific and designed to adapt the historic trajectories34in view of the different viewing angles that the cameras of the system may have of the scene. In other words, the historic trajectories34are transformed into the form or shape they would have had if they were found in images of a specific camera before being applied in filtering of images being plausible to have a false negative object detection from images captured by that camera.

Thus, unmatched tracks from any of the cameras in a system may be compared with the same reference tracks, or with camera specific transformed reference tracks that originate from the same reference tracks. The historic trajectories34may be updated by any camera of the system, meaning that all cameras can benefit from an update from a single camera within the system.

It may be noted that although this disclosure relates to the method of determining images plausible to have a false negative object detection as executed by an image processing device, or a camera comprising such a device, the method may also be executed in other suitable processing devices. For example, the method may be executed on a server to which the image sequence is provided by one or more cameras. Moreover, it is appreciated that the concept of the invention may be embodied in different forms, for example in one embodiment historic trajectories34can have a common reference track. The reference track is thus the same for a plurality of historic trajectories34. This may be the case in a scenario when monitoring a scene of a road where vehicles of different object classes cause similar tracks. Upon an unmatched track being determined to match with such a reference track, all object classes related to that reference track may be set as object class to the images of the determined track as being plausible to have a false negative object detection.

Thus, the concepts set forth herein should not be limited to the shown embodiments but should only be defined by the appended claims.