Data collection and user feedback in edge video devices

A digital video camera architecture for updating an object identification and tracking model deployed with the camera is disclosed. The invention comprises optics, a processor, a memory, and an artificial intelligence logic which may further comprise artificial neural networks. The architecture identifies objects according to a first confidence threshold of the model and identifies candidate objects according to the first confidence threshold and a second confidence threshold. The model may track the motion of the candidate objects within a visual field, separate the candidate objects into false positive candidate objects and false negative candidate objects according to their tracked motions, and present at least a portion of the false positive candidate objects and false negative candidate objects for further annotation.

RELATED APPLICATIONS

This application is related to application Ser. No. 17/515,977, filed on the same day, owned by the same assignee, and is included herein by reference in its entirety.

FIELD

The present disclosure relates to image processing. More particularly, the present disclosure technically relates to efficient training and retraining of artificial neural networks in video data processing in edge video devices.

BACKGROUND

As technology has grown over the last decade, the quantity of time-series data such as video content has increased dramatically. This increase in time-series data has generated a greater demand for automatic object identification and classification. In response, neural networks and other artificial intelligence methods have been increasingly utilized to generate automatic classifications, specific detections, and segmentations. In the case of video processing, computer vision trends have progressively focused on object detection, image classification, and other segmentation tasks to parse semantic meaning from video content. In particular, there is a need to improve the models used for object detection. It is desirable for this process to be automated as much as possible.

Corresponding reference characters indicate corresponding components throughout the several figures of the drawings. Elements in the several figures are illustrated for simplicity and clarity and have not necessarily been drawn to scale. For example, the dimensions of some of the elements in the figures might be emphasized relative to other elements for facilitating understanding of the various presently disclosed embodiments. In addition, common, but well-understood, elements that are useful or necessary in a commercially feasible embodiment are often not depicted in order to facilitate a less obstructed view of these various embodiments of the present disclosure.

DETAILED DESCRIPTION

In response to the issues described above, systems and methods are discussed herein that describe processes for streamlining the process of updating the models of video image data processing within artificial neural networks and other Artificial Intelligence (AI) techniques. Specifically, systems and methods are presented for the improvement of the models employed by digital video cameras with a minimum of human intervention.

In particular, it may be desirable to increase the certainty with which models detect objects and track their motions. A confidence threshold may be used in the models to differentiate between True Positive (TP) detections (e.g., correct detection of an object), False Positive (FP) detections (e.g., incorrect detections of an object), True Negative (TN) detections (e.g., correct detections of the absence of an object), and False Negative (FN) detections (e.g., incorrect detection of an object that is not present or miss an object that is present). Collecting examples of FP and FN cases may be used to periodically train and update the model

In practical applications, digital video cameras may be installed in clusters for purposes of surveillance of an area or areas. A single customer may have multiple installations, and additional customers may increase the installed base even further. The digital video cameras may be coupled to one or more computers which may be (optionally) operated by either the camera owner(s), the camera manufacturer and/or service provider. The amount of stored data may be immense with hundreds of hours of video stored in thousands of cameras in dozens or hundreds of installations worldwide.

In all of that data, there may be mistakes where the model made a FP or FN detection. Ideally, it may be desirable to identify some of these cases to improve the training data used to create the models. Given the volume of data, automation may be necessary to find those errant cases, select some for further annotation. In some embodiments, the further annotation may be performed by a human or a user making a judgement if the FP or FN candidate is really incorrect. In other embodiments, the further annotation may be performed automatically with computational resources greater than available on an individual digital video camera like, for example, a server farm or cloud computing environment with the resources to run a more complex AI and/or have access to a much larger database of annotated images to draw upon. Then the data must be gathered from the user base without querying the users so frequently as to lessen the user experience.

Embodiments of the present disclosure can be utilized in a variety of fields including general video analytics, facial recognition, object segmentation, object detection, autonomous driving, traffic flow detection, drone navigation/operation, stock counting, inventory control, and other automation-based tasks that generate time-series based data. The use of these embodiments can result in fewer required computational resources to produce similarly accurate results compared to a traditional convolutional or other neural network. In this way, more deployment options may become available as computational resources increase and become more readily available on smaller and less expensive electronic devices.

Aspects of the present disclosure may be embodied as an apparatus, system, method, or computer program product. Accordingly, aspects of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, or the like) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “function,” “module,” “apparatus,” or “system.” Furthermore, aspects of the present disclosure may take the form of a computer program product embodied in one or more non-transitory computer-readable storage media storing computer-readable and/or executable program code. Many of the functional units described in this specification have been labeled as functions, in order to emphasize their implementation independence more particularly. For example, a function may be implemented as a hardware circuit comprising custom VLSI circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, field-programmable gate arrays (“FPGAs”) or other discrete components. A function may also be implemented in programmable hardware devices such as programmable array logic, programmable logic devices, or the like.

“Neural network” refers to any logic, circuitry, component, chip, die, package, module, system, sub-system, or computing system configured to perform tasks by imitating biological neural networks of people or animals. Neural network, as used herein, may also be referred to as an artificial or deep neural network (DNN). Examples of neural networks that may be used with various embodiments of the disclosed solution include, but are not limited to, convolutional neural networks, feed forward neural networks, radial basis neural network, recurrent neural networks, modular neural networks, and the like. Certain neural networks may be designed for specific tasks such as object detection and/or image classification. Examples of neural networks suitable for object detection include, but are not limited to, Region-based Convolutional Neural Network (RCNN), Faster Region-based Convolutional Neural Network (Faster R-CNN), You Only Look Once (YOLO), and the like. Examples of neural networks suitable for image classification may include, but are not limited to, Googlenet Inception, Resnet, Mobilenet, Densenet and Efficientnet. A neural network may include both the logic, software, firmware, and/or circuitry for implementing the neural network as well as the data and metadata for operating the neural network. One or more of these components for a neural network may be embodied in one or more of a variety of repositories, including in one or more files, databases, folders, or the like. The neural network used with embodiments disclosed herein may employ one or more of a variety of learning models including, but not limited to, supervised learning, unsupervised learning, and reinforcement learning. These learning models may employ various backpropagation techniques.

Functions or other computer-based instructions may also be implemented at least partially in software for execution by various types of processors. An identified function of executable code may, for instance, comprise one or more physical or logical blocks of computer instructions that may, for instance, be organized as an object, procedure, or function. Nevertheless, the executables of an identified function need not be physically located together but may comprise disparate instructions stored in different locations which, when joined logically together, comprise the function and achieve the stated purpose for the function.

Indeed, a function of executable code may include a single instruction, or many instructions, and may even be distributed over several different code segments, among different programs, across several storage devices, or the like. Where a function or portions of a function are implemented in software, the software portions may be stored on one or more computer-readable and/or executable storage media. Any combination of one or more computer-readable storage media may be utilized. A computer-readable storage medium may include, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing, but would not include propagating signals. In the context of this document, a computer readable and/or executable storage medium may be any tangible and/or non-transitory medium that may contain or store a program for use by or in connection with an instruction execution system, apparatus, processor, or device.

Further, as used herein, reference to reading, writing, loading, storing, buffering, and/or transferring data can include the entirety of the data, a portion of the data, a set of the data, and/or a subset of the data. Likewise, reference to reading, writing, loading, storing, buffering, and/or transferring non-host data can include the entirety of the non-host data, a portion of the non-host data, a set of the non-host data, and/or a subset of the non-host data.

Referring toFIG.1, a conceptual illustration of a video image data processing system100in accordance with an embodiment of the disclosure is shown. In many embodiments, it may be desired to monitor one or more visual areas by installing security cameras150within those areas. The security cameras150can generate a plurality of video image data (i.e., “video content”) that can be processed. In a number of embodiments, the processing of the video image data will be configured to determine if one or more specific objects are within the monitored visual areas. In a number of embodiments, this detection may be presented as an inference map image from an object detection neural network which can be a segmentation or panoptic map. These maps can be utilized as a classification as to whether a specific object is present within the input data or not. These maps can be generated as an output from a neural network such as, but not limited to, a convolutional neural network (CNN). By way of example, and not limitation, video image data processing can be established to detect the presence of one or more pedestrians within the monitored visual areas. It will be understood by those skilled in the art that the video image data processing may be performed by processors internal to security cameras150or elsewhere in the system or in some combination thereof. The video image data processing may be implemented in software operating in conventional processors (e.g., CPU, MPU, GPU, RISC, etc.), and/or software operating in specifically purposed processors optimized to implement neural networks—or some combination thereof. In fact, the entire system may be considered a processor or a distributed processor.

Monitoring video content can be inefficient when done manually. Therefore, various embodiments attempt to minimize the amount of video content that needs to be manually reviewed by attempting to identify and/or determine if one or more specific objects within the video content and then trigger a notification for manual review. Often, the video image data processing system100will process the video image data within a centralized video processing server110, although some embodiments may offload various processing tasks to other devices such as, but not limited to edge network devices140(like, for example, server farms, specialized AI hardware accelerators, online databases, etc.), servers130, or internal to the security cameras150(themselves edge network devices). The video processing server110is often connected to a network120such as the Internet as depicted inFIG.1. A plurality of security cameras150can also be attached to the network120such that they are communicatively coupled to the video processing server110comprising one or more processors like, for example, CPUs, MPUs, GPUs, etc. Although the embodiment ofFIG.1depicts security cameras150, it will be understood by those skilled in the art that any video image data capture device may be utilized as required by the desired application.

The security cameras150can be wired directly to the network120or may be wirelessly connected via one or more wireless access points160. In this way, a variety of potential deployment arrangements may be realized to properly cover the desired areas for surveillance. In theory, there is no limit to the number of deployed security cameras150or other video image data capture devices that may be communicatively coupled with the video processing server110. The limitations experienced may relate to the available bandwidth of the network120and computational resources of the video processing server110. As discussed below, superior performance for system100is for the bulk of the processing to be done locally by security cameras150or other edge network devices140to minimize network traffic and reduce the need for centralized computing resources like video processing server110and servers130.

Referring toFIG.2, a conceptual illustration of an artificial neural network200in accordance with an embodiment of the disclosure is shown. At a high level, the artificial neural network200comprises an input layer220, one or more intermediate layers230, and an output layer240. The artificial neural network200comprises a collection of connected units or nodes called artificial neurons250which loosely model the neurons in a biological brain. Each connection, like the synapses in a biological brain, can transmit a signal from one artificial neuron to another. An artificial neuron that receives a signal can process the signal and then trigger additional artificial neurons within the next layer of the neural network. As those skilled in the art will recognize, the artificial neural network200depicted inFIG.2is shown as an illustrative example and various embodiments may comprise artificial neural networks that can accept more than one type of input and can provide more than one type of output.

In a typical embodiment, the signal at a connection between artificial neurons is a real number, and the output of each artificial neuron is computed by some non-linear function (called an activation function) of the sum of the artificial neuron's inputs210. The connections between artificial neurons are called “edges” or axons. Artificial neurons and edges typically have a weight that adjusts as learning proceeds. The weight increases or decreases the strength of the signal at a connection. Artificial neurons may have a threshold (trigger threshold) such that the signal is only sent if the aggregate signal crosses that threshold. Typically, artificial neurons are aggregated into layers. Different layers may perform different kinds of transformations on their inputs. Signals propagate from the first layer (the input layer220) to the last layer (the output layer240), possibly after traversing one or more intermediate layers (also called hidden layers)230.

The inputs to an artificial neural network may vary depending on the problem being addressed. In object detection, the inputs may be data representing pixel values for certain pixels within an image or frame. In one embodiment the artificial neural network200comprises a series of hidden layers in which each neuron is fully connected to neurons of the next layer. The artificial neural network200may utilize an activation function such as sigmoid, nonlinear, or a rectified linear unit (ReLU), upon the sum of the weighted inputs, for example. The last layer in the artificial neural network may implement a regression function such as SoftMax regression to produce the classified or predicted classifications output for object detection as output260. In further embodiments, a sigmoid function can be used, and position prediction may need raw output transformation into linear and/or non-linear coordinates.

In certain embodiments, the artificial neural network200is trained prior to deployment and to conserve operational resources. However, some embodiments may utilize ongoing training of the artificial neural network200especially when operational resource constraints such as die area and performance are less critical.

Referring toFIG.3, a schematic block diagram of a digital camera in accordance with an embodiment of the disclosure is shown. Digital video camera300may comprise optics310which may further comprise the lenses (not shown), the image sensor used to capture images (not shown), and the support circuitry necessary for capturing successive frames of video data. Processor321may execute various control programs and applications such as digital camera clients330. Such programs and applications may include an operating system, a file system, one or more databases, and a variety of applications, some of which may be artificial intelligence applications.

Volatile memory322may be used by processor321for code execution, scratch pad memory, temporary storage of video data, and the like. Non-volatile memory323may be used by processor321to store the programs, data, and various digital camera clients330. It may also be used as mass storage for the video data captured by optics310. Optional vector processor324may be used for high-speed parallel calculations. In some embodiments, vector processor324may be implemented as part of the artificial intelligence logic340. Vector processor324may be a graphics processing unit (GPU) and/or have a single instruction/multiple data (SIMD) processor architecture and be capable of operating on very long data words like, for example, 128-bits, 256-bits, 512-bits, 1024-bit, or even more in some embodiments. Computer-readable storage medium325may be used by processor321for program storage, data, and other purposes.

Artificial intelligence logic340may be either a hardware function, a software function, or a combination thereof. It may be responsible for managing all artificial intelligence (AI) functions, controlling the artificial neural networks350-1,350-2through350-N and using them for various functions, image processing functions, updating the AI modeling, and the like. Artificial neural networks350-1,350-2through350-N may be neural networks of the sort described in conjunction withFIG.2above and may be implemented in both hardware and/or software.

Referring toFIG.4, a schematic block diagram of a video processing and storage system400in accordance with an embodiment of the disclosure is shown. A customer camera installation410may comprise all of a customer's cameras. They may be coupled directly to a network420such as the Internet, or the cameras may be indirectly coupled to network420through a computer, a gateway, a server, or the like. Network420is used to transmit data back and forth between the camera installation and various remote computers and/or servers. Such data may be related to videos such as video clips, images and frames, metadata associated with the various clips, images, and frames, and the like. Requests for user feedback and the user feedback itself may also be transmitted back and forth as well as any other sort of data needed in the administration of the camera installation.

In some embodiments, the manufacturer of the cameras may also have one or more computers or servers430coupled to network420. This is typically to enhance the customer's user experience with the camera installation by performing various tasks in the background with the customer's permission. Such tasks may include, for example, monitoring the functionality of the various cameras and creating service alerts or performing remote repairs, automatic updates to the camera software or firmware, assisting the customer with various aspects of managing their installed system, etc. In other embodiments, computers/servers430could represent the servers/computers of a service provider who receives data from the cameras and provides various services on the data including analytics and processing. In some instances, the service provider may not be the manufacturer of the cameras.

In particular, the cameras may be deployed with a particular AI model in the software and/or firmware and/or hardware configured, for example, to detect objects (cars, people, packages, etc.), track their motion, identify characteristics of the object (gender, size, color, etc., depending on the identified object), and the like. The model may be trained at the factory, deployed with the camera, and updated periodically during the lifetime of the camera. Typically, the model may consist of the coefficients for an artificial neural network like, for example, the one illustrated inFIG.2. These may be determined by running a training data set of correctly annotated examples through the neural network. The neural network may compare the known correct result to the model output and “learns” by adjusting the coefficients to improve the accuracy of its outputs, in a process commonly known as “machine learning.”

Over time, the quality of the model outputs may degrade. This may occur for a variety of reasons including, but not limited to, changes in traffic patterns, changes in the visual fields of the cameras, changes in time of day, changes of the seasons, etc. To keep the accuracy of the model's outputs high, periodic retraining may be needed. It may be desirable to include new annotated examples from a wide distribution of cameras surveilling a large variety of visual fields and target objects and/or characteristics to the training data set. This may be done by utilizing real annotated examples from the video logs of the cameras deployed in the field. However, this may involve hundreds of hours of video stored on thousands of cameras in dozens or even hundreds of installations. Since human annotation may be needed for the annotated examples in the annotated training set, it may be difficult and/or impossible for humans to manually review all of that data looking for appropriate annotated examples to annotate. The process may be simplified by limiting the annotated examples to cases that the model has had difficulties in making correct identifications. This may reduce the number of potential candidates for new annotated examples.

The feedback and training may be implemented in other ways. In other embodiments, the digital video cameras may be owned and/or operated by, for example, a service company to provide the surveillance for the customer. There may be one or more service company computers or servers coupled to network420(not shown). In such an arrangement, the service company may choose to oversee the feedback and training process internally and separately from the manufacturer. In alternate embodiments, the customer may play the role of the service company and not only own and operate the cameras, but also perform their own feedback and training process. In such embodiments, there will be customer computers and servers (not shown) as part of the customer camera installation410. Persons skilled in the art will also realize there are other such embodiments and/or hybrid embodiments that all fall within the scope of the invention.

Referring toFIG.5, a graph of a Precision/Recall (PR) curve for the object detection task in accordance with an embodiment of the disclosure is shown. The graph500comprises a vertical Precision-axis510and a horizontal Recall-axis512. Both axes range from 0% to 100%. The dashed lines514and516along with the axes form an extent box for the PR curve518. In general, precision may be a measure of the accuracy of a model's predictions while recall may be a measure of its sensitivity. Typically, the curve may represent the increasing of the confidence threshold while moving to the left on PR curve518(and as indicated by the curved dashed arrow520).

Also shown inFIG.5is confidence threshold point T1(at the center of the labeled circle). This may be considered a figure of merit for a particular model. The solid vertical arrow (labeled FP) may indicate the portion of False Positives (FP) returned by the current model at T1. The portion of True Positives (TP) returned by the current model may be indicated by vertical dashed line522(also labeled TP). Vertical dashed line522is an extension of the FP arrow down to the recall-axis.

As the PR curve518moves to the left, the number of FPs decreases while the number of false negatives524(also labeled FN) increases. A FN my occur when an object is actually present, and the model does not correctly detect it. Thus, there may be a tradeoff to be made in choosing the right value for T1.

An object detection module may output such values as class (person, animal, car, etc.), bounding box (the location and extents of the object in a frame), and a confidence score (typically a percentage). T1may be chosen to be the confidence score value that delineates an optimal balance between correct detections (true positives) and false alarms (false positives—or objects detected and not actually present). In order to improve the model, the best candidates to become annotated examples may be objects with confidence scores very near to T1. These are cases that may be the most difficult because they are on the boundary between TPs and FPs.

For choosing annotated example candidates, a second confidence threshold T2may be defined at a distance from T1. The distance between T1and T2increases the number of false positives by ΔFP (represented by the vertical arrow so labeled) and decreases the number of false negatives by ΔFN (represented by the horizontal arrow so labeled). Detected objects that have a confidence score between T1and T2become the pool of annotated example candidates. The actual values of T1and T2are a matter of design choice.

Referring toFIG.6, a flowchart of a process600of data collection and requesting user feedback in accordance with an embodiment of the disclosure is shown. In some embodiments, process600may begin with the necessary metadata already downloaded from installed cameras stored on the camera manufacturer's computers and/or servers (block610). In other embodiments, the digital video cameras may be owned and/or operated by, for example, a service company to provide the surveillance for the customer. There may be one or more service company computers or servers coupled to the customer's camera installation via a network or the Internet where the metadata may be downloaded. In such an arrangement, the service company may choose to oversee the feedback and training process internally and separately from the manufacturer. In some alternative embodiments, the customer may play the role of the service company and not only own and operate the camera installation, but also perform their own feedback and training process. In such embodiments, there will be customer computers and servers as part of the customer camera installation where the metadata may be downloaded. Persons skilled in the art will realize there are other such arrangements and/or hybrid arrangements that all fall within the scope of the invention.

This metadata may include information about the camera installations, locations, the sort of video data archived internal to the camera itself, AI model outputs, etc., and may be used to perform the sampling method and feedback tracking (block620). The sampling method may determine the false positive and false negative candidates while the feedback tracking may determine which candidates are presented for annotation to a user or some other person and when.

A request and/or notification for a candidate for annotation may be presented, and a notification to the user to annotate the candidate may then be sent (block630). To streamline the process for the user, a simple dialog box may be sent with a picture of the frame overlaid with the candidate's bounding box. In the case of a FP request, the dialog box might read, “Click ‘NO OBJECT’ if there is no object in this alert,” and a single response button saying, “NO OBJECT.” In the case of a FN request, the dialog box might read, “Click ‘OBJECT’ if there is an object in this alert,” and a single response button saying, “OBJECT.”

The user may respond to the notification and the results of the feedback annotations may be collected in the manufacturer's servers (block640). The associated image or images may be retrieved from the camera (block650) and transferred to the manufacturers servers where they may be paired up with the annotation and may be added to the training set (block660).

Referring toFIG.7, a flowchart of a process700of data collection and requesting user feedback in accordance with an embodiment of the disclosure is shown. In some embodiments, process700may begin with the necessary metadata already downloaded from installed cameras and stored on the camera manufacturer's servers (block710). In other embodiments, the digital video cameras may be owned and/or operated by, for example, a service company to provide the surveillance for the customer. There may be one or more service company computers or servers coupled to the customer's camera installation via a network or the Internet where the metadata may be downloaded. In such an arrangement, the service company may choose to oversee the feedback and training process internally and separately from the manufacturer. In some alternative embodiments, the customer may play the role of the service company and not only own and operate the camera installation, but also perform their own feedback and training process. In such embodiments, there will be customer computers and servers as part of the customer camera installation where the metadata may be downloaded. Persons skilled in the art will realize there are other such arrangements and/or hybrid arrangements that all fall within the scope of the invention.

This metadata may include information about the camera installations, locations, the sort of video data archived internal to the camera itself, AI model outputs, etc. , and may be used to perform the sampling method and feedback tracking.

A search and analysis may be performed to determine false positive and false negative candidates (block720). Once the FP and FN candidates have been identified, a distribution analysis may be performed to ensure that the candidates that become annotated examples are selected from a broad base of all the installed cameras that participate in the process (block730). This may produce a smaller but most robust set of new annotated examples because of the exposure to a wider range of locations, objects, characteristics, etc.

Soliciting user feedback may be difficult. Too many feedback requests may annoy the user and detract from the user experience. Similarly, too few requests may delay the deployment of new versions of the models which may produce inferior results from the user's camera installation which may also detract from the user experience. There may be a need to balance these factors. In some embodiments, an interval analysis may be performed on a log of previous feedback requests from the cameras that generated the FP and FN candidates (block740). If it has been longer than a specified interval that may be determined by the manufacturer, a service company, or the user, a feedback request may be generated and/or sent out (block750). A user feedback notification may be generated and/or sent based on the feedback request, either to the camera associated with the request or a server monitoring that camera (block760).

Referring toFIG.8, a flowchart of a process800for automatically determining False Positive (FP) and False Negative (FN) candidates in accordance with an embodiment of the disclosure is shown. Process800may begin by performing an analysis of object detection results (block810). This analysis may be performed in the installed camera base and/or in the camera manufacturer's servers.

An object detection result may include the class of an object, its bounding box, its motion, and a confidence score. The currently deployed model will have a confidence threshold point (referred to as T1) on the PR curve that differentiates between true positives and false positives based on the confidence score. A second confidence threshold (referred to as T2) may also be defined for purposes of training at a different and lower confidence point on the PR curve. This may identify candidate objects that were originally true positives under T1but were hard for the model to determine. It may be determined if:
T1>Object's Confidence Score>T2  (block815)

If not, the object may not be a candidate, and no further processing may be required (block830). If so, then the object may be identified as an annotated example candidate (block820).

Referring toFIG.9A, a conceptual diagram of a process for automatically determining False Positive (FP) candidates in accordance with an embodiment of the disclosure is shown. Diagram900may comprise video data910that may be analyzed for motion by the motion model920and be filtered to determine FP and FN candidate objects by the candidate model930.

The motion model920may be part of the object detection model deployed with installed digital video cameras. Motion model920may compare multiple neighboring frames in a video sequence and may use differences between two or more successive neighboring frames to track the motion of detected objects. An output of motion model920may be a Hadamard matrix of motion922where a logic-1 may represent an area of motion and a logic-0 may represent an area of non-motion. These areas of motion and non-motion may represent a single pixel or a group of pixels.

An output of candidate model930may be a single video frame932with a bounding box934which may indicate the presence and extent of a FP candidate object in frame932. The frame932may be overlaid upon the Hadamard matrix922to form a comparison frame924. In this case, the bounding box934falls in an area of substantially non-motion (designated by all logic-0 values inside of bounding box934). One or more previous or subsequent neighboring frames may be analyzed for the presence of bounding box934. If the bounding box934is present in the previous or subsequent neighboring frame or frames, that means the bounding box934may represent a static object and may represent a correctly identified object (or True Positive). Finding an object identified at the same location in multiple neighboring frames prior or subsequent to frame932that is also present at the location of bounding box934in frame932is to be expected if an object is truly there. Thus, if the bounding box934is not present in the previous or subsequent frame or frames, the appearance of bounding box934in frame932may be unexpected and thus may be a possible false positive because an object may not actually be at that location. In this latter case, frame932is retained for annotation as a FP candidate.

Referring toFIG.9B, a conceptual diagram of a process for automatically determining False Negative (FN) candidates in accordance with an embodiment of the disclosure is shown.FIG.9Bis nearly identical toFIG.9A. Frame932and bounding box934have been removed, while a frame942with a bounding box944and an area of motion946have been added. The area of motion946(an area with a preponderance of logic-1s in comparison frame924) may not be present in frame942, and its presence in the figure may be to illustrate its relative physical location relative to frame942and bounding box944.

Bounding box944may be detected in frame942, while an area of motion946may absent but be detected in one or more previous or subsequent neighboring frames. In this example, the bounding box944may substantially overlap the area of motion946. If an object in the area of motion946is not detected in frame942but is present in previous or subsequent neighboring frames where bounding box944is absent, a false negative candidate may be present. Finding area of motion946in frame942without a moving object present is unexpected, since an object may be present that was not correctly identified by the model. In this case, frame942is retained for annotation as a FN candidate.

Referring toFIG.10, a flowchart of a process1000for automatically determining False Negative (FN) candidates in accordance with an embodiment of the disclosure is shown. Process1000may begin by performing an analysis of object detection results (block1010). This analysis may be performed in the installed camera base and/or in the camera manufacturer's servers.

An object detection result may include the class of an object, its bounding box, one or more of its characteristics, its motion, and a confidence score. The currently deployed model will have a confidence threshold point (referred to as T1) on the PR curve that differentiates between true positives and false positives based on the confidence score. A second confidence threshold (referred to as T2) may also be defined for purposes of training at a different and lower confidence point on the PR curve. This may identify candidate objects that were originally true positives under T1but were hard for the model to determine. It may be determined if:
T1>Object's Confidence Score>T2  (block1015)

If not, the object may not be a candidate, and no further processing may be required (block1070). If so, then the object may be identified as either a false positive or false negative candidate (block1020). The bounding box of the object may be determined (block1030), the bounding box being the location and extents of the object. A motion model may be used to determine the motion of the object (block1040).

A determination may be made if the object's bounding box is present in previous frames in an area of motion (block1045). If so, a request for a false negative is sent (block1050) and no further processing is required (block1070). If so, a determination may be made if the object's bounding box is present in previous frames in an area of non-motion (block1055). If not, a request for a false positive is sent (block1060) and no further processing is required (block1070). If so, then no further processing is required (block1070).

Referring toFIG.11, a flowchart of a process1100for automatically determining the presentation of False Positive (FP) and False Negative (FN) candidates to a user in accordance with an embodiment of the disclosure is shown. Process1100may begin with the detection of some number of false positive and false negative candidates (block1110). An analysis may be performed to determine the distribution formation of the candidates (block1120). It may be desirable to have a high level of diversity in the FP and FN candidates submitted for annotation. Ideally, the candidates would be taken from thousands of cameras in hundreds of installations all coupled to the camera manufacturer's servers. A diversity of annotated candidates may make for more robust training of the next generation model.

In the case of a particular FP or FN candidate, a determination is made if the candidate follows the distribution (block1125). If not, no feedback request is needed, and the process can return to detecting false positive and false negative candidates (block1110). If so, a determination is made if a sufficient time has elapsed since the last feedback request for that camera or installation (block1130). If not, no feedback request is needed, and the process similarly goes back to detecting false positive and false negative candidates (block1110). If so, a feedback request is sent (block1140).

Information as herein shown and described in detail is fully capable of attaining the presently described embodiments of the present disclosure, and is, thus, representative of the subject matter that is broadly contemplated by the present disclosure. The scope of the present disclosure fully encompasses other embodiments that might become obvious to those skilled in the art, and is to be limited, accordingly, by nothing other than the appended claims. Any reference to an element being made in the singular is not intended to mean “one and only one” unless explicitly so stated, but rather “one or more.” All structural and functional equivalents to the elements of the above-described preferred embodiment and additional embodiments as regarded by those of ordinary skill in the art are hereby expressly incorporated by reference and are intended to be encompassed by the present claims.