CUSTOM EVENT DETECTION FOR SURVEILLANCE CAMERAS

A system trains and uses event recognition models for recognizing custom events types defined by a user within a camera feed of a surveillance camera, The camera can be fixed-view, with a relatively constant position and angle, and the background of the video images video can be likewise relatively constant. A user interface receives, from a user, positive and negative samples of the event in question, such as a designation of live or pre-recorded portions of a camera feed as being positive or negative examples of the event in question. Based on the samples, the user system trains an event recognition model (e.g., using few-shot learning techniques) to detect occurrences of custom event types in the camera feed. A response is performed based on detected occurrences of the event. The user can flag mistakes (false positive or false negative) which can be incorporated into the model to enhance its accuracy.

BACKGROUND

Field of the Various Embodiments

This disclosure relates generally to the field of electronic data processing, and more specifically, to custom event detection for surveillance cameras.

Description of the Related Art

Many users have (or can easily obtain) surveillance cameras for their homes or other locations to monitor the state of those locations, Users may wish that particular events—such as a pet hopping on a couch or otherwise misbehaving, a person picking up a package or other comparatively complex events—could be recognized by such systems. However, recognizing events is much more complex than recognizing individual static objects, and camera visual recognition systems have thus far been limited to the latter type.

As the foregoing illustrates what is needed are improved techniques for custom event detection for surveillance cameras.

SUMMARY

In some embodiments, a computer-implemented method for detecting events by a video camera includes accessing a training data set of training data samples, each training data sample including at least one image obtained from the video camera and an indication of an occurrence of an event within the at least one image; training an event recognition model to generate the indication of the occurrence of the event within each training data sample of the training data set; applying the event recognition model to a camera feed of the video camera to generate an indication of an occurrence of the event in the camera feed; and performing a response based on the indication of the occurrence of the event in the camera feed.

In some embodiments, one or more non-transitory computer readable media stores instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of accessing a training data set of training data samples, each training data sample including at least one image obtained from a video camera and an indication of an occurrence of an event within the at least one image; training an event recognition model to generate the indication of the occurrence of the event within each training data sample of the training data set; applying the event recognition model to a camera feed of the video camera to generate an indication of an occurrence of the event in the camera feed; and performing a response based on the indication of the occurrence of the event in the camera feed.

In some embodiments, a system includes a memory that stores instructions, and a processor that is coupled to the memory and, when executing the instructions, is configured to access a training data set of training data samples, each training data sample including at least one image obtained from a video camera and an indication of an occurrence of an event within the at least one image; train an event recognition model to generate the indication of the occurrence of the event within each training data sample of the training data set; apply the event recognition model to a camera feed of the video camera to generate an indication of an occurrence of the event in the camera feed; and perform a response based on the indication of the occurrence of the event in the camera feed.

At least one technical advantage of the disclosed techniques is that the event recognition model is trained to detect specific types of events that indicated by the user. Training the event recognition model based on events that are of interest to the user can expand and focus the range of detected event types to those that are of particular interest to the user, and to exclude events that are not of interest to the user. As another advantage, training the event recognition model on the camera feed of a camera can enable the camera to detect events within the particular context of the camera and camera feed, such as a particular room of a house or area of a factory. As yet another advantage, performing responses based on the trained event recognition model can enable a surveillance system to take event-type-specific responses based on the events of interest to the user. These technical advantages provide one or more technological improvements over existing techniques for event detection for surveillance cameras.

DETAILED DESCRIPTION

The disclosed embodiments involve the use of event recognition models to recognize events that are visible to surveillance cameras. Surveillance cameras are widely used for consumer, commercial, and industrial applications. In such scenarios, basic event detection framework (based on motion detection or specific objects detection such as person detection) can lead to many false alarms. For example, a user who is using a doorbell or a camera at their home entrance may not be interested to know every time there is a motion at their door or every time a person is passing by which could result in hundreds of alerts per day. The techniques disclosed herein provide a custom event detection in which an event recognition model is trained to determine occurrences of events that a user cares about, such as a person picking up a package, which can significantly reduce the number of false alarms.

FIG. 1is a system100configured to implement one or more embodiments. As shown, a server102within system100includes a processor104, a memory106, and an event recognition model118. The memory106includes a data sample set106, including one or more event labels112associated with each of one or more camera images108. The memory106includes a set of camera images108, a user interface110, one or more event labels112, a training data set114, and a machine learning trainer116. The server102interacts with a camera120that generates a camera feed122, As shown, the system100interacts with the camera120, but in some embodiments (not shown), the camera120is included in the system100.

The camera120produces a camera feed122of camera images108, which can be individual images and/or video segments including a sequence of two or more images. The camera120can produce a camera feed of a monitored scene130, such as a portion of the user's home such as a living room or deck, commercial space, a factory assembly line, or the like. In some embodiments, the camera120is used in a fixed view manner, in that that its position and orientation do not change (or change very little) during its use, so that the area captured by the camera is relatively constant, and thus the background image also tends to be constant (assuming that the background itself is essentially static, such as a wall, a deck or other scene with primarily stationary objects). For example (without limitation), a typical security camera has a fixed (or nearly-fixed) viewpoint, Thus, in such cases, after a surveillance camera is installed, the scene/environment stays the same and the background is mostly constant except for slight changes in the angle or location of the camera. The relatively unchanging viewpoint and background of a surveillance camera can make it possible to identify events with greater precision than would be possible in other contexts with more movement and variation.

For each of the still images and/or video segments, the processor140executes the user interface110to receive, from a user, a selection of an event label112as an indication of an occurrence of an event within the image and/or video segment. In some embodiments, the user indicates a portion of an image in which the event occurs, such as a rectangular or free-form boundary designating an area of the image in which the occurrence of the event is visible. In some embodiments, the user indicates a portion of a video segment in which the event occurs, such as a rectangular or free-form boundary designating an area of the video segment in which the event occurs, and/or a subset of the two or more images during which the event occurs. Some embodiments of the user interface are further shown inFIG. 2.

The event labels112selected for the one or more camera images108of the training data set114can include a variety of event types in a variety of use cases. In some embodiments, the system100allows users of surveillance cameras to create custom alerts for specific events they care about. As an example (without limitation), in home monitoring scenarios, events that can be visible to a surveillance camera and recognized include an entrance door being left open, a faucet being left running, a person dropping off or picking up a package from a porch, a garage door left open, a car door left open, a person getting into a car in the driveway, a light being left on, and/or a hot tub being left uncovered. As an example (without limitation), in pet care scenarios, events that can be visible to a surveillance camera and recognized include a dog getting on furniture, a dog chewing on shoes, a dog defecating in the house, and/or a cat scratching the couch. As an example (without limitation), in elderly care scenarios, events that can be visible to a surveillance camera and recognized include an elderly person falling and/or an elderly taking medications. As an example (without limitation), in childcare scenarios, events that can be visible to a surveillance camera and recognized include a toddler climbing furniture, an infant lying on its stomach, and/or a kid playing with a knife. As an example (without limitation), in commercial scenarios (e.g., security in stores, malls, airports, train stations, patient care), events that can be visible to a surveillance camera and recognized include luggage being left unattended in an airport terminal, a person carrying a large item onto a train, and/or a person carrying a weapon into a mall. As an example (without limitation), in industrial scenarios (e.g.. manufacturing/assembly lines, power plants), events that can be visible to a surveillance camera and recognized include a fire in a plant, machinery being jammed in a manufacturing line, and/or an item being mis-assembled in an assembly line.

The camera images108and the selected event labels112for each camera image108(e.g., each individual image and/or video segment) comprise a training data set114. As shown, the training data set114is stored by the server102; however, in some embodiments, the training data set114is stored outside of the server102and is accessed by the server, such as (without limitation) over a wired or wireless network.

The processor104executes the machine learning trainer116to train an event recognition model118using the training data set114. The event recognition model118can be a neural network including a series of layers of neurons. In various embodiments, the neurons of each layer are at least partly connected to, and receive input from, an input source and/or one or more neurons of a previous layer. Each neuron can multiply each input by a weight; process a sum of the weighted inputs using an activation function; and provide an output of the activation function as the output of the artificial neural network and/or as input to a next layer of the artificial neural network. In some embodiments, the event recognition model118can include a convolutional neural network (CNN) in which convolutional filters that are applied by one or more convolutional layers to the input; memory structures, such as long short-term memory (LSTM) units or gated recurrent units (GRU); one or more encoder and/or decoder layers; or the like. For example, a deep convolutional neural network (CNN), such as a Res Net model, can be used as a “backbone” machine learning model that is trained to classify images for different tasks using a large dataset. Using the outputs of the last hidden layer as embeddings for a given input image, another machine learning model (e.g., a support vector machine (SVM)) can be trained using the positive and negative samples provided by the user for the event. The CNN model can be applied to one or more images from the camera120to generate embeddings, and the second classifier can be applied to the embeddings to determine an occurrence of the event. In various embodiments, the event recognition model118includes one or more other types of models, such as, without limitation, a Bayesian classifier, a Gaussian mixture model, a k-nearest-neighbor model, a decision tree or a set of decision trees such as a random forest, a restricted Boltzmann machine, or the like, or an ensemble of two or more event recognition models of the same or different types. In various embodiments, the event recognition model118can perform a variety of tasks, such as, without limitation, data classification or clustering, anomaly detection, computer vision (CV), semantic analysis, knowledge inference, or the like.

In some embodiments, the event recognition model118includes one or more binary classifiers, where each binary classifier outputs a probability of the training data sample to include an occurrence of an event of a particular event type. An occurrence of an event can be detected based on a maximum probability among the probabilities generated by the respective binary classifiers. In some other embodiments, the event recognition model includes a multi-class classifier that outputs, for a plurality of event types, a probability of the training data sample to include an occurrence of events of each of several event types. An occurrence of an event can be detected based on a maximum probability among the probabilities generated by the multi-class classifier for each of the event types. A CNN model can be applied to the images to generate the embeddings, and a second multi-class classifier is applied to determine an occurrence of any of the events.

In some embodiments, detection can be based on the maximum probability exceeding a probability threshold, e.g., a minimum probability at which confidence in the detection of an occurrence of the event is sufficient to prompt a response126. Alternatively or additionally, in some embodiments, a “null” event type can be defined to indicate a non-occurrence of any of the events, and a determination of an occurrence of a “null” event (e.g., with the highest probability among the event types) can indicate a non-occurrence of the other event types. The “null” event type can reduce the incidence of false positives.

As shown, the machine learning trainer116is a program stored in the memory106and executed by the processor104to train the event recognition model118. The machine learning trainer116trains the event recognition model118to output predictions of event labels112for camera images108included in the training data set114. For each camera image108(e.g., each still image or video segment), the machine learning trainer116compares the event label112received through the user interface110with an event label112predicted by the event recognition model118. If the associated event label112and the predicted event label112do not match, then the machine learning trainer116adjusts the internal weights of the neurons of the event recognition model118. The machine learning trainer116repeats this weight adjustment process over the course of training until the prediction124of the event label112by the event recognition model118is sufficiently close to or matches with the event label112received through the user interface110for the camera image108. In various embodiments, during training, the machine learning trainer116monitors a performance metric, such as a loss function, that indicates the correspondence between the associated event labels112and the predicted event labels112for each camera image108of the training data set114. The machine learning trainer116trains the event recognition model118through one or more epochs until the performance metric indicates that the correspondence of the event labels112received through the user interface110and the predicted event labels112is within an acceptable range of accuracy. The trained event recognition model118is capable of making predictions124of event labels112for unlabeled camera images108in a manner that is consistent with the associations of the training data set114.

In some embodiments, the machine learning trainer116can train the event recognition model118based on temporal information, such as the chronological sequence of two or more images in a video segment. In such embodiments, the event recognition model118can include, in addition to a “backbone” portion such as a convolutional neural network, one or more RNN based layers (including but not limited to LSTM cells) that capture the sequential nature of the data. In such embodiments, the short video clips or sequential frames tagged by the users can be used as samples and fed into the backbone portion of the event recognition model118.

In some cases, two or more camera feeds122from two or more cameras120can be monitored to detect events, such as cameras at different locations within a facility. In some embodiments, the machine learning trainer116trains an event recognition model118based on the camera feeds122of a plurality of cameras120. Similarly, in some embodiments, the system100applies an event recognition model118to the camera feeds122of a plurality of cameras120. In some other embodiments, the machine learning trainer116trains an event recognition model118for each camera feed122of a subset of multiple cameras120, including one camera120of the plurality of cameras120. Similarly, in some embodiments, the system100applies a different event recognition model118to each camera feed122of a plurality of cameras120, where the event recognition model118has been trained specifically on the camera feed122of the particular camera120.

As shown, the processor104applies the event recognition model118to the camera feed122of the video camera120to generate predictions124including an indication of an occurrence of the event in the camera feed122. Based on the prediction124, the processor104performs a response126.

Some embodiments of the disclosed techniques include different architectures than as shown inFIG. 1. As a first such example and without limitation, various embodiments include various types of processors104. In various embodiments, the processor104includes a CPU, a GPU, a TPU, an ASIC, or the like. Some embodiments include two or more processors104of a same or similar type (e.g., two or more CPUs of the same or similar types). Alternatively or additionally, some embodiments include processors104of different types (e.g., two CPUs of different types; one or more CPUs and one or more GPUs or TPUs; or one or more CPUs and one or more FPGAs). In some embodiments, two or more processors104perform a part of the disclosed techniques in tandem (e.g., each CPU training the event recognition model118over a subset of the training data set114) Alternatively or additionally, in some embodiments, two or more processors104respectively perform different parts of the disclosed techniques (e.g.. one CPU executing the machine learning trainer116to train the event recognition model118, and one CPU applying the event recognition model118to the camera feed122of the camera120to make predictions124).

As a second such example and without limitation, various embodiments include various types of memory106. Some embodiments include two or more memories106of a same or similar type (e.g., a Redundant Array of Disks (RAID) array). Alternatively or additionally, some embodiments include two or more memories106of different types (e.g., one or more hard disk drives and one or more solid-state storage devices). In some embodiments, two or more memories106distributively store a component (e.g., storing the training data set114to span two or more memories106). Alternatively or additionally, in some embodiments, a first memory106stores a first component (e.g., the training data set114) and a second memory106stores a second component (e.g., the machine learning trainer116).

As a third such example and without limitation, some disclosed embodiments include different implementations of the machine learning trainer116. In some embodiments, at least part of the machine learning trainer116is embodied as a program in a high-level programming language (e.g., C. Java, or Python), including a compiled product thereof. Alternatively or additionally, in some embodiments, at least part of the machine learning trainer116is embodied in hardware-level instructions (e.g., a firmware that the processor104loads and executes). Alternatively or additionally, in some embodiments, at least part of the machine learning trainer116is a configuration of a hardware circuit (e.g., configurations of the lookup tables within the logic blocks of one or more FPGAs). In some embodiments, the memory106includes additional components (e.g., machine learning libraries used by the machine learning trainer116).

As a fourth such example and without limitation, instead of one server102, some disclosed embodiments include two or more servers102that together apply the disclosed techniques. Some embodiments include two or more servers102that distributively perform one operation (e.g., a first server102and a second server102that respectively train the event recognition model118over different parts of the training data set114). Alternatively or additionally, some embodiments include two or more servers102that execute different parts of one operation (e.g., a first server102that displays the user interface110for a user, and a second server102that executes the machine learning trainer116). Alternatively or additionally, some embodiments include two or more servers102that perform different operations (e.g., a first server102that trains the event recognition model118and a second server102that applies the event recognition model118to the camera feed122to make predictions124). In some embodiments, two or more servers102communicate through a localized connection, such as through a shared bus or a local area network. Alternatively or additionally, in some embodiments, two or more servers102communicate through a remote connection, such as the Internet, a virtual private network (VPN), or a public or private cloud. In some embodiments, the system100and the video camera120are separate, and a communications network provides communication between the camera120and the system100, The communications network can be a local personal area network (PAN), a wider area network (e.g., the internet) in cases of remote control of the camera120, or the like. In various embodiments, training is performed by a device including the camera), at a cloud edge (e.g., on a gateway device or local server connected to the camera120via a local area network), and/or in the cloud (e.g., a separate machine outside of the local network which camera is connected to via a wide-area network such as the internet). Similarly, in various embodiments, prediction is performed by a device including the camera), at a cloud edge (e.g., on a gateway device or local server connected to the camera120via a local area network), and/or in the cloud (e.g., a separate machine outside of the local network which camera is connected to via a wide-area network such as the internet).

FIG. 2is an illustration of a user interface200of the system ofFIG. 1, according to some embodiments. In some embodiments, the user interface200is presented on a display of the system100and receives input via input devices of the system100, such as (without limitation) a keyboard, mouse, and/or touchscreen. In some other embodiments, the user interface200is a web-based user interface that is generated by the system100and sent by a webserver to a client device, which presents the user interface within a web browser.

As shown, the user interface200enables a user to train an event recognition models118for new event types. The user interface200displays one or more images of a camera feed122of a camera120. For example, the images of the camera feed122can be received by receiving an activation of buttons and recording a video clip for the sample from the camera feed122, e.g., for an indicated length of time. The user interface200receives, from the user, a name for a new event type and an indication of three positive samples taken from the camera feed122(e.g., three images or video segments in which an occurrence of the event is visible) and an indication of three negative samples taken from the camera feed122(e.g., three images or video segments in which an occurrence of the event is not visible). The machine learning trainer116can then train an event recognition model118based on the positive samples and negative samples.

In some embodiments, the user interface200receives, from a user, a selection of a portion of the at least one image of at least one training data sample and the indication of the occurrence of the event within the portion. In some embodiments, the user indicates a spatial portion of an image in which the event occurs, such as a rectangular or free-form boundary designating an area of the image in which the occurrence of the event is visible. In some embodiments, the user indicates a spatial portion of a video segment in which the event occurs, such as a rectangular or free-form boundary designating an area of the video segment in which the event occurs, and/or a chronological portion of the video segment, such as a subset of the two or more images during which the event occurs. In at least these embodiments, the training can include training the event recognition model118to generate the indication of the occurrence of the event based on the selected portion of the at least one image.

Event detection is a superset of object detection and has a broader scope. For example (without limitation), some events may not involve a new object appearing or disappearing, but rather a change in the state of an object, such as a transition from a door being open to being closed. For example (without limitation), event detection can involve a configuration or orientation of an object, such as person raising her hand, or an interaction between two objects, such as a dog getting on a couch.

In some embodiments, the user interface200flags prior portions of the camera feed122(e.g., based on motion detection) as candidates for training samples. The user interface200can present the flagged candidates to a user for verification and/or labeling, and can use the flagged portions to train the event recognition model118. These embodiments can make it easier to identify samples for the event of interest in the environment in which the camera120is intended to be used, which can simplify the training process for the user.

In some embodiments, the user interface200allows the user to create new event types. For example (without limitation), the user interface200receives from the user a tagging of one or more positive samples and/or one or more negative samples and an indication of the new event type that is represented by the one or more positive samples and not represented by the one or more negative samples. The user interface200can display, for the user, the video history or past motion alerts that might include positive and negative samples for each event type of interest. After an event type is created, the system100can train or retrain an event recognition model118for the new event type based on the user-provided samples.

In some embodiments, the user interface200allows the user to prioritize performance criteria for the event recognition model118. For example (without limitation), the user interface200can allow the user to prioritize precision (e.g., the accuracy of identified events) for the event recognition model118, where a higher precision reduces a likelihood of false positives. As another example (without limitation), the user interface200can allow the user to prioritize recall (e.g., the sensitivity to detecting events) for the event recognition model118, where a higher recall reduces a likelihood of false negatives. In some cases, training the event recognition model118to produce higher precision vs. higher recall can be a tradeoff, and the user interface200can allow the user to specify such priorities in order to adapt the sensitivity of the event recognition model118to the circumstances of the user.

In some embodiments, the user interface200is part of a mobile app. For consumer surveillance cameras, a mobile app allows users to monitor the real time camera feed122of the camera120as well as short video segments corresponding to recent events (motion alerts, person detection, etc.) In some embodiments, the user interface200includes a page that is designed in the mobile app to allow the user to define a custom event and to tag positive and negative samples (e.g. using either the live stream or using the short video segments recorded in the recent alert section).

The event recognition model118is trained or retrained based on the user selections in the user interface200. In various embodiments, the training or retraining is performed directly on the system100or on a remote server. In some embodiments, the machine learning trainer116uses a few-shot learning framework. The use of few-shot learning allows the event recognition model118to be trained with relatively few training data samples, such as three positive and three negative training data samples, as discussed with respect toFIG. 2. In some embodiments, training can be limited, for example, to a selected number of training data samples of events of interest (e.g., ten samples for each event type).

In some embodiments, the machine learning trainer116can retrain the event recognition model188by applying higher training weights to the samples provided by the user for a specific camera. Applying higher training weights to the samples provided by the user for a specific camera can be advantageous for improving the detection of occurrences of events by the camera according to samples provided by the user. Also, in some embodiments, retraining can be performed based on explicit or implicit feedback from a user regarding alerts generated by a pretrained model.

In some embodiments, the system100has access to a set of pretrained models. The pretrained models can provide basic recognition of predefined event types, such as common types of human or animal movement. In some embodiments, the pretrained models can be hosted by the system100and made available to all users who have access to the system100. In some embodiments, the system100receives a selection by the user of one or more of the predefined event types that are of interest to the user, and then uses one or more pretrained models as the event recognition model118for the camera120of the user. In some embodiments, the pretrained models are used as base models to generate embeddings that are further used to train the specific models on the camera feed122of the camera120. That is, a pretrained model can be generally trained to detect a presence of a person, and an event recognition model118can continue training of a pretrained model using images from the camera feed122of a particular camera120. Using a pretrained model can accelerate the training of the event recognition model118and/or can allow the training of the event recognition model118to be completed with fewer data samples of the events of interest. Also, continuing the training of a pretrained model using the camera feed of a particular camera can adapt the learned criteria of the pretrained model for detecting occurrences of the event to the specific details of the particular camera.

FIG. 3is an illustration of the system100ofFIG. 1detecting events in a monitored scene, according to some embodiments. The system100ofFIG. 3includes a processor104, memory106, and an event recognition model118, and interacts with a camera120.

As shown, the camera120provides a camera feed122of a monitored scene300. The system100applies the event recognition models118and/or pretrained models to the camera feed122of a camera120recognize the custom events of interest to the user. The system100performs a response126based on the indication by the event recognition model118of an occurrence of the event in the camera feed122of the monitored scene300. In some embodiments, the response126includes alerting a user of the occurrence of events or otherwise taking actions in reaction to the events. In some embodiments, the system100takes an appropriate action in response to the recognition of a custom event type by the event recognition model118, such as sending an alert to the user or a first responder, activating an alarm (e.g., playing a sound), controlling a portion of the user's premises (in the case of smart homes, businesses, or factories), or the like. In some embodiments, the system100includes a user interface module that provides a user interface by which the user can interact with the camera120, e.g., to zoom, pan, or tilt the camera120and/or to adjust camera properties such as exposure or resolution.

When a motion is detected within the camera feed122of the camera120, the event recognition model118is applied to determine if any of the events of interest to the user are occurring. In some embodiments, the system100continuously and/or in real time applies an event recognition model118to a camera feed122from the camera120. In some other embodiments, the system100applies the event recognition model118to past camera feeds, such as time-delayed analysis or historic analysis. In some embodiments, the system100applies the event recognition model118to the camera feed122only after motion is detected (e.g., in order to minimize computation costs). For example (without limitation), the system100can detect motion via a passive infrared (PIR) sensor, and can apply the event recognition model118only when or after the PIR sensor detects motion. As another example, the system100can compare two or more images (e.g., consecutive frames) in a camera feed122on a pixel-by-pixel and/or area-by-area basis. A change in the pixels that is greater than a threshold can be considered to indicate motion, resulting in applying the event recognition model118to the camera feed122.

When the event recognition model118detects an occurrence of its associated event, the system100performs a response126. In some embodiments, the response126includes an action, such as (without limitation) alerting the user by beeping/playing a sound/sending a message. In some embodiments, the response126includes a remedial action, such as sending an emergency call to a first responder such as police, firefighters, healthcare providers, or the like. In some embodiments, the response126includes controlling a location of the monitored scene300, such as activating an alarm, locking doors of a smart home, and/or shutting off power to certain parts of the home.

In some embodiments, an event recognition model118can be retrained. In some embodiments, the system100retrains the event recognition model118based on an identification of instances where the event recognition model118in question was incorrect. In such cases, retraining can include receiving (e.g., from a user) an updated indication of an occurrence of an event within a first training data sample of the training data set114, and re-training the event recognition model118to generate predictions124(e.g., an updated event label112) of the updated indication of the occurrence of the event for the first training data sample. That is, the user interface200can receive from the user an indication of a false positive (e.g., the event recognition model118incorrectly predicts an event is occurring while it is not), such as one or more tags indicating false detection. The machine learning trainer116can use the tags and the tagged one or more images as negative samples to retrain the event recognition model118to refrain from detecting non-occurrences of the event. As another example, the updated indication can include an indication of an occurrence of the event in the first training data sample for which the event recognition model failed to generate an indication of the occurrence of the event (e.g., a false negative). That is, the user interface200can receive from the user an indication of a false negative (e.g., a failure to detect an occurrence of event), such as one or more tags indicating failed detection, These identified instances can serve as a negative training set that can be used to re-train the event recognition model118for greater accuracy. The machine learning trainer116can use the tags and the tagged one or more images as positive samples to retrain the event recognition model118to detect occurrences of the event. As yet another example, the updated indication can include an indication of a non-occurrence of the event in the first training data sample for which the event recognition model incorrectly generated an indication of an occurrence of the event (e.g., a false positive). As yet another example, the updated indication can include an identification of a first event type for the occurrence of the event for which the event recognition model118determined a second event type (e.g., a correction of the event type determined by the event recognition model118for a selected training data sample). For example (without limitation), the updated indication can include a new event type for the occurrence of the event (e.g., a new event type as selected by a user). In these and other cases, re- training can involve continued training of a current event recognition model118and/or training a new event recognition model118to replace a current event recognition model118. In some embodiments, the user can delete a subset of (positive and negative) training data samples of the training data set114(e.g., samples that are incorrect and/or ambiguous). In such cases, a new event recognition model118can be trained using the updated training data set114.

FIG. 4is a flow diagram of method steps for detecting events by a video camera, according to one or more embodiments. Although the method steps are described with reference toFIGS. 1-3, persons skilled in the art will understand that any system may be configured to implement the method steps, in any order, in other embodiments.

As shown, at step402, a training data set of training data samples is accessed, wherein each training data sample includes at least one image obtained from a video camera and an indication of an occurrence of an event within the at least one image. For example (without limitation), the video camera can provide one or more individual images and/or one or more video segments including a sequence of two or more images. In some embodiments, the training data set is generated by presenting the training data samples through a user interface and receiving, from a user, a selection of an event label as an indication of an event occurring in each training data sample.

As shown, at step404, an event recognition model is trained to generate the indication of the occurrence of the event within each training data sample of the training data set. In some embodiments, the event recognition model includes one or more binary classifiers, where each binary classifier outputs a probability of the training data sample to include an occurrence of an event of a particular event type. In some other embodiments, the event recognition model includes a multi-class classifier that outputs, for a plurality of event types, a probability of the training data sample to include an occurrence of events of each of several event types. In some embodiments, training is performed using a few-shot learning framework, which enables the event recognition model to be trained using a small number of samples per event type.

As shown, at step406, the event recognition model is applied to a camera feed of the video camera to generate an indication of an occurrence of the event in the camera feed. In various embodiments, the camera feed is an individual image and/or a sequence of two or more images. In some embodiments, the camera feed is a live camera feed. In some embodiments, the indication of the occurrence of the event is generated based on a probability of the occurrence of the event in the camera feed, and, further, based on the probability exceeding a probability threshold.

As shown, at step408, a response is performed based on the indication of the occurrence of the event in the camera feed. In various embodiments, the response includes one or more of sending an alert to the user or a first responder, activating an alarm (e.g., playing a sound), controlling a portion of the user's premises (in the case of smart homes, businesses, or factories), or the like.

As shown, at step410, the event recognition model is retrained based on an updated indication of an occurrence of the event within at least one image of the camera feed. For example (without limitation), the updated indication can be an indication of an occurrence of the event within an individual image or video segment for which the event recognition model did not determine the occurrence of the event. As another example (without limitation), the updated indication can be an indication of a non-occurrence of the event within an individual image or video segment for which the event recognition model incorrectly determined an occurrence of the event. As yet another example (without limitation), the updated indication can be an indication of a new event type for which an occurrence is visible within an individual image or video segment, or a different event type than was detected by the event recognition model. In such cases, the retraining can involve continuing the previous training of the event recognition model using the updated indication, or training a substitute event recognition model to be used in place of the current event recognition model.

In sum, an event recognition model is trained to recognize occurrences of events in images from a camera based on user-selected event labels for events as indicated by user. The trained event recognition model is applied to a camera feed of the camera to generate indications of occurrences of the events of interest to the user. A response is performed based on determined occurrences of the events.

At least one technical advantage of the disclosed techniques is that the event recognition model is trained to detect specific types of events that indicated by the user. Training the event recognition model based on events that are of interest to the user can expand and focus the range of detected event types to those that are of particular interest to the user, and to exclude events that are not of interest to the user. As another advantage, training the event recognition model on the camera feed of a camera can enable the camera to detect events within the particular context of the camera and camera feed, such as a particular room of a house or area of a factory. As yet another advantage, performing responses based on the trained event recognition model can enable a surveillance system to take event-type-specific responses based on the events of interest to the user. These technical advantages provide one or more technological improvements over existing techniques for event detection for surveillance cameras.

[Claim combinations to be inserted by Artegis prior to filing]

One possible embodiment has been described herein. Those of skill in the art will appreciate that other embodiments may likewise be practiced. First, the particular naming of the components and variables, capitalization of terms, the attributes, data structures, or any other programming or structural aspect is not mandatory or significant, and the mechanisms described may have different names, formats, or protocols. Also, the particular division of functionality between the various system components described herein is merely for purposes of example, and is not mandatory; functions performed by a single system component may instead be performed by multiple components, and functions performed by multiple components may instead be performed by a single component.

Certain aspects described herein include process steps and instructions in the form of an algorithm. It should be noted that the process steps and instructions could be embodied in software, firmware or hardware, and when embodied in software, could be downloaded to reside on and be operated from different platforms used by real time network operating systems.

The algorithms and operations presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may also be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will be apparent to those of skill in the art, along with equivalent variations. In addition, the concepts described herein are not described with reference to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings as described herein, and any references to specific languages are provided for purposes of enablement and best mode.