Patent Description:
Many industries employ artificial intelligence technologies to analyze video information for a wide array of applications, e.g., object detection, object classification, etc. For example, in the retail sector, machine learning models may be employed to monitor entry and exit at a retail location, support traffic flow applications that monitor customer journeys within a retail location, and/or enable surveillance systems that detect unauthorized activity by retail customers with respect to retail articles offered for sale. Typically, video capture systems employ pre-built ML models that are not tailored for a particular video capture device capturing video information and/or particular camera scenes or environments represented in the video information. As a result, system accuracy may be significantly reduced due to false positives and/or false negatives.

<CIT> describes technology for building an edge convolutional neural network (CNN) system for IoT includes training on-site processors to analyze image data and identify motorized vehicles, bicycles and people in near-real-time, using a big cloud CNN, a small cloud CNN and an on-site CNN. At least five hundred site-specific images from cameras are analyzed using the big cloud CNN to produce a machine-generated training set that includes an image that has regions, and for each region, coordinates of bounding boxes for objects in the region, and classification of contents of the bounding boxes as a motorized vehicle, bicycle or person; and the training set. The machine-generated training set gets used to train the small cloud CNN; and coefficients from the trained small cloud CNN get transferred to the on-site CNN, thereby configuring the on-site CNN to recognize motorized vehicles, bicycles and people in images from the cameras in near-real-time.

The present disclosure provides systems, apparatuses, methods, and computer-readable media for dynamic refinement of artificial intelligence models. These systems, methods, and apparatuses will be described in the following detailed description and illustrated in the accompanying drawings by various modules, blocks, components, circuits, processes, algorithms, among other examples (collectively referred to as "elements"). Examples of processors include microprocessors, microcontrollers, graphics processing units (GPUs), central processing units (CPUs), and other suitable hardware configured to perform the various functionality described throughout this disclosure. Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software components, applications, software applications, software packages, routines, subroutines, objects, executables, threads of execution, procedures, functions, among other examples, whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise. Computer-readable media includes computer storage media, which may be referred to as non-transitory computer-readable media. Non-transitory computer-readable media may exclude transitory signals. By way of example, and not limitation, such computer-readable media can include a random-access memory (RAM), a read-only memory (ROM), an electrically erasable programmable ROM (EEPROM), optical disk storage, magnetic disk storage, other magnetic storage devices, combinations of the aforementioned types of computer-readable media, or any other medium that can be used to store computer executable code in the form of instructions or data structures that can be accessed by a computer.

In an aspect, a method is provided as set out in claim <NUM>.

In another aspect, a system is provided as set out in claim <NUM>.

In another aspect, a non-transitory computer-readable medium is provided as set out in claim <NUM>.

These features are indicative, however, of but a few of the various ways in which the principles of various aspects may be employed, and this description is intended to include all such aspects provided that they fall within the scope of the claims.

In some instances, well known components may be shown in block diagram form in order to avoid obscuring such concepts.

Implementations of the present disclosure provide systems, methods, and apparatuses that provide cloud-based dynamic refinement of ML models employed for object detection. These systems, methods, and apparatuses will be described in the following detailed description and illustrated in the accompanying drawings by various modules, blocks, components, circuits, processes, algorithms, among other examples (collectively referred to as "elements"). Examples of processors include microprocessors, microcontrollers, graphics processing units (GPUs), central processing units (CPUs), and other suitable hardware configured to perform the various functionality described throughout this disclosure. Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software components, applications, software applications, software packages, routines, subroutines, objects, executables, threads of execution, procedures, functions, among other examples, whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

In some implementations, one problem solved by the present solution is continued use of inaccurate ML models employed in edge devices ill-suited for ML model training. For example, the present disclosure describes systems and methods for dynamically updating ML models to reduce false negatives and false positives during an object detection workflow. Typically, video monitoring systems employ pre-built ML models that are not tailored for a particular video capture device and/or particular video capture scenes or environments. Further, the edge devices employed to support the video capture devices have limited means for updating the pre-built ML models in view of the particular attributes of a video capture device, particular video capture scenes or environments, and/or crowd sourced information related to a plurality of video capture devices. The present disclosure provides systems and methods for dynamically updating ML models employed in edge devices to improve ML model accuracy (e.g., reduce false negatives and false positives).

Referring to <FIG>, in one non-limiting aspect, a system <NUM> may be configured to provide dynamic refinement of ML models. As illustrated in <FIG>, the system <NUM> may include a management service <NUM>, one or more video capture devices <NUM>(<NUM>)-(n) configured to capture video data <NUM> in one or more physical environments <NUM>(<NUM>)-(n), and one or more edge devices <NUM>(<NUM>)-(N) configured to manage and analyze the video data <NUM>. For example, the first video capture device <NUM>(<NUM>) may capture video data <NUM> in a first physical environment <NUM>(<NUM>), the nth video capture device <NUM>(n) may capture video data <NUM> in an nth physical environment <NUM>(n), and so forth. In addition, in some aspects, an edge device <NUM> may be a component of a video capture device <NUM> or located within a shared physical environment <NUM> as the video capture device <NUM>. Further, the management service <NUM>, the plurality of video capture devices <NUM>(<NUM>)-(n), and/or the plurality of edge devices <NUM>(<NUM>)-(n) may communicate via a communication network <NUM>. In some implementations, the communication network <NUM> may include one or more of a wired and/or wireless private network, personal area network, local area network, wide area network, or the Internet.

As illustrated in <FIG>, an edge device <NUM> may receive video data <NUM> from a corresponding video capture device <NUM>. Further, the edge device <NUM> may employ an edge model <NUM> to determine object information <NUM> within the video data <NUM>. In some aspects, the edge model <NUM> may be configured to detect objects within the video data <NUM>, track motion of the detected objects within the video data <NUM>, and store data corresponding to the detected objects and tracked motion of the detected objects as the object information <NUM>. In some aspects, as described in detail herein, the edge model <NUM> is trained specifically to detect and/or track objects within video data <NUM> captured by a particular video capture device <NUM>. For example, the edge model <NUM>(<NUM>) may be trained to detect and track objects based at least in part on one or more attributes of the physical environment <NUM> where the video capture device <NUM>(<NUM>) is located, and/or one or more attributes of the video capture device <NUM>(<NUM>). Additionally, an edge device <NUM> may include and/or be coupled with a graphical user interface (GUI) <NUM> for presenting the video data <NUM> and/or the object information <NUM>.

Further, as illustrated in <FIG>, the video capture devices <NUM>(<NUM>)-(n) and/or edge devices <NUM>(<NUM>)-(n) may periodically sample the video data <NUM> and transmit the sampled video information <NUM>(<NUM>)-(n) to the management service <NUM>. As described in detail herein, the management service <NUM> may dynamically refine the edge models <NUM>(<NUM>)-(n) based on the sampled video information <NUM>(<NUM>)-(n). In some embodiments, the video data <NUM> may be sampled according to one or more criteria and/or requirements. For example, the video data <NUM> may be sampled based on video capture device placement (e.g., overhead placement of a video capture device), placement attributes of a video capture device (e.g., placement height), field of view attributes, diversity of objects within the video data <NUM>, object distance within the video data <NUM>, illumination levels, date and/or time, diversity of background, diversity of object occlusion, indoor and outdoor scenes, diversity of persons (e.g., height, clothing, ethnicity, sex), variety of posture, etc..

The management service <NUM> may include a cloud model <NUM>, the edge models <NUM>(<NUM>)-(n), a model management component <NUM>, an image selection component <NUM>, a motion detection component <NUM>, an image annotation component <NUM>, and a model training component <NUM>. The cloud model <NUM> may be configured to detect objects within the video data <NUM> and store data corresponding to the detected objects as the object information <NUM>. Further, the cloud model <NUM> may have higher object detection accuracy capabilities than the edge models <NUM>(<NUM>)-(n), while also being more resource intensive than the edge models <NUM>(<NUM>)-(n). As such, in some aspects, the cloud model <NUM> may not be employed by the edge device <NUM>, which may include less resources than the management service <NUM>. For instance, in some aspects, the management service <NUM> may be a cloud computing environment, and the edge devices <NUM>(<NUM>)-(n) may be local server devices. In addition, while the present disclosure describes edge models <NUM> and a cloud model <NUM> having object detection capabilities, the present disclosure may be applied to ML models having other uses.

The model management component <NUM> may be configured to manage the process of generating the edge models <NUM>(<NUM>)-(n) and/or updated edge models <NUM>(<NUM>)-(n). For instance, the model management component <NUM> may generate and deploy an edge model <NUM> in response to installation of a video capture device <NUM>. In some other instances, the model management component <NUM> may periodically generate an updated edge model <NUM> in response to a passage of time, receipt of a predefined amount of sampled video information from a video capture device <NUM> associated with an edge model <NUM>, and/or user feedback. Further, the model management component <NUM> may deploy an updated edge model <NUM> to a video capture device <NUM> based on the updated edge model <NUM> having an accuracy that exceeds the edge model <NUM> current employed in the video capture device <NUM> by a predefined threshold.

The image selection component <NUM> is configured to determine a plurality of training images <NUM>(<NUM>)-(n) from the sampled video information <NUM>. In particular, the image selection component <NUM> may select the plurality of training images <NUM>(<NUM>)-(n) based upon comparing the object information <NUM> generated by a local copy of the edge model <NUM> to the object information <NUM> generated by the cloud model <NUM>.

The object information <NUM> includes one or more bounding representations (e.g., bounding boxes, bounding segmentation, etc.) detected by the edge model <NUM> within an image frame of the sampled video information <NUM>, and the object information <NUM> includes one or more bounding representations detected by the cloud model <NUM> within the image frame. The image selection component <NUM> selects the image frame as one of the plurality of training images <NUM> based on a count of the one or more bounding representations detected by the edge model <NUM> not equaling a count of the one or more bounding representations detected by the cloud model <NUM>.

Further, the image selection component <NUM> may not select the image frame as one of the plurality of training images <NUM> based on a count of the one or more bounding representations detected by the edge model <NUM> equaling a count of the one or more bounding representations detected by the cloud model <NUM>.

The motion detection component <NUM> is configured to detect and track objects, as the motion detection information <NUM>, within the plurality of training images <NUM>(<NUM>)-(n). For example, the motion detection component <NUM> is configured to detect bounding representations within the plurality of training images <NUM>(<NUM>)-(n). In some aspects, the motion detection component <NUM> may employ an optical flow technique or frame segmentation approach for object detection. Further, the image annotation component <NUM> may be configured to annotate the plurality of training images <NUM>(<NUM>)-(n) to generate the plurality of annotated images <NUM>(<NUM>)-(n). In particular, the image annotation component <NUM> may annotate the plurality of training images <NUM>(<NUM>)-(n) based upon comparing the object information <NUM> generated by the cloud model <NUM>, the motion detection information <NUM> generated by the motion detection component <NUM>, and/or the object information <NUM> generated by the edge model <NUM>. As used herein, in some aspects, "annotating" may refer to applying bounding representations to the objects detected within an image frame and/or other forms of labeling of training data.

For example, the object information <NUM> may include one or more bounding representations detected by the cloud model <NUM> within an image frame of the plurality of training images <NUM>(<NUM>)-(n), and the motion detection information <NUM> may include one or more bounding representations detected by the motion detection component <NUM> within the same image frame. Further, the image annotation component <NUM> may generate an annotated image of the plurality of annotated images <NUM>(<NUM>)-(n) including the one or more bounding representations detected by the cloud model <NUM> based on the one or more bounding representations of the object information <NUM> matching the one or more of bounding representations of the motion detection information <NUM>. As used herein, in some aspects, "matching" may refer to correspondence between two bounding representations, e.g., as illustrated by the bounding representations of detection results <NUM> (e.g., each bounding representation generated by the cloud model <NUM> has a corresponding bounding representation generated by the edge model <NUM> in a similar location). Further, in some aspects, matching may be determined by comparing a count of bounding representations generated by a first model to a count of bounding representations generated by a second model. Additionally, or alternatively, matching may be determined based on the difference between a location of a first bounding representation and a second bounding representation being less than a predefined threshold.

As another example, the object information <NUM> may include one or more bounding representations detected by the edge model <NUM> within an image frame of the sampled video information <NUM>, the object information <NUM> may include one or more bounding representations detected by the cloud model <NUM> within the image frame, and the motion detection information <NUM> may include one or more bounding representations detected by the motion detection component <NUM> within the second image frame. Further, the image annotation component <NUM> may determine that the one or more bounding representations detected by the cloud model <NUM> do not match the one or more bounding representations detected by the motion detection component <NUM>, and determine that each of the one or more bounding representations detected by the cloud model <NUM> but not the motion detection component <NUM> matches a bounding representation detected by the edge model <NUM>. In response, the image annotation component <NUM> may generate an annotated image of the plurality of annotated images including the one or more bounding representations detected by the cloud model <NUM>.

As another example, the object information <NUM> may include one or more bounding representations detected by the edge model <NUM> within an image frame of the sampled video information <NUM>, the object information <NUM> may include one or more bounding representations detected by the cloud model <NUM> within the image frame, and the motion detection information <NUM> may include one or more bounding representations detected by the motion detection component <NUM> within the image frame. Further, the image annotation component <NUM> may determine that one or more bounding representations of the image frame detected by the edge model <NUM> are not detected by the cloud model <NUM> within the image frame, and determine that the one or more bounding representations detected by the edge model <NUM> and not the cloud model <NUM> do not match the one or more bounding representations detected by the edge model <NUM>. In response, the image annotation component <NUM> may generate an annotated image of the plurality of annotated images including the one or more bounding representations detected by the cloud model <NUM>.

As another example, the object information <NUM> may include one or more bounding representations detected by the edge model <NUM> within an image frame of the sampled video information <NUM>, the object information <NUM> may include one or more bounding representations detected by the cloud model <NUM> within the image frame, and the motion detection information <NUM> may include one or more bounding representations detected by the motion detection component <NUM> within the image frame. Further, in some aspects, the image annotation component <NUM> may identify a review context based at least in part on two of the one or more bounding representations detected by the cloud model <NUM>, the one or more bounding representations detected by the edge model <NUM>, or the one or more bounding representations detected by the motion detection component <NUM>. In some examples, a review context may correspond to an instance in which the more accurate cloud model <NUM> fails to detect an object that is detected by the less accurate edge model <NUM>. For instance, the image annotation component <NUM> may detect a review context based on the count of the one or more bounding representations detected by the motion detection component <NUM> being greater than the count of the one or more bounding representations detected by the cloud model <NUM> (i.e., potential false negative by the cloud model <NUM>). In some other instances, the image annotation component <NUM> may detect a review context based on one or more particular bounding representations of an image frame detected by the cloud model <NUM> and not being detected by the motion detection component <NUM>, and the one or more particular bounding representations detected by the cloud model <NUM> not matching the one or more bounding representations detected by the edge model <NUM> (i.e., potential false positive by the cloud model <NUM>). In yet still another instance, the image annotation component <NUM> may detect a review context based on a count of the one or more bounding representations detected by the edge model <NUM> being greater than a count of the one or more bounding representations detected by the cloud model <NUM>, and the one or more bounding representations of the image frame detected by the motion detection component <NUM> not matching the one or more bounding representations of the image frame detected by the edge model <NUM> (i.e., potential moving false positive or accurate detection by edge model <NUM>).

Once the image annotation component <NUM> determines the existence of a review context with respect to an image frame of the plurality of training images <NUM>(<NUM>)-(n), a review GUI component <NUM> may prompt a user for annotation information identifying and/or confirming the correct bounding representations for the image frame, and receive the annotation information provided by user via the review GUI component <NUM>. Further, the image annotation component <NUM> may generate an annotated image of the plurality of annotated images <NUM> based at least in part on the annotation information.

Further, the model management component <NUM> will split the plurality of annotated images <NUM> into a training set, a validation set, and a test set. Further, the model training component <NUM> may use the plurality of annotated images <NUM> to generate (e.g., train, validate, and test) the edge models <NUM> and the updated edge models <NUM>. If the model training component <NUM> has not previously generated an edge model <NUM> for an edge device <NUM>, the model training component <NUM> may perform a global update to a standard edge model <NUM> using the plurality of annotated images <NUM> associated with the plurality of edge devices <NUM>(<NUM>)-(n). Alternatively, if the model training component <NUM> has previously generated an edge model <NUM> for a particular edge device <NUM>, the model training component <NUM> may perform a local update to the edge model <NUM> previously deployed at the particular edge device using the plurality of annotated images <NUM> derived from sample video information <NUM> received from the particular edge device <NUM>. As such, the model training component <NUM> may perform an iterative process to improve the accuracy of the edge model <NUM> deployed to a particular edge device <NUM> over time. For example, the model training component <NUM> may use the training set of the plurality of annotated images <NUM> derived from sample video information <NUM> received from the particular edge device <NUM> to re-train an edge model <NUM> to generate an updated edge model <NUM> to be deployed at the particular edge device <NUM>.

In some aspects, the edge models <NUM> may be deep learning ML models, and the model training component <NUM> may employ transfer learning to train the ML models. As used herein, in some aspects, "transfer learning" may refer to using a result obtained by source items data items in feature extraction of target data items. In some aspects, a deep learning architecture may be a layered neural network in which the output of a first layer of neurons becomes an input to a second later of neurons, the output of the second layer of neurons becomes input to a third layer of neurons, and so forth. Further, the layered neural network may be trained to recognize a hierarchy features within an object recognition/detection application. For example, the first layer may learn to recognize simple features (e.g., edges), and the second layer, taking the output of the first layer as input, may learn to recognize combinations of features (e.g., simple shapes). Further, in some examples, higher layers may learn to represent complex shapes and/or common visual objects. In addition, in a transfer learning application, the model training component <NUM> may generate an updated edge model <NUM> based on the lower layers of a pre-existing edge model <NUM> and newer upper layers learned from the training set of the plurality of annotated images <NUM> derived from the most recent sampled video information <NUM> received from the corresponding edge device <NUM>. Further, the model training component <NUM> may validate and test the updated edge model <NUM> using the validation and testing set of the plurality of annotated images <NUM> derived from the most recent sampled video information <NUM> received from the corresponding edge device <NUM>. If the testing results indicate that the accuracy of the updated edge model <NUM> exceed a predefined value, the model management component <NUM> may send the updated edge model <NUM> to the edge device <NUM> for deployment.

<FIG> is a flow diagram <NUM> of an example of image selection, according to some implementations. As illustrated in <FIG>, at step <NUM>, the model management component <NUM> may select an image frame from within the sample video information <NUM>. At step <NUM>, the image frame is processed by the cloud model <NUM> to determine the object information <NUM> (e.g., one or more bounding representations) and the edge model <NUM> to determine the object information <NUM> (e.g., one or more bounding representations). At step <NUM>, the image selection component <NUM> may determine a first count of the one or more bounding representations detected within the image frame by the cloud model <NUM> and determine a second count of the one or more bounding representations detected within the image frame by the edge model <NUM>. At steps <NUM>-<NUM>, the image selection component <NUM> may compare the first count to the second count. If the first count is equal to the second count, as illustrated by the detection results <NUM>, the image selection component <NUM> may discard the image frame and will not select the image frame for the plurality of training images <NUM>, at step <NUM>. In addition, if the first count is greater than the second count, as illustrated in detection results <NUM>, the image selection component <NUM> may select the image frame for the plurality of training images <NUM> and label the image frame as potentially including a false negative (i.e., failure of the edge model <NUM> to detect an object), at step <NUM>. Further, if the first count is less than the second count, as illustrated in detection results <NUM>, the image selection component <NUM> may select the image frame for the plurality of training images <NUM> and label the image frame as potentially including false positive (i.e., the edge model <NUM> inaccurately identified a region of the image frame as corresponding to an object), at step <NUM>.

<FIG> is a flow diagram <NUM> of a first example of image annotation, according to some implementations. As illustrated in <FIG>, at step <NUM>, the model management component <NUM> may select an image frame labeled as potentially including a false negative. At step <NUM>, the image frame is processed by the cloud model <NUM> to determine the object information <NUM> (e.g., one or more bounding representations), the motion detection component <NUM> to determine the motion detection information (e.g., one or more bounding representations), and the edge model <NUM> to determine the object information <NUM> (e.g., one or more bounding representations). In some aspects, the image annotation component <NUM> may re-use the object information <NUM> and the object information <NUM> determined during image selection, as described with respect <FIG>. At step <NUM>, the image annotation component <NUM> may determine the one or more bounding representations detected within the image frame by the cloud model <NUM> and determine the one or more bounding representations detected within the image frame by the motion detection component <NUM>. At steps <NUM>-<NUM>, the image annotation component <NUM> may compare the one or more bounding representations detected within the image frame by the cloud model <NUM> to the one or more bounding representations detected within the image frame by the motion detection component <NUM>. If the bounding representations match, the image annotation component <NUM> may annotate the image frame as one of the plurality of annotated images <NUM> using the one or more bounding representations detected by the cloud model <NUM>, at step <NUM>.

In addition, if the bounding representations do not match and a first count of bounding representations detected by the cloud model <NUM> is greater than a second count of bounding representations detected by the motion detection component <NUM>, the image annotation component <NUM> may identify the one or more bounding representations detected by the cloud model <NUM> and not detected by the motion detection component, at step <NUM>. At step <NUM>, the image annotation component <NUM> may determine if the one or more identified bounding representations have overlapping bounding representations detected by the edge model <NUM>. If overlap is detected by the image annotation component, the image annotation component <NUM> may annotate the image frame as one of the plurality of annotated images <NUM> using the one or more bounding representations detected by the cloud model <NUM>, at step <NUM>. Otherwise, the image annotation component <NUM> may prompt a user for annotation information via the GUI component <NUM>, at step <NUM>. Further, if the first count of bounding representations detected by the cloud model <NUM> is less than the second count of bounding representations detected by the motion detection component <NUM>, the image annotation component <NUM> may prompt a user for annotation information via the GUI component <NUM>, at step <NUM>.

<FIG> is a flow diagram <NUM> of a first example of image annotation, according to some implementations. As illustrated in <FIG>, at step <NUM>, the model management component <NUM> may select an image frame labeled as potentially including a false positive by the image selection component <NUM>. At step <NUM>, the image frame is processed by the cloud model <NUM> to determine the object information <NUM> (e.g., bounding representations), the motion detection component <NUM> to determine the motion detection information (e.g., bounding representations), and the edge model <NUM> to determine the object information <NUM> (e.g., bounding representations). In some aspects, the image annotation component may re-use the object information <NUM>, motion detection information <NUM>, and the object information <NUM> with respect to <FIG>. At step <NUM>, the image annotation component <NUM> may identify the one or more bounding representations detected by the cloud model <NUM> and not detected by the edge model <NUM>, at step <NUM>. At step <NUM>, the image annotation component <NUM> may determine if the one or more identified bounding representations have an overlapping bounding representations detected by the motion detection component <NUM>. If overlap is not detected by the image annotation component <NUM>, the image annotation component <NUM> may annotate the image frame as one of the plurality of annotated images <NUM> using the one or more bounding representations detected by the cloud model <NUM>, at step <NUM>. Otherwise, the image annotation component <NUM> may prompt a user for annotation information via the GUI component <NUM>, at step <NUM>. In some aspects, the annotation information may correct a potential false positive caused by object motion or confirm that the edge model <NUM> correctly detected one or more objects that were not detected by the cloud model <NUM>.

Referring to <FIG>, a computing device <NUM> may implement all or a portion of the functionality described herein. The computing device <NUM> may be or may include or may be configured to implement the functionality of at least a portion of the system <NUM>, or any component therein. For example, the computing device <NUM> may be or may include or may be configured to implement the functionality of the management service <NUM>. The computing device <NUM> includes a processor <NUM> which may be configured to execute or implement software, hardware, and/or firmware modules that perform any functionality described herein. For example, the processor <NUM> may be configured to execute or implement software, hardware, and/or firmware modules that perform any functionality described herein with reference to the management service <NUM>, or any other component/system/device described herein, e.g., the edge model <NUM>, the cloud model <NUM>, the model management component <NUM>, the image selection component <NUM>, the motion detection component <NUM>, the image annotation component <NUM>, the model training component <NUM>, the object information <NUM>, and the GUI component <NUM>.

The processor <NUM> may be a micro-controller, an application-specific integrated circuit (ASIC), a digital signal processor (DSP), or a field-programmable gate array (FPGA), and/or may include a single or multiple set of processors or multi-core processors. Moreover, the processor <NUM> may be implemented as an integrated processing system and/or a distributed processing system. The computing device <NUM> may further include a memory <NUM>, such as for storing local versions of applications being executed by the processor <NUM>, related instructions, parameters, etc. The memory <NUM> may include a type of memory usable by a computer, such as random access memory (RAM), read only memory (ROM), tapes, magnetic discs, optical discs, volatile memory, nonvolatile memory, and any combination thereof. Additionally, the processor <NUM> and the memory <NUM> may include and execute an operating system executing on the processor <NUM>, one or more applications, display drivers, and/or other components of the computing device <NUM>, e.g., the edge model <NUM>, the cloud model <NUM>, the model management component <NUM>, the image selection component <NUM>, the motion detection component <NUM>, the image annotation component <NUM>, the model training component <NUM>, the object information <NUM>, and the GUI component <NUM>.

Further, the computing device <NUM> may include a communications component <NUM> configured to establish and maintain communications with one or more other devices, parties, entities, etc. utilizing hardware, software, and services. The communications component <NUM> may carry communications between components on the computing device <NUM>, as well as between the computing device <NUM> and external devices, such as devices located across a communications network and/or devices serially or locally connected to the computing device <NUM>. In an aspect, for example, the communications component <NUM> may include one or more buses, and may further include transmit chain components and receive chain components associated with a wireless or wired transmitter and receiver, respectively, operable for interfacing with external devices.

Additionally, the computing device <NUM> may include a data store <NUM>, which can be any suitable combination of hardware and/or software, that provides for mass storage of information, databases, and programs. For example, the data store <NUM> may be or may include a data repository for applications and/or related parameters not currently being executed by processor <NUM>. In addition, the data store <NUM> may be a data repository for an operating system, application, display driver, etc., executing on the processor <NUM>, and/or one or more other components of the computing device <NUM>, e.g., the edge model <NUM>, the cloud model <NUM>, the model management component <NUM>, the image selection component <NUM>, the motion detection component <NUM>, the image annotation component <NUM>, the model training component <NUM>, the object information <NUM>, and the GUI component <NUM>.

The computing device <NUM> may also include a user interface component <NUM> operable to receive inputs from a user of the computing device <NUM> and further operable to generate outputs for presentation to the user (e.g., via a display interface to a display device). The user interface component <NUM> may include one or more input devices, including but not limited to a keyboard, a number pad, a mouse, a touch-sensitive display, a navigation key, a function key, a microphone, a voice recognition component, or any other mechanism capable of receiving an input from a user, or any combination thereof. Further, the user interface component <NUM> may include one or more output devices, including but not limited to a display interface, a speaker, a haptic feedback mechanism, a printer, any other mechanism capable of presenting an output to a user, or any combination thereof.

Referring to <FIG>, in operation, the management service <NUM> or computing device <NUM> may perform an example method <NUM> for dynamically updating deployed ML models. The method <NUM> may be performed by one or more components of the management service <NUM>, the computing device <NUM>, or any device/component described herein according to the techniques described with reference to the previous figures.

At block <NUM>, the method <NUM> includes receiving sampled video information captured by a video capture device. For example, the model management component <NUM> may receive the sampled video information <NUM> from the video capture device <NUM> and/or the edge device <NUM>. Accordingly, the management service <NUM> or the processor <NUM> executing the model management component <NUM> may provide means for receiving sampled video information captured by a video capture device.

At block <NUM>, the method <NUM> includes generating first object detection information based on a cloud model and the sampled video information, the cloud model configured to detect objects within the sampled video information. For example, the cloud model <NUM> may determine the object information <NUM> based on the sampled video information <NUM>. In some aspects, the object information <NUM> may include one or more bounding representations detected within the image frames of the sampled video information <NUM>. Accordingly, the management service <NUM> or the processor <NUM> executing the cloud model <NUM> may provide means for generating first object detection information based on a cloud model and the sampled video information, the cloud model configured to detect objects within the sampled video information.

At block <NUM>, the method <NUM> includes generating second object detection information based on a first edge model and the sampled video information, the first edge model configured to detect objects within the sampled video information and employed at an edge device coupled with the video capture device. For example, the edge model <NUM> may determine the object information <NUM> based on the sampled video information <NUM>. In some aspects, the object information <NUM> may include one or more bounding representations detected within the image frames of the sampled video information <NUM>. Accordingly, the management service <NUM> or the processor <NUM> executing the edge model <NUM> may provide means for generating second object detection information based on a first edge model and the sampled video information, the first edge model configured to detect objects within the sampled video information and employed at an edge device coupled with the video capture device.

At block <NUM>, the method <NUM> includes selecting, based on comparing the first object detection information to the second object detection information, a plurality of training images from the sampled video information. For example, the image selection component <NUM> may compare the object information <NUM> and the object information <NUM> to determine the plurality of training images <NUM> from the sampled video information <NUM>. Accordingly, the management service <NUM> or the processor <NUM> executing the image selection component <NUM> may provide means for selecting, based on comparing the first object detection information to the second object detection information, a plurality of training images from the sampled video information.

At block <NUM>, the method <NUM> includes detecting motion information corresponding to motion of one or more detected objects within the plurality of training images. For example, the motion detection component <NUM> may determine the motion detection information <NUM> based on the plurality of training images <NUM>. In some aspects, the motion detection information <NUM> may include one or more bounding representations detected within the image frames of the plurality of training images <NUM>. Accordingly, the management service <NUM> or the processor <NUM> executing the motion detection component <NUM> may provide means for detecting motion information corresponding to motion of one or more detected objects within the plurality of training images.

At block <NUM>, the method <NUM> includes generating a plurality of annotated images based at least in part on comparing the first object detection information to the motion information. For example, the image annotation component <NUM> may determine the plurality of annotated images <NUM> based on the object information <NUM> and the motion detection information <NUM>. Accordingly, the management service <NUM> or the processor <NUM> executing the image annotation component <NUM> may provide means for generating a plurality of annotated images based at least in part on comparing the first object detection information to the motion information.

At block <NUM>, the method <NUM> includes generating a second edge model based upon training the first edge model using the plurality of annotated images, the second edge model to be employed at the video capture device or another video capture device. For example, the model training component <NUM> may generate the updated edge model <NUM> based on the plurality of annotated images <NUM> and the edge model <NUM>. Accordingly, the management service <NUM> or the processor <NUM> executing the model training component <NUM> may provide means for generating a second edge model based upon training the first edge model using the plurality of annotated images, the second edge model to be employed at the video capture device or another video capture device.

At block <NUM>, the method <NUM> optionally includes sending the second edge model to the edge device. For example, the model management component <NUM> may send the updated edge model <NUM>(<NUM>) to the edge device <NUM>(<NUM>). Upon receipt of the updated edge model <NUM>(<NUM>), the edge device <NUM>(<NUM>) may replace the edge model <NUM>(<NUM>) with the updated edge model <NUM>(<NUM>), and employ the updated edge model <NUM>(<NUM>) to process video data <NUM> received from the video capture device <NUM>. Accordingly, the management service <NUM> or the processor <NUM> executing the model management component <NUM> may provide means for sending the second edge model to the edge device.

The first object detection information includes a first plurality of bounding representations detected in a first image frame, the second object detection information includes a second plurality of bounding representations detected in the first image frame, and in order to select the plurality of training images from the sampled video information, the method <NUM> comprises selecting the first frame for the plurality of training images based on a count of the first plurality of bounding representations not equaling a count of the second plurality of bounding representations. Accordingly, the management service <NUM> or the processor <NUM> executing the image selection component <NUM> may provide means for selecting the first frame for the plurality of training images based on a count of the first plurality of bounding representations not equaling a count of the second plurality of bounding representations.

In an additional aspect, the first object detection information includes a first plurality of bounding representations detected in a first image frame, and the motion information includes a second plurality of bounding representations detected in the first image frame, and generating the plurality of annotated images comprises, and in order to generate the plurality of annotated images comprises, the method <NUM> comprises generating, based on a count of the first plurality of bounding representations equaling a count of the second plurality of bounding representations, a first annotated image of the plurality of annotated images including the first plurality of bounding representations. Accordingly, the management service <NUM> or the processor <NUM> executing the image annotation component <NUM> may provide means for generating, based on a count of the first plurality of bounding representations equaling a count of the second plurality of bounding representations, a first annotated image of the plurality of annotated images including the first plurality of bounding representations.

In an alternative or additional aspect, the first object detection information includes a first plurality of bounding representations detected in a first image frame, the second object detection information includes a second plurality of bounding representations detected in the first image frame, the motion information includes a third plurality of bounding representations detected in the first image frame, and in order to generate the plurality of annotated images comprises, the method <NUM> comprises determining that one or more bounding representations of the first plurality of bounding representations do not match any of the third plurality of bounding representations, determining that the one or more bounding representations of the first plurality of bounding representations match one or more bounding representations within the second object detection information, and generating a first annotated image of the plurality of annotated images including the first plurality of bounding representations.

Accordingly, the management service <NUM> or the processor <NUM> executing the image annotation component <NUM> may provide means for determining that one or more bounding representations of the first plurality of bounding representations do not match any of the third plurality of bounding representations, determining each of the one or more bounding representations of first plurality of bounding representations match a bounding representation within the second object detection information, and generating a first annotated image of the plurality of annotated images including the first plurality of bounding representations.

In an alternative or additional aspect, the first object detection information includes a first plurality of bounding representations detected in a first image frame, the second object detection information includes a second plurality of bounding representations detected in the first image frame, the motion information includes a third plurality of bounding representations detected in the first image frame, and in order to generate the plurality of annotated images comprises, the method <NUM> comprises determining that one or more bounding representations of the first image frame are within the second plurality of bounding representations and not within the plurality of bounding representations, determining that the one or more bounding representations do not match the third plurality of bounding representations, and generating a first annotated image of the plurality of annotated images including the one or more bounding representations. Accordingly, the management service <NUM> or the processor <NUM> executing the image annotation component <NUM> may provide means for determining that one or more bounding representations of the first image frame are within the second plurality of bounding representations and not within the plurality of bounding representations, determining that the one or more bounding representations do not match the third plurality of bounding representations, and generating a first annotated image of the plurality of annotated images including the one or more bounding representations.

In an alternative or additional aspect, the first object detection information includes a first plurality of bounding representations detected in a first image frame, the second object detection information includes a second plurality of bounding representations detected in the first image frame, the motion information includes a third plurality of bounding representations detected in the first image frame, and the method <NUM> further comprises determining a review context based at least in part on two of the first plurality of bounding representations, the second plurality of bounding representations, or the third plurality of bounding representations, receiving annotation information via a graphical user interface, and generating a second annotated image of the plurality of annotated images including the annotation information.

In an alternative or additional aspect, in order to generate the second edge model based upon training the first edge model using the plurality of annotated images, the method <NUM> comprises determining the second edge model based on at least a layer of the first edge model based on a transfer learning operation. Accordingly, the management service <NUM> or the processor <NUM> executing the model training component <NUM> may provide means for determining the second edge model based on at least a layer of the first edge model based on a transfer learning operation.

Referring to <FIG>, a computing device <NUM> may implement all or a portion of the functionality described herein. The computing device <NUM> may be or may include or may be configured to implement the functionality of at least a portion of the system <NUM>, or any component therein. For example, the computing device <NUM> may be or may include or may be configured to implement the functionality of the video capture devices <NUM>. The computing device <NUM> includes a processor <NUM> which may be configured to execute or implement software, hardware, and/or firmware modules that perform any functionality described herein. For example, the processor <NUM> may be configured to execute or implement software, hardware, and/or firmware modules that perform any functionality described herein with reference to the video capture device <NUM>, or any other component/system/device described herein, e.g., the edge model <NUM>, and/ or the GUI <NUM>.

The processor <NUM> may be a micro-controller, an application-specific integrated circuit (ASIC), a digital signal processor (DSP), or a field-programmable gate array (FPGA), and/or may include a single or multiple set of processors or multi-core processors. Moreover, the processor <NUM> may be implemented as an integrated processing system and/or a distributed processing system. The computing device <NUM> may further include a memory <NUM>, such as for storing local versions of applications being executed by the processor <NUM>, related instructions, parameters, etc. The memory <NUM> may include a type of memory usable by a computer, such as random access memory (RAM), read only memory (ROM), tapes, magnetic discs, optical discs, volatile memory, nonvolatile memory, and any combination thereof. Additionally, the processor <NUM> and the memory <NUM> may include and execute an operating system executing on the processor <NUM>, one or more applications, display drivers, and/or other components of the computing device <NUM>.

Additionally, the computing device <NUM> may include a data store <NUM>, which can be any suitable combination of hardware and/or software, that provides for mass storage of information, databases, and programs. For example, the data store <NUM> may be or may include a data repository for applications and/or related parameters not currently being executed by processor <NUM>. In addition, the data store <NUM> may be a data repository for an operating system, application, display driver, etc., executing on the processor <NUM>, and/or one or more other components of the computing device <NUM>, e.g., the edge model <NUM>, and/ or the GUI <NUM>.

Claim 1:
A method comprising:
receiving (<NUM>) sampled video information captured by a video capture device;
generating (<NUM>) first object detection information based on a cloud model and the sampled video information, the cloud model configured to detect objects within the sampled video information and included in a cloud computing environment connected to the video capture device, wherein the first object detection information includes a first plurality of bounding representations detected in a first image frame;
generating (<NUM>) second object detection information based on a first edge model and the sampled video information, the first edge model configured to detect objects within the sampled video information and employed at an edge device coupled with the video capture device, wherein the second object detection information includes a second plurality of bounding representations detected in the first image frame;
selecting (<NUM>), based on comparing the first object detection information to the second object detection information, a plurality of training images from the sampled video information, wherein selecting the plurality of training images from the sampled video information comprises: selecting the first image frame for the plurality of training images based on a count of the first plurality of bounding representations not equaling a count of the second plurality of bounding representations;
detecting (<NUM>) motion information corresponding to motion of one or more detected objects within the plurality of training images;
generating (<NUM>) a plurality of annotated images based at least in part on comparing the first object detection information to the motion information; and
generating (<NUM>) a second edge model based upon training the first edge model using the plurality of annotated images.