Real-time persistent object tracking for intelligent video analytics systems

Apparatuses, systems, and techniques for real-time persistent object tracking for intelligent video analytics systems. A state of a first object included in an environment may be tracked based on a first set of images depicting the environment. The first set of images may be generated during a first time period. It may be determined that the first object is not detected in the environment depicted in a second set of images. The second set of images may be generated during a second time period that is subsequent to the first time period. One or more predicted future states of the first object may be obtained in view of the state of the first object in the environment depicted in the first set of images. A second object may be detected in the environment depicted in a third set of images generated during a third time period that is subsequent to the second time period. A determination may be made as to whether a current state of the second object corresponds to at least one of the one or more predicted future states of the first object. In response to a determination that a current state of the second object corresponds to at least one of the predicted future states of the first object, an identifier associated with the second object is updated to correspond to an identifier associated with the first object.

TECHNICAL FIELD

At least one embodiment pertains to processing resources used for real-time persistent object tracking for intelligent video analytics systems. For example, at least one embodiment pertains to processors or computing systems used to provide and enable association of new objects detected in an environment to lost objects tracked by an intelligent video analytics system in real-time, according to various novel techniques described herein.

BACKGROUND

Efficient and effective object tracking is a critical task in video analytics applications, such as video analytics, video surveillance, activity recognition, vehicle navigation, etc. Some systems may utilize one or more object detection models to detect objects included in images depicting an environment. Such systems may estimate a state (e.g., a position, a location, a size, a scale, a velocity, etc.) of the detected object within the environment relative to a camera that generated the images, relative to other objects included in the environment, etc. The system may track the detected object's (also referred to as a target) state in subsequent images depicting the environment and may provide information associated with the target's state to a user of the system (e.g., via a client device, etc.).

DETAILED DESCRIPTION

Accurately detecting and tracking objects included in images is a challenging task. Modern video analytics systems, such as object monitoring systems, etc., may track an object in an environment by detecting the object in an image (e.g., a video frame) generated by a camera surveilling the environment and monitoring the state (e.g., location, position, speed, velocity, etc.) of the detected object in subsequent images generated by the camera. In some instances, an occlusion event can occur during monitoring of a tracked object (referred to herein as a target object or a target). An occlusion occurs when two or more objects in an environment come too close and seemingly merge or combine with each other, which prevents the video analytics system from differentiating between the two or more objects. For example, a partial occlusion (i.e., an object is partially merged or combined with another object) or full occlusion (i.e., an object is fully merged or combined with another object) can occur when a target in an environment moves in front of or behind a static object or another moving object or target, relative to a location of the camera that is surveilling the environment. During the occlusion event, the video analytics system may not detect the presence of a target in the environment, as the target may be partially or fully obscured. After the occlusion event, the target may move away from the other object or target (or vice versa) and the target may no longer be partially or fully obscured.

A video analytics system may lose track of a target during or after an occlusion event. For example, when a target is no longer partially or fully obscured by another object or target, the video analytics system may again detect the presence of the same object. However, the system may incorrectly identify the object as a new object instead of identifying the detected object as a target that had been tracked before the occlusion event. In some environments (e.g., a busy sidewalk, a crowded event space, etc.), targets can undergo multiple occlusion events as the video analytics system tracks such targets. Each time that a target becomes occluded (e.g., by another object) from the field of view of the monitoring system (camera) and appears (re-emerges) from the occluding object, the system may incorrectly identify the target as a newly detected object instead of identifying the object as an existing (e.g., previously detected) target. Accordingly, the system may incorrectly detect and subsequently track more targets than are actually present in the environment that is being surveilled. This incorrect object tracking can decrease an overall accuracy of the video analytics system. In addition, tracking more targets than are actually present in the environment being surveilled can consume a significant amount of computing resources (e.g., processing cycles, storage space, etc.). As a result, an overall efficiency of the system can decrease and an overall latency of the system can increase.

Embodiments of the present disclosure propose a technique to associate lost targets with newly detected objects after an occlusion event in real time or near real time (e.g., as a video of an environment is generated). A video analytics system may track a state (e.g., location, position, scale, velocity, etc.) of a target included in an environment relative to a position or location of a camera that is surveilling the environment. The system may track the state of the target by obtaining a first set of images (e.g., video frames) depicting the environment during a first time period and monitoring a state of the target (e.g., in view of bounding boxes obtained from an output of an object detection model, etc.) in the first set of images. During a second time period, the target may undergo an occlusion event (e.g., the first target may move in front of or behind another object or target within the environment relative to the camera). The system may obtain a second set of images depicting the environment during the second time period and determine that the target is not detected in the environment depicted in the second set of images (e.g., there are no bounding boxes associated with the second set of images that correspond to the first target).

The system may estimate one or more future states of the target within the environment. In some embodiments, the system may estimate one or more future states in response to determining that the target is not detected in the environment depicted in the second set of images. In other or similar embodiments, the system may estimate the one or more future states while tracking the target based on the first set of images (i.e., before the target is lost in the environment). The one or more future states of the target may be estimated in view of state data associated with the target in view of the first set of images. For example, the system may maintain state data associated with the target based on the location or position of the target detected in each of the first set of images. In some embodiments, the state data associated with the target may include coordinates (e.g., Cartesian coordinates, etc.) corresponding to regions of the environment that include the target before or at the time of the occlusion, a size and/or shape of one or more bounding boxes (i.e., associated with the first set of images) corresponding to the target detected at the regions of the environment, and/or a velocity of the first target within the environment. In some embodiments, the state data may also include coordinates corresponding to regions of the environment that are expected to include the target after the time of the occlusion (e.g., at a future time period). The system may obtain the state data associated with the target using one or more state functions (e.g., one or more Kalman filter functions, etc.) and/or a machine learning model (e.g., a recurrent neural network, etc.).

The system may calculate a path (e.g., a linear path, a non-linear path, etc.) that the target is expected to follow in the environment during or after the occlusion event based on the state data obtained for the target. For example, the system may calculate the path based on one or more coordinates corresponding to regions of the environment that include the target before or at the time of the occlusion (i.e., during the first time period) and one or more coordinates corresponding to regions of the environment that are expected to include the target after the time of the occlusion (i.e., during a future time period). In another example, the system may calculate the path based on one or more coordinates corresponding to regions of the environment that include the target before or at the time of the occlusion and the velocity of the target within the environment during the first time period. In some embodiments, the calculated path may include a set of coordinates that the target is expected to follow within the environment during the future time period. Each of the set of coordinates may correspond to a future state of the target during the future time period. The system may store an indication of the calculated path in a data store configured to store future paths for targets that are lost by the video analytics system.

In some embodiments, the system may detect another object in the environment depicted in a third set of images that are generated after the occlusion of the target (i.e., during a third time period) and may determine a current state of the detected object in the environment. The current state of the object may correspond to a set of coordinates for regions of the third set of images that depict a presence of the object. In some embodiments, the system may determine that the object is detected in a region of the environment that is at or near the region that included the target before the target was lost by the occlusion. In such embodiments, the system may obtain the calculated path associated with the target from the data store and may compare the current state of the detected object with each of one or more future states of the target (i.e., indicated by the calculated path). In one example, the system may calculate similarity metrics by performing a time series analysis (e.g., a dynamic time warping analysis) based on the current state of the detected object and the future states of the target. Responsive to determining that a calculated similarity metric for the state of the object and a respective future state of the target satisfies a similarity criterion (e.g., exceeds a similarity metric threshold), the system may determine that the detected object corresponds to the target (i.e., that the lost target has reappeared as the detected object). The system may associate the detected object with the target and may continue to track the target in accordance with embodiments described above.

Aspects and embodiments of the present disclosure provide a technique to reassociate lost targets with newly detected objects after an occlusion event in real time or near real time (e.g., as frames of a video are generated by one or more cameras). By re-associating lost targets with newly detected objects in real time, a video analytics system does not incorrectly track more targets than are actually present in an environment being surveilled. Additionally, an accuracy of the video analytics system is improved in embodiments. In addition, the amount of time of a post-processing phase for a video or video stream may be reduced, as a user or operator of the video analytics system does not manually associate lost targets with other objects detected in an environment after a complete video is generated. In view of the above, the overall accuracy of such video analytics systems may be high and such systems may consume fewer computing resources (e.g., processing cycles, etc.) as compared to prior object tracking systems, which increases an overall efficiency and increases an overall latency of the system.

System Architecture

FIG.1is a block diagram of an example system architecture100, according to at least one embodiment. The system architecture100(also referred to as “system” herein) may include a computing device102, an image source104, one or more data stores112, and/or server machines (e.g., server machines130-150), each connected to a network110. In implementations, network110may include a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or wide area network (WAN)), a wired network (e.g., Ethernet network), a wireless network (e.g., an 802.11 network or a Wi-Fi network), a cellular network (e.g., a Long Term Evolution (LTE) network), routers, hubs, switches, server computers, and/or a combination thereof.

Computing device102may be a desktop computer, a laptop computer, a smartphone, a tablet computer, a server, or any suitable computing device capable of performing the techniques described herein. In some embodiments, computing device102may be a computing device of a cloud computing platform. For example, computing device102may be, or may be a component of, a server machine of a cloud computing platform. In such embodiments, computing device102may be coupled to one or more edge devices (not shown) via network110. An edge device refers to a computing device that enables communication between computing devices at the boundary of two networks. For example, an edge device may be connected to computing device102, data store112, server machine130, server machine140, and/or server machine150via network110, and may be connected to one or more endpoint devices (not shown) via another network. In such example, the edge device can enable communication between computing device102, data store112, server machine130, server machine140, and/or server machine150and the one or more endpoint devices. In other or similar embodiments, computing device102may be, or may be a component of, an edge device. For example, computing device102may facilitate communication between data store112, server machine130, server machine140, and/or server machine150, which are connected to computing device102via network110, and one or more endpoint devices that are connected to computing device102via another network.

In still other or similar embodiments, computing device102may be, or may be a component of, an endpoint device. For example, computing device102may be, or may be a component of, devices, such as, but not limited to: televisions, smart phones, cellular telephones, personal digital assistants (PDAs), portable media players, netbooks, laptop computers, electronic book readers, tablet computers, desktop computers, set-top boxes, gaming consoles, autonomous vehicles, surveillance devices, and the like. In such embodiments, computing device102may be connected to data store112, server machine130, server machine140and/or server machine150via network110. In other or similar embodiments, computing device102may be connected to an edge device (not shown) of system100via a network and the edge device of system100may be connected to data store112, server machine130, server machine140and/or server machine150via network110.

Image source104may be or may include one or sensors that are configured to generate data, such as visual data, audio data, etc., associated with an environment. The sensors can include an image sensor (e.g., a camera), a light detection and ranging (LIDAR) sensor, a radio detection and ranging (RADAR) sensor, sound navigation and ranging (SONAR) sensor, an ultrasonic sensor, a microphone, and other sensor types. In some embodiments, the data collected and/or generated by the sensors may represent a perception of the environment by the sensors. It should be noted that although some embodiments of the present disclosure are directed to image data (e.g., an image) generated by one or more sensors of image source104, embodiments of the present disclosure may be applied to any type of data generated by one or more sensors of image source104(e.g., LIDAR data, RADAR data, SONAR data, ultrasonic data, audio data, etc.).

In some embodiments, image source104may be a component of, or may be otherwise connected to, computing device102. For example, as described above, computing device102may be, or may be a component of, an endpoint device. In such embodiments, image source104may be a camera component of computing device102that is configured to generate an image and/or video data associated with the environment. In other or similar embodiments, image source104may be a device, or a component of or otherwise connected to a device that is separate and distinct from computing device102. For example, as described above, computing device102may be, or may be a component of, a cloud computing platform or an edge device. In such embodiments, image source104may be a device (e.g., a surveillance camera, a device of an autonomous vehicle, etc.) that is connected to computing device102, data store112, and/or server machines130-150via network110or another network.

In some implementations, data store112is a persistent storage that is capable of storing content items (e.g., images) and data associated with the stored content items (e.g., object data, image metadata, etc.) as well as data structures to tag, organize, and index the content items and/or object data. Data store112may be hosted by one or more storage devices, such as main memory, magnetic or optical storage based disks, tapes or hard drives, NAS, SAN, and so forth. In some implementations, data store112may be a network-attached file server, while in other embodiments data store112may be some other type of persistent storage such as an object-oriented database, a relational database, and so forth, that may be hosted by computing device102or one or more different machines coupled to the computing device102via network110or another network.

Data store112may be or may include a domain-specific or organization-specific repository or data base. In some embodiments, computing device102, image source104, server machine130, server machine140, and/or server machine150may only be able to access data store via network110, which may be a private network. In other or similar embodiments, data stored at data store112may be encrypted and may be accessible to computing device102, image source104, server machine130, server machine140, and/or server machine150via an encryption mechanism (e.g., a private encryption key, etc.). In additional or alternative embodiments, data store112may be a publicly accessible data store that is accessible to any device via a public network.

Server machine130may include an image processing engine131that is configured to process data generated by image source104. For example, image source104and/or computing device102may encode image data (e.g., using a codec) generated by image source104prior to transmitting the image data to another device of system100via network110(or another network). Image processing engine131may decode the encoded image data (e.g., using the codec). In some embodiments, image processing engine131may re-encode decoded image data (e.g., using a different codec), prior to providing the image to another component or device of system100. In some embodiments, image process engine131may be configured to select, combine, and transmit signals (e.g., via a multiplexer component, etc.) associated with image data generated by image source104to another component or device of system100. In additional or alternative embodiments, image processing engine131may be configured to modify a quality of the image data generated by image source104before the image data is used for object detection and/or object tracking (e.g., by object detection engine141and/or object tracking engine151). For example, image processing engine131may be configured to apply one or more transformations to an image generated by image source104to remove or reduce an amount of noise present in the image, to crop the image, and so on. It should be noted that although some embodiments of the present disclosure provide that image processing engine131may modify a quality of image data, other components of system100(e.g., object detection engine141, object tracking engine151, etc.) may also be configured to modify the quality of the image data.

Server machine140may include an object detection engine141configured to detect one or more objects included in images depicting an environment, such as images generated by image source104. In some embodiments, object detection engine141may provide an image depicting an environment as input to a trained object detection model. The object detection model may be trained using historical data (e.g., historical images, historical object data, etc.) from one or more datasets to detect an object (referred to here as a detected object) included in a given input image depicting an environment, and estimate a region of the given input image that includes the detected object (referred to herein as a region of interest). In some embodiments, one or more outputs of the object detection model can indicate object data associated with the detected object. The object data may indicate a region of interest of a given input image that includes the detected object. For example, the object data can include a bounding box or another bounding shape (e.g., a spheroid, an ellipsoid, a cylindrical shape, etc.) that corresponds to the region of interest of the given input image. In some embodiments, the object data can include other data associated with the detected object, such as an object class corresponding to the detected object, mask data associated with the detected object (e.g., a two-dimensional (2D) bit array that indicates pixels (or groups of pixels) that corresponds to the detected object), and so forth.

Server machine150may include an object tracking engine151configured to track a state of one or more objects detected in one or more images (e.g., generated by image source104). For purposes of explanation, an object that is detected by object detection engine141is referred to herein as a detected object. An object that is tracked by object tracking engine151is referred to herein as a target object or a target. A state of a target, as provided herein, may correspond to a location of an object within an environment depicted by the one or more images, a position of the object within the environment, a scale or size of the object within the environment, a velocity of the object within the environment, and so forth.

In some embodiments, object tracking engine151may track a target based on an image including the target and object data (e.g., one or more bounding boxes) associated with the target. Object tracking engine151may instantiate an object tracker component (referred to as an object tracker herein) for each detected object in an image depicting the environment. An object tracker may be a logical component that is configured to maintain state data associated with a target within a set of images (e.g., a sequence of video frames) depicting the environment. For example, when an object is initially detected in an image (e.g., a video frame), object tracking engine151may instantiate an object tracker to monitor and determine a state associated with the detected object (referred to herein as a current state of the target). Object detection engine141may detect the target in other images depicting the environment (e.g., subsequent video frames) and the object tracker associated with the target may determine, for each of the other images, the current state of the target. The object tracker may update state data associated with the object to correspond to the determined current state and store the updated state data (e.g., at data store112). In some embodiments, the object tracker may further estimate a future state of the target in the environment and may store an indication of the future state (e.g., at data store112) with the updated state data. Further details regarding determining the current state of a target and estimating the future state of the target are provided herein.

In some embodiments, a target may become “lost” within an environment (i.e., object detection engine141and/or object tracking engine151may no longer detect a presence of the target in images depicting the environment). In one example, the target may move to a location within the environment that is not detectable by image source104(e.g., the target moves out of the environment depicted in images generated by image source104). In another example, the target may undergo an occlusion event within the environment. As described above, an occlusion occurs when an object is obscured from the point of view of the camera (image source104) by another object in the environment (e.g., one object walks behind one or more other objects from the perspective of the camera), which prevents object detection engine141and/or object tracking engine151from detecting or tracking the object being occluded. A partial occlusion (i.e., an object is partially occluded by another object) or a full occlusion (i.e., an object is fully occluded by another object) can occur when the target moves in front of or behind a static object or target or another moving object or target, relative to a position and/or location of image source104. Responsive to determining that a particular target is lost (i.e., is no longer visible by the camera or image source104) within an environment, the object tracker associated with the target may determine whether the lost target corresponds to another object that is detected in additional images depicting the environment. The additional images may be generated at a later time. Accordingly, there may be a period of time (e.g., one or more frames of a video) in which the tracked object is lost, after which tracking may again resume. If the lost object corresponds to another detected object, object tracking engine151may associate the “lost” target to the detected object and the object tracker may continue to track the target, as described herein. If the lost target does not correspond to another detected object, object tracking engine151may determine that the target is no longer present in the environment and, in such embodiments, may terminate the object tracker associated with the lost target. Further details about terminating a lost target and/or associating a lost target with another object detected in images depicting an environment are provided herein.

In some implementations, computing device102, image source104, data store112, and/or server machines130-150, may be one or more computing devices computing devices (such as a rackmount server, a router computer, a server computer, a personal computer, a mainframe computer, a laptop computer, a tablet computer, a desktop computer, etc.), data stores (e.g., hard disks, memories, databases), networks, software components, and/or hardware components that may be used to enable object detection based on an image (e.g., image106). It should be noted that in some other implementations, the functions of computing device102, image source104, server machines130,140, and/or150may be provided by a fewer number of machines. For example, in some implementations server machines130,140, and/or150may be integrated into a single machine, while in other implementations server machines130,140, and150may be integrated into multiple machines. In addition, in some implementations one or more of server machines130,140, and150may be integrated into computing device102. For example, as illustrated inFIG.1, image processing engine131, object detection engine141, and/or object tracking engine151may reside at on computing device102, in some embodiments. In general, functions described in implementations as being performed by computing device102and/or server machines130,140,150may also be performed on one or more edge devices (not shown) and/or client devices (not shown), if appropriate. In addition, the functionality attributed to a particular component may be performed by different or multiple components operating together. Computing device102and/or server machines130,140,150may also be accessed as a service provided to other systems or devices through appropriate application programming interfaces.

FIG.2is a block diagram of an image source104, an object detection engine131and an object tracking engine151, according to at least one embodiment. As described with respect toFIG.1, image source104may be or may include one or more sensors (e.g., image sensors, etc.) that are configured to generate data associated with an environment. For example, image source104may be, or may include, a camera component that is configured to generate a video stream (i.e., a sequence of video frames or image frames) depicting the environment over a period of time.

Image source104may generate an image202, in accordance with previously described embodiments, and may provide the image202to object detection engine141. In some embodiments, image source104may provide image202to image processing engine131, as described with respect toFIG.1. Image processing engine131may process image202, in accordance with previously described embodiments, and provide image202to object detection engine141. In response to obtaining image202, object detection engine141may provide image202as input to a trained object detection model and obtain one or more outputs of the model that indicate object data204associated with one or more objects detected in image202, as previously described. The trained object detection model may be, for example, an artificial neural network such as a convolutional neural network trained to identify one or more types of objects, such as cars, people, animals, and so on. In some embodiments, object data204may include a bounding box (or a bounding shape) that indicates a region of image202that includes a detected object. Image202and/or object data204may be stored at data store250, in some embodiments. Data store250may correspond to data store112, described with respect toFIG.1, or may be different from data store112.

FIGS.3A-3Cdepict example images202A-202C generated by image source104, according to at least one embodiment. As illustrated inFIG.3A, image202A depicts an example environment302including objects304,306,308and310. In some embodiments, image202A may be a first video frame of a sequence of video frames depicting environment302. Object detection engine131may obtain image202A and provide image202A as input to a trained object detection model, as described above. One or more outputs of the object detection model may indicate regions of image202A that include detected objects. The regions of image202A indicated by the one or more outputs may correspond to a bounding box or other bounding shape associated with the detected objects. For example, a first region indicated by the one or more outputs may correspond to a first bounding box312associated with object304, a second region may correspond to a second bounding box314associated with object306, a third region may correspond to a third bounding box316associated with object308and a fourth region may correspond to a fourth bounding box318associated with object310. Object data204generated for image202A may include an indication of bounding boxes312-318, in some embodiments.

Referring back toFIG.2, object tracking engine151may obtain image202and/or object data204from object tracking engine141, from image source104, and/or via a data store, such as data store112described with respect toFIG.1. As illustrated inFIG.2, object tracking engine151may include an object localization module210, a data association module214, a target manager module216, one or more object trackers218, a state estimation module220, and/or a tracklet manager module226. Object localization module210may be configured to estimate a location of existing targets (referred to herein as localizing targets) tracked by object tracking engine151in a sequence of images202generated by image source104.

In some embodiments, in response to obtaining object data204, object localization module210may determine whether any object trackers218have been instantiated to track targets in the environment depicted in image202. As described with respect toFIG.3A, image202A may be a first video frame of a sequence of video frames depicting environment302. As image202A may be the first video frame, object localization module210may determine that no object trackers218have been instantiated to track targets at the time object tracking module151obtains image202A. In such embodiments, object localization module210may extract one or more visual features associated with each detected object (i.e., objects304,306,308,310) depicted in image202A. The visual features may include an indication of one or more colors present in a set of pixels of a region of image202A indicated by a bounding box (referred to herein as a bounding box region), a Histogram-of-Oriented-Gradient (HOG) of the bounding box region, or other visual features. Object localization module210may extract the visual features associated with the detected objects from the regions of image202A indicated by bounding boxes312,314,316, and318.

The detected objects from different video frames may be compared to one another based on their visual features by similarity component212. In some embodiments, similarity component212of object localization module210may generate a set of similarity metric values each indicating a similarity between a detected object304,306,308,310from a current image or video frame and an existing target (e.g., associated with visual features extracted from one or more previous image or video frame). As described above, object localization module210may determine that no object trackers218have been instantiated for targets at the time image202A is obtained, and therefore object tracking engine151may not be tracking any targets. Accordingly, similarity component212may assign each of the detected objects304,306,308,310a particular similarity metric value (e.g., a low similarity metric value) which indicates that each of the detected objects304,306,308,310do not correspond to an existing target.

Referring back toFIG.2, in some embodiments, object detection engine141may not attempt to detect objects and generate object data204for each image202generated by image source104(e.g., in accordance with a protocol for the video analytics system). For example, object detection engine may be configured to detect objects in every other image202generated by image source104, every few images202generated by image source104, etc. In such embodiments, object localization module210may be configured to detect and localize one or more objects depicted in image202using a correlation filter. A correlation filter refers to a class of classifiers that are configured to produce peaks in correlation outputs or responses. In some embodiments, a peak in a correlation output or response can correspond to an object depicted in image202. In some embodiments, a correlation filter can include at least one of a Kernelized Correlation Filter (KCF), a discriminative correlation filter (DCF), a Correlation Filter neural network (CFNN), a Multi-Channel Correlation Filter (MCCF), a Kernel Correlation Filter, an adaptive correlation filter, and/or other filter types. A correlation filter may be implemented using one or more machine learning models, such as a machine learning model that uses linear regression, logistic regression, decision trees, support vector machines (SVM), Naïve Bayes, K-nearest neighbor (KNN), K-means clustering, random forest, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., auto-encoders, convolutional, recurrent, perceptrons, long/short term memory/LSTM, Hopfield, Boltzmann, deep belief, deconvolutional, generative adversarial, liquid state machine, etc.), and/or other types of machine learning models.

A correlation filter may be trained to produce or identify a peak correlation response at a region of an image that corresponds to a reference coordinate (e.g., a center) of an object depicted in the image. Object localization module210may obtain an image202(i.e., from image source104or via data store250) and apply the correlation filter to image202to obtain one or more outputs. The one or more outputs of the correlation filter can indicate one or more peak locations of a correlation response for image202(referred to herein simply as a correlation response). The locations of one or more correlation responses may correspond to regions of image202that depict an object in the environment and, in some embodiments, the peak location of the correlation response may correspond to the reference coordinate (e.g., the center) of the depicted object. Object localization module210may identify the regions of image202that are associated with a respective correlation response as regions of image202that depict a respective object (referred to herein as a correlation response region). In some embodiments, similarity component212may extract features from a correlation response region and assign a similarity metric value to the respective object depicted in the correlation response region and existing targets tracked by object tracking engine151, as described above and in further detail below.

In some embodiments, object localization module210may apply the correlation filter to an image202even if object detection engine141generates object data204associated with image202. In such embodiments, object localization module210may use object data204and the output of the correlation filter to improve (i.e., re-train) the correlation filter for subsequent images (e.g., video frames) generated by image source104. For example, object localization module210may identify the correlation response regions of image202based on one or more outputs of the correlation filter. Object localization module210may compare the correlation response at the respective correlation response regions of image202to each bounding box indicated by object data204and determine an accuracy of the respective correlation responses based on the comparison. In some embodiments, object localization module210may provide an indication of the correlation responses, the bounding boxes indicated by object data, and/or the determined accuracy of each respective correlation responses to re-train the correlation filter.

It should be noted that although some embodiments of the present disclosure are directed to localizing visual features of detected or depicted objects to the existing targets, other techniques may be used to localize the existing targets. For example, in response to obtaining object data204for image202, object localization module210may extract one or more visual features from regions of image202indicated by bounding boxes of object data204. Object localization module210may provide the extracted visual features as input to a machine learning model (e.g., a recurrent neural network, etc.) and obtain one or more outputs of the machine learning model. Object localization module210may extract, from the one or more obtained outputs, an identifier associated with one or more attributes of the extracted visual features. Object localization module210may compare the extracted identifier to identifiers associated with existing targets and provide an indication of the comparison to data association module214, in some embodiments.

Object localization module210may provide an indication of the object data204associated with image202(e.g., the bounding box regions of image202, correlation response regions of image202, etc.) and the set of similarity metric values to data association module214. Data association module220may be configured to determine whether a bounding box region and/or a correlation response region corresponds to an estimated location of an existing target (i.e., indicated by a future target state258for the target, as described below). In some embodiments, data association module214can compare a location of a respective bounding box region to a location of an estimated target location and determine, based on the comparison, whether the bounding box region is located within a threshold proximity of the estimated target location. In response to determining that the bounding box region is located within the threshold proximity of the estimated target location, data association module214may determine that bounding box region matches, or approximately matches, the region of image202that corresponds to the estimated target location. Such bounding box regions are referred to herein as matched bounding box regions. Responsive to determining that the bounding box region is located outside of the threshold proximity of the estimated target location, data association module214may determine that the bounding box region does not match the region of image202. Such bounding box regions and estimated target locations are referred to herein as unmatched bounding box regions and unmatched estimated target locations, respectively.

In additional or alternative embodiments, data association module214may determine whether a bounding box region of image202corresponds to an estimated target location based on a similarity metric value associated with the detected object included in the bounding box region and a respective target (e.g., determined from one or more previous images). For example, in response to determining that a similarity metric value associated with a detected object and a respective target satisfy a similarity criterion (e.g., the similarity metric value meets or exceeds a threshold value), data association module214may determine that the bounding box region that includes the detected object matches or approximately matches the estimated target location (i.e., the bounding box region is a matched bounding box region). Responsive to determining that the similarity metric value does not satisfy the similarity criterion (e.g., the similarity metric falls below the threshold value), data association module214may determine that the bounding box region that includes the detected object does not match the estimated target location (i.e., the bounding box region and/or the estimated target location is an unmatched bounding box region and/or an unmatched estimated target location, respectively).

In some embodiments, data association module214may provide an indication of each unmatched bounding box region and unmatched estimated target location to target manager module216, in some embodiments. Target manager module316may be configured to instantiate and/or terminate each object tracker218of object tracking engine151. As indicated above, an object tracker refers to a logical component that is configured to track a state of a target included in a set of images (e.g., a sequence of video frames) depicting an environment. In response to receiving the indication of the unmatched bounding box regions and/or unmatched estimated target locations, target manager module216may determine whether to instantiate one or more new object trackers218(i.e., to create a new target) or terminate an instantiated object tracker218for an existing target (e.g., in accordance with a target termination policy). In an illustrative example, an unmatched bounding box region may indicate to the target manager module216that a new object has been detected in the surveilled environment. Accordingly, target manager module216may instantiate a new object tracker218to track the state of the detected object in image202and subsequent images (e.g., video frames) generated by image source104. In some embodiments, target manager module216may instantiate a new object tracker218by assigning the target a target identifier (ID) and storing the target ID at data store250as target ID252. In another illustrative example, an unmatched estimated target location may indicate to target manager module216that a target is no longer present in the environment surveilled by image source104. In response to determining that the target satisfies one or more conditions of a target termination policy, target manager module216may terminate an object tracker218that was instantiated to track the state of the target. In some embodiments, target manager module216may terminate the object tracker218by removing the target ID252for the terminated target from data store250and/or recycling the target ID252of the terminated target to be used for a new target.

As indicated previously with respect toFIG.3A, image202A may be the first video frame in a sequence of video frames generated by image source104and no object trackers218may be instantiated for any targets in environment302at the time image202A is generated. Data association module214may identify each bounding box312,314,316,318as unmatched bounding box regions and accordingly, target manager module216may instantiate a respective object tracker218for each of objects304,306,308, and310, in accordance with previously described embodiments.

Referring back toFIG.2, an object tracker218may be configured to track a state of a respective target in an environment. A target state may refer to a location, a position, a scale or size, a velocity, etc. associated with a target during a time period that an image202is generated. An object tracker218associated with a respective target may determine one or more target states (e.g., a prior target state254, a current target state256, a predicted target state258, etc.) based on state estimations and/or predictions made by state estimation module220. As illustrated inFIG.2, state estimation module220may include a state estimation component222and a state prediction component224. State estimation component222may be configured to determine a current target state256based on state data associated with a target at the time an image202depicting the target is generated. For example, a current target state256may be defined by one or more coordinates for a bounding box associated with the target in image202, a size of the bounding box associated with the target, and/or a change in the one or more coordinates for the bounding box relative to prior coordinates of a bounding box associated with the target in one or more prior images depicting a surveilled or monitored environment. In another example, the current target state256may be further defined by a change in the size of the bounding box associated with the target relative to a bounding box associated with the target in the one or more prior images. In some embodiments, the current target state256may also include one or more target features (e.g., extracted from the bounding box region of image202, extracted from a correlation response region of image202, etc.).

In some embodiments, state estimation component222may determine a current target state based on data obtained for the target from image202. For example, an object tracker218, data association module214and/or object localization module210may provide an indication of one or more bounding boxes associated with the target to state estimation component222, in some embodiments. State estimation component222may determine the coordinates of the one or more bounding boxes and/or the size of the one or more bounding boxes based on the provided data. In some embodiments, state estimation component222may determine whether the target is a new target in image202or the target is an existing target that was tracked before image202was generated. In response to determining that the target was an existing target, state estimation component222may obtain prior target state data254for the target (e.g., from data store250). Prior target state data254refers to target state data that as estimated (e.g., by state estimation component222) for a target based on images generated prior to image202. State estimation component222may determine the change in the one or more coordinates for the bounding box associated with the target by determining a distance between the one or more coordinates of the bounding box associated with image202and coordinates of a bounding box associated with the target depicted in one or more prior images. State estimation component222may determine a speed and direction (i.e., a velocity) at which the target is moving based on the determined distance. In some embodiments, state estimation component222may further determine a change in the size or scale of the target based on the determined distance.

As indicated above, the change in the one or more coordinates for the bounding box associated with the target depends on the location of a bounding box for an image generated prior to image202. Accordingly, if a target is a new target in image202, state estimation component222may not determine the change in the location and/or size or scale of the target (i.e., as no prior images generated by image source104depict the target). If the target is depicted in subsequent images of the surveilled environment, state estimation component222may determine the velocity and/or size or scale change of the target when the subsequent images are generated, in accordance with previously described embodiments.

It should be noted that in some embodiments described below, object localization module210may identify one or more correlation response regions of image202(e.g., using a correlation response filter, etc.). In such embodiments, state estimation component222may determine the current state of the target based on the identified correlation response regions in addition to or in lieu of the bounding box regions of image202.

State estimation component222may store the coordinates of the one or more bounding boxes, the coordinates of one or more correlation response regions, the size of the one or more bounding boxes and/or the correlation response region, the velocity of the target, and/or the change in size or scale of the target as current target state256in data store250. In some embodiments, state prediction component224may be configured to predict a future state of the target in the environment based on the current target state256for the target. In some embodiments, state prediction component224may obtain the current target state256and provide the current state256as an input to one or more state prediction functions. A state prediction function may be configured to execute a recursive filter, such as a Kalman Filter (KF), to estimate a future state of a target in the environment. State prediction component224may obtain an output from the one or more state prediction functions and determine, based on the output, a future state of the target during a time that is subsequent to when image202is generated, in some embodiments. In other or similar embodiments, state prediction component224may determine multiple future states of the target during a time period that is subsequent to when image202is generated. For example, state prediction component224may determine, based on the output of the one or more state prediction functions, a future state of the target at each instance of time of a time period that is subsequent to when image202is generated. State prediction component224may store the one or more future states of the target at data store250as future target state258.

In additional or alternative embodiments, state prediction component224may use one or more machine learning models to predict the future state of the target. The one or more machine learning models may include a long term short term memory (LSTM) model, or another type of recurrent neural network (RNN) model. In some embodiments, the one or more machine learning models may be trained using historical object data and/or historical target state data to predict a future state of a target based on given target state and/or object data. State prediction component224may provide object data204, the prior target state254and/or the current target state256for a target as input to the one or more machine learning models and may obtain an output of the one or more models. State prediction component224may extract, from the one or more outputs, multiple sets of state data for the target. Each set of target state data may correspond to a future state of the target at an instance of time that is subsequent to when image202is generated. In some embodiments, state prediction component224may also extract an indication of a level of confidence that a respective set of state data corresponds to the target. State prediction component224may identify one or more sets of state data associated with a level of confidence that satisfies a level of confidence criterion. For example, state prediction component224may identify a set of state data that is associated with the higher level of confidence than other sets of state data extracted from the one or more outputs. In another example, state prediction component224may identify each set of state data associated with a level of confidence that meets or exceeds a threshold level of confidence. Responsive to identifying the one or more sets of state data, state prediction component224may store the one or more sets of state data as predicted target state258at data store250, as described above.

As described above, target manager module216may instantiate a new object tracker218to track a state of a newly detected object in image202(e.g., in view of an unmatched bounding box determined for image202). However, in some embodiments, the newly instantiated object tracker218may not be configured to begin tracking the target until the target has been detected (e.g., by object detection engine141and/or object localization module210) for a threshold number of images202generated by image source104. In an illustrative example, target manager module216may instantiate an object tracker218to track an object that is first detected in a first video frame. However, the object tracker218for the target may not obtain and/or provide state data associated with the target based on first video frame (i.e., the object tracker218may not be tracking the state of the target based on the first video frame). If the target is detected (e.g., by object detection engine141and/or object localization module210) in a threshold number of subsequent video frames, the object tracker218may be configured to obtain state data for the target and provide the state data to state estimation module220, in accordance with embodiments described above. The technique of delaying tracking of a target until the target is detected in a threshold number of images202generated by image source104is referred to herein as late object tracker activation or simply late activation.

FIG.3Bdepicts one or more estimated locations of targets304,306,308,310in environment302at a time period after image202A is generated. The estimated locations of targets304,306,308,310may correspond to a predicted target state258associated with each target, as determined by state prediction component224, in accordance with previously described embodiments. As illustrated inFIG.3B, state prediction component224can predict that target304may be present at location320of environment302, target306may be present at location322of environment302, target308may be present at location324of environment302, and target310may be present at location326of environment302in a future image or video frame. In some embodiments, object localization module210may localize targets304,306,308,310based on the estimated locations320,322,324,326(e.g., if object detection engine141does not generated object data204associated with an image generated after image202A, as described above).

FIG.3Cillustrates another image202B depicting example environment302. In some embodiments, image202B may be a video frame that is subsequent to the first video frame (i.e., image202A) of the sequence of video frames depicting environment302. Object detection engine141may generate object data204for one or more objects detected in image202B. As illustrated inFIG.3C, object data204may include bounding boxes350,352, and354. Object localization module210may obtain image202A and the corresponding object data204, as described above, and initiate one or more processes to localize existing targets (e.g., targets304,306,308,310) in image202B. In some embodiments, object localization module210may obtain predicted state data258associated with each respective target (e.g., from data store250) and may estimate a location of each respective target in image202B based on the obtained predicted state data258. In accordance with embodiments described with respect toFIG.3B, object localization module210may estimate that target304is present at location320, target306is present at location322, target308is present at location324, and target310is present at location326of environment302depicted in image202B.

Similarity component212may extract visual features from the bounding box regions of image202B and the correlation response regions of image202B that correspond to the estimated locations320,322,324,326of each respective target (referred to herein as an estimated target region). Similarity component212may compare the extracted visual features of the detected objects in image202B with visual features associated with tracked objects. Similarly component212may determine a similarity metric value associated with the extracted visual features based on the comparison, and may provide an indication of the bounding box regions, the estimate target regions, and the determined similarity metric values to data association module214, as previously described. In accordance with previously described embodiments and examples, data association module214may determine that bounding box350matches with estimated location320and bounding box352matches with estimated location352based on the similarity values satisfying one or more similarity criteria (e.g., a difference being less than a difference threshold). Accordingly, object trackers218associated with targets304and308may provide state data associated with targets304and308to state estimation module220to update the current states of targets304and308in view of image202B. For example, state estimation component222may determine a new state associated with targets304and308in view of image202B and may update the current target state256for each target based on the determined new state. State estimation component222may store the state determined for targets304and308with respect to image202A as prior state data254and may store the updated current target state256at data store250, as described above. In some embodiments, state prediction component224may predict a future location of targets304and308in environment302update the future target states258in view of the predicted future locations.

In some embodiments, data association module214may determine that bounding box354does not match with an estimated location associated with an existing target in environment302. Accordingly, target manager module216may determine that bounding box354corresponds to a new detected object in the environment and may instantiate an object tracker218to track the detected object. In additional or alternative embodiments, data association module214may determine estimated locations322and326do not match with a bounding box of object data204(i.e., estimated locations322and326are unmatched estimated locations). Accordingly, target manager module216may determine to terminate object trackers associated with310and/or306, in accordance with a target management policy and/or embodiments described herein.

As indicated above, target manager module216may be configured to terminate an object tracker218for a target if the target is determined to be “lost,” in accordance with a target termination policy of the video analytics system. In some embodiments, the target termination policy may provide that target manager module216may not terminate an object tracker218until the target associated with the object tracker is “lost” for a threshold number of images generated by image source104. In such embodiments, the object tracker218may continue to track the target based on the predicted target state258determined for the target (e.g., based on the most recent current object state256determined for the target). In an illustrative example, if a target is tracked based on one or more video frames, in accordance with previously described embodiments, and is determined to be “lost” in a subsequent video frame, the object tracker218associated with the target may continue to track the target, even though the target is “lost” in the subsequent video frame. If object detection engine141and/or object localization module210does not detect the “lost” target in a threshold number of subsequent video frames, the target manager module216may terminate the object tracker218associated with the target, in accordance with the target termination policy. The technique of tracking a target in an environment even though the target is not detected (e.g., by object detection engine141and/or object localization module210) is referred to herein as shadow tracking.

Tracklet manager module226may be configured to generate and maintain one or more tracklets associated with targets tracked by object tracking engine151. A tracklet refers to a set of coordinates that indicates a path that is predicted to be taken by a target in a surveilled environment. Tracklet generator228may be configured to generate a tracklet for a target based on a prior target state254, a current target state256, and/or a predicted target state258associated with the target. In some embodiments, tracklet generator228may use a prior target state254and/or a current target state256for a target to generate a tracklet that indicates a prior and/or current path that the target has taken. Tracklet generator228may store such tracklet as a current target tracklet260at data store250. In additional or alternative embodiments, tracklet generator228may use a prior target state254, a current target state256, and/or a predicted target state258associated with the target to generate a tracklet that indicates a predicted or future path that the target is expected to take. Tracklet generator228may store such tracklet as a predicted target tracklet262at data store250.FIGS.5A-5Cillustrate a current target tracklet260and a predicted target tracklet262associated with a target tracked by object tracking engine151, as described in further detail herein. In some embodiments, tracklet generator228may generate predicted target tracklets262for all targets tracked by object tracking engine151. In other or similar embodiments, tracklet generator may be configured to generate predicted target tracklets262for lost targets before object trackers218associated with the lost tracklets are terminated (i.e., in accordance with the shadow tracking technique described above).

In some additional embodiments, tracklet generator228may also be configured to generate a tracklet for new targets that are associated with an object tracker218that is not yet configured to track the target (i.e., an object tracker218that has not yet been activated in accordance with the late activation technique described above). In some embodiments, tracklet generator228may obtain state data associated with the new target (e.g., from object localization module210, from data association module214, etc.). In some embodiments, tracklet generator228may determine a current state associated with the new target based on the obtained state data and may update a tracklet associated with the new target based on the determined current state. In other or similar embodiments, tracklet generator228may provide the current state to state estimation component222, which may determine the current target state256, as described above. Tracklet generator228may update the tracklet associated with the target based on the current target state256, as described above.

Tracklet matching component230may be configured to identify matching tracklets (e.g., in data store250). For example, as indicated above, tracklet generator228may generate a predicted target tracklet262for a lost target that is tracked according to the shadow tracking technique. Tracklet generator228may also generate a current target tracklet260for a new target that is not yet being tracked by an object tracker218, in accordance with the late activation technique. In some embodiments, tracklet matching component230may compare current target tracklets260for new targets to predicted target tracklets262for lost targets and determine whether a current target tracklet260for a new target corresponds to a predicted target tracklet262(i.e., whether the new target is the same object as the lost target) based on the comparison. In response to tracklet matching component230determining that a current target tracklet260for a new target corresponds to a predicted target tracklet262(i.e., the lost target is now a found target), tracklet fusion component234may associate an identifier associated with the new target to correspond to an identifier for the lost target (i.e., to “fuse” or “merge” the current target tracklet260with the predicted target tracklet262). The object tracker218associated with the lost target may continue to track the object (i.e., the found target), in accordance with previously described embodiments. In some embodiments, target manager module216may terminate the object tracker218that was instantiated for the new object. Further details about identifying and fusing matching tracklets are provided in further details below.

Real-Time Persistent Object Tracking

FIG.4illustrates a flow diagram of an example method400of managing lost objects tracked by a video analytics system, according to at least one embodiment. In at least one embodiment, method400may be performed by client device102, server machine130, server machine140, and/or server machine150. For example, one or more operations of method400may be performed by one or more components or modules of image processing engine131, object detection engine141, and/or object tracking engine151. Method400may be performed by one or more processing units (e.g., CPUs and/or GPUs), which may include (or communicate with) one or more memory devices. In at least one embodiment, method400may be performed by multiple processing threads (e.g., CPU threads and/or GPU threads), each thread executing one or more individual functions, routines, subroutines, or operations of the method. In at least one embodiment, processing threads implementing method400may be synchronized (e.g., using semaphores, critical sections, and/or other thread synchronization mechanisms). Alternatively, processing threads implementing method400may be executed asynchronously with respect to each other. Various operations of method400may be performed in a different order compared with the order shown inFIG.4. Some operations of the methods may be performed concurrently with other operations. In at least one embodiment, one or more operations shown inFIG.4may not always be performed.

Processing units performing method400may track, at block410, a state of a first object included in an environment based on a first set of images depicting the environment during a first time period. In some embodiments, the first set of images may be a sequence of images202generated by image source104, as described with respect toFIGS.1and2. Object tracking engine151may track the state of the first object, in accordance with previously described embodiments.FIG.5Adepicts an example image502depicting an environment and an object504included in the environment. Image502may correspond to the first set of images generated during the first time period. In some embodiments, object504may be tracked by object tracking engine151. In such embodiments, target manager module216of object tracking engine151may instantiate an object tracker218to track a state of object504(also referred to herein as target504). Object tracker218may assign a tracker identifier506A to target504. In an illustrative example, object tracker218may assign a tracker identifier506A of “1” to target504. In some embodiments, object detection engine141may generate object data204associated with target504, such as bounding box508A. Tracklet generator228may generate a current target tracklet260associated with target504, in accordance with previously described embodiments. Current target tracklet260is depicted inFIG.5Aas tracklet510. As further illustrated inFIG.5A, target504may be moving towards object512(e.g., a lamp post), relative to a position or location of image source104that generated image502.

Referring back toFIG.4, at block412, processing units performing method400may determine that the first object is not detected in the environment depicted in a second set of images generated during a second time period.FIG.5Bdepicts another example image514generated by image source104. Image514may correspond to a second set of images generated during a second time period. As illustrated inFIG.5B, target504may undergo an occlusion event with object512. In some embodiments, object detection engine141and/or object localization module210may not detect target504due to the occlusion event. Accordingly, target manager module216may determine that target504is a lost target and may continue to track target504based on a shadow tracking technique, as described above.

Referring back toFIG.4, at block414, processing units performing method400may obtain one or more future states of the first object in view of the state of the first object in the environment depicted in the first set of images. The one or more future states of the first object (e.g., target504) may correspond to a future target tracklet262generated by tracklet generator228. As described above, tracklet generator228may generate predicted target tracklet262based on prior target state254, current target state256, and/or predicted target state258associated with target504.FIG.5Cdepicts another example image516generated by image source104. The predicted target tracklet262may be depicted inFIG.5Cas tracklet518. As illustrated inFIG.5C, tracklet518begins at approximately the location at which object detection engine141and/or object localization module210did not detect target504in image514. Tracklet518may correspond to a set of coordinates of image514for a path that target504is expected to follow in the environment (e.g., in view of the prior target state254, current target state256, and/or predicted target state258associated with target504).

Referring back toFIG.4, at block416, processing units performing method400may detect a second object included in the environment based on a third set of images generated during a third time period.FIG.5Ddepicts another example image520generated by image source104. Image520may correspond to a third set of images generated during a third time period. As illustrated inFIG.5D, target504may reappear in the environment (e.g., after moving from behind object512). Object detection engine141and/or object localization module210may detect target504as a new object in the environment. In some embodiments, object detection engine141may generate object data204associated with the new object, such as bounding box508B. Tracker manager module216may instantiate an object tracker218to track the new object, as described above. As illustrated inFIG.5D, object tracker218may assign the new object a target ID506B of “60.” In some embodiments, target manager module216may not yet activate the object tracker218for the new object, in accordance with the late activation technique described above. Tracklet generator228may generate a current target tracklet260for the new object, as described above. The current target tracklet260for the new object may be depicted inFIG.5Das tracklet522.

Referring back toFIG.4, in some embodiments, processing units performing method400may determine whether a current state of the second object corresponds to at least one of the one or more future states of the first object. For example tracklet matching component230of tracklet manager module226may compare tracklet522to predicted target tracklets262for lost targets (e.g., stored in data store250). In some embodiments, tracklet manager module226may identify one or more predicted target tracklets262that were lost during a time period that is the same or similar to a time period that the new object was detected in image520and compare tracklet522to the identified tracklets262. In other or similar embodiments, tracklet manager module226may identify one or more predicted target tracklets262that were lost at a location that is within a threshold proximity of a location that the new object was detected in520and compare tracklet522to the identified tracklets262.

In some embodiments, tracklet matching component230may compare tracklet522and each of the identified tracklets262by performing a time series analysis (e.g., a dynamic time warping analysis) to tracklet522and each respective identified tracklet262. To perform the time series analysis, tracklet matching component230may compare each coordinate of tracklet522to one or more coordinates of a respective tracklet262and determine whether a coordinate of tracklet522corresponds to (i.e., matches or approximately matches) a coordinate of the respective tracklet262. In response to determining that a coordinate of tracklet522corresponds to a coordinate of the respective tracklet262, tracklet matching component230may determine that tracklet522corresponds to the respective tracklet262, and therefore that the new object corresponds to the lost target associated with the respective tracklet262.

In some embodiments, tracklet matching component230may determine that tracklet522corresponds to multiple tracklets262associated with lost targets (i.e., multiple lost targets may correspond to the new object). In such embodiments, tracklet matching component230may compare other attributes associated with the new object and the multiple lost targets to identify a single lost target that corresponds to the new object. For example, tracklet matching component230may determine a difference between a size of a most recent bounding box or a correlation response determined for a lost target and a size of a bounding box or a correlation response determined for the new object. Responsive to determining that the difference satisfies a difference criterion (i.e., the difference falls below a difference threshold value), tracklet matching component230may determine that the lost target corresponds to the new object. Tracklet matching component230may consider different attributes associated with the new object and the lost target (i.e., as determined before the target was lost) to identify a lost target that corresponds to the new object. Such attributes may include an angle or direction of the new object and/or the lost target, a speed of the new object and/or the lost target, an average intersection-over-union rating associated with a bounding box for the new object and a bounding box for the lost target, and so forth.

At block418, processing units performing method400may determine that the current state of the second object corresponds to at least one of the one or more future states of the first object. Tracklet matching component230may determine that a tracklet262associated with a lost target (i.e., the first object) corresponds to tracklet522associated with the new object (i.e., the second object), in accordance with previously described embodiments. Accordingly, tracklet matching component230may determine that the current location of the new object, as indicated by one or more coordinates of tracklet522, corresponds to a future state of the lost target, as indicated by one or more coordinates of tracklet262.

At block420, processing units performing method400may update an identifier associated with the second object to correspond to an identifier associated with the first object. Tracklet fusion component232may update a target identifier506B associated with the new object to correspond to the target identifier506A associated with target504. In accordance with the previous illustrative example, responsive to determining that tracklet522corresponds to tracklet518(i.e., associated with target504), tracklet fusion component232may update identifier “60” for the new object to correspond to identifier “1” for target504. Tracklet fusion component232may also “fuse” or “merge” tracklet522to tracklet518. For example, tracklet fusion component232may identify a location of the environment depicted in images502,514,516,520where target504was last detected (e.g., by object detection engine141and/or object localization module210) and determine one or more coordinates for the identified location. Tracklet fusion component232may also determine a coordinate of tracklet522that corresponds to a current state of the new object in image520(e.g., the last coordinate of tracklet522). In some embodiments, tracklet fusion component232may perform a linear interpolation based on the coordinates for the location where target504was last detected and the coordinate of tracklet522that corresponds the current state of the new object to determine a path taken by target504when target504was not detected. Tracklet fusion component232may provide the determined path to the object tracker218associated with target504and the object tracker218may provide data associated with the determined path to state estimation module220. State estimation module220may update a prior target state254, a current target state256, and/or a future target state258associated with target504based on the provided data. Object tracker218may continue to track target504, in accordance with embodiments described herein. In some embodiments, target manager module216may terminate the object tracker218that was instantiated to track the new object, in accordance with previously described embodiments.

FIG.6Aillustrates inference and/or training logic615used to perform inferencing and/or training operations associated with one or more embodiments. Details regarding inference and/or training logic615are provided below in conjunction withFIGS.6A and/or6B.

In at least one embodiment, inference and/or training logic615may include, without limitation, a code and/or data storage605to store backward and/or output weight and/or input/output data corresponding to neurons or layers of a neural network trained and/or used for inferencing in aspects of one or more embodiments. In at least one embodiment, code and/or data storage605stores weight parameters and/or input/output data of each layer of a neural network trained or used in conjunction with one or more embodiments during backward propagation of input/output data and/or weight parameters during training and/or inferencing using aspects of one or more embodiments. In at least one embodiment, training logic615may include, or be coupled to code and/or data storage605to store graph code or other software to control timing and/or order, in which weight and/or other parameter information is to be loaded to configure, logic, including integer and/or floating point units (collectively, arithmetic logic units (ALUs). In at least one embodiment, code, such as graph code, loads weight or other parameter information into processor ALUs based on an architecture of a neural network to which the code corresponds. In at least one embodiment, any portion of code and/or data storage605may be included with other on-chip or off-chip data storage, including a processor's L1, L2, or L3 cache or system memory. In at least one embodiment, any portion of code and/or data storage605may be internal or external to on one or more processors or other hardware logic devices or circuits. In at least one embodiment, code and/or data storage605may be cache memory, DRAM, SRAM, non-volatile memory (e.g., Flash memory), or other storage. In at least one embodiment, choice of whether code and/or data storage605is internal or external to a processor, for example, or comprised of DRAM, SRAM, Flash or some other storage type may depend on available storage on-chip versus off-chip, latency requirements of training and/or inferencing functions being performed, batch size of data used in inferencing and/or training of a neural network, or some combination of these factors.

In at least one embodiment, code and/or data storage601and code and/or data storage605may be separate storage structures. In at least one embodiment, code and/or data storage601and code and/or data storage605may be same storage structure. In at least one embodiment, code and/or data storage601and code and/or data storage605may be partially same storage structure and partially separate storage structures. In at least one embodiment, any portion of code and/or data storage601and code and/or data storage605may be included with other on-chip or off-chip data storage, including a processor's L1, L2, or L3 cache or system memory.

In at least one embodiment, inference and/or training logic615may include, without limitation, one or more arithmetic logic unit(s) (“ALU(s)”)610, including integer and/or floating point units, to perform logical and/or mathematical operations based, at least in part on, or indicated by, training and/or inference code (e.g., graph code), a result of which may produce activations (e.g., output values from layers or neurons within a neural network) stored in an activation storage620that are functions of input/output and/or weight parameter data stored in code and/or data storage601and/or code and/or data storage605. In at least one embodiment, activations stored in activation storage620are generated according to linear algebraic and or matrix-based mathematics performed by ALU(s)610in response to performing instructions or other code, wherein weight values stored in code and/or data storage605and/or code and/or data storage601are used as operands along with other values, such as bias values, gradient information, momentum values, or other parameters or hyperparameters, any or all of which may be stored in code and/or data storage605or code and/or data storage601or another storage on or off-chip.

In at least one embodiment, ALU(s)610are included within one or more processors or other hardware logic devices or circuits, whereas in another embodiment, ALU(s)610may be external to a processor or other hardware logic device or circuit that uses them (e.g., a co-processor). In at least one embodiment, ALUs610may be included within a processor's execution units or otherwise within a bank of ALUs accessible by a processor's execution units either within same processor or distributed between different processors of different types (e.g., central processing units, graphics processing units, fixed function units, etc.). In at least one embodiment, code and/or data storage601, code and/or data storage605, and activation storage620may be on same processor or other hardware logic device or circuit, whereas in another embodiment, they may be in different processors or other hardware logic devices or circuits, or some combination of same and different processors or other hardware logic devices or circuits. In at least one embodiment, any portion of activation storage620may be included with other on-chip or off-chip data storage, including a processor's L1, L2, or L3 cache or system memory. Furthermore, inferencing and/or training code may be stored with other code accessible to a processor or other hardware logic or circuit and fetched and/or processed using a processor's fetch, decode, scheduling, execution, retirement and/or other logical circuits.

In at least one embodiment, activation storage620may be cache memory, DRAM, SRAM, non-volatile memory (e.g., Flash memory), or other storage. In at least one embodiment, activation storage620may be completely or partially within or external to one or more processors or other logical circuits. In at least one embodiment, choice of whether activation storage620is internal or external to a processor, for example, or comprised of DRAM, SRAM, Flash or some other storage type may depend on available storage on-chip versus off-chip, latency requirements of training and/or inferencing functions being performed, batch size of data used in inferencing and/or training of a neural network, or some combination of these factors. In at least one embodiment, inference and/or training logic615illustrated inFIG.6Amay be used in conjunction with an application-specific integrated circuit (“ASIC”), such as Tensorflow® Processing Unit from Google, an inference processing unit (IPU) from Graphcore™, or a Nervana® (e.g., “Lake Crest”) processor from Intel Corp. In at least one embodiment, inference and/or training logic615illustrated inFIG.6Amay be used in conjunction with central processing unit (“CPU”) hardware, graphics processing unit (“GPU”) hardware or other hardware, such as data processing unit (“DPU”) hardware, or field programmable gate arrays (“FPGAs”).

FIG.6Billustrates inference and/or training logic615, according to at least one or more embodiments. In at least one embodiment, inference and/or training logic615may include, without limitation, hardware logic in which computational resources are dedicated or otherwise exclusively used in conjunction with weight values or other information corresponding to one or more layers of neurons within a neural network. In at least one embodiment, inference and/or training logic615illustrated inFIG.6Bmay be used in conjunction with an application-specific integrated circuit (ASIC), such as Tensorflow® Processing Unit from Google, an inference processing unit (IPU) from Graphcore™, or a Nervana® (e.g., “Lake Crest”) processor from Intel Corp. In at least one embodiment, inference and/or training logic615illustrated inFIG.6Bmay be used in conjunction with central processing unit (CPU) hardware, graphics processing unit (GPU) hardware or other hardware, such as data processing unit (“DPU”) hardware, or field programmable gate arrays (FPGAs). In at least one embodiment, inference and/or training logic615includes, without limitation, code and/or data storage601and code and/or data storage605, which may be used to store code (e.g., graph code), weight values and/or other information, including bias values, gradient information, momentum values, and/or other parameter or hyperparameter information. In at least one embodiment illustrated inFIG.6B, each of code and/or data storage601and code and/or data storage605is associated with a dedicated computational resource, such as computational hardware602and computational hardware606, respectively. In at least one embodiment, each of computational hardware602and computational hardware606comprises one or more ALUs that perform mathematical functions, such as linear algebraic functions, only on information stored in code and/or data storage601and code and/or data storage605, respectively, result of which is stored in activation storage620.

In at least one embodiment, each of code and/or data storage601and605and corresponding computational hardware602and606, respectively, correspond to different layers of a neural network, such that resulting activation from one “storage/computational pair601/602” of code and/or data storage601and computational hardware602is provided as an input to “storage/computational pair605/606” of code and/or data storage605and computational hardware606, in order to mirror conceptual organization of a neural network. In at least one embodiment, each of storage/computational pairs601/602and605/606may correspond to more than one neural network layer. In at least one embodiment, additional storage/computation pairs (not shown) subsequent to or in parallel with storage computation pairs601/602and605/606may be included in inference and/or training logic615.

Data Center

FIG.7illustrates an example data center700, in which at least one embodiment may be used. In at least one embodiment, data center700includes a data center infrastructure layer710, a framework layer720, a software layer730, and an application layer740.

In at least one embodiment, as shown inFIG.7, data center infrastructure layer710may include a resource orchestrator712, grouped computing resources714, and node computing resources (“node C.R.s”) 716(1)-716(N), where “N” represents any whole, positive integer. In at least one embodiment, node C.R.s 716(1)-716(N) may include, but are not limited to, any number of central processing units (“CPUs”) or other processors (including accelerators, field programmable gate arrays (FPGAs), data processing units, graphics processors, etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid state or disk drives), network input/output (“NW I/O”) devices, network switches, virtual machines (“VMs”), power modules, and cooling modules, etc. In at least one embodiment, one or more node C.R.s from among node C.R.s 716(1)-716(N) may be a server having one or more of above-mentioned computing resources.

In at least one embodiment, resource orchestrator712may configure or otherwise control one or more node C.R.s 716(1)-716(N) and/or grouped computing resources714. In at least one embodiment, resource orchestrator712may include a software design infrastructure (“SDI”) management entity for data center700. In at least one embodiment, resource orchestrator may include hardware, software or some combination thereof.

In at least one embodiment, as shown inFIG.7, framework layer720includes a job scheduler722, a configuration manager724, a resource manager726and a distributed file system728. In at least one embodiment, framework layer720may include a framework to support software732of software layer730and/or one or more application(s)742of application layer740. In at least one embodiment, software732or application(s)742may respectively include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud and Microsoft Azure. In at least one embodiment, framework layer720may be, but is not limited to, a type of free and open-source software web application framework such as Apache Spark™ (hereinafter “Spark”) that may utilize distributed file system728for large-scale data processing (e.g., “big data”). In at least one embodiment, job scheduler722may include a Spark driver to facilitate scheduling of workloads supported by various layers of data center700. In at least one embodiment, configuration manager724may be capable of configuring different layers such as software layer730and framework layer720including Spark and distributed file system728for supporting large-scale data processing. In at least one embodiment, resource manager726may be capable of managing clustered or grouped computing resources mapped to or allocated for support of distributed file system728and job scheduler722. In at least one embodiment, clustered or grouped computing resources may include grouped computing resource714at data center infrastructure layer710. In at least one embodiment, resource manager726may coordinate with resource orchestrator712to manage these mapped or allocated computing resources.

In at least one embodiment, software732included in software layer730may include software used by at least portions of node C.R.s 716(1)-716(N), grouped computing resources714, and/or distributed file system728of framework layer720. The one or more types of software may include, but are not limited to, Internet web page search software, e-mail virus scan software, database software, and streaming video content software.

In at least one embodiment, application(s)742included in application layer740may include one or more types of applications used by at least portions of node C.R.s 716(1)-716(N), grouped computing resources714, and/or distributed file system728of framework layer720. One or more types of applications may include, but are not limited to, any number of a genomics application, a cognitive compute, and a machine learning application, including training or inferencing software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.) or other machine learning applications used in conjunction with one or more embodiments.

In at least one embodiment, any of configuration manager724, resource manager726, and resource orchestrator712may implement any number and type of self-modifying actions based on any amount and type of data acquired in any technically feasible fashion. In at least one embodiment, self-modifying actions may relieve a data center operator of data center700from making possibly bad configuration decisions and possibly avoiding underutilized and/or poor performing portions of a data center.

Such components may be used to generate synthetic data imitating failure cases in a network training process, which may help to improve performance of the network while limiting the amount of synthetic data to avoid overfitting.

Computer Systems

In at least one embodiment, computer system800may include, without limitation, processor802that may include, without limitation, one or more execution units808to perform machine learning model training and/or inferencing according to techniques described herein. In at least one embodiment, computer system800is a single processor desktop or server system, but in another embodiment computer system800may be a multiprocessor system. In at least one embodiment, processor802may include, without limitation, a complex instruction set computer (“CISC”) microprocessor, a reduced instruction set computing (“RISC”) microprocessor, a very long instruction word (“VLIW”) microprocessor, a processor implementing a combination of instruction sets, or any other processor device, such as a digital signal processor, for example. In at least one embodiment, processor802may be coupled to a processor bus810that may transmit data signals between processor802and other components in computer system800.

In at least one embodiment, processor802may include, without limitation, a Level 1 (“L1”) internal cache memory (“cache”)804. In at least one embodiment, processor802may have a single internal cache or multiple levels of internal cache. In at least one embodiment, cache memory may reside external to processor802. Other embodiments may also include a combination of both internal and external caches depending on particular implementation and needs. In at least one embodiment, register file806may store different types of data in various registers including, without limitation, integer registers, floating point registers, status registers, and instruction pointer register.

In at least one embodiment, execution unit808, including, without limitation, logic to perform integer and floating point operations, also resides in processor802. In at least one embodiment, processor802may also include a microcode (“ucode”) read only memory (“ROM”) that stores microcode for certain macro instructions. In at least one embodiment, execution unit808may include logic to handle a packed instruction set809. In at least one embodiment, by including packed instruction set809in an instruction set of a general-purpose processor802, along with associated circuitry to execute instructions, operations used by many multimedia applications may be performed using packed data in a general-purpose processor802. In one or more embodiments, many multimedia applications may be accelerated and executed more efficiently by using full width of a processor's data bus for performing operations on packed data, which may eliminate need to transfer smaller units of data across processor's data bus to perform one or more operations one data element at a time.

In at least one embodiment, execution unit808may also be used in microcontrollers, embedded processors, graphics devices, DSPs, and other types of logic circuits. In at least one embodiment, computer system800may include, without limitation, a memory820. In at least one embodiment, memory820may be implemented as a Dynamic Random Access Memory (“DRAM”) device, a Static Random Access Memory (“SRAM”) device, flash memory device, or other memory device. In at least one embodiment, memory820may store instruction(s)819and/or data821represented by data signals that may be executed by processor802.

In at least one embodiment, system logic chip may be coupled to processor bus810and memory820. In at least one embodiment, system logic chip may include, without limitation, a memory controller hub (“MCH”)816, and processor802may communicate with MCH816via processor bus810. In at least one embodiment, MCH816may provide a high bandwidth memory path818to memory820for instruction and data storage and for storage of graphics commands, data and textures. In at least one embodiment, MCH816may direct data signals between processor802, memory820, and other components in computer system800and to bridge data signals between processor bus810, memory820, and a system I/O822. In at least one embodiment, system logic chip may provide a graphics port for coupling to a graphics controller. In at least one embodiment, MCH816may be coupled to memory820through a high bandwidth memory path818and graphics/video card812may be coupled to MCH816through an Accelerated Graphics Port (“AGP”) interconnect814.

In at least one embodiment, computer system800may use system I/O822that is a proprietary hub interface bus to couple MCH816to I/O controller hub (“ICH”)830. In at least one embodiment, ICH830may provide direct connections to some I/O devices via a local I/O bus. In at least one embodiment, local I/O bus may include, without limitation, a high-speed I/O bus for connecting peripherals to memory820, chipset, and processor802. Examples may include, without limitation, an audio controller829, a firmware hub (“flash BIOS”)828, a wireless transceiver826, a data storage824, a legacy I/O controller823containing user input and keyboard interfaces825, a serial expansion port827, such as Universal Serial Bus (“USB”), and a network controller834, which may include in some embodiments, a data processing unit. Data storage824may comprise a hard disk drive, a floppy disk drive, a CD-ROM device, a flash memory device, or other mass storage device.

In at least one embodiment,FIG.8illustrates a system, which includes interconnected hardware devices or “chips,” whereas in other embodiments,FIG.8may illustrate an exemplary System on a Chip (“SoC”). In at least one embodiment, devices may be interconnected with proprietary interconnects, standardized interconnects (e.g., PCIe) or some combination thereof. In at least one embodiment, one or more components of computer system800are interconnected using compute express link (CXL) interconnects.

Such components may be used to generate synthetic data imitating failure cases in a network training process, which may help to improve performance of the network while limiting the amount of synthetic data to avoid overfitting.

FIG.9is a block diagram illustrating an electronic device900for utilizing a processor910, according to at least one embodiment. In at least one embodiment, electronic device900may be, for example and without limitation, a notebook, a tower server, a rack server, a blade server, a laptop, a desktop, a tablet, a mobile device, a phone, an embedded computer, an edge device, an IoT device, or any other suitable electronic device.

In at least one embodiment, system900may include, without limitation, processor910communicatively coupled to any suitable number or kind of components, peripherals, modules, or devices. In at least one embodiment, processor910coupled using a bus or interface, such as a 1° ° C. bus, a System Management Bus (“SMBus”), a Low Pin Count (LPC) bus, a Serial Peripheral Interface (“SPI”), a High Definition Audio (“HDA”) bus, a Serial Advance Technology Attachment (“SATA”) bus, a Universal Serial Bus (“USB”) (versions1,2,3), or a Universal Asynchronous Receiver/Transmitter (“UART”) bus. In at least one embodiment,FIG.9illustrates a system, which includes interconnected hardware devices or “chips,” whereas in other embodiments,FIG.9may illustrate an exemplary System on a Chip (“SoC”). In at least one embodiment, devices illustrated inFIG.9may be interconnected with proprietary interconnects, standardized interconnects (e.g., PCIe) or some combination thereof. In at least one embodiment, one or more components ofFIG.9are interconnected using compute express link (CXL) interconnects.

In at least one embodiment,FIG.9may include a display924, a touch screen925, a touch pad930, a Near Field Communications unit (“NFC”)945, a sensor hub940, a thermal sensor946, an Express Chipset (“EC”)935, a Trusted Platform Module (“TPM”)938, BIOS/firmware/flash memory (“BIOS, FW Flash”)922, a DSP960, a drive920such as a Solid State Disk (“SSD”) or a Hard Disk Drive (“HDD”), a wireless local area network unit (“WLAN”)950, a Bluetooth unit952, a Wireless Wide Area Network unit (“WWAN”)956, a Global Positioning System (GPS)955, a camera (“USB 3.0 camera”)954such as a USB 3.0 camera, and/or a Low Power Double Data Rate (“LPDDR”) memory unit (“LPDDR3”)915implemented in, for example, LPDDR3 standard. These components may each be implemented in any suitable manner.

In at least one embodiment, other components may be communicatively coupled to processor910through components discussed above. In at least one embodiment, an accelerometer941, Ambient Light Sensor (“ALS”)942, compass943, and a gyroscope944may be communicatively coupled to sensor hub940. In at least one embodiment, thermal sensor939, a fan937, a keyboard936, and a touch pad930may be communicatively coupled to EC935. In at least one embodiment, speaker963, headphones964, and microphone (“mic”)965may be communicatively coupled to an audio unit (“audio codec and class d amp”)962, which may in turn be communicatively coupled to DSP960. In at least one embodiment, audio unit964may include, for example and without limitation, an audio coder/decoder (“codec”) and a class D amplifier. In at least one embodiment, SIM card (“SIM”)957may be communicatively coupled to WWAN unit956. In at least one embodiment, components such as WLAN unit950and Bluetooth unit952, as well as WWAN unit956may be implemented in a Next Generation Form Factor (“NGFF”).

Such components may be used to generate synthetic data imitating failure cases in a network training process, which may help to improve performance of the network while limiting the amount of synthetic data to avoid overfitting.

FIG.10is a block diagram of a processing system, according to at least one embodiment. In at least one embodiment, system1000includes one or more processors1002and one or more graphics processors1008, and may be a single processor desktop system, a multiprocessor workstation system, or a server system having a large number of processors1002or processor cores1007. In at least one embodiment, system1000is a processing platform incorporated within a system-on-a-chip (SoC) integrated circuit for use in mobile, handheld, edge, or embedded devices.

In at least one embodiment, system1000may include, or be incorporated within a server-based gaming platform, a game console, including a game and media console, a mobile gaming console, a handheld game console, or an online game console. In at least one embodiment, system1000is a mobile phone, smart phone, tablet computing device or mobile Internet device. In at least one embodiment, processing system1000may also include, couple with, or be integrated within a wearable device, such as a smart watch wearable device, smart eyewear device, augmented reality device, or virtual reality device. In at least one embodiment, processing system1000is a television or set top box device having one or more processors1002and a graphical interface generated by one or more graphics processors1008.

In at least one embodiment, one or more processors1002each include one or more processor cores1007to process instructions which, when executed, perform operations for system and user software. In at least one embodiment, each of one or more processor cores1007is configured to process a specific instruction set1009. In at least one embodiment, instruction set1009may facilitate Complex Instruction Set Computing (CISC), Reduced Instruction Set Computing (RISC), or computing via a Very Long Instruction Word (VLIW). In at least one embodiment, processor cores1007may each process a different instruction set1009, which may include instructions to facilitate emulation of other instruction sets. In at least one embodiment, processor core1007may also include other processing devices, such a Digital Signal Processor (DSP).

In at least one embodiment, processor1002includes cache memory1004. In at least one embodiment, processor1002may have a single internal cache or multiple levels of internal cache. In at least one embodiment, cache memory is shared among various components of processor1002. In at least one embodiment, processor1002also uses an external cache (e.g., a Level-3 (L3) cache or Last Level Cache (LLC)) (not shown), which may be shared among processor cores1007using known cache coherency techniques. In at least one embodiment, register file1006is additionally included in processor1002which may include different types of registers for storing different types of data (e.g., integer registers, floating point registers, status registers, and an instruction pointer register). In at least one embodiment, register file1006may include general-purpose registers or other registers.

In at least one embodiment, one or more processor(s)1002are coupled with one or more interface bus(es)1010to transmit communication signals such as address, data, or control signals between processor1002and other components in system1000. In at least one embodiment, interface bus1010, in one embodiment, may be a processor bus, such as a version of a Direct Media Interface (DMI) bus. In at least one embodiment, interface1010is not limited to a DMI bus, and may include one or more Peripheral Component Interconnect buses (e.g., PCI, PCI Express), memory busses, or other types of interface busses. In at least one embodiment processor(s)1002include an integrated memory controller1016and a platform controller hub1030. In at least one embodiment, memory controller1016facilitates communication between a memory device and other components of system1000, while platform controller hub (PCH)1030provides connections to I/O devices via a local I/O bus.

In at least one embodiment, memory device1020may be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, flash memory device, phase-change memory device, or some other memory device having suitable performance to serve as process memory. In at least one embodiment memory device1020may operate as system memory for system1000, to store data1022and instructions1021for use when one or more processors1002executes an application or process. In at least one embodiment, memory controller1016also couples with an optional external graphics processor1012, which may communicate with one or more graphics processors1008in processors1002to perform graphics and media operations. In at least one embodiment, a display device1011may connect to processor(s)1002. In at least one embodiment display device1011may include one or more of an internal display device, as in a mobile electronic device or a laptop device or an external display device attached via a display interface (e.g., DisplayPort, etc.). In at least one embodiment, display device1011may include a head mounted display (HMD) such as a stereoscopic display device for use in virtual reality (VR) applications or augmented reality (AR) applications.

In at least one embodiment, platform controller hub1030enables peripherals to connect to memory device1020and processor1002via a high-speed I/O bus. In at least one embodiment, I/O peripherals include, but are not limited to, an audio controller1046, a network controller1034, a firmware interface1028, a wireless transceiver1026, touch sensors1025, a data storage device1024(e.g., hard disk drive, flash memory, etc.). In at least one embodiment, data storage device1024may connect via a storage interface (e.g., SATA) or via a peripheral bus, such as a Peripheral Component Interconnect bus (e.g., PCI, PCI Express). In at least one embodiment, touch sensors1025may include touch screen sensors, pressure sensors, or fingerprint sensors. In at least one embodiment, wireless transceiver1026may be a Wi-Fi transceiver, a Bluetooth transceiver, or a mobile network transceiver such as a 3G, 4G, or Long Term Evolution (LTE) transceiver. In at least one embodiment, firmware interface1028enables communication with system firmware, and may be, for example, a unified extensible firmware interface (UEFI). In at least one embodiment, network controller1034may enable a network connection to a wired network. In at least one embodiment, a high-performance network controller (not shown) couples with interface bus1010. In at least one embodiment, audio controller1046is a multi-channel high definition audio controller. In at least one embodiment, system1000includes an optional legacy I/O controller1040for coupling legacy (e.g., Personal System 2 (PS/2)) devices to system. In at least one embodiment, platform controller hub1030may also connect to one or more Universal Serial Bus (USB) controllers1042connect input devices, such as keyboard and mouse1043combinations, a camera1044, or other USB input devices.

In at least one embodiment, an instance of memory controller1016and platform controller hub1030may be integrated into a discreet external graphics processor, such as external graphics processor1012. In at least one embodiment, platform controller hub1030and/or memory controller1016may be external to one or more processor(s)1002. For example, in at least one embodiment, system1000may include an external memory controller1016and platform controller hub1030, which may be configured as a memory controller hub and peripheral controller hub within a system chipset that is in communication with processor(s)1002.

Inference and/or training logic615are used to perform inferencing and/or training operations associated with one or more embodiments. Details regarding inference and/or training logic615are provided below in conjunction withFIGS.6A and/or6B. In at least one embodiment portions or all of inference and/or training logic615may be incorporated into graphics processor1100. For example, in at least one embodiment, training and/or inferencing techniques described herein may use one or more of ALUs embodied in a graphics processor. Moreover, in at least one embodiment, inferencing and/or training operations described herein may be done using logic other than logic illustrated inFIGS.6A or6B. In at least one embodiment, weight parameters may be stored in on-chip or off-chip memory and/or registers (shown or not shown) that configure ALUs of a graphics processor to perform one or more machine learning algorithms, neural network architectures, use cases, or training techniques described herein.

Such components may be used to generate synthetic data imitating failure cases in a network training process, which may help to improve performance of the network while limiting the amount of synthetic data to avoid overfitting.

FIG.11is a block diagram of a processor1100having one or more processor cores1102A-1102N, an integrated memory controller1114, and an integrated graphics processor1108, according to at least one embodiment. In at least one embodiment, processor1100may include additional cores up to and including additional core1102N represented by dashed lined boxes. In at least one embodiment, each of processor cores1102A-1102N includes one or more internal cache units1104A-1104N. In at least one embodiment, each processor core also has access to one or more shared cached units1106.

In at least one embodiment, internal cache units1104A-1104N and shared cache units1106represent a cache memory hierarchy within processor1100. In at least one embodiment, cache memory units1104A-1104N may include at least one level of instruction and data cache within each processor core and one or more levels of shared mid-level cache, such as a Level 2 (L2), Level 3 (L3), Level 4 (L4), or other levels of cache, where a highest level of cache before external memory is classified as an LLC. In at least one embodiment, cache coherency logic maintains coherency between various cache units1106and1104A-1104N.

In at least one embodiment, processor1100may also include a set of one or more bus controller units1116and a system agent core1110. In at least one embodiment, one or more bus controller units1116manage a set of peripheral buses, such as one or more PCI or PCI express busses. In at least one embodiment, system agent core1110provides management functionality for various processor components. In at least one embodiment, system agent core1110includes one or more integrated memory controllers1114to manage access to various external memory devices (not shown).

In at least one embodiment, one or more of processor cores1102A-1102N include support for simultaneous multi-threading. In at least one embodiment, system agent core1110includes components for coordinating and operating cores1102A-1102N during multi-threaded processing. In at least one embodiment, system agent core1110may additionally include a power control unit (PCU), which includes logic and components to regulate one or more power states of processor cores1102A-1102N and graphics processor1108.

In at least one embodiment, processor1100additionally includes graphics processor1108to execute graphics processing operations. In at least one embodiment, graphics processor1108couples with shared cache units1106, and system agent core1110, including one or more integrated memory controllers1114. In at least one embodiment, system agent core1110also includes a display controller1111to drive graphics processor output to one or more coupled displays. In at least one embodiment, display controller1111may also be a separate module coupled with graphics processor1108via at least one interconnect, or may be integrated within graphics processor1108.

In at least one embodiment, a ring based interconnect unit1112is used to couple internal components of processor1100. In at least one embodiment, an alternative interconnect unit may be used, such as a point-to-point interconnect, a switched interconnect, or other techniques. In at least one embodiment, graphics processor1108couples with ring interconnect1112via an I/O link1113.

In at least one embodiment, I/O link1113represents at least one of multiple varieties of I/O interconnects, including an on package I/O interconnect which facilitates communication between various processor components and a high-performance embedded memory module1118, such as an eDRAM module. In at least one embodiment, each of processor cores1102A-1102N and graphics processor1108use embedded memory modules1118as a shared Last Level Cache.

In at least one embodiment, processor cores1102A-1102N are homogenous cores executing a common instruction set architecture. In at least one embodiment, processor cores1102A-1102N are heterogeneous in terms of instruction set architecture (ISA), where one or more of processor cores1102A-1102N execute a common instruction set, while one or more other cores of processor cores1102A-1102N executes a subset of a common instruction set or a different instruction set. In at least one embodiment, processor cores1102A-1102N are heterogeneous in terms of microarchitecture, where one or more cores having a relatively higher power consumption couple with one or more power cores having a lower power consumption. In at least one embodiment, processor1100may be implemented on one or more chips or as a SoC integrated circuit.

Inference and/or training logic615are used to perform inferencing and/or training operations associated with one or more embodiments. Details regarding inference and/or training logic615are provided below in conjunction withFIGS.6A and/or6B. In at least one embodiment portions or all of inference and/or training logic615may be incorporated into processor1100. For example, in at least one embodiment, training and/or inferencing techniques described herein may use one or more of ALUs embodied in graphics processor1108, graphics core(s)1102A-1102N, or other components inFIG.11. Moreover, in at least one embodiment, inferencing and/or training operations described herein may be done using logic other than logic illustrated inFIGS.6A or6B. In at least one embodiment, weight parameters may be stored in on-chip or off-chip memory and/or registers (shown or not shown) that configure ALUs of graphics processor1100to perform one or more machine learning algorithms, neural network architectures, use cases, or training techniques described herein.

Such components may be used to generate synthetic data imitating failure cases in a network training process, which may help to improve performance of the network while limiting the amount of synthetic data to avoid overfitting.

Virtualized Computing Platform

FIG.12is an example data flow diagram for a process1200of generating and deploying an image processing and inferencing pipeline, in accordance with at least one embodiment. In at least one embodiment, process1200may be deployed for use with imaging devices, processing devices, and/or other device types at one or more facilities1202. Process1200may be executed within a training system1204and/or a deployment system1206. In at least one embodiment, training system1204may be used to perform training, deployment, and implementation of machine learning models (e.g., neural networks, object detection algorithms, computer vision algorithms, etc.) for use in deployment system1206. In at least one embodiment, deployment system1206may be configured to offload processing and compute resources among a distributed computing environment to reduce infrastructure requirements at facility1202. In at least one embodiment, one or more applications in a pipeline may use or call upon services (e.g., inference, visualization, compute, AI, etc.) of deployment system1206during execution of applications.

In at least one embodiment, some of applications used in advanced processing and inferencing pipelines may use machine learning models or other AI to perform one or more processing steps. In at least one embodiment, machine learning models may be trained at facility1202using data1208(such as imaging data) generated at facility1202(and stored on one or more picture archiving and communication system (PACS) servers at facility1202), may be trained using imaging or sequencing data1208from another facility(ies), or a combination thereof. In at least one embodiment, training system1204may be used to provide applications, services, and/or other resources for generating working, deployable machine learning models for deployment system1206.

In at least one embodiment, model registry1224may be backed by object storage that may support versioning and object metadata. In at least one embodiment, object storage may be accessible through, for example, a cloud storage (e.g., cloud1326ofFIG.13) compatible application programming interface (API) from within a cloud platform. In at least one embodiment, machine learning models within model registry1224may uploaded, listed, modified, or deleted by developers or partners of a system interacting with an API. In at least one embodiment, an API may provide access to methods that allow users with appropriate credentials to associate models with applications, such that models may be executed as part of execution of containerized instantiations of applications.

In at least one embodiment, training pipeline1304(FIG.13) may include a scenario where facility1202is training their own machine learning model, or has an existing machine learning model that needs to be optimized or updated. In at least one embodiment, imaging data1208generated by imaging device(s), sequencing devices, and/or other device types may be received. In at least one embodiment, once imaging data1208is received, AI-assisted annotation1210may be used to aid in generating annotations corresponding to imaging data1208to be used as ground truth data for a machine learning model. In at least one embodiment, AI-assisted annotation1210may include one or more machine learning models (e.g., convolutional neural networks (CNNs)) that may be trained to generate annotations corresponding to certain types of imaging data1208(e.g., from certain devices). In at least one embodiment, AI-assisted annotations1210may then be used directly, or may be adjusted or fine-tuned using an annotation tool to generate ground truth data. In at least one embodiment, AI-assisted annotations1210, labeled clinic data1212, or a combination thereof may be used as ground truth data for training a machine learning model. In at least one embodiment, a trained machine learning model may be referred to as output model1216, and may be used by deployment system1206, as described herein.

In at least one embodiment, training pipeline1304(FIG.13) may include a scenario where facility1202needs a machine learning model for use in performing one or more processing tasks for one or more applications in deployment system1206, but facility1202may not currently have such a machine learning model (or may not have a model that is optimized, efficient, or effective for such purposes). In at least one embodiment, an existing machine learning model may be selected from a model registry1224. In at least one embodiment, model registry1224may include machine learning models trained to perform a variety of different inference tasks on imaging data. In at least one embodiment, machine learning models in model registry1224may have been trained on imaging data from different facilities than facility1202(e.g., facilities remotely located). In at least one embodiment, machine learning models may have been trained on imaging data from one location, two locations, or any number of locations. In at least one embodiment, when being trained on imaging data from a specific location, training may take place at that location, or at least in a manner that protects confidentiality of imaging data or restricts imaging data from being transferred off-premises. In at least one embodiment, once a model is trained—or partially trained—at one location, a machine learning model may be added to model registry1224. In at least one embodiment, a machine learning model may then be retrained, or updated, at any number of other facilities, and a retrained or updated model may be made available in model registry1224. In at least one embodiment, a machine learning model may then be selected from model registry1224—and referred to as output model1216—and may be used in deployment system1206to perform one or more processing tasks for one or more applications of a deployment system.

In at least one embodiment, training pipeline1304(FIG.13), a scenario may include facility1202requiring a machine learning model for use in performing one or more processing tasks for one or more applications in deployment system1206, but facility1202may not currently have such a machine learning model (or may not have a model that is optimized, efficient, or effective for such purposes). In at least one embodiment, a machine learning model selected from model registry1224may not be fine-tuned or optimized for imaging data1208generated at facility1202because of differences in populations, robustness of training data used to train a machine learning model, diversity in anomalies of training data, and/or other issues with training data. In at least one embodiment, AI-assisted annotation1210may be used to aid in generating annotations corresponding to imaging data1208to be used as ground truth data for retraining or updating a machine learning model. In at least one embodiment, labeled data1212may be used as ground truth data for training a machine learning model. In at least one embodiment, retraining or updating a machine learning model may be referred to as model training1214. In at least one embodiment, model training1214—e.g., AI-assisted annotations1210, labeled clinic data1212, or a combination thereof—may be used as ground truth data for retraining or updating a machine learning model. In at least one embodiment, a trained machine learning model may be referred to as output model1216, and may be used by deployment system1206, as described herein.

In at least one embodiment, deployment system1206may include software1218, services1220, hardware1222, and/or other components, features, and functionality. In at least one embodiment, deployment system1206may include a software “stack,” such that software1218may be built on top of services1220and may use services1220to perform some or all of processing tasks, and services1220and software1218may be built on top of hardware1222and use hardware1222to execute processing, storage, and/or other compute tasks of deployment system1206. In at least one embodiment, software1218may include any number of different containers, where each container may execute an instantiation of an application. In at least one embodiment, each application may perform one or more processing tasks in an advanced processing and inferencing pipeline (e.g., inferencing, object detection, feature detection, segmentation, image enhancement, calibration, etc.). In at least one embodiment, an advanced processing and inferencing pipeline may be defined based on selections of different containers that are desired or required for processing imaging data1208, in addition to containers that receive and configure imaging data for use by each container and/or for use by facility1202after processing through a pipeline (e.g., to convert outputs back to a usable data type). In at least one embodiment, a combination of containers within software1218(e.g., that make up a pipeline) may be referred to as a virtual instrument (as described in more detail herein), and a virtual instrument may leverage services1220and hardware1222to execute some or all processing tasks of applications instantiated in containers.

In at least one embodiment, a data processing pipeline may receive input data (e.g., imaging data1208) in a specific format in response to an inference request (e.g., a request from a user of deployment system1206). In at least one embodiment, input data may be representative of one or more images, video, and/or other data representations generated by one or more imaging devices. In at least one embodiment, data may undergo pre-processing as part of data processing pipeline to prepare data for processing by one or more applications. In at least one embodiment, post-processing may be performed on an output of one or more inferencing tasks or other processing tasks of a pipeline to prepare an output data for a next application and/or to prepare output data for transmission and/or use by a user (e.g., as a response to an inference request). In at least one embodiment, inferencing tasks may be performed by one or more machine learning models, such as trained or deployed neural networks, which may include output models1216of training system1204.

In at least one embodiment, tasks of data processing pipeline may be encapsulated in a container(s) that each represents a discrete, fully functional instantiation of an application and virtualized computing environment that is able to reference machine learning models. In at least one embodiment, containers or applications may be published into a private (e.g., limited access) area of a container registry (described in more detail herein), and trained or deployed models may be stored in model registry1224and associated with one or more applications. In at least one embodiment, images of applications (e.g., container images) may be available in a container registry, and once selected by a user from a container registry for deployment in a pipeline, an image may be used to generate a container for an instantiation of an application for use by a user's system.

In at least one embodiment, developers (e.g., software developers, clinicians, doctors, etc.) may develop, publish, and store applications (e.g., as containers) for performing image processing and/or inferencing on supplied data. In at least one embodiment, development, publishing, and/or storing may be performed using a software development kit (SDK) associated with a system (e.g., to ensure that an application and/or container developed is compliant with or compatible with a system). In at least one embodiment, an application that is developed may be tested locally (e.g., at a first facility, on data from a first facility) with an SDK which may support at least some of services1220as a system (e.g., system1300ofFIG.13). In at least one embodiment, because DICOM objects may contain anywhere from one to hundreds of images or other data types, and due to a variation in data, a developer may be responsible for managing (e.g., setting constructs for, building pre-processing into an application, etc.) extraction and preparation of incoming data. In at least one embodiment, once validated by system1300(e.g., for accuracy), an application may be available in a container registry for selection and/or implementation by a user to perform one or more processing tasks with respect to data at a facility (e.g., a second facility) of a user.

In at least one embodiment, developers may then share applications or containers through a network for access and use by users of a system (e.g., system1300ofFIG.13). In at least one embodiment, completed and validated applications or containers may be stored in a container registry and associated machine learning models may be stored in model registry1224. In at least one embodiment, a requesting entity—who provides an inference or image processing request—may browse a container registry and/or model registry1224for an application, container, dataset, machine learning model, etc., select a desired combination of elements for inclusion in data processing pipeline, and submit an imaging processing request. In at least one embodiment, a request may include input data (and associated patient data, in some examples) that is necessary to perform a request, and/or may include a selection of application(s) and/or machine learning models to be executed in processing a request. In at least one embodiment, a request may then be passed to one or more components of deployment system1206(e.g., a cloud) to perform processing of data processing pipeline. In at least one embodiment, processing by deployment system1206may include referencing selected elements (e.g., applications, containers, models, etc.) from a container registry and/or model registry1224. In at least one embodiment, once results are generated by a pipeline, results may be returned to a user for reference (e.g., for viewing in a viewing application suite executing on a local, on-premises workstation or terminal).

In at least one embodiment, to aid in processing or execution of applications or containers in pipelines, services1220may be leveraged. In at least one embodiment, services1220may include compute services, artificial intelligence (AI) services, visualization services, and/or other service types. In at least one embodiment, services1220may provide functionality that is common to one or more applications in software1218, so functionality may be abstracted to a service that may be called upon or leveraged by applications. In at least one embodiment, functionality provided by services1220may run dynamically and more efficiently, while also scaling well by allowing applications to process data in parallel (e.g., using a parallel computing platform1330(FIG.13)). In at least one embodiment, rather than each application that shares a same functionality offered by a service1220being required to have a respective instance of service1220, service1220may be shared between and among various applications. In at least one embodiment, services may include an inference server or engine that may be used for executing detection or segmentation tasks, as non-limiting examples. In at least one embodiment, a model training service may be included that may provide machine learning model training and/or retraining capabilities. In at least one embodiment, a data augmentation service may further be included that may provide GPU accelerated data (e.g., DICOM, RIS, CIS, REST compliant, RPC, raw, etc.) extraction, resizing, scaling, and/or other augmentation. In at least one embodiment, a visualization service may be used that may add image rendering effects—such as ray-tracing, rasterization, denoising, sharpening, etc. —to add realism to two-dimensional (2D) and/or three-dimensional (3D) models. In at least one embodiment, virtual instrument services may be included that provide for beam-forming, segmentation, inferencing, imaging, and/or support for other applications within pipelines of virtual instruments.

In at least one embodiment, where a service1220includes an AI service (e.g., an inference service), one or more machine learning models may be executed by calling upon (e.g., as an API call) an inference service (e.g., an inference server) to execute machine learning model(s), or processing thereof, as part of application execution. In at least one embodiment, where another application includes one or more machine learning models for segmentation tasks, an application may call upon an inference service to execute machine learning models for performing one or more of processing operations associated with segmentation tasks. In at least one embodiment, software1218implementing advanced processing and inferencing pipeline that includes segmentation application and anomaly detection application may be streamlined because each application may call upon a same inference service to perform one or more inferencing tasks.

In at least one embodiment, hardware1222may include GPUs, CPUs, DPUs, graphics cards, an AI/deep learning system (e.g., an AI supercomputer, such as NVIDIA's DGX), a cloud platform, or a combination thereof. In at least one embodiment, different types of hardware1222may be used to provide efficient, purpose-built support for software1218and services1220in deployment system1206. In at least one embodiment, use of GPU processing may be implemented for processing locally (e.g., at facility1202), within an AI/deep learning system, in a cloud system, and/or in other processing components of deployment system1206to improve efficiency, accuracy, and efficacy of image processing and generation. In at least one embodiment, software1218and/or services1220may be optimized for GPU processing with respect to deep learning, machine learning, and/or high-performance computing, as non-limiting examples. In at least one embodiment, at least some of computing environment of deployment system1206and/or training system1204may be executed in a datacenter one or more supercomputers or high performance computing systems, with GPU optimized software (e.g., hardware and software combination of NVIDIA's DGX System). In at least one embodiment, hardware1222may include any number of GPUs that may be called upon to perform processing of data in parallel, as described herein. In at least one embodiment, cloud platform may further include GPU processing for GPU-optimized execution of deep learning tasks, machine learning tasks, or other computing tasks. In at least one embodiment, cloud platform may further include DPU processing to transmit data received over a network and/or through a network controller or other network interface directly to (e.g., a memory of) one or more GPU(s). In at least one embodiment, cloud platform (e.g., NVIDIA's NGC) may be executed using an AI/deep learning supercomputer(s) and/or GPU-optimized software (e.g., as provided on NVIDIA's DGX Systems) as a hardware abstraction and scaling platform. In at least one embodiment, cloud platform may integrate an application container clustering system or orchestration system (e.g., KUBERNETES) on multiple GPUs to enable seamless scaling and load balancing.

FIG.13is a system diagram for an example system1300for generating and deploying an imaging deployment pipeline, in accordance with at least one embodiment. In at least one embodiment, system1300may be used to implement process1200ofFIG.12and/or other processes including advanced processing and inferencing pipelines. In at least one embodiment, system1300may include training system1204and deployment system1206. In at least one embodiment, training system1204and deployment system1206may be implemented using software1218, services1220, and/or hardware1222, as described herein.

In at least one embodiment, system1300(e.g., training system1204and/or deployment system1206) may implemented in a cloud computing environment (e.g., using cloud1326). In at least one embodiment, system1300may be implemented locally with respect to a healthcare services facility, or as a combination of both cloud and local computing resources. In at least one embodiment, access to APIs in cloud1326may be restricted to authorized users through enacted security measures or protocols. In at least one embodiment, a security protocol may include web tokens that may be signed by an authentication (e.g., AuthN, AuthZ, Gluecon, etc.) service and may carry appropriate authorization. In at least one embodiment, APIs of virtual instruments (described herein), or other instantiations of system1300, may be restricted to a set of public IPs that have been vetted or authorized for interaction.

In at least one embodiment, various components of system1300may communicate between and among one another using any of a variety of different network types, including but not limited to local area networks (LANs) and/or wide area networks (WANs) via wired and/or wireless communication protocols. In at least one embodiment, communication between facilities and components of system1300(e.g., for transmitting inference requests, for receiving results of inference requests, etc.) may be communicated over data bus(ses), wireless data protocols (Wi-Fi), wired data protocols (e.g., Ethernet), etc.

In at least one embodiment, training system1204may execute training pipelines1304, similar to those described herein with respect toFIG.12. In at least one embodiment, where one or more machine learning models are to be used in deployment pipelines1310by deployment system1206, training pipelines1304may be used to train or retrain one or more (e.g. pre-trained) models, and/or implement one or more of pre-trained models1306(e.g., without a need for retraining or updating). In at least one embodiment, as a result of training pipelines1304, output model(s)1216may be generated. In at least one embodiment, training pipelines1304may include any number of processing steps, such as but not limited to imaging data (or other input data) conversion or adaption In at least one embodiment, for different machine learning models used by deployment system1206, different training pipelines1304may be used. In at least one embodiment, training pipeline1304similar to a first example described with respect toFIG.12may be used for a first machine learning model, training pipeline1304similar to a second example described with respect toFIG.12may be used for a second machine learning model, and training pipeline1304similar to a third example described with respect toFIG.12may be used for a third machine learning model. In at least one embodiment, any combination of tasks within training system1204may be used depending on what is required for each respective machine learning model. In at least one embodiment, one or more of machine learning models may already be trained and ready for deployment so machine learning models may not undergo any processing by training system1204, and may be implemented by deployment system1206.

In at least one embodiment, output model(s)1216and/or pre-trained model(s)1306may include any types of machine learning models depending on implementation or embodiment. In at least one embodiment, and without limitation, machine learning models used by system1300may include machine learning model(s) using linear regression, logistic regression, decision trees, support vector machines (SVM), Naïve Bayes, k-nearest neighbor (KNN), K means clustering, random forest, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., auto-encoders, convolutional, recurrent, perceptrons, Long/Short Term Memory (LSTM), Hopfield, Boltzmann, deep belief, deconvolutional, generative adversarial, liquid state machine, etc.), and/or other types of machine learning models.

In at least one embodiment, training pipelines1304may include AI-assisted annotation, as described in more detail herein with respect to at leastFIG.14B. In at least one embodiment, labeled data1212(e.g., traditional annotation) may be generated by any number of techniques. In at least one embodiment, labels or other annotations may be generated within a drawing program (e.g., an annotation program), a computer aided design (CAD) program, a labeling program, another type of program suitable for generating annotations or labels for ground truth, and/or may be hand drawn, in some examples. In at least one embodiment, ground truth data may be synthetically produced (e.g., generated from computer models or renderings), real produced (e.g., designed and produced from real-world data), machine-automated (e.g., using feature analysis and learning to extract features from data and then generate labels), human annotated (e.g., labeler, or annotation expert, defines location of labels), and/or a combination thereof. In at least one embodiment, for each instance of imaging data1208(or other data type used by machine learning models), there may be corresponding ground truth data generated by training system1204. In at least one embodiment, AI-assisted annotation may be performed as part of deployment pipelines1310; either in addition to, or in lieu of AI-assisted annotation included in training pipelines1304. In at least one embodiment, system1300may include a multi-layer platform that may include a software layer (e.g., software1218) of diagnostic applications (or other application types) that may perform one or more medical imaging and diagnostic functions. In at least one embodiment, system1300may be communicatively coupled to (e.g., via encrypted links) PACS server networks of one or more facilities. In at least one embodiment, system1300may be configured to access and referenced data from PACS servers to perform operations, such as training machine learning models, deploying machine learning models, image processing, inferencing, and/or other operations.

In at least one embodiment, a software layer may be implemented as a secure, encrypted, and/or authenticated API through which applications or containers may be invoked (e.g., called) from an external environment(s) (e.g., facility1202). In at least one embodiment, applications may then call or execute one or more services1220for performing compute, AI, or visualization tasks associated with respective applications, and software1218and/or services1220may leverage hardware1222to perform processing tasks in an effective and efficient manner.

In at least one embodiment, deployment system1206may execute deployment pipelines1310. In at least one embodiment, deployment pipelines1310may include any number of applications that may be sequentially, non-sequentially, or otherwise applied to imaging data (and/or other data types) generated by imaging devices, sequencing devices, genomics devices, etc. —including AI-assisted annotation, as described above. In at least one embodiment, as described herein, a deployment pipeline1310for an individual device may be referred to as a virtual instrument for a device (e.g., a virtual ultrasound instrument, a virtual CT scan instrument, a virtual sequencing instrument, etc.). In at least one embodiment, for a single device, there may be more than one deployment pipeline1310depending on information desired from data generated by a device. In at least one embodiment, where detections of anomalies are desired from an MRI machine, there may be a first deployment pipeline1310, and where image enhancement is desired from output of an MRI machine, there may be a second deployment pipeline1310.

In at least one embodiment, an image generation application may include a processing task that includes use of a machine learning model. In at least one embodiment, a user may desire to use their own machine learning model, or to select a machine learning model from model registry1224. In at least one embodiment, a user may implement their own machine learning model or select a machine learning model for inclusion in an application for performing a processing task. In at least one embodiment, applications may be selectable and customizable, and by defining constructs of applications, deployment, and implementation of applications for a particular user are presented as a more seamless user experience. In at least one embodiment, by leveraging other features of system1300—such as services1220and hardware1222—deployment pipelines1310may be even more user friendly, provide for easier integration, and produce more accurate, efficient, and timely results.

In at least one embodiment, deployment system1206may include a user interface1314(e.g., a graphical user interface, a web interface, etc.) that may be used to select applications for inclusion in deployment pipeline(s)1310, arrange applications, modify, or change applications or parameters or constructs thereof, use and interact with deployment pipeline(s)1310during set-up and/or deployment, and/or to otherwise interact with deployment system1206. In at least one embodiment, although not illustrated with respect to training system1204, user interface1314(or a different user interface) may be used for selecting models for use in deployment system1206, for selecting models for training, or retraining, in training system1204, and/or for otherwise interacting with training system1204.

In at least one embodiment, pipeline manager1312may be used, in addition to an application orchestration system1328, to manage interaction between applications or containers of deployment pipeline(s)1310and services1220and/or hardware1222. In at least one embodiment, pipeline manager1312may be configured to facilitate interactions from application to application, from application to service1220, and/or from application or service to hardware1222. In at least one embodiment, although illustrated as included in software1218, this is not intended to be limiting, and in some examples (e.g., as illustrated inFIG.11) pipeline manager1312may be included in services1220. In at least one embodiment, application orchestration system1328(e.g., Kubernetes, DOCKER, etc.) may include a container orchestration system that may group applications into containers as logical units for coordination, management, scaling, and deployment. In at least one embodiment, by associating applications from deployment pipeline(s)1310(e.g., a reconstruction application, a segmentation application, etc.) with individual containers, each application may execute in a self-contained environment (e.g., at a kernel level) to increase speed and efficiency.

In at least one embodiment, each application and/or container (or image thereof) may be individually developed, modified, and deployed (e.g., a first user or developer may develop, modify, and deploy a first application and a second user or developer may develop, modify, and deploy a second application separate from a first user or developer), which may allow for focus on, and attention to, a task of a single application and/or container(s) without being hindered by tasks of another application(s) or container(s). In at least one embodiment, communication, and cooperation between different containers or applications may be aided by pipeline manager1312and application orchestration system1328. In at least one embodiment, so long as an expected input and/or output of each container or application is known by a system (e.g., based on constructs of applications or containers), application orchestration system1328and/or pipeline manager1312may facilitate communication among and between, and sharing of resources among and between, each of applications or containers. In at least one embodiment, because one or more of applications or containers in deployment pipeline(s)1310may share same services and resources, application orchestration system1328may orchestrate, load balance, and determine sharing of services or resources between and among various applications or containers. In at least one embodiment, a scheduler may be used to track resource requirements of applications or containers, current usage or planned usage of these resources, and resource availability. In at least one embodiment, a scheduler may thus allocate resources to different applications and distribute resources between and among applications in view of requirements and availability of a system. In some examples, a scheduler (and/or other component of application orchestration system1328) may determine resource availability and distribution based on constraints imposed on a system (e.g., user constraints), such as quality of service (QOS), urgency of need for data outputs (e.g., to determine whether to execute real-time processing or delayed processing), etc.

In at least one embodiment, services1220leveraged by and shared by applications or containers in deployment system1206may include compute services1316, AI services1318, visualization services1320, and/or other service types. In at least one embodiment, applications may call (e.g., execute) one or more of services1220to perform processing operations for an application. In at least one embodiment, compute services1316may be leveraged by applications to perform super-computing or other high-performance computing (HPC) tasks. In at least one embodiment, compute service(s)1316may be leveraged to perform parallel processing (e.g., using a parallel computing platform1330) for processing data through one or more of applications and/or one or more tasks of a single application, substantially simultaneously. In at least one embodiment, parallel computing platform1330(e.g., NVIDIA's CUDA) may enable general purpose computing on GPUs (GPGPU) (e.g., GPUs1322). In at least one embodiment, a software layer of parallel computing platform1330may provide access to virtual instruction sets and parallel computational elements of GPUs, for execution of compute kernels. In at least one embodiment, parallel computing platform1330may include memory and, in some embodiments, a memory may be shared between and among multiple containers, and/or between and among different processing tasks within a single container. In at least one embodiment, inter-process communication (IPC) calls may be generated for multiple containers and/or for multiple processes within a container to use same data from a shared segment of memory of parallel computing platform1330(e.g., where multiple different stages of an application or multiple applications are processing same information). In at least one embodiment, rather than making a copy of data and moving data to different locations in memory (e.g., a read/write operation), same data in same location of a memory may be used for any number of processing tasks (e.g., at a same time, at different times, etc.). In at least one embodiment, as data is used to generate new data as a result of processing, this information of a new location of data may be stored and shared between various applications. In at least one embodiment, location of data and a location of updated or modified data may be part of a definition of how a payload is understood within containers.

In at least one embodiment, AI services1318may be leveraged to perform inferencing services for executing machine learning model(s) associated with applications (e.g., tasked with performing one or more processing tasks of an application). In at least one embodiment, AI services1318may leverage AI system1324to execute machine learning model(s) (e.g., neural networks, such as CNNs) for segmentation, reconstruction, object detection, feature detection, classification, and/or other inferencing tasks. In at least one embodiment, applications of deployment pipeline(s)1310may use one or more of output models1216from training system1204and/or other models of applications to perform inference on imaging data. In at least one embodiment, two or more examples of inferencing using application orchestration system1328(e.g., a scheduler) may be available. In at least one embodiment, a first category may include a high priority/low latency path that may achieve higher service level agreements, such as for performing inference on urgent requests during an emergency, or for a radiologist during diagnosis. In at least one embodiment, a second category may include a standard priority path that may be used for requests that may be non-urgent or where analysis may be performed at a later time. In at least one embodiment, application orchestration system1328may distribute resources (e.g., services1220and/or hardware1222) based on priority paths for different inferencing tasks of AI services1318.

In at least one embodiment, shared storage may be mounted to AI services1318within system1300. In at least one embodiment, shared storage may operate as a cache (or other storage device type) and may be used to process inference requests from applications. In at least one embodiment, when an inference request is submitted, a request may be received by a set of API instances of deployment system1206, and one or more instances may be selected (e.g., for best fit, for load balancing, etc.) to process a request. In at least one embodiment, to process a request, a request may be entered into a database, a machine learning model may be located from model registry1224if not already in a cache, a validation step may ensure appropriate machine learning model is loaded into a cache (e.g., shared storage), and/or a copy of a model may be saved to a cache. In at least one embodiment, a scheduler (e.g., of pipeline manager1312) may be used to launch an application that is referenced in a request if an application is not already running or if there are not enough instances of an application. In at least one embodiment, if an inference server is not already launched to execute a model, an inference server may be launched. Any number of inference servers may be launched per model. In at least one embodiment, in a pull model, in which inference servers are clustered, models may be cached whenever load balancing is advantageous. In at least one embodiment, inference servers may be statically loaded in corresponding, distributed servers.

In at least one embodiment, inferencing may be performed using an inference server that runs in a container. In at least one embodiment, an instance of an inference server may be associated with a model (and optionally a plurality of versions of a model). In at least one embodiment, if an instance of an inference server does not exist when a request to perform inference on a model is received, a new instance may be loaded. In at least one embodiment, when starting an inference server, a model may be passed to an inference server such that a same container may be used to serve different models so long as inference server is running as a different instance.

In at least one embodiment, during application execution, an inference request for a given application may be received, and a container (e.g., hosting an instance of an inference server) may be loaded (if not already), and a start procedure may be called. In at least one embodiment, pre-processing logic in a container may load, decode, and/or perform any additional pre-processing on incoming data (e.g., using a CPU(s) and/or GPU(s) and/or DPU(s)). In at least one embodiment, once data is prepared for inference, a container may perform inference as necessary on data. In at least one embodiment, this may include a single inference call on one image (e.g., a hand X-ray), or may require inference on hundreds of images (e.g., a chest CT). In at least one embodiment, an application may summarize results before completing, which may include, without limitation, a single confidence score, pixel level-segmentation, voxel-level segmentation, generating a visualization, or generating text to summarize findings. In at least one embodiment, different models or applications may be assigned different priorities. For example, some models may have a real-time (TAT<1 min) priority while others may have lower priority (e.g., TAT<12 min). In at least one embodiment, model execution times may be measured from requesting institution or entity and may include partner network traversal time, as well as execution on an inference service.

In at least one embodiment, transfer of requests between services1220and inference applications may be hidden behind a software development kit (SDK), and robust transport may be provided through a queue. In at least one embodiment, a request will be placed in a queue via an API for an individual application/tenant ID combination and an SDK will pull a request from a queue and give a request to an application. In at least one embodiment, a name of a queue may be provided in an environment from where an SDK will pick it up. In at least one embodiment, asynchronous communication through a queue may be useful as it may allow any instance of an application to pick up work as it becomes available. Results may be transferred back through a queue, to ensure no data is lost. In at least one embodiment, queues may also provide an ability to segment work, as highest priority work may go to a queue with most instances of an application connected to it, while lowest priority work may go to a queue with a single instance connected to it that processes tasks in an order received. In at least one embodiment, an application may run on a GPU-accelerated instance generated in cloud1326, and an inference service may perform inferencing on a GPU.

In at least one embodiment, visualization services1320may be leveraged to generate visualizations for viewing outputs of applications and/or deployment pipeline(s)1310. In at least one embodiment, GPUs1322may be leveraged by visualization services1320to generate visualizations. In at least one embodiment, rendering effects, such as ray-tracing, may be implemented by visualization services1320to generate higher quality visualizations. In at least one embodiment, visualizations may include, without limitation, 2D image renderings, 3D volume renderings, 3D volume reconstruction, 2D tomographic slices, virtual reality displays, augmented reality displays, etc. In at least one embodiment, virtualized environments may be used to generate a virtual interactive display or environment (e.g., a virtual environment) for interaction by users of a system (e.g., doctors, nurses, radiologists, etc.). In at least one embodiment, visualization services1320may include an internal visualizer, cinematics, and/or other rendering or image processing capabilities or functionality (e.g., ray tracing, rasterization, internal optics, etc.).

In at least one embodiment, hardware1222may include GPUs1322, AI system1324, cloud1326, and/or any other hardware used for executing training system1204and/or deployment system1206. In at least one embodiment, GPUs1322(e.g., NVIDIA's TESLA and/or QUADRO GPUs) may include any number of GPUs that may be used for executing processing tasks of compute services1316, AI services1318, visualization services1320, other services, and/or any of features or functionality of software1218. For example, with respect to AI services1318, GPUs1322may be used to perform pre-processing on imaging data (or other data types used by machine learning models), post-processing on outputs of machine learning models, and/or to perform inferencing (e.g., to execute machine learning models). In at least one embodiment, cloud1326, AI system1324, and/or other components of system1300may use GPUs1322. In at least one embodiment, cloud1326may include a GPU-optimized platform for deep learning tasks. In at least one embodiment, AI system1324may use GPUs, and cloud1326—or at least a portion tasked with deep learning or inferencing—may be executed using one or more AI systems1324. As such, although hardware1222is illustrated as discrete components, this is not intended to be limiting, and any components of hardware1222may be combined with, or leveraged by, any other components of hardware1222.

In at least one embodiment, AI system1324may include a purpose-built computing system (e.g., a super-computer or an HPC) configured for inferencing, deep learning, machine learning, and/or other artificial intelligence tasks. In at least one embodiment, AI system1324(e.g., NVIDIA's DGX) may include GPU-optimized software (e.g., a software stack) that may be executed using a plurality of GPUs1322, in addition to DPUs, CPUs, RAM, storage, and/or other components, features, or functionality. In at least one embodiment, one or more AI systems1324may be implemented in cloud1326(e.g., in a data center) for performing some or all of AI-based processing tasks of system1300.

In at least one embodiment, cloud1326may include a GPU-accelerated infrastructure (e.g., NVIDIA's NGC) that may provide a GPU-optimized platform for executing processing tasks of system1300. In at least one embodiment, cloud1326may include an AI system(s)1324for performing one or more of AI-based tasks of system1300(e.g., as a hardware abstraction and scaling platform). In at least one embodiment, cloud1326may integrate with application orchestration system1328leveraging multiple GPUs to enable seamless scaling and load balancing between and among applications and services1220. In at least one embodiment, cloud1326may tasked with executing at least some of services1220of system1300, including compute services1316, AI services1318, and/or visualization services1320, as described herein. In at least one embodiment, cloud1326may perform small and large batch inference (e.g., executing NVIDIA's TENSOR RT), provide an accelerated parallel computing API and platform1330(e.g., NVIDIA's CUDA), execute application orchestration system1328(e.g., KUBERNETES), provide a graphics rendering API and platform (e.g., for ray-tracing, 2D graphics, 3D graphics, and/or other rendering techniques to produce higher quality cinematics), and/or may provide other functionality for system1300.

FIG.14Aillustrates a data flow diagram for a process1400to train, retrain, or update a machine learning model, in accordance with at least one embodiment. In at least one embodiment, process1400may be executed using, as a non-limiting example, system1300ofFIG.13. In at least one embodiment, process1400may leverage services1220and/or hardware1222of system1300, as described herein. In at least one embodiment, refined models1412generated by process1400may be executed by deployment system1206for one or more containerized applications in deployment pipelines1310.

In at least one embodiment, model training1214may include retraining or updating an initial model1404(e.g., a pre-trained model) using new training data (e.g., new input data, such as customer dataset1406, and/or new ground truth data associated with input data). In at least one embodiment, to retrain, or update, initial model1404, output or loss layer(s) of initial model1404may be reset, or deleted, and/or replaced with an updated or new output or loss layer(s). In at least one embodiment, initial model1404may have previously fine-tuned parameters (e.g., weights and/or biases) that remain from prior training, so training or retraining1214may not take as long or require as much processing as training a model from scratch. In at least one embodiment, during model training1214, by having reset or replaced output or loss layer(s) of initial model1404, parameters may be updated and re-tuned for a new data set based on loss calculations associated with accuracy of output or loss layer(s) at generating predictions on new, customer dataset1406(e.g., image data1208ofFIG.12).

In at least one embodiment, pre-trained models1306may be stored in a data store, or registry (e.g., model registry1224ofFIG.12). In at least one embodiment, pre-trained models1306may have been trained, at least in part, at one or more facilities other than a facility executing process1400. In at least one embodiment, to protect privacy and rights of patients, subjects, or clients of different facilities, pre-trained models1306may have been trained, on-premise, using customer or patient data generated on-premise. In at least one embodiment, pre-trained models1306may be trained using cloud1326and/or other hardware1222, but confidential, privacy protected patient data may not be transferred to, used by, or accessible to any components of cloud1326(or other off premise hardware). In at least one embodiment, where a pre-trained model1306is trained at using patient data from more than one facility, pre-trained model1306may have been individually trained for each facility prior to being trained on patient or customer data from another facility. In at least one embodiment, such as where a customer or patient data has been released of privacy concerns (e.g., by waiver, for experimental use, etc.), or where a customer or patient data is included in a public data set, a customer or patient data from any number of facilities may be used to train pre-trained model1306on-premise and/or off premise, such as in a datacenter or other cloud computing infrastructure.

In at least one embodiment, when selecting applications for use in deployment pipelines1310, a user may also select machine learning models to be used for specific applications. In at least one embodiment, a user may not have a model for use, so a user may select a pre-trained model1306to use with an application. In at least one embodiment, pre-trained model1306may not be optimized for generating accurate results on customer dataset1406of a facility of a user (e.g., based on patient diversity, demographics, types of medical imaging devices used, etc.). In at least one embodiment, prior to deploying pre-trained model1306into deployment pipeline1310for use with an application(s), pre-trained model1306may be updated, retrained, and/or fine-tuned for use at a respective facility.

In at least one embodiment, a user may select pre-trained model1306that is to be updated, retrained, and/or fine-tuned, and pre-trained model1306may be referred to as initial model1404for training system1204within process1400. In at least one embodiment, customer dataset1406(e.g., imaging data, genomics data, sequencing data, or other data types generated by devices at a facility) may be used to perform model training1214(which may include, without limitation, transfer learning) on initial model1404to generate refined model1412. In at least one embodiment, ground truth data corresponding to customer dataset1406may be generated by training system1204. In at least one embodiment, ground truth data may be generated, at least in part, by clinicians, scientists, doctors, practitioners, at a facility (e.g., as labeled clinic data1212ofFIG.12).

In at least one embodiment, AI-assisted annotation1210may be used in some examples to generate ground truth data. In at least one embodiment, AI-assisted annotation1210(e.g., implemented using an AI-assisted annotation SDK) may leverage machine learning models (e.g., neural networks) to generate suggested or predicted ground truth data for a customer dataset. In at least one embodiment, user1410may use annotation tools within a user interface (a graphical user interface (GUI)) on computing device1408.

In at least one embodiment, user1410may interact with a GUI via computing device1408to edit or fine-tune (auto)annotations. In at least one embodiment, a polygon editing feature may be used to move vertices of a polygon to more accurate or fine-tuned locations.

In at least one embodiment, once customer dataset1406has associated ground truth data, ground truth data (e.g., from AI-assisted annotation, manual labeling, etc.) may be used by during model training1214to generate refined model1412. In at least one embodiment, customer dataset1406may be applied to initial model1404any number of times, and ground truth data may be used to update parameters of initial model1404until an acceptable level of accuracy is attained for refined model1412. In at least one embodiment, once refined model1412is generated, refined model1412may be deployed within one or more deployment pipelines1310at a facility for performing one or more processing tasks with respect to medical imaging data.

In at least one embodiment, refined model1412may be uploaded to pre-trained models1306in model registry1224to be selected by another facility. In at least one embodiment, his process may be completed at any number of facilities such that refined model1412may be further refined on new datasets any number of times to generate a more universal model.

FIG.14Bis an example illustration of a client-server architecture1432to enhance annotation tools with pre-trained annotation models, in accordance with at least one embodiment. In at least one embodiment, AI-assisted annotation tools1436may be instantiated based on a client-server architecture1432. In at least one embodiment, annotation tools1436in imaging applications may aid radiologists, for example, identify organs and abnormalities. In at least one embodiment, imaging applications may include software tools that help user1410to identify, as a non-limiting example, a few extreme points on a particular organ of interest in raw images1434(e.g., in a 3D MRI or CT scan) and receive auto-annotated results for all 2D slices of a particular organ. In at least one embodiment, results may be stored in a data store as training data1438and used as (for example and without limitation) ground truth data for training. In at least one embodiment, when computing device1408sends extreme points for AI-assisted annotation1210, a deep learning model, for example, may receive this data as input and return inference results of a segmented organ or abnormality. In at least one embodiment, pre-instantiated annotation tools, such as AI-Assisted Annotation Tool1436B inFIG.14B, may be enhanced by making API calls (e.g., API Call1444) to a server, such as an Annotation Assistant Server1440that may include a set of pre-trained models1442stored in an annotation model registry, for example. In at least one embodiment, an annotation model registry may store pre-trained models1442(e.g., machine learning models, such as deep learning models) that are pre-trained to perform AI-assisted annotation on a particular organ or abnormality. These models may be further updated by using training pipelines1304. In at least one embodiment, pre-installed annotation tools may be improved over time as new labeled clinic data1212is added.

Such components may be used to generate synthetic data imitating failure cases in a network training process, which may help to improve performance of the network while limiting the amount of synthetic data to avoid overfitting.