Patent Description:
Stable and reliable robotic systems are becoming increasingly common, which has contributed to the recent advancement and proliferation of unmanned system technologies. In many instances these systems are equipped with recording devices (e.g., video, infrared, heat, audio, point cloud and/or other recording devices). Output data streams of these recording devices may enable an operator to get a good tactical view of what is happening on the ground. Furthermore, it may be useful for these output data streams to be annotated with different detected objects (e.g., tanks, operators, helicopters, etc.). One method of annotating these data streams may be achieved using machine learning models. These machine learning models are generally hosted on dedicated hardware with dedicated data sources, as they may require a large amount of processing resources (e.g., power, processor, memory, etc.). Thus, computing devices in the field may not be able to process data quickly and efficiently from multiple unmanned vehicles, especially if that data is of different types (e.g., video, infrared, audio, heat, etc.) requiring different machine learning models to process.

Further background information can be found in the following document: <CIT> which describes autonomous vehicles with sensors used to detect information that allows the vehicle to navigate its environment and determine occupant status.

The invention is set out in the appended claim set.

Methods and systems are described herein for generating composite frames that include objects detected in multiple different types of data frames. For example, a data stream processing system may be used to perform the operations described herein. The data stream processing system may reside at a central location (e.g., a command-and-control point). For example, a command-and-control point may be in a vehicle equipped with one or more computing devices, a datacenter that houses computing devices or in another suitable environment. In some embodiments, the data stream processing system may receive multiple data streams from, for example, multiple unmanned vehicles and determine, based on the type of data within each data stream, a machine learning model for each data stream for processing that type of data. Each machine learning model may receive the frames of a corresponding data stream and output indications and locations of objects within those data streams. The data stream processing system may then generate a composite data stream with indications of the detected objects.

The data stream processing system may receive a plurality of data streams. The data streams may include recording data of a particular location and may include data of different types. In some embodiments, the data streams may be received from different unmanned vehicles. For example, an operator may control multiple unmanned vehicles using a controller device. Those unmanned vehicles may include one or more onboard recording devices (e.g., a video camera, a microphone, an infrared camera, a thermal imaging device, a LiDAR and/or another suitable recording device). Thus, in some embodiments, the plurality of data streams may include data streams having captured data of different wavelength ranges. The onboard recording devices may generate data streams and transmit those data streams to a central location (e.g., a data center or a command-and-control vehicle).

The data stream processing system may determine a type of data captured within each data stream. For example, the data stream processing system may determine that one data stream includes video data and/or audio data, another data stream include thermal data, and yet another data stream include infrared data. In some embodiments, the data stream processing system may make the determination based on the metadata associated with each stream. However, in some embodiments, the data stream processing system may determine, based on the different radio or light wavelength ranges for each data stream of the plurality of data streams, a type of data captured within each data stream.

The data stream processing system may identify machine learning models for detecting objects within the data streams. In particular, the data stream processing system may determine a plurality of machine learning models for the plurality of data streams based on the type of data captured within each data stream. Each machine learning model may have been trained to identify objects within a frame of a particular type of data. To continue with the example above, if one of the data streams includes video data, data stream processing system may identify a machine learning model trained for processing data streams including video data. If another data stream includes thermal imaging data, data stream processing system may identify a machine learning model trained for processing data streams including thermal imaging data.

The data stream processing system may use the identified machine learning models to process the received data streams. Thus, the data stream processing system may input each of the plurality of data streams into a corresponding machine learning model of the plurality of machine learning models. For example, each data stream may be divided into a plurality of frames. Thus, the data stream processing system may input each frame (e.g., sequentially or in parallel) into each machine learning model.

The data stream processing system may receive, from each machine learning model, a plurality of object identifiers and a plurality of object locations corresponding to one or more objects detected in each data stream of the plurality of data streams. For example, each machine learning model may detect objects within each frame of each data stream. Each machine learning model may output an identifier of the object and a location of the object within the frame. In some embodiments, each machine learning model may output a probability that the object has been detected. The object identifiers may include a description of the object (e.g., tank, operator, missile, etc.).

The data stream processing system may then group the frames of different data streams based on timestamps to add objects detected in different data streams to composite frames. Thus, the data stream processing system may select, from the plurality of data streams, a plurality of sets of frames. Each set of frames of the plurality of sets of frames may match a corresponding timestamp of a plurality of timestamps. For example, each data stream may include a plurality of frames with each frame having a timestamp (e.g., in metadata or stamped onto the frame). Thus, the data stream processing system may identify frames with matching timestamps and group them for composite data stream generation.

In some embodiments, the data stream processing system may synchronize the data streams when selecting the plurality of sets of frames from the plurality of data streams. The data stream processing system may determine a time period common to the plurality of data streams. For example, each data stream may have been recorded during the same time interval (e.g., between <NUM>:<NUM>:<NUM> and <NUM>:<NUM>:<NUM>). In some embodiments, the recording time may be different for different time periods. Thus, data stream processing system may synchronize the overlapping frames.

The data stream processing system may then determine a frequency of each data stream of the plurality of data streams. For example, a video stream may have been recorded at <NUM> frames per second, a thermal imaging data stream may have been recorded at <NUM> frames per second, and a sound data stream may have been recorded at <NUM> thousand samples per second. In some embodiments, the data stream processing system may select a composite data stream frequency based on a lowest frequency data stream. For example, if the video stream has a frequency of <NUM> frames per second but other data streams have a higher frequency, the data stream processing system may select <NUM> frames per second as the composite frequency. Furthermore, to add sound to each frame, data stream processing system may use a different way to process audio streams. The data stream processing system may then select a first timestamp within the time period common to the plurality of data streams. For example, if the data streams have been synchronized as starting at <NUM>:<NUM>:<NUM>, the data stream processing system may select <NUM>:<NUM>:<NUM> as the first timestamp.

The data stream processing system may then locate, within each data stream of the plurality of data streams, a corresponding frame associated with the first timestamp. In some embodiments, there may be more than one available frame for a particular timestamp. Thus, the data stream processing system may select a first frame in a chronological order, the last frame in chronological order, or another suitable frame. The data stream processing system may then generate a first set of frames that match the first timestamp based on frames with the plurality of data streams that match the first timestamp.

The data stream processing system may then generate a data stream of composite frames. Thus, the data stream processing system may generate a composite data stream based on the plurality of sets of frames. The composite data stream may include frames with indicators representing objects detected by each machine learning model at a corresponding location associated with each object.

In some embodiments, the data stream processing system may generate the composite data stream using a base frame and add the data from other frames (e.g., frames from other data streams) to the base frame. For example, the data stream processing system may retrieve, for a particular set of frames (e.g., a set of frames from different data streams, but for a particular timestamp), a plurality of objects detected by the plurality of machine learning models. The data stream processing system may then select a video frame from the first set of frames to be a base frame and determine for the plurality of objects corresponding locations within the video frame. The data stream processing system may then overlay an indicator of each corresponding object at the corresponding location within the video frame.

In some embodiments, multiple machine learning models may identify the same object at the same time. Thus, it may be possible that multiple indicators be displayed for each object. In some embodiments, each indicator may include information about the type of data stream in which the object was identified. In some embodiments, each indicator may be selectable. Thus, when a selection of the indicator is detected, the data stream processing system may generate for display the frame associated with the indicator. For example, if the base frame is a video frame and the object was also detected in a thermal imaging frame, the data stream processing system may generate for display the thermal frame when an indicator from the thermal frame is selected.

In some embodiments, only one indicator may be displayed; however, that indicator may indicate that the object has been detected in multiple streams. For example, the data stream processing system may determine that a first machine learning model detected a first object in a first frame of a first data stream and that a second machine learning model has detected the first object (the same object) in a second frame of a second data stream. Thus, the data stream processing system may add a first indicator to a first composite frame indicating that the first object has been detected by both the first machine learning model and the second machine learning model.

In some embodiments, the data stream processing system may generate for display the composite data stream. For example, the command-and-control location may include one or more display devices where the composite data stream may be displayed. In some embodiments, the data stream processing system may transmit the data stream to another location.

Various other aspects, features and advantages of the system will be apparent through the detailed description and the drawings attached hereto. It is also to be understood that both the foregoing general description and the following detailed description are examples and not restrictive of the scope of the disclosure. As used in the specification and in the claims, the singular forms of "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. In addition, as used in the specification and the claims, the term "or" means "and/or" unless the context clearly dictates otherwise. Additionally, as used in the specification, "a portion" refers to a part of, or the entirety of (i.e., the entire portion) a given item (e.g., data), unless the context clearly dictates otherwise.

In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the disclosed embodiments. It will be appreciated, however, by those having skill in the art, that the embodiments may be practiced without these specific details.

In other cases, well-known models and devices are shown in block diagram form in order to avoid unnecessarily obscuring the disclosed embodiments. It should also be noted that the methods and systems disclosed herein are also suitable for applications unrelated to source code programming.

<FIG> is an example of environment <NUM> for generating composite images of objects detected in multiple different types of data frames. Environment <NUM> includes data stream processing system <NUM>, data node <NUM>, and recording devices 108a-108n. Data stream processing system <NUM> may execute instructions for generating composite images of objects detected in multiple different types of data frames. Data stream processing system <NUM> may include software, hardware or a combination of the two. In some embodiments, although shown separately, data stream processing system <NUM> and data node <NUM> may reside on the same computing device.

Data node <NUM> may store various data. For example, data node <NUM> may store a repository of machine learning models that may be accessed by data stream processing system. In some embodiments, data node <NUM> may also be used to train machine learning models and or adjust parameters (e.g., hyperparameters) associated with those machine learning models. Data node <NUM> may include software, hardware, or a combination of the two. For example, data node <NUM> may be a physical server, or a virtual server that is running on a physical computer system. In some embodiments, data node <NUM> may reside in a datacenter to be used by commanding officers for situational awareness. Network <NUM> may be a local area network, a wide area network (e.g., the Internet), or a combination of the two. Recording devices 108a-108n may be devices attached to unmanned vehicles and may include video cameras, infrared cameras, microphones, thermal imaging devices, and/or other suitable devices.

Data stream processing system <NUM> may receive a plurality of data streams that include recording data of a location. The plurality of data streams may include data streams having data of different types. For example, data stream processing system <NUM> may receive the data streams from different unmanned vehicles. Data stream processing system <NUM> may receive the data streams using communication subsystem <NUM>. Communication subsystem <NUM> may include software components, hardware components, or a combination of both. For example, communication subsystem <NUM> may include a network card (e.g., a wireless network card/processor) that is coupled with software to drive the card/processor. The network card may be built into a server or another suitable computing device. The data streams may be received from one or more unmanned vehicles. For example, an unmanned vehicle (e.g., a drone) may include an onboard camera that may send images to communication subsystem <NUM> via a wireless connection. Another unmanned vehicle may include an onboard microphone and/or a thermal imaging device. Communication subsystem <NUM> may pass each stream or a pointer to each stream in memory, to machine learning selection subsystem <NUM>.

In some embodiments, each data stream may include a corresponding plurality of frames. For example, a video data stream may include video frames, while a thermal imaging data stream may include thermal imaging frames. Thus, the plurality of data streams may include data streams having captured data of different wavelength ranges. For example, a data stream that includes infrared data may include data of a specific wavelength range (e.g., wavelength of infrared spectrum), while an audio data stream may include data of another wavelength range (e.g., sound spectrum). That data streams may be recording data of a particular location or of a particular range of locations. For example, if a mission is being executed against a particular target, a plurality of unmanned vehicles (e.g., having onboard different recording devices) may be used to record the area around the target. Thus, the location may be a target building, a target vehicle or another suitable target.

Machine learning selection subsystem <NUM> may include software components, hardware components, or a combination of both and may be used to determine a corresponding machine learning model to use for each data stream. Thus, machine learning selection subsystem <NUM> may determining a type of data captured within each data stream. For example, each data stream may include accompanying metadata that may indicate a type of data is included in the stream. Thus, machine learning selection subsystem <NUM> may access the metadata of each stream and determine the type of data within that stream.

In some embodiments, machine learning selection subsystem <NUM> may use the wavelength of data within the data stream to determine the type of data within that data stream. For example, machine learning selection subsystem <NUM> may sample a portion (e.g., a frame) of a data stream and determine the wavelength of the data within the data stream. Machine learning selection subsystem <NUM> may sample a particular data stream and determine that the wavelength of data within that data stream is between <NUM> and <NUM>. Based on that wavelength, machine learning selection subsystem <NUM> may determine that the data stream include data captured by an infrared camera (e.g., the data type is infrared).

Based on the type of data within each data stream, the machine learning selection subsystem may identify a machine learning model to process that data stream. Thus, machine learning selection subsystem <NUM> may determine, for the plurality of data streams based on the type of data captured within each data stream, a plurality of machine learning models for processing the data within those data streams. Each machine learning model may have been trained to identify objects within a frame of a particular type of data. For example, a particular machine learning model may have been trained to identify objects within a particular data stream. A particular machine learning model may have been trained to identify objects within thermal imaging data (e.g., thermal data frames), while another machine learning model may have been trained to identify objects within video data, and yet another machine learning model may have been trained to identify objects within infrared data streams.

<FIG> illustrates an excerpt of a table <NUM> that may store data types and corresponding machine learning models for processing those data types. Column <NUM> may store types of data while column <NUM> may store machine learning model identifiers for the corresponding data types. Thus, machine learning selection subsystem <NUM> may, after determining a type of data associated with a particular stream, iterate through data types in column <NUM> of table <NUM> to locate a matching data type. When machine learning selection subsystem <NUM> locates the matching data type, machine learning selection subsystem <NUM> may retrieve a corresponding machine learning model identifier. Machine learning selection subsystem <NUM> may repeat this process for each received data stream. In some embodiments, a first type of data stream may be video data, a second type of data may be infrared data, a third type of data may be thermal data, a fourth type of data may be audio data, and a fifth type of data may be point cloud data. Machine learning selection subsystem <NUM> may pass indicators of the streams and corresponding identified machine learning models to machine learning subsystem <NUM>.

Machine learning subsystem <NUM> may include software components, hardware components, or a combination of both. For example, machine learning subsystem <NUM> may include software components that access data in memory and/or storage and may use one or more processors to perform its operations. Machine learning subsystem <NUM> may receive the indicators of the streams and the corresponding machine learning models and use the machine learning models to identify objects within those data streams. In particular, machine learning subsystem <NUM> may input each of the plurality of data streams into a corresponding machine learning model of the plurality of machine learning models. For example, machine learning subsystem <NUM> may use the identifiers associated with the identified machine learning models to access (e.g., using an application programming interface) those machine learning models and input the data streams into those machine learning models.

In some embodiments, each data stream may be a digital file. The digital file may be input into to the corresponding machine learning model (e.g., a machine learning model hosted by data stream processing system <NUM> or hosted on data node <NUM>). Each machine learning model may break the digital file into frames and process each frame to identify objects in each frame. A frame may be in image or another suitable portion of the data stream. In some embodiments, each machine learning model may be trained to identify objects in frames (e.g., video images, thermal images, etc.). Thus, machine learning subsystem <NUM> may split the data stream (e.g., the digital file) into portions (e.g., frames, images, etc.) that will enable each machine learning model to process the frames and identify objects in those frames.

The machine learning models and algorithms descried above may take many forms. However, these machine learning models and/or algorithms may be used by machine learning subsystem <NUM> with any application programming interface. <FIG> illustrates an exemplary machine learning model. Machine learning model <NUM> may take input <NUM> (e.g., frame data, image data, sound data, point cloud data, etc.) and may output <NUM> coordinates (e.g., coordinates within a frame) and object identifiers of objects detected within a frame. In some embodiments, each machine learning model may output a probability that the object has been detected. The output parameters may be fed back to the machine learning model as input to train the machine learning model (e.g., alone or in conjunction with user indications of the accuracy of outputs, labels associated with the inputs, or other reference feedback information). The machine learning model may update its configurations (e.g., weights, biases, or other parameters) based on the assessment of its prediction (e.g., of an information source) and reference feedback information (e.g., user indication of accuracy, reference labels, or other information). Connection weights may be adjusted, for example, if the machine learning model is a neural network, to reconcile differences between the neural network's prediction and the reference feedback. One or more neurons of the neural network may require that their respective errors are sent backward through the neural network to facilitate the update process (e.g., backpropagation of error). Updates to the connection weights may, for example, be reflective of the magnitude of error propagated backward after a forward pass has been completed. In this way, for example, the machine learning model may be trained to generate better predictions of information sources that are responsive to a query.

In some embodiments, the machine learning model may include an artificial neural network. In such embodiments, the machine learning model may include an input layer and one or more hidden layers. Each neural unit of the machine learning model may be connected to one or more other neural units of the machine learning model. Such connections may be enforcing or inhibitory in their effect on the activation state of connected neural units. Each individual neural unit may have a summation function, which combines the values of all of its inputs together. Each connection (or the neural unit itself) may have a threshold function that a signal must surpass before it propagates to other neural units. The machine learning model may be self-learning and/or trained, rather than explicitly programmed, and may perform significantly better in certain areas of problem solving, as compared to computer programs that do not use machine learning. During training, an output layer of the machine learning model may correspond to a classification of machine learning model, and an input known to correspond to that classification may be input into an input layer of the machine learning model during training. During testing, an input without a known classification may be input into the input layer, and a determined classification may be output.

A machine learning model may include embedding layers in which each feature of a vector is converted into a dense vector representation. These dense vector representations for each feature may be pooled at one or more subsequent layers to convert the set of embedding vectors into a single vector.

The machine learning model may be structured as a factorization machine model. The machine learning model may be a non-linear model and/or supervised learning model that can perform classification and/or regression. For example, the machine learning model may be a general-purpose supervised learning algorithm that the system uses for both classification and regression tasks. Alternatively, the machine learning model may include a Bayesian model configured to perform variational inference on the graph and/or vector.

When each machine learning model is finished processing a corresponding data stream, the resulting data is output from each machine learning model. Thus, machine learning subsystem <NUM> may receive, from each machine learning model, a plurality of object identifiers and a plurality of object locations corresponding to one or more objects detected in each data stream of the plurality of data streams. In some embodiments, each machine learning model may also output a probability that a particular object was detected in a particular frame. For sound data streams, the machine learning model may output the location of a particular sound and/or a direction if the sound being detected is located outside a particular area (e.g., outside of an area that is being recorded by other recording devices (e.g., video cameras, thermal imaging device, infrared cameras, etc.)).

In some embodiments, model output may be stored in a data structure and linked to an identifier of a frame where that object was detected. For example, as each frame is processed by each machine learning model, machine learning subsystem <NUM> may receive machine learning model output for that frame. Thus, machine learning subsystem <NUM> may generate a portion of a data structure for each frame. In some embodiments, the data structure may be a table. <FIG> illustrates an excerpt of a table <NUM> storing object indicators, locations of those indicators, and corresponding frame identifiers. Column <NUM> stores frame identifiers. Each frame identifier may identify a frame and a data stream that the frame was extracted from. In some embodiments, column <NUM> may store a timestamp associated with the frame (e.g., the time and/or date that the information in the frame was recorded). Column <NUM> stores indications of the objects detected within a corresponding frame. Column <NUM> may include an object name and/or an object description. In some embodiments, column <NUM> may include a link to an object description. Column <NUM> may include a location of the object within the frame. In some embodiments, the location may be a single point associated with the object. However, in some embodiments, the location may be multiple points associated with the object.

Based on output of each machine learning model as each machine learning model processes frames of the data streams, machine learning subsystem <NUM> may add information to table <NUM>. That is, machine learning subsystem <NUM> may be building table <NUM> as the machine learning models output information associated with identified objects. In some embodiments, machine learning subsystem <NUM> may calculate the location of each detected object within three-dimensional space. For example, if the frame is a video frame, machine learning subsystem <NUM> may determine the latitude and longitude of each object. This determination may be made using the location of the unmanned vehicle that recorded the video frame together with the heading of the vehicle and direction of the recording device.

Machine learning subsystem <NUM> may indicate to composite stream generation subsystem <NUM> that machine learning model output is ready for generation of a composite stream. Composite stream generation subsystem <NUM> may include software components, hardware components, or a combination of both. For example, composite stream generation subsystem <NUM> may include software components that access data in memory and/or storage and may use one or more processors to perform its operations. Composite stream generation subsystem may use the detected objects and the data streams to generate composite frames and assemble those frames into a composite data stream.

Composite stream generation subsystem <NUM> may select, from the plurality of data streams, a plurality of sets of frames. Each set of frames of the plurality of sets of frames matches a corresponding timestamp of a plurality of timestamps. For example, each data stream may include a corresponding plurality of frames. Each of those frames may be associated with a particular timestamp (e.g., a time of when that frame was captured). In some embodiments, composite stream generation subsystem <NUM> may iterate through each received data stream and determine a timestamp of an earliest frame recorded in each of those data streams. Composite stream generation subsystem <NUM> may then select the latest timestamp of the earliest frames and use that timestamp as the first timestamp for composite frame generation. <FIG> illustrates frame selection for a first composite frame. Illustration <NUM> of <FIG> shows three data streams (data stream <NUM>, data stream <NUM>, and data stream <NUM>). Each data stream may include a plurality of frames (e.g., video frames, thermal imaging frames, infrared frames, etc.). Thus, composite stream generating subsystem <NUM> may determine for each of data stream <NUM>, data stream <NUM>, and data stream <NUM> a starting timestamp. <FIG> illustrates that the starting timestamp for data stream <NUM> is the latest timestamp. Thus, indicator <NUM> illustrates the frames that will be selected for a starting composite frame. Composite stream generation subsystem <NUM> may iterate through each timestamp and generate a plurality of sets of frames.

In some embodiments, the received data streams may have different associated frequencies. For example, video data streams may have a frequency of <NUM> frames per second, while a thermal imaging data streams may have a frequency of <NUM> frames per second. Thus, composite stream generating subsystem <NUM> may adjust for different frequencies of different data streams by subsampling or interpolating some data streams for generating composite frames. The system may choose to generate output at a rate equal to the lowest input data update rate, and subsample or average data in the higher-rate streams. Alternatively, the system may choose to output at a rate higher than the lowest input update rate using interpolation or other means of up-sampling data. In particular, composite stream generating subsystem <NUM> may determine a time period common to the plurality of data streams. For example, composite stream generating subsystem <NUM> may retrieve a starting timestamp associated with a starting frame of each data stream and an ending timestamp associated with an ending frame of each data stream. In some embodiments, composite stream generating subsystem <NUM> may retrieve the starting timestamp and the ending timestamp from metadata associated with each data stream.

Composite stream generation subsystem <NUM> may determine a frequency of each data stream of the plurality of data streams. For example, composite stream generating subsystem <NUM> may determine how many frames are recorded per second for each data stream by, for example, retrieving timestamps for a plurality of frames and dividing a time range of the timestamps by the number of frames. In some embodiments, composite stream generation subsystem <NUM> may retrieve the frequency of each data stream from metadata received form the corresponding data stream. Composite stream generation subsystem <NUM> may select a composite data stream frequency based on a lowest frequency data stream. For example, composite stream generation subsystem <NUM> may select, as the base frequency, the lowest frequency. That is, if data stream <NUM> has a frequency of <NUM> frames per second, data stream <NUM> has a frequency of <NUM> frames per second, and data stream <NUM> has a frequency of <NUM> frames per second, composite stream generation subsystem <NUM> may select <NUM> frames per second as the frequency. In some embodiments, composite stream generation subsystem <NUM> may select the highest frequency data stream.

Composite stream generation subsystem <NUM> may then select a first timestamp within the time period common to the plurality of data streams. For example, as illustrated in <FIG>, different data streams may have different start times. Thus, composite stream generation subsystem <NUM> may select a start time common to all received data streams. In some embodiments, composite stream generation subsystem <NUM> may select, as a start timestamp, the earliest timestamp within any of the received data streams.

Composite stream generation subsystem <NUM> may then locate, within each data stream of the plurality of data streams, a corresponding frame associated with the first timestamp. Thus, composite stream generation subsystem <NUM> may iterate through each data stream until the frame with the first timestamp is located. Composite stream generation subsystem <NUM> may then generate a first set of frames that match the first timestamp based on frames with the plurality of data streams that match or are suitably close to the first timestamp (e.g., within <NUM>/8th, <NUM>/16th, <NUM>/32nd of second). The first set of frames may include frames from different data streams having matching timestamps. For two timestamps to match, those timestamps do not necessarily need to be identical. Two timestamps may match when they are within a particular threshold of each other (e.g., within <NUM>/1000th of a second, <NUM>/100th of a second, or within another suitable threshold).

Composite stream generation subsystem <NUM> may generate a composite data stream based on the plurality of sets of frames. The composite data stream may include frames with indicators representing objects detected by each machine learning model at a corresponding location associated with each object. In some embodiments, composite stream generation subsystem <NUM> may select a base frame based on the frequency and then add indicators of objects (e.g., using overlays) detected in other received data streams into the selected frame. For example, there may be three received data streams. A first data stream may be a video data stream with a frequency of <NUM> frames per second. A second data stream may be an infrared data stream with a frequency of <NUM> frames per second. A third data stream may be a thermal data stream with a frequency of <NUM> frames per second. Composite stream generation subsystem <NUM> may select the video data stream (e.g., having the lowest frames per second) as a base data stream (i.e., use the frames of that data stream as base frames). Thus, for the other two received data streams, the frames beyond <NUM> frames per second will not be included in the composite data stream. Accordingly, some data may be lost.

In some embodiments, composite stream generation subsystem <NUM> may always select video data stream frames as base frames, for example, because people are used to looking at video frames versus, for example, thermal imaging frames. Thus, composite stream generation subsystem <NUM> may retrieve (e.g., from memory or from data node <NUM>), for a first set of frames, a plurality of objects detected by the plurality of machine learning models. In some embodiments, the first set of frames may be a data structure with pointers to the selected frames. In some embodiments, the pointers may be to frames stored in a data structure together with identified objects (e.g., as illustrated in <FIG>).

Composite stream generation subsystem <NUM> may then determine the types of frames within the first set of frames and select a video frame from the first set of frames as a base frame. Composite stream generation subsystem <NUM> may then determine, for the plurality of objects, corresponding locations within the video frame. For example, composite stream generation subsystem <NUM> may access table <NUM> (as illustrated in <FIG>) and retrieve the object types and/or object descriptions together with object locations. Composite stream generation subsystem <NUM> may then overlay an indicator of each corresponding object at the corresponding location within the video frame. For example, an indicator may be some text and/or graphics (e.g., an icon) indicating each object.

In some embodiments, composite stream generation subsystem <NUM>, may overlay indicators of objects detected by other machine learning models (e.g., from other data streams) onto the base frame. Thus, composite stream generation subsystem <NUM> may retrieve object identifiers and locations of objects associated with other frames of the first set of frames (i.e., from frames of different data streams with matching timestamps). Composite stream generation subsystem <NUM> may then add indicators of those objects to the base data frame.

In some embodiments, prior to adding those indicators, composite stream generation subsystem may determine whether some of the detected objects are the same as the object detected in the base data frame and may also convert the location of the object within the data frame where the object was detected to a location within the base data frame. Thus, composite stream generation subsystem <NUM> may determine that a first machine learning model and a second machine learning model have both detected the same object in a frame of a first data stream and in a frame of a second data stream, respectively. To perform this determination, composite stream generation subsystem <NUM> may compare a type of object detected (e.g., a tank, a missile system, a drone, a helicopter, etc.). Furthermore, composite stream generation subsystem <NUM> may calculate and compare coordinates in a three-dimensional space (e.g., latitude and longitude) of the object detected within the first frame (i.e., by the first machine learning model) and coordinates in a three-dimensional space of the object(s) detected within the second frame (i.e., by the second machine learning model). If the coordinates and/or the object type matches, composite stream generation subsystem <NUM> may determine that the same object was detected in both data streams.

When composite stream generation subsystem <NUM> determines that the two detected objects are the same object, composite stream generation subsystem <NUM> may add a first indicator to a first composite frame indicating that the first object has been detected by both the first machine learning model and the second machine learning model. For example, the indicator may include an indication of the types/names of streams that the object was detected in.

In some embodiments, composite stream generation subsystem <NUM> may determine that the machine learning model associated with the base frame having a particular timestamp did not detect a particular object, but another machine learning model detected that object in a frame having the same particular timestamp but being from a different data stream. In these instances, composite stream generation subsystem <NUM> may overlay an indicator of the object detected in another frame onto the base frame. In some embodiments, the indicator of the object may be an image of the object itself overlayed at the coordinates of the object. For example, a first data stream may be a video data stream and a second data stream may be a thermal imaging data stream. A particular object may not be detected in a video data stream. However, that object may be detected in a thermal imaging data stream. Thus, composite stream generation subsystem <NUM> may overlay that object onto the base frame (e.g., the video frame) in the location on the base frame corresponding to the location of the object in three-dimensional space.

Thus, composite stream generation subsystem <NUM> may determine that a first machine learning model did not detect a first object in a first frame of a first data stream and that a second machine learning model has detected the first object in a second frame of a second data stream. That is, the first frame and the second frame may be part of the same set of frames (i.e., both frames having a matching timestamp). Composite stream generation subsystem <NUM> may determine, based on a first location associated with the first object in the second frame, a second location of the first object in the second frame. That is, the first location and the second location have matching geographic coordinates. For example, each data stream may have associated meta data describing the position (e.g., position of the camera), direction of the recording and recording settings of the recording device. Based on that information, composite stream generation subsystem <NUM> may determine three-dimensional coordinates of each frame and then determine the position of any particular object within the three-dimensional space. Based on those positions, composite stream generation subsystem <NUM> may translate a position with the first frame to a position within the second frame. Thus, composite stream generation subsystem <NUM> may generate a composite frame from the first frame and, in some embodiments, generate an indicator of the object on the base frame at the location of the object. In some embodiments, composite stream generation subsystem <NUM> may overlay an image of the first object onto the composite frame at the second location. For example, composite stream generation subsystem <NUM> may identify coordinates of the object within a thermal imaging frame corresponding to an object (e.g., a person) and extract a portion of the image from that frame. Composite stream generation subsystem <NUM> may then add the extracted image onto the base frame at the appropriate location (e.g., the translated location).

In some embodiments, composite stream generation subsystem <NUM> may identify clusters of objects (e.g., two or more objects in the same location) that may trigger an alert. For example, one machine learning model may detect a person and another machine learning model may detect a weapon in the same location. Based on these two objects being within the same location, composite stream generation subsystem <NUM> may generate an alert. The alert may be displayed within the composite frame in the vicinity of the objects. For example, composite stream generation subsystem <NUM> may generate for display a red icon and/or a red box around the cluster of objects.

Thus, in some embodiments, composite stream generation subsystem <NUM> may determine that a first location in three-dimensional space of a first object detected by a first machine learning model of the plurality of machine learning models matches a second location within the three-dimensional space of a second object detected by a second machine learning model of the plurality of machine learning models. To perform this operation, composite stream generation subsystem <NUM> may calculate a three-dimensional location of each object from the data received from each machine learning model. Composite stream generation subsystem <NUM> may store the location in a data structure associated with a corresponding object.

Composite stream generation subsystem <NUM> may determine that the first object is associated with a first type and the second object is associated with a second type. For example, composite stream generation subsystem <NUM> may determine, based on object identifier of each object, a corresponding object type. In some embodiments, the object type may be stored in the data structure corresponding to the object. Composite stream generation subsystem <NUM> may then determine, based on type data, that a combination the first type and the second type indicates an alert condition. For example, composite stream generation subsystem <NUM> may access a database storing identifiers of object clusters (e.g., combinations of object types) that have been added to the system for generating alerts. Composite stream generation subsystem <NUM> may compare the object types associated with the first object and the second object with the data stored in the database (e.g., combinations of types). Based on determining that the combination of the first type and the second type matches a combination in the database, composite stream generation subsystem <NUM> may determine that an alert should be generated. In addition, composite stream generation subsystem <NUM> may extract (e.g., from the database) alert information associated with a particular cluster (e.g., combination of object types). For example, the database may store alert text data and/or alert image data to be displayed with the alert.

Based on determining that the combination the first type and the second type indicates the alert condition, composite stream generation subsystem <NUM> may generate for display, with the composite data stream, an alert associated with the first location. For example, composite stream generation subsystem <NUM> may retrieve alert text and/or alert image data from the database and cause that data to be displayed in the vicinity of the detected cluster of objects.

In some embodiments, an operator of the unmanned vehicles may generate a command for those vehicles to take up positions or adjust positions (e.g., during a mission) based on a type of recording device onboard each vehicle. For example, if a drone includes a thermal imaging camera, it may be useful for that drone to be hovering above a target location so that operators are able to see people in the image. If a land vehicle includes a video camera, it may be useful to position that vehicle in the front of an extraction team so that it is easier to see where the team is going.

Thus, composite stream generation subsystem <NUM> may transmit a payload request to each unmanned vehicle of the plurality of unmanned vehicles. Each payload request may query the corresponding unmanned vehicle for an identifier of recording device onboard. For example, each payload request may ask to return a device type. Composite stream generation subsystem <NUM> may receive, from the plurality of unmanned vehicles, a plurality of identifiers of recording devices onboard each unmanned vehicle. For example, composite stream generation subsystem <NUM> may receive types such as video camera, thermal imaging camera, infrared camera, long range microphone and/or other suitable responses. Composite stream generation subsystem <NUM> may generate, for each unmanned vehicle in the plurality of unmanned vehicles based on a corresponding recording device, a movement command for moving a corresponding vehicle into a position for receiving data of the location. For example, using a target location, composite stream generation subsystem <NUM> may instruct a drone with a thermal imaging camera to move above the target location and record the target location.

<FIG> shows an example computing system that may be used in accordance with some embodiments of this disclosure. In some instances, computing system <NUM> is referred to as a computer system. The computing system may be hosted on an unmanned vehicle, a controller or another suitable device. A person skilled in the art would understand that those terms may be used interchangeably. The components of <FIG> may be used to perform some or all operations discussed in relation to <FIG>. Furthermore, various portions of the systems and methods described herein may include or be executed on one or more computer system(s) similar to computing system <NUM>. Further, processes and modules described herein may be executed by one or more processing systems similar to that of computing system <NUM>.

Computing system <NUM> may include one or more processors (e.g., processors 610a-610n) coupled to system memory <NUM>, an input/output (I/O) device interface <NUM>, and a network interface <NUM> via an I/O interface <NUM>. A processor may include a single processor, or a plurality of processors (e.g., distributed processors). A processor may be any suitable processor capable of executing or otherwise performing instructions. A processor may include a central processing unit (CPU) that carries out program instructions to perform the arithmetical, logical, and input/output operations of computing system <NUM>. A processor may execute code (e.g., processor firmware, a protocol stack, a database management system, an operating system, or a combination thereof) that creates an execution environment for program instructions. A processor may include a programmable processor. A processor may include general or special purpose microprocessors. A processor may receive instructions and data from a memory (e.g., system memory <NUM>). Computing system <NUM> may be a uni-processor system including one processor (e.g., processor 610a), or a multi-processor system including any number of suitable processors (e.g., 610a-610n). Multiple processors may be employed to provide for parallel or sequential execution of one or more portions of the techniques described herein. Processes, such as logic flows, described herein may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating corresponding output. Processes described herein may be performed by, and apparatus can also be implemented as, special purpose logic circuitry (e.g., an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit)). Computing system <NUM> may include a plurality of computing devices (e.g., distributed computer systems) to implement various processing functions.

I/O device interface <NUM> may provide an interface for connection of one or more I/O devices <NUM> to computer system <NUM>. I/O devices may include devices that receive input (e.g., from a user) or output information (e.g., to a user). I/O devices <NUM> may include, for example, a graphical user interface presented on displays (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor), pointing devices (e.g., a computer mouse or trackball), keyboards, keypads, touchpads, scanning devices, voice recognition devices, gesture recognition devices, printers, audio speakers, microphones, cameras, or the like. I/O devices <NUM> may be connected to computer system <NUM> through a wired or wireless connection. I/O devices <NUM> may be connected to computer system <NUM> from a remote location. I/O devices <NUM> located on remote computer systems, for example, may be connected to computer system <NUM> via a network and network interface <NUM>.

Network interface <NUM> may include a network adapter that provides for connection of computer system <NUM> to a network. Network interface <NUM> may facilitate data exchange between computer system <NUM> and other devices connected to the network. Network interface <NUM> may support wired or wireless communication. The network may include an electronic communication network, such as the Internet, a local area network (LAN), a wide area network (WAN), a cellular communications network, or the like.

System memory <NUM> may be configured to store program instructions <NUM> or data <NUM>. Program instructions <NUM> may be executable by a processor (e.g., one or more of processors 610a-610n) to implement one or more embodiments of the present techniques. Program instructions <NUM> may include modules of computer program instructions for implementing one or more techniques described herein with regard to various processing modules. Program instructions may include a computer program (which in certain forms is known as a program, software, software application, script, or code). A computer program may be written in a programming language, including compiled or interpreted languages, or declarative or procedural languages. A computer program may include a unit suitable for use in a computing environment, including as a stand-alone program, a module, a component, or a subroutine. A computer program may or may not correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, subprograms, or portions of code). A computer program may be deployed to be executed on one or more computer processors located locally at one site or distributed across multiple remote sites and interconnected by a communication network.

System memory <NUM> may include a tangible program carrier having program instructions stored thereon. A tangible program carrier may include a non-transitory, computer readable storage medium. A non-transitory, computer readable storage medium may include a machine-readable storage device, a machine-readable storage substrate, a memory device, or any combination thereof. Non-transitory, computer readable storage medium may include nonvolatile memory (e.g., flash memory, ROM, PROM, EPROM, EEPROM memory), volatile memory (e.g., random access memory (RAM), static random access memory (SRAM), synchronous dynamic RAM (SDRAM)), bulk storage memory (e.g., CD-ROM and/or DVD-ROM, hard drives), or the like. System memory <NUM> may include a non-transitory, computer readable storage medium that may have program instructions stored thereon that are executable by a computer processor (e.g., one or more of processors 610a-610n) to cause the subject matter and the functional operations described herein. A memory (e.g., system memory <NUM>) may include a single memory device and/or a plurality of memory devices (e.g., distributed memory devices).

I/O interface <NUM> may be configured to coordinate I/O traffic between processors 610a-610n, system memory <NUM>, network interface <NUM>, I/O devices <NUM>, and/or other peripheral devices. I/O interface <NUM> may perform protocol, timing or other data transformations to convert data signals from one component (e.g., system memory <NUM>) into a format suitable for use by another component (e.g., processors 610a-610n). I/O interface <NUM> may include support for devices attached through various types of peripheral buses, such as a variant of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard.

Embodiments of the techniques described herein may be implemented using a single instance of computer system <NUM>, or multiple computer systems <NUM> configured to host different portions or instances of embodiments. Multiple computer systems <NUM> may provide for parallel or sequential processing/execution of one or more portions of the techniques described herein.

Those skilled in the art will appreciate that computer system <NUM> is merely illustrative and is not intended to limit the scope of the techniques described herein. Computer system <NUM> may include any combination of devices or software that may perform or otherwise provide for the performance of the techniques described herein. For example, computer system <NUM> may include or be a combination of a cloud-computing system, a data center, a server rack, a server, a virtual server, a desktop computer, a laptop computer, a tablet computer, a server device, a client device, a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a vehicle-mounted computer, a Global Positioning System (GPS) or the like. Computer system <NUM> may also be connected to other devices that are not illustrated or may operate as a stand-alone system. In addition, the functionality provided by the illustrated components may, in some embodiments, be combined in fewer components, or distributed in additional components. Similarly, in some embodiments, the functionality of some of the illustrated components may not be provided, or other additional functionality may be available.

<FIG> is a flowchart <NUM> of operations for generating composite frames of objects detected in multiple different types of data streams. The operations of <FIG> may use components described in relation to <FIG>. In some embodiments, data stream processing system <NUM> may include one or more components of computing system <NUM>. At <NUM>, data stream processing system <NUM> receives a plurality of data streams. For example, the data stream processing system may receive the frames from any recording devices 108a-108n, for example, mounted on unmanned vehicles. Data stream processing system <NUM> may receive the frames over network <NUM> using network interface <NUM>. In some embodiments, data stream processing system <NUM> may receive the data streams from data node <NUM>.

At <NUM>, data stream processing system <NUM> determines a type of data captured within each data stream. For example, data stream processing system <NUM> may use one or more processors 610a, 610b and/or 610n to perform the determination. At <NUM>, data stream processing system <NUM> determines a plurality of machine learning models for the plurality of data streams. For example, data stream processing system <NUM> may use one or more processors 610a-610n to perform the determination.

At <NUM>, data stream processing system <NUM> inputs each of the plurality of data streams into a corresponding machine learning model of the plurality of machine learning models. Data stream processing system <NUM> may perform this operation using one or more processors 610a, 610b and/or 610n and one or more application programming interfaces. At <NUM>, data stream processing system <NUM> receives, from each machine learning model, a plurality of object identifiers and a plurality of object locations corresponding to one or more objects detected in each data stream. Data stream processing system <NUM> may use one or more processors 610a, 610b and/or 610n and/or system memory <NUM> to perform this operation.

At <NUM>, data stream processing system <NUM> selects a plurality of sets of frames from the plurality of data streams. Data stream processing system <NUM> may perform this operation using one or more processors 610a, 610b and/or 610n. At <NUM>, data stream processing system <NUM> generates a composite data stream based on the plurality of sets of frames. Data stream processing system <NUM> may use one or more processors 610a, 610b and/or 610n and/or system memory <NUM> to perform this operation.

Although the present invention has been described in detail for the purpose of illustration based on what is currently considered to be the most practical and preferred embodiments, it is to be understood that such detail is solely for that purpose and that the invention is not limited to the disclosed embodiments, but, on the contrary, is intended to cover modifications that are within the scope of the appended claims. For example, it is to be understood that the present invention contemplates that, to the extent possible, one or more features of any embodiment can be combined with one or more features of any other embodiment.

Claim 1:
A system for generating composite images of objects detected in multiple different types of data frames, the system comprising:
one or more processors; and
a non-transitory, computer-readable storage medium storing instructions, which when executed by the one or more processors, cause the one or more processors to perform operations comprising:
receiving a plurality of data streams from a plurality of unmanned vehicles recording data of a location using a corresponding recording device, wherein each of the plurality of data streams comprises a corresponding plurality of frames, and wherein the plurality of data streams comprises data streams having captured data of different wavelength ranges;
determining, based on the different wavelength ranges for each data stream of the plurality of data streams, a type of data captured within each data stream;
determining, for the plurality of data streams based on the type of data captured within each data stream, a plurality of machine learning models for identifying objects within each data stream, wherein each machine learning model of the plurality of machine learning models has been trained to identify one or more objects within a frame of a particular type of data;
inputting each of the plurality of data streams into a corresponding machine learning model of the plurality of machine learning models;
receiving, from each machine learning model, a plurality of object identifiers and a plurality of object locations corresponding to a plurality of objects detected in each data stream of the plurality of data streams;
selecting, from the plurality of data streams, a plurality of sets of frames, wherein each set of frames of the plurality of sets of frames matches a corresponding timestamp;
generating a corresponding composite frame for each timestamp, wherein each composite frame comprises indicators for objects detected by each machine learning model;
generating for display each composite frame;
retrieving, for a first set of frames, a plurality of objects detected by the plurality of machine learning models;
selecting a video frame from the first set of frames;
determining for the plurality of objects corresponding locations within the video frame; and
overlaying an indicator of each corresponding object at the corresponding location within the video frame.