Patent Description:
In the field of image processing, there is an ongoing need for efficient and reliable ways to detect and classify objects of interest within a field of view (e.g., a scene) of an imaging device. Traditional systems combine a machine vision imaging component and a single board computer running rules-based image processing software. These systems are used for simple problems like barcode reading or identifying a particular feature of a known object.

Developments in machine learning have led to systems capable of more complex image analysis. In one approach, various images of an object of interest are collected into a training dataset for training a neural network to classify the object. The training images may be generated with a camera capturing images of the object at various angles and in various setting. A training dataset often includes thousands of images for each object classification, and the generation of the training dataset can be time consuming, expensive and burdensome to produce and update. The trained neural network may be loaded on a server system that receives and classifies images from imaging devices on a network. In some implementations, the trained neural network may be loaded on an imaging system itself allowing for real time object detection.

Simplified machine vision and image classification systems are available for use, but such systems are not capable of running robust trained neural networks and are difficult to adapt to various end-use scenarios. In practical implementations, limitations on memory, processing, communications, and other system resources often lead system designers to produce classification systems focused on particular tasks. A neural network may be trained for particular classification tasks for example and implemented to allow for real time operation within the constraints of the system. However, accuracy for more general and/or complex tasks is still an ongoing issue. For example, an object detection and mapping system may require detection and classification of a broad range of objects in a wide range of settings, while requiring localization accuracy for mapping the detected objects. Such systems encounter numerous issues in practice, including false detections, duplicate detections, and localization errors.

<NPL> discloses video understanding algorithms for automatically detecting people and vehicles, seamlessly tracking them using a network of cooperating active sensors, determining their three-dimensional locations with respect to a geospatial site model, and presenting this information to a human operator who controls the system through a graphical user interface.

There remains a continued need for improved object detection, classification and localization solutions, including systems and methods for use in surveilling an area, detecting and classifying objects, and mapping the results.

Various systems and methods are provided for detecting and tracking objects within a local map, global map, and/or other reference coordinate system. According to the invention, there is provided a system according to claim <NUM> comprising an unmanned vehicle or other sensor system configured with a multi-modal, multi-camera system and adapted to traverse a search area and generate sensor data associated with an object that may be present in the search area.

The unmanned vehicle may include a first logic device configured to fuse the sensor inputs to detect, classify and localize objects, and communicate object detection information to a control system. The object detection information is used to detect, classify and localize objects using probabilities and Kalman filtering to track objects within a local and global map (and/or other reference coordinate systems). In some embodiments, the first logic device includes a plurality of inference models configured to detect object information from sensor data, a local object tracking module configured to track detected objects in a local map, a Kalman filter module configured to track object data over time to resolve errors, duplicates and other issues, and a global object tracking module configured to maintain and optimize a global map, including object classification, localization and orientation.

In various embodiments, the unmanned vehicle comprises an unmanned ground vehicle (UGV), and unmanned aerial vehicle (UAV), and/or an unmanned marine vehicle (UMV), and further comprises a sensor configured to generate the sensor data, the sensor comprising a visible light image sensor, an infrared image sensor, a radar sensor, and/or a Lidar sensor. The present disclosure may also be implemented using other vehicles (e.g., manned vehicle) and/or devices.

The first logic device may be further configured to execute a trained neural network configured to receive a portion of the sensor data and output a location of an object in the sensor data and a classification for the located object, wherein the trained neural network is configured to generate a confidence factor associated with the classification. In addition, the first logic device may be further configured to construct one or more maps based on generated sensor data using the local object tracking module, the Kalman filtering module, and the global object tracker module.

The system may also include a control system configured to facilitate user monitoring and/or control of the unmanned vehicle during operation including a display screen, a user interface, and a second logic device configured to receive real-time communications from the unmanned vehicle relating to detected objects, access the stored object analysis information during a period when the unmanned vehicle is in communication range of the control system, display at least a portion of the object analysis information for the user to facilitate detection and classification of the detected object by the user, and/or update object detection information in accordance with user input. The second logic device may be further configured to generate a training data sample from the updated object detection information for use in training an object classifier, retrain the object classifier using a dataset that includes the training data sample, and determine whether to replace a trained object classifier with the retrained object classifier, the determination based at least in part on a comparative accuracy of the trained object classifier and the retrained object classifier in classifying a test dataset. In addition, the second logic device may be further configured to, if it is determined to replace the trained object classifier with the retrained object classifier, download the retrained object classifier to the unmanned vehicle to replace the trained object classifier; and add the training data sample to the training dataset.

In some embodiments, a detection device, such as an unmanned vehicle, is disclosed and adapted to traverse a search area and generate sensor data associated with objects that may be present in the search area. The generated sensor data is used by a system including object detection inference models configured to receive the sensor data and output object data, a local object tracker configured to track detected objects in a local map, and a global object tracker configured to track detected objects on a global map. The local object tracker is configured to fuse object detections from the object detection inference models to identify locally tracked objects, and a Kalman filter processes frames of fused object data to resolve duplicates and/or invalid object detections. The global object tracker includes a pose manager, configured to track global objects in the global map and update the pose based on a map optimization process. User-in-the-loop processing includes a user interface for displaying and manual editing of detected object data.

A more complete understanding of the disclosure will be afforded to those skilled in the art, as well as a realization of additional advantages thereof, by a consideration of the following detailed description of one or more embodiments. Reference will be made to the appended sheets of drawings that will first be described briefly.

Embodiments of the disclosure and their advantages are best understood by referring to the detailed description that follows.

Aspects of the present disclosure relate generally to object detection and classification, including detecting and tracking objects in three dimensions within a local or global map. Object data is captured using a multi-modal, multi-camera system, for example implemented on a mobile robot (e.g., unmanned ground vehicle (UGV), unmanned aerial vehicle (UAV), unmanned marine vehicle (UMV), etc.) or other device. Data capture with a multi-modal, multi-camera system will likely result in objects being seen by multiple sensors, from multiple perspectives, at the same time. One or more errors in the calibration of these sensors and their transforms could result in object detection errors, such as duplication of detection data.

Kalman filter that takes into account the position of previous object estimates, the quality of the estimate, and an object classification to filter the object detections to remove duplicates and other invalid detections. The Kalman filter may be implemented, for example, as an unscented Kalman filter to estimate object data from a series of sensor measurements to track object's pose and inference probability and merge duplicates with respect to the "base_footprint" frame.

In various aspects of the present disclosure, systems and methods are implemented to calculate object detection probabilities and perform Kalman filtering to track objects in three dimensions within a local map, a global map, and/or other reference coordinate system. The system is implemented as any system or device configured to capture sensor data for object detection and localization. In some embodiments, an unmanned ground vehicle with a fused multi-modal multi-camera system is used to explore an area and map detected object locations. The sensors may be configured to use a common map representation, whether local or global, and the object detections from one or more sensor can be looked up to determine if the object already exists in the map. If a new object is detected, it can be added to the map. If a previously detected object is detected, the new object information (e.g., object location, bounding box, object type, confidence) can be combined with the existing object information to refine the three-dimensional (3D) position, classification, or pose of the object.

In some embodiments, an object detection and localization solution includes a group of components designed to achieve object permanence within the map, while allowing for position updates due to map optimizations. A local object tracker component fuses inference detections across multiple inference models into a common frame ID. In various embodiments, the interference models may include Deep Neural Network (DNN) models, such as Convolutional Neural Network (CNN) models, or other machine learning/artificial intelligence models trained to generate object detection information from sensor data, including one or more of object location, object type, object pose, object bounding box, confidence score, and/or other relevant object information (e.g., information relevant to a particular task/scenario).

In various embodiments, any suitable neural network inference model may be used as long as the model generates a detection vector including an object type, confidence score, and bounding box location. The system uses the Kalman filter to merge multiple locally tracked objects into a single tracked object, based on available information including object type sizes. In some embodiments, the system is configured to track one object in a single location. Use of the Kalman filter also allows the local object tracker to override the inference probabilities from one or more sensors with stronger detections from other sensors. For example, if one camera sees the edge of one object type, with <NUM>% probability, and another camera sees the same object as a different object type, with <NUM>% probability, in the same location, the local object tracker can filter to the more probable object type with an adjusted probability (e.g., less than <NUM>% probability to take into account the detection of the other object type).

In some embodiments, the objects tracked in the local frame of the sensor system are treated as non-persistent observations. As the sensor system moves, the system is configured to minimize bias in the Kalman filter if another object appears in the same spatial relativity as a previous object. The local object tracker is configured to track the objects in the vicinity of the system. The tracked objects may be displayed to a user as markers in a diagnostic viewer user interface, allowing the user to see what is being detected around the sensor system, without having to look at multiple camera streams with typical detection boxes.

In various embodiments, the global object tracker receives the locally tracked objects from the local object tracker and tracks them in the global map frame. The global object tracker also uses a Kalman filter (e.g., unscented Kalman filter) to track object pose and inference probabilities. In some embodiments, the mapping algorithm is configured to periodically run optimizations and update the objects tracked in the global object tracker. It is observed that updating the object locations with map optimizations improves spatial accuracy in tested systems (e.g., otherwise, tracked objects may no longer be spatially accurate after certain map optimization processing).

In some embodiments, the global object tracker is configured to include a pose manager component that tracks the map keyframes. After a new object is identified, the global object tracker uses the pose manager component to get the current map position to translate the locally tracked object into the map frame. The global object tracker may also be configured to regularly update the known list of tracked objects with the pose manager and update the object's pose in the case that optimization has occurred. In some embodiments, the global object tracker pose updates, based on map optimizations, can be manually triggered.

In various embodiments, the system determines an image depth to extract the depth of a detected object, and the system is configured to manage null and NaN values. For example, for images captured using an infrared camera some materials in the scene may be infrared absorbent creating invalid depth readings. There are several image processing techniques that can be utilized to fill holes and repair the depth image, but they often result in changing the depth values of the surrounding pixels. In some embodiments of the present disclosure, the system samples an object's depth at a frame rate (e.g., <NUM> fps) sufficient to allow the system to ignore frames where the depth extraction fails and only those use frames that provide accurate readings. Using this approach simplifies the depth calculation problem and provides accurate depth readings of the objects.

In various embodiments, a human-in-the-loop solution facilitates operator review of the object detections, providing a check to identify false or inaccurate object detections. For example, a system implemented in an unmanned ground vehicle may operate in full autonomous exploration mode where it detects, classifies and localizes objects as described herein (e.g., using a local object tracker, Kalman filter, global object tracker, etc.), and provide a representation of the detected object on an operator terminal for review by the operator to identify false detections.

Embodiments include systems where data describing objects detected by a sensor system are automatically detected, localized and mapped, and may include a human-in-the-loop feature to send object information to a user to approve and/or update various parameters, such as detected type, position, color, and/or other parameters. The system may be configured to record and play back data that was captured by the sensor system during the detection of the object, providing the control station user a view of what led up to the detection of the object and the ability to supplement or correct the object information (e.g., object classification, object position, etc.). The user interface may include a real-time virtual reality, augmented reality or other three-dimensional interface of the 3D map and other telemetry data from the UGV to make it provide the user with additional views to make it easier for the user to approve/refine the collected sensor data.

In various embodiments, a remote device captures sensor data from an environment and performs object detection, classification, localization and/or other processing on the captured sensor data. For example, a system may include an unmanned ground vehicle (UGV) configured to sense, classify and locate objects in its environment, while in wireless communication with a control station that facilitates additional processing and control. The UGV may include a runtime object detection and classification module that is limited by the runtime environment, and may produce false positives, misclassifications, and other errors that are undesirable for a given application. In some embodiments, these and other object detection errors are reduced through use of one or more DNN/CNN inference models, a local object tracking module, a Kalman filtering module, and a global object tracker module, as described herein.

In operation, a user at the control station may be able to correct the data in real time. When there is lack of communications between the UGV and the control station, it may be desirable and convenient to later review and verify/correct the runtime detections of the UGV. Inaccuracies in the detections could lead to errors in the positioning of detected objects in three-dimensional (3D) space. To correct these errors, it may be desirable to have a user of the control station reposition the object within the two-dimensional (2D) and/or 3D map. To do this, the UGV is configured to record data about the detected object (e.g., data sufficient to understand what a detected object looks like visually) and details about the position of the object. In some embodiments, the system is configured to capture the visible images of the object, but also position and location information from one or more sensors, such as point cloud data from a light detection and ranging (Lidar) system, real-world coordinate information from a global positioning satellite (GPS) system, and/or other data from other sensor systems, that applies to the scenario.

The object detection systems and methods described herein may be used in various object detection contexts. For example, the system may include a robot (e.g., a UGV) that senses aspects of an environment, detects objects in the sensed data, and stores related object data in a database and/or local/global map of those object detections. The data may be directed to a control station where a user may approve and/or augment those detections before forwarding to the database.

In some embodiments, the detection of objects is performed using a trained artificial intelligence system, such as a deep neural network (DNN) or convolutional neural network (CNN) classifier that outputs a location of a box around detected objects in a captured image. In some cases, further detail may be desired, such as an understanding of the location of a reference point on the detected object. The systems described herein may be configured to create an augmentation to the data created by the artificial intelligence system, providing the user with an interface to verify or correct a particular point of interest and then transform the data into the correct reference frame. The present disclosure may be used with an artificial intelligence, machine learning, neural network or similar system that identifies an object type and/or class and the object position. In various embodiments, the classifier also outputs a probability indicating a confidence factor in the classification. The system is adaptable to a variety of machine learning frameworks, even systems that produce a large amount of error in the classification, provided the system produces a reasonable starting point for local/global object tracking framework and/or the user to make the correction.

Referring to <FIG>, an example object detection system <NUM> will now be described, in accordance with one or more embodiments. A robot <NUM> with imaging and depth sensors <NUM> is controlled by a controller <NUM> with user interface <NUM> with an interactive display <NUM> that commands the robot <NUM> to explore autonomously, such as through a real-world location <NUM>. While the robot <NUM> is exploring autonomously, it may lose communication with the controller <NUM> for a period of time (e.g., due to distance, obstruction, interference, etc.), during which the controller <NUM> receives no or partial information from the robot <NUM>. While the robot <NUM> is out of range of the controller <NUM>, it continues to collect data about the location <NUM>. In some embodiments, the robot <NUM> is configured to detect an object of interest (e.g., car <NUM>) and place that object in a local map that the robot <NUM> is generating and storing in memory. The robot <NUM> may continue searching for and detecting objects of interest, such as building <NUM> and building <NUM> and/or other objects of various sizes, before returning to within communications range of the controller <NUM>.

After the controller <NUM> re-establishes communications with the robot <NUM>, the controller <NUM> accesses the updated map, which includes the new objects that have been detected, including their positions, type, and confidence level as determined by the robot <NUM>. In some embodiments, a real-time VR view of the 3D map and other telemetry from the robot <NUM> is utilized to make it easier for the user <NUM> to approve/refine using the controller <NUM>. The user input may be used to re-train both the detections and the positioning of the objects and update the model for use on the next detection cycle.

The operation of a user-in-the loop object detection system will now be described in further detail with reference to <FIG>, which illustrates an example system operation in accordance with one or more embodiments. A process <NUM> receives sensor data <NUM> from one or more sensor systems of a remote device, such as a UGV, an unmanned aerial vehicle (UAV), unmanned marine vehicle, or other remote device that includes a sensor for acquiring environmental data, and a processing component for detecting objects in the sensor data. The remote device processing components include a trained inference model <NUM> configured to receive sensor data and output detected objects, object locations, object classifications, and/or a classification confidence factor. In some embodiments, the trained inference model <NUM> includes a convolutional neural network trained on a training dataset <NUM> to detect, classify and locate objects in the sensor data. The trained inference model <NUM> may further include sensor data processing components for one or more of the sensors such as image processing algorithms, radar data processing algorithms, Lidar processing algorithms, and/or other sensor data processing algorithms.

The remote device is configured to store object data, map data and/or user-in-the-loop data in a remote device data storage <NUM>. In one embodiment, the remote device is configured to detect when communications with the controller system are lost and store data for use in a user-in-the-loop correction process. This data may include an identification of object detections and data acquired or produced during the period without communications, addition data collection such pictures and video of the scene preceding, during and after detection, and other data. In other embodiments, the remote device is configured to store the user-in-the-loop data during operation, even when communication with the controller system is active.

After the remote device is back in communication with the controller system, which may include returning to a home location, the user-in-the-loop-correction process <NUM> can access and the data stored on the remote device data storage <NUM>. The user-in-the-loop-correction process <NUM>, identifies for the user objects detected and classified during periods where communications were lost, and provides an interface allowing the user to walk through the detection and make corrections to the classification, location of the object, a point of interest on the object, and/or other collected data. The user interface may include a VR/AR interface allowing the user to explore the captured data and map to aid in the user corrections. The user interface may include a display and control over video of the detection, including forward, reverse, pause, zoom, and other video controls as known in the art. The user interface may also display the local map constructed by the remote device and/or global map constructed by the system. The controller stores the data in a host data storage <NUM>, which may include one or more of a local storage device, a networked storage device, a cloud storage device, or other suitable storage device or system.

After corrections are made by the user to the objects detected during periods without and/or with communications between the remote device and the controller system, the corrected object classification information may be formatted for use in the training dataset <NUM>. In an optional retaining process <NUM>, the control system is configured to retrain the inference model <NUM> using the updated training dataset <NUM> and replace the trained inference model <NUM> if certain criteria are met. In one embodiment, the performance of the updated artificial intelligence training model is tested using a test dataset, and the results are compared against the performance of the current trained inference model <NUM> using the same dataset. The system may be configured, for example, to replace the trained inference model <NUM> if the performance of the updated model is above a certain threshold factor. In the illustrated embodiment, the user accesses the system using a control system <NUM>, that includes a display, user interface, communications components, data processing applications, and user applications.

An example embodiment of a remote device will now be described with reference to <FIG>. In some embodiments, a remote device <NUM> is configured to communicate with a control station <NUM> over a wireless connection <NUM> or other suitable connection. As illustrated, the remote device <NUM> may include an unmanned vehicle, such as a UGV, UAV or UMV or other device configured to travel and collect environmental data under control of user at a control station. In various configurations, the user may control, interact and/or observe the activity of the remote device <NUM> through the control station <NUM> in real-time and/or at a later time to review and correct object detections.

The remote device <NUM> is generally configured to capture and analyze sensor data to detect and classify objects. The remote device <NUM> includes a logic device <NUM>, a memory <NUM>, communications components <NUM>, sensor components <NUM>, GPS <NUM>, mechanical components <NUM>, and a housing/body <NUM>.

Logic device <NUM> may include, for example, a microprocessor, a single-core processor, a multi-core processor, a microcontroller, a programmable logic device configured to perform processing operations, a digital signal processing (DSP) device, one or more memories for storing executable instructions (e.g., software, firmware, or other instructions), a graphics processing unit and/or any other appropriate combination of processing device and/or memory configured to execute instructions to perform any of the various operations described herein. Logic device <NUM> is adapted to interface and communicate with components <NUM>, <NUM>, <NUM>, and <NUM> to perform method and processing steps as described herein.

It should be appreciated that processing operations and/or instructions may be integrated in software and/or hardware as part of logic device <NUM>, or code (e.g., software or configuration data) which may be stored in memory <NUM>. Embodiments of processing operations and/or instructions disclosed herein may be stored by a machine-readable medium in a non-transitory manner (e.g., a memory, a hard drive, a compact disk, a digital video disk, or a flash memory) to be executed by a computer (e.g., logic or processor-based system) to perform various methods disclosed herein.

Memory <NUM> includes, in one embodiment, one or more memory devices (e.g., one or more memories) to store data and information. The one or more memory devices may include various types of memory including volatile and non-volatile memory devices, such as RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically-Erasable Read-Only Memory), flash memory, or other types of memory.

In various embodiments, logic device <NUM> is adapted to execute software stored in memory <NUM> and/or a machine-readable medium to perform various methods, processes, and operations in a manner as described herein. The software includes device control and operation instructions <NUM> configured to control the operation of the remote device, such as autonomous driving, data acquisition, communications and control of various mechanical components <NUM> of the remote device <NUM>. The software further includes sensor data processing logic <NUM> configured to receive captured data from one or more sensor components <NUM> and process the received data for further use by the remote device <NUM>. The software further includes trained object detection models <NUM> configured to receive processed sensor data and output object detection and classification information that may include object location and a confidence factor for the classification.

The memory <NUM> also stores software instructions for execution by the logic device <NUM> for mapping the environment and user-in-loop data acquisition. The mapping system <NUM> is configured to use the sensor data, object detection and classification information, GPS data from GPS <NUM>, and other available information to construct a map of the sensed environment as the remote device <NUM> traverses the area. In some embodiments, the mapping system <NUM> includes a local object tracker, a Kalman filtering module, and a global tracker with pose manager, as disclosed herein. The user-in-loop data acquisition logic <NUM> is configured to detect whether the remote device <NUM> has lost communications with control station <NUM> and store additional data, such as video captured before, during and after object detection, other sensor data relevant to object classification, GPS location data, and other information to aid the user of the control station <NUM> in visually confirming and/or correcting object detection and classification information. The memory <NUM> is further configured to store object detection data <NUM>, map data <NUM> and user-in-the-loop data <NUM>. In some embodiments, the remote device <NUM> includes a separate remote data storage <NUM>.

The sensor components <NUM> include a plurality of sensors configured to sense and capture information about the surrounding environment. The sensor components <NUM> include one or more image sensors for capturing visible spectrum and/or infrared spectrum images of a scene as digital data. Infrared sensors may include a plurality of infrared sensors (e.g., infrared detectors) implemented in an array or other fashion on a substrate. For example, in one embodiment, infrared sensors may be implemented as a focal plane array (FPA). Infrared sensors may be configured to detect infrared radiation (e.g., infrared energy) from a target scene including, for example, mid wave infrared wave bands (MWIR), long wave infrared wave bands (LWIR), and/or other thermal imaging bands as may be desired in particular implementations. Infrared sensors may be implemented, for example, as microbolometers or other types of thermal imaging infrared sensors arranged in any desired array pattern to provide a plurality of pixels.

The sensor components <NUM> may further include other sensors capable of sensing characteristics of one or more objects in the environment, such as a radar system, a Lidar system, or other sensor system. Radar and/or Lidar systems are configured to emit a series of pulses or other signals into the scene and detect pulses/signals that are reflected back off of objects in the scene. The components produce signal data representing objects in the scene and corresponding sensor data processing logic <NUM> is configured to analyze the signal data to identify the location of objects within the scene. Logic device <NUM> may be adapted to receive captured sensor data from one or more sensors, process captured signals, store sensor data in memory <NUM>, and/or retrieve stored image signals from memory <NUM>.

The communications components <NUM> include circuitry and components (e.g., an antenna) for communicating with other devices using one or more communications protocols (e.g., a wireless communications protocol). The communication components <NUM> may be implemented as a network interface component adapted for communication with a network <NUM>, which may include a single network or a combination of multiple networks, and may include a wired or wireless network, including a wireless local area network, a wide area network, a cellular network, the Internet, a cloud network service, and/or other appropriate types of communication networks. The communications components <NUM> are also configured, in some embodiments, for direct wireless communications with the control station <NUM> using one or more wireless communications protocols such as radio control, Bluetooth, Wi-Fi, Micro Air Vehicle Link (MAVLink), and other wireless communications protocols.

GPS <NUM> may be implemented as a global positioning satellite receiver, a global navigation satellite system (GNSS) receiver, and/or other device capable of determining an absolute and/or relative position of the remote device <NUM> based on wireless signals received from space-born and/or terrestrial sources, for example, and capable of providing such measurements as sensor signals. In some embodiments, GPS <NUM> may be adapted to determine and/or estimate a velocity of remote device <NUM> (e.g., using a time series of position measurements).

The mechanical components <NUM> include motors, gears, wheels/tires, tracks and other components for moving remote control across the terrain and/or operating physical components of the remote device <NUM>. In various embodiments, one or more of the mechanical components <NUM> are configured to operate in response to instructions from logic device <NUM>. The remote device <NUM> includes a housing <NUM> that protects the various components of remote device <NUM> from environmental or other conditions as desired.

An example controller system for use with remote device <NUM> will now be described with reference to <FIG>. A controller system <NUM> is configured to communicate with remote device <NUM> across a wireless communications link <NUM>, and/or through a network, such as cloud/network <NUM>, to interface with the remote device <NUM>. In the illustrated embodiment, the controller system <NUM> includes a logic device <NUM>, a memory <NUM>, communications components <NUM>, display <NUM> and user interface <NUM>.

The logic device <NUM> may be include, for example, a microprocessor, a single-core processor, a multi-core processor, a microcontroller, a programmable logic device configured to perform processing operations, a DSP device, one or more memories for storing executable instructions (e.g., software, firmware, or other instructions), a graphics processing unit and/or any other appropriate combination of processing device and/or memory configured to execute instructions to perform any of the various operations described herein. Logic device <NUM> is adapted to interface and communicate with various components of the controller system including the memory <NUM>, communications components <NUM>, display <NUM> and user interface <NUM>.

Communications components <NUM> may include wired and wireless interfaces. Wired interfaces may include communications links with the remote device <NUM>, and may be implemented as one or more physical network or device connect interfaces. Wireless interfaces may be implemented as one or more WiFi, Bluetooth, cellular, infrared, radio, MAVLink, and/or other types of network interfaces for wireless communications. The communications components <NUM> may include an antenna for wireless communications with the remote device during operation.

Display <NUM> may include an image display device (e.g., a liquid crystal display (LCD)) or various other types of generally known video displays or monitors. Under interface <NUM> may include, in various embodiments, a user input and/or interface device, such as a keyboard, a control panel unit, a graphical user interface, or other user input/output. The display <NUM> may operate as both a user input device and a display device, such as, for example, a touch screen device adapted to receive input signals from a user touching different parts of the display screen.

The memory <NUM> stores program instructions for execution by the logic device <NUM> including remote device control/operation instructions <NUM>, user applications <NUM>, model training system <NUM>, data processing system <NUM>, and user-in-the-loop applications <NUM>. Data used by the controller system <NUM> may be stored in the memory <NUM> and/or stored in a separate controller data storage <NUM>. The remote device control and operation instructions <NUM> facilitate operation of the controller system <NUM> and interface with the remote device <NUM>, including sending and receiving data such as receiving and displaying a real-time video feed from an image sensor of the remote device <NUM>, transmitting control instructions to the remote device, and other operations desired for a particular implementation. The user applications <NUM> include system configuration applications, data access and display applications, remote device mission planning applications, and other desired user applications.

The model training system <NUM> generates trained inference models for implementation on the remote device <NUM> and the controller system <NUM>. In some embodiments, one or more aspects of the model training system <NUM> may be implemented through a remote processing system, such as a cloud platform <NUM>, that includes cloud systems <NUM>, data analytics <NUM> modules, and data storage <NUM>. In some embodiments, the cloud platform <NUM> is configured to perform one or more functions of the controller system <NUM> as described herein. The data processing system <NUM> is configured to perform processing of data captured by the remote device <NUM>, including viewing, annotating, editing and configuring map information generated by the remote device <NUM>.

The user-in-the-loop applications <NUM> are configured to facilitate user review, confirmation, refinement, and correction of the object detection data, and improvement to the trained inference models. In some embodiments, the user-in-the-loop applications <NUM> include processes for accessing object detection data and user-in-the-loop data (e.g., UIL data <NUM>) from the remote device <NUM> corresponding to periods without communication between the controller system <NUM> and the remote device <NUM> (and/or other periods as defined by the system, such as periods associated with object classifications that have a confidence factor below a threshold) and facilitating an interactive display providing the user with a visual representation of the object detection data. In some embodiments, the visual representation includes stored video from before, during, and after detection, display of other sensor data, and display of the object detection data. The user may control the display to focus on desired aspects of the object and/or object detection data and input confirmation on object classification, refinement of object classification data (e.g., manual adjusting object location, manually identifying a point of interest on the object, etc.) and corrections to object classification data. In some embodiments, the object detection and classification data (e.g., detection data <NUM>) may be combined with map data <NUM> generated by the remote device <NUM> and/or provided from another source (e.g., through the cloud platform <NUM>). The map data <NUM> may include detection object information, local map data, global map data and/or map data representing a different reference coordinate system as disclosed herein. In some embodiments, the display <NUM> and user interface <NUM> include a virtual reality headset or similar device allowing the user to interact with the data in a three-dimensional space.

In some embodiments, the user-in-the-loop applications <NUM> are further configured to generate labeled training data to the model training system <NUM> representing corrections and/or refinements to the object detection data generated by one or more trained inference models. In one implementation, user corrections and refinements are provided to the model training system <NUM> for consideration in adding to the training and/or testing datasets <NUM>. The model training system <NUM> is configured to compare training results with and without the user corrections. If the accuracy of the inference model is determined to be improved by including of the new training data, then the new training data is added to the training dataset and the model training system <NUM> generates an updated inference model to replace the object detection model implemented by the remote device <NUM>.

Referring to <FIG>, an example a neural network that may be used to generate trained training models will be described, in accordance with one or more embodiments. The neural network <NUM> is implemented as a deep neural network, convolutional neural network or other suitable neural network that receives a labeled training dataset <NUM> to produce object detection information <NUM> for each data sample. The training dataset represents captured sensor data associate with one or more types of sensors, such as infrared images, visible light images, radar signal data, Lidar signal data, GPS data, and/or other data used by the remote device <NUM>. For object classification in images, the images may comprise a region of interest from a captured image that includes an object to be identified.

The training includes a forward pass through the neural network <NUM> to produce object detection and classification information, such as an object location, an object classification, and a confidence factor in the object classification. Each data sample is labeled with the correct classification and the output of the neural network <NUM> is compared to the correct label. If the neural network <NUM> mislabels the input data, then a backward pass through the neural network <NUM> may be used to adjust the neural network to correct for the misclassification. Referring to <FIG>, a trained neural network <NUM>, may then be tested for accuracy using a set of labeled test data <NUM>. The trained neural network <NUM> may then be implemented in a run time environment of the remote device to detect and classify objects.

Referring to <FIG>, an example object detection system will now be described in accordance with one or more embodiments. The object detection system <NUM> may be implemented in a remote device or other system as described herein. The object detection system <NUM> includes a plurality of sensors <NUM> configured to sense objects in a scene <NUM>, a plurality of data processors <NUM> configured to receive the sensor data and transform the raw data into form that is useable by the system <NUM>, and object detection classification logic <NUM> configured to detect objects, classify objects, provide an associated confidence factor, determine an object location, and/or produce other desired object data.

The sensors <NUM> may include any sensor or device that is capable of sensing environmental data related to the scene <NUM> and producing corresponding data that assists in generating desired object information used herein. In the illustrated embodiment, the sensors include a visible light camera <NUM>, an infrared camera <NUM>, a Lidar system <NUM>, a radar system <NUM> and other sensors <NUM>. Each of the sensors <NUM> produces raw data that is transformed using appropriate data processing components into a format that useable by the object classification system. In the illustrated embodiment, for example, the data processors <NUM> include an image processor <NUM>, a thermal image processor <NUM>, a Lidar signal processor <NUM>, a radar signal processor <NUM>, and a sensor data processor <NUM> which corresponds to another sensor type, as needed. In one or more embodiments, the data processors <NUM>, may perform addition data manipulation, including feature extraction for input into the object detection and classification logic <NUM>.

The object detection and classification logic <NUM> includes one or more trained models <NUM> and (optionally, as needed) object detection logic <NUM> and object classification logic <NUM> to perform additional object detection and classification operations that are more efficiently and/or more accurately performed outside of the trained models <NUM>. For example, object detection for some sensor types may be performed using background learning algorithms, motion detection algorithms, and other suitable algorithms. In various embodiments, the data from individual sensors may be processed separately (e.g., through separate trained AI models) and/or data from two or more sensors be combined through a fusion processor to produce a single classification.

Referring to <FIG>, an example operation of object detection and classification in a remote device using user-in-the-loop processing will now be described in accordance with one or more embodiments. An object detection and classification process <NUM> starts by capturing sensor data associated with a scene, in step <NUM>. The data includes at least one image of all or part of the scene to facilitate user review and correction of object detection data. Next, in step <NUM>, the system analyzes the received sensor data and performs object detection and classification. If there are active communications with a control station (step <NUM>), then relevant data is transmitted to the control station for real-time monitoring by the user (step <NUM>). For example, a live video feed may be sent to the controller, allowing the user to provide real-time control of the remote device. Similarly, the remote device processes commands and data received from the controller, in step <NUM>. The remote device also performs autonomous actions as required. The remote device updates its map and stored data in step <NUM> to capture relevant data captured or detected by the remote device.

Referring back to step <NUM>, if there is not active communication with the controller (e.g., if the remote device is out of range), then the remote device stores data for use in user-in-the-loop object detection processing (step <NUM>) after communications have been reestablished with the control station. The stored data may include storing a stream of sensor data from a period prior to object detection through a period after object detection, including recorded video of the object detection, and other data as appropriate. The remote device then performs autonomous actions in accordance with its mission (step <NUM>). The map and stored data are updated as needed in order to capture relevant data captured or detected by the remote device (step <NUM>).

Referring to <FIG>, an example operation of object detection and classification in a control system using user-in-the-loop processing will now be described in accordance with one or more embodiments. A user-in-the-loop process <NUM> starts by accessing object detection data from the remote device, in step <NUM>. This step may take place during a time when the remote device is in range for communications, when the remote device returns home, through a network, or through another method. Next, the controller identifies object candidates for review, refinement and/or correction, in step <NUM>. In some embodiments, the controller tracks the time when the remote device is out of range and identifies corresponding activity. The remote device may identify activity associated with periods without communication by tracking time periods without communication, tagging data acquired during the period without communication, and/or through other approaches suitable for identifying the relevant data. In some embodiments, the remote device may identify other objects for review, such as detected objects that have a low (e.g., compared to a threshold) classification confidence.

In step <NUM>, the controller generated an interactive display to aid the user in visualizing, confirming, refining, and/or correcting the object data. The display is generated using the user-in-the-loop data stored by the remote device, map data generated by the remote device and other available data (e.g., GPS data, satellite images, etc.). In some embodiments, the display includes a video of the object detection with controls (e.g., pause, forward, back, zoom) to provide the user the ability for focus on areas of interest. The display may further include sensor data or other data generated by the remote device (e.g., active radar data associated with the detection), and user input features for identifying areas of interest or inputting corrected classifications and data values. In some embodiments, the display includes a virtual reality environment (e.g., accessible using a VR headset) allowing the user to virtually traverse the map and explore the detected object.

If the object detection data is modified, the controller may prepare a labeled data sample, including the sensor data and correct classification information, for use by the AI training process, in step <NUM>. In some embodiments, the controller is configured to identify the types of modifications that should be forwarded, such as object misclassifications, and which types of modifications should be ignored in AI training (e.g., slight adjustment to an object location). In step <NUM>, the controller updates the AI training models as appropriate and downloads the updated trained AI models to the remote device. In one embodiment, the AI training system tests new trained data samples received through the user-in-the-loop process to determine whether the new sample will improve or reduce the accuracy of the trained AI model. If a determination is made to add the sample to the training dataset, the AI training system generates a new trained AI model and downloads it to the remote device.

In various embodiments, the systems and methods disclosed herein use probabilities and Kalman filtering (e.g., unscented Kalman filtering) to track objects in three dimensions, within a local and global map (or other reference coordinate system), with a device (e.g., unmanned ground robot) including a multi-modal, multi-camera system. With a multi-modal, multi-camera system objects will likely be seen by multiple cameras, from multiple perspectives, at the same time. Errors in the data (e.g., due to one or more errors in the calibration of those sensors and their transforms) could result in duplication of detection data. To address these issues, systems and methods of the present disclosure take into account the position of the previous estimates, the quality of the estimate, and/or the type of object, when adding object data to a map.

Example embodiments of object tracking in a local and global map will now be described with reference to <FIG>. <FIG>, illustrates an example process <NUM> for object tracking within a local or global map, in accordance with one or more embodiments. The system may include, for example, an unmanned vehicle with a multi-modal, multi-camera system to capture sensor data from a scene, a manned vehicle or device, a mobile system, a system for processing data received from a multi-modal, multi-camera system, an operator terminal, and/or other processing systems. In step <NUM>, the system captures sensor data associated with a scene, including at least one image of the scene. In various embodiments, the system captures frames of sensor data at a rate of multiple frames per second (e.g., <NUM> fps).

In step <NUM>, the system analyzes received sensor data and performs object detection, classification, and/or localization of detected objects in three-dimensional space. In various embodiments, each sensor of the system is part of a transformation tree that is relative to a root node (e.g., relative to the orientation of the device). In various embodiments, the fusion and analysis of sensor data is performed by a local object tracker component, which includes one or more neural networks configured for object detection, classification, confidence determinations, depth, location, and/or other suitable object information.

Because the group of cameras (and other sensors) use a common map representation, both local and global, the detections from each camera can be searched to determine if they already exist in the map, within some 3D positional threshold (step <NUM>). If the detected objects don't exist, they can be added to an object storage (step <NUM>), but if they do, the quality of the estimate can be combined with the existing probability estimate and refine the 3D position or pose of the object. The stored detection object information may include, for example, object classification, confidence, bounding box, location, etc..

With a multi-modal, multi-camera system (e.g., on a mobile robot), objects will likely be seen by multiple cameras and/or sensors, from multiple perspectives, at the same time, and error(s) in the calibration of those sensors and their transforms could result in duplication of detection data and/or differences in object classification and localization information. In step <NUM>, the system applies a Kalman filter process to take into account previous object detection estimates (e.g., object data from previous frames and/or map storage <NUM>), the quality of the estimates, the type of object detected, and/or other available information. In step <NUM>, the system removes duplicates and/or invalid detections and synthesize the data into updated object information. In step <NUM>, the system stores the updated object data in the map storage <NUM>.

Referring back to step <NUM>, if the detected object is new (e.g., the location of the object is not within a proximity threshold of a previously detected object), then the new object information added in step <NUM> to the object data and map storage <NUM>.

In various embodiments, the object detection and localization systems and methods are organized into three main logical components. An example processing system is illustrated in <FIG>. The logic device <NUM> may represent any processing system disclosed herein including an unmanned vehicle, the host system or another processing system. The components are configured to achieve object permanence within the map(s), allowing for position updates due to map optimizations.

The logic device <NUM> includes a plurality of object detection inference models <NUM> configured to detect objects from the sensors <NUM>. In various embodiments, the object detection inference models <NUM> may be implemented as a plurality of CNN and/or DNN object detection inference models, and any suitable CNN/DNN may be used that provides a detection vector or similar data output comprising an object type, confidence score, and location (e.g., a bounding box). Different CNN/DNN implementations can improve the detections but the overall functionality of the tracking system in this disclosure can operate with any suitable CNN/DNN implementation, including implementations that the generate results with greater error and/or lower accuracy.

The local object tracker <NUM> fuses inference detections across multiple inference models into a common frame ID using the Kalman filter <NUM>. The Kalman filter <NUM> (e.g., unscented Kalman filter or other suitable filter) is used to track an object's pose and inference probability and merge duplicates with respect to the "base_footprint" frame. With the system including several DNNs, for example, detecting objects around the system, there is a chance that two cameras will detect the same object from different perspectives. The Kalman filter is configured to merge multiple locally tracked objects into a single tracked object, based on object type sizes. In some embodiments, the system operates based on an assumption that the system will only see one object in a particular location.

The system is also configured in some embodiment to allow the local object tracker <NUM> to override the inference probabilities with stronger detections from other cameras. For example, if one camera sees the edge of one object type, with <NUM>% probability, and another camera/sensor sees the same object as a different object type, with <NUM>% probability, in the same location, the local object tracker will filter to the new object type with slightly less than <NUM>% probability. The local object data and local may data may be stored in a storage device <NUM>, which may include a local storage, network storage, cloud storage, or other suitable storage devices.

In various embodiments, the objects tracked in the local frame are treated as non-persistent observations. Because the system can move while sensing objects, the system doesn't create a bias in the Kalman filter if another object appears in the same spatial region as a previous object. The local object tracker is configured to indicate when an object might be in the vicinity of the system, within some threshold range.

In some embodiments, a human-in-the-loop implementation provides a user interface (UI) in the form of a diagnostic viewer allowing an operator to confirm object detections. Since the objects are displayed as markers in the diagnostic viewer, it allows the user to see what is being detected around the system, without having to look at multiple camera streams with typical detection boxes. The diagnostic viewer UI allows multiple viewing options, including in various embodiments each individual camera view, with or without detection boxes, a local dynamic view of the current detections, shown in 3D on a map around the system, a cached global view of past detections, shown in 3D on a map of places that the system has been, and a combined view of both local and global maps. Any of these layers can be shown or hidden, depending on the level of detail that is needed. For the map views, 3D detections boxes, sized to match the detected object's maximum volumetric boundary, are shown to represent the 3D position of each object. In addition, video frames from each individual camera stream, including RGB, thermal, and stereo depth images can be displayed in the map view, projected out from their relative transform and shown at the far clipping plane of the camera frustum. The multimodal cameras can be combined into an aggregate image, overlaid onto each other or each one displayed individually, depending on the desired view. Displaying objects on a map provides a way to abstract the data for the user and make it easier for the user to visualize the 3D position.

<FIG> illustrates an example of a user interface <NUM> displaying local object tracker results of a 3D inference with depth camera. In this example, a local coordinate system <NUM> is used, and a bounding box <NUM> is displayed and classified with a degree of probability (e.g., a drill). The user interface <NUM> may further include a visible image <NUM> and/or video of the captured object to aid the user in the review. Object data <NUM> may also be displayed for the user, including distance, shape, yaw, pitch, tolerances, time, as well as user controls <NUM>.

In practice has been observed that tracking objects globally in the map frame may be affected by inaccurate map optimizations, which can lead to inaccurate object reports. In addition, in practice different DNNs for different sensors can produce conflicting detections, and the system may be tuned to give one or more DNN more priority over others. In some embodiments, when the system operates in full autonomous exploration mode, it will find objects, and it is useful to provide the operator with a way to detect false detections. A proposed solution is to accompany a screen shot of the object for review by the operator as shown in <FIG>.

<FIG> illustrates an example view <NUM> of local object tracker data showing 3D object inference data fused from multiple depth cameras/sensors. The 3D view includes a local coordinate system <NUM>, an object depicted by a bounding box <NUM>, and other point cloud data gathered <NUM> from the various sensors.

<FIG> illustrates an example of a falsely classified detection of an object. A user interface <NUM> presents a 3D view of the local object data, including a local coordinate system <NUM>, a bounding box <NUM> depicting an object location, and a classification and confidence score <NUM>. A 2D representation is also available to provide the user with a visible image <NUM> of the object/scene. The user may review objects through the user interface to delete, correct and/or confirm the object data.

Referring back to <FIG>, the global object tracker <NUM> receives the locally tracked objects from the local object tracker <NUM> and tracks the objects in the global map frame. The global object tracker <NUM> may also use an unscented Kalman filter <NUM> to track object pose and inference probabilities. In various embodiments, global object tracker <NUM> includes a mapping algorithm that runs optimizations frequently and updates the objects tracked in the global object tracker <NUM>. When the map is optimized, the object locations are also updated to maintain spatial accuracy. In some embodiments, the global object tracker <NUM> includes a pose manager <NUM> configured to track the map keyframes. As a new object is identified, the global object tracker <NUM> uses the pose manager <NUM> to get the current map position to translate the locally tracked object into the global map frame. The global object tracker <NUM> also regularly updates a known list of tracked objects with the pose manager <NUM> and updates the object's pose in the case that optimization has occurred. Optionally, the global object tracker <NUM> pose updates, based on map optimizations, can be triggered through other criteria, such as via manual user input. The global object data and global map data may be stored in a storage device <NUM>, which may include a local storage, network storage, cloud storage, or other suitable storage devices.

In various embodiments, the depth image is used to extract the depth of an object, however, care is taken to manage null and NaN values, as some materials are IR absorbent and will create invalid depth readings. <FIG> illustrates example infrared data <NUM> with invalid depth data. In some cases, an object may be detected that includes infrared absorbing material that generates invalid depth data <NUM> for the image. There are several image processing techniques that can be utilized to fill holes and repair the depth image, but they often result in changing the depth values of the surrounding pixels. In the present disclosure, an object's depth is sampled at a frame rate (e.g., 15fps) that allows for multiple object detections at a plurality of perspectives. In practice, it has been observed that the high frame rate allows the system to ignore frames where the depth extraction failed and wait for accurate depth readings in subsequent frame. This approach simplifies the depth problem and provides accurate depth readings of the objects.

Where applicable, various embodiments provided by the present disclosure can be implemented using hardware, software, or combinations of hardware and software. Also, where applicable, the various hardware components and/or software components set forth herein can be combined into composite components comprising software, hardware, and/or both. Where applicable, the various hardware components and/or software components set forth herein can be separated into sub-components comprising software, hardware, or both.

Claim 1:
A system (<NUM>) comprising:
a mobile system (<NUM>) comprising a multi-modal, multi-camera system configured to capture (<NUM>) images from multiple perspectives in a local area (<NUM>) traversed by the mobile system;
a plurality of object detection inference models (<NUM>; <NUM>; <NUM>) configured to analyze the captured images from each camera (<NUM>...<NUM>; <NUM>) to detect, classify, and localize objects therein and output associated object data comprising information associated with detected objects; and
a local object tracker (<NUM>) configured to track detected objects in a local map (<NUM>; <NUM>) associated with the mobile system based on the object data and update object data across a plurality of detections using Kalman filtering (<NUM>; <NUM>) to remove duplicates.