Using light detection and ranging (LIDAR) to train camera and imaging radar deep learning networks

Disclosed are techniques for annotating image frames using information from a light detection and ranging (LiDAR) sensor. An exemplary method includes receiving, from the LiDAR sensor, at least one LiDAR frame, receiving, from a camera sensor, at least one image frame, removing LiDAR points that represent a ground surface of the environment, identifying LiDAR points of interest in the at least one LiDAR frame, segmenting the LiDAR points of interest to identify at least one object of interest in the at least one LiDAR frame, and annotating the at least one image frame with a three-dimensional oriented bounding box of the at least one object of interest detected in the at least one image frame by projecting the three-dimensional oriented bounding boxes from the at least one LiDAR frame to the at least one image frame using cross-calibration transforms between the LiDAR sensor and the camera.

INTRODUCTION

1. Technical Field

Aspects of this disclosure relate generally to autonomous or semi-autonomous driving techniques, and more specifically, to using light detection and ranging (LiDAR) to train camera and imaging radar deep learning networks.

Modern motor vehicles are increasingly incorporating technology that helps drivers avoid drifting into adjacent lanes or making unsafe lane changes (e.g., lane departure warning (LDW)), or that warns drivers of other vehicles behind them when they are backing up, or that brakes automatically if a vehicle ahead of them stops or slows suddenly (e.g., forward collision warning (FCW)), among other things. The continuing evolution of automotive technology aims to deliver even greater safety benefits, and ultimately deliver automated driving systems (ADS) that can handle the entire task of driving without the need for user intervention.

There are six levels that have been defined to achieve full automation. At Level 0, the human driver does all the driving. At Level 1, an advanced driver assistance system (ADAS) on the vehicle can sometimes assist the human driver with either steering or braking/accelerating, but not both simultaneously. At Level 2, an ADAS on the vehicle can itself actually control both steering and braking/accelerating simultaneously under some circumstances. The human driver must continue to pay full attention at all times and perform the remainder of the driving tasks. At Level 3, an ADS on the vehicle can itself perform all aspects of the driving task under some circumstances. In those circumstances, the human driver must be ready to take back control at any time when the ADS requests the human driver to do so. In all other circumstances, the human driver performs the driving task. At Level 4, an ADS on the vehicle can itself perform all driving tasks and monitor the driving environment, essentially doing all of the driving, in certain circumstances. The human need not pay attention in those circumstances. At Level 5, an ADS on the vehicle can do all the driving in all circumstances. The human occupants are just passengers and need never be involved in driving.

These and other safety technologies use a combination of hardware (sensors, cameras, and radar) and software to help vehicles identify certain safety risks so they can warn the driver to act (in the case of an ADAS), or act themselves (in the case of an ADS), to avoid a crash. A vehicle outfitted with an ADAS or ADS includes one or more camera sensors mounted on the vehicle that capture images of the scene in front of the vehicle, and also possibly behind and to the sides of the vehicle. Radar systems may also be used to detect objects along the road of travel, and also possibly behind and to the sides of the vehicle. Radar systems utilize radio frequency (RF) waves to determine the range, direction, speed, and/or altitude of the objects along the road. More specifically, a transmitter transmits pulses of RF waves that bounce off any object(s) in their path. The pulses reflected off the object(s) return a small part of the RF waves' energy to a receiver, which is typically located at the same location as the transmitter. The camera and radar are typically oriented to capture their respective versions of the same scene.

A processor, such as a digital signal processor (DSP), within the vehicle analyzes the captured camera images and radar frames and attempts to identify objects within the captured scene. Such objects may be other vehicles, pedestrians, road signs, objects within the road of travel, etc. The radar system provides reasonably accurate measurements of object distance and velocity in various weather conditions. However, radar systems typically have insufficient resolution to identify features of the detected objects. Camera sensors, however, typically do provide sufficient resolution to identify object features. The cues of object shapes and appearances extracted from the captured images may provide sufficient characteristics for classification of different objects. Given the complementary properties of the two sensors, data from the two sensors can be combined (referred to as “fusion”) in a single system for improved performance.

To further enhance ADAS and ADS systems, especially at Level 3 and beyond, autonomous and semi-autonomous vehicles may utilize high definition (HD) map datasets, which contain significantly more detailed information and true-ground-absolute accuracy than those found in current conventional resources. Such HD maps may provide accuracy in the 7-10 cm absolute ranges, highly detailed inventories of all stationary physical assets related to roadways, such as road lanes, road edges, shoulders, dividers, traffic signals, signage, paint markings, poles, and other data useful for the safe navigation of roadways and intersections by autonomous/semi-autonomous vehicles. HD maps may also provide electronic horizon predictive awareness, which enables autonomous/semi-autonomous vehicles to know what lies ahead.

SUMMARY

In an aspect, a method for annotating image frames using information from a LiDAR sensor of a first vehicle includes receiving, from the LiDAR sensor, at least one LiDAR frame representing an environment of the first vehicle, the at least one LiDAR frame comprising a plurality of LiDAR points, receiving, from a camera sensor or an imaging radar sensor of the first vehicle, at least one image frame of the environment of the first vehicle, removing LiDAR points from the at least one LiDAR frame that represent a ground surface of the environment of the first vehicle, identifying LiDAR points of interest in the at least one LiDAR frame, segmenting the LiDAR points of interest to identify at least one object of interest in the at least one LiDAR frame, and annotating the at least one image frame with a three-dimensional oriented bounding box of the at least one object of interest detected in the at least one image frame by projecting the three-dimensional oriented bounding boxes from the at least one LiDAR frame to the at least one image frame using cross-calibration transforms between the LiDAR sensor and the camera sensor or between the LiDAR sensor and the imaging radar sensor.

In an aspect, a method performed by at least one processor of an ego vehicle includes receiving, from an imaging radar sensor of the ego vehicle, at least one radar frame of the environment of the ego vehicle, detecting at least one object of interest in the at least one radar frame, determining a three-dimensional oriented bounding box, a location, and a velocity of the at least one object of interest, and outputting the three-dimensional oriented bounding box, the location, and the velocity of the at least one object of interest.

In an aspect, an apparatus for annotating image frames using information from a LiDAR sensor of a first vehicle includes at least one processor configured to receive, from the LiDAR sensor, at least one LiDAR frame representing an environment of the first vehicle, the at least one LiDAR frame comprising a plurality of LiDAR points, receive, from a camera sensor or an imaging radar sensor, at least one image frame of the environment of the first vehicle, remove LiDAR points from the at least one LiDAR frame that represent a ground surface of the environment of the first vehicle, identify LiDAR points of interest in the at least one LiDAR frame, segment the LiDAR points of interest to identify at least one object of interest in the at least one LiDAR frame, and annotate the at least one image frame with a three-dimensional oriented bounding box of the at least one object of interest detected in the at least one image frame by projecting the three-dimensional oriented bounding box from the at least one LiDAR frame to the at least one image frame using cross-calibration transforms between the LiDAR sensor and the camera sensor or between the LiDAR sensor and the imaging radar sensor.

In an aspect, an ego vehicle includes at least one processor configured to receive, from an imaging radar sensor of the ego vehicle, at least one radar frame of the environment of the ego vehicle, detect at least one object of interest in the at least one radar frame, determine a three-dimensional oriented bounding box, a location, and a velocity of the at least one object of interest, and output the three-dimensional oriented bounding box, the location, and the velocity of the at least one object of interest.

In an aspect, an apparatus for annotating image frames using information from a LiDAR sensor includes means for receiving, from the LiDAR sensor of a first vehicle, at least one LiDAR frame representing an environment of the first vehicle, the at least one LiDAR frame comprising a plurality of LiDAR points, means for receiving, from a camera sensor or an imaging radar sensor, at least one image frame of the environment of the first vehicle, means for removing LiDAR points from the at least one LiDAR frame that represent a ground surface of the environment of the first vehicle, means for identifying LiDAR points of interest in the at least one LiDAR frame, means for segmenting the LiDAR points of interest to identify at least one object of interest in the at least one LiDAR frame, and means for annotating the at least one image frame with a three-dimensional oriented bounding box of the at least one object of interest detected in the at least one image frame by projecting the three-dimensional oriented bounding box from the at least one LiDAR frame to the at least one image frame using cross-calibration transforms between the LiDAR sensor and the camera sensor or between the LiDAR sensor and the imaging radar sensor.

In an aspect, an ego vehicle includes means for receiving, from an imaging radar sensor of the ego vehicle, at least one radar frame of the environment of the ego vehicle, means for detecting at least one object of interest in the at least one radar frame, means for determining a three-dimensional oriented bounding box, a location, and a velocity of the at least one object of interest, and means for outputting the three-dimensional oriented bounding box, the location, and the velocity of the at least one object of interest.

In an aspect, a non-transitory computer-readable medium storing computer-executable instructions for annotating image frames using information from a LiDAR sensor of a first vehicle includes computer-executable instructions comprising at least one instruction instructing at least one processor to receive, from the LiDAR sensor, at least one LiDAR frame representing an environment of the first vehicle, the at least one LiDAR frame comprising a plurality of LiDAR points, at least one instruction instructing the at least one processor to receive, from a camera sensor or an imaging radar sensor, at least one image frame of the environment of the first vehicle, at least one instruction instructing the at least one processor to remove LiDAR points from the at least one LiDAR frame that represent a ground surface of the environment of the first vehicle, at least one instruction instructing the at least one processor to identify LiDAR points of interest in the at least one LiDAR frame, at least one instruction instructing the at least one processor to segment the LiDAR points of interest to identify at least one object of interest in the at least one LiDAR frame, and at least one instruction instructing the at least one processor to annotate the at least one image frame with a three-dimensional oriented bounding box of the at least one object of interest detected in the at least one image frame by projecting the three-dimensional oriented bounding box from the at least one LiDAR frame to the at least one image frame using cross-calibration transforms between the LiDAR sensor and the camera sensor or between the LiDAR sensor and the imaging radar sensor.

In an aspect, a non-transitory computer-readable medium storing computer-executable instructions includes computer-executable instructions comprising at least one instruction instructing at least one processor of an ego vehicle to receive, from an imaging radar sensor of the ego vehicle, at least one radar frame of the environment of the ego vehicle, at least one instruction instructing the at least one processor to detect at least one object of interest in the at least one radar frame, at least one instruction instructing the at least one processor to determine a three-dimensional oriented bounding box, a location, and a velocity of the at least one object of interest, and at least one instruction instructing the at least one processor to output the three-dimensional oriented bounding box, the location, and the velocity of the at least one object of interest.

DETAILED DESCRIPTION

Aspects of the disclosure are provided in the following description and related drawings directed to various examples provided for illustration purposes. Alternate aspects may be devised without departing from the scope of the disclosure. Additionally, well-known aspects of the disclosure may not be described in detail or may be omitted so as not to obscure more relevant details.

Further, many aspects are described in terms of sequences of actions to be performed by, for example, elements of a computing device. It will be recognized that various actions described herein can be performed by specific circuits (e.g., application specific integrated circuits (ASICs)), by program instructions being executed by one or more processors, or by a combination of both. In addition, for each of the aspects described herein, the corresponding form of any such aspect may be implemented as, for example, “logic configured to” perform the described action.

Autonomous and semi-autonomous driving safety technologies use a combination of hardware (sensors, cameras, and radar) and software to help vehicles identify certain safety risks so they can warn the driver to act (in the case of an ADAS), or act themselves (in the case of an ADS), to avoid a crash. A vehicle outfitted with an ADAS or ADS includes one or more camera sensors mounted on the vehicle that capture images of the scene in front of the vehicle, and also possibly behind and to the sides of the vehicle. Radar systems may also be used to detect objects along the road of travel, and also possibly behind and to the sides of the vehicle. Radar systems utilize radio frequency (RF) waves to determine the range, direction, speed, and/or altitude of the objects along the road. More specifically, a transmitter transmits pulses of RF waves that bounce off any object(s) in their path. The pulses reflected off the object(s) return a small part of the RF waves' energy to a receiver, which is typically located at the same location as the transmitter. The camera and radar are typically oriented to capture their respective versions of the same scene.

A processor, such as a digital signal processor (DSP), within the vehicle analyzes the captured camera images and radar frames and attempts to identify objects within the captured scene. Such objects may be other vehicles, pedestrians, road signs, objects within the road of travel, etc. The radar system provides reasonably accurate measurements of object distance and velocity in various weather conditions. However, radar systems typically have insufficient resolution to identify features of the detected objects. Camera sensors, however, typically do provide sufficient resolution to identify object features. The cues of object shapes and appearances extracted from the captured images may provide sufficient characteristics for classification of different objects. Given the complementary properties of the two sensors, data from the two sensors can be combined (referred to as “fusion”) in a single system for improved performance.

To further enhance ADAS and ADS systems, especially at Level 3 and beyond, autonomous and semi-autonomous vehicles may utilize high definition (HD) map datasets, which contain significantly more detailed information and true-ground-absolute accuracy than those found in current conventional resources. Such HD maps may provide accuracy in the 7-10 cm absolute ranges, highly detailed inventories of all stationary physical assets related to roadways, such as road lanes, road edges, shoulders, dividers, traffic signals, signage, paint markings, poles, and other data useful for the safe navigation of roadways and intersections by autonomous/semi-autonomous vehicles. HD maps may also provide electronic horizon predictive awareness, which enables autonomous/semi-autonomous vehicles to know what lies ahead.

Referring now toFIG. 1, a vehicle100(referred to as an “ego vehicle” or a “host vehicle”) is illustrated that includes a radar-camera sensor module120located in the interior compartment of the vehicle100behind the windshield112. The radar-camera sensor module120includes a radar component configured to transmit radar signals through the windshield112in a horizontal coverage zone150(shown by dashed lines), and receive reflected radar signals that are reflected off of any objects within the coverage zone150. The radar-camera sensor module120further includes a camera component for capturing images based on light waves that are seen and captured through the windshield112in a horizontal coverage zone160(shown by dashed lines).

AlthoughFIG. 1illustrates an example in which the radar component and the camera component are collocated components in a shared housing, as will be appreciated, they may be separately housed in different locations within the vehicle100. For example, the camera may be located as shown inFIG. 1, and the radar component may be located in the grill or front bumper of the vehicle100. Additionally, althoughFIG. 1illustrates the radar-camera sensor module120located behind the windshield112, it may instead be located in a rooftop sensor array, or elsewhere. Further, althoughFIG. 1illustrates only a single radar-camera sensor module120, as will be appreciated, the vehicle100may have multiple radar-camera sensor modules120pointed in different directions (to the sides, the front, the rear, etc.). The various radar-camera sensor modules120may be under the “skin” of the vehicle (e.g., behind the windshield112, door panels, bumpers, grills, etc.) or within a rooftop sensor array.

The radar-camera sensor module120may detect one or more (or none) objects relative to the vehicle100. In the example ofFIG. 1, there are two objects, vehicles130and140, within the horizontal coverage zones150and160that the radar-camera sensor module120can detect. The radar-camera sensor module120may estimate parameters (attributes) of the detected object(s), such as the position, range, direction, speed, size, classification (e.g., vehicle, pedestrian, road sign, etc.), and the like. The radar-camera sensor module120may be employed onboard the vehicle100for automotive safety applications, such as adaptive cruise control (ACC), forward collision warning (FCW), collision mitigation or avoidance via autonomous braking, lane departure warning (LDW), and the like.

Collocating the camera and radar permits these components to share electronics and signal processing, and in particular, enables early radar-camera data fusion. For example, the radar sensor and camera may be integrated onto a single board. A joint radar-camera alignment technique may be employed to align both the radar sensor and the camera. However, collocation of the radar sensor and camera is not required to practice the techniques described herein.

FIG. 2illustrates an on-board computer (OBC)200of a vehicle100, according to various aspects of the disclosure. In an aspect, the OBC200may be part of an ADAS or ADS. The OBC200includes a non-transitory computer-readable storage medium, i.e., memory204, and one or more processors206in communication with the memory204via a data bus208. The memory204includes one or more storage modules storing computer-readable instructions executable by the processor(s)206to perform the functions of the OBC200described herein. For example, the processor(s)206in conjunction with the memory204may implement the various neural network architectures described herein.

One or more radar-camera sensor modules120are coupled to the OBC200(only one is shown inFIG. 2for simplicity). In some aspects, the radar-camera sensor module120includes at least one camera212, at least one radar214, and an optional light detection and ranging (LiDAR) sensor216. The OBC200also includes one or more system interfaces220connecting the processor(s)206, by way of the data bus208, to the radar-camera sensor module120and, optionally, other vehicle sub-systems (not shown).

The OBC200also includes, at least in some cases, a wireless wide area network (WWAN) transceiver230configured to communicate via one or more wireless communication networks (not shown), such as an NR network, an LTE network, a GSM network, and/or the like. The WWAN transceiver230may be connected to one or more antennas (not shown) for communicating with other network nodes, such as other vehicle UEs, pedestrian UEs, infrastructure access points, roadside units (RSUs), base stations (e.g., eNBs, gNBs), etc., via at least one designated RAT (e.g., NR, LTE, GSM, etc.) over a wireless communication medium of interest (e.g., some set of time/frequency resources in a particular frequency spectrum). The WWAN transceiver230may be variously configured for transmitting and encoding signals (e.g., messages, indications, information, and so on), and, conversely, for receiving and decoding signals (e.g., messages, indications, information, pilots, and so on) in accordance with the designated RAT.

The OBC200also includes, at least in some cases, a wireless local area network (WLAN) transceiver240. The WLAN transceiver240may be connected to one or more antennas (not shown) for communicating with other network nodes, such as other vehicle UEs, pedestrian UEs, infrastructure access points, RSUs, etc., via at least one designated RAT (e.g., cellular vehicle-to-everything (C-V2X), IEEE 802.11p (also known as wireless access for vehicular environments (WAVE)), dedicated short-range communication (DSRC), etc.) over a wireless communication medium of interest. The WLAN transceiver240may be variously configured for transmitting and encoding signals (e.g., messages, indications, information, and so on), and, conversely, for receiving and decoding signals (e.g., messages, indications, information, pilots, and so on) in accordance with the designated RAT.

As used herein, a “transceiver” may include a transmitter circuit, a receiver circuit, or a combination thereof, but need not provide both transmit and receive functionalities in all designs. For example, a low functionality receiver circuit may be employed in some designs to reduce costs when providing full communication is not necessary (e.g., a receiver chip or similar circuitry simply providing low-level sniffing).

The OBC200also includes, at least in some cases, a global positioning systems (GPS) receiver250. The GPS receiver250may be connected to one or more antennas (not shown) for receiving satellite signals. The GPS receiver250may comprise any suitable hardware and/or software for receiving and processing GPS signals. The GPS receiver250requests information and operations as appropriate from the other systems, and performs the calculations necessary to determine the vehicle's100position using measurements obtained by any suitable GPS algorithm.

In an aspect, the OBC200may utilize the WWAN transceiver230and/or the WLAN transceiver240to download one or more maps202that can then be stored in memory204and used for vehicle navigation. Map(s)202may be one or more high definition (HD) maps, which may provide accuracy in the 7-10 cm absolute ranges, highly detailed inventories of all stationary physical assets related to roadways, such as road lanes, road edges, shoulders, dividers, traffic signals, signage, paint markings, poles, and other data useful for the safe navigation of roadways and intersections by vehicle100. Map(s)202may also provide electronic horizon predictive awareness, which enables the vehicle100to know what lies ahead.

In an aspect, the camera212may capture image frames (also referred to herein as camera frames) of the scene within the viewing area of the camera212(as illustrated inFIG. 1as horizontal coverage zone160) at some periodic rate. Likewise, the radar214may capture radar frames of the scene within the viewing area of the radar214(as illustrated inFIG. 1as horizontal coverage zone150) at some periodic rate. The periodic rates at which the camera212and the radar214capture their respective frames may be the same or different. Each camera and radar frame may be timestamped. Thus, where the periodic rates are different, the timestamps can be used to select simultaneously, or nearly simultaneously, captured camera and radar frames for further processing (e.g., fusion).

FIG. 3illustrates a sensed observation radar grid300. A transmitter (e.g., an array of transmit antennas) of the radar214transmits pulses of electromagnetic RF waves that reflect from object(s) in the transmission path, such as vehicles130and140inFIG. 1. A portion of the electromagnetic RF waves that are reflected from the object(s) are returned to the receiver (e.g., an array of receive antennas) of the radar214, which is usually located at the same site as the transmitter of the radar214.

In an aspect, the radar214may be an imaging radar that uses beamforming to scan horizontally and vertically. Beamforming is a technique used to aim the effective direction of a radar beam by changing the delay between different transmitting antennas so that the signals add constructively in a specified direction. Thus, the radar214may scan horizontally and vertically across the sensing area (e.g., horizontal coverage zone150) by using a transmitter comprising an array of electronically steered antennas.

The returned responses (which may also be referred to as “pings”) measured by the radar214is characterized as an observation (or occupancy) grid340having a plurality of observation cells342. Each cell342represents the measured returned response value at a specific range (r) and angle/azimuth (θ). Each cell342is alternately referred to as a range-angle bin. Features344are extracted from the cells342to determine whether the feature344is an object (e.g., a vehicle130/140). Each feature344within a respective cell342can be identified as having up to four parameters: range, Doppler, azimuth, and elevation. This is called a radar frame. As an example, a feature344within a cell342may be the signal-to-noise ratio (SNR) computed by a constant false alarm rate (CFAR) algorithm. However, it should be understood that other methods may be used to target and identify features344within a cell342.

The processor(s)206generate four dimensional (4D) tensors for features344within cells342of the observation grid340detected by the radar214. The generated tensors represent the range (distance from the vehicle100to the detected feature344), azimuth (the horizontal distance between a feature344and a reference RF ray emitted by the radar214, such as the initial RF ray of a radar sweep), Doppler (indicating the speed of the detected feature344), and elevation (vertical direction from the radar214to the detected feature) of each detected feature344. The processor(s)206then performs object detection, object classification, localization, and property/attribute estimation based on the tensors and undistorted camera frames received from the camera212.

In contrast to images (e.g., from camera212), radar signals (e.g., from radar214) have several unique characteristics. One is specular reflections, in which only certain surfaces on the target having an advantageous orientation reflect the radar signal, which often results in a small number of reflections. A second is non-uniformity, in which objects that appear at the far range of the {range, azimuth} signal space are always smaller and have a different shape than those that appear at closer ranges. This is not the case with camera images.

FIG. 4illustrates exemplary differences between a camera image410and a radar image420of the same scene. The camera image410may have been captured by the camera212, and the radar image420may have been captured by the radar214. The camera212captures raw camera frames of the scene within the viewing area of the camera212(as illustrated inFIG. 1as horizontal coverage zone160). The processor(s)206correct any distortion in the raw camera frames to generate undistorted camera frames that are used for processing with the radar tensors. The camera image410is an example of an undistorted camera frame. Note that the terms “frame” and “image” are used interchangeably herein.

The radar image420is captured and processed as discussed above with respect toFIG. 3. The camera image410and the radar image420are then processed to identify objects within the respective images. Object detection in radar images was discussed above with respect toFIG. 3. Object detection in camera images is well-known in the art and is therefore not further described here for the sake of brevity. Any object detection methods and/or techniques may be used to detect objects in the camera images.

The results of the object detection are various attributes of the detected object(s), including bounding boxes in Cartesian x-y(-z) coordinates that tightly enclose the object(s). In the camera image410, three objects have been detected, each surrounded by a bounding box412,414, and416. In the radar image420, the same three objects have been detected and are surrounded by bounding boxes422,424, and426. As can be seen inFIG. 4, the shapes of the objects/bounding boxes422,424, and426in the radar image420are different sizes and shapes based on whether or not they are below (i.e., bounding box422) or above (i.e., bounding boxes424and426) the midline428of radar image420. In addition, the bounding boxes associated with the closer objects (e.g., bounding box422) are larger than the bounding boxes associated with farther objects (e.g., bounding boxes424and426) due to reception at the radar214of fewer reflected radio waves due to the greater distance between the radar214and the further objects.

Once one or more objects (or none) have been identified in the camera image410, the processor(s)206may use pattern-recognition and/or object recognition algorithms to determine the classification (another attribute) of the object(s) as road signs, traffic barrels, cars, trucks, motorcycles, bicyclists, and pedestrians. The fine pixel resolution of an image enables precise angular localization of recognized objects. Range (another attribute of the object) may be estimated from stereo disparity if two cameras are used. Otherwise, a monocular system can estimate range from expected object size or displacement from the horizon. Object classification for radar images is more difficult, and often relies on correlating the object(s) detected in the radar image (e.g., radar image420) to the object(s) detected in the corresponding (i.e., simultaneously, or nearly simultaneously, captured) camera image (e.g., camera image410).

More specifically, the radar214provides reasonably accurate measurements of object distance and velocity in various weather conditions. However, radar systems typically have insufficient resolution to identify features of the detected objects. The camera212, however, may provide sufficient resolution to identify object features. The cues of object shapes and appearances extracted from the captured images may provide sufficient characteristics for classification of different objects. Given the complementary properties of the two sensors, data from the two sensors can be combined (referred to as “fusion”) in a single system for improved performance.

Further, recent advances in machine-learning techniques have made object-classification systems for both camera images and radar images much more effective. For example, deep neural networks (mathematical functions with many layers of nodes that resemble the connectivity of brain neurons) are now practical to train due to recently developed algorithms and the availability of “big data” image sets. The heavy mathematics can now be applied to every pixel in a video/radar stream in real time due to miniature supercomputers comprised of inexpensive graphics processing units (GPUs).

FIG. 5illustrates an exemplary sensor fusion architecture500, according to aspects of the disclosure. In an aspect, the sensor fusion architecture500may be implemented by the radar-camera sensor module120and/or the OBC200ofFIG. 2. A perception interface510of the sensor fusion architecture500provides inputs to a sensor fusion/real world model (RWM) module530. In the example ofFIG. 5, the perception interface510includes a camera object detection module511(which detects objects in the images captured by the camera212), an optional radar object detection module512(which detects objects based on data from the optional LiDAR sensor216), and an imaging radar object detection module513(which detects objects based on data from the radar214). In order to reduce the data transfer budget, the camera object detection module511extracts the geometric and semantic information from the camera images and sends it to the sensor fusion/RWM module530without the camera images themselves. The perception interface510also includes information about static objects514, drivable space516, and road geometry518(obtained from the HD map(s)520). The information about static objects514may be bounding box information, the same as for vehicles. Information about drivable space516may include a per pixel label indicating whether that pixel is drivable or not. The information about road geometry518may be a spline representation of the lane boundaries. The HD map(s)520for the region of travel (e.g., continental United States, North America, Europe, etc.) may be either pre-loaded or downloaded as needed.

In the example ofFIG. 5, the sensor fusion/RWM module530of the sensor fusion architecture500includes a dynamic object tracking module531that includes a data association module532, a motion model/tracking engine533, and an object management module534. The data association module532identifies which target object(s) in previous camera/radar frames are matched to currently detected object(s) and associates them with each other across the camera/radar frames. The dynamic object tracking module531receives the object detections from the camera object detection module511, the optional radar object detection module512, and the imaging radar object detection module513and generates dynamic object detections (e.g., detections of moving objects, including their path, speed, rotation, etc.). The sensor fusion/RWM module530also includes an occupancy grid manager module535that includes a dynamic object filter module536and an occupancy grid update module537. The occupancy grid manager module535receives the information about the static objects514and the drivable space516from the perception interface510and the occupancy grid update module537updates the occupancy grid information (e.g., which grid boxes are occupied and which are open). Finally, the sensor fusion/RWM module530includes a 3D parametric object management module538and a base path generation module539that receive as input information about the road geometry518and HD maps520(e.g., map(s)202), respectively, and outputs base paths (e.g., the center-of-the-lane path that the host vehicle and/or nearby objects are expected to follow).

The sensor fusion architecture500also includes a positioning engine550(e.g., a GPS, motion sensors (e.g., accelerometer, gyroscope, etc.), etc.) and a transform tree module560that provide further inputs to the sensor fusion/RWM module530. For example, a “sensor coordinate frame” (e.g., a camera coordinate frame, a radar coordinate frame, etc.) as used herein refers to a coordinate system with an origin at the sensor's current position and orientation along the sensor's current axes.

The sensor fusion/RWM module530outputs the dynamic object detections, occupancy grid, and base paths to a planner module540of the sensor fusion architecture500. The planner module540includes a behavior planner module542and a motion planner module544that direct other systems (e.g., braking, accelerations, steering, cruise control, signaling, etc.) of the host vehicle (e.g., vehicle100ofFIGS. 1 and 2) to act (e.g., brake, accelerate, change lanes, etc.) based on the information about the dynamic objects, the occupancy grid, and the base paths received from the sensor fusion/RWM module530. The sensor fusion/RWM module530also outputs to a visualization interface570. The visualization interface570may be connected to a display (e.g., light emitting diode (LED) display, liquid crystal display (LCD), etc., not shown) within the host vehicle (e.g., vehicle100), such as on the dashboard or behind the steering wheel, to provide information to a driver or occupant within the vehicle.

Although the sensor fusion architecture500shown inFIG. 5is primarily directed to reacting to events (e.g., other vehicles changing lanes, turning onto the roadway, braking, etc.) that may occur while the vehicle is driving, as will be appreciated, the sensor fusion architecture500may also receive input from a driver of the vehicle (e.g., direction to change lanes, turn, etc.) and act on those instructions based on the information generated by the sensor fusion/RWM module530. For example, in the case of an instruction to change lanes, the sensor fusion architecture500may first determine that there is a lane to change into and that there are no objects (e.g., vehicles, pedestrians, road signs, etc.) in it before the planner module540directs the vehicle to change lanes as instructed.

FIG. 6illustrates an exemplary object tracking architecture600, according to aspects of the disclosure. In an aspect, the object tracking architecture600may be implemented by the radar-camera sensor module120and/or the OBC200ofFIG. 2, and more specifically, the dynamic object tracking module531of the sensor fusion/RWM module530inFIG. 5. At an input stage610, the object tracking architecture600receives as input positioning information612(e.g., GPS information), camera perception information614(e.g., camera images from camera212), radar data616(e.g., radar frames from radar214), and HD maps618(e.g., HD maps520ofFIG. 5).

At a pre-processing stage620, the object tracking architecture600creates a transform tree621(a binary tree representation of multiple coordinate frames as nodes, and links storing the transformation matrix to move from one coordinate frame to another) from the positioning information612. The pre-processing stage620performs camera object localization622, input sanitization624, and clustering626operations on the camera perception information614and radar data616. The pre-processing stage620uses the HD maps618for creation of a k-D tree628(a space-partitioning data structure for organizing points in a k-dimensional space).

At an associator stack stage630(which performs associations of new observations of target objects with existing tracks associated with those objects), the object tracking architecture600generates multi camera associations632, observation track associations634, imaging radar observation associations636, and camera-radar associations638. At a tracker stage640, the object tracking architecture600performs motion model estimation642of a tracked object and an extended Kalman filter (EKF) based solution644(discussed further below, and referred to as an “EKF tracker”). Finally, at a periodic output stage650, the object tracking architecture600outputs a list of all dynamic objects with respective map associations652. This information is then fed to the planner module540inFIG. 5.

At the preprocessing stage620, input data coming into the sensor fusion module (e.g., sensor fusion/RWM module530) is noisy and needs further preprocessing before it can be sent to the associator stack stage630and the estimation pipeline in the sensor fusion module. Accordingly, the present disclosure provides various techniques to sanitize the input data at the input sanitization624level. Additionally, some of the inputs are processed to estimate additional quantities that can then be later used in the association and estimation pipeline.

The training of deep learning networks (also referred to as “deep neural networks”) for object detection and state estimation based on imaging radar and/or camera input requires large amounts of training data. In the case of generating training data/annotations for camera frames, current techniques are to manually draw bounding boxes on images of cars, trucks, and other vehicles. This is a very labor intensive and relatively expensive work. In the case of imaging radar, generating training data/annotations manually does not seem feasible, as it is difficult for a human to correctly place the bounding box on an imaging radar frame. It can, however, be done indirectly if camera images have been recorded as well and cross calibration transformations (e.g., formula(s) that transform a camera frame and/or a radar frame such that the two sensors appear to have the same location and orientation, and the two frames have the same sensor coordinate frame—cross calibration is captured in six parameters, translation in x, y, and z axes and rotation around x, y, and z axes) between the camera and radar sensors are available.

Camera, radar, and LiDAR have different strengths and weaknesses as sensors. While LiDAR is currently used on autonomous or semi-autonomous vehicles, the goal is to equip vehicles with cameras and radars alone. However, the main advantage of LiDAR over a camera and/or radar is that it provides depth and geometric information. A camera provides azimuth and elevation information, but is poor at providing depth and orientation information, especially at greater distances. It is, however, human-interpretable. Radar provides azimuth, depth, and velocity information, but is poor at providing orientation and size information. Raw imagining radar output is also hard for humans to interpret. LiDAR provides 3D information in the form of a point cloud.

It would be beneficial to automatically annotate large amounts of camera and/or radar data using LiDAR data. These annotations could then be used to train the deep learning networks to perform object detection in camera and radar frames, which would outperform traditional camera and/or radar training methods because, as noted above, manual annotation is slow, expensive, and sometimes difficult.

Accordingly, the present disclosure provides techniques to use LiDAR data to annotate object detections in camera and radar frames captured by a test (ego) vehicle while, for example, the test vehicle is driving down a freeway. The test vehicle is expected to have, in addition to the LiDAR sensor and a high quality GPS sensor for positioning, at least one imaging radar sensor and multiple camera sensors (note that the test vehicle may have enough radar and camera sensors to capture a 360 degree view around the vehicle). In addition, the cross calibration between the LiDAR sensor, the imaging radar, and the cameras is assumed to be known. Because of the cross calibration between the sensors, the bounding boxes around the objects detected in a LiDAR frame can be projected onto the corresponding radar frame and camera frames to generate annotations that can be used to train the respective deep learning networks. In addition to the bounding boxes (which provide the position and dimensions of the detected object(s)), the speed and heading of target objects can also be shared for training.

Ideally, the generated annotations would be of sufficiently high quality that they could be used for training “as is.” However, if they are not, the generated annotations could still be used, just with some imperfections that would be fixed manually. This would still be much more efficient than manually annotating a data set of camera and radar frames from scratch.

FIG. 7illustrates an exemplary LiDAR sensor700(which may correspond to LiDAR sensor216) mounted in a rotatable housing, according to aspects of the disclosure. The LiDAR frame of reference is centered at the LiDAR sensor700with the x-axis pointing to the right, the y-axis pointing forward, and the z-axis pointing vertically upward. The LiDAR sensor700may emit lasers vertically along 32 lines (i.e., directions) of measurement at a frame rate of 10 Hz (i.e., 10 times per second). The 32 lines of measurement may be within a +10 to −30 degree vertical field of view (FOV), meaning there is a 1.33 degree separation between lines. The lasers may be emitted sequentially or in an interleaving pattern, such as 1, 17, 2, 18, 3, 19, etc.

The LiDAR sensor700may rotate 360 degrees horizontally around the z-axis (the horizontal FOV), with the lasers firing once for every 0.15 degrees of revolution (referred to as ticks). The reflections from the lasers are used in the LiDAR firmware to compute the distance to each object detected in the scene along each direction of measurement. A single revolution of the LiDAR sensor700may take 100 ms and provide on the order of 360 degrees divided by 0.15 times 32 distance measurements, or approximately 70,000 distance measurements. For processing, LiDAR data is processed one revolution at a time. The information collected in one revolution is referred to as a LiDAR frame. Thus, there may be up to 70,000 points per LiDAR frame. However, as will be appreciated, there may not be returns for each point.

The range of a LiDAR sensor (e.g., LiDAR sensor700) is around 80 to 100 m, but the point cloud density decreases significantly with distance.FIGS. 8A and 8Billustrate exemplary LiDAR frames800A and800B, respectively, captured at two different time steps, according to aspects of the disclosure. The circular lines are ground returns. There are two objects (e.g., vehicles) in front of the ego vehicle traveling in the same direction as the ego vehicle. As can be seen in the center of the LiDAR frames800A and800B, there is a blind spot created by the ego vehicle surrounding returns from its roof. Slopes and shrubbery on the right are visible, as is the guard rail on the left. Note that the spacing of the LiDAR lines increases rapidly the further away from the center, and correspondingly, the number of LiDAR returns from far away objects also decreases. Note also the intensity difference on the lane markers versus on unmarked ground.

As illustrated inFIG. 8A, detected objects (e.g., vehicles) that are near the ego vehicle are scanned very well. However, as illustrated inFIG. 8B, which is captured 7.3 seconds, or 73 frames (i.e., revolutions), after LiDAR frame800A, the detected objects (e.g., vehicles), now further from the ego vehicle, are harder for the LiDAR sensor to “see.”

FIG. 8Cillustrates a comparison between a camera frame810, a radar frame820, and a LiDAR frame830, according to aspects of the disclosure. Specifically, the same object has been detected in each frame, andFIG. 8Cshows the difference of how that object is represented in each frame.

LiDAR data can be used to obtain the size, position, and yaw information (i.e., the oriented bounding box or OBB) for observed objects (e.g., vehicles). This information can then be used to annotate some large number of radar frames, and the annotated radar frames can be used to train a deep neural network (DNN) to infer this information directly from the radar frames. Training a DNN to infer this information from the radar frames themselves eliminates the need for OBB information from the LiDAR sensor after the DNN is trained.

Similarly, the LiDAR OBB information can be used to annotate a large number of camera frames. Currently, deep learning techniques provide two-dimensional (2D) bounding boxes for detected objects. Using the techniques described herein, a DNN can instead be trained to provide 3D OBBs.

FIG. 9is a method900illustrating the sequence of operations for using LiDAR data to generate annotations, according to aspects of the disclosure. The first stage910of processing LiDAR data, regardless of the annotation goal (e.g., identifying nearby vehicles), is ground extraction from the LiDAR point cloud. This is because the removal of ground points is needed before objects of interest (e.g., vehicles) can be identified. The second stage920is to obtain LiDAR points of interest. The third stage930is to segment the chosen LiDAR points to obtain an object of interest. The fourth stage940is to track detected objects across frames. The fifth stage is to fit 3D OBBs to the objects (e.g., vehicles) detected in the camera/radar frames.

With reference to stage910ofFIG. 9,FIG. 10illustrates a comparison of a camera frame1010and a LiDAR frame1020, in which ground points and non-ground points in the LiDAR frame1020have been identified. In the example ofFIG. 10, the camera frame1010and the LiDAR frame1020have been captured at the same point in time.

The ground extraction method begins by creating a mesh of all LiDAR points, with each point from the point cloud acting as a node in the mesh. In creating the mesh, the nearest neighbors on the same laser contour and adjacent laser contours are identified. More specifically, the nodes (i.e., points) are first traversed along laser contours. Nearest neighbors to either side of each node on the contour that are within a distance tolerance are added as left and right neighbors. Nodes are then traversed radially outward starting from the innermost contour and up and down neighbors are again defined similarly for each node on adjacent contours within a distance threshold. In general, every node has a left neighbor, right neighbor, up neighbor, and down neighbor unless there are no nodes within the distance threshold in one of the directions.

Next, once the mesh is created, the surface gradients of the z coordinate are estimated at each point and points with gradients below a threshold are considered ground candidates. Finally, the ground region is identified locally in the inner laser contour and is grown outward from the first contours across the mesh by labeling a point as a ground point if it has gradient below the gradient threshold and it has a neighboring ground point with a z coordinate that is within a pre-specified z threshold of its own z coordinate. As shown inFIG. 10, the ground points are in gray and form concentric rings around the ego vehicle. At the end of this operation, each point in the LiDAR point cloud is labeled either ground or occupied. The occupied points are considered for object segmentation while the ground points are considered for lane marker detection.

Referring now to stage920ofFIG. 9,FIG. 11illustrates a camera frame1110and a camera projection frame1120. In the camera frame1110, three 2D bounding boxes1112-1116(2D OBBs) have been identified by a vertex-SSD (single-shot detector) based vehicle detector. Note that in the example ofFIG. 11, bounding box1112has falsely identified a vehicle. In the camera projection frame1120, non-ground LiDAR points have been projected onto the image plane. Using (enlarged versions of) the same bounding boxes from the camera frame1110, illustrated as 2D bounding boxes1122-1126, LiDAR points of interest (i.e., LiDAR points corresponding to the 2D bounding boxes in the camera frame1110) are identified. Note that LiDAR points from the background may also get picked up in the 2D bounding boxes1122-1126.

Referring to stage930ofFIG. 9,FIG. 12illustrates an adjacency graph1210derived from the camera projection frame1120ofFIG. 11and a zoomed-in adjacency graph1220of the adjacency graph1210, according to aspects of the disclosure. At this stage, the LiDAR points of interest are segmented into objects by creating the voxel-based adjacency graph1210and finding connected components. In a voxel-based approach, the space is carved into voxels as follows. Consider an axis aligned grid in X, Y, and Z dimensions, and the voxels created by that. If the grid lines had identical and uniform spacing in each dimension, each voxel would be a perfect cube. In general, the spacing may be neither identical nor uniform, so the voxels are cuboids. In any case, note that each cuboid has 26 neighbors, if the adjacent cuboids in each dimension and also along the diagonals, i.e., all cuboids that share a face, edge, or corner with that cuboid, are considered as neighbors. Each non-ground, non-transition LiDAR point is assigned to the cuboid within which it lies. All empty cuboids are discarded, and an edge is drawn between two occupied cuboids if they are neighbors. Thus, an undirected graph is obtained. Connected components (a subgraph in which any two vertices are connected to each other by paths, and which is connected to no additional vertices in the supergraph) of this graph are obtained, and each connected component is considered to represent an independent object. All of the LiDAR points assigned to the voxels of the connected component are together the point cloud for that object.

After segmentation, the correct object needs to be identified. This is done by finding the objects that have the most points inside their respective 2D bounding boxes (e.g., 2D bounding boxes1122-1126).

Referring to stage940ofFIG. 9,FIG. 13illustrates an adjacency graph1310at a first time step (frame1960) and an adjacency graph1320at a second time step (frame2033), according to aspects of the disclosure. Camera based tracking can be used to associate objects (e.g., vehicles130/140) across frames (e.g., frames1960and2033). The dimensions of an object (e.g., a vehicle) can be estimated when the object is near the ego vehicle and the LiDAR scan is good. Those dimensions can then be used in all subsequent frames containing the detected object.FIG. 13illustrates the difference in the number of LiDAR points associated with the same two detected vehicles at a first time step (adjacency graph1310) and a second time step 7.3 seconds later (adjacency graph1320). As shown, there are many fewer LiDAR points associated with the two target vehicles in the second adjacency graph1320(frame2033), where the target vehicles are further from the ego vehicle, than there are in the first adjacency graph1310(frame1960), where the target vehicles are close to the ego vehicle. As such, it is beneficial to use the dimensions of the target vehicles calculated from the adjacency graph1310.

Referring to stage950ofFIG. 9, 3D OBBs can now be fitted to the observed/detected objects (e.g., vehicles).FIG. 14illustrates a camera frame1410and the corresponding LiDAR frame1420, according to aspects of the disclosure. In the example ofFIG. 14, two objects (e.g., vehicles130/140) have been detected and identified by bounding boxes1422and1424using the stages of the disclosed algorithm (i.e., method900) described above. Based on these detections, the camera frame1410can be annotated with 3D OBBs1412and1414, which correspond to bounding boxes1422and1424, respectively.

Although not described in detail herein, the disclosed annotation technique can be applied to radar frames in the same way as described with reference to camera frames. Once annotated, the annotated camera and radar frames can be used to train a DNN that detects objects in camera and radar frames. Once trained, the DNN should be able to detect/identify objects in camera and radar frames without needing assistance (i.e., the annotations) from LiDAR data, thereby rendering the LiDAR sensor unnecessary.

There are various challenges that should be considered in the above-disclosed algorithm (i.e., method900ofFIG. 9), such as the intricacies of tracking vehicles as their range relative to the ego vehicle increases and the corresponding point clouds become more sparse. Even if the algorithm calculates a good dimension estimate of a vehicle (i.e., a 3D OBB), accurately estimating the correct location of the OBB for a vehicle far away with a correspondingly sparse point cloud is an ongoing challenge.FIG. 15illustrates an exemplary adjacency graph1510for a first LiDAR frame captured at a first time step and an exemplary adjacency graph1520for a second LiDAR frame captured at a second time step, according to aspects of the disclosure. InFIG. 15, the x-y axes represent coordinates in the East-North-Up (ENU) frame with respect to a reference base station. Specifically, the x axis corresponds to East and the y axis corresponds to North. Since the ego vehicle will have GPS, it is straightforward to transform the LiDAR point cloud from the LiDAR coordinate frame to an ENU coordinate frame.

As shown inFIG. 15, there are many more LiDAR points associated with the best view of a vehicle (obtained when the object is nearby), as shown in adjacency graph1510, versus the worst view of the same vehicle (obtained when the object is far away), as shown in adjacency graph1520. The disclosed techniques attempt to estimate which corner of the vehicle is being sensed the best, and match the corner of the correct-sized bounding box to that corner. In addition, trajectory smoothing, outlier rejection, and various yaw estimation methods can be performed in the current algorithm.

The disclosed algorithm can fail in some cases. For example, ground removal may not be perfect at far distances. Specifically, some ground points may be grouped with vehicle points, causing incorrect localization. As another example, for some vehicles, the LiDAR system may never get a good “look” at them, so the estimated dimensions may be wrong. This can be addressed by having size priors. For example, a size prior for a car may be a length of 4.5 m, a width of 1.7 m, and a height of 1.6 m. If a car in front of the ego vehicle moves into range but then accelerates out of LiDAR range, the LiDAR sensor may only detect the back of it. However, if the height does not exceed 2 m, for example, the ego vehicle can assume the detected object is a car and apply these size priors. Otherwise, the ego vehicle may determine the object is a truck or something else. As yet another example, tracking can fail at far distances. That is, a tracked object (e.g., vehicle) may be the same object as in a previous frame, but the algorithm may not know it.

To address these issues, more data can be collected and the algorithm and associated parameters can be fine-tuned for, for example, freeway and non-freeway conditions. In addition, an annotation confidence measure can be developed, which would exclude poor frames from the training set, or those frames could be manually annotated.

In an aspect, an occupancy map can be annotated using the disclosed algorithm (i.e., method900ofFIG. 9).FIG. 16illustrates an exemplary occupancy map1600at a given time step (frame1960), according to aspects of the disclosure. The goal is to have the radar and camera sensors infer the occupancy of the environment without assistance from the LiDAR sensor (as future autonomous vehicles will preferably not need to be equipped with LiDAR). As discussed above, the LiDAR sensor can detect the ground and other vehicles. As such, the following labels are possible: (1) label all ground as unoccupied, (2) label target vehicle locations as occupied, and (3) label as unknown the cone beyond the target vehicles. It would also be beneficial to have more detailed labels, such as “drivable space” (e.g., road shoulder versus paved road).

In an aspect, the disclosed algorithm (i.e., method900ofFIG. 9) may include an additional stage of transition detection. This stage may be performed as part of the first stage910of identifying and removing ground points. Specifically, the initial set of ground points (from910ofFIG. 9) is often not exhaustive, as some true ground points with neighboring points on vehicles tend to be incorrectly classified as non-ground. If all such non-ground points are considered occupied points, this could lead to incorrect object size estimation and object orientation estimation in downstream processing. This is avoided by performing a transition point detection step. This involves traversing along each contour while maintaining a finite length first-in-first-out (FIFO) buffer of ground points encountered along the contour. Any non-ground point on the contour with a z coordinate that is within the z-threshold of the median buffer value is labeled as a transition point.

FIG. 17illustrates a first LiDAR frame1710and a second LiDAR frame1720in which a transition has been detected and removed, according to aspects of the disclosure. The first and second LiDAR frames1710and1720may correspond to frame1960in the earlier examples described herein. In the first LiDAR frame1710, a transition has been detected, and in the second LiDAR frame1720, the transition has been removed. This improves the differentiation of the LiDAR points associated with the ground and the detected object.

Incorporating this operation into the method900ofFIG. 9, at the end of stage910, each point in the LiDAR point cloud would be labeled either ground, transition, or occupied. The occupied points are considered for object segmentation while the ground points are considered for lane marker detection. Transition points are not used for either processing.

In an aspect, object detections and annotations from multiple vehicles can be combined. For example, where two test vehicles are being driven in close proximity, they are likely to “see” the same target vehicles, but from slightly different perspectives. As such, the annotations generated according to method900by one vehicle can be used to augment the annotations generated according to method900by the other vehicle(s).

More specifically, the disclosed techniques aim at generating annotations (e.g., location, orientation, velocity, and dimensions) for objects (e.g., vehicles) detected in LiDAR frames to train deep learning neural network-based object detection for imaging radar or camera. Both camera and imaging radar may have a range of 200-300 m, which is required for autonomous highway driving. However, this range is not matched by current LiDAR sensors, and is even more challenging for more recent LiDAR sensors focusing on range. To work around this LiDAR limitation, LiDAR data from more than one test vehicle can be collected, and these test vehicles, each equipped with a full suite of sensors (LiDAR, camera, imaging radar, and high precision GPS), could be spaced apart, for example, 50 to 100 m, while driving.

After collecting data from at least two test vehicles, each equipped with a full suite of sensors, there would be at least two separate data sets, including two sets of LiDAR measurements, one from each vehicle. To merge the LiDAR measurements, they must be in the same coordinate system. Therefore, both sets of LiDAR measurements are projected into the ENU frame using the GPS positioning systems on the vehicles with LiDAR sensors. Since LiDAR sensors use rotating beams (see, e.g.,FIG. 7), the LiDAR point cloud measurements made by the two or more LiDAR sensors would not be obtained at the same time. From an algorithmic standpoint, it is therefore easier to merge LiDAR measurements after the annotations have been generated for the individual LiDAR sensors, as opposed to merging multiple LiDAR point clouds in an ENU frame and then running the LiDAR annotation algorithm (i.e., method900ofFIG. 9) once.

Target objects (e.g., vehicles) within range of both LiDAR sensors will overlap, while target objects in range of just one sensor will appear only once. To address this, the algorithm first concatenates target object annotations from both test vehicles, detects overlapping annotations, and removes the annotation(s) with smaller dimensions. For example, if a LiDAR sensor only saw, for example, the back of a vehicle, the annotation algorithm would likely underestimate the target vehicle's length. After merging, the 3D bounding box can be projected from the ENU frame to the camera or imaging radar coordinate frame of either vehicle.

FIG. 18illustrates an exemplary method1800for annotating image frames using information from a LiDAR sensor (e.g., LiDAR sensor216), according to aspects of the disclosure. The method1800may be performed by one or more processors to train an object-detection neural network.

At1802, the one or more processors receive at least one LiDAR frame (e.g., captured by LiDAR sensor216) representing an environment of a first vehicle (e.g., vehicle100). The at least one LiDAR frame may comprise a plurality of LiDAR points (i.e., a “point cloud”). At1804, the one or more processors receive at least one image frame (e.g., captured by camera212or radar sensor214) of the environment of the first vehicle. At1806, the one or more processors remove LiDAR points from the at least one LiDAR frame that represent a ground surface of the environment of the first vehicle, as described above with reference to910ofFIG. 9. At1808, the one or more processors identify LiDAR points of interest in the at least one LiDAR frame, as described above with reference to920ofFIG. 9. At1810, the one or more processors segment the LiDAR points of interest to identify at least one object of interest (e.g., vehicle130,140) in the at least one LiDAR frame, as described above with reference to930ofFIG. 9. At1812, the one or more processors optionally track the at least one object of interest across the at least one LiDAR frame and at least one subsequent LiDAR frame, as described above with reference to940ofFIG. 9. At1814, the one or more processors annotate the at least one image frame with a three-dimensional oriented bounding box of the at least one object of interest detected in the at least one image frame by projecting the three-dimensional oriented bounding boxes from the at least one LiDAR frame to the at least one image frame using cross-calibration transforms between the LiDAR sensor216and the camera212or between the LiDAR sensor216and the imaging radar sensor214, as described above with reference to950ofFIG. 9.

FIG. 19illustrates an exemplary method1900according to aspects of the disclosure. The method1900may be performed by at least one processor (e.g., processor(s)206) of an ego vehicle (e.g., vehicle100) implementing an object-detection neural network that has been trained as described herein. The ego vehicle need not be equipped with a LiDAR sensor (e.g., LiDAR sensor216), as the method1900does not require information from a LiDAR sensor.

At1902, the at least one processor receives, from an imaging radar sensor (e.g., radar214) of the ego vehicle, at least one radar frame of the environment of the ego vehicle.

At1904, the at least one processor detects at least one object of interest in the at least one radar frame. The at least one process may use an object-detection neural network to detect the at least one object of interest in the at least one radar frame.

At1906, the at least one processor determines a three-dimensional oriented bounding box, a location, and a velocity of the at least one object of interest.

At1908, the at least one processor outputs the three-dimensional oriented bounding box, the location, and the velocity of the at least one object of interest.

Accordingly, it will be appreciated, for example, that an apparatus or any component of an apparatus may be configured to (or made operable to or adapted to) provide functionality as taught herein. This may be achieved, for example: by manufacturing (e.g., fabricating) the apparatus or component so that it will provide the functionality; by programming the apparatus or component so that it will provide the functionality; or through the use of some other suitable implementation technique. As one example, an integrated circuit may be fabricated to provide the requisite functionality. As another example, an integrated circuit may be fabricated to support the requisite functionality and then configured (e.g., via programming) to provide the requisite functionality. As yet another example, a processor circuit may execute code to provide the requisite functionality.

Accordingly, it will also be appreciated, for example, that certain aspects of the disclosure can include a computer-readable medium embodying the methods described herein.

While the foregoing disclosure shows various illustrative aspects, it should be noted that various changes and modifications may be made to the illustrated examples without departing from the scope defined by the appended claims. The present disclosure is not intended to be limited to the specifically illustrated examples alone. For example, unless otherwise noted, the functions, steps, and/or actions of the method claims in accordance with the aspects of the disclosure described herein need not be performed in any particular order. Furthermore, although certain aspects may be described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated.