Patent Description:
Modern motor vehicles are increasingly incorporating technology that helps drivers avoid drifting into adjacent lanes or making unsafe lane changes (e.g., lane departure warning (LDW)), or that warns drivers of other vehicles behind them when they are backing up, or that brakes automatically if a vehicle ahead of them stops or slows suddenly (e.g., forward collision warning (FCW)), among other things. The continuing evolution of automotive technology aims to deliver even greater safety benefits, and ultimately deliver automated driving systems (ADS) that can handle the entire task of driving without the need for user intervention.

There are six levels that have been defined to achieve full automation. At Level <NUM>, the human driver does all the driving. At Level <NUM>, an advanced driver assistance system (ADAS) on the vehicle can sometimes assist the human driver with either steering or braking/accelerating, but not both simultaneously. At Level <NUM>, an ADAS on the vehicle can itself actually control both steering and braking/accelerating simultaneously under some circumstances. The human driver must continue to pay full attention at all times and perform the remainder of the driving tasks. At Level <NUM>, an
ADS on the vehicle can itself perform all aspects of the driving task under some circumstances. In those circumstances, the human driver must be ready to take back control at any time when the ADS requests the human driver to do so. In all other circumstances, the human driver performs the driving task. At Level <NUM>, an ADS on the vehicle can itself perform all driving tasks and monitor the driving environment, essentially doing all of the driving, in certain circumstances. The human need not pay attention in those circumstances. At Level <NUM>, an ADS on the vehicle can do all the driving in all circumstances. The human occupants are just passengers and need never be involved in driving. <CIT> discloses a system that generates a high resolution <NUM>-D point cloud to operate an autonomous driving vehicle (ADV) from a low resolution <NUM>-D point cloud and camera-captured image(s). The system receives a first image captured by a camera for a driving environment. The system receives a second image representing a first depth map of a first point cloud corresponding to the driving environment. The system downsamples the second image by a predetermined scale factor until a resolution of the second image reaches a predetermined threshold. The system generates a second depth map by applying a convolutional neural network (CNN) model to the first image and the downsampled second image, the second depth map having a higher resolution than the first depth map such that the second depth map represents a second point cloud perceiving the driving environment surrounding the ADV.

The following presents a simplified summary relating to one or more aspects disclosed herein.

In an aspect, a method of radar-aided single-image 3D depth reconstruction performed by at least one processor of an on-board computer of an ego vehicle includes receiving, from a light detection and ranging, LiDAR, sensor of the ego vehicle, at least one LiDAR image of an environment of the ego vehicle , wherein the at least one LiDAR image represents range measurements of laser signals emitted by the LiDAR sensor, wherein an azimuth axis of the at least one LiDAR image is quantized into uniformly spaced azimuth angle bins, and at least one depth value is calculated for each of the uniformly spaced azimuth angle bins, wherein the at least one depth value calculated for each of the uniformly spaced azimuth angle bins is computed as an average range measurement of all range measurements falling in that azimuth angle bin; using the at least one LiDAR image to train a convolutional neural network, CNN; receiving , from a radar sensor of the ego vehicle , at least one radar image of the environment of the ego vehicle; receiving, from a camera sensor of the ego vehicle, at least one camera image of the environment of the ego vehicle; and generating, using the trained CNN executed by the at least one processor, a depth image of the environment of the ego vehicle based on the at least one radar image and the at least one camera image.

In an aspect, an on-board computer of an ego vehicle includes means for receiving, from a light detection and ranging, LiDAR, sensor of the ego vehicle, at least one LiDAR image of an environment of the ego vehicle, wherein the at least one LiDAR image represents range measurements of laser signals emitted by the LiDAR sensor; means for quantizing an azimuth axis of the at least one LiDAR image into uniformly spaced azimuth angle bins; means for calculating at least one depth value for each of the uniformly spaced azimuth angle bins, wherein the at least one depth value calculated for each of the uniformly spaced azimuth angle bins is computed as an average range measurement of all range measurements falling in that azimuth angle bin; means for using the at least one LiDAR image to train a convolutional neural network, CNN; means for receiving at least one radar image of the environment of an ego vehicle; means for receiving at least one camera image of the environment of the ego vehicle; and means for generating, using the trained CNN, a depth image of the environment of the ego vehicle based on the at least one radar image and the at least one camera image.

In an aspect, computer-executable instructions for radar-aided single-image 3D depth reconstruction includes computer-executable instructions carrying out the method.

This disclosure provides techniques for applying 3D depth reconstruction to autonomous driving. In this context, access to both front-facing camera and radar sensors can be expected. These two sensors are complementary in several respects: the camera is a passive sensor measuring azimuth and elevation, while the radar is an active sensor measuring azimuth and range. In this disclosure, the camera's and radar's complementary characteristics are used by fusing their measurements.

Additionally, well-known aspects of the disclosure may not be described in detail or may be omitted so as not to obscure more relevant details.

In addition, for each of the aspects described herein, the corresponding form of any such aspect may be implemented as, for example, "logic configured to" perform the described action.

Autonomous and semi-autonomous driving safety technologies use a combination of hardware (sensors, cameras, and radar) and software to help vehicles identify certain safety risks so they can warn the driver to act (in the case of an ADAS), or act themselves (in the case of an ADS), to avoid a crash. A vehicle outfitted with an ADAS or ADS includes one or more camera sensors mounted on the vehicle that capture images of the scene in front of the vehicle, and also possibly behind and to the sides of the vehicle. Radar systems may also be used to detect objects along the road of travel, and also possibly behind and to the sides of the vehicle. Radar systems utilize radio frequency (RF) waves to determine the range, direction, speed, and/or altitude of the objects along the road. More specifically, a transmitter transmits pulses of RF waves that bounce off any object(s) in their path. The pulses reflected off the object(s) return a small part of the RF waves' energy to a receiver, which is typically located at the same location as the transmitter. The camera and radar are typically oriented to capture their respective versions of the same scene.

A processor, such as a digital signal processor (DSP), within the vehicle analyzes the captured camera images and radar frames and attempts to identify objects within the captured scene. Such objects may be other vehicles, pedestrians, road signs, objects within the road of travel, etc. The radar system provides reasonably accurate measurements of object distance and velocity in various weather conditions. However, radar systems typically have insufficient resolution to identify features of the detected objects. Camera sensors, however, typically do provide sufficient resolution to identify object features. The cues of object shapes and appearances extracted from the captured images may provide sufficient characteristics for classification of different objects. Given the complementary properties of the two sensors, data from the two sensors can be combined (referred to as "fusion") in a single system for improved performance.

To further enhance ADAS and ADS systems, especially at Level <NUM> and beyond, autonomous and semi-autonomous vehicles may utilize high definition (HD) map datasets, which contain significantly more detailed information and true-ground-absolute accuracy than those found in current conventional resources. Such HD maps may provide accuracy in the <NUM>-<NUM> absolute ranges, highly detailed inventories of all stationary physical assets related to roadways, such as road lanes, road edges, shoulders, dividers, traffic signals, signage, paint markings, poles, and other data useful for the safe navigation of roadways and intersections by autonomous / semi-autonomous vehicles. HD maps may also provide electronic horizon predictive awareness, which enables autonomous / semi-autonomous vehicles to know what lies ahead.

Referring now to <FIG>, a vehicle <NUM> (referred to as an "ego vehicle" or a "host vehicle") is illustrated that includes a radar-camera sensor module <NUM> located in the interior compartment of the vehicle <NUM> behind the windshield <NUM>. The radar-camera sensor module <NUM> includes a radar component configured to transmit radar signals through the windshield <NUM> in a horizontal coverage zone <NUM> (shown by dashed lines), and receive reflected radar signals that are reflected off of any objects within the coverage zone <NUM>. The radar-camera sensor module <NUM> further includes a camera component for capturing images based on light waves that are seen and captured through the windshield <NUM> in a horizontal coverage zone <NUM> (shown by dashed lines).

Although <FIG> illustrates an example in which the radar component and the camera component are collocated components in a shared housing, as will be appreciated, they may be separately housed in different locations within the vehicle <NUM>. For example, the camera may be located as shown in <FIG>, and the radar component may be located in the grill or front bumper of the vehicle <NUM>. Additionally, although <FIG> illustrates the radar-camera sensor module <NUM> located behind the windshield <NUM>, it may instead be located in a rooftop sensor array, or elsewhere. Further, although <FIG> illustrates only a single radar-camera sensor module <NUM>, as will be appreciated, the vehicle <NUM> may have multiple radar-camera sensor modules <NUM> pointed in different directions (to the sides, the front, the rear, etc.). The various radar-camera sensor modules <NUM> may be under the "skin" of the vehicle (e.g., behind the windshield <NUM>, door panels, bumpers, grills, etc.) or within a rooftop sensor array.

The radar-camera sensor module <NUM> may detect one or more (or none) objects relative to the vehicle <NUM>. In the example of <FIG>, there are two objects, vehicles <NUM> and <NUM>, within the horizontal coverage zones <NUM> and <NUM> that the radar-camera sensor module <NUM> can detect. The radar-camera sensor module <NUM> may estimate parameters (attributes) of the detected object(s), such as the position, range, direction, speed, size, classification (e.g., vehicle, pedestrian, road sign, etc.), and the like. The radar-camera sensor module <NUM> may be employed onboard the vehicle <NUM> for automotive safety applications, such as adaptive cruise control (ACC), forward collision warning (FCW), collision mitigation or avoidance via autonomous braking, lane departure warning (LDW), and the like.

Collocating the camera and radar permits these components to share electronics and signal processing, and in particular, enables early radar-camera data fusion. For example, the radar sensor and camera may be integrated onto a single board. A joint radar-camera alignment technique may be employed to align both the radar sensor and the camera. However, collocation of the radar sensor and camera is not required to practice the techniques described herein.

<FIG> illustrates an on-board computer (OBC) <NUM> of a vehicle <NUM>, according to various aspects of the disclosure. In an aspect, the OBC <NUM> may be part of an ADAS or ADS. The OBC <NUM> includes a non-transitory computer-readable storage medium, i.e., memory <NUM>, and one or more processors <NUM> in communication with the memory <NUM> via a data bus <NUM>. The memory <NUM> includes one or more storage modules storing computer-readable instructions executable by the processor(s) <NUM> to perform the functions of the OBC <NUM> described herein. For example, the processor(s) <NUM> in conjunction with the memory <NUM> may implement the various neural network architectures described herein.

One or more radar-camera sensor modules <NUM> are coupled to the OBC <NUM> (only one is shown in <FIG> for simplicity). In some aspects, the radar-camera sensor module <NUM> includes at least one camera <NUM>, at least one radar <NUM>, and an optional light detection and ranging (LiDAR) sensor <NUM>. The OBC <NUM> also includes one or more system interfaces <NUM> connecting the processor(s) <NUM>, by way of the data bus <NUM>, to the radar-camera sensor module <NUM> and, optionally, other vehicle sub-systems (not shown).

The OBC <NUM> also includes, at least in some cases, a wireless wide area network (WWAN) transceiver <NUM> configured to communicate via one or more wireless communication networks (not shown), such as a New Radio (NR) network, a Long-Term Evolution (LTE) network, a Global System for Mobile Communication (GSM) network, and/or the like. The WWAN transceiver <NUM> may be connected to one or more antennas (not shown) for communicating with other network nodes, such as other vehicle UEs, pedestrian UEs, infrastructure access points, roadside units (RSUs), base stations (e.g., eNBs, gNBs), etc., via at least one designated radio access technology (RAT) (e.g., NR, LTE, GSM, etc.) over a wireless communication medium of interest (e.g., some set of time/frequency resources in a particular frequency spectrum). The WWAN transceiver <NUM> may be variously configured for transmitting and encoding signals (e.g., messages, indications, information, and so on), and, conversely, for receiving and decoding signals (e.g., messages, indications, information, pilots, and so on) in accordance with the designated RAT.

The OBC <NUM> also includes, at least in some cases, a wireless local area network (WLAN) transceiver <NUM>. The WLAN transceiver <NUM> may be connected to one or more antennas (not shown) for communicating with other network nodes, such as other vehicle UEs, pedestrian UEs, infrastructure access points, RSUs, etc., via at least one designated RAT (e.g., cellular vehicle-to-everything (C-V2X), IEEE <NUM>. 11p (also known as wireless access for vehicular environments (WAVE)), dedicated short-range communication (DSRC), etc.) over a wireless communication medium of interest. The WLAN transceiver <NUM> may be variously configured for transmitting and encoding signals (e.g., messages, indications, information, and so on), and, conversely, for receiving and decoding signals (e.g., messages, indications, information, pilots, and so on) in accordance with the designated RAT.

As used herein, a "transceiver" may include a transmitter circuit, a receiver circuit, or a combination thereof, but need not provide both transmit and receive functionalities in all designs. For example, a low functionality receiver circuit may be employed in some designs to reduce costs when providing full communication is not necessary (e.g., a receiver chip or similar circuitry simply providing low-level sniffing).

The OBC <NUM> also includes, at least in some cases, a global positioning systems (GPS) receiver <NUM>. The GPS receiver <NUM> may be connected to one or more antennas (not shown) for receiving satellite signals. The GPS receiver <NUM> may comprise any suitable hardware and/or software for receiving and processing GPS signals. The GPS receiver <NUM> requests information and operations as appropriate from the other systems, and performs the calculations necessary to determine the vehicle's <NUM> position using measurements obtained by any suitable GPS algorithm.

In an aspect, the OBC <NUM> may utilize the WWAN transceiver <NUM> and/or the WLAN transceiver <NUM> to download one or more maps <NUM> that can then be stored in memory <NUM> and used for vehicle navigation. Map(s) <NUM> may be one or more high definition (HD) maps, which may provide accuracy in the <NUM>-<NUM> absolute ranges, highly detailed inventories of all stationary physical assets related to roadways, such as road lanes, road edges, shoulders, dividers, traffic signals, signage, paint markings, poles, and other data useful for the safe navigation of roadways and intersections by vehicle <NUM>. Map(s) <NUM> may also provide electronic horizon predictive awareness, which enables the vehicle <NUM> to know what lies ahead.

In an aspect, the camera <NUM> may capture image frames (also referred to herein as camera frames) of the scene within the viewing area of the camera <NUM> (as illustrated in <FIG> as horizontal coverage zone <NUM>) at some periodic rate. Likewise, the radar <NUM> may capture radar frames of the scene within the viewing area of the radar <NUM> (as illustrated in <FIG> as horizontal coverage zone <NUM>) at some periodic rate. The periodic rates at which the camera <NUM> and the radar <NUM> capture their respective frames may be the same or different. Each camera and radar frame may be timestamped. Thus, where the periodic rates are different, the timestamps can be used to select simultaneously, or nearly simultaneously, captured camera and radar frames for further processing (e.g., fusion).

<FIG> illustrates a sensed observation radar grid <NUM>. A transmitter (e.g., an array of transmit antennas) of the radar <NUM> transmits pulses of electromagnetic RF waves that reflect from object(s) in the transmission path, such as vehicles <NUM> and <NUM> in <FIG>. A portion of the electromagnetic RF waves that are reflected from the object(s) are returned to the receiver (e.g., an array of receive antennas) of the radar <NUM>, which is usually located at the same site as the transmitter of the radar <NUM>.

In an aspect, the radar <NUM> may be an imaging radar that uses beamforming to scan horizontally and vertically. Beamforming is a technique used to aim the effective direction of a radar beam by changing the delay between different transmitting antennas so that the signals add constructively in a specified direction. Thus, the radar <NUM> may scan horizontally and vertically across the sensing area (e.g., horizontal coverage zone <NUM>) by using a transmitter comprising an array of electronically steered antennas.

The returned responses (which may also be referred to as "pings") measured by the radar <NUM> is characterized as an observation (or occupancy) grid <NUM> having a plurality of observation cells <NUM>. Each cell <NUM> represents the measured returned response value at a specific range (r) and angle/azimuth (θ). Each cell <NUM> is alternately referred to as a range-angle bin. Features <NUM> are extracted from the cells <NUM> to determine whether the feature <NUM> is an object (e.g., a vehicle <NUM>/<NUM>). Each feature <NUM> within a respective cell <NUM> can be identified as having up to four parameters: range, Doppler, azimuth, and elevation. This is called a radar frame. As an example, a feature <NUM> within a cell <NUM> may be the signal-to-noise ratio (SNR) computed by a constant false alarm rate (CFAR) algorithm. However, it should be understood that other methods may be used to target and identify features <NUM> within a cell <NUM>.

The processor(s) <NUM> may generate two-dimensional (2D), three-dimensional (3D), or four dimensional (4D) tensors for features <NUM> within cells <NUM> of the observation grid <NUM> detected by the radar <NUM>. Specifically, a 2D tensor represents the range (distance from the vehicle <NUM> to the detected feature <NUM>) and azimuth (the horizontal distance between a feature <NUM> and a reference RF ray emitted by the radar <NUM>, such as the initial RF ray of a radar sweep) of each detected feature <NUM>. A 3D tensor represents the range, azimuth, and Doppler (indicating the speed of the detected feature <NUM>) or elevation (vertical direction from the radar <NUM> to the detected feature) of each detected feature <NUM>. A 4D tensor represents all four quantities. The processor(s) <NUM> then performs object detection, object classification, localization, and property/attribute estimation based on the tensors and undistorted camera frames received from the camera <NUM>.

Note that conventional automotive radars generally only provide range and azimuth measurements (2D tensors); they do not provide elevation information (3D tensors). In addition, Doppler information is usually integrated out, meaning it may be measured but is then removed. However, it may also not be measured at all. As such, the description of the techniques disclosed herein assume that only range and azimuth information is obtained from the radar sensor (e.g., radar <NUM>).

In contrast to images (e.g., from camera <NUM>), radar signals (e.g., from radar <NUM>) have several unique characteristics. One example is specular reflections, in which only certain surfaces on the target having an advantageous orientation reflect the radar signal, which often results in a small number of reflections.

<FIG> is a diagram illustrating a camera image plane <NUM> and a radar image plane <NUM>, according to aspects of the disclosure. Camera image formation maps a point <NUM> with Cartesian coordinates (x, y, z) into the point <NUM> (x/z, y/z) on the camera image plane <NUM>. This so-called perspective projection removes all depth information. Radar image formation maps the point <NUM> with spherical coordinates, i.e., range, azimuth, and elevation (ρ, θ, Φ), into the point <NUM> with polar coordinates (ρ, θ) on the radar image plane <NUM>. This spherical-to-polar projection removes all elevation information. Thus, at a high level, the camera <NUM> and radar sensor <NUM> measure projections of the 3D scene onto different 2D planes (vertical for camera and horizontal for radar).

Single-image depth reconstruction is an important problem in computer vision. It has applications in scene understanding, robotics, and 3D reconstruction. In autonomous driving, depth reconstruction can be used to aid in sensor fusion, drivable space detection, and navigation.

As discussed above with reference to <FIG>, a radar sensor (e.g., radar sensor <NUM>) is an active sensor (insofar as it transmits and receives RF signals) that measures range and azimuth. However, the radar sensor does not output elevation information. Rather, as discussed above, the resulting radar image represents a target point (e.g., point <NUM>) in polar coordinates (ρ, θ). This characteristic makes the radar sensor complementary to the camera (e.g., camera <NUM>), which is a passive sensor (insofar as it simply captures light rays and transforms them into an image) that measures azimuth and elevation (e.g., as a point <NUM> (x/z, y/z)), but does not measure range. From the complementary nature of the camera and radar sensors, there is a clear benefit of fusing their measurements for depth reconstruction.

However, radar-aided single-image 3D depth reconstruction remains ill posed, since both sensors provide only 2D projections of a 3D scene, as discussed above with reference to <FIG>. For front-facing sensors, the camera image yields a vertical projection, whereas the radar image yields a horizontal projection, as illustrated in <FIG>.

To understand the complementary nature of the camera and radar sensors, it is beneficial to consider their respective image formation processes. For ease of presentation, the following description is restricted to a simplified and somewhat stylized description. Specifically, an ideal camera model is assumed with unit focal length, and similarly, an ideal noiseless radar sensor is assumed. These assumptions are relaxed later in this disclosure, where real camera and radar images are considered.

As discussed above with reference to <FIG>, camera image formation maps a point with Cartesian coordinates (x, y, z) into the point (x/z, y/z) on the camera image plane. This perspective projection removes all depth information. Radar image formation maps a point with spherical coordinates (ρ, θ, Φ) into the point with polar coordinates (ρ, θ) on the radar image plane. This spherical-to-polar projection removes all elevation information. Thus, at a high level, the camera and radar sensors measure projections of a 3D scene onto different 2D planes.

The complementary nature of the camera and radar sensors means that fusing them can remove some of the inherent ambiguity in the problem of monocular depth reconstruction. An important one of these is the well-known scale ambiguity problem depicted in <FIG>, where the different sized (scaled) objects A and B result in identical projections <NUM> onto the camera image plane <NUM>. The radar sensor does not suffer from this scale ambiguity. This is again depicted in <FIG>, which shows that the two objects A and B have easily distinguished projections onto the radar image plane <NUM>. Thus, fusing camera and radar sensors entirely eliminates the problem of scale ambiguity.

Unfortunately, even when using both the camera and radar sensors, the 3D depth reconstruction problem remains in general ambiguous and ill posed. For example, <FIG> illustrates two distinct 3D scenes (a) and (b), each with two objects (<NUM> and <NUM> in scene (a) and <NUM> and <NUM> in scene (b)) in the scenes. These two distinct scenes yield identical camera projections <NUM> and <NUM> and radar projections <NUM> and <NUM> on the camera image plane <NUM> and the radar image plane <NUM>, respectively. Thus, from the camera and radar projections alone, the corresponding 3D scene cannot be uniquely determined. This indicates that the radar-aided single-image 3D depth reconstruction problem remains challenging, and complex global prior information is needed to solve it.

Note that the foregoing description of the radar sensor has ignored the non-ideal effects of the radar antenna beam pattern, which limits the sensor resolution, particularly in the azimuth direction. This limited resolution results in significant azimuthal "smearing" in the radar image, rendering the correct association of points on the image plane and points on the radar plane with a single 3D object even more difficult.

Several examples of corresponding camera and radar images shown in <FIG> illustrate this difficulty. In <FIG>, camera images are on the left and the corresponding radar images are on the right. In each pair of images, the x-axis represents azimuth with a field of view of <NUM> degrees. For the radar images, the y-axis represents range starting at <NUM> meters (m) at the bottom and ending at <NUM> at the top. The radar images depict radar return intensities in decibels (dB).

<FIG> shows a camera image <NUM> of an empty road flanked on the right by a concrete guard rail. The radar signature of the guard rail is faintly visible as a curved blurry line in the lower-right quadrant of the radar image <NUM>. <FIG> shows a camera image <NUM> of two cars at a distance of approximately <NUM> and <NUM>, respectively. The radar signature of the farther car is visible as a bright horizontal line in the upper part of the radar image <NUM>. The radar signature of the closer car is visible as a white blob and a blurry horizontal line in the lower part of the radar image <NUM>. This clearly illustrates the aforementioned "smearing" effect due to the radar sensor's limited azimuthal resolution. <FIG> shows a camera image <NUM> of a large truck at close range. The corresponding radar signature occupies most of the lower part of the radar image <NUM>.

The techniques of the present disclosure use a deep CNN to perform the task of fusing the camera and radar images into a 3D depth reconstruction (represented as a depth image). A CNN is a class of deep neural networks, most commonly used for analyzing visual imagery. A CNN uses a variation of multilayer perceptrons (a class of feedforward artificial neural networks, consisting of at least an input layer, a hidden layer, and an output layer) designed to require minimal preprocessing (e.g., cleaning, instance selection, normalization, transformation, feature extraction and selection, etc.) of the raw input data to generate the final training set. This means that a CNN learns the filters that in traditional algorithms were hand-engineered. This independence from prior knowledge and human effort in feature design is a major advantage of CNNs.

The proposed CNN uses a modified encoder-decoder network architecture, consisting of two separate camera and radar branches, whose outputs are combined in a fusion encoder branch, followed by a decoder branch producing the final depth image. The CNN can be trained with pairs of camera and radar image collected in highway environments. The CNN is trained to fuse the camera and radar images, and also to incorporate prior knowledge about highway environments, such as knowledge of what a highway scene looks like from the perspective of the ego vehicle, such as the likely locations of signs, guide rails, other vehicles, etc. During training, ground truth information can be obtained from a LiDAR sensor (e.g., LiDAR sensor <NUM>).

Radar-aiding for single-image depth reconstruction provides a number of advantages. For example, radar-aiding improves depth reconstruction quality. As another example, radar-aiding improves the robustness of the reconstruction, since the camera and radar sensors have different failure modes. More specifically, as noted above, the camera is affected by lighting conditions (e.g., shadows, rain, glare, darkness, etc.) while the radar sensor, being an active sensor, is unaffected by lighting conditions.

The techniques of the present disclosure utilize up to three sensors: a camera sensor (e.g., camera <NUM>) and a radar sensor (e.g., radar <NUM>) as the input sensors, and a LiDAR sensor (e.g., LiDAR sensor <NUM>) as the ground truth depth sensor <FIG> illustrates exemplary mounting positions of a camera <NUM>, a radar <NUM>, and a LiDAR sensor <NUM> on an ego vehicle <NUM>. The camera <NUM> may be an automotive-grade camera with an <NUM> by <NUM> pixel resolution, for example. As disclosed above with reference to <FIG> and illustrated in <FIG>, the camera <NUM> may be mounted behind the front windshield of the vehicle <NUM>. The radar <NUM> may be an automotive radar, such as a RadarLog® from INRAS®, operating in the <NUM> gigahertz (GHz) frequency band, for example. As disclosed above with reference to <FIG> and illustrated in <FIG>, the radar <NUM> may be mounted on the front of the vehicle <NUM> (e.g., in the grill or front bumper). The LiDAR sensor <NUM> may be mounted on the roof of the vehicle <NUM>, and may be, for example, a Velodyne® <NUM>-laser LiDAR.

<FIG> also shows each sensor's elevation field of view after preprocessing (described further below). Specifically, the elevation field of views of the camera <NUM> and the radar sensor <NUM> (the input sensors) are indicated by reference numbers <NUM> and <NUM>, respectively, and the elevation field of view of the LiDAR sensor <NUM> (the ground truth sensor) is indicated by reference number <NUM>. The frame rates of the three sensors may be, for example, <NUM> hertz (Hz) for the camera <NUM>, <NUM> for the radar <NUM>, and <NUM> for the LiDAR sensor <NUM>. During training, each LiDAR frame is matched to the nearest camera and radar frames, and during operation, each camera image is matched to the nearest radar frame.

At an initial stage, some basic preprocessing is performed on each of the sensors' measurements in order to approximately align their fields of view (illustrated in <FIG>), and to convert the data into the appropriate form for neural network processing. Note that, as shown in <FIG>, the sensors need not be collocated, and as described above, need not be synchronized or calibrated with each other. Rather, each LiDAR frame is matched to the nearest camera and radar frames during training of the CNN, and each camera image is matched to the nearest radar frame after training.

The camera image is undistorted (i.e., any distortions are removed) and then cropped to a region spanning, for example, <NUM> degrees in elevation and <NUM> degrees in azimuth. It is then centered (e.g., manually) to align with the corresponding LiDAR image to be reconstructed. The cropped image is then resized to, for example, <NUM> by <NUM> pixels.

Standard radar signal processing techniques (see, e.g., M. Richards, "Fundamentals of Radar Signal Processing," McGraw-Hill, second ed. , <NUM>) can be used to create a <NUM> by <NUM> image of radar return intensities with the horizontal axis representing azimuth and the vertical axis representing range. The intensities are expressed in dB scale and clipped to restrict dynamic range to, for example, <NUM> dB.

Of the total number of LiDAR lasers (e.g., <NUM>), only the data from the central ones is kept (e.g., the central <NUM> lasers). The other lasers typically point above the horizon or towards the hood of the vehicle and hence do not provide significant information about the traffic around the ego vehicle (e.g., vehicle <NUM>). The LiDAR measurements may be further restricted to within ±<NUM> degrees around the forward direction to discard data from outside the field of view of the radar <NUM>.

The azimuth axis is then quantized into uniformly spaced bins (e.g., <NUM> bins of width <NUM> degrees). The depth value of a bin is computed as the average range measurement of all LiDAR returns falling in that bin. This process produces a depth image (e.g., of dimension <NUM> by <NUM>). Some bins may contain no LiDAR returns (due to the lack of reflective targets in that direction). The depth for such bins is attempted to be imputed by interpolating from neighboring bins at the same elevation. If that method fails, the depth value of the bin is set to the maximum value of <NUM>, assuming that there was no reflector in that direction.

The LiDAR depth image has a high dynamic range as the lower laser rays travel a short distance to the ground immediately in front of the vehicle, while higher lasers could point to far away targets. This dynamic range can be reduced by subtracting the distance from the known LiDAR sensor to an assumed flat ground plane. The resulting ground-depth subtracted depth image can be used as the ground truth in training the network and in visualizing the results. This helps local variations in depth stand out in the depth image.

<FIG> illustrates an exemplary neural network architecture <NUM>, according to aspects of the disclosure. The neural network architecture <NUM> is the architecture of an exemplary CNN. In the example of <FIG>, the input images to the neural network architecture <NUM> are <NUM> by <NUM> pixels for the camera image <NUM> and <NUM> by <NUM> pixels for the radar image <NUM>. The depth image <NUM> output is <NUM> by <NUM> pixels, to match the size of the LiDAR depth image described above.

The camera image <NUM> and the radar image <NUM> are first processed through separate network branches, camera branch <NUM> and radar branch <NUM>, respectively, in which they are down-sampled (i.e., resampled to produce an approximation of the image that would have been obtained by originally sampling at a lower rate) progressively until their feature maps have the same dimension as the output depth image <NUM> (i.e., <NUM> by <NUM> in the example of <FIG>). The camera branch <NUM> applies two down-sampling steps, each of which consists of a <NUM> by <NUM> convolutional layer with stride (<NUM>, <NUM>) followed by a <NUM> by <NUM> convolutional layer with stride (<NUM>, <NUM>). A convolution layer applies a convolution operation (a mathematical operation on two functions to produce a third function that expresses how the shape of one is modified by the other) to the input, passing the result to the next layer, and which emulates the response of an individual neuron to visual stimuli. Similarly, the radar branch <NUM> applies five down-sampling steps, each of which consists of a <NUM> by <NUM> convolutional layer with stride (<NUM>, <NUM>) followed by a <NUM> by <NUM> convolutional layer with stride (<NUM>, <NUM>). The number of feature maps at the output of each layer on the radar and camera branches <NUM> and <NUM> is kept fixed (e.g., at <NUM>).

The outputted feature maps (e.g., <NUM> of <NUM> by <NUM> pixels) from each of the two sensor branches <NUM> and <NUM> are concatenated at <NUM> and then reduced from, for example, <NUM>, back to, for example, <NUM> feature maps through a subsequent <NUM> by <NUM> convolutional layer. They are then passed through a fusion branch <NUM> in the form of an encoder-decoder network. The encoder-decoder network may be similar to U-Net (see <NPL>). In the encoder part of the fusion branch <NUM>, the feature maps are down-sampled two more times, each down-sampling step consisting of a <NUM> by <NUM> convolutional layer with stride (<NUM>, <NUM>) followed by a <NUM> by <NUM> convolutional layer with stride (<NUM>, <NUM>). The number of feature maps is doubled after every layer that down-samples the image. At the waist of the encoder-decoder network there are, for example, <NUM> feature maps, each of dimension, for example, <NUM> by <NUM>. In the example of <FIG>, these <NUM> variables (<NUM> by <NUM> by <NUM> = <NUM>) are used as inputs to a fully connected layer with, for example, <NUM> outputs, and are then passed through a <NUM>% dropout layer, followed by another fully connected layer with <NUM> output variables. These variables are then used to populate, for example, <NUM> feature maps of dimension, for example, <NUM> by <NUM> pixels, which form the input to the decoder part of the fusion branch <NUM>. The use of fully connected and dropout layers helps the neural network architecture <NUM> to learn global features and to improve generalization, respectively.

The decoder part of the fusion branch <NUM> uses deconvolution and up-sampling layers (resampling to produce an approximation of the image that would have been obtained by originally sampling at a higher rate) mirroring the steps in the encoder part. Similar to U-Net, skip connections (implemented via tensor addition) are used from the encoder layers to the decoder layers to transmit higher frequency features.

Once trained on some number of camera and radar images, the neural network architecture <NUM> can be used to process camera images and radar images in real-time without reliance on corresponding LiDAR images. The processed images can be used in an autonomous vehicle's computer vision system to perform autonomous driving operations, such as lane changes, breaking, accelerating, etc..

<FIG> illustrates a sample result of the neural network architecture <NUM>, according to aspects of the disclosure. The illustrated scene contains a concrete guard rail, a car, and a truck (from left to right). The sun is blocked by the truck, resulting in poor illumination of the truck and a pronounced shadow on the central lane. Both these effects are detrimental, and the depth reconstruction from the camera alone is quite poor in this scenario, as shown in camera image <NUM>. The radar sensor is unaffected by either of these effects, as shown in the radar image <NUM>, and with radar aiding, 3D depth reconstruction is successful, as shown in the depth image <NUM>. A LiDAR image <NUM> provides the ground truth depth for the scene.

<FIG> illustrates an exemplary method <NUM> for radar-aided single-image 3D depth reconstruction, according to aspects of the disclosure. In an aspect, the method <NUM> may be performed by the processor(s) <NUM> of the OBC <NUM> in conjunction with the radar-camera sensor module <NUM>. In an aspect, the camera sensor and the radar sensor may be collocated on the ego vehicle (e.g., vehicle <NUM>). However, in an alternative aspect, the camera sensor and the radar sensor may not be collocated on the ego vehicle (e.g., as in the example of <FIG>).

At <NUM>, the processor(s) <NUM> optionally receive, from a LiDAR sensor (e.g., LiDAR sensor <NUM>) of the ego vehicle, at least one LiDAR image (e.g., LiDAR image <NUM>) of the environment of the ego vehicle. In an aspect, the LiDAR image represents range measurements of laser signals emitted by the LiDAR sensor. In an aspect, an azimuth axis of the at least one LiDAR image may be quantized into uniformly spaced azimuth angle bins and at least one depth value may be calculated for each of the uniformly spaced azimuth angle bins. In an aspect, an elevation axis of the at least one LiDAR image may be quantized into uniformly spaced elevation steps, and a depth value may be calculated for each pair of azimuth angle bins and elevation steps. In an aspect, the depth value of each of the uniformly spaced azimuth angle bins may be computed as an average range measurement of all range measurements falling in that azimuth angle bin.

At <NUM>, the processor(s) <NUM> optionally use the LiDAR image to train a CNN executed by the processor(s) <NUM>. Stages <NUM> and <NUM> are optional because they need not be performed after the CNN is trained.

At <NUM>, the processor(s) <NUM> receive, from a radar sensor (e.g., radar <NUM>) of the ego vehicle, at least one radar image (e.g., radar image <NUM>) of the environment of the ego vehicle. In an aspect, the radar sensor may be a commercially available electronically scanning radar (ESR), a short-range radar (SRR), a long-range radar, or a medium-range radar.

At <NUM>, the processor(s) <NUM> receive, from a camera sensor (e.g., camera <NUM>) of the ego vehicle, at least one camera image (e.g., camera image <NUM>) of the environment of the ego vehicle. In an aspect, the camera sensor and the radar sensor may capture images at different frequencies, and the at least one camera image may be the nearest camera image in time to the at least one radar image.

At <NUM>, the processor(s) <NUM>, using the CNN, generates a depth image (e.g., depth image <NUM>) of the environment of the ego vehicle based on the at least one radar image and the at least one camera image, as described above. In an aspect, the CNN may use an encoder-decoder network architecture, which may include a camera branch, a radar branch, a fusion encoder branch, and a decoder branch, as described above with reference to <FIG>. In an aspect, as described above, the camera branch may generate at least one feature map representing the at least one camera image by down-sampling the at least one camera image until dimensions of the at least one feature map match dimensions of the depth image. The radar branch may generate at least one feature map representing the at least one radar image by down-sampling the at least one radar image until dimensions of the at least one feature map match dimensions of the depth image. /The fusion encoder branch may combine the at least one feature map representing the at least one camera image and the at least one feature map representing the at least one radar image into at least one fused feature map. The decoder branch may then generate the depth image from the at least one fused feature map based on up-sampling the at least one fused feature map.

At <NUM>, the processor(s) optionally cause the ego vehicle to perform an autonomous driving operation based on the depth image of the environment of the ego vehicle. Stage <NUM> is optional because the ego vehicle may not need to perform a driving operation based on the generated depth image. In an aspect, the autonomous driving operation may be one or more of displaying the depth image, detecting drivable space, path planning, braking, accelerating, steering, adjusting a cruise control setting, or signaling.

It should be understood that any reference to an element herein using a designation such as "first," "second," and so forth does not generally limit the quantity or order of those elements. Rather, these designations may be used herein as a convenient method of distinguishing between two or more elements or instances of an element. Thus, a reference to first and second elements does not mean that only two elements may be employed there or that the first element must precede the second element in some manner. Also, unless stated otherwise a set of elements may comprise one or more elements. In addition, terminology of the form "at least one of A, B, or C" or "one or more of A, B, or C" or "at least one of the group consisting of A, B, and C" used in the description or the claims means "A or B or C or any combination of these elements. " For example, this terminology may include A, or B, or C, or A and B, or A and C, or A and B and C, or 2A, or 2B, or 2C, and so on.

In view of the descriptions and explanations above, those of skill in the art will appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the aspects disclosed herein may be implemented as electronic hardware, computer software, or combinations of both.

Accordingly, it will be appreciated, for example, that an apparatus or any component of an apparatus may be configured to (or made operable to or adapted to) provide functionality as taught herein. This may be achieved, for example: by manufacturing (e.g., fabricating) the apparatus or component so that it will provide the functionality; by programming the apparatus or component so that it will provide the functionality; or through the use of some other suitable implementation technique. As one example, an integrated circuit may be fabricated to provide the requisite functionality. As another example, an integrated circuit may be fabricated to support the requisite functionality and then configured (e.g., via programming) to provide the requisite functionality. As yet another example, a processor circuit may execute code to provide the requisite functionality.

Moreover, the methods, sequences, and/or algorithms described in connection with the aspects disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in random access memory (RAM), flash memory, read-only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. In the alternative, the storage medium may be integral to the processor (e.g., cache memory).

Accordingly, it will also be appreciated, for example, that certain aspects of the disclosure can include a computer-readable medium embodying the methods described herein.

Claim 1:
A method of radar-aided single-image three-dimensional, 3D, depth reconstruction performed by at least one processor of an on-board computer of an ego vehicle (<NUM>), comprising:
receiving (<NUM>), from a light detection and ranging, LiDAR, sensor (<NUM>) of the ego vehicle, at least one LiDAR image (<NUM>) of an environment of the ego vehicle (<NUM>), wherein the at least one LiDAR image (<NUM>) represents range measurements of laser signals emitted by the LiDAR sensor (<NUM>), wherein
an azimuth axis of the at least one LiDAR image (<NUM>) is quantized into uniformly spaced azimuth angle bins, and
at least one depth value is calculated for each of the uniformly spaced azimuth angle bins, wherein the at least one depth value calculated for each of the uniformly spaced azimuth angle bins is computed as an average range measurement of all range measurements falling in that azimuth angle bin;
using (<NUM>) the at least one LiDAR image to train a convolutional neural network, CNN;
receiving (<NUM>), from a radar sensor (<NUM>) of the ego vehicle (<NUM>), at least one radar image (<NUM>, <NUM>) of the environment of the ego vehicle (<NUM>);
receiving (<NUM>), from a camera sensor (<NUM>) of the ego vehicle (<NUM>), at least one camera image (<NUM>, <NUM>) of the environment of the ego vehicle (<NUM>); and
generating (<NUM>), using the trained CNN executed by the at least one processor, a depth image (<NUM>) of the environment of the ego vehicle (<NUM>) based on the at least one radar image (<NUM>, <NUM>) and the at least one camera image (<NUM>, <NUM>).