Patent Description:
<CIT> discloses a method for detecting a road course for a vehicle where radar sensors are used in order to determine the position and velocity of objects ahead. In particular, stationary targets are determined based on a comparison between the velocity of the vehicle and a target velocity wherein the stationary targets corresponding to an edge of the road are identified in order to determine the distance of the vehicle to the edge of the road by using a neural network. <CIT> teaches a computer implemented method for object detection where point-cloud data and map data are brought into respective representations suitable for concatenating the representations. The concatenated representation data is further used as input to a neural network in order to improve detection results.

An object of the invention is to provide an method, system and computer-readable storage media for radar systems in automotive applications improving downstream applications such as assisted-driving and autonomous-driving systems.

This object is satisfied by the subject-matters of the independent claims.

This document describes a method for a radar system using a machine-learned model for stationary object detection. The method comprises: receiving radar data that comprises multiple time-series frames associated with electromagnetic, EM, energy reflected by one or more objects in a roadway and received at an antenna of a radar system attached to a portion of a vehicle, each time-series frame of the radar data including multiple range bins; generating, using the time-series frames of the radar data, a Doppler beam vector of the EM energy; generating, using the Doppler beam vector and a super-resolution operation, a range-azimuth map of the EM energy for each time-series frame of the multiple time-series frames; generating, using the range-azimuth map, an interpolated range-azimuth map of the EM energy by: determining, for each range bin of multiple range bins within the range-azimuth map, a range-bin time, the range-bin time being the time-series frame at which the vehicle reaches the range bin; selecting, for a respective range-bin time, bracketing data captures of the radar data that have a time-series frame immediately before and immediately after the respective range-bin time; determining a range position of the vehicle at the bracketing data captures; and shifting range values of the bracketing data captures in a range dimension so that a location of the vehicle and a stationary object among the one or more objects match those in an interpolated time-series frame; generating, using the interpolated range-azimuth map, a range-time map of the EM energy, the range-time map representing a potential detection of the stationary object among the one or more objects as a straight, <NUM>-degree line; and detecting, using a machine-learned model, stationary objects among the one or more objects, the machine-learned model being configured to receive as inputs extracted features corresponding to the one or more objects from the range-time map for multiple range bins at each of the time-series frames, the extracted features including at least two of range, target angle, velocity, geometrical features, or intensity-based features of the one or more objects.

This document also describes a system comprising a radar system including one or more processors configured to detect stationary objects by performing the foregoing method.

This document also describes a computer-readable storage media comprising computer-executable instructions that, when executed, cause a processor of a radar system to perform the foregoing method.

This Summary introduces simplified concepts related to a radar system that uses a machine-learned model for stationary object detection, further described in the Detailed Description and Drawings. This Summary is not intended to identify essential features of the claimed subject matter, nor is it intended for use in determining the scope of the claimed subject matter.

The details of one or more aspects of a radar system using a machine-learned model for stationary object detection are described in this document with reference to the following figures. The same numbers are often used throughout the drawings to reference like features and components:.

Radar systems can be configured as an important sensing technology that vehicle-based systems rely on to acquire information about the surrounding environment. For example, vehicle-based systems can use radar systems to detect stationary objects in or near a roadway and, if necessary, take necessary actions (e.g., reducing speed, changing lanes) to avoid a collision. Radar systems generally use a point-cloud representation of radar data (e.g., detection-level data) to detect objects such as this. From a moving vehicle, such systems can generally detect relatively large stationary objects, such as parked vehicles, but often cannot detect smaller stationary objects while the vehicle is not stationary and driving at speed. If the host vehicle travels at a non-uniform speed, these radar systems may also struggle to differentiate stationary objects from moving objects.

Radar systems generally process radar data as a series of frames collected at equal time intervals. To pre-process, label, and extract features from the radar data, these radar systems use the speed of the host vehicle to track and associate stationary objects. Due to changing vehicle speed, the vehicle displacement in each frame is different. The changing vehicle speed can make it difficult for these radar systems to detect and label stationary objects accurately.

To improve the accuracy and speed with which even small stationary objects are detected by radar, this document describes techniques and systems for a machine-learned model that a radar system can use for stationary object detection based on low-level radar data. Low-level radar data (e.g., range-Doppler maps, range-azimuth maps, Doppler-azimuth maps) provide more information than point cloud representations. By using low-level radar data, the described machine-learned model can accurately detect stationary objects of various sizes and identify stationary objects sooner.

The described techniques and systems can also interpolate the low-level radar data so that each frame is normalized based on the vehicle speed. In this way, an interpolated range-time map represents a potential detection of a stationary object as a straight <NUM>-degree line. The interpolated range-time map simplifies stationary object detection, improving its accuracy and confidence.

This is just one example of the described techniques and systems for a radar system to use a machine-learned model for stationary object detection. This document describes other examples and implementations.

<FIG> illustrates an example environment <NUM> in which a radar system uses a machine-learned model for stationary object detection in accordance with techniques of this disclosure. In the depicted environment <NUM>, the radar system <NUM> is mounted to, or integrated within, a vehicle <NUM> traveling on a roadway <NUM>. Within a field-of-view <NUM>, the radar system <NUM> can detect one or more stationary objects <NUM> near the vehicle <NUM>.

Although illustrated as a truck, the vehicle <NUM> can represent other types of motorized vehicles (e.g., a car, motorcycle, bus, tractor, semi-trailer truck), non-motorized vehicles (e.g., a bicycle), railed vehicles (e.g., a train), watercraft (e.g., a boat), aircraft (e.g., an airplane), or spacecraft (e.g., satellite). In general, manufacturers can mount the radar system <NUM> to any moving platform, including moving machinery or robotic equipment.

In the depicted implementation, the radar system <NUM> is mounted on the front of the vehicle <NUM> and illuminates the stationary object <NUM>. The radar system <NUM> can detect the stationary object <NUM> from any exterior surface of the vehicle <NUM>. For example, vehicle manufacturers can integrate, install, or attach the radar system <NUM> into a front portion, bumper, side mirror, headlights, rear lights, or any other interior or exterior location where the stationary object <NUM> requires detection. In some cases, the vehicle <NUM> includes multiple radar systems <NUM>, such as a first radar system <NUM> and a second radar system <NUM>, that provide a larger instrument field-of-view <NUM>. In general, vehicle manufacturers can design the locations of one or more radar systems <NUM> to provide a particular field-of-view <NUM> that encompasses a region of interest. Example fields-of-view <NUM> include a <NUM>-degree field-of-view, one or more <NUM>-degree fields-of-view, one or more <NUM>-degree fields-of-view, and so forth, which can overlap into the field-of-view <NUM> of a particular size.

The stationary object <NUM> includes one or more materials that reflect radar signals. Depending on the application, the stationary object <NUM> can represent a target of interest. For example, the stationary object <NUM> can be a parked vehicle, a roadside sign, a roadway barrier, or debris on the roadway <NUM>.

The radar system <NUM> emits electromagnetic (EM) radiation by transmitting EM signals or waveforms via antenna elements. In the environment <NUM>, the radar system <NUM> can detect and track the stationary object <NUM> by transmitting and receiving one or more radar signals. For example, the radar system <NUM> can transmit EM signals between <NUM> and <NUM> gigahertz (GHz), between <NUM> and <NUM>, or between approximately <NUM> and <NUM>.

The radar system <NUM> can include a transmitter <NUM>, which includes at least one antenna to transmit EM signals. The radar system <NUM> can also include a receiver <NUM>, which includes at least one antenna to receive reflected versions of the EM signals. The transmitter <NUM> includes one or more components for emitting the EM signals. The receiver <NUM> includes one or more components for detecting the reflected EM signals. Manufacturers can incorporate the transmitter <NUM> and the receiver <NUM> together on the same integrated circuit (e.g., configured as a transceiver) or separately on different integrated circuits.

The radar system <NUM> also includes one or more processors <NUM> (e.g., an energy processing unit) and computer-readable storage media (CRM) <NUM>. The processor <NUM> can be a microprocessor or a system-on-chip. The processor <NUM> can execute computer-executable instructions stored in the CRM <NUM>. For example, the processor <NUM> can process EM energy received by the receiver <NUM> and determine, using a stationary object detection module <NUM>, a location of the stationary object <NUM> relative to the radar system <NUM>. The stationary object detection module <NUM> can also detect various features (e.g., range, target angle, velocity) of the stationary object <NUM>.

The processor <NUM> can also generate radar data for at least one automotive system. For example, the processor <NUM> can control, based on processed EM energy from the receiver <NUM>, an autonomous or semi-autonomous driving system of the vehicle <NUM>. For example, the autonomous driving system can control operation of the vehicle <NUM> to maneuver around the stationary object <NUM> or to slow down or come to a stop to avoid a collision with the stationary object <NUM>. As another example, the semi-autonomous driving system can alert an operator of the vehicle <NUM> that the stationary object <NUM> is in the roadway <NUM>.

The stationary object detection module <NUM> receives radar data, for example, raw or time-series frames associated with EM energy received by the receiver <NUM>, and determines whether a stationary object <NUM> is in the roadway <NUM> and various features associated with the stationary object <NUM>. The stationary object detection module <NUM> can use a machine-learned model <NUM> to assist with the described operations and functions. The radar system <NUM> can implement the stationary object detection module <NUM> and the machine-learned model <NUM> as computer-executable instructions in the CRM <NUM>, hardware, software, or a combination thereof that is executed by the processor <NUM>.

The machine-learned model <NUM> can perform, using input range-time maps and extracted features, stationary object detection for the stationary objects <NUM>. The machine-learned model <NUM> can use a neural network (e.g., long short-term memory (LSTM) network) to detect the stationary objects <NUM>. The machine-learned model <NUM> is trained to receive the range-time maps and extracted features to perform stationary object detection. The output of the machine-learned model <NUM> can include an identification of the stationary objects <NUM>.

The machine-learned model <NUM> can be or include one or more various types of machine-learned models. The machine-learned model <NUM> can perform classification, clustering, tracing, and/or other tasks in some implementations. For classifications, the machine-learned model <NUM> can be trained using supervised learning techniques. For example, the stationary object detection module <NUM> can train the machine-learned model <NUM> with training data (e.g., truth data) that includes range-time maps and extracted features corresponding to stationary objects with example detected objects labeled as stationary (or not stationary). The labels can be manually applied by engineers or provided by other techniques (e.g., based on data from other sensor systems). The training dataset can include range-time maps similar to those input to the machine-learned model <NUM> during operation of the vehicle <NUM>.

The machine-learned model <NUM> can be trained offline, e.g., at a training computing system and then provided for storage and implementation at one or more computing devices. For example, the training computing system can include a model trainer. The training computing system can be included in or separate from the computing device that implements the machine-learned model <NUM>. The training of the machine-learned model <NUM> is described in greater detail with respect to <FIG>.

In some implementations, the machine-learned model <NUM> can be or include one or more artificial neural networks. A neural network can include a group of connected nodes organized into one or more layers. Neural networks that include multiple layers can be referred to as deep networks. A deep network can include an input layer, an output layer, and one or more hidden layers positioned between the input and output layers. The nodes of the neural network can be connected or non-fully connected.

In other implementations, the machine-learned model <NUM> can be or include one or more recurrent neural networks. In some instances, at least some of the nodes of a recurrent neural network can form a cycle. Recurrent neural networks (e.g., a LSTM network with multiple layers) can be especially useful for processing input data that is sequential in nature (e.g., a series of frames in radar data). In particular, a recurrent neural network can pass or retain information from a previous portion of the input data sequence (e.g., an initial frame) to a subsequent portion of the input data sequence (e.g., a subsequent frame) through the use of recurrent or directed cyclical node connections.

By using low-level radar data, the radar system <NUM> and the stationary object detection module <NUM> can extract more information about the EM energy distribution across the range, Doppler, azimuth-angle, and elevation-angle dimensions than is possible using radar data in a compressed data cube (CDC) format. In addition to range, angle, and Doppler features regarding the stationary objects <NUM>, the low-level radar data allows the stationary object detection module <NUM> to capture intensity-based or geometrical features for the stationary objects <NUM>.

For example, <FIG> illustrates the vehicle <NUM> traveling on the roadway <NUM>. The radar system <NUM> detects the stationary object <NUM>. The radar system <NUM> can also track the stationary object <NUM> and extract features associated with it. As described above, the vehicle <NUM> can also include at least one automotive system that relies on data from the radar system <NUM>, such as a driver-assistance system, an autonomous-driving system, or a semi-autonomous-driving system. The radar system <NUM> can include an interface to an automotive system that relies on the data. For example, the processor <NUM> outputs, via the interface, a signal based on EM energy received by the receiver <NUM>.

Generally, the automotive systems use radar data provided by the radar system <NUM> to perform a function. For example, the driver-assistance system can provide blind-spot monitoring and generate an alert that indicates a potential collision with the stationary object <NUM>. The radar data can also indicate when it is safe or unsafe to change lanes. The autonomous-driving system may move the vehicle <NUM> to a particular location on the roadway <NUM> while avoiding collisions with the stationary object <NUM>. The radar data provided by the radar system <NUM> can also provide information about a distance to and the location of the stationary object <NUM> to enable the autonomous-driving system to perform emergency braking, perform a lane change, or adjust the speed of the vehicle <NUM>.

<FIG> illustrates an example configuration of the vehicle <NUM> with the radar system <NUM> that can use the machine-learned model <NUM> for stationary object detection. As described with respect to <FIG>, the vehicle <NUM> can include the radar system <NUM>, the processor <NUM>, the CRM <NUM>, the stationary object detection module <NUM>, and the machine-learned model <NUM>. The vehicle <NUM> can also include one or more communication devices <NUM> and one or more vehicle-based systems <NUM>.

The communication devices <NUM> can include a sensor interface and a vehicle-based system interface. The sensor interface and the vehicle-based system interface can transmit data over a communication bus of the vehicle <NUM>, for example, when the individual components of the stationary object detection module <NUM> are integrated within the vehicle <NUM>.

The stationary object detection module <NUM> can include an input processing module <NUM> with a feature extraction module <NUM>. The stationary object detection module <NUM> can also include a post processing module <NUM> with the machine-learned model <NUM>. The input processing module <NUM> can receive radar data from the receiver <NUM> as an input. Generally, the radar data is received as low-level, time-series data. In contrast, some radar systems process the radar data from the receiver <NUM> as a cloud-point representation. The low-level, time-series data can be processed by the stationary object detection module <NUM> to provide better detection resolution and extract features associated with the stationary objects <NUM>.

The input processing module <NUM> can process the radar data to generate interpolated range-angle maps, including interpolated range-azimuth maps and/or interpolated range-elevation maps. By setting the range based on a speed of the vehicle <NUM>, the interpolated range-azimuth maps represent a potential detection of stationary objects as straight lines with a <NUM>-degree angle. The interpolated range-azimuth format improves the accuracy of the stationary object detection module <NUM> by simplifying the labeling of stationary objects <NUM> for the machine-learned model <NUM>.

The stationary object detection module <NUM> or the feature extraction module <NUM> can process the interpolated range-azimuth maps to generate range-time maps for the radar data. The feature extraction module <NUM> can input the range-time maps into the machine-learned model <NUM> to make stationary object predictions. The feature extraction module <NUM> can also perform additional processing to extract features associated with the stationary objects <NUM>.

The post processing module <NUM> can perform additional processing on the stationary object predictions to remove noise. The post processing module <NUM> can then provide the stationary object detections to the vehicle-based systems <NUM>.

The vehicle <NUM> also includes the vehicle-based systems <NUM>, such as an assisted-driving system <NUM> and an autonomous-driving system <NUM>, that rely on data from the stationary object detection module <NUM> to control the operation of the vehicle <NUM> (e.g., braking, lane changing). Generally, the vehicle-based systems <NUM> can use data provided by the stationary object detection module <NUM> to control operations of the vehicle <NUM> and perform certain functions. For example, the assisted-driving system <NUM> can alert a driver of the stationary object <NUM> and perform evasive maneuvers to avoid a collision with the stationary object <NUM>. As another example, the autonomous-driving system <NUM> can navigate the vehicle <NUM> to a particular destination to avoid a collision with the stationary object <NUM>.

<FIG> illustrates an example conceptual diagram <NUM> of the radar system <NUM> that uses the machine-learned model <NUM> for stationary object detection. As described with respect to <FIG>, the vehicle <NUM> can include the input processing module <NUM>, the feature extraction module <NUM>, and the post processing module <NUM> of the stationary object detection module <NUM>. The conceptual diagram <NUM> illustrates example inputs, outputs, and operations of the stationary object detection module <NUM>, but the stationary object detection module <NUM> is not necessarily limited to the order or combinations in which the inputs, outputs, and operations are shown herein. Further, any one or more of the operations may be repeated, combined, or reorganized to provide other functionality.

The radar system <NUM> provides time-series frames of EM energy as radar data <NUM> to the input processing module <NUM>. The radar data <NUM> is low-level radar data that can include more information than point-cloud data, which some radar systems use for stationary object detection. Because the stationary object detection module <NUM> uses the radar data <NUM>, it does not require additional input data from other sensors (e.g., a camera or lidar system) to detect stationary objects. The radar data <NUM> includes information associated with the stationary objects <NUM> in multiple dimensions, including range space, Doppler space, elevation space, and azimuth space. The radar data <NUM> can include beam vectors that encompass all ranges and Doppler bins. In some implementations, the stationary object detection module <NUM> may use only the magnitude information of the radar data <NUM> and not the phase information. In this way, the stationary object detection module <NUM> can assume that the non-zero yaw rate is not applicable to detecting stationary objects <NUM>.

At operation <NUM>, the input processing module <NUM> performs initial processing on the radar data <NUM>. The input processing module <NUM> generates stationary Doppler beam vectors from the radar data <NUM>. The beam vectors can encompass all ranges and Doppler bins and include intensity data related to nearby objects (e.g., both stationary and moving objects) fused together. The input processing module <NUM> performs additional processing on the beam vectors to separate the intensity data related to the various detected objects.

At operation <NUM>, the input processing module <NUM> performs super resolution on the stationary Doppler beam vectors across the azimuth plane. In other implementations, the super-resolution operation can be performed across a different plane (e.g., the elevation plane). The super resolution operation can include Fourier Transforms and iterative adaptive approach (IAA). Super resolution is applied to produce azimuth and/or elevation angle data at each range bin by collapsing the Doppler dimension. For stationary objects, the stationary object detection module <NUM> focuses on the range-azimuth spectrums, and the input processing module <NUM> generates range-angle maps, including range-azimuth maps and/or range-elevation maps, of the radar data for each time frame.

Consider that the stationary object detection module <NUM> processes a series of range-azimuth maps collected at different times (e.g., successive time frames) by the radar system <NUM>. In general, object-detection algorithms work best when the host vehicle (e.g., the vehicle <NUM>) and each stationary object (e.g., the stationary object <NUM>) move a constant range (e.g., distance) relative to one another in between data captures. Although data captures are generally equally spaced in time, the host vehicle and stationary objects are not always equally spaced in range because the vehicle velocity can vary among data captures.

At operation <NUM>, the input processing module <NUM> performs range sampling and interpolation on the range-azimuth maps. The input processing module <NUM> can create a set of looks-one for each range bin through which the vehicle <NUM> moves by interpolating the actual data captures or time frames. For a given range bin, the input processing module <NUM> can generate an interpolated time frame by determining the time at which the vehicle reaches the range bin (e.g., "the range-bin time"). The input processing module <NUM> then selects data captures with time frames that bracket the range-bin time (e.g., "the bracket data captures"). The range position of the vehicle <NUM> is determined at the bracket data captures. The input processing module <NUM> then shifts the bracket data captures in the range dimension so that the location of the vehicle <NUM> and the stationary objects <NUM> match those in the interpolated time frame to be generated. The intensities in the two shifted looks can be combined using a weighted average based on the relative differences between the bracket data captures and the interpolated time frame. Alternatively, a filtered intensity value (e.g., the minimum intensity value, average intensity value, or maximum intensity value) of the two bracket data captures can be used to create the interpolated time frame. This latter approach can suppress transient signals. The input processing module <NUM> can combine the interpolated time frame to generate interpolated range-azimuth maps <NUM> of the radar data.

The input processing module <NUM> can also perform range down-sampling before performing the interpolation operations. The range-azimuth maps can be down-sampled in range by a given factor. The input processing module <NUM> can, for example, perform the range down-sampling by taking a filtered intensity value (e.g., the maximum intensity, average intensity, or minimum intensity) at each azimuth point (or elevation point in different implementations) over N consecutive range bins, where N is a positive integer. In this way, the input processing module <NUM> effectively compresses the range-azimuth data by a factor of N in the range dimension. The range-bin size and other range-dependent variables are multiplied by a factor of N. The increased range-bin size causes the input processing module <NUM> to produce a factor of N fewer interpolated time frames. Coupled with the compression in range by a factor of N, the reduction in interpolated time frames by a factor of N can result in an approximately N-squared (N<NUM>) decrease in run time for subsequent processing. In this way, the described operations of the stationary object detection module <NUM> can be performed in real-time or near real-time despite using radar data <NUM> or time-series frames of the received EM energy as the input. Similarly, the machine-learned model <NUM> can process the radar data quicker and/or have a smaller size.

At operation <NUM>, the input processing module <NUM> performs image-signal processing on the interpolated range-azimuth maps <NUM> to generate range-time maps <NUM>. The input processing module <NUM> stacks the vectors generated by the interpolated range-azimuth maps <NUM> to generate the range-time maps <NUM>. Because the stationary objects <NUM> move one range bin for each interpolated time frame, the interpolation causes potential detections of the stationary objects <NUM> to appear as straight, <NUM>-degree lines in the range-time maps <NUM>. Depicting the stationary objects <NUM> as straight, <NUM>-degree lines simplifies and improves the accuracy of labeling for the machine-learned model <NUM>. The interpolation also ensures that the stationary object <NUM> is present at each range bin and time frame in the range-time maps <NUM>, which also simplifies the labeling of the stationary objects <NUM> for the machine-learned model <NUM>. The interpolation operation performed by the input processing module <NUM> also results in the detection of stationary objects being independent of the velocity of the vehicle <NUM>. Example range-time maps generated from range-azimuth maps and interpolated range-azimuth maps are described below with respect to <FIG>, respectively.

The input processing module <NUM> processes the range-time maps <NUM> to identify the stationary objects <NUM> from the azimuth, elevation, and range-rate dimensions and collapse the data cube along these dimensions. The data cube results in a vector of energy distribution of each potential stationary object <NUM> along the range dimension.

At operation <NUM>, the feature extraction module <NUM> extracts, using the range-time maps <NUM>, features of the stationary objects <NUM> and labels the stationary objects <NUM>. As described in greater detail with respect to <FIG>, similar operations can be performed to train the machine-learned model <NUM>. Because it can be difficult to differentiate stationary objects from radar reflections received for other objects (e.g., a guardrail, moving vehicles), the feature extraction module <NUM> performs several steps to label and identify features associated with the stationary objects <NUM>. The range-time maps <NUM> are processed using a sliding window to produce window-based range-time maps. For example, the window can include desired Doppler bins at each range bin. The window-based range-time maps are then fed into a pre-trained encoder-based feature extractor.

The machine-learned model <NUM> can then generate predicted detections <NUM> regarding the stationary objects <NUM>. The machine-learned model <NUM> can use an acyclic graph model to assist with the stationary object detection. An acyclic graph model identifies and retains a trace of each stationary object <NUM>. As a result, for a given value in a time-series signal, the time frame closer to it will have a more significant impact than time frames further away.

The machine-learned model <NUM> can include a convolutional neural network. The machine-learned model <NUM> can also use pre-trained deep learning models within the convolutional neural network to extract features and feed the extracted features into a deep sequence model (e.g., a long short-term memory (LSTM) network with multiple layers) that is trained to detect the presence of stationary objects and produce detection probabilities <NUM>. The detection probabilities <NUM> provide a range-time probability map that includes predictions across each range bin and each time frame.

At operation <NUM>, the post processing module <NUM> performs additional processing to smooth out (e.g., remove noise) the detection probabilities <NUM> by applying a smoothing function and generating detections <NUM>. The detections <NUM> can include detection probabilities at each range bin for each time frame for the stationary objects <NUM>. The smoothing function to generate the final output probability of the detections <NUM> at a specific range is given by Equation (<NUM>): <MAT> where p(t - <NUM>) represents the prediction probability of the machine-learned model <NUM> at each range for a set of time frames, including the present time frame; p(t) represents the output of the post processing operation <NUM> that is a smooth pseudo probability of an object detected at the input range; and α and β represent parameters that control the level of smoothness. For example, the post processing module <NUM> can set α equal to <NUM> and β equal to <MAT>. The time frames are the past intensity values that are used to form the input feature set to the machine-learned model <NUM>. The post processing module <NUM> can apply post processing to each frame to give a smoothed probability value for the detections <NUM>. The probability value represents the confidence level of the detections <NUM>.

<FIG> illustrate an example range-time map <NUM> and <NUM> generated from range-azimuth maps and interpolated range-azimuth maps, respectively, for a detection <NUM> of the stationary object <NUM>. The maps <NUM> and <NUM> provide range <NUM> as the y-axis to indicate the range between the vehicle <NUM> and the stationary object <NUM>.

The range-time map <NUM> provides time frame <NUM> as the x-axis to indicate the look or data capture. In contrast, the interpolated range-time map <NUM> provides interpolated time frame <NUM> as the x-axis to indicate the interpolated look. Based on the interpolation operation described with respect to <FIG>, the interpolated frame <NUM> does not have a constant interval between consecutive looks. In other words, the interpolated time frame <NUM> is not equispaced in time.

The range-time map <NUM> depicts the detection <NUM> of the stationary object <NUM> as a curved line. As it approaches the stationary object <NUM>, the vehicle <NUM> can experience velocity changes and the range <NUM> to the stationary object <NUM> will change non-linearly over the time frames <NUM>. In addition, the slope of the detection <NUM> in the range-time map <NUM> will vary depending on the velocity of the vehicle <NUM>, and changes in the slope will depend on changes in the velocity of the vehicle <NUM>. In contrast to the range-time map <NUM>, interpolation by the input processing module <NUM> causes the detection <NUM> of the stationary object <NUM> in the range-time map <NUM> to appear as a straight, <NUM>-degree line because the stationary object <NUM> moves one range bin for each interpolated time frame <NUM>.

<FIG> illustrates an example conceptual diagram <NUM> for training the machine-learned model <NUM> for stationary object detection. In particular, the conceptual diagram <NUM> illustrates feature extraction and labeling process of the stationary object detection module <NUM> to train the machine-learned model <NUM>. The conceptual diagram <NUM> illustrates example inputs, outputs, and operations of the stationary object detection module <NUM>, but the stationary object detection module <NUM> is not necessarily limited to the order or combinations in which the inputs, outputs, and operations are shown herein. Further, any of one or more of the operations may be repeated, combined, or reorganized to provide other functionality.

At operation <NUM>, the feature extraction module <NUM> extracts a feature vector from the range-time maps <NUM>. The feature vector represents the intensity in various ranges across time frames in a range-time map. The feature extraction module <NUM> uses the feature vector to represent what is in the range-time map <NUM> and where that feature appears in a range. To determine what is in the range-time map <NUM>, the feature extraction module <NUM> takes the intensity values along a <NUM>-degree line for a specific time frame in the past and augments it with the range bin to know precisely where the target appears. By including range in the feature vector, the machine-learned model <NUM> can detect distant stationary objects <NUM> even when the target intensity is weak.

The feature set <NUM> is taken from a <NUM>-degree line in the range-time map <NUM>. Because potential detections of stationary objects in the range-time maps <NUM> move one range bin for each time frame and are represented by a straight <NUM>-degree line, the feature extraction module <NUM> can label each stationary object <NUM> and non-stationary object as "<NUM>" or "<NUM>," respectively. Each target in a set of range-time maps <NUM> is taken through the feature extraction and labeling process to prepare a training dataset (e.g., truth data) for the machine-learned model <NUM>.

The machine-learned model <NUM> can be trained offline or online. In offline training (e.g., batch learning), the machine-learned model <NUM> is trained on a static training data set. In online training, the machine-learned model <NUM> is continuously trained (or re-trained) as new training data become available (e.g., while the machine-learned model <NUM> is used to perform stationary object detection).

Centralized training of multiple machine-learned models <NUM> (e.g., based on a centrally stored dataset) may be performed. In other implementations, the trainer can use decentralized training techniques, including distributed training or federated learning, to train, update, or personalize the machine-learned model <NUM>.

Once the training is completed, the machine-learned model <NUM> can be deployed in an inference stage. In the inference stage, the machine-learned model <NUM> can receive as input multiple range bins, including all available range bins, at each time frame of the range-time maps <NUM>. During the inference stage, the radar data <NUM> is passed through the stationary object detection module <NUM> to generate predictions by the machine-learned model <NUM>. In the inference stage, each input feature is multiplied with trained weights to produce a probability at each range bin and each time frame.

<FIG> illustrates a flow diagram of an example method <NUM> of a radar system <NUM> that uses a machine-learned model <NUM> for stationary object detection. Method <NUM> is shown as sets of operations (or acts) performed, but not necessarily limited to the order or combinations in which the operations are shown herein. Further, any of one or more of the operations may be repeated, combined, or reorganized to provide other methods. In portions of the following discussion, reference may be made to the environment <NUM> of <FIG> and entities detailed in <FIG>, reference to which is made for example only. The techniques are not limited to performance by one entity or multiple entities.

At <NUM>, a model is generated to process radar data and detect stationary objects. For example, a machine-learning model can be generated to process range-time maps and label stationary objects. The machine-learning model can be generated using the techniques and systems described with respect to <FIG>.

At <NUM>, the model can be trained to process the radar data and detect the stationary objects. For example, the machine-learning model can be trained to process the radar data, including range-time maps, and detect and label the stationary objects <NUM>. In addition, the machine-learning model can be trained to extract features associated with the stationary objects <NUM>. Blocks <NUM> and <NUM> are optional operations of the method <NUM>, which may be performed by different systems or components, at different times, and/or at different locations than blocks <NUM> through <NUM>.

At <NUM>, one or more processors of a radar system receive the radar data that includes time-series frames associated with EM energy. For example, the processor <NUM> of the radar system <NUM> receives radar data that includes time-series frames associated with the EM energy. The EM energy can be received by the receiver <NUM> of the radar system <NUM>. The EM energy is the radar data <NUM> as described with respect to <FIG>. The processor <NUM> can also process the EM energy by generating one or more Doppler beam vectors of the EM energy by using the time-series frames. The processor <NUM> can use the one or more Doppler beam vector and a super-resolution operation to generate a range-azimuth map of the EM energy for each of the time-series frames. Using an interpolation operation as described in greater detail with respect to <FIG>, the processor <NUM> can then generate an interpolated range-azimuth map of the EM energy. Before performing the interpolation operation, the processor <NUM> can also down-sample the range-azimuth map in range by taking a maximum intensity at each azimuth point over a number of consecutive range bins to effectively compress the range-azimuth map.

At <NUM>, the one or more processors generate a range-time map of the EM energy using the radar data. For example, the processor <NUM> processes the radar data <NUM> to generate the interpolated range-azimuth maps <NUM>. The interpolated range-azimuth maps <NUM> can be generated after the input processing module <NUM> performs initial processing, super resolution across one or more planes (e.g., the azimuth plane), range sampling, and/or interpolation on the radar data <NUM>. The input processing module <NUM> can then perform additional processing to generate the range-time maps <NUM> using the interpolated range-azimuth maps <NUM>.

At <NUM>, the one or more processors detect, using a machine-learned model, stationary objects. The machine-learned model is configured to input extracted features corresponding to the stationary objects from the range-time map for multiple range bins at each time frame of the radar data. For example, the processor <NUM> can input extracted features corresponding to the stationary objects <NUM> from the range-time maps <NUM> to the machine-learned model <NUM> to detect the stationary objects <NUM>. The machine-learned model <NUM> can input the extracted features for multiple range bins at each time frame of the radar data. The extracted features can include range, target angle, velocity, geometrical features, and intensity-based features of the stationary objects <NUM>.

The method <NUM> can return to block <NUM> to optionally retrain or update the machine-learned model <NUM> with additional truth data. For example, an implementation of the machine-learned model <NUM> can be updated based on additional or new truth data obtained and processed at another computing system and/or location.

In the following section, examples, not according to the invention as claimed, are provided.

Example <NUM>. A method comprising: receiving radar data, the radar data comprising time-series frames associated with electromagnetic (EM) energy; generating, using the radar data, a range-time map of the EM energy; and detecting, using a machine-learned model, stationary objects in a roadway, the machine-learned model being configured to input extracted features corresponding to the stationary objects from the range-time map for multiple range bins at each of the time-series frames.

Example <NUM>. The method of example <NUM>, the method further comprising: receiving the radar data by processing the EM energy when received at an antenna by at least: generating, using the time-series frames, a Doppler beam vector of the EM energy; generating, using the Doppler beam vector and a super-resolution operation, a range-angle map of the EM energy for each of the time-series frames, an angle of the range-angle map including an elevation angle or an azimuth angle; and generating, using the range-angle map and an interpolation operation, an interpolated range-angle map of the EM energy.

Example <NUM>. The method of example <NUM>, wherein: the angle is the azimuth angle; the range-angle map is a range-azimuth map; the interpolated range-angle map is an interpolated range-azimuth map; and the range-time map represents a potential detection of a stationary object of the stationary objects as a straight, <NUM>-degree line.

Example <NUM>. The method of example <NUM>, wherein the super-resolution operation can include at least one of a Fourier transform or an iterative adaptive approach to produce azimuth-angle data at each range bin.

Example <NUM>. The method of example <NUM> or <NUM>, wherein the interpolation operation comprises: determining, for each range bin of multiple range bins within the range-azimuth map, a range-bin time, the range-bin time being the time-series frame at which a vehicle reaches the range bin, the radar system being attached to a portion of the vehicle; selecting data captures of the radar data with time-series frames that bracket the range-bin time; determining a range position of the vehicle at the data captures that bracket the range-bin time; and shifting the data captures that bracket the range-bin time in a range dimension so that a location of the vehicle and the stationary object match those in an interpolated time-series frame.

Example <NUM>. The method of example <NUM>, wherein shifting the data captures that bracket the range-bin time in the range dimension comprises: using a weighted average of the data captures that bracket the range-bin time, the weighted average based on relative differences between the data captures that bracket the range-bin time and the interpolated time frame; or selecting a filtered intensity value of the data captures that bracket the range-bin time.

Example <NUM>. The method of any one of examples <NUM> through <NUM>, wherein processing the EM energy received by the antenna of the radar system further comprises: down-sampling the range-azimuth map in range by taking a filtered intensity at each azimuth point over a number of consecutive range bins to effectively compress the range-azimuth map.

Example <NUM>. The method of example <NUM>, wherein processing the EM energy received by the antenna of the radar system further comprises: multiplying a range-bin size by the number to effectively reduce a number of interpolated time frames by the number.

Example <NUM>. The method of any one of the preceding examples, wherein the time-series frames includes information associated with the stationary objects in multiple dimensions including at least three of a range dimension, a Doppler dimension, an elevation-angle dimension, or an azimuth-angle dimension.

Example <NUM>. The method of any one of the preceding examples, wherein the extracted features include at least two of range, target angle, velocity, geometrical features, or intensity-based features of the stationary objects.

Example <NUM>. The method of any one of the preceding examples, wherein the machine-learned model comprises a long short-term memory (LSTM) network with multiple layers.

Example <NUM>. A system comprising a radar system including one or more processors configured to detect stationary objects by performing the method of any one of the preceding examples.

Example <NUM>. The system of example <NUM>, wherein the radar system is configured to be integrated in or installed in a vehicle.

Example <NUM>. The system of example <NUM> or <NUM>, wherein the machine-learned model comprises a long short-term memory (LSTM) network with multiple layers.

Claim 1:
A method comprising:
receiving radar data (<NUM>) that comprises multiple time-series frames associated with electromagnetic, EM, energy reflected by one or more objects in a roadway and received at an antenna of a radar system (<NUM>) attached to a portion of a vehicle (<NUM>), each time-series frame of the radar data (<NUM>) including multiple range bins;
generating, using the time-series frames of the radar data (<NUM>), a Doppler beam vector of the EM energy;
generating, using the Doppler beam vector and a super-resolution operation, a range-azimuth map of the EM energy for each time-series frame of the multiple time-series frames;
generating, using the range-azimuth map, an interpolated range-azimuth map (<NUM>) of the EM energy by:
determining, for each range bin of multiple range bins within the range-azimuth map, a range-bin time, the range-bin time being the time-series frame at which the vehicle (<NUM>) reaches the range bin;
selecting, for a respective range-bin time, bracketing data captures of the radar data (<NUM>) that have a time-series frame immediately before and immediately after the respective range-bin time;
determining a range position of the vehicle (<NUM>) at the bracketing data captures; and
shifting range values of the bracketing data captures in a range dimension so that a location of the vehicle (<NUM>) and a stationary object (<NUM>) among the one or more objects match those in an interpolated time-series frame;
generating, using the interpolated range-azimuth map (<NUM>), a range-time map of the EM energy, the range-time map representing a potential detection of the stationary object (<NUM>) among the one or more objects as a straight, <NUM>-degree line; and
detecting, using a machine-learned model (<NUM>), stationary objects (<NUM>) among the one or more objects, the machine-learned model (<NUM>) being configured to receive as inputs extracted features corresponding to the one or more objects from the range-time map for multiple range bins at each of the time-series frames, the extracted features including at least two of range, target angle, velocity, geometrical features, or intensity-based features of the one or more objects.