Patent Description:
<CIT> relates to a method and a device for evaluating radar systems. Further prior art is mentioned in <CIT>, <CIT> and <CIT>.

This document describes techniques and systems for fuzzy labeling of low-level electromagnetic sensor data. Sensor data in the form of an energy spectrum is obtained and the points within an estimated geographic boundary of a scatterer represented by the smear is labeled with a value of one. The remaining points of the energy spectrum are labeled with values between zero and one with the values decreasing the further away each respective remaining point is from the geographic boundary. The fuzzy labeling process may harness more in-depth information available from the distribution of the energy in the energy spectrum. A model can be trained to efficiently label an energy spectrum map in this manner. This may result in lower computational costs than other labeling methods. Additionally, false detections by the sensor may be reduced resulting in more accurate detection and tracking of objects.

This Summary introduces simplified concepts related to fuzzy labeling of low-level electromagnetic sensor data, further described in the Detailed Description and Drawings. This Summary is not intended to identify essential features of the claimed subject matter, nor is it intended for use in determining the scope of the claimed subject matter. Although primarily described in the context of automotive radar sensors, the techniques for fuzzy labeling of low-level electromagnetic sensor data can be applied to other applications where robust labeled data for training models related to radar sensors is desired.

The details of one or more aspects of fuzzy labeling of low-level electromagnetic sensor data are described in this document with reference to the following figures, in which same numbers are used throughout to indicate like components:.

Labeling radar sensor data independently can be rare due to spectrum smearing (e.g., radar leakage in side angular bins) of targets in various dimensions (e.g., azimuth, elevation) and the difficulty of separating and segmenting objects. For example, a scatterer (e.g., object) of finite size produces a radar return that is a sine function. The energy distribution of this function along a particular dimension (e.g., azimuth angle), may show a peak at the location of the scatterer and sidelobes that diminish in intensity on either side of the peak. Thus, it can be challenging to segment an object or discern its boundaries in a radar return. In contrast, light detection and ranging (LiDAR) systems produce distinct three-dimensional point clouds, and cameras produce objects with clear boundaries in the two angular dimensions. The segmentation and labeling of objects are more accessible with LiDARs and cameras.

Some current approaches to labeling radar data may use the rigid labels produced by LiDARs or cameras that simultaneously collect data with the radar system. The object labels can be produced using the LiDAR or camera images and then be projected onto the radar data to define the rigid boundaries of the various labeled objects.

Some existing methods for labeling LiDAR and camera images segment the object using its geometric boundary and apply a binary label to a pixel in the image depending on whether the pixel lies within or outside of the boundary. Pixels inside the boundary may be labeled as <NUM> (positive) or <NUM> (negative). Due to radar spectrum smearing, the geometric boundary of the object is difficult, if not impossible, to determine from radar data alone. Further, if the geometric boundary derived from a LiDAR or camera image is projected onto radar data and binary labeling is applied, useful information in the extended radar reflection spectrum may be lost. Likewise, there is no geometric information in the range rate dimension, which is essential for detecting and classifying a moving object. If the radar data is labeled in a binary manner (positive and negative only), the neural network is trained to learn a rigid boundary, for example <MAT> which is not easily discernible in radar data and more difficult to learn for a neural network.

In contrast, this document describes a fuzzy labeling system for radar data that segments and labels objects in the radar signal space (e.g., a radar spectrum map, radar image) on a pixel level. In this context, "fuzzy labeling" refers to a non-binary labeling method that assigns labels between <NUM> and <NUM> to pixels in the radar return (e.g., the radar spectrum smear) outside of the geometric boundary of the object as determined from LiDAR or camera data. Each pixel is not labeled as an independent data point. The properties and labels of the pixels are correlated spatially and temporally.

The fuzzy labeling system, as described herein, is based on radar signals (e.g., radar returns) and considers the soft boundary of objects seen in radar imagery. This fuzzy labeling process uses low-level radar data (e.g., time series, uncompressed data cube, lossless fast Fourier transform (FFT)) that harnesses in-depth information available from the distribution of the energy corresponding to the target across range, range rate, azimuthal, and elevation range. The fuzzy labeling system can localize the object in each of the four dimensions and can segment the energy distribution, containing useful information for detection, tracking, classification, and the like, corresponding to objects. This procedure may maximize the information extracted from the radar signal and can be used as inputs into a neural network that can be trained to detect and classify moving and stationary objects. In this manner, false positives may be reduced during object detection and tracking without degrading the system's dynamic range. Additionally, a model trained with the fuzzy labeling system, as described herein, may adapt to different driving scenarios, including scenarios not included in the training, while maintaining performance of the trained model.

<FIG> illustrates an example training environment <NUM> for fuzzy labeling of low-level electromagnetic sensor data, in accordance with techniques of this disclosure. In this example, a first sensor on a vehicle <NUM> is described as a LiDAR sensor <NUM>, and a second sensor on the vehicle <NUM> is described as a radar sensor <NUM>. However, the first sensor can be any imaging sensor such as an optical camera or a thermal camera.

The example training environment <NUM> can be a controlled environment that the vehicle <NUM> including the LiDAR sensor <NUM> and the radar sensor <NUM> uses to collect sensor data about the training environment <NUM>. The training environment <NUM> includes one or more objects such as object <NUM> and object <NUM>.

The LiDAR sensor <NUM> collects LiDAR data that includes geometric locations of the objects <NUM> and <NUM>. The geometric locations may include specific geometric boundaries of the objects <NUM> and <NUM> such as can be determined in the LiDAR point cloud. The radar sensor <NUM> collects low-level radar data that may have higher resolution in some dimensions (e.g., range, Doppler) and lower resolution in other dimensions (e.g., azimuth, elevation). This lower resolution can be represented as an energy spectrum smear when the radar data is represented as an energy spectrum map. The energy spectrum map can be two dimensional representing dimensions such as range and time or elevation and time.

A fuzzy labeling model training system <NUM> can obtain the LiDAR data and the low-level radar data by over-the-air (OTA) means or by other methods from the vehicle <NUM>. In other aspects, the fuzzy labeling model training system <NUM> may reside in the vehicle <NUM>. The LiDAR data can be input into the fuzzy labeling model training system <NUM> as LiDAR input data <NUM>. The LiDAR input data <NUM> serves as the ground truth for training a model. Likewise, the radar data can be input into the fuzzy labeling model training system <NUM> as radar input data <NUM>. The fuzzy labeling model training system <NUM> can include a combination of hardware components and software components executing thereon. For example, a non-transitory computer-readable storage media (CRM) of the fuzzy labeling model training system <NUM> may store machine-executable instructions that, when executed by a processor of the fuzzy labeling model training system <NUM>, cause the fuzzy labeling model training system <NUM> to output a trained model <NUM> based on fuzzy labeling of low-level electromagnetic sensor data. The fuzzy labeling model training system <NUM> also includes a LiDAR data processing module <NUM>, a radar image generator <NUM>, a bounding box generator <NUM>, a fuzzy labeling module <NUM>, and a machine learning module <NUM>. In other examples, the operations associated with the fuzzy labeling model training system <NUM> can be performed using a different arrangement or quantity of components than that shown in <FIG>.

The LiDAR data processing module <NUM> can identify the geographic locations, and specifically, the geographic boundaries of objects <NUM> and <NUM>. The LiDAR data processing module can output this geographic boundary information to the bounding box generator <NUM>.

The radar image generator <NUM> can generate an energy spectrum map based on the radar input data <NUM>. The energy spectrum map may be, for example, a range-time map generated by selecting certain azimuth and Doppler bins and collapsing the elevation dimension. This can generate the amplitude of various ranges in time to make the range-time map. Similarly, other dimensions may be represented in an energy spectrum map. The energy spectrum map may also be referred to as a radar image since it can be analyzed and manipulated like a camera image. The spectrum smear (e.g., radar spectrum smear) may be observed on the radar image as high intensity pixels (e.g., bright pixels) surrounded by pixels that lose their intensity as they spread away from the high intensity pixels. The spectrum smear can be caused by radar leakage in side angular bins and may contribute to the noise levels of the radar sensor <NUM>. The energy spectrum map may be represented as a radar image, similar to a camera image, having pixels that can be analyzed based on various characteristics (e.g., location within the image, intensity). The radar image generated by the radar image generator <NUM> can be output to the bounding box generator <NUM>.

The bounding box generator <NUM> can receive the geographic boundaries of objects <NUM> and <NUM> and generate a first bounding box based on the geographic boundaries of the objects <NUM> and <NUM>. Likewise, the bounding box generator <NUM> can determine radar smears that may be associated with the objects <NUM> and <NUM> and generate a second bounding box that encompasses the associated radar smears. The bounding box generator <NUM> can project the first bounding box and the second bounding box onto the radar image. Based on the projection of the first bounding box and the second bounding box onto the radar image, a first portion of the radar smear can be identified. The first portion is the union of the first bounding box and the second bounding box. A second portion of the radar smear encompasses the remainder of the radar smear not included in the first portion.

The fuzzy labeling module <NUM> can label the first portion of the radar smear with a highest value (e.g., a value of one on a scale from zero to one). The fuzzy labeling module <NUM> can analyze the second portion of the radar smear and label the pixels in the second portion with a value between a lowest value and the highest value. The pattern of labeling of the pixels in the second portion may generally be from higher values to lower values depending upon their distance from the first portion. For example, the pixels in the second portion may be labeled, from closest to the first portion to the farthest from the first portion as a pattern such as <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>. The decreasing pattern may be linear, exponential, Gaussian, or any other continuous function. Any pixels in the radar image not associated with the first portion or second portion of the radar smear can be labeled a lowest value (e.g., zero).

The labels generated by the fuzzy labeling module <NUM> can be used as labels for the machine learning module <NUM>. Because the fuzzy labels are based on an image (e.g., the radar image) neural networks that are conducive to processing images may be used by the machine learning module. One such example of a neural network that may be used is a convolutional neural network (CNN), although others may be used. The machine learning module <NUM> outputs the trained model <NUM>, which can then be deployed to automotive systems. Because the trained model <NUM> has been trained in a controlled environment using machine learning techniques, it can be very efficient in terms of computations and memory. This efficiency enables the trained model <NUM> to be used in automotive applications that have limited computing resources. Further, the trained model <NUM> may result in less false positives detected by a radar system when used in an uncontrolled environment.

In some aspects, the fuzzy labeling model training system <NUM> may reside in the vehicle <NUM>, and the trained model <NUM> may be constantly retrained with sensor data received by the vehicle <NUM> in uncontrolled environments. Further, the retrained model may be deployed to other vehicles. Other vehicles may also include other fuzzy labeling model training system <NUM>. Each respective vehicle may upload their respective retrained model and/or their respective sensor data to a cloud. In these aspects, the trained model <NUM> may be retrained with very large data sets.

<FIG> illustrates an example environment <NUM> in which fuzzy labeling of low-level electromagnetic sensor data can be applied, in accordance with techniques of this disclosure. In the depicted environment <NUM>, a vehicle <NUM> travels on a roadway by at least partially relying on output from a radar system <NUM>. Although illustrated as a passenger car, the vehicle <NUM> can represent other types of motorized vehicles (e.g., truck, motorcycle, bus, tractor, semi-trailer truck), non-motorized vehicles (e.g., a bicycle), railed vehicles (e.g., a train), watercraft (e.g., a boat), aircraft (e.g., an airplane), spacecraft (e.g., satellite), and the like.

The radar system <NUM> can be installed on, mounted to, or integrated with any moving platform, including moving machinery or robotic equipment. Components of the radar system <NUM> can be arranged in a front, back, top, bottom, or side portion of the vehicle <NUM>, within a bumper, integrated into a side mirror, formed as part of a headlight and/or taillight, or at any other interior or exterior location where objects require detection. The vehicle <NUM> may include multiple radar systems <NUM>, such as a first radar system and a second radar system, to provide a custom field-of-view <NUM> that encompasses a particular region of interest outside the vehicle <NUM>. Acting as part of a perception system of the vehicle <NUM>, the radar system <NUM> aids in driving the vehicle <NUM> by enabling advanced safety or autonomous driving features. Vehicle subsystems may rely on the radar system <NUM> to detect whether any objects (e.g., an object <NUM>) appear in the environment <NUM> within a particular field of view (FOV) <NUM>.

The radar system <NUM> is configured to detect the object <NUM> by radiating the object <NUM> within the field of view <NUM>. For example, the object <NUM> can be a stationary or moving object and includes one or more materials that reflect radar signals. The object <NUM> may be another vehicle, a traffic sign, a barrier, an animal, a pedestrian, or any other object or debris.

The radar system <NUM> can include a combination of hardware components and software components executing thereon. For example, a non-transitory computer-readable storage media (CRM) of the radar system <NUM> may store machine-executable instructions that, when executed by a processor of the radar system <NUM>, cause the radar system <NUM> to output information about objects detected in the field of view <NUM>. As one example, the radar system <NUM> includes a signal processing component that may include a radar monolithic microwave integrated circuit (MMIC) <NUM>, a trained model processor <NUM>, and a radar processor <NUM>. The radar MMIC <NUM>, the trained model processor <NUM>, and the radar processor <NUM> may be physically separated components, or their functionality may be included within a single integrated circuit. Other processors may, likewise, be present in some aspects. In this example, the radar system <NUM> also includes a fuzzy labeling module, a thresholding module <NUM>, and an output manager <NUM>. In other examples, the operations associated with the radar system <NUM> can be performed using a different arrangement or quantity of components than that shown in <FIG>. These components receive radar signals to generate detections <NUM> and a refined radar image <NUM> (e.g., a radar image with reduce false positives). The detections <NUM> and the refined radar image <NUM> can be used to update object tracks and classify objects.

For example, the radar MMIC <NUM> may receive low-level radar signals from the radar system <NUM> that were transmitted by the radar system <NUM> and reflected from the object <NUM>. These low-level radar signals may be digitized raw signals, or signals that have been pre-processed (e.g., lossless FFT, uncompressed data cube) without loss of data. The low-level radar signals can be input into the fuzzy labeling module <NUM> being executed by the trained model processor <NUM>. A low-level radar image based on the low-level radar data can be generated, and the pixels of the low-level radar image can be labeled based on the trained model <NUM>. The labeled data (e.g., labeled pixels) can be output to the thresholding module <NUM> being executed on the radar processor <NUM>. The thresholding module <NUM> applies a threshold value to the labeled data and generates the refined radar image <NUM> that includes only the labeled data that is greater than the threshold value. The detections <NUM> can, likewise, be determined from the labeled data that is greater than the threshold value. The output manager <NUM> can output the detections <NUM> and the refined radar image <NUM> to other systems of the vehicle <NUM> for automotive and safety applications. In this manner, he outputted detections <NUM> and the refined radar image <NUM> may include relevant information included in the low-level radar data but lower the quantity of false detections that the radar system <NUM> may have reported without using the trained model <NUM>.

<FIG> illustrates an example vehicle <NUM>-<NUM> including a system configured to utilize a model trained using fuzzy labeling of low-level electromagnetic sensor data, in accordance with techniques of this disclosure. The vehicle <NUM>-<NUM> is an example of the vehicle <NUM>. Included in the vehicle <NUM>-<NUM> is a radar system <NUM>-<NUM>, which is an example of the radar system <NUM>. The vehicle <NUM>-<NUM> further includes a communication link <NUM> that the radar system <NUM>-<NUM> can use to communicate to other vehicle-based systems <NUM>. The communication link <NUM> may be a wired or wireless link and, in some cases, includes a communication bus (e.g., CAN bus). The other vehicle-based systems <NUM> perform operations based on information received from the radar system <NUM>-<NUM>, over the link <NUM>, such as data output from the radar system <NUM>-<NUM>, including information indicative of one or more objects identified and tracked in the FOV.

The radar system <NUM>-<NUM> includes a radar MMIC <NUM>-<NUM>, a trained model processor (e.g., embedded processor for machine learned models) <NUM>-<NUM>, and a radar processor <NUM>-<NUM>, similar to the radar system <NUM>. The radar MMIC <NUM>-<NUM> includes one or more transceivers/receivers <NUM>, timing/control circuitry <NUM> and analog-to-digital converters (ADC) <NUM>.

The radar system <NUM>-<NUM> further includes a non-transitory computer-readable storage media (CRM) <NUM> (e.g., a memory, long-term storage, short-term storage), which stores instructions for the radar system <NUM>-<NUM>. The CRM <NUM> stores a fuzzy labeling module <NUM>-<NUM>, a thresholding module <NUM>-<NUM>, and an output manager <NUM>-<NUM>. Other instructions, relevant to the operation of the radar system <NUM>-<NUM> may, likewise, be stored in the CRM <NUM>. The components of the radar system <NUM>-<NUM> communicate via a link <NUM>. For example, the trained model processor <NUM>-<NUM> receives low-level radar data <NUM> from the MMIC <NUM>-<NUM> over the link <NUM> and instructions from the CRM <NUM> to execute the fuzzy labeling module <NUM>-<NUM>. The radar processor <NUM>-<NUM> receives labeled radar images <NUM> (e.g., fuzzy labeled radar images) from the trained model processor <NUM>-<NUM>. The radar processor <NUM>-<NUM> also receives instructions from the CRM <NUM> to execute the thresholding module <NUM>-<NUM> and the output manager <NUM>-<NUM> over the link <NUM>.

The fuzzy labeling module <NUM>-<NUM> generates a low-level radar image based on the low-level radar data <NUM> and executes a model (e.g., the trained model <NUM>) trained to perform fuzzy labeling of low-level electromagnetic sensor data. The trained model labels the pixels of the low-level radar image from zero to one based on the training techniques described herein. The output of the fuzzy labeling module <NUM>-<NUM> is the labeled radar image <NUM>. The trained model may be periodically updated via over-the-air (OTA) updates or by other methods.

The thresholding module <NUM>-<NUM> receives the labeled radar image <NUM> and applies a thresholding value to the labeled pixels. The thresholding module <NUM>-<NUM> outputs the refined radar image <NUM>. The refined radar image <NUM> includes the pixels of the labeled radar image that are greater than the threshold value. The refined radar image <NUM> is made available by the output manager <NUM>-<NUM> to the other vehicle-based systems <NUM>. Detections based on the refined radar image <NUM> may also be made available to the other vehicle-based systems <NUM>.

The other vehicle-based systems <NUM> can include autonomous control system <NUM>-<NUM>, safety system <NUM>-<NUM>, localization system <NUM>-<NUM>, vehicle-to-vehicle system <NUM>-<NUM>, occupant interface system <NUM>-<NUM>, multi-sensor tracker <NUM>-<NUM>, and other systems not illustrated. Objects in the FOV can be inferred and classified based on the refined radar image <NUM> output to the other vehicle-based systems <NUM>. In this manner, the other vehicle-based systems <NUM> can receive an indication of one or more objects detected by the radar system <NUM>-<NUM> in response to the radar system <NUM>-<NUM> combining and analyzing the radar data generated by the received signals. The other vehicle-based systems <NUM> may perform a driving function or other operation that may include using output from the radar system <NUM>-<NUM> to assist in determining driving decisions. For example, the autonomous control system <NUM>-<NUM> can provide automatic cruise control and monitor the radar system <NUM>-<NUM> for output that indicates the presence of objects in the FOV, for instance, to slow the speed and prevent a collision with an object in the path of the vehicle <NUM>-<NUM>. The safety system <NUM>-<NUM> or the occupant interface system <NUM>-<NUM> may provide alerts or perform a specific maneuver when the data obtained from the radar system <NUM>-<NUM> indicates that one or more objects are crossing in front of the vehicle <NUM>-<NUM>.

<FIG> illustrates a graph <NUM> of an energy spectrum <NUM> being labeled using fuzzy labeling of low-level electromagnetic sensor data, in accordance with techniques of this disclosure. The graph <NUM> demonstrates fuzzy labeling logic in the range dimension with a linear decreasing pattern. However, any radar dimension may be demonstrated.

The energy spectrum <NUM> may be a typical energy spectrum of finite extent from a scatterer (e.g., object). The reflection center <NUM> of the energy spectrum <NUM> represents an intensity peak of the energy spectrum <NUM>. The geometric boundary <NUM> of the scatterer may not be discernible in the energy spectrum <NUM> and may be determined from data from another sensor projected onto the energy spectrum <NUM> (e.g., LiDAR data projected onto a radar image).

Some traditional labeling techniques may label the energy spectrum <NUM> using binary labeling. For example, after estimating the geometric boundary <NUM> of the scatterer on the energy spectrum <NUM>, the label portion <NUM> representing the geometric boundary <NUM> and the area within the geometric boundary may be labeled with a one. The label portions <NUM> of the energy spectrum <NUM> that lies outside the geometric boundary <NUM> may have no label (e.g., label with a value of zero).

In contrast, using fuzzy labeling logic, the reflection center <NUM> of the energy spectrum <NUM> can be labeled with a one. The fuzzy labels <NUM> descend from one to zero. Instead of labeling only the geometric boundary <NUM>, fuzzy labeling labels the entire energy spectrum <NUM> and accounts for the soft boundary of the scatterer included in the energy spectrum <NUM>. Using fuzzy labeling logic in this manner enables all the information in a radar signal to be considered. Fuzzy labeling logic may provide a machine-learned model (e.g., artificial neural network) with less ambiguous labeling and result in a more robust model for radar sensors, traffic scenarios, and types of objects (e.g., targets). Additionally, Fuzzy labeling may produce fewer false positives than traditional labeling techniques.

<FIG> illustrates a LiDAR bounding box <NUM> and a radar bounding box <NUM> projected onto a radar image <NUM> for fuzzy labeling of low-level electromagnetic sensor data, in accordance with techniques of this disclosure. The radar image exhibits areas of radiation reflected by objects and received by a radar sensor.

LiDAR point cloud <NUM> detects several objects including object <NUM>. The LiDAR bounding box <NUM> of the object <NUM> is projected on to the radar image <NUM> and represents the geometric location (e.g., geometric boundary) of the object <NUM> on a radar smear <NUM> on the radar image <NUM>.

The radar bounding box <NUM> encompasses the radar smear <NUM>. The radar bounding box <NUM> includes a first portion <NUM>-<NUM> and a second portion <NUM>-<NUM> of the radar smear <NUM>. The first portion <NUM>-<NUM> corresponds to the portion of the radar smear that is the geometric location of the object <NUM>. The first portion <NUM>-<NUM> is determined by finding the union of the LiDAR bounding box <NUM> and the radar bounding box <NUM>. The second portion <NUM>-<NUM> includes the remainder of the radar smear outside of the geometric location of the object <NUM>. For fuzzy labeling of the radar smear <NUM>, the pixels in the first portion <NUM>-<NUM> can be labeled with a value of one. The pixels in the second portion <NUM>-<NUM> can be labeled with values between zero and one with the values being higher for the pixels closer to the first portion <NUM>-<NUM> and descending the further away each respective pixel is from the first portion <NUM>-<NUM>. This fuzzy labeling system enables continuous, smooth labeling of all the pixels in the radar smear <NUM> and considers all the information contained in the radar smear <NUM>.

<FIG> illustrates an example method <NUM> for fuzzy labeling of low-level electromagnetic sensor data, in accordance with techniques of this disclosure. The fuzzy labels are used to train a machine-learned model to label electromagnetic images efficiently using little computational resources. The data can be collected in a controlled training environment where the objects are specifically placed in certain geometric locations. At step <NUM>, a geometric location (e.g., geometric boundary) of an object is identified based on first sensor data obtained from a first sensor. The first sensor may be any imaging sensor, such as a LiDAR or a camera, that can provide distinct outlines of objects in its field of view.

At step <NUM>, an energy spectrum smear on a spectrum map derived from second sensor data is identified that corresponds to the object. The spectrum map (e.g., radar image) can be derived from low-level data obtained from an electromagnetic sensor in the form of a radar sensor according to the invention. The spectrum map can be a two-dimensional map that represents dimensions of the electromagnetic sensor such as the range and time dimensions. The energy spectrum smear, or radar smear, may represent the object and energy leakage (e.g., noise) in side angular bins (e.g., data derived from side lobes of the second sensor antenna) of the second sensor data. By collapsing the elevation information in the second sensor data, the spectrum map may include the range and time dimensions.

At step <NUM>, a first portion of the energy spectrum smear is identified. The first portion corresponds to the geometric location of the object. The first portion may be identified by projecting the geometric location identified in the first sensor data onto the spectrum map. Specifically, a bounding box may be generated around the geometric location in the first sensor data image and projected onto the spectrum map to identify the first portion of the energy spectrum smear.

At step <NUM>, each pixel in the first portion of the energy spectrum smear is labeled with a value of one. Because this first portion encompasses the geometric boundary of the object, these labels can be used as ground truth for training the model.

At step <NUM>, each pixel in a second portion of the energy spectrum smear is labeled with a value between zero and one. The second portion includes all the pixels in the energy spectrum smear not included in the first portion. That is, the first portion and the second portion include all the pixels in the energy spectrum smear. The pixels in the second portion that are closer to the first portion are labeled with higher values. The values of the pixels decrease as the location of each respective pixel is further away from the first portion. For example, a pixel in the second portion that is close to the first portion may have a value of <NUM>. The next closest pixel may have a value of <NUM>, and so forth. The decreasing pattern of values of pixels may be linear, exponential, Gaussian, or any other continuous function. In some cases, the energy spectrum smear may correspond to a plurality of objects. The objects may be indistinguishable in the energy spectrum smear in these cases. However, the labeling of the pixels is the same as when the energy spectrum smear corresponds to only a single object.

At step <NUM>, a model is trained, based on the labels of the first portion and the second portion of the energy spectrum smear, to label pixels in a spectrum map used in a real world application. Because the spectrum map is essentially an image, machine learning techniques used to train image classification models can be used. For example, the model can be a convolutional neural network that is effective at classifying pixels in images. Further, extensive computing resources can be used to train the model because the training can take place outside of an automotive system (e.g., in a lab). The resulting model executable software may be very inexpensive in terms of computing resources and, thus, ideal for automotive applications. Using this method to train a model using fuzzy labeling may consider all the data that is in a received electromagnetic signal. The output of the trained model may be used to accurately detect objects while minimizing false detections.

Example <NUM>: A method comprising: identifying, based on first sensor data obtained from a first sensor being a light detection and ranging (LiDAR) sensor or a camera, a geometric location of at least one object; identifying, on a spectrum map derived from second sensor data obtained from a second sensor being a radar sensor, an energy spectrum smear that corresponds to the object, the second sensor being an electromagnetic sensor; identifying a first portion of the energy spectrum smear that corresponds to the geometric location of the object; labeling, in the first portion of the energy spectrum smear, each pixel with a value of one; labeling, in a second portion of the energy spectrum smear that includes all pixels of the energy spectrum smear not included in the first portion, each pixel with a value between zero and one, the value decreasing the further each respective pixel is from the first portion of the energy spectrum smear; and training, by machine learning and based on the labeling of each pixel in the first portion and each pixel in the second portion, a model to label a spectrum map used for detecting and tracking objects.

Example <NUM>: The method of example <NUM>, further comprising labeling each pixel in the spectrum map that is not included in the energy spectrum smear with a value of zero.

Example <NUM>: The method of any one of the preceding examples, wherein the model is a convolutional neural network.

Example <NUM>: The method of any one of the preceding examples, wherein the second sensor is a radar sensor.

Example <NUM>: The method of any one of the preceding examples, wherein the energy spectrum smear comprises radar energy leakage in side angular bins.

Example <NUM>: The method of any one of the preceding examples, wherein deriving the spectrum map comprises generating a radar image by selecting azimuth and Doppler bins.

Example <NUM>: The method of any one of the preceding examples, wherein generating the radar image further comprises collapsing elevation information in the second sensor data such that the radar image includes range and time dimensions.

Example <NUM>: The method of any one of the preceding examples, wherein identifying the geometric location of the object comprises determining a first bounding box that represents a geometric boundary of the object.

Example <NUM>: The method of any one of the preceding examples, wherein identifying the first portion of the energy spectrum smear comprises projecting the first bounding box onto the spectrum map.

Example <NUM>: The method of any one of the preceding examples, wherein identifying the energy spectrum smear comprises determining a second bounding box that includes the first portion of the energy spectrum smear and the second portion of the energy spectrum smear.

Example <NUM>: The method of any one of the preceding examples, wherein the second portion of the energy spectrum smear is a portion of the energy spectrum smear included in the second bounding box but not included in the first bounding box.

Example <NUM>: The method of any one of the preceding examples, wherein the geometric boundary of the object is used as ground truth data for training the trained model that labels each pixel of the spectrum map.

Example <NUM>: The method of any one of the preceding examples, wherein the energy spectrum smear corresponds to a plurality of objects near to one another such that each respective object of the plurality of objects is indistinguishable from the other objects in the energy spectrum smear.

Example <NUM>: A system comprising: at least one processor configured to: obtain first sensor data from a first sensor, the first sensor data based on a controlled environment including at least one object; obtain second sensor data from a second sensor, the second sensor data based on the controlled environment, the second sensor being an electromagnetic sensor; identify, based on the first sensor data, a geometric location of the object; generate, based on the second sensor data, a spectrum map exhibiting an area of radiation reflected by objects in the controlled environment; identify, on the spectrum map, an energy spectrum smear that corresponds to radiation reflected by the object; identify a first portion of the energy spectrum smear that corresponds to the geometric location of the object; label, in the first portion of the energy spectrum smear, each pixel with a value of one; label, in a second portion of the energy spectrum smear that includes all pixels of the energy spectrum smear not included in the first portion, each pixel with a value between zero and one, the value decreasing the further each respective pixel is from the first portion of the energy spectrum smear; and train, by machine learning and based on the labeling of each pixel in the first portion and each pixel in the second portion, a model to label a spectrum map used for detecting and tracking objects,.

wherein: the first sensor is one of a light detection and ranging (LiDAR) sensor or a camera; and the second sensor is a radar sensor.

Example <NUM>: The system of any one of the preceding examples, wherein the processor is configured to generate the spectrum map by at least: generating a radar image by selecting azimuth and Doppler bins; and collapsing elevation information in the second sensor data such that the radar image includes range and time dimensions.

Example <NUM>: The system of any one of the preceding examples, wherein the energy spectrum smear comprises radar energy leakage in side angular bins determined in the second sensor data.

Claim 1:
A computer-implemented method comprising:
identifying, based on first sensor data obtained from a first sensor, a geometric location of at least one object, the first sensor being a light detection and ranging (LiDAR) sensor (<NUM>) or a camera;
identifying, on a spectrum map derived from second sensor data obtained from a second sensor, an energy spectrum smear that corresponds to the object, the second sensor being an radar sensor (<NUM>);
identifying a first portion of the energy spectrum smear that corresponds to the geometric location of the object (<NUM>; <NUM>);
labeling, in the first portion of the energy spectrum smear, each pixel with a value of one;
labeling, in a second portion of the energy spectrum smear that includes all pixels of the energy spectrum smear not included in the first portion, each pixel with a value between zero and one, the value decreasing the further each respective pixel is from the first portion of the energy spectrum smear; and
training, by machine learning and based on the labeling of each pixel in the first portion and each pixel in the second portion, a model to label spectrum maps used for detecting and tracking objects.