Patent Description:
Autonomous vehicles and related systems utilize various computer-operated technologies for determining conditions around the vehicle. For instance, 3D object detection is a central task for applications such as autonomous driving, in which the system needs to localize and classify surrounding traffic agents. One technology commonly applied to this operation is LiDAR, which stands for Light Detection And Ranging. LiDAR uses lasers to create 3D representations of the surrounding environment by emitting pulsed light beams and capturing their reflection via receivers. These systems can be heavily impacted by adverse weather conditions due to the impact that precipitation and other environmental effects can have on the light beams. Further difficulties arise when attempting to collect and annotate training data in this type of setting.

According to an aspect of the present invention a computer-implemented method for simulating the effect of snowfall on LiDAR point clouds is provided, as set out in the appended set of claims.

Since adverse-weather data is difficult to collect, and thus adverse weather samples are rare and well underrepresented. This method allows for the generation of weather-based LiDAR data by simulating snowfall on real clear-weather LiDAR point clouds. This physically based simulation is realistic enough to relieve the need for real snowy training samples.

In an example, the method further comprises utilizing the partially synthetic snow LiDAR data as training data for optimizing 3D object detection methods in order to train the 3D object detection methods to better discriminate between snow particles and target objects in a real environment.

Object detection models trained using this method for generating training data with simulated snow may consistently achieve significant performance gains over baseline models trained only on clear weather and competing simulation methods.

According to the invention, simulating the effect of snowfall on the LiDAR point cloud data comprises sampling snow particles in 2D space for each scanning layer of the LiDAR sensor.

In an example, the snow particles are rendered as opaque spheres.

In an example, the method further comprises selecting a snowfall rate, wherein said snowfall rate determines the size and amount of the snow particles. This allows data to be produced that simulates snowfall of varying intensity, such as light snowfall and heavy snowfall.

In an example, the sampling is configured to obey the exclusion principle that no two snow particles intersect with each other.

In an example, augmenting the LiDAR point cloud data comprises using induced geometry to modify an attribute for each LiDAR beam.

In an example, using induced geometry comprises computing for each LiDAR beam the set of particles that intersect with it and derive the angle of the beam cross-section that is reflected by each particle, taking occlusions into account in order to produce an impulse response of the linear system in the presence of snowfall that allows an analytical calculation of the received power at the receiver.

An aspect of the present invention is directed toward a training dataset configured to be used for optimizing 3D object detection methods, produced by the method disclosed.

An aspect of the present invention is directed toward a system for generating a partially synthetic snowy LiDAR dataset, as defined in claim <NUM>.

In an example, the system further comprises a pulse emitter and a receiver, wherein the pulse emitter is configured to emit LiDAR beams and the receiver is configured to detect a reflected LiDAR beams.

In an example, the controller is configured to generate a LiDAR dataset containing real LiDAR point cloud data for a clear weather environment based on attributes about the reflections of said LiDAR beam that are detected by the receiver.

In an example, the system is configured to generate a LiDAR dataset containing real LiDAR point cloud data for a clear weather environment based on attributes of the reflections of the LiDAR beam.

In an example, the system is configured to utilize the partially synthetic snow LiDAR data as training data for optimizing 3D object detection methods in order to train the 3D object detection methods to better discriminate between snow particles and target objects in a real environment.

According to the invention, the system is configured to simulate the effect of snowfall on the LiDAR point cloud data comprises sampling snow particles in 2D space for each scanning layer of the LiDAR sensor.

In an example, the system is configured to render the snow particles as opaque spheres.

The disclosure may be more completely understood in consideration of the following detailed description of aspects of the disclosure in connection with the accompanying drawings, in which:.

The term "exemplary" is used in the sense of "example", rather than "ideal". While aspects of the disclosure are amenable to various modifications and alternative forms, specifics thereof have been shown by way of example in the drawings and will be described in detail. It should be understood, however, that the intention is not to limit aspects of the disclosure to the particular embodiment(s) described. On the contrary, the intention of this disclosure is to cover all modifications, equivalents, and alternatives falling within the scope of the disclosure.

As used in this disclosure and the appended claims, the singular forms "a", "an", and "the" include plural referents unless the content clearly dictates otherwise. As used in this disclosure and the appended claims, the term "or" is generally employed in its sense including "and/or" unless the content clearly dictates otherwise.

The following detailed description should be read with reference to the drawings. The detailed description and the drawings, which are not necessarily to scale, depict illustrative aspects and are not intended to limit the scope of the disclosure. The illustrative aspects depicted are intended only as exemplary.

When an element or feature is referred to herein as being "on," "engaged to," "connected to," or "coupled to" another element or feature, it may be directly on, engaged, connected, or coupled to the other element or feature, or intervening elements or features may be present. In contrast, when an element or feature is referred to as being "directly on," "directly engaged to," "directly connected to," or "directly coupled to" another element or feature, there may be no intervening elements or features present. Other words used to describe the relationship between elements or features should be interpreted in a like fashion (for example, "between" versus "directly between," "adjacent" versus "directly adjacent," etc.).

Although the terms "first," "second," etc. may be used herein to describe various elements, components, regions, layers, sections, and/or parameters, these elements, components, regions, layers, sections, and/or parameters should not be limited by these terms. These terms are only used to distinguish one element, component, region, layer, or section from another region, layer, or section. Thus, a first element, component, region, layer, or section discussed herein could be termed a second element, component, region, layer, or section without departing from the teachings of the present disclosure.

The examples and conditional language recited herein are principally intended to aid the reader in understanding the principles of the present technology and not to limit its scope to such specifically recited examples and conditions. It will be appreciated that those skilled in the art may devise various arrangements which, although not explicitly described or shown herein, nonetheless embody the principles of the present technology and are included within its spirit and scope.

A LiDAR system <NUM>, such as the one shown in <FIG>, may include a pulse emitter Tx and a receiver Rx. The pulse emitter Tx may include one or more light sources for transmitting light or photon beams. Examples of suitable light sources include lasers, laser diodes, light emitting diodes, organic light emitting diodes, or the like. For instance, the pulse emitter may include one or more visible and/or non-visible laser sources. In at least some embodiment, light source includes one or more non-visible laser sources, such as a near-infrared (NIR), short-wave infrared (SWIR), mid-wave infrared (MWIR), long-wave infrared (LWIR) laser, or the like. A light source may provide continuous or pulsed light beams of a predetermined frequency, or range of frequencies. The provided light beams may be coherent light beams. In an example, the pulse emitters Tx may be an array synchronized near field infrared pulse emitters.

The receivers Rx may include one or more photon-sensitive, or photon-detecting, arrays of sensor pixels. An array of sensor pixels may be configured to detect continuous or pulsed light beams on a constant or sporadic basis. The array of pixels may be a one dimensional-array or a two-dimensional array. The pixels may include single photon avalanche diodes (SPADs) or other photo-sensitive elements that avalanche upon the illumination of one or a few incoming photons. The pixels may have ultra-fast response times in detecting a single or a few photons that are on the order of a few nanoseconds. The pixels may be sensitive to the frequencies emitted or transmitted by pulse emitter Tx and relatively insensitive to other frequencies. In an example, the receivers Rx may be avalanche photodiodes (APDs).

During operation of the LiDAR system <NUM>, the pulse emitters Tx may emit a LiDAR beam with power P<NUM>. The LiDAR beam is reflected by a solid scene object, often referred to as target, that has a reflectivity ρ<NUM>. The LiDAR beam may be captured by the receiver Rx. Attributes of the recovered LiDAR beam may be detected by the receiver Rx, such as the time delay τ of the captured echo and its corresponding power PR. The object distance R of the target may be calculated by applying R = cτ , where c is the speed of light. A 3D position [x, y, z] of the target relative to the LiDAR system <NUM> may be further determined from this information, such as by using the direction in which the LiDAR beam with power P<NUM> was emitted in coordination with the object distance R.

By collating a dataset of bounced LiDAR beams, wherein each bounced LiDAR beam represents a target data point, a LiDAR point cloud may be produced that is embodies a virtual representation of the surroundings of the LiDAR system <NUM>. An increase in the number of points within the LiDAR point cloud may represent an increase of the fidelity between the representation and the "real-world" surroundings of the LiDAR system <NUM>. This fidelity is also affected by the medium that the LiDAR beams pass through. For instance, the presence of scattering particles, such as snow, rain, fog, etc., may decrease the fidelity of the LiDAR point cloud. Therefore, the LiDAR system <NUM> may optimally produce a LiDAR point cloud dataset for a clear weather environment.

A process for producing a linear model for pulse propagation in the presence of scattering particles, such as the process shown in <FIG>, may be described.

In step S1, a LiDAR point cloud dataset is obtained. The LiDAR point cloud dataset represents a LiDAR dataset containing LiDAR point cloud data for a clear weather environment that has been collected by a LiDAR system <NUM> in a real environment. For example, the LiDAR point cloud dataset may be a dataset collected by the LiDAR system <NUM> previously described for a clear weather environment in the real world.

In step S2, the snowfall effect on the LiDAR point cloud data is simulated. This includes step S3, in which a sample of snow particles in a given space is produced. The snow particles is sampled in 2D space for each LiDAR beam of the dataset. Snowfall may be simulated that is based on a sampling of snow particles. The sampled snow particles may be modeled as spheres, and more specifically as opaque spheres. The size of the spheres may be correlated to a snowfall rate, and therefore a snowfall rate may be selected, wherein said snowfall rate determines the size and amount of the snow particles. The model may further obey the exclusion principle such that no two snow particles may intersect with each other. Thus, the sample of snow particles may be produced.

In step S4, an attribute for each LiDAR beam that is impacted by a snow particle may be modified, such as by using induced geometry.

In step S5, a partially synthetic snowy LiDAR data is generated, such as by augmenting the LiDAR point cloud data based on the simulated effect of snowfall. This may also include collating the LiDAR beam data that was previously modified. The collated LiDAR beam data may constitute a "training dataset.

In step S6, the generated partially synthetic point clouds that include snowfall simulation effects may be used as training data for optimizing 3D object detection methods in order to train the 3D object detection methods to better discriminate between snow particles and target objects in a real environment.

Further aspects of this method and may be described herein.

As discussed previously, a LiDAR point cloud dataset may include attributes of the recovered LiDAR beam that may be detected by the receiver Rx, such as the time delay τ of the captured echo and its corresponding power PR. For extended objects, geometric optics can be applied to model the received power PR by using the formulation:
<MAT>
which holds for objects with a diameter larger than the beam diameter of the LiDAR beam with power P<NUM> at distance R and. Additional information may be needed, namely (i) the incident angle αin and (ii) the system constant CA independent of range and time. Because the system constant CA may differ for each scanning layer due to different optics and beam divergences, a correction coefficient is typically applied to the received power PR that corresponds with the specific hardware that is used due to differences in optics and beam divergence.

Thus, pulse propagation in "free space", i.e., lacking scattering particles, may be modeled based on the discussed principles of geometrical optics in order to model the transmission of the LiDAR beams with power P<NUM> and the associated received power PR at the LiDAR system <NUM>.

In order to prepare the LiDAR data for processing, the intensity calibration of the sensor may be inverted to obtain the raw intensity data. In snowfall, the optical medium contains particles which are smaller than the beam diameter, so Mie scattering and the exact spatial distribution of the particles are taken into account.

For each LiDAR beam, the set of snow particles that intersect with it may be computed, such as to derive the angle of the LiDAR beam cross-section that is reflected by each snow particle while taking potential occlusions into account.

Pulse propagation in the presence of scattering particles may be determined, such as by determining the modified impulse response of the linear system in the presence of snowfall, which allows the analytical calculation of the received power PR at the sensor. The range-dependent received power PR may be expressed as a time-wise convolution between the time-dependent transmitted signal power PT and the impulse response H of the optical system:
<MAT>
with the time signature of the transmitted pulse given by:
<MAT>
wherein τH is the half-power pulse width. The impulse response H can be factored into the impulse responses of the optical channel, HC , and the target, HT :
<MAT>
HC depends on the beam divergence, the overlap of transmitter and receiver described by ξ(R) as well as the transmittance T(R) of the medium through:
<MAT>.

The transmittance T(R) is equal to <NUM> in the part of the medium that is not occupied by snow particles. The absence of further "scattering elements" may be assumed. The overlap ξ(R) may be geometrically derived from the physical attributes of the LiDAR system, such as that shown in <FIG>, as
<MAT>.

The impulse response of the target, HT, enables the modelling of the model snow particles.

"Scene reflection" defines a particle interaction with the laser pulse by means of the impulse response HT. For an extended solid target object, the following equation may be used:
<MAT>
with ρ<NUM> being the reflectivity of the object and δ the Dirac delta function. In snowfall, the laser beam is partially reflected by snow particles in addition to the solid target object.

<FIG> shows a representation of the snow particles j relative to a LiDAR sensor <NUM>. A snow particle j may be modeled as a spherical object with a reflectivity ρs, a diameter Dj that is based upon a given distribution such that the snow particle j has an azimuth angle and distance Rj relative the sensor. Each of the snow particles j may be placed, such as uniformly and/or at random, around the sensor. Each of the snow particles j may be placed so that no snow particle j "intersects," or shares physical space, with any other snow particle j or the sensor <NUM> itself. The number of snow particles that are placed may be determined by a desired snowfall rate, which may be generated, detected, or otherwise determined. For instance, a typical snowfall rate may include a range of <NUM>-<NUM>/h, although a higher rate is considered.

A representation of a single LiDAR beam <NUM> can be seen. The individual snow particles j may partially or fully block the path of the LiDAR beam j between the sensor <NUM> and the target object. Additionally, the individual snow particles j may partially or fully block the LiDAR beam with respect to each other. Thus, each particle j reflects only a fraction θj/Θ of the opening angle Θ of the beam, also letting a fraction θ<NUM>/Θ of the beam reach the target.

Assuming Dj « cτH for all particles j, the following equation may be used:
<MAT>
with <MAT>.

When the results of Equation <NUM>, Equation <NUM>, Equation <NUM>, and Equation <NUM> are entered into Equation <NUM>, the received power <MAT> in snowfall may be determined to be
<MAT> <MAT><MAT> can be derived by substituting (θj , Rj , ρs) with (θ<NUM>, R<NUM>, ρ<NUM>) in the right-hand side of Equation <NUM>.

The received power PR is thus a superposition of multiple echoes <NUM>, wherein each echo <NUM> is each associated with an object, such as a snow particle j or target object, as depicted in <FIG>. The magnitude of each echo <NUM> may depend on the azimuth angle θj and the inverse square of the distance Rj of the respective object from the sensor <NUM>. The maximum peak of the received power may be taken to be the LiDAR beam <NUM> return. Thus, if a peak that is present due to a snow particle j is higher than a peak associated with the target object, the true echo <NUM> is missed and a cluttered point is added to the simulated point cloud at the range of peak of the snow particle j. Otherwise, the target object intensity is attenuated according to its occlusion percentage.

<FIG> demonstrates a schematic illustration of example code <NUM> for simulating snowfall for a LiDAR system, as described previously. The code is a "pseudo code" in that the logic of the code may be implemented within any number of programming languages.

<FIG> demonstrates a schematic illustration of example code <NUM> for snowflake sampling, such as to simulate the effect of snowfall on the LiDAR point cloud data by sampling snow particles in 2D space for each scanning layer of the LiDAR sensor. The code is a "pseudo code" in that the logic of the code may be implemented within any number of programming languages.

<FIG> demonstrates a schematic illustration of example code <NUM> for computing LiDAR beam occlusions, such as by using induced geometry in order to compute for each LiDAR beam the set of particles that intersect with it and derive the angle of the beam cross-section that is reflected by each particle, taking occlusions into account in order to produce an impulse response of the linear system in the presence of snowfall that allows an analytical calculation of the received power at the receiver. The code is a "pseudo code" in that the logic of the code may be implemented within any number of programming languages.

Using the method previously described, a partially synthetic snowy LiDAR data is generated.

<FIG> demonstrates the output of the process described. In the drawing labeled "Clear Input," a 3D representation of the input, which may be clear weather LiDAR data from a real world environment, is displayed. In the drawing labeled "Augmented Output," a 3D representation of the output, which may be the clear weather LiDAR data that has been augmented to add the simulated snowfall as previously described, is displayed.

<FIG> shows a real world heavy snowfall scene. This may be a scene as viewed from a LiDAR system <NUM>. <FIG> demonstrates the results of a conventional object detection model. As can be seen by comparing <FIG>, the conventional object detection model is able to accurately detect objects from image, such as vehicle <NUM>. However, it appears that due to interference from the snowfall, the object detection model has erroneously detected "ghost" objects <NUM> where this is clearly no object present. <FIG> represents the results of an object detection model that has been trained using as training dataset the partially synthetic snowy LiDAR data, such as that produced by the disclosed method. As can be seen, the object detection model in <FIG> is able to detect the "real" objects from <FIG> without erroneously detecting any ghost objects due to the effects of snowfall.

Aspects of the present disclosure may be implemented by or incorporate elements of a computer system. The computer system may comprise various hardware components including one or more single or multi-core processors collectively represented by processor, a solid-state drive, a memory, which may be a random-access memory or any other type of memory. Communication between the various components of the computer system may be enabled by one or more internal and/or external buses (e.g. a PCI bus, universal serial bus, IEEE <NUM> "Firewire" bus, SCSI bus, Serial-ATA bus, etc.), to which the various hardware components are electronically coupled. The drive may store program instructions suitable for being loaded into the memory and executed by the processor. It is noted that the computer system may have additional and/or optional components, such as a network communication module for communication, via a communication network with other electronic devices and/or servers, localization modules, and the like.

The functions of the various computer elements, including any functional block labeled as a "processor", may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. Moreover, explicit use of the term "processor" or "controller" should not be construed to refer exclusively to hardware capable of executing software, and may implicitly include, without limitation, digital signal processor (DSP) hardware, network processor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), read-only memory (ROM) for storing software, random access memory (RAM), and non-volatile storage.

<FIG> demonstrates a system <NUM> for generating a partially synthetic snowy LiDAR dataset. The system <NUM> may include a pulse emitter <NUM>, a receiver <NUM>, and a controller <NUM>. The pulse emitter <NUM> and receiver <NUM> may be similar to those previously discussed. The pulse emitter <NUM>, receiver <NUM>, and controller <NUM> may not necessarily be located proximately or temporally close. For instance, the controller <NUM> may receive data from the receiver <NUM> and/or pulse emitter <NUM> that was collected at a different location and/or a different time. Thus, the pulse emitter <NUM>, receiver <NUM>, and controller <NUM> do not need to be directly connected or concurrently operation.

The controller <NUM> may include a processing module <NUM> for obtaining the raw data from the receiver <NUM> and generating a LiDAR point cloud dataset. It is contemplated that the controller <NUM> may otherwise obtain the LiDAR point cloud dataset.

The controller <NUM> may include a simulation module <NUM> configured to simulate the effect of snowfall on the LiDAR point cloud data, such as by the method outlined previously. The simulation module <NUM> may include a snowfall module. The snowfall module <NUM> may be configured to simulate the effects of snowfall on each LiDAR beam of the LiDAR point cloud data. The controller <NUM> may be configured to output the augmented LiDAR point cloud data as partially synthetic snowy LiDAR data.

The controller <NUM> may include a training module <NUM> configured to use the partially synthetic snowy LiDAR data as a training dataset to train a 3D object detection model. Non-limiting examples of 3D object detection methods that may use the partially synthetic snowy LiDAR data as a training dataset are PV-RCNN, CenterPoint, Part-A<NUM>, PointRCNN, SECOND, PointPillars, and VoxelRCNN-Car.

Claim 1:
A computer-implemented method for simulating the effect of snowfall on LiDAR point clouds for a LiDAR system comprising:
- obtaining a LiDAR dataset containing LiDAR point cloud data for a clear weather environment that has been collected by a LiDAR system in a real environment (S1),
- simulating the effect of snowfall on the LiDAR point cloud data (S2), and
- augmenting the LiDAR point cloud data based on the simulated effect of snowfall to generate partially synthetic snowy LiDAR data (S5);
wherein simulating the effect of snowfall on the LiDAR point cloud data comprises:
- sampling snow particles in 2D space for each LiDAR beam of the dataset (S3), each snow particle reflecting only a fraction of an opening angle of the beam, also letting a fraction of the beam reach a target; and
- an analytical calculation of a received power received by a sensor of the LiDAR system, such received power being a superposition of multiple echoes, wherein each echo is associated with one of the snow particles or the target.