Neural network device and method using a neural network for sensor fusion

In accordance with an embodiment, a neural network is configured to: process a first grid representing at least a first portion of a field of view of a first sensor; process a second grid representing at least a second portion of a field of view of a second sensor; and fuse the processed first grid with the processed second grid into a fused grid, where the fused grid includes information about the occupancy of the first portion of the field of view of the first sensor and the occupancy of the second portion of the field of view of the second sensor.

This application claims the benefit of European Patent Application No. 19177666, filed on May 31, 2019, which application is hereby incorporated herein by reference in its entirety.

TECHNICAL FIELD

Various embodiments generally relate to a neural network device and a method.

BACKGROUND

Various systems, such as advanced driver assistance systems (ADAS), may include a variety of sensors of different sensor type. Each sensor type may have advantages and shortcomings. In order to overcome the shortcomings of each sensor type the data provided by the respective sensors of different sensor type may be combined, e.g. fused.

An ADAS may include sensors of different types for environment perception, such as LIDAR (light detection and ranging) sensors, radar sensors, monocular camera sensors, and stereo camera sensors. Cost efficient sensors like radar sensors and camera sensors usually provide sparse environmental information, and therefore environmental information should be gathered over time in order to obtain a meaningful environmental model. For a free space estimation dense environmental information are necessary, however, sensors providing dense environmental information, such as LIDAR sensors, which have a high cost and are thus not suitable for mass market. It is to be noted that some cameras may be configured to provide images. In case so called smart sensors are used, usually there are no raw images, since smart sensors are processing the detected data within the sensor itself.

In various embodiments a neural network device and a method using a neural network for sensor fusion are provided, which are capable of generating dense environmental information out of sparse input sensor data.

SUMMARY

According to an embodiment, a neural network device includes a neural network. The neural network is configured to process a first grid including a plurality of grid cells. The first grid represents at least a first portion of a field of view of a first sensor. At least one grid cell has information about an occupancy of the first portion of the field of view assigned to the at least one grid cell, the information being based on data provided by the first sensor. The neural network is further configured to process a second grid including a plurality of grid cells. The second grid represents at least a second portion of a field of view of a second sensor. At least one grid cell has information about an occupancy of the second portion of the field of view assigned to the at least one grid cell, the information being based on data provided by the second sensor. The neural network is further configured to fuse the processed first grid with the processed second grid into a fused grid. The fused grid includes information about the occupancy of the first portion of the field of view of the first sensor and the occupancy of the second portion of the field of view of the second sensor.

A system may include the neural network device. The first sensor is configured to provide data for the information of the first grid and the second sensor is configured to provide data for the information of the second grid.

A vehicle may include a driver assistance system. The driver assistance system includes the above system.

According to an embodiment, a method includes a neural network processing a first grid including a plurality of grid cells. The first grid represents at least a first portion of a field of view of a first sensor. At least one grid cell has information about an occupancy of the first portion of the field of view assigned to the at least one grid cell, the information being based on data provided by the first sensor. The neural network is further processing a second grid including a plurality of grid cells. The second grid represents at least a second portion of a field of view of a second sensor. At least one grid cell has information about an occupancy of the second portion of the field of view assigned to the at least one grid cell, the information being based on data provided by the second sensor. The neural network is further fusing the processed first grid with the processed second grid into a fused grid. The fused grid includes information about the occupancy of the first portion of the field of view of the first sensor and the occupancy of the second portion of the field of view of the second sensor.

According to an embodiment, a method of training a neural network includes: receiving first information about an occupancy of a first field of view from a first sensor, providing the first information to a first grid including a plurality of grid cells, receiving second information about an occupancy of a second field of view from a second sensor, providing the second information to a second grid including a plurality of grid cells, receiving ground truth data, fusing the first grid and the second grid into a fused grid, and training the neural network by comparing the ground truth data with a network output provided by the fused grid.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

In an embodiment, a “circuit” may be understood as any kind of a logic implementing entity, which may be hardware, software, firmware, or any combination thereof. Thus, in an embodiment, a “circuit” may be a hard-wired logic circuit or a programmable logic circuit such as a programmable processor, e.g. a microprocessor (e.g., a Complex Instruction Set Computer (CISC) processor or a Reduced Instruction Set Computer (RISC) processor). A “circuit” may also be software being implemented or executed by a processor, e.g. any kind of computer program, e.g. a computer program using a virtual machine code such as e.g. Java. Any other kind of implementation of the respective functions which will be described in more detail below may also be understood as a “circuit” in accordance with an alternative embodiment.

Various embodiments relate to a neural network device and a method using a neural network for sensor fusion, wherein at least two grids, which are provided by at least two sensors and which include feature-level data, are fused to a fused grid. The fusion of feature-level data has the effect that the fused grid includes sensor-specific information. An unsupervised training of the neural network using a dense grid provided by another sensor has the further effect that the trained neural network is capable of generating dense environmental information out of sparse input sensor data.

Thus, a neural network device and a method using a neural network for sensor fusion may be capable of generating a dense environmental model out of sparse input sensor data.

It is another aspect to provide a neural network device and a method using a neural network for sensor fusion, which are capable of classifying a large unobserved regions, in other words, which are capable of classifying large regions without explicit measurements.

It is another aspect to provide a neural network device and a method using a neural network for sensor fusion, which are capable of providing a dense environmental model within one measurement cycle.

It is another aspect to provide a neural network device and a method using a neural network for sensor fusion, which does not require gathering information over time.

FIG.1Ashows a vehicle100including an environment perception system according to various embodiments. The vehicle100may be for example a combustion engine vehicle, an electric vehicle, a hybrid vehicle, a hybrid electric vehicle or a combination thereof. Furthermore, the vehicle may be a car, a truck, a ship, a drone, an aircraft, and the like. The vehicle100may include a first sensor104. The first sensor104may include at least one of a first camera sensor or a first radar sensor. The first sensor104may have a first field of view108. The vehicle100may further include a second sensor106. The second sensor may include at least one of a second camera sensor or a second radar sensor. The second sensor106may have a second field of view no. According to an embodiment the first field of view108and the second field of view no may at least partially overlap forming a shared field of view112. In various embodiments, the first field of view108and the second field of view no do not overlap and thus do not form a shared field of view112. The vehicle100may further include additional sensors (in general an arbitrary number of sensors), wherein each sensor of the plurality of sensors may have a field of view. The plurality of sensors may include various sensors of the same type and/or various sensors of different type. The various sensors of the plurality of sensors may differ for example in the sensor type, in the detection principle, and/or in the detection specification (e.g., the sensors may detect different colors, e.g. a sensor may be specified to detect light of an indicator, e.g. a sensor may be specified to detect a backlight). The vehicle100may further include a processing circuit114. The processing circuit114may be configured to process the data provided by the plurality of sensors.

FIG.1Bshows a processing circuit114including a neural network according to various embodiments. The processing circuit114may be configured to process the first sensor data124provided by the first sensor104and the second sensor data126provided by the second sensor106. The first sensor104and/or the second sensor106may be smart sensors. The first sensor104and/or the second sensor106may be configured to provide digital sensor data. The first sensor104and/or the second sensor106may be configured to provide pre-processed sensor data. The pre-processed sensor data may include feature-level sensor data. The pre-processed sensor data may include target lists in case of radar sensors and object lists on case of camera sensors. In various embodiments, the first sensor104and/or the second sensor106may be configured to provide analog sensor data and the processing circuit114may include an analog-digital converter to convert the analog sensor data into digitized sensor data. In various embodiments, the first sensor104and/or the second sensor106may be configured to provide raw sensor data and the processing circuit114may be configured to pre-process the raw sensor data. The processing circuit114may include a memory device120. The memory device120may include a memory which is for example used in the processing carried out by a processor. A memory used in the embodiments may be a volatile memory, for example a DRAM (Dynamic Random Access Memory) or a non-volatile memory, for example a PROM (Programmable Read Only Memory), an EPROM (Erasable PROM), EEPROM (Electrically Erasable PROM), or a flash memory, e.g., a floating gate memory, a charge trapping memory, an MRAM (Magnetoresistive Random Access Memory) or a PCRAM (Phase Change Random Access Memory). The memory device120may be configured to store the first data124and/or the second data126. The processing circuit114may further include at least one processor122. The at least one processor122may be any kind of circuit, i.e. any kind of logic implementing entity, as described above. In various embodiments, the at least one processor122may be configured to process the data provided by the plurality of sensors.

FIG.1Cshows a processing system140according to various embodiments. The processing system140may include the memory device120. The memory device120may be configured to store the sensor data provided by the plurality of sensors. The memory device120may be configured to store the first sensor data124provided by the first sensor104. The memory device120may be further configured to store the second sensor data126provided by the second sensor106. The processing system140may further include the processor122. The processor122may be configured to receive the sensor data stored in the memory120and may be configured to process the sensor data.

The processor122may implement an inverse sensor model (ISM)130. The ISM130may include a plurality of ISM. The ISM130may be specified for each sensor of the plurality of sensors. In other words, each ISM of a plurality of ISM may be assigned to a sensor of the plurality of sensors. The ISM130may be obtained for each sensor of the plurality of sensors using a neural network (e.g., a convolutional neural network, an auto-encoder neural network, or a combination of both). Each ISM of the plurality of ISM may be configured to generate an occupancy grid (e.g., from the input data124and126).

The ISM130may be configured to process pre-processed first sensor data124and/or pre-processed second sensor data126. The pre-processed first sensor data124may be provided by the first sensor104. The pre-processed second sensor data126may be provided by the second sensor106. A first ISM may be applied to the first sensor data124. The first ISM may be configured to provide information about the occupancy of a first portion of a first field of view108of the first sensor104. A second ISM may be applied to the second sensor data126. The second ISM may be configured to provide information about the occupancy of a second portion of a second field of view no of the second sensor106. Applying the first ISM may provide a first grid132. The first grid132may include a plurality of grid cells. The first grid132may represent at least a first portion of the first field of view108of the first sensor104. At least one grid cell of the plurality of grid cells of the first grid may have information about an occupancy of the first portion of the first field of view108assigned to the at least one grid cell. The information may be based on data provided by the first sensor104. Applying the second ISM may provide a second grid134. The second grid134may include a plurality of grid cells. The second grid134may represent at least a second portion of the second field of view no of the second sensor106. At least one grid cell of the plurality of grid cells of the second grid134may have information about an occupancy of the second portion of the second field of view no assigned to the at least one grid cell. The information may be based on data provided by the second sensor106. The structure of the first grid132may be similar to the structure of the second grid134. The grid cells of the first grid132may have the same size and/or shape. The grid cells of the second grid134may have the same size and/or shape. The first grid132may be a first occupancy grid. The second grid134may be a second occupancy grid.

The processor122may implement at least a portion of a neural network136. The neural network136may be configured to process the first grid132provided by the first ISM and the second grid134provided by the second ISM to a fused grid138. The neural network136may be configured to determine a free space within the fused grid138based on the fused grid138and/or based on the first grid132and the second grid134. The fused grid138may include information about the occupancy of the first portion of the first field of view108of the first sensor104and the occupancy of the second portion of the second field of view110of the second sensor106. The first portion of the first field of view108and the second portion of the second field of view110may at least partially overlap. In various embodiments, the first portion of the first field of view108and the second portion of the second field of view110do not overlap. Various embodiments may provide a neural network device and a method using a neural network for sensor fusion and advanced free space estimation. In other words, various embodiments may provide a neural network device and a method using a neural network for sensor fusion with combined free space estimation.

The neural network136may be further configured to determine, for at least one grid cell of the fused grid138, a probability of the occupancy of the first portion of the first field of view108of the first sensor104and the second portion of the second field of view110of the second sensor106assigned to the at least one grid cell of the fused grid138.

The neural network136may be further configured to take into account the information about an occupancy of a portion of the field of view next to the first portion and/or next to the second portion when determining information about the occupancy assigned to a respective fused grid cell of the fused grid138.

The processor122may be further configured to process the sensor data provided by additional sensors. The ISM130may be configured to process the sensor data provided by the plurality of sensors and may be configured to provide a plurality of grids, wherein each grid of the plurality of grids is assigned to a respective sensor of the plurality of sensors. The neural network136may be configured to process the plurality of grids provided by the ISM130, in other words, generated by the inverse sensor models130. The neural network136may be configured to process the plurality of grids to a fused grid138. The fused grid138may include information about the occupancy of a portion of the field of view of each sensor of the plurality of sensors. The neural network136may be further configured to determine, for at least one grid cell of the fused grid138, a probability of the occupancy of a portion of a field of view of each sensor of the plurality of sensors assigned to the at least one grid cell of the fused grid138. The neural network136may be further configured to take into account the information about an occupancy of a portion of the field of view next to the portion of the field of view of each sensor of the plurality of sensors when determining information about the occupancy assigned to a respective fused grid cell of the fused grid138.

The neural network136may be trained by comparing the fused grid138with ground truth data. The training may include updating the neural network136based on the comparison of the fused grid138with the ground truth data. The training of the neural network136may be an unsupervised training. The training will be described in more detail below.

FIG.2shows a method200using a neural network according to various embodiments. The method200may be performed by a processor such as e.g., the processing circuit114. The method200may use the neural network136. The method200may include processing a first grid132in202. The method200may further include processing a second grid134in204. The method200may further include fusing the processed first grid132with the processed second grid134into a fused grid138in206.

The method200may further include processing additional grids. The method200may further include fusing the processed plurality of grids into a fused grid138.

The neural network136may be a fully convolutional neural network.FIG.3shows an architecture of a fully convolutional neural network300for sensor fusion according to various embodiments. The fully convolutional neural network300may include two input streams. In various embodiments, the fully convolutional neural network300includes more than two input streams. A first input stream of the fully convolutional neural network300may process a first grid132and a second input stream of the fully convolutional neural network300may process a second grid134. The first grid132and/or the second grid134may include more than one feature channel (for example two feature channels, for example three feature channels, for example more than three feature channels). The number of feature channels may be defined by the sensor data provided by the respective sensor of the plurality of sensors. Each feature channel of the plurality of feature channels may be assigned to an occupancy state of a plurality of occupancy states. The plurality of occupancy states may include a first state, a second state, and a third state, wherein the first state includes a free state, the second state includes an unknown state, and the third state includes an occupied state. According to an embodiment the first grid132and the second grid134include three features channels, wherein the three feature channels include three occupancy states, and wherein the first state includes a free state, the second state includes an unknown state, and the third state includes an occupied state. The dimensions of the first grid132and the second grid134may be 256×256 grid cells. In various embodiments, the first grid132and the second grid134may have different dimensions. The fully convolutional neural network300may include a plurality of network layers (for example two network layers, for example three network layers, for example more than three network layers). The fully convolutional neural network300may include a padding process, so that all layers of the plurality of network layers include the same dimensions. The plurality of network layers may include a plurality of convolutional layers (for example two convolutional layers, for example more than two convolutional layers) and at least one concatenating layer (for example exactly one concatenating layer, for example more than one concatenating layer). Each convolutional layer of the plurality of convolutional layers may be configured to process an input and may be configured to provide an intermediate layer output. Each convolutional layer of the plurality of convolutional layers includes a filter size and an activation function. According to an embodiment each convolutional layer includes a filter size of 3×3 grid cells. The activation function may include a ReLU (rectified linear unit) activation function. Each concatenating layer of the plurality of concatenating layers may be configured to process at least two convolution layer outputs (for example exactly two convolutional layer outputs, for example more than two convolutional layer outputs). Each concatenating layer of the plurality of concatenating layers may be configured to concatenate the at least two convolutional layer outputs and may be configured to provide a concatenating layer output. A concatenation can also be applied to the input occupancy grids (i.e., the first grid132and the second grid134) directly. There is generally no need to have convolutional layers upfront.

The fully convolutional neural network300may include a first neural network portion. The first neural network portion may be configured to process the first grid132. The first neural network portion may include a first convolutional layer306aof the first input stream. The first convolutional layer306amay provide a first convolutional layer output306. The fully convolutional neural network300may further include a second neural network portion. The second neural network portion may be configured to process the second grid134. The second neural network portion may include a second convolutional layer308aof the second input stream. The second convolutional layer308amay provide a second convolutional layer output308. The first convolutional layer306aand the second convolutional layer308amay include eight feature channels. In various embodiments, the first convolutional layer306aand the second convolutional layer308ainclude a different number of feature channels. According to an embodiment the first neural network portion includes additional convolutional layers, wherein the additional convolutional layers of the first neural network portion process the first convolutional layer output306of the first convolutional layer306a. The second neural network portion may include additional convolutional layers, wherein the additional convolutional layers of the second neural network portion process the second convolutional layer output308of the second convolutional layer308a.

The fully convolutional neural network300may include a fusion neural network portion. The fusion neural network portion may be configured to fuse the processed first grid132with the processed second grid134into a fused grid138. According to an embodiment, the fusion neural network portion of the fully convolutional network300includes one concatenating layer. The concatenating layer may concatenate the output of the convolutional layers of the input streams. The concatenating layer may include a first concatenating connection310a. The first concatenating connection310amay process the first convolutional layer output306. The concatenating layer may include a second concatenating connection310b. The second concatenating connection310bmay process the second convolutional layer output308.

The concatenating layer may concatenate the first convolutional layer output306of the first neural network portion and the second convolutional layer output308of the second neural network portion. The concatenating layer may be configured to provide a concatenating layer output310. The concatenating layer output may include sixteen feature channels. In various embodiments, the concatenating layer output310includes a different number of feature channels.

The fusion neural network portion may further include a plurality of convolutional layers processing the concatenating layer output310. According to an embodiment, the fusion neural network portion includes a third convolutional layer312a, a fourth convolutional layer314a, a fifth convolutional layer316a, and a sixth convolutional layer318a, wherein the sixth convolutional layer318aoutputs a fused grid138. The fusion neural network portion includes a different number of convolutional layer according to various embodiments. The neural network may be configured to fuse the first grid132with the second grid134into the fused grid138, wherein the fused grid138may include the same number of feature channels as the first grid132and/or the second grid134.

The neural network300may include a plurality of input streams (for example two input streams, for example three input streams, for example more than three input streams), wherein each input stream of the plurality of input streams processes a grid of a plurality of grids. Each input stream of the plurality of input streams may include a plurality of convolutional layers processing the respective grid of the plurality of grids. The concatenating layer may concatenate the output of each input stream of the plurality of input streams. The fusion neural network portion may include a plurality of convolutional layers, wherein the convolutional layers process the concatenating layer output310. The fusion neural network portion may output a fused grid138.

In various embodiments, the neural network136may include or may be an auto-encoder neural network. The auto-encoder neural network may include an encoding portion and a decoding portion. The encoding portion may include at least one encoding layer (for example exactly one encoding layer, for example two encoding layers, for example more than two encoding layers). The decoding portion may include at least one decoding layer (for example exactly one decoding layer, for example two decoding layers, for example more than two decoding layers). The auto-encoder neural network may include a plurality of encoding layers and a plurality of decoding layers, wherein the number of encoding layers or decoding layers defines the depth of the neural network. According to an embodiment the encoding portion and the decoding portion of an auto-encoder neural network may each include 8 layers, i.e. 8 encoding layers and 8 decoding layers. Each encoding layer of the plurality of encoding layers may include an encoding block and an encoding layer output. Each encoding block of the plurality of encoding blocks may be configured to provide the encoding layer output. Each decoding layer of the plurality of decoding layers may include a decoding block and a decoding layer output. Each decoding block of the plurality of decoding blocks may be configured to provide the decoding layer output.

FIG.4shows an architecture of an encoding block400of an auto-encoder neural network according to various embodiments. The encoding block400may include a convolutional layer402. The convolutional layer402may have a filter size of 3×3 grid cells. The encoding block400may further include an activation function404. The activation function404may be a ReLU activation function. The encoding block400may further include a pooling layer406. The pooling layer406may include a Max-pooling layer. The pooling layer406may include 2×2 grid cells (in other words, the pooling layer406may include a stride of 2). The pooling layer406may include a stride of 2. In various embodiments, the pooling layer406includes a different number of grid cells and/or a different number of strides. The encoding block400may further include a batch normalization layer408.

After an encoding block with parameters as described above (stride of 2), the input dimensions are halved. The number of feature channels is an arbitrary choice, defined by the number of filters of the convolutional layer.

It is to be noted that the order of the layers may vary within the neural network and it is not limited to the specific order as illustrated in the examples ofFIG.4andFIG.5.

FIG.5shows an architecture of a decoding block500of an auto-encoder neural network according to various embodiments. The decoding block500may include a transposed convolutional layer502. The transposed convolutional layer502may have a filter size of 3×3 grid cells (and strides of 2 (similar to pooling layers in encoding blocks)). The decoding block500may further include a convolutional layer504. The convolutional layer504may have a filter size of 3×3 grid cells. The decoding block500may further include an activation function506. The activation function506may be a ReLU activation function. The decoding block500may further include a batch normalization layer508.

The complete decoder (all decoding blocks together) up-samples to the dimensions of the input grids (i.e., the first grid132and the second grid134).

A single decoding block doubles grid dimensions (strides are equal to 2). The number of feature channels is an arbitrary choice. In an embodiment, the number of feature channels is doubled after each decoding block.

FIG.6shows an architecture of an auto-encoder neural network600according to various embodiments. The auto-encoder neural network600may include a first encoder602. The first encoder602may be configured to process a first grid132. In various embodiments, as shown inFIG.6, the first encoder602is configured to process a first processed grid, wherein the first processed grid may be a first convolutional layer output306, and wherein the first convolutional layer306amay process the first grid132. The auto-encoder neural network600may further include a second encoder604. The second encoder604may be configured to process a second grid134. In various embodiments, as shown inFIG.6, the second encoder604is configured to process a second processed grid, wherein the second processed grid may be a second convolutional layer output308, and wherein the second convolutional layer308amay process the second grid134.

Each of the first encoder602and the second encoder604may include at least one encoding layer (for example exactly 1 encoding layer, for example 2 encoding layers, for example more than 2 encoding layers), wherein each encoding layer may include an encoding block400and an encoding layer output. The first encoder602may include a first encoding block610aand a second encoding block612a. The first encoding block610amay be configured to process the first convolutional layer output306and may be further configured to provide a first encoding layer output610. The first encoding layer output610may have dimensions of 128×128 grid cells and may have sixteen feature channels. The second encoding block612amay be configured to process the first encoding layer output610and may be further configured to provide a second encoding layer output612. The second encoding layer output612may have dimensions of 64×64 grid cells and may have thirty-two feature channels. The second encoder604may include a third encoding block614aand a fourth encoding block616a. The third encoding block614amay be configured to process the second convolutional layer output308and may be further configured to provide a third encoding layer output614. The third encoding layer output614may have dimensions of 128×128 grid cells and may have sixteen feature channels. The fourth encoding block616amay be configured to process the third encoding layer output614and may be further configured to provide a fourth encoding layer output616. The fourth encoding layer output616may have dimensions of 64×64 grid cells and may have thirty-two feature channels.

The auto-encoder neural network600may further include at least one concatenating layer (for example exactly one concatenating layer, for example two concatenating layers, for example more than two concatenating layers). Each concatenating layer of the plurality of concatenating layers may be configured to process at least two encoding layer outputs (for example exactly two encoding layer outputs, for example more than two encoding layer outputs). Each concatenating layer of the plurality of concatenating layer may process each encoding layer of the plurality of encoding layers via a respective concatenating connection of a plurality of concatenating connections. Each concatenating layer of the plurality of concatenating layers may be configured to concatenate the at least two encoding layer outputs via at least two concatenating connections and may be configured to provide a concatenating layer output. According to an embodiment, the auto-encoder neural network600includes one concatenating layer, wherein the one concatenating layer may include a first concatenating connection618aand a second concatenating connection618b. The concatenating layer may be configured to provide a concatenating layer output618. The concatenating layer output618may have dimensions of 64×64 grid cells and may have sixty-four feature channels.

The auto-encoder neural network600may further include a decoder606. The decoder606may include at least one decoding layer (for example exactly one decoding layer, for example two decoding layers, for example more than two decoding layers), wherein each decoding layer may include a decoding block500and a decoding layer output. According to an embodiment, the decoder606includes a first decoding block620a, a second decoding block622aand a further convolutional layer624a, wherein the further convolutional layer624aoutputs the fused grid138. The first decoding block620amay be configured to process the concatenating layer output618and may be further configured to provide a first decoding layer output62o. The first decoding layer output620may have dimensions of 128×128 grid cells and may have thirty-two feature channels. The second decoding block622amay be configured to process the first decoding layer output620and may be further configured to provide a second decoding layer output622. The second decoding layer output622may have dimensions of 256×256 grid cells and may have sixteen feature channels. The further convolutional layer624amay be configured to process the second decoding layer output622and may be further configured to provide a fused grid138. The fused grid138may have dimensions of 256×256 grid cells and may have three feature channels.

According to an embodiment the auto-encoder neural network600includes additional encoders, wherein each encoder of the plurality of encoders processes a grid of a plurality of grids and wherein the at least one concatenating layer618may concatenate the plurality of processed grids.

FIG.7shows an architecture of an auto-encoder neural network700according to various embodiments. The auto-encoder neural network700may include a first encoder702. The first encoder702may include a plurality of encoding layers according to the first encoder602. The first encoder702may include the first encoding block610a, the first encoding layer output610, the second encoding block612a, and the second encoding layer output612. The first encoding block610amay be configured to process the first convolutional layer output306. The auto-encoder neural network700may further include a second encoder704. The second encoder704may include a plurality of encoding layers according to the second encoder604. The second encoder704may include the third encoding block614a, the third encoding layer output614, the fourth encoding block616a, and the fourth encoding layer output616. The third encoding block614amay be configured to process the second convolutional layer output308. The auto-encoder neural network700may further include a concatenating layer. The concatenating layer may concatenate the output of the first encoder702and the second encoder704. The concatenating layer may include a first concatenating connection710a. The first concatenating connection710amay be configured to process the second encoding layer output612. The concatenating layer may include a second concatenating connection710b. The second concatenating connection710bmay be configured to process the fourth encoding layer output616. The concatenating layer may be configured to provide a concatenating layer output710. The concatenating layer is one example of a fusion neural network portion.

The auto-encoder neural network700may further include a decoder706. The decoder706may be configured to provide the fused grid138based on a processed first grid132and a processed second grid134. The decoder706may include at least one decoding layer (for example exactly one decoding layer, for example two decoding layers, for example more than two decoding layers). Each decoding layer may include a decoding block500and a decoding block output. Each decoding layer may further include a skip concatenating layer, wherein the skip concatenating layer includes at least one skip connection (for example exactly one skip connection, for example two skip connections, for example more than two skip connections). Each skip connection may bypass code from an encoder to a decoder. Each skip connection may skip at least one encoding layer (for example skip exactly one encoding layer, for example skip two encoding layers, for example skip more than two encoding layers). Each skip connection may bypass code from the first encoder702and/or the second encoder704to the decoder706. The skip concatenating layer may concatenate a code of at least a part of an encoding layer output of the first encoder702and a code of at least a part of an encoding layer output of the second encoder704. In other words, at least a part of the code of an encoding layer output of the first encoder702and at least a part of the code of an encoding layer output of the second encoder704are bypassed to a decoding layer of the decoder706via skip connections.

The respective skip concatenating layer may concatenate at least a part of the code of the encoding layer output of the first encoder702and at least a part of the code of the encoding layer output of the second encoder704, which have the same dimensions and the same number of feature channels as the decoding block output of the decoding layer assigned to the respective encoding layer output or encoding layer outputs. Each skip concatenating layer may be configured to provide a skip concatenating layer output.

According to an embodiment, the decoder706includes a first decoding layer. The first decoding layer may include a first decoding block712aand a first decoding block output712d. The first decoding block712amay be configured to process the concatenating layer output710and may be further configured to provide the first decoding block output712d. The first decoding block output712dmay have dimensions of 128×128 grid cells and may have sixteen feature channels. The first decoding layer may further include a first skip concatenating layer. The first skip concatenating layer may concatenate at least a part of the code of the encoding block of the first encoder702and at least a part of the code of the encoding block of the second encoder704, which have the same dimensions and the same number of feature channels as the first decoding block. The first skip concatenating layer may include a first skip concatenating connection712band a second skip concatenating connection712c. The first skip concatenating connection712bmay be configured to process the first encoding layer output61o. The second skip concatenating connection712cmay be configured to process the third encoding layer output614. The first skip concatenating layer may be configured to provide a first skip concatenating layer output712e. The first decoding layer may include a first decoding layer output712, wherein the first decoding layer output712may include the first decoding block output712dand the first skip concatenating layer output712e.

The decoder706may further include a second decoding layer. The second decoding layer may include a second decoding block714aand a second decoding block output714d. The second decoding block714amay be configured to process the first decoding layer output712. In other words, the second decoding block714amay process the code of the first decoding block output712dand the code of the first skip concatenating layer output712e. The second decoding block output714dmay have dimensions of 256×256 grid cells and may have eight feature channels. The second decoding layer may further include a second skip concatenating layer. The second skip concatenating layer may concatenate at least a part of the code of the encoding block of the first encoder702and at least part of the code of the encoding block of the second encoder704, which have the same dimensions and the same number of feature channels as the second decoding block. The second skip concatenating layer may include a first skip concatenating connection714band a second skip concatenating connection714c. The first skip concatenating connection714bmay be configured to process the first convolutional layer output306. The second skip concatenating connection714cmay be configured to process the second convolutional layer output308. The second skip concatenating layer may be configured to provide a second skip concatenating layer output714e. The second decoding layer may include a second decoding layer output714, wherein the second decoding layer output714may include the second decoding block output714dand the second skip concatenating layer output714e.

The decoder706may further include a third decoding layer. The third decoding layer may include a yet further convolutional layer716a. The yet further convolutional layer (which may also be referred to as a decoder convolutional layer)716amay be configured to process the second decoding layer output714. In other words, the yet further convolutional layer716amay process the code of the second decoding block output714dand the code of the second skip concatenating layer output714e. The third decoding layer may be configured to provide a yet further convolutional layer output716. The yet further convolutional layer output716may have dimensions of 256×256 grid cells and may have eight feature channels. It should be noted that in various embodiments, a plurality of decoder convolutional layers may be provided in the neural network in order to reduce the number of feature channels to a desired number, e.g. three.

The decoder706may further include a further decoder convolutional layer718a. The further decoder convolutional layer718amay be configured to process the yet further convolutional layer output716. The further decoder convolutional layer718amay be configured to provide a fused grid138. The fused grid138may include the same dimensions and/or the same number of feature channels as the first grid132and/or the second grid134. The fused grid138may have dimensions of 256×256 grid cells and may have three feature channels.

Skipping at least a part of the code from the first encoder702and/or the second encoder704to the decoder706via skip connections has the effect that small features are preserved. In other words, skip connections generate a more detailed fused grid138.

According to an embodiment the auto-encoder neural network700includes additional encoders, wherein each encoder of the plurality of encoders processes a grid of a plurality of grids and wherein the at least one concatenating layer may be configured to concatenate the plurality of processed grids. The auto-encoder neural network700may further include a plurality of decoding layers, wherein each decoding layer of the plurality of decoding layers may include a decoding block and may further include a skip concatenating layer. The skip concatenating layer may concatenate at least a part of a code of a plurality of encoding blocks. In other words, at least a part of the code of a plurality of encoding block outputs is bypassed to a decoding layer of the decoder706via skip connections.

FIG.8shows a method800of training a neural network according to various embodiments. The training method800may include receiving first information about an occupancy of a first field of view from a first sensor in802. The training method800may further include providing the first information to a first grid in804. The first grid may include a plurality of grid cells. The training method800may further include receiving second information about an occupancy of a second field of view from a second sensor in806. The training method800may further include providing the second information to a second grid in808. The second grid may include a plurality of grid cells. The training method800may further include fusing the first grid and the second grid into a fused grid in81o. The training method800may further include receiving ground truth data in812. The training method800may include training the neural network by comparing the ground truth data with a network output provided by the fused grid in814. The ground truth data may include a ground truth grid. Training the neural network may include updating the neural network. The neural network may be trained using an ADAM optimizer.

According to an embodiment the training method800includes receiving information about an occupancy of a field of view from additional sensors. The training method800may further include providing the information of each sensor of the plurality of sensor to a respective grid of a plurality of grids. The training method800may further include fusing the plurality of grids into a fused grid. The training method Boo may further include receiving ground truth data and training the neural network by comparing the ground truth data with a network output provided by the fused grid.

FIG.9shows examples of a first grid902, a second grid904, a fused grid906, and a ground truth grid908, wherein a neural network was trained using the first grid902, the second grid904and the ground truth grid908and wherein the trained neural network fused the first grid902and the second grid904to the fused grid906.

As shown inFIG.9,white colored elements in a respective grid indicate an occupancy probability of “0” of the respective grid cell (illustratively: the respective grid cell is considered to be free);black colored elements in a respective grid indicate an occupancy probability of “1” of the respective grid cell (illustratively: the respective grid cell is considered to be occupied);grey colored elements in a respective grid indicate an occupancy probability of “0.5” of the respective grid cell (illustratively: the respective grid cell is considered to have an unknown occupancy state).

It is to be noted that in these embodiments, the occupancy probability values are rounded to “0”, “0.5” and “1”, respectively. However, in various embodiments, the occupancy probability values may have any value in the range from “o” to “1”.

In this way, the neural network may be configured to determine a free space within the fused grid. The free space may be a classification result for a grid cell based on the respectively assigned and determined occupancy probability value for the grid cell of the fused grid138.

FIG.10Ashows a system1000A for training a neural network according to various embodiments. The training system1000A may include the processing system140. The processing system140may include an ISM130and a neural network136, wherein the ISM130and the neural network136may be implemented by a processor122. The ISM130may be configured to process the first sensor data124and may be further configured to provide a first grid132. The ISM130may be configured to process the second sensor data126and may be further configured to provide a second grid134. The neural network136may be configured to fuse the first grid132and the second grid134to a fused grid138. The training system1000A may further include a third sensor1002. The third sensor1002may be configured to provide third sensor data. The third sensor data may include a third grid1004. The memory device120may be configured to receive the third sensor data and/or the third grid1004from the third sensor1002and may be further configured to provide the third sensor data and/or the third grid1004to the processor122. The third grid1004may include a plurality of grid cells. The third grid1004may represent at least a third portion of a third field of view of the third sensor1002. At least one grid cell of the plurality of grid cells of the third grid1004may have information about an occupancy of the third portion of the third field of view assigned to the at least one grid cell. The system1000A may further include a ground truth grid1006. The ground truth grid1006may include ground truth data. The processor122may be configured to process the third grid1004and may be configured to provide the ground truth grid1006. The ground truth grid1006may be based on the third grid1004. In various embodiments, the third grid1004is the ground truth grid1006. The neural network136may be trained by comparing the ground truth grid1006with the fused grid138. The neural network136may be updated based on the result of the comparison.

According to an embodiment, the training system1000A includes a plurality of sensors, wherein the processor122may process the data provided by the plurality of sensors and wherein the neural network136may output a fused grid based on the data provided by the plurality of sensors.

The first sensor104may include at least one of a radar sensor or a camera sensor. The second sensor106may include at least one of a radar sensor or a camera sensor. The third sensor1002may include a LIDAR sensor. The first sensor104and/or the second sensor106may provide pre-processed sensor data. The pre-processed sensor data may include target lists in case of radar sensors and object lists on case of camera sensors. The neural network may process 2-dimensional data. The camera sensor may be a 2D camera sensor. The camera sensor may be a 3D camera sensor, wherein the 3D data are projected in a 2D plane.

According to an embodiment, the third sensor1002includes a LIDAR sensor and the ground truth data include sensor data provided by the third sensor1002. The ground truth data may be obtained from a measured LIDAR point cloud, wherein a ground plane may be estimated by RANSAC-based plane fitting. The third grid1004may be obtained from the ground plane. The ground truth grid1006may be based on the third grid1004and additional grids of the plurality of grids obtained from the plurality of sensor data provided by plurality of sensors (for example based on all grids of the plurality of grids, for example based on some grids of the plurality of grids).

In various embodiments, the third grid1004may be obtained from the measured LIDAR point cloud. It may include all points belonging to the ground plane. They are assumed to be located on a drivable road. All those points may be converted into an occupancy grid with a LIDAR inverse sensor model.

The neural network136may be trained based on the ground truth grid1006. The training may be done using data from a dataset, wherein a plurality of individual frames is stored. The individual frames may include synchronized measurements from a LIDAR sensor, a camera sensor, a radar sensor, and position sensors (IMU and GPS). The camera data may be provided as bounding boxes (2D or 3D), the radar data may be provided as a target list, wherein individual targets consist of at least information about the spatial location of the target (Cartesian coordinates or polar coordinates), and the LIDAR data may be provided as a point cloud (2D or 3D), wherein individual points consist of at least information about spatial location of the reflection point. Radar data may further include information about velocities, uncertainties, and other information about the target. Camera object detections may include parameters such as position, size, orientation, and velocities.

A grid of a plurality of grids obtained from a plurality of sensor data including information about a field of view of a sensor of a plurality of sensors may be obtained for each frame of the plurality of frames. A ground truth grid1006may be obtained for each frame of the plurality of frames. The number of frames may be increased by data augmentation. The data augmentation may include random rotations and/or random mirroring. A radar grid is generated by applying an ISM and thus converting the raw data detections to a spatial occupancy probability. A camera grid is obtained by projecting the 3D bounding box of each camera object detection to a 2D ground plane, wherein these footprints are assumed to be occupied and the remaining cells are defined as an unknown occupancy state.

The neural network136may be configured to process the plurality of grids and may provide a fused grid138based on the plurality of grids. The fused grid138may be compared to the ground truth grid1006. A loss function may include a pixel-wise, i.e. for each grid cell of the plurality of grid cells, softmax classification. The loss function may be applied for each grid of the plurality of grids. The estimated loss of each grid cell of the plurality of grid cells may be summed up to a total grid cell loss. The total grid cell loss of each grid cell of the plurality of grid cells may be summed up for all grid cells of the plurality of grid cells providing a total loss. Now, illustratively, the loss has been calculated. The network may now be changed, so that the loss function is getting smaller for each training iteration. This minimization process may be performed with an ADAM optimizer.

Training the neural network with ground truth data that are based on LIDAR sensor data has the effect that the trained neural network is capable of providing a dense environmental model and hence no LIDAR sensor is further needed once the network is trained. LIDAR sensors usually have a high cost. Thus, it is an aspect of this disclosure to provide a neural network device and a method using a neural network for sensor fusion with reduced cost. Providing a dense environmental model out of sparse sensor data has the further effect that object shapes are provided with higher accuracy and that a drivable free space is estimated.

FIG.10Bshows a system1000B for training a neural network according to various embodiments. The training system1000B may correspond substantially to the training system1000A. The training system1000B may differ from the training system1000A in that the ground truth grid1006is generated from ground truth data obtained from the third grid1004and ground truth data obtained from the first grid132and/or the second grid134. In other words, the processor is configured to process the third grid1004and the first grid132and/or the second grid134and the processor122may be further configured to provide a ground truth grid1006based on the third grid1004and the first grid132and/or the second grid134.

According to an embodiment the training system1000B includes additional sensors, wherein the processor122may be configured to process the data provided by the plurality of sensors and wherein the neural network136may output a fused grid based on the data provided by the plurality of sensors. The processor122may be configured to process the plurality of grids (for example from each grid of the plurality of grids, for example from some grids of the plurality of grids) and may be further configured to provide a ground truth grid1006based on the plurality of grids (for example based on each grid of the plurality of grids, for example based on some grids of the plurality of grids).

In the following, various aspects of this disclosure will be illustrated:

Example 1 is a neural network device. The neural network device includes a neural network configured to process a first grid comprising a plurality of grid cells. The first grid represents at least a first portion of a field of view of a first sensor. At least one grid cell has information about an occupancy of the first portion of the field of view assigned to the at least one grid cell, the information being based on data provided by the first sensor. The neural network is further configured to process a second grid comprising a plurality of grid cells. The second grid represents at least a second portion of a field of view of a second sensor. At least one grid cell has information about an occupancy of the second portion of the field of view assigned to the at least one grid cell, the information being based on data provided by the second sensor. The neural network is further configured to fuse the processed first grid with the processed second grid into a fused grid. The fused grid includes information about the occupancy of the first portion of the field of view of the first sensor and the occupancy of the second portion of the field of view of the second sensor.

In Example 2, the subject matter of Example 1 can optionally include that at least a portion of the neural network is implemented by one or more processors.

In Example 3, the subject matter of any one of Examples 1 or 2 can optionally include that the neural network is further configured to determine, for at least one grid cell of the fused grid, a probability of the occupancy of the first portion of the field of view of the first sensor and the second portion of the field of view of the second sensor assigned to the at least one grid cell of the fused grid.

In Example 4, the subject matter of any one of Examples 1 to 3 can optionally include that the neural network is further configured to take into account the information about an occupancy of a portion of the field of view next to the first portion and/or next to the second portion when determining information about the occupancy assigned to a respective fused grid cell of the fused grid.

In Example 5, the subject matter of any one of Examples 1 to 4 can optionally include that the structure of the first grid is similar to the structure of the second grid.

In Example 6, the subject matter of any one of Examples 1 to 5 can optionally include that the grid cells of the first grid have a same size and/or shape, and/or that the grid cells of the second grid have a same size and/or shape.

In Example 7, the subject matter of any one of Examples 1 to 6 can optionally include that the first grid forms a first occupancy grid, and/or that the second grid forms a second occupancy grid.

In Example 8, the subject matter of any one of Examples 1 to 7 can optionally include that the neural network includes a first neural network portion configured to process the first grid, a second neural network portion configured to process the second grid, and a fusion neural network portion configured to fuse the processed first grid with the processed second grid into the fused grid.

In Example 9, the subject matter of Example 8 can optionally include that the neural network includes a convolutional neural network.

In Example 10, the subject matter of any one of Examples 1 to 9 can optionally include that the first sensor includes at least one of a first camera sensor or a first radar sensor, and/or that the second sensor includes at least one of a second camera sensor or a second radar sensor.

In Example 11, the subject matter of any one of Examples 1 to 10 can optionally include that the neural network includes or essentially consists of an auto-encoder.

In Example 12, the subject matter of Example 11 can optionally include that the auto-encoder includes a first encoder configured to process the first grid, a second encoder configured to process the second grid, and a decoder configured to provide the fused grid based on the processed first grid and the processed second grid.

In Example 13, the subject matter of any one of Examples 1 to 12 can optionally include that the neural network includes one or more skip connections.

In Example 14, the subject matter of Example 13 can optionally include that the one or more skip connections bypass code from the first neural network portion and/or the second neural network portion to the fusion neural network portion.

In Example 15, the subject matter of any one of Examples 13 or 14 can optionally include that the one or more skip connections bypass code from the first encoder and/or the second encoder to the decoder.

In Example 16, the subject matter of Example 15 can optionally include that bypassing code from the first encoder and/or the second encoder to the decoder includes bypassing code from an encoding layer output or encoding layer outputs of the first encoder and/or the second encoder to a decoding layer or decoding layers of the decoder.

In Example 17, the subject matter of Example 16 can optionally include that bypassing code from an encoding layer output or encoding layer outputs of the first encoder and/or the second encoder to a decoding layer or decoding layers of the decoder includes bypassing code from an encoding layer output or encoding layer outputs having the same neural network depth as the respective decoding layer output.

In Example 18, the subject matter of any one of Examples 16 or 17 can optionally include that bypassing code from an encoding layer output or encoding layer outputs of the first encoder and/or the second encoder to a decoding layer or decoding layers of the decoder includes bypassing code from an encoding layer output or encoding layer outputs having the same dimensions and/or the same number of feature channels as the decoding block output of the respective decoding layer.

In Example 19, the subject matter of any one of Examples 1 to 18 can optionally include that a first inverse sensor model is applied to the data provided by the first sensor to provide the information about the occupancy of the first portion of the field of view of the first sensor, and that a second inverse sensor model is applied to the data provided by the second sensor to provide the information about the occupancy of the second portion of the field of view of the second sensor.

In Example 20, the subject matter of any one of Examples 1 to 19 can optionally include that the neural network is further configured to determine a free space within the fused grid based on the first grid and the second grid.

Example 21 is a system. The system includes a neural network device of any one of Examples 1 to 20. The first sensor may be configured to provide data for the information of the first grid. The second sensor may be configured to provide data for the information of the second grid.

Example 22 is a vehicle. The vehicle includes a driver assistance system including the system of Example 21.

Example 23 is a method. The method includes a neural network processing a first grid including a plurality of grid cells. The first grid represents at least a first portion of a field of view of a first sensor. At least one grid cell has information about an occupancy of the first portion of the field of view assigned to the at least one grid cell, the information being based on data provided by the first sensor. The neural network further processes a second grid including a plurality of grid cells. The second grid represents at least a second portion of a field of view of a second sensor. At least one grid cell has information about an occupancy of the second portion of the field of view assigned to the at least one grid cell, the information being based on data provided by the second sensor. The neural network further fuses the processed first grid with the processed second grid into a fused grid, wherein the fused grid includes information about the occupancy of the first portion of the field of view of the first sensor and the occupancy of the second portion of the field of view of the second sensor.

In Example 24, the subject matter of Example 23 can optionally include that at least a portion of the neural network is implemented by one or more processors.

In Example 25, the subject matter of Example 24 can optionally include that the neural network determines, for at least one grid cell of the fused grid, a probability of the occupancy of the first portion of the field of view of the first sensor and the second portion of the field of view of the second sensor assigned to the at least one grid cell of the fused grid.

In Example 26, the subject matter of any one of Examples 23 to 25 can optionally include that the neural network takes into account the information about an occupancy of a portion of the field of view next to the first portion and/or next to the second portion when determining information about the occupancy assigned to a respective fused grid cell of the fused grid.

In Example 27, the subject matter of any one of Examples 23 to 26 can optionally include that the structure of the first grid is similar to the structure of the second grid.

In Example 28, the subject matter of any one of Examples 23 to 27 can optionally include that the grid cells of the first grid have the same size and/or shape, and/or that the grid cells of the second grid have the same size and/or shape.

In Example 29, the subject matter of any one of Examples 23 to 28 can optionally include that the first grid forms a first occupancy grid, and/or that the second grid forms a second occupancy grid.

In Example 30, the subject matter of any one of Examples 23 to 29 can optionally include that the neural network includes a first neural network portion processing the first grid, a second neural network portion processing the second grid, and a fusion neural network portion fusing the processed first grid with the processed second grid into the fused grid.

In Example 31, the subject matter of Example 30 can optionally include that the neural network includes a convolutional neural network.

In Example 32, the subject matter of any one of Examples 23 to 31 can optionally include that the first sensor includes at least one of a first camera sensor or a first radar sensor, and/or that the second sensor includes at least one of a second camera sensor or a second radar sensor.

In Example 33, the subject matter of any one of Examples 23 to 32 can optionally include that the neural network includes an auto-encoder.

In Example 34, the subject matter of Example 33 can optionally include that the auto-encoder includes a first encoder processing the first grid, a second encoder processing the second grid, and a decoder processing the fused grid based on the processed first grid and the processed second grid.

In Example 35, the subject matter of any one of Examples 30 to 34 can optionally include that the neural network includes one or more skip connections.

In Example 36, the subject matter of Example 35 can optionally include that the one or more skip connections bypass code from the first neural network portion and/or the second neural network portion to the fusion neural network portion.

In Example 37, the subject matter of any one of Examples 35 or 36 can optionally include that the one or more skip connections bypass code from the first encoder and/or the second encoder to the decoder.

In Example 38, the subject matter of Example 37 can optionally include that bypassing code from the first encoder and/or the second encoder to the decoder includes bypassing code from an encoding layer output or encoding layer outputs of the first encoder and/or the second encoder to a decoding layer or decoding layers of the decoder.

In Example 39, the subject matter of Example 38 can optionally include that bypassing code from an encoding layer output or encoding layer outputs of the first encoder and/or the second encoder to a decoding layer or decoding layer output of the decoder includes bypassing code from an encoding layer output or encoding layer outputs having the same neural network depth as the respective decoding layers.

In Example 40, the subject matter of any one of Examples 38 or 39 can optionally include that bypassing code from an encoding layer output or encoding layer outputs of the first encoder and/or the second encoder to a decoding layer or decoding layers of the decoder includes bypassing code from an encoding layer output or encoding layer outputs having the same dimensions and/or the same number of feature channels as the decoding block output of the respective decoding layer.

In Example 41, the subject matter of any one of Examples 23 to 40 can optionally include that a first inverse sensor model is applied to the data provided by the first sensor to provide the information about the occupancy of the first portion of the field of view of the first sensor, and that a second inverse sensor model is applied to the data provided by the second sensor to provide the information about the occupancy of the second portion of the field of view of the second sensor.

In Example 42, the subject matter of any one of Examples 23 to 41 can optionally include that the neural network further determines a free space within the fused grid based on the first grid and the second grid.

Example 43 is a method of training a neural network. The method may include: receiving first information about an occupancy of a first field of view from a first sensor, providing the first information to a first grid including a plurality of grid cells, receiving second information about an occupancy of a second field of view from a second sensor, providing the second information to a second grid including a plurality of grid cells, receiving ground truth data, fusing the first grid and the second grid into a fused grid, and training the neural network by comparing the ground truth data with a network output provided by the fused grid.

In Example 44, the subject matter of Example 43 can optionally include that the ground truth data are received from a third sensor.

In Example 45, the subject matter of any one of Examples 43 or 44 can optionally include that the ground truth data are obtained from a third grid received from a third sensor and from the first grid received from the first sensor and/or the second grid received from the second sensor.

In Example 46, the subject matter of any one of Examples 44 or 45 can optionally include that the first sensor includes at least one of a radar sensor or a camera sensor, that the second sensor includes at least one of a radar sensor or a camera sensor, and that the third sensor includes a LIDAR sensor.