Patent Description:
In the field of object recognition, sensor data fusion for object recognition is a technique for recognizing an object on the basis of the sensor data of several different sensors. This provides the advantage that each sensor may sense different signals or characteristics of the object such that the recognition may be based on a broader basis of input information. For example, for detecting a person in an environment, a camera for capturing visible light and a sensor for sensing infrared radiation may both provide sensor data. Recognizing the person is more robust or reliable as both the visual appearance (camera data) as well as the heat emitted from the body of the person (infrared data) may be used to verify that the object actually is a person or human being.

Generally, two ways of fusing the sensor data are known. One solution is to perform an individual object recognition on the respective sensor data of each sensor individually. Then, these separate recognition results for the different sensors may be combined in order to verify the different recognition results against each other.

The alternative solution is to perform a combined object recognition on the basis of a mixture or combination of the sensor data from the different sensors. This second solution may be difficult to implement as prior to any object recognition it is difficult to decide which sensor data from one sensor should be combined with which other sensor data from another sensor. No intermediate recognition result is available that may indicate which data may belong together.

It is an objective of the present invention to provide a sensor data fusion for a data processing that is based on an artificial neural network, ANN, in particular for an ANN-based object recognition, in a vehicle.

The objective is accomplished by the subject matter of the independent claims. Advantageous developments with convenient and non-trivial further embodiments are specified in the following description, the dependent claims and the figures.

As one solution, a method is provided for operating a processing unit of a vehicle in order to process sensor data of several different sensors with an artificial neural network, ANN. A respective detection range of each sensor is directed towards an environment of the vehicle. Preferably, the detection ranges of the different sensors overlap fully or partly. The method is characterized in that a set of specific data structures, namely "volume data cells", is provided as a volumetric representation of different volume elements or voxels of the environment. In other words, a "volume data cell" is a data structure for storing data belonging to the same volume element of the environment. A volume element, as represented by a volume data cell, can be defined, e.g., as a cube or a cuboid with an edge length in the range from, e.g., <NUM> to <NUM>. The volume data cells may be provided in a memory of the processing unit. In a preferred embodiment, the volume data cells are provided by at least one layer of said artificial neural network itself.

The solution is based on the fact that for each combination of one of the (real) volume elements of the environment and one of the sensors a respective mapping of a world coordinate system of the environment to a respective sensor coordinate system of the respective sensor is given. The mapping describes the path of the detected signal or radiation (e.g. visible light or radar or infrared light or ultrasonic sound) from the respective volume element to the sensor or the sensor chip of the sensor.

When sensor data is generated by the sensor, this sensor data is transferred to the respective volume data cells using an inverse mapping function, which provides an inverse mapping to said mapping, such that each inverse mapping function is a mapping of the respective sensor coordinate system to an internal volumetric coordinate system that corresponds to the world coordinate system. The inverse mapping function identifies in which one of the volume data cells specific sensor data are to be stored. This identifies the volume data cell to which the sensor data belong, i.e. the volume data cell that represents the volume element from which the signal or radiation must have originated. For example, in a camera sensor with a sensor chip for detecting pixels, a specific region or set of pixels on that chip will hold the picture of a specific volume element depending on, e.g., the design of the optical lens of the camera. For example the top left corner of the sensor chip may hold all those pixels that original from the volume element that is on the bottom right as seen from the camera (as the optical lens may provide a flipped image). The mapping and consequently the inverse mapping function can be derived from geometric calculations and/or on the basis of a training of parts of the artificial neural network. Note that the inverse mapping function can also be ambiguous, as, e.g., from a <NUM>-dimensional sensor coordinate system (as it is given in a sensor chip of a camera) is cannot be decided from which volume element the light exactly came from as the depth information is lost. In this case, the corresponding sensor data may be stored in several volume data cells.

By the transfer of the sensor data from all sensors to the volume data cells, each volume data cell receives the sensor data of each sensor and thus the received sensor data from the different sensors are collected or accumulated (i.e. stored together or are made available together) in the respective volume data cell as accumulated or fused or combined data. Thus, each volume data cell holds the combined object information as captured by the different sensors.

For each volume data cell an individual processing of the combined data of that volume data cell is performed by the artificial neural network wherein the individual processing comprises deriving feature data of at least one feature of at least one object that is described by the combined data in that volume data cell such that a combined processing of sensor data of the different sensors for the same volumetric element is performed. Thus, for the processing of the sensor data, the artificial neural network is already provided with a mixture or accumulation of sensor data (i.e. the combined data) from different sensors. Preferable, the sensor data are kept separate in the combined data (e.g. no mean value of the sensor data is calculated to obtain the combined data).

This provides the benefit that for recognizing objects the different aspects or characteristics (for example sensor data describing visible light and infrared light) are already available in combination for the processing. The processing is an "individual processing" as it may be performed separately for each volume data cell.

The invention also comprises further developments with additional technical features that provide further benefits.

One further development comprises that at least a part of the processing of the sensor data from the volume data cells is "agnostic" in that it is performed independently from which sensor the sensor data or the combined data are from. In other words, the sensor data may indicate or describe a measurement value, for example an intensity and/or a position, for single data points, but to agnostic processing of sensor data it is unknown, which sensor performed the actual measurement, in particular which type of sensor (e.g. camera or radar) did the measurement. This provides the advantage that sensor data from different sensors can be processed in the same processing step as no differentiation or distinction is made between their sensor data. Agnostic processing can be advantageous, for example, for edge detection and/or a detection of a contour of an object.

One further development comprises that at least for a part of the processing of the sensor data respective tagging data is provided, wherein the tagging data informs or signals to the ANN from which sensor the respective sensor data in the combined data is coming from. In other words, for a single respective single data point or measurement it is denoted or signaled, which sensor performed the actual measurement that yielded the sensor data. The tagging data can be provided, for example, as a flag indicating which sensor the sensor data are from, or as a value of a variable indicating the sensor or its sensor type. Providing sensor data together with tagging data provides the advantage that the processing based on the ANN can be adapted to specific characteristics of the respective sensor. For example, a sensor-specific distortion and/or characteristic curve, for example a color mapping, can be compensated and/or can be taken into account for processing and/or interpreting the sensor data.

One further development comprises that for the processing of at least one volume data cell the ANN also receives at least a part of the combined data from at least one neighboring volume data cell. In other words, the processing is not only based on the combined data contained in the one volume data cell associated with the respective volume element of the environment, but also sensor data from at least one neighboring volume element are considered. This provides the advantage that the influence of neighboring volume elements and/or the context as given by neighboring volume data cells can be considered in the processing of the combined data of a respective volume data cell. For example, it can be verified, if an object that has been recognized in a specific volume element also extends into the neighboring volume element.

One further development comprises that a 3D-convolution is applied to combine at least a part of the sensor data of at least two neighboring volume data cells for generating at least a part of the combined data of these volume data cells. In other words, new data are generated by combining sensor data using a convolution operation. This has been proven to be a beneficial pre-processing for some types of sensor data. For example, a smoothing over several volume elements can be provided, which can combine separate signals that originate from the same object, for example reflections of radar radiation from different parts of the same vehicle. By applying a 3D-convolution, these signals can be smeared in order to make them appear as coming from one object, e.g. a vehicle. The operation does not need to necessarily be a 3D convolution. For example, if the voxel space is sparse, one could use a sparse convolution implementation. In general, a mathematical operation that fuses or combines spatially proximate or neighboring values.

One further development comprises that in at least one iteration step at least one invariant operator is applied to the resulting feature data for deriving modified feature data. Another expression for "invariant operator" also is invariant differential operator. An invariant differential operator is one kind of mathematical map of a function to a function of changed or different type, but with the same basic characteristics of the function, e.g. monotonicity. Applying an invariant operator can help scaling and/or emphasizing pre-defined characteristics of an object for the recognition process. One example is a gamma correction curve.

One further development comprises that the respective inverse mapping function models sensor intrinsics (intrinsic parameters) and sensor extrinsics (extrinsic parameters) of the respective sensor. Intrinsic parameters are, for example, the focal length, the image sensor format and the focal point. Extrinsic parameters are, for example, the position, main axis and rotational position of the camera on this main axis. By considering at least one of or several of these parameters, an image distortion can be compensated such that a correct association of a volume element to a volume data cell is achieved.

One further development comprises that the inverse mapping functions for the different volume data cells are implemented as an artificial neural network. In other words, the artificial neural network for processing the combined data can be preceded in the processing chain or signal chain by an artificial neural network that maps the sensor data to the respective volume data cells. This can be provided as a separate artificial neural network or, alternatively, as in the claimed invention, as specific layers of the artificial neural network that is also trained for the processing of the resulting combined data. In other words, the present invention comprises that the volume data cells are provided by one or more than one layer of the ANN, wherein the ANN comprises both the inverse mapping functions and the individual processing for the volume data cells. For implementing the inverse mapping functions, the artificial neural network can be trained for providing the inverse mapping function. For example, sensor training data may be provided together with labeling data that indicate to which volume data cell the sensor data belong to. The sensor training data and the corresponding labeling data may be generated on the basis of test drives and/or numerical simulations of the signal processing inside the sensors and/or on the basis of the extrinsics and intrinsics of the respective sensor.

As a further solution a processing unit for a vehicle is provided that is designed to perform an embodiment of the described method. The processing unit may comprise a data processing device or a processor circuit adapted to perform an embodiment of the method according to the invention. For this purpose, the processor circuit may comprise at least one microprocessor and/or at least one microcontroller and/or at least one FPGA (field programmable gate array) and/or at least one DSP (digital signal processor). Furthermore, the processor circuit may comprise program code which comprises computer-readable instructions to perform the embodiment of the method according to the invention when executed by the processor device. The program code may be stored in a data memory of the processor device.

As a further solution a motor vehicle is provided. The motor vehicle comprises an electronic control circuit that is designed to operate the vehicle in an autonomous driving mode. The motor vehicle also comprises several sensors for detecting objects in the environment of the vehicle. The control circuit is coupled to the sensors over an embodiment of said processing unit. The motor vehicle can be designed as a passenger vehicle or a truck.

The invention also comprises the combinations of the features of the different developments.

In the following, an exemplary implementations of the invention are described. The figures show:.

The embodiments explained in the following is a preferred embodiment of the invention.

However, in the embodiments, the described components of the embodiment each represent individual features of the invention which are to be considered independently of each other and which each develop the invention also independently of each other and thereby are also to be regarded as a component of the invention in individual manner.

Furthermore, the described embodiments can also be supplemented by further features of the invention already described.

In the figures identical reference signs indicate elements that provide the same function. In the following, the principle of the sensor data fusion is explained on the basis of an example that is illustrated in <FIG>.

<FIG> shows a vehicle <NUM> with sensors <NUM> and a processing unit <NUM>. The processing unit <NUM> can be, e.g., a central computer of the vehicle <NUM>. Detection ranges <NUM> of the sensors <NUM> may be directed towards the environment <NUM> of the vehicle <NUM> such that they observe or cover surroundings of the vehicle <NUM>. Different regions in the environment <NUM> can be regarded as volume elements <NUM>, but this only a virtual classification. As <FIG> shows a top view, the volume elements <NUM> are depicted as flat regions (as height is not illustrated). Each volume element <NUM> can be associated with a respective volume data cell <NUM> inside the processing unit <NUM> such that the volume data cells <NUM> constitute a volumetric representation of the environment <NUM>. Each volume data cell <NUM> can be a predefined section of a data memory of the processing unit <NUM>.

In each volume data cell <NUM>, information can be stored, the information regarding possible objects that might exist or might be present in the corresponding volume element <NUM> of the environment <NUM>. A world coordinate system <NUM> of the environment <NUM> and an internal volumetric coordinate system <NUM> of the volumetric representation illustrate how a correspondence between a virtual volume element <NUM> and a volume data cell <NUM> can be established in that a point X, Y, Z in the environment <NUM> (world coordinate system <NUM>) corresponds to a data set containing coordinates X', Y', Z' (internal volumetric coordinate system <NUM>) that may be stored in the corresponding volume data cell <NUM> together with further information about the point X, Y, Z (e.g. color and/or brightness). Note that in <FIG> only a few volume elements <NUM> and volume data cells <NUM> are labeled with a reference sign in order to keep <FIG> readable.

For detecting objects <NUM>, <NUM> in the environment <NUM>, the several sensors <NUM> can generate sensor data <NUM> that are correlated with radiation or signals <NUM> that are emitted by the objects <NUM>, <NUM>. The signals <NUM> can be, for example, light beams or reflected radar waves or ultrasonic waves, respectively. Correspondingly, the sensors <NUM> can be, for example, a camera or a radar detector or an ultrasonic transducer, respectively. Other possible sensor types are, for example, a LIDAR and an infrared sensor. Preferably, the sensors <NUM> are of different sensor types or sensing technology (e.g. radar and light).

It is now necessary to know which incoming sensor data <NUM> belong to which volume data cell <NUM>. This is accomplished on the basis of the following principle.

The path of a signal <NUM> from the respective object <NUM> to one of the sensors <NUM> and/or the processing of that signal <NUM> inside the sensor <NUM> for generating corresponding sensor data <NUM> can be described by a respective mapping <NUM> (mapping from origin of the signal <NUM> to the resulting values in the sensor data <NUM>, e.g. a resulting value of an image pixel). For distinguishing between the different mappings, <FIG> shows mapping functions F1, F2, F3, F4 as examples of possible mappings <NUM>. For each pair of a volume element <NUM> and a sensor <NUM>, a corresponding mapping <NUM> can be defined that describes which sensor data <NUM> would result, if a signal <NUM> from that volume element <NUM> was received by that sensor <NUM>. Each mapping <NUM> can describe the influence or transformation effect on a signal <NUM> on its way from the respective volume element <NUM> to the respective sensor <NUM> and/or the signal processing characteristics of the sensor <NUM> when generating the sensor data <NUM>, like, for example, a distortion in a lens of a sensor <NUM> and/or the signal damping in an electric circuit of a sensor <NUM> can be modelled by such a mapping function that describes the mapping <NUM>. Overall, the mapping function of a specific mapping <NUM> describes how the coordinates of a respective point X, Y, Z in the environment <NUM> expressed by the world coordinate system <NUM> is mapped to a point in a sensor coordinate system. For example, in the case of a <NUM>-dimensional image sensor with pixels in a <NUM>-dimensional U-V-image-plane (U, V are the coordinates in the image plane resulting in a <NUM>-dimensional sensor coordinate system), the signal <NUM> (e.g. light) from a point X, Y, Z in the environment <NUM> will be mapped to a pixel U, V in the sensor image plane (or to several pixels, if an effect like diffusion and/or chromatic aberration is also modelled by the mapping function for mapping <NUM>).

In other words, the signals <NUM> (e.g. light or radar waves) of the real world environment <NUM> coming from the volume elements <NUM> are projected to the sensors <NUM> which results in a mapping from world coordinate system <NUM> into a respective sensor coordinate system of the respective sensor <NUM>. A sensor coordinate system can be <NUM>-dimentional or <NUM>-dimensional or <NUM>-dimensional or even so-called <NUM>-dimensional (which refers to a reduced spatial resolution in a third dimension).

If sensor data <NUM> are generated by the sensors <NUM>, these sensor data <NUM> must then be transferred or mapped to the correct volume data cell <NUM> that represents the volume element <NUM> where the signal <NUM> that is described by these sensor data <NUM> possibly came from. It is worth noticing that this association is not necessarily unambiguous. For example, sensor data <NUM> from a <NUM>-dimensional image sensor <NUM> lack depth information (i.e. a third dimension) such that a distance of an object <NUM>, <NUM> is not described by the sensor data <NUM> of such a <NUM>-dimensional image sensor. The sensor data <NUM> of a specific pixel of that image sensor <NUM> might therefore result from any one volume element <NUM> that lies along a straight line of propagation described by the beam of that signal <NUM>.

For mapping the sensor data <NUM> to one or (in the case of ambiguity) to more than one volume data cell <NUM>, use can be made of the knowledge about the mapping functions <NUM>. By inverting such a mapping function <NUM>, a respective inverse mapping can be generated that indicates which sensor data <NUM> belong to which volume data cell <NUM> by describing the back projection from a respective point in the respective sensor coordinate system back into the environment <NUM> and thus also into the volumetric representations given by the volume data cells <NUM> (i.e. into the (internal volumetric coordinate system <NUM>).

If <NUM>-D camera image data (i.e. pixel data) are back projected, this can result in the fact that these pixels are associated with more than one volume data cell <NUM> as the already-described lack of depth information results in an ambiguous mapping (X,Y,Z → U,V → X',Y',Z', where Z' is not unambiguous). Therefore, several volume data cells <NUM> can receive the sensor data <NUM> of such <NUM>-dimensional pixels. Due to this ambiguity, this type of mapping can be complex.

But this principle can also be described the other way round, namely from the perspective of the volume data cells <NUM>, by regarding the sensors <NUM> from the perspective of this volumetric representation: from a given volume data cell <NUM> a look-up is performed in order to determine where this volume data cell <NUM> gets its sensor data <NUM> from. In other words, a mapping of the respective volume data cell <NUM> X',Y',Z' into the sensor coordinate system U,V of the respective sensor <NUM> is performed, i.e. from the <NUM>-dimensional internal coordinate system of the volumetric representation to, e.g. <NUM>-dimensional coordinates of an image sensor coordinate system of a camera or to a sensor coordinate system in general. The difference is, that the sensor data <NUM> are transferred in the opposite direction from the respective sensor <NUM> to the volume data cell <NUM>. These transfer functions <NUM> with inverse transfer direction (from sensor to volume data cell) are called inverse transfer functions or inverse mapping functions <NUM> (i.e. looking at the respective volume data cell <NUM> into the internal volumetric coordinate system <NUM> and transferring the sensor data from the sensor coordinate system to the internal volumetric coordinate system <NUM> of the volume data cells <NUM>).

This mapping of sensor data <NUM> of all the sensors <NUM> results in combined data <NUM> for each volume data cell <NUM>, i.e. a collection of sensor data <NUM> from different sensors <NUM>. In other words, combined data <NUM> can be determined that describe what possible object <NUM>, <NUM> could have been present in the corresponding volume element <NUM> of the environment <NUM>. Overall, for obtaining the combined data <NUM> for the respective volume data cell <NUM>, a look-up is performed from the volume data cells <NUM> of 3D volumetric representation (central representation as provided by the definition of the volume data cells <NUM>) into the sensor coordinate systems of all the sensors <NUM>.

As combined data <NUM> from different sensors <NUM> are combined or stored together in each volume data cell <NUM>, the information from different sensors <NUM> are fused and can be analyzed together in the respective volume data cell <NUM>.

By applying a respective artificial neural network ANN to the combined data <NUM> of one respective volume data cell <NUM> makes is possible to perform an object detection / recognition for the corresponding volume element <NUM> on the basis of the sensor data <NUM> of several different sensors <NUM> collected as the combined data <NUM> in the corresponding volume data cell <NUM>. Thus the sensor data <NUM> of each sensor <NUM> may contribute to the ANN-based object detection and/or recognition. Note that a volume data cell <NUM> preferably stores the sensor data <NUM> of more than only one single sensor pixel or sensor voxel. Instead, a whole region of several neighboring sensor pixels or sensor voxels is considered in each volume data cell <NUM> in order to be able to perform, e.g., an edge detection and/or an object classification on the basis of the combined data <NUM> of a single volume data cell <NUM>.

Using inverse mapping functions <NUM> links the volume data cells <NUM> to the sensors <NUM> in a way that may also provide the benefit that the training of the respective ANN may also include a training of the inverse mapping functions <NUM> (if these inverse mapping functions <NUM> are provided as a part of the ANN that analyzes the combined data as is done in the claimed invention). If the inverse mapping function <NUM> or look-up of each volume data cell <NUM> into the coordinate system of the sensors <NUM> is modelled as a differentiable function, such an inverse mapping function <NUM> can be adapted by the training procedure for the ANN. Especially, a back propagation algorithm for training the neural network may back-propagate the error gradient from an output (detection / recognition result) to an input (according to the so-called chain rule), wherein this input can be the sensors themselves, if the inverse mapping functions <NUM> are included in the training process.

Additionally or alternatively, for providing the inverse mapping functions <NUM>, a model of the respective sensor (e.g. camera) and the information where it is pointing to may be provided. For example, a parametric mapping function may be configured on the basis of these information.

<FIG> and <FIG> further illustrate in which way sensor data <NUM> are ambiguous such that from the sensor data <NUM> of one single sensor <NUM> alone, the presence of objects <NUM>, <NUM> can only be estimated with limited spatial resolution. This ambiguity results from the described sensors with less dimensions than the volumetric representation. Examples are a radar sensor and an ultrasonic sensor (both provide no height information) and an image sensor (no depth / distance information provided). Such an estimation in the form of combined data <NUM> leads to information about "possible objects" in contrast to the real objects <NUM>, <NUM>, as the exact position (exact volume element <NUM>) cannot be derived.

<FIG> introduces a scheme for labelling volume elements <NUM> of environment <NUM>. Each volume element <NUM> is represented by a pair of coordinates indicating the column C and row R (height is ignored in <FIG> as a top view perspective is shown like in <FIG>). A volume element <NUM> can be represented by its column-row coordinate C_R, for example, 3_2 for C=<NUM> and R=<NUM>. Correspondingly, a mapping functions <NUM> of name F can be labelled on the basis of the coordinates as FC_R, for example, F3_2 indicating mapping function F starting at volume element C_R (and leading to one specific sensor <NUM>). <FIG> illustrates the mapping functions <NUM> of row number <NUM>, i.e. the mapping functions F0_2 up to F3_2.

If a sensor <NUM> has no sensitivity for depth (i.e. the distance between sensor <NUM> and respective volume element <NUM> is not measured), such a sensor <NUM> can only create sensor data <NUM> that do not describe how far the respective volume element <NUM> or more precisely an object contained in one of the volume elements <NUM> of the same row R is away from the sensor <NUM>. An example of such a sensor <NUM> is an image sensor of a camera. In other words, the information about the column C is not represented in the sensor data <NUM>.

<FIG> illustrates the consequence with regard to applying inverse mapping functions <NUM> to sensor data <NUM> of such a sensor <NUM> that has no depth sensitivity. As an example, sensor data <NUM> may represent a camera image <NUM> showing a Street <NUM>, a tree <NUM> and the object <NUM> (see <FIG>). From the sensor data <NUM> containing camera image <NUM>, it cannot be seen, how far the object <NUM> is away from sensor <NUM>. Consequently, when inverse mapping functions <NUM> are applied to the sensor data <NUM>, combined data <NUM> result for each column C' in the volume data cells <NUM>. Column C' is the column in the internal volumetric coordinate system <NUM> describing the column C in the environment <NUM> (see <FIG>).

Applying the inverse mapping functions <NUM> to the sensor data <NUM> reveals the ambiguity as regards depth information. However, the inverse mapping functions <NUM> generate combined data <NUM> that indicate that if an object <NUM> is very close to sensor <NUM> (left column), the object <NUM> must be smaller than in the case that the object <NUM> is far away from sensor <NUM> (right column). The combined data <NUM> would be different as regards the size of the possible object <NUM>.

Using only the combined data <NUM> of one single sensor for performing an object detection / recognition on the basis of an artificial neural network might therefore deteriorate the detection / recognition performance as the size of an object <NUM> can be a valuable hint for the artificial neural network.

<FIG> illustrates how the detection / recognition performance can be improved, if combined data <NUM>, <NUM>' of different sensors <NUM> are combined in the volume data cells <NUM>. <FIG> illustrates the possible content of volume data cells <NUM> for object <NUM> (see <FIG>): the combined data <NUM> of a sensor <NUM> with low depth sensitivity, like an image sensor, and the combined data <NUM>' of a second sensor <NUM> with higher depth resolution, but lower angular resolution (resulting in ambiguous information regarding the row R (i.e. Y'-coordinate)) can be combined. An example for a sensor with low angular resolution is a radar sensor or an ultrasonic sensor.

Combining the combined data <NUM>, <NUM>' will result in only one volume data cell <NUM> where the combined data <NUM>, <NUM>' of both sensors <NUM> indicate the presence of a possible object <NUM>'. All the other volume data cells <NUM> provide combined data <NUM>, <NUM>' with a weaker or lower indication for a presence of a possible object. Applying an artificial neural network to all the data cells <NUM>, <NUM>, will provide a clearer or more confident detection/recognition result for volume data cell <NUM> as in this volume data cell <NUM> the combined data <NUM>, <NUM>' of two sensors describe more features or characteristics of the object <NUM>. The artificial neural network receives more hints indicating the presence and features of the object <NUM>.

Thus, by mapping sensor data <NUM> of several different sensors <NUM> using the inverse mapping functions <NUM>, an artificial neural network can be supplied with the combined information from several sensors. There is no need to perform a detection/recognition for each sensor <NUM> individually and then combine the detection/recognition results afterwards. One single detection/recognition process for each volume data cell <NUM> is needed only and this detection / recognition process can additionally be based on the combined information from several sensors (i.e. the combined data <NUM>, <NUM>'). If more than two sensors are available, even more contribution data can be generated for each volume data cell <NUM> making the detection/recognition process even more robust as even more features of the object are described.

<FIG> shows the motor vehicle <NUM> that can be designed, for example, as a passenger vehicle or a truck. The motor vehicle <NUM> may drive or perform a motion <NUM> through an environment <NUM>, e.g. along a road. A steering and/or an acceleration and/or a braking of the vehicle <NUM> may be performed autonomously by an electronic control circuit <NUM> that is providing an autonomous driving function. Like is known from the prior art, the control circuit <NUM> may control actuators <NUM> for controlling the steering and/or an engine and/or brakes of the vehicle. For deciding or calculating a trajectory <NUM> for the motion <NUM>, the control circuit <NUM> may receive feature data <NUM> of at least one object <NUM> in the surroundings or the environment <NUM> of the vehicle <NUM>. In <FIG>, as an exemplary object <NUM> a traffic light is depicted. Other possible objects may be the road, traffic participants (e.g. vehicles, cyclists, pedestrians), buildings, road-side installations. The feature data <NUM> describing the respective recognized object <NUM> may be provided by an object recognition <NUM>, which may be based, for example, on an artificial neural network ANN.

The object recognition <NUM> may receive sensor data <NUM> from several different sensors <NUM> of the vehicle <NUM>. Respective detection ranges <NUM> of the sensors <NUM> may be directed towards or into the environment <NUM> such that radiation or signals <NUM>, for example electromagnetic waves and/or sound waves, emitted or reflected by the respective object <NUM>, may be received by the sensors <NUM>. The signals <NUM> may be transformed into the sensor data <NUM> or may be used for generating the sensor data <NUM> in the respective sensor <NUM>. The sensor data <NUM> may be generated or updated by the sensors <NUM> periodically, for example at a given frame rate or in periodic measurement cycles, in a range from, e.g. <NUM> per second to <NUM> every <NUM> minutes. Exemplary sensors can be: camera, radar, ultrasonic, lidar, infrared, just to name examples.

As the sensor data <NUM> originate from different sensors <NUM>, in the vehicle <NUM>, a sensor data fusion <NUM> is provided as a pre-processing for the object recognition <NUM>.

The sensor data fusion <NUM> and the artificial neural networks ANN may be operated or provided in a processing unit <NUM> of the vehicle <NUM>. To this end, the processing unit <NUM> may comprise at least one microprocessor and/or at least one data storage.

For the sensor data fusion <NUM>, virtual volume elements <NUM> may be defined in the environment <NUM>. Each volume element <NUM> may form the basis or a separate unit for the recognition process in the object recognition <NUM> in that the sensor data <NUM> belonging to a respective volume element <NUM> are provided in combination in a respective volume data cell <NUM> to the object recognition <NUM>. To achieve this, the sensor data fusion <NUM> is designed as a preprocessing <NUM> that provides an inverse transfer function <NUM> for mapping or associating sensor data <NUM> depending on their position in the sensor coordinate system of the respective sensor to a respective volume data cell <NUM> that represents a specific volume element <NUM>.

The sensor data fusion <NUM> may be designed as an extension or as additional layers of the artificial neural network ANN. For training such an extension of the artificial neural network ANN, sensor training data may be generated, either artificially/numerically or in test drives or in a testing phase, where for a is specific object, the resulting sensor data <NUM> and the correct volume data cell <NUM> are known. This can be provided as sensor training data and labeling data for training a separate artificial neural network or said extension for the ANN that is to perform the sensor data fusion <NUM>.

Alternatively, an inverse mapping function <NUM> may be defined on the basis of an equation or deterministic algorithm describing, for example, the extrinsics and/or the intrinsics of the respective sensor <NUM>. The relation of a volume element <NUM> to a volume data cell <NUM> can be expressed by using a coordinate transformation or mapping for transforming the world coordinate system <NUM> of the environment <NUM> to the sensor coordinate system of the respective sensor <NUM>. The inverse mapping function <NUM> describes the inverse process whereby a possible reduction in dimensions from the <NUM>-dimensional world coordinate systems <NUM> to the, for example, two-dimensional or <NUM>-dimensional coordinate system of the respective sensor <NUM> can be taken into account. The respective inverse mapping function <NUM> may be expressed on the basis of a vector transform or implicitly by the respective training of the layers of the artificial neural network that performs the sensor data fusion <NUM>.

Thus, in each volume data cell <NUM>, the sensor data <NUM> of different sensors <NUM> are combined as combined data <NUM> that may then be provided together or in combination as an input to the object recognition <NUM>. Thus, the object recognition <NUM> may be based not only on the sensor data <NUM> of one single sensor <NUM> at a time, but the characteristics of an object <NUM> as reflected by the sensor data <NUM> of several different sensors <NUM> (especially of sensor of different sensor types) can be processed together or at the same time or in combination. The object recognition <NUM> is a processing P that follows the generation of the combined data <NUM>.

Thus, the artificial neural network ANN is provided with data chunks or data blocks of combined data <NUM> belonging to the respective volume data cell <NUM>. The object recognition of possible objects <NUM> in the environment <NUM> can be performed for each volume element <NUM> separately. In this context, the artificial neural network ANN can also be regarded as several separate artificial neural networks that perform the processing P for the object recognition <NUM> individually for one separate volume element <NUM>.

The generation of the combined data <NUM> may also comprise that the output of the inverse mapping function <NUM>, may be further pre-processed using, e.g., a convolution for convolving the output of the different volume data cells <NUM>. This can be performed for sensor data originating from the same sensor and/or from different sensors. Additionally or alternatively, the combined data <NUM> of neighboring volume data cells <NUM> may be provided together to the same part of the artificial neural networks ANN that is associated with one specific volume element in order to provide information about the context or the neighborhood of the respective corresponding volume data cell <NUM>.

Claim 1:
Method for operating a processing unit (<NUM>) of a vehicle (<NUM>) for processing sensor data (<NUM>, <NUM>) of several different sensors (<NUM>, <NUM>) with an artificial neural network, wherein a respective detection range (<NUM>) of each sensor (<NUM>, <NUM>) is directed towards an environment (<NUM>, <NUM>) of the vehicle (<NUM>), the method comprising:
- a set of volume data cells (<NUM>, <NUM>) is provided as a volumetric representation of different volume elements (<NUM>, <NUM>) of the environment (<NUM>, <NUM>), wherein for each combination of one of the volume elements (<NUM>, <NUM>) and one of the sensors (<NUM>, <NUM>) a respective mapping of a world coordinate system (<NUM>) of the environment (<NUM>, <NUM>) to a respective sensor coordinate system of the respective sensor (<NUM>, <NUM>) is given, and
- when sensor data (<NUM>, <NUM>) is generated by the sensors (<NUM>, <NUM>)
- the sensor data (<NUM>, <NUM>) is transferred to the respective volume data cells (<NUM>, <NUM>) using an inverse mapping function (<NUM>, <NUM>), which provides an inverse to the mapping (<NUM>), such that each inverse mapping function (<NUM>, <NUM>) is a mapping of the respective sensor coordinate system to an internal volumetric coordinate system (<NUM>) corresponding to the world coordinate system (<NUM>), and by the transfer of the sensor data (<NUM>, <NUM>) each volume data cell (<NUM>, <NUM>) receives the sensor data (<NUM>, <NUM>) that are associated with this volume data cell (<NUM>, <NUM>) according to the inverse mapping function (<NUM>, <NUM>) of each sensor (<NUM>, <NUM>), wherein the received sensor data (<NUM>, <NUM>) from each sensor (<NUM>, <NUM>) are accumulated in the respective volume data cell (<NUM>, <NUM>) as combined data (<NUM>, <NUM>'), and
- for each volume data cell (<NUM>, <NUM>) an individual processing (P) of the combined data (<NUM>, <NUM>') of that volume data cell (<NUM>, <NUM>) is performed by the artificial neural network, ANN, wherein the individual processing comprises deriving feature data (<NUM>) of at least one feature of at least one object (<NUM>') that is described by the combined data (<NUM>, <NUM>'), wherein the ANN performs both the inverse mapping and the individual processing (P) for the volume data cells (<NUM>, <NUM>).