Patent Description:
By way of example, for autonomous driving, imaging sensors, such as camera sensors and/or video sensors, may be used to provide digital images of the surroundings of a vehicle. The digital images may illustrate objects, such as cars, bicycles, pedestrians, street signs etc., and may be segmented and classified by semantic segmentation using a segmentation model and the vehicle may be controlled depending on the segmented and classified digital images. Thus, for safety reasons, it is necessary that the segmentation model is capable of providing segments for the objects in the digital image with sharp boundaries.

Various models, such as neural networks, are applied in the field of computer vision. By way of example, neural networks may be used to generate a segmented image using an image detected by an imaging sensor. However, a segment of the segmented image may also include pixels not belonging to an object shown in the detected image. Thus, it may be necessary to provide a model capable of generating a segmented image for a detected image with sharp object boundaries of the segmented objects.

In <NPL>, a generative adversarial network is described, wherein a segmented part of an image is copied to another image.

Exemplary training methods are disclosed in <NPL>; in<NPL>; in <NPL>; in <NPL>; and in <NPL>.

According to the present invention, a method defined in claim <NUM>, a training device defined in claim <NUM>, systems defined in claims <NUM> , and a vehicle defined in claim <NUM> are provided.

The method and the device enable a model to be trained to generate a segmented image for a digital image with improved object boundaries of a segmented object.

A model may be any kind of algorithm, which provides output data for input data. A model may be a differential model. For example, a model may be a neural network.

The segmentation image may be a binary segmentation image. The segmentation image may include a first binary value "<NUM>" for the segment and a second binary value "<NUM>" for the background.

The digital image may be a color image, for example an RGB image.

Determining a predictability of the digital image from the segment of the segmentation image may include generating a segment of the digital image using the segment of the segmentation image and the digital image. Determining a predictability of the digital image from the segment of the segmentation image may further include determining a predictability of the digital image from the segment of the digital image. The segmentation model may be trained to reduce, for example to minimize, the predictability of the digital image from the segment of the digital image. Thus, the pixels of the segment of the digital image share higher level of abstractions that are more independent of the other pixels. This has the effect that the segmentation method is trained unsupervised, i.e. using digital images only and no human supervision.

Determining a predictability of the digital image from the segment of the digital image may include a generative model generating a reconstructed digital image for the segment of the digital image. The reconstructed digital image may be a prediction of the digital image. The segmentation model may be trained to increase the difference between the reconstructed digital image and the digital image. For example, training the segmentation model to increase the difference between the reconstructed digital image and the digital image may include determining a first loss value by comparing the reconstructed digital image with the digital image and the segmentation model may be trained by increasing, for example maximizing, the first loss value.

The method may further include training the generative model by reducing, for example minimizing, the first loss value.

Determining a predictability of the digital image from the segment of the digital image may include a generative model generating a reconstructed digital image for the segment of the digital image. The reconstructed digital image may be a prediction of the digital image. Determining a predictability of the digital image from the segment of the digital image may further include a discriminative model determining a probability for the reconstructed digital image to belong to a class including the digital image. The segmentation model may be trained to increase the probability determined by the discriminative model. For example, training the segmentation model to increase the probability determined by the discriminative model may include determining a second loss value using the probability determined by the discriminative model and the segmentation model may be trained by increasing, for example maximizing, the second loss value.

The method may further include training the generative model by reducing, for example minimizing, the second loss value.

The method may further include training the discriminative model by increasing, for example maximizing, the second loss value.

Determining a predictability of the digital image from the segment of the segmentation image may include determining a predictability of the background of the segmentation image from the segment of the segmentation image. The segmentation model may be trained to reduce, for example to minimize, the predictability of the background of the segmentation image from the segment of the segmentation image.

The method may further include generating a background of the digital image for the background of the segmentation image using the digital image. Determining a predictability of the background of the segmentation image from the segment of the segmentation image may include determining a predictability of the background of the digital image from the segment of the digital image. The segmentation model may be trained to reduce, for example to minimize, the predictability of the background of the digital image from the segment of the digital image.

Determining a predictability of the background of the digital image from the segment of the digital image includes generating a background of an additional digital image using the background of the segmentation image and the additional image. Determining a predictability of the background of the digital image from the segment of the digital image includes generating a combined digital image for the segment of the digital image and the background of the additional digital image. Determining a predictability of the background of the digital image from the segment of the digital image includes a discriminative model determining a probability for the combined digital image to belong to a class including the digital image and/or the additional digital image. The segmentation model is trained to increase the probability determined by the discriminative model.

For example, training the segmentation model to increase the probability determined by the discriminative model may include determining a third loss value using the probability determined by the discriminative model and the segmentation model may be trained by increasing, for example maximizing, the third loss value. This has the effect that the segmentation model is trained to segment regions such that if they are pasted to an additional image, the combined image looks like a realistic sensor-detected image. Thus, the segmentation model provides segments that are independent of the background of the segments.

The method may further include training the discriminative model by reducing, for example minimizing, the third loss value.

The method may further include a sensor detecting the digital image.

The sensor may include a camera sensor, a video sensor, a radar sensor, a LiDAR sensor, an ultrasonic sensor, a motion sensor, or a thermal sensor.

The digital image may include a plurality of objects. The segmentation model may generate a segmentation image for each object of the plurality of objects, wherein each segmentation image may include a segment describing the object. The method may be performed for each segmentation image of the plurality of segmentation images.

At least a part of the segmentation model may be implemented by one or more processors.

The segmentation model may be a neural network.

The method may further include the trained segmentation model generating training data.

The method may include training an additional model using the training data.

A computer program product may store program instructions configured to, if executed, perform the method of any one of the first example to the twentieth example.

A system may include a device of the twenty-second example or the twenty-third example. The system may further include at least one sensor. The at least one sensor may be configured to provide a digital image for the device.

A system may include a generative adversarial network (GAN). The GAN may include a plurality of generator networks, wherein at least one generator network of the plurality of generator networks may include the segmentation model trained by the method of any one of the first example to the nineteenth example.

A system may include a generative adversarial network (GAN). The GAN may include a plurality of generator networks, wherein at least one generator network of the plurality of generator networks may include the additional model trained by the method of the twentieth example.

A vehicle includes at least one imaging sensor, configured to provide digital images.

The vehicle includes a driving assistance system. The driving assistance system includes a segmentation model trained by the method of claim <NUM> The segmentation model is configured to provide a segmentation image for each digital image. The driving assistance system is configured to control the vehicle using the segmentation images. Thus, the vehicle is capable of detecting surrounding objects in an improved manner. In other words, the trained segmentation model is capable to provide segmented images with sharp object boundaries improving the classification of the segmented objects and thus the control of the vehicle.

Various embodiments of the invention are described with reference to the following drawings, in which:.

In an embodiment, a "circuit" may be understood as any kind of a logic implementing entity, which may be hardware, software, firmware, or any combination thereof. Thus, in an embodiment, a "circuit" may be a hard-wired logic circuit or a programmable logic circuit such as a programmable processor, e.g. a microprocessor (e.g. a Complex Instruction Set Computer (CISC) processor or a Reduced Instruction Set Computer (RISC) processor). A "circuit" may also be software being implemented or executed by a processor, e.g. any kind of computer program, e.g. a computer program using a virtual machine code such as e.g. Java. Any other kind of implementation of the respective functions which will be described in more detail below may also be understood as a "circuit" in accordance with an alternative embodiment.

In the field of computer vision segmentation models are applied for image segmentation and various systems may be controlled based on the segmented images. However, segments of segmented images may also include pixels not belonging to an object shown in the processed image, which may result in an unwanted control of the respective system. Illustratively, a segmentation model is trained to provide a segmented image for a digital image, wherein the segments, which describe a respective object in the digital image, represent the object with sharp object boundaries.

<FIG> shows a device <NUM>. The device <NUM> may include one or more sensors <NUM>. The sensor <NUM> may be configured to provide (digital) images, for example a plurality of digital images <NUM> including the digital image <NUM>. The sensor <NUM> may be any kind of sensor, which is capable of providing (digital) images, for example an imaging sensor, such as a camera sensor or a video sensor, a radar sensor, a LiDAR sensor, an ultrasonic sensor, a motion sensor, a thermal sensor, etc. The plurality of sensors may be of the same type of sensor or of different sensor types. The device <NUM> may further include a memory device <NUM>. The memory device <NUM> may include a memory which is for example used in the processing carried out by a processor. A memory used may be a volatile memory, for example a DRAM (Dynamic Random Access Memory) or a non-volatile memory, for example a PROM (Programmable Read Only Memory), an EPROM (Erasable PROM), EEPROM (Electrically Erasable PROM), or a flash memory, e.g., a floating gate memory, a charge trapping memory, an MRAM (Magnetoresistive Random Access Memory) or a PCRAM (Phase Change Random Access Memory). The memory device <NUM> may be configured to store the plurality of digital images <NUM>, such as the digital image <NUM>, provided by the one or more sensors <NUM>. The device <NUM> may further include at least one processor <NUM>. The at least one processor <NUM> may be any kind of circuit, i.e. any kind of logic implementing entity, as described above. In various embodiments, the processor <NUM> may be configured to process the digital image <NUM>.

<FIG> shows a processing system <NUM> for training a segmentation model.

The processing system <NUM> may include the memory device <NUM>. The memory device <NUM> may store the digital image <NUM>. The digital image <NUM> may include at least one object <NUM>. The processing system <NUM> may further include the at least one processor <NUM>. The processor <NUM> may be configured to process the digital image <NUM>. The processor <NUM> may be configured to implement at least a part of a segmentation model <NUM>. The segmentation model <NUM> may be a neural network, wherein the neural network may be any kind of neural network, such as a convolutional neural network. The neural network may include any number of layers and the training of the neural network, i.e. adapting the layers of the neural network, may be based on any kind of training principle, such as backpropagation, i.e. the backpropagation algorithm.

The segmentation model <NUM> is configured to process the digital image <NUM> and is further configured to generate a segmentation image <NUM> for the digital image <NUM>. The segmentation image <NUM> include a segment <NUM> describing the at least one object <NUM> in the digital image <NUM>. The segmentation image <NUM> further includes a background <NUM> of the segment <NUM>. In other words, the segmentation image <NUM> includes a segment <NUM>, which describes at least one object <NUM> in the digital image, and a background <NUM>, which describes a background <NUM> of the at least one object <NUM> in the digital image. The digital image <NUM> may be a color image (for example RGB) and the segmentation image <NUM> may be a binary image. The segmentation image <NUM> may include a first binary value "<NUM>" for the segment <NUM> and a second binary value "<NUM>" for the background <NUM> or may include a first binary value "<NUM>" for the segment <NUM> and a second binary value "<NUM>" for the background <NUM>. In other words, the digital image <NUM> and the segmentation image <NUM> each include a plurality of pixels, wherein the segment <NUM> of the segmentation image <NUM> includes a first plurality of pixels associated to a first plurality of pixels in the digital image <NUM>, wherein the first plurality of pixels in the digital image <NUM> describe the at least one object <NUM> in the digital image <NUM>. The background <NUM> of the segmentation image <NUM> includes a second plurality of pixels associated to a second plurality of pixels in the digital image <NUM>, wherein the second plurality of pixels in the digital image <NUM> describe the background of the at least one object <NUM> in the digital image <NUM>. Thus, for example, the segment <NUM> described by the first plurality of pixels in the segmentation image <NUM> may include the first binary value "<NUM>" and the background <NUM> described by the second plurality of pixels in the segmentation image <NUM> may include the second binary value "<NUM>".

The processor <NUM> is be further configured determine a predictability <NUM> of the digital image <NUM> from the segment <NUM> of the segmentation image <NUM>. The processor <NUM> is be configured to adapt, i.e. to train, the segmentation model <NUM> to reduce, for example to minimize, the predictability <NUM> of the digital image <NUM> from the segment <NUM> of the segmentation image <NUM>.

The processing system <NUM> may correspond substantially to the processing system <NUM>, wherein the processor <NUM> is further configured to generate a segment <NUM> of the digital image <NUM>. The segment <NUM> of the digital image <NUM> may be generated using the segment <NUM> of the segmentation image <NUM> and the digital image <NUM>. The segment <NUM> (S(X)⊙X) of the digital image <NUM> may be determined by the Hadamard product of the segmentation image <NUM> (S(x)) for the digital image <NUM> and the digital image <NUM> (X), wherein the digital image <NUM> may be a color image and wherein the segmentation image <NUM> may include a first binary value "<NUM>" for the segment <NUM> and a second binary value "<NUM>" for the background <NUM>. In other words, the segmentation image <NUM> may include a first binary value "<NUM>" for the pixels associated to the segment <NUM> and a second binary value "<NUM>" for the pixels associated to the background <NUM>. Illustratively, the segmentation image <NUM> may be used as a mask and the segment <NUM> of the digital image <NUM> may be a part cut out of the digital image. The processor <NUM> may be further configured to determine a predictability <NUM> of the digital image <NUM> from the segment <NUM> of the digital image <NUM>. The predictability <NUM> of the digital image <NUM> from the segment <NUM> of the digital image <NUM> may be determined by equation (<NUM>): <MAT>.

The processor <NUM> may be configured to train the segmentation model <NUM> by reducing, for example minimizing, the predictability <NUM> of the digital image <NUM> from the segment <NUM> of the digital image <NUM>, wherein the training may include the objective function given by equation (<NUM>): <MAT> wherein S is the segmentation model <NUM> generating the segmentation image <NUM> S(X) for the digital image <NUM> X.

The processing system <NUM> may correspond substantially to the processing system <NUM>, wherein the processor <NUM> may be further configured to implement at least a part of a generative model <NUM>. The generative model <NUM> may be a neural network, for example a generator. The generative model <NUM> may be configured to process the segment <NUM> of the digital image <NUM>. The generative model <NUM> may be configured to generate a reconstructed digital image <NUM> for the segment <NUM> of the digital image <NUM>. The reconstructed digital image <NUM> may be a prediction of the digital image <NUM>. In other words, the generative model <NUM> may presume a background <NUM> of the digital image <NUM> using the segment <NUM> of the digital image <NUM> to reconstruct the digital image <NUM>. In even other words, the generative model <NUM> may be an Inpainter model generating a background of the segment <NUM> of the digital image <NUM>. The processor <NUM> may be configured to determine a first loss value <NUM> by comparing the reconstructed image <NUM> with the digital image <NUM>. The first loss value <NUM> may be determined by a mean square error (MSE). Thus, MSE may be used as quantification of the predictability <NUM> of the digital image <NUM> from the segment <NUM> of the digital image <NUM>. The first loss value <NUM> (L<NUM>) may be determined by equation (<NUM>): <MAT> wherein S is the segmentation model <NUM> and G is the generative model <NUM>.

The processor <NUM> may be further configured to train the segmentation model <NUM> by increasing, for example maximizing, the first loss value <NUM>. The processor <NUM> may be configured to train the generative model <NUM> by reducing, for example minimizing, the first loss value <NUM>. Thus, the predictability <NUM> of the digital image <NUM> from the segment <NUM> of the digital image <NUM> is reduced, for example minimized, by increasing, for example maximizing, the first loss value <NUM>. In other words, the generative model <NUM> may be trained to reduce, for example minimize, the loss between a generated reconstructed digital image <NUM> and the digital image <NUM>, and the segmentation model <NUM> may be trained to increase, for example maximize, the loss between a generated reconstructed digital image <NUM> and the digital image <NUM>. Illustratively, the quality of the segment <NUM> of the segmentation image <NUM> generated by the segmentation model <NUM> is improved by generating a segmentation image <NUM> for which the generative model <NUM> fails to reconstruct the digital image <NUM> from the segment <NUM> of the digital image <NUM>. In other words, a part of the digital image <NUM>, for example the background <NUM> of the digital image <NUM>, may be hidden such that the generative model <NUM> fails to reconstruct the digital image <NUM>. Thus, the generative model <NUM> may fail to reconstruct the digital image <NUM> from the segment <NUM> of the digital image <NUM>, if the segment <NUM> generated by the segmentation model <NUM> includes sharp object boundaries. In other words, the generative model <NUM> may fail to reconstruct the digital image <NUM> from the segment <NUM> of the digital image <NUM>, if the segment <NUM> generated by the segmentation model <NUM> includes the pixels associated to the at least one object <NUM> in the digital image <NUM> and does not include pixels associated to the background <NUM> of the digital image <NUM>.

The objective function for training the segmentation model <NUM> (S) and the generative model <NUM> (G) may be described by equation (<NUM>): <MAT>.

The processing system <NUM> may correspond substantially to the processing system <NUM>, wherein the processor <NUM> may be further configured to implement at least a part of the generative model <NUM> and may be configured to implement at least a part of a discriminative model <NUM>. The discriminative model <NUM> may be a neural network, for example a discriminator. The generative model <NUM> may be configured to process the segment <NUM> of the digital image <NUM>. The generative model <NUM> may be configured to generate a reconstructed digital image <NUM> for the segment <NUM> of the digital image <NUM>, wherein the reconstructed digital image <NUM> may be a prediction of the digital image <NUM>. The discriminative model <NUM> may be configured to determine a probability <NUM> for the reconstructed digital image <NUM> to belong to a class including the digital image <NUM>. In other words, the discriminative model may include a first class and the first class may include the digital image <NUM>. The discriminative model <NUM> may further include a second class, wherein the second class may include at least one image generated by a generative model, such as the generative model <NUM>. Thus, the first class may be associated to "real" images, such as sensor-detected images, and the second class may be associated to "fake" images, such as generated images. Illustratively, the discriminative model <NUM> may be configured to determine a probability of the reconstructed digital image <NUM> to be a "real" image, i.e. a sensor-detected image. Thus, the processing system <NUM> may include a generative adversarial network (GAN), including the generative model <NUM> and the discriminative model <NUM>. The objective function of a GAN may be described by equation (<NUM>): <MAT> wherein D is a discriminative model, wherein <MAT> is the distribution of the "real" image, for example the digital image <NUM>, and wherein <MAT> is a random distribution.

Considering an optimal discriminative model D*, a loss (L(D*, G)) for the optimal discriminative model and the generative model may be described by equation (<NUM>): <MAT> wherein JSD is the Jensen-Shannon-Divergence, and wherein <MAT> is the distribution of a generated image, for example the reconstructed digital image <NUM>.

The generative model <NUM> may be configured to generate the reconstructed digital image <NUM> for the segment <NUM> of the digital image <NUM>. Thus, <MAT> may determine, for example approximate, Pg(X|S(X)⊙X) by reducing, for example minimizing, equation (<NUM>), wherein <MAT> is reduced, for example minimized, for sharp object boundaries of the segment <NUM> of the digital image <NUM> generated using the segment <NUM> of the segmentation image <NUM>, wherein JSD may be considers as quantification of the predictability <NUM> of the digital image <NUM> from the segment <NUM> of the digital image <NUM>. Thus, relative entropy may be used as quantification of the predictability <NUM> of the digital image <NUM> from the segment <NUM> of the digital image <NUM>. According to various embodiments, equation (<NUM>) is reduced, for example minimized, if the segmentation model <NUM> generates the segment <NUM> of the segmentation image <NUM> with sharp object boundaries. Illustratively, the generative model fails to reconstruct the digital image <NUM> from the segment <NUM> of the digital image for a segment <NUM> of the segmentation image <NUM> with sharp object boundaries. The processor <NUM> may be configured to determine a second loss value <NUM> using the probability <NUM> determined by the discriminative model <NUM>. The second loss value <NUM> (L<NUM>) may be determined by equation (<NUM>): <MAT>.

The processor <NUM> may be further configured to train the segmentation model <NUM> by increasing, for example maximizing, the second loss value <NUM>. As described above, the predictability <NUM> of the digital image <NUM> from the segment <NUM> of the digital image <NUM> is reduced, for example minimized, by training the segmentation model <NUM> to increase, for example maximize, the second loss value <NUM>. The processor <NUM> may be configured to train the generative model <NUM> by reducing, for example to minimize, the second loss value <NUM>. The second loss value <NUM> may be minimized using the logD trick. The predictability <NUM> of the digital image <NUM> from the segment <NUM> of the digital image <NUM> may be determined using the Wasserstein distance. The processor <NUM> may be further configured to train the discriminative model <NUM> by increasing, for example maximizing, the second loss value <NUM>. In other words, the generative model <NUM> may be trained to generate a reconstructed digital image <NUM> from the segment <NUM> of the digital image <NUM> for which the discriminative model <NUM> determines a high probability <NUM> to belong to the first class associated with "real" images; the discriminative model <NUM> may be trained to determine a low probability <NUM> for the reconstructed digital image <NUM> to belong to the first class; and the segmentation model <NUM> may be trained to generate a segmentation image <NUM> for which the discriminative model <NUM> determines a low probability <NUM> for the reconstructed digital image <NUM> to belong to the first class.

Hence, the objective function for training the segmentation model <NUM> (S), the generative model <NUM> (G) and the discriminative model <NUM> (D) may be described by equation (<NUM>): <MAT>.

The processing system <NUM> may correspond substantially to the processing system <NUM>, wherein the processor <NUM> may be further configured to generate a background <NUM> of the digital image <NUM>. The background <NUM> of the digital image <NUM> may be generated using the background <NUM> of the segmentation image <NUM> and the digital image <NUM>. The background <NUM> of the segment <NUM> (S(X)⊙ X) of the digital image <NUM> may be determined by the Hadamard product of the segmentation image <NUM> (S(x)) for the digital image <NUM> and the digital image <NUM> (X), wherein the digital image <NUM> may be a color image and wherein the segmentation image <NUM> may include a first binary value "<NUM>" for the segment and a second binary value "<NUM>" for the background. In other words, the background <NUM> may be described by <NUM>-S(X) for a first binary value "<NUM>" for the segment and a second binary value "<NUM>" for the background. In even other words, for <NUM>-S(X), the pixels associated to the background <NUM> of the segmentation image <NUM> may include the first binary value "<NUM>" and the pixels associated to the segment <NUM> of the segmentation image <NUM> may include the second binary value "<NUM>". Thus, the background <NUM> of the digital image <NUM> may be described by <NUM>-S(X)⊙ X. Illustratively, the segmentation image <NUM> may be used as a mask and the background <NUM> of the digital image <NUM> may be a part cut out of the digital image <NUM>.

The processor <NUM> may be further configured determine a predictability <NUM> of the background <NUM> of the digital image <NUM> from the segment <NUM> of the digital image <NUM>. The predictability <NUM> of the background <NUM> of the digital image <NUM> from the segment <NUM> of the digital image <NUM> may be determined by equation (<NUM>): <MAT>.

The processor <NUM> is configured to train the segmentation model <NUM> by reducing, for example minimizing, the predictability <NUM> of the background <NUM> of the digital image <NUM> from the segment <NUM> of the digital image <NUM>, wherein the training may be described by the objective function given by equation (<NUM>): <MAT>.

The predictability <NUM> of the background <NUM> of the digital image <NUM> from the segment <NUM> of the digital image <NUM> is quantified by the mutual information (I) of the background <NUM> of the digital image <NUM> and the segment <NUM> of the digital image <NUM> and may be determined by equation (<NUM>): <MAT>.

<FIG> shows a processing system <NUM> for training a segmentation model according to an embodiment. The processing system <NUM> may include the memory device <NUM>. The memory device <NUM> may store the digital image <NUM>. The memory device <NUM> may further store an additional digital image <NUM>. The additional digital image702 may be provided by the one or more sensors <NUM>. The additional digital image <NUM> may have the same dimensions, i.e. the same resolution, as the digital image <NUM>. The processing system <NUM> may further include the at least one processor <NUM>. The processor <NUM> may be configured to implement at least a part of the segmentation model <NUM>, wherein the segmentation model <NUM> is configured to generate the segmentation image <NUM> for the digital image <NUM>. The processor <NUM> is configured to generate the segment <NUM> of the digital image <NUM> using the segmentation image <NUM> and the digital image <NUM>.

The processor <NUM> is further configured to generate a background <NUM> of the additional digital image <NUM>. The processor <NUM> is configured to generate the background <NUM> of the additional digital image <NUM> using the background <NUM> of the segmentation image <NUM> and the additional digital image <NUM>. The background <NUM> of the segmentation image <NUM> may be described by (<NUM>-S(X))⊙ X, wherein the pixels associated to the background <NUM> of the segmentation image <NUM> include the first binary value "<NUM>" and wherein the pixels associated to the segment <NUM> of the segmentation image <NUM> include the second binary value "<NUM>". The background <NUM> of the additional image <NUM> is determined by the Hadamard product of the background <NUM> of the segmentation image <NUM> (<NUM>-S(X)) and the additional digital image <NUM>, wherein the additional digital image <NUM> may be described by Y. Thus, the background <NUM> of the additional digital image <NUM> may be described by (<NUM>-S(X))⊙ Y. Illustratively, the segmentation image <NUM> may be a mask and the background <NUM> of the digital image <NUM> may be a part cut out of the additional digital image <NUM> using the mask.

The processor <NUM> is further configured to generate a combined digital image <NUM>.

The processor <NUM> is configured to generate the combined digital image <NUM> using the segment <NUM> of the digital image <NUM> and the background <NUM> of the additional digital image <NUM>. In other words, the combined digital image <NUM> includes the pixels of the digital image <NUM> associated to the segment <NUM> describing the at least one object <NUM> and includes the pixels of the additional digital image <NUM> associated to the background <NUM> of the additional digital image <NUM>. Illustratively, the segment <NUM> of the digital image <NUM> may be a first part cut out of the digital image <NUM> using the segmentation image <NUM> and the background <NUM> of the additional digital image <NUM> may be a second part cut out of the additional digital image <NUM> using the segmentation image <NUM>. In other words, each pixel of the plurality of pixels of the combined digital image <NUM> may be either associated to the digital image <NUM> or to the additional digital image <NUM>. The processor <NUM> may be configured to implement at least a part of the discriminative model <NUM>. The discriminative model <NUM> is configured to determine a probability <NUM> for the combined digital image <NUM> to belong to a class including the digital image <NUM> and/or the additional digital image <NUM>. In other words, the discriminative model <NUM> may include a first class and the first class may include the digital image <NUM> and/or the additional digital image <NUM>. The discriminative model <NUM> may further include a second class, wherein the second class may include at least one image generated by a generative model. Thus, the first class may be associated to "real" images, such as sensor-detected images, and the second class may be associated to "fake" images, such as generated images. Illustratively, the discriminative model <NUM> may be configured to determine a probability <NUM> of the combined digital image <NUM> to be a "real" image, i.e. a sensor-detected image. The processor <NUM> may be configured to determine a third loss value <NUM> using the probability <NUM> determined by the discriminative model <NUM>. The third loss value <NUM> (L<NUM>) may be determined by equation (<NUM>): <MAT>.

The processor <NUM> is further configured to train the segmentation model <NUM> by increasing, for example maximizing, the third loss value <NUM>. As described above, the predictability <NUM> of the background <NUM> of the digital image <NUM> from the segment <NUM> of the digital image <NUM> may be quantified by the mutual information (I) of the background <NUM> of the digital image <NUM> and the segment <NUM> of the digital image <NUM>. The mutual information of the background <NUM> of the digital image <NUM> and the segment <NUM> of the digital image <NUM> and thus the predictability <NUM> of the background <NUM> of the digital image <NUM> from the segment <NUM> of the digital image <NUM> may be reduced, for example minimized, by generating a combined digital image <NUM> for which the discriminative model <NUM> determines a high probability <NUM> to belong to the first class associated with "real" images. Thus, the segmentation model <NUM> may be trained by reducing, for example minimizing, the third loss value <NUM>. The discriminative model <NUM> may be trained to determine a low probability <NUM> for the combined digital image <NUM> to belong to the first class. Thus, the discriminative model <NUM> may be trained by increasing, for example maximizing, the third loss value <NUM>.

Hence, the objective function for training the segmentation model <NUM> (S) and the discriminative model <NUM> (D) may be described by equation (<NUM>): <MAT>.

Training the segmentation model <NUM> by increasing, for example maximizing, the third loss value <NUM> has the effect that the segment <NUM> of the segmentation image <NUM> generated by the segmentation model <NUM> includes sharp object boundaries. In other words, the segment <NUM> of the segmentation image <NUM> includes the pixels associated to the at least one object <NUM> in the digital image <NUM> and does not include pixels associated to the background <NUM> of the digital image <NUM>.

<FIG> shows a method <NUM> of training a segmentation model.

The method <NUM> may include the segmentation model <NUM> generating a segmentation image <NUM> for the digital image <NUM> (in <NUM>). The segmentation image <NUM> may include a segment <NUM> describing at least one object <NUM> in the digital image <NUM>. The method <NUM> may further include determining a predictability <NUM> of the digital image <NUM> from the segment <NUM> in the segmentation image <NUM> (in <NUM>). The method <NUM> may include training the segmentation model <NUM> to reduce, for example to minimize, the predictability <NUM> of the digital image <NUM> from the segment <NUM> of the segmentation image <NUM> (in <NUM>).

The method <NUM> may include generating a segment <NUM> of the digital image <NUM> using the segment <NUM> of the segmentation image <NUM> and the digital image <NUM>. The method <NUM> may include determining a predictability <NUM> of the digital image <NUM> from the segment <NUM> of the digital image <NUM>. The method <NUM> may include training the segmentation model <NUM> to reduce, for example to minimize, the predictability <NUM> of the digital image <NUM> from the segment <NUM> of the digital image <NUM>.

The method <NUM> may include generating a background <NUM> of the digital image <NUM> using the segmentation image <NUM> and the digital image <NUM>. The method <NUM> may include determining a predictability <NUM> of the background <NUM> of the digital image <NUM> from the segment <NUM> of the digital image <NUM>. The method <NUM> may include training the segmentation model <NUM> to reduce, for example to minimize, the predictability <NUM> of the background <NUM> of the digital image <NUM> from the segment <NUM> of the digital image <NUM>.

The digital image <NUM> may include a plurality of objects and the method <NUM> may be performed for each object of the plurality of objects. The digital image <NUM> may be processed by the processing system <NUM>, the processing system <NUM> and/or the processing system <NUM>, wherein the segmentation model <NUM> may generate a segmentation image <NUM> for each object of the plurality of objects in the digital image. In other words, each segmentation image <NUM> of the plurality of segmentation images generated by the segmentation model <NUM> for the plurality of objects may include a segment <NUM> describing one object of the plurality of objects in the digital image <NUM> and may further include a background <NUM> of the segment <NUM> for the associated object.

<FIG> shows a system <NUM> including a segmentation model.

The system <NUM> may include a first device <NUM>. The first device <NUM> may be a computer-controlled device like a robot, a vehicle, a domestic appliance, a power tool, a manufacturing machine, a personal assistant, an access control system etc. The first device <NUM> may be a device for conveying information like a surveillance system or a medical (imaging) system. The system <NUM> may further include a sensor <NUM>. The sensor <NUM> may be configured to detect digital images associated to the first device <NUM>. The system <NUM> may include a second device <NUM>. The second device <NUM> may be configured to process the digital images provided by the sensor <NUM>. The second device <NUM> may include a segmentation model, such as the segmentation model <NUM>. The second device <NUM> may be configured to perform the method <NUM> of training the segmentation model and may use the processing system <NUM>, the processing system <NUM> and/or the processing system <NUM>. The second device <NUM> may be configured to provide a segmentation image <NUM> for a digital image <NUM>. The system <NUM> may further include a control device <NUM>. The control device <NUM> may be configured to control the first device <NUM> using a segmentation image provided by the second device <NUM>.

According to various embodiments, the second device <NUM> includes a trained segmentation model, such as the trained segmentation model <NUM>, trained by the method <NUM> and the control device <NUM> may be configured to control the first device <NUM> using a segmentation image provided by the trained segmentation model.

The vehicle <NUM> may be a vehicle with a combustion engine, an electric vehicle, a hybrid vehicle, or a combination thereof. Further, the vehicle <NUM> may be a car, a truck, a ship, a drone, an aircraft or the like. The vehicle <NUM> includes at least one sensor <NUM>. The sensor <NUM> may be an imaging sensor, such as a camera sensor or a video sensor, wherein the sensor <NUM> is configured to provide a plurality of digital images <NUM>, such as the digital image <NUM>, wherein each digital image may include at least one object <NUM>. The vehicle <NUM> includes a driving assistance system <NUM>. The driving assistance system <NUM> may include the memory device <NUM>. The driving assistance system <NUM> may further include the at least one processor <NUM>. The processor <NUM> may implement a segmentation model. The segmentation model is configured to process the plurality of digital images <NUM> provided by the sensor <NUM> and to generate a plurality of segmentation images, such as the segmentation image <NUM>, for the plurality of digital images <NUM>, wherein each segmentation image may include a segment describing the at least one object <NUM> in the associated digital image. The segmentation model may be the trained segmentation model <NUM>.

Claim 1:
A method of computer-implemented training of a segmentation model, the method comprising:
• the segmentation model processing a digital image to generate a segmentation image, the segmentation image comprising a segment associated with at least one object shown in the digital image and comprising a background of the segment, wherein the segment of the segmentation image comprises a first plurality of pixels associated with a first plurality of pixels in the digital image, wherein the first plurality of pixels of the digital image is associated with the at least one object, and wherein the background of the segment of the segmentation image comprises a second plurality of pixels associated with a second plurality of pixels in the digital image, wherein the second plurality of pixels of the digital image is associated with a background of the at least one object;
• generating a background of the digital image using the background of the segmentation image and the digital image, wherein the background of the digital image comprises the second plurality of pixels of the digital image;
• determining a predictability of the digital image from the segment of the segmentation image, comprising
∘ determining a predictability of the background of the digital image from the digital image segment of the digital image,
∘ wherein determining the predictability of the background of the digital image from the digital image segment of the digital image comprises:
▪ generating a background of an additional digital image using the background of the segmentation image and the additional digital image, wherein the background of the additional digital image comprises a third plurality of pixels in the additional digital image is determined by the Hadamard product of the segmentation image and the additional image;
▪ generating a combined digital image using the digital image segment of the digital image and the background of the additional digital image, wherein the combined digital image comprises the first plurality of the digital image and the third plurality of pixels of the additional digital image;
▪ a discriminative model determining a probability for the combined digital image to belong to a class comprising the digital image and the additional digital image;
• training the segmentation model to increase the probability determined by the discriminative model.