Patent Description:
Machine learnable models find a wide application in many fields of technology. For example, in parts production a machine learnable model may classify a produced part as fault from a sensor reading of the part, e.g., an image taken with an image sensor. Automated quality control has the potential to greatly reduce the percentage of faulty parts produced, e.g., components and the like. An image sensor coupled to a machine learnable model can eliminate almost all parts with visible defects.

For example, in autonomous driving a machine learnable model may classify an object in the environment of the autonomous vehicle as another car, cyclist, pedestrian and so on, e.g., from a sensor reading of the environment. The sensor reading may be obtained with sensors such as an image sensor, LIDAR and so on. A controller of the vehicle will use the classification in making driving decisions. For example, the car may need to reduce speed if a pedestrian appears to be about to cross the road, while there is no need to adjust the driving for a sign waiting at the side of the road-unless the sign is classified as a traffic sign, in which case such a need may arise, and so on.

To train or test a machine learnable model, e.g., comprising a neural network, training data is needed. Such training data may be obtained by taking sensor measurement in an environment that is similar to the one expected to be encountered in practice. However, obtaining the right kind or the right amount of training data is sometimes hard. For example, there may be too little training data for the complexity of a particular machine learnable model, while obtaining additional training data is costly or even impossible. Another problem is that getting enough training data of the right kind is difficult. For example, in the case of an autonomous vehicle, such as a car, if it is currently summer, then obtaining additional training data in a winter landscape will not be possible until it is winter. Another problem is that dangerous situations, e.g., crashes and near crashes, occur only seldom and hard to enact artificially. Between <NUM> and <NUM>, a fleet of autonomous cars traveled <NUM> million miles and was involved in <NUM> crashes. (see the article <NPL>).

Although any crash is one too many, for testing and training purposes, the sensor data of <NUM> crashes is not much.

Additional training data may be generated using a generated neural network. In particular, generative adversarial networks (GANs) are a powerful tool for data synthesis, e.g., for generating natural looking images, as well as for learning feature representations. A generator neural network may be configured to generate synthesized sensor data, that looks like measured sensor data. For example, a generator neural network may be configured to generate synthesized sensor data from scratch, e.g., taking a noise vector as input and producing the synthesized sensor data as output. For example, a generator neural network may be configured transform existing measured sensor data from one domain to another, e.g., from summer to winter, or from LIDA sensor data to image sensor data, etc. A generator neural network may be configured to take an additional input, e.g., a class label, indicating to the generator neural network the type of synthesized measured, e.g., the time of year, the type of car in the synthesized sensor data, and so on. Another application is to transfer measured sensor data from one modality to another, e.g., from radar to image data or vice versa. This may make complementary data from different origins interchangeable for the use of the system.

A preferred way to train a generator neural network is together with a discriminator neural network, e.g., in a so-called generative adversarial network (GAN). A GAN comprises at least two neural networks: a generator, and a discriminator. The two networks are trained jointly through adversarial training, e.g., the two networks compete in a game, where the discriminator is optimized to distinguish original images from the training set from images generated by the generator network. Conversely, the generator is trained to make its generated images less and less distinguishable from those contained in the training dataset. In their standard form, GANs generate independent samples from their model of the training data distribution and need no label information for training or generation. For example, see the article <NPL>, et al.

The generator network may be An important subclass of GANs are conditional GANs (cGANs), which receive one or more additional inputs to both generator and discriminator networks, and can thus generate new data conditioned on user-specified information. The basic use case is to provide a class label, and then generate images of only that particular class. Recently cGANs have been successfully used to generate photo-realistic synthetic images of different classes; see, e.g., the article <NPL>, et al. cGANs have been also successfully employed for image-to-image and label-to-image translation tasks, aiming to generate realistic looking images while conditioning on dense information such as images or label maps and making use of some prior information about the scene; see, e.g., the article <NPL>, et al.

First approaches aimed to learn a direct mapping from source images to target images given paired input-output image pairs as training data. Given two domains X and Y, for each image x ∈ X there is a corresponding y ∈ Y, and the goal is to find a translator G:X→Y such that G(x)≈y. Because this group of methods employs paired data for training they can be considered as supervised learning.

Other methods also tackle the unpaired setting of image-to-image translation, where the goal is to relate two independent sets of images from two different domains and there exist no paired examples showing how an image could be translated to a corresponding image in another domain. This problem is inherently ill-posed and requires additional constraints. A popular approach is CycleGAN, which assumes cycle-consistency and the existence of an inverse mapping F that translates from Y to X, e.g., F:Y→X such that F(y)≈x. It trains two generators which are bijections and inverse to each other (G and F) and uses two discriminators (DX and DY) to ensure the translated image appears to be drawn from the target domain as well as the cycle-consistency constraint to ensure the translated image can be mapped back to the original image using the inverse mapping, e.g., F(G(x))≈x and G(F(y))≈y. See, for example the article "<NPL>, et al.

A problem with learning image mappings such as domain transfers, but also with pure image generation is that there is no guarantee that the system will learn to generate images that make sense. As a result, some generated data may correspond to a situation that is not physically possible. This is a problem, especially if the generated images are used to train or test a further system. For example, when a generated image is used to test the safety of a further device, then the test will be meaningless if the generated image does not correspond to a situation which the device can encounter in normal use.

State of the art neural networks are known to be over-parameterized and require vast amounts of data to train. One solution to this is to incorporate prior knowledge about the data distribution into the model, making the training procedure easier and the model more robust. One form of such prior knowledge relies on the concept of symmetry properties of the data, e.g., invariance/equivariance, e.g., the known transformations of the input which leave the desired output (class label, label map, image, etc.) unchanged or change the output in a specific and predictable way with respect to the input transformation.

Current approaches to exploiting data symmetries use an explicit analytic representation of the desired transformations formulated as a mathematical group and apply this to convolutional neural network filter weights. An example of such a transformation group could be the set of rotations by <NUM> degrees. These transformations would then need to be represented analytically as a parametrized set of functions which are used to generate additional transformed versions of learned convolutional filters. In this way, the network has a multiplicative increase in the number of filters used for producing output representations while keeping the number of trainable weights constant. Additionally, by capturing all transformations of the features learned by the filters, the network is now equivariant with respect to that transformation group. The analytic method is both cumbersome, as well as impractical. As soon as the type of equivariance is less well defined, or more complicated, this approach becomes infeasible. Moreover, the resulting configuration is inflexible; since the equivariance is hard-coded in the network architecture it is complicated to change it.

It would be advantageous to have an improved training method for training a generator neural network configured to generate synthesized sensor data. Advantageous embodiments are disclosed by the dependent claims.

Interestingly, transformations can be defined that transform measured sensor data. The transforms can be used within training to increase control of the kind of images that the discriminator will accept. This will improve the training signal of discriminator which in turn improves the generator.

A fidelity preserving transformation preserves the properties that the generator aims to learn. For example, in an embodiment the measured sensor data in the training data may be a collection of street-images obtained through an image sensor, and applying the fidelity preserving transformation produces images that could have been drawn from the same distribution. In other words, applying a fidelity preserving transformation to a measured sensor data should give an image that would be regarded as success if that image were generated by a trained generator neural network.

On the other hand, a fidelity destroying transformation degrades the properties that the generator aims to learn. For example, in an embodiment the measured sensor data in the training data may be a collection of street-images obtained through an image sensor, and applying the fidelity destroying transformation produces images that could not have been drawn from the same distribution. In other words, applying a fidelity destroying transformation to a measured sensor data should give an image that would be regarded as a failure if that image were generated by a trained generator neural network.

Interestingly, in an embodiment, the optimization is over the difference between the discriminator output, e.g., a distinguishing data, a discriminator value, etc. for example, if the transformation is fidelity preserving, said difference should be small, whereas if the transformation is fidelity destroying the difference should be large, e.g., in absolute value or in magnitude. In an embodiment, these two differences are combined in single loss term, e.g., by taking their ratio. An advantage of optimizing a difference instead of the desired discriminator output directly, is that even when the quality of the discriminator is not yet very good, the discriminator will already be trained on having equivariant properties.

Through the transformations, e.g., the fidelity preserving transformation or the fidelity destroying transformation, additional prior knowledge can be integrated in the trained model. For example, prior knowledge regarding data symmetry, e.g., invariance and equivariance, properties of the data can be incorporated into training of the (c)GAN models with little effort. No changing of the generator or discriminator network is needed. This leads to an improved and more robust performance of the generator. Using transformations as in an embodiment also allows a straightforward correction of problems in the model; for example, suppose after training of an image-translation network, an input is found which after rotation over a small amount, say <NUM>-degree causes a breakdown of the network, e.g., a clearly wrong translation. In that case, <NUM>-degree transformation may be added as fidelity-preserving transformations.

In an embodiment, one or both transformations are applied in one or more new losses for the discriminator and/or generator training. The losses force the generator to become equivariant under the transformation that preserves the conditional information, e.g., identity of object class, while the decision of the discriminator about the sample coming from a real or fake class should be invariant to those transformations. This prior knowledge may be provided as additional supervision for training the discriminator and generator. Thus, given a fidelity preserving transformation, a generator neural network may be obtained that is equivariant for that transformation, without having to change the architecture of the generator neural network. Thus, given a fidelity destroying transformation, a discriminator can be obtained that is forced to learn that particular types of sensor data are not acceptable, which in turn leads to a better training signal for the generator.

The measured sensor data may be obtained from one or more sensors, some of which may be spatial sensors, e.g., image sensors LIDAR and so. For example, the measured sensor data may comprise a measured image obtained from an image sensor; and the synthesized sensor data may comprise a synthesized image. Typically, the synthesized sensor data and the measured sensor data have the same resolution. For example, the sensor data may include sensor signals from one or more sensors such as, e.g., video, radar, LiDAR, ultrasonic, motion, imaging camera, and so on. A sensor data may comprise sensor signals from multiple sensor modalities, e.g., image and radar sensor signals. The measured sensor data may comprise audio data. The measured and synthesized sensor data may comprise a plurality of values indicating a plurality of sensor values. For example, pixels in image like data, or samples, in audio like data, and so on.

Training the discriminator and/or generator network may use otherwise conventional training techniques such a backpropagation, e.g., using techniques used in GAN training, e.g., using a system such as ADAM. The GAN may be a CycleGAN, but this is not necessary.

Examples of fidelity-destroying transformations include: a blurring operating, perturbation of pixels, a horizontal mirror operation, a spline operation, a color-change, a partial overwriting, a rotation over a threshold, etc..

For example, a horizontal mirror operation when executed in a street-scene image removes the image from the domain of images that could have been obtained from the image sensor. In other words, a generator neural network that produces an upside-down street scene is a failure. The same holds for large rotations, e.g., rotations over a threshold, e.g., rotations between <NUM> and <NUM> degrees. Consider a rotation of <NUM> degrees of a street scene. In such a case, the sky would be located on the side, and cars would drive in a physically impossible direction. Yet on a small local scale, the image might look perfectly realistic. Thus, the fidelity destroying operation encourages the discriminator to take global aspects of the image into account.

A further example of fidelity-destroying operations is CutMix operations, in which a composed sensor data is created from two other sensor data, e.g., measured or synthesized sensor data. For example, a street scene of two different streets combined into a single image is not globally realistic even if locally it looks valid.

Examples of fidelity-preserving transformations include one or more of: an affine transformation, a rotation, a translation, a vertical mirror operation. For example, a street scene that is mirror over a vertical axis remains a valid street scene; a generator neural network that produces a synthesized sensor data which is identical to a real measured sensor data except for this operation would perform well. For example, a small rotation, or a small affine transformation, e.g., below a threshold. For example, a rotation of less than <NUM> degrees may be considered fidelity-preserving. The effect of introducing this would cause the discriminator to allow such small rotations, and as a consequence this will be reflected in the training signal to the generator.

The discriminator may be configured to generate a discriminator value indicating at least if a discriminator input is a synthesized sensor data or a measured sensor data. In an embodiment, the discriminator values obtained for one or more transformation is included in one or more additional loss terms.

For example, the discriminator network may be applied to a measured sensor data, e.g., a real image. The fidelity-destroying transformation may be applied to the measured sensor data, and the discriminator network may be applied to the result. If the discriminator network is well-trained the first ought to give an indication of a real image, say, a <NUM> value, while the latter ought to give an indication of a fake image, say, a <NUM> value. Thus, by maximizing this difference the discriminator network is made aware of this class of non-real images. This in turn will impact the generator network due to a better training signal. The maximizing may be for the magnitude, e.g., the absolute value.

The fidelity-preserving transformation may be applied to the measured sensor data, and the discriminator network may be applied to the result. In this case, the real and transformed image should give the same discriminator output, e.g., that both are real images. Thus, the difference between discriminator values may be minimized.

Good results have been obtained by combining both types of optimizing in a single loss term. For example, one could use as an additional loss-term the following: <MAT> wherein.

The parameters α, β and ε may be, e.g., <NUM>. Note that this single term causes (α∥D(T(x))- D(x)∥+<NUM>) to be maximized and (β∥D(F(x))- D(x)∥ + ε) to be minimized. However, in this way the forces caused by the fidelity-destroying transformation versus the fidelity-preserving transformation can be balanced against each other.

For example, the discriminator network may be applied to a synthesized sensor data, e.g., a fake image. The fidelity-preserving transformation may be applied to the synthesized sensor data, and the discriminator network may be applied to the result. In this case, the original fake image and the transformed fake image should give the same discriminator output, e.g., that both are fake images. Thus, the difference between discriminator values may be minimized.

For example, one may use as a loss term: <MAT> wherein.

The parameter ε may be taken as <NUM>. Note that the above term is large whenever (∥D(F(G(z)))- D(G(z))∥ + ε) is small. In these examples ∥. ∥ denotes a norm, e.g., an L2 norm or Euclidian norm. Other options are possible.

In an embodiment, the generator network may be configured for a domain translation task. For example, the generator network may be configured to receive measured sensor data from a first domain as input and wherein the generator network is trained to generate synthesized sensor data from a second domain. This can be used for many purposes. For example,.

For example, to test a machine learnable model on hard to obtain test data, e.g., sensor data corresponding with dangerous situations, e.g., crashes and near crashes, the generator may be applied to an example of the test data, and transfer it to a different domain, e.g., a slightly different domain. For example, types of cars may be changed, time of day or time of year may be changed, etc. Thus, measured sensor data obtained during a near collision, say around noon in spring, may be converted to synthesized sensor data corresponding to an evening in fall, yet still show a near collision. Using the synthesized sensor data, the machine learnable model may be tested for a wider range of near-collisions, this improving the safety of the autonomous Apparatus in which the machine learnable model is used.

Another way in which a trained generator network may be used, is to generate synthesized sensor data for which a given controller would take a rare action. For example, trained generator network may be used to generate a large set of synthesized sensor data, e.g., from a corresponding set of noise vectors and apply a controller of an autonomous apparatus to the set, e.g., to obtain a simulated control signal. A set of control signals may be selected therefrom according to a criterion, e.g., cases that indicate a dangerous situation, e.g., control signals that correspond to sharp breaking or sharp steering. Next the synthesized sensor data that correspond to the selected simulated control signals may be obtained and analyzed. In this way, one can quickly obtain synthesized sensor data for which the controller would take a particular action, e.g., a non-average action. For example, one may evaluate if the action taken is appropriate given the synthesized sensor data. Using measured sensor data, it is difficult to obtain test cases in which the controller takes a rare action, e.g., an emergency brake.

In an embodiment, the training set comprises ground-truth class-labels for the measured sensor data. A class-label may be provided as an additional input to the discriminator network. For example, the discriminator network may be a conditional network receiving the class label as input. Typically, a class label is also be provided as an additional input to the generator network, e.g., to indicate to the generator network to generate synthesized sensor data according to the class label. The latter is not necessary though, for example, a conditional discriminator network may be combined with multiple unconditional generator networks.

The class label indicates a class of the discriminator input data, the discriminator neural network being optimized to distinguish if the discriminator input data corresponds to the class. For example, the discriminator network may be trained to distinguish between measured sensor data with the correct class label on the one hand and synthesized sensors data or measured sensor data with an incorrect class label on the other hand.

The domain translation and data synthesis tasks can be performed between any sensor signals. However, as an example we focus mostly on images as the main application. The proposed framework can be also used for data augmentation as well as domain transfer tasks, e.g., from real samples to synthetic samples. The generated samples can be then used for training any data-driven method.

A class label may also be used for a generator network configured for a domain translation task. For example, in an embodiment, a class-label may indicate a transformation goal to the generator network. There may be a plurality of transformation goals, e.g., corresponding to a plurality of domains. The training data may be labeled with a domain of the plurality of domains. The generator network may be configured to transform measured sensor data to a domain according to the transformation goal. The discriminator network may be configured to determine if the input sensor data satisfies the domain according to the transformation goal.

In an embodiment, a transformation goal may comprise a time difference; the training data may be labeled with a timestamp. The generator network may be configured to transform measured sensor data from a first timestamp to a second timestamp according to the time difference. The discriminator network may be configured to receive as input a first sensor data, a second sensor data and a time difference and to determine if the first sensor data and the second sensor data satisfy the time difference. Any one of the first and second sensor data may be synthesized data in which case, the discriminator network may be trained to reject the images. Measured sensor data labeled with a timestamp is relatively easy to obtain, e.g., by testing a controller and saving measured sensor data with a timestamp. By artificially changing the time of a particular measured sensor data, one can obtain a large amount of synthesized sensor data from a given measured sensor data of interest. For example, multiple time differences may be applied to the given measured sensor data.

An interesting application of sensor data translation is occlusion and desocclusion. The class label, e.g., a transformation goal may indicate an object which is to be occluded, e.g., moved behind another object, or to be desoccluded, e.g., moved in front of another object. For example, a pedestrian may be moved in front of or behind of a tree; a cyclist in front or behind a car, and so on. The discriminator network may be trained to verify if the object is indeed occluded or desoccluded. The class label in this case may be a map indicating the object to be occluded/ desoccluded. In an embodiment, generator network and discriminator network receive data indicating an object in the sensor data, and indication if the object is to be occluded or desoccluded.

In an embodiment, the generator network and/or the discriminator network comprise one or more neurons receiving at least part of the sensor data and optionally at least part of the class label, e.g., transformation goal. For example, the generator network and/or the discriminator network may be arranged to receive multiple channels as input, at least one of the channels encoding the sensor data; optionally at least one of the channels may encode for a class label or transformation goal. For example, the generator network and/or the discriminator network may comprise multiple layers.

The method of training a generator network may be used in a method to generate further training data for a machine learnable model. For example, the machine learnable model may be a classifier. For example, the machine learnable model may be configured to receive measured sensor data as input and to generate a classification of the measured sensor data as output. For example, the measured sensor data may be an image taken with an image sensor. For example, image may be of a produced part and the classification may be if the part is defective. For example, the measured sensor data may be an image taken in an environment of an autonomous apparatus and the classification may indicate if there is a dangerous situation. The machine learnable model may also be a neural network, but this is not necessary. The machine learnable model may use other techniques, e.g., SVM, random forests, and so on. The further training data may be used for training but may also or instead be used for testing.

For example, a method may comprise obtaining an initial training set for the machine learnable model, the initial training set comprising measured sensor data obtained from a sensor, and training a generator network from the initial training set using an embodiment of the training method. The trained generator network may be applied to generate the further training data. The further training data may then be used for training and/or testing the machine learnable model at least on the further training data.

A further aspect concerns a training system for training a generator neural network configured to generate synthesized sensor data. A further aspect concerns a generator system for a generator neural network arranged to generate synthesized sensor data. A further aspect concerns an autonomous apparatus, e.g., an autonomous vehicle. For example, the autonomous apparatus may be a computer-controlled machine, such as a robot, a vehicle, a domestic appliance, a power tool, a manufacturing machine.

Embodiments of the methods and/or systems may be performed on one or more electronic devices. For example, the electronic device, may be a computer.

An embodiment of the methods may be implemented on a computer as a computer implemented method, or in dedicated hardware, or in a combination of both. Executable code for an embodiment of the method may be stored on a computer program product. Examples of computer program products include memory devices, optical storage devices, integrated circuits, servers, online software, etc. Preferably, the computer program product comprises non-transitory program code stored on a computer readable medium for performing an embodiment of the method when said program product is executed on a computer.

In an embodiment, the computer program comprises computer program code adapted to perform all or part of the steps of an embodiment of the method when the computer program is run on a computer. Preferably, the computer program is embodied on a computer readable medium.

Another aspect of the presently disclosed subject matter is a method of making the computer program available for downloading.

Further details, aspects, and embodiments will be described, by way of example only, with reference to the drawings. In the drawings,.

While the presently disclosed subject matter is susceptible of embodiment in many different forms, there are shown in the drawings and will herein be described in detail one or more specific embodiments, with the understanding that the present disclosure is to be considered as exemplary of the principles of the presently disclosed subject matter and not intended to limit it to the specific embodiments shown and described.

Further, the subject matter that is presently disclosed is not limited to the embodiments only, but also includes every other combination of features described herein or recited in mutually different dependent claims.

<FIG> schematically shows an example of an embodiment of a generator neural network <NUM> and of a discriminator neural network <NUM>. Generator neural network <NUM> and discriminator neural network <NUM> are trained together as a GAN <NUM>.

Generator network <NUM> may be configured to generate synthesized sensor data <NUM>. The generator network may be optimized so that generated synthesized sensor data <NUM> is indistinguishable from measured sensor data by discriminator network <NUM>. For example, that as far as discriminator network <NUM> can distinguish synthesized sensor data <NUM> appears as if it was drawn from measured sensor data.

Discriminator neural network <NUM> is optimized to distinguish between measured sensor data and synthesized sensor data. Discriminator neural network <NUM> may be configured to receive a discriminator neural network input <NUM>. For example, discriminator neural network input <NUM> may comprise measured sensor data, e.g., sensor data obtained from a sensor. In this case, discriminator neural network <NUM> may distinguish the discriminator neural network input <NUM> as measured sensor data. For example, discriminator neural network input <NUM> may comprise synthesized sensor data, e.g., synthesized sensor data <NUM>. In this case, discriminator neural network <NUM> may distinguish the discriminator neural network input <NUM> as synthesized.

G aims to map a latent variable z ~ p(z) sampled from a prior distribution to a realistic-looking image, e.g., like measured sensor data, while D aims to distinguish between a real x and generated G(z) images, e.g., between measured sensor data and synthesized sensor data. In an embodiment, G and D may be modeled as a decoder and an encoder convolutional network, respectively. For example, the discriminator may output a value between <NUM> and <NUM> to indicate how real the input seems.

Generator neural network <NUM> is configured to receive a generator neural network input <NUM> and to produce synthesized sensor data <NUM>. The generator neural network input <NUM> may comprise a random element, e.g., noise, e.g., a noise vector, which may be used for generation of new synthesized sensor data.

Generator neural network input <NUM> may comprise measured sensor data; generator neural network <NUM> may be configured for a translation task, e.g., domain translation. For example, generator neural network input <NUM> may be configured to generate synthesized sensor data like generator neural network input <NUM> but in a different domain. For a domain translation task some kind of cycle regularization may be used during training. Cycle regularization may be obtained by using multiple generator networks, or by configuring the generator neural network as a conditional neural network, e.g., wherein a class label indicates the desired domain transfer. A CycleGAN is not necessary though.

Generator neural network <NUM> may be configured to receive an additional input: class label <NUM>, but this is optional. Class label <NUM> indicates a desired property of synthesized sensor data <NUM>, e.g., a desired domain. There may be more than one class label. Input <NUM> is optional.

Discriminator network <NUM> may be configured to receive an additional input: a class label <NUM>. In case a class label <NUM> is used, the discriminator network may additionally verify if the discriminator input <NUM> is according to the class label. For example, discriminator network <NUM> may only output an 'is-real' output, e.g., a discriminator value of1, in case the discriminator input <NUM> is both measured sensor data and according to the class label.

Discriminator network <NUM> may be configured to produce a discriminator value <NUM>. For example, the discriminator value <NUM> may indicate if the discriminator input <NUM> appears to be measured sensor data, e.g., output <NUM>, or synthesized sensor data, e.g., output <NUM>. A value in between may indicate an uncertainty about the image.

<FIG> schematically shows an example of an embodiment of a generator neural network. In this embodiment, the generator neural network receives measured sensor data as part of the input <NUM>; this is not necessary, e.g., noise may be used instead or in addition. <FIG> also show a class label <NUM> as input; this input is optional.

The generator network of <FIG> comprises three parts: an encoder part <NUM>, a processing part <NUM> and a decoder part <NUM>.

Encoder part <NUM> is configured to receive the input sensor data <NUM>. Encoder part <NUM> may be configured with a so-called bottleneck at its output. Processor part <NUM> receives the output of the encoder part <NUM>, decoder part <NUM> may receive the output of the processing part. The optional class-label <NUM>, may comprise a transformation goal which is to be applied to one or more parts of the input. As shown in <FIG>, the class-label <NUM> is provided as input to the encoder part and as an input to the processing part. Although not shown in <FIG>, it was found to be particularly advantageous to supply the transformation goal as an input to the decoder part <NUM> as well.

In an embodiment, the class-label could be an input to the decoder part <NUM>. In an embodiment, the class-label could be an input to the decoder part <NUM> and to the encoder part <NUM>.

In an embodiment, encoder part <NUM> comprises multiple convolution layers, processor part <NUM> comprises multiple residual layers and the decoder part comprises multiple convolution layers. Various known types of layers may be added. For example, in an embodiment, encoder part <NUM> comprises <NUM> convolution layers, processor part <NUM> comprises <NUM> residual layers and the decoder part comprises <NUM> convolution layers. The network may be larger or smaller as desired or may even be much larger.

Discriminator neural network <NUM> may use a conventional architecture, e.g., an encoder network or down sampling network. The architecture of discriminator neural network <NUM> may be the same as encoder <NUM> except that output discriminator <NUM> needs only to be one bit. In an embodiment, the output discriminator <NUM> could be more detailed than one bit.

<FIG> schematically shows an example of an embodiment of a discriminator neural network.

Training of GAN <NUM> may use conventional GAN training methods with the addition of additional training data and/or additional loss terms in the optimization. <FIG> shows a measured sensor data <NUM>. The measured sensor data <NUM> may be provided to discriminator <NUM> as input <NUM> to obtain a discriminator output, e.g., a value <NUM>. For example, discriminator <NUM> may be optimized so that the corresponding discriminator <NUM> output <NUM> indicates measured sensor data, e.g., an output of <NUM>.

Additional training data may be obtained by applying a transformation to measured sensor data <NUM>.

<FIG> shows a fidelity-preserving transformation <NUM>. Fidelity-preserving transformation <NUM> may be applied to measured sensor data <NUM>. The resulting sensor data should not be distinguishable from real measured sensor data. When discriminator <NUM> is applied to the transformed measured sensor data <NUM> as input <NUM>, a value <NUM> is obtained. Thus, one may optimize discriminator <NUM> that the corresponding output <NUM> indicates measured training data, e.g., also a value of <NUM>. Instead of directly optimizing the latter on could instead optimize discriminator <NUM> to minimize the difference between the two values <NUM> and <NUM>. This has the advantage that from early iterations, the discriminator is trained to produce equivariant outputs-even if the discriminator values themselves are not very accurate yet.

Fidelity-destroying transformation <NUM> may be applied to measured sensor data <NUM>. The resulting sensor data should be distinguished from real measured sensor data. When discriminator <NUM> is applied to the transformed measured sensor data as input <NUM>, a value <NUM> is obtained. Thus, one may optimize discriminator <NUM> that the corresponding output <NUM> indicates not measured training data, e.g., a value of <NUM>, e.g., the same value as for synthesized sensor data. Instead of directly optimizing the latter on could instead optimize discriminator <NUM> to maximize the difference between the two values <NUM> and <NUM>.

The two loss terms may be combined so that a single loss term can be optimized, e.g., as a ratio. This has the effect of balancing the above two loss terms.

Training of GAN <NUM> may use conventional GAN training methods with the addition of additional training data and/or additional loss terms in the optimization. <FIG> shows generator neural network <NUM>, the output of which may be provided to discriminator network <NUM> as input <NUM> to obtain output <NUM>. For example, discriminator <NUM> may be optimized so that the corresponding discriminator <NUM> output <NUM> indicates synthesized sensor data, e.g., an output of <NUM>.

Additional training data may be obtained by applying a transformation to the synthesized sensor data. <FIG> shows the fidelity-preserving transformation <NUM>. Fidelity-preserving transformation <NUM> may be applied to the synthesized sensor data. The resulting sensor data should still be distinguished from measured sensor data. When discriminator <NUM> is applied to the transformed synthesized sensor data as input <NUM>, a value <NUM> is obtained. Thus, one may optimize discriminator <NUM> so that the corresponding output <NUM> indicates synthesized training data, e.g., also a value of <NUM>. Instead of directly optimizing the latter one could instead optimize discriminator <NUM> to minimize the difference between the two values <NUM> and <NUM>.

<FIG> schematically shows an example of an embodiment of a generator neural network. Shown in <FIG> is sensor data <NUM>, typically measured sensor data. For example, sensor data <NUM> may comprise an image. In the embodiment, generator neural network <NUM> is configured to take sensor data as input and to produce synthesized sensor data as output. For example, generator neural network <NUM> may be configured for an image translation task, e.g., a domain transfer. Generator neural network <NUM> may be configured to receive a class label as additional input, e.g., a transformation goal, but this is optional. Also shown in <FIG> is an equivariant transformation <NUM>. For example, equivariant transformation <NUM> may be a fidelity preserving transformation. For example, equivariant transformation <NUM> may be a small rotation, e.g., a rotation angle below a threshold, or a mirroring over a vertical axis, etc. Shown in <FIG> in the top row is that generator neural network <NUM> is first applied to sensor data <NUM> followed by equivariant transformation <NUM>. Shown in <FIG> in the bottom row is that equivariant transformation <NUM> is first applied to sensor data <NUM> followed by generator neural network <NUM>. The equivariant transformation <NUM> is chosen so that the two rows should give the same result. The two images are compared in difference operation <NUM>. For example, they may be compared pixel by pixel and the differences may be summed. For example, this may be a sum of squared differences. Generator network <NUM> may be optimized to minimize the difference. And advantage of this operation is that the generator network can be optimized directly, without having to go through the discriminator network. Moreover, from an early stage in training the generator network can be trained to respect the equivariance. Note that the type of equivariancy that is obtained in this way may be difficult or impossible to obtain by adapting the architecture of the generator neural network.

Generally, e.g., in <FIG>, there may be multiple fidelity preserving, fidelity destroying and equivariant transformations. For example, if the transformation comprises a rotation, the rotation may be selected randomly, e.g., selected randomly from a range. A transformation may be used for a single loss term, but if strong equivariance learning is desired, multiple loss terms may be introduced, e.g., using multiple transformations of the various types.

<FIG> schematically shows an example of an embodiment of a training system <NUM>. Training system <NUM> is configured for training a generator neural network arranged to transform measured sensor data into synthesized sensor data. For example, system <NUM> may comprise a generator unit <NUM> configured for applying the generator neural network, and a discriminator unit <NUM> configured for applying a discriminator neural network. For example, generator unit <NUM> and/or discriminator unit <NUM> may comprise storage for storing parameters of the respective neural networks. For example, generator unit <NUM> and/or discriminator unit <NUM> may be configured to receive network inputs, apply the inputs and the parameters according to the neural network type and to provide the network result on an output.

System <NUM> comprises an optimizer <NUM>. Optimizer <NUM> is configured to train the generator network together with the discriminator neural network. The generator network is optimized to generated synthesized sensor data, and the discriminator network is optimized to distinguish between measured sensor data and synthesized sensor data. In order to train the two neural networks, optimizer <NUM> has access to a training set, e.g., as stored in a training set storage <NUM>. The training set comprises measured sensor data. Sensor data may be image data, e.g., images, but may comprise instead or in addition a wide variety of data, e.g., radar data, ultrasonic sensor data, etc. In an embodiment, sensor data may be obtained from a sensor configured to produce two-dimensional data characterizing an environment of the sensor. The sensor may be employed in a machine. In an embodiment, at least part or all of the sensor measurements have domain information and/or sensor time information indicating the domain in which the condition, e.g., the environment or environment type, and/or the time when the sensor data was obtained.

Optimizer <NUM> may apply one or more transformation to the measured sensor data and/or to synthesized sensor data to augment the training set. In this way, it may be enforced that the resulting neural networks have desired equivariant properties.

A sensor data may be a multiple of conjoint sensor data, possibly of different sensor modalities. For example, in the example of autonomous vehicles one sensor data item may comprise, one or more of image, radar, and other sensor data, typically concurrent data recorded from multiple sensors. For example, system <NUM> may comprise a communication interface for accessing the training set. Sensor data may be measured, e.g., as received from a sensor, e.g., real, or true; or sensor data may be generated, e.g., as generated by a generator unit, e.g., fake.

Once the generator network is sufficiently trained, e.g., after convergence or after exhausting the training data, or after a preset number of training iterations, the generator network may be used in an application, typically without the corresponding discriminator network. For example, <FIG> schematically shows an example of an embodiment of a generator system <NUM>. Generator system <NUM> is configured to apply a generator neural network, such as the generator neural network trained by system <NUM>, e.g., the generator neural network of generator unit <NUM>. Generator system <NUM> is thus arranged to generate synthesized sensor data. System <NUM> may comprise an input unit <NUM>. Input unit <NUM> may be configured for receiving as input measured sensor data, e.g., in case of a domain transferring task. Input unit <NUM> may be configured for receiving a noise component, e.g., in case of a generating task. Input unit <NUM> may be configured for both noise and sensor data as well. Input unit <NUM> might also be used to receive sensor data that was not measured but generated. After generating the synthesized sensor data, the generated output sensor data may be put on output <NUM>, e.g., transmitted. For example, system <NUM> may comprise a communication interface for receiving and/or transmitting the sensor data.

System <NUM> comprises a generator system <NUM> configured to apply the trained generator network to the received input measured sensor data. Typically, system <NUM> is configured to perform further tasks. For example, system <NUM> may be configured to augment further training data for a further neural network, e.g., for a classifier. System <NUM> and system <NUM> may be the same system, or they may not be. Systems <NUM> and/or <NUM> may be a single device or may comprise multiple devices.

Systems <NUM> and/or <NUM> may communicate with each other or with external storage or input devices or output devices over a computer network. The computer network may be an internet, an intranet, a LAN, a WLAN, etc. The computer network may be the Internet. The systems comprise a connection interface which is arranged to communicate within the system or outside of the system as needed. For example, the connection interface may comprise a connector, e.g., a wired connector, e.g., an Ethernet connector, an optical connector, etc., or a wireless connector, e.g., an antenna, e.g., a Wi-Fi, <NUM> or <NUM> antenna, etc..

The execution of system <NUM> and <NUM> is implemented in a processor system, e.g., one or more processor circuits, examples of which are shown herein. <FIG>, <FIG>, <FIG> show functional units that may be functional units of the processor system. For example, <FIG> may be used as a blueprint of a possible functional organization of the processor system. The processor circuit(s) are not shown separate from the units in these figures For example, the functional units shown in <FIG> may be wholly or partially implemented in computer instructions that are stored at system <NUM> and <NUM>, e.g., in an electronic memory of system <NUM> and <NUM>, and are executable by a microprocessor of system <NUM> and <NUM>. In hybrid embodiments, functional units are implemented partially in hardware, e.g., as coprocessors, e.g., neural network coprocessors, and partially in software stored and executed on system <NUM> and <NUM>. Parameters of the network and/or training data may be stored locally, e.g., at system <NUM> and <NUM>, or may be stored in cloud storage.

<FIG> schematically shows an example of an embodiment of a training system <NUM>. Training system <NUM> may comprise a processor system <NUM>, a memory <NUM>, and a communication interface <NUM>. For example, the execution of system <NUM> may be implemented in a processor system, e.g., one or more processor circuits, e.g., microprocessors, examples of which are shown herein. Parameters of the networks and/or training data may be stored locally at system <NUM> or may be stored in cloud storage.

<FIG> schematically shows an example of an embodiment of a training method <NUM>. Method <NUM> is a training method for training a generator neural network configured to generate synthesized sensor data. Method <NUM> comprises.

In the various embodiments of system <NUM>, <NUM> and <NUM>, one or more communication interfaces may be selected from various alternatives. For example, the interface may be a network interface to a local or wide area network, e.g., the Internet, a storage interface to an internal or external data storage, a keyboard, an application interface (API), etc..

The systems <NUM>, <NUM> and <NUM> may have a user interface, which may include well-known elements such as one or more buttons, a keyboard, display, touch screen, etc. The user interface may be arranged for accommodating user interaction for configuring the systems, training the networks on a training set, or applying the system to new sensor data, etc..

Storage may be implemented as an electronic memory, say a flash memory, or magnetic memory, say hard disk or the like. Storage may comprise multiple discrete memories together making up the storage, e.g., storage <NUM>, <NUM>, etc. Storage may comprise a temporary memory, say a RAM. The storage may be cloud storage.

Systems <NUM>, <NUM> or <NUM> may be implemented in a single device. Typically, the systems <NUM>, <NUM> and <NUM> each comprise a microprocessor which executes appropriate software stored at the system; for example, that software may have been downloaded and/or stored in a corresponding memory, e.g., a volatile memory such as RAM or a non-volatile memory such as Flash. Alternatively, the systems may, in whole or in part, be implemented in programmable logic, e.g., as field-programmable gate array (FPGA). The systems may be implemented, in whole or in part, as a so-called application-specific integrated circuit (ASIC), e.g., an integrated circuit (IC) customized for their particular use. For example, the circuits may be implemented in CMOS, e.g., using a hardware description language such as Verilog, VHDL, etc. In particular, systems <NUM>, <NUM> and <NUM> may comprise circuits for the evaluation of neural networks.

A processor circuit may be implemented in a distributed fashion, e.g., as multiple sub-processor circuits. A storage may be distributed over multiple distributed sub-storages. Part or all of the memory may be an electronic memory, magnetic memory, etc. For example, the storage may have volatile and a non-volatile part. Part of the storage may be read-only.

<FIG> schematically shows an example of an embodiment of a training system. Shown in <FIG> is a measured sensor data <NUM>, in this case an image of a person, but data <NUM> could be other sensor data, e.g., a street scene, a manufactured product, etc., the sensor modality may be different as well.

A fidelity-preserving transformation <NUM>, F may be applied to measured sensor data <NUM>. In this case, a vertical mirroring, e.g., a mirror operation over the vertical axis. It is the intention that the discriminator learns that such an operation should not change it classification. The result is a fidelity-preserved transformed measured sensor data <NUM>. Likewise, a fidelity-destroying transformation <NUM>, T may be applied to measured sensor data <NUM>. In this case a horizontal mirroring operation is applied. The discriminator may thus be trained that this operation will remove the data, e.g., data <NUM> from its domain.

Also shown in <FIG> is a generator <NUM>. The generator receives as input at least input <NUM>. Input <NUM> may comprise a noise vector, but may also or instead comprise sensor data, in particular measured sensor data. The result is synthesized sensor data <NUM>. The latter may be used to train the discriminator that such data is not real, e.g., fake. The fidelity-preserving transformation <NUM> may be applied to the synthesized sensor data <NUM>. The latter may be used to train that discriminator that data <NUM> and <NUM> should receive the same classification.

The discriminator <NUM> may be applied to all of the data, <NUM>, <NUM>, <NUM>, <NUM> and <NUM>. For example, the discriminator <NUM> may be taught that <NUM> is real, and <NUM> is fake. The discriminator may be taught that the difference in classification between <NUM> and <NUM> should be small, between <NUM> and <NUM> should be large, and between <NUM> and <NUM> should be small. These differences could be optimized for directly, they can also be combined in one or more loss terms.

<FIG> may be applied to a GAN or to a cGAN, for example, image <NUM> may be provided in a training set together with a class label. The same class label may be used for images <NUM> and <NUM>. Input <NUM> may comprise a class label, which may be used for images <NUM> and <NUM>. Discriminator <NUM> may receive an image of <NUM>-<NUM> together with the corresponding class label. The discriminator may in addition to judging the realness of an image also judge the appropriateness of the class label. Otherwise the same loss functions may be used.

In case the generator <NUM> is trained for a domain translation task, e.g., if input <NUM> comprises measured sensor data, e.g., an image, then equivariancy may also be directly trained for the generator. For example, given an equivariant function R, e.g., a fidelity-preserving function, one may require that G(R(x)) is close to R(G(x)), e.g., by averaging squared pixel differences. This provides an additional loss term that can be directly applied to G.

Below several further optional refinements, details, and embodiments are illustrated. Below it is often assumed that the measured and synthesized sensor data comprise an image. This is not necessary however, instead of an image sensor another type of sensor may have been used to obtain the measured sensor data; such different sensor data may also be used in addition to image sensor data.

The domain translation and data generation tasks with additional supervision as in an embodiment for generator and/or discriminator can be performed in principle between any sensor signals. It is noted that the embodiments below can be adapted to different sensor modalities, e.g. radar sensor data instead of images.

Embodiment can be used for data augmentation as well as domain transfer tasks. Generated samples can be used for training any data-driven methods.

In an embodiment, GAN and cGAN models are trained with additional supervision, such as the proposed data symmetry loss formulations for the generator and discriminator.

In an embodiment, a GAN comprises at least two neural networks, respectively modelling the discriminator and the generator. The generator G may take z as the input and may synthesize x'. The discriminator D may take the input x and/or x' as input to classify it into real and fake samples.

In an embodiment, a cGAN comprises at least two neural networks, respectively modelling the discriminator and the generator. The generator G may take (y, z) as the input and may synthesize x'. The discriminator D may take the input (x,y) and/or (x',y) to classify it into real and fake samples. For example, in this case, z may comprise a noise vector, x may comprise real measured sensor data, and x' fake synthesized sensor data. The input y may comprise additional information such as a class label, an image from another domain or label map.

Conventionaly, the following objectives may be used:.

A transformation F is defined which doesn't alter the domain (sample identity) of the data x ∈ X, F(x) ∈ X. For example, if x is an image, then F can be an affine transformation applied to x. The discriminator decision should not be altered by the transformation F applied on x: D(x) = D(F(x)), e.g., D's decision should be invariant to the transformation F. The transformation may be referred to as 'fidelity preserving' or 'domain preserving' or 'distribution preserving' because by applying the transformation F, the resulting image still appears real, still seems to belong to the domain of the original images, and can still be regarded as drawn from the same probability distribution.

A transformation T is defined which does alter the domain of the data x ∈ X, T(x) ∉ X. For example, if x is an image, then T can be a severe blurring or perturbation of pixels applied to x. In this case, the discriminator decision should be altered by transformation T on x: D(x) ≠ D(T(x)). The transformation may be referred to as 'fidelity destroying' or 'domain destroying' or 'distribution destroying' because by applying the transformation T, the resulting image no longer appears real, no longer seems to belong to the domain of the original images, and cannot be regarded as drawn from the same probability distribution.

The functions F and T provide additional prior knowledge to the training without having to change the architecture of the neural network. For example, the preserving transformation can be used to train the network that particular variations are within the range of the expected distribution of the images. The destroying transformation can be used to train the network that certain changes are not expected.

Thus, in an embodiment these data symmetry properties may be used to add additional supervision to the discriminator:
For example, on may define a loss function term: LD(x)_data_symmetry_prior(D,F,T) = Ex~pdata(x)[ (α∥D(T(x))- D(x)∥+<NUM>)/(β∥D(F(x))- D(x)ll + ε)], where ∥∥ can be any Lp norm (e.g. L<NUM>) or any distance measure, and α, β and ε are tunable hyperparameters.

Note that the value ∥D(T(x))- D(x)ll will go to <NUM> if the discriminator perfectly classifies the real image x, and the destroyed image T(x); For instance, one may have D(x)=<NUM> and D(T(x))= <NUM> for a real image x. Note that the value ∥D(F(x))- D(x)∥ will go to <NUM> if the discriminator perfectly classifies the real image x and the preserved image F(x). For instance, one may have D(x)=<NUM> and D(F(x))= <NUM> for a real image x. In both cases, even if the value produced by discriminator D is not correct, e.g., <NUM> or <NUM> for a real or fake image, then the loss terms still instill equivariant properties in the discriminator.

The aim of the generator is to generate a data sample x'=G(z), z~P(z), in such way that the discriminator would classify it as a real data sample. The aim of the discriminator is to classify x' as a synthetic sample. Under the fidelity-preserving transformation F, these properties of the generated samples should be preserved as well. Accordingly, one may further include a loss term like: LD(G)_data_symmetry_prior(D,G,F) = Ez~p(z)[ <NUM>/(∥D(F(G(z)))- D(G(z))∥ + ε)]. Note that the term ∥D(F(G(z)))- D(G(z))∥ will tend towards <NUM> for a well-trained discriminator and/or generator, and so <NUM>-over that term will tend to become large in that circumstance.

The new discriminator objective may be: maxD [LGAN(G,D) + λ<NUM>LD(x)_data_symmetry_prior(D,F,T) + λ<NUM>LD(G)_data_symmetry_prior(D,G,F)], where λ<NUM>, and λ<NUM> control the relative importance of the two objectives. For example, one may choose λ<NUM>, and λ<NUM> as <NUM>.

In case of the conditional GAN case, particularly for image to image and label map to image translations the same data symmetry properties can be used to train the generator. For example, a generator objective may be to map y to x, G:{z,y}→x. For example, in particular the input y may comprise an image from another domain. In an embodiment, y may also or instead comprise a label map.

Consider a transformation R to be such that the transformation function does not alter the domain of y ∈ Y, R(y) ∈ Y, e.g., R can be a rotation operation. Thus the transformation applied on y, R(y), should be preserved in the generated sample x'=G(R(y),z) as well. Using the equivariance property under the transformation R: G(R(y),z)=R(G(y,z)), we can formulate a new loss for the generator:
LG_data_symmetry_prior(G,R) = Ez~p(z), y~pdata(y)[∥G(R(y),z)- R(G(y,z))∥], where ∥ ∥ can be any Lp norm (e.g. L<NUM>) or any distance measure. Note that the difference in G(R(y),z)-R(G(y,z)) is computed in the sensor data space, e.g., on images. The rotation may be limited to small rotations, e.g., below a threshold. For example, first performing a vertical mirroring operation and then performing the mapping of G should be the same as a vertical mirroring operation after first performing the mapping of G.

Note that if G has been trained to be equivariant, than G(R(y),z)- R(G(y,z) should be small. A new generator objective may be: <MAT> where λ<NUM>,<NUM>,<NUM> control the relative importance of the objectives; they may be taken as <NUM>. Note that the first term does not depend on G, so it can be omitted if this is only used as the generator objective.

<FIG> schematically shows an example of an embodiment of a training system, in this case a cGAN. In these embodiments, y may represent an image that is to be transformed according to a noise vector z.

<FIG> shows a generator <NUM> and a discriminator <NUM>. The discriminator may be configured to receive as input a pair (x,y) or a pair (x',y), and decide if x/x' is real based on y.

Generator <NUM> is applied to two inputs (y, z) and to (R(y), z). The outputs include x'=G(y,z). From the latter R(x') is computed which may be used to compute the G-specific loss term for the equivariant R shown above, that is G(R(y), z) and R(G(y,z)) should be close. Note that the discriminator need not be applied to G(R(y), z) or to R(G(y,z)); although it could, e.g., as additional synthesized training data, or an additional pair of synthesized sensor data versus fidelity-preserved transformed synthesized sensor data (e.g., x' and R(x')).

Also computed is F(x'). The discriminator is applied to F(x') and x' which may be used to compare the discriminator classifications of x' and F(x'). It is expected that the discriminator should work equally well on F(x') as on x'. Discriminator <NUM> may be trained on the difference between the classifications of F(x') and x', which should be small, or on the output directly, which should both be fake.

The discriminator may further be applied on x, F(x) and T(x), e.g., with the loss terms as indicated above. For example, a loss term may drive discriminator values for x and T(x) apart, but for x and F(x) together.

A further example of a fidelity-destroying transformation is a perturbation of a given real image, wherein the perturbation area is large enough, e.g., having an area larger than threshold, to destroy the local/global information in the image.

Further examples of a fidelity-destroying transformation are composed images, in particular, CutMix operations. For example, one way to compose sensor data, e.g., images, is to cut and paste one or more patches from images of different classes, e.g., real and fake. For example, the transformation may be the so-called CutMix transformation of real and fake samples. An advantage of CutMix in the context of GAN training is that it does not alter the real and fake image patches used for mixing, preserving their original class domain, and provides a large variety of possible outputs. We visualize the CutMix augmentation strategy in <FIG>, e.g., between <NUM>% and <NUM>%.

For example, for the T transformation, one may compose two real-images, or a real and a synthesized image. The discriminator may be trained to classify it as not-real because of the global inconsistencies.

<FIG> schematically shows examples of an embodiment of data in an embodiment of a training method. <FIG> shows a schematic visualization of the CutMix transformation.

The first row of <FIG> shows a real image <NUM>, e.g., measured sensor data, and a fake image <NUM>, e.g., synthesized sensor data. The second row shows binary masks M, which may be used for the CutMix operation. In this row of <FIG> a white color is used for real, and a black color for fake. Mask <NUM> will cause the majority of the composed image to come from the fake image. The second row schematically shows the composed sensor data, in this case a CutMix image from a real and fake sample. For example, image <NUM> may have been obtained by taking the part from fake image <NUM> indicated by mask <NUM> and taking the part from real image <NUM> indicated by mask <NUM>.

For example, if the images <NUM> and <NUM> are images taken from the environment of a vehicle, e.g., street scenes, then the composed image <NUM> may partly show a realistic street scene and partly a less-realistic looking generated street scene. The same holds for products, e.g., goods, which may or may not have production defects. A composed sensor data may show the product but part of it corresponds to measured sensor data, e.g., an image of the product as it rolls of a production line, with the other part synthesized.

For example, in an embodiment a new training sample is composed x~ for the discriminator by mixing measured sensor data x and synthesized sensor data G(z) ∈ RW×H×C with the mask M: <MAT> where M ∈ {<NUM>,<NUM>}W×H is the binary mask indicating if the pixel (i,j) comes from the real (Mi,j = <NUM>) or fake (Mi,j = <NUM>) image, <NUM> is a binary mask filled with ones, and ⊙ is an element-wise multiplication.

<FIG> shows an example of composed sensor data, in this case, the left half of the image is a real image, e.g., measured sensor data, while the part on the right is a synthesized image, e.g., synthesized sensor data. The same could be done with two real images. For example, the image of <FIG> may be a fidelity destroyed image for image <NUM>.

Embodiments may be used in GAN models for data synthesis and data augmentation. Its use is particularly advantages when collecting additional data is expensive or legally not possible. In the context of autonomous driving this includes extreme situations, like dangerously maneuvering cars or near-hit situations involving pedestrians.

For example, the methods, e.g., training methods, may be computer implemented methods. For example, accessing training data, and/or receiving input data may be done using a communication interface, e.g., an electronic interface, a network interface, a memory interface, etc. For example, storing or retrieving parameters may be done from an electronic storage, e.g., a memory, a hard drive, etc., e.g., parameters of the networks. For example, applying a neural network to data of the training data, and/or adjusting the stored parameters to train the network may be done using an electronic computing device, e.g., a computer.

The neural networks, either during training and/or during applying may have multiple layers, which may include, e.g., convolutional layers and the like. For example, the neural network may have at least <NUM>, <NUM>, <NUM>, <NUM>, <NUM> or <NUM> hidden layers, or more, etc. The number of neurons in the neural network may, e.g., be at least <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, or more, etc..

Many different ways of executing the method are possible, as will be apparent to a person skilled in the art. For example, the order of the steps can be performed in the shown order, but the order of the steps can be varied or some steps may be executed in parallel. Moreover, in between steps other method steps may be inserted. The inserted steps may represent refinements of the method such as described herein, or may be unrelated to the method. For example, some steps may be executed, at least partially, in parallel. Moreover, a given step may not have finished completely before a next step is started.

Embodiments of the method may be executed using software, which comprises instructions for causing a processor system to perform the method, e.g., method <NUM>. Software may only include those steps taken by a particular sub-entity of the system. The software may be stored in a suitable storage medium, such as a hard disk, a floppy, a memory, an optical disc, etc. The software may be sent as a signal along a wire, or wireless, or using a data network, e.g., the Internet. The software may be made available for download and/or for remote usage on a server. Embodiments of the method may be executed using a bitstream arranged to configure programmable logic, e.g., a field-programmable gate array (FPGA), to perform the method.

Below a particular detailed embodiment is discussed, which is built upon the state-of-the-art BigGAN model, and extend its discriminator. For details on BigGAN see, <NPL>.

Different than described in BigGAN are, e.g., the addition of transforms F and/or T. To train the system, one could take.

Experiments were performed in a conditional and in an unconditional embodiment. Note that the original BigGAN is a class-conditional model. For the unconditional model, the class-conditional BatchNorm may be replaced with selfmodulation, wherein the BatchNorm parameters are conditioned only on the latent vector z, and do not use the class projection of in the discriminator.

During the training, for each iteration, a mini-batch of CutMix images (x~; c; M) may be created, e.g., with a probability rmix. This probability may be increased linearly from <NUM> to <NUM> between the first n epochs in order to give the generator time to learn how to synthesize more real looking samples and not to give the discriminator too much power from the start. CutMix images are created from the existing real/real and/or real/fake images in the mini-batch using binary masks M. For sampling M, one may first sample the combination ratio c between the real and generated images from the uniform distribution (<NUM>, <NUM>) and then uniformly sample the bounding box coordinates for the cropping regions of x and G(z) to preserve the c ratio.

The original training parameters of BigGAN may be adopted. In particular, one may use a uniformly distributed noise vector z in [-<NUM>, <NUM>]<NUM> as input to the generator, and the Adam optimizer with learning rates of 1e-<NUM> and 5e-<NUM> for G and DU. It was found beneficial in experiments to operate with considerably smaller mini-batch sizes than BigGAN, e.g., batch sizes between <NUM> and <NUM>.

<FIG> schematically shows an example of an embodiment of a training system <NUM>.

<FIG> shows an autonomous apparatus, in this case an autonomous car <NUM>', situated in an environment <NUM>, e.g., a traffic situation. In environment <NUM> there may be various objects, both static and dynamic, that affect how the apparatus <NUM> may be controlled. A similar apparatus, in this case car <NUM>, may be used to obtain measured sensor data. For example, shown in <FIG> is a pedestrian <NUM> crossing the environment behind car <NUM>. Apparatus <NUM> may be autonomous but does not need to be. Apparatus <NUM> and <NUM>' may be the same except for an update in controller <NUM>.

Car <NUM> may comprise a sensor system <NUM>, e.g., comprising one or more image sensors, radar, LIDAR and so on, to sense the environment of the car, e.g., environment <NUM>. For example, sensor system <NUM> may be configured to produce measured sensor data comprising information on environment <NUM>. Car <NUM> may comprise one or more actuators to move the car through environment <NUM>, e.g., wheels and motor.

Sensor data obtained from sensor system <NUM> may be stored in a first training database <NUM>. A training system <NUM>, e.g., configured for an embodiment of a training method for training a generator neural network may be configured to train a generator to generate synthesized sensor data which appears to be drawn from first training database <NUM>. Training system may be configured to obtain an initial training set form first database <NUM> and train a generator network from the initial training set. For example, a training system <NUM> may produce a generator network for use in a generator system <NUM>. Generator system <NUM> may be used to generate additional sensor data, e.g., synthesized sensor data. The synthesized sensor data may be stored in a second training database <NUM>. The second training database <NUM> may also comprise the original measured sensor data, e.g., taken from first database <NUM>.

The synthesized training data may be generated with or without the use of class-labels. For example, the measured training data in first database <NUM> may be labeled, e.g., by apparatus <NUM>, by sensor <NUM>, by a further device (not shown), or by a human. The class labels may be used to generate synthesized sensor data of a particular kind, e.g., with a nearby pedestrian. An unconditional generator neural network may be configured to receive as input a measured sensor data or a noise vector, or both. Also a conditional generator neural network may be configured to receive as input a measured sensor data or a noise vector, or both. Both types may be trained for pure generation or for domain transfer or for a combination, e.g., generation in the context of a measured sensor data.

A machine learning system <NUM> may be configured to train a machine learnable model on the training data in second database <NUM>. For example, the machine learnable model may be a classifier. The machine learnable model may comprise a neural network, but this is not necessary; For example, it may comprise an SVM, random forests, and so on. Machine learning system <NUM> may be configured with a learning algorithms consistent with the type of machine learnable model, e.g., SVM training or random forest training. Machine learning system <NUM> may use the synthesized sensor data for training, for testing, or for both. Machine learning system <NUM> produces a trained classifier <NUM>. For example, classifier <NUM> may be configured to classify an object in the environment of the apparatus from the measured sensor data.

The classifier <NUM> may be included in a controller for an autonomous apparatus <NUM>', e.g., like car <NUM>. For example, a controller <NUM> may comprise classifier <NUM>. Controller <NUM> may be configured to generate a control signal to control the autonomous apparatus <NUM>'. Controller <NUM> may be configured to generate a control signal at least from the object classified by the classifier. For example, if classifier <NUM> classifies that an environment <NUM> comprises a pedestrian like <NUM>, then it is not safe to revert the car. The control signal may be configured to control the actuators, e.g., turning and steering of the wheels and/or motor.

It will be appreciated that the presently disclosed subject matter also extends to computer programs, particularly computer programs on or in a carrier, adapted for putting the presently disclosed subject matter into practice. The program may be in the form of source code, object code, a code intermediate source, and object code such as partially compiled form, or in any other form suitable for use in the implementation of an embodiment of the method. An embodiment relating to a computer program product comprises computer executable instructions corresponding to each of the processing steps of at least one of the methods set forth. These instructions may be subdivided into subroutines and/or be stored in one or more files that may be linked statically or dynamically. Another embodiment relating to a computer program product comprises computer executable instructions corresponding to each of the devices, units and/or parts of at least one of the systems and/or products set forth.

<FIG> shows a computer readable medium <NUM> having a writable part <NUM> comprising a computer program <NUM>, the computer program <NUM> comprising instructions for causing a processor system to perform a training method according to an embodiment. The computer program <NUM> may be embodied on the computer readable medium <NUM> as physical marks or by magnetization of the computer readable medium <NUM>. However, any other suitable embodiment is conceivable as well. Furthermore, it will be appreciated that, although the computer readable medium <NUM> is shown here as an optical disc, the computer readable medium <NUM> may be any suitable computer readable medium, such as a hard disk, solid state memory, flash memory, etc., and may be non-recordable or recordable. The computer program <NUM> comprises instructions for causing a processor system to perform said training method.

<FIG> shows in a schematic representation of a processor system <NUM> according to an embodiment of a training system, or generator system. The processor system comprises one or more integrated circuits <NUM>. The architecture of the one or more integrated circuits <NUM> is schematically shown in <FIG>. Circuit <NUM> comprises a processing unit <NUM>, e.g., a CPU, for running computer program components to execute a method according to an embodiment and/or implement its modules or units. Circuit <NUM> comprises a memory <NUM> for storing programming code, data, etc. Part of memory <NUM> may be read-only. Circuit <NUM> may comprise a communication element <NUM>, e.g., an antenna, connectors or both, and the like. Circuit <NUM> may comprise a dedicated integrated circuit <NUM> for performing part or all of the processing defined in the method. Processor <NUM>, memory <NUM>, dedicated IC <NUM> and communication element <NUM> may be connected to each other via an interconnect <NUM>, say a bus. The processor system <NUM> may be arranged for contact and/or contact-less communication, using an antenna and/or connectors, respectively.

For example, in an embodiment, processor system <NUM>, e.g., a training device may comprise a processor circuit and a memory circuit, the processor being arranged to execute software stored in the memory circuit. For example, the processor circuit may be an Intel Core i7 processor, ARM Cortex-R8, etc. In an embodiment, the processor circuit may be ARM Cortex M0. The memory circuit may be an ROM circuit, or a non-volatile memory, e.g., a flash memory. The memory circuit may be a volatile memory, e.g., an SRAM memory. In the latter case, the device may comprise a non-volatile software interface, e.g., a hard drive, a network interface, etc., arranged for providing the software.

While device <NUM> is shown as including one of each described component, the various components may be duplicated in various embodiments. For example, the processor <NUM> may include multiple microprocessors that are configured to independently execute the methods described herein or are configured to perform steps or subroutines of the methods described herein such that the multiple processors cooperate to achieve the functionality described herein. Further, where the device <NUM> is implemented in a cloud computing system, the various hardware components may belong to separate physical systems. For example, the processor <NUM> may include a first processor in a first server and a second processor in a second server.

It should be noted that the above-mentioned embodiments illustrate rather than limit the presently disclosed subject matter, and that those skilled in the art will be able to design many alternative embodiments.

Use of the verb 'comprise' and its conjugations does not exclude the presence of elements or steps other than those stated in a claim. The article 'a' or 'an' preceding an element does not exclude the presence of a plurality of such elements. Expressions such as "at least one of" when preceding a list of elements represent a selection of all or of any subset of elements from the list. For example, the expression, "at least one of A, B, and C" should be understood as including only A, only B, only C, both A and B, both A and C, both B and C, or all of A, B, and C. The presently disclosed subject matter may be implemented by hardware comprising several distinct elements, and by a suitably programmed computer. In the device claim enumerating several parts, several of these parts may be embodied by one and the same item of hardware.

Claim 1:
A training method (<NUM>) for training a generator neural network (<NUM>; <NUM>) configured to generate synthesized sensor data (<NUM>), the method comprising
- accessing (<NUM>) a training set (<NUM>; <NUM>; <NUM>) of measured sensor data obtained from a sensor,
- training (<NUM>) the generator network together with a discriminator neural network (<NUM>; <NUM>; <NUM>), wherein the training comprises
- generating (<NUM>) the synthesized sensor data using the generator neural network,
- optimizing (<NUM>) the discriminator network to distinguish between the measured sensor data and the synthesized sensor data,
- optimizing (<NUM>) the generator network to generate the synthesized sensor data which is indistinguishable from the measured sensor data by the discriminator network,
characterised in that
- a fidelity destroying transformation is defined configured to transform the measured sensor data to obtain (<NUM>) fidelity-destroyed transformed measured sensor data, the training comprising optimizing the discriminator network to distinguish between the measured sensor data and the fidelity-destroyed transformed measured sensor data,
wherein the fidelity destroying transformation transforms its input to a result that cannot be regarded as drawn from the same probability distribution as the input.