DEVICE AND COMPUTER-IMPLEMENTED METHOD FOR DETERMINING A DATA SET FOR USE IN TRAINING, FOR TRAINING AND FOR OPERATING A MACHINE LEARNING SYSTEM

A device and computer-implemented method for determining a data set for use in training a machine learning system. A first and second data set are combined to form the data set for use in training. The first data set includes a first digital image identified by at least one label from a first set of labels. The second data set includes a second digital image identified by at least one label from a second set of labels. The two sets of labels differ by at least one label. A first encoding for identifying the first digital image is determined with labels from both sets of labels. The first encoding is mapped to a first representation in a state space. The data set for use in training includes the first digital image and the first representation.

FIELD

The present invention relates to a device and a computer-implemented method for determining a data set for use in training a machine learning system, for training the machine learning system and for operating the machine learning system.

SUMMARY

According to an example embodiment of the present invention, a computer-implemented method for determining a data set for use in training a machine learning system provides that at least two data sets are combined to form the data set for use in training, wherein the at least two data sets comprise a first data set and a second data set, wherein the first data set comprises a first digital image, which is identified by at least one label, in particular by a one-hot encoding with one label or by a multi-hot encoding with a plurality of labels, from a first set of labels, and the second data set comprises a second digital image, which is identified by at least one label, in particular by a one-hot encoding with one label or by a multi-hot encoding with a plurality of labels, from a second set of labels, wherein the two sets of labels differ by at least one label, wherein, depending on the two sets of labels, a first encoding for identifying the first digital image is determined with labels from both sets of labels, wherein the first encoding is mapped to a first representation in a state space, wherein the data set for use in training comprises the first digital image and the first representation. The representation is a continuous representation of at least one label from the union for identifying the first image. The data set for use in training the machine learning system is particularly suitable for training a semantic image synthesis depending on the first representation in the state space and the first image. The semantic image synthesis takes place, for example, with a generative model designed to map the first representation to a synthetic digital image.

According to an example embodiment of the present invention, it may be provided that, depending on the two sets of labels, a second encoding for identifying the second digital image is determined with labels from both sets of labels, wherein the second encoding is mapped to a second representation in the state space, wherein the data set for use in training comprises the second digital image and the second representation. This makes both images and the representations respectively assigned to them available for training the machine learning system, in particular for semantic image synthesis.

According to an example embodiment of the present invention, a computer-implemented method for training a machine learning system provides that a data set for use in training is determined with the method for determining a data set for use in training the machine learning system, wherein the system comprises a model designed to map at least one label, in particular a one-hot encoding of one label or a multi-hot encoding of a plurality of labels, to a representation of the at least one label in the state space provided in the data set for use in training and to map the representation from the state space to a synthetic digital image, wherein the model is trained with the first representation and the first digital image and/or with the second representation and the second digital image from the data set to map the representation from the state space to the synthetic digital image, or wherein the model is designed to map a digital image to a representation of at least one label in the state space provided in the data set for use in training and, depending on the representation from the state space, to determine the at least one label, in particular a one-hot encoding of one label or a multi-hot encoding of a plurality of labels, for the digital image, wherein the model is trained with the first representation and the first digital image and/or with the second representation and the second digital image from the data set to map the digital image to the representation in the state space. This means that the model is trained with a representation of at least one combined label, e.g., for semantic image synthesis.

According to an example embodiment of the present invention, a computer-implemented method for operating a machine learning system provides that the machine learning system comprises a model designed to map at least one label for a synthetic digital image, in particular a one-hot encoding of one label or a multi-hot encoding of a plurality of labels, to a representation of the at least one label in the state space provided in the data set for use in training, and to map the representation from the state space to the synthetic digital image, or to determine at least one label, in particular a one-hot encoding of one label or a multi-hot encoding of a plurality of labels, for a digital image, wherein the model is trained with the method for training, wherein, with the model, the at least one label is mapped to a representation of the at least one label in the state space and the representation in the state space is mapped to the synthetic digital image, or wherein, with the model, the at least one label for a digital image is determined. This means that the model is trained with a representation of at least one combined label, e.g., for semantic image synthesis. Depending on synthetic digital images generated by the model and/or depending on the labels, the model can be trained to label, in particular to classify or to semantically segment, the synthetic digital image.

According to an example embodiment of the present invention, it may be provided that the machine learning system comprises a technical system designed for at least partially autonomous operation depending on at least one label, in particular a one-hot encoding of one label or a multi-hot encoding of a plurality of labels, for a digital image, wherein the digital image is captured in the method for operating the machine learning system, wherein the at least one label for the digital image is determined with the model, and wherein the technical system is operated at least partially autonomously depending on the at least one label for the digital image. This means that the system with the trained model responds autonomously to previously unknown digital images.

According to an example embodiment of the present invention, a device for determining a data set for use in training a machine learning system, comprising at least one processor and at least one memory, wherein the at least one memory comprises instructions that can be executed by the at least one processor and that, when they are executed by the at least one processor, cause the device to perform the method for determining the data set according to the present invention, wherein the at least one processor is designed to execute the instructions, has advantages corresponding to the advantages of the method for determining the data set.

According to an example embodiment of the present invention, a device for training a machine learning system, the device comprising at least one processor and at least one memory, wherein the at least one memory comprises instructions that can be executed by the at least one processor and that, when they are executed by the at least one processor, cause the device to perform the method for training the machine learning system according to the present invention, wherein the at least one processor is designed to execute the instructions, has advantages corresponding to the advantages of the method for training the machine learning system.

According to an example embodiment of the present invention, a device for operating a machine learning system, comprising at least one processor and at least one memory, wherein the at least one memory comprises instructions that can be executed by the at least one processor and that, when they are executed by the at least one processor, cause the device to perform the method for operating the machine learning system according to the present invention, wherein the at least one processor is designed to execute the instructions, has advantages corresponding to the advantages of the method for operating the machine learning system.

According to an example embodiment of the present invention, a computer program comprising instructions that can be executed by a computer and that, when they are executed by the computer, cause the method according to the present invention to run has advantages corresponding to the advantages of the method.

Further advantageous embodiments of the present invention can be found in the disclosure herein.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

FIG.1shows a schematic representation of a first device100. The first device is designed to determine a data set102for use in training.

The first device100comprises at least one first processor104and at least one first memory106.

The at least one first memory106comprises instructions that can be executed by the at least one first processor104and that, when they are executed by the at least one first processor104, cause the first device100to perform a method for determining the data set102for use in training. The at least one first processor104is designed to execute these instructions. In the example, the data set102for use in training is stored in the at least one first memory106.

FIG.2shows an architecture for determining the data set102for use in training. The data set102for use in training is determined depending on at least two data sets. This is described using the example of determining the data set102for use in training depending on a first data set202and a second data set204.

An example of the first data set202is Berkeley DeepDrive, BDD. An example of the second data set204is Cityscapes.

The first data set202comprises digital images, each of which is assigned at least one label, which identifies the relevant digital image. In the example, a first digital image206, which is identified by at least one label208, is provided. The first image206in the example is identified by a one-hot encoding with one label or by a multi-hot encoding with a plurality of labels. For the first image206, a first set of labels is provided, which comprises the label or labels for the first image.

The labels from the first set of N labels each identify a class. In one-hot encoding, a class is mapped to a vector {0,1}N, which contains exactly one one and otherwise zeros. For example, the elements of a vector representing a particular class are each assigned to a class, wherein the element assigned to the particular class represented by the vector is one. In multi-hot encoding, it is allowed that a plurality of the elements of the vector are one, i.e., the vector can represent a plurality of classes.

An example of the first set of labels is two-wheeler, automobile, background person. An exemplary first digital image206shows a two-wheeler. An exemplary first label208is two-wheeler. An example of a one-hot encoding of the exemplary first digital image206is a vector [1, 0, 0, 0], which represents the exemplary first label208“two-wheeler”.

The second data set204comprises digital images, each of which is assigned at least one label, which identifies the relevant digital image. In the example, a second digital image210, which is identified by at least one label212, is provided. The second digital image210in the example is identified by a one-hot encoding with one label or by a multi-hot encoding with a plurality of labels. For the second image210, a second set of labels is provided, which comprises the label or labels for the second digital image210.

The labels from the second set of M labels each identify a class. In one-hot encoding, a class is mapped to a vector {0,1}M, which contains exactly one one and otherwise zeros. For example, the elements of a vector representing a particular class are each assigned to a class, wherein the element assigned to the particular class represented by the vector is one. In multi-hot encoding, it is allowed that a plurality of the elements of the vector are one, i.e., the vector can represent a plurality of classes.

An example of the second set of labels is sky, vegetation, person, vehicle. An exemplary second digital image210shows a person. An exemplary second label212is person. An example of a one-hot encoding of the exemplary second digital image210is a vector [0, 0, 1, 0], which represents the exemplary second label212“person”.

The two sets of labels in the example differed by at least one label. In the example, N=M=4 labels are provided in both sets of labels. The number of labels in the two sets may also be different.

The digital images from the first data set202are each assigned an encoding, which encodes labels from both sets, for identifying the relevant digital image. The first digital image206is assigned a first encoding214for identifying the first digital image206. The first encoding214encodes labels from both sets. This is described by way of example for the first digital image206.

The labels contained in both sets are summarized in the first encoding214. In the example, the first encoding214comprises the labels of the two sets that differ from one another. In the example, the order in which the labels are contained in the first encoding214corresponds to the order in which the labels are arranged in the first set of labels, followed by the labels from the second set of labels that differ from those in the first set of labels, in the order in which the labels that differ from the first set of labels are arranged in the second set of labels. A different order is likewise possible.

The first encoding214of the exemplary first digital image206is a vector [1, 0, 0, 0, 0, 0, 0] whose elements are assigned to the labels two-wheeler, automobile, background, person, sky, vegetation and vehicle.

The digital images from the second data set204are each assigned an encoding, which encodes labels from both sets, for identifying the relevant digital image. This is described by way of example for the second digital image210.

The second digital image210is assigned a second encoding220for identifying the second digital image210. The second encoding220encodes labels from both sets.

The labels contained in both sets are summarized in the second encoding220, as described for the first encoding214. The second encoding220of the exemplary second digital image210is a vector [0, 0, 0, 1, 0, 0, 0].

It may be provided that the first encoding214is generated depending on the labels from the two sets of labels. For example, a semantic assignment of the labels from the two sets is specified. The semantic assignment is determined, for example, by an expert or with a model designed to assign the labels to one another. An example of the model is WordNet, which is described in Redmon et al., “YOLO9000: Better, Faster, Stronger” (arxiv 1612.08242).

For example, the label “two-wheeler” and the label “vehicle” are assigned to one another in the semantic assignment. An exemplary first encoding214for the first digital image206is a multi-hot encoding, i.e., a vector [1, 0, 0, 0, 0, 1].

The first encoding214is represented by a first representation216in a state space218.

The first representation216for the first encoding214of the exemplary first digital image206is a first vector [0.0232, 0.1, 0.43] in the state space218.

The second encoding220is represented by a second representation222in the state space218.

The second representation222for the second encoding216of the exemplary second digital image210is a second vector [0.73, −0.21, −0.113] in the state space218.

The encodings for the other digital images in the example are each represented by a representation of the relevant encoding in the state space218.

In the example, the data set102for use in training comprises the first digital image206and the first representation216as well as the second digital image210and the second representation222. The first digital image206and the first representation216are assigned to one another as a first training data point224. The second digital image210and the second representation222are assigned to one another as a second training data point226.

The first data set202and the second data set204are combined with a computer-implemented method for determining the data set102for use in training.

FIG.3shows a flowchart with steps of a computer-implemented first method. The first method is provided for determining the data set102for use in training. The first method is described using the example of the first digital image206and of the second digital image210. The first method is correspondingly performed for the other digital images.

The first method comprises a step302.

In step302, the first data set202and the second data set204are provided.

In step304, depending on the two sets of labels, the first encoding214for identifying the first digital image206is determined with labels from both sets of labels.

In step304, depending on the two sets of labels, the second encoding218for identifying the second digital image210is optionally determined with labels from both sets of labels.

In step306, the first encoding214is mapped to the first representation216in the state space218.

In step306, the second encoding220is optionally mapped to the second representation222in the state space218.

In step308, the first data set202and the second data set204are combined to form the data set102for use in training.

The data set102for use in training comprises the first digital image206and the first representation216.

Optionally, the data set102for use in training comprises the second digital image210and the second representation222.

FIG.4shows a schematic representation of a second device400. The second device400is designed to train a machine learning system402.

The second device400comprises at least one second processor404and at least one second memory406.

The at least one second memory406comprises instructions that can be executed by the at least one second processor404and that, when they are executed by the at least one second processor404, cause the second device400to perform a method for training. The at least one second processor404is designed to execute these instructions. In the example, the second device400comprises the machine learning system402. In the example, the at least one second memory406comprises a part of the machine learning system402.

The machine learning system402comprises a model. In the example, the model comprises a generative adversarial network GAN410, which is designed for semantic image synthesis. The GAN410is designed to map at least one label412for a synthetic or real digital image414to a representation416of the at least one label412in the state space218. The at least one label412for the synthetic or real digital image414is, for example, a one-hot encoding of one label or a multi-hot encoding of a plurality of labels. The term “real image” refers to a digital image that reproduces an actually existing view.

The GAN410is designed, for example, to determine the relevant representation in the state space218by a linear projection of the relevant encoding. The GAN410comprises, for example, an encoder for the state space218or uses knowledge of an already trained language model such as Vord2Vec.

The GAN410is designed to map the representation416from the state space218to the synthetic digital image414.

The GAN410comprises, for example, an already trained neural network.

It may be provided that the model comprises a classifier418. The classifier418is designed to determine at least one label420, in particular a one-hot encoding of one label or a multi-hot encoding of a plurality of labels, for the synthetic digital image414or for a digital image422. The classifier comprises, for example, an already trained neural network.

It may be provided that the machine learning system402comprises a technical system424designed for at least partially autonomous operation depending on the at least one label420for the digital image422.

The technical system424is, for example, a physical system. The technical system424is, for example, a robot, in particular an autonomous vehicle or a household appliance or a tool or a manufacturing machine or a personal assistance system or an access control system.

The technical system424comprises, for example, a sensor426designed to capture the digital image422. The sensor426is, for example, a camera, a radar sensor, a LiDAR sensor, an ultrasonic sensor, an infrared sensor, a motion sensor or a thermal imaging sensor.

FIG.5shows a flowchart with steps of a computer-implemented second method. The second method is provided for training the machine learning system402.

The training may comprise training the GAN410alone, training the classifier418alone, or jointly training the GAN410and classifier418with the data set102for use in training.

The second method is described using the example of the first digital image206and of the second digital image210. The second method is correspondingly performed for the other digital images.

The second method for training the system402comprises a step502.

In step502, the data set102for use in training is determined with the first method.

In step504, the GAN410is trained with the first representation216and the first digital image206from the data set102for use in training.

The GAN410is trained to map the representation416from the state space218to the synthetic digital image414.

The GAN410is optionally trained with the second representation222and the second digital image210from the data set102for use in training.

This means that, in the example, the GAN410alone is trained with the data set102for use in training.

FIG.6shows a schematic representation of a third device500. The third device500is designed to operate the machine learning system302.

The third device600comprises at least one third processor602and at least one third memory604.

The at least one third memory604comprises instructions that can be executed by the at least one third processor602and that, when they are executed by the at least one third processor602, cause the third device600to perform a method for machine learning. The at least one third processor602is designed to execute these instructions.

FIG.7shows a flowchart with steps of a computer-implemented third method. The third method is provided for machine learning, in particular for operating the machine learning system402.

The third method comprises a step702.

In step702, the GAN410is trained with the second method.

In step704, the at least one label412for the synthetic digital image414for training the classifier418is specified.

In step706, the synthetic digital image414is determined with the GAN410depending on the at least one label412for the synthetic digital image414.

In step708, the classifier418is trained depending on the synthetic digital image414to determine the at least one label412for the synthetic digital image414.

It may be provided that the classifier418or the GAN410are already trained. It may be provided that only the classifier418alone or only the GAN410alone is trained.

In step710, the digital image422is captured.

In step712, the at least one label420for the digital image422is determined with the classifier418.

In step714, the technical system424is operated at least partially autonomously depending on the at least one label420for the digital image422.

The methods are each implemented, for example, as a computer program comprising instructions that can be executed by a computer and that, when they are executed by the computer, cause the relevant method to run.