Patent Description:
In supervised learning artificial neural networks for video and/or image processing are trained using training data comprising labelled images. Creating sets of labelled images for training is cumbersome and expensive. It is thus desirable to identify and label images that are particular useful for the training.

In one aspect acquisition functions are used to identify images from sets of images. Acquisition functions are deterministic functions that select images based for example on entropy or mutual information defined in terms of information about an image content such as color or the like.

The usefulness of an individual acquisition function for a particular set of images depends on the data of the images. In order to use a particularly useful acquisition function, for example,<NPL> discloses a method that learns a policy to choose an acquisition function from a set of predetermined and deterministic acquisition functions based on a set of available heuristics. Furthermore, <NPL>), discloses an active learning method to train a LiDAR 3D object detector with the least amount of labeled training data necessary.

The invention is set out in independent claims <NUM>, <NUM>, <NUM> and <NUM>. Preferred aspects of the invention are set out in the dependent claims <NUM>-<NUM>.

<FIG> schematically depicts parts of an exemplary learning system <NUM>.

With the data-driven method described below an acquisition function is determined. As prediction uncertainty is an essential input to acquisition heuristics, in the example a deep Bayesian Neural Net is used as base predictor.

In order to acquire high-quality estimates of prediction uncertainty with an acceptable computational cost, a deterministic approximation scheme is described that can both effectively train a deep Bayesian Neural Net and calculate its posterior predictive density following a chain of closed-form operations. With this, the probabilistic information, i.e. the uncertainty provided by the predictions of the deep Bayesian Neural Net is used for a state design, which brings about a full-scale probability distribution. This distribution is then fed into a probabilistic policy network, e.g. another Bayesian Neural Net, which is trained by reinforcement feedback collected from every labeling round in order to inform the system about the success of its current acquisition function. This feedback fine-tunes the acquisition function, bringing about improved performance in the subsequent labeling rounds.

The training uses samples that are processed recursively with epochs that comprise batches.

An exemplary training is based on a large unlabeled set of images, e.g. <NUM> million images. The training starts from a small labeled subset of this set, for instance <NUM> images. This subset is for example chosen at random from the set, i.e. without active learning. A predictor, e.g. Bayesian neural net or gaussian process is trained based on this small subset. The subset of images is in this example removed from the set.

Afterwards active learning rounds are started. For example, <NUM> images are chosen to be labeled in a labeling round based on an active learning algorithm that will decides which images will be labeled.

At each labeling round, the large unlabeled set, e.g. in the first labeling round <NUM> million - <NUM> images, is used to train the predictor that was previously trained initially on the randomly selected small subset of images, e.g. the <NUM> images.

In a model that uses the predictions of the predictor, each unlabeled image of the large unlabeled set is assigned an acquisition score by an acquisition scoring function and a ranked order of the images is determined by ranking the images with respect to these acquisition scores. The acquisition scoring function is referred to as acquisition function.

The top-ranking images, e.g. the top-ranking <NUM> images, are presented to the oracle. The oracle provides the labels for these images and adds them to the previous small set of images. More specifically the oracle is adapted to present an image to a human via a human machine interface, e.g. a display. The human machine interface is adapted to detect an input, more specifically a label, that is provided by the human, e.g. via a keyboard, when viewing the image. As the top-ranking images are most relevant, the learning system presents these to the human user for labeling. Hence the human user must not label all images but only the images that are considered most relevant.

Then the predictor is trained with the new small set, e.g. after the first labeling round with <NUM>+<NUM> = <NUM> labeled images.

As the model changes recursively, its acquisition scoring function is also changed.

This cycle is repeated for example until our labeling budget is exhausted.

For the prediction, a data set <MAT> is used that consists of N tuples of feature vectors <MAT> and labels yn ∈ {<NUM>,<NUM>}C for a C dimensional binary output label.

Parameterizing an arbitrary neural network f(·) with parameters w following some prior distribution p(w), the following data generation process is used in the example <MAT> <MAT> where fc is the c-th output channel of the neural network, where φ(u) = <MAT> is the normal cumulative distribution function, and Ber(· | ·) is a Bernoulli distribution.

For a test data point x* a posterior predictive distribution is used that performs model averaging on latent variables w based on their posterior distribution for example
with <MAT> <MAT> for X = {x<NUM>,. , xN} and Y = {y<NUM>,.

Considering a normal mean-field approximate posterior <MAT> where V is a number of synaptic connections in the neural net. For a new observation x* the following approximation is available to predict the output label y* <MAT> where N(. | ·,·) is a normal distribution having two free parameters, the tuple <MAT> represents the variational parameter set for weight wi of the network f(·) and <MAT>, for i = <NUM>,. , V is a collection of the hyperparameters of weight priors p(wi), and where the functions <MAT> and <MAT> encode a cumulative map from an input layer to the moments of the top-most layer L of an exemplary Bayesian Neural Net predictor.

The data driven label acquisition described herein can perform an adaptation during an active learning labeling round. This is achieved by a reinforcement learning framework, where in the example a policy net, i.e. a deep Bayesian Neural Net, is trained by rewards observed from the training environment.

An exemplary definition of a state, action and reward is given below for this framework.

The state for the active learning is defined in the example based on a collection of unlabeled input points Du. In an initial learning before the unlabeled input points are used, a collection of labeled input points Dl may be used as well. In an example the input points refer to representations xn of the set of images.

For efficient processing, a representation of the state is in the example determined by first ranking an unlabeled sample set into a ranked order of the unlabeled input points Du by an information-theoretic heuristic, e.g. a maximum entropy criterion. As such heuristics assign similar scores to samples with similar character, consecutive samples in ranking inevitably have high correlation.

In order to break trends and enhance diversity, in one aspect, from the ranked order of input points, starting from the top of the order, every k-th input point is picked up until M samples {x<NUM>,. , xM} are collected.

Feeding these samples into the Bayesian Neural Net predictor a posterior predictive density estimate is determined for each sample and the state of the unlabeled sample space is determined in the following distributional form: <MAT> where <MAT> is the mean and <MAT> is the variance of the activation of an output neuron, calculated as described above for the Bayesian Neural Net predictor. The state S follows a C x M-dimensional normal distribution.

An action comprises in this example, sampling in each labeling round, a number of data points from the set of samples {xi1,. , xiM} e.g. according to probability masses assigned on them.

The reward is calculated in the example by the oracle <NUM> from a measure of model fit and is calculated for a new data point having a newly labeled pair (x, y) as <MAT> where qold(·) is the variational posterior before training with the new data point (x, y) and where qnew(·) is the variational posterior after training with the new data point (x, y).

Optionally, a second component that encourages the diversity across the chosen labels throughout the whole labeling round is calculated as <MAT>.

The reward in this in this option calculated depending on the sum of Rimprov and Rdiv.

Additionally, a policy net π(·), in the example a second Bayesian Neural Net, is parameterized by φ∼p(φ). The input to the policy net π(·) is the state S. The output of the policy net π(·) is a parametrization of an M-dimensional categorical distribution over possible actions. The output is for example determined for M binary probabilities for taking an action ãm at a time point t <MAT>.

The action is then chosen based on the categorical distribution from <MAT>.

Wherein Cat(·) is a Categorical distribution.

In one aspect, the episodic version of a REINFORCE algorithm according to <NPL>, is adapted to train the policy net π(·).

A moving average over all the past rewards may be uses. A labeling episode in the example consists of choosing a sequence of points to be labeled e.g. with a discount factor of γ = <NUM>:<NUM> after which the BNN is retrained and the policy net π(·) takes one update step. The policy πφ(· |St) may be parameterized by a neural network with parameters φ.

The method iterates between labeling episodes, training the policy net π(·), and training the deep Bayesian Neural Net f (·). An exemplary pseudo code is given below:
<IMG>
<IMG>.

Where Gt refers to the cumulative reward at time step t of the episode that is processed, e.g. <MAT>.

<FIG> depicts an exemplary learning system <NUM> and an exemplary workflow for operating this system.

The learning system <NUM> of the depicted example comprises as components of an oracle <NUM>, a predictor <NUM>, a bulk filter <NUM> and a guide <NUM>. Instructions for executing a function these components provide, may be executed on one or more processors and may be stored on one or more memory. The function of the individual components is described below. Nonetheless the functions may be combined or be separated differently. Interfaces between these functions are adapted accordingly.

It is assumed that images are available for the input points from a memory or from a capturing device. For the following description, xn is an input observation, i.e. an independent variable, e.g. raw image pixel data representing an image and y or yn are related output label, e.g. numeral variables representing a class or descriptive name for the image or the like.

The guide <NUM> is adapted for selecting a representation xn of an image from a plurality of representations of images M. The guide <NUM> in the example selects a plurality of images as input points to be labeled by the oracle <NUM>.

The oracle <NUM> is adapted for determining a first representation y of at least one first label for the image represented by xn. In one aspect the oracle <NUM> receives the plurality of images as input points and provides a set of labeled data for the predictor <NUM> to learn on. The oracle <NUM> comprises the human machine interface to display the image represented by xn and determines the first representation y from input detected by the human machine interface.

The oracle <NUM> is equipped with an interface, for example the human machine interface, to provide images, e.g. to a human user, and receive corresponding labels of images, e.g. from said human user.

The predictor <NUM> in turn provides predictive uncertainties, in the example the variance <MAT> with respect to the means <MAT> to the bulk filter <NUM>.

The predictor <NUM> is adapted for determining a second representation yn of at least on second label, and for determining the uncertainty of the prediction of the second representation yn based on the representation xn of the image, the first representation y and the second representation yn.

The predictor <NUM> is for example a probabilistic predictor, in particular a Gaussian process or the deep Bayesian Neural Net described above.

The guide <NUM> uses an acquisition function, e.g. the deep Bayesian Neural Net described above, that learns with the probabilistic state S and reinforcement feedback, e.g. the reward R = Rimprov + Rdiv, from the oracle <NUM> how to optimally choose the next points. It is thus able to adapt itself flexibly to the data set at hand.

The guide <NUM> communicates to the oracle <NUM> which input points to label next, restarting the cycle.

The system <NUM> is in one aspect adapted for training the predictor <NUM> based on randomly selected representations of images, and afterwards training the predictor <NUM> based on representations of images selected based on the reward R. The predictor is trained on this set using one of the many known learning or inference methods. The randomly selected input is preferably selected small compared to the input selected based on the Reward.

The guide <NUM> is adapted for selecting representations of images depending on the uncertainty and on the reward R.

The oracle <NUM> is adapted for determining the reward R depending on the representation xn of the image and/or the second representation yn.

The oracle <NUM> is adapted to determine the reward R depending on the model, and a comparison of a first measure of model fit and a second measure of model fit, wherein the first measure is determined depending on at least one first parameter, e.g. qold(θold) of the model determined before training the model with the representation xn and depending on at least one second parameter qnew(θnew) of the model determined after training the model with the representation xn. In this aspect, the reward R is calculated from a difference between the first measure and the second measure. The difference is the reward for the improvement in model fit e.g. as described above for Rimprov.

Additionally, the reward aspect for diversity in the labels of the chosen batch may be used for determining the reward Rdiv.

The guide <NUM> is adapted to select a subset of input points from the set of input points for training of the predictor <NUM>. The set of input points refers to the plurality of representations of images in this context.

The guide <NUM> is in one aspect adapted to select the subset depending on a ranked order of input points and depending on the acquisition function that is learned depending on the reward R and the probabilistic state S. The acquisition function is in the example learned by reinforcement learning during the learning process.

The ranking is for example provided by the bulk filter <NUM>. In the example a heuristic bulk filter <NUM> is used to determine the probabilistic state S from the uncertainty.

The bulk filter <NUM> in the example is adapted to apply active learning. The bulk filter <NUM> takes the prediction, e.g. a mean <MAT> and its attached uncertainties, e.g. variance <MAT> for each data point in the unlabeled set, and sorts them with respect to an acquisition score using a heuristic. The heuristic is for example mutual information or variance <MAT>.

The intuition is that all acquisition scores will agree on the large portion of the unlabeled set that is less informative.

In one aspect, starting from the top in the ranking data points are chosen to obtain a set of M chosen data points as actions until the order k*M is reached. This means the input points are ranked in the ranked order and the input points are selected depending on the acquisition function. The input points are selected in particular from the top of the ranked order by selecting every k'th input point, where k is a natural number larger than <NUM>.

In one aspect, the means and variances of the predictions of the Bayesian Neural Net on these M data points are used to construct a multivariate normal distribution. This normal distribution is used as the probabilistic state S in this aspect. The input points for the acquisition function in this aspect correspond to the probabilistic state S.

The guide <NUM> uses this probabilistic state S to learn to choose one input data point to label from the provided set of M actions. As a result of the action it has chosen, the guide <NUM> receives the reward R from the oracle <NUM>.

The learning system <NUM> as described above is adapted to process images of a set of unsupervised samples comprising unlabeled images recursively. In one aspect the learning system <NUM> is adapted to process beforehand without active learning a plurality of randomly selected images and labels of a set of samples comprising labeled images recursively for training of the predictor <NUM>. This means the predictor <NUM> is trained depending on these images and labels.

A computer implemented method of operating a learning system is described below referencing <FIG> contains steps of the method. In one aspect these steps are repeated recursively as described in the pseudo code above for training the predictor <NUM> and for learning an acquisition function. According to one aspect, the plurality of representations of images is recursively processed, wherein in one recursion a set of samples is selected from the plurality of representations of images, labeled, and added to a plurality of input points comprising labeled images, wherein the plurality of input points comprising labeled images is processed for training of the predictor <NUM> and wherein the set of samples is removed from the plurality of representations of images in particular before a next recursion is started. Not all of the steps need to be executed in each recursion and the order of the steps may vary from recursion to recursion.

After the start of the method, in a step <NUM> the representation xn of an image is selected from the plurality of representations of images M. In one aspect, the plurality of representations of images M correspond to input points that are ranked in the ranked order and the subset of input points is selected from the top of the ranked order in particular by selecting every k'th input point, where k is a natural number larger than <NUM>.

In step <NUM> the first representation y of at least one first label for the image is determined, e.g. by the oracle <NUM> as described above.

In step <NUM> a second representation yn of at least on second label is determined. e.g. by the predictor <NUM> as output the deep Bayesian Neural Net f(·) as described above.

In step <NUM> the uncertainty of the prediction of the second representation yn is determined in particular based on the representation xn of the image, the first representation y and the second representation yn.

In step <NUM> the reward R is determined depending on the representation xn of the image and/or the second representation yn. The reward R is preferably determined depending on the model and the comparison of the first measure of model fit and the second measure of model fit for example as described above from Rimprov and optionally from Rdiv. The first measure is determined for example depending on the at least one first parameter, e.g. qold(θold) of the model determined before training the model with the representation xn and depending on at least one second parameter qnew(θnew) of the model determined after training the model with the representation xn.

In step <NUM> another representation of another image is selected depending on the uncertainty and on the reward R.

The acquisition function is determined depending on the reward R and the probabilistic state S. The acquisition function is for example learned by the probabilistic policy predictor, in particular the deep Bayesian Neural Net π(·), depending on the probabilistic state S and based on the reward R.

The steps <NUM> to <NUM> may be recursively repeated until the labeling budget is exhausted, all batches of all episodes of input data is processed or until another criterion is met.

The learning system generally may be operated in a variety of appliances.

In one aspect, depicted in <FIG> the learning system <NUM> is part of a control system <NUM> adapted to determine at least one control signal for an output <NUM>.

The control signal is in particular for a video-based or an image-based system <NUM>. The control signal is determined as output of the learning system <NUM> in response to an input received at an input <NUM>, in particular of at least one image captured for the system <NUM> by a sensor <NUM>. The sensor <NUM> is for example an image, video, radar, LiDAR, ultrasonic, or motion sensor.

The learning system <NUM> in the example comprises a processor <NUM> and a memory <NUM> for storing instructions that when executed by the processor <NUM> cause the processor to operate the learning system according to the method described above.

Claim 1:
A computer implemented method of operating a learning system for training a predictor based on randomly selected representations of images, and afterwards training the predictor based on representations of images selected based on a reward (R) characterized by selecting (<NUM>) a representation (xn) being raw image pixel data representing of an image from a plurality of the representations of images,
determining (<NUM>) by an oracle a first representation (y) of at least one first label representing a class for the image,
determining (<NUM>) by the predictor a second representation (yn) of at least one second label representing a class for the image in a prediction,
determining (<NUM>) by the predictor an uncertainty of the prediction of the second representation (yn) based on the representation (xn) of the image, the first representation (y) and the second representation (yn),
determining (<NUM>) the reward (R) depending on the representation (xn) of the image and/or the second representation (yn), and
selecting (<NUM>) another representation of another image depending on the uncertainty and on the reward (R),
characterized in that for training of the predictor a subset of input points is selected from the plurality of representations of images depending on an acquisition function,
wherein the acquisition function is determined depending on the reward (R) and a probabilistic state (S),
wherein the probabilistic state (S) is determined depending on at least the uncertainty,
wherein the acquisition function is learned by a probabilistic policy predictor, in particular a deep Bayesian Neural Net, depending on the probabilistic state (S) and based on the reward (R).