Patent Description:
Classification of objects and/or of events may be achieved by means of a neural network, sometimes also referred to as artificial intelligence AI. The trend right now is of an increased use of these technologies for classifying objects or events from captured still images or video. These classifying neural networks are often used in applications like character recognition, monitoring, surveillance, image analysis, natural language processing etc. There are many neural network algorithms/technologies that may be used for classifying objects, e.g. Convolutional Neural Networks, Recurrent Neural Networks, etc..

A general training setup <NUM> for training a general neural network <NUM> for classification is shown in Fig. 1a. The neural network <NUM> is fed labeled data <NUM>. The labeled data <NUM> is for example an image of an object to be classified, wherein the image is labeled with the correct class, i.e. the labeled data <NUM> includes the ground truth <NUM> of the image data <NUM> and the image data <NUM> itself. The image data <NUM> is inputted to the classifier and the ground truth <NUM> is sent to a loss function calculator <NUM>. The classifier <NUM> processes the data representing an object to be classified and generates a classification identifier <NUM>. The processing in the classifier includes applying weights to values as the data is fed through the classifier <NUM>. The classification identifier <NUM> may be a feature vector, a classification vector, or a single value identifying a class. In the loss function the classification identifier <NUM> is compared to the ground truth <NUM> using, e.g. a loss function. The result from the loss function <NUM> is then transferred to a weight adjustment function <NUM> that is configured to adjust the weights used in the classifier <NUM>. Then when the classifier <NUM> is fully trained it may be used as depicted in <FIG>, wherein a classification is performed by loading the data <NUM> to be classified into the classifier <NUM>. The data <NUM> to be classified is in the same form as the labeled data used during training, but without the label. The classifier <NUM> then output data <NUM> identifying the class determined for the data inputted.

To achieve a properly trained classifier a very large number of labeled data instances is required, e.g. labeled images. Generally hundreds of thousands of instances of labeled data is required, in many cases even millions. This training data is very cumbersome to generate. For some classifications you may buy large labeled data sets. The most common data sets includes images that are classified. One problem with these existing data sets is that they may not be labeled with the classes you would like to train your classifier to recognize. Another problem with the existing data sets is that they may not use the form of input data that you would like to make your classification on.

The classifier may be any type of neural network, artificial intelligence, or machine learning scheme. In the present description an artificial intelligence includes a neural network, hence when we describes neural networks it also applies to any artificial intelligences including such neural networks. A neural network to be used as a classifier may be implemented in a lot of different ways known to the skilled person. Neural networks sometimes are referred to as artificial neural networks.

<NPL>, describes and categorizes developments in computational intelligence-based transfer learning research and applications.

<NPL> describes and characterizes approaches to transfer-based activity recognition by sensor modality, by differences between source and target environments, by data availability and type of information that is transferred.

<NPL>, presents a reference implementation of an opportunistic activity recognition system and demonstrates the transfer of recognition capabilities from a fused multi-sensor ensemble to an untrained sensing device within the system.

One object of the present invention is to make training of a classifier less labour intensive.

The object is achieved by means of a method for training a classifier according to claim <NUM> and a system arranged to train a classifier according to claim <NUM>. Further embodiments of the invention are presented in the dependent claims.

One advantage of the method of claim <NUM> and the system of claim <NUM> is that it enables training a classifier for a new set of sensor data using an already trained classifier operating on a first set of sensor data, wherein the data of the second set of sensor data represent the same object or event as the first set of sensor data and during a corresponding time period.

In some embodiments the tutor classifier is based on a neural network and according to some embodiments the apprentice classifier is based on a neural network.

Hence, it is to be understood that this invention is not limited to the particular component parts of the device described or steps of the methods described as such device and method may vary. Thus, for example, reference to "a sensor" or "the sensor" may include several sensors, and the like. Furthermore, the word "comprising" does not exclude other elements or steps.

Other features and advantages of the present invention will become apparent from the following detailed description of a presently preferred embodiment, with reference to the accompanying drawings, in which:.

Further, in the figures like reference characters designate like or corresponding parts throughout the several figures.

The present invention relates to classifiers that are being configured to classify objects and/or events by means of processing data from a sensor. In particular the invention relates to having a first classifier already configured to classify objects and/or events from a first set of data provided by a sensor to assist in configuring a second classifier to classify objects and/or events from a second set of data provided by the same or a different sensor. The first classifier may be seen as teaching the second classifier and therefore the first classifier will be referred to as tutor classifier and the second classifier will be referred to as apprentice classifier.

The tutor classifier is configured to classify data from an image sensor, e.g. a sensor for capturing images from visible light or infra-red light. One advantage of having the tutor classifier configured to classify objects and/or events based on data from a visible light image sensor is that there are a lot of labelled data sets already prepared for this type of data and, thus, this will facilitate the training of such a classifier. There may even be pre-configured classifiers that may be used.

The apprentice classifier is trained to classify data from the same sensor as the tutor classifier is trained to classify, but in a different form. According to some examples not according to but relevant to the invention the apprentice classifier is to be trained to classify data from a second sensor. The second sensor may for example be a visible light image sensor, an infra-red light image sensor, a radar sensor, a microphone, a microphone matrix, a sound field sensor, ultra sound, a Lidar sensor, a sensor for laser absorption spectroscopy, a sonar sensor, a radiation sensor, an X-ray sensor, etc..

Now referring to <FIG> depicting an example of a system arranged for making one classifier configure or train another classifier to classify. The system includes a configured and/or trained tutor classification module <NUM> and an apprentice classification module <NUM> to be configured and/or trained. The tutor classification module <NUM> is arranged to receive sensor data from a first sensor <NUM> and the apprentice classification module <NUM> is arranged to receive data from a second sensor <NUM>. The first sensor <NUM> is, in this example, an image sensor delivering two dimensional image of a scene.

The tutor classification module includes an object detector <NUM>, a cropping module <NUM>, and a tutor classifier <NUM>. The object detector <NUM> may be any kind of object detector arranged to detect objects in a two dimensional image that is known to the skilled person. The object detector <NUM> is arranged to detect objects appearing in the image of the scene captured in data received from the first sensor <NUM>. The cropping module <NUM> is arranged to extract a subset of the data from the first sensor <NUM> and transmit the extracted subset of data to the tutor classifier <NUM>. The subset to be extracted is determined by the position and size of the object detected in the object detector <NUM>. The cropping area may be of rectangular shape and be somewhat larger than the object. In addition to cropping the data by extracting said subset of data and sending the subset of data to the tutor classifier the cropping module <NUM> transfer information of the position of the detected object to the apprentice classification module <NUM>. Moreover, in addition to transferring the position of the detected object to the apprentice classification module the cropping module may be arranged to transfer any one of the following features: area of the cropped area, width of the cropped area, height of the cropped area, direction to the cropped area.

The tutor classifier <NUM> processes the cropped image data and generates an indicator that indicates the class of the detected object. The indicator may be a feature vector, a classification vector, a single value or it may be a data set indicating the class and the likelihood of the object being correctly classified, i.e. the confidence value for the indicated class. The class indicator or a filtered version of the class indicator is then transferred to the apprentice classification module <NUM>. The filtered version may be a class indicator in which a limited number or a single one class indicator having the best confidence value/s are transferred to the apprentice classification module. Moreover, in some embodiments a class indicator may be stopped from being transferred if the confidence value is below a predetermined threshold.

The apprentice classification module <NUM> includes a crop translator <NUM>, a cropping module <NUM>, an apprentice classifier <NUM>, a loss function calculator <NUM>, and a weight adjuster <NUM>. The crop translator <NUM> is arranged to translate the information relating to the cropping performed in the cropping module <NUM> in the tutor classification module102 into a corresponding position represented in the data from the second sensor <NUM>.

This translation is illustrated in <FIG> in an example not according to but relevant to the invention in which <FIG> depicts an image <NUM> captured by an image sensor, e.g. visible light image sensor or infra-red image sensor and <FIG> depicts a representation <NUM> of sensor data from a radar sensor at a corresponding time period as the capturing of the image in <FIG>. The image in <FIG> shows two persons <NUM>,<NUM> in the captured scene. An object detector, such as the one <NUM> in the tutor classification module <NUM> will probably detect both persons <NUM>,<NUM>, however, the system should only classify one object at a time. Let us assume that the classifying module is in the stage to process the person <NUM> to the right and the hatched area <NUM> indicates that this is the detected object and the cropping module <NUM> has determined the area to crop, indicated by the hatched area <NUM>. The information relating to the cropping determined in the cropping module <NUM> is then transferred to the crop translator <NUM> of the apprentice classification module <NUM>, as previously mentioned. The crop translator <NUM> will in this case translate the position to a corresponding direction in the data of the radar sensor, indicated by the hatched sector <NUM> in <FIG>. The angle α may be determined from the width of the cropping area <NUM> in <FIG> or may be determined on an independent object detection for the radar data in the determined direction.

Now returning to <FIG> and the description of the features of the apprentice classification module <NUM> in an example not according to but relevant to the invention. The cropping module <NUM> is arranged to extract a subset of data from the second sensor <NUM>. The selection of the subset of data to be extracted is at least partially determined using data from the crop translator. The apprentice classifier <NUM> is arranged to receive the cropped data from the cropping module <NUM> and generate a value or a vector of values indicating an estimated class for the corresponding object in the data from the second sensor <NUM>. The loss function calculator <NUM> is arranged to evaluate the result from the apprentice classifier <NUM> in comparison to the ground truth provided from the tutor classifier <NUM>. In machine learning the ground truth is, for the example of classifiers, the class of the object which is provided as the correct class for the particular image, i.e. by manual labelling of objects. This evaluation results in an output that is used for adjustment of the processing in the apprentice classifier <NUM>, e.g. by adjusting weights in various process steps. The weight adjuster <NUM> is arranged to use this evaluation result from the loss function <NUM> and generate the appropriate weight adjustments for the apprentice classifier <NUM> in order to get closer to a correct classification.

Now referring to <FIG> and <FIG> showing an example process for the training of a classifier in an example not according to but relevant to the invention. <FIG> shows a portion of the training process relating to the processing of the data from the first sensor <NUM> and generating of data for use in training the classification of the data from the second sensor <NUM>. <FIG> shows a portion of the training process relating to the processing of the data from the second sensor <NUM> and the adjustment of weights in the apprentice classifier using inputs from the processing of data from the first sensor <NUM>. A training loop starts with the input of two data sets from the two sensors <NUM>, <NUM>. Accordingly the process receives data from the first sensor <NUM>, step <NUM> in <FIG>, and from the second sensor <NUM>, step <NUM> in <FIG>. The first sensor <NUM> being an image sensor as described above. The data registered by the two sensors <NUM>, <NUM>, are registered by each sensor <NUM>, <NUM>, at corresponding points in time, i.e. the data is captured at the two sensors, respectively, at a short temporal distance or for overlapping time periods in order to make it likely that the two sensors capture data relating to the same object, or event.

Now let us take a look at the processing of the data from the first sensor <NUM> in <FIG>. After the image sensor data have been received, when it has been captured by the first sensor <NUM>, step <NUM>, the process continues by detecting an object in the data from the first sensor <NUM>, step <NUM>. The detected object is then analysed for cropping and a subset of data representing the detected object is selected, i.e. cropped, step <NUM>. Further, data relating to one of or any combination of the following features relating to the detected object is determined and transferred, step <NUM>, for use in a cropping operation to be performed on the data from the second sensor <NUM> in the apprentice classification module104. The features are: a position of the detected object, an indication of the area occupied by the object, the width of the cropped area, the height of the cropped area.

The cropped data from the first sensor is then inputted to the tutor classifier <NUM>, step <NUM>, and a classification identifier is generated, step <NUM>, by the tutor classifier <NUM>. The classification identifier relating to the detected object is then sent to the loss function <NUM> in the apprentice classification module <NUM>, step <NUM>. The process then returns to step <NUM> for receipt of new image data. The repetition of a training loop may be performed as fast as possible. In some embodiment, the training may be limited to the classifications having good enough confidence values. In some embodiments the training is performed on live data and in some embodiments it is performed on recorded data that registered or captured at points in time that suggest that the data represents the same event or object. Continuing the description of the training process by looking at the processing of the data from the second sensor in <FIG> in an example not according to but relevant to the invention, the sensor data from the second sensor is received, step <NUM>, as previously mentioned. Then the features relating to the cropping and object detection, described in connection with the description of <FIG>, is received from the tutor classification module <NUM>, step <NUM>. The features are translated to values that are relevant for the type of data delivered by the sensor in order to enable correlation between positions in the data sets from the two sensors <NUM>, <NUM>, step <NUM>. Cropping parameters are then determined using at least some translated values from the tutor classification module <NUM>, step <NUM>. The data from the second sensor <NUM> is then cropped using these cropping parameters, step <NUM>, and the cropped data is inputted into the apprentice classifier <NUM>, step <NUM>. The apprentice classifier <NUM> then generates a classification identifier of the object in the processed cropped data, step <NUM>. The classification identifier generated by the apprentice classifier <NUM> is then received at the loss function calculator <NUM>, where the loss function is calculated from the classification identifier received from the apprentice classifier <NUM> and the classification identifier received from the tutor classifier <NUM>, step <NUM>. The calculated loss function is then used for generating weight adjustments for the apprentice classifier, step <NUM>. The weight adjustments are then introduced in the apprentice classifier in order to improve the classification accuracy, step <NUM>. The process then returns to step <NUM> for receipt of new image data.

As described above in relation to some examples not according to but relevant to the invention, the image data captured by the first sensor <NUM> and the data captured by the second sensor <NUM> should both include data representing the same object at the same moment in time, e.g. the data should be captured at essentially the same point in time or covering overlapping time periods.

In some examples not according to but relevant to the invention of the invention the apprentice classification module may be arranged to process a sequence of sensor data sets, i.e. a plurality of sensor data sets captured sequentially at different time points. Examples of this type of sequential data are, motion video, audio, radar, sequences of still images, etc..

An example of an apprentice classification module arranged to handle these situations, i.e. processing a sequence of sensor data sets, are depicted in <FIG>. The apprentice classification module <NUM> includes a crop translator <NUM> similar to the one described in <FIG> and a cropping module <NUM>. The cropping module may be arranged to initially find the object or the event in the sensor data by using the translated crop data from the tutor classification module and then track the object or the event initially found in order to crop relevant data. When the data relates to motion video the cropped data could be a subset of data representing a tighter rectangle surrounding the object of interest. For some sensor data cropping of data makes no sense and will in those cases be skipped, e.g. sensor data from a single microphone or other sensor data having no spatial information relating a data point spatially to another data point. The apprentice classification module further includes a buffer <NUM>, a feature extractor <NUM>, the apprentice classifier <NUM>, the loss function <NUM>, and the weight adjuster. The buffer <NUM> is arranged to buffer the sequence of cropped data from the sensor, or the sequence of data if no cropping is performed. The feature extractor <NUM> operates on the data in the buffer to find and extract features from the sequence, e.g. angular movement, change in distance, object speed, changes in shape of the object, movement characteristics, etc. These features from the feature extractor will then be used as inputs to the apprentice classifier which will be trained on these features in order to identify them as belonging to a class identified in the tutor classification module <NUM>. The apprentice classifier <NUM>, the loss function <NUM>, and the weight adjuster are very much the same devices and/or modules as described in connection with <FIG>.

Claim 1:
Method for training a classifier, the method comprising:
generating, by an image sensor (<NUM>, <NUM>) registering visible light and infra-red light, first image data by registering visible light, the first image data including data representing an object or an event in a monitored environment,
generating, by the image sensor (<NUM>, <NUM>), second image data by registering infra-red light, the second image data representing a corresponding time period as a time period represented by the first image data,
detecting at an object detector (<NUM>) an object appearing in the first image data,
cropping at a first cropping module (<NUM>) from the first image data a subset of data representing the detected object, a cropping area thereof being determined by the position and size of the detected object,
determining at said first cropping module (<NUM>) features of said cropping area, said features comprising a position of the detected object, an indication of the area occupied by the object, a width of the cropping area and a height of the cropping area,
inputting (<NUM>) to a tutor classifier (<NUM>) the cropped data of the first image data,
generating (<NUM>) a classification of the object or event in the tutor classifier (<NUM>),
receiving (<NUM>) the second image data at an apprentice classifier training process,
translating (<NUM>) at a crop translator (<NUM>) said features of the cropping area into a corresponding position represented in the second image data,
determining (<NUM>) at a second cropping module (<NUM>) cropping parameters at least partially based on the translated features, wherein the second image data are then cropped (<NUM>) using said cropping parameters,
receiving (<NUM>) the classification generated in the tutor classifier (<NUM>) at the apprentice classifier training process,
training (<NUM>, <NUM>, <NUM>) the apprentice classifier (<NUM>) in the apprentice classifier training process using the cropped data of the second image data as input and using the classification received from the tutor classifier (<NUM>) as a ground-truth for the classification of the second image data.