Patent ID: 12249184

DESCRIPTION OF EMBODIMENTS

Embodiments of the present invention will now be described by reference to the above-referenced figures.

InFIG.1, four images11,12,13,14from a user-facing camera are shown. The camera images have been processed with a prediction module. The prediction module determines associated information11i,12i,13i,14ifor each processing image. The associated information comprises determined likelihoods of predetermined activities occurring in a processed image. The predetermined activities comprise a set with a predetermined number of activities. InFIG.1, the predetermined activities are: “smoking”; “eating”; “drinking”; “talking on the phone”; and “no activity”. For each processed image, the activity with the highest determined likelihood is determined to be the activity occurring in the processed image. As examples, from analysis of the associated information images11and12have been determined to show smoking, whereas images13and14have been determined to show drinking.

It will be appreciated that the term activity includes any uniquely identifiable behavior pattern presenting temporal correlation, for example, falling asleep or a certain pattern of sudden sickness.

The sensitivity of the user-facing cameras used in embodiments of the present invention need not be limited to any specific range of wavelengths, but most commonly will be sensitive to near infra-red, NIR, light and/or visible, RGB, light. In some embodiments, RGB or intensity image information is provided directly from the camera image sensor, whereas in other embodiments, an event camera of the type disclosed in in Posch, C, Serrano-Gotarredona, T., Linares-Barranco, B., & Delbruck, T. “Retinomorphic event-based vision sensors: bioinspired cameras with spiking output”, Proceedings of the IEEE, 102(10), 1470-1484, (2014), European Patent No. EP3440833, PCT Application W02019/145516 and PCT Application W02019/180033 from Prophesee can be employed and image information can be generated from event information provided by the camera, as disclosed in U.S. patent application Ser. No. 16/904,122 filed on 17 Jun. 2020 entitled “Object Detection for Event Cameras”, (Ref: FN-662-US), the disclosure of which is herein incorporated by reference.

The user-facing camera will generally be in the form of a camera module comprising a housing for a lens and a sensor, the lens serving to focus light onto the sensor. The camera module may also have electronics to power the sensor and enable communication with the sensor. The camera module may also comprise electronics to process acquired images. The processing can be low level image signal processing, for example, gain control, exposure control, color balance, denoise, etc. and/or it can involve more powerful processing for example, for computer vision.

Some embodiments of the present invention are used in a driver monitoring systems, OMS. The user-facing cameras in these systems are typically configured to communicate data along a vehicle system bus (BUS), for example, a controller area network (CAN) bus. Data from other electronic components, such as additional cameras, may also be transmitted to the OMS along the BUS. In other embodiments, the OMS can be a separate system that can transfer and record data independently of the BUS.

Embodiments of the present invention predict activity not just in a single still image but images in a stream of images being acquired in real time. Thus, as well as having access to a given image, embodiments can access a sliding window sequence of previously acquired images which can help to inform the prediction of the activity in a given image.

By using a sequence of image, the confidence in the determined activity can be higher. As an example,FIG.2shows a sequence of seven images. Images20-25clearly show a user smoking. Whether the user is smoking is unclear in image26due to the hand of the user being close to the camera and far from the user's mouth. If the user's activity is determined only from imaged26the determination would be difficult, and the result may not be smoking. However, if the user activity is determined from the sequence of seven images the determination of smoking would be relatively straight forward.

Embodiments of the present invention use a prediction module that can take any sequence with any number of images. In the example, sequential sequences of five images are processed. There are three such sequences shown inFIG.2. The first sequence27runs from image20to image24. The second sequence28runs from image21to image25. The third sequence29running from image22to image26.

A configuration of a prediction module3is shown inFIG.3. In a first step, a sequence30of five sequential images (IM[i] to IM[i+4]) is passed to the prediction module. The images are first conditioned by cropping and/or resampling to a preferred format, such as an image that has dimensions 98×98. Each input image is then processed in a respective feature extraction module, FE,31to extract i.e. encode features from the processed input image. Each FE comprises a convolution neural network, CNN, with convolution, batch normalization and pooling layers. The output of the FEs are five feature blocks32. In some embodiments, each feature block is a vector that has dimensions 16×48×48. Up to this stage in processing the processing is image specific i.e. it has only involved independent spatial analysis of the input images.

In the example, the five feature blocks are concatenated into a concatenated block37that has dimensions 5×16×48×48. However, it will be appreciated that techniques other than concatenation can be used to combine the feature blocks including multiplication, convolution or aggregation. In any case, the concatenated block37is then processed with a time-based module, TBM,38. The TBM performs temporal analysis of the input images. The TBM38comprises a CNN with convolution, batch normalization, pooling layers and one or more fully connected layers with a final fully connected layer comprising 5 nodes, each for a respective activity. The output of the TBM can be normalized by a SoftMax process to produce the sequence associated information39. The sequence associated information39provides the likelihoods of each activity of the predetermined activities occurring in the processed sequence of images30.

The most likely sequence activity39′ in the sequence associated information39represents the most likely activity in the processed sequence of images30. The most likely sequence activity39′ may be displayed to the user, recorded, or passed to another system for processing and/or recording.

The FE processes each image of the sequence of images30independently. Any length of sequence of images may be processed provided the resultant feature blocks when concatenated are of a manageable size for the TBM. Longer sequences may improve the confidence in the determined activity provided the length of the sequence does not extend past the end of the activity. However, by using shorter sequences, e.g. sequences of 4, 5, 6 or 7 images, the processing demands will be reduced allowing the prediction module to operate using less energy and/or operating with a reduced latency.

In some embodiments, the images are not processed in parallel. For example, a single FE can be used to process one input image into a feature block, store the produced feature block, and then proceed to process another image. This process can be repeated until all the required input images have been processed. The stored feature block can then be concatenated and processed as described above.

In some applications, such as processing frames from a user-facing camera, the sequences of images to be processed are typically sequentially streamed to the prediction module. In such applications, the prediction module can use the intermediate results from a previous instance of FE processing to reduce the processing required. So for example the output of FE processing of frame IM[i] is available for use as the output of processing of frame IM[i+1] for processing a subsequent frame similar to the technique described in U.S. Patent Application No. 63/017,165 filed on 29 Apr. 2020 and entitled “Image Processing System”, (Ref: FN-661-US) the disclosure of which is herein incorporated by reference.

As an example, considerFIGS.2and3and consider the processing of the second sequence28with the multi-class prediction module3after the first sequence27has been previously processed. During the processing of the first sequence27, the FE31has produced feature locks for images20to24. By storing these feature blocks, the system can reuse these results when processing the second sequence28. In this way, the processing of the second sequence only requires the FE31to process image25into a feature block. Effectively, a sliding window can be used to reuse previous processing and therefore minimize the energy and time taken to determine an activity from a sequence of images. The use of this sliding window technique also means the processing required to produce the blocks is independent of the length of the sequence of images.

In the processing of the concatenated blocks by the TBM, the information in the feature blocks is treated equivalently across the feature blocks. The prediction module3therefore treats each input image of a sequence of images as equivalent in importance when determining the activity in the sequence of images. However, this is not necessarily correct as often some images in a sequence of images are much stronger or weaker indicators of a sequence activity than others.

For example, inFIG.4, images41to43, which show a user holding a cigarette near a mouth, are good indicators of smoking. The smoking activity determination is high for images41to43as relevant target object are close together i.e. the target object of a cigarette is close to the target object of a mouth. Images41to43would therefore be highly relevant when determining if the user in the sequence of images shown inFIG.4is smoking. In contrast, in image44, the cigarette is far from the user's mouth. Judged in isolation, image44may be considered to simply show a user holding a cigarette rather than smoking. The determination of the smoking activity in image44will therefore have a lower likelihood. When determining the activity of the user (smoking) in the sequence of images shown inFIG.4, image44is therefore less important to the sequence determination than images41to43. Therefore, even though each image in the sequence shown inFIG.4is determined to have the same activity, when determining the activity of this sequence, the images41to43are more important.

To account for the variation in the importance of images in a determination of the activity in a sequence of images, the prediction module3can be improved. The operation of the improved prediction module is explained in the steps shown inFIG.5. In a first step50, a sequence of images is acquired. In some embodiments, these images are a series of frames that are acquired from a user-facing camera of an OMS system. The acquired images are then processed with a FE to encode51the images into feature blocks. The determination of the feature blocks with the improved prediction module can be the same as the process the prediction module3shown inFIG.3uses to produce its feature blocks32.

The feature blocks are then processed52to determine the likelihoods the predetermined activities for each feature block as well as block specific activity likelihoods, a weighting is determined for the most likely activity for each block. The determined weighting is associated with the likelihood that the image indicates the most likely activity.

Using the images fromFIG.4as an example, the weightings determined for images41to43may be 0.95 and the weighting determined for image44may be 0.6. The determined weightings are however an indication of image importance and need not correspond to other measurements. In other words, the determined weightings may correspond to image260specific activity likelihoods or they may take other values. The determined weightings are then compared53and adjusted to normalize the weightings. In some embodiments, a SoftMax process is used to process the determined weightings into normalized weightings.

Each of the normalized weightings is then combined with the associated feature block to form weighted blocks. The combination can be achieved by multiplication, concatenation, or any process that combines the information in the feature block with the associated normalized weighting.

In the embodiment, the normalized weighted blocks are then concatenated54in the order of the sequential images. However, it will be appreciated that in variations of the embodiment the normalized weighted blocks could be combined using other techniques including multiplication, convolution or aggregation. The concatenated weighted blocks are then processed55with a TBM to determine the likelihoods of predetermined activities in the sequence of image. The processing of the concatenated weighted blocks with the improved prediction module can be the same as the process the prediction module3shown inFIG.3uses to produce its associated information39from its concatenated blocks37.

The activity with the highest likelihood of the determined likelihood is marked as the most likely sequence activity. The most likely sequence activity may be displayed to the user, recorded, or passed to another system for processing and/or recording.

In an optional step, the predetermined activities and weighting for at least one feature block are compared to the most likely sequence activity, to validate or trigger updating of most likely sequence activity, step56. As an example, a most likely sequence activity is compared with the likelihood for the same activity determined from at least one feature block and a difference is taken to indicate that the most likely sequence activity has finished, is unusual in some manner, or that the user is about to transition to another activity. This comparison serves as a useful validation of the most likely sequence activity and may also trigger the updating or recalculation of the most likely sequence activity. The comparison therefore improves the confidence or accuracy in the activity determined for the sequence of images.

Two parts of an improved prediction module6are shown inFIGS.6and7. InFIG.6, the feature blocks62are produced by processing input images of a sequence60with respective FEs61. The feature blocks can be produced in the same way as described above. Each feature block62is then processed with an image-based module, IBM63. The IBM comprises a CNN containing convolution, batch normalization, pooling layers and a final fully connected layer. The output of each IBM63is a weighting value64and the likelihoods of the predetermined activities65. The IBM63forms its predictions based only on the processed feature block62and ultimately only from the image from which the FE formed the processed block62. The IBM63does not take any temporal analysis into account i.e. the difference between images is ignored by the IBM63. The IBM therefore helps to optimize the spatial analysis and ultimately to make later processing easier.

Again, each feature block62is processed independently by an IBM63but, in some embodiments, the blocks are not processed in parallel. For example, the feature blocks may in a serially processed in a single IBM63. In other words, the sliding window technique described above can be used.

The output of the IBM63is the image specific information. The image specific information comprises likelihoods of the predetermined activities65and the weighting64. The image specific information is specific to the input block62, which is specific to an image of the input sequence of images60that formed the input block62. The weighting64relates to the highest65′ of the likelihoods of the predetermined activities. Each weighting may be considered as representing a measure of how well a particular image represents the most likely image activity65′.

After the processing inFIG.6, each image will have image specific information comprising the likelihoods of the predetermined activities65including a most likely image activity65′, and a weighting64associated with the most likely image activity65′.

The processing inFIG.7starts with the weightings64produced by the process shown inFIG.6. The weightings64are normalized using a SoftMax process75to produce normalized weightings76. The feature blocks62are then multiplied by the corresponding nominalized weighting76. The weighted blocks are then combined, in this case using concatenation. The concatenated weighting blocks77are then processed with a TBM78to determine the likelihoods of predetermined activities in the sequence of image.

The processing of the concatenated weighted blocks77by a TBI78in the improved prediction module6occurs in a similar manner to the processing of concatenated blocks37by the TBM38of the prediction module3. Specifically, the output of the TBM78is normalized by a SoftMax process to produce the sequence associated information79. The sequence associated information79provides the likelihood of each activity of the predetermined activities occurring in the processed sequence of images60. The activity with the highest determined likelihood is marked as the most likely sequence activity79′ in the processed sequence of images60. The most likely sequence activity79′ may be displayed to the user, recorded, or passed to another system for processing and/or recording.

In an optional step, the image specific information64+65for at least one image is compared to the most likely sequence activity79′. The result of this comparison can be used to validate or trigger an update or recalculation of the most likely sequence activity79′.

Alternatively, if the most likely image activity65′ for at least one image in a sequence differs from the most likely sequence activity, a further decision needs to be taken. An error or warning flag can be raised, or an “unknown activity” can be recorded as the most likely sequence activity. If the number of images with a most likely image activity differing from the most likely sequence activity is greater than half the sequence length, the most likely sequence activity may be updated. The most likely sequence activity may be updated to match the most likely image activity for the most recent image, or to the most common most likely image activity across all the images. Alternatively, the improved prediction module may repeat the determination of the most likely sequence activity.

In addition to, or as an alternative to, validating the most likely sequence activity79′, in some embodiments, the image specific information and/or most likely image activity65′ may be displayed, recorded and used in addition to the most likely sequence activity79′. This is especially useful for fast acting or safety systems in vehicles, in which it is useful to know the most recently determined activity of the user alongside the activity determined over a recent period.

In some applications of embodiments of the present invention, the output of the most likely sequence activity79′ and/or the image specific information can be used to modify the function of other systems. For example, a vehicle may activate driver assistance functions because an improved prediction module indicates that the driver is currently drinking or talking on the phone. In another example, the vehicle may deactivate air flow if the car is cold and the improved multi-class prediction module indicates that the driver has just stopped smoking.

The improved prediction module6comprises a machine learning module. In some embodiments, this module is trained using the process shown inFIG.8. The training process comprises acquiring80labelled training data. Typically, the labelled training data is in the form of a plurality of sequences of images, with each image in each sequence having a label that indicates the activity occurring in the image. The labels of the images serve as the source of truth for training the module6. Labelled images may be obtained from real images that have been manually labelled and/or automatically labelled using a separate labelling system operating according to known methods. Labelled images may also comprise generated images that have been created by processing a labelled real image. For example, at least one labelled image may be processed into a sequence with additional images, with each additional image in the sequence being assigned a label derived from the labels of the at least one labelled image. The use of generated images helps reduce the number of labelled images that need to be acquired for training.

The improved prediction module is a predictive model that may be considered to comprise two models: a first model to model81spatial features and a second model to model82temporal features. The first model comprises the FE and the IBM components of the improved multi-class prediction module. The second model comprises the TBM of the improved prediction module6.

The labelled trained data is processed with the improved prediction module lo produce training results in the manner described above. The training results comprise image specific information and sequence associated information The training results are then tested83against the labels of the training data. The difference between the image specific and sequence associated information and the respective labels of the training data is used to update the predictive model and thereby train the improved prediction module. The training therefore has two constraints: minimizing the difference between the labelled and predicted image specific information; and minimizing the difference between the labelled and predicted sequence associated information.

An example of a loss function that may be used in the training of the improved prediction module to update the FE, IBM and TBM components is:
LOSS[i]=(CrossEntropy(TBcrR[if,TBPRED[i])+I]JCrossEntropy(FBcrR[j],FBPRED[j])),where:TBGTR [i] is the labelled activity for a training sequence i;TBPRm [i] is the predicted activity for a training sequence i;FBGTRU] is the ground-truth activity class for image}; andFBPRmU] is the predicted activity class for image j.

Training in this manner is advantageous as it ensures the FE and the IBM are well optimized using the image specific information. The optimized FE and IBM produce weightings reflecting the importance of the images in a sequence to the determination of the activity in the sequence. As these accurate weightings are input to the TBM, this means the optimization of the TBM no longer has to try and account for the variation in importance in input data itself. Consequently, the TBM is easier to optimize.

As the FE and the IBM are optimized using the image specific information, this training method also helps isolate the predictions of the most likely image activity from the most likely sequence activity. This isolation means the most likely sequence activity for a sequence is less likely to be biased in the same way as the most likely image activity for images from the sequence. This helps ensure that the validation of the most likely sequence activity for a sequence using the most likely image activity for images from the sequence is a robust validation.

While the above embodiment has been described in terms of determining an activity of a user in a scene, it will be appreciated that in variations of the disclosed embodiment, any activity which might be occurring in a scene may be determined.

In the embodiment, a percentage likelihood for an activity is provided by the predictive models (IBM63and TBM78) for each image and for each sequence of images. This can be more or less granular as required. So, for example, the final output79′ of the module can be a simple indication of a single predicted activity at any given time. Alternatively, the predictive models (IBM63and TBM78) may also determine the likelihoods of a plurality of predetermined confidences for that activity (e.g. High confidence, Medium confidence, Low confidence, Uncertain confidence).