Patent Description:
Referring now to <FIG>, vehicle occupant monitoring systems typically comprise an image acquisition device <NUM> mounted in a cabin <NUM> of a vehicle in which one or more occupants <NUM>, <NUM> are seated within the field of view <NUM> of the camera. In <FIG>, two occupants <NUM>, <NUM> are shown in front seats of the vehicle, with the image acquisition device <NUM> located forwardly and facing rearwardly. It will be appreciated that other occupants of the vehicle including back seat passengers (not shown) may also appear within the field of view <NUM> of the camera <NUM>.

<FIG> shows a typical image <NUM> acquired by the image acquisition device <NUM> where it will be seen that occupant <NUM> is the driver of the vehicle, while occupant <NUM> is a passenger.

It is desirable and indeed future vehicle safety standards are based on being able to identify the activity of the occupants at any given time, especially to determine if any of these endanger the safe driving of the vehicle, for example, in the case of semi/fully autonomous cars, to determine if the person sitting behind the wheel is able to take control of the vehicle (if needed).

<NPL> discloses a driver monitoring system where in a first stage, body key points of the driver are localised and in a second stage, image regions around the localized hands are extracted. These regions and the determined 3D body key points are used as the input to a recurrent neural network for driver activity recognition.

<NPL> utilizes a feature pyramid network (FPN) backbone with multiple detection heads for driver/passenger state detection tasks.

<NPL>, discloses using of 3D hand poses to recognize first-person dynamic hand actions interacting with 3D objects.

<CIT> discloses an operator behaviour recognition system comprising hardware including at least one processor, a data storage facility in communication with the processor and input/output interfaces in communication with the processor, the hardware being configured to implement a set of convolutional neural networks including: an object detection group into which at least one image is received from an image source for detecting at least one object in the image and to delineate the object from the image for further processing, at least one of the objects being detected being a face of a person; a facial features extraction group into which the image of the person's face is received and from which facial features from the person's face are extracted; and a classifier group which assess the facial features received from the facial feature extraction group in combination with objects detected by the object detection group to classify predefined operator behaviours.

<NPL>, Master's Thesis Utrecht University discloses a computer implemented method for detecting if a driver is using his handheld phone, the method comprising receiving images of a driver region, detecting body regions including hands and face of the driver, detecting a phone, and classifying the images into risky and non-risky behavior based on the relative positions of the body regions.

<CIT> discloses a computer implemented method for determining an activity of an occupant of a vehicle comprising: capturing sensor data of the occupant using at least one sensor; determining respective two-dimensional or three-dimensional coordinates for a plurality of pre-determined portions of the body of the occupant based on the sensor data; determining at least one portion of the sensor data showing a pre-determined body part of the occupant based on the sensor data and the two-dimensional or three-dimensional coordinates; and determining the activity of the occupant based on the two-dimensional or three-dimensional coordinates and the at least one portion of the sensor data.

According to a first aspect of the present invention, there is provided a method according to claim <NUM>.

In further aspects, there are provided a vehicle occupant monitoring system and a computer program product for implementing the method according to the invention.

Embodiments of the invention identify an occupant state which may be closely related to the occupant handling an object, without explicitly identifying the object.

An embodiment of the invention will now be described, by way of example, with reference to the accompanying drawings, in which:.

Referring now to <FIG>, a vehicle occupant monitoring system according to an embodiment of the present invention acquires successive images <NUM> from an image acquisition device <NUM> mounted in a cabin <NUM> of the vehicle.

The image acquisition device <NUM> can comprise any of: a visible wavelength color camera with RGB pixels, an infra-red (IR) camera, a camera including a Bayer array of pixels including R,G,B and IR sensitive pixels or a thermal camera. Alternatively, the camera could comprise a monochrome, for example, green sensitive, camera. In a still further alternative, the image acquisition device <NUM> could comprise an event camera of the type described in PCT Application No.<CIT> (Ref: FN-<NUM>-PCT), where event information is accumulated in a tensor before being provided to downstream classifiers. Again, the pixels of such a camera can be sensitive to any of visible, infra-red or thermal radiation. As disclosed in <CIT> (Ref: FN-<NUM>-PCT), dedicated hardware can be provided to produce such an image tensor from event camera information. In a still further alternative, the image acquisition device <NUM> could comprise a combined frame based and event camera such as a Davis <NUM> camera available at iniVation.

It will also be appreciated that it is possible to swap between IR images and visible wavelength image source in accordance with a measured ambient light level of the vehicle cabin.

In a typical implementation, the camera comprises a 1600x1300 array of pixels. However, this is typically not sufficient to be able to explicitly identify the various objects which an occupant may have in their hands, and which can signal their current activity - especially for back seat occupants who appear significantly smaller than front seat occupants and so would require even greater camera resolution in order to classify their activity through explicitly identifying objects in their hand. Also, for example, while identifying a cigarette can indicate that an occupant is smoking, identifying any one of a number of foodstuffs can indicate that an occupant is eating, and identifying a phone may indicate that an occupant is either talking on the phone without using hands-free mode or texting or otherwise interacting with their phone, this approach may often not produce satisfactory or reliable results. For example, objects may be held in an occupant's hand in a manner where most of the object is occluded.

In embodiments of the present invention camera image information is downsampled to QVGA resolution, i.e. 320x240, before being provided to respective a body detector <NUM>, hand detector <NUM> and face detector <NUM>.

The body detector <NUM> is configured to identify one or more regions of the image bounding a human body. In <FIG>, the region <NUM> has been identified as bounding the driver <NUM> and the region <NUM> has been identified as bounding the passenger <NUM>. Similarly the face detector <NUM> identifies regions <NUM> and <NUM> bounding the faces of the passenger <NUM> and driver <NUM> respectively. Finally, the hand detector <NUM> identifies hand regions <NUM> and <NUM>.

It will be appreciated that any suitable forms of detectors can be employed ranging from classical Haar type detectors to neural network based detectors. In the case of neural network detectors, networks with similar or identical structures, but trained to use different weights can be used for each of the body detector <NUM>, hand detector <NUM> and face detector <NUM>. These detectors can be run serially, where the intermediate or final output of one may boost the performance of another, or in parallel in particular using a multi-core neural network processor of the type disclosed in <CIT> (Ref: FN-<NUM>-EP) and <CIT> (Ref: FN-<NUM>-EP).

Typically, such networks will produce a set of confidence levels for bounding boxes at different scales and proportions at respective locations across the image. In some embodiments, up to <NUM> mutually exclusive bounding boxes <NUM> with confidence levels above a threshold amount will be chosen as body regions. Thus, for each body region, an x,y location of its origin within the image, along with a width and height and a confidence level for the body region will be provided from the body detector <NUM>.

Similarly, up to <NUM> mutually exclusive bounding boxes with confidence levels above a threshold amount will be chosen as face regions <NUM>. Thus, for each face region, an x,y location of its origin within the image, along with a width and height and a confidence level for the face region will be provided from the face detector <NUM>.

Finally, up to <NUM> mutually exclusive bounding boxes <NUM> with confidence levels above a threshold amount will be chosen as hand regions. Again, for each hand region, an x,y location of its origin within the image, along with a width and height and a confidence level for the hand region will be provided from the hand detector <NUM>.

It will be understood that where it is possible for there to be fewer or more occupants of a vehicle, the maximum number of body, face and hand boxes can be varied accordingly.

In the embodiment, each bounding box <NUM> for a body region is provided to a body skeleton detector <NUM>. When such a detector is implemented with a neural network, it typically requires its input image to be normalised. In the illustrated embodiment, the input image map for the skeleton detector <NUM> is 192x192 pixels. Given that the input image for the body detector <NUM> comprises only 320x240 pixels, it is most likely that a detected body region <NUM> will be smaller than this size and typically will have a width less than its height. However, rather than scaling and resizing the detected bounding box <NUM> directly, in the embodiment, the system first re-shapes the body region, by expanding the width of body region to provide an expanded square region <NUM> centered around the originally detected body region <NUM>. The re-shaped body region can then be expanded to comprise a region <NUM> of 192x192 pixels corresponding to the input map size of the skeleton detector <NUM>. It will be appreciated that the re-shaping and the expansion may mean that the boundary of the body region extends beyond the boundary of the input image <NUM> - such regions <NUM> can be simply treated as blank. Nonetheless, the increased image area provided around the detected body region <NUM> can provide the skeleton detector <NUM> with improved context with which to make its inference.

Referring now to <FIG>, conventional skeleton detectors <NUM> typically provide at their output a respective heat map for each joint to be identified with the peak value in each heat map generally indicating the location of the associated joint. These results can be filtered and analysed to the extent required, but in essence, an image crop <NUM> from the input image <NUM> corresponding to a re-shaped and expanded bounding box from the body detector <NUM> can be used to provide a set of x,y locations within the input image for each one a number of joints of the body region of interest. Note that, as per <CIT> (Ref: FN-<NUM>-US), the x,y locations for joints do not necessarily need to fall within the area of the re-shaped and expanded bounding box and some locations for joints can be inferred, even if they are not visible within the image crop <NUM>.

As will be seen, in the present embodiment, two joints in particular are of interest and these are the left <NUM> and right <NUM> wrists of the occupant.

Referring back to <FIG>, in the embodiment, a hand filter <NUM> looks at the bounding boxes <NUM> for each hand identified by the hand detector <NUM> and associates a hand bounding box <NUM> nearest and/or most overlapping a wrist location <NUM>, <NUM> as the left or right hand region corresponding to the body.

It should also be noted, as shown in <FIG>, that rather than searching the input image <NUM> for all hand regions in one pass, it is also possible to use the expanded image crop <NUM> around each body region <NUM> as an input image for respective instances of the hand detector <NUM>. Now any hand regions that are detected within each input image crop <NUM> are automatically associated with the body from which the image crop <NUM> is derived. Again nonetheless, a hand bounding box <NUM> nearest and/or most overlapping a wrist location <NUM>, <NUM> is designated as the left <NUM>-L or right <NUM>-R hand region corresponding to the body.

Similarly, a face region <NUM> which most overlaps with a body region or which is closest or most overlaps with the eye sockets or chin locations identified by the skeleton detector <NUM> for a given body region can be treated as the face region corresponding to the identified body.

Pairs of left and right hand regions identified for a given body are fed to respective networks <NUM>, <NUM> which will be described in more detail below. Again, these networks <NUM>, <NUM> require their input images to be normalised. In the illustrated embodiment, each input image map for the networks <NUM>, <NUM> is 128x128 pixels. Again, it is most likely that a detected hand region will be smaller than this size. Again, as with the detected body region(s), rather than scaling and resizing a detected bounding box <NUM> directly, in the embodiment, the system first re-shapes each hand region, by expanding the width of hand region, so that it comprises a square centered around the originally detected hand region, for example, region <NUM>-L in <FIG>. (In this case, hand region <NUM>-R is already square and so this step can be skipped. ) The re-shaped hand region is then expanded to comprise a region <NUM>-L, <NUM>-R of 128x128 pixels corresponding to the input map size of the networks <NUM>, <NUM>. Again, the increased image area provided around the detected hand region(s) can provide the networks <NUM>, <NUM> with improved context with which to make their inference. It will also be appreciated that where no hand regions are associated with a given body, the networks <NUM>, <NUM> will not be executed. However, when only one hand region is associated with a given body, one or both of the networks <NUM>, <NUM> can be executed with the second hand region input being blank.

In the embodiment, each of the networks <NUM>, <NUM> comprise identical structures and produce corresponding outputs, with each output corresponding to a user activity of concern, for example: smoking, eating, talking on the phone, texting.

However, the network <NUM> is trained with images of occupants involved in such activities where the objects involved in the activity, for example, a cigarette, food or a phone are not visible. On the other hand, the network <NUM> is trained with images of occupants involved in such activities where the objects are visible in at least one hand region of the input image.

As will be seen later, this allows the networks <NUM>, <NUM> to provide improved inference because the appearance of a hand region during different activities tends to either show the hand or the object and so, rather than tending to provide an averaged output inference value when a hand region clearly shows a hand involved in an activity or an object indicative of an activity, in the present embodiment, in the first case when only an occupants hands are visible, network <NUM> should produce quite a high confidence level and the network <NUM> a low confidence level, whereas in the second case when an object is visible in at least one hand, network <NUM> should produce quite a high confidence level and the network <NUM> a lower confidence level.

In the embodiment, the output produced by the network <NUM> is labelled handCompare and qualified by the type of (unseen) object associated with the activity, e.g. cigarette, food, phone. The output produced by the network <NUM> is labelled gestureCompare and qualified by the type of visible object associated with the activity, again: cigarette, food, phone.

For each body region <NUM>, each of the classifications produced by the networks <NUM>, <NUM>, the left/right hand inference produced by the hand filter <NUM>, the skeleton data produced by the detector <NUM> and the location of the face region <NUM> are fed to occupant behaviour logic <NUM> to make a final classification of the activity for the occupant corresponding to the identified body region.

It will be understood that all of the above information can be produced from a single image (or image tensor) <NUM> from image acquisition device <NUM>.

However, it will be appreciated, that inference made on previously acquired image information can also be used to improve the operation of the various modules within the system of <FIG>.

Furthermore, other modules can also be employed with a view to determining user activity without necessarily and explicitly identifying an object indicative of the activity. For example, as is well known, a face region <NUM> for a body can be tracked from one image frame to the next. In one embodiment, a smoking detector <NUM> makes a simple contrast measurement on each instance of face region detected. It will be appreciated that when an occupant smokes, they will periodically exhale smoke and this tends to lower the contrast of the image in the region of their face during this period. When such a periodic fluctuation in contrast is detected, then the smoking detector <NUM> can signal that the occupant corresponding to the face region may be smoking.

The occupant behaviour logic <NUM> can now further process the information provided to generate further inferences about user activity.

In a first example, the logic <NUM> attempts to identify whether any arm of an occupant is in a V shape indicative of a user talking on a phone, smoking, eating or drinking. In order to do so, the logic <NUM> attempts to estimate a Euclidian distance between each of <NUM> skeleton points from the arm: (W)rist, (S)houlder, (E)lbow. The Euclidian distance (d) can be estimated from a single 2D image, especially taking into account that the approximate location and orientation of a body relative to the image acquisition device <NUM> is known. So if d(W, S) < d(S, E) and d(W, E), it can be inferred that the occupants arm is in a V shape. In such a case, a parameter armSkeletonCompare can be set to true for the occupant.

It will be appreciated that non-geometrical techniques can also be employed to determine if the occupant's arm is in a V shape including applying a dedicated neural network to the body region <NUM> or re-shaped and expanded body region <NUM>.

In another example, the behaviour logic <NUM> may analyze the skeleton data to determine if an occupant's arm is in an L shape which can be indicative of a user interacting with their phone other than talking. When such an L shape is detected, a parameter similar to armSkeletonCompare can be set to true.

In the embodiment, another parameter which is determined by the behaviour logic <NUM> is based on detecting whether a hand region associated with a body overlaps or is closely adjacent the face region for a body. If so, a parameter handFaceCompare is set to true.

So, for any given frame, the behaviour logic <NUM> is provided by the networks <NUM>, <NUM> with handCompare and gestureCompare values indicating a respective confidence levels for the various occupant activities being monitored for each occupant associated with a detected body region. In the embodiment, the behaviour logic <NUM> can analyse the skeleton joint locations to produce an indication, armSkeletonCompare, of whether one or more of the monitored occupant activities are true or not. Furthermore, in the embodiment, the behaviour logic <NUM> can analyse the spatial relationship of the hand and face regions to produce an indication, handFaceCompare, of an activity.

All of these parameters, as well as the current value of smoking provided by the smoking detector <NUM>, when this is available, can be combined to produce a final value for the current activity of the occupant.

Thus, these behaviors are triggered only when the food / drink / phone is near the face, not just held in the hand.

In some embodiments, the behaviour logic <NUM> can wait until eating / drinking / talking on the phone is detected in more than a threshold number of frames within a pre-defined time window before flagging the behaviour.

In this case, rather than using armSkeletonCompare as false, the value of the L shape skeleton detecting parameter being true could be used.

In any case, it will be seen how this behaviour is detected only when a phone is held in the hand, away from the face.

Again, the behaviour logic <NUM> can wait until this behaviour is detected in more than a threshold number of frames within a pre-defined time window before flagging the behaviour.

Further analysis of the skeleton points provided by the detector <NUM> can also be employed to detect other behaviours.

For example, the skeleton points (neck, left hip, right hip) can be used compute an upper body forward angle, where <NUM>°-<NUM>° can indicate the occupant is bending down, while angles of approximately <NUM>° indicate the occupant is upright.

If no hip skeleton points are detected, then the behaviour logic <NUM> can check if any of the head or shoulder skeleton points are located inside a "bending" region of the body region, for example, a lower <NUM>% of the body region. If so, then bending down is detected.

It is also possible to check if the face region associated with a body region intersects the lower bending region of the body region and if so, bending down can be indicated.

These checks can also be combined and again the activity can be monitored so that if it is detected in more than a threshold number of frames within a pre-defined time window, the behaviour is flagged.

As well as determining activities from the instantaneous values of the above parameters and the instantaneous location of skeleton joints for a given frame, the behaviour logic <NUM> can also keep track of parameter values over time to determine an occupant activity. So again, in relation to bending, if the aspect ratio of the body region changes over time, with the height decreasing from a height for the body region when it was detected as upright, then this too can be used to indicate bending.

Claim 1:
A method for monitoring occupants (<NUM>, <NUM>) of a vehicle comprising:
obtaining successive images (<NUM>) of an interior cabin (<NUM>) of the vehicle from an occupant facing camera;
identifying a respective body region (<NUM>) for one or more occupants of said vehicle within an obtained image;
identifying within at least one of said body regions, a plurality of skeletal points (<NUM>) including one or more points on an arm of a body;
identifying one or more hand regions (<NUM>) within said obtained image;
determining (<NUM>) a hand region to be either a left (<NUM>-L) or a right hand (<NUM>-R) of an occupant in accordance with its spatial relationship with identified skeletal points of said body region of said occupant;
providing at least said determined left or right hand region for said occupant to each of a pair of classifiers (<NUM>, <NUM>) to provide a respective provisional activity classification for said occupant, a first classifier (<NUM>) of said pair of classifiers being trained with images of hands of occupants in states where objects involved are not visible and a second classifier (<NUM>) of said pair of classifiers being trained with images of occupants in said states where objects are visible in at least one hand region, wherein the second classifier (<NUM>) is configured to output its provisional activity classification based on an appearance of a detected object; and
combining (<NUM>) said provisional activity classifications to provide a final state for said occupant.