Patent Description:
At security check points, border crossings, high occupancy vehicle (HOV) lanes, and the like it, is desirable to know how many occupants are in each vehicle that passes. At a traditional checkpoint an officer can count occupants that are visible in a vehicle. In security applications, it can be desirable to know who the occupants of a vehicle are. An officer can verify this by inspection of identification documents such as a photo ID for each occupant of the vehicle. However, these techniques require each vehicle to stop for inspection before passing through.

The conventional techniques have been considered satisfactory for their intended purpose.

<CIT> discloses a system to automatically control occupancy of vehicles at a toll station, the system comprising means for acquiring images of a vehicle to be controlled, means for processing the acquired images, and means for detecting faces inside the vehicle and counting said faces. A discount is then applied to the toll according to the number of validated occupants.

<CIT> discloses a method for performing face recognition in a video. Face detection data in frames of input data are used to generate face galleries, which are labeled and used in recognizing faces throughout the video. Metadata that associates the video frame and a face are generated and maintained for subsequent identification. Faces other than those found by face detection may be found by face tracking, in which facial landmarks found by the face detection are used to track a face over previous and/or subsequent video frames. Once generated, the maintained metadata may be accessed to efficiently determine the identity of a person corresponding to a viewer selected face.

<CIT> discloses a passenger counting system including: a first photographing means for photographing a vehicle and acquiring a first image; a second photographing means for photographing the vehicle in a state of receiving only light of a first specific wavelength band and acquiring a second image; and a passenger counting device. The passenger counting device includes: an image separation means for acquiring a plurality of separate images based on the first image and the second image; and a passenger number determination means for determining the number of passengers of the vehicle based on the plurality of separate images.

<CIT> discloses a method for determining a pixel classification threshold for vehicle occupancy determination. An IR image of a moving vehicle is captured using a multi-band IR imaging system. A driver's face is detected using a face recognition algorithm. Multispectral information extracted from pixels identified as human tissue of the driver's face is used to determine a pixel classification threshold. This threshold is then used to facilitate a classification of pixels of a remainder of the IR image. Once pixels in the remainder of the image have been classified, a determination can be made whether the vehicle contains additional human occupants other than the driver. An authority is alerted in the instance where the vehicle is found to be traveling in a HOV/HOT lane requiring two or more human occupants and a determination has been made that the vehicle contains an insufficient number of human occupants.

However, there is a need for improved systems and methods for detecting, counting, and identifying occupants in vehicles. The present invention provides a solution for this need.

In an aspect of the present invention, there is provided a system according to appended claims <NUM>-<NUM>.

In another aspect of the present invention, there is provided a method according to appended claims <NUM>-<NUM>.

These and other features of the system and method of the present invention will become more readily apparent to those skilled in the art from the following detailed description of the preferred embodiments taken in conjunction with the drawings.

So that those skilled in the art to which the present invention appertains will readily understand how to make and use the devices and methods of the present invention without undue experimentation, preferred embodiments thereof will be described in detail herein below with reference to certain figures, wherein:.

Reference will now be made to the drawings wherein like reference numerals identify similar structural features or aspects of the present invention. For purposes of explanation and illustration, and not limitation, a partial view of an exemplary embodiment of a system in accordance with the present invention is shown in <FIG> and is designated generally by reference character <NUM>. Other embodiments of systems in accordance with the present invention, or aspects thereof, are provided in <FIG>, as will be described. The systems and methods described herein can be used for automated counting and identification of occupants in vehicles.

The system <NUM> for detecting occupants in a vehicle <NUM> includes a controller <NUM> and a plurality of camera systems <NUM>, <NUM>, and <NUM> that are external to the vehicle <NUM> in the vehicle approach area <NUM>. Each camera system <NUM>, <NUM>, and <NUM> is operatively connected to the controller <NUM>. A trigger <NUM> in the vehicle approach area <NUM> is operatively connected to the controller <NUM> to detect an approaching vehicle <NUM> and to control the camera systems <NUM>, <NUM>, and <NUM> to acquire images of the approaching vehicle <NUM>. The controller <NUM> includes machine readable instructions configured to cause the controller <NUM> to perform any method as disclosed herein. As shown in <FIG>, each camera system <NUM>, <NUM>, and <NUM> is in a different location for acquiring images with sensors viewing the vehicle from different respective angles.

With reference now to <FIG>, camera system <NUM> includes an imaging sensor <NUM>, a pulsed illumination device <NUM>, and a processor <NUM> operatively connecting the imaging sensor <NUM> to the pulsed illumination source <NUM> for synchronizing an illumination pulse from the pulsed illumination device <NUM> with exposure of the imaging sensor <NUM>. The illumination device <NUM> can be located on camera as in camera system <NUM> shown in <FIG>, or can be located off-camera as in camera systems <NUM> and <NUM> shown in <FIG>. The camera system <NUM> include a lens <NUM> optically coupled to the imaging sensor <NUM>, an optical bandpass filter <NUM> operatively connected to filter light passing through the lens <NUM> to the imaging sensor <NUM>. The camera system <NUM> also includes a linear polarization filter <NUM> operatively connected to filter light passing through the lens <NUM> to the sensor <NUM>, e.g., to reduce glare from glass windshields and windows of the vehicle <NUM>. Imaging sensors <NUM> and <NUM> can include the same components as camera system <NUM>.

With reference now to <FIG>, a method of detecting occupants in a vehicle includes detecting an oncoming vehicle, e.g., detecting oncoming vehicle <NUM> using trigger <NUM> as shown in <FIG>. When trigger <NUM> detects an oncoming vehicle <NUM>, it signals the controller <NUM>. Controller <NUM> then commands the camera systems <NUM>, <NUM>, and <NUM> to acquire a plurality of images of occupants in the vehicle <NUM>. Each camera system <NUM>, <NUM>, and <NUM> can acquire a respective image <NUM>, <NUM>, and <NUM>, forming a set <NUM> of acquired images as shown in <FIG>. Controller <NUM> can illuminate the vehicle <NUM> with a respective pulse of illumination from each respective illumination device <NUM> for each image acquired, wherein each pulse of illumination is performed at a different time to reduce shadows cast onto the occupants while acquiring the plurality of images.

The method includes having controller <NUM> perform automated facial detection on the plurality of images <NUM>, <NUM>, and <NUM>, and to add a facial image for each face detected to a gallery <NUM> of facial images for the occupants of the vehicle <NUM>. For the image <NUM>, three faces are detected and four faces are detected from each of images <NUM> and <NUM>. Controller <NUM> performs automated facial recognition on the facial images of gallery <NUM> to group the facial images into groups <NUM>, <NUM>, <NUM>, and <NUM> based on which occupant is in the respective facial images, as indicated by facial recognition groupings <NUM> in <FIG>. While multiple images are shown in <FIG> in each of the groups <NUM>, <NUM>, <NUM>, and <NUM>, it should be understood that, in an alternative example not covered by the claims, the groups <NUM>, <NUM>, <NUM>, and <NUM> need not ever actually contain multiple images in each group. For example during facial recognition, each time a new image of a given individual is identified, controller <NUM> can decide whether the new image is better than the previous best image of the individual (based on facial detection confidence scores, facial offset angle, or the like, as described below), and if so the new image replaces the previous image in the respective group. In this way each group <NUM>, <NUM>, <NUM>, and <NUM> only ever includes one image.

Facial detection and facial recognition need not necessarily be performed one after another, but instead can be performed together on the fly. One of the sensors <NUM> can be a primary sensor, e.g., the sensor <NUM> of camera system <NUM>, that acquires a primary image, e.g., image <NUM>, of occupants in the vehicle <NUM>. The faces detected in primary image <NUM> can serve as references in the gallery <NUM> for facial recognition for subsequent ones of the images <NUM> and <NUM> of occupants in the vehicle. The controller <NUM> can add a new face to the gallery <NUM> each time a detected face in a subsequent one of the images <NUM> and <NUM> does not match with a face already in the gallery <NUM>. The controller <NUM> can iteratively compare faces detected in subsequent ones of the images <NUM>, <NUM>, and <NUM> and add each face detected to the gallery <NUM> that is not already in the gallery <NUM> until there is an image in the gallery <NUM> of each face detected by performing automated facial detection.

Whenever a face is detected for which there is already an image in the gallery <NUM>, the best image of the face can be retained in the image gallery <NUM>. Controller <NUM> selects a representative image <NUM>, <NUM>, <NUM>, and <NUM> from each group <NUM>, <NUM>, <NUM>, and <NUM> and outputs a set <NUM> of cropped selected images, one uniquely cropped selected image for each of the occupants. Set <NUM> includes no duplicate images, i.e. no more than one image is in set <NUM> for a given occupant, so duplicate images of a given occupant need be stored or displayed. The controller <NUM> can select the representative image <NUM>, <NUM>, <NUM>, and <NUM> from each group <NUM>, <NUM>, <NUM>, and <NUM> by selecting images based on corresponding confidence scores from the automated facial detection. It is also contemplated that controller <NUM> can select the representative image <NUM>, <NUM>, <NUM>, and <NUM> from each group <NUM>, <NUM>, <NUM>, and <NUM> by selecting images based on which image in the group has least facial offset angle from line of sight of the imaging sensor <NUM> which acquired the respective image. This selection process can be run on the fly with facial detection and facial recognition to winnow the gallery <NUM> down to the set <NUM>.

The controller <NUM> can determine how many occupants are in the vehicle <NUM> by counting the groups <NUM>, <NUM>, <NUM>, and <NUM>. In this example, there are four groups <NUM>, <NUM>, <NUM>, and <NUM> indicating there are four occupants in the vehicle <NUM>. If groups <NUM>, <NUM>, <NUM>, and <NUM> are conflated down to the set <NUM> on the fly as described above, then the groups <NUM>, <NUM>, <NUM>, and <NUM> can be counted indirectly by simply counting the final cropped images in set <NUM> to determine how many occupants are in the vehicle <NUM>.

The controller <NUM> can output the number of occupants in the vehicle <NUM>, and can provide other output actions as needed. For example, controller <NUM> can initiate a response, e.g., via the output device <NUM>, upon determining an improper number of occupants in the vehicle. For example, if controller <NUM> determines there are not enough occupants in a vehicle in an ROV lane, controller <NUM> can use the output device <NUM> to output an alert on a visual display, sound an audible alarm, close a physical barrier, transmit a citation, mail a citation, update a database, and/or dispatch an officer.

It is also contemplated that with the set of images <NUM>, controller <NUM> can run the final cropped facial images through a facial recognition database, either locally or remotely, to check for matches between the occupants and known individuals in the database. If a match is found, e.g., one of the occupants in the vehicle <NUM> is on a watch list, the controller <NUM> can initiate an output response, e.g., using output device <NUM>, such as outputting an alert on a visual display, sounding an audible alarm, closing a physical barrier, transmitting a citation, mailing a citation, updating a database, and/or dispatching an officer.

While shown and described herein in an exemplary context where there are n=<NUM> camera systems and m=<NUM> occupants in the vehicle <NUM>, those skilled in the art will readily appreciate that any suitable number n of camera systems can be used, and any suitable number m of occupants in a vehicle can be counted/identified without departing from the scope of the invention as defined by the claims.

Claim 1:
A method of detecting occupants in a vehicle (<NUM>) using a system (<NUM>) comprising a first camera system (<NUM>) and a second camera system (<NUM>), the first and second camera systems being external to the vehicle in a vehicle approach area (<NUM>), the method comprising:
detecting an oncoming vehicle;
using the first camera system and the second camera system, acquiring (<NUM>) a plurality of images (<NUM>, <NUM>) of occupants in the vehicle in response to detection of the vehicle, wherein the first camera system acquires a first portion (<NUM>) of the plurality of images from a first angle, wherein the second camera system acquires a second portion (<NUM>) of the plurality of images from a second angle, and wherein the first angle is different from the second angle; characterized by:
performing automated facial detection on the plurality of images and adding a facial image for each face detected in the plurality of images to a gallery (<NUM>) of facial images for the occupants of the vehicle, wherein the gallery of facial images comprises at least a first facial image of at least one of the occupants taken from the first angle and a second facial image of said at least one of the occupants taken from the second angle;
performing automated facial recognition on the gallery of facial images to form facial groupings (<NUM>, <NUM>, <NUM>, <NUM>), wherein each of the facial groupings comprises facial images of only one of the occupants, and wherein one of said facial groupings comprises said first and second facial images of said at least one of the occupants;
selecting a representative facial image from each of the facial groupings;
outputting the representative facial images selected from each of the facial groupings, each of the outputted representative facial images being representative of only one of the occupants; and
counting the facial groupings to determine how many occupants are in the vehicle.