Vision-based occupant classification method and system for controlling airbag deployment in a vehicle restraint system

A vehicle restraint system has a vision-based occupant classification system for control of airbag deployment during a crash scenario. The classification system utilizes two imaging sensors which together create a stream of paired images received and stored by an occupant classification controller. A computer program product of the controller utilizes the paired images to extract disparity/range features and stereo-vision differential edge density features. Moreover, the controller extracts wavelet features from one of the two paired images. All three features or maps are classified amongst preferably seven classifications by algorithms of the computer program product producing class confidence data fed to a sensor fusion engine of the controller for processing and output of an airbag control signal input into a restraint controller of the vehicle restraint system.

FIELD OF THE INVENTION

The present invention relates to method and system for controlling a vision-based occupant classification airbag deployment in a vehicle restraint system.

BACKGROUND OF THE INVENTION

The United States National Highway Traffic Safety Administration, NHTSA, has recognized that earlier generation airbag systems designed to enhance safety and vehicle seat belt restraint systems can potentially cause injury during a crash scenario especially when the occupants are not properly situated within the vehicle and/or restrained by a seat belt. As such, the NHTSA allowed automotive manufacturers the option to reduce the inflation power or aggressiveness of the first generation airbags to lessen the likelihood of an airbag related injury. These less powerful airbags are typically known as “de-powered” airbags and have been in most vehicles since 1997.

As an added precaution, the NHTSA required manufacturers to introduce an “advanced frontal airbag” which is designed to meet the needs of the occupant in a variety of specific crash scenarios. The advanced airbag systems automatically determine if and with what level of power the driver frontal airbag and the passenger frontal airbag will inflate. The appropriate level of power is generally based upon sensor inputs that can typically detect: 1) occupant size, 2) seat position, 3) seat belt use of the occupant, and 4) crash severity. Advance frontal airbags were generally designed to reduce the risk of airbag induced injury to children and adults of small stature. All passenger cars and light trucks produced after Sep. 1, 2006 in the United States are required to have the advanced frontal airbag system.

Various occupant-detection devices are known to communicate with a controller of the advanced frontal airbag system requiring the system to take appropriate action(s) (i.e. disabling the airbag in a crash). A weight-based occupant-detection system is one such device that utilizes a bladder installed in a passenger-side seat that senses weight distribution in the seat. A microcontroller in the device uses an algorithm to analyze the weight distribution and determine if the occupant may be injured by the airbag. Unfortunately, the weight-based devices are generally not designed to detect if an occupant is out of position. Yet further, such devices are not capable of differentiating between an empty seat with an inanimate object and a seat with a child. Consequently, during a crash scenario, a passenger frontal airbag could actuate without need.

Visual or imaging based systems are known that measure various features of at least one image, establish confidence levels and fuse the features together to compute an “occupant type” which in-turn is used to, for instance, enable or disable a vehicle airbag. Such a system is disclosed in U.S. Patent Application Publication 2003/0204384 A1, published Oct. 30, 2003 and incorporated herein in its entirety. Such features include an edge density taken from a single image, a wavelet feature taken from a single image, and a disparity feature that requires “stereo” images from two independent cameras. A sub-classifier of each feature independently assigns a confidence value to each one of five occupant classifications known to be: rear-facing infant seat; front-facing infant seat; adult; out of position adult; and empty seat. The fifteen class confidences are then input into a fusion classifier that is trained to intelligently combine the confidences to form a final airbag enable/disable decision.

Unfortunately, the five classification system is limited and does not include other categories such as “a child inside of an at-risk-zone” or “a child outside of an at-risk-zone” which could further refine airbag safety. Moreover, known software algorithms used to classify the three known features are likely to become confused between categories if required to handle the additional two classifications. Furthermore, additional classifications will increase the size of the neural network making training of the network more difficult. Therefore, more efficient and more effective features and methods to fulfill the seven-category-classification task are desired.

SUMMARY OF THE INVENTION

A vision-based occupant classification method and/or system controls airbag deployment during a crash scenario. Two imaging sensors create a stream of paired images received and stored by an occupant classification controller. A computer program product of the controller utilizes the paired images to extract disparity/range features and stereo-vision differential edge density features. Moreover, the controller extracts wavelet features from one of the two paired images. All three features or maps are classified amongst preferably seven classifications by algorithms of the computer program product producing class confidence data fed to a sensor fusion engine of the controller for processing and output of an airbag control signal input into a restraint controller of the vehicle restraint system.

The vision-based occupant classification method and/or system preferably has a seven-category classifier which departs from the traditional five by including “a child inside of the at-risk-zone” and “a child outside of the at-risk-zone.” Potential confusion between categories created by the additional two categories is eliminated by a disparity based segmentation of the image which reduces the influence of irrelevant background information to the classification. Moreover, a spatial filter is applied over the disparity features or map in order to extract more consistent disparity/range features over time, along with a predetermined adaptive offset map and average threshold map to counter possible inaccuracy of the disparity/range measurement. The stereo-vision edge density features are better suited to suppress noise and associate the occupant contour with depth perception data.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Referring toFIGS. 1-3, a restraint system20of a vehicle22preferably has a frontal operator airbag24and a frontal passenger airbag26along with corresponding seat belt devices known in the art. Each airbag24,26preferably inflates independently during a frontal crash of the vehicle22to protect the occupants which include a driver28and a passenger30facing the respective airbags24,26. The restraint system20has a restraint controller32that initiates the airbag inflation by sending an electrical signal to an inflator of the restraint system20. A microprocessor36controls the controller that deploys each airbag24,26separately. The microprocessor36determines when a crash is occurring by electrical input signals received from at least one and preferably a plurality of accelerometers38that indicate a change in velocity with time or rapid deceleration. A large deceleration indicates that a crash is in progress. The microprocessor36of the restraint controller32has software calibrations that determine the magnitude of deceleration indicative of a crash for a particular vehicle.

The vehicle restraint controller32can preferably diagnose the restraint system20. If a problem is detected by the restraint controller32, a telltale indicator40on an instrument cluster42of the vehicle22can warn the driver28audibly and/or visually. The controller32can also send an electrical diagnostic signal to a remote tester that a technician can connect to a vehicle data link or serial bus44used for communication.

A classification controller46of a vision-based occupant-classification device or system48of the restraint system20classifies the occupant28,30and periodically electrically transmits its classification50to the vehicle restraint controller32preferably using a data bus. Dependent upon the input classification signal50, the restraint controller32can disable the passenger-side or driver-side airbags26,24, preventing either from deploying even in the event of a crash. The vision-based occupant-classification system48also communicates with the restraint controller32to control the intensity of the airbag deployment, or prevent deployment altogether, depending upon a variety of pre-established parameters. The system preferably utilizes two imaging sensors52,54, or video cameras, to analyze the image of a person or object near an airbag for classification. Image classifications generally include: 1) rear-facing infant seat (RFIS); 2) front-facing infant seat (FFIS); 3) adult normal or twisted position (Adult NT); 4) adult out of position (Adult OOP); 5) child normal or twisted position (Child NT); 6) child out of position (Child OOP); and 7) empty seat (Empty). For instance, front-seat passenger26may be considered out of position if his/her head or torso is within twelve inches of the dashboard40. An empty seat is not necessarily completely empty but may have inanimate objects (for example, a box or newspaper) placed upon it.

Generally, the vision-based occupant-classification device48disables either airbag24,26for “Adult OOP” and “Child OOP” classifications and for RFIS (because the head of the infant in such a seat is near the dashboard) upon the classification input signal50from the classification controller46to the restraint controller32. Child NT is also a case for disabling the airbag26, while Adult NT is preferably a case for enabling the airbag26. The airbag is preferably disabled for the FFIS classification to eliminate any possible harm to the child, and the airbag26is preferably disabled for an empty seat to save the cost of replacing the airbag after a crash.

The occupant-classification device uses the two cameras52,54to obtain “stereovision” images of the occupants28,30from two different perspectives. The cameras52,54are preferably mounted near a rear-view mirror56of the vehicle22and next to an illuminator58that transmits light at an infrared wavelength which is generally invisible to the naked eye (as best shown inFIG. 2). Because the infrared light filters out other types of light, the illumination can be kept more uniform as ambient light levels change. The cameras52,54register the images as a two-dimensional matrix of digital grayscale data, meaning that they are stored in discrete shades of gray, with each data value representing the intensity of a particular point in each image. Each of these points is called a pixel. The darkest pixels have the minimum value (zero) and the brightest pixels have the maximum value. The system preferably uses eight-bit data having a maximum value of about 255. The classification controller46of the occupant-classification system48controls the system and electrically receives the stream of image data60from the cameras. Knowing the distance between the cameras52,54and having two perspectives, the classification controller46is able to estimate the range or distance of objects in the images from the cameras, providing the depth of the objects in the images. This third dimension or depth perception is desirable for classifying the image.

Referring toFIGS. 4,5and7, a computer readable medium62of the occupant detection system48has encoded image-classification algorithms which perform a series of mathematical operations on each “snap-shot” or image of video data or paired stream of images60. The system computer readable medium or software62generally uses three distinct algorithms. The output of each algorithm is an array or matrix of confidence values called features. The features or maps are generally termed: differential edge density features64; wavelet features66; and disparity features68. Preferably, the software62and controller46operate with sufficient speed to process pairs of images80,82produced by the cameras at a pre-established rate of preferably about thirty to fifty updates per second.

Referring toFIGS. 4 and 5, the first algorithm is a range or disparity feature calculation. The disparity feature calculation requires snap shots of the paired images80,82from the stream of images84produced by respective cameras52,54. A pre-determined region of interest86(ROI) of the paired images is divided into a matrix or set of small areas by the encoded instructions62. As an example, the ROI for the disparity feature is preferably about 192 pixels horizontally by 220 pixels vertically. The first algorithm calculates a separate disparity value for each of 352 equal-sized areas within the ROI. Each of these areas has dimensions of twelve pixels horizontally by ten pixels vertically. These dimensions are for a particular vehicle application and are likely to change dependent upon the application.

The ultimate goal of the disparity algorithm is to determine a set of confidences or disparity features related to the range of each of the small areas from the paired images80,82of the cameras52,54, thus giving a third dimension to the ROI86. This feature is not expressed in terms of range but of disparity, which is proportional to the inverse of the range. Referring toFIG. 8, the concept of disparity between point A′ and point A″ in the set of stereo images80,82is illustrated. The first image80taken from the first camera52denotes the first point A′. The second image82taken from the second camera54illustrates the corresponding first point A″. A horizontal and a vertical coordinate is assigned to the point A′ on the first image80and differing coordinates on the second image82for point A″. The positions of the two points differ due to the difference in perspective of each camera. Preferably, the cameras52,54are placed so that the vertical coordinates of identical features in each image are identical. The horizontal coordinates do differ and their difference is the disparity. Because the disparity is proportional to the inverse of the range of the features from the cameras52,54, the closer a feature in the image is to the cameras, the greater the disparity. As an example, for points A′ and A″ inFIG. 8, the disparity is twenty.

In order to estimate the disparities in a paired set of small areas, the first algorithm must identify pairs of points in one image that correspond to points in the other image. After partitioning the ROI86into a two-dimensional matrix of small disparity areas with equal size, the range algorithm identifies those areas with significant texture producing a texture matrix88,90of the ROI86of each image80,82. An area with a significant amount of texture exhibits a large variation in the intensities of the pixels within the area. Areas with little texture are difficult to differentiate from others due to their lack of distinguishing features.

Once the classification controller46identifies the high-texture areas in one image80, the algorithm identifies areas in the other image82that are likely matches. For each high-texture area of the first texture matrix88of the first image80, the algorithm finds the best match in the other texture matrix90of the other image82and stores to memory92the computed disparity between these sets of areas as a disparity map estimate94. At this point, the disparities of the low-texture areas are not assigned. The algorithm then uses an iterative technique, or iterative neighborhood update96, to compute the disparity of all the areas in the disparity map estimate90, including those with disparities already assigned to them. The technique bases the computed disparity of a given image not only on a match with an area in the other image, but also on the computed disparity of neighboring or adjacent areas. The result of the analysis is a two-dimensional disparity map98.

A smoothing operation or 3×3-smoothing filter100is then applied to the disparity map98to make the ultimate disparity features more consistent over time. Assuming (i, j) is the center location of the smoothing filter and Di,jrepresents the disparity value at that location, the smoothing operation is defined as replacing Di,jwith:

(∑m=-11⁢Di-1,j+m+∑m=-11⁢Di,j+m+∑m=-11⁢Di+1,j+m)/9
This operation is applied to each pixel of the disparity map98that has eight-way neighbors. A resultant smoothed disparity map102is generated and is used for segmentation and extracting range or disparity features.

A pre-determined segmentation template103which generally includes an adaptive offset map104and an average disparity threshold map106, which takes various lighting environments into account and empirically compensates for inherent flaws of the disparity algorithm, is applied to the smooth disparity map102producing a binary map108for ultimately discarding portions of the ROI86to achieve higher confidence values of both the disparity features and the wavelet features.

Referring toFIG. 6, the pre-determined disparity threshold map106is generally created from a physical boundary110, which contains the maximum volume within which the occupant(s)28,30will be classified. To reduce the influence of noise, the disparity threshold map uses disparity values averaged over about one hundred frames or snap-shots. The physical boundary is exposed to various driving environments112to get a set of disparity threshold maps114. The averaged disparity values of this set of maps compose the final disparity threshold map116. Furthermore, the standard deviation of each pixel in the disparity threshold map is calculated and used as an adaptive offset map118for image segmentation.

The segmentation is achieved by comparing the measured disparity values of the smoothed disparity map102and the disparity threshold values of the disparity threshold map116for each small area of the smooth disparity map102. For instance, in a given location (i, j), assume the measured disparity value is Dm(i, j) and the disparity threshold is Dth(i, j). A new binary map B(i, j)108with the same matrix correlation as the disparity map of the original images80,82is created with the following rules:
IfDm(i, j)<=Dth(i, j)+δ(i, j) thenB(i, j)=0 otherwiseB(i, j)=1

The parameter δ(i, j) is the standard deviation of disparity in the location (i, j) of the adaptive offset map104. It is used to control the degree of background suppression. A larger δ(i, j) indicates more uncertainty of the disparity measurement in the location and therefore, the image segmentation should be less dependent on disparity. Such an adaptive offset will ensure that the image segmentation will be applied only to the region where the disparity estimate is reliable. For example, it reduces the over segmentation of images of child occupants and makes it possible to classify child positions correctly. The binary map108is applied as a mask over the smooth or area disparity map102with areas, B′(i, j), corresponding to “zeros” being blocked out and the areas corresponding to “ones” remaining unchanged to calculate, or segment into, the refined or segmented disparity map68. A seven-category neural network classifier112receives the refined disparity map68and together with the wavelet and differential edge-density features outputs preferably seven class confidences114for processing by the classification controller46.

Wavelet Feature

The second algorithm is a wavelet feature calculation which generally utilizes the binary map108calculated from the disparity algorithm to develop a segmented image118. The binary map108applies as a mask over any one of the original paired images80,82. The original image areas corresponding to “zeros” in B′(i, j) are blocked out and the areas corresponding to “ones” in B′(i, j) remain unchanged. Therefore, the segmented image118is still a gray level image with the irrelevant background being masked out. The image applied is from only one of the two cameras52,54and must be consistent over time (i.e. from the same image stream).

A wavelet ROI of the segmented image118is divided into a set of wavelet areas preferably being larger the small disparity areas previously described. As the continuing example, there may be four sets of wavelet areas with two sets consisting of 32×32 pixels, and two sets having 64×64 pixels each. The pixels used for the wavelet calculation are preferably down-sampled by two. That is, every second pixel is skipped during sampling, although the total number of pixels sampled is not reduced. Thus, each sampled wavelet area is 64×64 and 128×128 in size, respectively. Preferably. The small wavelet regions preferably overlap each other for providing an “over-complete” sampling of the image data providing more information than would a set of areas that did not overlap.

In each of the wavelet areas, pixels in an upper half of the wavelet area are subtracted from pixels in a lower half of the wavelet area, producing a vertical wavelet coefficient for that area. In the same areas, pixels in a left half are subtracted from pixels in a right half, producing a horizontal wavelet coefficient for the area. Each wavelet area thus has two features associated with it, one horizontal and one vertical producing the combined wavelet transform or map66. The seven-category neural network classifier112receives the wavelet features and together with the disparity features and differential edge-density features outputs preferably seven class confidences114for processing by the classification controller or processor46.

Stereo-Vision Edge Density Feature

Referring toFIGS. 4 and 7, the stereo-vision edge density feature is preferred for the novel seven-category system. The differential edge density algorithm of the computer readable medium62examines each pixel of each original image80,82that has eight neighboring pixels (i.e. pixels not on the outer border of the images). The algorithm analyzes the neighbor of each pixel in the pre-determined ROI86of the original images to determine if the intensity of the image is changing significantly near that point. Pixels with a large difference in intensity relative to their neighbors are considered to be edge pixels. A matrix of pixel intensities, termed pixel map122,124is generally developed for the ROI86of each respective original image80,82.

In practice, Ai,jand Bi,jare a stereo image pair at pixel location (i,j) of respective pixel maps122,124. An offset-compensated and intensity normalized stereo differential image Ci,j126is constructed according to:

For a horizontal stereo configuration, H_offset is the horizontal correspondence value averaged in the field of view when an object is at a distance of infinity. Similarly, the parameter V_offset is the averaged pixel location difference in the vertical direction of a correspondence point between the stereo images80,82at different object distances. These offset values can be determined through initial camera configuration calibrations. The Full_Pixel_Depth is simply the maximum scale of the pixel grey level. For example, for an eight-bit pixel depth, this parameter would be about 255. The offset compensation is intended to suppress far field objects (background) so that the foreground object can be naturally segmented. The image intensity normalization and scaling improves the image visibility and contrast. Such a stereo differential image is essentially an estimated edge density map where the width of the edge shown is wider when the object is closer to the cameras.

Note that the image normalization and scaling described by equation (1) may amplify noises in the extreme dark region of the image. In order to suppress these noises, a pre-determined pixel intensity threshold referred to as Threshold_A is used to construct a final stereo differential image C′i,j126with the following rules:
IfBi+H—offset, j+V—offset+Ai,j>Threshold—A, then

New image features termed as “stereo differential edge densities” are then extracted from the final stereo differential image C′i,j126. This extraction first converts the final stereo differential image C′i,j126into a binary image Di,j, or binary map128, with a pre-determined differential intensity threshold, Threshold_B, by the following rule:
If C′i,j>Threshold_B then Di,j=255 Otherwise Di,j=0.
Unlike traditional edge filters such as the Sobel filter, this process produces “solid” edges from the stereo differential images and reserves the thickness of the edges. The thickness contains wanted range information of the object.

Secondly, the extraction then applies morphological operations on the binary map128to remove smaller particles generally denoting noise. Preferably, one erosion process followed by one dilation process is applied to the binary map128to achieve a refined binary map130.

Lastly, to gain the stereo differential edge density feature or map64, the resultant binary image130is divided into a number of bins or density areas of a predetermined ROI131. For instance, for an edge density ROI131having dimensions of 180 pixels horizontally by 220 pixels vertically, the dimensions of the areas where edge density is calculated are fifteen pixels horizontally by twenty-two pixels vertically. These dimensions are preferably pre-determined for a given vehicle application. The total number of non-zero pixels in each area is then counted as an image input feature. The differential edge density map64is a matrix of all the areas and is fed into the seven-category neural network classifier112

Processing the Features to Classify the Image

The image feature generated by the three image-processing algorithms require further processing to obtain an image classification. A fusion engine, which is preferably the neural network classifier112, accomplishes this for the vision system48. The input to the network is an array of all the features computed for a particular image (differential edge density, wavelets, and range/disparity). The output is another array, with one value assigned to each of the possible classifications for the image. As discussed earlier, an image can have one of seven different classifications in the system (Adult NT, Adult OOP, Child NT, Child OOP, RFIS, FFIS and Empty). Hence, the neural network112supporting the classifications produces an array of seven values. Each computed value for each distinct classification is a number ranging from zero to one. The computed value represents a confidence value or “figure of merit” for its assigned image classification, with one indicating that the classification is very likely to apply to the image, and zero meaning the classification is very unlikely. The system48classifies the computed image based upon the highest classification value, which should be substantially greater than the other computed values.

The neural network classifier112performs a series of mathematical operations on the feature data to produce the classification array. These operations are always the same and use a large number of weighting calibrations that determine the outputs114of the neural network classifier112. That is, the neural network classifier112generates the classification values based upon the calibrations defined for it. These constant weighting values are multiplied by the inputs to the network and by intermediate results that the network computes. They determine how the network generates its classification values.

The array of class confidences114is inputted into a processor116of the classification controller46which processes and outputs a restraint control signal or airbag enable/disable signal50to the restraint controller32, as best shown inFIGS. 3 and 4.

Deriving the values of the calibrations amounts to training the network so that it “learns” how to classify images. Firstly, a separate computer-based system is preferably used to obtain training data. In the alternative, the occupant detection system20can include a special training mode to conduct the same operation. When running the training mode, a user enters the correct classification for each image that the system48is about to process. The system records the occupant's image that the system is about to process in the computer memory of the system that runs the training mode. The system records the occupant's image and computes the corresponding features, which it stores in a file with the user-defined classification. In a second step, another software program uses the feature arrays and associated classifications to determine the network's weighting values. This program chooses the calibrations so that the neural network generates the expected classifications for the feature data from the training set. The occupant-classification system includes these calibrations in its memory for use during normal operation of image classification.

To train the system effectively, the system needs to process a wide variety of images. This occupant-classification system20performs and demonstrates the ability to classify about ninety-eight percent of images correctly in tests.

Alternative—Mono-Vision Edge Density Feature

The stereo-vision edge density feature can be substituted with a known mono-vision edge density feature that produces lower confidence values but requires less software processing time. The mono-vision edge density feature uses only one of the two imaging sensors or cameras. The algorithm applied to calculate the mono-vision edge density feature examines each pixel of the image which has eight neighboring pixels (i.e. pixels not on the outer border of the image). The algorithm analyzes the neighbor of each pixel in the predetermined region of interest (ROI) of the image to determine if the intensity of the image is changing significantly near that point. Pixels with a large difference in intensity relative to their neighbors are considered to be edge pixels.

This algorithm builds an edge map, a new two-dimensional matrix of pixels, with edge pixels assigned a maximum value and others assigned a minimum value. Once the pixels have been analyzed, the algorithm divides the edge map into areas of equal size and computes the average value of the pixels in each area. This matrix of average values, representing the density of edges in each small area of the ROI of the image, are the features for this algorithm.

Although the preferred embodiments of the present invention have been disclosed, various changes and modifications can be made by one skilled in the art without departing from the scope and spirit of the invention as set forth in the appended claims. Furthermore, it is understood that the terms used here are merely descriptive rather than limiting and various changes may be made without departing from the scope and spirit of the invention.