Abstract:
Traffic sign warnings are provided to a driver by optically scanning an area in front of the moving vehicle to obtain an image of said area in front of the moving vehicle. Road signs are recognized and their importance categorized. A driver&#39;s eyes are continuously scanned to determine whether the driver&#39;s eyes&#39; ever focus on, or are directed to a detected and recognized road sign. Multiple different visual and/or audible warnings or alarms are generated based on the nature of a road sign, its proximity to the moving vehicle, and the driver&#39;s determined awareness of the road sign.

Description:
BACKGROUND 
     Road signs help keep traffic flowing smoothly and freely by helping drivers reach their destinations and let them know entry, exit and turn points in advance. Drivers who are warned in advance of a road entry, exit, turn, stop sign, traffic signal or speed limit by way of a road sign naturally tend to avoid mistakes that can cause collisions. 
     Virtually every vehicle driver will at some point inadvertently overlook a road sign. A method and apparatus for providing an active warning to a driver that a road sign has been missed or might have been missed would reduce collisions, improve traffic flow and safety. 
    
    
     
       BRIEF DESCRIPTION OF THE FIGURES 
         FIG. 1  is a cross sectional view of a vehicle having a road camera configured to provide images of area in front of the vehicle and a driver camera configured to provide images of a driver&#39;s eyes; 
         FIG. 2  is a block diagram of a first embodiment of an apparatus for providing advanced traffic sign warnings to a driver of a moving vehicle; 
         FIG. 3  is a block diagram of a preferred apparatus for providing advanced traffic sign warnings to a driver of a moving vehicle; 
         FIG. 4  depicts steps of a method of providing traffic sign warnings to a driver of a moving vehicle; 
         FIG. 5  depicts steps of a method for performing traffic sign recognition and localization; 
         FIG. 6  shows steps of a method for determining whether a driver has seen a particular road sign; and 
         FIG. 7  depicts steps of a method  700  for synchronizing a traffic signs location and driver&#39;s awareness to each other. 
     
    
    
     DETAILED DESCRIPTION 
       FIG. 1  depicts a vehicle in cross-section. The vehicle  100  comprises a windshield  102  through which a driver, not shown, can see the road ahead of the vehicle as well as road signs. A first road camera  104  is mounted inside the windshield  102  and configured to be able to “see” area in front of the vehicle, including pavement, right-of-ways, vehicle shoulders and overhead signs. A second camera  106  mounted inside the passenger compartment  108  is directed toward the driver such that the second  106  is able to “see” the drivers head, face and eyes. 
     Various gauges and instruments are mounted in or form part of a vehicle display panel or dashboard  110 , which provide vehicle operating information to a driver. A computer and other electronics  112  is coupled to both cameras  104 ,  106  and the display  110 . The computer and its associated electronics  112  perform various process steps described below. 
       FIG. 2  is a block diagram of an apparatus  200  for providing advanced traffic sign warnings to a driver of a moving vehicle, such as the vehicle  100  depicted in  FIG. 1 . The apparatus  200  comprises a road camera  202  mounted in a vehicle and which is able to continuously scan areas in front of the vehicle as it moves. The road camera  202  produces data that represents images of the roadway and terrain in front of the vehicle. The images generated by the road camera  202  are frames of individual picture elements or “pixels.” Each pixel is represented by digital data  203  that is output from the road camera  202 . 
     A driver camera  204  is mounted in the vehicle at a location where the driver camera  204  is able to send images of a driver&#39;s head, face and eyes. As with the road camera  202 , the driver camera  204  produces successive images, each individual picture element of which is represented by digital data. 
     The output  203  of the road camera  202  is provided to a sign detector and recognizer  206 , which is an electronic device configured or programmed to recognize various predetermined shapes in images sent to it from the road camera  202 . Once a shape in an image is recognized, the content of the recognized sign is also extracted by the sign detector and recognizer. By way of example, in the United States, a stop sign is a hexagon, i.e., it has six equal sides. The “content” of the stop sign is a command to a driver to bring a vehicle to a complete stop at a stop line, usually adjacent to a stop sign. In another example, speed limit signs in the United States are rectangular and usually include the words “SPEED LIMIT” and one or two digits. The “content” of a recognized speed limit sign is the numeric value posted on the sign. 
     The type or identity of a road sign and its content is provided as digital data  207  to a sign location determiner  208 . The sign location determiner  208  is an electronic device configured to determine a physical location of a sign in three-dimensional space. The sign location determiner  208  in combination with the sign detector and recognizer  206  ranks the importance of a recognized road sign by its geographical location relative to the location of the moving vehicle, as determined by a global positioning system  210 , its distance from the vehicle, and the content of the sign. 
     The location of a recognized road sign, its relative importance and location relative to the vehicle are provided as digital data  209  to a focal point and sign location comparator  212 . The focal point and sign location determiner is an electronic device that receives information from the driver camera  204  and data from the sign location determiner and determines whether the driver&#39;s eyes appear to have fixed on the approaching, recognized road sign. 
     Referring now to the driver camera  204 , images that it produces is sent to the focal point determiner  214 . In a preferred embodiment, the focal point determiner  214  is a processor that executes program instructions stored in a non-transitory device  213 , which is coupled to the focal point determiner  214  through a conventional address/data/control bus  215 . 
     Using a high-definition (HD) image of the driver&#39;s face and eyes, the focal point determiner  214  determines a point in three-dimensional space at which the driver&#39;s eyes are focused. Such a determination is made by measuring the distance between the driver&#39;s eye pupils, the pupil&#39;s angle of inclination and relative angles between the pupils. 
     The output of the focal point determiner  214  is a digital representation of a point in three dimensional space where the driver&#39;s eyes are focused or looking Those of ordinary skill in the art will recognize that the focal point of the driver&#39;s eyes will thus change or should change as the driver operates the vehicle. The output  216  of the focal point determiner  214  is provided to the focal point to sign location comparator  212 . The comparator continuously monitors the output of the focal point determiner  216  and the output of the sign location determiner  208  to determine if the focal point of the driver&#39;s eyes ever equalizes relative to the location of the recognized sign. 
     The output  218  of the focal point to sign location comparator  212  is provided to a vehicle operation determiner  220 . The vehicle operation determiner  220  thus receives a determination of whether the driver appears to have seen a road sign that was recognized and, based upon the identity of the sign as provided by the sign detector and recognizer  206 , the vehicle operation determiner  220  determines whether any aspect of the vehicle&#39;s operation should be changed based upon the driver seeing or not seeing a particular recognized road sign. 
     The output  221  of the vehicle operation determiner  220  is provided to an alarm system  222 . In a preferred embodiment, the warning alarm system  222  comprises visible displays on a vehicle dashboard or control panel  110  as well as audible alarms output from the vehicles speakers, not shown in  FIG. 1  but well known to those of ordinary skill in the art. 
       FIG. 3  is a block diagram of another apparatus  300  for providing advanced traffic sign warnings to a driver of a moving vehicle but which is implemented by a computer. In  FIG. 3 , a road camera  302 , a driver camera  304 , display device  306  and warning enunciator  308  are all coupled to a central processing unit (CPU) or processor  310 . The processor  310  is coupled to a non-transitory memory device  312  through a conventional address/data/control but  313 . 
     The non-transitory memory device  312 , typically embodied as semiconductor RAM or ROM stores a database  314 . The database  314  stores geographic coordinates (location data) for road signs, road speed limit signs and other signs and the information displayed on them. 
     The database  314  is continuously updated. The locations of recognized signs are provided to the database  314  and stored therein using geographic location information obtained from a GPS navigation system  316  which is also coupled to the processor  310 . 
       FIG. 4  depicts steps of a method of providing traffic sign warnings to a driver of a moving vehicle. The method  400  begins at step  402  where the area in front of a vehicle is continuously scanned through step  404  for road signs that are detected and recognized by a road sign detector and recognizer. 
     When a sign is detected and recognized, the method proceeds to step  406  where the geographic location of the sign relative to the vehicle is determined in three dimensions. The location of the sign relative to the vehicle in three dimensions is accomplished by obtaining the cars geographic location from a global positioning system and shape recognition software executed by a processor. 
     If the location of a particular recognized sign is such that the vehicle&#39;s operation should be changed in some way, a decision is made at step  408  that the vehicle&#39;s operation should be changed. Examples of changing a vehicle&#39;s operation would be slowing the vehicle down, braking, changing lanes, or completely stopping the vehicle. 
     At step  410 , the driver&#39;s eyes are scanned to determine a point in three dimensional space where the driver is presumably looking The scanning and focal point determination performed at step  410  is used at step  412  to determine whether the driver appears to have seen a particular sign the location of which was determined in step  406 . If the test at step  412  was negative, which means the driver apparently did not see the sign, a warning is provided to the driver at step  414 . Examples of warnings would include visible warning displayed on a dashboard, flashing lights inside or outside the vehicle and an audible warning provided to the driver such as one through the vehicles speakers. 
     If as a result of the test at step  412  the driver has at least apparently seen a posted road sign, a test is performed at step  416  whether the vehicle&#39;s operation has changed or should change. If no vehicle operation has changed after the driver has seen a particular road sign, the method proceeds to step  414  or a warning is issued to the driver again. If the vehicle&#39;s operation is being changed appropriately, the method proceeds to step  418  which returns the process to the beginning step  402 . 
       FIG. 5  depicts steps of a method for performing traffic sign recognition and localization. At step  502 , a camera scans a surrounding environment of the vehicle for traffic signs or road signs. Examples of such signs would include a display device such as a lettered board or surface including a stop sign, speed limit sign or a road identification sign. A sign can also be a light or series of lights such as a stop light or flashing warning lights. The method  500  shown in  FIG. 5  continuously loops through  502  and  504  until a traffic sign or road sign is detected. 
     At step  506 , a detected road sign is ranked in importance according to its contextual relevance, temporal relevance and urgency. Contextual relevance means that the importance of a sign will depend upon its context. By way of example, a sign identifying the distance to a particular landmark or waypoint will increase in importance as the distance to the landmark or waypoint decreases. Temporal or time relevance will also depend on a distance as well as a time required to reach a location or point on a roadway where a vehicle&#39;s operation must change. Urgency determines importance based upon the nature of the recognized sign. By way of example, a stop sign has more urgency than does a speed limit sign or merging sign. A stoplight has more urgency that does a stop sign. 
     At step  508 , the recognized sign is localized in three dimensional space. Localizing a sign in three dimensional space is accomplished using the geographic location of the vehicle and the distance to a particular sign by the size of the recognized sign in an image as well as a location of a sign as determined by a database of road signs. 
     At step  510 , a database of recognized road sign locations is updated. 
       FIG. 6  shows steps of a method for determining whether a driver is aware of or has seen a particular road sign. At step  602 , the driver&#39;s face, including eyes are scanned by a camera to determine or identify points or locations in 3-D space where the driver appears to be looking or gazing. If a driver&#39;s gaze as determined at step  602  is the same or substantially the same as the location of a road sign in 3-D as determined by the method in  FIG. 5 , the test performed at step  604  results in the method proceeding to step  606  where the database of awareness points is updated. 
       FIG. 7  depicts steps of a method  700  for synchronizing a traffic signs location and driver&#39;s awareness to each other. At step  702 , a database of traffic sign locations and traffic sign types is continuously scanned or sorted for matches or occurrences candidates in the database of awareness points, the database of awareness points being created or updated in step  606  as shown in  FIG. 6 . At step  704 , a determination is made whether a match candidate is considered to be a match. If the database of existing and known traffic sign locations matches a geographic location in the database of awareness points, the sign detected in the method shown in  FIG. 6  proceeds to step  706  where a recognized sign is added to the database and marked as being “found.” If as a result of this test performed at step  704 , a determination is made that a recognized sign is not in the database, the method proceeds to step  708  wherein the database of traffic signs is updated to indicate the location, relative importance, contextual relevance and temporal relevance and urgency are updated. 
     At step  710 , the database of traffic signs is scanned and three successive decisions  712 ,  716 ,  720  are determined in order to determine the type and nature of the alarm to be provided to the driver. 
     At step  712 , a determination is made whether the urgency and the temporal relevance and contextual relevance are such that an alarm  714  should be issued to the driver. If the sign is not particularly urgent or important, a warning is issued at step  718 . Similarly if the nature and relevance and importance of the sign do not warrant a warning, a simple notice is issued at step  722 . 
     The foregoing description is for purposes of illustration only. The true scope of the invention is set forth in the following claims.