Abstract:
A vehicle includes a camera capturing an image of a road sign associated with a road that the vehicle is traveling on. A first processing device is communicatively coupled to the camera and recognizes text characters in the captured image. A second processing device is communicatively coupled to the first processing device and converts the recognized text characters into an electronic speech signal. A loudspeaker is communicatively coupled to the second processing device and converts the electronic speech signal into audible speech that can be heard by the driver of the vehicle.

Description:
CROSS-REFERENCED TO RELATED APPLICATIONS 
       [0001]    This application claims benefit of U.S. Provisional Application No. 62/280,436 filed on Jan. 19, 2016, which the disclosure of which is hereby incorporated by reference in its entirety for all purposes. 
     
    
     FIELD OF THE INVENTION 
       [0002]    The disclosure relates to the field of automotive information systems, and, more particularly, to automotive information systems that provide road information to the driver. 
       BACKGROUND OF THE INVENTION 
       [0003]    Reading road signs while still watching the road is a difficult task for a driver, particularly at highway speed and high traffic situations. It is known to display the road sign information on the instrument cluster or head up display (HUD). However, this also requires the driver to use his eyes for reading rather than watching the road. 
       SUMMARY 
       [0004]    The present invention may capture images of nearby road signs and read the signs out loud to the driver through the vehicle&#39;s built-in sound system. The invention may interpret the text of the signs in real time and convert the text to speech in the form of a synthesized voice. Alternatively, the invention may detect the road signs and play previously made recordings of readings of each sign from a database instead of interpreting the text of the signs in real time and converting the text to speech in the form of a synthesized voice. 
         [0005]    The invention may read road signs and translate the road signs into another language, such as the native language of the driver. 
         [0006]    In one embodiment, the invention comprises a vehicle including a camera capturing an image of a road sign associated with a road that the vehicle is traveling on. A first processing device is communicatively coupled to the camera and recognizes text characters in the captured image. A second processing device is communicatively coupled to the first processing device and converts the recognized text characters into an electronic speech signal. A loudspeaker is communicatively coupled to the second processing device and converts the electronic speech signal into audible speech that can be heard by the driver of the vehicle. 
         [0007]    In another embodiment, the invention comprises a vehicle including a geographical positioning system determining a geographical location of the vehicle. A first processing device is communicatively coupled to the geographical positioning system and wirelessly transmits the geographical location of the vehicle to an external database. The first processing device wirelessly receives from the external database a textual message associated with a road sign that is visible to a driver of the vehicle at the geographical location of the vehicle. A second processing device is communicatively coupled to the first processing device and converts the textual message from the external database into an electronic speech signal. A loudspeaker is communicatively coupled to the second processing device and converts the electronic speech signal into audible speech that can be heard by the driver of the vehicle. 
         [0008]    In yet another embodiment, the invention comprises a vehicle including a a camera capturing an image of a road sign associated with a road that the vehicle is traveling on. A first processing device is communicatively coupled to the camera and recognizes text characters in the captured image. A geographical positioning system determines a geographical location of the vehicle. A second processing device is communicatively coupled to the geographical positioning system and wirelessly transmits the geographical location of the vehicle to an external database. The second processing device wirelessly receives from the external database a textual message associated with a road sign that is visible to a driver of the vehicle at the geographical location of the vehicle. A third processing device is communicatively coupled to each of the first processing device and the second processing device and produces a confidence score indicative of a likelihood that the textual message from the database is associated with the road sign in the image captured by the camera. A fourth processing device is communicatively coupled to the third processing device and converts the textual message from the external database into an electronic speech signal only if the confidence score exceeds a threshold. A loudspeaker is communicatively coupled to the fourth processing device and converts the electronic speech signal into audible speech that can be heard by the driver of the vehicle. 
         [0009]    The present invention has the advantage that the driver can keep his eyes on the road without having to deviate his gaze to read road signs. 
         [0010]    Another advantage is that the driver may be enabled to process all signs along the road. 
         [0011]    Yet another advantage is that, by providing auditory assist, the invention provides fewer distracting visual stimuli and can help those who are visually impaired. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0012]    A better understanding of the present invention will be had upon reference to the following description in conjunction with the accompanying drawings. 
           [0013]      FIG. 1  is a block diagram of one embodiment of a road sign announcing arrangement of the present invention. 
           [0014]      FIG. 2  is a plan view of a road sign that may be read by the road sign announcing arrangement of  FIG. 1 . 
           [0015]      FIG. 3  is a block diagram of another embodiment of a road sign announcing arrangement of the present invention. 
           [0016]      FIG. 4  is a flow chart of one embodiment of a road sign announcing method of the present invention. 
       
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
       [0017]      FIG. 1  illustrates one embodiment of a road sign announcing arrangement  10  of the present invention disposed in a vehicle, and including a camera  12 , a sign shape recognition module  14 , a text recognition module  16 , a sign color and symbol recognition module  18 , a text synthesization module  20 , a text translation module  22 , and a text-to-speech module  24 . 
         [0018]    Camera  12  may be mounted on the front of the vehicle, and may capture images within its field of view along the roadside and/or above the road that the vehicle is traveling on. In one embodiment, two cameras are provided. One of the cameras is directed to the right of and slightly above the vehicle along the roadside, and the other camera is directed straight ahead of and above the vehicle. Thus, the cameras are pointed in directions where road signs are typically posted. 
         [0019]    Sign shape recognition module  14  may recognize the shape of the sign in the images captured by the camera and determine therefrom the type of sign it is. For example, a diamond-shaped sign may be interpreted as a warning of road hazards; a triangular sign may be interpreted as a yield sign; a circular sign may be interpreted as a railroad crossing; an octagonal sign may be interpreted as being a stop sign, etc. For example, camera  12  may capture an image of the diamond-shaped sign shown in  FIG. 2  and sign shape recognition module  14  may interpret the diamond shape of the sign as an indication of a road hazard. 
         [0020]    Text recognition module  16  may recognize the letters and numbers printed on the sign in the images captured by the camera and determine therefrom the text message conveyed by the sign. Text recognition module  16  may determine words that would add more clarity to the text message conveyed by the sign, but that may have been omitted from the sign because of limited area available for printing on the sign. For example, text recognition module  16  may determine that the message conveyed by the sign is “speed limit 25 miles per hour”, wherein “miles per hour” provides more clarity to the message, but may have been omitted from the sign because there was not enough area on the sign to print these words. Such words may also be omitted from the sign because the words may be easily assumed by a driver who sees the sign. However, such words may be more easily assumed by a driver who sees the sign and its familiar format than by a driver who merely has the literal text of the sign read to him. Thus, it may be advantageous to add such words to the audible message spoken to the driver. 
         [0021]    Arrangement  10  may store in memory the speed limit that the vehicle is currently in, which may be determined from previous speed limit signs that have been seen, or from an external database that looks up the speed limit in an external database based on the vehicle&#39;s GPS location. Thus, a processor included in arrangement  10  may ascertain whether the posted speed limit represents an increase or a reduction in the immediately preceding speed limit. 
         [0022]    Sign color and symbol recognition module  18  may recognize the color of the sign and symbols on the sign in the images captured by the camera and determine therefrom the type of sign it is and the information that the sign is conveying. For example, a yellow sign conveys caution; a black and white sign conveys regulations; an orange sign conveys a road construction area; a brown sign conveys a recreation area; a blue sign conveys guidance; an arrow symbol indicates a direction that the sign pertains to, or a direction that the road takes ahead; a stick figure of a human walking indicates a cross walk; an X symbol indicates a railroad crossing; and a circle with an angled line through it indicates that whatever is within the circle is prohibited, etc. For example, camera  12  may capture an image of the sign with the arrow symbol shown in  FIG. 2  and determine that the color of the sign outside of the arrow and speed limit rectangle is orange, indicating road construction, and the arrow indicates a straight-ahead direction. Putting these two elements together, sign color and symbol recognition module  18  may determine that the sign indicates road construction is taking place on the road ahead. 
         [0023]    Text synthesization module  20  receives all of the outputs of sign shape recognition module  14 , text recognition module  16 , and sign color and symbol recognition module  18  and synthesizes a text message that conveys the total or overall message conveyed by the sign. For example, given the outputs of sign shape recognition module  14 , text recognition module  16 , and sign color and symbol recognition module  18  in response to camera  12  capturing images of the sign of  FIG. 2 , text synthesization module  20  may produce a textual message, such as “construction zone ahead; speed is reduced to 25 miles per hour”. 
         [0024]    Text translation module  22  may receive the output of text synthesization module  20  and may translate the output into the language of the driver&#39;s choice. This may be particularly advantageous when the driver is driving in an area where signs are not in his native language. If the output of text synthesization module  20  is already in the driver&#39;s native language, then text translation module  22  may pass the output of text translation module  22  without change to text-to-speech module  24 . 
         [0025]    Text-to-speech module  24  may convert the text output of text translation module  22  into a synthesized speech signal or a pre-recorded speech signal. The signal may then drive a loudspeaker (not shown) to produce audible speech that the driver can hear. 
         [0026]    Another embodiment of a road sign announcing arrangement  200  of the present invention is shown in  FIG. 3 . Arrangement  200  includes a camera  212  and a text recognition module  216  which may be substantially similar to camera  12  and text recognition module  16  of  FIG. 1 , and thus are not described in detail herein. 
         [0027]    The output of text recognition module  216  may be received by an electronic processor  226 , which also receives vehicle location information from an in-vehicle GPS module  228 . Processor  226  is also in bi-directional wireless communication with an external database  230  of sign messages stored in association with geographical locations. Processor  226  may forward the vehicle location information along with vehicle traveling direction information, which may be derived from the changes in vehicle location, to database  230 . Database  230  may interpret this information as a query as to what message is conveyed by a sign that may be visible to the driver. Database  230  may respond to the query by transmitting to processor  226  both the literal word-for-word message on the visible sign and a longer textual interpretation of the sign that is more easily understood in audio form than is the literal word-for-word message. For example, in the case of the sign of  FIG. 2 , database  230  may transmit to processor  226  both “speed limit 25” and “construction zone ahead; speed is reduced to 25 miles per hour”. 
         [0028]    Processor  226  may compare the output of text recognition module  216  to the literal word-for-word message as provided by database  230 . Processor  226  may then, based on the comparison, produce a confidence score indicating the degree of confidence that the word-for-word message as provided by database  230  is the same text that is on the sign in the images captured by camera  212  (i.e., as indicated by the output of text recognition module  216 ). In general, the greater the percentage of words and/or letters in the word-for-word message as provided by database  230  that match the output of text recognition module  216 , the greater the confidence score. 
         [0029]    If the confidence score equals or exceeds a threshold, then processor  226  may transmit the longer textual interpretation of the sign received from database  230  to text translation module  222  for translation to the driver&#39;s chosen or native language. Conversely, if the confidence score is less than the threshold, then processor  226  may transmit the output of text recognition module  216  to text translation module  222  for translation to the driver&#39;s chosen or native language. In either case, text-to-speech module  224  may present the audible translation to the driver. 
         [0030]    In another embodiment, the features of arrangements  10  and  200  are combined. If the confidence score is less than the threshold, then processor  226  may forward the output of text synthesization module  20  to text translation module  222 . 
         [0031]    In yet another embodiment, database  230  provides only the literal word-for-word message on the visible sign to processor  226 . If the confidence score is at least as high as the threshold, then processor  226  may then transmit to text translation module  222  the word-for-word message as provided by database  230  or may transmit the output of text recognition module  216  to text translation module  222 . Alternatively, in the embodiment which combines the features of arrangements  10  and  200 , then processor  226  may forward the output of text synthesization module  20  to text translation module  222 . 
         [0032]    It is further possible in another embodiment, to eliminate camera  212  and text recognition module  216  from arrangement  200 . In this case, processor  226  may simply transmit the output of database  230  to text translation module  230 . 
         [0033]    In still another embodiment, database  230  provides only the longer textual interpretation of the sign to processor  226 . The confidence score may then be based on the degree to which the words and/or letters in the output of text recognition module  216  are present in the longer textual interpretation of the sign from database  230 . If the confidence score is at least as high as the threshold, then processor  226  may transmit to text translation module  222  the longer textual interpretation of the sign. 
         [0034]      FIGS. 1 and 3  each include a plurality of functional blocks or modules. It is to be understood that these modules may all be considered processing devices and may be incorporated in individual electronic processors, in a common single processor, or in some intermediate number of processors. 
         [0035]    In one embodiment, the confidence threshold is adjustable by the driver such that he may receive and hear more or fewer textual messages from the external database. This may be particularly advantageous when a road sign is no longer readable, or when the camera is not in working condition, such as due to the presence of snow or dust, etc. 
         [0036]      FIG. 4  illustrates one embodiment of a road sign announcing method  400  of the present invention. In a first step  402 , an image of a road sign is captured. For example, camera  12  may capture images of road signs within its field of view along the roadside and/or above the road that the vehicle is traveling on. 
         [0037]    Next, in step  404 , the type of the sign in the image is determined based on the shape of the sign. That is, sign shape recognition module  14  may recognize the shape of the sign in the image captured by the camera and determine therefrom the type of sign it is. For example, an isosceles triangle or pennant-shaped sign may be interpreted as a no passing sign. 
         [0038]    In a next step  406 , a text message on the sign is read. For example, text recognition module  16  may recognize the letters and numbers printed on the sign in the image captured by the camera and determine therefrom the text message conveyed by the sign. 
         [0039]    In step  408 , the type of the sign in the image is determined based on the color of the sign and based on symbols on the sign. That is, sign color and symbol recognition module  18  may recognize the color of the sign and symbols on the sign in the image captured by the camera and determine therefrom the type of sign it is and the information that the sign is conveying. For example, the yellow color of a no passing sign conveys caution, and a circle with an angled line through it indicates that whatever is within the circle (e.g., the word “passing” is prohibited). 
         [0040]    Next, in step  410  an overall message conveyed by the sign is synthesized. For example, text synthesization module  20  receives all of the outputs of sign shape recognition module  14 , text recognition module  16 , and sign color and symbol recognition module  18  and synthesizes a text message that conveys the total or overall message conveyed by the sign. 
         [0041]    In step  412 , the message conveyed by the sign is translated into the driver&#39;s language. For example, text translation module  22  may receive the output of text synthesization module  20  and may translate the output into the language of the driver&#39;s choice. 
         [0042]    In a final step  414 , the translated text message conveyed by the sign is converted into audible speech. For example, text-to-speech module  24  may convert the text output of text translation module  22  into a synthesized speech signal or a pre-recorded speech signal. The signal may then drive a loudspeaker to produce audible speech that the driver can hear. 
         [0043]    The foregoing description may refer to “motor vehicle”, “automobile”, “automotive”, or similar expressions. It is to be understood that these terms are not intended to limit the invention to any particular type of transportation vehicle. Rather, the invention may be applied to any type of transportation vehicle whether traveling by air, water, or ground, such as airplanes, boats, etc. 
         [0044]    The foregoing detailed description is given primarily for clearness of understanding and no unnecessary limitations are to be understood therefrom for modifications can be made by those skilled in the art upon reading this disclosure and may be made without departing from the spirit of the invention.