Abstract:
The invention comprises a method and system of recognition and translation, stored on a digital storage device with an operating system and running computer applications, such as a personal computer, which recognizes input by the human computer user and transmits output to the human user, which performs non-optical and optical character recognition of characters displayed on the output device of the digital storage device, which automatically recognizes and translates phrases contiguous to and including the phrase upon which the System is activated and which translates words from one written phrase set to a second written phrase set.

Description:
FIELD OF THE INVENTION 
     The present invention is directed generally to an invention with on-screen optical character recognition capability and with translation and automatic recognition and translation of phrases and is directed more particularly to a recognition and translation system having a digital storage device with input and output capabilities, such as a personal computer with computer software running over an operating system. 
     BACKGROUND OF THE INVENTION 
     At present, optical character recognition systems and translation systems are available. The available optical character recognition systems typically require high resolution printed words. Such optical character recognition systems, thus, can only recognize text printed on paper or other document form and cannot optically recognize characters on-screen which is displayed in a lower resolution. Further, available optical character recognition systems typically recognize text by recognition of the character. Such optical character recognition systems, thus, are dependent upon a specific recognition of the shape of a character. These optical character recognition systems, thus, are limited to paper and to recognition of a character based upon its foreground characteristics. 
     Available translation systems typically are limited to performing translation of words to another language without taking into consideration the use of the word in a phrase or a sentence. These available translation systems, thus, fail to take into consideration the use of a word in a phrase or a sentence to provide the most accurate translation or translations of the word into another language. 
     Translation systems include the Roboword product created by TechnoCraft and a product created by Accent. However, these products do not have the capability of translating words depending upon their use in a phrase or sentence. Furthermore, these systems do not have optical character recognition technology that allows on-screen text recognition or optical character recognition capability which utilizes characteristics such as the background of a screen to recognize a character. Accordingly, in instances where, for example, an accurate translation of a word is dependent upon its use in a phrase, these systems will not accurately provide a translation of the word. Furthermore, these systems are not capable of optically recognizing on-screen text or optically recognizing text by use of characteristics such as the background of a letter. 
     Optical character recognition technology is available, but this existing technology is dependent upon recognition of the foreground of the character. Accordingly, if a character is printed such that the foreground of the character is not understood by the technology, the technology will not be capable of reading it. Furthermore, the existing technology cannot optically recognize characters using characteristics such as the background of a letter in determining the correct printed letter. In addition, existing technology cannot typically optically recognize characters displayed on a screen because of the lower resolution of on-screen displays. 
     Additionally, upon unsuccessful recognition and translation of a word or phrase, recognition and translation systems do not typically provide a list of words or phrases which are linguistically similar to the displayed word or phrase and do not allow the System user to add the displayed word or phrase and the corresponding translation to a user dictionary for future translations. 
     Furthermore, recognition systems and translation systems do not typically incorporate the ability to optically recognize characters in combination with the ability to translate words and phrases. 
     Accordingly, an object of the invention is a recognition and translation system with the ability to optically recognize characters displayed on a digital screen, such as a computer monitor. 
     A further object of the invention is a recognition and translation system which is capable of using characteristics such as the background of the screen to optically recognize characters. 
     Another object of the invention is a recognition and translation system which is capable of optically recognizing characters produced by many different applications displayed on-screen. 
     Still another object of the invention is to provide a recognition and translation system which is capable of accurately translating words from one spoken language to another depending on the use of the word in a phrase or sentence. 
     Another object of the invention is a recognition and translation system which is capable of automatically recognizing and translating phrases in text. 
     Furthermore, an object of the invention is a recognition and translation system which combines the ability to optically recognize characters displayed on screen and the ability to translate words from one spoken language to another depending on the use of the word in a phrase or sentence. 
     A further object of the invention is a recognition system which can be utilized by many different modes of digital display. 
     An addition object of the invention is a recognition and translation system which provides a list of words or phrases which are linguistically similar to the displayed word or phrase upon unsuccessful recognition and translation of a word or phrase. 
     Another object of the invention is a recognition and translation system which allows the System user to add the displayed word or phrase and the corresponding translation to a user dictionary for future translations. 
     SUMMARY OF THE INVENTION 
     These and other objects of the invention, which shall become apparent hereafter, are achieved by providing a recognition and translation system, including a personal computer having a computer monitor and keyboard, a means for connecting a mouse or other point-and-click interface mechanism to the computer system, optical and non-optical character recognition, and translation of words from one spoken language to another in the context in which the words are used in the phrase or sentence. 
     In an embodiment, the invention is stored on or accessible by a computer or other digital storage device which utilizes a digital display device and any operating system, uses three different optical and non-optical methodologies to recognize characters, automatically recognizes and translates phrases, translates words from one spoken language to another by a method which translates the word incorporating its use in the phrase or sentence, provides a list of linguistically similar words or phrases upon unsuccessful recognition and translation, allows the system user to add the word or phrase and its corresponding translation to a dictionary for future reference upon unsuccessful recognition and translation, and has a keystroke or other interactive activation mechanism. 
     In a preferred embodiment, the invention is stored on or accessible by a personal computer or other digital display device running an operating system with a graphical user interface, such as Windows 95 or NT operating system and connected to the Internet, connected to a mouse with a right and left button, providing optical character recognition capability through font-specific and non-font specific analysis of the background behind each letter or letters, non-optical character recognition capability, the ability to automatically recognize and translate words, the ability to translate words from one spoken language to another as the words are used in phrases and sentences, the ability to provide a list of linguistically similar words or phrases upon unsuccessful recognition and translation, and the ability to allow the system user to add the word or phrase and its corresponding translation to a dictionary for future reference upon unsuccessful recognition and translation. 
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS 
     The invention would be better understood from the following Detailed Description of the Preferred Embodiment, with reference to the accompanying drawings, in which: 
     FIG. 1 is a flowchart depicting the operation of the translation system with multiple methods for character recognition and a translation method; 
     FIG. 2 is a flowchart depicting a clipboard recognition method of character recognition; 
     FIG. 3 is a flowchart depicting a font recognition method of character recognition; 
     FIG. 4 is a flowchart depicting an optical recognition method of character recognition; and 
     FIG. 5 is a flowchart depicting a translation method. 
     FIG. 6 is a flowchart depicting a method of providing linguistically similar words and phrases and of allowing the user to add the word and translation to a dictionary upon unsuccessful recognition and translation. 
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
     Referring to the drawings, wherein like numerals reflect like elements throughout the several views, FIG. 1 is a flowchart showing the process embodied by the Recognition and Translation System. The System is initialized  101  and prepares to recognize the method of activation, such as activation by clicking the right button of a mouse  102  or other specific keystroke or human-directed action on any letter in a word and runs in the background of the computer. A computer user then commences an activity, such as connecting to the Internet or commencing creation of a document using a variety of types of document creation software. Once the System is running in the background and the document creation or other application is commenced, the System waits for the computer user to activate it by the activation mechanism, such as the clicking of the right button of a mouse  103 . If the System is running on a Windows application  103 A, upon input by the computer user, the System captures characteristics of the words and phrases as stored by the application  150 , determines the address at which the words and phrases will be displayed by Windows  150 , creates a address database  170  containing the contents of the words and phrases, the corresponding location (address) in the application, and the corresponding final Windows location at which each word or phrase will be displayed  150 . Once the computer user activates the System, it first determines whether this is a Windows application  103 A and if it can recognize the character through using the address database  170 . If the System cannot recognize the characters through use of the address database, it determines whether the application is a clipboard application  104 , that is, those applications for which the System has drivers allowing immediate recognition of the characters by the System and for which the System does not require use of its optical character recognition capability. If the System fails to recognize the application, it determines the font of the text  105  displayed by the application, which utilizes the optical character recognition technology  107 . If the System fails to recognize the font used by the application, it then determines the character solely using optical character recognition technology  107 . If the System is successful in any of these methods of recognizing the character, it commences translation of the word, and automatic recognition and translation of the word&#39;s surrounding phrases, from which the system was activated  112 ,  113 ,  114 , FIG.  5 . If the System is unsuccessful in all three methods of recognizing the characters, it provides the capabilities depicted in FIG. 6,  115 ,  116 . 
     In FIG. 2, the flowchart depicts the process by which the System determines whether the text is displayed by a clipboard application automatically recognized by the System  104 . Referring specifically to FIG. 2, the System determines whether the word from which the application was activated is displayed in a bitmap format  201  by performing a select and mark copy mechanism. If it is a bitmap display, the System must use alternate methods of recognition  201 A, such as the font method (FIG. 3) or the optical character recognition method (FIG.  4 ). If it is not a bitmap display, the System sends the application a message to copy the word and sends a unique application identifier to the application, signaling to the application that it must use application-specific methods of copying  201 . After the System copies the word, it determines whether the word can be part of a phrase  202 ,  500  and then reads and copies the words to the left and right  203 ,  501 . 
     In FIG. 3, the flowchart depicts the process by which the System determines the font used by the application to display the word, using in part the optical character recognition technology. The System first samples all pixel colors in the vicinity of the click point  300  by repeatedly targeting a coordinate on the screen and reading and determining the pixel color at the location of the coordinate  300 A. By repeatedly sampling the pixel colors in the pre-determined area of analysis, the System determines whether any pixel is of the same color as its neighboring pixel in order to determine whether the application utilizes at least one potential foreground color  301 . If the System determines that the application displays at least one potential foreground color, then for each foreground color, the system iteratively creates and examines an expanding rectangle of the display  303  to determine the background, as distinguished from the foreground of the display by reading and comparing each pixel and its respective color  304 . Once the background of the display has been processed sufficiently to determine the character displayed, the System processes the potential character by first comparing the font of the character with fonts already understood by the System  305 , such as Netscape and Internet Explorer fonts, and the Windows standard fonts. The System performs this processing  305  by comparing the pixels in the potential letter with data compiled by the System for various fonts through use of binary characteristics  305 A. The data compiled by the System for various fonts  350  are comprised of various characteristics, which are accessible by an index based upon the width of the letter and the number of pixels in the letter which are not active  305 B. In the event that the pixel comparison method  305 ,  305 A,  305 B is not successful, the System performs the comparison of the displayed letter to the data table with only a small portion of the original “read” rectangle and performs the comparison in an increasingly larger portion of the rectangle until the character is successfully read  305 C and, if successful, the System than performs the same comparison with the remainder of the “read” rectangle  305 D. If the System recognizes the font in these methods, it determines the word displayed by iteratively examining an expanding rectangle of the display  307  by reading and comparing each pixel and its respective color to the stored characteristics of the background of the letter and determining when the word has been “read”  307 A by when sufficient background has been displayed. Once the System has “read” the letter, it processes each potential character in the word by first comparing the font of the character with fonts already understood by the system  308 ,  305 , such as Netscape and Internet Explorer fonts, and the Windows standard fonts. The System performs this processing  308 ,  305  by comparing the pixels in each potential letter of the word with data compiled by the System for various fonts through use of binary characteristics  305 A. The data compiled by the System for various fonts are comprised of various characteristics, which are accessible by an index based upon the width of the letter and the number of pixels in the letter which are not active  305 B. In the event that the pixel comparison method  308 ,  305 A,  305 B is not successful, the System performs the comparison of the displayed letter to the data table with only a small portion of the original “read” rectangle and performs the comparison in an increasingly larger portion of the rectangle until each character of the word is successfully read  305 C and, if successful, the System than performs the same comparison with the remainder of the “read” rectangle  305 D. 
     In FIG. 4, the flowchart depicts the process by which the System utilizes optical character recognition technology to recognize the text displayed by the application. The System iteratively creates and examines an expanding rectangle of the background of the display to read characteristics of a letter (the foreground) in the word displayed  400 . The System determines the characteristics of a letter by first analyzing the background, as distinguished from the foreground, of the displayed letter in the word  401 , pixel by pixel. The System compares the “white space” characteristics of the letter with known characteristics of a letter, using information compiled on each letter, its defined areas of “white space”, and the position of the “white space” in the letter  401 A,  401 B. Once it has retrieved sufficient data about the letter of the displayed word, the system processes the potential letter by first comparing the characteristics of the “white space” of the displayed potential letter with characteristics of “white space” of other letters already understood by the system, such as Netscape and Internet Explorer fonts, and the Windows standard fonts  402 . The data compiled by the System for various “white space” are comprised of various characteristics of “white space” in various letters, which is indexed by as unique an index as possible to facilitate the comparison  401 B. The System calculates the similarity of each “read” letter to the potential letters  402 A. 
     For each letter in the word, the system repeats  403  the foregoing process  402  by examining and analyzing the next letter to the right and to the left of the displayed letter  405  until the entire word has been read, by iteratively repeating the process until the System reads a “period”, “comma”, or punctuation  403 . Once the entire word has been optically recognized in this manner by the System, it determines all permutations of potential words  404  based upon characteristics of the “read” word. The System rejects any potential letter combinations which cannot logically exist based upon a table which is created by the system  406 . Once any illogical letter combinations are eliminated, the System analyzes all remaining potential words  407  by, for each letter, determining the probability of the “read” letter to each potential letter  407 A, assigning a probability to each potential letter  407 B, and rejecting any potential letters below a certain probability  407 C. Once all probable letters are tabulated, potential words based upon probable letters are determined and the System analyzes the probability, based upon letter characteristics, of each potential word  407 D. The word with the highest probability is returned to the computer user either as an exact match (100% probability)  409  or a most similar match (less than 100% similarity)  408 . 
     In FIG. 5, the flowchart depicts the translation process by which the System determines the most accurate translation of the displayed and recognized word based upon the use of the word in a phrase or sentence, The System determines whether the displayed and selected word is appears in any word or phrase understood by the System  500 ,  202  and determines a list of potential phrases  503 ,  500 A. If the displayed word is recognized, the System iteratively examines each word to the right and to the left of the word  501 ,  203  in combination with the word already “read” by the System. For each new and expanding combination of words (except for the first combination of words), the System searches and determines whether the current combination of words are contained in any phrases in the list maintained by the System  500 A. If found, the phrase is maintained in the list  500 A; if not found, the phrase is deleted from the list  500 A. The System repeats the process  502  until (1) a certain predetermined stop point is reached; or (2) the combination does not match any phrase contained in the list  500 A maintained by the system  504  for either the right or left side  504 , upon which the system stops iteratively examining the combination of words for the unsuccessful side  504 , but continues to search on the successful side  502 . When no combination of words matches any phrase contained in the list  500 A maintained by the System for words either on the left or right of the displayed word, the System determines whether any potential phrases remain in the list  500 A,  507 . The System then determines which potential phrases  500 A are most likely to most accurately translate the displayed word on a word by word basis  508 . The System determines the probability of an accurate translation  508  by determining the importance of the words which do not match and assessing a penalty against the potential phrase dependent upon the importance of the non-matching word  508 A. The system then returns the translations most likely to accurately translate the displayed word  509 . 
     In FIG. 6, the flowchart depicts the capabilities of the System upon unsuccessful recognition and translation of the word or phrase. If the System is unsuccessful at optically and non-optically recognizing the displayed word or phrase  600 ,  115 , it determines the linguistics of the displayed word or phrase  601 , determines the linguistics of similar words  602 , compares the linguistics of the displayed word or phrase with the linguistics of similar words or phrases  603 , compiles a list of linguistically similar words and phrases  604 ,  605  and displays the list of linguistically similar words to the System user  606 . If the System is unsuccessful at translating the word or phrase  600 A,  116 , it allows the user to add the word or phrase  620  and its translation  621  to a dictionary to be accessed in the future  622 ,  623 . 
     While the preferred and alternate embodiments of the system have been depicted in detail, modifications and adaptations may be adapted thereto, without departing from the spirit and scope of the system as delineated in the following claims: