Context based spelling correction

Some mistakes in spelling and typing produce correct words, such as typing “fig” when “fog” was intended. These errors are undetectable by traditional spelling correction techniques. In this paper we present a statistical technique capable of detecting and correcting some of these errors when they occur in sentences. Experimental results show that this technique is capable of detecting 76% of simple spelling errors and correcting 73%. A traditional spelling error detection program determines that a word is misspelled only when the word does not appear in the program’s dictionary of known words. That is, if a word is in the dictionary it is determined to be correctly spelled, and conversely, if a word is not in the dictionary it is noted as potentially misspelled. Since some misspellings (or typing errors) result in a word contained in the dictionary, there are errors that are undetectable using the traditional technique. For example, when attempting to detect errors in the sentence I saw rhe r~lan it t&e park., the error produced by transforming in into it is undetectable. However, some of these errors are detectable by employing syntactic, semantic, or pragmatic knowledge about language. In the previous example, syntactic knowledge could be employed to determine that the sentence is syntactically ill-formed, and thus may potentially contain an error. in addition to performing spelling error detection, many programs also assist in spelling error correction by offering a set of candidate corrections that are “close” to the misspelled word. Here also, linguistic knowledge could be used to suggest likely corrections. Continuing the example, syntactic knowledge determines that substituting irs for it produces a well-formed sentence, whereas substituting ds does not. The task underlying context based spelling correction is determining the relative degree of syntactic or semantic well-formedness among alternative sentences. The question is in what form linguistic knowledge should be represented and utilized to aid in this determination, For example, one possible form for representing syntactic knowledge is as a grammar. A parser could then determine which of the alternative sentences are those accepted by the grammar. There is a vast amount of knowledge that has been gathered by linguists and applied in computational settings. Nevertheless, we judge that applying linguistic knowledge in this form is a substantial undertaking. We have studied the effectiveness of applying a statistical model of language that has demonstrate success in the task of speech recognition [ 11. In this model, syntactic, semantic, and pragmatic knowledge is conflated into word trigram conditional probabilities. These conditional probabilities are derived from statistics gathered from large bodies of text. Using the traditional approach to spelling error detection, a tension exists between the size of the dictionary and the number of undetectable spelling errors. As the number of words in the dictionary increases so does the number of undetectable spelling errors. For example, in the sentence, Isubmit chat is what is happening in this case., the misspelling of that as chat would be detected if chat were not in the ~ctionary. On the other hand, if chat were in the dictionary the misspelling would not be detected. In [2] it was found that the (frequency weighted) number of potential undetected typing errors as a percentage of all typing errors ranges from 10% for a 50,000 word dictionary to 15% for a 350,000 word