A Fast Morphological Algorithm with Unknown Word Guessing Induced by a Dictionary for a Web Search Engine

This paper describes a {simple yet practical} algorithm of morphological analysis and synthesis that uses a limited dictionary to obtain {rather precise} morphology of a wide lexical coverage. It doesn't imply any particular morphological model of the dictionary and is implemented for Russian, Polish and English. The design of the algorithm fits the constrains of a modern web search engines. Preamble: does a web search engine need morphology? Some IR studies showed very little or no improvements in search effectiveness with morphological analysis applied [harman1991]. Though later results proved success even in languages that typically have few word forms such as English [hull1996] it is still under the question if a web search engine needs morphologic analysis at all. Google, the #1 ranked search engine in the world, claims that it intentionally doesn't use stemming to provide the most accurate results to its users [google1998]. Such an extreme position was a good revisionist point in the late 90s before the link analysis had improved the precision of web search. But is recall really unnecessary? Here are some arguments pro. Number of pages that a web search engines return are distributed according to a Zipf law, which means that though on popular queries it is very big, the amount of queries with only few found documents is substantial. For example more than 30% of responses received at www.yandex. ru on the 06.09. 2001 contained less than 100 documents while only 5-10% of queries have misspelling. The user behaviour, coordinate ranking and morphology Another observation in favour of morphological processing on the web is the searchers behaviour and expectations. Users have their own mental models of how search engines should work [muramatsu2001] and they generally prefer coordinate-level ranking [heimstra2002]. To satisfy this model (please note that we don’t mention precision and recall as the primary goals) Web search engines first look for the exact phrase matching. When it is impossible (more 50% of queries doesn't have a “phrase match” anywhere on the web) search engines rank results according to the term proximity [muramatsu2001] so the responses are as "similar" to queries as possible. One may notice looking at the typical web search engine response that documents are roughly listed in the decreasing order of the edit distance (“likelihood”) between the query and snippets of the documents. In the same time 75% of web search queries consist from more than 1 word [onestat2003]. So why the "similarity" of responses to longer queries allows insertion of big spans of characters (words, sentences) between search terms but doesn’t' allow small changes of terms' suffixes when terms are close to each other? Introduction & current trends The state-of-the-art algorithms for morphological analysis are based on the Finite State Transducers of different flavours ([kimo1983], [kartunen1992], [mohri2000], [daciuk2000]), where phonological rules are described by hand and each word from the dictionary is assigned its corresponding (inflectional/derivational) model. These transducers allow both analysis and synthesis of the word forms; they are very fast and robust. Unfortunately number of unknown lexical items brought from the web with every crawl are very big and can be estimated by Heaps law VR = Kn, where VR is the portion of the vocabulary represented by the instance text of size n. K and β are free parameters determined empirically. On the web β is closer to 1.0 than to 0.5 due to (un) intentional misspellings, foreign words, privately invented words such as taken from blogs etc. This means that the dictionary grows sub linear with grow of index. One approach here is simply to ignore all the words that are not from main dictionary [www.aport. ru] hoping that the coverage will suffice. Recently the problem of the morphological analysis of the unknown words draws more attention mostly from the success in unsupervised learning of morphology from the corpora itself. [Jackuemin1997] analysed statistics of k-n conflations (where k – length of the hypothesized suffix, usually 3 letters, and n – the width of the context window – usually 2 words) and obtained decent results compared to Lovins and Porter famous hand-made stemmers ([lovins1968], [porter1980]). [gaussier1999] used similar approach (p-similarity – the number of common letters between two words) to learn derivational morphological families and even derivational trees. The more generalized approaches of corpus suffix analysis are presented in [goldsmith2001] and [snover2002]. They use MDL (Minimum Description Length) principle, known from theory of information, to produce the set of stems, signatures (paradigms) and affixes that can be encoded in the minimal number of bits [goldsmith2001] or that are most probable (Brent) thus making morphological description of corpus minimal in some sense. Linguistica [goldsmith2001] is also a publicly available tool that we will use later in this paper. Another promising approach [schone2001] makes use of word contexts to distillate hypothesized models obtained from pure suffixal analysis. It uses the primitive of PPMV — pair of potential morphological variants and measures semantic similarity between them, which is understood as the normalized correlation between their vectors in the latent semantic space (which in fact is word-to-wordneighbours matrix factorised with the help of LSI). All of the approaches mentioned above are unsupervised in a sense that they don’t use any prior knowledge about a language and its morphological models. Unfortunately there are at least two obstacles that don’t let direct use of the unsupervised methods in Web search engines as is. The first is the quality of the morphology of the most used words that must be as precise as possible. The second is that the complicating of index updates: index is distributed over hundred and thousands of computers, so the morphological interpretation of the new words that appear in daily crawls must not change too often or it may require daily patching of the index. It worth noting that if a language has a web corpus substantial enough to be of a public interest and of a commercial search engine to strive on, it usually has plenty of dictionaries that already accumulated some knowledge. For example widespread spelling tools that use compact description of all possible word forms such as [ispell] or [aspell] have dictionaries for 37 and 26 languages respectively. Of course the task of the compact listing of all allowed forms in a language is not exactly the task of describing all words and its respecting paradigms, nevertheless these dictionaries are successfully used in the web search engines ([mnogosearch], [glimpse], [aspseek] etc). Important observation about existing dictionaries that they always (and hopefully correct) describe the most frequently used words of the language. Here we suppose that all the complex and non-trivial [inflectional/derivational] models always belong to the most used words of the language -which is in fact another law of Zipf -“the greater the frequency of a word, the greater the number of morphologically complex forms it will be used in” [manning1999]). The idea of applying a relatively small dictionary to guess new words morphology is not new. For example [woods1999] used a set of aggressive rules together with a small lexicon to improve lexical coverage, [mikheev1995] presented a technique of automatic acquisition of unknown words morphology through a general-purpose lexicon and word frequencies, the classic Russian morphology algorithm [belonogov1985] used the same dictionary both for known and unknown words, also all the FST machines allow to write catch-all rules for the words that are not in the dictionary. While the process of describing particular word morphology seem to be deterministic, the catchall rules are rather hard to write and debug. The example of applying a one-time learning procedure together with the existing morphological dictionary to the corpus in order to obtain morphological guesser is presented in [kovalenko2002]. The approach described here is based on [segalovich1998] and uses a very simple idea of morphological similarity. Unknown word morphology is taken from all the closest words from dictionary, where the closeness is the number of letters on its end. From the [belonogov1985] approach it differs in that it doesn’t choose one variant, but rather take all possible variants that have the same common length, and unlike [kovalenko2002] it uses for the induction of the hypothesizes the whole dictionary not just N several last letters of stems. To build such dictionary we don’t need to know the morpheme structure or any particular formalism of any particular morphology, the compiler accepts input in the form of full listing of all word paradigms from the dictionary. If the grammar tags are available they are accepted either. Such a dictionary is be able not only to analyse known words, but also to synthesize all forms both known and hypothesized. The algorithmic idea behind this algorithm is that after we analysed the word trying to find its exact form in the dictionary we already have enough information (we already looked at all its letters) to find the most probable inexact equivalent. The FST approach also use one precompiled automata with exact and catch-all rules but unlike [Karttunen1992] we don’t require catch-all rules, we rather consider them harmful and completely rely on the probabilistic nature of the unknown words analysis. This is the idea of the algorithm – use deterministic knowledge when possible (for all the ‘known’ words) and probabilistic approach when the exact interpretation is not available. An interesting application of this algorithm is the combined dictionary of several languages which is useful in the situations when the language is ambiguous, so the combined dictionary will choos