TO FIND THE POS TAG OF UNKNOWN WORDS IN PUNJABI LANGUAGE

The accuracy of unknown words in the task of Part of Speech tagging is one significant area where there is still room for improvement. Because of their high information content, unknown words are also disproportionately important for how often they occur, and increase in number when experimenting with corpora from different domains. One area however, where all POS tagging methods suffer a significant decrease in accuracy, is with unknown words. These words are those that are seen for the first time in the testing phase of the tagger, having never appeared in the training data. In general, on POS tagging as well as other similar NLP tasks, accuracy on unknown words is about 10% less than words that have been seen in the training data (Brill, 1994). Unknown words also occur a significant amount of the time, comprising approximately 5% of a test corpus (Mikheev, 1997). Introduction to Unknown Words Part of Speech (POS) tagging involves assigning basic grammatical classes such as verb, noun and adjective to individual words, and is a fundamental step in many Natural Language Processing (NLP) tasks. The tags it assigns are used in other processing tasks such as chunking and parsing, as well as in more complex systems for question answering 117 Research Cell: An International Journal of Engineering Sciences ISSN: 2229-6913 Issue July 2011, Vol. 1 © 2011 Anu Books and automatic summarisation. All POS taggers suffer a significant decrease in accuracy on unknown words, that is, words that have not been previously seen in the annotated training set. A loss of up to 10% is typical for most POS taggers e.g. Brill (1994) and Ratnaparkhi (1996)[1]. This decreased accuracy has a flow on effect for the accuracy of both following POS tags and later processes which utilize them. Unknown words also occur a significant amount of the time, ranging from 2%– 5%(Mikheev, 1997), depending on the training and test corpus. These figures are much higher for domains with large specialist vocabularies, for example biological text. We improve the performance of a Maximum Entropy POS tagger by implementing features with non-negative real values. Although Maximum Entropy is typically described assuming binary-valued features, they are not in fact required to be binary valued. The only limitations come from the optimization algorithm. For example, the Generalised Iterative Scaling (Darroch and Ratcliff, 1972) algorithm used in these experiments imposes a non-negativity constraint on feature values. Real-valued features can encapsulate contextual information extracted from around unknown word occurrences in an unannotated corpus. Using a large corpus is important because this increases the reliability of the real-values. By looking at the surrounding words, we can formulate constraints on what POS tag(s) could be assigned. This can be seen in the sentence below: (1) pkg{ i h efj zd fe g[Zso N? B; aB fdb B{z eJh fpwkohnK bk fdzdh j ? Here, N? B; aB is the unknown word (word from other language) which, as competent speakers of the language, we can surmise is probably a noun or adjective. This is because it sits between two nouns, which is a position frequently assumed by words with these syntactic categories. Also, if we can find the word N? B; aB in other places, then we can get an even better, more reliable idea of what its correct tag should be. Since we see it so often, we know the types of words that follow it quite well. Other words that occur less frequently don’t give as strong an indication of what is to follow, simply because the evidence is sparser. Our aim then, is to take this intuitive reasoning for determining the correct tag for an unknown word. Unknown word processing POS taggers have reached such a high degree of accuracy that there remain few areas where performance can be improved significantly. Unknown words are one of these areas, with state-of-the-art accuracy in the range 85 – 88%, which is well below the 97% accuracy achievable over all words. The prevalence of unknown words is also problematic, although somewhat dependant on the size and type of corpus being used. Rports shows that training on sections 0–18 of the Penn Treebank (Marcus et al., 1993), and test on sections 22–24. This test set then contains 2.81% (approximately 4000) unknown words. Also, when applying a POS tagger to a specialized