It us widely acknowledged that word sense d~samblguatmn (WSD) us a central problem m natural language processing In order for computers to be able to understand and process natural language beyond simple keyword matching, the problem of d~samblguatmg word sense, or dlscermng the meamng of a word m context, must be effectively dealt with Advances in WSD v, ill have slgmficant Impact on apphcatlons hke information retrieval and machine translation For natural language subtasks hke part-of-speech tagging or s)ntactm parsing, there are relatlvely well defined and agreed-upon cnterm of what it means to have the "correct" part of speech or syntactic structure assigned to a word or sentence For instance, the Penn Treebank corpus (Marcus et al, 1993) pro~ide~,t large repo.~tory of texts annotated w~th partof-speech and s}ntactm structure mformatlon Tv.o independent human annotators can achieve a high rate of agreement on assigning part-of-speech tags to words m a g~ven sentence Unfortunately, th~s us not the case for word sense assignment F~rstly, it is rarely the case that any two dictionaries will have the same set of sense defimtmns for a g~ven word Different d~ctlonanes tend to carve up the "semantic space" m a different way, so to speak Secondly, the hst of senses for a word m a typical dmtmnar~ tend to be rather refined and comprehensive This is especmlly so for the commonly used words which have a large number of senses The sense dustmctmn between the different senses for a commonly used word m a d~ctmnary hke WoRDNET (Miller, 1990) tend to be rather fine Hence, two human annotators may genuinely dusagree m their sense assignment to a word m context The agreement rate between human annotators on word sense assignment us an Important concern for the evaluatmn of WSD algorithms One would prefer to define a dusamblguatlon task for which there us reasonably hlgh agreement between human annotators The agreement rate between human annotators will then form the upper ceiling against whmh to compare the performance of WSD algorithms For instance, the SENSEVAL exerclse has performed a detaded study to find out the raterannotator agreement among ~ts lexicographers taggrog the word senses (Kllgamff, 1998c, Kllgarnff, 1998a, Kflgarrlff, 1998b) 2 A Case Study In this-paper, we examine the ~ssue of raterannotator agreement by comparing the agreement rate of human annotators on a large sense-tagged corpus of more than 30,000 instances of the most frequently occurring nouns and verbs of Enghsh This corpus is the intersection of the WORDNET Semcor corpus (Miller et al, 1993) and the DSO corpus (Ng and Lee, 1996, Ng, 1997), which has been independently tagged wlth the refined senses of WORDNET by two separate groups of human annotators The Semcor corpus us a subset of the Brown corpus tagged with ~VoRDNET senses, and consists of more than 670,000 words from 352 text files Sense taggmg was done on the content words (nouns, ~erbs, adjectives and adverbs) m this subset The DSO corpus consists of sentences drawn from the Brown corpus and the Wall Street Journal For each word w from a hst of 191 frequently occurring words of Enghsh (121 nouns and 70 verbs), sentences containing w (m singular or plural form, and m its various reflectional verb form) are selected and each word occurrence w ~s tagged w~th a sense from WoRDNET There ~s a total of about 192,800 sentences in the DSO corpus m which one word occurrence has been sense-tagged m each sentence The intersection of the Semcor corpus and the DSO corpus thus consists of Brown corpus sentences m which a word occurrence w is sense-tagged m each sentence, where w Is one of.the 191 frequently oc-,currmg English nouns or verbs Since this common pomon has been sense-tagged by two independent groups of human annotators, ~t serves as our data set for investigating inter-annotator agreement in this paper 3 Sentence Matching To determine the extent of inter-annotator agreement, the first step ~s to match each sentence m Semcor to its corresponding counterpart In the DSO corpus This step ~s comphcated by the following factors 1 Although the intersected portion of both corpora came from Brown corpus, they adopted different tokemzatmn convention, and segmentartan into sentences differed sometimes 2 The latest versmn of Semcor makes use of the senses from WORDNET 1 6, whereas the senses used m the DSO corpus were from WoRDNET 15 1 To match the sentences, we first converted the senses m the DSO corpus to those of WORDNET 1 6 We ignored all sentences m the DSO corpus m which a word is tagged with sense 0 or -1 (A word is tagged with sense 0 or -1 ff none of the given senses m WoRDNFT applies ) 4, sentence from Semcor is considered to match one from the DSO corpus ff both sentences are exactl) ldent~cal or ff the~ differ only m the pre~ence or absence of the characters " (permd) or -' (hyphen) For each remaining Semcor sentence, taking into account word ordering, ff 75% or more of the words m the sentence match those in a DSO corpus sentence, then a potential match ~s recorded These i -kctua\[ly, the WORD~q'ET senses used m the DSO corpus were from a shght variant of the official WORDNE'I 1 5 release Th~s ssas brought to our attention after the pubhc release of the DSO corpus potential matches are then manually verffied to ensure that they are true matches and to ~eed out any false matches Using this method of matching, a total of 13,188 sentence-palrs contasnmg nouns and 17,127 sentence-pa~rs containing verbs are found to match from both corpora, ymldmg 30,315 sentences which form the intersected corpus used m our present study 4 The Kappa Statistic Suppose there are N sentences m our corpus where each sentence contains the word w Assume that w has M senses Let 4 be the number of sentences which are assigned identical sense b~ two human annotators Then a simple measure to quantify the agreement rate between two human annotators Is Pc, where Pc, = A/N The drawback of this simple measure is that it does not take into account chance agreement between two annotators The Kappa statistic a (Cohen, 1960) is a better measure of rater-annotator agreement which takes into account the effect of chance agreement It has been used recently w~thm computatmnal hngu~stlcs to measure raterannotator agreement (Bruce and Wmbe, 1998, Carletta, 1996, Veroms, 1998) Let Cj be the sum of the number of sentences which have been assigned sense 3 by annotator 1 and the number of sentences whmh have been assigned sense 3 by annotator 2 Then P~-P~ 1-P~ where M j=l and Pe measures the chance agreement between two annotators A Kappa ~alue of 0 indicates that the agreement is purely due to chance agreement, whereas a Kappa ~alue of 1 indicates perfect agreement A Kappa ~alue of 0 8 and above is considered as mdmatmg good agreement (Carletta, 1996) Table 1 summarizes the inter-annotator agreement on the mtersected corpus The first (becond) row denotes agreement on the nouns (xerbs), wh~le the lass row denotes agreement on all words combined The a~erage ~ reported m the table is a s~mpie average of the individual ~ value of each word The agreement rate on the 30,315 sentences as measured by P= is 57% This tallies with the figure reported ~n our earlier paper (Ng and Lee, 1996) where we performed a quick test on a subset of 5,317 sentences,n the intersection of both the Semcor corpus and the DSO corpus 10 \[\] mm m m m m m mm m m m m mm m m m Type Num of v, ords A N \[ P~ Avg Nouns 121 7,676 13,188 I 0 582 0 300 Verbs 70 9,520 17,127 I 0 555 0 347 All I 191 I 17,196 30,315 I 056T 0317 Table 1 Raw inter-annotator agreement 5 Algorithm Since the rater-annotator agreement on the intersected corpus is not high, we would like to find out how the agreement rate would be affected if different sense classes were in use In this section, we present a greedy search algorithm that can automatmalb derive coarser sense classes based on the sense tags assigned by two human annotators The resulting derived coarse sense classes achmve a higher agreement rate but we still maintain as many of the original sense classes as possible The algorithm is given m Figure 1 The algorithm operates on a set of sentences where each sentence contains an occurrence of the word w whmh has been sense-tagged by two human annotators At each Iteration of the algorithm, tt finds the pair of sense classes Ct and Cj such that merging these two sense classes results in the highest t~ value for the resulting merged group of sense classes It then proceeds to merge Cz and C~ Thin process Is repeated until the ~ value reaches a satisfactory value ~,~t,~, which we set as 0 8 Note that this algorithm is also applicable to deriving any coarser set of classes from a refined set for any NLP tasks in which prior human agreement rate may not be high enough Such NLP tasks could be discourse tagging, speech-act categorization, etc 6 Results For each word w from the list of 121 nouns and 70 verbs, ~e applied the greedy search algorithm to each set of sentences in the intersected corpus contaming w For a subset of 95 words (53 nouns and 42 verbs), the algorithm was able to derive a coarser set of 2 or more senses for each of these 95 words such that the resulting Kappa ~alue reaches 0 8 or higher For the other 96 words, m order for the Kappa value to reach 0 8 or higher, the algorithm collapses all senses of the ~ord to a single (trivial) class Table 2 and 3 summarizes the results for the set of 53 nouns and 42 ~erbs, respectively Table 2 md~cates that before the collapse of sense classes, these 53 nouns have an average of 7 6 senses per noun There is a total of 5,339 sentences in the intersected corpus containing these nouns, of which 3,387 sentences were assigned the same sense by the two groups of human annotators The average Kappa statistic (computed as a simple average of the Kappa statistic of ~he mdlwdual nouns) is 0 463 After the collapse of sense classes by the greedy search algorithm, the average number of senses per noun for these 53 nouns drops to 40 Howe~er, the number of sentences which have been asmgned the same coarse sense by the annotators increases to 5,033 That is, about 94 3% of the sentences have been assigned the same coarse sense, and that the average Kappa statistic has improved to 0 862, mgmfymg high rater-annotator agreement on the derived coarse senses Table3 gl~es the analogous figures for the 42 verbs, agmn mdmatmg that high agreement is achieved on the coarse sense classes den~ed for verbs 7 Discussion Our findings on rater-annotator agreement for word sense tagging indicate that for average language users, it is quite dl~cult to achieve high agreement when they are asked to assign refned sense tags (such as those found in WORDNET) given only the scanty definition entries m the WORDNET dlctionary and a few or no example sentences for the usage of each word sense Thin observation agrees wlth that obtmned m a recent study done by (Veroms, 1998), where the agreement on sense-tagging by naive users was also not hlgh Thus It appears that an average language user is able to process language wlthout needing to perform the task of dlsamblguatmg word sense to a very fine-grained resolutmn as formulated m a tradltlonal dmtlonary In contrast, expert lexicographers tagged the ~ ord sense in the sentences used m the SENSEVAL exerclse, where high rater-annotator agreement was reported There are also fuller dlctlonary entries m the HECTOR dlctlonary used and more e<amples showing the usage of each word sense m HECTOR These factors are likely to have contributed to the difference in rater-annotator agreement observed m the three studies conducted We also examined the coarse sense classes derived by the greedy search algorithm Vv'e found some interesting groupings of coarse senses for nouns which ~e hst in Table 4 From Table 4, it is apparent that the greedy search algorithm can derive interesting groupings of word senses that correspond to human mtmtwe judgment of sense graz}.ulanty It Is clear that some of the disagreement between the two groups of human annotators can be attributed solely to the overly refined senses of WoRDNET As an example, there is a total Ii loop: let Ct,, C M denote the current M sense classes ~* +--oo for all z,3 such that 1 <, < 3 < M let C\[,,C~w_ 1 denote the resulting M 1 sense classes by mergmg C, and C 3 compute ~(C\[,, C~/_t) ff ~(C,, C~4_x) > ~* then ~" +~(C~,,C~_t), z* +~, ~* +end for merge the sense class C,.