,.~.eqmvalent ot duty in a parallel French text, the correct sense of the Enghsh word is identified These studies exploit th~s lnformatmn m order to gather co-occurrence data for the different senses, which ts then used to dtsamb~guate new texts In related work, Dywk (1998) used patterns of translational relatmns in an EnghshNorwegian paralle ! corpus (ENPC, Oslo Umverslty) to define semantic propemes such as synonymy, ambtgmty, vagueness, and semantic helds and suggested a derivation otsemantic representations for signs (eg, lexemes), captunng semantm relatmnshlps such as hyponymy etc, fiom such translatmnal relatmns Recently, Resnlk and Yarowsky (1997) suggested that fol the purposes ot WSD, the different senses of a wo~d could be detelmlned by considering only sense d~stmctmns that are lextcahzed cross-hngmstlcally In particular, they propose that some set of target languages be ~dent~fied, and that the sense d~stmctmns to be considered for language processing appllcatmns and evaluatmn be restricted to those that are reahzed lexlcally in some minimum subset of those languages This idea would seem to p~ovtde an answer, at least m part, to the problem of determining different senses of a word mtumvely, one assumes that ff another language lexlcahzes a word m two or more ways, there must be a conceptual monvatmn If we look at enough languages, we would be likely to fred the s~gmficant lexlcal differences that dehmtt different senses of a word However, th~s suggestmn raises several questions Fo~ instance, ~t ~s well known that many amb~gumes are preserved across languages (for example, the French tntdrYt and the Enghsh interest), especmlly languages that are relatively closely related Assuming this problem can be overcome, should differences found m closely related languages be given lesser (or greater) weight than those found m more distantly related languages 9 More generally, which languages should be considered for this exermse 9 All languages 9 Closely related languages9 Languages from different language famlhes '~ A mixture of the two 9 How many languages, and of which types, would be "enough" to provide adequate lnfotmanon tot this purpose~ There ts also the questmn ot the crlterm that would be used to estabhsh that a sense distinction is "lexlcahzed cross-hngu~stmally" How consistent must the d~stlnCtlOn be 9 Does it mean that two concepts are expressed by mutually non-lntetchangeable lexmal items in some slgmficant number ot other languages, or need tt only be the case that the option ot a different lexlcahzatlon exists m a certain percentage of cases 9 Another conslderatmn ts where the cross-hngual mformatlon to answer these questmns would come from Using bdmgual dictionaries would be extremely tedmus and error-prone, g~ven the substantial d~vergence among d~ctlonanes in terms of the kinds and degree of sense dlstmctmns they make Resmk and Yalowsky (1997) suggest EutoWordNet (Vossen, 1998) as a possible somce of mformatmn, but, given that EuroWordNet ts pttmatdy a lexmon and not a corpus, ~t is subject to many of the same objections as for bl-hngual dictionaries An alternative would be to gather the reformation from parallel, ahgned corpma Unlike bilingual and muttt-hngual dictionaries, translatmn eqmvalents xn parallel texts a~e determined by experienced translatols, who evaluate each instance ot a word's use m context rather than as a part of the meta-hngmst~c actlvlty of classifying senses for mclusmn in a dictionary However, at present very few parallel ahgned corpora exist The vast majority ot these are bl-texts, mvolwng only two languages, one of which is very often English Ideally, a serious 53 evaluation of Resnik and Yarowsky's proposal would include parallel texts m languages from several different language families, and, to maximally ensure that the word m question is used in the exact same sense across languages, ~t would be preferable that the same text were used over all languages in the study The only currently avadable parallel corpora for more than two languages are Olwell's Nmeteen Eighty-Four (Erjavec and Ide, 1998), Plato's Repubhc (Erjavec, et al, 1998), the MULTEXT Journal .o/ the Commt.~ston corpus (Ide and V6roms, 1994), and the Bible (Resnlk, et al, m press) It is likely that these corpora do not provide enough appropriate data to reliably determine sense distinctions Also, ~t Is not clear how the lexlcahzatlon of sense distractions across languages Is affected by genre, domain, style, etc Thls paper attempts to provide some prehmlnary answers to the questions outhned above, In order to eventually determine the degree to which the use of parallel data ts vmble to determine sense distinctions, and, ff so, the ways in which th~s reformation might be used Given the lack of lalge parallel texts across multiple languages, the study is necessarily hmlted, however, close exammanon of a small sample of parallel data can, as a first step, provide the basis and dlrectmn for more extensive studies 1 Methodology I have conducted a small study using parallel, aligned versmns ot George Orwell's Nineteen Etghtv-Fo,lr (Euavec and Ide, 1998)m five languages Enghsh, Slovene, Estonian, Romanlan, and Czech I The study therefole Involves languages from four language families The O~well parallel corpus also includes vers|ons o) Ntneteen-E~gho Four m Hungarian, Bulgarmn, Latwan, Llthuaman, Se~bmn, and Russmn (Germanic, Slavic, Fmno-Ugrec, and Romance), two languages from the same family (Czech and Slovene), as well as one non-Indo-European language (Estoman) Nmeteen Eighty-Four Is a text of about 100,000 words, translated directly from the original English m each of the other languages The parallel versions of the text are sentence-aligned to the English and tagged for part of speech Although Nineteen Eighty-Four is a work of fiction, Orwell's prose IS not highly stylized and, as such, it provides a reasonable sample ot modern, ordinary language that ~s not tied to a given topic or sub-domain (such as newspapers, technical reports, etc ) Furthermore, the translations of the text seem to be relatively faithful to the original for instance, over 95% ot the sentence alignments in the full pmallel corpus of seven languages are one-to-one (Prlest-Dorman, et al, 1997) Nine ambiguous English words were considered hard, head, country, hne, promise, shght, seize, scrap, float The first four were chosen because they have been used in other dlsamb~guatlon studies, the latter five were chosen from among the words used m the Senseval dlsamblguatlon exercise (Kllgamff and Palmer, forthcoming) In all cases, the study was necessarily hmlted to words that occurred frequently enough in the Orwell text to warrant consideration F~ve hundred forty-two sentences conta|nmg an occurrence or occurrences (Including morphological variants) of each of the nine words were extracted from the Enghsh text, together w~th the parallel sentences m which they occur m the texts ot the four comparison languages (Czech, Estonian, Romantan, Slovene) As Walks and Stevenson (1998) have pointed out, pa~t-of-speech tagging accomplishes a good portion of the work ot semantic dlsamb~guatmn, therefore occmrences of wolds that appemed in the data in more than 54 one part of speech were grouped separately 2 The Enghsh occurrences were then grouped usmg the sense distinctions m WordNet, (version 1 6) \[Miller et al, 1990, Fellbaum, 1998\]) The sense categonzatmn was performed by the author and two student assistants, results from the three were compared and a final, mutually agreeable set of sense assignments was estabhshed For each of the four comparison languages, the corpus of sense-grouped parallel sentences were sent to a llngmst and natl,ve speaker of the comparison language The hngmsts were asked to provide the lexlcal item m each parallel sentence that corresponds to the ambiguous Enghsh word If inflected, they were asked to provide both the inflected form and the root form In addttmn, the lmgmsts were asked to indicate the type of translatmn, according to the dtstmctmns given m Table 1 For over 85% of the Enghsh word occurrences (corresponding to types 1 and 2 m Table 1), a specific lexlcal item or items could be identified as the translation equivalent for the corresponding Enghsh word For comparison purposes, each translanon equivalent was represented by ~ts lemma (or the lemma of the toot form in the case of derivatives) and associated w~th the WordNet sense to which it corresponds In order to determine the degree to which the assigned sense dlstlncttons correspond to translation eqmvalents, a coherence index ( Cl) was computed that measures how often each pmr of senses is translated usmg the same word as well as the consistency with which a g~ven se,ls,z ~s translated with the same word ~ Note that the z The adJective and adverb senses of hard are consadeied together because the distinction is not consistent across the translations used m the study Note that the CI ~s similar to semanuc entropy (Melamed, 1997) However, Melamed computes CIs do not determine whether or not a sense dtstmctton can be lextcahzed in the target language, but only the degree to whmh they are lexicahzed differently m the translated text However, tt can be assumed that the CIs provide a measure of the tendency to lex~cahze different WordNet senses differently, which can m turn be seen as an mdtcatmn of the degree to which the distraction ts vahd For each ambiguous word, the CI Is computed for each pair of senses, as follows S<q t> Cl(sqS, ) = '=1 m rnrt where @ n ~s the number of comparison languages under consideration, nl~q and m,, are the nt~mber of occurrences olsense sqand sense s~ m the Enghsh corpus, respectively, including occurrences that have no idenufiable translation, s<~ ~>m ts the number of times that senses q and r are translated by the same lex~cal Item m language t, i e, x=y t ~tJan ~( q ), r~oan~( r ) The CI ts a value between 0 and 1, computed by examining clusters of occurrences translated by the same word In the othel languages If sense and sense ) are consistently translated w~th the same wo~d in each comparison language, then Cl(s, s~) = 1, if they are translated with a different word m every occurrence, Cl(s, ~) = 0 In general, the CI for pans of different senses provides an index of thmr relatedness, t e, the greater the value of Cl(s, sj), the more frequently occurrences of-sense t and sense j are translated with the same lextcal item When t = j, we entropy tOl wold types, lather than word senses 55 obtain a measure of the coherence of a ~lven sense Type Meaning 1 A slngle lexlcal Item is used to translate the En@izsh equivalent (possibly a 2 The English word is translated by a phrase of two or more words or a compound, meaning as the slngle English word 3 The En@izsh word is not lexzcalized in the translation 4 A pronoun is substituted for the English word In the translation An English phrase contalnmng the ambiguous word Is translated by a single language which has a broader or more specific meanlng, or by a phrase in whl corresponding to the English word Is not explicltl~ lexlcallzed Table 1 Translation types and their trequencles % dizen whl%h h 6% 6% 6% of s p same Word # Description hard 1 1 difficult 2 head i i i 1 Table 2 1 2 _meta~horlcally hard _\] 3 not yielding to pressure, 1 4 very strong or ~lgorous, ar 2 I wlth force or vigor (adv) 3 earnestly, intently (adv) i_ ~art of the body  3 intellect 4 _r~le_!r, ch,%ef 7 front, front part WoldNet senses ot hard and head CIs were also computed for each language individually as well as for different language groupings Romaman, Czech, and Estonian (three different language families) Czech and Slovene (same family), Romaman, Czech, Slovene (Indo-European, and Estonian (nonIndo-European) To better visualize the relationship between senses, a hierarchical clustering algorithm was applied to the CI data to generate trees reflecting sense proximity 4 Finally, in order to determine the degree to which the linguistic relaUon between languages may affect coherence, a correlation was run among CIs for all pairs of the four target languages Fol example, Table 2 gives the senses of hard and head that occurred in the data s The CI data .s 'sobS' hard and head are given in Tables 3 and 4 ~uous CIs measuring the aff, mty of a sense with itself--that is, the tendency for all occurrences of that sense to be translated wlth the same word--show that all of the s,x senses of ha,d have greatel internal consistency tfian athmty with other senses, with senses 1 1 ("dlff|cult" CI = 56) and 13 (,'not soft,, ci = 63) registenng the h,ghest internal consistency 6 The same holds true for three of the four senses of head, while the CI for senses 1 3 ("Intellect") and 1 1 ("part of the body") is higher than the CI for 1 3/1 3 WordNet Sense 2 1 2 3 1 4 1 3 1 1 1 2 21 23 1 4 13 0 50 o 13 i ool 0 O0 0 25 i O0 0 04 0 50 0 17 0 56 0 19 0 00 0 00 0 00 0 00 0 00 0 25 0 21 Table 3 CIs for hard I i 12 0,,63 0 00 0 50 2 Results Although the data sample is small, It gives some insight into ways m which a larger sample might contribute to sense discrimination 4 Developed by Andleas Stolcke Results tor all words m the study are avadable at http//www cs vassar edu/~~de/wsd/cross-hng html 6 Senses 2 3 and 1 4 have CIs ot 1 because each ot these senses exists m a single occurrence m the corpus, and have theretote been dlscarded horn consideration ot CIs to~ individual senses We a~e currently mvesugatmg the use oI the Kappa staUst~c (Carletta, 1996) to normahze these sparse data 56 WordNet Sense 1 1 1 3 1 4 1 7 1 1 0 69 1 3 0 53 0 45 1 4 0 12 0 07, 0 50 1 7 0 40 0 001 0 00 1 00 Table 4 CIs for head Figure 2 shows the sense clusters for hard generated from the CI data 7 The senses fall into two mare clusters, w~th the two most internally consistent senses (1 1 and 1 3) at the deepest level of each ot the respecuve groups The two adverbml forms 8 are placed in separate groups, leflectmg thmr semantic proximity to the different adjecuval meanings of hard The clusters for head (Figure 2) stmdarly show two dlstmct groupings, each anchored in the two senses with the h~ghest internal consistency and the lowest mutual CI ("part of the body" (1 1) and "ruler, chief" (1 4)) The h~erarchtes apparent m the cluster graphs make intuitive sense Structured hke dictmnary enmes, the clusters for hard and head might appeal as m F~gure 1 This ts not dissimilar to actual dlctLonary entries for hard and head, for example, the enmes for hard in four differently constructed dlctmnanes ( Colhns Enghsh (CED), Longman's (LDOCE), OxJotd Advanced Learner's (OALD), and COBUILD) all hst the "'d~fficult" and "not soft" senses first and second, whmh, since most dictionaries hst the most common Ol frequently used senses hrst, reflects the gross dlwslon apparent m the clusters Beyond this, ~t ~s difficult to assess the 7 Foi the purposes ot the cluster analys~s, CIs of l 00 resulting from a single occurrrence were normahzed to 5 8 Because ~oot to, ms were used m the analysis, no dzstlncUon m UanslaUon eqmvalents was made tor part ot speech correspondence between the senses In the dictionary entries and the clusters The remamlng WordNet senses are scattered at various places within the entries or, m some cases, split across various senses The h~erarchlcal relatmns apparent m the clusters are not reflected m the d~cttonary enmes, smce the senses are for the most part presented in flat, hnear hsts However, It is interesting to note that the first five senses of hard In the COBUILD d~cuonary, which is the only d~cttonary in the group constructed on the bas~s of colpus examples 9 and presents senses m ruder of frequency, correspond to hve of the six WordNet senses in thls study WordNet's "metaphorically hard" is spread over multiple senses in the COB UILD, as it.is In the other d~ctlonarles HARD HEAD I 1 dlfflcult 2 vlgorously II 1 a not soft b strong 2 a earnestly b metaphorlcally hard I 1 a part of the body b zntellect 2 front, front part II ruler, chlef Flgme 1 Clusteis tol hard and head suuctured as dlcuonary entt ~es The results tor dlftment language groupings show that the tendency to lextcahze senses differently is not aftected by language d~stance (Table 5) In fact, the mean CI fol Estonian, the only non-Indo-European language m the study, ~s lower than that for any other group, mdmatmg that WordNet sense dtstmctmns are slightly less hkely to be lexlcahzed differently m Estonian 9 Edmons ot the LDOCE (1987 vexsmn) and OALD (1985 version) dictlonalles consulted m this study ple-date edmons ol those same d~ctlonanes based on colpus evidence 57 Correlations of CIs for each language pair (Table 5) also show no relationship between the degree to which sense d~stmcuons are lexlcahzed differently and language distance This is contrary to results obtained by Resmk and Yarowsky (subm,tted), who, using a memc slmdar to the one used in this study, found that that non-Indo-European languages tended to lexlcallze English sense d~stmctlons more than Indo-European languages, especially at finergrained levels However, their translation data was generated by native speakers presented with Isolated sentences in English, who were asked to provide the translation for a given word In the sentence It is not clear how this data compares to translations generated by trained translators working with full context Lanquaqe qroup Averaqe CI ALL 0 27 RO/ES/SL 0 28 SL/CS 0 28 RO/SL/CS 0 27 ES 0 26 Table 5 Average CI values Lanqs Hard Country Llne Head Ave ES/CS 0 86 0 72 0 68 0 69 0 74 RO/SL 0 73 0 78 0 68 1 00 0 80 RO/CS 0 83 0 66 0 67 0 72 0 72 SL/CS 0 88 0 51 0 72 0 71 0 71 RO/ES 0 97 0 26 0 70 0 98 0 73 ES/SL 0 73 0 59 0 90 0 99 0 80 Table 6 CI correlauon tor the tour target languages I -I I  I I m~nlmum dlstance = 0 249399 m~nlmum d~stance = 0 434856 mlnlmum dlstance = 0 555158 mlnlmum dlstance = 0 602972 m~nlmum dlstance = 0 761327 I  >21 I  >ii I  >23 l  >13 l  >14 I  >12 (13) (23) (12) (1,4) (ii) (21) (1412) (2313) ( 2 3 1 3 1 4 1 2 ) ( 2 111 ) Figure 2 Cluster tree and distance measures tor the sm senses of hard I  >14 -i I  > i i I--- 1 J  > i 3 I  >17 mlnlmum dlstance = 0 441022 mlnlmum dlstance = 0 619052 mln~mum dlstance = 0 723157 (13) (ll) (17) (1113) (111317) (14) F,gure 3 Cluster tree and dmtance measures tot the tout senses ot head 58 Conclusion The small sample m this study suggests that cross-hngual lexlcahzat~on can be used to define and structure sense d~stmct~ons The cluster graphs above provide mformat~on about relations among WordNet senses that could be used, for example, to determine the granularity of sense differences, whtch m turn could be used in tasks such as machine translatton, mtormaUon retrieval, etc For example, it is hkely that as sense dtstmcttons become finer, the degree of error ~s less severe Resmk and Yarowsky (1997) suggest that confusing freer-grained sense dtstmctlons should be penahzed less severely than confusing grosser d~stmct~ons when evaluatmg the performance of sense dtsambtguatt0n systems The clusters also provide insight into the lexlcallzatlon of sense dtstmcttons related by various semantic relations (metonymy, meronymy, etc ) across languages, for instance, the "part of the body" and "intellect" senses of head are lex~cahzed with the same ~tem a s~gnlficant portion of the t~me across all languages, reformation that could be used m machine translatton In addtt~on, cluster data such as that presented here could be used m lexicography, to determine a mole detaded hierarchy of relations among senses in dtct~onary entries It is less clear how cross-hngual reformation can be used to determine sense d~st~nctlons independent of a pre-deflned set, such as the WordNet senses used here In an effort to explore how thts mlght be done, I have used the small sample from thts study to create word groupmgs from "back translations" (l e, additional translations m the original language ot the translations m the target language) and developed a metric that uses th~s mformatton to determine relatedness between occurrences, whtch ~s m turn used to cluster occurrences into sense groups I have also compared sets of back translations for words representing the various WordNet senses, which provtde word groups s~mdar to WordNet synsets Interestingly, there ts virtually no overlap between the WordNet synsets and word groups generated from back translations The results show, however, that sense dlstmctlons useful for natural language processing tasks such as machme translanon could potentsally be determined, ot at least influenced, by constdeHng this mformatton The automatically generated synsets themselves may also be useful m the same apphcatlons; where WordNet synsets (and ontologtes) have been used tn the past More work needs to be done on the topic of cross-hngual sense determination, utthzmg substantially larger parallel corpora that include a variety ot language types as well as texts fiom several genres This small study explores a possible methodology to apply when such resources become avatlable Acknowledgements The author would hke to gratefully acknowledge the contrtbut~on of those who provided the translatton mfotmat~on Tomaz Eua~ec (Slovene), Kadrt Muxschnek (Estonian), Vladtmlr Petkevtc (Czech), and Dan Tubs (Romanlan), as well as Dana Fleut and Darnel Khne, who helped to transcrtbe and evaluate the data Special thanks to Dan Melamed and Hlnrtch Schutze for their helpful comments 59 \[\] \[\] in \[\] in i i Hg nn i an i am References Ca~letta, Jean (1996) Assessing Agreement on Classthcatton Tasks The Kappa Stat~st~t. Computational Lmgulstlcs, 22(2), 249-254 Dagan, Ido and Ita~, Alon (1994) Wo~d sense dlsambxguat~on using a second language monohngual corpus Computattonal Ltngmsttcs, 20(4), 563-596 Dagan, Ido, Ital, Alon, and Schwall, Ulnke (1991) Two languages a~e more mformattve than one Proceedings of the 29th Annual Meettng of the Assoctatton for Computattonal Ltngutsttcs, 18-21 June 1991, Berkeley, Cahfornm, 130-137 Dyvtk, Helge (1998) Translations as Semantic Mirrors Proceedmgs of Workshop W13 Multzlmguahty in the Lextcon II, The 13th Biennial European Conference on Arttftctal lntelhgence (ECA198), Brighton, UK, 24-44 Eqavec, Tomaz and Ide, Nancy (1998) The MULTEXT-EAST Corpus Proceedlng~ of the Fltst International Conference on Language Resources and Evaluatton, 27-30 May 1998, Granada, 971-74 Erjavec, Tomaz, Lawson, Ann, and Romary, Laurent (1998) East meets West Producing Multflmgual Resources m a European Context Pioceedtngs of the Ftrst Internattonal Conference on Language Resources and Evaluation, 27-30 May 1998, Gtanada, 981-86 Fellbaum, Chttstmne (ed) (1998) WordNet An Electrontc Lexlcal Database MIT Press, Cambridge, Massachusetts Gale, Wdham A, Church, Kenneth W and Yatowsky, Davtd (1993) A method tor dlsamblguatmg word senses m a large cmpus Computers and the Humamtles, 26, 415-439, Hearst, M'attl A (1991) Noun homograph  ' dlsamblguatlon using local:'~.'0ntext m large corpora Proceedtngs of the 7th Annual Conference of the Umver~lt~ of Waterloo Centre for the New OED and Text ReaeaJch, Oxford, Umted Kingdom, 1-19 Ide, Nancy and V61oms, Jean (1998) Word sense d~samb~guat~on The state of the alt Computational Lmgut~ttc~, 24 1, 1-40 Kdgar~ttt, Adam and Palmer, Ma~tha, Eds (forthcoming) Proceedmgs ot the Senseval Word Sense D~samb~guatlon Workshop, Specml double ~ssue otComputer~ and the Humamttes, 33 4-5 Leacock, Claudia, Towell, Geoffrey and Voorhees, Ellen (1993) Corpus-based stattstlcal sense resolution Proceedtng~ of the ARPA Human Language Technology Worsl~shop, San Francisco, Morgan Kautman Melamed, I Dan (1997) Measuring Semantic Entropy ACL-SIGLEX Workshop Taggmg Tert wtth Lextcal Semanttcs Why, What, and How ~ April 4-5, 1997, Washington, D C, 41-46 Mtllet, George A, Beckwlth, Richard T Fellbaum.