and CAUS ate slgmficantly different for unaccusattve and object-dtop verbs, indicating that we need additional featules that have different values across these two classes In Section 2 1, we noted the differing semantic role asmgnments for the verb classes, and hypothesized that these differences would affect the expression of syntactic features that ate countable in a corpus For example, the c ~bs feature approximates sen\]antic role reformation b.~ encoding the oxerlap beh~een nouns that can occur m the ~ubject and object positions of a cau~ative xetb Here x~e suggest another feature, that of ammacy of subject, that is intended to distinguish nouns that receive an Agent role flora those that receive a Theme role Recall that objectdrop verbs assign Agent to their subject in both the transitive and intransitive alternations, while unaccusattves assign Agent to their subject only in the transitive, and Theme m the intransitive We expect then that object-drop verbs will occur more often with an animate subject Note again that ~e are 20 II Features \[Acc% SE% II I VBD ACT INTR CAUS I 63 7 0 6 \] VBD ACT INTR CAUS PRO 70 7 0 4 Table 6 Percentage Accuracy (Acc%) and Standard Error (SE%) of C5 0, W~th and W~thout New PRO Feature, All Verb Classes (33 8% basehne) making use of frequency dmtnbutmns--the clatm ~s not that only Agents can be ammate, but rather that nouns that receive the Agent role will more often be ammate than nouns that receive the Theme role A problem w~th a feature hke ammacy ~s that ~t requires etther manual determmatmn of the antmacy of extracted subjects, or reference to an on-hne resource such as WordNet for determining ammacy To approximate ammacy w~th a feature that can be extracted automatically, and w~thout reference to a resource external to the corpus, we instead count pronouns (other than ~t) m subject positron The assumptmn ~s that the words I, we, you, she, he, and they most often refer to ammate ent~tms The values for the new feature, P~.O, were determined by automatmally extracting all subject/verb tuples including our 59 examples verbs (from the WSJ88 parsed corpus), and computing the ratm of occurrences of pronouns to all subjects We again apply t-tests to our new data to determine whether the sets of PRo values d~ffer across the verb classes Interestingly, we find that the Prto values for unaccusat~ve verbs (the only class to ass~gn Theme role to the sub tect m one of tts alternatmns) are s~gmficantly dtffe~ent from those for both unergatlve and object-drop verbs (p< 05) Moreover, the PRo values for unergat~ve and object-drop verbs (whose subjects are Agents m bo~h alternatmns) are not s~gmficantly d~fferent Th~s pattern confirms the abd~ty of the feature to capture the thematm d~stmctmn between unaccusat~ve verbs and the other two classes Table 6 shows the result of applying C5 0 (10-fold eross-vahdatmn repeated 50 t~mes) to the three-x~ay classfficatmn task using the PRo feature m conjunctmn w~th the four previous features ~.ccuracy ranproves to over 70%, a teductmn m the error rate of almost 20% due to th~s single nex~ feature Moteover, classifying the unaccusat~ve an2 object-drop verbs using the new feature m conjunctmn w~th the prevmus four leads to accuracy of over 68% (compared to 58% w~thout PRo) We conclude that this feature ~s ~mportant in d~stmgmshlng unaccusat~ve and object-drop verbs, and hkely contributes to the tmprovement m the three-way classtficatton because of th~s Future work wdl examine the performance w~thm the verb classes of th~s new set of features to see whether accuracy has also tmproved for unergatire verbs 5 Conclusions In thin paper, we have presented an m-depth case study, m whmh we investigate varmus machine learnmg techmques to automatically classify a set of verbs, based on dlstnbutmnal features extracted from a very large corpus Results show that a small number of hngmstlcally motivated grammatical features are sufficmnt to reduce the error rate by mote than 50% over chance, acluevmg a 70% acctuacy rate m a three-way classfficatmn task Tins leads us to conclude that corpus data is a usable repository of verb class mformatmn On one hand ~e observe that semantlc propemes of verb classes (such as causatlvlty, or ammacy of subject) may be usefully approximated through countable syntactic features Even with some noise, lexmal propertms are reflected m the corpus robustly enough to positively contribute m classlficatmn On the other hand, however, we remark that deep hngumtm analysis cannot be ehmmated--m our approach, it is embedded m the selection of the features to count We also think that using hngumtlcally motivated features makes the approach very effective and easdy scalable we report a 56% reductmn m error rate, w~th only five features that are relatwely straightforward to count Acknowledgements This research was partly sponsored by the S~ lss Natmnal Scmnce Foundatmn, under fello~slup 821046569 to Paola Merlo, by the US Natmnal Scmnce Foundatmn, under grants #9702331 and #9818322 to $uzanne Stevenson, and by the Infotmatton Sciences Councd of Rutgers Umverslty ~,~,e thank Martha Palmer for getting us started on tlus ~ork and Mmhael Colhns for gwmg us access to the output of his parser We gratefully acknowledge the help of Ixlva Dickinson, ~ho calculated no~mahzatmns of the corpus data Appendix A The une~gatx~es are manner of morton ~erbs jumptd rushed, malched, leaped floated, laced, huslwd uandered, vaulted, paraded, galloped, gl,ded, hzked hopped jogged, scooted, ncurlzed, ~kzpped, hptoed, trotted The unaccusau~es are verbs of change of state opened, exploded, flooded, dzs~olved, cracked, hardened bozled, melted,.fractured,,ol,dzfied, collapsed cooled folded, w~dened, changed, clealed, dzwded, ~,mmered stabdzzed The object-dlop verbs are unspecffied object altelnatron verbs played, painted, k,cked, carved, reaped, washed, danced, yelled, typed, kmtted bolrowed mhet21 tted, organtzed, rented, sketched, cleaned, packed, studted, swallowed, called References Thomas G Bever 1970 The cogmtwe basis for hngmstlc structure In J R Hayes, e&tor, Cognttson and the Development of Language John Wdey, New York Michael Brent 1993 From grammar to le~con Unsupervmed learmng of \[ex~cal syntax Computational Linguistics, 19(2) 243-262 Edward Bnscoe and Ann Copestake 1995 Lex~cal rules m the TDFS framework Techmcal report, AcquflexI I Working" Papers Anne-Marm Brousseau and Ehzabeth R~tter 1991 A non-umfied analysis of agent~ve verbs In West Coast Conference on Formal Lmgutstzcs, number 20, pages 53-64 M~chael John Colhns 1997 Three generaUve, lexacahsed models for statistical parsmg In Proc of the ~5th Annual Meeting of the ACL, pages 16-23 Hoa Trang Dang, Kann K~pper, Martha Palmer, and Joseph Rosenzwe~g 1998 Investtgatmg regular sense extenmons based on mteresecttve Levm classes In Proc of the 361h Annual Meeting of the ACL and the 171h \[nternatwnal Conference on Computatwnal L,ngu,st,cs (COLING-A CL '98), pages 293-299, Montreal, Canada Umvers~t6 de Montreal Bonme Dorr and Doug Jones 1996 Role of word sense d~samb~guatmn m lexacal acqms~tmn Predmtmg semantics from syntactic cues In Proc of the 161h Internattonal Conference on Computat*onal Lmgutsttcs, pages 322-327, Copenhagen, Denmark COLING Bonnie Dorr 1997 Large-scale chctmnary constructmn for foreign language tutonng and mterhngual machine translatmn Machine Translatton, 12 1-55 Hana Fd~p M~chael Tanenhaus, Greg Carlson, Paul AIlopenna, and Joshua Blatt 1999 Reduced relatives judged hard require constraint-based analyses In P Merlo and S Stevenson, echtors, Sentence Processmg and the Lextcon Formal, Computational, and Ezpertmental Perspectives, John Benjamms, Holland Ken Hale and Jay Keyser 1993 On argument structure and the lexacal representatmn of s:~ ntact~c relatmns In K Hale and J Keyser, editors, The t',ew from Budding ~0, pages 53-110 MIT Press Juchth L Ixlavans and Martin Chodorow 1992 Degrees of stat~vlty The lexacal representatmn of verb aspect In Proceedmg~ of the Fourteenth International Conference on Computahonal Lmgmst,cs Juchth Ixlavans and Mm-Yen Kan 1998 Role of ~erbs m document analysis In Proc of the 361h Annual Meeting of the ACL and the 171h \[nternatzonal Conference on Computational Lmgutsttcs ( C O L L'v G4 C L '98), pages 680-686, Montreal, Canada Umvers~te de Montreal Beth Levm and/Vlalka Rappapti(t'Hovav 1995 (Jnaccusatwlty MIT Press, Cambridge, MA Beth Le~m 1993 Enghsh Verb Clas~e~ and 4lternatwns Chacago Umvers~ty Press, Chicago, IL Maryellen C MacDonald 1994 Probablhstlc constramts and syntactic amblgtuty resolution Language and Cognltzve Processes, 9(2) 157-201 Paola Merlo and Suzanne Stevenson 1998 What grammars tell us about corpora the case of reduced relative clauses In P1oceedmgs of the Slzth Workshop on Very Large Corpora, pages 134-142, Montreal, CA George Miller, R Beckw~th, C Fellbaum, D Gross, and Ix I~hller 1990 Fwe papers on Wordnet Techmcal report, Cogmtzve Scmnce Lab, Princeton Ual~erstt~ Martha Palmer 1999 Coasmtent criteria for sense distmctmns Computmg \]or the Hamamttes Fernando Perelra, Naftah Tlshby, and Ldhan Lee 1993 Dlstrabutmnal clustering of enghsh words \[n Proc of the 31th 4nnual Meeting of the 4CL, pages 183-190 Fernando Perexra, Ido Dagan, and Lalhan Lee 1997 Slmdanty-based methods for word sense dlsamblguatmn In Proc of the 35th Annual Meeting of the 4 CL and the 8th Conf of the E 4 CL (A CL/EA CL '97) pages 56 -63 Geoffrey K Pullum 1996 Learnabthty, hyperlearnrag, and the poverty of the sttmulus In Jan Johnson, Matthew L Jute, and Jen L Moxley, editors, ~nd Annual Meeting of the Berkeley Lmgutstzcs Soctety General Sesston and Parasesswn on the Role of Learnabdzty m Grammatzcal Theory, pages 498-513, Berkeley, Cahforma Berkeley Linguistics Socmty James Pustejovsky 1995 The Generatwe Lexicon MIT Press J Ross Qumlan 1992 C$ 5 Programs fo~ Machine Learning Series m Machme Learning Morgan Ixaufmann, San Mateo, C 4.