A New Method of N-gram Statistics for Large Number of n and Automatic Extraction of Words and Phrases from Large Text Data of Japanese

In the process of establish in g the it, form ation theory, C. F,. Sh a n n o n prol)ose.d the Markov I)ro(:ess as a good model to character ize ~t na tu ra l la.nguage. The core or this ide.a is t;o cah:ula.te the ['re(lU('Iides of str ings compose(l of 'n characters ( 'n-grams), bu t this stat ist ical analysis of large text. (lata a.,id for a large n lilts l lever be(HI carried ()tit })eca. /ise of the memory l imi ta t ion of (:omputer and the shortage of text data. Taking advantage of the recent powerful computers we developed a. new aJgorithm of n-grams of large text data for arbitr~try hu'ge 'n a,nd (:alculated successl'ully, within ,'ela, t iv( . ly short thlle~ n-grams of some Japa,nese text (la, t~t containing between two an(l thirty million chara,(:ters. From this exl)eriment it 1)ecame (:loa,r t]l&t the automatic extract ion or detern, i , ta t ion of words, (:oml)ound words and (;ol]ocations is possible by mutually compar ing n -g ram stat is t ics for dill'etch t values of l t . c a t e g o r y : topical pa,per~ quant i tat ive linguisth:s, large text corpora, text t)rocesshlg 1 I n t r o d u c t i o n Claude E. Shannon estal)lished tile in foH.a t io , theory in 19d8 [1]. Iris theory included the co,lcept tlutt ~ hmgnage could be a,pproximated by an nth order Markov model by n to l)e extended t(~ infinity. Since his proposal there were ma.ny tri~tls to ea.h:ulate n-grams (statist ics of 'n c},ara.(:ter str ings of a language) lbr a big text data, of a, l a . guage, l[owever computers u 1) to tim present t o . h i not ca.h:ulate them for a large n 1)e(:ause the cah:u|ation require(1 hug(; amount of memory space au([ t ime. For example the I'r(,quen('y ea, h;ulati(m of 10grams of English requires a,t least 2 ( ;win l0 s ~ I() (~ giga word memory space. Therefore ti le ca,lcuh~tiou was done at most for n :: d ,.o 5 wil;h modest text qua, n tit;,, We developed a new method of calcula, ting w gra.ms For large 'n's. We (1o not In'el)are a table for an 'n. gra.m. Our methods consists of two stages. The first stage perh)rms I;he sort ing of sul)strings of a text, aim fin(Is out ti le lenlKth of t:he prefix parts which axe the same for th(; a, dja(:ent sul)stritGs iN the st)rted ta,ble. The second sta,ge is the (:a,lcuh~,tion of an 'n-gram when i t is aal(ed for a sl)ecific n. Only the existing 'n, chara,cter combinations require the ta, hle entrie,g t'm' (,lie l'requen(:y count, so that we .eed not r(,serve a, big si)ace for 'n-gram table. The progranl we ha,ve develol)e(1 requires 71 bytes for an l cha, ra,cter text of two byte (:ode such as Japa, llcse and Chinese texts and 6! bytes for an l character text o1' English and other F',uropean la,nguages. By the present program '., ca, n be extended up to 255. The program can l)e (:hant~;ed very easily for la,rger '., i f it is required. We l)erf.rme(l '.,-l,;ram Irequen(:y (:a,h'ulations for three (11 fl'(;ren t text data. We were not so m u(:h interested in tile. en t ropy wdue of a ]a.nguage ])ut were illter('ste(I in the extra.orion of varieti(,s o l langua~e i)rOl)(,rties, su(:]l as wor(ls, (:olnl)oun(I words, (:o[]oca.l.ions and so on. The cah:ula.tion of fre(luelu:y of o(:(:urren(:es of clul.ra('.t(,r sl, rings is t)art icularly i ln lmr ta l l t to ( leterlni lm what is ;~ wor(l in such la, nguage.s as 3al)alle. ~e a,nd Chinese where there is no sl)a. (:es between words a.nd the de terminath)n of word boundarh~. ~ is not so easy. In this l)aper we will explain some of our results on 1,hose probh;nls. 2 C a l c u l a t i o n of 'n-grams for an arbit, r a r y large n u m b e r o f ' n It was w! ry diff icult to calculate 'n-grams for a large number o1" 'n because of the. memory l imi ta t ion of a computer. For examph. ', Ja, panese langua,ge ha.s m~t'e thall d000 di/ l 'ere.t characters a,nd i f we want