2 Methods
A Study on the Distribution of Cooccurrence Weight Patterns ofWe use the Hachidaishū as the material of the present study, which compr
Classical JapanesePoetrythe eight anthologies
compiled underorderof the Emperors (ca. 90512
BorHodošček

Hilofumi Yamamotoand contains about 9,500 poems. We developed the corpus and a method

Tokyo Institute of Technology

Osaka University
y cooccurrence weighting similar to thetf-idf method, cw (Yamamoto 200Introduction• P: Hairball eﬀect or spoke eﬀect (Yamamoto 2005)• P: Diﬃcult to observe all word adjacency features.
•
O: Checking the distribution of cooccurrence
weight (Yamamoto 2006)•G: Drawing clear figures and extracting function words.
•S: Cooccurrence weight is like the meanvalue of two person’s weights.

which calculates the weight of patterns of any two words occurring in a poMethods
sentence(Spärck Jones 1972, Robertson 2004, Manning and Schütze 19
Rajaraman andUllman 2012).
Material:
the Hachidaishū (ca. 905–1205)

Calculation of Cooccurrence Weight: cw
w(t, d) = (1+log tf (t, d)) · idf (t)cw(t1 , t2 , d) = (1+log ctf (t1 , t2 , d)) · cidf (t1 , t2 )√idf (t1 ) · idf (t2 )N
idf (t) = log
df (t)

cidf (t1 , t2 ) =Tall Headring!ring!

Where w is a ~weight, t a token, and N the number of tokens. T
Distributionof cw becomes Bell curve.
curve
function idf is called the “inverse document frequency.”(Spärck Jones 19
Robertson 2004, Manning and Schütze 1999) The function cw is called• Over σ =⇒ Content Tail.
“co-occurrence weight,” which allows us to examine the patterns of poheavy
Under −σmodeling.
⇒ Function Tail.
C.
F. Gausswordconstructions through •mathematical(1777–1855)
As in Fig. 1, there is a concept (Losee 2001: 1019) of terms locatedeach layer being eﬀective query terms. Luhn (1968) cuts the top and bContent Tailtom words of the frequency and uses mid-range vocabulary for the dex (1983:
opment of an automatic outline generation system.(Fig. 1) Nagao
to be eﬀective in generating automa
σ0 also mentioned mid-range vocabularyabstracts.
Nagao(1983)’s
viewpointisslightly diﬀerent with Luhn (1968
Table 2: Lower cutoﬀ patterns of ame (sakura) in Kokinshū: 30 out of 164 patterns
1000~
Long Tail Dinosaur← more genericmore specificlight→

Function Tail−σResult

1
2
3
4
5
6
7
8
910cw

z

pattern

0.62
0.62
0.62
0.60
0.60
0.60
0.59
0.59
0.59
0.59

-0.91
-0.92
-0.92
-0.94
-0.94
-0.94
-0.96
-0.96
-0.96
-0.96

mo–kerihana–oo–koso
zu–kerisu–ha
to–ba
ari–haari–moware–monasi–o

11
1213
14
15
16
17181920cw

zpattern

0.59
0.57
0.57
0.57
0.56
0.56
0.55
0.54
0.52
0.52

-0.96
-0.98
-0.98
-0.98
-1.00
-1.00
-1.01
-1.02
-1.05
-1.05

nasi–hao–ramumo–ramu
ha–ki
zu–mo
o–te
hito–mozu–te
zo–ha
omou–o

21
22
23
24
25
26
27
28
29
30

cw

z

pattern

0.52
0.52
0.52
0.48
0.48
0.43
0.43
0.43
0.34
0.34

-1.05
-1.05
-1.05
-1.09
-1.09
-1.16
-1.16
-1.16
-1.27
-1.27

nu–o
o–zo
miru–oba–moo–kerizu–ha
to–ote–hao–hao–mofrequency

Table 1: Upper cutoﬀ patterns of ame (sakura): cw = co-occurrence weight; z = z-value(normalized value of frequency). word annotations: ari(be), ba(cond.), ha(topic.),
hana(flower), hito(human), keri(past.), ki(past.), koso(emphatic.), miru(see), mo(also), nasi(no exist), nu(neg.), o(obj.), omou(think), ramu(aux.will), su(do), te(p.),
to(and), ware(we), zo(emphatic.), zu(neg.)

2600

Sakura(cherry)

400

200

0−2−101z−value

2316001400
1200as well as inclassical texts (Fig. 2). Therefore we will attempt to divide this 1000
torawareru
Ume(be captured)hodonishape into three layers by (LK)
inflection points.
(plum)(as much as..)800
The co-occurrence patterns of sakura (cherry) under-0.9 (near -1) cw
taisetsu(precious)
value are adjacent patterns comprising function words, and over 1 wacw value 600(FP)
are those of the patterns with content words as we expected (Table 1 and 400
2). As for the upper cutoﬀ, we used an under -0.9 (near da-1) σ value of 200(ASSER)all patterns
cw, which could extract patterns of functional tokens:
almostkara0(because.1)
123
ncluded functional words, while as lower cutoﬀ, we used over 1 σ values, −2 −1 0yo(ra)reru
z−value(PASS/POT)
no/kotowhich could
extract patterns of content tokens: almost all patterns(FP)includedomou.
(DN)regarded as inflection
Figure 1: Bell curvestacontent words.
over 1 σ aresa points(think) Both under -1 and(PERF)(FP)
which have mathematically interesting property.

4extracted; cw = co-occurrence weight; z = z-value (normalized value of frequency) word annotations: ba(cond.), bakari(only), besi(should be), chiru(fall),fukakusa(deepgreen), hana(flower), isa(already), kakusu(hide), katu(win), koku(pull),komoru(go deep inside), magiru(mix), makasu(entrust), maku(wind up), manimani(asit is), masi(as), mazu(mix), me(eye), minami(south), miyako(city), mono(thing),
nagara(even if), sakura(cherry), si(emphasic.), sumi(black ink), tatu(start,stand),tazumu(being around), tu(past.), uturou(change), watasu(give), yamakaze(mountainwind), yamu(stop), yanagi(willow), yononaka(world)800

frequency

1

1

1

120

1

112
13

1

134

14

7

2

538
5

1

5
12

1

515

24

iru
(be-AS)

1 37 1

Discussion
haseru
(run)

4
5

2

te/de(LK)

23
1
1
1

1
2
3
4
5
6
7
8
9
10–
101
102
103
104
105

cw

z

pattern

3.86
3.75
3.67
3.61
3.42
3.38
3.38
3.27
3.26
3.25

3.18
3.04
2.93
2.86
2.62
2.57
2.56
2.43
2.42
2.40

yamu–manimaniminami–magiru
minami–maku
maku–magiru
yanagi–koku
yamu–makasu
mazu–koku
yanagi–mazusakura–yamu
minami–yamakaze

2.40
2.40
2.40
2.39
2.39

1.33
1.33
1.33
1.32
1.31

uturou–komoru
sakura–watasu
katu–nagara
sakura–masi
sakura–makasu

5Conclusion

wave

u(CJR)

fuan(anxiety)

16

ka(Q)

1.04
1.04
1.03
1.03
1.03
1.03
1.03
1.02
1.00
1.00

chiru–katubakari–sumi
maku–besitatu–maku
tatu–tazumu
tazumu–tu
miyako–sakura
kakusu–si
yononaka–sakura
mono–sakura
1after1mountain village
day11(adv.will)
Using the distribution characteristics of cooccurrence
weights, we were able
to classify cooccurrence patterns into three layers of cooccurrence patterns:
4high-, mid-, and low-range patterns.
We found that 1) the distribution of classical texts fits a Gaussian curve
mountainas well as in modern texts; 2) the cw value can separate patterns into threelayers (low-, mid-, and high-range) using inflection points (-1σ and 1σ); 3)of the three layers, the high-range could be extracted without a list of stopcomewords; 4) the mid-range lexical layer might include
mathematical traits not
yet revealed in the present study.

1 1
11painful.1
11either 1way
1 1(emph)
1 11usedã ¤ãknow
¤get1thing.1
1 1 be
1 1trust.2
1stop.vi.2
1 1lonely
seem
1
perplexed
begin
1
1
1
1 1
1 desolate
1love.1
1
forcibly1 1 1 1 1 1 1 1 1 1 1this yearwill
11 1 1notprosper
11 1 1 1
number
11 11 11 1111 1 111roadcross
11 1 11 11 11 1 1 ã11 111 111 1 1 11 11111 1 111 11 1 1 1
1 1 1suchsend.v2
111 1 1 111111111 11 1 11 1 11 1say
kurabu.pl1 word1
1 1 1 1 1 1as.1
111
ã 1ˆã ˝
11111
1 11 1 1 111 1 1111 1 1 111 1 111 11cloud.vi
11 spring
1go over1111 1 1311 11 1
1 1 11 1 11111111111 1111212 1 1till
1 111 11 1
1 11111111
1
1 increase 2ã
1
11
1
1
1
1
111 1 11 2111 12 1 1112 12 1 1 11113 11 1 1 takeã ã old1 age
name1
1 11 11 1 111 1 12(interj.)
1 11hear11
11 1 1 1
1 121 1121 1 22112121 1122 11myself
1 2 1 21 1 1 11 111like.2
1think1111311111 11111 1 1 11 1 1211 11 111 1 11 11 12 2 2122 1 211 222 1 1 121 11 1 11 1(suffix.forbid)
1 11 11 1vain
1 1 11 111 21 111 22 2 2 221 12 12 11112121
1 1 1 1 11 11 1 exchange
11 11 wait
2 11 21 111 111 1222 21 121211 13113 12 12 63 1213 121 121 1211111111 1 111 1 1111 1borrow
1 111add
111 11
1 12 11 111 1112 11212112112112 2 11237139213231111 22111 2311111 11112 1 1111111112 11 11 1 1 11111 change.1
1 1 bear11
111 1 1 1 11 1111 1 11 1 13 3221 16 7124121112 245124121111 42 121112111 1 2111111111 1 day
11ã˙
11rare.1111lodge.n3 321
1
1 1 112 11 11 1 1 21 1 1 1
111 1 seemed
31 1 11 12 13 264121 1111 11 1111212 11111 1 111 11 14710
111 11 Yoshino.PN
today
1 21 11 3 11 1 111121221 111112 114 14 3213211 851211
year
115 43 22619 1 1111 11 11211 1 1 11 1
1 1
21 2 fromhere
go 1person
oldvillage
1 we
1 1 2 1 1 1 1 2 12 11112111 1131 1323 313 231212186 217111 1111
1211416113 11 110
1 11211 1 11 11 1 111 1 11 1 111 1111foot
1 114 1 111111111 211121 16 172811 4122 2331 7 411 41410
224117 11512 31231 2211353 114111211 11111111 111 1 22 112 11 11 21colour
1112211trail
fast/early1 1 1rain
1 112 11 11 1 112225 1 28 112 1221 2135 1312 21 111 1 11111 11 1
see.1
spring1
1 1 111get
954322421 1 3 12111 211111 121311 11 11 211 11regrettable
111111 1112 3 11212 2 2 1 15
1 1 1tired.1
a11glance
12 1 4111211 11 1121212 1 1221231511523 3231 412118
1 1 1 11 1 11156 21 3 11 11gorge
stand.vi
1 11 11 22 4 11 3 21 1 1213 18 1141213 11 111 1 11
1what
11 1121111 1 1111 11 11
1 11 11 211 215 11 11 1 21 1 21 1 21 12 111 2 5113
11111 12 1 1 1 1111211 111 11212 1136121111111 121 2311 2211 212511723111 1218 5134 4113 1221 311 11 1311 11 12 cloud.1
1regret111 3 3 111 11 11 211 21 2 3 141 11 3114 12 3111 2113 1 11 1 11confuse
1(aux.perf)
white1 cloud221 111 1 21 1 1 hide.vt
1 1 11 2111 2 11 3 1 1 21221
1
1 1 111 11 2 2111 11211
1you.3
11 1 113642 12111 11 11211
2 12 1 13 12blow
1
11 1 1 1 11 112 1 1 17 11 111211 21111121 1211 11111 1 1111211 1 1 1 1out
every111teartransfer.1
1 112 1 11111 111111 111111 1 11 1111121 131 1111 14even
1pull
1
11 1 132 121111121 11 111 11111212 111 11 1111121111111111(ku)1 1
1 111 1 1 11 11 11
1 12
1
1 thing.2
1 11 1fall
111 111 11211111 2 21 1111
112 111 11 11 211
1 wave1 11 11
1 1
1 11 12 1 21 21111211 11 121 11111 11 1 1(adv.would)
present
mistake
1 remnant1 1 111 1 1 1 11 1 21 111112 1111111 11(must)ã 1ã 1 1111
1 1 run 1
1 1 1 111 quiet
1 1 111 11 1 111 11 although
1 11 1 hand
snow
1
1
1
after
1 11for111 1 11 11 waterfall
1
1 1 1
mountain
11 1
1 waterskytellarea
mountainstonewish
1
1 village
1such as.2
1
lovelylodgingå¦ ä½¯ã«1 1 1 1
way home
forget1 village.1
1
1 1 here
11 1
sorrowful.1
1 11 1 1
world.2
1
remainbe over
1 1 1mi(p.wish) high.1

2.17
2.17
2.16
2.16
2.16
2.16
2.16
2.16
2.14
2.14

trust.2
seem1
1 1(emph)

1

1

1

1

1

1

1

1

1

1

1

1
13
high.1mi3

1

1change.1

1

1

11

for

1

1

1wish

1

1

hand

what

1

trail

1

till

1

1

2

1

spring rain

1

1

1

regret1

1transfer.1

11spring haze
quiet

1

1

increase1

1

1

get tired.1

1will not

1

1

1

1

forcibly1(aux.perf)

1

1hide.vt

such as.2

Yoshino.PN

between1

kurabu.pl2

1

perplexed
1

1

6

1

1

gomind between
mountainlodge.n

1

1
1
1
1

2

bloom

mistake

1

1

1

1

waterfall1
1
run 1
1
1

every

2

1

1

1

1
1

1

1

myself1

1
1 stoneborrowlike.2
1

1

tell

present

1

only

1

1

1

1

1

1

1

1

break off 1

1

1

1

1

2

3

gorge

1

1

1

1

1
seemedregrettable1

1

prosper1

mountain areatake 1 1 1
here we go

cloud.11

1 1

desolate

1

1
1

foot

1

1

1

1

1

1

1

(suffix.forbid) 1

1

1

1

be lonely painful.1
11

white cloud

1
pull

1

1

koi
(love.2)

patternsi–fukakusa
sakura–hanasakura–isa
sakura–ba
sakura–me

1

1

1z
1.31
1.31
1.31
1.31
1.30

1

1

kuru
(toward-AS)

cw
2.38
2.38
2.38
2.38
2.38

out

remnant

snow

51

106
107
108
109
110
—
155
156
157
158
159
160
161
162
163
164

1

1

love.111either
way

11
1

see.3

1

nflection points are defined as the points on the curve
where the curvaturenodeãã ¯¨ comescatter.1
personã ªãwind.nã(adv.will)changes its sign
while a tangent exists.(Bronshtein et(because.2)
al. 2004: 231) We conãscatter.1´see.3suru(adv.neg)(aux.past) while(do.2)
Referencesã ã® springoff hazeã `ã only ã ç ¡ã ˙ãbreaksider the threshold values that separate upper cutoﬀ, mid-range, and lower°ãbloom
¦be.3
VERB
a3
LK
a2
DN
gamma
b1b2
END
do.1 ã ˛
cutoﬀ not as
coincidental
but
as a1evidential
points.
It is,
however,necessarytoomou (think.1) B1
Bronshtein, I.N., K. A. Semendyayev, G. Musiol, and H. Muehlig (2004)
conduct further experiments and continue to discuss the mathematical traitsHandbook of Mathematics: Springer-Verlag, 4th edition.
behind the distributionsof cooccurrenceofweights.
In termsofofomou
removing
theFigure 2: Construction
the predicate(think)
Function Tailow-range (upper cutoﬀ)withand extracting
the high-range (lower cutoﬀ) from
Hodošček, Bor and Hilofumi Yamamoto (2013) “Analysis and Application ofpoetic texts, we found that we do not need to use any filters to eliminate
Midrange Terms of ModernJapanese”,
in ComputerandHumanities 2013
Figure4: Only withContentTailFigure 3: Hairball effectterms, since cw values returned semantically cooccurring patterns. Apart
Symposium Proceedings, No. 4, pp. 21–26.
romand high-range, of
theclassicalcharacteristicsof thefits
mid-range
lexical1)low-range
the distribution
texts
a Gaussian(Bell) curve as well as in modern texts (Hodošček and Yamamoto 2013);
ayer
still cw
unknown.
2)arethevalue can separate patterns into three layers (low-, mid-, and high-range) using inflection points (-1σ and 1σ);11cherry blossom (28/131,4.28): CT ctf.>0.00;
non-dist=off; idf=off; pruned under U:1:

Conclusion

go over1

1send.v2 wordeven

1

1

1

1

3

1

cross

person from old village

add

1
1
2 1

1

1

think

1

1

1 1

1

1

1

you.3

1

1

1

1

1

1

blowsay

1

1

get usedsorrowful.1

1

remainvillage.1

lodging(must)1

cloud.vi

1

hereforgetthis year wait

1
1

1

1

1 1
lovely

1

1

bear

1

1

today

1

1

1

1

vain

1

1

1

1

1

1

confuse1

way home1

61(p.wish)1 1such as.11 old age begin
1
1 exchange

11

fast/early

springa glance

1

1

1

wind.n1

number

1year

name

1

3

1

rare.12

2stop.vi.2
11road2

3

1

1(aux.past)103(ku)world.2

be overcherry blossom (28/131,4.28): CT ctf.>0.00;
non-dist=off; idf=off; pruned under U:1:

53) of the three layers, the high-range could be extracted without a list of stop words;
4) the mid-range lexical layer might include mathematical traits not yet revealed in the present study.

Reference4• Yamamoto, H. (2005), Visualisation of the construction of poetic vocabulary using the database of the Kokinshū., Jinbun kagaku todētabēsu (Humanities and Database) the 11th symposium, 81–8, The council of humanities and database.
• Yamamoto, H. (2006), Extraction and Visualisation of the Connotation of Classical Japanese Poetic Vocabulary, Symposium for
Computer and Humanities, vol. 2006, 21–28, The information processing society of Japan.
• Hodošček, B. and H. Yamamoto (2013) “Analysis and Application of Midrange Terms of Modern Japanese”, in Computer and Humanities 2013 Symposium Proceedings, No. 4, pp. 21–26.