A Three-Parameter Rank-Frequency Relation in Natural Languages
Chenchen Ding, Masao Utiyama, Eiichiro Sumita
Advanced Translation Technology Laboratory,
Advanced Speech Translation Research and Development Promotion Center,
National Institute of Information and Communications Technology
3-5 Hikaridai, Seika-cho, Soraku-gun, Kyoto, 619-0289, Japan{chenchen.ding, mutiyama, eiichiro.sumita}@nict.go.jp

Abstract
We present that, the rank-frequency relation in
textual data follows f ∝ r−α (r + γ)−β , where
f is the token frequency and r is the rank byfrequency, with (α, β, γ) as parameters. The
formulation is derived based on the empiricalobservation that d2 (x+y)/dx2 is a typical impulse function, where (x, y) = (log r, log f ).
The formulation is the power law when β = 0
and the Zipf–Mandelbrot law when α = 0. We
illustrate that α is related to the analytic features of syntax and β + γ to those of morphology in natural languages from an investigationof multilingual corpora.

1

7

7

6

6

5

5

4

4

3

3

2

2

1

1

0

0
0

1

2

3

4

5

6

7

0

1

2

3

4

5

6

7

Figure 1: Rank-frequency plots on English words (left)
and Chinese characters (right). The x- and y-axes are
log10 r and log10 f , respectively. The gray curves are
the proposed formulation under logarithm: y = C −αx − β log10 (10x + 10γ ), where C is a constant. The
dashed lines are the asymptotes of C − (αx + βγ) andC −(α+β)x. (α, β, γ) is (0.93, 2.04, 3.82) for English
words and (0.59, 32.31, 4.42) for Chinese characters.

Introduction

Zipf’s law (Zipf, 1935, 1949) is an empirical law
to formulate the rank-frequency (r-f) relation inphysical and social phenomena. Linguistically,
Zipf’s law can be observed on the distribution ofwords in corpora of natural languages, where thefrequency (f ) of words is inversely proportional to
its rank (r) by frequency; that is, f ∝ r−1 . Zipf’s
law is a special form of a general power law, thatis, f ∝ r−α , with α = 1.
The Zipf’s/power law is usually examined under a log-log plot of rank and frequency, where
the data points lie on a straight line. The simple proportionality of the Zipf’s/power law can beobserved on randomly generated textual data (Li,1992) and it only roughly depicts the r-f relation inreal textual data. A two-parameter generalizationof the Zipf’s/power law is the Zipf-Mandelbrot
law, where f ∝ (r + β)−α (Mandelbrot, 1965).
Li et al. (2010) considered the reversed rank ofrmax +1−r, where rmax is the maximum of ranking index, and proposed a two-parameter formulation of f ∝ r−α (rmax + 1 − r)β .
As a straightforward observation, the coefficients of proportionality should be distinguishedfor common and rear words (Powers, 1998; Liet al., 2010). Therefore, an extension of the original Zipf’s/power law requires at least two parameters. In this study, a three-parameter formulationof f ∝ r−α (r + γ)−β is derived based on the observation and analysis of multilingual corpora. It
is a natural generalization of the power law and
the Zipf-Mandelbrot law. The third parameter provides a depiction of the rigidness of different coefficients of proportionality. The proposed formulation can also fit non-Zipfian phenomena in natural
languages, such as the r-f relation on Chinese characters. Figure 1 shows examples on English words
from Europarl (Koehn, 2005) 1 and Chinese characters of Academia Sinica from the data of Sproatand Emerson (2003).22

Proposed and Related Formulation

Under a logarithmic form, the Zipf’s law statesthat x + y = C, where (x, y) = (log r, log f ), andC is roughly a constant. We further investigate the1
http://www.statmt.org/europarl/v8/
europarl.tgz
2
http://sighan.cs.uchicago.edu/
bakeoff2005/data/icwb2-data.zip

460
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 460–464
July 5 - 10, 2020. c 2020 Association for Computational Linguistics

0

1

2

3

4

5

English Word
Chinese Character
Artificial

gi00 =

0
gi0 − gi−1.
xi − xi−1(1)

exp(bc − d)ra (r + exp(c))b∝ r−α (r + γ)−β ,

4

3

3

2

2

1

1
0
1

2

3

4

5

6

7

0

1

2

3

4

5

6

7

Zipf-Mandelbrot law. Because the softplus function is a differentiable form of a rigid ramp function, Eq. (3) can also be considered as a smoothed
piecewise broken power law. As shown in Fig. 1,
α and (α + β) depict the proportional coefficientsat the two ends, and the proportional coefficientsare switched smoothly around x = γ.
f ∝ r−α (rmax + 1 − r)β proposed in Li
et al. (2010) is also a two-component formulation. One more parameter (i.e., γ) in Eq. (3) isused to identify the location of the impulse observed in g 00 . Under Li’s formulation, we obtaing = y + αx = β log(rmax + 1 − exp(r)) andg 00 = −C1 exp(x)(C2 −exp(x))−2 , where C1 andC2 are constants. g 00 is a monotonically decreasing function with x = log(C2 ) as the asymptote
for x < log(C2 ). Therefore, Li’s formulation always has a steep tail and lacks the capacity to depict the switching of two stable proportional coefficients. Figure 3 shows examples using Li’s formulation to fit data in Fig. 1. It can be observedthat the non-Zipfian Chinese characters are fitted
well, but not for the tail part in more Zipfian English words. This can be explained from the shapeof g 00 in Fig. 2. It is reasonable to model the g 00
of Chinese characters using a monotonically decreasing function because the γ in Eq. (3) is quite
large (around rmax ). However, it is not proper for
English words, where a proper γ is required.
Based on the analysis, it can be concluded thatthe formulation f ∝ r−α (r + γ)−β is a generalized form that covers the Zipf’s/power law, ZipfMandelbrot law, piecewise broken power law, andLi’s two-parameter formulation. In the next section, we show the linguistic interpretation of theparameter (α, β, γ).

(2)

and to substitute (x, y) by (log r, log f ), we obtain,
f=

5

4

Figure 3: English word (left) and Chinese character (right) data in Figure 1 fitted by the gray curveof y = C − αx + β log10 (rmax + 1 − 10x ). The
dashed lines are of C − (αx + β log10 (rmax + 1)) andC − β log10 (rmax + 1 − 10x ) for two ends. (α, β) is(1.15, 9.16) for English words and (0.62, 157.13) for
Chinese characters.

Here (xi , yi ) is the data point of the i-th frequent token, gi = xi +yi for i > 1, and g10 = g100 =0.3 Because the differences are intrinsically nonsmooth, Bézier curves are applied for smoothing
in the investigation.
Figure 2 shows examples of the smoothed g 00
on English words and Chinese characters from thesame dataset used for Fig. 1. An artificial Zipfian
dataset generated in the manner of Li (1992)4 is
also used for comparison. It can be observed thatthe g 00 on English words and Chinese characters
has an impulse, but not that on the artificial data.
Generally, the impulse becomes more obvious if
the data are more non-Zipfian.
If we consider g 00 as a general impulse function,then g 0 is a general sigmoid function and g can bemodeled by a general softplus function in the formof b log(exp(x − c) + 1). To replace x by a generalized linear form as ax + d,
y = −d − ax − b log(exp(x − c) + 1)

6

5

0

property of C = g(x). The first and second-orderdifferences on g(x) are calculated asgi − gi−1,
xi − xi−1

7

60

Figure 2: Smoothed second-order differences on therank-frequency relation. The x-axis is log10 r.

gi0 =

7

(3)

where (α, β, γ) = (a, b, exp(c)). exp(bc − d) is aconstant unrelated to r.
The obtained proportional form is a natural twocomponent extension of the power law and the3

To avoid too many meaningless zeros in the differences,
only the data point with the minimum x is used for data points
with the same y, i.e., tokens with the same frequency.
4
Two letters a and b are used. The frequency of a, b, andspace is 3 : 1 : 1, and 107 characters are randomly generated.

461

0.92±.00
0.86±.00
0.99±.00
0.99±.00
0.98±.00
0.93±.00
0.94±.00
0.90±.00
0.87±.00
1.01±.00
0.92±.00
0.94±.00
0.84±.00
0.87±.00
0.98±.00
0.87±.00
0.93±.00
0.94±.00
0.89±.00
0.91±.00
0.99±.00

2.05±.06
1.20±.01
1.10±.01
1.08±.01
1.96±.03
2.04±.01
1.38±.01
1.06±.01
0.89±.01
2.05±.02
0.96±.02
1.47±.01
1.04±.01
1.69±.02
1.18±.01
1.18±.01
1.33±.01
5.24±.32
1.38±.01
1.77±.04
1.05±.01γ
4.25±.02
3.89±.01
3.85±.01
3.94±.01
4.43±.01
3.82±.00
3.82±.01
4.13±.01
4.07±.01
4.14±.01
4.16±.02
3.84±.00
3.77±.01
4.22±.01
3.73±.01
3.97±.01
3.77±.01
4.78±.03
4.14±.01
4.31±.01
3.86±.01

γ
rmax

0.85

sl
Slavic
0.80

0.85
0.74
0.69
0.70
0.82
0.75
0.73
0.75
0.70
0.80
0.76
0.73
0.70
0.81
0.68
0.76
0.72
0.97
0.79
0.84
0.70

0.70

sk

bg
en

2.00

1.50
0.80

1.20

lv

2.50

huetplencs
Uralic esitptdeRomance
sv
fi lt
danl
Germanic

0.65
1.60

β

2.00

el

fr

sl

sk it
es
Slavicpt Romance
cs
pl
nl
et
da
de
sv
lt
hu
Uralic fi
Germanic
0.80

0.85

0.90

0.95

1.00

1.05

α

Figure 4: Distribution of languages in Europarl.

the asymptotic standard error (± ) are listed. Because γ may depend on the vocabulary size, norγ
malized γnorm = rmax
is also listed. It can be observed that all the language data were fitted wellwith an α of around 1.0, which is in accordancewith the original Zipf’s law. β and γnorm for eachlanguage are plotted on the left of Fig. 4.9 On the
β-γnorm plane, we can observe the rough tendency
that β and γnorm are linear, in addition to a separation for different language branches. Further principal component analysis on (α, β, γnorm ) suggests that α and β + γnorm can be generally considered as two dominant components.10 The plot
on the right of Fig. 4 shows that the language
branches can be separated roughly by lines parallel to the axes of α and β + γnorm . This indicatesthe linguistic explainability of the two axes.
From the nature of these languages, we consider
that α can be explained as an axis of analysissynthesis on syntax and β + γnorm as that on morphology. A large α suggests a couple of extremely
frequent words in the corpus. As typical examples,languages with a relatively large α, that is, Romance and Germanic, generally contain abundant
prepositions, particles, and determiners to marksyntactic roles, whereas those with a smaller α,that is, Slavic and Uralic, tend to use complex declension and conjugation within words to afford
syntactic information. Interesting evidence is thatbg, as a very analytic Slavic language, has a larger
α than other Slavic languages. In another dimension, a large β + γnorm suggests a dramatic decrease in the frequency of rare words. Hence, lan-

Table 1: Fitted parameters on Europarl data.

3

0.75

lv

3.00

bg
el
fr

β + γnorm

bg
cs
da
de
el
en
es
et
fi
fr
hu
it
lt
lv
nl
pl
pt
ro
sk
sl
sv

β

γnorm

α

Experiment and Discussion

We used the proposed formulation to fit data ofvarious European languages and typical Asian languages. The Europarl corpus (Koehn, 2005) anddata from the Second International Chinese Word
Segmentation Bakeoff (ICWB2) (Sproat and Emerson, 2003) were mentioned in Section 1. We also
used English-Japanese patent data from the 7th
NTCIR Workshop (Fujii et al., 2008). The Europarl data and English data from NTCIR werelower-cased and tokenized using the toolkit provided by MOSES5 (Koehn et al., 2007). Fitting
was performed under a logarithmic scale using
the fit function6 in gnuplot.7 Specifically,
relation-frequency data were used to fit (α, β, γ)and C in y = C −αx−β log10 (10x +10γ ). For the
initialization, (α, β, γ) = (1, 1, rmax2 ) and C = 3γ
were applied.
Table 1 lists the fitting results for all the languages8 in the Europarl corpus. The (α, β, γ) withGreek (el), English (en), Spanish (es), Estonian (et),Finnish (fi), French (fr), Hungarian (hu), Italian (it),Lithuanian (lt), Latvian (lv), Dutch (nl), Polish (pl), Portuguese (pt), Romanian (ro), Slovak (sk), Slovene (sl),and Swedish (sv).
9
The non-typical Germanic en, Baltic lt and lv, andHellenic el are in gray. ro with a large β is excluded.
10
First principal component: −0.1α − 0.7β − 0.7γnorm ,and second principal component: 1.0α + 0.1β − 0.3γnorm .

5

http://www.statmt.org/moses/
An implementation of the nonlinear least-squares
Marquardt-Levenberg algorithm was used.
7
http://www.gnuplot.info/
8
Bulgarian (bg), Czech (cs), Danish (da), German (de),
6

462

γ

γ
rmax

0.92±.00
0.84±.00
0.80±.00
0.81±.00
0.59±.00
0.49±.00
0.50±.00
0.50±.00

0.73±.01
1.09±.04
1.22±.04
1.32±.06
32.31±2.0431.30±2.73
15.51±0.52
21.02±1.18

3.73±.02
3.84±.03
3.77±.03
3.76±.04
4.42±.03
4.32±.04
3.95±.02
4.10±.03

0.72
0.79
0.81
0.79
1.17
1.17
1.08
1.12

2.50

en.8

3.50

en.4
3.00

ja.juman
en

en.2
en.0

2.50

β + γnormβ

β + γnorm

a.w
c.w
m.w
p.w
a.c
c.c
m.c
p.c

α

es.8
es.4 de.8
cs.8
es.2
es.0
cs.4
de.4
cs.2
cs.0
de.2
fi.8
de.0
fi.4
fi.2
fi.0

2.00

1.50
0.80

0.90

1.00

ja.kytea

2.00

ja.mecab

1.50
1.10

0.80

0.85

α

0.90

0.95

α

Figure 5: Effects on α and β + γnorm .

Table 2: Fitted parameters on ICWB2 data.

ally halved and the change of the fitted parameters is shown in the plot on the left of Fig. 5.
*.0 denotes the original data and *.n denotes
the data of one n-th size. α does not change substantially for smaller data because of the stablesyntax features and functional words. However,β + γnorm becomes larger, which suggests that
there are fewer morphological varieties becauseof the smaller data size. The plot on the right ofFig. 5 shows how different word segmentationsin Japanese affect the parameters. There are threecommon Japanese morphological analysis tools:
kytea, mecab, and juman. kytea provides
the most fragmentary segmentation and juman
tends to attach suffixes to stems. For example, thethree tools segment wakarimashita (understood, inpolite form) as follows: waka / ri / ma / shi / ta
(5 tokens) by kytea, wakari / mashi / ta (3 tokens) by mecab, and wakari / mashita (2 tokens)by juman. As the most fragmentary segmentation by kytea contains more functional suffixes
as words, it has the largest α, and by contrast, thesegmentation by juman has the smallest α. Furthermore, mecab has a smaller β +γnorm because
it may keep proper nouns unsegmented, which canbe considered as introducing more compoundedwords. For example, tōkyōdaigaku (The University of Tokyo) is kept as one word by mecab, butsegmented as tōkyō / daigaku (Tokyo / university)by the other two tools.

guages with a small β + γnorm , that is, Germanicand Uralic, have a more gradual decrease in rarewords, which are instances of various phenomena of derivation and compounding from complex
morphology. By contrast, languages with a large
β + γnorm , such as en and fr, tend to use phrases
composed of multiple common words to express
complex concepts, so that the drop in frequency ofrare words is relatively dramatic. As β + γnorm
is sensitive to the portion of rare words, this dimension may be easily affected by the property of
specific data. An example is ro, for which a muchlarger β than other languages was fitted.
Table 2 lists the fitting results on ICWB2 Chinese data. a.*, c.*, m.*, and p.* denote
Academia Sinica, City University of Hong Kong,
Microsoft Research, and Peking University data,
respectively. *.w and *.c denote manually segmented words and characters, respectively. For the
results on words, a trade-off on α and β + γnorm
can be observed. Based on the previous analysis,
we can consider that a.w has more segmentations
on function words. An evidence is the segmentation of the expression shibushi (whether or not),which is composed of three characters shi (to be)bu (not), and shi (to be). The expression is segmented into shi / bu / shi in most cases in a.w, but
always kept together in m.w. Regarding characters, we have small α and huge β + γnorm . Notethat both common functional words and rare specific concepts in Chinese are commonly composedof multiple characters. Therefore, the contrast between common and rare characters is not so obvious, which leads to small α (no overwhelmingly
functional words in syntax) and huge β + γnorm(extremely analytic in morphology).
Figure 5 provides further evidence. The data
size of typical languages in Europarl is gradu-

4

Conclusion and Future Work

We have shown that f ∝ r−α (r + γ)−β for the
rank-frequency relation in natural languages. Thisis an explainable extension of several related formulations, with α related to the analytic featuresof syntax and β + γ to Q
that of morphology. A
more general form, f ∝ k (r + γk )−βk , can beconsidered for further investigation. The k terms
can depict k different proportional coefficients.
463

References
Atsushi Fujii, Masao Utiyama, Mikio Yamamoto, andTakehito Utsuro. 2008. Overview of the patenttranslation task at the NTCIR-7 workshop. In Proc.
of NTCIR, pages 389–400.
Philipp Koehn. 2005. Europarl: A parallel corpus forstatistical machine translation. In Proc. of MT summit, volume 5, pages 79–86.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris
Callison-Burch, Marcello Federico, Nicola Bertoldi,
Brooke Cowan, Wade Shen, Christine Moran,
Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra
Constantin, and Evan Herbst. 2007. Moses: Open
source toolkit for statistical machine translation. In
Proc. ACL (Demo and Poster), pages 177–180.
Wentian Li. 1992. Random texts exhibit Zipf’s-lawlike word frequency distribution. IEEE Transactions
on information theory, 38(6):1842–1845.
Wentian Li, Pedro Miramontes, and Germinal Cocho. 2010. Fitting ranked linguistic data with twoparameter functions. Entropy, 12(7):1743–1764.
Benoı̂t Mandelbrot. 1965. Information theory and psycholinguistics.
David M. W. Powers. 1998. Applications and explanations of Zipf’s law. In Proc. of NeMLaP3/CoNLL98,
pages 151–160.
Richard Sproat and Thomas Emerson. 2003. The firstinternational Chinese word segmentation bakeoff.
In Proc. of the SIGHAN workshop on Chinese language processing, pages 133–143.
George K. Zipf. 1935. The psycho-biology of language.
George K. Zipf. 1949. Human behaviour and the principle of least-effort.

464