# A Language as a Self-Organized Critical System

Authors: Vasilii A. Gromov; Anastasia M. Migrina
Journal: Complexity
 (2017)

---

## Abstract

A natural language (represented by texts generated by native speakers) is considered as a complex system, and the type thereof to which natural languages belong is ascertained. Namely, the authors hypothesize that a language is a self-organized critical system and that the texts of a language are “avalanches” flowing down its word cooccurrence graph. The respective statistical characteristics for distributions of the number of words in the texts of English and Russian languages are calculated; the samples were constructed on the basis of corpora of literary texts and of a set of social media messages (as a substitution to the oral speech). The analysis found that the number of words in the texts obeys power-law distribution.


---

## Body

## 1. Introduction

Since natural languages gradually came to be regarded as complex systems, a means to study linguistic processes changed from descriptive approaches to formal analysis aiming to construct mathematical model for the operation and the development of language—this change is both a reason and an effect of “the linguistic turn” (the term is due to [1]).In the frameworks of this approach, it is possible to pursue two main avenues of inquiry: The first one is aimed at constructing a theory of natural languages grammars [2]. The second one is associated with analysis of language statistical characteristics: Primary emphasis is placed, from seminal studies by Zipf [3], on distributions for separate words for written language [4–8] and for various graphs reflecting language features [9]. The principal result of these studies is a class of distributions describing natural language features [10]; the class comprises power (heavy-tail) distributions with various values of exponents: power laws manifest themselves in word’s frequency in language [3], in syntactic networks [5–7, 11], in frequency of letter sequences in vocabularies [8], and so forth.Meanwhile, a shift in XX century philosophy (akin to the Copernican revolution in astronomy) suggests considering a language as an integral whole: “a man is just a place where a language speaks itself” [12]. Therefore, since above all else a language is a unified communicative tool to convey the meanings and its text (either a literary work or a tweet) is usually a meaningful, complete message, a text becomes a basic unit for such analysis.The authors hypothesize that a natural language is a self-organized critical system (SOC) [13, 14] and the texts of a language are “avalanches” (as those are defined by Bak) flowing down the word cooccurrence graph of the respective language; large avalanches correspond to literary works, while smaller ones are associated with messages from social media. It is worth noting that a self-organized critical system conventionally features [13, 14](1)
a space of elements able to be in two states, active and passive, along with a set of rules to describe how a change of state of an element affects states of the other ones,(2)
the “avalanches” in the space that are chain reactions of elements’ state changes triggered by changes of other elements,(3)
a power-law distribution governing avalanche sizes.For a language system, a semantic space plays a part of the space at issue, and the rules are reduced to syntactic and semantic rules of the respective language. In the present study the space was formalized as a cooccurrence graph—the authors are aware that one-to-one correspondence between semantic space and vocabulary is absent, but they assume that the latter approximates the former somehow. Vertices of the graph correspond to words and an edge is present if and only if the words associated with its incident vertices occur simultaneously in the same text of the sample involved, once or more. As indicated above, an avalanche, in this context, is a text of a language, and the hypothesis that sizes of avalanches obey a power-law distribution forms the subject of the present study. One should mention that real-world (unfolding over time) SOC-systems usually exhibit long periods of slow evolution as opposed to short periods of fast evolution when the system space is changed drastically; similar phenomenon is reported to take place for evolving language systems.Another point of interest here is the emergence of a gigantic volume of information reflecting, in essence, spoken language (parole as opposed to langue according to Saussurean terminology [15], (second) Orality versus Literacy [16]) that is texts posted by users in social networks of every sort and kind (Facebook, Reddit, and so on); this makes it possible to explore this domain of human communication. In this context, the primary problem is to compare statistical characteristics calculated by means of corpus of literary texts, on the one hand, and by means of a set of texts written by social networks users. If these characteristics appear to be statistically equal, this may give proof to the idea of language unity as a complex system; and if so, written and spoken language are merely different projections of internal dynamics of this complex system (synchronic unity of language). On the other hand, the characteristic at issue calculated for different time periods (one is to be restricted in this case to the analysis of written language) can be compared in order to verify unity of the linguistic system unfolding in time (diachronic unity of language). The present work is focused on the study (and comparison) of statistical characteristics of texts sets for English and Russian languages being considered in their synchronic and diachronic aspects.The first paper to be cited among recent studies of distributions (mainly power-law distribution) observed for oral and written language is a brilliant review [17]. Here, the object of study is a particular text, and its basic unit is a word or a sentence—distributions of characteristics of these objects form the subjects of the overwhelming majority of papers for this line of investigation [18]. For example, Font-Clos et al. [19] examine dependence text length versus statistical properties of word occurrences; the authors reveal that the distribution obeys the power and investigate its relationship with Zipf’s and Heaps’ (Herdan’s) laws (the former states that the vocabulary grows as a power function of text’s length) [20].Another object of study that generates power-law distributions is a representation of language semantic and syntactic relations using various discrete structures: semantic nets [17], global syntactic dependency trees [21, 22], cooccurrence graphs [18], and others. Both conventional methods aimed at exploring these structures as complex networks [23] and random walks on these structures result in power-law distributions [17, 21, 22, 24, 25]. A basic unit is a word or a sentence likewise.The subject of the present paper is language as a whole; texts (semantic “avalanches”) are considered as its basic units. The rest of the paper is organized as follows. The next section outlines methods used to estimate distribution parameters; the third provides results for both English and Russian languages. The fourth section discusses results; finally, the last section presents conclusions.


## 2. Methods

The choice of the languages is determined, apart from the availability of voluminous corpora of texts (for written and spoken languages) for them, by their qualitative difference in grammar structure: Russian is an inflected language, while in English inflections are rather rare; Russian is characterized by flexible word order in a sentence, whereas word order in English language is strict, and rare exceptions are constrained by stringent rules [26, 27].We used two different approaches to test statistical hypothesis in question. The first one utilizing the concept of data collapse is considered in greater detail in the monograph by Pruessner [14]; the second one (grounded on the Kolmogorov-Smirnov (KS) criterion) is proposed in the source [28]. Both methods not only evaluate the exponents but also cut off the smallest elements of the sample that usually do not fit a power-law. The first method also cuts off the largest nonfitting elements. It is worth noting that this phenomenon (the sharp distinction between the largest [smallest] elements and all others) seems to be a salient characteristic of real-world data following power-law distributions [13, 14].The first approach we dwell on briefly is that using the concept of data collapse [14]. It assumes that the distribution generating the data has the following probability density function:(1)px=ax-τgxbLDwith x which can be a continuous or discrete random variable, metric factors a and b, characteristic dimension of a system L, scaling function g, and scaling exponents τ and D as well. The distribution follows the modified power-law within the interval bounded by the lower and upper cutoffs xmin and xmax. The scaling function g (that distinguishes the distribution from a canonical power-law) fits many real-world systems obeying heavy-tail distributions [14]. The quantity bLD determines the characteristic upper cutoff.The raw data is taken to be previously binned, that is, grouped together and averaged over the observations belonging to the same group; we used the exponential binning for its suitability for this kind of data [14].If the null hypothesis (that the sample under study is generated from the distribution equation (1)) is true, then p-(x)x-τ plotted against ξ=x/xc(L), where xc(L)=bLD, gives the same function ag(ξ) for various L, where p-(x) is the empirical probability density function, and τ is the true value of the power exponent. The phenomenon is given the title of data collapse. Thus, for given data generated from the power-law distribution, p-(x)x-τ (as a function of ξ) plotted for various L is superimposed on each other.Therefore, the respective goodness-of-fit test for power-law distribution involves the following steps:(1)
binning of the raw data(2)
plottingp-(x)x-τ against ξ=x/xc(L) for various L using “apparent exponent” τ^ (rough estimate of τ)—such a plot comprises a nonhorizontal straight line and a characteristic nonlinear curve, whose extremum is called a landmark (xc represents its coordinate)(3)
the refinement of the value with the employment of the least squares method applied to the landmarks.As a result, the plots merge into a single horizontal straight line for the section of the domain of definition for which power-law holds true (between the lower and upper cutoffs).The second approach [28] is applicable to power-law distribution without scaling function:(2)px=0,x≤xminax-τ,xmin<xwith normalization constant a=1/ζ(τ,xmin), where (3)ζτ,xmin=∑i=1ni+xmin-τis the generalized (Hurwitz) zeta function. The method implies that all practicable values of the lower cutoff xmin are considered; for each xmin the estimate of the power exponent (with the maximum likelihood principle in mind) is calculated from (4)τ^=1+n∑i=1nln⁡xixmin-1;see [28]; for each xmin the Kolmogorov-Smirnov statistic S=supx≥xmin⁡P(x)-P-(x) (where P(x) is the cumulative distribution function (CDF) with estimated value of τ and P-(x) is the empirical CDF) is calculated. The eventual estimate of xmin minimizes S(xmin). For real-world data, the function S(xmin) possesses, usually, several local minima; it is often reasonable not to choose a global minimum but the local minimum closest to xmin, the lower boundary of the domain of definition, provided a value of the statistic at it does not differ significantly from that at a global minimum.


## 3. Power-Law Distributions for the English and Russian Languages

To test the null hypothesis in question (that the number of words in the texts obeys a power-law distribution; (1) is used to verify data collapse, while the method based on the KS statistics employs (2), (3)), two samples were generated on the basis of corpora of literary texts for these languages and of a set of Reddit messages (or its Russian counterpart Pikabu). The resulting samples sizes for English language are 9820 (literary works), 5016 (Reddit), and 14836 (joint sample); for Russian language they are 12683 (literary works), 6005 (Pikabu), and 18688 (joint sample). For the method based on the concept of data collapse, the size of vocabulary used to generate texts serves as characteristic dimension of a system L. To obtain samples for various L, one resamples down the initial sample deleting randomly λ% of words from the complete vocabulary and then from all the texts used. This brings about the generation of new samples corresponding to the characteristic dimension of L1-λ/100.Figure1 presents a dependence of the number of words in a text in double-logarithmic scale on a rank of the text in the sample; namely, Figures 1(a), 1(b), and 1(c) correspond to the joint sample, to the sample constructed on the basis of literary works, and to poetry works for the English language, respectively; Figures 1(d), 1(e), and 1(f) exhibit the same dependence for the Russian, respectively; a dashed straight line in each subfigure corresponds to power distribution with an exponent estimated using data collapse.Figure 1
The empirical probability density function for the random variable “the number of words in a text” (double-logarithmic scale). (a) English texts, joint sample, (b) English literary works, (c) English poetry, (d) Russian texts, joint sample, (e) Russian literary works, and (f) Russian poetry. Dashed straight lines correspond to power exponents estimated using data collapse.
(a)
(b)
(c)
(d)
(e)
(f)Figure2 (in the coordinates (x-τp-(x), ξ=x/xc), xc is a landmark coordinate) shows data collapse for the joint samples for English (Figure 2(a)) and Russian (Figure 2(d)) languages. Raw data was binned exponentially with bin sizes Bj=cRjR-1 (c=1 and R=1.4). Red colour (with discs) stands for the complete vocabulary (of size L), grey colour (with squares) is for the vocabulary of size 0.9L, blue colour (with diamonds) is for 0.8L, black colour (with triangles) is for 0.7L, orange colour (with upturned triangles) is for 0.6, and, finally, purple colour (with circles) is for 0.5L; the curves are dragged apart a little bit in order to make it possible to distinguish as they are superimposed owing to data collapse. Figures 2(b) and 2(e) present the same dependence for samples constructed using corpora of literary texts for English (Figure 2(b)) and Russian (Figure 2(e)) languages. Figures 2(c) and 2(f) demonstrate data collapse for poetry samples for English and Russian languages, respectively.Figure 2
Collapsed distributions. (a) English texts, joint sample, (b) English literary works, (c) English poetry, (d) Russian texts, joint sample, (e) Russian literary works, and (f) Russian poetry. Red colour (with discs) stands for the complete vocabulary (of sizeL), grey colour (with squares) is for the vocabulary of size 0.9L, blue colour (with diamonds) is for 0.8L, black colour (with triangles) is for 0.7L, orange colour (with upturned triangles) is for 0.6L, and, finally, purple colour (with circles) is for 0.5; the curves are artificially dragged apart a little bit in order to make it possible to distinguish them as they are superimposed owing to data collapse. x-τp-(x) versus ξ=x/xc; xc is a landmark coordinate.The results obtained using both approaches are presented in Table1; the table includes results for samples constructed on the basis of literary works and of messages of social media and for joint sample as well for both languages. Each cell contains estimates for a power exponent and (in parentheses) for a lower cutoff xmin. The above results suggest synchronic unity for both languages because of a good agreement of estimates for power exponents and lower cutoffs calculated for literary works and for social media messages.Table 1
Estimated power-law exponents and lower cutoffs for distributions of the number of words in texts of literary works and social media.
Sample type
English
Russian
Estimates based on data collapse
Estimates based on KS criterion
Estimates based on data collapse
Estimates based on KS criterion
x
m
i
n
τ
x
m
i
n
τ
x
m
i
n
τ
x
m
i
n
τ
(nw)
(dl)
(nw)
(dl)
(nw)
(dl)
(nw)
(dl)
Literary works
114
1.30
55
1.29
72
1.78
56
1.97
Poetry works
125
1.91
109
1.89
112
2.61
98
2.63
Social media
111
2.65
124
2.63
183
2.10
151
2.15
Joint sample
109
1.39
42
1.34
85
1.82
62
1.95
nw: the number of words; dl: dimensionless.In order to deal with the problem of diachronic unity of a language, authors confined themselves to the samples generated on the basis of literary works created before the 20th century and in the 20th century for both the English and Russian languages (respective samples sizes amount to 7179 and 2641 for the English language, 5758 and 6925 for the Russian language). Table2 exhibits the respective results.Table 2
Estimated power-law exponents and lower cutoffs for distributions of the number of words in texts for literary works for XIXth (and before it) and for XXth century.
Time limit
English
Russian
Estimates based on data collapse
Estimates based on KS criterion
Estimates based on data collapse
Estimates based on KS criterion
x
m
i
n
τ
x
m
i
n
τ
x
m
i
n
τ
x
m
i
n
τ
(nw)
(dl)
(nw)
(dl)
(nw)
(dl)
(nw)
(dl)
XIX century and early
118
1.37
75
1.37
91
1.71
53
1.87
XX century
121
1.19
54
1.21
74
1.75
53
2.03
nw: the number of words; dl: dimensionless.


## 4. Discussion

Data collapse implies that if the distribution obeys power-law, the transformed distributions possess an interval with horizontal line and coincide inside this interval. For real-world data, the line inside the interval may not be so straight, but coincidence must occur as Figure2 shows (one should take into account the fact that the curves are artificially dragged apart a little bit in order to make it possible to distinguish them as they are superimposed owing to data collapse). Analogously, the results produced by the method that uses KS statistics also count in favour of the hypothesis that distributions are power-law.We would like to emphasize that we regard this assumption as a plausible hypothesis as before; the results of the previous sections are arguments in its favour, not final results. It seems to us extremely important to ensure global visibility of this hypothesis. We strongly hope that other papers concerning this hypothesis will occur, with broader data sets and, probably, with more rigorous statistical methods. Fairly good agreement (for this class of distributions) of parameters for separate distributions of literary works created before the 20th century and in the 20 century (Table2) suggests that a language (at least written language) is a single system diachronically.We would like to dwell on a cooccurrence graph as a semantic space in greater detail as opposed to rather popular global syntactic dependency tree and similar structures. In the present paper, a text is considered as a basic unit of a language; thus a sentence is a means to break (rather arbitrary) this semantic “avalanche” down. Global syntactic dependency tree is a great tool to explore this avalanche locally, but as far as it generally fails to reveal cross-sentence semantic dependencies, it does not seem to be the best tool to examine the avalanche as a whole. Therefore cooccurrence graph is a natural choice; an edge belongs to this graph if the words (corresponding to its vertices) belong to the same text. Generally, the underlying structure does not seem to be of principal importance for the problem considered.In our opinion, this allows one to distinguish results of the present work and those for global syntactic dependency trees and cooccurrence graphs with the employment of random walks [24, 25, 29, 30]. We explore real-world semantic “avalanches” generated by a particular language, while the algorithms using random walks produces artificial “avalanches” with the employment of a graph complying with a natural language. In particular, such motion continues (theoretically) perpetually, whereas the avalanches considered in this article are of finite sizes, explicitly defined by the authors of the respective texts. In order to emphasize fundamental distinction between these approaches, one could draw the following analogy: the approach of the present paper and those associated with random walks are related to study of joint distribution of random variables and of product of distributions of these variables. Nevertheless, the results of this study are likely to be useful for switcher-random-walks models [30] to estimate realistically switching time.We would also like to emphasize the difference between the classic Zipf’s laws and the distributions considered in this paper: Zipf studied the laws governing means to represent information, while we attempt to explore laws governing semantic flows and, moreover, semantic flows of a language as a whole.


## 5. Conclusions

As a result of the above analysis several conclusions may be reached on linguistic systems of English and Russian languages. A language system (given by texts generated in its frameworks) is a self-organized critical system defined on its word cooccurrence graph. Texts of a language are “avalanches” flowing down this graph; the large avalanches correspond to literary works, while the smaller ones are associated with spoken language. A fairly good agreement of parameters for separate distributions of literary works and of social media offers a clearer view of synchronic unity of each linguistic system; on the other hand, an analogous comparison between distributions for literary works of XIXth (and before) and of XXth centuries suggests diachronic unity for the systems. Poetry distributions appear closest to a canonical power-law and therefore poetry may be treated as a kind of supporting column of a language.



---

*Source: 9212538-2017-11-19.xml*