Venue: NAACL 2016

# Statistical Modeling of Creole Genesis

Yugo Murawaki

Graduate School of Informatics, Kyoto University

Yoshida-honmachi, Sakyo-ku, Kyoto, 606-8501, Japan murawaki@i.kyoto-u.ac.jp

###### Abstract

Creole languages do not fit into the traditional tree model of evolutionary history because multiple languages are involved in their formation. In this paper, we present several statistical models to explore the nature of creole genesis. After reviewing quantitative studies on creole genesis, we first tackle the question of whether creoles are typologically distinct from non-creoles. By formalizing this question as a binary classification problem, we demonstrate that a linear classifier fails to separate creoles from non-creoles although the two groups have substantially different distributions in the feature space. We then model a creole language as a mixture of source languages plus a special _restructurer_. We find a pervasive influence of the restructurer in creole genesis and some statistical universals in it, paving the way for more elaborate statistical models.

## 1 Introduction

While most linguistic applications of computational phylogeny rely on lexical data Gray and Atkinson (2003); Bouckaert et al. (2012), there is a growing trend to make use of topological data Tsunoda et al. (1995); Dunn et al. (2005); Teh et al. (2008); Longobardi and Guardiano (2009); Murawaki (2015). One advantage of typological features over lexical traits (cognates) is that they allow us to compare an arbitrary pair of languages even if they do not share enough cognates. For this reason, they have the potential of uncovering external relations involving language isolates and tiny language families such as Ainu, Basque, and Japanese.

However, our understanding of typological changes is far from satisfactory in at least two respects. First, typological changes are less intuitive than the birth and death of a lexical trait. Modeling word-order change with a single transition matrix Maurits and Griffiths (2014), for example, appears to be an oversimplification because some complex mechanisms must be hidden behind the changes Murawaki (2015).

The second point, the main focus of this paper, is that it is not clear whether typological data fit into the traditional tree model for a group of languages, which has long been used as the default choice to summarize evolutionary history Schleicher (1853). To be precise, regardless of whether typological features are involved, linguists have viewed the tree model with suspicion. A central problem of the tree model is its assumption that after a branching event, two resultant languages evolve completely independently. However, linguists have noted that horizontal contact is a constitutive part of evolutionary history. Various models for contact phenomena have been proposed to address this problem, including the wave theory Schmidt (1872) and the gravity model Trudgill (1974). As for linguistic typology, areal linguistics has worked on the diffusion of typological features across languages within a geographical area Campbell (2006).

In this paper, we study creole languages as an extreme case of non-tree-like evolution Wardhaugh and Fuller (2015). A creole is developed as a result of intense contact between multiple languages: typically one socioculturally dominant language (superstrate) and several low-prestige languages (sub

strates_). Superstrates are also known as _lexifiers_ because the lexicon of a creede is largely derived from its superstrate. In spite of this, the grammar of a creede is drastically different from that of its lexifier. It is often said that creede grammars are simpler than non-creole ones although it is not easy to measure grammatical simplicity.

Creoles are not irrelevant to historical linguistics because some have speculated about the plausibility of creede status for Middle English Bailey and Maroldt (1977) and (pre-)Old Japanese Kawamoto (1974); Akiba-Reynolds (1984); Kawamoto (1990) while they have been criticized harshly by others.

This controversy can only be settled by fully understanding creede genesis, or the question of how creoles emerge, which also remains unresolved Wardhaugh and Fuller (2015). One theory called the gradualist hypothesis suggests their staged development from pidgin languages. A pidgin arises when speakers of different languages with no common language try to have a makeshift conversation, which results in a drastic simplification of grammar called _pidgin formation_. A pidgin is then acquired by children as their first language and is transformed into a fully functional language. This process of grammatical elaboration is known as _creole formation_.

We follow Bakker, Daval-Markussen and colleagues in taking data-driven approaches to this problem Bakker et al. (2011); Daval-Markussen and Bakker (2012); Daval-Markussen (2013). They argue that creoles are topologically distinct from non-creoles. They compare four theories of creede genesis:

1. [topsep=0pt]
2. _Superstratist_. The lexifier plays a major role in creede genesis.
3. _Substratist_. Substrates do, instead of the lexifier.
4. _Feature pool_. Both the lexifier and substrates, but nothing else, provide a feature pool from which each feature value of a creede is selected.
5. _Universalist_. Innate ability of humans to create language is emphasized. The hypothetical linguistic universals for creole formation are called _restructuring universals_. To test these explanations, they apply NeighborNet Bryant and Moulton (2004), a bottom-up, distance-based clustering method developed in the field of computational biology. By demonstrating that, as in Figure 1, creoles form a cluster distinguishable from lexifiers, substrates and other non-creole languages, they argue for the universalist position and for creode distinctiveness.

However, we find both theoretical and methodological problems in their discussion. Theoretically, the synchronic question of creede distinctiveness is confused with the diachronic question of creede genesis. If creoles are distinct from non-creoles, then something specific to creoles (e.g., restructuring universals) would play a role, but not vice versa. It is logically possible that even with restructuring universals, creoles are indistinguishable from (a subgroup of) non-creoles. Methodologically speaking, NeighborNet does not straightforwardly explain fundamentally non-tree-like evolution because it does assume tree-like evolution even though it represents conflicting signals with reticulations.

We begin by addressing the question of creede (non-)distinctiveness. Whether one is distinct from another is straightforwardly formalized as a binary classification problem. We show that an SVM classifier fails to separate creo 

lowing a practice in population genetics, we visualize the data using principal component analysis (PCA). The result suggests that although creoles have a substantially different distribution from non-creoles, they nevertheless overlap.

Next, we propose to model creole genesis with mixture models. In this approach, a creole is stochastically generated by mixing its kexifier, substrate(s) plus a special _restructure_. Conceptually, this is the opposite of the tree model, as illustrated in Figure 2. Specifically, we present two Bayesian models. The first one considers one mixing proportion per creole, and the other decomposes the proportions into per-feature and per-creole factors. Our experimental results suggest that the restructured dominates creep genesis, dismissing the superstratist, substratisl and feature pool theories. We also find some statistical universals in the restrucuter although we refrain from identifying them as _restructuring_ universals. In this way, we represent a first step toward understanding the complex process of creole genesis through statistical models.

## 2 Related Work

### Population Genetics

Like Bakker, Daval-Markussen and colleagues [14, 30], we borrow ideas from computational biology. For reasons unknown to us, they chose clustering models that basically assume tree-like evolution [65, 79]. However, creole genesis is more comparable to models that explicitly take into account genetic admixture (i.e., contact phenomena). See Jones et al. (2015), for example, to take a look at standard practices in population genetics.

Population genetic analysis of genotype data (binary sequences comparable to sets of linguistic features) can be grouped into two types: population-level and individual-level analysis. Populations, such as Sardinian, Yoruba and Japanese, are predefined sets of individuals. Population-level analysis utilizes genetic variation within a population [8]. From a modeling point of view, languages are more comparable to individuals. Although a language is spoken by a population, no linguistic data available are comparable to _a set of_ individuals.

Individual-level analysis, where population labels are used only for the purpose of visualization, is often done using PCA and admixture analysis. PCA is used for dimensionality reduction: by selecting the first two principal components, high-dimensional sequences are projected onto an informative two-dimensional diagram [8].

Admixture analysis [8, 10] closely resembles topic models, most notably Latent Dirichlet Allocation (LDA) [1], in NLP. It assumes that each individual is a mixture of \(K\) ancestral components (i.e., topics). One difference is that while each LDA topic is associated with a single word distribution (\(K\) distributions in total), each SNP (i.e., feature type) has its own distribution (\(K\times J\) distributions in total for sequences with length \(J\)).

### Linguistic Typology and Non-tree-like Evolution

Like lexical data, typological features are usually analyzed with a tree model, but Reesink et al. (2009) are a notable exception. They applied admixture analysis to Australian and Pappuan languages, for which tree-building techniques had not been successful. They related inferred ancestral components to putative prehistoric dispersals and contacts.

Independently of biologically-inspired studies, Daume III (2009) incorporated linguistic areas into a phylogenetic tree. In his Bayesian generative model, each feature of a language has a latent variable which determines whether it is derived from an areal cluster or the tree. Thus his model can be seen as a mixture model.

Figure 2: Schematic comparison of two approaches to creole genesis.

 



## 3 Data and Preprocessing

We used the online edition1 of the Atlas of Pidgin and Creole Language Structures (APiCS) [2], a database of pidgin and creole languages. It was larger than the datasets of Bakker et al. (2011). As of 2015, it contained 76 languages (104 varieties). It was essentially a pidgin-and-creole version of the online edition2 of the World Atlas of Language Structures (WALS) [9], but it contained sociolinguistic features, phonological inventories and example texts in addition to typological features.

Footnote 1: [http://apics-online.info/](http://apics-online.info/)

Footnote 2: [http://wals.info/](http://wals.info/)

As ApiCS did not mark creoles, we used the sociolinguistic feature "Ongoing creolization of pidgins" as a criterion to select creoles. Specifically, we filtered out languages whose feature value was neither "Not applicable (because the language is not a pidgin)" nor "Widespread."

In APiCS, 48 out of 130 typological features were mapped to WALS features. We used these features to combine creoles from APIiCS with non-creoles from WALS. Since the WALS database was sparse, we selected languages for which at least 30% of the features were present. As a result, we obtained 64 creoles and 541 non-creoles.

We imputed missing data using the R package _missMDA_[9]. It handled missing values using multiple correspondence analysis (MCA). Specifically, we used the imputeMCA function to predict missing feature values.

When investigating creepic distinctiveness, we used binary representations of features. Using a one-of-\(K\) encoding scheme, we transformed 48 categorical features into 220 binary features.

Our mixture models require each creole to be associated with a lexifier and substrate(s). Unfortunately, APiCS described these languages in an obscure way (and many of them are indeed not fully resolved). We had no choice but to manually select several modern languages as _proxies_ for them. For simplicity, we chose only one substrate per creole, but it is not difficult to extend our model for multiple substrates. We are aware that these are oversimplification, but we believe they would be adequate for a proof-of-concept demonstration.

## 4 Creole Non-distinctiveness

### Binary Classification

To determine whether creoles are distinct from non-creoles, we apply a linear SVM classifier to the typological data. Here, linearity is assumed for two reasons. First, since the supposed distinctiveness is explained by restructuring universals, there is no way for creoles to have an XOR-like distribution. Second, daval-markussen (2013) claims that as few as three features are sufficient to distinguish creoles from non-creoles. If this is correct, it is expected that given 48 categorical features, even a simple lin

Figure 3: PCA of creoles (red squares) and non-creoles (green circles) with explained variance in the labels of the axes. (a) Scatterplot of both types of languages. (b) Kernel density estimates (KDEs) of creoles. (c) KDEs of non-creoles.



ear classifier can work nearly perfectly.

The classifier is trained to classify whether a given language, represented by binarized features, is a creode (\(+1\)) or non-creole (\(-1\)). We use 5-fold cross validation with grid search to tune hyperparameters.

In our experiments, the accuracy, recall, precision and F1-measure were 97.2%, 88.5%, 84.4% and 86.4%, respectively. Table 1 shows the confusion matrix. We can see that the classifier failed to separate creoles from non-creoles. Although the classifier worked well, borderline cases remained.

### Pca

For exploratory analysis and visualization, we applied PCA to creoles and non-creoles, again represented by binarized features. Figure 3 depicts the scatterplot of the first two principal components. We can see that creoles were characterized by quite a different distribution from that of non-creoles. The creoles were concentrated on the lower center while most non-creoles belonged to one of two clusters in the middle. However, the distribution of creoles did overlap with that of non-creoles.

Having a closer look at the diagram, we found that Negerhollands (Dutch), Cape Verdean Creole of Brava (Portuguese) and Vincentian Creole (English) were among the most "typical" creoles (lexifiers in parentheses). Tok Pisin (English) and Bislama (English) were at the periphery of the cluster. The outliers on the upper left included Korliat (Portuguese) while Kikongo-Kituba (Bantu) lay on the upper right.

On the other hand, "creole-like" non-creoles included Chontal Maya (a Mayan language of Mexico), Mussau (an Oceanic language of Papua New Guinea),3 Catalan and other European languages. The non-creole cluster on the middle left consisted of Japanese, Kannada, Maltese and others. Another non-creole cluster on the middle right included Swahili, Hawaiian and Khmer. The creoleless upper central area was occupied by Lalo (Sino-Tibetan), Maninka (Western) (Mande in West Africa), Salt-Yui (Trans-New Guinea) among others.

Footnote 3: Interestingly, Mussau is noted for contact-induced changes [Brownie, 2012].

## 5 Mixture Models for Creole Genesis

### Basic Idea

Forsaking the quest for synchronic distinctiveness, we take a more direct approach to the diachronic question of creole genesis. Since multiple languages are involved in creole genesis, it is reasonable to apply a mixture model. We assume that a creole is stochastically generated by mixing three sources: (1) a lexifier, (2) a substrate and (3) a global _restructure_. Under this assumption, the main question is with what proportions these sources are mixed.

An unusual property of our model as a mixture model is that not only outcomes (creoles) but most sources (lexifiers and substrates) are observed. We only need to infer the restructurer. Thus another question is what the restructurer looks like.

Note that our model is constructed such that it does not commit to a particular theory of creole genesis. If the superstratist theory is correct, then leafiers would dominate the inferred mixing proportions. The same is true of the substratist theory. Similarly, the feature pool theory entails that the restructurer only occupies negligible portions. Also note that even if the restructurer plays a significant role, it does not necessarily imply the universalist position. The restructurer is a set of catch-all feature distributions for those which are explained neither by the verifier nor by the substrate (that is why we avoid calling it restructuring universals). In order for it to be linguistic universals, it must show some consistent patterns in its distributions.

### Mono Model

Our idea is materialized in two Bayesian generative models. The first one, called Mono, is similar to the Structure algorithm of admixture analysis [Pritchard et al., 2000].4

Footnote 4: As seen in Section 2.1, Mono is more similar to Structure than to LDA in that each feature type has its own distributions. The difference is that while Structure infers all \(K\) global components, Mono always has one global component

\begin{table}
\begin{tabular}{c|c|c c}  & & \multicolumn{2}{c}{**System**} \\ \cline{3-4}  & & **C** & **NC** \\ \hline \multirow{2}{*}{**Reference**} & **C** & 54 & 10 \\  & **NC** & 7 & 534 \\ \hline \end{tabular}
\end{table}
Table 1: Confusion matrix of binary classification. C stands for creoles and NC for non-creoles.

 

Every language in the model is represented by a sequence of categorical features. The number of possible values varies among feature types. For feature \(j\) of creede \(i\), the latent assignment variable \(z_{i,j}\) determines from which source the feature is derived, a verifier (L), a substrate (S) or the restructurer (R). Each creede \(i\) is associated a priori with a lexifier and a substrate. Let \(y_{i,j,\text{L}}\) and \(y_{i,j,\text{S}}\) be the values of feature \(j\) of creede \(i\)'s lexifier and substrate, respectively. If the source is the lexifier (or substrate), the creede simply copies \(y_{i,j,\text{L}}\) (or \(y_{i,j,\text{S}}\)). For the sake of uniformity, we can think of a lexifier (or substrate) as a set of feature distributions each of which concentrates all probability mass on its observed value (i.e., the \(\delta\) function). The remaining source, the restructure, is a set of categorical feature distributions each of which is drawn from a Dirichlet prior.

The assignment variable \(z_{i,j}\) is generated from \(\theta_{i}\), which in turn is generated from a Dirichlet prior. \(\theta_{i}=(\theta_{i,\text{L}},\theta_{i,\text{S}},\theta_{i,\text{R}})\) is the parameter of a categorical distribution which specifies the mixing proportion of the three sources for creede \(i\).5

Footnote 5: By letting another categorical distribution subdivide \(\theta_{i,\text{S}}\), we can incorporate multiple substrates into the model.

More concretely, the generative story of Mono is as follows:

1. For each feature type \(j\in\{1,\cdots,J\}\) of the restructurer: 1. draw a distribution from a symmetric Dirichlet distribution \(\phi_{j}\sim\operatorname{Dir}(\beta_{j})\)
2. For each creede \(i\in\{1,\cdots,N\}\): 1. draw a mixing proportion from a symmetric Dirichlet distribution \(\theta_{i}\sim\operatorname{Dir}(\alpha_{i})\) 2. then for each feature type \(j\in\{1,\cdots,J\}\): 1. draw a topic assignment \(z_{i,j}\sim\operatorname{Categorical}(\theta_{i})\) 2. draw a feature value \[x_{i,j}\sim\begin{cases}\delta(y_{i,j,\text{L}})&\text{if $z_{i,j}=\text{ L}$}\\ \delta(y_{i,j,\text{S}})&\text{if $z_{i,j}=\text{ S}$}\\ \operatorname{Categorical}(\phi_{j})&\text{if $z_{i,j}=\text{ R}$}\end{cases}\] As usual, we marginalize out \(\phi_{j}\) and \(\theta_{i}\) using conjugacy of Dirichlet and categorical distributions Griffiths and Steyvers (2004). We use Gibbs sampling to infer \(z_{i,j}\), whose probability conditioned on the rest is proportional to \[\begin{cases}\left(\alpha_{i}+c^{-(i,j)}_{i,\text{L}}\right)\;\mathrm{I}(x_{ i,j}=y_{i,\text{J}})&\text{if $z_{i,j}=\text{ L}$}\\ \left(\alpha_{i}+c^{-(i,j)}_{i,\text{S}}\right)\;\mathrm{I}(x_{i,j}=y_{i,j, \text{S}})&\text{if $z_{i,j}=\text{ S}$}\\ \left(\alpha_{i}+c^{-(i,j)}_{i,\text{R}}\right)\xrightarrow[B_{j}+c^{-(i,j)}_{ \text{R},j,\ast}]{\beta_{j}+c^{-(i,j)}_{\text{R},j,\ast}}&\text{if $z_{i,j}=\text{ R}$}\end{cases}\] (1) where I is an indicator function, \(B_{j}=\sum\beta_{j}\), \(c^{-(i,j)}_{i,k}\) is the number of assignments for creede \(i\), except \(z_{i,j}\), whose values are \(k\), and \(c^{-(i,j)}_{\text{R},j,l}\) is the number of observed features for feature type \(j\), except \(x_{i,j}\), that is derived from the restructure and has \(l\) as its value. Intuitively, the first term gives priority to the source from which many other features of creede \(i\) are derived. The second term concerns how likely the source generates the feature value. For the lexifier or the substrate, it is \(1\) only if the source shares the same feature value with the creede; otherwise \(0\). To tune hyperparameters \(\alpha_{i}\) and \(\beta_{j}\), we set a vague gamma prior \(\operatorname{Gamma}(1,1)\) and sample these parameters using slice sampling Neal (2003).

### Fact Model

It is said that some features are more easily borrowed than others Matras (2011). For creoles, some seems to reflect substrate influence on phonology while reduced inflections might be attributed to the restructurer. Inspired by these observations, we extend the Mono model such that some feature types can have strong connections to particular sources. We call this extended model Fact.

To do this, we decomposes the mixing proportions into per-feature and per-creole factors. We apply additive operations to these factors in log-space in a way similar to the Sparse Additive Generative model Eisenstein et al. (2011). As a result of this extension, every feature \(j\) of creede \(i\) has its own mixing proportion, \(\theta_{i,j}=(\theta_{i,j,\text{L}},\theta_{i,j,\text{S}},\theta_{i,j,\text{R}})\):

\[\theta_{i,j,k}=\frac{\exp(m_{j,k}+n_{i,k})}{\sum_{k}\exp(m_{j,k}+n_{i,k})}, \tag{2}\]

where \(m_{j,k}\) is a factor specific to feature type \(j\) and \(n_{i,k}\) is the one specific to creede \(i\). To penalize extreme values, we put Laplacian priors on \(m_{j,k}\) and \(n_{i,k}\), with mean \(0\) and scale \(\gamma\).

To sum up, the generative story of Fact is as follows: 

1. For each feature type \(j\in\{1,\cdots,J\}\): 1. draw \(\phi_{j}\sim\mathrm{Dir}(\beta_{j})\) 2. for each source \(k\in\{\mathrm{L},\mathrm{S},\mathrm{R}\}\): 1. draw \(m_{j,k}\sim\mathrm{Laplace}(0,\gamma)\)
2. For each creole \(i\in\{1,\cdots,N\}\): 1. for each source \(k\in\{\mathrm{L},\mathrm{S},\mathrm{R}\}\): 1. draw \(n_{i,k}\sim\mathrm{Laplace}(0,\gamma)\) 2. then for each feature \(j\in\{1,\cdots,J\}\): 1. normalize \(m_{j,k}\) and \(n_{i,k}\) to obtain \(\theta_{i,j}\) (Equation (2)) 2. draw a topic assignment \(z_{i,j}\sim\mathrm{Categorical}(\theta_{i,j})\) 3. draw \(x_{i,j}\) as in Mono \(\phi_{j}\) is integrated out as before, but the conjugacy no longer holds for \(\theta_{i,j}\).

For inference, a modification is needed to infer \(z_{i,j}\): the first term \(\alpha_{i}+c^{-(i,j)}_{i,k}\) of Equation (1) is replaced with \(\theta_{i,j,k}\). \(m_{j,k}\) and \(n_{i,k}\) are sampled using the Metropolis algorithm, with a Gaussian proposal distribution centered at the previous value. Hyperparameter \(\gamma\) is set to \(10\).

### Results

Table 2 summarizes mixing proportions. For Mono and Fact (combined), we use a fraction of assignment variables pointing to a particular souce. Per-feature and per-creole factors are converted into probabilities as follows: per-feature proportions \(\tilde{\phi}_{j}=(\tilde{\phi}_{j,\mathrm{L}},\tilde{\phi}_{j,\mathrm{S}}, \tilde{\phi}_{j,\mathrm{R}})\), where \(\tilde{\phi}_{j,k}=\frac{\exp(m_{j,k})}{\sum_{k}\exp(m_{j,k})}\). Similarly, per-creole proportions \(\tilde{\theta}_{i}=(\tilde{\theta}_{j,\mathrm{L}},\tilde{\theta}_{j,\mathrm{ S}},\tilde{\theta}_{j,\mathrm{R}})\), where \(\tilde{\theta}_{i,k}=\frac{\exp(n_{i,k})}{\sum_{k}\exp(n_{i,k})}\).

We can see that the overwhelming majority of features were derived from the restructured both in Mono and Fact (combined). The restructurer was followed by lexifiers, and substrates were the least influential.6 These results can be interpreted as counter-evidence to the superstratist, substratist and feature pool theories.

Footnote 6: The substrates would probably occupy a larger portion if multiple substrates are incorporated in future work.

Mono and Fact (combined) exhibited similar patterns. When the mixing proportions are decomposed into per-feature and per-creole factors, per-creole factors exhibited less uneven distributions than per-feature factors. This implies heterogeneous behavior of features in creole genesis. Table 3 lists top-5 feature types for each source.

Figure 4 plots recoles on a simplex of mixing proportions in Mono. Creoles scattered across the simplex but leaned toward the restructurer. This implies that a kelifier cannot be mixed with substrates without interference from the restructurer.

Compared with Mono, Fact tended to push points to the edges of the simplex. This can be confirmed in Figure 5. In particular, Figure 5(c) is directly comparable to Figure 4. It is possible that halfway points in Mono were artifacts of its limited expressive power.

Table 4 lists the top-10 feature type-value pairs that were derived from the restructurer. In other words, we stochastically removed the influence of the lexifiers and substrates from creole data. These features can be regarded as (statistical) universals

\begin{table}
\begin{tabular}{l|l|l l c} \hline  & \multicolumn{4}{c}{**Sources**} \\ \cline{3-5}  & **Model** & L & \(\mathrm{S}\) & \(\mathrm{R}\) \\ \hline \multirow{3}{*}{Fact} & Mono & 16.6\% & 9.3\% & 74.1\% \\ \cline{2-5}  & Combined & 17.6\% & 6.0\% & 76.4\% \\ \cline{1-1}  & Per-feature & 22.5\% & 6.8\% & 70.8\% \\ \cline{1-1}  & Per-creole & 25.0\% & 20.4\% & 54.6\% \\ \hline \end{tabular}
\end{table}
Table 2: Summary of mixing proportions. The arithmetic mean of 50 samples after 5,000 iterations, with an interval of 100 iterations.

 

although our model leaves the possibility that they were not _restructuring_ universals. To answer this question, we need to break down the restructure by types of linguistic universals.

Among the 10 feature type-value pairs, only four apply to Japanese (Negative Indefinite Pronouns and Predicate Negation, Intensifiers and Reflexive Pronouns, Alignment of Case Marking of Pronouns, and Order of Numeral and Noun). For reference, English has seven. Combined with the PCA analysis in Section 4.2, this suggests that Japanese is a very non-creole-like language. However, we are unsure if the possibility of creole status for (pre-)Old Japanese is completely rejected. This question might be answered if we figure out how long it takes to make creole-like traits disappeared.

It is often said that Creoles have SVO word order. According to APiCS, the number of creoles with SVO order was 61 (exclusive) and 71 (exclusive plus shared) in the 76 language dataset. However, this feature value only gained the ratio of 67.3%. This is mainly because SVO is the word order of most lexifiers, but it can also be attributed to data representation: since WALS did not allow multi-valued features (e.g., SVO _and_ SOV), some creoles with multiple word orders were mapped to a separate category "No dominant order," underestimating the influence of SVO.

\begin{table}
\begin{tabular}{c|l|l}
**Source** & **Ratio** & **Feature type** \\ \hline  & 100.0\% & Order of Adposition and Noun Phrase \\  & 100.0\% & Order of Relative Clause and Noun \\
**Lexifier** & 99.9\% & Applicative Constructions \\  & 99.5\% & The Prohibitive \\  & 98.2\% & Alignment of Case Marking of Full Noun Phrases \\ \hline  & 84.9\% & Order of Genitive and Noun \\  & 56.6\% & Tone \\
**Substrate** & 54.9\% & Order of Subject, Object and Verb \\  & 26.5\% & Pronominal and Andnominal Demonstratives \\  & 24.0\% & Relativization on Subjects \\ \hline  & 100.0\% & Intensifiers and Reflexive Pronouns \\  & 100.0\% & Numerical Classifiers \\
**Restructurer** & 100.0\% & Suppletion According to Tense and Aspect \\  & 100.0\% & Expression of Pronominal Subjects \\  & 100.0\% & Polar Questions \\ \hline \end{tabular}
\end{table}
Table 3: Top-5 feature types for each source according to per-feature factors of Fact. The arithmetic mean of 50 samples after 5,000 iterations, with an interval of 100 iterations.

 



### Discussion

The main contribution of our work is the introduction of mixture models to creole studies. This is, however, only the first step toward understanding the complex process of creole genesis by means of statistical modeling. Better data are needed with respect to proxies for substrates, missing values, multi-valued features among others.

With better data, more elaborate models could uncover the detailed process of creole genesis. Our models mix several sources in one step, but we may want to model the staged development of pidgin formation and creole formation. As a result of continued influence from its superstrate, a creole might undergo _decreolization_. It is argued that pidgins themselves have several development stages, from each of which creoles can emerge [14]. Hopefully, these hypotheses could be tested with statistical models.

Our finding that the restructurer plays a dominant role in creole genesis has a negative implication for tree-based inference of language relationships. If most features of a language come from nowhere, we are unable to trace its origin back into the deep past. In the meanwhile, it has been argued that creole genesis only occurred in modern and early-modern, exceptional circumstances and cannot be responsible for most historical changes. Thus identifying the social conditions under which creoles arise [23] is another research direction to be explored.

## 6 Conclusion

In this paper, we present several statistical models of linguistic typology to answer questions concerning creole genesis. First, we formalized creep (non-)distinctiveness as a binary classification problem. Second, we propose to model creole genesis with mixture models, which makes more sense than tree-building techniques.

Recent studies on linguistic applications of computational phylogeny have been heavily influenced from computational biology. They often depend on ready-to-use software packages developed in that field. We observe that, as a result, linguistic phenomena that lack exact counterparts in biology tend to be left untouched. In this paper, we have hopefully demonstrated that computational linguists could fill the gap.

## Acknowledgment

This work was partly supported by JSPS KAKENHI Grant Number 26730122.

## References

* Akiba-Reynolds (1984) Katsue Akiba-Reynolds. 1984. Internal reconstruction in pre-Japanese syntax. In Jacek Fisiak, editor, _Historical Syntax_, pages 1-23. Walter de Gruyter.
* Alexander et al. (2009) David H. Alexander, John Novembre, and Kenneth Lange. 2009. Fast model-based estimation of ancestry in unrelated individuals. _Genome Research_, 19(9):1655-1664.
* Almahri et al. (2012)

\begin{table}
\begin{tabular}{c|l|p{142.3pt}}
**Ratio** & \multicolumn{1}{c|}{**Feature type**} & \multicolumn{1}{c}{**Feature value**} \\ \hline
91.2\% & Numerical Classifiers & Absent \\
74.3\% & Gender Distinctions in Independent Personal Pronouns & No gender distinctions \\
72.3\% & Negative Indefinite Pronouns and Predicate Negation & Predicate negation also present \\
70.5\% & Occurrence of Nominal Plurality & All nouns, always optional \\
69.7\% & Intensifiers and Reflexive Pronouns & Identical \\
68.4\% & Distributive Numerals & No distributive numerals \\
67.2\% & Expression of Pronominal Subjects & Obligatory pronouns in subject position \\
66.9\% & Politeness Distinctions in Pronouns & No politeness distinction \\
66.6\% & Alignment of Case Marking of Pronouns & Nominative - accusative (standard) \\
66.3\% & Order of Numeral and Noun & Numeral-Noun \\ \end{tabular}
\end{table}
Table 4: Top-10 features derived from the restructurer in Fact. The ratio of the feature type-value pair \((j,l)\) is defined as \(|\{(i\,|\,x_{i,j}=l,z_{i,j}=\mathrm{R}\}|~{}/~{}N\). The arithmetic mean of 50 samples after 5,000 iterations, with an interval of 100 iterations.



Charles J. Bailey and Karl Maroldt. 1977. The French lineage of English. In Jurgen M. Meisel, editor, _Languages en contact - Pidgins - Creoles_, pages 21-53. Narr.
* Bakker et al. (2011) Peter Bakker, Aymeric Daval-Markussen, Mikael Parkvall, and Ingo Plag. 2011. Creoles are typologically distinct from non-creoles. _Journal of Pidgin and Creole Languages_, 26(1):5-42.
* Blei et al. (2003) David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent Dirichlet allocation. _Journal of Machine Learning Research_, 3:993-1022.
* Bouckaert et al. (2012) Remco Bouckaert, Philippe Lemey, Michael Dunn, Simon J. Greenhill, Alexander V. Alekseyenko, Alexei J. Drummond, Russell D. Gray, Marc A. Suchard, and Quentin D. Atkinson. 2012. Mapping the origins and expansion of the Indo-European language family. _Science_, 337(6097):957-960.
* Brownie (2012) John Brownie. 2012. Multilingualism and identity on Mussau. _International Journal of the Sociology of Language_, 2012(214).
* Bryant and Moulton (2004) David Bryant and Vincent Moulton. 2004. NeighborhoodNet: An agglomerative method for the construction of phylogenetic networks. _Molecular Biology and Evolution_, 21(2):255-265.
* Campbell (2006) Lyle Campbell. 2006. Areal linguistics. In _Encyclopedia of Language and Equistics, Second Edition_, pages 454-460. Elsevier.
* Daume III (2009) Hal Daume III. 2009. Non-parametric Bayesian areal linguistics. In _HLT-NAACL_, pages 593-601.
* Dayal-Markussen and Bakker (2012) Aymeric Dayal-Markussen and Peter Bakker. 2012. Explorations in creole research with phylogenetic tools. In _Proc. of LINGVIS & UNCLH_, pages 89-97.
* Dayal-Markussen (2013) Aymeric Dayal-Markussen. 2013. First steps towards a typological profile of creoles. _Acta Linguistica Hafniensia_, 45(2):274-295.
* Dunn et al. (2005) Michael Dunn, Angela Terrill, Ger Reesink, Robert A. Foley, and Stephen C. Levinson. 2005. Structural phylogenetics and the reconstruction of ancient language history. _Science_, 309(5743):2072-2075.
* Eisenstein et al. (2011) Jacob Eisenstein, Amr Ahmed, and Eric P. Xing. 2011. Sparse additive generative models of text. In _Proc. of ICML_, pages 1041-1048.
* Gray and Atkinson (2003) Russell D. Gray and Quentin D. Atkinson. 2003. Language-tree divergence times support the Anatolian theory of Indo-European origin. _Nature_, 426(6965):435-439.
* Griffiths and Steyvers (2004) Thomas L. Griffiths and Mark Steyvers. 2004. Finding scientific topics. _PNAS_, 101:5228-5235.
* Haspelmath et al. (2005) Martin Haspelmath, Matthew Dryer, David Gil, and Bernard Comrie, editors. 2005. _The World Atlas of Language Structures_. Oxford University Press.
* Jones et al. (2012) Eppie R. Jones, Gloria Gonzalez-Fortes, Sarah Connell, Veronika Siska, Anders Eriksson, Rui Martiniiano, Russell L. McLaughlin, Marcos Gallego Llorente, Lara M. Cassidy, Cristina Gamba, Tengiz Meshveliani, Ofer Bar-Yosef, Werner Muller, Anna Belfer-Cohen, Zinovi Matskevich, Nino Jakeli, Thomas F. G. Higham, Mathias Currat, David Lordkipanidze, Michael Hofreiter, Andrea Manica, Ron Pinhasi, and Daniel G. Bradley. 2015. Upper Palaeolithic genomes reveal deep roots of modern Eurasians. _Nature Communications_, 6.
* Josse et al. (2012) Julie Josse, Marie Chavent, Benot Liquet, and Francois Husson. 2012. Handling missing values with regularized iterative multiple correspondence analysis. _Journal of Classification_, 29(1):91-116.
* Kawamoto (1974) Takao Kawamoto. 1974. Agreements and disagreements in morphology between Japanese and Austronesian (chiefly Melanesian) languages. _The Japanese Journal of Ethmology_, 39(2):113-129. (in Japanese).
* Kawamoto (1990) Takao Kawamoto. 1990. Pijin kureoru-ka to Nihong no keisei [Pidginian-creolization and the formation of Japanese]. In Osamuaki Sakiyama, editor, _Nihong no Keisei [Formation of Japanese]_, pages 130-168. Sanseido. (in Japanese).
* Longobardi and Guardiano (2009) Giuseppe Longobardi and Cristina Guardiano. 2009. Evidence for syntax as a signal of historical relatedness. _Lingua_, 119(11):1679-1706.
* Matras (2011) Yaron Matras. 2011. Universals of structural borrowing. In Peter Siemund, editor, _Linguistic Universals and Language Variation_, pages 204-233. Walter de Gruyter.
* Maurits and Griffiths (2014) Luke Maurits and Thomas L. Griffiths. 2014. Tracing the roots of syntax with Bayesian phylogenetics. _PNAS_, 111(37):13576-13581.
* Michaelis et al. (2013) Susanne Maria Michaelis, Philippe Maurer, Martin Haspelmath, and Magnus Huber, editors. 2013. _APiCS Online_. Max Planck Institute for Evolutionary Anthropology.
* Muhlhausler (1997) Peter Muhlhausler. 1997. _Pidgin and Creole Linguistics: Expanded and revised Edition_. University of Westminster Press.
* Murawaki (2015) Yugo Murawaki. 2015. Continuous space representations of linguistic typology and their application to phylogenetic inference. In _Proc. of NAACL-HLT_, pages 324-334.
* Neal (2003) Radford M. Neal. 2003. Slice sampling. _Annals of Statistics_, 31(3):705-767.
* Patterson et al. (2006) Nick Patterson, Alkes L. Price, and David Reich. 2006. Population structure and eigenanalysis. _PLoS Genetics_, 2(12):e190, 12.
* Patterson et al. (2012) Nick Patterson, Priya Moorjani, Yuntao Luo, Swapan Mallick, Nadin Rohland, Yiping Zhan, Teri Genschoreck, Teresa Webster, and David Reich. 2012. Ancient admixture in human history. _Genetics_, 192(3):1065-1093 

Jonathan K. Pritchard, Matthew Stephens, and Peter Donnelly. 2000. Inference of population structure using multilocus genotype data. _Genetics_, 155(2):945-959.
* Reesink et al. [2009] Ger Reesink, Ruth Singer, and Michael Dunn. 2009. Explaining the linguistic diversity of Sahul using population models. _PLoS Biology_, 7(11).
* Saitou and Nei [1987] Naruya Saitou and Masatoshi Nei. 1987. The neighborjoining method: A new method for reconstructing phylogenetic trees. _Molecular Biology and Evolution_, 4(4):406-425.
* Schleicher [1853] August Schleicher. 1853. Die ersten Spaltungen des indogermanischen Urvolkes. _Allgemeine Monatsschrift fur Wissenschaft und Literatur_, 3:786-787. (in German).
* Schmidt [1872] Johannes Schmidt. 1872. _Die Verwandtschaftsverhaltnisse der inagdermanischen Sprachen_. Hermann Bohlau. (in German).
* Teh et al. [2008] Yee Whye Teh, Hal Daume III, and Daniel Roy. 2008. Bayesian agglomerative clustering with coalescents. In _NIPS_, pages 1473-1480.
* Tria et al. [2015] Francesca Tria, Vito D.P. Servedio, Salikoko S. Mufwene, and Vittorio Loreto. 2015. Modeling the emergence of contact languages. _PLoS ONE_, 10(4):e0120771, 04.
* Trudgill [1974] Peter Trudgill. 1974. Linguistic change and diffusion: Description and explanation in sociolinguistic dialect geography. _Language in Society_, 3:215-246.
* Tsunoda et al. [1995] Tasaku Tsunoda, Sumie Ueda, and Yoshiaki Itoh. 1995. Adpositions in word-order typology. _Linguistics_, 33(4):741-762.
* Wardhaugh and Fuller [2015] Ronald Wardhaugh and Janet M. Fuller. 2015. _An Introduction to Sociolinguistics, 7th Edition_. John Wiley & Sons.