言語処理学会 第30回年次大会 発表論文集（2024年3月）

Finding structure in spelling and pronunciation using Latent
Dirichlet Allocation
Kow Kuroda
Medical School, Kyorin University

Abstract
Latent Dirichet Allocation (LDA) was used to reveal
hidden regularities in spelling and pronunciation among
languages, such as English, French, German, Russian andSwahili. Words, either in spell or pronunciation, wereregarded as “documents” and their characters as “terms.”
For spelling, spells from the languages were mixed andencoded using LDA and analyzed. For pronunciation, IPA
symbols for English, French and German were encoded andanalyzed in the same way. Both gave promising results.

1Introduction

Languages diﬀer. This is why most people have hardtime in learning any of them. But at the same time, theirdiﬀerences are a matter of degree. Given a language, say
English, some language such as French and German areless diﬀerent from it than others such as Japanese and Russian. Languages employ similar sounds or similar spellings
are clearly more alike. What this research aims at, ultimately, is to quantify such diﬀerences among languages
using Latent Dirichlet Allocation (LDA) [4, 5].

2Analysis

English spelling and pronunciation data were part of thedata used in Kuroda [1] based on CMU Pronouncing Dictionary1) and other sources. For spelling analysis, roughly
817 spells were randomly sampled from 4,279 and usedin each run. For pronunciation, roughly 1,000 forms wererandomly sampled from 4,199 and used in each run.
For languages other than English, spelling data were obtained from “1000 most common words” site2) or pronunciation data from open-dict-data3) . For French, 1,000spells and 900 sounds were used. For German, 793 spells
1) http://www.speech.cs.cmu.edu/cgi-bin/cmudict2) https://1000mostcommonwords.com/3) https://open-dict-data.github.io/and 798 sounds were used. For Russian, 1,000 spells wereused. For Swahili, 708 spells were used.
On analysis of spelling, words of English, French, German, Russian and Swahili were treated as “documents” andtheir orthographic characters as “terms.” For terms, bothcharacter 𝑛-grams and 𝑘-skip 𝑛-grams [2] were used (𝑘 wasset to the 0.8 of the longest size of the document), but 𝑛grams are inclusive in that 𝑛-grams contain (𝑛 − 1)-grams.
Under this, a document-term matrix (DTM) was constructed, ﬁltered by minimum frequency = 2 and abuse
threshold = 0.05, and fed to LDA with varying numbersof topics. To look for eﬀective values, LDA tuning4) wasused. Then, Latent Dirichlet Allocation (LDA) [4, 5] wasapplied using gensim package (v4.2.x)5) . Through this, allwords under analysis were assigned encodings via LDA.
Encodings thus obtained were visualized by t-SNE [3].
For this, scikit-learn package6) was used. The reason
for this choice is that grouping of items should be harder tounderstand with hard clustering such as hierarchical clustering than with soft clustering. For reference, results ofhierarchical clustering are presented in Appendix 6.
On analysis of pronunciation, strings of IPA symbols ofEnglish, French and German were analyzed in the sameway.
All data and codes used in this paper are open and available at a GitHub repository7) : mainly, Jupyter Notebooks
that implemented the analysis and the relevant data.

3

Results: Spelling

3.1

Spelling: #topics = 5

Figures 1–3 show 2D views on t-SNE 3D map (perplexity = 5) for cases for 1-, 3-gram and skippy 2-gram where4)
5)
6)
7)

― 149 ―

https://github.com/nikita-moor/ldatuning
https://radimrehurek.com/gensim/
https://scikit-learn.org/
https://github.com/kow-k/LDA-spell-sound-typology

This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

#topics = 5.

Figure 1

Figure 2

Figure 3

t-SNE 3D [#topics: 5; 1-gram; perplexity: 5]
Figure 4

t-SNE 3D [#topics: 5; 1-gram; perplexity: 335]

Figure 5

t-SNE 3D [#topics: 5; 3-gram; perplexity: 335]

t-SNE 3D [#topics: 5; 3-gram; perplexity: 5]

t-SNE 3D [#topics: 5; skippy 2-gram; perplexity: 5]

Figures 4–6 show 2D views on t-SNE 3D map (perplexity = 335) for cases where #topics = 5.

3.2

Figure 6

t-SNE 3D [#topics: 5; skippy 2-gram; perplexity:335]

Spelling: #topics = 15

Figures 7–9 show 2D views on t-SNE 3D map (perplexity = 5) for cases for 1-, 3-gram and skippy 2-gram where#topics = 15.
Figures 10–12 show 2D views on t-SNE 3D map (perplexity = 335) for cases for 1-gram, 3-gram and skippy
2-gram where #topics = 15.

3.3

Discussion

With smaller number of topics such as 5, clusteringresults do not diﬀer qualitatively for smaller perplexity

Figure 7

t-SNE 3D [#topics: 15; 1-gram; perplexity: 5]

Figure 8

t-SNE 3D [#topics: 15; 3-gram; perplexity: 5]

values such as 5, yet they diﬀer for larger values such as335. It is hard to say which of Figures 1–3 is the best,
but diﬀerences among the target languages seems to be
successfully captured. What is common is that Russianwords are isolated, which is predictable from the fact thatit employs diﬀerent set of characters. This is even captured
in Figures 2 and 3 in which most of Russian words are
somehow localized.
The localization of Swahili words is not as clear as that ofRussian words, but it is observable, especially in Figure 4.
What makes Figure 6 diﬀerent is that it exaggerates theFigure 9t-SNE 3D [#topics: 15; skippy 2-gram; perplexity: 5]

dissimilarity of Russian words.
With larger number of topics such as 15, results do not― 150 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Figure 10

t-SNE 3D [#topics: 15; 1-gram; perplexity: 335]

Figure 11

t-SNE 3D [#topics: 15; 3-gram; perplexity: 335]

Figure 12t-SNE 3D [#topics: 15; skippy 2-gram; perplexity:

Figure 13

t-SNE 3D [#topics: 5; 1-gram; perplexity: 5]

Figure 14

t-SNE 3D [#topics: 5; 3-gram; perplexity: 5]

Figure 15

t-SNE 3D [#topics: 5; skippy 2-gram; perplexity:5]335]

diﬀer for smaller perplexity such as 5. This the same as
the cases with #topic = 5. But the eﬀectiveness of 1-grambased LDA seems to be improved. Comparison among
Figures 10, 11 and 12
In sum, data compression is more eﬀective with smaller
number of topics but this gives less room for representations, thereby less encoding less ﬂexible.

4Results 2: Pronunciation

Figure 16

t-SNE 3D [#topics: 5; 1-gram; perplexity: 335]

Figure 17

t-SNE 3D [#topics: 5; 3-gram; perplexity: 335]

IPA encodings of English, French and German words
are analyzed using LDA.

4.1

Pronunciation: #topics = 5

Figures 13–15 show 2D views on t-SNE 3D map (perplexity = 5) for cases for 1-, 3-gram and skippy 2-gramwith #topics = 5.
Figures 16–18 show 2D views on t-SNE 3D map (perplexity = 5) for cases with #topics = 335.

4.2

Pronunciation: #topics = 15

Figures 19–21 show 2D views on t-SNE 3D map (perplexity = 5) for cases for 1-, 3-gram and skippy 2-gram
Figure 18

with #topics = 15.

t-SNE 3D [#topics: 5; skippy 2-gram; perplexity:335]

Figures 22–18 show 2D views on t-SNE 3D map (perplexity = 335) for cases for 1-, 3-gram and skippy 2-gram― 151 ―

This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Figure 19

t-SNE 3D [#topics: 15; 1-gram; perplexity: 5]

Figure 22

t-SNE 3D [#topics: 15; 1-gram; perplexity: 335]

Figure 20

t-SNE 3D [#topics: 15; 3-gram; perplexity: 5]

Figure 23

t-SNE 3D [#topics: 15; 3-gram; perplexity: 335]

t-SNE 3D [#topics: 15; skippy 2-gram; perplexity:

Figure 24

t-SNE 3D [#topics: 15; skippy 2-gram; perplexity:

Figure 215]

335]

analysis. This is why connection among data points lookwith #topics = 15.

4.3

smoother in skip 2-gram and 3-gram settings.
Under this remark, though, it is not unreasonable toDiscussion

observe that Figure 24 gradually and most eﬀectively lo-

Let us examine cases with #topics = 5 ﬁrst. With smaller
perplexity values such as 5, clustering results do not diﬀer
qualitatively, yet they diﬀer with larger values such as 335,
as shown by the contrast among Figures 13, 14 and 15.

calizes the distribution of Germanic and Romance soundpatterns.

5Conclusions

With larger perplexity values such as 335 clustering re-LDA is applied to encode spellings and pronunciations

sults diﬀer qualitatively, and dimension reduction looksof a few languages and the encodings thus obtained weresuccessful. This is indicted by Figures 16, 17 and 18. The
diﬀerences among 1-gram, 3-gram and skippy 2-gram arecompared. English, French, German, Russian and Swahili
were compared for spelling; English, French and Ger-

basically smoothness of encoding: 3-gram and skip 2-gram

man for pronunciation. Though the results at hand are

based LDA capture richer structures than 1-gram base one.

rather preliminary, they still suggest that this unsupervised

It is hard to tell which of 3-gram or skippy 2-gram gives
better results, but it is clear that skippy 2-gram based en-

method successfully capture both similarities and dissimi-

coding can capture more complex structures.

ing hidden regularities shared among them.

larities among a set of languages examined, thereby reveal-

Let us turn to cases with #topics = 15. Like cases with

The virtue of this method is that we can make target data

#topics = 5, clustering results do not diﬀer qualitatively

as large as we want, because data preparation is virtuallywith smaller perplexity values such as 5. This is showncost-free. This research has targeted English, French, Ger-by the contrast among Figures 19, 20 and 21. Diﬀerencesman, Russian, and Swahili, only because the submission

arise with larger perplexity values such as 335, but this time

deadline did not allow to add more languages.

with Figures 22, 23 and 24, unlike cases with #topics = 5,

In this paper, I deliberately forbid myself to explicate im-

it is harder to tell how they diﬀer. This is probably because

portant insights gained from the obtained results, judging

more number of topics give more degrees of similarity, or

this is not the right place to do so.

rather precisely more connectivities, to data points under― 152 ―

This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Acknowledgements
To run LDA tuning, R (https://www.R-project.
org/) version 4.3.x, developed by R Core Team, was used.
To run t-SNE, the manifold module in scikit-learn(https://scikit-learn.org/) was used.

For otherdata analysis and visualizations, Anaconda 3 (https://www.anaconda.com) version 23.11.00 was used, running Jupyter Notebook 6.5.4 on Python 3.10.

References[1] 黒田航. 英単語の綴りと発音のズレの定量的評価. 認
知科学会第 40 回大会発表論文集, pp. 644–647, 2023.
[2] D. Gutherie, B. Allison, W. Liu, L. Gutherie, and Y. Wilks.
A closer look at skip-gram modeling. In Proceedingsof the 5th International Conference on Language Resources and Evaluation, pp. 1–4, 2006.

[3] Laurens van der Maaten and Geoﬀrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, Vol. 9, pp. 2579–2605, 2008.
[4] David M. Blei, Andrew Y. Ng, and Michael I. Jordan. Latent Dirichlet allocation. Journal of Machine Learning
Research, Vol. 3, pp. 993–1022, 2003.
[5] 岩田具治. トピックモデル. 講談社, 2015.

― 153 ―

This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

6Appendix

Hierarchical clusterings of subsampled
spells and sounds
Hierarchical clustering of 140 spells (subsamples) isshown in Figure 25. Leaf colors correspond to the colors assigned to the languages in Figures 1–12.

Figure 25

Hierarchical clustering of 140 sample spells encodedvia skippy 2gram-based LDA (#topic: 15)

Hierarchical clustering of 166 sounds (subsamples) isshown in Figure 26. Leaf colors correspond to the colors
assigned to the languages in Figures 13–24.

Figure 26

Hierarchical clustering of 166 sample sounds encoded via skippy 2gram-based LDA (#topic: 15)― 154 ―

This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).