SlowGuess/ACL_2020_OCR huggingface dataset:
Table 3: Prediction accuracies for varying experimental setups and thresholds  $l_{a}$ ,  $l_{r}$ . All: results for model trained on all predictors;  $|\overline{F}|$ : family size only;  $f_{r}$ : relative token frequency of parent only;  $\bar{z}^{(p)}$ : trending behavior of parent only;  $D_{w}^{U}$ : user dissemination only;  $D_{w}^{T}$ : thread dissemination only;  $Q_{w}$ : topicality only.  $\tilde{F}_{n}$ : results for model trained on new children only. 12 m.: results for model trained on context intervals of 12 months; 24 m.: 24 months. Best accuracies per column are highlighted in gray, second-best accuracies in light gray.
la(δa)	μ±σ	lr(δr)	μ±σ
0.0	0.8	1.6	2.4	3.2	4.0	4.8	1.0	1.2	1.4	1.6	1.8	2.0	2.2
|F|	.202	.296	.344	.393	.458	.495	.489	.382±.101	.202	.208	.223	.253	.286	.261	.204	.234±.031
fr	.153	.145	.140	.152	.127	.128	.164	.144±.013	.153	.153	.148	.144	.129	.122	.125	.139±.012
z(p)	.168	.150	.149	.145	.154	.136	.138	.149±.010	.168	.172	.178	.179	.195	.218	.264	.196±.032
DuW	.156	.134	.119	.098	.084	.070	.058	.103±.033	.156	.156	.149	.136	.130	.117	.096	.134±.021
DTw	.162	.139	.124	.105	.081	.090	.067	.110±.031	.162	.158	.150	.145	.132	.147	.145	.148±.009
Qw	.158	.137	.125	.108	.096	.081	.083	.113±.027	.158	.155	.152	.143	.128	.135	.167	.148±.013
Table 4: Importance loadings for individual features. Features as in Table 3. Importance loadings are calculated for different values of the thresholds  $l_{a}$  and  $l_{r}$ . Highest importance loadings for features are highlighted in gray, second-highest in light gray.
The strongest predictor for both models is type frequency with a feature importance of  $39.3\%$  and  $25.3\%$ , respectively (Table 4). Models trained only on this feature already achieve accuracies of  $74.2\%$  and  $64.5\%$ , respectively (Table 3). However, the effect of  $\lceil F\rceil$  is reversed: while larger morphological families have higher absolute growth values, which is in accordance with theories of lexical growth based on preferential attachment (Steyvers and Tenenbaum, 2005), smaller morphological families have higher relative growth rates. This can be explained by the observation that a large family needs much higher increases in family size to have the same relative growth rate as a small family. The fact that larger families are generally more likely to grow does not seem to counteract this imbalance.
We now analyze how different design choices impact the performance of our model.
Thresholds  $l_{a}$  and  $l_{r}$ . We systematically vary  $l_{a}$  and  $l_{r}$  in the range  $0.0 \leq l_{a} \leq 4.8$  and  $1.0 \leq l_{r} \leq 2.2$  to examine their influence on per
formance. $^{7}$  We find that accuracies for predicting larger  $\delta_{a}$  and  $\delta_{r}$  are considerably higher than for smaller increases (Table 3). For  $l_{a} = 4.8$  (i.e., the family size increases by more than 4.8 members on average), the model has an error rate of  $15.4\%$ , which is less than half compared to the error rate of  $38.2\%$  for  $l_{a} = 0.0$ . This striking result is in line with studies on the predictability of extreme events in social media (Miotto and Altmann, 2014) and statistical physics (Hallerberg and Kantz, 2008) showing that extreme events are generally better predictable than non-extreme events.
We then train single-feature models for varying  $l_{a}$  and  $l_{r}$ . The best predictor for all values of  $l_{a}$  and  $l_{r}$  is  $\overline{|F|}$  (Table 3). The overall second-best predictor is  $\bar{z}^{(p)}$ , even though it is (sometimes only marginally) outperformed by  $f_{r}$  and  $D_w^T$  on several values of  $l_{a}$  and  $l_{r}$ .
To further analyze the relative importance of individual features, we examine the RF feature
The number of positive examples with  $\delta_{a} > 4.8$  and  $\delta_r > 2.2$  is too small for achieving reliable results.
7278
Figure 2: Feature importances of MFEP RF models  $(\delta_{a})$  for varying the absolute growth threshold  $l_{a}$ . The predictors are grouped into the classes type frequency-based  $(|F|)$ , token frequency-based  $(f_{r},\bar{z}^{(p)})$ , and dissemination-based  $(D_w^U,D_w^T,Q_w)$ .
Figure 3: Feature importances of MFEP RF models  $(\delta_r)$  for varying the relative growth threshold  $l_{r}$ . The predictors are grouped into the classes type frequency-based  $(|\overline{F}|)$ , token frequency-based  $(f_{r},\bar{z}^{(p)})$ , and dissemination-based  $(D_w^U,D_w^T,Q_w)$ .
importance loadings for varying  $l_{a}$  and  $l_{r}$  (Table 4, Figure 2, Figure 3). While  $|\overline{F}|$  is again the best predictor overall,  $\bar{z}^{(p)}$  much more clearly comes out as the second-best predictor: especially for  $\delta_{r}$ , it steadily increases with higher values for  $l_{r}$  and even surpasses  $|\overline{F}|$  for  $l_{r} = 2.2$ .
These results indicate that while the family size is most predictive of morphological family growth in general, high growth rates are particularly likely for families of trending parents (most of which have initially small family sizes). An example for the second case is the burst in the "trump" family before the 2016 presidential election illustrated in Section 1 (Figure 1, Table 1). This would explain how small morphological families can grow in the first place given the overall dominating importance of a large family size: small families need exogenous forcing (Altmann et al., 2011), i.e., external events leading to a burst in token frequency and a subsequent increase in type frequency. In order to test this hypothesis, we retrain the model for  $\delta_{a}$  on small families  $(1.5\leq \overline{|F|}\leq 1.6)$ , varying
la	μ±σ
0.0	0.2	0.4	0.6	0.8	1.0
fr	.175	.187	.196	.193	.165	.190	.184±.011
z(p)	.222	.216	.223	.253	.269	.234	.236±.019
DUw	.183	.198	.190	.181	.179	.190	.187±.006
DTw	.212	.196	.201	.192	.195	.233	.205±.014
Qw	.208	.202	.190	.181	.192	.153	.188±.018
Table 5: Importance loadings for individual features with small families  $(1.5 \leq |F| \leq 1.6)$ . Features as in Table 3. Importance loadings are calculated for different values of the thresholds  $l_{a}$ . Highest importance loadings for features are highlighted in gray, second-highest in light gray.
$0.0 \leq l_{a} \leq 1$  (which is the range of  $\delta_{a}$  for families of that size);  $\bar{z}^{(p)}$  has the highest feature importance for all values of  $l_{a}$  (Table 5).<sup>8</sup>
Furthermore, it is interesting to note that the frequency-based as well as the dissemination-based measures are considerably clumped together in feature importance space for absolute growth  $\delta_{a}$ , with the frequency-based predictors topping the dissemination-based ones. This is in line with recent work on the relative importance of frequency and social dissemination in lexical change (Stewart and Eisenstein, 2018). Higher values in these features correlate with a higher likelihood of growth, except for topicality: here, growth is more likely with lower topicality, which as discussed above indicates higher topical dissemination.
Length of  $i^{(c)}$  and  $i^{(p)}$ . In previous experiments, the length of  $i^{(c)}$  and  $i^{(p)}$  was set to 18 and 6 months, respectively. We now analyze how the choice of the interval length, specifically of the length of  $i^{(c)}$ , influences the performance of our MFEP model. We retrain the model for  $0.0 \leq l_a \leq 4.8$  and  $1.0 \leq l_r \leq 2.2$  with  $|i^{(c)}| = 12$  and  $|i^{(c)}| = 24$ , i.e., the context intervals are six months shorter and longer than previously. The length of the probe interval is kept unchanged,  $|i^{(p)}| = 6$ .
The performances of the two MFEP models is comparable with  $|i^{(c)}| = 18$  (Table 3). Both show top performance at  $l_{a} = 4.8$  and  $l_{r} = 2.2$ . However, it is interesting to note that the performance with  $|i^{(c)}| = 12$  tends to be better than  $|i^{(c)}| = 24$  with large values for  $l_{a}$  and  $l_{r}$ , but worse with smaller values, suggesting that shorter context intervals have an advantage in predicting large increases in family size while longer context intervals have an advantage in predicting smaller increases.
Even smaller families do not allow to vary  $l_{a}$  on such a large range due to data sparsity, but show similar results.
7279
New children. In our main study design, growth in the families is not necessarily due to new children being added to the family, i.e., due to an increase of  $|\tilde{F}_n|$ . A rare but established English word  $w \in \tilde{F}_o$  that only occurs a couple of times in the data counts as much to the growth as an innovative form. Here, we try to exclude fluctuations due to  $\tilde{F}_o$  by excluding all words in the data that are listed on a comprehensive list of English words encompassing over 400,000 word types, an independent estimate of established words. Training the model on the resulting data, we find that accuracies tend be higher for  $\delta_a^{10}$  but lower for  $\delta_r$  than corresponding accuracies on the full dataset (Table 3). This result indicates that our model is not only capable of forecasting the evolution of the entire family but also of predicting the birth of new morphological children.
Error analysis. The segmentation algorithm is doomed to produce a certain number of false positives. To get a clearer picture of its accuracy, we manually examine 500 randomly selected families from one month in the data. Macro-averaged over families,  $8.8\%$  of the words are errors, i.e., they do not belong to the morphological family assigned by the algorithm. However, the error rate is not distributed evenly: only 10 of the 500 families are responsible for more than  $60\%$  of the errors.
One frequent source of erroneous segmentations is incorrect orthography. The word "representatives", e.g., is frequently written as "representatives" due to its being pronounced without the consonant "t". The algorithm then segments "representatives" into "re+pre+senate+ive+s" and adds it to  $\tilde{F} ($  "senate"). Another frequent case is the erroneous segmentation of emphatical repetitions of vowels, e.g., "heyy" is segmented as "hey+y" and added to  $\tilde{F} ($  "hey"). Such false positives are a major source of distortion in the data.
Morphological families and productivity. The concept of morphological families was introduced in psycholinguistic work on lexical processing (Schreuder and Baayen, 1997; de Jong et al., 2000; del Prado Martin et al., 2004). These studies show that response latencies in lexical decision are not only influenced by token frequency but also by type
frequency, i.e., the size of their morphological family. Morphological families have also been used for analyzing lexical change on historical time scales (Keller and Schultz, 2013, 2014). However, this work is not comparable to our study since it relies on dictionaries, which typically exclude the transparently formed, non-entrenched words in  $\tilde{F}_n$  we are interested in (Bauer, 2001).
The main question of our study (how can we predict the growth of morphological families?) is related to a long-standing problem in traditional linguistic scholarship, i.e., what factors influence morphological productivity. The productivity of a morpheme is defined as its propensity to be used in novel combinations and traditionally understood to refer to bound morphemes (Haspelmath and Sims, 2010). Pierrehumbert and Granell (2018) highlight the fact that morphological productivity, just as morphology (Rácz et al., 2015) and other components of language (Labov, 1963) in general, is heavily influenced by social variation. Social groups differ in the morphological patterns they use and in the extent to which they extend these patterns to new words. This makes morphological productivity an exciting new area for future research in computational social science, and it further underscores the relevance of MFEP for that field.
Derivational morphology in NLP. Derivational morphology has received increasing attention in NLP recently. Key challenges include segmenting (Cotterell et al., 2016; Luo et al., 2017; Cotterell and Schütze, 2018), modeling the meaning (Lazaridou et al., 2013; Kisselew et al., 2015; Padó et al., 2016; Cotterell and Schütze, 2018), and predicting the form (Vylomova et al., 2017; Cotterell et al., 2017; Deutsch et al., 2018) as well as morphological well-formedness (Hofmann et al., 2020) of derivatives. Whereas all these studies approach derivational morphology from a synchronous standpoint, MFEP is to the best of our knowledge the first computational task that addresses diachronic aspects of derivation.
Lexical change in social media. Language change (Croft, 2000; Bybee, 2015) is most visible on the lexical level. New words like "detrumpify" attract attention, often becoming the subject of public discourse (Metcalf, 2002). Since innovations are taking place at a much faster rate on internet media (Crystal, 2004), social media have become a central resource for studies on lexical change over the last decade (Altmann et al., 2011; Garley
The list is available on https://github.com/dwyl/english-words. We only trained the model for  $0 \leq l_{a} \leq 3.2$  since there was not enough data for larger threshold values.
7280
and Hockenmaier, 2012; Danescu-Niculescu-Mizil et al., 2013; Grieve et al., 2016; Kershaw et al., 2016; Sang, 2016; Stewart and Eisenstein, 2018; del Tredici and Fernandez, 2018).
One central question in this field is: what factors determine whether a word will survive in the lexicon of an online community? Usage frequency is a well-known factor that influences the evolution of a word at historical time scales (Pagel et al., 2007). Studies on lexical change in online groups have shown that this is also true for shorter time scales (Altmann et al., 2011; Stewart and Eisenstein, 2018). Another main factor is the dissemination of a word, i.e., how widely a word is spread across different social and linguistic contexts. Generally, the more disseminated a word is, the more likely it is to grow. This holds for social dissemination across users and threads (Altmann et al., 2011) as well as linguistic dissemination across different lexical collocations (Stewart and Eisenstein, 2018).
The studies mentioned so far focus on token frequency. An exciting new approach looks instead at the meaning of words using diachronic word embeddings (Hamilton et al., 2016). del Tredici et al. (2019), e.g., explore short-term meaning shifts on Reddit and identify considerable changes even within a period of eight years.
A main goal of this study is to add a third approach to studies on lexical change in social media besides word frequency and word embeddings: word families. From a linguistic point of view, these three approaches can be viewed to be complementary: whereas word frequency is context-independent, both word embeddings and word families reflect context-sensitive measures. However, while word embeddings reflect proximity on the utterance level (which words are close to each other in spoken sentences?), word families reflect proximity on the system level (which words are close to each other in the mental lexicon?).
In this paper, we have proposed MFEP (Morphological Family Expansion Prediction), a new task that aims at predicting how morphological families evolve over time. We have shown that changes in morphological family size provide a fresh look at topical dynamics in social media, thus complementing token frequency as a metric.
Furthermore, we have presented a random forest model for MFEP that achieves very good accuracy
cies, particularly in predicting extreme growth in morphological family size. The strongest predictor of growth is the morphological family size itself, an endogenous factor. However, the initial growth of small families is mainly driven by the trending behavior of the parent, an exogenous factor. This reflection of external events makes morphological families a promising tools for various fields drawing upon NLP techniques for tracing temporal dynamics in text (e.g., virality detection).
Overall, we see our study as an exciting step in the direction of bringing together computational social science and derivational morphology. In future work, we intend to further fine-tune our methodological apparatus for tackling MFEP.
Valentin Hofmann was funded by the Arts and Humanities Research Council and the German Academic Scholarship Foundation. This research was also supported by the European Research Council (Grant No. 740516). We thank the reviewers for their detailed and helpful comments.
Eduardo G. Altmann, Janet B. Pierrehumbert, and Adilson E. Motter. 2011. Niche as a determinant of word fate in online groups. *PloS ONE*, 6(5).
John R. Anderson and Robert Milson. 1989. Human memory: An adaptive perspective. Psychological Review, 96(4):703-719.
Georgios Baskozos, John M. Dawes, Jean S. Austin, Ana Antunes-Martins, Lucy McDermott, Alex J. Clark, Teodora Trendafilova, Jon G. Lees, Stephen B. McMahon, Jeffrey S. Mogil, Christine Orengo, and David L. Bennett. 2019. Comprehensive analysis of long noncoding RNA expression in dorsal root ganglion reveals cell-type specificity and dysregulation after nerve injury. Pain, 160(2):463-485.
Laurie Bauer. 2001. Morphological productivity. Cambridge University Press, Cambridge, UK.
Laurie Bauer. 2019. Rethinking morphology. Edinburgh University Press, Edinburgh, UK.
Leo Breiman. 2001. Random forests. Machine Learning, 45:5-32.
Marc Brysbaert, Michael Stevens, Paweł Mandera, and Emmanuel Keuleers. 2016. The impact of word prevalence on lexical decision times: Evidence from the Dutch lexicon project 2. Journal of Experimental Psychology: Human Perception and Performance, 42(3):441-458.
Joan Bybee. 1995. Regular morphology and the lexicon. Language and Cognitive Processes, 10(425-455).
Joan Bybee. 2015. Language change. Cambridge University Press, Cambridge, UK.
Kenneth Church. 2000. Empirical estimates of adaptation: The chance of two noriegas is closer to p/2 than p2. In Conference on Computational Linguistics (COLING) 18, pages 180-186.
Kenneth W. Church and William A. Gale. 1995. Poisson mixtures. Natural Language Engineering, 1(2):163-190.
Ryan Cotterell and Hinrich Schütze. 2018. Joint semantic synthesis and morphological analysis of the derived word. Transactions of the Association for Computational Linguistics, 6:33-48.
Ryan Cotterell, Tim Vieira, and Hinrich Schütze. 2016. A joint model of orthography and morphological segmentation. In Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HLT) 15.
Ryan Cotterell, Ekaterina Vylomova, Huda Khayral-lah, Christo Kirov, and David Yarowsky. 2017. Paradigm completion for derivational morphology. In Conference on Empirical Methods in Natural Language Processing (EMNLP) 2017.
William Croft. 2000. Explaining language change: An evolutionary approach. Pearson, Harlow, UK.
David Crystal. 1997. The Cambridge encyclopedia of the English language. Cambridge University Press, Cambridge, UK.
David Crystal. 2004. Language and the internet. Cambridge University Press, Cambridge, UK.
Cristian Danescu-Niculescu-Mizil, Robert West, Dan Jurafsky, Jure Leskovec, and Christopher Potts. 2013. No country for old members: User lifecycle and linguistic change in online communities. In The Web Conference (WWW) 22.
Fermín del Prado Martín, Raymond Bertram, Tuomo Häikio, Robert Schreuder, and R. Harald Baayen. 2004. Morphological family size in a morphologically rich language: The case of finnish compared with Dutch and hebrew. Journal of Experimental Psychology: Learning, Memory, and Cognition, 30(6):1271-1278.
Marco del Tredici and Raquel Fernandez. 2018. The road to success: Assessing the fate of linguistic innovations in online communities. In International Conference on Computational Linguistics (COLING) 27.
Marco del Tredici, Raquel Fernandez, and Gemma Boleda. 2019. Short-term meaning shift: A distributional exploration. In Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HTL) 17.
Daniel Deutsch, John Hewitt, and Dan Roth. 2018. A distributional and orthographic aggregation model for english derivational morphology. In Annual Meeting of the Association for Computational Linguistics (ACL) 56.
Devin Gaffney and J. Nathan Matias. 2018. Caveat emptor, computational social science: Large-scale missing data in a widely-published reddit corpus. *PloS ONE*, 13(7):e0200162.
Matt Garley and Julia Hockenmaier. 2012. Beefmovers: Dissemination, diversity, and dynamics of english borrowings in a german hip hop forum. In Annual Meeting of the Association for Computational Linguistics (ACL) 50.
Jack Grieve, Andrea Nini, and Diansheng Guo. 2016. Analyzing lexical emergence in modern american english online. English Language and Linguistics, 21(1):99-127.
Sarah Hallerberg and Holger Kantz. 2008. Influence of the event magnitude on the predictability of an extreme event. Physical Review E, 77(1):011108.
William Hamilton, Jure Leskovec, and Dan Jurafsky. 2016. Diachronic word embeddings reveal statistical laws of semantic change. In Annual Meeting of the Association for Computational Linguistics (ACL) 54.
Bo Han and Timothy Baldwin. 2011. Lexical normalisation of short text messages: Makn sens a #twitter. In Annual Meeting of the Association for Computational Linguistics (ACL) 49.
Martin Haspelmath and Andrea D. Sims. 2010. Understanding morphology. Routledge, New York, NY.
Valentin Hofmann, Hinrich Schütze, and Janet B. Pierrehumbert. 2020. A graph auto-encoder model of derivational morphology. In Annual Meeting of the Association for Computational Linguistics (ACL) 58.
Jörg D. Jescheniak and Willem J. Levelt. 1994. Word frequency effects in speech production: Retrieval of syntactic information and of phonological form. Journal of Experimental Psychology: Learning, Memory and Cognition, 20(4):824-843.
Nivja de Jong, Robert Schreuder, and R. Harald Baayen. 2000. The morphological family size effect and morphology. Language and Cognitive Processes, 15(4/5):329-365.
Daniela B. Keller and Jörg Schultz. 2013. Connectivity, not frequency, determines the fate of a morpheme. *PloS ONE*, 8(7):e69945.