SlowGuess/ACL_2020_OCR huggingface dataset:
Valentin Hofmann\*, Janet B. Pierrehumbert†*, Hinrich Schütze‡
*Faculty of Linguistics, University of Oxford †Department of Engineering Science, University of Oxford ‡Center for Information and Language Processing, LMU Munich valentin.hofmann@ling-phil.ox.ac.uk
We present the first study that examines the evolution of morphological families, i.e., sets of morphologically related words such as "trump", "antitrumpism", and "detrumpify", in social media. We introduce the novel task of Morphological Family Expansion Prediction (MFEP) as predicting the increase in the size of a morphological family. We create a ten-year Reddit corpus as a benchmark for MFEP and evaluate a number of baselines on this benchmark. Our experiments demonstrate very good performance on MFEP.
Lexical change is a prime indicator of topical dynamics in social media. When people or events attract the attention of a user community, this is reflected by the token frequency evolution of individual words. The burst in token frequency of the word "trump" in social media before the 2016 presidential election (see Figure 1), e.g., mirrors the increasing presence of Donald Trump in public discourse during that time.
However, token frequency is only one way of measuring changes in topical prominence. Accompanying the increase in token frequency of "trump", there was a parallel increase in the number of words morphologically related to "trump", i.e., words like "trumpification", "antit Trumpism", and "detrumpify" (see Figure 1, Table 1). Most of these words have a very low token frequency and are removed in the first steps of a typical NLP pipeline.
Here, we present the first study of lexical change in social media that takes as its main unit of analysis the type frequency evolution of morphological families, i.e., changes in the number of morphologically related words such as "trump", "trumpification", "antit Trumpism", and "detrumbify". We show that morphological families allow for a fresh view
Figure 1: Token frequency of "trump" and type frequency of derivations of "trump" in the r/politics Subreddit between 08/2007 and 07/2018. Vertical line: date of the 2016 presidential election.
Month	Words
04/2015	“trump”
05/2015	“trump”, “trumpish”, “trumpster”, “trumpy”
06/2015	“trump”, “trumpening”, “trumper”, “trumpish”, “trumpness”, “trumpology”, “trumpster”, “trumpy”
07/2015	“trump”, “trumpening”, “trumper”, “trumpic”, “trumpification”, “trumpiness”, “trumpish”, “trumpsism”, “trumpistan”, “trumpness”, “trumpster”, “trumpy”
Table 1: Derivations of "trump" in four subsequent months of the r/politics Subreddit.
of lexical change in social media, making them a promising tool for studies in the social sciences that draw on NLP techniques. At the same time, our work adds to the growing body of computational research on derivational morphology (Cotterell et al., 2017; Vylomova et al., 2017; Cotterell and Schütze, 2018; Deutsch et al., 2018; Pierrehumbert and Granell, 2018; Hofmann et al., 2020) by introducing a temporal perspective.
Contributions. We introduce the novel task of Morphological Family Expansion Prediction (MFEP), which aims at predicting whether a morphological family will increase in size or not. We publish a benchmark for MFEP and show that the growth of morphological families can be successfully modeled using social and linguistic factors relating to the morphological parent. Furthermore,
7273
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7273-7283 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
our results add a new perspective to the growing body of research on the link between cultural and linguistic change in social media.
2 Morphological Families
We define a morphological family as a set  $F$  of words  $w$  with a shared free morpheme. Thus, "trump", "trumpification", "antitruppism", and "detrumpify" are in the same morphological family because they share the free morpheme "trump". By contrast, "antitruppism" and "antiprocessivism" are in different morphological families: even though both words have two morphemes in common ("anti" and "ism"), they do not belong to the same morphological family according to our definition since "anti" and "ism" are bound morphemes, not free morphemes like in the case of "trump", "trumpification", "antitruppism", and "detrumpify". In this study, we only consider derivational morphology.2 Compounds such as "trump-wall" are not split into their parts, but they can become a parent (see below). Thus, each word belongs to exactly one morphological family. The cardinality  $|F|$  of a family will be referred to as the morphological family size, a term also used in other studies (Schreuder and Baayen, 1997; de Jong et al., 2000; del Prado Martin et al., 2004).
The morphological parent  $w^{*}$  is the morphologically most basic word of a family  $F$ . The word "trump" is the parent of "antitrustpism", "trumpification", and "detrumbify". We denote the morphological family of  $w^{*}$  as  $F(w^{*})$ .
Except for the parent, all members of a family are morphological children and form a subset  $\tilde{F}$  of the entire family  $F$ . The words "antitruppism", "trumpification", and "detrumpify" are all morphological children. We further distinguish between old children  $\tilde{F}_o$ , established words in the lexicon, and new children  $\tilde{F}_n$ , innovative forms. While "trumpify" can still be considered a new child of "trump", "trumpster" is on its way to becoming an old child in the family. $^3$
As the example of "trumpster" shows, morpho
logical families are in constant flux. Specifically, there are three types of change in morphological families: word birth  $\emptyset \rightsquigarrow \tilde{F}_n$ , word entrenchment  $\tilde{F}_n \rightsquigarrow \tilde{F}_o$ , and word death  $\tilde{F}_n, \tilde{F}_o \rightsquigarrow \emptyset$ . The topic of this paper is word birth, i.e., we ask: given a set  $\mathcal{F}$  of morphological families, which will increase in size during a specified time interval? This differentiates our work from previous research on lexical change in social media which has focused on word entrenchment (see Section 7). One question we are particularly interested in is whether endogenous (language-internal) or exogenous (language-external) factors are better predictors of morphological family growth; these factors have been previously compared for changes in word token frequency (Altmann et al., 2011), but not for changes in morphological type frequency.
We develop MFEP using Reddit, a social media platform hosting discussions about a variety of topics. Reddit is divided into smaller communities centered around a shared interest, so-called subreddits (SRs), which are highly conducive to linguistic innovation (del Tredici and Fernandez, 2018). Concretely, we draw upon the Baumgartner Reddit Corpus, a collection of (almost) all publicly available comments posted on Reddit since 2005. A three-year slice of this corpus was used in a study on lexical change by Stewart and Eisenstein (2018). Gaffney and Matias (2018) show that the corpus's coverage of Reddit is not complete, but we do not expect this to affect our analysis.
Our study examines data from 2007 to 2018 in the four SRs r/gaming, r/movies, r/nba, and r/politics. These SRs were chosen because they are of comparable size, belong to the largest SRs of Reddit, and at the same time all reflect distinct areas of interest (Table 2). For each month, we also draw a random sample of comments from all SRs that will be used for computing word topicality (Section 4). The size of the sample equals the average size of the four selected SRs within the respective month.
Preprocessing. As in previous work (Tan and Lee, 2015), we filter posts for known bots and spammers. We remove abbreviations ("viz.", strings containing numbers ("b4"), references to users ("u/user") and SRs ("r/subreddit"), and both
4The corpus is available at https://files.pushshift.io/redisit/comments/.
<sup>1</sup>We make all our data and code publicly available at https://github.com/valentinhofmann/mfep.
2The distinction between inflection and derivation is gradual, not binary (Haspelmath and Sims, 2010). The suffix "ly", e.g., is variously defined as inflectional or derivational (Bauer, 2019). We try to exclude inflectional morphology as far as possible (e.g., by lemmatizing), but we are aware that a clear separation does not exist in linguistic reality.
3"Trumpster" is already listed in the English Wiktionary at https://en.wiktionary.org/wiki/Trumpster.
7274
SR	Start	End	NI	NW	NU	NT	|F|	μ|F|	σ|F|
r/politics	08/2007	07/2018	19	2,970,509,554	1,519,039	2,020,843	19,000	4.64	4.05
r/gaming	09/2007	08/2018	19	1,119,096,999	2,875,931	1,859,228	19,000	4.00	3.48
r/movies	06/2008	05/2018	17	738,365,964	1,772,085	613,158	17,000	3.44	3.09
r/nba	10/2010	09/2018	13	898,483,442	486,746	721,641	13,000	3.20	2.93
Table 2: SR summary statistics. Start: first month included; end: last month included;  $N_I$ : number of intervals;  $N_W$ : number of word tokens;  $N_U$ : number of users;  $N_T$ : number of threads;  $|\mathcal{F}|$ : number of family examples (1,000 ×  $N_I$ );  $\mu_{|F|}$ : mean family size per month;  $\sigma_{|F|}$ : standard deviation of family size per month.
full and shortened hyperlinks. We convert British English spelling variants to American English and lemmatize all words to remove inflectional morphology. We follow Han and Baldwin (2011) in reducing repetitions of more than three letters ("niiiiice") to three letters. Except for stopwords, we do not employ a frequency threshold; in particular, we include words that occur only once.
Computing morphological families. Given a collection of texts  $S$ , we define the morphological families as follows. Let  $V_{S}$  be the vocabulary of  $S$ , i.e., all words occurring in it. We define the set of parents  $O_{S} \subset V_{S}$  as the 1,000 most frequent words in  $S$ , regardless of whether the word is decomposable or not. This means that parents are not necessarily morphological roots (Haspelmath and Sims, 2010). We attempt to segment all other words  $w$  using affixes from a representative list of productive prefixes and suffixes in English (Crystal, 1997). We define the set  $C$  of candidate parents of  $w$  as follows. If  $w \in O_{S}$ , then  $C(w) = \{w\}$ . Otherwise,  $C(w) = \bigcup_{b \in B(w)} C(b)$ , where  $B(w)$  is the set of bases that remain when one of  $w$ 's affixes is removed. For  $w^{*} \in O_{S}$ , we then define its morphological family as
$$
F (w ^ {*}) = \{w \in V _ {S} | w ^ {*} \in C (w) \land | C (w) | = 1 \}.
$$
Procedurally, families can be identified by a recursive bottom-up algorithm. The algorithm is sensitive to morpho-orthographic rules of English (Plag, 2003); e.g., when "ness" is removed from "trumpsiness", the result is "trumpy", not "trumps".
4 Predictors for MFEP
In a setup similar to Altmann et al. (2011), we formalize MFEP as a binary classification task. Given a context interval  $i^{(c)}$ , a following temporally adjacent probe interval  $i^{(p)}$ , and a morphological parent  $w^*$ , we ask: what properties of  $w^*$  can predict whether  $|F(w^*)|$  increases in  $i^{(p)}$  or not?
Here, we set the length of  $i^{(c)}$  to 18 months and the length of  $i^{(p)}$  to 6 months. Morphological families are computed separately for each pair of  $i^{(c)}$  and  $i^{(p)}$ . The lowest frequency count of a parent in  $i^{(c)}$  is 244. Table 2 summarizes statistics of the morphological families for each SR.
We define a number of predictors for family expansion that are measurements of properties of  $w^{*}$ . All predictors are motivated by work in psycholinguistics and NLP. They fall into three natural classes: (i) a type frequency-based predictor  $(\overline{|F|})$ , (ii) token frequency-based predictors  $(f_{r},\bar{z}^{(p)})$ , and (iii) dissemination-based predictors  $(D_w^U,D_w^T,Q_w)$ . All predictors except for  $\bar{z}^{(p)}$  (which is measured in  $i^{(c)}$  and  $i^{(p)}$ ) are measured in  $i^{(c)}$ .
Family size  $\overline{F}$ . The family size is a prime example of an endogenous (language-internal) factor, i.e., one that depends on the linguistic system. A morpheme with a large family might combine more readily with new affixes than a morpheme that occurs only with a small number of affixes. This idea bears a theoretical connection to smoothing techniques such as Witten-Bell and Kneser-Ney smoothing, which model the probability of previously unseen  $n$ -grams containing a given word  $(\approx |\tilde{F}_n|)$  by assuming a rich-get-richer process (Manning and Schütze, 1999; Teh, 2006). It is also in line with lexical growth models based on preferential attachment (Steyvers and Tenenbaum, 2005). Intuitively, the fact that morphological children themselves can become the basis for new derivations also suggests a rich-get-richer process.
Notice that  $\overline{|F|}$  is equivalent to the type fre
In a situation where both "sense" and "sensation", e.g., fall above the frequency threshold, we get two separate morphological families of the parent "sense" (a root) with "nonsense", "sensitive", etc. and the parent "sensation" (not a root) with "sensational", "sensationalism", etc. (the children of "sensation" are not added to the family of "sense"). However, most morphological parents are in fact roots.
quency of  $w^{*}$ . In linguistics, type frequency is known to be a good predictor of the productivity of inflectional patterns (Bybee, 1995). Furthermore, it has been shown that the morphological family size facilitates lexical processing (Schreuder and Baayen, 1997). To probe whether type frequency also influences the likelihood of a family to grow, we include the predictor  $|\overline{F}|$ , morphological family size averaged over the 18 months of  $i^{(c)}$ .
Relative token frequency of parent  $f_{r}$ . Frequent words are known to be more accessible in lexical processing than rare words (Jescheniak and Levelt, 1994). Therefore, they might be more available for use in novel derivations, causing an increase in morphological family size.
Trending behavior  $\bar{z}^{(p)}$ . Changes in the relative frequency of a morphological parent might be indicative of concomitant changes in the morphological family size. If a word gains in popularity and becomes more frequent, this could increase the chances of new morphologically related words being created. The trending behavior is a prime example of an exogenous (language-external) factor, i.e., one that depends on non-linguistic events (e.g., a presidential election). Therefore, we measure whether the parent increases in frequency.
This is done in the following way: we calculate the z-score of the frequency distribution of the parent in the probe interval  $i^{(p)}$  relative to the frequency distribution in the context interval  $i^{(c)}$ . The mean of these z-scores is then used as a continuous variable in the model,
$$
\bar {z} ^ {(p)} = \frac {1}{| i ^ {(p)} |} \sum_ {j = 1} ^ {| i ^ {(p)} |} z _ {j} ^ {(p)} = \frac {1}{| i ^ {(p)} |} \sum_ {j = 1} ^ {| i ^ {(p)} |} \frac {x _ {j} ^ {(p)} - \mu^ {(c)}}{\sigma^ {(c)}},
$$
where  $|i^{(p)}| = 6$  is the length in months of the probe interval and  $x_{j}^{(p)}$  is the relative frequency of the parent in month  $j$  ( $1 \leq j \leq |i^{(p)}|$ ) of  $i^{(p)}$ ;  $\mu^{(c)}$  and  $\sigma^{(c)}$  are mean and standard deviation of relative frequency of the parent in the 18 months of the context interval  $i^{(c)}$ . The measure detects increases in frequency relative to the intrinsic variation in usage frequency of a particular word. This is necessary since some words naturally exhibit stronger short-term fluctuations, which we do not want to count as frequency bursts. Similar methods for peak detection in time series are frequently used, e.g., in Baskozos et al. (2019). We use both  $i^{(c)}$  and  $i^{(p)}$  for calculating  $\bar{z}^{(p)}$  because this captures the idea of exogenous forcing without any additional
assumptions; notice that the metric is calculated on the parent only and does not include any information about what is being predicted in MFEP, namely changes in the morphological family size.
User dissemination  $D_w^U$ . Following findings by Church and Gale (1995) and Altmann et al. (2011), we define user dissemination  $D_w^U$  as the extent to which the number of users of a specific word  $w$  deviates from a Poisson process,
$$
D _ {w} ^ {U} = \frac {U _ {w}}{\tilde {U} (f _ {w})},
$$
where  $U_{w}$  is the number of users who posted at least one comment including  $w$  in  $i^{(c)}$ ,  $f_{w}$  is the relative frequency of  $w$  in  $i^{(c)}$ , and  $\tilde{U}(f_{w})$  is the expected number of users under a Poisson model given the relative frequency  $f_{w}$ .  $\tilde{U}(f_{w})$  can be calculated as
$$
\tilde {U} (f _ {w}) = \sum_ {j = 1} ^ {N _ {U}} \tilde {U} _ {j} \approx \sum_ {j = 1} ^ {N _ {U}} \left(1 - e ^ {- f _ {w} m _ {j} ^ {U}}\right),
$$
where  $N_U$  is the number of users,  $\tilde{U}_j$  is the probability that the posts of user  $j$  contain  $w$  at least once, and  $m_j^U$  is the total number of words posted by user  $j$  in  $i^{(c)}$ . The approximation is valid for  $f_w \ll 1$  and  $m_j^U / \sum_{j}^{N_U} m_j^U \ll 1$ . Our data satisfy both requirements.
User dissemination and the following dissemination measures have a cognitive justification: it has been shown that items that are used in more diverse situations and contexts are stored in human memory in a way that makes them more retrievable (Anderson and Milson, 1989; Brysbaert et al., 2016). Thus, words with a higher dissemination are more accessible to speakers and could figure more prominently among bases for new formations. The dissemination measures fall into a gray area between exogenous and endogenous factors since they reflect the cognitive representation of language-external properties (Altmann et al., 2011).
Thread dissemination  $D_w^T$ . Similar to user dissemination, thread dissemination  $D_w^T$  is defined as the extent to which the number of threads containing a specific word  $w$  deviates from a Poisson process (Altmann et al., 2011),
$$
D _ {w} ^ {T} = \frac {T _ {w}}{\tilde {T} (f _ {w})},
$$
where  $T_{w}$  is the number of threads that include at least one instance of  $w$ , and  $\tilde{T}(f_w)$  is the expected
7276
number of threads under a Poisson model.  $\tilde{T}(f_w)$  can again be calculated as
$$
\tilde {T} (f _ {w}) = \sum_ {j = 1} ^ {N _ {T}} \tilde {T} _ {j} \approx \sum_ {j = 1} ^ {N _ {T}} \left(1 - e ^ {- f _ {w} m _ {j} ^ {T}}\right),
$$
where  $N_{T},\tilde{T}_{j}$  , and  $m_j^T$  are defined analogously to  $N_U,\tilde{U}_j$  , and  $m_j^U$  . The approximation is again valid since the data satisfy  $m_j^T /\sum_j^{N_T}m_j^T\ll 1$
Topicality  $Q_{w}$ . Because SRs are communities centered around interests, words that are characteristic of a SR's topic are more frequent in that SR than in the others. Topicality has been shown to have an impact on lexical dynamics at long time scales (Church, 2000; Montemurro and Zanette, 2016). It could also influence the productivity of morphological families: higher topical dissemination, i.e., lower topicality, could facilitate growth. To capture this effect, we introduce a metric of topical distinctiveness,  $Q_{w}$ , which we define as
$$
Q _ {w} = \frac {f _ {w}}{\tilde {f} _ {w}},
$$
where  $f_{w}$  is the relative frequency of the word  $w$  in a SR in  $i^{(c)}$ , and  $\tilde{f}_{w}$  is the expected relative frequency of  $w$  based on a random sample of posts from all SRs in  $i^{(c)}$ .
The polarity of  $Q_{w}$  is reversed to  $D_w^U$  and  $D_w^T$ , i.e., a word that is very clumped in SR space will have a high value of  $Q_{w}$ , but a word that is very clumped in user or thread space will have a low value of  $D_w^U$  or  $D_w^T$ , respectively.
Finding growing families. We use two different notions of growth for MFEP: absolute growth and relative growth. We define absolute growth as
$$
\delta_ {a} (F) = \mu_ {| F |} ^ {(p)} - \mu_ {| F |} ^ {(c)},
$$
where  $\mu_{|F|}^{(p)}$  and  $\mu_{|F|}^{(c)}$  are the mean morphological family size in  $i^{(p)}$  and  $i^{(c)}$ , respectively. Relative growth is defined similarly as
$$
\delta_ {r} (F) = \frac {\mu_ {| F |} ^ {(p)}}{\mu_ {| F |} ^ {(c)}}.
$$
For both  $\delta_{a}$  and  $\delta_r$ , we define binary features based on thresholds  $l_{a}$  and  $l_{r}$ , i.e., we define a morphological family  $F$  to be a positive example if
$\delta_{a}(F) > l_{a}$  for a pair of  $i^{(p)}$  and  $i^{(c)}$  (in the case of absolute growth). We thus train two models: one for predicting whether  $\delta_{a}(F) > l_{a}$ , one for predicting whether  $\delta_{r}(F) > l_{r}$ .
Model. We use Random Forests (RF) to perform the classification (Breiman, 2001). RF offers two main advantages in comparison with other models. Firstly, as opposed to other tree-based models, RFs decorrelate trees, which is important if the features are correlated (as is the case here). Secondly, the feature importance scores of a RF provide a transparent way to compare the predictive power of features. We do not use more complex albeit potentially better performing methods such as deep architectures since our primary goal is to compare various features and show that MFEP is a feasible computational task.
Since the data contain considerably more negative than positive examples, we randomly sample one negative example for every positive example for the final data. The interval pairs from all SRs were merged into one dataset, which was then split into 0.8 and 0.2 for train/dev and test sets. The train/dev set was split again into 0.8 and 0.2 for train and dev sets. Thus, all sets contain a balanced sample of interval pairs from all SRs. $^6$
We use a total of 68,000 pairs of intervals  $(i^{(c)}, i^{(p)})$  where  $i^{(c)}$  is the context interval and  $i^{(p)}$  the probe interval (see also Table 2). Recall that  $i^{(c)}$  has length 18 months and  $i^{(p)}$  6 months. Temporally adjacent interval pairs are overlapping by  $|i^{(p)}|$  months, i.e., every month in the original data is used exactly once in a probe interval and three times in a context interval.
We do not perform hyperparameter tuning and instead choose typical values for the hyperparameters of RF: 80 for the number of trees, and 20 for tree depth. For our initial MFEP models, we set the thresholds  $l_{a} = 2.4$  and  $l_{r} = 1.6$ , two values in the mid-range of existing values for  $\delta_{a}$  and  $\delta_{r}$ . We will later analyze the sensitivity to these hyperparameters in greater detail.
Overall performance. As shown in Table 3, the RF models exhibit a good performance with an overall prediction accuracy of  $80.9\%$  for  $l_{a} = 2.4$  and  $70.8\%$  for  $l_{r} = 1.6$  (random baselines:  $50.0\%$ ).
<sup>6</sup>Since the chosen SRs cover diverse topics, the model should have a high transferability to other SRs, but we have not tested this.
7277
la(δa)	μ±σ	lr(δr)	μ±σ
0.0	0.8	1.6	2.4	3.2	4.0	4.8	1.0	1.2	1.4	1.6	1.8	2.0	2.2
All	.618	.696	.745	.809	.769	.844	.846	.761±.077	.618	.631	.651	.708	.701	.724	.744	.683±.045
|F|	.602	.674	.724	.742	.705	.754	.831	.719±.066	.602	.607	.629	.645	.628	.681	.744	.648±.046
fr	.522	.542	.524	.551	.534	.574	.492	.534±.024	.522	.507	.509	.465	.558	.483	.605	.521±.044
z(p)	.520	.540	.523	.528	.506	.475	.631	.532±.045	.520	.521	.521	.555	.544	.595	.488	.535±.031
DwU	.506	.513	.506	.509	.466	.484	.569	.508±.030	.506	.499	.494	.505	.485	.569	.558	.517±.031
DwT	.521	.546	.508	.518	.542	.590	.431	.522±.045	.521	.525	.492	.490	.478	.578	.535	.517±.032
Qw	.512	.504	.506	.516	.522	.484	.415	.494±.034	.512	.507	.505	.516	.496	.543	.558	.520±.021
Fn	.557	.730	.883	.846	.909	—	—	.785±.129	.557	.567	.621	.618	.657	.701	.703	.632±.054
12 m.	.608	.668	.749	.811	.840	.838	.946	.780±.106	.608	.629	.684	.673	.771	.733	.912	.716±.095
24 m.	.630	.688	.732	.780	.797	.818	.847	.756±.071	.630	.645	.679	.710	.690	.748	.786	.698±.051