SlowGuess/NAACL_2021_OCR huggingface dataset:
Jian Zhu
lingjzhu@umich.edu
jurgens@umich.edu
New words are regularly introduced to communities, yet not all of these words persist in a community's lexicon. Among the many factors contributing to lexical change, we focus on the understudied effect of social networks. We conduct a large-scale analysis of over 80k neologisms in 4420 online communities across a decade. Using Poisson regression and survival analysis, our study demonstrates that the community's network structure plays a significant role in lexical change. Apart from overall size, properties including dense connections, the lack of local clusters and more external contacts promote lexical innovation and retention. Unlike offline communities, these topic-based communities do not experience strong lexical levelling despite increased contact but accommodate more niche words. Our work provides support for the sociolinguistic hypothesis that lexical change is partially shaped by the structure of the underlying network but also uncovers findings specific to online communities.
Lexical change is a prevalent process, as new words are added, thrive, and decline in day-to-day usage. While there is a certain randomness at play in word creation and adoption (Newberry et al., 2017), there are also psychological, social, linguistic and evolutionary factors that systematically affect lexical change (Labov, 2007; Christiansen and Kirby, 2003; Lupyan and Dale, 2010).
In sociolinguistics, one structural factor that has long been recognized as influencing lexical changes is the language community's social network. For example, drawing on pioneering works on social networks (Granovetter, 1977, 1983), the weak tie model of change holds that the structural properties of social networks can account for the general tendency of some language communities to be more resistant to linguistic change than oth-
ers (Milroy and Milroy, 1985, 1992; Milroy and Llamas, 2013). A classic finding is that loose-knit networks with mostly weak ties are more conducive to information diffusion, thereby facilitating innovation and change, while close-knit networks with strong bonds impose norm-enforcing pressure on language usage, strengthening the localized linguistic norms (Milroy and Milroy, 1985).
One compelling observation in favor of this argument concerns the comparison between two Germanic languages, Icelandic and English. Icelandic has changed little since the late thirteenth century, which could be due to the norm-enforcing pressure inherent in the strong kinship and friendship ties. In contrast, in Early Modern London English, the loosening of network ties, accompanied by the rise of the mobile merchant class, was argued to be responsible for some radical change in the language (Milroy and Milroy, 1985).
This study extends network-based sociolinguistic research to online communities, which remain understudied despite their expansion in past decades. While we draw an analogy between offline and online communities, our focus is on communities of practice (Eckert and McConnell-Ginet, 1992; Holmes and Meyerhoff, 1999; Schwen and Hara, 2003), or "an aggregate of people who come together around mutual engagement in an endeavor" (Eckert and McConnell-Ginet, 1992), rather than offline speech communities. We examine how network structures affect lexical innovation, retention and levelling in online communities. Specifically, we ask 1) how network structure contributes to the introduction of new words to online communities (innovation), 2) how structural properties affect the survival of these newly introduced words (retention) and 3) whether the increased inter-connectedness causes online communities to adopt a similar set of new words (levelling).
This work offers the following contributions. First, using a massive longitudinal dataset of 4420
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2201-2218 June 6-11, 2021. ©2021 Association for Computational Linguistics
Figure 1: Applying the hypothesis of Milroy and Milroy (1985) to these two gaming subreddits of similar size suggests that the network with lower density (left; r/masseffect) will be more innovative than the more closely-connected community shown right (r/F13thegame). However, after controlling for size, the one with higher average degree (more inner-connections) (right: r/F13thegame) tends to develop more lexical innovations.
communities, we precisely quantify the structural mechanisms that drive these lexical processes. Our work adds to network studies in sociolinguistics focusing on in-person observations of local communities (Conde-Silvestre, 2012; Sharma and Dodsworth, 2020) and shows that conclusions drawn from offline communities are insufficient to account for behavior seen in online social networks (Figure 1). We find that larger size, denser connections, lack of local clustering and greater external contacts promote lexical innovation and retention in online communities, while density, as discussed most in offline studies, could be an emergent byproduct of network size. These topic-based communities also do not experience strong levelling due to increased contact. Second, emerging studies in online communities (Danescu-Niculescu-Mizil et al., 2013; Stewart and Eisenstein, 2018; Del Tredici and Fernandez, 2018) focus exclusively on lexical change at the individual or word level. Few investigate how global network properties affect lexical change at the community level. Finally, sampling offline networks presents practical difficulties, we extract complete networks for thousands of online communities, providing a large-scale dataset to explore the structural factors of lexical change. Our code is available at https://github.com/lingjzhu/reddit_network and replication details are available in Appendix A.
2 Lexical Change
Lexical change and social networks Since the landmark study of sound change in the Belfast community by Milroy and Milroy (1985), the impact of network structures on language change has been a key consideration in sociolinguistics. Milroy and Milroy (1985) found that speakers in loose-knit networks tend to experience more linguistic change than those in close-knit networks. Most early social network studies focus predominantly on speakers in local, less mobile communities where ties between people tend to be strong (Nevalainen, 2000; Conde-Silvestre, 2012; Sharma and Dodsworth, 2020). Except for a few recent simulation studies (Reali et al., 2018), researchers have rarely explored how the global properties of social networks systematically affect lexical change, although the weak tie model does predict an influence of social network at the macro-level. In addition, while there are lexicographic studies attempting to enumerate factors that affect the acceptance of neologisms (Metcalf, 2004; Barnhart, 2007), network structures are rarely taken into consideration. A key limitation of previous works has been access to a large longitudinal dataset of communities with different network properties as well as a precise estimate of the network structure of larger communities, which are limitations this study overcomes.
Lexical change in online communities The rise of social media and the proliferation of Internet speech has drawn increasing attention to lexical change in online communities, including Twitter (Eisenstein et al., 2014; Goel et al., 2016), Reddit (Altmann et al., 2011; Stewart and Eisenstein, 2018; Del Tredici and Fernandez, 2018) and review sites (Danescu-Niculescu-Mizil et al., 2013). It has been shown that the usage of certain words is associated with community loyalty and norms (Zhang et al., 2017; Bhandari and Armstrong, 2019) and indicative of user behaviors (Danescu-Niculescu-Mizil et al., 2013; Noble and Fernandez, 2015; Chang and Danescu-Niculescu-Mizil, 2019; Klein et al., 2019). Specifically for lexical change over time, Stewart and Eisenstein (2018) investigate the survival of lexical items in Reddit, and conclude that a word's appearance in more diverse linguistic contexts is the strongest predictor of its survival while social dissemination is a comparatively weaker predictor. Del Tredici and Fernandez (2018) examined the use of neologisms in 20 subreddit communities. Their finding that weak-tie users tend to innovate whereas strong-tie users tend to propagate is consistent with the weak tie theory of language change. Other studies along this line tend to focus on the role of individual users (Paolillo, 1999; Paradowski and Jonak, 2012). The study closest to our current study is that by Kershaw et al. (2016), which investigates word innovations in Reddit and Twitter by looking at grammatical and topical factors. Yet Kershaw et al. (2016) only used network information to partition the dataset without exploring the role of these structural attributes in depth. Less is known about how network structures are systematically related to community-level lexical change in online communities, which we address here.
3 The Reddit Network Corpus
To analyze lexical innovation in a network setting across long time scales, we use comments made to Reddit, one of the most popular social media sites. There, 330M users are active in about 1M distinct topic-based sub-communities (subreddits). Here we define each subreddit as a community of practice (Schwen and Hara, 2003), as each subreddit is relatively independent with various norms formed through interactions. The subreddit communities span across a wide range of social network structures (Hamilton et al., 2017) and linguistic use patterns (Zhang et al., 2017), making them ideal
for studying the propagation of sociolinguistic variations in online communities. Detailed statistics are given in Appendix B.
Data To strike a balance between acquiring active subreddits and preserving the diversity of these communities, we initially select the top 4.5K subreddits based on their overall size from their inception to October 2018 via the Convokit package (Chang et al., 2020). Let  $\mathcal{C}_{\text{Reddit}} = \{C_1, C_2, \ldots, C_n\}$  be the set of subreddit communities included in the corpus. A subreddit community  $C_n$  is further discretized into multiple monthly subreddit communities  $c_n(t)$  based on its actual life span in the monthly time step  $t$ , such that  $C_n = \{c_n(1), c_n(2), \ldots, c_n(t_{\text{max}})\}$ . For each  $c_n(t)$ , we extracted all individual comments except those marked as [deleted] and performed tokenization via SpaCy. During text cleaning, we removed numbers, emojis,urls, punctuations and stop words, and set a cutoff frequency of 10 over the entire dataset to exclude infrequent typos or misspellings. Only those monthly subreddits  $c_n(t)$  with more than 500 words or 50 users after preprocessing are retained. Some communities known for their content in foreign languages are also removed. After preprocessing, 4420 subreddits were left in our analysis.
Community networks For a community  $c$  from month  $t = 1,2,\ldots ,t_{max}$  , its temporal network can be represented as a discrete-time sequence of network snapshots  $\mathcal{G}_c =$ $\{G_c(1),G_c(2),\dots ,G_c(t_{max})\}$  . Each snapshot network at time  $t,G_{c}(t) = \{V_{c}(t),E_{c}(t)\}$  consists of a set of user nodes  $V_{c}(t)$  and a set of edges  $E_{c}(t)$  characterizing direct interactions between users.  $G_{c}(t)$  is initiated as an undirected and unweighted graph under the assumption that these commenting communications are mutual and bi-directional.
A user  $u_{i}$  is represented as a node if this user has posted at least one comment at month  $t$ . An edge  $e_{ij}$  exists between user  $u_{i}$  and user  $u_{j}$  if these two users have interacted in close proximity in a common discussion thread, that is, separated by at most two comments (Hamilton et al., 2017; Del Tredici and Fernandez, 2018). Since online communications are asynchronous, a discussion thread created at time  $t$  may still have active comments from users at time  $t + 1$  or later. For such threads, we only included interactions at time  $t$  in  $G_{c}(t)$  and grouped later interactions
into the future time steps at which these interactions happened. Users marked as [deleted] or AutoModerater were all removed. After filtering, a total of 289.8k community networks have been extracted for all 4420 communities.
Inter-community networks We also identify the network dynamics between communities. We created temporal network  $\mathcal{G}_{IC}$  to characterize the connections between communities at consecutive months  $t = 1,2,\ldots ,t_{max},$ $\mathcal{G}_{IC} = \{G_{IC}(1),G_{IC}(2),\dots,G_{IC}(t_{max})\}$  , in which  $G_{IC}(t) = \{V_{IC}(t),E_{IC}(t)\}$  .  $V_{IC}(t)$  contains the set of nodes whereas  $E_{IC}(t)$  is the set of edges between communities. A community is represented as a node  $u_{i}$  in  $G_{IC}(t)$  , except for communities that do not exist or are no longer active at time  $t$  . Two communities are determined to be connected if they share active users, that is, users who had posted at least 2 comments in both communities during that month. Each network snapshot is initiated as a weighted and undirected network with the edge weights set to the numbers of shared users, as an approximation of connection strength. Finally, 152 inter-community networks have been constructed since the inception of Reddit in 2005 until October 2018.
Internet neologisms Neologisms are newly emerging language norms that fall along a continuum from the common words known to the overwhelming majority of users to nonce words that are mostly meaningless and rarely adopted. We only focus on Internet neologisms, e.g. lol, lmao, idk, as community slangs in Reddit communities. Such neologisms are abundant in the ever-evolving online communications as people use them for convenience or to signify in-group identity. The non-standard, idiosyncratic spelling patterns of Internet neologisms also make them easier to track than nuanced meaning shifts.
We obtained the Internet slang from two online dictionary sources, NoSlang.com and Urban Dictionary. The neologisms in NoSlang.com have been used in a previous study (Del Tredici and Fernandez, 2018). After filtering some lexical entries, we ended up with approximately 80K Internet neologisms for subsequent analysis. We set the minimum frequency threshold of neologisms to 10 over the entire dataset; this low setting ensures that the analysis is not biased by selectively looking only
Frequency	Neologisms
Most frequent	lol, /r, kinda, bitcoin, idk, lmao, tbh
tl;dr, alot, /s, OMG, lvl, haha, iirc
Least frequent	thugmonster, blein, sotk, f’tang
yobbish, ferranti, sonse, yampy
Table 1: Examples of neologisms.
at surviving words, which may obscure the lexical change process. Details can be found in Appendix B.
Many of these neologisms were not first coined in Reddit but were coined elsewhere and introduced into subreddits subsequently by users. Since it was neither feasible nor possible to trace the exact origins of these words, we instead focused on how words were introduced and adopted. This approach is also consistent with previous studies of lexical change (Altmann et al., 2011; Grieve et al., 2017; Del Tredici and Fernandez, 2018).
4 Network statistics
Communities in Reddit can be defined in terms of how their members relate within the community (intra) and how the community relates to other communities (inter) through multi-community memberships by its users (Tan and Lee, 2015). We formalize both as potential influences. As network attributes may be affected by the hyperparameters for network construction, we additionally validate this approach in Appendix C.
Intra-community features We take the following network measurements for each  $G_{c}(t)$  to characterize the global properties of community networks: density, average local clustering coefficient, transitivity, average degree, maximum degree, degree assortativity, fraction of the largest connected components and fraction of singletons. These network measures can characterize the size, fragmentation and connectedness of Reddit networks (Hamilton et al., 2017; Cunha et al., 2019).
Parameters like average local clustering coefficient, transitivity, and assortativity are highly influenced by the underlying degree distribution (Hamilton et al., 2017). We adjusted these parameters by computing their relative differences with respect to the mean values of five random baseline networks, which were generated by randomly rewiring the original network for  $10 \times$  edge count iterations and preserving the original degree sequence. These features are referred to as adjusted local cluster
ing coefficient, adjusted transitivity, and adjusted assortativity in the following text.
Inter-community features In addition to the intra-community network features, it is also necessary to measure a community's external connections to other communities. User mobility and external influence have been found to play a role in the process of lexical change (Conde-Silvestre, 2012). For each between-community network snapshot  $G_{IC}(t)$  at time  $t$ , we focus on the properties of individual nodes (communities). We computed the degree centrality, closeness centrality, eigenvector centrality, betweenness centrality and PageRank centrality for each community node. These centrality measures quantify the connectedness of a community to other communities, which can be used as an indicator of their degree of external contact and user mobility.
5 Lexical innovations
In what types of communities are neologisms likely to be introduced? Here, we investigate the extent to which the number of innovations introduced per month can be predicted with only the structural properties of community networks.
Experiment setup Given a set of communities  $C = \{c_1, c_2, \ldots, c_n\}$  spanning time steps  $T = \{1, 2, \ldots, t_{max}\}$ , we aim to predict the count of monthly lexical innovations for each community  $Y = \{y_1^{c_1}, y_2^{c_1}, \ldots, y_{t_{max}}^{c_n}\}$  from the corresponding network attributes  $\mathbf{X} = \{\mathbf{x}_1^{c_1}, \mathbf{x}_2^{c_1}, \ldots, \mathbf{x}_{t_{max}}^{c_n}\}$ . The predicted variable  $y_t^{c_n}$  is computed by counting only innovations first introduced into community  $c_n$  at month  $t$ . Any subsequent usage of the same innovations after their first introduction is not counted as innovations in community  $c_n$ . The feature vector  $\mathbf{x}_t^{c_n}$  is the structural features of the network at time  $t$  for  $c_n$ . After removing about  $0.03\%$  invalid data points and outliers, we ended up with 289.1k samples for the task.
Implementation We used both intra-community and inter-community features for innovation prediction. However, in empirical networks, certain structural features tend to be correlated. For example, network size and density are usually strongly correlated on a log-log scale in online social networks (Backstrom et al., 2012), which is also apparent in our dataset (Spearman  $\rho = -0.87$ ). Such correlations may confound the interpretation of the
feature contributions (see Appendix D). To generate orthogonal features, we first standardized all 15 network features and then used principal component analysis (PCA) with whitening to decompose them into principal components (PCs). Standardization was necessary as it could prevent a few variables with a large range of variance from dominating the PCs. We found that the first five PCs accounted for  $87\%$  of total variance and 10 PCs explained  $99\%$  of the total variance.
Since counts of innovations are non-negative integers, Poisson regression and Histogram-based Gradient Boosted Trees (HGBT) with Poisson loss were used to predict the number of innovations with PCs. The model parameters were selected through ten-fold cross-validation. The data were randomly partitioned into training and test sets with a ratio of  $90\% / 10\%$ . We report the mean absolute error (MAE) and the mean Poisson deviance (MPD) averaged across 20 runs with different random partitions of data. Both metrics should be minimized by the models. Replication details are in Appendix E.
Model	MAE	MPD
Baseline (mean)	19.37	30.16
Poisson reg. (PCs=5)	11.79	12.29
Poisson reg. (PCs=10)	11.14	11.03
Poisson reg. (raw feat.)	11.72	12.21
HGBT (PCs=5)	10.57	9.63
HGBT (PCs=10)	9.65	8.19
HGBT (raw feat.)	9.24	7.49
Table 2: Results of lexical innovation prediction.
Results As summarized in Table 2, all models outperformed the mean baseline by a significant margin, suggesting that the internal network structures and the external connections to other communities are systematically correlated to the count of lexical innovations per month. The three largest coefficients of the Poisson model with 5 PCs correspond to the first three PCs (see Figure 2). PC1 represents the overall size of the network, such that the Poisson model predicts that networks having larger overall size tend to have more innovations (Coefficient: -0.87). PC2 indicates the fragmentation and the local clusteredness of the network, and contributes negatively to lexical innovation (Coefficient.: -0.20). In other words, fragmented networks with local clusters tend to have fewer innovations as this structure inhibits the spread of information.
<sup>1</sup>Note that the coefficient sign for a PC must be interpreted with respect to its loading on structural components.
Figure 2: The decomposition of the PCs used in predicting innovations. Inter-community features are highlighted in orange bars. Adj.lc, Adj.gc and Adj.assort are local clustering coefficient, global clustering coefficient and assortativity adjusted with respect to a random network. PC1 represents the overall size, PC2 the density of intracommunity connections and PC3 the inter-community connections.
PC3 is generally related to inter-community connections with positive correlation to innovation (Coefficient.: 0.19). Yet what matters is not the number of communities connected (degree centrality) but the quality of those connections (Pagerank centrality). High Ragerank centrality suggests that the network might be connected to many influential communities, as these connections are weighted higher in the Pagerank algorithm (Page et al., 1999).
While structural properties can account for many regularities in the creation of lexical innovations, there are also surges of innovations that cannot be explained by structural factors alone. Inspection of the data suggests that the surges of innovations at the tail of empirical distributions are often related to some factors beyond network structures, including topical variations or external events, such as community migration or new game releases for some game communities.
6 Survival Analysis
Not all lexical innovations survive through time, with only a few neologisms eventually becoming widely adopted by community members. Here, we test the structural factors that systematically affect the survival of words in online communities.
Model specification Survival analysis models the elapsed time before a future event happens (Kleinbaum and Klein, 2010), which has been used to predict word survival (Stewart and Eisenstein, 2018). Compared to the traditional Cox model, deep survival analysis approximates the risk (hazard) with neural networks, thereby achieving improved performance. We estimated word survival with the Logistic Hazard model (LH) proposed
by Klamme and Borgan (2019). Given samples  $\{\mathbf{x}_1,\mathbf{x}_2,\dots ,\mathbf{x}_n\}$  and time steps  $\{1,2,\ldots ,T\}$ , the LH method estimates  $h(t|\mathbf{x})$ , the hazard function of the death event with respect to time  $t$ , with a deep neural network. The hazard function can be interpreted as the word's "danger of dying" at  $t$ .
After the model is trained, the survival function  $S(t|\mathbf{x}_i)$  for sample  $\mathbf{x}_i$  can be computed as
$$
S (t | \mathbf {x} _ {i}) = \prod_ {t = 1} ^ {T} [ 1 - h (t | \mathbf {x} _ {i}) ] \tag {1}
$$
$S(t|\mathbf{x}_i)$  can be interpreted as the chance of survival at time  $t$  for sample  $\mathbf{x}_i$ , that is, the survival probability of a word given the corresponding network features at time  $t$ . The detailed derivation and experiment settings are given in Appendix F.