SlowGuess/NAACL_2021_OCR huggingface dataset:
Data coding We consider only communities that have existed longer than six months and words that survived more than three months. The subreddit duration restriction avoids right-censoring of the data from new communities forming and quickly dying (a common event), which would skew estimates of word survival. A word's survival time is defined as the total number of months a word persists in a community, excluding the intervening month in which the word is not used. The last time step  $t$  at which the word shows up is considered the "death" event. However, if this last time step is also the last three recorded months, this word is considered right-censored such that a death event has not happened. This three-month buffer period is added to avoid false negatives. The network features for predictions were derived from averaging all the monthly features for the months that a particular word has existed. After preprocessing, we
Model	Concordance	IBS
Random baseline	0.50	0.25
Cox Model (PCs=5)	0.600	0.297
Cox Model (PCs=10)	0.662	0.289
Cox Model (raw)	0.665	0.209
LH (PCs=5)	0.584	0.245
LH (PCs=10)	0.691	0.192
LH (raw)	0.718	0.152
ended up with 1.47M samples with 69,683 distinct words. All features were then transformed into 10 orthogonal principal components using PCA with whitening. The first 5 PCs accounted for  $90\%$  of the total variance whereas all 10 PCs explained  $99\%$  of the variance.
Implementation Models of deep survival analysis were implemented via the package pycox (Kvamme et al., 2019). We trained a three-layered LH model with 256 hidden dimensions to model the word survival. We used the Adam optimizer with a learning rate of 0.001 and a batch size of 2048 samples. The data were randomly partitioned into  $80\%$ ,  $10\%$  and  $10\%$  portions as training, development and test sets, respectively, with no overlap between sets in terms of subreddits. Each model was run for 3 epochs and was run 10 times with different data partitioning. The performance metrics were averaged. We also ran baseline Cox models under the same conditions for comparison.
The performance is evaluated with time-dependent concordance (Antolini et al., 2005) and Integrated Brier Score (IBS) (Kvamme et al., 2019). Concordance measures the model's capacity to provide a reliable ranking of individual risk scores. A good concordance score should be above the 0.5 random baseline and close to 1. The IBS is the average squared distances between the observed survival events and the predicted survival probability and should be minimized by the model.
Results Results in Table 3 show that structural factors of the community in which a neologism is introduced can predict its chance of survival or death, with all models outperforming the baseline by a significant margin. Since samples in training and test sets do not overlap in subreddits, such performance indicates that there are strong associations between network structures and word survival such that our models can generalize across communities. The coefficients for the Cox model with 10 PCs are shown in Table 4. To interpret the LH
Table 3: Survival analysis results. All models outperform the concordance baseline.
Variables	Coef.	Exp(coef)	S.E.
PC1	-0.122***	0.885	0.002
PC2	-0.072***	0.930	0.002
PC3	0.170***	1.186	0.003
PC4	0.009***	1.001	0.001
PC5	-0.017***	0.984	0.001
PC6	-0.160***	0.852	0.001
PC7	-0.516***	1.675	0.002
PC8	-0.048***	0.953	0.001
PC9	-0.004***	1.004	0.001
PC10	-0.054***	0.947	0.002
Table 4: Results of the Cox model. All coefficients are highly significant. Exp(coef) refers to the hazard, or the probability of death. Lower Exp(coef) suggests that this variable is protective. S.E. refers to the standard error of the regression coefficients.
model with 10 PCs, we generate the survival function  $S(t|x)$  by varying a single feature from low to high but keep the remainder fixed at their median value (Figure 4). While the Cox model predicts the hazard (death rate) and the LH model predicts  $S(t|x)$  (the survival rate) (in reverse direction), we found that both models were highly consistent in assessing the input PCs, both in terms of relative weights and directions.
A large overall size (PC1) tends to preserve neologisms, as large communities provide a basic threshold population for words to be used. In addition to sheer size, global network topology also contributes to neologism survival. PC2, PC3, PC6 and PC7 correspond to three different network structures. PC3 represents networks that have many external connections but are split into multiple clusters within the community, which contributes negatively to the survival probabilities. In contrast, less clustered networks with dense edges and rich external connections (PC2) increase word survival rates. Both PC6 and PC7 boost word survival rate and they both represent networks that are relatively densely connected, but PC6 has high connections to many external communities and is more fragmented whereas PC7 is more isolated in the inter-community network (low degree centrality) but its external connections are influential communities (high Pagerank and Betweenness centrality). This may suggest that inter- and intra-community connections complement each other. In general, within a community, dense connections in the network keep words alive whereas local clusters in the network are adverse to word survival. In the multi-community landscape, more external connections tend to promote word survival.
Figure 3: The five highest weighted PCs used by the survival model. Inter-community features are indicated by orange bars. Adj.lc, Adj.gc and Adj.assort are local clustering coefficient, global clustering coefficient and assortativity adjusted to a random network. PC1 represents the overall size, PC2 the within-community connections and PC3 the inter-community connections. PC6 and PC7 are specific combinations of intra- and inter-community connections.
Figure 4: The contribution of predictors to the survival probability  $S(t|\mathbf{x})$  with remaining features fixed. Brighter regions indicate high survival rates.
7 Lexical levelling
Levelling refers to the gradual replacement of localized linguistic features (marked) by mainstream linguistic features (unmarked) over the whole community (Kerswill, 2003), which has been observed in a wide range of offline linguistic communities due to increasing mobility and external contacts (Milroy, 2002; Kerswill, 2003).
The subreddit communities have become increasingly inter-connected over time, as the average inter-community degree has increased from 6 in January 2008 to 2,323 in October 2018 (Figure 5). While some of these could be accounted for by the simultaneous growth in the number of subreddits, the growth in connectedness is also apparent. Such an increase of contact could promote the spread of neologisms across Reddit. In the same period, the number of variants that spread to more than  $60\%$  of the communities has grown slightly from 7 to 22. Some of the notable examples include words like lol, alot, imao and cuz. Meanwhile, the
variants that are only confined to one community grew rapidly from 1992 in 2008 to 23,397 in 2018. The widespread use of some neologisms does not necessarily cause the loss of local expressions, as in offline communities. Instead, the community-specific terms and community-general terms develop in tandem. Many community-specific terms are nested within topic-based communities with little meaning overlap with those widespread variants, and are therefore unlikely to be replaced by more general terms through levelling.
Figure 5 also shows that the probabilistic density distribution (PDF) of word dissemination (the percentage of communities sharing a neologism) conforms to the power law fit  $p(x) \propto x^{-\alpha}$ , as a few words spread to most communities while most words are confined to a few communities. Further, the shape parameter  $\alpha$  decreases asymptotically despite the growth of average inter-community degree (Figure 5), which implies that, as the size of Reddit grows, more community-specific words, as well as more widespread words, emerge.
Summary The number of community specific words grew rapidly despite increased intercommunity connectedness, which seems to go against the levelling trend observed in offline networks (Conde-Silvestre, 2012). In contrast to offline communities, these subreddit networks are of a different nature, as they are topic-based groups bounded by common interests. By joining these communities, users opt for fragmentation into some niche groups. Such segregation in topics and interests naturally brings in more community specific words. In other words, there is no strong evidence
Figure 5: [Left] Change of average community degree and the shape parameter  $\alpha$  of the power law fit  $p(x) \propto x^{-\alpha}$  over time. The average community degree is increasing, indicating that more communities are connected to each other. The  $\alpha$  is decreasing, suggesting that the tail of the distribution has become thicker, or more community specific words have emerged. [Right] Snapshots of PDFs of dissemination across communities over time.
for lexical levelling; instead, online communities go in the reverse direction, by developing more niche neologisms.
8 Discussions and Conclusions
In traditional sociolinguistics, weak ties within a social network have been linked to innovation and language change. Yet most studies only use indirect evidence to infer the underlying network types (Milroy and Milroy, 1985; Nevalainen, 2000; Dodsworth, 2019). Our quantitative analysis suggests that multiple structural properties play a role in lexical change. The overall network size is the most prominent factor in lexical innovation and survival, as large communities provide the base population to create and use those neologisms. The effect of network size has also been emphasized in other network studies of language (Reali et al., 2018; Raviv et al., 2019; Laitinen et al., 2020). However, sheer size is only part of the story, as dense edges between users, the lack of separate local clusters, and rich external connections also promote both lexical innovation and survival. Dense connections within and across communities increase the visibility of neologisms so that they can be imitated by other users, as exposure alone predicts users' information spreading behavior (Bakshy et al., 2012). In contrast, local clustering tends to separate networks into disconnected parts, slowing the spread of new words. These structural attributes are found to facilitate information spread in online social networks (Lerman and Ghosh, 2010). On a broader scale, our results suggest that the lexical change process in online social networks may be similar to other information spread processes (Guille et al., 2013).
Our results show that conclusions drawn from offline communities might be insufficient to account for behavior seen in online social networks. While the classic weak tie model emphasizes the role of loose social networks in language change (Milroy and Milroy, 1985; Nevalainen, 2000) and has been confirmed in online communities (Del Tredici and Fernandez, 2018), our work further extends this model by showing that a variety of network structural attributes also play a role in language change. Our quantitative analysis also suggests a different leveling process in online communities with implications for sociolinguistic theories.
Limitations and future work One limitation of this study is that topical variation is not explored in depth, because we aimed to look at the contributes of networks alone by smoothing out topical variation with diverse communities. Yet topics have been found to affect users' posting behavior in online communities (Mathew et al., 2019) and niche topics do affect word retention (Altmann et al., 2011). In Reddit, communities involving certain niche or foreign topics, such as r/pokemon, might inherently introduce more lexical innovations than others. Secondly, we only focus on Internet neologisms in Reddit. How these neologisms propagate across multiple social media platforms and how online and offline neologisms interact remain important questions to be addressed. Thirdly, while our study reveals the general patterns of lexical change, there are multiple sub-categories of neologisms such as discourse markers and name entities. It is of interest to ask whether different subcategories may exhibit different patterns of usage in online communities. These research questions are worth exploring in future work.
9 Ethical concerns
In terms of ethical concerns, a great number of low frequency neologisms collected from Urban Dictionary may be considered offensive to specific groups of populations. We collected the word usage data as they were in order to recover as realistic of a lexical landscape in Reddit as possible. However, these offensive words by no means reflect our values. Nor do we endorse the use of these words.
We thank Professor Patrice Beddor, Professor Will Styler, Julia Mendelsohn, Jiaxin Pei, Zuoyu Tian and Allison Lahnala for their comments on earlier versions of this draft. We are also grateful to all anonymous reviewers for their insightful comments, which helped improve this manuscript greatly. This material is based upon work supported by the National Science Foundation under Grant No 185022.
Eduardo G Altmann, Janet B Pierrehumbert, and Adilson E Motter. 2011. Niche as a determinant of word fate in online groups. *PloS one*, 6(5):e19009.
Laura Antolini, Patrizia Boracchi, and Elia Biganzoli. 2005. A time-dependent discrimination index for survival data. Statistics in Medicine, 24(24):3927-3944.
Lars Backstrom, Paolo Boldi, Marco Rosa, Johan Ugander, and Sebastiano Vigna. 2012. Four degrees of separation. In Proceedings of the 4th Annual ACM Web Science Conference, pages 33-42.
Eytan Bakshy, Itamar Rosenn, Cameron Marlow, and Lada Adamic. 2012. The role of social networks in information diffusion. In Proceedings of the 21st international conference on World Wide Web, pages 519-528.
David K Barnhart. 2007. A calculus for new words. Dictionaries: Journal of the Dictionary Society of North America, 28(1):132-138.
Abhinav Bhandari and Caitrin Armstrong. 2019. Tkol, httt, and r/radiohead: High affinity terms in Reddit communities. In Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019), pages 57-67, Hong Kong, China. Association for Computational Linguistics.
Jonathan Chang and Cristian Danescu-Niculescu-Mizil. 2019. Trajectories of blocked community members: Redemption, recidivism and departure. In *The World Wide Web Conference*, pages 184–195.
Jonathan P Chang, Caleb Chiam, Liye Fu, Andrew Z Wang, Justine Zhang, and Cristian Danescu-Niculescu-Mizil. 2020. Convokit: A toolkit for the analysis of conversations. In Proceedings of SIG-DIAL.
Morten H Christiansen and Simon Kirby. 2003. Language evolution: Consensus and controversies. Trends in cognitive sciences, 7(7):300-307.
Juan Camilo Conde-Silvestre. 2012. The role of social networks and mobility in diachronic sociolinguistics. The Handbook of Historical Sociolinguistics, pages 332-352.
Tiago Cunha, David Jurgens, Chenhao Tan, and Daniel Romero. 2019. Are all successful communities alike? characterizing and predicting the success of online communities. In *The World Wide Web Conference*, pages 318-328.
Cristian Danescu-Niculescu-Mizil, Robert West, Dan Jurafsky, Jure Leskovec, and Christopher Potts. 2013. No country for old members: User lifecycle and linguistic change in online communities. In Proceedings of the 22nd international conference on World Wide Web, pages 307-318. ACM.
Marco Del Tredici and Raquel Fernandez. 2018. The road to success: Assessing the fate of linguistic innovations in online communities. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1591-1603, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Robin Dodsworth. 2019. Bipartite network structures and individual differences in sound change. *Glossa: a journal of general linguistics*, 4(1).
Penelope Eckert and Sally McConnell-Ginet. 1992. Think practically and look locally: Language and gender as community-based practice. Annual review of anthropology, 21(1):461-488.
Jacob Eisenstein, Brendan O'Connor, Noah A Smith, and Eric P Xing. 2014. Diffusion of lexical change in social media. *PloS one*, 9(11).
Rahul Goel, Sandeep Soni, Naman Goyal, John Paparrizos, Hanna Wallach, Fernando Diaz, and Jacob Eisenstein. 2016. The social dynamics of language change in online networks. In International Conference on Social Informatics, pages 41-57. Springer.
Mark Granovetter. 1983. The strength of weak ties: A network theory revisited. Sociological theory, pages 201-233.
Mark S Granovetter. 1977. The strength of weak ties. In Social networks, pages 347-367. Elsevier.
Jack Grieve, Andrea Nini, and Diansheng Guo. 2017. Analyzing lexical emergence in modern american english online 1. English Language & Linguistics, 21(1):99-127.
Adrien Guille, Hakim Hacid, Cecile Favre, and Djamel A Zighed. 2013. Information diffusion in online social networks: A survey. ACM Sigmoid Record, 42(2):17-28.
William L Hamilton, Justine Zhang, Cristian Danescu-Niculescu-Mizil, Dan Jurafsky, and Jure Leskovec. 2017. Loyalty in online communities. In Eleventh International AAAI Conference on Web and Social Media.
Janet Holmes and Miriam Meyerhoff. 1999. The community of practice: Theories and methodologies in language and gender research. *Language in society*, 28(2):173-183.
Daniel Kershaw, Matthew Rowe, and Patrick Stacey. 2016. Towards modelling language innovation acceptance in online social networks. In Proceedings of the Ninth ACM International Conference on Web Search and Data Mining, pages 553-562.
Paul Kerswill. 2003. Dialect levelling and geographical diffusion in British english. Social dialectology: in honour of Peter Trudgill, pages 223-243.
Colin Klein, Peter Clutton, and Adam G Dunn. 2019. Pathways to conspiracy: The social and linguistic precursors of involvement in reddit's conspiracy theory forum. *PloS one*, 14(11):e0225098.
David G Kleinbaum and Mitchel Klein. 2010. Survival analysis. Springer.
Havard Kvamme, Ornulf Borgan, and Ida Scheel. 2019. Time-to-event prediction with neural networks and cox regression. Journal of Machine Learning Research, 20(129):1-30.
Havard Kvamme and Ornulf Borgan. 2019. Continuous and discrete-time survival prediction with neural networks. arXiv preprint arXiv:1910.06724.
William Labov. 2007. Transmission and diffusion. Language, 83(2):344-387.
Mikko Laitinen, Masoud Fatemi, and Jonas Lundberg. 2020. Size matters: Digital social networks and language change. Frontiers in Artificial Intelligence, 3:46.
Kristina Lerman and Rumi Ghosh. 2010. Information contagion: An empirical study of the spread of news on Digg and Twitter social networks. In Proceedings of 4th International Conference on Weblogs and Social Media (ICWSM), 2010.
Gary Lupyan and Rick Dale. 2010. Language structure is partly determined by social structure. *PloS one*, 5(1).
Binny Mathew, Ritam Dutt, Suman Kalyan Maity, Pawan Goyal, and Animesh Mukherjee. 2019. Deep dive into anonymity: Large scale analysis of Quora questions. In International Conference on Social Informatics, pages 35-49. Springer.
Allan A Metcalf. 2004. Predicting new words: The secrets of their success. Houghton Mifflin Harcourt.
James Milroy and Lesley Milroy. 1985. Linguistic change, social network and speaker innovation. Journal of Linguistics, 21(2):339-384.
Lesley Milroy. 2002. Introduction: Mobility, contact, and language change—working with contemporary speech communities. Journal of Sociolinguistics, 6(1):3-15.
Lesley Milroy and Carmen Llamas. 2013. Social networks. The Handbook of Language Variation and Change, pages 407-427.
Lesley Milroy and James Milroy. 1992. Social network and social class: Toward an integrated sociolinguistic model. Language in society, 21(1):1-26.
Terttu Nevalainen. 2000. Mobility, social networks and language change in Early Modern England. European Journal of English Studies, 4(3):253-264.
Mitchell G Newberry, Christopher A Ahern, Robin Clark, and Joshua B Plotkin. 2017. Detecting evolutionary forces in language change. Nature, 551(7679):223-226.
Bill Noble and Raquel Fernandez. 2015. Centre stage: How social network position shapes linguistic coordination. In Proceedings of the 6th Workshop on Cognitive Modeling and Computational Linguistics, pages 29-38, Denver, Colorado. Association for Computational Linguistics.
Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. 1999. The pagerank citation ranking: Bringing order to the web. Technical report, Stanford InfoLab.
John C Paolillo. 1999. The virtual speech community: Social network and language variation on irc. In Proceedings of the 32nd Annual Hawaii International Conference on Systems Sciences. 1999. HICSS-32. Abstracts and CD-ROM of Full Papers, pages 10–pp. IEEE.
Michał B. Paradowski and Łukasz Jonak. 2012. Diffusion of linguistic innovation as social coordination. Psychology of Language and Communication, 16(2):131-142.
Limor Raviv, Antje Meyer, and Shiri Lev-Ari. 2019. Larger communities create more systematic languages. Proceedings of the Royal Society B, 286(1907):20191262.
Florecia Reali, Nick Chater, and Morten H Christiansen. 2018. Simpler grammar, larger vocabulary: How population size affects language. Proceedings of the Royal Society B: Biological Sciences, 285(1871):20172586.
Thomas M Schwen and Noriko Hara. 2003. Community of practice: A metaphor for online design? The Information Society, 19(3):257-270.
Devyani Sharma and Robin Dodsworth. 2020. Language variation and social networks. Annual Review of Linguistics, 6.
Ian Stewart and Jacob Eisenstein. 2018. "fetch" happen: The influence of social and linguistic context on nonstandard word growth and decline. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4360-4370, Brussels, Belgium. Association for Computational Linguistics.
Chenhao Tan and Lillian Lee. 2015. All who wander: On the prevalence and characteristics of multi-community engagement. In Proceedings of the 24th International Conference on World Wide Web, pages 1056-1066.
Justine Zhang, William L Hamilton, Cristian Danescu-Niculescu-Mizil, Dan Jurafsky, and Jure Leskovec. 2017. Community identity and user engagement in a multi-community landscape. In Eleventh International AAAI Conference on Web and Social Media.
A Replicability
We take measures to ensure the replicability of our study. Some of the validation results are presented in the following supplementary materials. The following resources can be used to replicate the current study.
- Our code for preprocessing and analysis as well as the preprocessed data can be found at: https://github.com/lingjzhu/reddit_network.
- The original Reddit data can be retrieved from the following corpus (about 2TB): https://convokit.cornell.edu/documentation/subreddit.html.
- The list of neologisms was collected from the Urban Dictionary and NoSlang.com. Warning: the following two sites may contain offensive content. https://www.urbandictionary.com/ https://www.noslang.com/.
- The network construction was carried out using the networkx while the feature extraction was through the networkit packages: https://networkx.org/ https://networkit.github.io/.
- The statistical tests were implemented with Pingouin and predictive models with sklearn.
  - https://pingouin-stats.org/
  - https://scikit-learn.org/stable/
- The deep survival analysis was implemented using the pycox package: https://github.com/havakv/pycox.
- The baseline cox model was implemented using the lifelines package: https://lifelines.readthedocs.io/en/latest/Survival%20Regression.html.
B The Reddit Network Corpus
The detailed information of the Reddit Network Corpus is given in this section. The code and data will soon be released to the public.
B.1 Neologisms