SlowGuess/NAACL_2021_OCR huggingface dataset:
Table 1 shows some samples of most frequent and least frequent neologisms in Reddit. These linguistic innovations were collected from NoSlang.com and Urban Dictionary. We filtered out lexical entries that: 1) span more than one word, 2) can be found as an entry in an English dictionary after lemmatization, 3) are identified as person names, 4) contain non-alphabetical characters, numbers or emojis and 5) do not show up in our Reddit dataset.
We set loose criteria for word inclusion. Many of the frequent neologisms have already been incorporated into the daily lexicon, such as wiki, google and instagram. We manually filtered out these words in our wordlist and the number of such words is less than 100. We also keep typos in the curated list, as these words often carry special meanings. For example, alot, atleast and recieve are the typos that are used more than 1 million times, so frequent that they carry some special meanings and functions such as identity assertion.
After automatic filtering, we manually inspected the 5000 most frequent words with greater care so as to filter out some invalid entries. In addition, we also sampled a few hundred words at different frequency bins for close inspection. For the rest of the words, we only scanned through them for a quick sanity check.
C Additional validation of the networks
C.1 Intra-community networks
We constructed the network representations of Reddit communities with the same method as that used by Hamilton et al. (2017) and Del Tredici and Fernandez (2018), so that our study is consistent and comparable with previous works. The rationale behind this setting is that "two users who comment in such proximity interacted with each other, or at least directly with the same material" (Hamilton et al., 2017).
Here we compare the inter-community networks in our study with two types of baseline networks extracted from the same Reddit communities. We randomly sampled 100 networks from our data and created the following two baseline networks.
- DRG: The Direct-reply Graph (DRG) was constructed by treating every user as a node. An edge was created between two users if one user directly replied to the other. This network
Variables	Kendall τ correlations
TG	DRG
# Nodes	1 (0.0)	1 (0.0)
# Edges	0.81 (0.27)	0.957 (0.05)
density	0.50 (0.34)	0.928 (0.08)
assortativity	0.04 (0.26)	0.189 (0.25)
local clustering	0.40 (0.18)	0.21 (0.36)
global clustering	0.07 (0.29)	0.56 (0.25)
average degree	0.54 (0.30)	0.84 (0.14)
max degree	0.75 (0.12)	0.66 (0.22)
min degree	0.34 (0.03)	0.44 (0.09)
LCC %	0.60 (0.19)	0.55 (0.19)
Singletons %	0.53 (0.22)	0.49 (0.22)
could underestimate the user interactions as users are likely to read nearby posts in the same comment chain when replying.
- TG: The Thread Graph (TG) was constructed by setting each user as a node and two users were connected by an edge if they had commented in the same thread. This network might overestimate the user interactions because in some mega threads that span hundreds or thousands of posts, users might not interact with all the people in the same thread but only with nearby users.
As these two baseline networks might either underestimate or overestimate the connections, we used these two networks to provide an estimate of the possible errors of our networks.
The results are presented in Table 5. Despite the different settings, most of the network parameters have correlations ranging from moderate to strong. But the correlations for assortativity and clustering coefficients are weaker. However, TG is not considered a good indication of the connections in Reddit as users are unlikely to interact with all users in a long thread. DRG and our networks are more similar to each other. Hamilton et al. (2017) had noted that changing the original networks to DRG did not significantly change their analysis results of Reddit networks.
Table 5: Correlations between two baseline intracommunity networks and the intra-community networks used in our study. The reported numbers are mean correlations with standard deviations inside the bracket.
Centrality	Kendall τ correlations
Threshold: 3	Threshold: 4
Betweenness	0.62 (0.26)	0.48 (0.37)
Closeness	0.64 (0.17)	0.50 (0.25)
Degree	0.85 (0.05)	0.79 (0.06)
Eigenvector	0.92 (0.03)	0.88 (0.05)
Pagerank	0.90 (0.04)	0.86 (0.06)
Table 6: Correlations between two baseline intercommunity networks and the inter-community networks used in our study. The reported numbers are mean correlations with standard deviations inside the bracket.
C.2 Inter-community networks
In order to validate our approach to construct the inter-community graph, we constructed different inter-community graphs by setting the posting threshold of active users to 2, 3 and 4. One concern is that setting the threshold too low ( $>=1$ ) results in extremely dense graphs, which are challenging to process.
After extracting the network features from these networks, we compared them by computing the Kendall rank correlation coefficients between these features. The results in Table 6 show that these networks are highly correlated in structural features, especially for degree, eigenvector and pagerank centralities. The correlations for betweenness and closeness are more unstable but still moderately correlated. So adjusting the threshold does not significantly bias our results qualitatively.
C.3 Network statistics
Table 7 provides some general statistics of the whole Reddit Network Corpus.
Total
Months	152
Subreddits	4420
Inter Networks	152
Intra Networks	289170
Users	>50 millions
Neologisms	80071
Table 7: Summary of the whole corpus.
The average duration of the 4420 subreddit communities is 65 months. Statistical summaries of all 289170 networks are presented in Table 8.
Median	Inter-quartile range
Nodes	765	[351, 1776]
Density	0.0059	[0.0025, 0.0121]
Average Degree	4.19	[3.04, 6.29]
Largest Connected Component	88.6%	[81.9%, 92.4%]
Singletons	9.4%	[6.5%, 13.9%]
Inter-Community Degree	1351	[492, 2322]
D Correlations between variables
For empirical networks, some network attributes are often correlated. Here we present the correlation matrix between variables used in innovation prediction in Figure 6 for illustration. The correlation matrix for features in survival analysis also exhibit a similar pattern of correlations.
E Predicting lexical innovations
E.1 Feature preprocessing
We used mean-variance normalization to normalize all prediction features. Since the distribution of some features were highly skewed, before normalization, we log-transformed the following intracommunity features: number of nodes, number of edges, density, average degree, maximum degree, and the following inter-community features: degree centrality, closeness centrality, Pagerank centrality, betweenness centrality, eigenvector centrality. The rest of the features were directly normalized. Whether to perform log-transformation was determined by visual inspection of the density plot. A small number  $10^{-6}$  was added before taking the logarithm to improve numerical stability. We found that such a practice improved the performance during cross-validation relative to directly normalizing all features.
The following features were used to predict the number of innovations per-month. Some of the features were correlated and the correlations varied from weak to strong.
- Inter: degree centrality, closeness centrality, Pagerank centrality, betweenness centrality, eigenvector centrality
- Intra.: number of nodes, number of edges, density, average degree, maximum degree, proportion of the largest connected components, proportion of singletons, adjusted as
Table 8: Statistical Summary of 289170 Subreddit networks.
Variables	Coefficients
PC1	-0.877
PC2	-0.195
PC3	0.193
PC4	-0.024
PC5	-0.003
Table 9: [Predicting innovations] Coefficients for the Poisson regression with first 5 PCs.
sortativity, adjusted transitivity, adjusted clustering coefficients.
Then PCA with whitening was applied to decompose all of the features into principal components. We did consider the delta features, which were the change in these variables with respect to the last month. However, these added temporal features did not improve the performance. So we assumed that changes in each month might not be highly relevant.
All models were implemented in sklearn. The baseline was the mean number of innovations across all time and all subreddits as the prediction. For the rest of models, we performed ten-fold cross-validation to select the best parameters. After parameter selection, the regularization parameter for the Poisson regression was  $10^{-2}$  and the maximum number of iterations was 300. For the histogram based gradient boosting trees, the maximum number of split was set to 256 and the loss was the Poisson loss. Otherwise we kept the default hyperparameters.
The data were partitioned into training and test sets with a ratio of  $90\% / 10\%$ . We ran each model 20 times with a different random partition each time. The resulting metrics were averaged across the metrics obtained from the test sets over 20 runs.
Figure 6: The correlation matrix between variables. Correlation coefficients were computed using the Spearman correlation, as some variables are related in log-linear relations. Variables in dark are within-community features while variables in red are inter-community features.
Figure 7: [Predicting innovations] Distribution of observed innovations in test samples (left) and predicted distributions by Poisson regression with all network features (middle) and by HGBT with all network features (right). Both models well approximate the empirical distribution of lexical innovation counts but fall short of predicting the trailing long tail.
F Deep survival analysis
In this section, we describe the details of deep survival analysis.
F.1 Model specification
We adopted the Logistic Hazard model developed in following works (Kvamme and Borgan, 2019; Kvamme et al., 2019). The original derivation comes from Kvamme and Borgan (2019).
In survival analysis, given a set of discrete time steps  $T = \{t_1, t_2, \dots, t_n\}$  and the event time  $t^*$ , the goal is to estimate the probability mass distribution (PMF) of the event time  $f(t)$  and the survival function  $S(t)$ .
$$
f (t) = P (t ^ {*} = t _ {i}),
$$
$$
S (t) = P \left(t ^ {*} > t _ {i}\right) = \sum_ {j > i} f \left(t _ {j}\right) \tag {2}
$$
The model can also be expressed as the hazard function  $h(t)$ .
$$
\begin{array}{l} h (t) = P \left(t ^ {*} = t _ {i} \mid t ^ {*} > t _ {i - 1}\right) \\ = \frac {f \left(t _ {i}\right)}{S \left(t _ {i - 1}\right)} \tag {3} \\ = \frac {S (t _ {i - 1}) - S (t _ {i})}{S (t _ {i - 1})} \\ \end{array}
$$
With the above equations, the survival function can be rewritten as follows.
$$
f \left(t _ {i}\right) = h \left(t _ {i}\right) S \left(t _ {i - 1}\right) \tag {4}
$$
$$
S (t _ {i}) = [ 1 - h (t _ {i}) ] S (t _ {i - 1})
$$
$$
S \left(t _ {i}\right) = \prod_ {k = 1} ^ {i} \left[ 1 - h \left(t _ {k}\right) \right] \tag {5}
$$
For each individual  $i$ , the likelihood function can be formulated as
$$
L _ {i} = f \left(t _ {i}\right) ^ {d _ {i}} S \left(t _ {i}\right) ^ {1 - d _ {i}} \tag {6}
$$
The above equation can be rewritten with respect to the hazard function.
$$
\begin{array}{l} L _ {i} = f \left(t _ {i}\right) ^ {d _ {i}} S \left(t _ {i}\right) ^ {1 - d _ {i}} \\ = \left[ h \left(t _ {i}\right) S \left(t _ {i - 1}\right) \right] ^ {d _ {i}} \left(\left[ 1 - h \left(t _ {i}\right) \right] S \left(t _ {i - 1}\right)\right) ^ {1 - d _ {i}} \\ = h \left(t _ {i}\right) ^ {d _ {i}} \left[ 1 - h \left(t _ {i}\right) \right] ^ {1 - d _ {i}} S \left(t _ {i - 1}\right) \\ = h \left(t _ {i}\right) ^ {d _ {i}} \left[ 1 - h \left(t _ {i}\right) \right] ^ {1 - d _ {i}} \prod_ {k = 1} ^ {i - 1} \left[ 1 - h \left(t _ {k}\right) \right] \tag {7} \\ \end{array}
$$
The loss function is negative log likelihood function, the negative of the sum of  $\log (L_i)$  over all samples. After some algebraic operations, the loss function of the Logistic Hazard model can be formulated as the common binary cross-entropy function.
$$
\begin{array}{l} \mathcal {L} = - \frac {1}{n} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {T} \left(y _ {i j} \log \left[ h \left(t _ {j} \mid \mathbf {x} _ {i}\right) \right] \right. \tag {8} \\ \left. + \left(1 - y _ {i j}\right) \log \left[ 1 - h \left(t _ {j} \mid \mathbf {x} _ {i}\right) \right]\right) \\ \end{array}
$$
where  $y_{ij}$  is the binary event indicator for sample  $i$  at time  $t$ .
Let  $\mathbf{x}$  be an input feature vector and  $\phi (\mathbf{x})\in$ $\mathbb{R}^h$  is the neural network that transforms input  $\mathbf{x}$  into  $h$  output vectors. Each output vector corresponds to a discrete time step such that  $\phi (\mathbf{x}) =$ $\{\phi_1(\mathbf{x}),\phi_2(\mathbf{x}),\dots ,\phi_h(\mathbf{x})\}$  . The hazard function then can be approximated by the sigmoid function.
$$
h \left(t _ {i} \mid \mathbf {x}\right) = \frac {1}{1 + \exp \left[ - \phi_ {i} (\mathbf {x}) \right]} \tag {9}
$$
F.2 Implementation
Models of deep survival analysis were implemented using the package pycox (Kvamme et al., 2019). The network features were normalized and partitioned in the same way as described in Section E.1.
The actual survival time for these neologisms varied from 3 to 152 months. First, we discretized the survival time measured in actual months into 100 intervals based on the distribution of the event times, with the assumption that each interval has the same decrease of the survival probability. The resulting grid was denser during months with more event times and sparser during months with fewer event times. Such a practice is recommended by Kvamme and Borgan (2019), as it reduces parameters and stabilizes training.
We trained a three-layered Logistic Hazard model. For each of the first two layers, we used a linear layer with 256 hidden dimensions and ReLU activation function, followed by batch normalization and a dropout with a probability of 0.1. The last layer was a linear layer with output dimension of 100 followed by a sigmoid activation function.
During training, we used the Adam optimizer with a learning rate of 0.001 and a batch size of 2048 samples. All hyperparameters were tuned with a simple grid search on the development set. Each model was trained for 5 epochs and was run
Figure 8: [Predicting survival] Remaining PCs for the network features. Inter-community features are highlighted in orange bars.
10 times with different random seeds and different partitions of data each time. The performance metrics were averaged over all 10 runs. These models were trained on a Nvidia V100 GPU and each run took about less than a minute to complete.
In each run, the data were randomly partitioned into around  $80\%$ ,  $10\%$  and  $10\%$  portions as training, development and test sets with different random seeds. In order to avoid information leaking, we ensured that samples in these three sets were from distinct subreddits.
F.3 Baseline models
We also ran baseline Cox's proportional hazard models (Kleinbaum and Klein, 2010) with the same data partitions and discretization scheme. The Cox's model estimates the hazard function  $h(t_i|\mathbf{x})$  with the following equations.
$$
h \left(t _ {i} | \mathbf {x}\right) = \beta_ {0} \left(t _ {i}\right) \cdot \exp \left(\sum_ {i = 1} ^ {n} \beta_ {i} (\mathbf {x} - \overline {{\mathbf {x}}})\right) \tag {10}
$$
We ran the model ten times and report the average performance. All baseline Cox's models were implemented using the CoxPHFitter function via the package lifelines.
F.4 Additional results of PCA
The additional results of PCA are shown in Figure 8.
F.5 Additional results of deep survival analysis
The additional results are shown in Figure 9.
Figure 9: [Predicting survival] The contribution of the predictors to the survival probability  $S(t|\mathbf{x})$  with remaining features fixed.