SlowGuess/NAACL_2022_OCR huggingface dataset:
Felix Drinkall*, Stefan Zohren*†, Janet B. Pierrehumbert*‡
*Department of Engineering Science, University of Oxford
†The Alan Turing Institute
‡Faculty of Linguistics, University of Oxford
felix.drinkall@eng.ox.ac.uk
We present a novel approach incorporating transformer-based language models into infectious disease modelling. Text-derived features are quantified by tracking high-density clusters of sentence-level representations of Reddit posts within specific US states' COVID-19 subreddits. We benchmark these clustered embedding features against features extracted from other high-quality datasets. In a threshold-classification task, we show that they outperform all other feature types at predicting upward trend signals, a significant result for infectious disease modelling in areas where epidemiological data is unreliable. Subsequently, in a time-series forecasting task we fully utilise the predictive power of the caseload and compare the relative strengths of using different supplementary datasets as covariate feature sets in a transformer-based time-series model.
Many papers have shown that web search data can be used to forecast the spread of infectious diseases (Lampos et al., 2017), (Lampos et al., 2021), (McDonald et al., 2021), (Reinhart et al., 2021), (Alruily et al., 2022). Alongside this literature, social media has been exploited for its predictive potential in several other fields such as quantitative finance Xu and Cohen (2018), Sawhney et al. (2020), logistics forecasting Ni et al. (2017) and election forecasting (Bermingham and Smeaton, 2011), (Huberty, 2015). Research conjoining these two strands has produced results showing that social media can help predict rises in disease caseloads. Iso et al. (2016) and Samaras et al. (2020) both used pre-defined keywords in order to predict outbreaks of influenza; words such as "Influenza", "fever", "headache" were selected a-priori. These papers assume that useful feature sets have no geographical variation and use the same features regardless of the regional social dynamics; they also assume that useful features are limited to words that refer to
Figure 1: HDBSCAN clusters of the SBERT-NLI-STSb-base representations of r/CoronavirusWA posts made at 50 dimensions but reduced to 2 for visualisation.
symptoms. To address these limitations, Drinkall and Pierrehumbert (2021) set more general and objective inclusion criteria. For each of four US state COVID-19 subreddits, all words over-represented in that US state's COVID-19 subreddit compared to the rest of Reddit were considered to be potential keywords for forecasting. The most informative keywords proved to be highly dependent on the target state, and included many that did not refer to symptoms. However, the paper still relied on static word counts that miss more complex information as the discussion unfolds over time. The present paper extracts more informative features from social media data and, to our knowledge, is the first work to incorporate modern NLP techniques in this setting.
New transformer-based language models (Devlin et al., 2019), (Yang et al., 2019), (Liu et al., 2019) provide the potential for identifying more informative features for infectious disease forecasting, and using them in a more effective manner. This paper uses transfer learning and clustering algorithms to isolate useful features for predicting COVID-19 caseloads. We first pilot-tested a straightforward way to exploit transformer-based language models for the task: the caseload target value was encoded alongside each post in a sequence classification framework. Trained using
Human Language Technologies, pages 1471 - 1484
historical data, this approach generates a prediction from every post, and the results are aggregated for an overall prediction. This method performed very poorly because of noise introduced by irrelevant posts, and we do not discuss it further here (see Appendix C). To achieve better performance, we developed a novel feature identification technique that filters out unrelated posts and generates informative features using high-density clusters of posts within a subreddit's embedding space.
Our work builds off Sia et al. (2020) and Thompson and Mimno (2020) who demonstrated that clusters of contextualised word embeddings are a good basis for topic modelling. In a similar vein, Aharoni and Goldberg (2020) showed that the domain type of a particular text could be identified using the clustering of sentence-level representations. Finally, Rother et al. (2020) showed that clusters of contextualised embeddings could detect meaning shifts in words. The success of these papers motivates our use of high-density clusters of sentence-level representations.
The present paper shows that our novel feature sets outperform more traditional methods by comparing our results to those in Drinkall and Pierrehumbert (2021) in a threshold-classification task. This task provides an understanding of which feature sets provide the most informative trend signals at different caseload growth rates, enabling us to understand the effectiveness of a particular feature type at identifying a distinct epidemiological event. Strong performance on this task is relevant in forecasting worst-case scenarios like hospital overflow, where the outcome is a binary variable.
The caseload information is not fully utilised in the threshold-classification task. This observation motivates a time-series forecasting task to compare feature sets at predicting a more continuous target. Feature selection is a crucial step in time-series modelling (Wang et al., 2013), (Sun et al., 2015); adding extraneous features to a multivariate prediction can result in performance deterioration as the models get more complex, a fact that inspired L1-regularisation. Only highly relevant features, which represent complementary information, improve performance.
Contributions. We introduce a novel unsupervised method for predicting COVID-19 trend signals and forecasting caseloads. We show that sole use of our feature set achieves very high accuracy in trend signal prediction, a significant result for
infectious disease modelling in regions where other reported data is unavailable or unreliable.
Comparing our Reddit features' performance against other open-source geographically-specific datasets allows us to understand their value. The following data sources were used to create the feature sets in this paper:
Pushshift API - The Pushshift API (Baumgartner et al., 2020) is used to compile datasets of target subreddits to create the Reddit features. The Pushshift API provides data on every comment and submission posted on Reddit. This paper uses comments to form the subreddit dataset since there are more comments than submissions, and they constitute more conversational and reactionary discourse. No individual comments or users are reported in this paper to observe the anonymity of the users. Update frequency: real-time.
COVID-19 Tracking Project - The state-level COVID-19 epidemiological data is provided by the COVID-19 Tracking Project<sup>1</sup> to create the prediction target and is also used as a feature set in baseline predictions. Update frequency: 24 hours. Start date: 13/01/2020.
Oxford COVID-19 Government Response Tracker (OxCGRT) - The OxCGRT (Hale et al., 2020) defines the local government response. The data covers policies including health, containment and economic measures, and overall stringency scores. Update frequency: "continuously" but can be variable due to human data collection; daily periodicity. Start date: 01/01/2020.
Google's COVID-19 Community Mobility Reports (GCCMR)  $^{2}$  - The GCCMR provides local movement data in different area types such as parks, workplaces, etc. and has been used to successfully predict COVID-19 caseloads (Wang et al., 2020), (Ilin et al., 2021). The data is freely available for the duration of the ongoing pandemic. Update frequency: 2-3 days. Start date: 15/02/2020.
3 Feature identification
Social media is a complex and noisy data source, requiring significant processing to isolate meaningful predictive features. The pipeline used in this paper consists of three main steps for feature
<sup>1</sup>https://covidtracking.com
$^{2}$ https://www.google.com/covid19/mobility/
identification: sentence-level encoding, dimensionality reduction and clustering. This process groups together Reddit comments that are semantically similar. Following these main steps that are outlined below, the Reddit features are reduced further to 25 using a chi-squared test. Once these 25 high-density clusters are identified, the daily counts of comments within these clusters are used as features in the evaluation frameworks in Sections 4 & 5.
3.1 Sentence-level representation
A common technique for identifying sentence representations is to take the average-pooled BERT hidden-state embedding (Aharoni and Goldberg, 2020); however, papers such as Reimers and Gurevych (2019) have shown that the average-pooled BERT embeddings are a relatively poor way of encoding sentences and advocate for further fine-tuning to produce a more semantically meaningful embedding. In Reimers and Gurevych (2019), the best results are achieved by training the language model on Natural Language Inference (NLI) (Bowman et al., 2015), (Williams et al., 2018) and Sentence Textual Similarity (STS) (Cer et al., 2017) data. The NLI data contains many sentence pairs with their semantic relationship labelled. The STS data provides a semantic relatedness score between 0-5. It is possible to use both datasets to fine-tune the language model using both dataset types by manipulating the objective functions. The NLI data is trained using a classification objective function, and the STS data is trained using a regression objective function. Reimers and Gurevych (2019) shows that averaging the final layer BERT embeddings leads to a Spearman rank correlation  $\rho$  between the cosine similarity of the sentence representations and the actual labels of the STS data of around  $\rho = 54.81$ , whereas SBERT-NLI-STSb-base achieves  $\rho = 88.31$ .
For this paper, there is no domain-specific training. The SBERT-NLI-STSb-base, SRoBERTa-NLISTSb-base and SDistilBERT-NLI-STSb-base encode the Reddit posts with no further fine-tuning.
3.2 Dimensionality reduction
The language models specified in Section 3.1 have a dimensionality of 768, which means that their embedding space is very sparse, making it challenging to find dense clusters. Lowering the embedding dimensionality is consistent with the findings in Sia et al. (2020), who show that the dimensionality of the embeddings can be reduced by  $\sim 80\%$  and still
maintain the topic modelling coherence. Therefore, in line with these findings, the dimensionality of the embedding space is reduced to 50.
UMAP (Uniform Manifold Approximation and Projection for Dimension Reduction) is used as in Rother et al. (2020) to lower the dimensionality of the embedding space. UMAP is appropriate for this task since it preserves global structure better than other manifold learning dimensionality reduction methods such as t-SNE (McInnes et al., 2018) (McConville et al., 2021). UMAP's preservation of global structure has been shown in Reif et al. (2019) to produce clear clusters related to different word senses. It is tested against a PCA algorithm in Appendix B on the Threshold-Classification task outlined in Section 4. The results justify its use as it outperforms PCA when used in conjunction with the best performing clustering algorithm.
3.3 Clustering
For this paper, the HDBSCAN algorithm (Campello et al., 2013) is used for clustering due to the complex structure of the subreddit embedding space. The benefit of using a density-based clustering algorithm is that sparse areas are not fitted into clusters, removing a significant source of noise from the prediction.
HDBSCAN offers an advantage over other density-based clustering algorithms; the cut-off density that characterises the edge of the clusters is non-constant and defined by a stability metric that rewards large and dense clusters. This stability metric is calculated from the data points' Minimum Spanning Tree (MST). The following equation defines the stability of cluster  $C_i$ :
$$
S \left(C _ {i}\right) = \sum_ {x _ {j} \in C _ {i}} \left(\lambda_ {\max } \left(x _ {j}, C _ {i}\right) - \lambda_ {\min } \left(C _ {i}\right)\right) \tag {1}
$$
Here  $\lambda$  represents the density statistic:  $\lambda = 1 / \epsilon$  where  $\epsilon$  is equal to the distance between points on the MST. In this equation,  $\lambda_{max}(x_j,C_i)$  is the density at which the point  $x_{j}$  would fall out of the cluster  $C_i$ , and  $\lambda_{min}(C_i)$  is the minimum density threshold at which the cluster still exists.
Clusters with maximum stability are used as the final clusters, and points that fall out of these clusters are discarded. New data points can subsequently be added to the cluster by identifying where they fall in the MST. A point is treated as noise unless it can be grouped into a cluster larger than
min_cluster_size, which, for this paper, we have set at 25 so that the clusters are not too small and the resulting features are not too sparse. Removing noisy comments from the clusters is shown in Appendix B to have performance benefits over other clustering algorithms that do not reject comments: we have compared HDBSCAN to a Spherical K-Means (KM) algorithm and a Gaussian Mixture Model (GMM), two popular algorithms within the literature base.
4 Threshold-Classification Framework
The threshold-classification framework (henceforth Threshold task) uses the same evaluation methodology as in Drinkall and Pierrehumbert (2021). The problem is presented as a classification task on balanced classes, with a randomised train/test split and test size of 0.25 on data from 07/03/2020 to 17/01/2021. Balanced classes allow us to report accuracy as the performance metric for this task. The feature sets, derived from a 7-day moving average of the datasets in Section 2, are concatenated to a target value that encodes whether the caseload increase exceeded the threshold within a given time interval. The threshold is defined by a relative increase,  $\delta_r(t)$ :
$$
\delta_ {r} (t) = \frac {\mu (t + \tau) - \mu (t)}{\mu (t)} \tag {2}
$$
Where  $\mu (t)$  is the 7-day moving average of the caseload, and  $\tau$  is the prediction horizon.
The model used for classification is a Random Forest (RF) (Breiman, 2001). The advantage of using an RF model over other tree-based models is that it decorrelates the trees, making it robust to correlated feature sets. Social media data is highly correlated as overall take-up surges and wains; therefore, robustness to correlated features is critical. Of course, many more complex models would likely outperform an RF model; however, given that the goal of this task is to compare feature sets, the increased transparency that an RF model offers over more complex models justify its use.
Each data type is used in isolation to predict the target labels so that the implicit information within each feature set can be compared.  $T_{DisB}$ ,  $T_{BERT}$  and  $T_{RoB}$  correspond respectively to the features extracted using the methodology above
Feature set	Average	7D	14D	21D	28D
T RoB ++	.875	.895	.880	.849	.874
T BoW ++	.810	.836	.809	.805	.791
T RoB	.803	.845	.792	.789	.787
TBERT	.789	.821	.798	.780	.761
TDisB	.780	.808	.771	.774	.768
TBoW	.768	.816	.755	.749	.753
TKW	.633	.733	.628	.591	.580
M	.702	.703	.691	.713	.698
G	.702	.713	.710	.695	.691
P	.545	.516	.549	.557	.557
C	.555	.651	.536	.529	.503
Table 1: The average performance across all relative thresholds and states at different prediction horizons. The features are:  $T_{<\text{language model}>}$  → our features;  $T_{BoW}$  → Drinkall and Pierrehumbert (2021) features;  $M$  → GCCMR data;  $G$  → OxCGRT data;  $P$  → daily post count;  $C$  → current caseload;  $T_{RoB}++\rightarrow T_{RoB}+M + G + P + C; T_{BoW}++\rightarrow T_{BoW} + M + G + P + C$ . The light grey indicates the highest performing instance of each model setup. The dark grey indicates the highest performance for each prediction horizon.
from the SDistilBERT-NLI-STSb-base, SBERT-NLI-STSb-base and SRoBERTa-NLI-STSb-base language models. The performance of our clustered embedding features is compared against the bag-of-words features used in Drinkall and Pierrehumbert (2021),  $T_{BoW}$ , as well as word-count features taken from a prescriptive list of COVID-19 words defined by the non-hastag queries in the keyword list in Lamsal (2020),  $T_{KW}$ . The evaluation is conducted in four states where Reddit uptake is high: Washington, California, Texas and Florida. The states represent culturally different communities, instilling confidence that the behaviour is true in multiple domains. A successful result across all four states indicates that any observed behaviour is likely not just a symptom of an anomalous community.
The results in Table 1 detail the average performance across the different states and relative thresholds. Firstly, it is clear that word counts from the prescribed list in Lamsal (2020) only capture fractionally more information than a single post-count feature, and that simply using a chi-squared test of over-represented words,  $T_{BoW}$ , results in a significant performance increase. However, our  $T_{DisB}$ ,  $T_{BERT}$ ,  $T_{RoB}$  feature sets perform the best, and when  $T_{RoB}$  is used in combination with the comparison feature sets, the performance improves further. It is also evident that as better language models are used, the performance on this task increases. Showcasing the relationship between language model complexity and overall performance supports our
δr(t)	μ + σ
m	0.2	0.4	0.6	0.8	1
TRoB++	.803	.828	.867	.962	.970	.880 + .039
TBOW++	.704	.795	.821	.862	.910	.809 + .024
TRoB	.753	.752	.787	.876	.876	.792 + .034
TBOW	.683	.699	.761	.828	.865	.755 + .025
M	.649	.683	.663	.712	.727	.691 + .026
G	.678	.607	.735	.761	.789	.710 + .019
P	.548	.539	.537	.530	.577	.549 + .030
C	.437	.466	.527	.631	.640	.536 + .039
Table 2: Performance across a range of thresholds at a prediction horizon,  $\tau = 14$  days, averaged across all states.  $\mu$  &  $\sigma$  represent the mean and standard deviation of each feature set's results. The variables and highlighting criteria are the same as Table 1, but for the dark grey which denotes the highest performance at each threshold.
a-priori belief that improved semantic information from the text is linked with better epidemiological insights. Due to its success,  $T_{RoB}$  alone will henceforth be used in the evaluation as it provides the best performing feature set from our methodology.
4.1.1 Varying Thresholds
Table 2 breaks down the performance of classifying the data across different threshold increases. Intuitively, the more extreme events are easier to predict, explaining the behaviour across all feature sets. Indeed, when the threshold is large enough, the  $T_{RoB}++$  features achieve an accuracy of .970, significantly higher than the comparison feature sets, showing that social media data is a strong candidate for predicting a sharp rise in caseloads. Again, the performance across all thresholds is highest when using the  $T_{RoB}++$  features as opposed to the  $T_{BoW}++$ , highlighting the performance gain from the increased semantic information of transformer-based language models.
4.1.2 Feature Importance
To understand which features more are heavily weighted by the RF model when given the  $T_{RoB}++$  and  $T_{BoW}++$  feature sets, the feature importances
Feature set	m	Average	7D	14D	21D	28D
\( T_{RoB} + + \)	\( T_{RoB} \)	.331	.334	.307	.309	.386
M	.285	.264	.277	.299	.301
G	.307	.321	.354	.313	.240
P	.009	.009	.010	.008	.009
C	.064	.071	.052	.070	.063
\( T_{BoW} + + \)	\( T_{BoW} \)	.535	.584	.502	.509	.543
M	.244	.222	.265	.254	.235
G	.171	.131	.199	.192	.162
P	.014	.012	.009	.013	.021
C	.036	.050	.025	.031	.038
Table 3: Feature importances across varying prediction horizons, at  $\delta_r = 0.6$ . The variables and highlighting criteria are the same as Table 1.
are shown in Table 3. The tabulated data represents the sum of all individual feature importances in that class.
Table 3 shows that despite  $T_{RoB}$  performing better than  $T_{BoW}$ , the other comparison features,  $G$  and  $M$ , are more heavily weighted in  $T_{RoB} + +$  than in  $T_{BoW} + +$  at some prediction horizons. The  $T_{RoB} + +$  feature set performs better than the  $T_{BoW} + +$  features, so it appears that the information provided by the  $T_{RoB}$  features is complementary to the other feature types. It is also possible that there is some skew in the feature importance owing to the reported over-weighting of more continuous features by a Gini Importance algorithm (Strobl, 2007). Regardless of the slight differences, both text-derived feature sets are the most highly weighted when averaged over all prediction horizons, further showing the value of social media in this context.
5 Time-Series Forecasting Task
This section showcases our feature identification methodology within a time-series forecasting framework (henceforth Time-Series task) since this is a widely used prediction task in disease modelling. The high-density clusters are used as covariates in two multivariate time-series models. This setup better utilises the caseload feature and learns the temporal patterns within its historical movement. One difference with the Threshold task in the feature identification pipeline is the feature pruning step that reduces the number of features to 25. In the Threshold task, the target is a binary classification; therefore, a chi-squared test is appropriate. Given that the target is continuous in this task, f-regression is used. F-regression works by firstly calculating the cross-correlation  $\rho_{i}$  of the  $i^{th}$  feature  $X[:i]$  and target  $y$ :
$$
\rho_ {i} = \frac {\left(X [: , i ] - \bar {X} [: , i ]\right) \cdot \left(y - \bar {y}\right)}{\sigma_ {X [: , i ]} \cdot \sigma_ {y}} \tag {3}
$$
The F-statistic is then calculated along with the associated p-value. Then the top 25 most significant features are filtered to make up the feature set. For each model, the training features and targets are normalised between 0 and 1, and the test set is scaled using the same transformation. The target data is changed from the daily caseload to the daily increase in caseload to make sure the time-series is stationary. No moving average is used since the time-series models should account for the weekly
seasonality. The models are trained over 50 epochs on data from 07/03/2020 to 31/12/2020 and tested on data from 01/01/2021 to 01/03/2021. Whilst it is possible to improve the performance by retraining the model on recently evaluated data and sliding the train-test split across the dataset, our proposed framework highlights how the models perform on completely out-of-sample data.
We compare a Transformer and Gaussian Process (GP) model against the Martingale property baseline model which assumes that the caseload will not change, i.e. that additional features have zero predictive power. At a forecast horizon  $T$  days in the future, the last observed caseload,  $\mu_t$ , is used to forecast the caseload:  $\mu_{t + T} = \mu_t$ .
Gaussian Process Model - GP models were shown by Roberts et al. (2013) to perform well in contexts where prior knowledge regarding the appropriate model is limited. The difficulty in inferring the appropriate parametric model in infectious disease modeling led Lampos et al. (2017), Lampos et al. (2021) and Zou et al. (2018) to adopt a GP time-series model to predict future infectious disease caseloads. More modern methods have since outperformed GP models in time-series forecasting, so this GP model provides a further benchmark to the Transformer model outlined below. Our work uses a radial basis function (RBF) Kernel to specify the covariance function.