SlowGuess/NAACL_2022_OCR huggingface dataset:
Qing-Guo Wang, Xian Li, and Qin Qin. 2013. Feature selection for time series modeling. Journal of Intelligent Learning Systems and Applications, 5(03):152-164.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
Neo Wu, Bradley Green, Xue Ben, and Shawn O'Banion. 2020. Deep transformer models for time series forecasting: The influenza prevalence case. arXiv preprint arXiv:2001.08317.
Yumo Xu and Shay B. Cohen. 2018. Stock movement prediction from tweets and historical prices. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1970-1979, Melbourne, Australia. Association for Computational Linguistics.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
L. Ye, M. Rochan, Z. Liu, and Y. Wang. 2019. Cross-modal self-attention network for referring image segmentation. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10494–10503, Los Alamitos, CA, USA. IEEE Computer Society.
George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. 2021. A transformer-based framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery Data Mining, KDD '21, page 2114-2124, New York, NY, USA. Association for Computing Machinery.
Bin Zou, Vasileios Lampos, and Ingemar Cox. 2018. Multi-task learning improves disease models from web search. In Proceedings of the 2018 World Wide Web Conference, WWW '18, page 87-96, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
A Clustering algorithms hyperparameter tuning
An exhaustive search has been conducted to find the optimal  $k$  parameter (number of clusters) for KM and GMM clustering to compare their optimal
Clustering algorithm	k	Average	7 days	14 days	21 days	28 days
GMM	25	.659	.799	.645	.594	.596
50	.691	.838	.691	.614	.622
75	.698	.845	.689	.618	.639
100	.702	.831	.703	.612	.662
125	.714	.827	.709	.659	.664
150	.716	.850	.718	.619	.676
KM	25	.706	.883	.685	.620	.635
50	.636	.681	.714	.622	.528
75	.677	.781	.769	.607	.550
100	.702	.787	.678	.647	.695
125	.663	.757	.754	.611	.531
150	.658	.768	.687	.621	.556
Table 8: The average performances of an RF classification model using KM and GMM clustering across all thresholds at different values of  $k$  on the r/CoronavirusWA subreddit. The comment-level SDistilBERT-NLI-STSb-base representations' dimensionality was reduced via UMAP. The light grey indicates the highest performing instance of each model setup. The dark grey indicates the highest average performing model configuration.
Language model	Dim. reduction	Clustering	Average	7 days	14 days	21 days	28 days
DistilBERT	PCA	HDBCSAN	.722	.821	.724	.678	.667
KM	.716	.807	.675	.677	.704
GMM	.714	.808	.715	.651	.680
UMAP	HDBCSAN	.807	.905	.827	.759	.737
KM	.706	.883	.685	.620	.635
GMM	.716	.850	.718	.619	.676
Average	.730	.846	.724	.667	.683
Table 9: The average performance, on the r/CoronavirusWA subreddit, of an RF model across all thresholds at different prediction horizons for each of the model pipelines using only  $T_{RoB}$  features. The variables and highlights are the same as in Table 8.
configurations against HDBSCAN. The standard Silhouette score method was trialled for fine-tuning the  $k$  parameter, but the result was  $k = 1$ , perhaps indicating the unsuitability of KM and GMM for this task. Figure 3 is a plot of the Silhouette score (Rousseeuw, 1987) of a KM clustering algorithm for different values of  $k$  on the UMAP reduced SDistilBERT-NLI-STSb-base embeddings space. The maximum Silhouette score should be the most
Figure 3: Silhouette score vs.  $k$  using a KM clustering algorithm on UMAP-DistilBERT embedding space.
appropriate k value if the data is divided into distinct clusters. The maximum Silhouette score at  $k = 1$  indicates that the data is structured into one central cluster with high and low-density areas.
Since the Silhouette score does not provide an
obvious  $k$  parameter, and yet there needs to be some proof that HDBSCAN is a better algorithm than KM and GMM, an exhaustive search for the optimal  $k$  on the development data is conducted to prove that KM and GMM are not suitable for the task.  $k$  is tuned using the performance on the r/CoronavirusWA data from 01/03/2021 to 17/01/2021 with UMAP dimensionality reduction.
The same search was conducted to find the optimal  $k$  for the PCA space; for KM, the value was 75, and for GMM, the value was 125 on the DistilBERT embedding space. These values of  $k$  were used for the testing in Appendix B.
B Dimensionality reduction and clustering algorithms
A test was carried out to see which combination of dimensionality reduction and clustering algorithms resulted in the best overall performance. The different algorithms were tested using SDistilBERT-NLISTSB-base representations of the comments. The two dimensionality reduction techniques used were PCA and UMAP; the three clustering techniques used were GMM, KM and HDBSCAN. The  $k$  values derived in Appendix A were used for the GMM
and KM clustering, and the evaluation pipeline used is the Threshold task described in Section 4.
The results from Table 9 show that the combination of UMAP and HDBSCAN is the best combination of algorithms; the UMAP-HDBSCAN combination is the best performing pipeline across all prediction horizons.
C Aggregated Sequence Classification models
As mentioned in Section 1, the most obvious way to incorporate the modern transformer-base language models is to formulate the problem as an Aggregated Sequence Classification (ASC) task. It has been shown that BERT and other similar models are well adapted to performing sequence classification, and this has become a common usage of these language models (Devlin et al., 2019). Therefore, it is important to trial a model that incorporates this more standard methodology before trialling other feature identification methods.
For evaluation, we trialled two language models: BERT-base-uncased, and a domain adapted version of BERT-base-uncased trained on the r/Coronavirus subreddit - CoFReBERT (CoVID-19 Forecasting from Reddit BERT). The language models are then fine-tuned on a Sequence Classification task in which the [CLS] token encodes the "up" or "down" class, indicating a possible increase or decrease in the number of cases. The adapted models are referred to as ASC-BERT and ASC-CoFReBERT. The model is trained on balanced classes with a 4:1 train-test split, where each day is assigned to be a test or a train day, and all comments written on a particular day are categorised together. Once the model labels each comment within the test set as either "up" or "down", the majority class on a given test day is assigned as the prediction for that day.
Models	Av.	7D	14D	21D	28D
ASC-BERT	.631	.769	.655	.561	.537
ASC-CoFReBERT	.701	.846	.690	.634	.634
\(T_{RoB}\)	.869	.923	.896	.810	.855
\(T_{BoW}\)	.765	.780	.808	.804	.791
Table 10: The average performance, on the r/CoronavirusWA subreddit, across all thresholds at four different prediction horizons. The variables and highlights are the same as in Table 8.
From the results in Table 10, it is clear that the models do not perform as well as hoped in comparison to the traditional static word features and the features outlined in this paper. The main reason for
this is likely to be noise from the unsupervised labelling process. Comments that are either unrelated to the prediction or indicate an opposite caseload trend are included in the prediction. Without manual labelling, it is hard to reduce this noise; however, that would result in investigator bias entering the prediction. Furthermore, it is not completely clear whether a comment is indicative of a rise in cases, shown by the variety of topics considered important to the prediction in Table 11. Therefore, the structure of the ASC models is not well adapted to the task of predicting COVID-19 cases.
D Training and software details
Python Packages The sentence-embedding models from (Reimers and Gurevych, 2019) were used to encode the Reddit post representations using the sentence-transformers Python package. The time-series models were both implemented using the gluonts Python package (Alexandrov et al., 2019). The ASC models outlined in Appendix C use the BERT-base-uncased model from the transformers package and the ASC-CoFReBERT model was trained using the run_mlm.py file in the library.
Training Parameters Besides the analysis detailed earlier in the Appendix, we do not perform hyperparameter tuning but use common hyperparameter values for all calculations in this paper. For the Random Forest model in the Threshold task, the number of trees is 100, and the maximum tree depth is 20. The Time-Series models were trained over 50 epochs and used default parameter values. The ASC-CoFReBERT model was trained with standard parameter values, using a batch size of 128 and a dropout probability of 0.1.
Computation All experiments in the main body of the paper were run on a personal computer, the ASC model in Appendix C was run on the. The ASC model was run on a Tesla P100 and took between 3 to 6 hours to run, depending on the size of the subreddit.
Licenses There are licenses associated with the use of some of the data and Python packages used in this paper. The OxCGRT dataset and Pushshift API are open access under the Creative Commons Attribution CC BY and 4.0 International standards. The COVID-19 Track
ing Project, gluonts, transformers and sentence-transformers Python packages are licensed under the Apache License 2.0.