SlowGuess/NAACL_2022_OCR huggingface dataset:
Transformer model - Transformers have predominantly been used with textual (Vaswani et al., 2017) and image-based data (Ye et al., 2019); however, the auto-regressive properties of a masked self-attention layer mean that structurally transformers can obey causality. As a result, many papers have used transformers successfully to model time-series data (Lim et al., 2021), (Zerveas et al., 2021). Both papers reported that transformer models significantly outperformed the statistical, recurrent and convolutional comparison methods. This success has been replicated in disease modelling by Wu et al. (2020). Thus, transformer-based time-series models represent the state-of-the-art in many comparable contexts, motivating their use in this framework. The architecture that is used in this paper mimics that of Vaswani et al. (2017) and Alexandrov et al. (2019).
Data Source	Martingale	GP	Transformer
univariate	.0366	.0336	.0291
+T RoB	”	.0298‡	.0284†
+M	”	.0321‡	.0289
+G	”	.0308‡	.0290
+T RoB + G	”	.0322‡	.0288*
+T RoB + M	”	.0298‡	.0288*
+M + G	”	.0331*	.0287*
+T RoB + G + M	”	.0326†	.0288*
5.2 Time-Series Evaluation
For the Time-Series task, the prediction error of the forecasts is reported in an ablation study, using the same forecast horizons as the Threshold task. Different feature types make up the covariate set and are compared against the univariate case.
Table 4 shows the main ablation study, which averages the root-mean-square-error (RMSE) across the different forecast horizons and states. The results show the overall behaviour of the different feature types. The first conclusion is that the Transformer model always outperforms the Martingale and GP model. Poor GP model results are also seen in Lampos et al. (2021), where their persistence model outperforms the univariate and multivariate GP forecasts in multiple countries. Due to the GP model's weaker performance, further analysis will involve the Transformer model. State-level results are displayed in Figure 2 and show that the Transformer performs well at modelling the time-series data.
The more meaningful conclusion that can be drawn from Table 4 is that whilst the success of the  $T_{RoB}$  features is still present, it is more marginal than in the Threshold task both when the  $T_{RoB}$  features are used in isolation and when the feature sets are combined. Using the  $T_{RoB}$  features
Table 4: The RMSE error averaged across all forecast horizons and states. The significance of each result in comparison to the univariate case is denoted by asterisks: * - P<.2; † - P<.05; ‡ - P<.01
Data Source	Av.	7D	14D	21D
uni	.0291	.0274	.0287	.0301
+TRoB	.0284†	.0270*	.0274‡	.0288‡
+M	.0289	.0274	.0286	.0292‡
+G	.0290	.0274	.0283*	.0300
+TRoB+G	.0288*	.0271*	.0287	.0289‡
+TRoB+M	.0288*	.0270*	.0278‡	.0298*
+M+G	.0287*	.0271*	.0274‡	.0299
+TRoB+G+M	.0288*	.0273	.0274‡	.0299
Table 5: The RMSE error of a Transformer model averaged across all states at varying forecast horizons, using the same highlighting criteria as Table 1. The significance notation is: * - P<.2; † - P<.05; ‡ - P<.01
Size rank	Topic	ID	Frequency	Top 5 words
1	Masks	95	10699	mask, wear, masks, wearing, gloves
2	Unemployment	138	7591	unemployment, claim, pay, money, rent
3	Appreciation	181	3508	thank, thanks, appreciate, good, sharing
4	Schools	120	2808	school, kids, schools, teachers, students
5	Temporal statistics	152	1290	weeks, phase, ago, months, week
6	Lockdown frustration	75	1217	closed, shut, f**k, close, die
7	Agreement	197	892	yes, agree, yeah, exactly, sure
8	Festivities	96	879	thanksgiving, christmas, family, people, party
9	Vaccines	196	877	vaccine, vaccines, vaccinated, vaccination, people
14	Illness	178	569	cough, fever, symptoms, asthma, throat
17	Gyms	50	487	gym, gyms, fitness, open, exercise
19	Trump	218	378	trump, people, stupid, asleep, president
Table 6: The notable clusters from the r/CoronavirusWA subreddit using a SROBERTa-NLI-STSb-base language model. The Frequency column represents the number of comments that are included in the cluster.
in the covariate set does deliver a statistically significant result, however, the decrease in error rate is small. Whilst all feature sets deliver a performance increase, none of the other feature sets can be considered to deliver a statistically significant result. Combining further data types doesn't deliver the expected performance increase, with the performance plateauing, and in some cases decreasing as the number of feature types increases. One possibility is that the information that the  $T_{RoB}$  features provide is counteracted by the performance costs of having a large number of variables. Table 5 reinforces what is seen in Table 4, showing that across all forecast horizons there is a slight improvement in performance and that using the  $T_{RoB}$  features alongside the time-series data generally results in the lowest error across all tested feature sets.
The significance of the results in Table 4 is calculated by taking 10,000 samples at every forecast horizon for each state. The error of each forecast is calculated, resulting in an error distribution for all feature sets. To discern whether the addition of a feature type results in a statistically significant performance shift a Z-test is used. The univariate forecast is assumed to be the population distribution and each feature set's forecast errors are treated as the sample distribution. The Z-score is calculated using parameters from both distributions:
$$
Z = \frac {\overline {{X}} _ {p o p} - \overline {{X}} _ {s a m p l e}}{\sqrt {\sigma_ {X _ {p o p}} ^ {2} + \sigma_ {X _ {s a m p l e}} ^ {2}}} \tag {4}
$$
Figure 2: State forecasts at  $\tau = 7$ . The univariate forecast is compared against three multivariate forecasts where the  $M, G \& T_{RoB}$  features make up the covariate set.
State	Topic	ID	Importance	Top 5 words
Washington	Working	107	.21	work, office, home, headquarters, let
Illness	178	.14	cough, fever, symptoms, asthma, throat
Quarantine	136	.08	quarantine, facility, people, outside, think
Schools	120	.08	school, kids, schools, teachers, students
Statistics	150	.07	trendline, graph, using, ggplot2, plotted
California	Illness	163	.12	cough, throat, fever, chest, symptoms
School closure	122	.09	schools, close, school, closing, closed
Guns	60	.09	gun, guns, shoot, firearms, buy
Safety	103	.08	safe, stay, luck, protect, safer
Flu	166	.06	flu, pneumonia, influenza, season, spanish
Texas	Data	130	.20	source, data, information, info, sources
Voter Fraud	72	.10	vote, mail, voter, voting, fraud
Houston	78	.07	houston, harris, county, area, houstonian
Doctors	138	.06	doctor, doctors, medical, physician, telemedicine
Illness	155	.04	fever, cough, allergies, asthma, symptoms
Florida	Spring Break	22	.22	spring, break, bike, week, breakers
Social Media News	111	.19	reddit, facebook, news, echo, chamber
Statistics	106	.05	numbers, data, believe, trust, graph
Illness	94	.04	drug, people, fever, virus, sick
Desantis	67	.04	desantis, care, deathsantis, dbpr, ron
Table 11: Important features from each of the key states at  $\delta_r = 0.6$  &  $\tau = 7$  days.
Examining the contents of our Reddit features can help us understand the information they provide to the prediction task. We took the top 5 non-stopwords from the cluster to characterise each cluster and manually named them for better comprehension. Table 6 shows the largest clusters from the SRoBERTa-NLI-STSb-base representation of the r/CoronavirusWA subreddit. These topics identify precise semantic concepts that intuitively provide relevant information for a caseload prediction.
As mentioned, the advantage of the Threshold task is that it provides greater interpretability than the more black-box time-series models. Therefore, the Threshold task is used to understand which features are important to the prediction. Table 11 shows the weightings of the most important features at  $\delta_r = 0.6$  &  $\tau = 7$  days. The cultural differences between the states can be seen via these features, most obviously the Houston feature in Texas and the Desantis feature in Florida. The Spring Break cluster is only seen in Florida, a state that is famed for this holiday tradition and was a large contributor to an increase in non-COVID-19 compliant events that resulted in an increase in cases at the beginning of the pandemic. Equally, the Guns and Safety features in California likely identify the strong negative reaction from the libertarian community within California to what were the most stringent lockdown restrictions from any of the analysed states. The libertarian trait within California is best characterised by the Prop 22 bal
lot initiative<sup>3</sup> which identifies a political attitude not aligned with strict lockdown measures. Alongside these differences, the Illness feature is highly weighted in all states. The use of this feature in all short-term predictions might explain the success of prior work that used static tracking words such as "Influenza", "fever", "headache", etc. (Samaras et al., 2020), (Iso et al., 2016); discussion about symptoms is indicative of a rise in cases in all states. It is clear, however, that exclusive use of symptomatic features is not optimal, since other topics besides symptomatic conversation are useful for the prediction.
Reddit data performs well at discerning different trend signals for COVID-19 caseload increases in the Threshold task. Reddit features alone achieved high accuracy at most threshold increases but were especially strong when identifying whether the caseload was likely to double in the next 14 days, achieving an accuracy of .970. That value is seen in the Time-Series task but the performance benefit is not as stark, especially when the number of features increases. The characteristics of Reddit data make it appealing: it is readily available and updated in real-time, offering the means for monitoring infectious diseases in regions where reported data is unreliable; however global Reddit usage is not constant, and not every area has a subreddit, making our exact methodology hard
$^{3}$ URL: https://vig.cdn.sos.ca.gov/2020/general/pdf/top1-prop22.pdf (accessed: 29/12/2021)
to scale. As Reddit usage increases and disperses around the world or data from another social media site is adapted to fit within our pipeline, the methods used in this paper will become more scalable. Another notable conclusion is that the predictive information within Reddit data is better extracted by including transformer-based language models in the forecasting pipeline. Language model complexity appears to be linked with performance improvements in the Threshold task. Strong language models allow us to isolate highly specific features predictive of future caseload increases in an unsupervised setting.
8 Future work
More work can be done on feature selection for the Time-Series task. The value of combining all data types is evident in the Threshold task but that value is not seen in the Time-Series task. Developing models that are able to model a larger number of features more effectively could likely yield some performance gains. On top of this, our methodology relies on using textual data that refers to a specific geographic location. Reddit's structure makes this simple; however, more data is needed to replicate our findings in regions where Reddit take-up is low. Geotagged posts and the geolocation of a user's home region are possible avenues for enlarging the social-media dataset. Finally, the unsupervised methodology outlined in this paper can be adapted to other fields in which a social media derived feature set is used, such as quantitative finance, election and logistics forecasting.
The first author was funded by the Economic and Social Research Council of the UK via the Grand Union DTP. This work was also supported in part by a grant from the Engineering and Physical Sciences Research Council (EP/T023333/1). We are also grateful to the Oxford-Man Institute of Quantitative Finance and the Oxford e-Research Centre for their support.
Roee Aharoni and Yoav Goldberg. 2020. Unsupervised domain clusters in pretrained language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7747-7763, Online. Association for Computational Linguistics.
Alexander Alexandrov, Konstantinos Benidis, Michael Bohlke-Schneider, Valentin Flunkert, Jan Gasthaus, Tim Januschowski, Danielle C. Maddix, Syama Rangapuram, David Salinas, Jasper Schulz, Lorenzo Stella, Ali Caner Turkmen, and Yuyang Wang. 2019. Gluonts: Probabilistic time series models in python. arXiv preprint arXiv:1906.05264.
Meshrif Alruily, Mohamed Ezz, Ayman Mohamed Mostafa, Nacim Yanes, Mostafa Abbas, and Yasser El-Manzalawy. 2022. Prediction of Covid-19 transmission in the united states using google search trends. Computers, Materials and Continua, 71(1):1751-1768. Funding Information: Funding Statement: This work is supported in part by the Deanship of Scientific Research at Jouf University under Grant No. (CV-28-41). Publisher Copyright: © 2022 Tech Science Press. All rights reserved.
Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. 2020. The pushshift reddit dataset. Proceedings of the International AAAI Conference on Web and Social Media, 14(1):830-839.
Adam Birmingham and Alan Smeaton. 2011. On using Twitter to monitor political sentiment and predict election results. In Proceedings of the Workshop on Sentiment Analysis where AI meets Psychology (SAAIP 2011), pages 2-10, Chiang Mai, Thailand. Asian Federation of Natural Language Processing.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Computational Linguistics.
Leo Breiman. 2001. Random forests. Machine Learning, 45(1):5-32.
Ricardo J. G. B. Campello, Davoud Moulavi, and Joerg Sander. 2013. Density-based clustering based on hierarchical density estimates. In Advances in Knowledge Discovery and Data Mining, pages 160-172, Berlin, Heidelberg. Springer Berlin Heidelberg.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017).
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Felix Drinkall and Janet B Pierrehumbert. 2021. Predicting COVID-19 cases using reddit posts and other online resources. In 2021 Swiss Text Analytics Conference, SwissText 2021.
Thomas Hale, Sam Webster, Anna Petheric, Toby Phillips, and Beatrix Kira. 2020. Oxford pandemic-19 government response tracker. *Blavatnik School of Government*.
Mark Huberty. 2015. Can we vote with our tweet? on the perennial difficulty of election forecasting with social media. International Journal of Forecasting, 31(3):992-1007.
Cornelia Ilin, Sébastien Annan-Phan, Xiao Hui Tai, Shikhar Mehra, Solomon Hsiang, and Joshua E Blumenstock. 2021. Public mobility data enables COVID-19 forecasting and management at local and global scales. Scientific reports, 11(1):1-11.
Hayate Iso, Shoko Wakamiya, and Eiji Aramaki. 2016. Forecasting word model: Twitter-based influenza surveillance and prediction. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 76-86, Osaka, Japan. The COLING 2016 Organizing Committee.
Vasileios Lampos, Maimuna S. Majumder, Elad YomTov, Michael Edelstein, Simon Moura, Yohhei Hamada, Molebogeng X. Rangaka, Rachel A. McKendry, and Ingemar J. Cox. 2021. Tracking Covid-19 using online search. npj Digital Medicine, 4.
Vasileios Lampos, Bin Zou, and Ingemar Johansson Cox. 2017. Enhancing feature selection using word embeddings: The case of flu surveillance. In Proceedings of the 26th International Conference on World Wide Web, WWW '17, page 695-704, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
Rabindra Lamsal. 2020. Coronavirus (covid-19) tweets dataset.
Bryan Lim, Sercan Ö. Arik, Nicolas Loeff, and Tomas Pfister. 2021. Temporal fusion transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting, 37(4):1748-1764.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.
Ryan McConville, Raul Santos-Rodriguez, Robert J Piechocki, and Ian Craddock. 2021. N2d: (not too) deep clustering via clustering the local manifold of an autoencoded embedding. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 5145-5152.
Daniel J. McDonald, Jacob Bien, Alden Green, Addison J. Hu, Nat DeFries, Sangwon Hyun, Natalia L. Oliveira, James Sharpnack, Jingjing Tang, Robert Tibshirani, Valérie Ventura, Larry Wasserman, and Ryan J. Tibshirani. 2021. Can auxiliary indicators improve COVID-19 forecasting and hotspot prediction? Proceedings of the National Academy of Sciences, 118(51).
Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger. 2018. Umap: Uniform manifold approximation and projection. Journal of Open Source Software, 3(29):861.
Ming Ni, Qing He, and Jing Gao. 2017. Forecasting the subway passenger flow under event occurrences with social media. IEEE Transactions on Intelligent Transportation Systems, 18(6):1623-1632.
Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019. Visualizing and measuring the geometry of bert. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
Alex Reinhart, Logan Brooks, Maria Jahja, Aaron Rumack, Jingjing Tang, Sumit Agrawal, Wael Al Saeed, Taylor Arnold, Amartya Basu, Jacob Bien, Angel A. Cabrera, Andrew Chin, Eu Jing Chua, Brian Clark, Sarah Colquhoun, Nat DeFries, David C. Farrow, Jodi Forlizzi, Jed Grabman, Samuel Gratzl, Alden Green, George Haff, Robin Han, Kate Harwood, Addison J. Hu, Raphael Hyde, Sangwon Hyun, Ananya Joshi, Jimi Kim, Andrew Kuznetsov, Wichada La Motte-Kerr, Yeon Jin Lee, Kenneth Lee, Zachary C. Lipton, Michael X. Liu, Lester Mackey, Kathryn Mazaitis, Daniel J. McDonald, Phillip McGuinness, Balasubramanian Narasimhan, Michael P. O'Brien, Natalia L. Oliveira, Pratik Patil, Adam Perer, Collin A. Politsch, Samyak Rajanala, Dawn Rucker, Chris Scott, Nigam H. Shah, Vishnu Shankar, James Sharpnack, Dmitry Shemetov, Noah Simon, Benjamin Y. Smith, Vishakha Srivastava, Shuyi Tan, Robert Tibshirani, Elena Tuzhilina, Ana Karina Van Nortwick, Valerie Ventura, Larry Wasserman, Benjamin Weaver, Jeremy C. Weiss, Spencer Whitman, Kristin Williams, Roni Rosenfeld, and Ryan J. Tibshirani. 2021. An open repository of real-time Covid-19 indicators. Proceedings of the National Academy of Sciences, 118(51).
Stephen Roberts, Michael Osborne, Mark Ebden, Steven Reece, Neale Gibson, and Suzanne Aigrain. 2013. Gaussian processes for time-series modelling. Philosophical Transactions of the Royal Society A:
Mathematical, Physical and Engineering Sciences, 371(1984):20110550.
David Rother, Thomas Haider, and Steffen Eger. 2020. CMCE at SemEval-2020 task 1: Clustering on manifolds of contextualized embeddings to detect historical meaning shifts. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 187-193, Barcelona (online). International Committee for Computational Linguistics.
Peter J. Rousseeuw. 1987. Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics, 20:53-65.
Loukas Samaras, Elena García-Barriocanal, and Miguel-Angel Sicilia. 2020. Comparing social media and google to detect and predict severe epidemics. Nature - Sci Rep 10.
Ramit Sawhney, Shivam Agarwal, Arnav Wadhwa, and Rajiv Ratn Shah. 2020. Deep attentive learning for stock movement prediction from social media text and company correlations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8415-8426, Online. Association for Computational Linguistics.
Suzanna Sia, Ayush Dalmia, and Sabrina J. Mielke. 2020. Tired of topic models? clusters of pretrained word embeddings make for fast and good topics too! In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1728-1736, Online. Association for Computational Linguistics.
Boulesteix AL. Zeileis A. et al. Strobl, C. 2007. Bias in random forest variable importance measures: illustrations, sources and a solution. BMC Bioinformatics, 8(1):1-21.
Youqiang Sun, Jiuyong Li, Jixue Liu, Christopher Chow, Bingyu Sun, and Rujing Wang. 2015. Using causal discovery for feature selection in multivariate numerical time series. Machine Learning, 101(1):377-395.
Laure Thompson and David Mimno. 2020. Topic modeling with contextualized word representation clusters. arXiv preprint arXiv:2010.12626.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Lijing Wang, Xue Ben, Aniruddha Adiga, Adam Sadilek, Ashish Tendulkar, Srinivasan Venkatramanan, Anil Vullikanti, Gaurav Aggarwal, Alok Talekar, Jiangzhuo Chen, Bryan Lewis, Samarth Swarup, Amol Kapoor, Milind Tambe, and Madhav Marathe. 2020. Using mobility data to understand and forecast COVID dynamics. medRxiv.