SlowGuess/NAACL_2022_OCR huggingface dataset:
Preprocessing We pre-process English tweets using NLTK (Twitter mode), for treatment of URLs, identifiers (@) and hashtags (#). We adopt the Bert-Tokenizer for tokenization and use the pre-trained BERT-base-cased model for text embeddings. We align days by dropping samples that lack tweets for a consecutive 5-day lookback window.
Training Setup We adopt grid search to find optimal hyperparameters based on the validation MCC for all methods. For NMS, we use an overlapping threshold of 2 units. We use default  $\alpha = 2, \gamma = 5$  for the focal loss and learning rate  $\in (1e^{-5}, 1e^{-3})$  to train the models using the Adam optimizer.
Evaluation Metrics We evaluate all methods using F1-score, Mathews Correlation Coefficient (MCC) and Exact Match (EM) (We provide metric
equations in Appendix). On the bubble span level, we use F1 and MCC, which measure the overlap between the predicted and the true bubble spans. While EM measures the percentage of overall predicted bubbles that exactly match the true bubble spans. We also evaluate on the "number of bubble" task via F1-Score and Accuracy (Acc).
6.1 Baseline Methods
We use BERT for text encoding in all models. Each baseline forecasts prices over lookahead  $T$  and uses the PSY model to detect bubbles.
ARIMA Auto-regressive moving average based model that uses past prices from 100 days as input for forecasting (Yenidogan et al., 2018).
- W-LSTM A LSTM model with autoencoders that encode noise-free data obtained via wavelet transform of historic prices (Bao et al., 2017).
- A-LSTM Adversarily trained LSTM on price inputs for forecasting (Feng et al., 2019b).
- Chaotic A Hierarchical GRU model applied on texts within and across days (Hu et al., 2018).
- MAN-SF(T) Hierarchical attention on texts within and across days (Sawhney et al., 2020b).
- CH-RNN An RNN coupled with cross-modal attention on price and texts (Wu et al., 2018).
- SN - HFA: StockNet - HedgeFundAnalyst, a variational autoencoder with attention on texts and prices (Xu and Cohen, 2018b).
Through Figure 5 we observe that augmenting price-based MBHN with financial texts leads to sig-
Model	F1 ↑	MCC ↑	EM ↑
ARIMA (Yenidogan et al., 2018)	0.02	0.00	0.02
W-LSTM (Bao et al., 2017)	0.09	0.04	0.03
A-LSTM (Feng et al., 2019b)	0.45	0.13	0.39
Chaotic (Hu et al., 2018)	0.49	0.14	0.45
MAN-SF(T) (Sawhney et al., 2020b)	0.49	0.16	0.46
CH-RNN (Wu et al., 2018)	0.50	0.19	0.47
SN - HFA (Xu and Cohen, 2018b)	0.51	0.21	0.49
MBHN (Ours)	0.53*	0.25*	0.53*
nificant  $(p < 0.01)$  improvements, suggesting the importance of leveraging textual sources to forecast the formation of speculative bubbles. This observation ties up with (Sawhney et al., 2021) who show that online texts often indicate market surprises that may not be well captured by prices alone. Noting insignificant  $(p > 0.01)$  improvements on using both price and text information, we further probe MBHN performance benefits from each of its components in Table 4 with only textual inputs. We note the biggest improvements on replacing the Euclidean encoders with their hyperbolic variants, suggesting that the inherent power-law dynamics and scale-free nature of online texts and financial bubbles are better represented in the hyperbolic space (Ganea et al., 2018b). Next, on adding temporal attention, we note improvements as MBHN can better distinguish noise inducing text from relevant market signals, minimizing false evaluations and overreactions (De Long et al., 1990). The temporal attention mechanism can likely diminish the impact of such noise (rumors, vague comments) and better capture the diverse influence of texts.
7.2 Performance Comparison
We compare the performance of MBHN with baselines in Table 5. We observe that MBHN through hyperbolic learning significantly outperforms  $(p < 0.01)$  all baselines. This observation validates our premise of formulating CryptoBubbles as an extractive multi-span task and suggests its practical applicability in predicting speculative financial risks. Further, we note that methods that use crypto affecting information from online texts (MBHN, SNHFA) generate better performance than approaches that only use price data (A-LSTM, W-LSTM). These improvements re-validate the effectiveness
Table 5: Performance comparison against baselines (mean of 10 independent runs). Bold and italics represent best and second-best results, respectively. * indicates statistically significant  $(p < 0.01)$  improvement over SN-HFA under Wilcoxon's Signed Rank Test.
Asset Type	Asset Name	F1↑	MCC↑	EM↑
Equity	Top 1: CCIV	0.73	0.48	0.71
Bottom 1: PLTR	0.26	-0.05	0.17
All Equities	0.52	0.07	0.63
Cryptocoin	Top 1: DOGE	0.54	0.26	0.55
Bottom 1: BASED	0.33	-0.01	0.43
All Cryptocurrencies	0.50	0.05	0.70
Table 6: MBHN's performance on Reddit meme stock data under zero shot settings. Top 1 and Bottom 1 denote MBHN's best and the worst performing assets.
of using online texts to encode investor sentiment and market information. Next, we note that MBHN requires less historical data ( $\tau = 5$ ) compared to conventional methods (ARIMA,  $\tau = 100$ ), while achieving better performance, suggesting MBHN applicability in low data scenarios.
7.3 Zero-shot Learning
To evaluate MBHN applicability to unseen asset classes and social media linguistic styles we test it on unseen Reddit meme-stock data (§4) under zero-shot settings and summarise the results in Table 6. We observe that MBHN can generalize over the true market sentiment to an extent and transfer knowledge across the two social media by being invariant to post styles, lengths, and social media structure. Further, these observations suggest that MBHN can be leveraged for cold start scenarios (completely new assets), which is quite frequent for cryptocurrencies (Gavin and Richard, 2019). We speculate MBHN's transferability to hyperbolic learning due to its ability to accurately reflect complex scale-free relations between social media texts. This observation ties up with (Khrulkov et al., 2020b) who show that hyperbolic learning is useful under zero-shot settings. These observations collectively demonstrate MBHN's ability to generalize across social media and asset classes, including meme stocks such as GameStop, which purely see bubbles due to social media hype (Chohan, 2021).
7.4 Sensitivity to Lookback and Lookahead
We study the impact of historical context on MBHN's performance in Figure 7 by varying the availability of historic information. We observe lower performance on using shorter lookback periods, indicating inadequacy to capture the market influencing signals, likely as public information requires time to absorb into price fluctuations (Luss and D'Aspremont, 2015). As we increase the look
Figure 6: Top: Reddit posts of GameStop from  $13^{\text{th}}$  Jan'21 to  $3^{\text{rd}}$  Feb'21. Bottom: Twitter tweets of LiteCoin from  $5^{\text{th}}$  Feb'21 to  $2^{\text{nd}}$  March'21 along with token level attention and temporal tweet level attention visualisation. We also show MBHN's performance with its Euclidean variant on CryptoBubbles bubble detection task.
Figure 7: Sensitivity to parameters  $T$  and  $\tau$
back, we find that larger periods allow the inclusion of stale information from older days having a relatively lower influence on prices (Bernhardt and Miao, 2004) that may deteriorate the model performance. However, our temporal attention mechanism, to an extent, learns to filter our salient texts, which induce bubbles in the market. Further, we analyze MBHN's sensitivity to the number of lookahead days  $T$  in Figure 7. We observe that our model is robust to different lookahead periods, suggesting
MBHN's generalizability for predicting speculative bubbles over different risk taking appetites.
We conduct an extended study to elucidate MBHN's explainable predictions and practical applicability. As shown in Figure 6, we analyze the recent posts corresponding to LiteCoin tweets and Gamestop zero-shot Reddit posts.
Practical applicability of CryptoBubbles and MBHN We first calculate temporal attention scores across tweets and observe that MBHN is able to attend to more influential tweets to an extent for both meme cryptocurrencies and stocks under zero-shot settings providing valuable pieces of information to better judge speculative risks (Kumar, 2009). For both asset classes, we observe an overall positive market sentiment and brief epidemic-like social media activity, leading to increased trade volume activity, causing the creation of multiple risky bubbles (Phillips and Gorse, 2018; Froot and Obst-
feld, 1989). Such epidemic like social media activity and bubble creation exhibit scale-free dynamics since their impact decays as a powerlaw distribution over time (Phillips and Gorse, 2017). MBHN outperforms its Euclidean variant as it is better able to represent the scale-free dynamics via hyperbolic learning (Chen and Hafner, 2019). Furthermore, we observe multiple cryptocurrency mentions in a single social media post suggesting that bubble explosivity in one cryptocoin may induce bubbles in another crypto (Agosto and Cafferata, 2020). These observations demonstrate the practical applicability of CryptoBubbles quantitative trading as it can scale to forecast multiple risk bubbles.
Contextualizing impact of social media hype and meme stocks As shown in Figure 6, we note that for meme-stocks like GameStop (GME), Reddit posts consistently display a social media hype growing like a chain reaction consecutively creating a bubble (Angel, 2021). The price value of such meme stocks follows sentiment-driven pricing where more media traffic, more positive tones, more argument leads to higher returns in a short squeeze (Chohan, 2021; Hu et al., 2021). We observe that MBHN via hyperbolic learning is able to capture the social media hype of meme-stocks where the intensity of the psychological contagion among retail investors on social media drive asset price fluctuations (Semenova and Winkler, 2021).
Building on the popularity of speculative social media trading, we present CryptoBubbles, a challenging multi-span crypto bubble forecasting task, and dataset. We explore the power-law dynamics of social media hype and propose MBHN which learns from the power-law dynamics via hyperbolic learning. We curate a Reddit test set to evaluate our models under zero-shot and cold-start settings. Through extensive analysis, we show the practical applicability of CryptoBubbles and publicly release CryptoBubbles and our hyperbolic models.
The research work of Paolo Rosso was partially funded by the Generalitat Valenciana under DeepPattern (PROMETEO/2019/121). We would like to thank the Financial Services Innovation Lab at Georgia Institute of Technology for their generous support.
While data is essential in making models like MBHN effective, we must work within the purview of acceptable privacy practices to avoid coercion and intrusive treatment. To that end, we utilize publicly available Twitter and Reddit data in a purely observational and non-intrusive manner. Although informed consent of each user was not sought as it may be deemed coercive, We follow all ethical regulations set by Twitter and Reddit.
There is an ethical imperative implicit in this growing influence of automation in market behavior, and it is worthy of serious study (Hurlburt et al., 2009; Cooper et al., 2020). Since financial markets are transparent (Bloomfield and O'Hara, 1999), and heavily regulated (Edwards, 1996), we discuss the ethical considerations pertaining to our work. Following (Cooper et al., 2016), we emphasize on three ethical criteria for automated trading systems and discuss MBHN's design with respect to these criteria.
Prudent System A prudent system "demands adherence to processes that reliably produce strategies with desirable characteristics such as minimizing risk, and generating revenue in excess of its costs over a period acceptable to its investors" (Longstreth, 1986). MBHN is directly optimized towards high risk bubble forecasting.
Blocking Price Discovery A trading system should not block price discovery and not interfere with the ability of other market participants to add to their own information (Angel and McCabe, 2013). For example, placing an extremely large volume of orders to block competitor's messages (Quote Stuffing) or intentionally trading with itself to create the illusion of market activity (Wash Trading). MBHN does not block price discovery in any form.
Circumventing Price Discovery A trading system should not hide information, such as by participating in dark pools or placing hidden orders (Zhu, 2014). We evaluate MBHN only on public data in regulated markets.
Despite these considerations, it is possible for MBHN, just as any other automated trading system, to be exploited to hinder market fairness. We follow broad ethical guidelines to design MBHN and encourage readers to follow both regulatory and ethical considerations pertaining to the market.
Arianna Agosto and Alessia Cafferata. 2020. Financial bubbles: a study of co-explosivity in the cryptocurrency market. *Risks*, 8(2):34.
Rodrigo Aldecoa, Chiara Orsini, and Dmitri Krioukov. 2015. Hyperbolic graph generator. Computer Physics Communications, 196:492-496.
Arash Aloosh and Samuel Ouzan. 2020. The psychology of cryptocurrency prices. *Finance Research Letters*, 33:101192.
Leif BG Andersen. 2007. Efficient simulation of the he-ston stochastic volatility model. Available at SSRN 946405.
Yanuar Andrianto and Yoda Diputra. 2017. The effect of cryptocurrency on investment portfolio effectiveness. Journal of finance and accounting, 5(6):229-238.
James Angel. 2021. Gamestonk: What happened and what to do about it. Available at SSRN 3782195.
James J Angel and Douglas McCabe. 2013. Fairness in financial markets: The case of high frequency trading. Journal of Business Ethics, 112(4):585-595.
Lennart Ante. 2021. How elon Musk's twitter activity moves cryptocurrency markets. Available at SSRN 3778844.
Wei Bao, Jun Yue, and Yulei Rao. 2017. A deep learning framework for financial time series using stacked autoencoders and long-short term memory. *PloS one*, 12(7):e0180944.
Johannes Beck, Roberta Huang, David Lindner, Tian Guo, Zhang Ce, Dirk Helbing, and Nino Antulov-Fantulin. 2019. Sensing social media signals for cryptocurrency news. In Companion Proceedings of The 2019 World Wide Web Conference, pages 1051-1054.
Dan Bernhardt and Jianjun Miao. 2004. Informed trading when information becomes stale. The Journal of Finance, 59(1):339-390.
Robert Bloomfield and Maureen O'Hara. 1999. Market transparency: who wins and who loses? The Review of Financial Studies, 12(1):5-35.
Flamur Bunjaku, Olivera Gjorgieva-Trajkovska, and Emilija Miteva-Kacarski. 2017. Cryptocurrencies-advantages and disadvantages. Journal of Economics, 2(1):31-39.
Ines Chami, Zhitao Ying, Christopher Ré, and Jure Leskovec. 2019a. Hyperbolic graph convolutional neural networks. In Advances in Neural Information Processing Systems, pages 4869-4880.
Ines Chami, Zhitao Ying, Christopher Ré, and Jure Leskovec. 2019b. Hyperbolic graph convolutional neural networks. In Advances in Neural Information Processing Systems 32. Curran Associates, Inc.
Cathy Yi-Hsuan Chen and Christian M Hafner. 2019. Sentiment-induced bubbles in the cryptocurrency market. Journal of Risk and Financial Management, 12(2):53.
Alexander Chepurnoy, Vasily Kharin, and Dmitry Meshkov. 2019. A systematic approach to cryptocurrency fees. In *Financial Cryptography and Data Security*, pages 19-30. Springer Berlin Heidelberg.
Adrian Cheung, Eduardo Roca, and Jen-Je Su. 2015. Crypto-currency bubbles: an application of the phillips-shi-yu (2013) methodology on mt. gox bitcoin prices. Applied Economics, 47(23):2348-2358.
Usman W Chohan. 2021. Counter-hegemonic finance: The gamestop short squeeze. Available at SSRN.
David LEE Kuo Chuen, Li Guo, and Yu Wang. 2017. Cryptocurrency: A new investment opportunity? The Journal of Alternative Investments, 20(3):16-40.
Ricky Cooper, Michael Davis, Andrew Kumiega, and Ben Van Vliet. 2020. Ethics for automated financial markets. Handbook on Ethics in Finance, pages 1-18.
Ricky Cooper, Michael Davis, Ben Van Vliet, et al. 2016. The mysterious ethics of high-frequency trading. Business Ethics Quarterly, 26(1):1-22.
Shaen Corbet, Brian Lucey, and Larisa Yarovaya. 2018. Datestamping the bitcoin and ethereum bubbles. Finance Research Letters, 26:81-88.
J Bradford De Long, Andrei Shleifer, Lawrence H Summers, and Robert J Waldmann. 1990. Noise trader risk in financial markets. Journal of Political Economy, 98(4):703-738.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Bhuwan Dhingra, Christopher Shallue, Mohammad Norouzi, Andrew Dai, and George Dahl. 2018. Embedding text in hyperbolic spaces. In Proceedings of the Twelfth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-12), New Orleans, Louisiana, USA. Association for Computational Linguistics.
Xin Du and Kumiko Tanaka-Ishii. 2020. Stock embeddings acquired from news articles and price history, and an application to portfolio optimization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3353-3363, Online. Association for Computational Linguistics.
Franklin R Edwards. 1996. The new finance: regulation and financial stability. American Enterprise Institute.
Pedro Febrero. 2019. What's the difference between trading cryptocurrency and stocks?
Fuli Feng, Huimin Chen, Xiangnan He, Ji Ding, Maosong Sun, and Tat-Seng Chua. 2019a. Enhancing stock movement prediction with adversarial training. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 5843-5849. International Joint Conferences on Artificial Intelligence Organization.
Fuli Feng, Huimin Chen, Xiangnan He, Ji Ding, Maosong Sun, and Tat-Seng Chua. 2019b. Enhancing stock movement prediction with adversarial training. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 5843-5849. International Joint Conferences on Artificial Intelligence Organization.
Shanshan Feng, Lucas Vinh Tran, Gao Cong, Lisi Chen, Jing Li, and Fan Li. 2020. Hme: A hyperbolic metric embedding approach for next-poi recommendation. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '20, page 1429-1438, New York, NY, USA. Association for Computing Machinery.
Kenneth A Froot and Maurice Obstfeld. 1989. Intrinsic bubbles: The case of stock prices. Technical report, National Bureau of Economic Research.
John Fry and Eng-Tuck Cheah. 2016. Negative bubbles and shocks in cryptocurrency markets. International Review of Financial Analysis, 47:343-352.
Xavier Gabaix, Parameswaran Gopikrishnan, Vasiliki Plerou, and H Eugene Stanley. 2003. A theory of power-law distributions in financial market fluctuations. Nature.
Octavian Ganea, Gary Becigneul, and Thomas Hofmann. 2018a. Hyperbolic neural networks. In Advances in Neural Information Processing Systems.
Octavian Ganea, Gary Becigneul, and Thomas Hofmann. 2018b. Hyperbolic neural networks. In Advances in Neural Information Processing Systems.
Brown Gavin and Whittle Richard. 2019. More than 1,000 cryptocurrencies have already failed – here's what will affect successes in future.
David I Harvey, Stephen J Leybourne, Robert Sollis, and AM Robert Taylor. 2016. Tests for explosive financial bubbles in the presence of non-stationary volatility. Journal of Empirical Finance, 38:548-574.
Danqi Hu, Charles M Jones, Valerie Zhang, and Xiaoyan Zhang. 2021. The rise of reddit: How social media affects retail investors and short-sellers' roles in price discovery. Available at SSRN 3807655.
Ziniu Hu, Weiqing Liu, Jiang Bian, Xuanzhe Liu, and Tie-Yan Liu. 2018. Listening to chaotic whispers: A deep learning framework for news-oriented stock trend prediction. In Proceedings of the eleventh ACM international conference on web search and data mining, pages 261-269.
George F Hurlburt, Keith W Miller, and Jeffrey M Voas. 2009. An ethical analysis of automation, risk, and the financial crises of 2008. IT professional, 11(1):14-19.
Valentin Khrulkov, Leyla Mirvakhabova, Evgeniya Ustinova, Ivan Oseledets, and Victor Lempitsky. 2020a. Hyperbolic image embeddings. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Valentin Khrulkov, Leyla Mirvakhabova, Evgeniya Ustinova, Ivan Oseledets, and Victor Lempitsky. 2020b. Hyperbolic image embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Ritchie King. 2013. By reading this article, you're mining bitcoins.
Shimon Kogan, Dimitry Levin, Bryan R. Routledge, Jacob S. Sagi, and Noah A. Smith. 2009. Predicting risk from financial reports with regression. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 272-280, Boulder, Colorado. Association for Computational Linguistics.
Kazuhiro Kohara, Tsutomu Ishikawa, Yoshimi Fukuhara, and Yukihiro Nakamura. 1997. Stock price prediction using prior knowledge and neural networks. Intelligent Systems in Accounting, Finance & Management, 6(1).
Olivier Kraaijeveld and Johannes De Smedt. 2020. The predictive power of public twitter sentiment for forecasting cryptocurrency prices. Journal of International Financial Markets, Institutions and Money, 65:101188.
Peter M Krafft, Nicolas Della Penna, and Alex Sandy Pentland. 2018. An experimental study of cryptocurrency market dynamics. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pages 1-13.
Alok Kumar. 2009. Who gambles in the stock market? The Journal of Finance, 64(4):1889-1933.
Nikolaos Kyriazis, Stephanos Papadamou, and Shaen Corbet. 2020. A systematic review of the bubble dynamics of cryptocurrency prices. Research in International Business and Finance, 54:101254.